Firstly, the root setting doesn't work at all. Despite specifying it, all of the Urls still point to ~/default.aspx rather than ~/forum/default.aspx.

<appSettings>
    <add key="YAF.Root" value="~/forum" />
    <add key="YAF.BaseUrl" value="~/forum" />  
</appSettings>

Edit

Consider updating your documentation so that it's actually accurate. In order to host YAF in a subdirectory the following appsettings must be set accordingly:

<add key="YAF.FileRoot" value="~/forum/" />
<add key="YAF.AppRoot" value="~/forum" />
<add key="YAF.BaseUrl" value="~/forum" />

Setting YAF.Root has no effect on the application whatsoever.