If you have YAF installed to a folder on your hosting called /forum and that folder is configured as an IIS Application in your hosting control panel, and you want it to run as a subdomain, this should work:
<add key="YAF.BaseUrlMask" value="http://forum.xxxxxx.com" />
<add key="YAF.ServerFileRoot" value="~/" />
<add key="YAF.ClientFileRoot" value="~/" />
<add key="YAF.AppRoot" value="~/" />
<add key="YAF.ForceScriptName" value="default.aspx" />
The ServerFileRoot and Client File Root are relative to the "IIS Virtual Application" root that you created when you made the /forum folder an IIS Virtual Application. This is the method where everything YAF (including the /bin folder in the distribution) runs in it's own folder completely and totally isolated from the rest of your sites. By setting the YAF.BaseUrlMask value above, you set YAF to properly set links to the subdomain name.
Backup your app.config, then find and update above values - make sure they are not commented out - then restart YAF application. (Adapted from Jaben's post about how the support forum is configured to run on subdomain - he has YAF and other apps mixed, so ServerFileRoot and ClientFileRoot have to point directly to YAF files)
If you can't find it using the forum search, try my signature link -- searches this site using Google:
Google is my Friend