I've been trying to figure out the same thing but it looks like someone forgot to put the instructions on the Wiki :(
I've given the Yaf membership provider the same settings as my existing site (app name, db connection, machinekey etc.) but when I try to login on the forum it just says "There's been a system error processing your request. Please contact the forum admin explaining what you were doing when it happened."