I've had chance to investigate further and what I'm seeing is:
1) Consistently very high CPU usage on the worker process - YAF is the only application running under this particular application pool. SO I think the earlier ahrefs bot may have been a non-issue
2) I've run DebugDiag 2 (Microsoft diagnostic tool) on the server and it is
i) Notifying "High CPU usage between dump files was detected on one or more threads."
ii) Showing the following in the Top 11 threads by CPU
Thread 4344 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In
tPtr, Int32)
Thread 6116 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In
tPtr, Int32)
Thread 4412 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus
ByRef)
Thread 6112 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,
SNI_Packet**, Int32
Thread 3452 - YAF.Core.dll!Unknown
Thread 116 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In
tPtr, Int32)
Thread 6108 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,
SNI_Packet**, Int32
Thread 5408 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In
tPtr, Int32)
Thread 4328 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus
ByRef)
Thread 6140 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,
SNI_Packet**, Int32
Thread 4500 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus
ByRef)
iv) In the top thread above, the top function stats are:
System_Data_ni!SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle
, IntPtr ByRef, Int32)
ServiceStack.OrmLite.dll!Unknown
YAF.Core.dll!Unknown
YAF.Web.dll!Unknown
YAF.dll!Unknown
App_Web_45hd2zr0.dll!Unknown
App_Web_sv0tltxt.dll!Unknown
YAF.Types.dll!Unknown
ServiceStack.OrmLite.OrmLiteResultsFilterExtensions.ConvertTo[[System.ValueTuple`2[[System.Int32, ms
corlib],[System.Int32, mscorlib]], mscorlib]](System.Data.IDbCommand, System.String)
ServiceStack.OrmLite.OrmLiteExecFilter.Exec[[System.ValueTuple`2[[System.Int32, mscorlib],[System.In
t32, mscorlib]], mscorlib]](System.Data.IDbConnection, System.Func`2alueTuple`2>
YAF.Core.Data.DbAccessBase.Execute[[System.ValueTuple`2[[System.Int32, mscorlib],[System.Int32, msco
rlib]], mscorlib]](System.Func`2>, System.Da
ta.IDbCommand, System.Data.IDbTransaction)
YAF.Core.Model.MessageRepositoryExtensions.FindUnread(YAF.Types.Interfaces.Data.IRepository`1pes.Models.Message>, Int32, System.Nullable`1, System.DateTime, Boolean)
YAF.Pages.Posts.GetFindMessageId(Boolean, Int32 ByRef)
YAF.Pages.Posts.BindData()
YAF.Pages.Posts.Page_Load(System.Object, System.EventArgs)
Any ideas or pointers on where to start looking in the source code? I'm thinking if I can find the function call which is causing the problems I may be able to simply comment out as an interim workaround.