YAFLogo

Posted by: BWG - Wednesday, 13 December 2023 08:19:52
Hi, I'm using v3.1.13 CP and the CPU usage has suddenly surged. On investigation, it is SQL server which is consuming about 70% of the available resources, so I've had to take the forum offline temporarily. This has not happened before and usage at the time the problem started was light. The event log shows multiple entries like the one below. (I've blanked out the full url path).  Any ideas?  [code=markup]No posts were found for topic 54.36.149.20 **************/default.aspx?g=posts&m=838147Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)No posts were found for topic [                             topicId:16393, userId:1, updateViewCount:False,                              showDeleted:False, sincePostedDate:1/1/1903 12:00:00 AM, toPostedDate:12/13/2023 6:34:41 AM,                              pageIndex:0, pageSize:20, messagePosition:0]                          [/code]

Posted by: tha_watcha - Wednesday, 13 December 2023 08:40:28
I had similar issues with such useless bots. Thats why since YAF 3.1.15 its possible to block user agents like that AhrefsBot

Posted by: tha_watcha - Wednesday, 13 December 2023 09:18:15
Regarding to the event log entries those come from dead topics which contains no posts or only deleted posts. You should find and delete the topics under Admin > Maintenance > Restore & Delete Topics.

Posted by: BWG - Wednesday, 13 December 2023 09:52:42
I've taken a backup of the database and restored to my local development server. When trying Admin > Maintenance > Restore & Delete Topics I eventually get the following timed out message. There are about 80,000 pages because the database has existed a long time! Is it possible to just delete these topics from SQL sever directly? [code=markup] [HttpException (0x80004005): Request timed out.]    System.Web.HttpContext.InvokeCancellableCallback(WaitCallback callback, Object state) +372    System.Web.UI.<ProcessRequestAsync>d__554.MoveNext() +454    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +32    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +62    System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar) +65    System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +426    System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +172    System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +158[/code]  

Posted by: tha_watcha - Wednesday, 13 December 2023 10:04:05
The page loads all topics the paging is not server side paging. I didn't expect someone with so many topics and messages. I put this on my todo list to implement sql paging. You could manually delete all topics and messages from the Topic/Message Table with the Isdeleted flag set to true.

Posted by: BWG - Wednesday, 13 December 2023 10:23:17
If I run: [code=markup]Delete FROM [YAF].[dbo].[yaf_Message] where IsDeleted = 1[/code] The query fails because of a foreign key constraint: [code=markup]Msg 547, Level 16, State 0, Line 1 The DELETE statement conflicted with the SAME TABLE REFERENCE constraint "FK_yaf_Message_yaf_Message". The conflict occurred in database "YAF", table "dbo.yaf_Message", column 'ReplyTo'. The statement has been terminated.[/code]  

Posted by: BWG - Wednesday, 13 December 2023 12:01:21
I've had chance to investigate further and what I'm seeing is: 1) Consistently very high CPU usage on the worker process - YAF is the only application running under this particular application pool. SO I think the earlier ahrefs bot may have been a non-issue 2) I've run DebugDiag 2 (Microsoft diagnostic tool) on the server and it is i)  Notifying "High CPU usage between dump files was detected on one or more threads." ii) Showing the following in the Top 11 threads by CPU [code=markup]Thread 4344 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In tPtr, Int32)   Thread 6116 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In tPtr, Int32)   Thread 4412 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus  ByRef)   Thread 6112 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,  SNI_Packet**, Int32   Thread 3452 - YAF.Core.dll!Unknown   Thread 116 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In tPtr, Int32)   Thread 6108 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,  SNI_Packet**, Int32   Thread 5408 - System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, In tPtr, Int32)   Thread 4328 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus  ByRef)   Thread 6140 - [[InlinedCallFrame] (.SNIReadSyncOverAsync)] .SNIReadSyncOverAsync(SNI_ConnWrapper*,  SNI_Packet**, Int32   Thread 4500 - System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus  ByRef)   [/code] iv) In the top thread above, the top function stats are: [code=markup]System_Data_ni!SNINativeMethodWrapper.SNIReadSyncOverAsync(System.Runtime.InteropServices.SafeHandle , IntPtr ByRef, Int32)  ServiceStack.OrmLite.dll!Unknown  YAF.Core.dll!Unknown  YAF.Web.dll!Unknown  YAF.dll!Unknown  App_Web_45hd2zr0.dll!Unknown  App_Web_sv0tltxt.dll!Unknown  YAF.Types.dll!Unknown  ServiceStack.OrmLite.OrmLiteResultsFilterExtensions.ConvertTo[[System.ValueTuple`2[[System.Int32, ms corlib],[System.Int32, mscorlib]], mscorlib]](System.Data.IDbCommand, System.String)  ServiceStack.OrmLite.OrmLiteExecFilter.Exec[[System.ValueTuple`2[[System.Int32, mscorlib],[System.In t32, mscorlib]], mscorlib]](System.Data.IDbConnection, System.Func`2alueTuple`2>  YAF.Core.Data.DbAccessBase.Execute[[System.ValueTuple`2[[System.Int32, mscorlib],[System.Int32, msco rlib]], mscorlib]](System.Func`2>, System.Da ta.IDbCommand, System.Data.IDbTransaction)  YAF.Core.Model.MessageRepositoryExtensions.FindUnread(YAF.Types.Interfaces.Data.IRepository`1pes.Models.Message>, Int32, System.Nullable`1, System.DateTime, Boolean)  YAF.Pages.Posts.GetFindMessageId(Boolean, Int32 ByRef)  YAF.Pages.Posts.BindData()  YAF.Pages.Posts.Page_Load(System.Object, System.EventArgs)  [/code] Any ideas or pointers on where to start looking in the source code?  I'm thinking if I can find the function call which is causing the problems I may be able to simply comment out as an interim workaround.  

Posted by: tha_watcha - Thursday, 14 December 2023 06:55:29
Thank you for the info. Can you try to modify the following code in the file ..\Pages\Posts.ascx.cs replace the code block [code=csharp]   // find first unread message  var lastRead = !this.PageBoardContext.IsCrawler      ? this.Get<IReadTrackCurrentUser>().GetForumTopicRead(          this.PageBoardContext.PageForumID,          this.PageBoardContext.PageTopicID,          null,          null)      : DateTime.UtcNow;  var unreadFirst = this.GetRepository<Message>().FindUnread(      this.PageBoardContext.PageTopicID,      null,      lastRead,      showDeleted);  findMessageId = unreadFirst.MessageID;  messagePosition = unreadFirst.MessagePosition;  if (this.PageBoardContext.PageUser.Activity)  {      this.GetRepository<Activity>().UpdateNotification(this.PageBoardContext.PageUserID, findMessageId);  }[/code] with  [code=csharp]  if (!this.PageBoardContext.IsCrawler) {     // find first unread message     var lastRead = !this.PageBoardContext.IsCrawler         ? this.Get<IReadTrackCurrentUser>().GetForumTopicRead(             this.PageBoardContext.PageForumID,             this.PageBoardContext.PageTopicID,             null,             null)         : DateTime.UtcNow;     var unreadFirst = this.GetRepository<Message>().FindUnread(         this.PageBoardContext.PageTopicID,         null,         lastRead,         showDeleted);     findMessageId = unreadFirst.MessageID;     messagePosition = unreadFirst.MessagePosition;     if (this.PageBoardContext.PageUser.Activity)     {         this.GetRepository<Activity>().UpdateNotification(this.PageBoardContext.PageUserID, findMessageId);     } }[/code] this is would exclude the bots from that function. If this this not help comment out this code.  

Posted by: BWG - Thursday, 14 December 2023 13:08:01
Thanks - I modified the code, recompiled and deployed earlier this morning (I'm in the UK). Also, yesterday I deleted the dead topics using SQL and applied a maximum memory limit in SQL Server Management Studio.  The problems with high SQL / worker process usage have gone away now, but I'm still not sure of the underlying cause (nothing obvious in the server log files). There are two other websites running on the same server, but they run under their own application pools and CPU usage for both remained low, despite relatively high volumes of traffic.

Posted by: BWG - Saturday, 16 December 2023 11:55:18
The problem with the IIS worker process spiking returned yesterday. On investigation, the IIS logfiles are showing a lot of requests from Bytespider. The code provided earlier in this thread doesn't seem to stop Bytespider, but I have left it in. Therefore, I've put a URL rewrite blocking rule (via the IIS manager user interface) which checks the User-agent Header for Bytespider and aborts the request if found. As I'm still not sure what the underlying cause of the problem is, it is too early to say whether this will help or not. 

Posted by: tha_watcha - Saturday, 16 December 2023 14:45:41
Yes bytespider is one of the most aggressive bots, you can find dozens of topics on the internet about this bot and how to stop it Can you apply the following fix for the Crawler detection in the file yafsrc\YAF.Core\Helpers\UserAgentHelper.cs replace  [code=csharp] else         {             isSearchEngine = isCrawler || IsSearchEngineSpider(userAgent);         }[/code] with [code=csharp]              isSearchEngine = isCrawler || IsSearchEngineSpider(userAgent); [/code]

Posted by: BWG - Sunday, 17 December 2023 15:27:49
I can't find that code to replace. In the source I have there is this block: [code=markup]  else   {       // check if it's a search engine spider or an ignored UI string...       var san = SearchEngineSpiderName(userAgent);       if (san.IsSet())       {           browser = san;       }       isSearchEngine = isCrawler || san.IsSet();   }[/code]

Posted by: tha_watcha - Monday, 18 December 2023 10:14:02
You might want to upgrade to the latest version. I already applied the fixes to the version 3.2.2. Or you make the changes from this commit  [url]https://github.com/YAFNET/YAFNET/commit/ef5774b4881a73a3b857840e530eddcfe1b1cce6[/url]

Posted by: BWG - Friday, 12 January 2024 17:02:18
[quote=tha_watcha;73931]You might want to upgrade to the latest version. I already applied the fixes to the version 3.2.2. Or you make the changes from this commit  [url]https://github.com/YAFNET/YAFNET/commit/ef5774b4881a73a3b857840e530eddcfe1b1cce6[/url] [/quote] Thanks and sorry for the delay in acknowledging your suggestions. Christmas got in the way of things, but I have now had a chance to look at v3.2.2 on my development machine. One thing I noticed is CKEditor has been replaced (since v3.1.15 I think) with BBCode editor. My forum members aren't technical and probably wouldn't like seeing BBcode when they are writing their posts.  Do you know if it possible to have a WYSIWYG view like CKEditor? (I had a look in settings and couldn't find anything to change the default)

Posted by: tha_watcha - Saturday, 13 January 2024 10:20:04
[quote=BWG;73956][quote=tha_watcha;73931]You might want to upgrade to the latest version. I already applied the fixes to the version 3.2.2. Or you make the changes from this commit  [url]https://github.com/YAFNET/YAFNET/commit/ef5774b4881a73a3b857840e530eddcfe1b1cce6[/url] [/quote] Thanks and sorry for the delay in acknowledging your suggestions. Christmas got in the way of things, but I have now had a chance to look at v3.2.2 on my development machine. One thing I noticed is CKEditor has been replaced (since v3.1.15 I think) with BBCode editor. My forum members aren't technical and probably wouldn't like seeing BBcode when they are writing their posts.  Do you know if it possible to have a WYSIWYG view like CKEditor? (I had a look in settings and couldn't find anything to change the default)[/quote] CKEditor was removed because it causes too many problems, also CKEditor 4 is no longer been developed and its a paid product now. At the moment there is no option to switch to an WYSIWYG view. But im still open for suggestions for implementing an Editor which supports bbcodes, but most available one are no longer maintained or not really usable or causing similar issues like ckeditor.

Posted by: BWG - Saturday, 13 January 2024 12:17:43
[quote=tha_watcha;73957] CKEditor was removed because it causes too many problems, also CKEditor 4 is no longer been developed and its a paid product now. At the moment there is no option to switch to an WYSIWYG view. But im still open for suggestions for implementing an Editor which supports bbcodes, but most available one are no longer maintained or not really usable or causing similar issues like ckeditor.[/quote] I wonder if this would be an option? [url]https://www.sceditor.com/[/url] PS: I've not used it and don't have any connection to it.

Posted by: tha_watcha - Sunday, 14 January 2024 10:56:24
[quote=BWG;73958][quote=tha_watcha;73957] CKEditor was removed because it causes too many problems, also CKEditor 4 is no longer been developed and its a paid product now. At the moment there is no option to switch to an WYSIWYG view. But im still open for suggestions for implementing an Editor which supports bbcodes, but most available one are no longer maintained or not really usable or causing similar issues like ckeditor.[/quote] I wonder if this would be an option? [url]https://www.sceditor.com/[/url] PS: I've not used it and don't have any connection to it.[/quote] I also found this editor, it looks promising. But it needs some testing and requires a lot of work to implement all yaf bbcodes and features. So this would take some time to implement an alternative editor.

Posted by: tha_watcha - Thursday, 1 February 2024 08:24:10
I added the SCEditor to 3.2.3 Beta for testing

Posted by: BWG - Sunday, 4 February 2024 12:44:34
[quote=tha_watcha;73969]I added the SCEditor to 3.2.3 Beta for testing[/quote] That's great news. 👍I'll try and take a look during the coming week.

Posted by: Zero2Cool - Wednesday, 6 March 2024 00:59:31
[quote=tha_watcha;73969]I added the SCEditor to 3.2.3 Beta for testing[/quote] So far I am liking it a lot. It's a little tricky at first cuz I didn't know I had to click on the upper left portion to enter text, but super easy adjustment there. [attach]2348[/attach]

Posted by: tha_watcha - Wednesday, 6 March 2024 09:38:53
[quote=Zero2Cool;74024]Its a little tricky at first cuz I didnt know I had to click on the upper left portion to enter text[/quote] That issue should be fixed now.