Helping another member with large forum --

Welcome Guest! To enable all features please Login or Register

squirrel
100% (Exalted)
YAF Leader Topic Starter

13 years ago

I am helping another YAF forum member with issues in a YAF application pool shutting down after 48-96 hours of operation. When the forum goes down, the ASP.NET error being given is the following:

System.Web.HttpException (0x80004005): The EventLogWebEventProvider provider failed to log an event with the error code 0x80070057.

In the person's Windows event logs, there are HUNDREDS of the following error:

Event code: 3008
Event message: A configuration error has occurred.
Event time: 1/5/2012 10:35:02 PM
Event time (UTC): 1/5/2012 3:35:02 PM
Event ID: 35ac7aa935ce432e9b0fab27883fbf39
Event sequence: 5915
Event occurrence: 2364
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/5/ROOT-1-129702502738385744
Trust level: Full
Application Virtual Path: /
Application Path: [[PROTECTED/REMOVED]]
Machine name: [[PROTECTED/REMOVED]]

Process information:
Process ID: 5500
Process name: w3wp.exe
Account name: [[PROTECTED/REMOVED]]

Exception information:
Exception type: ConfigurationErrorsException
Exception message: Thread was being aborted. ([[PROTECTED/REMOVED]]195\web.config line 87)
at System.Web.Security.Membership.Initialize()
at System.Web.Security.Membership.get_Provider()
at YAF.Classes.YafBoardSettings..ctor()
at YAF.Core.CurrentBoardSettings.<get_Instance>b__0()
at YAF.Core.HttpApplicationStateBaseExtensions.GetOrSet[T](HttpApplicationStateBase httpApplicationState, String key, Func`1 getValue)
at YAF.Core.CurrentBoardSettings.get_Instance()
at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass1`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.ComponentActivation.Activate(IEnumerable`1 parameters)
at Autofac.Core.Resolving.ComponentActivation.Execute(IEnumerable`1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Resolve(ISharingLifetimeScope activationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
at YAF.Types.Interfaces.IServiceLocatorExtensions.Get[TService](IServiceLocator serviceLocator)
at YAF.Core.UnloadSessionForumModule.UnloadSessionModule_Unload(Object sender, EventArgs e)
at System.EventHandler.Invoke(Object sender, EventArgs e)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Page.UnloadRecursive(Boolean dispose)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Thread was being aborted.
at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
at System.Web.Util.ReadWriteObjectLock.AcquireRead()
at System.Web.HttpApplicationStateLock.AcquireRead()
at System.Web.HttpApplicationState.Get(String name)
at System.Web.HttpApplicationStateWrapper.get_Item(String name)
at YAF.Providers.Membership.YafMembershipProvider.Initialize(String name, NameValueCollection config)
at System.Web.Configuration.ProvidersHelper.InstantiateProvider(ProviderSettings providerSettings, Type providerType)



Request information:
Request URL: http://forum.cadovn.com/default.aspx
Request path: /default.aspx
User host address: [[PROTECTED/REMOVED]]
User: [[PROTECTED/REMOVED]]
Is authenticated: True
Authentication Type: Forms
Thread account name: [[PROTECTED/REMOVED]]

Thread information:
Thread ID: 213
Thread account name: [[PROTECTED/REMOVED]]
Is impersonating: False
Stack trace: at System.Web.Security.Membership.Initialize()
at System.Web.Security.Membership.get_Provider()
at YAF.Classes.YafBoardSettings..ctor()
at YAF.Core.CurrentBoardSettings.<get_Instance>b__0()
at YAF.Core.HttpApplicationStateBaseExtensions.GetOrSet[T](HttpApplicationStateBase httpApplicationState, String key, Func`1 getValue)
at YAF.Core.CurrentBoardSettings.get_Instance()
at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass1`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
at Autofac.Core.Resolving.ComponentActivation.Activate(IEnumerable`1 parameters)
at Autofac.Core.Resolving.ComponentActivation.Execute(IEnumerable`1 parameters)
at Autofac.Core.Resolving.ResolveOperation.Resolve(ISharingLifetimeScope activationScope, IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(IComponentRegistration registration, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
at YAF.Types.Interfaces.IServiceLocatorExtensions.Get[TService](IServiceLocator serviceLocator)
at YAF.Core.UnloadSessionForumModule.UnloadSessionModule_Unload(Object sender, EventArgs e)
at System.EventHandler.Invoke(Object sender, EventArgs e)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Control.UnloadRecursive(Boolean dispose)
at System.Web.UI.Page.UnloadRecursive(Boolean dispose)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

What I have determined so far:

This forum has been progressively upgraded from versions back before 1.9.1.x, and the owner has used the proper web.config settings regarding using the upgrade version of the web.config(s). In the current web.config in place, line 87 (referenced above) has the following:

<add connectionStringName="yafnet" applicationName="YetAnotherForum" name="YafMembershipProvider" requiresUniqueEmail="false" type="YAF.Providers.Membership.YafMembershipProvider" hashHex="true" hashCase="upper" />

I ran the web.config through WinMerge against the recommended web.config for a fresh install to verify other differences and have not found anything that sticks out other than the configurations regarding the membership providers to maintain backwards compatability with the older YAF membership provider.

The forum in question is running on Windows 2008 Server R2 and ASP.NET v4. It has it's own dedicated pool, and by all appearances in browsing the site, seems to be a stock configuration of YAF, but it is running the latest packaged beta release as hosted on CodePlex.

If there is further diagnosis needed, please advise. I would like to help this member get to the bottom of this issue, as well as help the developers make sure there is not an issue with the newer versions against the older membership configurations under heavy traffic situations. The forum is a fairly high use system, with 300+ active users on average and over 3 million posts in the system.

Thank you for your time!

Edited by user 13 years ago | Reason: Not specified

If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend

Wanna join the discussion?! Login to your YAF.NET Support Forum forum account or Register a new forum account

Sponsor

bbobb
100% (Exalted)
YAF Developer

13 years ago

squirrel, thanks for your help to the member. Your huge help is greatly appreciated.

I can only suppose that it comes to thread safety or the thead pool is stuffed at some pont by multiplying threads.

I guess, only Jaben can help in the case. Thanks again for you huge help in diagnosis!

squirrel
100% (Exalted)
YAF Leader Topic Starter

13 years ago

squirrel, thanks for your help to the member. Your huge help is greatly appreciated.

I can only suppose that it comes to thread safety or the thead pool is stuffed at some pont by multiplying threads.

I guess, only Jaben can help in the case. Thanks again for you huge help in diagnosis!

Originally Posted by: bbobb

I'm thinking as a temporary measure to attempt to see if in the IIS configuration, we configure the pool in question to automatically recycle every 24 hours. Because the forum currently makes it between 48-96 hours before it dies, I'm wondering if forcing the recycle of the pool 'ahead of time' would help keep the site online for the time being until we are able to further diagnose the issue at hand.

A question I am wondering also: The old membership provider configuration did not require unique email addresses, but in the current releases, the setting is to require unique emails (per web.config) -- because the forum is in a language I cannot translate easily, I am unable to test the following theory of if this is an issue with registrations in the membership provider. I am curious of any issues known if we change the setting requiring unique emails would have any adverse effect ( going from the current of "false" to "true" ) or even if that would be an issue. The error logs so far point to an invalid configuration option in the web.config in the membership provider connection setting.

If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend

bbobb
100% (Exalted)
YAF Developer

13 years ago

Originally Posted by: squirrel

Good idea, but it desirable to know does it crush at rush-hour or not, first.

Originally Posted by: squirrel

Emails should be unique as Memebership requires unique emails. But I don't know how it was from beginning. It's not difficult to write an sql script to check emails in yaf_user and in Membership table usinf cursor and run it at night(his night).

Jaben
100% (Exalted)
YAF Developer

13 years ago

I'm going to say the IIS App is setup incorrectly with a "Queue Limit" set. It's hitting that queue limit very quickly and causing rapid recycling of the app process.

abcd
64.2% (Friendly)
YAF Lover

13 years ago

I'm going to say the IIS App is setup incorrectly with a "Queue Limit" set. It's hitting that queue limit very quickly and causing rapid recycling of the app process.

Originally Posted by: Jaben

Dear Jaben,

the default of Queue length of application pool IIS 7 is 1000, I just change it to 16000 and keep monitor on it.

Thanks & best regards,

squirrel
100% (Exalted)
YAF Leader Topic Starter

13 years ago

What's happening is whenever IIS 'automatically' recycles the application pool (such as 'scheduled' default of 29 hours), the forum starts responding with an error and the above error in the windows logs. If you manually recycle the pool it fixes itself --

If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend

squirrel
100% (Exalted)
YAF Leader Topic Starter

13 years ago

We solved 99% of the issues we were encountering - we found that the application was 'deadlocking' when trying to start up - and the worker process was abandoning the startup task before it was complete. We changed the executionTimeout to 600 (which is high honestly) and the errors stopped. I compiled a copy of the beta with all changesets up to 1/11/2012 and we upgraded to it. All is running quite well.

BUT:

The forum handles it's heaviest traffic periods during the weekends. This weekend we made it almost all the way through, then got shutdown with an error regarding the following:

EXCEPTION DETAILS

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

STACK TRACE


[InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.]
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +6371713
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6372046
   System.Data.SqlClient.SqlConnection.Open() +300
   YAF.Classes.Data.MsSqlDbConnectionManager.get_OpenDBConnection() +55
   YAF.Classes.Data.LegacyDb.forumpage_initdb(String& errorStr, Boolean debugging) +143
   YAF.Core.Services.StartupInitializeDb.RunService() +78
   YAF.Core.Services.BaseStartupService.Run() +40
   YAF.Utils.EnumerableExtensions.ForEach(IEnumerable`1 list, Action`1 action) +273
   YAF.Core.IHaveServiceLocatorExtensions.RunStartupServices(IHaveServiceLocator serviceLocator) +412
   System.Web.UI.Page.PerformPreInit() +49
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1716

The research I've done points to an issue with either SQL connections not being closed, or our SQL connection pool not being large enough. Weekend forum traffic ranges from 200 active users up to over 400+. Is there a chance we are just plain overloading the SQL connection pool or should I start looking into the source code for 'unclosed' connections? Considering what has gone into YAF development on the SQL side, I highly doubt there are unclosed connections, but wanted to see what the developers thought of this --

Currently, in our IIS configuration, our Queue Request Size is at 2000 requests, and my research shows that this queue does not relate to the error we have directly, as it is an SQL error more than an IIS error.

Edited by user 13 years ago | Reason: Not specified

If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend

bbobb
100% (Exalted)
YAF Developer

13 years ago

It's difficult to say from the top of my head what is the reason. I didn't dig into the connection singletons, as I use a custom code, but first I suspect a thread-safety i.e. it tries to open a connection opened by another thread. A monitor like this could be helpful.

private static readonly object m_lock = new object();

lock (m_lock)

{

connection.Open();

}

Important Information: