YAFLogo

JP
  • JP
  • 100% (Exalted)
  • YAF Leader
12 years ago
My "workaround" is to restart the application via the admin interface, or to clear the app pool in IIS...

Seems to work for the first couple of messages sent by YAF, then the number will rise steadily until it gets a bit random depending on time of day, color of the sky, number of beers in my fridge or whatever...

Had 150 "new registration" mails today, but most were 5-10 copies while some were just 1-3...

I also think this issue should be of priority to our dear developer guys, since also new registrants will be puzzled and think the forum owner is an idiot by sending so many confirmation mails...

I don't mind being called an idiot, but would prefer being taken somewhat seriously as a forum owner... 😉


He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
francoz
  • francoz
  • 82.4% (Honored)
  • YAF Lover
12 years ago
bbobb
  • bbobb
  • 100% (Exalted)
  • YAF Developer
12 years ago
You'd better ask Jaben as the feature was written by him.

Really it's wriiten in a very intransparent manner.

JP
  • JP
  • 100% (Exalted)
  • YAF Leader
12 years ago
I have received a mail log from my server owner.

Email & host adress names are renamed, but I capitalized name parts so the flow can be followed.

This mail transaction resulted in 3 identical emails.



193.12.247.11   [16F0] 19:17:56 Connected, local IP=193.12.247.11:25
193.12.247.11   [16F0] 19:17:56 >>> 220-mail.MAILHOST.com ESMTP MAILHOST.com mailserver v10; Mon, 21 May 2012 19:17:56 +0200
193.12.247.11   [16F0] 19:17:56 <<< EHLO MAILHOST01
193.12.247.11   [16F0] 19:17:56 >>> 250-mail.MAILHOST.com Hello MAILHOST01 [193.12.247.11], pleased to meet you.
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<RECIPIENT@aol.com>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <RECIPIENT@aol.com>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself
193.12.247.11   [16F0] 19:17:56 <<< 861 bytes (overall data transfer speed=99652778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <RECIPIENT@aol.com> 1 856 00:00:00 OK FWM36056
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 856 bytes received in 00:00:00; Message id FWM36056 accepted for delivery
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<RECIPIENT@aol.com>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <RECIPIENT@aol.com>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself SYSTEM          [2750] 19:17:56 Client session Message id FWM36056 item 201205211917569813.tm$
193.12.247.11   [16F0] 19:17:56 <<< 861 bytes (overall data transfer speed=99652778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing SYSTEM          [2750] 19:17:56 Client session DNS query 'aol.com' 0 (4) [OK - 1] SYSTEM          [2750] 19:17:56 Client session Connecting to 'mailin-01.mx.aol.com'
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <RECIPIENT@aol.com> 1 856 00:00:00 OK FWM37656
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 856 bytes received in 00:00:00; Message id FWM37656 accepted for delivery
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<RECIPIENT@aol.com>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <RECIPIENT@aol.com>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself
193.12.247.11   [16F0] 19:17:56 <<< 861 bytes (overall data transfer speed=99652778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <RECIPIENT@aol.com> 1 856 00:00:00 OK FWM37756
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 856 bytes received in 00:00:00; Message id FWM37756 accepted for delivery
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <FORUM.site@address.name>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself
193.12.247.11   [16F0] 19:17:56 <<< 591 bytes (overall data transfer speed=68402778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <FORUM.site@address.name> 1 586 00:00:00 OK FWM43856
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 586 bytes received in 00:00:00; Message id FWM43856 accepted for delivery
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <FORUM.site@address.name>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself
193.12.247.11   [16F0] 19:17:56 <<< 591 bytes (overall data transfer speed=68402778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <FORUM.site@address.name> 1 586 00:00:00 OK FWM45456
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 586 bytes received in 00:00:00; Message id FWM45456 accepted for delivery
193.12.247.11   [16F0] 19:17:56 <<< MAIL FROM:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.0 <FORUM.site@address.name>... Sender ok
193.12.247.11   [16F0] 19:17:56 <<< RCPT TO:<FORUM.site@address.name>
193.12.247.11   [16F0] 19:17:56 >>> 250 2.1.5 <FORUM.site@address.name>... RECIPIENT ok; will forward
193.12.247.11   [16F0] 19:17:56 <<< DATA
193.12.247.11   [16F0] 19:17:56 >>> 354 Enter mail, end with "." on a line by itself
193.12.247.11   [16F0] 19:17:56 <<< 591 bytes (overall data transfer speed=68402778 B/s)
193.12.247.11   [16F0] 19:17:56 Start of mail processing
193.12.247.11   [16F0] 19:17:56 *** <FORUM.site@address.name> <FORUM.site@address.name> 1 586 00:00:00 OK FWM45556
193.12.247.11   [16F0] 19:17:56 >>> 250 2.6.0 586 bytes received in 00:00:00; Message id FWM45556 accepted for delivery
205.188.59.194  [2750] 19:17:56 Client session Connected, local IP= 193.12.247.11:
193.12.247.11   [16F0] 19:17:56 <<< QUIT
193.12.247.11   [16F0] 19:17:56 >>> 221 2.0.0 mail.MAILHOST.com closing connection
193.12.247.11   [16F0] 19:17:56 Disconnected

He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
daveburke
12 years ago
I happened to be working in YafSendMailThreaded.cs this afternoon. I was pushing it pretty hard and experienced my first duplicate emails. I was studying bad email address handling by the YAF SendMail task. Very interesting, like bbobb said.

One thing I observed was when multiple emails occurred the mail records seemed to be entering yaf_mail with 1 SendTries attempt (rather than 0.) That shouldn't be possible if you look at the yaf_mail_create sql proc, but perhaps somehow simultaneous calls are made to LegacyDb.mail_create()? Multiple threads? I can't say. Another observation when the duplicates occurred was messages were leaving the yaf_mail table one at a time over several minutes rather than many at a time. I restarted YAF. Processing returned to normal and the dupes were gone.

Sorry I only have observations to report. One thing I did do which might be helpful was modify the yaf_mail_delete SQL proc so that the users of any mail items with two SendTries are placed in a "do-not-mail" lookup table to eliminate possible bad email address processing issues. As for the "do-not-mail" logic, I'm actually sending records to yaf_mail from outside of YAF, which is where I'm checking the do-not-mail status and not sending those to Yaf_mail. It was determined on this thread that bad addresses weren't the culprit in the duplicate emails, but after looking into bad email handling, if nothing else it will reduce processing. And the less a background task has to do is always better.

Thanks,

Dave

squirrel
12 years ago

I happened to be working in YafSendMailThreaded.cs this afternoon. I was pushing it pretty hard and experienced my first duplicate emails. I was studying bad email address handling by the YAF SendMail task. Very interesting, like bbobb said.

One thing I observed was when multiple emails occurred the mail records seemed to be entering yaf_mail with 1 SendTries attempt (rather than 0.) That shouldn't be possible if you look at the yaf_mail_create sql proc, but perhaps somehow simultaneous calls are made to LegacyDb.mail_create()? Multiple threads? I can't say. Another observation when the duplicates occurred was messages were leaving the yaf_mail table one at a time over several minutes rather than many at a time. I restarted YAF. Processing returned to normal and the dupes were gone.

Sorry I only have observations to report. One thing I did do which might be helpful was modify the yaf_mail_delete SQL proc so that the users of any mail items with two SendTries are placed in a "do-not-mail" lookup table to eliminate possible bad email address processing issues. As for the "do-not-mail" logic, I'm actually sending records to yaf_mail from outside of YAF, which is where I'm checking the do-not-mail status and not sending those to Yaf_mail. It was determined on this thread that bad addresses weren't the culprit in the duplicate emails, but after looking into bad email handling, if nothing else it will reduce processing. And the less a background task has to do is always better.

Thanks,

Dave

Originally Posted by: daveburke 

My theory with the bad emails was when YAF got into a process of dealing with many emails -- long running task, that it wasn't completing and those that were stuck were getting resent and resent - like you noted -- you placed the task under heavy usage before you saw the fluke. Since we moved our site to a new server, we get no duplicate emails of any sort. Running on a Win2k8 box, 8 core, 4gb ram hosting around 200 sites, sql2008 r2, etc. Our public mailserver is MailEnable 5, but YAF sends it's outgoing mail to the local IIS SMTP server and it plays middleman to the public mailserver. We used to use our public mailserver directly and did have plenty of issues even though it is hosted on same machine - had many duplicate email issues. Since putting IIS SMTP in the middle and moving to win2k8 the issue has disappeared for us.

Now if I que up a message to get sent to all the users on the site and there are bad emails in there, I guarantee I'll get stuck in a 'duplicate email' issue again until I go in there and clean out the bad email addresses -- then it'll go away.

Keep in mind, this is all on an older codebase than is current. I only offer this information as comparison. We haven't updated to current version just yet -- I've had ton of other projects on the table.


If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend 
Thantis
  • Thantis
  • 81.8% (Honored)
  • YAF Commander
12 years ago
Well, I am not sure that load issues are the root of the problem, or atleast not the only problem. The forum I am experiencing this on has 13 users 🙂
bbobb
  • bbobb
  • 100% (Exalted)
  • YAF Developer
12 years ago

Well, I am not sure that load issues are the root of the problem, or atleast not the only problem. The forum I am experiencing this on has 13 users :)

Originally Posted by: Thantis 

It's not a load issue. It has problems with logic.At first glance it launches 2 threads instead of 1.

Morover there can be more threads launched if something in sending fails. And the whole thing is not thread-safe, and there are possible double enumerations. Due to the complexity it's impossible to say what exactly happens.

JP
  • JP
  • 100% (Exalted)
  • YAF Leader
12 years ago
Clearly not a load issue, but rather some kind of issue of sending mails in parallel instead of in a strict sequential order in the mailout-queue... Mail is not time-critical, and should be thread safe in a single thread just reading the mail queue and send the mail once and mark it as sent. Should not have any non-RFC workarounds...

Just wonder why it did not happen in 1.9.4 but started with 1.9.5.x and beyond.... Was there any clever code changes in the mailing module between those versions?


He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
Jaben
  • Jaben
  • 100% (Exalted)
  • YAF Developer
12 years ago
If you're having trouble with duplicate emails, please replace the mail_list SPROC with the following SQL:


EDIT: doesn't fix the issue.

Let me know if that fixes the issues.

Jaben
  • Jaben
  • 100% (Exalted)
  • YAF Developer
12 years ago
Never mind about the SQL recently posted. I have a better solution forthcoming.
JP
  • JP
  • 100% (Exalted)
  • YAF Leader
12 years ago
Thanks for looking into it Jaben, rather annoying problem for many of us with 1.9.5 or higher... 👍

All I can say is that myself & members on my site have been quite annoyed by this for a while... I constantly get complaints from members receiving multiple emails, while I myself have grown accustomed to it and see it as a "cosmetic" annoyance.... 😉


He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Old Chinese Proverb]
Jaben
  • Jaben
  • 100% (Exalted)
  • YAF Developer
12 years ago
Please run these SQL changes (they can be run in the Admin->Run SQL Query area).

If you have v1.9.6 or greater, you need to run the following SQL:


ALTER PROCEDURE [{databaseOwner}].[{objectQualifier}mail_list]
(
	@ProcessID int,
	@UTCTIMESTAMP datetime
)
AS
BEGIN
	BEGIN TRANSACTION TRANSUPDATEMAIL
		UPDATE [{databaseOwner}].[{objectQualifier}Mail]
		SET 
			ProcessID = NULL
		WHERE
			ProcessID IS NOT NULL AND SendAttempt > @UTCTIMESTAMP

		UPDATE [{databaseOwner}].[{objectQualifier}Mail]
		SET 
			SendTries = SendTries + 1,
			SendAttempt = DATEADD(n,5,@UTCTIMESTAMP),
			ProcessID = @ProcessID
		WHERE
			MailID IN (SELECT TOP 10 MailID FROM [{databaseOwner}].[{objectQualifier}Mail] WHERE SendAttempt < @UTCTIMESTAMP OR SendAttempt IS NULL ORDER BY SendAttempt, Created)
	COMMIT TRANSACTION TRANSUPDATEMAIL

	-- now select all mail reserved for this process...
	SELECT TOP 10 * FROM [{databaseOwner}].[{objectQualifier}Mail] WHERE ProcessID = @ProcessID ORDER BY SendAttempt, Created desc
END

For v1.9.5.x, please use the following SQL:


ALTER PROCEDURE [{databaseOwner}].[{objectQualifier}mail_list]
(
	@ProcessID int
)
AS
BEGIN
	DECLARE @UTCTIMESTAMP datetime
	
	SET @UTCTIMESTAMP = GETUTCDATE()
	
	BEGIN TRANSACTION TRANSUPDATEMAIL
		UPDATE [{databaseOwner}].[{objectQualifier}Mail]
		SET 
			ProcessID = NULL
		WHERE
			ProcessID IS NOT NULL AND SendAttempt > @UTCTIMESTAMP

		UPDATE [{databaseOwner}].[{objectQualifier}Mail]
		SET 
			SendTries = SendTries + 1,
			SendAttempt = DATEADD(n,5,@UTCTIMESTAMP),
			ProcessID = @ProcessID
		WHERE
			MailID IN (SELECT TOP 10 MailID FROM [{databaseOwner}].[{objectQualifier}Mail] WHERE SendAttempt < @UTCTIMESTAMP OR SendAttempt IS NULL ORDER BY SendAttempt, Created)
	COMMIT TRANSACTION TRANSUPDATEMAIL

	-- now select all mail reserved for this process...
	SELECT TOP 10 * FROM [{databaseOwner}].[{objectQualifier}Mail] WHERE ProcessID = @ProcessID ORDER BY SendAttempt, Created desc	
END

Please note that v1.9.6 has YAF.NET code improvements that do help deal with these issues.

I've put this code into production here on this forum and have committed it to the latest SQL in codeplex.

Let me know how it goes with your forum.s

squirrel
12 years ago
I've got an older 1.9.5 forum still running -- I used to have the issue but don't anymore since we migrated servers. We're moving to 1.9.6.1 soon though and it'll have that in there so will be able to report back any issues --
If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend 
daveburke
12 years ago
Jaben,

You answered the question I was going to ask, that is, if you applied the updates to the YAF Forums. Because your post was the first time that I can remember when I didn't get multiple notifications! :)

Thanks much,

Dave