Upgrade 7.02 FP1 HTTP Crash with Access Violation

HCL Domino Domino Forum
1

Anybody seen this???

After upgrading Domino Server from 7.01 to 7.02 FP1 on Win 2k server in preperation for the DST changes, my Domino server constantly crashes (2-5 times per day). I have a mixed environment with 6.5x to 7.01 databases.

Working with IBM support, they believe it is caused by http client either forwarding or replying to and email. However we’ve been unable to resolve.

This is a random occurance and have been unable to reproduce this failure.

Lited below is the NSD text from the 1st pass;

############################################################

FATAL THREAD 29/86 [ nhttp: 0788: 0894]

FP=0x39df7cec, PC=0x60008cd0, SP=0x39df7cec

stkbase=39e00000, total stksize=262144, used stksize=33556

EAX=0x00000000, EBX=0xffffffff, ECX=0x39df80f4, EDX=0x00000001

ESI=0x39df80f4, EDI=0x00000002, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60008cd0 nnotes._Cstrlen@4+16 (0)

@[ 2] 0x0049aacf ninotes.NCDfield::Init+31 (39df7f90)

@[ 3] 0x00572291 ninotes.HuDocNote::ShimmerConvertEmbeddedImagesToAttachment+593 (39b8cf7c)

@[ 4] 0x00572449 ninotes.HuDocNote::ShimmerConvertImagesToAttachment+57 (39b8cf7c)

@[ 5] 0x00572f9f ninotes.HuDocNote::SetFieldValuesPass2+191 (4d)

@[ 6] 0x005756c6 ninotes.HuDocNote::SetItemValues+38 (39b8cf7c)

@[ 7] 0x0055e7e9 ninotes.Haiku::ProcessEditAction+249 (39b92014)

@[ 8] 0x0055f4a0 ninotes.Haiku::HandleDominoCmd+576 (39b92014)

@[ 9] 0x00556d39 ninotes.Haiku::HandleCmd+793 (39b92014)

@[10] 0x00441a56 ninotes.CmdHandlerBase::PrivHandle+102 (39b92014)

@[11] 0x00440189 ninotes.CmdHandler::PrivHandle+121 (39b92014)

@[12] 0x004411dd ninotes.CmdHandler::Handler+221 (39b92014)

@[13] 0x004378da ninotes.Cmd::Execute+58 (3dd38e14)

@[14] 0x0047bace ninotes.$L171905+792 (3dd38e24)

@[15] 0x0047df7f ninotes._InotesHTTPProcessRequest+47 (3dd38e24)

@[16] 0x10015354 nhttpstack.HTInotesRequest::ProcessRequest+36 (3dd38b28)

@[17] 0x10010c0d nhttpstack.HTRequestExtContainer::ProcessRequest+941 (5)

@[18] 0x1001fdf0 nhttpstack.HTRequest::ProcessRequest+1904 (0)

@[19] 0x10025351 nhttpstack.HTSession::StartRequest+897 (389872de)

@[20] 0x1002c70f nhttpstack.HTWorkerThread::CheckForWork+399 (3)

@[21] 0x1002cc28 nhttpstack.HTWorkerThread::ThreadMain+88 (389872d2)

@[22] 0x60103d20 nnotes._ThreadWrapper@4+208 (0)

@[23] 0x7c57b388 KERNEL32.CreateProcessW+310

Invalid stack frame detected: Unable to read process memory for frame

2

Subject: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Did you have any hotfixes for 7.0.2?

3

Subject: Upgrade 7.02 FP1 HTTP Crash with Access Violation

I’ve had exactly the same issue. I opened a PMR and have been told that that it is SPR COLY6Y2J3Q. It caused by replying with history or forwarding a mail with an embedded graphic. I sent them a copy of the mail that caused the crash and developement are currently working on it.

I also had other nHTTP crashes caused according to Lotus as "This crash occurs when an anonymous user is added to the

usercache and the anonymous user is disabled at the port level."

I open another PMR for this, it’s SPR is NORK6UFP5W and a fix is available.

Overall, I found R7.x to be very buggy, I found that R 7.0 to R7.02 had a lot of problems with nSMTP bringing the whole server down. FP1 has introduced the nHTTP issues.

God only knows what R8 will bring.

4

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Thanks for the SPR numbers Gary. I emailed a contact at support and was given the first SPR number you mentioned. I’m glad that they know what is causing it. In my 7.0.2 issue I talk about below, it’s related to the UserCache - like the SPR NORK6UFP5W that you mentioned. I mentioned the UserCache change at my blog where I stated:

Lotus has re-worked the caching for HTTP 1.0 in Domino 7.0.2. So, if you have DWA users who have HTTP 1.0 enabled, keep an eye out for issues. To determine if this updated cache mechanism is the issue, you can enable a new setting in your Domino 7.0.2 server’s notes.ini:

iNotes_WA_PreferWebEngineCacheControl=1 - This will revert the HTTP 1.0 cache behavior back to that of Domino 7.0.1.

Want a 7.0.2 horror story? After going to 7.0.2, we had 8 crashes before noon one day! A link to my resolution to the issue is here. I had to actually downgrade my cluster server to 7.0.1 and force http to that server through DNS. It was not fun… This particular hotfix is being ported to 7.0.2 FP1 as we speak. This means that immediately when I move to 7.0.2 FP1 that I will have to be running a hotfix…

Here is some text from my post on the cause of my initial 7.0.2 issue:

the issue was with the HTTP UserCache and how it was working with LTPA tokens in an SSO environment. Since we use LTPA we were receiving the error. It’s been almost a week now and it’s working well. Whew…

I don’t really want to start a rant here, but the last major Domino server issues we had were when we went to 7.0 this time last year and revolved around the HTTP task. Thankfully this time the fix took a week instead of 2.5 months!!

Yes, when we went to 7.0.0 we also had some major HTTP issues if you go to the links in the last paragraph above.

Chris Whisonant

http://cwhisonant.blogspot.com

5

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Based on Chris’ response, I went to look at a PC that was beleived to have caused the Domino server to crash. I looked at the “Advaned” tab on IE6 and looked at the HTTP settings. I noticed that “HTTP 1.1 thru Proxy” was not selected. HTTP 1.1 was selected.

As we use a proxy to access the Internet, does this indicate that HTTP 1.1 isn’t being used and it defaults to 1.0?

Or am I barking up the wrong tree?

6

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Hi guy’s!

I can crash the server when ever i want :expressionless: In Webmail I open a mail doc i reply to it, i choose To: button and the Addressbook opens… Bang!! Server crashes…

I think the form s_namepicker in forms5.nsf database is causing this, at least i can read that from the nsd log file…

Now, can anyone of you guy’s try the same on your servers?

7

Subject: Bad template…

I found the reason for our server crashes, Inotes6.ntf template version 6.0.3 was for some strange reasons being used by some of the Webmail users and was crashing the server everytime they had to use the addressbook:-|

After i corrected this the server hasn’t crashed! One problem i ran into though, when reply’ing or forwarding webmail and there is an icelandic character (þ, ö, ð, ó, í á) in the display name it won’t send the reply or forward, the error: /inotes/proxy/?EditDocument&Form:=s_ValidationXml&Charset=UTF8… and i can’t see the rest of the error…

I have looked into to forms5.ntf forms6 and forms7 to see if there’s anything i could correct there, but i can’t see anything that relates to this error.

I have gone through Mime settings in Message Config doc. and i have gone through the server doc. it self and i cant see anything wrong there.

The reply and forward works fine when using Firefox and IE7 on Vista but not IE 6 on XP.

Anyone heard of this?

Regards,

Björgvin

8

Subject: RE: Bad template…

Thanks for the information on the bad template. That shouldn’t affect me - I just dont’ want to be bitten with another HTTP issue!

9

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Nice to know I’m not alone.

I was finally able to replicate the crash. After Identifying the last database accessed in the ASCII code of the NSD file, I was able to find files in the users database with similar timestamps. I tried to copy these files to other databases, but could not cause the server to fail.

Then I replicated the entire database locally and deleted non-essentail emails. I sent this database to IBM, and they too were able to reproduce the failure in their labs. I am waiting for response at this time.

I have 2 different “Fatal” type errors and now am exibiting a new “PANIC” error. All seem to be related to HTTP.

I do have FP1 installed.

10

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

No you are not alone… :expressionless:

I have just inserted iNotes_WA_PreferWebEngineCacheControl=1 in the servers notes.ini. Crossing my fingers!

Another thing, I saw in the servers Event Viewer that netevent.dll is reported as a cause for the crash, I’m having one of my Microsoft specialist investigating this now. Anyone of you seeing the same event?

11

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Hi Dan!

I have been trying to find the database that was last open when the server crashed, is this what i should be looking for?

        0ed2fd0c 00000000 2F6D6169 6C2F6D61 72677265 | .... /mai l/ma rgre |

        0ed2fd1c 7467692E 6E73662F 694E6F74 65732F50 | tgi. nsf/ iNot es/P |

        0ed2fd2c 726F7879 2F3F4564 6974446F 63756D65 | roxy /?Ed itDo cume |

        0ed2fd3c 6E742646 6F726D3D 735F4E61 6D655069 | nt&F orm= s_Na mePi |

        0ed2fd4c 636B6572 44697265 63746F72 794C6973 | cker Dire ctor yLis |

        0ed2fd5c 74000000 00000000 00000000 00000000 | t... .... .... ....

margretgi.nsf and the form is S_namepicker?

12

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Yes, Looks like margretgi.nsf

Try to match a file with the correct time stamp. When I can’t find the file, I usually find an empty file in the Drafs folder.

13

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Ok, thank’s for that Dan, i will try to find that. You still having crashes?

The server crashed last night even after i inserted the notes.ini parameter :frowning: So that didn’t work…

14

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

I found the document in question, infact i think i have found all the documents in all the databases that have crashed the server the last couple of day’s… Now is the BIG question, what can i do with it? My customer does not have a Support contract with IBM so…

Any ideas Dan or Chris?

15

Subject: SPR COLY6Y2J3Q

Lotus technical support asked me to add 3 parameters to my notes.ini and wait for the next crash. They then will want the offending mail file, NSDs and maybe the weblog. I’ve agreed to this but I’ll only do it the once. After that FP1 is coming out because the tech support guy says this SPR is caused by FP1.

Has anyone had issues uploading files to tech support? I frequently find that large files time out during upload. FTP would make a lot more sense than HTTP uploading especially for large NSDs and mail files.

16

Subject: RE: SPR COLY6Y2J3Q

IBM completed a Hotfix for me to address one of my 3 crash issues. I will apply over the weekend and post update.

17

Subject: Upgrade 7.02 FP1 HTTP Crash with Access Violation

I have this problem too. Domino 702 FixPack1 on a Windows 2003 server. Crashes sometimes 3-4 times an hour! Been like this since i upgraded last Tuesday. If i stop the http task it looks like the server is OK. I’m trying that for the second time now. I hope we get an response from IBM…

18

Subject: Upgrade 7.02 FP1 HTTP Crash with Access Violation

My Server was “Domino 7.0.2” on Redhat. It was so stable!!! I have more than 300 users.

Yesterday, I really don’t know why I tough of installing “Domino 7.0.2 FP1” in the first place since I had no problems… Oh maybe I just wanted to keep the server up-to-date!!!

I installed the FB1 yesterday…

Today, I got a call from a user that had an error when he tried to use “Domino Web Access” … The error was some thing like “Document entry not found in Index”.

I tried to login to the server using “Domino Admin Client” to check on the user’s quota …etc … but the server stopped responding.

I SSH’ed to the Linux server and found that domino was down!!!

I immediately started the server again… and issued the unix “top” command and I got surprised !!!

I saw the average load on the server to be (50 to 60) !!! Man that was really too MUCH !!!

my server load usually between (1.3 to 5)…

After few minutes the server crashed again!!! see the bellow message:

"Stack base = 0xaa7af52c, Stack size = 491240 bytes

Fatal Error signal = 0x0000000b PID/TID = 9340/-1434780752

2/11/2007 9:57:32 Running NSD

NSD is in progress …

Please attach the following files to your bug report along with the server log:

Log file : /local/notesdata/IBM_TECHNICAL_SUPPORT/nsd_Linux_lotus_2007_02_11@09_57_33.log

2/11/2007 10:00:13 Termination is in progress

2/11/2007 10:00:13 Terminating tasks

2/11/2007 10:00:27 Freeing resources

2/11/2007 10:00:27 Termination completed"

I immediately decided to uninstall the FB1 !!!

I did uninstall it and started the server again… I called the person who had the “Document entry not found in Index” error to see if he still have it … He told me everything is okay now and the error is gone !!!

I checked the server load average… It was back to normal…

Only then I was able to breathe normally again !!!

I got one lesson learned from this experience !!

Stay away from FIX PACKS if your server if is operating normally !!!

19

Subject: Upgrade 7.02 FP1 HTTP Crash with Access Violation

Hello. I have the same problem whit Domino Server 7.0.2.FP1, but i tried a lot of things like change INI parameters, move son document configuration parameters but nothing works, this morning i revert this version to 7.0.2. You spoke about some hot fix from Lotus but I coudn’t found. Can you please send me this file (702FP1HF68_W32.exe).

Thanks.

e-mail: ljimenez@sprc.com.co

20

Subject: RE: Upgrade 7.02 FP1 HTTP Crash with Access Violation

You need to open a support incident with IBM to get the patch. Hotfixes are not publicly available, and end-users can’t redistribute it.