Server down without a reason

hcl-bot · May 14, 2009, 2:52pm

The Domino server was working properly. Nothing new was added (agents, databases, tasks) but ther server begun crashing , the strange thing is that the service didn’t continue running, it just stopped without leaving any trace of the possible error that caused the crash.Notes log has no information about the error. The last line that appeared is the last task the server was doing before the crash.

I tried to find common tasks that the server was doing before the crashes, but i found nothing, it seemed to be a random case.

What could be happening?, what else can I do to check whether the server can register anything else before crashing?, Could it be something in the operating system or in domino server?

Thanks in advance.

hcl-bot · May 14, 2009, 7:48pm

Subject: Check for .NSD files

hcl-bot · May 14, 2009, 8:18pm

Subject: Can you post the FATAL THREAD section of the NSD

Hi

Can you post the FATAL THREAD section of the NSD files with the few lines under it

For example:

############################################################

FATAL THREAD 1/2 [ domdsm:107a8:67076]

FP=0x0012a34c, PC=0x60069a86, SP=0x0012a2f0, stksize=92

EAX=0x01a3088c, EBX=0x01a48650, ECX=0x0000bffe, EDX=0x01a325e6

ESI=0x00004000, EDI=0x0032ff78, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60069a86 nnotes._fdDelete@4+118 (4000,12a368,608770ef,4000)

@[ 2] 0x60069a0c nnotes._OSFileClose@4+12 (4000,12adc0,12a380,60872ea7)

@[ 3] 0x608770ef nnotes._sqloclose+15 (4000,12b59c,5010,0)

@[ 4] 0x60872ea7 nnotes._sqlpgclf+71 (12adc0,1aaa2cb,70,1aaa2f3)

@[ 5] 0x60885458 nnotes._sqlpgfdl+616 (12b9a4,1aaa2cb,41,12be44)

@[ 6] 0x60876a6f nnotes._sqlpgcdl+927 (12bf44,12b9a4,0,12be44).

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

hcl-bot · May 19, 2009, 8:58am

Subject: NSD data

These are the lines with the FATAL THREAD data.

############################################################

FATAL THREAD 1/4 [ nUpdate: 0ed8: 17cc]

FP=0x0012d490, PC=0x6390cf1c, SP=0x0012d484

stkbase=00130000, total stksize=77824, used stksize=11132

EAX=0x00000100, EBX=0x06c26c18, ECX=0x0012d90c, EDX=0x00000000

ESI=0x00000001, EDI=0x27650fff, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x6390cf1c NLSCCSTR.ccSTRnLenBytes@16+140 (27650fff,27650fff,0,12d90c)

@[ 2] 0x6392cd34 NLSCCSTR.ICU_ccSTRConvExt+372 (6c26c18,12d994,8,27650fff)

@[ 3] 0x63905a06 NLSCCSTR.ccSTRConvExt@28+134 (6c26c18,12d994,8,27650fff,0,0,0)

@[ 4] 0x6391bda2 NLSCCSTR.LNL_find_matching_substr@24+354 (12dac0,6509,12da63,3,0,0)

@[ 5] 0x63a02e39 NLSCCSTR.NLS_find_matching_substr@24+73 (12dac0,fffffffe,12da63,3,0,0)

@[ 6] 0x606320fd nnotes.GetNextRangePiece@20+717 (27646299,12dbe0,27646295,12dc18,0)

@[ 7] 0x606333aa nnotes.CvtTextToTIMEDATEPAIRExt@32+122 (0,0,0,0,0,0,0,0)

@[ 8] 0x60635b4f nnotes.ConvertTextToTIMEDATEPAIRZoneExt@32+79 (0,0,0,0,0,0,0,0)

@[ 9] 0x6053b03d nnotes.TextElement::ToTimePair+77 (12dc90,12dcbc,12dcd4,27642320,0,0,0,0)

@[10] 0x6055385c nnotes.AtTextToTime::Execute+220 (27642320,27642068,26ac6538,ffffffff,0,0,0,0)

@[11] 0x60046152 nnotes.AtFunctionNode::ComputeVariants+82 (26ac6ca0,27642068,26ac6538,26ac6538,0,0,0,0)

@[12] 0x60066b30 nnotes.AssignmentExpressionNode::ComputeVariants+80 (0,27644014,1,12dff4,0,0,0,0)

@[13] 0x60046392 nnotes.RootNode::ComputeVariants+98 (12e058,27644014,2763b014,291,0,0,0,0)

@[14] 0x60045f6d nnotes.Compute::Eval+141 (12e438,12e138,152a3dc,12dde0,0,0,0,0)

@[15] 0x6053489e nnotes.NSFComputeMainFormula2@36+142 (27644014,1,12e058,0,0,0,0,0,0)

@[16] 0x6007bd02 nnotes.NSFComputeMainFormula@28+50 (27644014,1,12e058,0,0,0,0)

@[17] 0x6006efac nnotes.SearchNote@20+2220 (12e54c,12e438,12e138,5fca0b4,0)

@[18] 0x60072e8c nnotes.DbForEachDocumentInBucket@24+1772 (5fc1000,152a498,0,152a3dc,0,0)

@[19] 0x60073b7b nnotes.iSearchFill@8+1323 (2c75,2be5)

@[20] 0x60073f0f nnotes.SearchFill@8+63 (2c75,2be5)

@[21] 0x6005b354 nnotes.QueueFill@4+52 (2c75)

@[22] 0x6005b291 nnotes.QueueGet@8+33 (2c75,12e690)

@[23] 0x6006d51e nnotes.GetNextBuffer@8+894 (2a0510c,60ecb48c)

@[24] 0x6004ea4e nnotes.UpdateCollection@8+4270 (0,0)

@[25] 0x60080152 nnotes.NIFUpdateCollectionNext@8+2306 (299bd0c,27208878)

@[26] 0x00403c06 nUpdate.UpdateViewCollection@20+1110 (4c4,12f76c,16f652,c000003,0)

@[27] 0x0040406a nUpdate.DesignEnumProc@28+410 (12f490,4c4,16f652,12f070,0,0,0)

@[28] 0x600b15b1 nnotes.DesignEnum2@32+609 (4c4,8,0,40,0,0,0,0)

@[29] 0x00404672 nUpdate.UpdateCollections@44+1490 (12f76c,c000003,0,0,0,0,0,0,0,0,0)

@[30] 0x004020e4 nUpdate.PerformRequest@20+404 (0,c000003,0,0,0)

@[31] 0x00402e7d nUpdate.Update+2365 (907384,4c5c3a43,7375746f,6d6f445c,0)

@[32] 0x00403522 nUpdate.AddInMain@12+370 (400000,1,907384)

@[33] 0x00408d2f nUpdate.NotesMain@8+47 (1,400000)

@[34] 0x00408e64 nUpdate.notes_main+212 (0,0)

@[35] 0x00408d56 nUpdate.main+22 (1,374ff8)

@[36] 0x00409487 nUpdate.mainCRTStartup+323 (0,0)

[37] 0x77e323e5 kernel32.IsProcessorFeaturePresent+158 (409344,0)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 1/4 [ nUpdate: 0ed8: 17cc]

FP=0012d490, PC=6390cf1c, SP=0012d484

stkbase=00130000, total stksize=77824, used stksize=11132

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

Disassembly of c. 10 instructions before and after faulting address 6390cf1c:



    6390cf02 8a07             mov     al,[edi]                        ds:27650fff=00

    6390cf04 884510           mov     [ebp+0x10],al                   ss:019f7cb6=00

    6390cf07 7423             jz      6390cf2c

    6390cf09 8da42400000000   lea     esp,[esp]                 ss:0012d484=00000000

    6390cf10 3c1f             cmp     al,0x1f

    6390cf12 7604             jbe     6390cf18

    6390cf14 3c80             cmp     al,0x80

    6390cf16 7204             jb      6390cf1c

    6390cf18 84c0             test    al,al

    6390cf1a 7521             jnz     6390cf3d

FAULT ->6390cf1c 8a4701 mov al,[edi+0x1] ds:28f1b825=00

    6390cf1f 83c701           add     edi,0x1

    6390cf22 83ee01           sub     esi,0x1

    6390cf25 85f6             test    esi,esi

    6390cf27 884510           mov     [ebp+0x10],al                   ss:019f7cb6=00

    6390cf2a 75e4             jnz     6390d010

    6390cf2c 2b7d08           sub     edi,[ebp+0x8]             ss:019f7cb6=00000000

    6390cf2f 8b4514           mov     eax,[ebp+0x14]            ss:019f7cb6=00000000

    6390cf32 8938             mov     [eax],edi                 ds:00000100=????????

    6390cf34 5f               pop     edi

    6390cf35 5e               pop     esi

    6390cf36 33c0             xor     eax,eax

hcl-bot · May 20, 2009, 6:43am

Subject: Crash or CPU spike when opening view via the user interface or accessing the view programmatically

Crash or CPU spike when opening view via the user interface or accessing the view programmatically

JYR

hcl-bot · May 15, 2009, 9:50am

Subject: could be…

…windows or domino at this point.

Did the windows server go down as well? Is there AV or anything like that on the hardware that might have decided Domino was a threat and taken it down? With windows, you should also be able to get some system logging that might give you hints about what was going on around that time…what tasks were taking up lots of CPU, etc.

For Domino: Are you running any 3rd-party products on the Domino server? If so, try disabling them and see if the problem goes away. set Debug_Router=1 in the notes.ini so you can capture any issues there in the notes log. When you have a chance, run updall against the databases to clean up any corruption.

Hope that helps!