AdminP is causing Domino server to Crash

HCL Domino Domino Forum
1

Hi everyone,

I am a new employee that has been assigned to work on Lotus Domino.

I have been learning(on my own) about Domino 7 for the last 3 months.

The Domino server we have is version 7. And for the past week, every time I type ‘tell adminp process all’ the Domino server crashes. I don’t know why it is doing this… I have set up NSD logging, but I can’t read much from it that tells me what is making this happen. How can I try to resolve this Domino crash that occurs every time I enter the command ‘tell adminp process all’ ?

2

Subject: AdminP is causing Domino server to Crash

Hi

From the knowledge base

Can you post the FATAL THREAD section of the NSD files with the few lines under it

For example:

############################################################

FATAL THREAD 1/2 [ domdsm:107a8:67076]

FP=0x0012a34c, PC=0x60069a86, SP=0x0012a2f0, stksize=92

EAX=0x01a3088c, EBX=0x01a48650, ECX=0x0000bffe, EDX=0x01a325e6

ESI=0x00004000, EDI=0x0032ff78, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60069a86 nnotes._fdDelete@4+118 (4000,12a368,608770ef,4000)

@[ 2] 0x60069a0c nnotes._OSFileClose@4+12 (4000,12adc0,12a380,60872ea7)

@[ 3] 0x608770ef nnotes._sqloclose+15 (4000,12b59c,5010,0)

@[ 4] 0x60872ea7 nnotes._sqlpgclf+71 (12adc0,1aaa2cb,70,1aaa2f3)

@[ 5] 0x60885458 nnotes._sqlpgfdl+616 (12b9a4,1aaa2cb,41,12be44)

@[ 6] 0x60876a6f nnotes._sqlpgcdl+927 (12bf44,12b9a4,0,12be44).

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

or call IBM support.

JYR

3

Subject: RE: AdminP is causing Domino server to Crash

Hi,Exert of the NSD log generated when Domino crashed. I found 3 ‘fatal’ threads.

############################################################

FATAL THREAD 4/9 [ nadminp: 688: 544]

FP=0x0755ea5c, PC=0x60008af0, SP=0x0755ea5c, stksize=0

EAX=0x7b273d5d, EBX=0x05fee3ae, ECX=0x0755ef5e, EDX=0x7b273d5e

ESI=0x60adfdec, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60008af0 nnotes._Cstrlen@4+16 (7b273d5d)

@[ 2] 0x603097e7 nnotes._AdminpModifyItem@32+1607 (7d3a5f2a,247c2424,3d262e2d,225f2c7b,755e688,3,1,7bc)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 4/9 [ nadminp: 688: 544]

FP=0755ea5c, PC=60008af0, SP=0755ea5c, stksize=0

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

---------- Top of the Stack ----------

   # 0755ea5c  0755eec0 603097e7 7b273d5d 00ed1e34  |..U...0`]='{4...|

   # 0755ea6c  00000000 00000000 0755eba8 60304ac7  |..........U..J0`|

   # 0755ea7c  0755edd0 0755ea94 00000100 00ed0798  |..U...U.........|

   # 0755ea8c  00000001 01040000 513d4e43 38393541  |........CN=QA598|

   # 0755ea9c  2d585345 37444e4c 3d4f2f30 614c4151  |ESX-LND70/O=QALa|

   # 0755eaac  00007362 00000000 00000000 00000000  |bs..............|

   # 0755eabc  00000000 00000000 00000000 00000000  |................|

   # 0755eacc  00000000 00000000 00000000 00000000  |................|

   # 0755eadc  00000000 00000000 00000000 00000000  |................|

############################################################

PASS 3 : FATAL THREAD with PARAMETER DATA 4/9 [ nadminp: 688: 544]

############################################################

@[ 1] 0x60008af0 nnotes._Cstrlen@4+16 (7b273d5d)

@[ 2] 0x603097e7 nnotes._AdminpModifyItem@32+1607 (7d3a5f2a,247c2424,3d262e2d,225f2c7b,755e688,3,1,7bc)

   Parameter 5:

   # 0755e688  0755e774 0755e790 0755e6b4 7c82eeb2  |t.U...U...U....||

   # 0755e698  0755e774 0755ffdc 0755e790 0755e750  |t.U...U...U.P.U.|

############################################################

thread 5/9: [ nadminp: 688: 2740]

FP=0765c8cc, PC=7c82ed54, SP=0765c864, stkbase=07560000, stksize=262144

############################################################

@[ 1] 0x7c82ed54 ntdll.KiFastSystemCallRet+0 (a,0,765c910,6073ebcd)

@[ 2] 0x77e424b7 KERNEL32.Sleep+15 (a,5697e4c,56b2e4c,765cbe0)

@[ 3] 0x6073ebcd nnotes._DbAddToDbCache@12+925 (56b2e4c,1,2dc995e)

@[ 4] 0x600350bb nnotes._DbClose@20+3675 (1,0,0,56b3480,0)

@[ 5] 0x60036a20 nnotes._NSFDbCloseExtended3@16+96 (1b6,0,0,1)

@[ 6] 0x600369b6 nnotes._NSFDbCloseExtended2@12+22 (1b6,0,0)

@[ 7] 0x60036992 nnotes._NSFDbCloseExtended@8+18 (1b6,0)

@[ 8] 0x6003696e nnotes._NSFDbClose@4+14 (1b6)

@[ 9] 0x6030d2e9 nnotes._AdminpResidesOnAdminServer@8+761 (765d9c8,765d370)

@[10] 0x6030d348 nnotes._AdminpCheckForReadersAuthors@8+56 (765d9c8,765d398)

@[11] 0x6070df94 nnotes._DbDirGetProperties@12+180 (765d9c8,2,765dac8)

@[12] 0x60731d3c nnotes._IsNotefile@92+428 (765db3c,126f8c4,0,4,0,1,0,0,0,0,0,0,0,0,0,0,2adf254,765d3a8,0,2adf4c4,4663a5,2adf648,8c59954)

@[13] 0x607324e3 nnotes._DbDirFaultBucket@24+1043 (5690e4c,0,1522e08,1522f14,0,1)

@[14] 0x6082a072 nnotes._DbFaultBucket@28+50 (5690e4c,0,0,1522e08,0,1,0)

@[15] 0x600732d8 nnotes._DbForEachDocumentInBucket@24+2264 (765eb04,1522ec4,0,1522e08,0,1)

@[16] 0x60073d44 nnotes._iSearchFill@8+1300 (2bd9,2bdd)

@[17] 0x6007403f nnotes._SearchFill@8+63 (2bd9,2bdd)

@[18] 0x6005c2e8 nnotes._QueueFill@4+40 (2bd9)

@[19] 0x6005c256 nnotes._QueueGet@8+22 (2bd9,765f34c)

@[20] 0x6030dc67 nnotes._AdminpProcessDbCtx@32+247 (27,6,765f3f4,5fe6cf0,0,1,0,0)

@[21] 0x6030ee8c nnotes._AdminpProcessDatabases@28+108 (6,765f3f4,5fe6cf0,765f3fc,0,1,0)

@[22] 0x0045dd2e nAdminP.ModifyReadersAuthors::ProcessRequests+78 (3334c14,5fe2f30,0,3334c14)

@[23] 0x004613c3 nAdminP.BatchRequest::DoProcessRequest+499 (1a08ba,1b1,0,765fd1c)

@[24] 0x00422ee2 nAdminP._AdminpProcessNewRequest@44+4434 (1b1,5fe2f30,18,2b69,0,1,0,0,0,0,0)

@[25] 0x00426364 nAdminP._AdminpRequestAndResponse@40+820 (1b1,5fe32c0,1a0792,2b63,0,1,0,0,0,0)

@[26] 0x0040257c nAdminP._EntryThread@4+908 (0)

@[27] 0x60104fc0 nnotes._ThreadWrapper@4+208 (0)

@[28] 0x77e66063 KERNEL32.GetModuleFileNameA+235 (0,0,0,0)

4

Subject: RE: AdminP is causing Domino server to Crash

Hi,

nothing useful on the WEB or in the knowledge base,

i think you will have to call IBM Support or try to find which admin request is pending,

JYR

5

Subject: RE: AdminP is causing Domino server to Crash

Thank you very much for your prompt help.I appreciate it. I will look into it more.

Regards,

Marco

6

Subject: RE: AdminP is causing Domino server to Crash

I would advise you to not use 'tell adminp process all". It starts all of the processes, beginning with the oldest ones. Use the following commands instead:

Tell adminp process new

Tell adminp process daily

Tell adminp process delayed. (This is the hardest on your processor, so never do this during the workday)

For more details on adminp, see these resources:

Knowledge Collection — The Administration Process in Domino 6.0x and 6.5x - www-1.ibm.com/support/docview.wss?uid=swg21213224

Frequently Asked Questions — The Administration Process (AdminP)

www-1.ibm.com/support/docview.wss?rs=899&uid= swg21212760

All About AdminP, Parts 1 and 2

7

Subject: RE: AdminP is causing Domino server to Crash

Thank you Susan. I will try that today.Also, Our Domino server crashes every night just after 12:00am (midnight). I think it crashes because of an agent. We have agents scheduled to run at 12:00am. The server crash is very consistent. It goes down very night.

How can I workout which agent(if indeed t is) is causing the server crash? I do have the NSD log from the crash too.

Thank you.

8

Subject: RE: AdminP is causing Domino server to Crash

Susan is the mistress of AdminP :slight_smile:

For your other crash, post the FATAL THREAD section

Thank you

JYR

9

Subject: RE: AdminP is causing Domino server to Crash

Here is the NSD log fatal threads for the crash that occurs every night

FATAL THREAD 5/9 [ nadminp: 3428: 4076]

FP=0x08e5ea5c, PC=0x60008af0, SP=0x08e5ea5c, stksize=0

EAX=0x7b273d5d, EBX=0x04af269a, ECX=0x08e5ef5e, EDX=0x7b273d5e

ESI=0x60adfdec, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60008af0 nnotes._Cstrlen@4+16 (7b273d5d)

@[ 2] 0x603097e7 nnotes._AdminpModifyItem@32+1607 (7d3a5f2a,247c2424,3d262e2d,225f2c7b,8e5e688,3,1,79c)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 5/9 [ nadminp: 3428: 4076]

FP=08e5ea5c, PC=60008af0, SP=08e5ea5c, stksize=0

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

---------- Top of the Stack ----------

   # 08e5ea5c  08e5eec0 603097e7 7b273d5d 00e9fd60  |......0`]='{`...|

   # 08e5ea6c  00000000 00000000 08e5eba8 60304ac7  |.............J0`|

   # 08e5ea7c  08e5edd0 08e5ea94 00000100 04ae30ec  |.............0..|

   # 08e5ea8c  00000001 01040000 513d4e43 38393541  |........CN=QA598|

   # 08e5ea9c  2d585345 37444e4c 3d4f2f30 614c4151  |ESX-LND70/O=QALa|

   # 08e5eaac  00007362 00000000 00000000 00000000  |bs..............|

   # 08e5eabc  00000000 00000000 00000000 00000000  |................|

   # 08e5eacc  00000000 00000000 00000000 00000000  |................|

   # 08e5eadc  00000000 00000000 00000000 00000000  |................|

   # 08e5eaec  00000000 00000000 00000000 00000000  |................|

   # 08e5eafc  00000000 00000000 00000000 00000000  |................|

   # 08e5eb0c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb1c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb2c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb3c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb4c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb5c  00000000 00000000 00000000 00000000  |................|

   # 08e5eb6c  00000000 08e5edd0 0000001a 60005cf4  |.............\.`|

   # 08e5eb7c  08e5edd0 04ae337c 0000001a 08e5eba8  |....|3..........|

   # 08e5eb8c  60006f17 08e5edd0 04ae337c 0000001a  |.o.`....|3......|

   # 08e5eb9c  04ae337c 00000001 01040000 08e5eed4  ||3..............|

   # 08e5ebac  60304b93 08e5edd0 04ae337c 0000001a  |.K0`....|3......|

   # 08e5ebbc  00000000 08e5f3bc 00000000 73726550  |............Pers|

   # 08e5ebcc  00006e6f 11840004 6a360000 805e0001  |on........6j..^.|

   # 08e5ebdc  8b58c363 234aa377 739b0032 0002ca25  |c.X.w.J#2..s%...|

   # 08e5ebec  282d0000 739b0032 002eca25 282e0001  |..-(2..s%......(|

   # 08e5ebfc  739b0032 0086ca25 00000000 00000000  |2..s%...........|

   # 08e5ec0c  00000000 0032282d ca25739b 0032279f  |....-(2..s%..'2.|

   # 08e5ec1c  ca25739b 00000000 00000003 00000000  |.s%.............|

   # 08e5ec2c  00000000 00000000 00000000 00000000  |................|

   # 08e5ec3c  00000000 00000001 00000000 00000000  |................|

   # 08e5ec4c  00000000 00000000 0000fffd 00004000  |.............@..|

   # 08e5ec5c  00000000 00000000 00000000 00000000  |................|

   # 08e5ec6c  00000000 00000000 00000ce8 ffffffff  |................|

   # 08e5ec7c  00000000 00000000 00000000 00000000  |................|

   # 08e5ec8c  00000000 00000000 00000000 08e5ece4  |................|

   # 08e5ec9c  00000000 00000000 0000117c 00000001  |........|.......|

   # 08e5ecac  00000000 00000000 00000000 60050000  |...............`|

   # 08e5ecbc  00000000 00000000 00000000 00000000  |................|

   # 08e5eccc  2e3d4e43 27262d2e 275f2727 2726275f  |CN=..-&'''_'_'&'|

   # 08e5ecdc  2e275f2d 3d4f2f27 614c4151 00007362  |-_'.'/O=QALabs..|

   # 08e5ecec  00000000 00000000 00000000 00000000  |................|

   # 08e5ecfc  00000000 00000000 00000000 00000000  |................|

   # 08e5ed0c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed1c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed2c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed3c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed4c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed5c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed6c  00000000 00000000 00000000 00000000  |................|

   # 08e5ed7c  5b275b3c 243d275f 3e232727 272e213b  |<['[_'=$''#>;!.'|

   # 08e5ed8c  2325265d 2d3c2f24 3d275e60 2424215e  |]&%#$/<-`^'=^!$$|

   # 08e5ed9c  2e275f27 3c5e405f 3a5d5e26 277b2e5f  |'_'._@^<&^]:_.{'|

   # 08e5edac  26277d3c 2327267c 265b7d5f 7c2e2b21  |<}'&|&'#_}[&!+.||

   # 08e5edbc  2b232d3e 5e222624 405e3d5f 607d7b2b  |>-#+$&"^_=^@+{}`|

   # 08e5edcc  5f2e3e5f 5e3b2f3e 5b2e2724 603b277b  |_>._>/;^$'.[{';`|

   # 08e5eddc  2d2c602a 5b7e3b2c 60243c5e 2b3d5d27  |*`,-,;~[^<$`']=+|

   # 08e5edec  5b2a2324 3d7c235f 2d2b2624 7c5e2b7b  |$#*[_#|=$&+-{+^||

   # 08e5edfc  5e2c272d 24222c24 2b7c2f3d 2a3c5d3b  |-',^$,"$=/|+;]<*|

   # 08e5ee0c  5f7e232e 273d3a7d 5b2b3b40 5f245e3b  |.#~_}:='@;+[;^$_|

   # 08e5ee1c  277b7b7c 213a212e 3c7e3c2f 2d3c5d23  ||{{'.!:!/<~<#]<-|

   # 08e5ee2c  2e23227e 3d237b5f 26217e5b 5e272b23  |~"#._{#=[~!&#+'^|

   # 08e5ee3c  24267d24 233e2322 7b407c26 263d3b2e  |$}&$"#>#&|@{.;=&|

   # 08e5ee4c  215f3c7d 7d7b5e24 607d3d27 275e5f3e  |}<_!$^{}'=}`>_^'|

@[ 1] 0x60008af0 nnotes._Cstrlen@4+16 (7b273d5d)

@[ 2] 0x603097e7 nnotes._AdminpModifyItem@32+1607 (7d3a5f2a,247c2424,3d262e2d,225f2c7b,8e5e688,3,1,79c)

Short Callstack detected, dumping extra stack data

############################################################

PASS 3 : FATAL THREAD with PARAMETER DATA 5/9 [ nadminp: 3428: 4076]

############################################################

@[ 1] 0x60008af0 nnotes._Cstrlen@4+16 (7b273d5d)

@[ 2] 0x603097e7 nnotes._AdminpModifyItem@32+1607 (7d3a5f2a,247c2424,3d262e2d,225f2c7b,8e5e688,3,1,79c)

   Parameter 5:

   # 08e5e688  08e5e774 08e5e790 08e5e6b4 7c82eeb2  |t..............||

   # 08e5e698  08e5e774 08e5ffdc 08e5e790 08e5e750  |t...........P...|
10

Subject: RE: AdminP is causing Domino server to Crash

euh, it’s the same fatal thread. :slight_smile:

You said Our Domino server crashes every night just after 12:00am (midnight).

Do you have another NSD?

JYR

11

Subject: RE: AdminP is causing Domino server to Crash

I do have two different NSD logs. One that was generated for the midnight crash and the other for when I type 'tell adminp process all’But the fatal thread seems to be the same for both logs.

I can send you(or attach) both NSD logs… But where? They are too big to be pasted in here. The overnight crash log is greater than 2GB and the ‘tell adminp’ log is about 500MB.

Thank you.