Server crashes. Problem with ODBC, iTEXT, LS2J, DB2?

hcl-bot · January 8, 2008, 7:56am

I’m hoping someone can help.

We have a 100% web application that’s been running for over a year. All data is contained in DB2. We also use COM+ and IText in our app. Our severa crashes several times a day. All NSD’s (using Lotus Diagnostic Tool) have a similar output:

Build: 7.0.3

Host Name: HQDOMINO

Date: Tue Jan 08 14:15:40 2008

############################################################

FATAL THREAD 46/96 [ nHTTP: 0bd8: 0840]

FP=0x0cc3d3fc, PC=0x4bf7131a, SP=0x0cc3d3fc

stkbase=0cc50000, total stksize=262144, used stksize=76804

EAX=0x00000000, EBX=0x0000000b, ECX=0xffffffff, EDX=0x00000000

ESI=0x0ffcfce8, EDI=0xffffffff, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

[ 1] 0x4bf7131a odbc32 (ffcfd0c,cc3d414,4bf7325d,ffcfd0c)

[ 2] 0x4bf8cd80 odbc32.VRetrieveDriverErrorsRowCol+38636 (ffcfd0c,cc3d4c0,4bf7345f,ffcfce8)

[ 3] 0x4bf7325d odbc32.MpHeapAlloc+1470 (ffcfce8,cc3d870,0,4)

[ 4] 0x4bf7345f odbc32.SQLGetInfoW+333 (ffcfce8,b,ffc2e80,8)

[ 5] 0x4bfa0b18 odbc32.SQLGetInfo+167 (ffcfce8,b,cc3d858,4)

[ 6] 0x0aa497e4 nLSXODBC.MainEntryPoint+76244 (0,cc3e5e8,cc3e0b4,ff542d4)

[ 7] 0x0aa49ab7 nLSXODBC.MainEntryPoint+76967 (22d41724,cc3d8a4,0,0)

@[ 8] 0x0ff542d4 nDBODBC.add_connect_entry+260 (cc3e0e8,cc3e3e8,cc3e4e8,3853)

@[ 9] 0x0ff51732 nDBODBC.DALPerform@32+834 (18464d1,3853,2,4,0,0,0,0)

@[10] 0x600b5d41 nnotes.DBDFunction@36+1889 (18464ac,0,2,5,0,0,0,0,0)

@[11] 0x6016795c nnotes.FallbackComputeExtension+476 (d3,cc40370,2,5,0,0,0,0,0)

@[12] 0x600b3a3b nnotes.CompGeneralContext::ExtensionProc+459 (9d,b9,0,5,0,0,0,0,0)

@[13] 0x600b3ed5 nnotes.ExtensionProc::Execute+229 (29916c78,243460e0,1,0,0,0,0,0,0)

@[14] 0x60045ed2 nnotes.AtFunctionNode::ComputeVariants+82 (29916c78,299161e0,29916cb4,19409fe,0,0,0,0,0)

@[15] 0x6008c51b nnotes.AtIf::ComputeVariants+203 (243460a8,299161e0,29916cb4,cc3edf8,0,0,0,0,0)

@[16] 0x60065180 nnotes.AssignmentExpressionNode::ComputeVariants+80 (0,29915e2c,1,cc3edec,0,0,0,0,0)

@[17] 0x60046101 nnotes.RootNode::ComputeVariants+81 (29914e2c,29915e2c,0,4b6c80,0,0,0,0,0)

@[18] 0x60045ced nnotes.Compute::Eval+141 (cc42884,cc40370,77e6bdf0,29915e2c,0,0,0,0,0)

@[19] 0x600b3645 nnotes.NSFComputeEvaluateExt@32+181 (29915e2c,795,cc428fc,cc40338,0,0,0,0)

@[20] 0x004bb37d NINOTES.NFormula::Evaluate+141 (cc42884,0,cc4a3d0,2a04ed48,0,0,0,0)

@[21] 0x004c1eed NINOTES.NItem::GetDefaultValue+445 (cc42884,2a04ed48,0,cc4cb48,0,0,0,0)

@[22] 0x004c201a NINOTES.NItem::SetComputedValue+234 (cc4cb48,4bd362,cc4a3d0,cc441a0,0,0,0,0)

@[23] 0x004a8ac7 NINOTES.NDocNote::SetItemComputedValues+103 (ed,1,cc43e44,cc43fd4,0,0,0,0)

@[24] 0x004bdc2f NINOTES.INotesCompExtProc@36+79 (29919e2c,cc4a3d0,ed,1,0,0,0,0,0)

@[25] 0x600b399a nnotes.CompGeneralContext::ExtensionProc+298 (e,ed,0,1,0,0,0,0,0)

@[26] 0x600b3ed5 nnotes.ExtensionProc::Execute+229 (29920130,2434450c,1,0,0,0,0,0,0)

@[27] 0x60045ed2 nnotes.AtFunctionNode::ComputeVariants+82 (243444b4,2991ffa0,24342854,2991fefc,0,0,0,0,0)

@[28] 0x6008c51b nnotes.AtIf::ComputeVariants+203 (0,29919e2c,1,cc443e8,0,0,0,0,0)

@[29] 0x60046101 nnotes.RootNode::ComputeVariants+81 (2991462c,29919e2c,cc45b2c,4b6c80,0,0,0,0,0)

@[30] 0x60045ced nnotes.Compute::Eval+141 (cc45968,cc4a3d0,77e6bdf0,77c675b7,0,0,0,0,0)

@[31] 0x600b3645 nnotes.NSFComputeEvaluateExt@32+181 (29919e2c,795,cc459e0,cc45934,0,0,0,0)

@[32] 0x004bb37d NINOTES.NFormula::Evaluate+141 (cc45968,cc45b2c,1,2,0,0,0,0)

@[33] 0x004bb6b5 NINOTES.NFormula::Evaluate+69 (657b600,ee4afc0,ee05130,1,0,0,0,0)

@[34] 0x00436cdc NINOTES.CmdHandler::HandleClick+572 (ee4afc0,cc4cb48,ed79008,cc4c9c8,0,0,0,0)

@[35] 0x004f07c7 NINOTES.CmdHandlerBase::HandleOpenFormCmd+743 (ee4afc0,0,0,ee4afc0,0,0,0,0)

@[36] 0x00441c2c NINOTES.CmdHandlerBase::PrivHandle+268 (ee4afc0,0,0,657b600,0,0,0,0)

@[37] 0x00440199 NINOTES.CmdHandler::PrivHandle+121 (ee4afc0,eec8790,ee4afc0,0,0,0,0,0)

@[38] 0x0044130d NINOTES.CmdHandler::Handler+221 (ee4afc0,657b600,ee4afdc,6e,0,0,0,0)

@[39] 0x0043778a NINOTES.Cmd::Execute+58 (28572164,28571e78,3,66a408,0,0,0,0)

@[40] 0x0047ccce NINOTES.InotesHTTPProcessRequestImpl+1614 (28572174,28572164,28571e78,3,0,0,0,0)

@[41] 0x0047f20f NINOTES.InotesHTTPProcessRequest+47 (28572174,285724fc,0,0,0,0,0,0)

@[42] 0x100156a4 nhttpstack.HTInotesRequest::ProcessRequest+36 (28571e78,28571d10,0,0,0,0,0,0)

@[43] 0x10010f1d nhttpstack.HTRequestExtContainer::ProcessRequest+941 (5,1,ee25fa8,0,0,0,0,0)

@[44] 0x1002011f nhttpstack.HTRequest::ProcessRequest+1919 (0,65e8d48,0,6970652f,0,0,0,0)

@[45] 0x100258e1 nhttpstack.HTSession::StartRequest+897 (65e8d54,65e8d48,0,468,0,0,0,0)

@[46] 0x1002ce3f nhttpstack.HTWorkerThread::CheckForWork+399 (3,65e8d48,10029c50,10029c7a,0,0,0,0)

@[47] 0x1002d358 nhttpstack.HTWorkerThread::ThreadMain+88 (65e8d48,0,0,0,0,0,0,0)

@[48] 0x60103160 nnotes.ThreadWrapper@4+208 (0)

[49] 0x77e66063 kernel32.GetModuleFileNameA+235

as well as

Build: 7.0.3

Host Name: HQDOMINO

Date: Tue Jan 08 14:15:40 2008

############################################################

thread 51/96: [ nHTTP: 0bd8: 0590]

FP=0d147aa4, PC=7c82f357, SP=0d147a98

stkbase=0d150000, total stksize=262144, used stksize=34152

############################################################

[ 1] 0x7c82f357 ntdll.RtlEnterCriticalSection+32 (66679f0,50e4818,12c7448,11cc)

[ 2] 0x71b2255b mswsock+9563 (10c0,2,d147af8,50e4818)

[ 3] 0x71c078cf WS2_32.shutdown+75 (10c0,2,50e91e0,d147b34)

@[ 4] 0x630123b0 nNTCP.cmd_CloseFunctionPanicProcedure@4+16 (10c0)

@[ 5] 0x601801ad nnotes.OSShutdownPanicProcedure@0+141 ()

@[ 6] 0x60184b4a nnotes.StaticHangEnable@0+138 ()

@[ 7] 0x600070b4 nnotes.OSMemAllocExtended@16+212 (11e9,1f4a,0,d147ba4)

@[ 8] 0x600076e7 nnotes.OSMemAlloc@12+23 (11e9,1f49,d147ba4)

@[ 9] 0x0ff532ef nDBODBC.env_alloc+31 (0,d1484c0,d148330,10001)

@[10] 0x0ff530af nDBODBC.get_list_via_stmt+207 (d1480ec,2,10001,d148330)

@[11] 0x0ff5176f nDBODBC.DALPerform@32+895 (18464d1,3853,2,4,0,d148110,f10f10,ffffffff)

@[12] 0x600b5d41 nnotes.DBDFunction@36+1889 (18464ac,0,2,5,0,d1482a0,f10f10,ffffffff,0)

@[13] 0x6016795c nnotes.FallbackComputeExtension+476 (eb,d149e74,2,5)

@[14] 0x600b3a3b nnotes.CompGeneralContext::ExtensionProc+459 (7b,b9,0,5)

@[15] 0x600b3ed5 nnotes.ExtensionProc::Execute+229 (244e2c60,244e2f94,2,0)

@[16] 0x60045ed2 nnotes.AtFunctionNode::ComputeVariants+82 (244e2c60,244e2360,244e2c9c,19409fe)

@[17] 0x6008c51b nnotes.AtIf::ComputeVariants+203 (244e2f5c,244e2360,244e2c9c,d1488fc)

@[18] 0x60065180 nnotes.AssignmentExpressionNode::ComputeVariants+80 (0,244e2018,1,d1488f0)

@[19] 0x60046101 nnotes.RootNode::ComputeVariants+81 (20f5e128,244e2018,0,4b6c80)

@[20] 0x60045ced nnotes.Compute::Eval+141 (d14b194,d149e74,77e6bdf0,244e2018)

@[21] 0x600b3645 nnotes.NSFComputeEvaluateExt@32+181 (244e2018,1ab9,d14b20c,d149e3c,0,d148974,f10f10,ffffffff)

@[22] 0x004bb37d NINOTES.NFormula::Evaluate+141 (d14b194,0,20f5c428,20f5c428)

@[23] 0x004c1eed NINOTES.NItem::GetDefaultValue+445 (d14b194,d14cb48,2,0)

@[24] 0x004c57e1 NINOTES.NItem::SetDefaultValue+65 (1,d14c324,20f5d528,d14b414)

@[25] 0x004a5c80 NINOTES.NDocNote::AddItem+208 (20f5d528,20f5c528,20f5d528,b)

@[26] 0x004b4678 NINOTES.NForm::AddField+200 (20f5d528,0,d14c324,1)

@[27] 0x004b5156 NINOTES.NForm::ProcessFieldRecords+1206 (25462278,20f5d528,ef20000,d14c324)

@[28] 0x004b683b NINOTES.NForm::PrivLoadFields+939 (25461f36,0,d14cb48,d14b784)

@[29] 0x004b6b0e NINOTES.NForm::LoadFields+30 (d14cb48,edbc038,ee8b978,657b600)

@[30] 0x004aaf1f NINOTES.NDocNote::CreateUsing+319 (edbc038,d14c324,ee8c238,ee8b978)

@[31] 0x004f064d NINOTES.CmdHandlerBase::HandleOpenFormCmd+365 (ee8b978,0,0,ee8b978)

@[32] 0x00441c2c NINOTES.CmdHandlerBase::PrivHandle+268 (ee8b978,0,0,657b600)

@[33] 0x00440199 NINOTES.CmdHandler::PrivHandle+121 (ee8b978,ef26100,ee8b978,0)

@[34] 0x0044130d NINOTES.CmdHandler::Handler+221 (ee8b978,657b600,ee8b994,6e)

@[35] 0x0043778a NINOTES.Cmd::Execute+58 (ef4645c,ef46170,3,66a408)

@[36] 0x0047ccce NINOTES.InotesHTTPProcessRequestImpl+1614 (ef4646c,ef4645c,ef46170,3)

@[37] 0x0047f20f NINOTES.InotesHTTPProcessRequest+47 (ef4646c,ef467f4,0,0)

@[38] 0x100156a4 nhttpstack.HTInotesRequest::ProcessRequest+36 (ef46170,ef46008,0,0)

@[39] 0x10010f1d nhttpstack.HTRequestExtContainer::ProcessRequest+941 (5,1,eee2b90,0)

@[40] 0x1002011f nhttpstack.HTRequest::ProcessRequest+1919 (0,65ec33f,0,6970652f)

@[41] 0x100258e1 nhttpstack.HTSession::StartRequest+897 (65ec34b,65ec33f,0,4a4)

@[42] 0x1002ce3f nhttpstack.HTWorkerThread::CheckForWork+399 (3,65ec33f,10029c50,10029c7a)

@[43] 0x1002d358 nhttpstack.HTWorkerThread::ThreadMain+88 (65ec33f,0,0,0)

@[44] 0x60103160 nnotes.ThreadWrapper@4+208 (0)

[45] 0x77e66063 kernel32.GetModuleFileNameA+235 (0,0,0,0)

There is always a ‘c0000005 (ACCESS_VIOLATION)’ and something to do with ODBC. We have the latest Domino 7 release. We’ve checked our code. Our server is very powerful (CPU & memory-wise).

We are talking with IBM but I’m that someone can see something familiar here. Thank you. John

hcl-bot · January 8, 2008, 8:03am

Subject: Server crashes. Problem with ODBC, iTEXT, LS2J, DB2?

Is there anything new (Drivers, updates, etc)?

The firs tlines of the stack suggest that it’s not related to Domino because you don’t have the ‘@’ in front of these lines.

Can you debug your code to diagnose which lines is causing the error?

[ 1] 0x4bf7131a odbc32 (ffcfd0c,cc3d414,4bf7325d,ffcfd0c)

[ 2] 0x4bf8cd80 odbc32.VRetrieveDriverErrorsRowCol+38636 (ffcfd0c,cc3d4c0,4bf7345f,ffcfce8)

[ 3] 0x4bf7325d odbc32.MpHeapAlloc+1470 (ffcfce8,cc3d870,0,4)

[ 4] 0x4bf7345f odbc32.SQLGetInfoW+333 (ffcfce8,b,ffc2e80,8)

[ 5] 0x4bfa0b18 odbc32.SQLGetInfo+167 (ffcfce8,b,cc3d858,4)

[ 6] 0x0aa497e4 nLSXODBC.MainEntryPoint+76244 (0,cc3e5e8,cc3e0b4,ff542d4)

[ 7] 0x0aa49ab7 nLSXODBC.MainEntryPoint+76967 (22d41724,cc3d8a4,0,0)

Frequently asked question - How to analyze Notes/Domino NSDs?

http://www-10.lotus.com/ldd/nd6forum.nsf/DateAllFlatweb/92d3876c4e4f140a852572810008ea9d?OpenDocument

JYR

hcl-bot · January 8, 2008, 9:25am

Subject: RE: Server crashes. Problem with ODBC, iTEXT, LS2J, DB2?

Thank you for your quick response.

Of course, at first I thought the crashes were related to our (many) agents. I still haven’t ruled this out. I also don’t know if it’s Domino code related. That is, LS2J, LSX (ODBC) related.

I’ve ‘studied’ the NSD’s but I can’t tell what else they can tell me about where to look for the problem. I don’t know if ODBC32 is Domino related or COM related (in Domino). Also, the nLSXODBC mention pointed me back to Domino.

Let me know if you can recommend anything else.

hcl-bot · January 8, 2008, 9:43am

Subject: RE: Server crashes. Problem with ODBC, iTEXT, LS2J, DB2?

Use this tool:

Lotus Notes Diagnostic Tool (Nsd Analyzer)

Automatically perform repetitive tasks associated with files from a Lotus Domino server crash, hang, or performance issues.

http://www-10.lotus.com/ldd/sandbox.nsf/ByDate/c0c64aa07fc9abae85257356004ac7b3?OpenDocument

It will guide you on what debug parameters you could add to troubleshoot your issue

JYR

hcl-bot · January 8, 2008, 10:08am

Subject: RE: Server crashes. Problem with ODBC, iTEXT, LS2J, DB2?

Thanks JY. I’ve been using this tool and that’s where I noticed the pattern in the NSD’s. I’ve also used the debug parameters in the past but haven’t noticed anything informative there either. Do you think the ODBC32 mentioned in the NSD has something to do with the ODBC driver that Domino uses to communicate with DB2? That is, the ODBC driver.