FAQ - How to analyze Notes/Domino NSDs?

Hi,

It may be useful to troubleshoot your clients and servers comes partly from here

TN 7007508 Knowledge Collection: NSD for Notes Domino release 6 and 7

TN 7003396: Overview of NSD Analysis on UNIX and S/390
SG24-6051: Lotus Domino for AS/400: Problem Determination Guide
TN 7003599: Using NSD for Domino Servers and Notes Clients on Windows NT, 2000 and XP ()
TN 1086330: Memcheck: What Is It And How Does It Work?
TN 1153833: Troubleshooting Notes Client Crashes
TN 7003028: Troubleshooting Notes/Domino Server Crashes<—Link seems to be broken (On November 16th 2008)
TN 7002930: Domino Web Server (HTTP) Crash and Hang Troubleshooting
TN 1094630: Semaphores and Semaphore Timeouts

TN 1253056 :Troubleshooting Notes/Domino abnormal terminations on W32

HND202 - LAB.pdf Lotusphere 2007

HND 107 - Using IBM Lotus Domino Notes System Disagnostic (NSD) Soft copy Hand out LotusSphere 2008

TN 1252970 Using NSD to generate W32 process Core Files

From the FAQ of FAQs in the NSD - crashes and section:
http://www-10.lotus.com/ldd/sandbox.nsf/ecc552f1ab6e46e4852568a90055c4cd/42334489e0ed9f0a85257123004d9eed

Troubleshooting Lotus Domino hangs and crashes

From eview (registration)

Automatic Fault Recovery and Crash Data Collection — New and Improved!

Lotus Notes Diagnostic Tool (Nsd Analyzer)

Automatically perform repetitive tasks associated with files from a Lotus Domino server crash, hang, or performance issues.

http://www-10.lotus.com/ldd/sandbox.nsf/ByDate/c0c64aa07fc9abae85257356004ac7b3?OpenDocument

Update may 14th 2008-

Version 2.6

ftp://ftp.software.ibm.com/software/lotus/tools/LND_2.6_Setup.zip

november 3rd 2008

Version 2.7 is available

ftp://ftp.software.ibm.com/software/lotus/tools/LND_2.7_Setup.zip (4028 KB)

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs?

Hi,In some cases, NSD for Domino server R6.x on Linux does not collect process call stack information.

Use GDB to collect process stack information

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs?

Title: NSD Fixlist and NSD Update Strategy
URL: http://www.ibm.com/support/docview.wss?rs=463&uid=swg21233676

Title: Updated NSD for Domino Releases
URL: http://www.ibm.com/support/docview.wss?rs=463&uid=swg24013182

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs?

Description of the IPC section header fields in an NSD

TN 1204263 How to run a manual NSD for Notes/Domino on Windows

1115742 Running NSD from a restricted Windows user environment

1209535 NSD crashes with FATAL errors in two processes - which process caused the server outage?

Subject: FAQ - How to analyze Notes/Domino NSDs?
TN 7003599: Using NSD for Domino Servers and Notes Clients on Windows NT, 2000 and XP

It is from Google cache, sorry for the display, it is an old Technote but it is still acurate

JYR

Technical Paper

Lotus Notes Knowledge Base

Published

Technical Paper Document Details

Using NSD for Domino Servers and Notes Clients on Windows NT, 2000 and XP

NSD has been a diagnostic shell script on the Domino UNIX platforms for a long time, and has proven itself an invaluable tool that improves our ability to isolate problems on the Windows NT platform. These instructions and the examples referenced use versions 1.8 of NSD, which contains version 4.12 of Memcheck, the prerelease versions current when this technote was revised. Unlike the nsd.sh shell script for Unix, nsd.exe for Windows is different for each version of Notes and Domino. NSD is a tool used by Lotus Technical Support to diagnose particular problems at customer sites.

To collect the most complete information, run NSD with no options.

To collect a subset of this information, or to add some special processing, run NSD with options listed in the table below. Sample benefits NSD will bring to problem resolution are also in this table:

NSD command

What it does

How it helps

nsd -ver
nsd -version

Lists the versions for NSD, Windows, and every instance of Notes or Domino found running from data and program directories referenced in notes.ini files found on the system.

Identifies all partitions and clients active at the time NSD was run, along with Notes API programs that load nnotes.dll.

nsd -ps

Lists all processes running on the system, with process and user ID’s, start times, highlighting Notes/Domino processes.

Shows which processes would be dumped or killed. Shows whether servers were started at boot time, or whether server processes were loaded at server boot or dynamically.

nsd -info

Display all of the above, plus system information, notes.ini, environment variables, data and program directory contents, disk drive and system memory statistics.

Allows you to verify the installed versions of all executables and accessory files, including base release and hotfixes.

nsd -nomemcheck

(collect nsd data but don’t run the internal version of memcheck)

Display all of the above, plus the call stacks and function arguments for each thread of every active process running with the data directory where this copy of NSD is installed.

Shows everything that was active on the server, and what each thread was doing, instead of just dumping the stacks of the threads in a failing process, as with the QNC debugger.

memcheck

Display usage information for all of Notes/Domino memory, semaphores, databases, and other known objects.

Saves an incredible amount of trial and error logging, data collection, and reading, to get the same information.

nsd

Displays all of the above.

A simple command collects a large set of standard diagnostic data.

nsd -kill

After “nsd -ps” processing, terminates client and server processes active from the current data directory.

Terminates errant hanging processes after a client or server crash, allowing a restart without a reboot of the OS.

nsd -ini <pathname\filename>

Use a particular INI file rather than search for all.

Use NSD even if usage of notes.ini files is nonstandard.

nsd -i -auto

Install NSD as the default application debugger, automatically invoked.

Dumps thread call stacks of all server processes, not just the failing process.

nsd -help

Shows a short description of available commands in Unix versions, some not available in Windows NSD version 1.8.

Remember commands without this technote. See examples below.

nsd -lsof

Displays open files in Unix versions, not yet in Windows NSD version 1.8.

nsd -filter

Collapses duplicate call stacks in Unix versions, not yet in Windows NSD version 1.8.

Analyzes duplicate stacks in threads, which is especially useful for waiting threads in database server, http, etc.


Table of Contents:

Installing NSD for Windows to debug a server or client hang
Using basic NSD for Windows command line options
Terminating NSD after gathering diagnostic data
Capturing all server threads during a server Panic or crash
Repeating NSD more than once on the same system
Analyzing the output of NSD for Windows

All references to C:\winnt below are referring to the default Windows system directory (%windir%), which may be different on the actual system hosting Notes or Domino. The Notes or Domino installation directories shown are also representative, and may be changed.

Installing NSD for Windows to debug a server or client hang

NSD is automatically installed with Notes clients and Domino servers starting with release 5.0.9. For releases prior to that:

Get from Lotus support the version of NSD.EXE that matches the Notes or Domino release, along with PSAPI.DLL.
Copy PSAPI.DLL into any directory in the path, for example, C:\winnt\system32 (skip this step on Windows 2000 or XP)
Copy NSD.EXE into the Notes or Domino program directory (this is where Notes and Domino install it)

When the server or client hangs:

  • Open a Command Prompt window.
  • Set the current directory to the data directory for the hanging server or client.
  • Run NSD with no options.

Send to Lotus Technical Support the output file “nsd_all . . . mm_dd@hh_mm.log” that will be saved in the data directory.

In order to find the thread call stacks, NSD must attach to the processes as the debugger. Currently on Windows, a program cannot relinquish that power once acquired, and the NSD program cannot terminate without bringing down the processes as well, making the target server or client inoperable, and requiring a restart. When NSD is finished processing, it will wait with this prompt, which you must leave up until you are ready to stop the server or client:

nsd

At this point, if the Domino server window is still responsive, try to shut down the server, to gracefully stop as many processes as you can by entering the “q” or “quit” command on the Domino Server Console. If you run NSD against a Notes client, you may not be able to bring focus back to the client window, since NSD will be an active debugger. You can type “kill” at the NSD prompt to terminate any leftover Notes or Domino processes from the current data directory, then “q” or “quit” to terminate NSD and see the final messages, starting with the amount of time the window was active:

Started at: Sun Jan 23 16:51:18 2000
Ended at: Sun Jan 23 16:52:27 2000

Generated Messages:

INFO (0): Using NSD profile C:\WINNT\notesnsd.ini

If you type “q” or “quit” in the NSD window while the server or client is still running, you receive this response:

nsd> q
NSD is still attached to nn Notes processes. If Notes processes
can’t be terminated gracefully or are not responding, then use the
kill command to terminate them.

NSD default behavior is to store its messages in an output file in the Notes or Domino data directory. At the end of NSD messages in the command window, you will see a message listing any file created from this run. If NSD starts an output file, but is waiting with an “nsd” prompt, this message will not appear until NSD closes the file after you issue the “quit” command.

Please send the following files:

Log File : nsd_all…mm_dd@hh_mm.log

Windows Notepad will not properly display an NSD output log file, but Windows Wordpad, any programmer’s editor, or any Unix editor will.

Files with this name will be created in the Notes or Domino data directory with dumps of invalid memory blocks if any such blocks are found:

memcheck…mm_dd@hh_mm.dmp

NSD, especially the Memcheck contained within it, has release-dependent code, and you must run the version that goes with the release of Notes or Domino you are trying to debug. To check the version of NSD, and the version of Notes or Domino to which it applies, run NSD with the “-version” option:

C:>nsd -version
NSD Version : 1.8
Memcheck Version: MEMCHECK Version (4.12) for Lotus Notes Release 5.0.8 (June 18, 2001)
Host Name : MYSYSTEM
User User : Administrator
Date : Tue Oct 09 16:28:50 2001
Windows Dir : C:\WINNT
OSVersion : NT 5.0 (Build 2195), PlatID=2, Service Pack 2 (1 Processor)

If the version of NSD does not match the version of Notes or Domino, NSD may crash, and if it is already attached to Domino server processes as the debugger, it brings them down as well. For this reason, we recommend that you try out NSD for the first time in a safe environment, typically a test server that is configured to closely emulate the production environment. For clients, this means avoiding a live Notes client with unsaved data for a trial run. If NSD should crash, try “nsd -nomemcheck” to see if that runs successfully. The memcheck functions are the most highly release-dependent, so you may be able to avoid a problem caused by version mismatch, perhaps due to an installed server hotfix or debug version of code, by not running the memcheck functions.

However, if the server or client hangs, please run NSD even if you never ran it before, since there is no additional risk to your server or client, which is already failing. Even if NSD does not complete successfully, the process and thread stack information collected provide a valuable source of new information when debugging the hang. If NSD or Memcheck does crash, and if the correct version was used for your Notes or Domino release, please send in the notes.rip file or the drwtsn32.log file created in the current data directory, along with any output files, which will be named as noted above.

Using basic command line options of NSD for Windows

The help options in NSD version 1.8 give the following basics:

NSD -help displays the options supported in the UNIX versions of NSD:

Usage: nsd options
Usage: nsd [options] [ pid ]

Options:
-batch (run in batch mode – don’t write to console)
-info (report system info only)
-noinfo (don’t report system info)
-nolog (don’t log output to log file)
-version (show version header only)
-ps (show process tree only)
-kill (kill all/user notes processes and cleanup)
-memcheck (run the Notes memory checker only)
-nomemcheck (don’t run the Notes memory checker)
-perf (collect performance data only)
-noperf (don’t collect performance data)
-lsof (run lsof only – list Notes open files)
-nolsof (don’t run lsof by default)
-filter <log_file> (filter & annotate stack output of NSD log_file)
-filter1 <log_file> (filter (only) stack output of NSD log_file)
-annotate <log_file> (annotate stack symbols of NSD log_file)
-dumpsym <image_file> (dump symbols of dll/exe file image_file )
-i [-auto] [options] (install NSD as the Just-In-Time debugger.)
( -auto: kick-off automatically by the OS)
( options: options be passed by OS to NSD (i.e. -noinfo)
-u (uninstall NSD as the Just-In-Time debugger.)
-help (show this help list)
-help (where option is any one of the above)
-help gen
eral (general info about NSD and how it works)
-help lim*itations (general info on NSD limitations)
-help update (list NSD version update info)

NSD -help gen gives a basic overview of NSD:

NSD is intended to help you diagnose and report problems
you encounter when running Notes servers. Here is what it does:

  • If you don’t specify any options or arguments, NSD finds
    all running Notes processes and attaches the debugger to each process
    to dump all thread stacks in that process.

  • Unless the ‘-nolog’ flag is specified, NSD will automatically
    log its output to a file named as follows:

    nsd_all@.log (default)
    ps_@.log (with the -ps flag)
    kill_@.log (with the -kill flag)
    sysinfo_@.log (with the -info flag)
    memcheck_@.log (with the -memcheck )

  • It will run the Notes memory checker ‘memcheck’ that detects and reports
    memory corruption in Notes shared memory, process memory and core files.
    Corrupted memory segments (if any) are compressed and saved in a file
    named:

    memcheck__@.dump

  • By default, processes that have NNOTES.DLL loaded are candidates
    for tracing.

Note:
You can set the environment variable NSD_LOGDIR to point
to the directory where you want your logs/cores to be
automatically saved. By default, NSD_LOGDIR points to the
current directory.

in the above examples will be replaced with “W32I”, as in

nsd_all_W32I_NTSYSTEM_03_08@16_59.log

NSD -ps lists the processes that are running in the system:

PID PPID UID STIME COMMAND
00000000 00000000 0 ??? [System]
00000008 00000000 0 ??? [System]
0000008c 00000000 0 01/26 12:14 \SystemRoot\System32\smss.exe
000000a4 00000000 0 01/26 12:14 ??\C:\WINNT\system32\csrss.exe
000000b8 00000000 0 01/26 12:14 ??\C:\WINNT\system32\winlogon.exe
000000d4 00000000 0 01/26 12:14 C:\WINNT\system32\services.exe

…

00000468 00000000 1000 01/28 10:06 C:\WINNT\system32\cmd.exe
→ 00000388 00000000 1000 01/31 08:51 C:\Notes\notes.exe
→ 0000041c 00000000 1000 01/31 08:51 C:\Notes\nhldaemn.EXE
00000414 00000000 1000 01/31 11:19 C:\Notes\Client\nsd.exe

Note the arrows marking the processes running using the Notes/Domino data directory from which NSD was run.

NSD gathers all data, and calls the Memcheck within NSD itself.

When the embedded Memcheck is called by NSD, the command line used is similar to the following:

memcheck -k cur -d err -o memcheck_nsd_all_W32I_HOSTNAME_MM_DD@HH_MM.dmp

This means that files with this name will also be created with dumps of invalid memory blocks if any such blocks are found.

Terminating NSD after gathering diagnostic data

In order to find the thread call stacks, NSD must attach to the processes as the debugger. QNC, the default debugger installed by Domino, or Visual C++, cannot be used on the Domino processes once NSD dumps the thread stacks. Even NSD, if run a second time against the same processes, will not find the thread stacks. After completing its processing, NSD will wait with this prompt:

nsd>

At this point, you can type “ps” to get the process list, “kill” to terminate all Notes or Domino processes from the current data directory, or “q” or “quit” to terminate NSD, and see the final messages, starting with:

Started at: Sun Jan 23 16:51:18 2000
Ended at: Sun Jan 23 16:52:27 2000

Generated Messages:

INFO (0): Using NSD profile C:\WINNT\notesnsd.ini

If you type “q” in the NSD window while the server or client is still running, you get this response:

nsd> q
NSD is still attached to nn Notes processes. If Notes processes
can’t be terminated gracefully or are not responding, then use the
kill command to terminate them.

At the end of NSD output in the command window you see a message if there is an output log file from this run. If NSD has started an output file, but is waiting with an “nsd” prompt, this message will not appear until after you “quit” the window.

Please send the following files:

Log File : nsd_all_W32I_HOSTNAME_MM_DD@HH_MM.log

Windows Notepad will not properly display the NSD output file, but Windows Wordpad, or any Unix or programmer’s editor will. The optional memcheck dump file will not be listed, must be found separately, and may not be displayed with any editor.

Repeating NSD more than once on the same system

To dump thread call stacks for the same server a second time, type “dump” at an “nsd>” prompt. This can be helpful to show the differences in repeated call stacks when a thread is in a loop rather than just waiting.

You can also run NSD on each of several instances of Domino server or Notes client that are running on the same system. Each must be run from a separate command window, which you must also leave running at an “nsd>” prompt.

Analyzing the output of NSD for Windows

This section will be very limited, showing some of the features that can be observed. Analysis of NSD output is a lengthy subject covering vast amounts of detail of the architecture of Notes and Domino. Some of the documents relevant to NSD analysis are linked below.

This process table from version 1.0 of NSD shows Domino running as an NT service, a Netfinity API program that has NNOTES.DLL loaded, and a Notes client running, all from the same data directory (because they all have arrows → pointing to them).

@@@@@@@@@@@@@@@@@ Process Table @@@@@@@@@@@@@@@@@

    PID     PPID     STIME    COMMAND

00000000 00000000 ??? [System]
00000002 00000000 ??? [System]
0000001a 00000002 01/26 16:34 \SystemRoot\System32\smss.exe
00000022 0000001a 01/26 16:34 ??\C:\WINNT\system32\csrss.exe
00000028 0000001a 01/26 16:34 ??\C:\WINNT\system32\winlogon.exe
0000002e 00000028 01/26 16:34 C:\WINNT\system32\services.exe
00000048 0000002e 01/26 16:35 C:\WINNT\system32\spoolss.exe
0000002f 0000002e 01/26 16:35 C:\ADSM\baclient\dsmcsvc.exe
00000051 0000002e 01/26 16:35 C:\ADSM\baclient\dsmcsvc.exe
→ 00000058 0000002e 01/26 16:35 d:\notes\nSERVICE.EXE
→ 00000074 00000058 01/26 16:35 d:\notes\nSERVER.EXE
→ 00000090 00000074 01/26 16:36 d:\notes\nRouter.EXE
→ 000000a1 00000074 01/26 16:36 d:\notes\nReplica.EXE
→ 000000a3 00000074 01/26 16:36 d:\notes\nUpdate.EXE
→ 000000a5 00000074 01/26 16:37 d:\notes\nStats.EXE
→ 000000a9 00000074 01/26 16:37 d:\notes\nAmgr.EXE
→ 000000ab 000000a9 01/26 16:37 d:\notes\namgr.EXE
→ 00000043 00000074 01/26 16:37 d:\notes\nSched.EXE
→ 0000006a 00000074 01/26 16:37 d:\notes\nCalConn.EXE
→ 000000af 00000074 01/26 16:37 d:\notes\nAdminP.EXE
→ 000000b1 00000074 01/26 16:37 d:\notes\nReport.EXE
→ 000000b3 00000074 01/26 16:37 d:\notes\nEvent.EXE
→ 000000ad 00000074 01/26 16:37 d:\notes\nCldbdir.EXE
→ 000000c2 00000074 01/26 16:37 d:\notes\nClrepl.EXE
00000060 0000002e 01/26 16:35 C:\WINNT\system32\RpcSs.exe
0000006e 0000002e 01/26 16:35 c:\winnt\system32\pstores.exe
00000030 00000028 01/26 16:34 C:\WINNT\system32\lsass.exe
00000072 00000028 01/26 16:35 C:\WINNT\System32\nddeagnt.exe
00000084 00000080 01/26 16:35 C:\WNETFIN\NETFBASE.EXE
00000095 00000084 01/26 16:35 C:\WNETFIN\alertmgr.exe
→ 0000009a 00000084 01/26 16:35 C:\WNETFIN\monbase.exe
0000006b 00000084 01/26 16:35 C:\WNETFIN\CMBASE.EXE
→ 000000d0 00000054 01/27 15:17 D:\notes\NLNOTES.EXE

This process table from version 1.4 of NSD shows two server partitions and one client running. The current data directory processes are marked with arrows (->) and the other Notes and Domino processes are marked with asterisks (*).

    PID     PPID   UID     STIME    COMMAND

00000000 00000000 0 ??? [System]
00000008 00000000 0 ??? [System]
00000088 00000000 0 03/03 09:57 \SystemRoot\System32\smss.exe
000000b8 00000000 0 03/03 09:57 ??\C:\WINNT\system32\winlogon.exe
000004cc 00000000 1000 03/03 13:46 C:\WINNT\system32\cmd.exe

  • 0000059c 00000000 1000 03/06 12:12 C:\5\Notes\nlnotes.exe
  • 00000454 00000000 1000 03/06 12:14 C:\5\Notes\nhldaemn.EXE
    → 000004e4 00000000 1000 03/06 16:06 c:\4ie\domino\nserver.exe
    → 00000604 00000000 1000 03/06 16:06 c:\4ie\domino\nBilling.EXE
    → 00000370 00000000 1000 03/06 16:06 c:\4ie\domino\nReplica.EXE
    → 000005fc 00000000 1000 03/06 16:07 c:\4ie\domino\nRouter.EXE
    → 0000041c 00000000 1000 03/06 16:07 c:\4ie\domino\nUpdate.EXE
    → 00000198 00000000 1000 03/06 16:07 c:\4ie\domino\nStats.EXE
    → 00000544 00000000 1000 03/06 16:07 c:\4ie\domino\nAMgr.EXE
    → 00000290 00000000 1000 03/06 16:07 c:\4ie\domino\namgr.EXE
    → 0000053c 00000000 1000 03/06 16:07 c:\4ie\domino\nAdminp.EXE
    → 00000564 00000000 1000 03/06 16:07 c:\4ie\domino\nSched.EXE
    → 0000038c 00000000 1000 03/06 16:07 c:\4ie\domino\nCalConn.EXE
    → 000005f4 00000000 1000 03/06 16:07 c:\4ie\domino\nReport.EXE
  • 00000430 00000000 1000 03/06 16:07 c:\4ie\domino\nserver.exe
  • 000002bc 00000000 1000 03/06 16:07 c:\4ie\domino\nCladmin.EXE
  • 00000520 00000000 1000 03/06 16:07 c:\4ie\domino\nBilling.EXE
  • 000001a4 00000000 1000 03/06 16:07 c:\4ie\domino\nReplica.EXE
  • 000004e0 00000000 1000 03/06 16:07 c:\4ie\domino\nserver.exe
  • 00000554 00000000 1000 03/06 16:07 c:\4ie\domino\nRouter.EXE
  • 00000628 00000000 1000 03/06 16:07 c:\4ie\domino\nCladmin.EXE
  • 00000630 00000000 1000 03/06 16:07 c:\4ie\domino\nBilling.EXE
  • 00000638 00000000 1000 03/06 16:07 c:\4ie\domino\nUpdate.EXE
  • 00000648 00000000 1000 03/06 16:07 c:\4ie\domino\nReplica.EXE
  • 00000650 00000000 1000 03/06 16:08 c:\4ie\domino\nStats.EXE
    00000658 00000000 1000 03/06 16:08 C:\4ie\domino\thinkpad\nsd.exe

The Unix technique of searching for the words “fatal”, “panic”, or “segmentation” will not work. Instead, search for “except” or “error”. A sample crash stack follows, taken when NSD was run while the application exception dialog was displayed after the AeDebug Auto value was set to zero. “NtRaiseHardError” is the routine that displays that application exception dialog, and waits for OK or Cancel.

############################################################

thread 1/4: nimsgcnv pid=0x72, tid=0x41

############################################################
[1] ntdll.NtRaiseHardError+11 (12f468,12ffc0,456b20,0)
[2] nimsgcnv.__XcptFilter+330 (80000003,12f468,44c0ab,12f470)
[3] nimsgcnv._mainCRTStartup +249 (12f470,0,12f470,12ffb0)
[4] nimsgcnv.__except_handler3+75 (12f538,12ffb0,12f558,12f51c)
[5] ntdll.RtlConvertUlongToLargeInteger+66 (12f538,12ffb0,12f558,12f51c)
[6] ntdll.RtlRandom+191 (12f538,12f558,12f538,12f558)
[7] ntdll.KiUserExceptionDispatcher+14 (88880138,99e,2,12f9c8)
[8] nnotes.OSUnlockObject+422 (99e,12f9c0,12f9c4,2)
[9] nnotes.OSMemGetFaultHandle+24 (99e,99e,2,1a10008)
[10] nnotes.OSLockPool+18 (12fa00,99e,19c0242,99e)
[11] nnotes.NSFNoteSetInfo+99 (99e,2,12fa20,2697)
[12] nimsgcnv._NoteSetOrigin+50 (99e,42,19c0326,0)
[13] nimsgcnv._ICCopyMessage+195 (2697,12fa84,185ffff,19c0326)
[14] nimsgcnv._ICSplitMsgJ+82 (2697,0,2697,19c0242)
[15] nimsgcnv._ICConvertMsg+506 (2697,19c0242,2697,0)
[16] nimsgcnv._ICConvertMsgs+223 (2697,360000,493e0,0)
[17] nimsgcnv._ICHMTAShutdown +473 (2697,14a0640,4,7ffdf000)
[18] nimsgcnv._ICMain +1079 (458050,12fe4c,401082,400000)
[19] nimsgcnv._AddInMain@12+59 (400000,4,14a0640,616c0000)
[20] nimsgcnv._NotesMain@8+50 (4,14a0640,616c4624,17367)
[21] nimsgcnv._main+248 (0,0,0,4)
[22] nimsgcnv._main+23 (4,14b0d38,14b0d78,616c4624)
[23] nimsgcnv._mainCRTStartup +215 (616c4624,17367,7ffdf000,7ffdf000)
[24] KERNEL32.GetProcessPriorityBoost+279 (4478e0,0,0,77fa5aa0)

This is the summary from the equivalent annotated RIP file when QNC was allowed to debug the crash above. Note that the symbol files used in annotation allow the naming of an internal routine that was not exported from nnotes.dll. Also, argument addresses and values are listed in parentheses rather than in QNC stack frames. Finally, NSD offsets are in decimal, while QNC offsets are in hexadecimal.

→ _LockHandle@12+0116 ←

0012F9A8 0012F9C8 00506608 0000099E 0012F9C0 |…fP…|
0012F9B8 0012F9C4 00000002 |…|…q..|

→ _OSMemGetFaultHandle@4+0018 ←

0012F9C8 0012F9E4 00506B72 0000099E 0000099E |…rkP…|
0012F9D8 00000002 01B10008 00530A65 |…e.S…|

→ _OSLockPool@8+0012 ←

0012F9E4 0012FA08 00530AA3 0012FA00 0000099E |…S…|
0012F9F4 01AC0242 0000099E 01B1D188 00000014 |B…|
0012FA04 001C0242 |B…<…6B…|

→ _NSFNoteSetInfo@12+0063 ←
→ _NoteSetOrigin+0032
→ _ICCopyMessage+00C3
→ _ICSplitMsg+0052
→ _ICConvertMsg+01FA
→ _ICConvertMsgs+00DF
→ _ICHMTAShutdown+01D9
→ _ICMain+0437
→ _AddInMain@12+003B
→ _NotesMain@8+0032
→ _main+00F8
→ _main+0017
→ _mainCRTStartup+00D7

When NSD has been used as the default debugger, search for “Top of the Stack”. Here is a sample of what you’ll find.

############################################################

thread 1/4: [ ndecs:00d2:00d7]

FP=0012F2D4, PC=77F7D66E, SP=0012F2B8, stkbase=00030000, stksize=36864

############################################################

Got exception event: code=0xC0000005 (ACCESS_VIOLATION) from [ ndecs:00d2:00d7]

flags = 0x0

Address = 0x77F7D66E

Chain = 0x0

Access violation exception on write access to addr 0x8

---------- Top of the Stack ----------

0012F2B8 01900000 0CE614F0 00000000 10042B88 |…+..|

0012F2C8 0012F2E8 1002EC3A 00000009 0012F304 |…:…|

[ 1] 0x77F7D66E ntdll.RtlDestroyHeap+1268 (1900000,CE614F0,12F300,0)
[ 2] 0x77F64E17 ntdll.RtlFreeHeap+183 (1900000,0,CE614F8,1)
[ 3] 0x002796F2 NLCHTAPI+300786 (CE614F8,CE61F5C,26,12F3A8)
[ 4] 0x0026F9AA NLCHTAPI.HTErrorSetProcedure+30134 (16A74860,9,0,0)
[ 5] 0x0026ECC9 NLCHTAPI.HTErrorSetProcedure+26837 (1906D70,16A76CB0,F,0)
[ 6] 0x0026EA49 NLCHTAPI.HTErrorSetProcedure+26197 (1906D70,A97,0,0)
[ 7] 0x0024BB7A NLCHTAPI.HTDatabaseClose+89 (A97,0,0,0)
[ 8] 0x0040BF94 ndecs+49044 (167A,12F7E8,FFFFFFFF,72B13E0)
[ 9] 0x00402B56 ndecs+11094 (41206425,76697463,65522065,69546C61)
[10] 0x0040292B ndecs+10539 (400000,1,1800E74,0)
[11] 0x0040FA0F ndecs+64015 (1,400000,0,0)
[12] 0x0040FB16 ndecs+64278 (0,0,0,1)
[13] 0x0040FA36 ndecs+64054 (1,1A20BB0,1A20B50,0)
[14] 0x00410DD6 ndecs+69078 (0,0,7FFDF000,C0000005)
[15] 0x77F1B9EA KERNEL32.GetProcessPriorityBoost+279 (410D22,0,B0,100)

The User Environment section is the environment of the Command Prompt window running NSD, which may be different from the actual environment in which the server runs, depending on the user ID’s and the batch procedures used to start the Domino server and NSD.

Other sections of NSD output have standard system data that will not be explained here. Part of its value is that it is automatically collected whenever NSD is run. Output sections from the included Memcheck have much more Notes and Domino internal data, which will also not be explained here. See the examples for NSD and Memcheck output in the Online Resources document linked below for a summary of sections included and for complete output from sample runs on different Domino and Windows releases.

Related Documents:

NSD and Memcheck for Windows - Example Output
Document #: 21331

NSD and Memcheck for Windows - Selected Versions
Document #: 21332

Overview of NSD Analysis on UNIX
Document #: 157181

Memcheck: What Is It and How Does it Work?
Document #: 169549

UNIX NSD Described for NT Users: A Comparative Table for Reference
Document #: 161529

How to Set NSD as the Default Debugger in Windows
Document #: 190818

Are NSD and Memcheck Installed with Domino Releases 5.0.9 and Later?
Document # 189886

Other references:

How to Set or Remove QNC as the Default Notes Debugger in NT or Windows 95
Document #: 138694

Subject: FAQ - How to analyze Notes/Domino NSDs? Die Hard - Deciphering a server crash

From the Lotusphere 2008

a great presentation from

Franziska Whitlock Wave Technology

Paul Mooney Blue Wave Technology

Die Hard - Deciphering a server crash

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs? Information about NSD updates

Hi,

From http://www-10.lotus.com/ldd/nflsblog.nsf/dx/NSD_Updates

Information about NSD updates

Date

Topic

Update : Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs? NSD Express - Fault Analysis in 5 Minutes

Lotusphere 2008 - Emergence SpeedGeeking Session

NSD Express - Fault Analysis in 5 Minutes Daniel Nashed (Nash!Com)

Subject: FAQ - How to analyze Notes/Domino NSDs? NSD Analyzer V2.6

a new version of the Lotus Notes Diagnostic Tool (Nsd Analyzer) is available

Update may 14th 2008- Version 2.6

Date

Topic

2008-05-14

Update - Frequently asked question - How to analyze Notes/Domino NSDs? NSD Analyzer V2.6

From

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

Subject: FAQ - Determining the database being accessed at the time of a server outage

Hi,

Determining the database being accessed at the time of a server outage

Question How do you determine what database was being accessed during a Domino® server outage?

View this multimedia presentation for a high level overview of the basic Domino troubleshooting steps to identify the database that was being accessed by the fatal thread at the time of a server outage.

PDF format

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

Subject: FAQ - maintenance script from David Killingsworth using NSD

From David Killingsworth’s blog

Maintenance script using NSD features

JYR

Subject: FAQ - Memcheck memory analyzer

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? Memcheck memory analyzer (Jean-Yves Riverin)

Subject: FAQ - How to analyze Notes/Domino NSDs? Domino NSD file analysis on AS400, tip for people who do NSD analysis

From http://samurai.alise.lv/lotus.nsf/dx/domino-nsd-file-analysis-on-as400

If you have a crash of Domino on AS400, if you open it on you PC it looks like this.

if you you open it with Microsoft Word, it will suggest you to pick encoding,
just set IBM EDCDIC (International) and “Voir la”, you see a normal NSD not a garbage.
then just search specific keyword in IBM Support, and if your crash is not an IBM Internal Technote, you will know what was happen to your Domino.

Hope it will help some one. :slight_smile:

vlad

Subject: Update - FAQ - How to analyze Notes/Domino NSDs? NSD Analyzer V2.7

Where to obtain the Lotus Notes Diagnostic toolVersion 2.7 is available

ftp://ftp.software.ibm.com/software/lotus/tools/LND_2.7_Setup.zip (4028 KB)

Question

What is the Lotus Notes Diagnostic utility? Where can you download it?

Answer

Lotus Notes Diagnostic is an unsupported utility that you can download and use to automatically perform repetitive tasks associated with analyzing files from a Lotus® Domino® server crash, hang, or performance issues.

JYR

Subject: Update - FAQ - How to analyze Notes/Domino NSDs? How to determine what files were open in the Crashing Stack of an NSD

How to determine what files were open in the Crashing Stack of an NSD

You need to determine which files were open at the time of a crash within an NSD. In some NSD’s, the process stac, and memcheck virtual pool data can be used to build a search to find our answer.

http://www-1.ibm.com/support/docview.wss?uid=swg21257344

Subject: FAQ - How to analyze Notes/Domino NSDs? Outputting your custom applications callstacks in NSD

Outputting your custom applications callstacks in NSD

In the NSD Hands On Lab many customers asked how do we get detailed functions of our customer application in the NSD output. When NSD runs it annotates call stacks against LotusDomino.sym (symbol file). This produces functions and offsets to give further insight to what when and how the process crashed. After talking with Jim Rouleau when NSD runs it will look through all *.sym files in the Notes/Domino program directory. Now you can take the map file from your custom application, create a sym file from your map file. Once you have a custom sym file you can now get your custom applications annotated data in the NSD.

For more information check out Map2iSym.exe located in the CAPI Toolkit
http://www-12.lotus.com/ldd/doc/tools/c/7.0/api70ug.nsf/85255d56004d2bfd85255b1800631684/00cb00910049008485255e3d00774c2f?OpenDocument

Subject: FAQ - How to analyze Notes/Domino NSDs? How to locate the agent that crashes Agent Manager using the Lotus Domino NSD output file

Problem Is it possible to determine which agent crashed with just an NSD log from the Lotus Domino server?

How to locate the agent that crashes Agent Manager using the Lotus Domino NSD output file

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs? Third-party applications generate an NSD

Third-party applications generate an NSD

http://www-01.ibm.com/support/docview.wss?rs=463&context=SSKTMJ&context=SSKTWP&context=SSRNU3&context=SSRNUY&dc=DB560&dc=DB520&uid=swg21280032&loc=en_US&cs=utf-8&lang=en

Subject: FAQ - How to analyze Notes/Domino NSDs? Unix, System I, i5/OS, Semaphore

Title: Multimedia: Troubleshooting server crashes and hangs for Domino on IBM i or i5/OS

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21304574

Title: Best practices and problem recovery tips for IBM Lotus Domino on System i

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21293957

Title: Multimedia presentation: “Troubleshooting Lotus Domino server crashes on UNIX”

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21296420

Title: Multimedia presentation: “Troubleshooting semaphores and Lock Manager messages”

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21297031

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

Subject: FAQ - How to analyze Notes/Domino NSDs? Basic overview of troubleshooting Lotus Notes client crashes

Date

Topic

2009-01-23

FAQ - How to analyze Notes/Domino NSDs? Basic overview of troubleshooting Lotus Notes client crashes