Domino Server Hangs

HCL Domino Domino Forum
1

After Upgrading from R8.5.3 to R9 our one server have begin to become unresponsive a couple of times, but are normally coming to live again by it self.

In the log I can see:

[0E14:0004-0DCC] 04-04-2013 18:01:09 Compacting log.nsf (XXXXX’s Log), -S 10 [0F28:0046-08B8] LkMgr BEGIN Long Held Lock Dump ------------------ [0F28:0046-08B8] Lock(Mode=X * LockID(DB DB=C:\Program Files\IBM\Lotus\Domino\data\log.nsf)) Waiters countNonIntentLocks = 2 countIntentLocks = 0, queuLength = 20 [0F28:0046-08B8] Req(Status=Granted Mode=X Class=Manual Nest=0 Cnt=1 [0F28:0046-08B8] Tran=0 Func=N/A m\lkmgr.cpp:163 [0E14:0004-0DCC]) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:181 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00EC-0938] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0041-06A0] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0063-0454] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:007B-07F0] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0074-13D8] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00A4-1390] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0062-0D3C] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00AD-1264] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:008A-1318] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00A7-1190] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00D9-0B90] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:007A-122C] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:008B-0918] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00B1-0EE8] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00B2-0A4C] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:00D6-0D74] Delay=0min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0080-0EA4] Delay=0min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0067-0858] Delay=0min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:005D-13C8] Delay=0min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A nsfsem3.c:2160 [0F28:0069-10A8] Delay=0min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] nsfsem3_c:2156 [0F28:0046-08B8] Req(Status=Granted Mode=S Class=Manual Nest=1 Cnt=1 [0F28:0046-08B8] Tran=0 Func=N/A dbrrv.c:2704 [0E14:0004-0DCC]) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] nsfsem1_c:1516 [0F28:0046-08B8] dbrrv_c:2695 [0F28:0046-08B8] dbfixup_c:17338 [0F28:0046-08B8] dbfixup_c:680 [0F28:0046-08B8] LkMgr END Long Held Lock Dump ------------------ [0C58:0002-0EC4] 04-04-2013 18:03:22 Access control is set in catalog.nsf to not allow replication from XXXX/OU/ORG catalog.nsf Some normal lines [0E14:0004-0DCC] 04-04-2013 18:03:23 Compacted log.nsf, 0K bytes recovered (0%), -S 10 Some normal lines [0C58:0002-0EC4] 04-04-2013 18:04:33 Starting replication with server XXXX/OU/ORG [0F28:0046-08B8] LkMgr BEGIN Long Held Lock Dump ------------------ [0F28:0046-08B8] Lock(Mode=SIX* LockID(DB DB=C:\Program Files\IBM\Lotus\Domino\data\cldbdir.nsf)) Waiters countNonIntentLocks = 1 countIntentLocks = 0, queuLength = 7 [0F28:0046-08B8] Req(Status=Granted Mode=SIX Class=Manual Nest=0 Cnt=1 [0F28:0046-08B8] Tran=0 Func=N/A dbclose.c:652 [0F28:0062-0EA4]) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:181 [0F28:0046-08B8] dbclose_c:257 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [11BC:0002-0408] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [0F28:00AD-0D74] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [0ABC:0002-0AAC] Delay=2min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [0B64:0002-0B48] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [049C:0002-10DC] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [0E68:0002-0FD8] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] Req(Status=Waiting Mode=S Class=Manual Nest=0 Cnt=0 [0F28:0046-08B8] Tran=0 Func=N/A dbopen.c:4174 [1004:0002-1248] Delay=1min) [0F28:0046-08B8] rm_lkmgr_cpp:2121 [0F28:0046-08B8] rm_lkmgr_cpp:1354 [0F28:0046-08B8] nsfsem1_c:585 [0F28:0046-08B8] dbopen_c:858 [0F28:0046-08B8] LkMgr END Long Held Lock Dump ------------------ [0C58:0002-0EC4] 04-04-2013 18:30:35 Access control is set in catalog.nsf to not allow replication from XXXX/OU/ORG catalog.nsf Some normal lines [0F28:008B-0EE8] 04-04-2013 18:30:35 Closed session for XXXXY/OU/ORG Databases accessed: 1 Documents read: 0 Documents written: 0 Transactions: 3 [0F28:000B-089C] 04-04-2013 18:30:35 Error connecting to server XXXXX/DEN/SArmy: Remote system no longer responding [0120:0002-0F24] 04-04-2013 18:30:35 Admin Process: Searching Administration Requests database
Is there any idea what happening ?

/Leif

2

Subject: Very Bad Practice Compacting Domino System Databases while Domino Server up and running

I see you are compacting log.nsf and cld= bdir.nsf… Compacting any Domino System database while Domino Server up and running is highly discouraged because it can result in the behavior you are experiencing… My guess is you probably have a program record schedul= ing the compacts…

I suggest looking into utilizing the new DBMT (Database Maintenance Tool) in Release 9.0 for non-system databases and the new Compact Replication feature in Release 9.0 for system databases… See documentation as well as much more detail in my IBM Connect Presentation

ID107 - Deploying IBM Lotus Domino=AE 9.0 in Your Production Environment

Believe under Track 6 - Infrastructure

Connect Presentations can be downloaded here…

<>

Dear Colleague,
I’m excited to let you know that sessio= ns from IBM Connect 2013 are available online at SocialBizUG.org, including presentations from the Opening General Session!

IBM Connect combines deep technical content with the learning you need to drive real business values with social and collaborative technologies. Offering hundreds of technology and business sessions, Connect content empowers you to take your skills to the next level. Learn more <>

Connect 2013 session presentation materials <> are available, including:

  • Smarter Workforce
  • Smarter Commerce - Create an Exceptional Customer Experience
  • Lotusphere Technical Sessions
  • Track 4 - Strategy and Innovation
  • Track 5 - Application Development
  • Track 6 - Infrastructure View session det= ails
  • Track 7 - Best Practices
  • Track 8 - Customer Case Studies (from an IT perspective)
  • Track 10 - JumpStarts and Master Classes
  • Track 11 - Show 'n Tell<= a href=3D"http://tracking.priorityrequest.com/track?type=3Dclick&eas=3D= 1&mailingid=3D32709&messageid=3D31500&databaseid=3D31303&se= rial=3D16805183&emailid=3Djpaganet@us.ibm.com&userid=3D1=5F3703&= ;fl=3D&extra=3DMultivariateId=3D&&&2004&&&http:= //www.socialbizug.org/wikis/home?lang=3Den-us#!/wiki/IBM%20Connect%202013/p= age/IBM%20Connect%202013" target=3D=5Fblank>Visit SocialBizUG.org now to view session details and get started<= font size=3D2 face=3D"Arial">! SocialBiz User Group is proud to offer you this educational opportunity, and we hope that you find it beneficial. And, stay tuned because we will be adding even more Connect 2013 content in the coming weeks, including audio recordings and video of select presentations and even more sessions!
    Feed your brain,

Julia Weisman

Community Manager

Regards, John Pag jpaganet@us.ibm.com

3

Subject: Very Bad Practice compacting SYSTEM databases with Domino Server running

I see compacting log.nsf and cldbdir.nsf= … Compacting any Domino system database with Domino Server up and running can lead to this type of behavior… This is not new to Domino 9.0…

You probably have a program record to do the compaction…

Suggest you look into new DBMT (Database Maintenance Tool) in Release 9.0 to perform schedule maintenance (compact) on Non-System Databases and look at new Compact Replication Feature in Release 9.0 for the System Databases.= … See documentation and also suggest going to SocialBizUG.org and findi= ng my presentation ID107 - Deploying IBM Lotus Domino=AE 9.0 in Your Production Environment which has more details about both.
<>

Dear Colleague,
I’m excited to let you know that sessio= ns from IBM Connect 2013 are available online at SocialBizUG.org, including presentations from the Opening General Session!

IBM Connect combines deep technical content with the learning you need to drive real business values with social and collaborative technologies. Offering hundreds of technology and business sessions, Connect content empowers you to take your skills to the next level. Learn more <>

Connect 2013 session presentation materials <> are available, including:

  • Smarter Workforce
  • Smarter Commerce - Create an Exceptional Customer Experience
  • Lotusphere Technical Sessions
  • Track 4 - Strategy and Innovation
  • Track 5 - Application Development
  • Track 6 - Infrastructure View session det= ails
  • Track 7 - Best Practices
  • Track 8 - Customer Case Studies (from an IT perspective)
  • Track 10 - JumpStarts and Master Classes
  • Track 11 - Show 'n Tell<= a href=3D"http://tracking.priorityrequest.com/track?type=3Dclick&eas=3D= 1&mailingid=3D32709&messageid=3D31500&databaseid=3D31303&se= rial=3D16805183&emailid=3Djpaganet@us.ibm.com&userid=3D1=5F3703&= ;fl=3D&extra=3DMultivariateId=3D&&&2004&&&http:= //www.socialbizug.org/wikis/home?lang=3Den-us#!/wiki/IBM%20Connect%202013/p= age/IBM%20Connect%202013" target=3D=5Fblank>Visit SocialBizUG.org now to view session details and get started<= font size=3D2 face=3D"Arial">! SocialBiz User Group is proud to offer you this educational opportunity, and we hope that you find it beneficial. And, stay tuned because we will be adding even more Connect 2013 content in the coming weeks, including audio recordings and video of select presentations and even more sessions!
    Feed your brain,

Julia Weisman

Community Manager

jpaganet@us.ibm.com

4

Subject: RE: Domino Server Hangs

It seems that http://www-01.ibm.com/support/docview.wss?uid=swg21470902 http://www-01.ibm.com/support/docview.wss?uid=swg21470902 are fixing it even the article is for 8.5.2

Waiting a couple of days to see if it really is fixed

5

Subject: Very Bad Practice to compact SYSTEM databases with Domino Server up

I see log.nsf and cldbdir.nsf being compacted and that’s bad news…

I don’t believe this is new behavior to 9.0… Two things you should look at in 9.0 are the possible use of the new DBMT tool to schedule your maintenance on non-system databases (also make sure your Backups are scheduled to happen after any copy style compact as the DBIID changes) while leaving Domino Server up and also look at scheduling maintenance of you System Databases with Domino Server down or try using new 9.0 Compact Replication Feature for System Databases to minimize Domino Server Down Time…

Check out this presentation for more details on both DBMT and Compact Replication

6

Subject: Solution

Shut down your domino server.

Delete your log, rename your mail(n).box files to mail(n+5).box, also delete your transactional log directory contents if you have one.

Go to your notes.ini. add.

CREATE_R85_DATABASES=1
CREATE_R85_LOG=1

Go to your dos console, go to your Domino folder where ncompact.exe is

run on the console “ncompact -c -v -n -*” without target. (Very important, wait for it to finish! it will say it is finished, also it will show error messages, 10GB an hour if you are unlucky and have disk fragmentation.)

run it twice, (ods changes)

now run a “nfixup -J -L -F” without target.

now run “nUpdall -R” without target, this is just a view opimization.

Start the Domino Server, it will take a while to build mail.boxes and log.nsf, maybe also you transactional log.

Sincerely

Douglas Bryant

email@dbryant.de

www.dbryant.de http://www.dbryant.de