Strange. Two AIX 5.3 boxes running fine on 6.5.3FP1. I upgraded them to 6.5.4FP2 and they would both run for about an hour and then fail. I opened a PMR with IBM and agreed to try 6.5.5 which didn’t resolve the issue, so I reverted back to 6.5.3FP1 which also didn’t resolve the issue.We also upgraded to the latest AIX version of Symantec anti-virus, but the crashes still occured with the AV disabled in the NOTES.INI and also after the AV was uninstalled.
Frustrating as these boxes were rock-solid before the upgrades.
Subject: AIX crashing after upgrade to 6.5.4FP2 and 6.5.5
I am about to do the same upgrade. Do you have an SPR yet? If you could either post or email me at robATtechnoticsDOTcom if you find a cause or resolution.
Subject: RE: AIX crashing after upgrade to 6.5.4FP2 and 6.5.5
No resolution or SPR yet. We did restore the binaries from tape and applied the latest and greatest AIX patches per IBM’s technote. Last night I ran compact -c against all databases with the server down and this morning we made some INI changes for memory management, but the server still fails.
I don’t know at this point if a restore of the databases will resolve the issue, but we still have two servers that won’t run. I’m cancelling the rest of the upgrades until we get this figured out. We’ve been working with IBM’s second and third level support on the phone, as well as having some of their management involved, but we are still collecting data and sending it for analysis so I’m not optimistic that we are close to a solution.
Also, my advice for everyone is to clearly understand the SLA of your support from IBM for sev 1 outages (especially on the weekend).
Subject: Re: AIX crashing after upgrade to 6.5.4FP2 and 6.5.5 - Resolved
We replaced the names.nsf again and ran maintenance against all databases. It appears that we had corruption not only in the names.nsf on the server, but also in the names.nsf on other servers that we used to replace the names.nsf on this server. Confusing, huh? Basically we were replacing a corrupt NAB with another corrupt NAB. Once maintenance was run the root cause was resolved but the subsequent crashes were caused by corrupt mail files (probably caused by the server crashing since we had already run maintenance against them).We brought down all production servers and ran maintenance against the NABs so we should be squeaky clean for awhile.
Subject: RE: AIX crashing after upgrade to 6.5.4FP2 and 6.5.5
Hi Paul,
Can you confirm that you have now gone to 6.5.5 with your aix servers and everything is running OK? I’m thinking of doing a similar upgrade and any information would be helpful.
Basically we were replacing a corrupt NAB with another corrupt NAB. Once maintenance was run the root cause was resolved but the subsequent crashes were caused by corrupt mail files (probably caused by the server crashing since we had already run maintenance against them).We brought down all production servers and ran maintenance against the NABs so we should be squeaky clean for awhile.