I have a Connections environment which had problems with the Component pack part. I fixed those problems, but on the WebSphere side the messaging stores filled up. The WebSphere logs now show:
CWSIS1574E: The file stores permanent store file is full
Should the message store empty itself once events can be delivered again, or should I take other actions?
I've enlarged the permanent file store to 600 MB and restarted the WebSphere clusters, but so far the queues don't seem to be emptying.
Hi Martijn,
We had the same problems last year, same error message. We could not find a cause.
The HCL support suggested to delete the 3 filestore files (Log+PermanentStore+TemporaryStore) for the messaging engine and restart the Connections servers.
The required filestore files will be created automatically during startup.
<Connections>/data/shared/messageStores/<Cluster>/log/Log
<Connections>/data/shared/messageStores/<Cluster>/store/PermanentStore
<Connections>/data/shared/messageStores/<Cluster>/store/TemporaryStore
(Delete only the files but not the directories)
This worked for us, since then the error message has not appeared. But as I said, we still don't know what caused it.
Regards,
Daniel
Hello Martijn,
the message store just runs out of space, when the system create/produce more messages then it can be consumed at a specific timeframe. The Application server is no database for messages and best practice is to keep the queue and topic size as low as possible. If your message store run out of space, then the first thing that you need to check is the queue and topic size. If you find out the queue and topic in question that is causing the problem you need to find out next, which application belongs to it. Furthermore I think it make sense to check the Activation Specification and ListenerPort settings for that application to find out how much messages can be consumed at a time. You maybe need to tune the batch-size/maximum concurrent messages) for that to ensure that messages can be consumed faster. If you need further help with that feel free to open a case with us and/or with IBM. I worked for the WAS-messaging support team in the past. With detailed messaging traces enabled it should be possible to find out more.
Dropping the message store files should be just a workaround for the moment. But I am quite sure that you still run into that problem in the future. So I think it is better to find out the root-cause of it. It could be also possible that you still have a message producing application up and running and that the consuming application is currently stopped. With that I would also assume that the message store runs out of space after a certain time.
By the way... If your message store already run out of disk space it is possible that it is already corrupted. For that it make sense to stop the server and to delete the filestores as described above from Daniel. If you run your message store on a database, you need to drop your sib-tables.
Which are usually the following tables:
SIB000
SIB001
SIB002
SIBCLASSMAP
SIBKEYS
SIBLISTING
SIBOWNER
SIBOWNERO
SIBXACTS
I hope that helps,
Thanks, Thorsten
Hi Martijn
We had exactly the same issue recently and even raised a support ticket. After short investigation the support engineers came with the following recommendations:
There are suggestions for setting the message store sizes in the performance tuning guide kindly review these against your current settings: https://ds_infolib.hcltechsw.com/ldd/lcwiki.nsf/dx/IBM_Connections_V6_Tuning_guide
Please also check from the WAS integrated console:
Buses > Bus_Name > Messaging engines > Messaging engines_Name >Message store
Buses > Bus_Name > Messaging engines > Messaging engines_Name > Queue points
If the issue still persists another cause of the errors reported is a corrupt message store where in most cases deleting the contents of the message store directory addresses the issue:
NOTE: however that you may loose status update data:
Steps to clear out the message store:
1. Check the WAS variable, MESSAGE_STORE_PATH, for the correct path to the messageStores. (Ex. /opt/IBM/Connections/data/shared/messageStores)
2. Shutdown the Connections servers
3. Delete only the files from the
3 MESSAGE_STORE_PATH directories noted below, not the directories themselves: (you can also instead move the contents of the directories to a location outside such as a temp directory) /<MESSAGE_STORE_PATH>/<cluster>/log/Log /<MESSAGE_STORE_PATH>/<cluster>/store/PermanentStore /<MESSAGE_STORE_PATH>/<cluster>/store/TemporaryStore
4. Restart the Connections servers
Thanks all for your answers. In the end, it turned out I was just impatient.
As said, I knew what caused the problem. I described the cause and solution in this blog. This problem, however, had caused a different problem. The permanent message store was full, and it can't process the events if it's full already. I therefore had to create space first. The place to do that is Buses > ConnectionsBus > Messaging engines > <your clusters>-ConnectionsBus > File store.
After making the permanent store larger and ripplestarting the Connections clusters that are using the Connections busses, the queues started to empty. This just takes a very long time, and you don't see much happening at the beginning. I left it for the night and this morning the queues were empty and people's homepages were showing updates again.
I hope this explanation of my situation will help someone in the future.