I am just going through the component pack upgrade and I have to say for something that was supposed to be much easier in this release I am not convinced. I wanted to see if any one has actually completed an upgrade as I am going through it and upgrading the PVs basically the first step is failing, it appears that the naming conventions have changed and been hard coded to :-
- /pv-connections/customizations with permissions 0005
- /pv-connections/esbackup-7 with permissions 0700
- /pv-connections/esdata-7-0 with permissions 0700
- /pv-connections/esdata-7-1 with permissions 0700
- /pv-connections/esdata-7-2 with permissions 0700
- /pv-connections/esmaster-7-0 with permissions 0700
- /pv-connections/esmaster-7-1 with permissions 0700
- /pv-connections/esmaster-7-2 with permissions 0700
- /pv-connections/mongo-node-0 with permissions 0700
- /pv-connections/mongo-node-1 with permissions 0700
- /pv-connections/mongo-node-2 with permissions 0700
When the previous was :-
/pv-connections/esdata-{0,1,2}
sudo mkdir -p /pv-connections/esbackup
sudo mkdir -p /pv-connections/customizations
sudo mkdir -p /pv-connections/mongo-node-{0,1,2}/data/db
sudo mkdir -p /pv-connections/solr-data-solr-{0,1,2}
sudo mkdir -p /pv-connections/zookeeper-data-zookeeper-{0,1,2}
This is bad enough but when I have looked at changing the names to the above and running the upgrade Helm commands it is actually looking for es-data-7-0 etc so the documentation seems to also be incorrect.
I am hoping someone has done this and got through it already.
Hello,
when I understand the documentation correctly, the migration of data from Elastiscearch 5.5 to 7 is a manual process so you setup a new Elasticsearch 7 cluster (where you need new persistent volumes) and then migrate the data.
So the changer in naming makes sense.
https://help.hcltechsw.com/connections/v7/admin/install/cp_install_services_tasks.html#cp_install_services_tasks__section_grm_x5w_tnb
-> Enable Metrics with Elasticsearch
Martin
Yes, the Elasticsearch index needs to be updated, so you can't access a index of ES 5 with 7. Updates need to be done for major versions, so 5.5.1 -> 6.8 -> 7 when I understood it right.
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
I expect to get a tech note on this, because doing this with Search Guard (the tool to add certificate based authentication to ES) by hand will be a nightmare.
- /pv-connections/esbackup-7 with permissions 0700
- /pv-connections/esdata-7-0 with permissions 0700
- /pv-connections/esdata-7-1 with permissions 0700
- /pv-connections/esdata-7-2 with permissions 0700
- /pv-connections/esmaster-7-0 with permissions 0700
- /pv-connections/esmaster-7-1 with permissions 0700
- /pv-connections/esmaster-7-2 with permissions 0700
- /pv-connections/mongo-node-0 with permissions 0700
- /pv-connections/mongo-node-1 with permissions 0700
- /pv-connections/mongo-node-2 with permissions 0700
This can be written like:
sudo mkdir /pv-connections/esdata-7-{0,1,2}
sudo chmod 0700 /pv-connections/esdata-7-{0,1,2}
and so on.
I would recommend doing auto provision persistent Volumes in Kubernetes. The NFS is a single point of failure, so it's worth to think about storage a little bit more.
Or you just look into the Ansible scripts released yesterday, there you will find the code for making that pvs.
Christoph
Thanks for your replies that makes more sense however even with that, the documentation is still incorrect as it is telling us to create new PVs
- /pv-connections/esbackup-7 with permissions 0700
- /pv-connections/esdata-7-0 with permissions 0700
- /pv-connections/esdata-7-1 with permissions 0700
- /pv-connections/esdata-7-2 with permissions 0700
- /pv-connections/esmaster-7-0 with permissions 0700
- /pv-connections/esmaster-7-1 with permissions 0700
- /pv-connections/esmaster-7-2 with permissions 0700
But when then running the PV upgrade you get the following UPGRADE FAILED: no PersistentVolume with the name "es-master-7-2" found
So as you can see the naming convention appears to be wrong or am I again missing something... The way this is going it is possible :)
I haven't done this until now, but after creating the nfs share, you need to tell Kubernetes that there is one. So creating a PV in Kubernetes, so that the PVC can be fullfilled.
So:
helm upgrade connections-volumes component_pack_installation_folder/hybridcloud/helmbuilds/connections-persistent-storage-nfs-*.tgz -i -f
component_pack_installation_folder/hybridcloud/examples/multi_domain_environment/connections-volumes.yml
this is generating the missing PV?
I read through this: https://help.hcl-software.com/connections/v7/admin/install/cp_install_services_tasks.html and I think the NFS configuration is missing. So they create the folders for NFS, but the part to update /etc/exports seems not to be mentioned. Have you exported the new shares?
It is when I run the upgrade task you have listed above is when I see the error but I think you are correct there is nothing in the awful sample steps that they have put in the documentation regarding running etc/exports as you would if this was a new build. I will take a look at that next and see how far I get this time.
I have been putting this off after my last epic fail but have now had a chance to revisit it, reading through all the documentation again it appeared the best course of action for an in place upgrade is to uninstall everything leaving the Persistent Volumes intact as to preserve the data then start the install from scratch. This actually seems to have worked, however as predicted because of the change to Elasticsearch I have none of the data from the original elasticsearch data so in effect the data for search and metrics is now started from scratch.
This is fine for our dev environment but no good for customers so the next question is has anyone upgraded an existing install and managed to convert the data from the previous version of elasticsearch? I still have the data as it is sat in the old PVs so I am assuming it needs converting and migrating to the new PVs?
Any help is much appreciated as I feel like I have got further along but not enough for a customer based upgrade unless they don't care about old metrics data etc.
Hi everyone,
Connections 7 (and Component Pack 7) have a couple of changes when it comes to PVs and PVCs:
- Solr is not needed anymore (not mandatory actually from 6.5.0.1 as ES is used for OrientMe in its place). In CP in version 7 it is not even supported officially anymore, hence for new installs no folders for PVs needed. I've just rechecked, both Solr and Zookeeper (which was dependency for Solr) are not at all in the zip file of CP7.
- ElasticSearch 5 is very different then ElasticSearch 7. This has nothing to do with Connections or Component Pack, but point is you can not simply install new version, share the data (or in this case point to the same data folder) and continue using old data.
Saying this, how this reflects to the users:
- Users starting fresh follow this documentation from the first post, create those folders from the first post, and use Helm charts from v7.
- Users upgrading still do the extra steps, because they are upgrading; this means, they will continue with ES5, they will install ES7 as well, and they will then migrate to ES7 and at some point, once migrated, remove ES5 data and pods fully as it won't be needed anymore.
What is actually migrated from ES5 to ES7 data wise? Only two things: quick results and metrics data. Quick results, from my personal point of view, can be ignored, as it is going to be regenerated. Metrics are different beast, and in case someone doesn't need old metrics, then it is no brainer - drop ES5 fully, install only ES7 and you are good to go. This is how far I can go, since I, myself, didn't do data migration from 5 to 7 (simply don't have big enough datasets for something like that) but if there is some problem, I probably can help, both externally and internally to get it fixed.
And some more context now about why there is (are) extra folder(s) for ES7. With ES5, you were asked to create only folders for PVs for data pods. This was wrong on the first place, because data pods cluster with master pods. On pods recreation, data pods would keep the state, because they are keeping it in PVs, but master pods would not know about that, and you would hit the issue that cluster is in broken state because data pods kept one cluster ID while master pods have now very different one, as from their point of view they are forming new cluster. This is easily fixable but annoyance. So for v7 (where this issue is even more prominent if you go that way) also master pods got their PV, simply to preserve cluster state and cluster id, nothing more.
And the question regarding documentation and NFS - is it really necessary to mention that, if NFS folders need to be created, they also need to be exported and accessible? I can recheck the documentation, but this is, for me at least, quite obvious thing. You need folders, you need to export them, they need to be mountable, and then you use Helm charts which are going to use those same folders to create PVs and PVCs. Not sure really where is confusion coming here?
Thanks for the post it does indeed give a better explanation.
Your comment “ but this is, for me at least, quite obvious thing.” does however show what the issue is with the documentation. If you use Or have used in other instances Docker, Kubernetes, helm etc in the past outside of Connections I can see how this might indeed be obvious but for a lot of people a first venture, and probably their only venture even now, into this technology was Connections where IBM / HCL gave a detailed step by step set of instructions on how to do this. It achieved what most needed the ability to spin up the component pack and use it with little issue. When an upgrade appeared, again a step by step set of instructions on how to do this with no assumptions. I would say the last couple of releases have started making assumptions that everyone knows the architecture which unfortunately is a massive assumption. If HCL are not going to provide this type of documentation then that is up to them, however I would say that there needs to be explanations at least, much like you have provided above, not telling us how but to just say it needs to be done and why, that way someone doesn’t fall into a trap of thinking they can do it the same way they have for the past steps of an upgrade.
Losing Metrics data for me is fine but for a few of my customers this would be a massive blow but they are also in the SMB bracket which means they also don’t have the means to have a vast component pack infrastructure which is why an in place upgrade has always been the choice and only real option for them.
Hi Richard,
You are probably talking to the right person, as I am behind cleaning up/removing those specific instructions. Everything that you wrote about is true, however, HCL doesn't provide instructions on how to install and configure Linux, Windows, set up the disks inside the server, and all that stuff is way more simple and less change prone then Kubernetes and any of the things you mentioned. On top of that, from the CNX 7, we support latest three versions of Kubernetes all the time, each of our customers so far have different NFS implementation, AND we support Component Pack on Amazon EKS and RedHat OpenShift. And last but not least, for example, Kubernetes is going to drop the support for Docker itself. We would be working full time just 'supporting' all those use cases. However, for those who still want this, we did open sourced our own automation which is used on a daily basis by different teams within HCL, and customers can, of course, use that to get it all set up if they want. Writing step by step guide on how to install this on that is just not the business of HCL, and that can be taken to the product management probably, because maybe they will see that type of support as an additional business model, but that's not my cup of tea.
Now regarding metrics data. I'm not sure how we came to losing metrics data, and what is the issue here? I did write that, if someone doesn't need the data, can skip migration and go directly to ES7, if the data is needed, then migration needs to be performed. Are you saying that migration is not working? Is there something specific there that I am missing?