We are upgrading our old Domino 9.0.1 platform that is now out of support to 11.0.1 (unfortunately at the time of planning our vendors app is not approved to run on Domino 12 or 14 before anyone asks :) )
My question is this. Since all planning was done in 2022 and our cloud platforms have changed and we now only have Veeam as a backup solution. I have had discussions with our supplier for the Veeam product and they have advised that they cannot backup open nsf files. They also do not have anything on their horizon to create a backup agent for Domino. We do not have an option to use any other product. We are now also looking to deploy the platform to Azure. What options do I have to ensure I can restore the platform.
Our current solution utilises additional servers that replicate consistently and shutdown automatically at a predefined time we then backup the nsf files in an offline mode to ensure the nsf files are NOT corrupted. Then the platform will start up and start replicating immediately. No users can connect to the platform at all even in a DR scenario. This is extremely risky and can result in large amounts of data loss in the event of database corruption after a 12 hour shift. The manual task of recovery will take around two weeks to recover.
We are in a real predicament and need to find a better backup option than we currently have.
NOTE: We DO NOT use the platform for any mail, only for document management.
Unfortunately we cannot yet upgrade to v12, our vendor is still qualifying their application against the environment.
I have also glanced over the documentation for v12 and it seems there is little to no benefit for the system we run placing it on the version as there is no change to the backup method and the use of agents to backup the nsf files.
We run three independent Domino Clusters to handle the workload. Each of these backup servers sits outside of one of those clusters doing constant replication. The problem is the time taken from shutting the (what I call) backup servers down to restarting them. It used to be a lot simpler with an agent, however backup vendors seem to be getting less and less given that only one is currently supporting Domino Agents. I want to investigate the possibility of backing up the platform instead of once a day to every 4 hours. The data folders storing the nsf files are rather large.
Given that Veeam have such a large footprint in the backup market and there are still a fair amount of Domino platforms out there in my opinion it would make sense for them to invest in potentially finding a solution to backup nsf files.
I would also like to know from HCL's perspective what can be done about the backup and restore issues if anything of course.
"Unfortunately we cannot yet upgrade to v12, our vendor is still qualifying their application against the environment."
I understood that, but since noone accesses the applications on the backup server and there are just stored databases there, why do you need their approval for V12?
"I have also glanced over the documentation for v12 [...]"
The documentation isn't bad, but have an additional look at @Daniel Nashed's blog:
With transactions logging, DAOS and the new online backup option in V12, you could implement a really great backup procedure - even with point-in-time-recovery: restore the mail or application database just a minute before the user deleted hundred of documents (and emptied the trash).
Consider joining the backup servers to the clusters. I'm 100% sure, this is supported, but can still ask HCL. Again, since nobody accesses the applications on the backup servers, your vendors' okay shouldn't be needed.
Domino 12.0.1 introduced a Veeam integration for Domino. And for Domino on Windows I would recommend updating to Domino 12.0.2 to use the new native VSS Writer Support.
You find details about it here --> https://opensource.hcltechsw.com/domino-backup/backup-providers/veeam/install_vss_writer/
You should really look into Domino 12.0.2 or directly make the move to Domino 14.0. There are a lot of enhancements and updates starting with Domino 12.0.
I am not sure which type of application you are using (Web or Notes) but there are many benefits to move to a new version and it would be definitively worth looking into updating.
When the application is working with 11.0.1, it is very likely to work with 12.0.2 and 14.0. Check with your vendor of your application.
If a short downtime at night would be OK, you could also stop your Domino servers, take a snapshot and start them again to get a consistent backup. But Domino 12.0.2 Veeam integration goes beyond just supporting a consistent backup. You can restore databases directly from a Notes front end with all kind of restore options.
I assumed you are running on Windows. Domino Backup is also available on Linux and has a Veeam integration for Linux as well. Just without the VSS Writer option we have on Windows since Domino 12.0.2.
You really should look into Domino 12.0.2 or higher support with your vendor.
I almost missed the question about Azure.
Even this is a Microsoft platform they are not fully supporting their own VSS implementation completely. They don't support application consistent snapshots. You could still take a snapshot with VSS and Domino Backup would generate delta files, which would need to be backuped separately and restored as well.
There is no integration for Domino, because there is no good way to restore programmatically. On Azure you have to bring your own backup. Azure mainly uses snapshots.
Without more information about what platform you are on (Windows/Linux), how much data you have (NSF, DAOS?) it is hard to give you another recommendation.
Domino on Windows with VSS Backup there is a defined interface. On Linux there are a couple of interesting options I am working on.
We have to deploy on Windows 2019 again this is an organisation and Vendor limitation unfortunately.
I will follow up with our vendor to see if we can deploy to Domino 12.0.2 as opposed to 11.0.1 this sounds like it could be a viable option. Thank you @Daniel Nashed and @Thomas Bahn for your insights on this. I only looked at 12.0.1 these changes for Veeam seem to have been introduced in version 12.0.2.
On our one platform the app has around 24 nsf files per folder and we have 2 folders totaling 48 nsf files. The total capacity on the data drive is around 300Gig, biggest nsf file is around 4.5G. On our second platform the app is way larger which has 14 folders of 24 each totalling 336 nsf files and is data capacity is considerably larger.
Some nsf files are just archive files with little to no agents in then, operational nsf files can have a number of agent running in them performing complex tasks. This is what makes backup so complex.
There is no way to shut the servers down as the service runs 24x7 and is a critical component in the supply chain. It may mean we would potentially still need to keep the backup servers as those can then be shutdown or placed in backup mode as suggested in the documents and we can backup more frequently.
The first Veeam integration has been implemented for Domino 12.0.1 Windows and Linux.
We always wanted true VSS Writer support to simplify backup using this standardized interface. Once we implemented that in Domino 12.0.2 I looked again into the Veeam integration on Windows and simplified the backup part -- which now will not need any special integration scripting.
OK if no downtime is possible, you need an on-line application. Veeam works very well in this scenario with Domino 12.0.2 leveraging the VSS Writer support.
You have not answered my question about DAOS and other optimization. Usually DAOS reduces the size of the databases dramatically -- depending on the application.
And if you have applications with large view indexes, NIFNSF can help to keep the NSF files smaller. Domino online backup is only required for the NSF data. DAOS can be backed up separately online without a backup agent.
See this document for some details of how storage optimization can help you to reduce the backup space needed -- specially for the online backup of NSF files.
I don't know your application, but you should have a holistic view on your storage, backup setup.
For a server with this amount of data you should look into updating to Windows 2022. We have seen some issues with the file-system cache on Windows 2019, which could lead to performance issues. So if you look into updating to Domino 12.0.2, I would consider updating to Windows 2022 as well.
If your environment is that critical for your business you should consider working with a business partner to work out the best way specifically for your environment.
There is no one size fits everyone and it sounds like you would benefit from a consultant who is an expert in this field.
I am located in Germany but I would help in any region of the world. but I will also try to give you tips and point you to the right resources if you want to work it out on your own.
Daniel Nashed
HCL Lifetime Ambassador & IBM Lifetime Champion Nash!Com - Daniel Nashed Communication Systems https://blog.nashcom.de