How to do an off-site backup of the Zextras backup files now that full scan is gone

[Cine]

Hello again!

I've ran some tests and checked the logs you sent, and I confirm that there's an issue in ZxBackup's scheduling engine which prevents the SmartScan to be properly started on the day the Purge is scheduled.

I've passed all the info on to the devs, and a fix will be available in ZeXtras Suite 1.10.1, while in the meantime you can make up for the missing run by scheduling a cron job to run the `zxsuite backup doSmartScan` command...

Thank you very much for your report!

Have a nice day,
Cine
the ZeXtras Team

[msutherland25]

OK, but can you suggest at what time I should actually schedule a smartscan bearing in mind that the purge starts at 01:00? Before, after etc

Also, do you have any script that could be added to my backup script so that an off-site backup (rsync) only starts once any smartscan/purge has finished?

Thanks, Martin

[Cine]

Hello Martin!

Purge and SmartScan can be executed at the same time, as those act on different data, so feel free to schedule the SmartScan anytime your server is not under high load...

Unfortunately I don't have any script such as the one you mention: the best way to do this is to check your notifications and schedule the rsync a bit after the average completion time of the SmartScan (if the average SmartScan run starts at 2AM and takes 1 hour I'd schedule the rsync at 3.30AM - but that's just a personal preference).

Other than this, you can start the SmartScan and rsync at any time and even multiple times per day according to your infrastructure's I/O load, as the Backup Path will always be valid and consistent.

Still - and I know I'll sound dull - I'd go for the RealTime Scanner


Have a nice day,
Cine
the ZeXtras Team

[msutherland25]

Cine,

My understanding is that the realtime scanner (if enabled) is going to update the backup almost as soon as something changes in live Zimbra. This is brilliant indeed, but if I start an rsync of the backup area to a remote server and the backup changes during this rsync, surely the off-site backup is not consistent?

[Cine]

Hello Martin!

if I start an rsync of the backup area to a remote server and the backup changes during this rsync, surely the off-site backup is not consistent?

Not at all

As mentioned before, the ZeXtras Backup module and its dataset (a.k.a. the Backup Path) is specifically designed to be 100% self-consistent at any time, especially when the RealTime Scanner is on.

Using rsync to mirror your backup path to an external server will always return a consistent and valid mirror:

ZeXtras Backup's Datastore has the following characteristics:

• Atomicity: any transaction is committed and written to the disk only when completed.
• Consistency: any commited transaction is valid and no invalid transaction will be committed and written to the disk.
• Isolation: all transactions are executed sequentially so that no more than 1 transaction can affect the same item at once.
• Durability: once a transaction is committed, it will stay so even in case of a crash (e.g. power loss or hardware failure).

Due to this, it's very easy to make a backup of it. The best (and easiest) way to do so is by using rsync. Specific options and parameters depend on many factors, such as the amount of data to be synced and the storage in use, while connecting to an rsync daemon instead of using a remote shell as a transport as it's usually much faster in transferring the data.
You won't need to stop neither Zimbra nor the Real Time Scanner in order to make an additional backup of ZxBackup's datastore using rsync, and you will be always able to stop the sync at any time and reprise it afterwards if needed.

(extract from Taking Additional and Offsite backups of ZxBackup's Datastore - ZeXtras Suite Wiki)


Have a nice day,
Cine
the ZeXtras Team

[msutherland25]

Most interesting! But why then is a Smartscan ever needed (other than for initial setup)?

[Cine]

Hello Martin!

Most interesting! But why then is a Smartscan ever needed (other than for initial setup)?

Well, how can I say this on a public forum... Because "life" happens

The SmartScan is an additional safety layer for those times when there's an issue with either the RealTime Scaner or with the data within the backup path - e.g. accidentally unmounting the storage containing the backup path, a too long operation queue caused by an emergenvy situation, losing the link to a remote storage... On top of this, the SmartScan saves many settings and preferences that cannot be backed up by the RealTime Scanner itself.

Have a nice day,
Cine
the ZeXtras Team