Near real time synchronisation
Results 1 to 2 of 2

Thread: Near real time synchronisation

  1. #1
    Active Member
    Join Date
    Feb 2013

    Question Near real time synchronisation

    There has been a lot of discussion surrounding the use of rsync for mirroring the real time scanner backed up files to a secondary/and or tertiary system. I would like to explore a couple of alternative approaches and see whether they could be more efficient and less likely to potential file inconsistencies.

    The first would be the use of DRBD and GFS to create a clustered LVM specifically for the /opt/zimbra/backup/zextras directory. As long as the systems have a dedicated giga link between them then it could be pretty good for setting up a hot standby.

    The second approach would be the use of Inotify: Efficient, Real-Time Linux File System Event Monitoring to monitor the backup directory for filesystem changes. As it is able to trigger on a committed write then there should be no fear of synchronising a partial file. Obviously it would need to be scripted; perhaps use Tatsuhiko Miyagawa / Filesys-Notify-Simple - but it at least would only transfer files when something has been written. That would certainly negate the necessity of a cronjob running every through minutes and putting additional load on the system.

    These are just some muse ramblings to hopefully fuel some further discussion on this subject.


  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Hello uxbod!

    I have a very limited experience with Inotify and no experience at all with the Filesys::Notify::Simple perl module, so my contribution to the topic will only be about the ZeXtras Suite side of the discussion.

    First of all, ZeXtras Suite uses an atomic write algorithm to reduce the chances of reading incomplete files, mainly to avoid data disruption during Import/Restore operations (similarly, the BLOB Move operations of ZxPowerstore use a transactional algorythm).
    No hardlinks are used and the only filesystem imitation is Case Sensitiveness as explained at ZxBackup Store/Case Sensitive filesystem for ZeXtras Backup - ZeXtras Suite Wiki.

    A near real time synchronization of ZxBackup's backup path sould mainly focus on the "accounts/" and "items/" subdirectories, which contain item's history and blobs respectively. The "server/" subdorectory is also important, but for DR purpouses a copy of the last written file is all you need - as the Backup Path of a server can be used as an "External Restore" source but not as a Backup Path for another server.

    The content of the root of the backup path is also important, especially the "backupstat", "smartstat" and "dirtylist" files. This files are the ones actively used by the backup system, while map_* files are created by the "External Import" feature to keep track of OLD>NEW zimbraID mapping. The "lock_*" file is used to identify the directory as a valid backup path for a server, and it's never changed after being created during the backup system's initialization.

    Concerning the directory structure created by ZxBackup: the "server/" subdirectory is plain, while the "accounts/" and "items/" subdirectory use a nested hierarchy based on zimbraIDs and item names (more info HERE).

    This is all I can say right now, I'll be more than glad to post other informations if something useful comes to my mind or if somebody has any question

    Luckly enough, my colleague Trantor is both experienced with Inotify and an amazing perl programmer, I'll discuss this matter with him and see if he has any additional comment/suggestion...

    Have a nice day,
    Last edited by Cine; 02-08-2013 at 12:03 PM. Reason: Too many smilies. Friday is not a valid reason to let my smiley love loose...


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts