Secondary Backup Script Run Time Went Through The Roof on New Update
Results 1 to 5 of 5

Thread: Secondary Backup Script Run Time Went Through The Roof on New Update

  1. #1
    Active Member
    Join Date
    Sep 2014
    Posts
    3

    Secondary Backup Script Run Time Went Through The Roof on New Update

    Hello,

    For months now we have been using an rsync script to do a secondary backup of the backup directory to a NAS. For months, like clockwork, the secondary backup took about 1.5hours to run.

    Ever since the update and newly revamped backup, this secondary backup time has jumped from 1.5 hours to almost 6 hours overnight. I've been digging and digging trying to figure out what has changed and I simply cannot figure it out. At this point i am curious whether other folks are experiencing the same issue or not, and if not, maybe we can compare rsync commands and possibly how their NAS is set up relative to the server. On ours, the NAS is attached on a private subnet (gigabit) that only gets traffic from server backups. Any help would be greatly appreciated.

    Some output from my script:

    Here is our Sept 1 backup:

    Backup location is mounted, proceeding with sync.
    Starting file sync process.
    Start Time:
    Mon Sep 1 21:30:01 MDT 2014
    Emailing sync results..
    End Time:
    Mon Sep 1 23:07:56 MDT 2014
    Sync job complete. Exiting..

    Here is our Sept 2 backup:

    Backup location is mounted, proceeding with sync.
    Starting file sync process.
    Start Time:
    Tue Sep 2 21:30:01 MDT 2014
    Emailing sync results..
    End Time:
    Wed Sep 3 03:41:57 MDT 2014
    Sync job complete. Exiting..


    Any ideas what may have caused this?

  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,363
    Hello burd,
    welcome to the forums!

    This is interesting, could you please give us more info about your infrastructure such as:

    - OS, Version and Architecture (and any change about this in the last month).
    - Hardware changes in August/September.
    - Content of the "/opt/zimbra/.zextras_install_history" file.
    - Disk usage information before and after Sept. 2nd for the disk containing your Backup Path. You can find those in Zimbra's Admin Console under "Monitor" -> "Server Statistics" -> "Disk".
    - Weekly Fullscan - or Backup Purge when running Zimbra 1.8.17 - statistics for the latest run before Sept. 1st an and for the first run after the same date.

    I'd also suggest to run the usual health checks on your local disks and on your NAS (last time I saw something similar was due to a RAID Failure).

    Have a nice day,
    Cine
    the ZeXtras Team

  3. #3
    Active Member
    Join Date
    Sep 2014
    Posts
    3
    Hello and thanks for the reply. There has been absolutely no changes to this server other than the Zextras update and we implemented the Zextras chat system. All disks say they are healthy.

    Edit: Just as a note, I also use this NAS as a backup for other servers. There has been no slow transfer issues with the other backups. The other backups never run at the same time as the secondary backup script.

    OS, Architecture:
    Ubuntu 12 LTS 64bit
    ZCS 8.0.7_GA_6021.FOSS

    No hardware changes.

    .zextras_install_history:
    1381243572 INFO Command Line: all
    1381243574 INFO Detected Zimbra version 8.0.4
    1381243584 INFO Current release: 1.8.10
    1381243587 INFO Beginning Core installation procedure
    1381243652 INFO Stopping mailboxd
    1381243654 INFO ZeXtras Core installation successfully completed
    1381243667 INFO Starting mailboxd
    1381243728 INFO Beginning ZeXtras Suite Zimlet installation
    1381243742 INFO ZeXtras Suite core version 1.8.10 present on the server
    1381243775 INFO ZeXtras Suite Zimlet installation completed
    1382707679 INFO Command Line: all
    1382707682 INFO Detected Zimbra version 8.0.4
    1382707692 INFO Current release: 1.8.11
    1382707694 INFO ZeXtras Suite core version 1.8.10 present on the server
    1382707694 INFO Detected ZeXtras Suite Zimlet version 1.8.10: upgrading
    1382707697 INFO Beginning Core installation procedure
    1382707729 INFO Stopping mailboxd
    1382707731 INFO ZeXtras Core installation successfully completed
    1382707745 INFO Starting mailboxd
    1382707806 INFO Beginning ZeXtras Suite Zimlet installation
    1382707816 INFO ZeXtras Suite core version 1.8.11 present on the server
    1382707850 INFO ZeXtras Suite Zimlet installation completed
    1384622238 INFO Command Line: all
    1384622239 INFO Detected Zimbra version 8.0.5
    1384622249 INFO Current release: 1.8.11
    1384622251 INFO ZeXtras Suite core version 1.8.11 present on the server
    1384622258 INFO Beginning Core installation procedure
    1384622270 INFO Stopping mailboxd
    1384622271 INFO ZeXtras Core installation successfully completed
    1384622287 INFO Starting mailboxd
    1384622347 INFO Beginning ZeXtras Suite Zimlet installation
    1384622541 INFO Command Line: all
    1384622542 INFO Detected Zimbra version 8.0.5
    1384622551 INFO Current release: 1.8.11
    1384622553 INFO ZeXtras Suite core version 1.8.11 present on the server
    1384622564 INFO Beginning Core installation procedure
    1384622579 INFO Stopping mailboxd
    1384622581 INFO ZeXtras Core installation successfully completed
    1384622594 INFO Starting mailboxd
    1384622655 INFO Beginning ZeXtras Suite Zimlet installation
    1384622666 INFO Beginning ZeXtras Suite Zimlet reinstallation
    1384622673 INFO ZeXtras Suite core version 1.8.11 present on the server
    1384622706 INFO ZeXtras Suite Zimlet installation completed
    1388582979 INFO Command Line: all
    1388582981 INFO Detected Zimbra version 8.0.6
    1388582990 INFO Current release: 1.8.12
    1388582992 INFO ZeXtras Suite core version 1.8.11 present on the server
    1388582992 INFO Detected ZeXtras Suite Zimlet version 1.8.11: upgrading
    1388582996 INFO Beginning Core installation procedure
    1388583012 INFO Stopping mailboxd
    1388583014 INFO ZeXtras Core installation successfully completed
    1388583030 INFO Starting mailboxd
    1388583090 INFO Beginning ZeXtras Suite Zimlet installation
    1388583119 INFO ZeXtras Suite core version 1.8.12 present on the server
    1388583153 INFO ZeXtras Suite Zimlet installation completed
    1393972546 INFO Command Line: all
    1393972547 INFO Detected Zimbra version 8.0.6
    1393972557 INFO Current release: 1.8.13
    1393972559 INFO ZeXtras Suite core version 1.8.12 present on the server
    1393972559 INFO Detected ZeXtras Suite Zimlet version 1.8.12: upgrading
    1393972571 INFO Beginning Core installation procedure
    1393972617 INFO Stopping mailboxd
    1393972619 INFO ZeXtras Core installation successfully completed
    1393972635 INFO Starting mailboxd
    1393972695 INFO Beginning ZeXtras Suite Zimlet installation
    1393972702 INFO ZeXtras Suite core version 1.8.13 present on the server
    1393972737 INFO ZeXtras Suite Zimlet installation completed
    1393974012 INFO Command Line: all
    1393974014 INFO Detected Zimbra version 8.0.6
    1393974022 INFO Current release: 1.8.13
    1393974024 INFO ZeXtras Suite core version 1.8.13 present on the server
    1393974024 INFO Detected ZeXtras Suite Zimlet version 1.8.13: upgrading
    1393974040 INFO Beginning Core installation procedure
    1393974048 INFO Beginning Core reinstallation
    1393974067 INFO Stopping mailboxd
    1393974068 INFO ZeXtras Core installation successfully completed
    1393974082 INFO Starting mailboxd
    1393974143 INFO Beginning ZeXtras Suite Zimlet installation
    1393974164 INFO ZeXtras Suite core version 1.8.13 present on the server
    1393974198 INFO ZeXtras Suite Zimlet installation completed
    1398104752 INFO Command Line: all
    1398104753 INFO Detected Zimbra version 8.0.6
    1398104762 INFO Current release: 1.8.14
    1398104764 INFO ZeXtras Suite core version 1.8.13 present on the server
    1398104764 INFO Detected ZeXtras Suite Zimlet version 1.8.13: upgrading
    1398104769 INFO Beginning Core installation procedure
    1398104804 INFO Stopping mailboxd
    1398104806 INFO ZeXtras Core installation successfully completed
    1398104821 INFO Starting mailboxd
    1398104882 INFO Beginning ZeXtras Suite Zimlet installation
    1398104988 INFO ZeXtras Suite core version 1.8.14 present on the server
    1398105024 INFO ZeXtras Suite Zimlet installation completed
    1398487652 INFO Command Line: all
    1398487654 INFO Detected Zimbra version 8.0.7
    1398487663 INFO Current release: 1.8.15
    1398487665 INFO ZeXtras Suite core version 1.8.14 present on the server
    1398487665 INFO Detected ZeXtras Suite Zimlet version 1.8.14: upgrading
    1398487671 INFO Beginning Core installation procedure
    1398487691 INFO Stopping mailboxd
    1398487693 INFO ZeXtras Core installation successfully completed
    1398487706 INFO Starting mailboxd
    1398487767 INFO Beginning ZeXtras Suite Zimlet installation
    1398487776 INFO ZeXtras Suite core version 1.8.15 present on the server
    1398487813 INFO ZeXtras Suite Zimlet installation completed
    1402943992 INFO Command Line: all
    1402943994 INFO Detected Zimbra version 8.0.7
    1402944003 INFO Current release: 1.8.16
    1402944005 INFO ZeXtras Suite core version 1.8.15 present on the server
    1402944005 INFO Detected ZeXtras Suite Zimlet version 1.8.15: upgrading
    1402944010 INFO Beginning Core installation procedure
    1402944029 INFO Stopping mailboxd
    1402944031 INFO ZeXtras Core installation successfully completed
    1402944046 INFO Starting mailboxd
    1402944107 INFO Beginning ZeXtras Suite Zimlet installation
    1402944113 INFO ZeXtras Suite core version 1.8.16 present on the server
    1402944157 INFO ZeXtras Suite Zimlet installation completed
    1407764645 INFO Command Line: all
    1407764647 INFO Detected Zimbra version 8.0.7
    1407764657 INFO Current release: 1.8.17
    1407764658 INFO ZeXtras Suite core version 1.8.16 present on the server
    1407764659 INFO Detected ZeXtras Suite Zimlet version 1.8.16: upgrading
    1407764662 INFO Beginning Core installation procedure
    1407764754 INFO Stopping mailboxd
    1407764756 INFO ZeXtras Core installation successfully completed
    1407764772 INFO Starting mailboxd
    1407764832 INFO Beginning ZeXtras Suite Zimlet installation
    1407764839 INFO ZeXtras Suite core version 1.8.17 present on the server
    1407764886 INFO ZeXtras Suite Zimlet installation completed

    Disk usage:

    Has remained pretty much flat, no spikes around Sept 1 or anything like that. Looking at the monthly reports, the domain gained 7GB of disk usage for the month of August.

    Purge results. August 29 purge:
    This is an automated notification from ZxBackup about Purge.

    Operation Purge Completed.

    Operation Id: 07642e13-6fab-4538-bc92-d4a35f1f44e6
    Operation Host: BLOCKED
    Operation Start Time: 29/08/2014 19:01:00
    Operation Duration: 2 hours, 30 minutes, 23 seconds
    Scheduled operation

    ZeXtras Version: 1.8.17
    ZeXtras commit: 75c2d883fb87d042042446f1895402186b2d4c3e
    Zimbra version: 8.0.7_GA_6021 20140408123937 20140408-1243 FOSS


    Total items visited: 1091063
    Total items per seconds: 120
    Total accounts: 147
    Total accounts visited: 147

    Purged digests: 21964
    Purged items: 28296
    Purged accounts: 8
    Purged Server Configurations: 20
    Purged Server Customizations: 0

    Sept 5 Purge:


    Operation Id: 7ce137fe-b70d-4d81-8494-f5ef4a0e0ed7
    Operation Host: BLOCKED
    Operation Start Time: 05/09/2014 19:01:00
    Operation Duration: 2 hours, 28 minutes, 45 seconds
    Scheduled operation

    ZeXtras Version: 1.8.17
    ZeXtras commit: 75c2d883fb87d042042446f1895402186b2d4c3e
    Zimbra version: 8.0.7_GA_6021 20140408123937 20140408-1243 FOSS


    Total items visited: 1086654
    Total items per seconds: 121
    Total accounts: 139
    Total accounts visited: 139

    Purged digests: 17471
    Purged items: 19128
    Purged accounts: 0
    Purged Server Configurations: 16
    Purged Server Customizations: 0
    Last edited by burd; 09-18-2014 at 02:50 PM. Reason: Added information about NAS performance

  4. #4
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,363
    Hello burd!

    Things are getting puzzling...

    According to your zextras_install_history file ZeXtras Suite was updated to version 1.8.17 on August 11th, and the difference in both the number of backed up items and in the access throughtput is consistent between the August 29th and September 5th - so it looks like no relevant changes happened in neither ZeXtras Suite nor in your zimbra data storage and primary backup storage.

    As the next step, I'd go for a filesystem/RAID health check on the NAS and for a network monitoring session...

    Have a nice day,
    Cine
    the ZeXtras Team

  5. #5
    Active Member
    Join Date
    Sep 2014
    Posts
    3
    Thanks for your input. If I find the issue I'll post the results here.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •