ZeXtras Migration export is very slow
Results 1 to 9 of 9

Thread: ZeXtras Migration export is very slow

  1. #1
    Active Member
    Join Date
    Feb 2015
    Posts
    4

    ZeXtras Migration export is very slow

    I installed ZeXtras Migration tool on a Centos 5, Zimbra 7.2.2. When I started export process, it takes long time, and the scan speed from the log shows it is very slow, only 4 per second, and it takes several hours to finish one account:

    Operation log path: /opt/zimbra/log/zextras_ZxBackup_doExport_ebd5d68f-b7e1-456f-91b0-d1afa574f53e.log
    Account scanned: 2/107
    Account new: 3
    Item scanned: 45500
    Item new/unique: 44411/39823
    Items per second: 4


    processor Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
    8GiB System Memory
    /dev/sda6 271G 46G 212G 18% /
    /dev/sda5 46G 323M 43G 1% /var/log
    /dev/sda4 118G 9.1G 103G 9% /usr
    /dev/sda1 927M 30M 850M 4% /boot
    tmpfs 3.9G 0 3.9G 0% /dev/shm
    /dev/md0 5.5T 1.6T 3.9T 30% /raid5-drive
    /raid5-drive/opt 5.5T 1.6T 3.9T 30% /opt

  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello jiaowx,
    welcome to the forums!

    The most common cause of slow exports/imports is a low I/O throughtput: I see that you are using Raid5, which is by far one of the less performing tiers, could you please post the output of `iostat -xm 1` (let it run for around 10 seconds)?

    Also, could you please report the information mentioned in section 3.1 of the "How to report a ZeXtras Suite issue" guide?

    Have a nice day,
    Cine
    the ZeXtras Team

  3. #3
    Active Member
    Join Date
    Feb 2015
    Posts
    4
    your server's specs (cpu, ram, storage etc...)
    processor Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
    8GiB System Memory
    /dev/sda6 271G 46G 212G 18% /
    /dev/sda5 46G 323M 43G 1% /var/log
    /dev/sda4 118G 9.1G 103G 9% /usr
    /dev/sda1 927M 30M 850M 4% /boot
    tmpfs 3.9G 0 3.9G 0% /dev/shm
    /dev/md0 5.5T 1.6T 3.9T 30% /raid5-drive
    /raid5-drive/opt 5.5T 1.6T 3.9T 30% /opt

    your Zimbra version
    Zimbra 7.2.2

    your ZeXtras Suite/ZxMig version ( as the zimbra user run: "zxsuite/zxmig core getVersion" )
    ZeXtras Migration tool 1.10.2

    your OS, its version and architecture
    Centos 5

    the number of servers in your Zimbra infrastructure
    1

    the number of mailbox server in your Zimbra infrastructure
    107

  4. #4
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello jiaowx!

    Could you please also post the iostat output I mentioned in my last message?

    Have a nice day,
    Cine
    the ZeXtras Team

  5. #5
    Active Member
    Join Date
    Feb 2015
    Posts
    4
    avg-cpu: %user %nice %system %iowait %steal %idle
    13.27 0.00 2.13 5.01 0.00 79.60

    Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 26.00 0.00 17.00 0.00 0.17 20.24 0.02 1.29 0.88 1.50
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda5 0.00 26.00 0.00 17.00 0.00 0.17 20.24 0.02 1.29 0.88 1.50
    sda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sdb 53.00 38.00 15.00 56.00 0.27 0.37 18.37 0.58 8.11 4.82 34.20
    sdb1 53.00 38.00 15.00 56.00 0.27 0.37 18.37 0.58 8.11 4.82 34.20
    sdc 53.00 57.00 34.00 35.00 0.34 0.36 20.87 0.83 12.01 4.28 29.50
    sdc1 53.00 57.00 34.00 35.00 0.34 0.36 20.87 0.83 12.01 4.28 29.50
    sdd 21.00 95.00 24.00 48.00 0.18 0.56 20.89 0.66 9.12 4.33 31.20
    sdd1 21.00 95.00 24.00 48.00 0.18 0.56 20.89 0.66 9.12 4.33 31.20
    sde 14.00 81.00 26.00 44.00 0.16 0.49 18.86 0.77 10.56 4.07 28.50
    sde1 14.00 81.00 26.00 44.00 0.16 0.49 18.86 0.77 10.56 4.07 28.50
    md0 0.00 0.00 0.00 232.00 0.00 0.91 8.00 0.00 0.00 0.00 0.00

  6. #6
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello jiaowx!

    Could you please let the command run for ~10 seconds (during an export, of course) and post the full output as described in my post?

    Have a nice day,
    Cine

  7. #7
    Active Member
    Join Date
    Feb 2015
    Posts
    4
    Monitor log:
    --
    Operation log path: /opt/zimbra/log/zextras_ZxBackup_doExport_41a17e7b-136d-4495-8c61-1b2025584e5d.log
    Account scanned: 2/107
    Account new: 3
    Item scanned: 21500
    Item new/unique: 21500/17015
    Items per second: 6
    ----------------------------------
    output of `iostat -xm 1`

    avg-cpu: %user %nice %system %iowait %steal %idle
    0.12 0.00 0.12 6.75 0.00 93.00

    Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
    sda 0.00 4.00 0.00 2.00 0.00 0.02 24.00 0.00 0.50 0.50 0.10
    sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    sda6 0.00 4.00 0.00 2.00 0.00 0.02 24.00 0.00 0.50 0.50 0.10
    sdb 40.00 42.00 27.00 28.00 0.12 0.27 14.69 7.04 45.75 5.13 28.20
    sdb1 40.00 42.00 27.00 28.00 0.12 0.27 14.69 7.04 45.75 5.13 28.20
    sdc 61.00 32.00 21.00 30.00 0.30 0.24 21.96 3.38 32.90 5.35 27.30
    sdc1 61.00 32.00 21.00 30.00 0.30 0.24 21.96 3.38 32.90 5.35 27.30
    sdd 50.00 31.00 27.00 18.00 0.23 0.19 19.02 7.92 38.69 5.38 24.20
    sdd1 50.00 31.00 27.00 18.00 0.23 0.19 19.02 7.92 38.69 5.38 24.20
    sde 32.00 49.00 23.00 25.00 0.21 0.24 19.33 10.54 164.12 20.27 97.30
    sde1 32.00 49.00 23.00 25.00 0.21 0.24 19.33 10.54 164.12 20.27 97.30
    md0 0.00 0.00 0.00 331.00 0.00 1.29 8.00 0.00 0.00 0.00 0.00

  8. #8
    CTO ZeXtras Employee d0s0n's Avatar
    Join Date
    Apr 2011
    Posts
    570
    Hi jiaowx,

    as Cine said, I can confirm that your issue is caused by poor storage performance. You can see your sde device at almost 100% used with only 25 writes/sec and 23 reads/sec, It should be used only for backup purpose while you are using it as production environment. I can suggest you to take a look to THIS zimbra doc. Performances should improve if you use a different (better) device as export folder.

    Have a nice day.
    D0s0n
    ZeXtras Website # ZeXtras Wiki # ZeXtras Store

    Head of ZeXtras System Administrators

  9. #9
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello!


    I'm glad to inform you that the next release of ZeXtras Suite will include an optimization to the "External Restore" operation made to improve import times by reducing the operation's overhead: in optimal cpu/storage/memory conditions the improvement will be remarkable, but it should be noticeable even in sub-par situations (e.g. low I/O trhoughput or not enough iops).


    Keep an eye on the "News and announcements" section for more info about the release of ZeXtras Suite 2.0!


    Have a nice day,
    Cine
    the ZeXtras Team

    (This is a bulk update to several similar threads, but don't hesitate to reply!)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •