Zimbra 8.6 import rate slow and degrading
Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Zimbra 8.6 import rate slow and degrading

  1. #1
    Active Member
    Join Date
    Jan 2015
    Posts
    3

    Zimbra 8.6 import rate slow and degrading

    Background:
    Migrating from ZCS 7.2.7 on CentOS 5/x86. Exported mail with zxmig. No issues with the export.

    Migrating to ZCS 8.6.0

    a precise description of the issue
    The initial rate of message import was reasonable; approx 10,000 messages per hour. It has degraded to about 1 thousand messages per hour.

    Disk, CPU and RAM vitals appear fine (reasonable latency, plenty of CPU time, enough RAM to have a good size page cache [2GB]).

    The import rate is slow, but consistent. Could there be some SOAP DOS prevention being triggered? I didn't see any evidence of this in the logs.

    At the current rate of import, it appears the process will not complete for at least another 100 hours, which is unacceptable for my customer. I'm open to suggestions. I have been tuning the VM subsystem to try to alleviate disk load and keep the page cache large, but this appears to have no tangible effect. It seems more like lock contention, throttling or similar.

    steps to reproduce the issue
    NA

    your server's specs (cpu, ram, storage etc...)
    6GB of RAM (not great, but overlooked during provisioning and now can't interrupt the restore due to criticality of email to customer).
    16 x E7340 Xeon cores.
    Paravirt disk drivers (raw throughput reads ~280MB/s, writes ~27MB/s).
    I/O latency is reasonable (1 second intervals):
    Code:
    Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
    xvda              0.00     0.00   72.00    3.00  2488.00    11.00    66.64     7.45  103.23  107.22    7.33   4.56  34.20
    
    xvda              0.00     0.00    0.00   34.00     0.00   551.50    32.44     0.16    4.59    0.00    4.59   2.71   9.20
    
    xvda              0.00     0.00    5.00  228.00    24.00  9952.00    85.63     5.00   20.91    8.00   21.20   0.74  17.30
    
    xvda              0.00     0.00   72.00   17.00  3076.00    43.00    70.09     5.67   65.06   74.11   26.71   5.54  49.30
    
    xvda              0.00     0.00   10.00  245.00    68.00 10078.50    79.58     2.91   11.41    8.30   11.54   1.82  46.40
    
    xvda              0.00     0.00    1.00    6.00     4.00    19.00     6.57     0.05    7.29    0.00    8.50   7.29   5.10
    CPU load is fluctuating, but around 80-100%.

    your Zimbra version
    8.6.0

    your ZeXtras Suite/ZxMig version ( as the zimbra user run: "zxsuite/zxmig core getVersion" )
    zextras_version 1.10.2
    zextras_commit 5fa7c70acf715a78c825bdac8babbe223627b40b
    zal_version 1.5.2
    zal_commit 5fa7c70acf715a78c825bdac8babbe223627b40b

    your OS, its version and architecture
    CentOS 7/x64.

    the number of servers in your Zimbra infrastructure
    1.

    the number of mailbox server in your Zimbra infrastructure
    1.

    for GUI related issues, the browser you were using when experiencing the issue and its version
    NA

    for ZxChat Zimlet issues, the ZxChat Zimlet version
    NA

  2. #2
    Active Member
    Join Date
    Jan 2015
    Posts
    3
    Further to the above, if I do an imapsync on multiple accounts in parallel to the restore, the message rate is the same (per account) at about 3 seconds per message.

  3. #3
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello aussiemusclenator,
    welcome to the forums!

    Could you please send your mailbox.log and external restore log to community@zextras.com so that I'll be able to have a look at the full restore information?

    Have a nice day,
    Cine
    the ZeXtras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  4. #4
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello aussiemusclenator!

    I got the logs, will report back to you as soon as possible...

    Have a nice day,
    Cine
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  5. #5
    Active Member
    Join Date
    Jan 2015
    Posts
    3
    The restore ultimately aborted due to timeout:
    error: com.zextras.lib.Error.UnableToCommunicateWithServe rError: Unable to communicate with server
    .

  6. #6
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello aussiemusclenator!

    The Devs are still investigating your case, I'll forward your updated feedback to them immediately... Could you please send me the full exception and the "operation interrupted" notification (if any)?

    Have a nice day,
    Cine
    the ZeXtras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  7. #7
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello aussiemusclenator!

    I got a feedback by the devs: the mailbox.log you sent contains a lot of "Slow execution" messages, even in periods when the restore is not running - so it looks like there's something wrong with mysql performances. A bottleneck like this is usually cause by either I/O issues or low throughtput, RAM overuse/swapping or CPU load, and the imapsync accessing the server during the import unfortunately does not help.

    I seem to understand that this server is already in production, can you please confirm this?

    Also, a couple of additional questions in order to better identify the cause:
    - Is Zimbra's database stored on the "xvda" device mentioned in the OP?
    - You mention CPU load between 80% and 100%: could you please let me know which processes take up the most CPU% (I assume those are the mailboxd and mysql, but it's just to be sure)?
    - Could you please make sure that no memory limitations are currently enforced by the hypervisor on the VM?

    Have a nice day,
    Cine
    the ZeXtras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  8. #8
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello!


    I'm glad to inform you that the next release of ZeXtras Suite will include an optimization to the "External Restore" operation made to improve import times by reducing the operation's overhead: in optimal cpu/storage/memory conditions the improvement will be remarkable, but it should be noticeable even in sub-par situations (e.g. low I/O trhoughput or not enough iops).


    Keep an eye on the "News and announcements" section for more info about the release of ZeXtras Suite 2.0!


    Have a nice day,
    Cine
    the ZeXtras Team

    (This is a bulk update to several similar threads, but don't hesitate to reply!)
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  9. #9
    Junior Member
    Join Date
    Mar 2015
    Posts
    2
    Hello!!

    I 'm importing 1000 accounts in a new Zimbra Server 8.6 witch ZxBackup. There is 500G of data and the process is tooooo slow.

    The server is a 2 Xeon E5-2670 with 16G RAM and 4 disks. One disk for operative system, other for /opt/zimbra (xvdc), other for /opt/zimbra/index (xvdc-SSD) and the last for /opt/zimbra/backup(xvdf-SSD)

    I saw that ZxBackup perform a sustained 8-10mbps write in xvdf ( where it must read ). Why is that? It's a bug? Below, you can see the output from dstat. Some times, ZxBackup stop write to backup drive and the restore will faster.. but 1 minute after, became to write 10mbps again. What is that??

    Actually, the restore is about 300kbps.. with this rate it will finished in 2 Months!!!!!

    #dstat -D xvdc,xvdd,xvdf

    -dsk/xvdc----dsk/xvdc----dsk/xvdf-
    read writ: read writ: read writ|
    23k 303k: 561B 15k: 70k 561k|
    60k 388k: 0 40k: 20k 9008k|
    4096B 360k: 0 0 : 92k 9504k|
    4096B 372k: 0 0 : 24k 8308k|
    4096B 280k: 0 0 : 316k 8904k|
    12k 532k: 0 0 : 24k 8904k|
    4096B 272k: 0 164k: 140k 9004k|
    4096B 296k: 0 0 : 100k 8904k|
    64k 368k: 0 0 : 56k 8904k|
    4096B 308k: 0 0 : 84k 8904k|
    4096B 604k: 0 0 : 108k 8904k|
    188k 192k: 0 0 : 16k 7148k|
    464k 428k: 0 40k: 20k 6308k|
    8192B 540k: 0 0 : 100k 8904k|
    4096B 468k: 0 0 : 96k 8904k|
    8192B 904k: 0 84k: 24k 8908k|
    4096B 360k: 0 0 : 116k 9008k|
    4096B 344k: 0 160k: 108k 8912k|
    4096B 456k: 0 0 : 12k 4456k|
    4096B 532k: 0 0 : 40k 8912k|
    52k 564k: 0 0 : 168k 8912k|
    100k 348k: 0 0 : 112k 9016k|
    24k 332k: 0 40k: 88k 8912k|
    24k 300k: 0 0 : 32k 8912k|
    32k 388k: 0 0 : 212k 8912k|
    480k 288k: 0 0 : 20k 8912k|
    4096B 280k: 0 0 : 104k 9008k|
    184k 272k: 0 0 : 24k 8912k|
    4096B 200k: 0 160k: 12k 4808k|
    4096B 272k: 0 0 : 20k 8560k|
    4096B 312k: 0 0 : 20k 8912k|
    4096B 272k: 0 0 : 128k 9016k|
    196k 388k: 0 0 : 120k 12M|
    4096B 596k: 0 40k: 120k 6432k|
    188k 764k: 0 0 : 64k 8912k|
    48k 396k: 0 0 : 24k 8912k|
    176k 396k: 0 0 : 104k 9016k|
    740k 1164k: 308k 148k: 112k 8912k|
    1164k 256k: 0 172k: 116k 4456k|
    1392k 460k: 0 0 : 96k 8912k|
    48k 508k: 0 0 : 28k 8912k|
    184k 380k: 0 0 : 168k 9008k|
    392k 404k: 0 0 : 104k 8912k|

    As you see in iostat, the backup drive is about 7% %util

    # iostat -mx 1

    avg-cpu: %user %nice %system %iowait %steal %idle
    20.10 0.00 26.13 4.02 0.00 49.75

    Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
    xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    xvdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    xvdc 0.00 44.00 1.00 23.00 0.00 0.27 23.33 0.04 1.50 4.00 1.39 1.33 3.20
    dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    dm-1 0.00 0.00 1.00 67.00 0.00 0.27 8.24 0.15 2.18 4.00 2.15 0.47 3.20
    xvdf 14.00 20.00 21.00 218.00 0.14 9.14 79.50 4.26 17.82 12.76 18.31 0.28 6.80

  10. #10
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello Luciano!

    Some performance-improving tweaks have been recently added to ZeXtras Backup, and while at the moment those are only available in the 1.99.x beta tree we'll release a stable ZeXtras Suite version that includes such tweaks pretty soon.

    Have a nice day,
    Cine
    the ZeXtras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •