Import failing OutOfMemoryError: Java heap space
Results 1 to 8 of 8

Thread: Import failing OutOfMemoryError: Java heap space

  1. #1
    Active Member
    Join Date
    Mar 2014
    Posts
    6

    Import failing OutOfMemoryError: Java heap space

    I'm trying to do an import of a Zimbra 6.0.7 physical server to a virtualized Zimbra 8 server. The accounts have been created successfully, but no mail is in the boxes and I'm getting a host of warnings, and then a fatal error. It doesn't email me or put a notification up that it failed. It just silently stops until I go digging in the logs.

    This appears to be a heap issue, but I've already increased it days ago. I included samples of the warnings I'm getting and the Fatal error below.

    Any idea how to get the this to run completely? Also, when I do run an actual import (not a test), will I need to remove all this data or will it replace everything?

    I've only got 6 days left on the trial so I'm getting a bit antsy. :-)

    Code:
    2014-03-17 04:01:00,002 WARN  [ZeXtras Cron Task Thread] [] extensions - Scheduled SmartScan skipped because the backup path is no
    t initialized.
    2014-03-17 11:13:13,773 WARN  [qtp310912546-4399:https://10.0.0.13:7071/service/admin/soap/GetInfoRequest] [ip=10.0.0.60;ua=Zimbra
    WebClient - FF27 (Linux);] SoapEngine - no valid authtoken present: cannot dispatch request
    ....
    2014-03-17 11:17:22,446 WARN  [ZxBackup Thread] [] extensions - Unrestorable attribute zimbraId, removed.
    2014-03-17 11:17:22,447 WARN  [ZxBackup Thread] [] extensions - Unrestorable attribute zimbraDomainName, removed.
    2014-03-17 11:17:22,447 WARN  [ZxBackup Thread] [] extensions - Unrestorable attribute zimbraDomainType, removed.
    2014-03-17 11:17:22,447 WARN  [ZxBackup Thread] [] extensions - Unrestorable attribute zimbraCreateTimestamp, removed.
    ....
    2014-03-17 11:17:22,447 WARN  [ZxBackup Thread] [] misc - checkValue: no attribute info for: dc
    2014-03-17 11:17:32,709 WARN  [ZxBackup Thread] [] misc - SpamHandler called on unhandled item type=folder account=ca11031@domain.com id=4
    ....
    2014-03-17 11:19:32,672 FATAL [Index-9] [name=us5850@domain.com;mid=24;] system - out of memory
    java.lang.OutOfMemoryError: Java heap space
            at java.util.Arrays.copyOfRange(Arrays.java:2694)
            at java.lang.String.<init>(String.java:203)
            at java.lang.String.substring(String.java:1913)
            at java.lang.String.trim(String.java:2732)
            at com.zimbra.cs.index.Fragment.skipFragmentHeader(Fragment.java:60)
            at com.zimbra.cs.index.Fragment.getFragment(Fragment.java:187)
            at com.zimbra.cs.index.Fragment.getFragment(Fragment.java:173)
            at com.zimbra.cs.mime.ParsedMessage.analyzeBodyParts(ParsedMessage.java:398)
            at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:419)
            at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:461)
            at com.zimbra.cs.mailbox.CalendarItem.getIndexDocuments(CalendarItem.java:382)
            at com.zimbra.cs.mailbox.CalendarItem.generateIndexData(CalendarItem.java:257)
            at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:736)
            at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:373)
            at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:83)
            at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1404)
            at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1383)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:744)
    2014-03-17 11:21:02,764 ERROR [qtp679447471-150:https://10.0.0.13:7071/service/admin/soap/zextras] [name=admin@mail.domain.com;mid=2;ip=10.0.0.60;ua=ZimbraWebClient - FF27 (Linux);] extensions - Could not find backup statistics of account 797e4dd6-5fe8-41d4-a678-5d27a078f8fd
    2014-03-17 11:56:33,227 FATAL [Index-9] [name=us5850@domain.com;mid=24;] system - out of memory
    java.lang.OutOfMemoryError: Java heap space
            at java.util.Arrays.copyOfRange(Arrays.java:2694)
            at java.lang.String.<init>(String.java:203)
            at java.lang.String.substring(String.java:1877)
            at com.zimbra.cs.index.Fragment.skipFragmentHeader(Fragment.java:78)
            at com.zimbra.cs.index.Fragment.getFragment(Fragment.java:187)
            at com.zimbra.cs.index.Fragment.getFragment(Fragment.java:173)
            at com.zimbra.cs.mime.ParsedMessage.analyzeBodyParts(ParsedMessage.java:398)
            at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:419)
            at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:461)
            at com.zimbra.cs.mailbox.CalendarItem.getIndexDocuments(CalendarItem.java:382)
            at com.zimbra.cs.mailbox.CalendarItem.generateIndexData(CalendarItem.java:257)
            at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:736)
            at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:373)
            at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:83)
            at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1404)
            at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1383)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:744)
    Here are some diagnostic info:
    Code:
    mail:~$ sudo /opt/zimbra/bin/zmlocalconfig mailboxd_java_options mailboxd_java_heap_size mailboxd_java_heap_memory_percent
    Warning: null valued key 'mailboxd_java_heap_memory_percent'
    mailboxd_java_options = -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true
    mailboxd_java_heap_size = 1024
    mail:~$ free -m
                 total       used       free     shared    buffers     cached
    Mem:          3952       3580        371          0         80       1374
    -/+ buffers/cache:       2125       1826
    Swap:         1019        464        555

  2. #2
    Active Member
    Join Date
    Mar 2014
    Posts
    6
    Would some other information be helpful in understanding this problem?

  3. #3
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello flickerfly,
    welcome to the forums!

    Unfortunately there's not much to understand here, as the system runs out of heap space and the operation fails due to the subsequent mailboxd crash and restart...

    I see that you have 3952mb of RAM memory and 1024mb of heap space: my suggestion would be to provisionally raise the amount of RAM memory to at least 8gb and the heap size to 2048mb (you can change those settings back after completing the migration).

    Also, make sure that the amount of RAM is not limited in any way by the hypervisor.


    Regarding the "real" import: ZeXtras Backup keeps track of the previously imported items, which will be skipped by any subsequent import. This means that no items will be duplicated, but also that any change applied to imported items on the source server will not be applied to the destination server.
    If you wish to start the "real" import from scratch you will need to delete any file starting with "map_" contained in the backup path of the destination server (default: /opt/zimbra/backup/zextras/) before proceeding.

    Have a nice day,
    Cine
    the ZeXtras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  4. #4
    Active Member
    Join Date
    Mar 2014
    Posts
    6
    Thanks Cine. It looks like with more RAM it is doing a better job.

    So, if I delete the map_* files in my previous export using zxMig and run another export to the same location and then import that data overtop of the previous data any stuff deleted on the source server will also be then deleted on the destination server. Did I understand correctly?

  5. #5
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello flickerfly!
    Quote Originally Posted by flickerfly View Post
    Thanks Cine. It looks like with more RAM it is doing a better job.
    You're welcome, glad it works better ow...

    Quote Originally Posted by flickerfly View Post
    So, if I delete the map_* files in my previous export using zxMig and run another export to the same location
    ZxMig's exports are not incremental, so you'll need to choose a different export path each time. If you wish to perform an Incremental Migration, have a look at http://wiki.zextras.com/wiki/ZxBacku...eXtras_Backup: you'll need ZeXtras Suite on both servers (the trial is perfectly fine), and by setting your last export path as the "Backup Path" in ZxBackup you'll avoid running the export from scratch again.

    Quote Originally Posted by flickerfly View Post
    and then import that data overtop of the previous data any stuff deleted on the source server will also be then deleted on the destination server.
    As I mentioned, Import operations never update or delete any item, so just running the import over the previous data after deleting the map files will cause item duplication: you'll have to remove all accounts from the server so that ZeXtras Backup can start the import anew (I can provide you with a bash oneliner to do this if you wish).

    Have a nice day,
    Cine
    the ZeXtras Team.
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  6. #6
    Active Member
    Join Date
    Mar 2014
    Posts
    6
    I'd appreciate that bash one-liner. Thank you!

  7. #7
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,342
    Hello flickerfly!

    Quote Originally Posted by flickerfly View Post
    I'd appreciate that bash one-liner. Thank you!
    Running
    Code:
    zmprov -l gaa | awk {'print "da "$1'} > /tmp/delete_all_accounts.zmprov
    will create a file in /tmp/ called "delete_all_accounts.zmprov" with a set of zmprov commands that will delete ALL accounts on the server.

    Remove any system account entry (e.g. galsync, ham, spam and quarantine accounts) from the file and run
    Code:
    zmprov -f /tmp/delete_all_accounts.zmprov
    to apply

    Have a nice day,
    Cine
    the ZeXtras Team
    Last edited by Cine; 03-20-2014 at 01:29 PM.
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  8. #8
    Active Member
    Join Date
    Mar 2014
    Posts
    6
    Excellent! Thank you

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •