export crashes after "backup path not initialized"
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: export crashes after "backup path not initialized"

  1. #1
    Active Member
    Join Date
    Mar 2015
    Posts
    6

    export crashes after "backup path not initialized"

    Forum
    Hi @ all,
    I want to migrate my Zimbe 6.0.7 (running on debian 5.0.10) to a new machine and try to export a domain with ZxMig. After a while (between 2 and 30 min) the export crashes. In the notifications I see a warning „... backup path not initialized“ and at the same time there is an error „...operation interrupted abnormally“
    Since this happened a few times, I suppose these two things are tied together.

    The details given with the error-message don´t help me either:
    -- Operation Details --
    Name: Export
    Start Time: 31/03/2015 11:52:07
    Operation Id: c09c041f-b3c0-46ed-8e3c-deda7a577af2
    Requested By: admin@jupiter2.ow.bfi-bgld.at

    -- Parameters --
    backupDirectory = com.zextras.backup.v2.BackupStoreAccessorImpl@21f7 41
    The details in the warning about the backup path being not initialized say that it might be unmounted, but since I try to export to an directory on the same disc, that cant be.
    It is not possible to initialize like said in ZxBackup: Initialization - ZeXtras Suite Wiki , because I don´t have the „Backup Tab“ in the Admin console.

    The details from „export started“ are:
    Operation Export Started.

    Operation Id: c09c041f-b3c0-46ed-8e3c-deda7a577af2
    Operation Host: jupiter2.ow.bfi-bgld.at
    Operation Log path: /opt/zimbra/log/zextras_ZxBackup_doExport_c09c041f-b3c0-46ed-8e3c-deda7a577af2.log
    Monitor Command: zxmig migration monitor c09c041f-b3c0-46ed-8e3c-deda7a577af2

    Operation requested by: admin@jupiter2.ow.bfi-bgld.at

    ZeXtras Version: 1.10.3
    ZeXtras commit: 208fdd4f3c8caa453af298c643dcf19ce42d5ca7
    ZAL Version: 1.5.2
    ZAL commit: 5fa7c70acf715a78c825bdac8babbe223627b40b
    Zimbra version: 6.0.7_GA_2473.DEBIAN5 20100616212143 20100616-2023 FOSS

    The logfile only contains:
    2015-03-31 11:52:08,366 INFO Starting doExport for Server configurations.
    2015-03-31 11:52:14,236 k.schuh@schuh-tv.at INFO Starting Export for account ff7e4e8a-d8c4-4a39-a224-3120591e5c7b.

    Here http://forums.zextras.com/zextras-mi...ifiations.html I found message #2, which says …...
    1) if you check the incremental migration guide you'll find that with zxbackup you can "update" your export directory with a simple Smart Scan operation after the backup/export initialization, while with zxmig you should close all zimbra services from users without stopping the mailboxd service (for example with some firewall rules).

    Since I cannot stop the server at the moment, I wanted to know if this could cure my problem, or if I have to search for annother reason.

    BTW: Do you have special pricing for educational, non profit organisations?

    tia, Karl

  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello Karl,
    welcome to the forums!

    It seems that the mailboxd service crashes and is automatically restarted during the export process, could you please let me know:

    - The information about your server mentioned in section 3.1 of the "How to report a ZeXtras Suite issue" guide.
    - The output of the following command:
    Code:
    zmlocalconfig | grep -i mailboxd_java
    - Any error, exception or service restart appearing in the mailbox.log file
    - Also, please send the full mailbox.log and zmmailboxd.out files to community@zextras.com mentioning the URL of this thread in the Subject of the email.

    About the Edu/Non Profit pricing: Educational, Gov and Non Profit cusomers can benefit of a 40% discount on any purchase straight in the ZeXtras Store

    Have a nice day,
    Cine
    the ZeXtras Team

  3. #3
    Active Member
    Join Date
    Mar 2015
    Posts
    6
    Hi Cine,
    sorry for the delay - I´ve been out of office for some days.

    Here is what I found out:
    General Information
    I want to migrate my domain „schuh-tv.at“ from Zimbra 6.0.7 (running on debian 5.0.10) to a new machine and try to export a domain with ZxMig. After a while (between 2 and 30 min) the export crashes. In the notifications I see a warning „... backup path not initialized“ and at the same time there is an error „...operation interrupted abnormally“
    Since this happened a few times, I suppose these two things are tied together.

    Further details see above.

    My server is:
    CPU Intel Xeon X3230 2,66GHz
    RAM 8 GB
    swap 4 GB

    df -h gives:
    /dev/sda1 1,9G 1,1G 693M 62% /
    tmpfs 4,0G 0 4,0G 0% /lib/init/rw
    udev 10M 812K 9,3M 8% /dev
    tmpfs 4,0G 0 4,0G 0% /dev/shm
    /dev/sda7 438G 194G 222G 47% /opt
    /dev/sda5 4,6G 138M 4,3G 4% /tmp
    /dev/sda2 4,6G 1,8G 2,6G 42% /usr
    /dev/sda3 6,5G 1,4G 4,8G 22% /var

    I am running Zimbra 6.0.7_GA_2473.DEBIAN5 DEBIAN5 FOSS edition
    under perl 5.10.0

    zxmig core getVersion gives:
    zextras_version 1.10.3
    zextras_commit 208fdd4f3c8caa453af298c643dcf19ce42d5ca7
    zal_version 1.5.2
    zal_commit 5fa7c70acf715a78c825bdac8babbe223627b40b

    My OS is debian 5.0.10 with kernel
    Linux jupiter2 2.6.26-2-686-bigmem #1 SMP Sun Mar 4 23:05:22 UTC 2012 i686 GNU/Linux

    The complete Zimbra-Installation is on a single machine.

    zimbra@jupiter2:/root$ zmlocalconfig | grep -i mailboxd_java
    mailboxd_java_heap_memory_percent = 30
    mailboxd_java_heap_new_size_percent = 25
    mailboxd_java_heap_size = 3000
    mailboxd_java_options = -server -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+UseConcMarkSweepGC -XX:PermSize=128m -Dsun.net.inetaddr.ttl=
    zimbra@jupiter2:/root$

    To find additional information, you wanted, I tried an export today at 10:40. At 10:57 I got the warning and the error as above mentioned.

    Mailbox.log
    The only ERROR I get around 10:57 is:

    2015-04-09 10:56:24,982 ERROR [ScheduledTask-10] [name=galsync@bfi- b.at;mid=271;ds=zimbra;] gal - Error executing gal search

    Mid 271 is account galsync@bfi-b.at which belongs to another domain on this server (I tried to export the domain schuh-tv.at).

    When greping for „backup path“ I find:
    2015-04-09 04:01:00,047 WARN [ZeXtras Cron Task Thread] [] extensions - Scheduled SmartScan skipped because the backup path is not initialized.

    Grepig for „restart“ I find:
    2015-04-09 10:57:05,094 WARN [main] [] extensions - ZeXtras Suite has detected an abnormal interruption of a ZxBackup operation, usually caused by a mailboxd service restart.
    But that´s not new :-(

    I can´t find anything in zmmailboxd.out

    I will send the log as requested.

    tia, regards
    Karl

  4. #4
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello Karl,
    welcome to the forums!

    I'll have a look at the logs you sent, in the meantime I say that the "Operation interrupted abnormally" notification means that the mailboxd service is crashed or has been restarted during the export, so the issue might be related to a resource shortage (32bit systems are sometimes prone to OOM issues)... I'll get back to you as soon as I have any news...

    Have a nice day,
    Cine
    the ZeXtras Team

  5. #5
    Active Member
    Join Date
    Mar 2015
    Posts
    6
    Quote Originally Posted by Cine View Post
    Hello Karl,
    I'll get back to you as soon as I have any news...
    Hi Cine,
    any news for me?

    Regards, Karl

  6. #6
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello Karl!

    The Devs are still investigating your report, but I had a look at the logs myself and I confirm that the issue is very likely related to a resource outage:
    Code:
    2015-04-09 10:43:15,501 WARN  [Export] [] extensions - java.lang.OutOfMemoryError
            at java.io.RandomAccessFile.readBytes ( RandomAccessFile.java:-2 ) [native]
            at java.io.RandomAccessFile.read ( RandomAccessFile.java:322 )
            at com.zextras.lib.vfs.blockingfs.BlockingFileStreamReader.read ( BlockingFileStreamReader.java:67 )
            at com.zextras.lib.vfs.blockingfs.BlockingFileStreamReader.read ( BlockingFileStreamReader.java:33 )
            at com.zextras.lib.vfs.InputStreamFileReaderWrapper.read ( InputStreamFileReaderWrapper.java:45 )
            at com.zextras.lib.vfs.InputStreamFileReaderWrapper.read ( InputStreamFileReaderWrapper.java:38 )
            at com.zextras.backup.v2.legacybackupstore.GzippedInputStream.fill_buffer ( GzippedInputStream.java:77 )
            at com.zextras.backup.v2.legacybackupstore.GzippedInputStream.read ( GzippedInputStream.java:128 )
            at com.zextras.backup.v2.legacybackupstore.LegacyBackupStoreWriter.write ( LegacyBackupStoreWriter.java:46 )
            at com.zextras.backup.v2.legacybackupstore.LegacyBackupStoreWriter.writeCompressed ( LegacyBackupStoreWriter.java:30 )
            at com.zextras.backup.v2.legacybackupstore.LegacyBackupStore.writeDigest ( LegacyBackupStore.java:198 )
            at com.zextras.backup.v2.BackupStoreAccessorImpl.writeDigest ( BackupStoreAccessorImpl.java:81 )
            at com.zextras.backup.v2.BackupStoreAccessorWrapper.writeDigest ( BackupStoreAccessorWrapper.java:199 )
            at com.zextras.backup.v2.SmartScanOperationStoreAccessorWrapper.writeDigest ( SmartScanOperationStoreAccessorWrapper.java:164 )
            at com.zextras.backup.v2.ModifiedItemVisitor._visit ( ModifiedItemVisitor.java:57 )
            at com.zextras.backup.v2.ModifiedItemVisitor._visit ( ModifiedItemVisitor.java:11 )
            at com.zextras.backup.v2.AbstractAsyncVisitor.visit ( AbstractAsyncVisitor.java:22 )
            at com.zextras.backup.v2.ItemMetadataVisitor._visit ( ItemMetadataVisitor.java:40 )
            at com.zextras.backup.v2.ItemMetadataVisitor._visit ( ItemMetadataVisitor.java:7 )
            at com.zextras.backup.v2.AbstractAsyncVisitor.visit ( AbstractAsyncVisitor.java:22 )
            at com.zextras.backup.v2.zimbrastore.ZimbraStoreReadOnly.visitModifiedItems ( ZimbraStoreReadOnly.java:381 )
            at com.zextras.backup.v2.BackupStoreReadOnlyAccessorImpl.visitModifiedItems ( BackupStoreReadOnlyAccessorImpl.java:189 )
            at com.zextras.op.backup.scan.ScanOperationBackupItemSync.syncOrDelete ( ScanOperationBackupItemSync.java:78 )
            at com.zextras.op.backup.scan.ScanOperationBackupItemSync.syncOrDelete ( ScanOperationBackupItemSync.java:55 )
            at com.zextras.backup.v2.ExportAccountVisitor._visit ( ExportAccountVisitor.java:52 )
            at com.zextras.backup.v2.ExportAccountVisitor._visit ( ExportAccountVisitor.java:9 )
            at com.zextras.backup.v2.AbstractAsyncVisitor.visit ( AbstractAsyncVisitor.java:22 )
            at com.zextras.backup.v2.AbstractAsyncVisitor.visit ( AbstractAsyncVisitor.java:23 )
            at com.zextras.mobile.BackupAccountLogInitializerVisitor.visit ( BackupAccountLogInitializerVisitor.java:36 )
            at com.zextras.mobile.BackupAccountLogInitializerVisitor.visit ( BackupAccountLogInitializerVisitor.java:15 )
            at com.zextras.backup.v2.AccountMetadataVisitor._visit ( AccountMetadataVisitor.java:48 )
            at com.zextras.backup.v2.AccountMetadataVisitor._visit ( AccountMetadataVisitor.java:7 )
            at com.zextras.backup.v2.AbstractAsyncVisitor.visit ( AbstractAsyncVisitor.java:22 )
            at com.zextras.backup.v2.zimbrastore.AllAccountsMetadataVisitor.visit ( AllAccountsMetadataVisitor.java:44 )
            at com.zextras.backup.v2.zimbrastore.AllAccountsMetadataVisitor.visit ( AllAccountsMetadataVisitor.java:8 )
            at org.openzal.zal.Provisioning.visitAllLocalAccountsSlow ( Provisioning.java:445 )
            at com.zextras.backup.v2.zimbrastore.ZimbraStoreReadOnly.visitAccounts ( ZimbraStoreReadOnly.java:203 )
            at com.zextras.backup.v2.BackupStoreReadOnlyAccessorImpl.visitAccounts ( BackupStoreReadOnlyAccessorImpl.java:79 )
            at com.zextras.op.backup.ExportOperation.exec ( ExportOperation.java:268 )
            at com.zextras.op.StatefulOperationImpl.exec ( StatefulOperationImpl.java:70 )
            at com.zextras.op.OperationStarterActivity.doActivity ( OperationStarterActivity.java:45 )
            at com.zextras.lib.activities.ActivityThread.run ( ActivityThread.java:118 )
    There are several possible causes, but there's a couple of things that usually help in most of the cases:
    - Run a BLOB check (zmblobchk) and a Database Integrity check (zmdbintegrityreport) to make sure that your BLOBs and database are not corrupted.
    - Raise the Maximum PermGen size to 350m (-XX:MaxPermSize part of the "mailboxd_java_options" localconfig entry).
    - If your server is running on a virtual environment, make sure that no memory usage limits are enforced by the hypervisor.

    Have a nice day,
    Cine
    the ZeXtras Team

  7. #7
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,356
    Hello Karl!

    The devs confirm that the most likely cause of your Export issue is a resource outage: did you try the three suggestions I posted in my last reply?
    On top of that, adding some RAM to the server (updating the appropriate localconfig entries afterwards) could really help aswell...

    Have a nice day,
    Cine
    the ZeXtras Team

  8. #8
    Active Member
    Join Date
    Mar 2015
    Posts
    6
    Hello Cine,

    Quote Originally Posted by Cine View Post
    The devs confirm that the most likely cause of your Export issue is a resource outage: did you try the three suggestions I posted in my last reply?
    On top of that, adding some RAM to the server (updating the appropriate localconfig entries afterwards) could really help aswell...
    excuse me for answering so late - I have been on holiday. I have just started the blobcheck and already seen

    Code:
    Mailbox 4, volume 1, /opt/zimbra/store/0/4/msg/0/3739-94593.msg: unexpected blob.  File size is 1161.
    Mailbox 4, volume 1, /opt/zimbra/store/0/4/msg/0/3739-9015.msg: unexpected blob.  File size is 297.
    Mailbox 4, volume 1, /opt/zimbra/store/0/4/msg/0/3739-9014.msg: unexpected blob.  File size is 221.
    I found the option
    --unexpected-blob-list <path> Write the paths of any unexpected blobs to a file.

    Should I use this option and what do I do with the found paths?

    I will continue to run the blobcheck over night and start the other things you suggested tomorrow.

    Thanks a lot,
    Karl

  9. #9
    Active Member
    Join Date
    Mar 2015
    Posts
    6
    Hi Cine,
    the test just stopped. I got a log-file of 38.000 lines !? I also got a message

    Code:
    com.zimbra.common.soap.SoapFaultException: no such account: bbbc0880-4881-4ff9-8c62-575f6cd3d726
    ExceptionId:com.zimbra.cs.account.AccountServiceException: no such account: bbbc0880-4881-4ff9-8c62-575f6cd3d726
    ExceptionId:btpool0-1197://localhost:7071/service/admin/soap/CheckBlobConsistencyRequest:1431964183503:bcc83a0c354330c9
    Code:account.NO_SUCH_ACCOUNT
            at com.zimbra.cs.account.AccountServiceException.NO_SUCH_ACCOUNT(AccountServiceException.java:177)
            at com.zimbra.cs.mailbox.Mailbox.getAccount(Mailbox.java:516)
            at com.zimbra.cs.mailbox.MailboxManager.getMailboxById(MailboxManager.java:461)
            at com.zimbra.cs.mailbox.MailboxManager.getMailboxById(MailboxManager.java:380)
            at com.zimbra.cs.service.admin.CheckBlobConsistency.handle(CheckBlobConsistency.java:81)
            at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:420)
            at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:274)
            at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:158)
            at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:291)
            at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:212)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
            at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:181)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
            at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
            at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
            at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:79)
            at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
            at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
            at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:155)
            at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
            at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
            at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
            at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
            at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
            at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
            at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
            at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
            at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
            at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
            at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
            at org.mortbay.jetty.handler.DebugHandler.handle(DebugHandler.java:77)
            at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
            at org.mortbay.jetty.Server.handle(Server.java:326)
            at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:543)
            at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939)
            at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
            at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
            at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
            at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
            at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
    
    Code:account.NO_SUCH_ACCOUNT
            at com.zimbra.common.soap.Soap12Protocol.soapFault(Soap12Protocol.java:88)
            at com.zimbra.common.soap.SoapTransport.extractBodyElement(SoapTransport.java:303)
            at com.zimbra.common.soap.SoapTransport.parseSoapResponse(SoapTransport.java:262)
            at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:253)
            at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:162)
            at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:356)
            at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:319)
            at com.zimbra.cs.account.soap.SoapProvisioning.invoke(SoapProvisioning.java:319)
            at com.zimbra.cs.store.file.BlobConsistencyUtil.checkMailbox(BlobConsistencyUtil.java:207)
            at com.zimbra.cs.store.file.BlobConsistencyUtil.run(BlobConsistencyUtil.java:179)
            at com.zimbra.cs.store.file.BlobConsistencyUtil.main(BlobConsistencyUtil.java:282)
    Can you give any hints on this?

    Regards,
    Karl

  10. #10
    Active Member
    Join Date
    Mar 2015
    Posts
    6
    Hi, its me again - for the last time today

    I can´t do the Database Integrity check (zmdbintegrityreport) you suggested - I get "command not found" as user zimbra as well as as user root.

    Can you help?

    Regards,
    Karl

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •