Problem Java.lang.OutOfMemoryError: Java heap space at initial scan
Results 1 to 8 of 8
Like Tree1Likes
  • 1 Post By Sylv1

Thread: Problem Java.lang.OutOfMemoryError: Java heap space at initial scan

  1. #1
    Senior Member Known ZeXtras Reseller
    Join Date
    Sep 2012
    Posts
    65

    Problem Java.lang.OutOfMemoryError: Java heap space at initial scan

    Hello,

    I'm trying to do an initial fullscan after Zextras installation, but after some hours, it blocks.

    Error in Zimbra Admin Console :

    Code:
    2014-07-05 10:36:34,612 WARN  [main] [] extensions - ZeXtras Suite has detected an abnormal interruption of a ZxBackup operation, usually caused by a mailboxd service restart.
    Please make sure to find and fix the cause of said service restart before restarting the operation. 
    
    For additional information see http://wiki.zextras.com/wiki/Running_Operation_Awareness .
    
    -- Operation Details --
    Name: Full Scan
    Start Time: 04/07/2014 13:34:07
    Operation Id: c92cdb80-46b9-4133-8c49-5dbf8a34d5e5
    Requested By: MAIL
    
    -- Parameters --
    deepScan = false
    dataPath = /home/backup/zimbra/
    skipSmartScan = false
    notificationeMails = MAIL
    OpStartTime = 1404473647929
    origin = SOAP
    canBeFirstScan = true
    Error in oplog file :

    Code:
    {"info":{"id":"c92cdb80-46b9-4133-8c49-5dbf8a34d5e5","module":"ZxBackup","name":"FullScan","descName":"Full Scan","requestedBy":"MAIL","startTime":1404473648048,"parameters":{"deepScan":"false","skipSmartScan":"false","OpStartTime":"1404473648048","origin":"SOAP","canBeFirstScan":"true","dataPath":"/home/backup/zimbra/","notificationeMails":"[MAIL]"}},"timestamp":1404504635925,"type":"exception","exception":"java.lang.OutOfMemoryError: Java heap space\n        at com.zextras.backup.ZEBItem.writeContent ( ZEBItem.java:286 )\n        at com.zextras.backup.ZEBItem.synchronize ( ZEBItem.java:1109 )\n        at com.zextras.backup.ZEBAccount.synchronize ( ZEBAccount.java:1704 )\n        at com.zextras.op.backup.ZEScanOperation.doItemsScan ( ZEScanOperation.java:550 )\n        at com.zextras.op.backup.ZEScanOperation.doAllItemsScan ( ZEScanOperation.java:503 )\n        at com.zextras.op.backup.ZEScanOperation.doAccountScan ( ZEScanOperation.java:456 )\n        at com.zextras.op.backup.ZEScanOperation.doDomainScan ( ZEScanOperation.java:153 )\n        at com.zextras.op.backup.ZEFullScanOperation.doOperation ( ZEFullScanOperation.java:457 )\n        at com.zextras.op.ZEOperation.exec ( ZEOperation.java:720 )\n        at com.zextras.backup.services.BackupThread.execOperation ( BackupThread.java:253 )\n        at com.zextras.backup.services.BackupThread.run ( BackupThread.java:126 )\n        at java.lang.Thread.run ( Thread.java:744 )\n"}
    Error in mailbox log :

    Code:
    com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
    	at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:58)
    	at com.zimbra.cs.mailbox.Mailbox.getFolderTree(Mailbox.java:3655)
    	at com.zimbra.cs.service.mail.GetFolder.handle(GetFolder.java:111)
    	at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:522)
    	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:385)
    	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:258)
    	at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:294)
    	at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:210)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    	at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:207)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
    	at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:78)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:52)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:57)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
    	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:45)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:114)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:474)
    	at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:345)
    	at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:316)
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
    	at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    	at org.eclipse.jetty.server.Server.handle(Server.java:369)
    	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
    	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
    	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
    	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    	at java.lang.Thread.run(Thread.java:744)
    	
    2014-07-05 10:36:05,864 FATAL [LmtpServer-918] [ip=MYIP;] system - out of memory
    java.lang.OutOfMemoryError: Java heap space
    	at com.zimbra.common.util.BufferStream.<init>(BufferStream.java:55)
    	at com.zimbra.common.util.CopyInputStream.<init>(CopyInputStream.java:36)
    	at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:330)
    	at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:376)
    	at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:73)
    	at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:365)
    	at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:181)
    	at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:66)
    	at com.zimbra.cs.server.ProtocolHandler.processConnection(ProtocolHandler.java:188)
    	at com.zimbra.cs.server.ProtocolHandler.run(ProtocolHandler.java:127)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    	at java.lang.Thread.run(Thread.java:744)
    Output of zmlocalconfig mailboxd_java_options mailboxd_java_heap_size :

    Code:
    mailboxd_java_options = -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Dorg.apache.jasper.compiler.disablejsr199=true
    mailboxd_java_heap_size = 3942
    Server infos :

    • 16 Go of RAM
    • Around 230 boxes
    • Around 1To of data
    • Last ZeXtras modules
    • Zimbra 8.0.7 OSS


    How can I do my first scan ? I can add more RAM, but i want to be sure the problem is linked .

    Thanks in advance !

    Sylvain.
    Zimbra Collaboration Server 8.0.0 OSS on Ubuntu 12.04 LTS
    Zimbra Collaboration Server 7.2.0 OSS on Debian 5 & Ubuntu 10.04 LTS

  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,360
    Hello Sylv1!

    Can you please send the mailbox.log file for July 5th to community@zextras.com?

    Have a nice day,
    Cine
    the ZeXtras Team

  3. #3
    Senior Member Known ZeXtras Reseller
    Join Date
    Sep 2012
    Posts
    65
    Hello Cine

    Sent.

    Thanks in advance,

    Sylvain.
    Zimbra Collaboration Server 8.0.0 OSS on Ubuntu 12.04 LTS
    Zimbra Collaboration Server 7.2.0 OSS on Debian 5 & Ubuntu 10.04 LTS

  4. #4
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,360
    Hello Sylv1!

    It looks like there's a Mysql-related issue: the Heap is filled by either a memory leak or a bad query... Could you please also send the mysqlow.log file for further inspection?

    Have a nice day,
    Cine
    the ZeXtras Team

  5. #5
    Senior Member Known ZeXtras Reseller
    Join Date
    Sep 2012
    Posts
    65
    Hi,

    Sent too (not sure about the right day, let me know if you want more logs).

    Thanks !
    Zimbra Collaboration Server 8.0.0 OSS on Ubuntu 12.04 LTS
    Zimbra Collaboration Server 7.2.0 OSS on Debian 5 & Ubuntu 10.04 LTS

  6. #6
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,360
    Hello Sylv1!

    I've had a look at the logs and talked about your case with the devs, temporarily raising the Heap Size a bit (to 25-30% of the total ram) should help in solving the issue.

    I take this opportunity to say that we are eagerly working to improve ZxBackup's memory management in borderline cases and to informally announce that ZeXtras Suite 1.8.17 will include a full refactoring of both the SmartScan and External Backup operations with optimized memory management that will handle cases like this much better than the current SmartScan and FullScan.


    Have a nice day,
    Cine
    the ZeXtras Team
    Last edited by Cine; 07-11-2014 at 10:39 AM.

  7. #7
    Senior Member Known ZeXtras Reseller
    Join Date
    Sep 2012
    Posts
    65
    Hello Cine,

    Thanks for feedback.

    I set around 7 gigs (before it was around 4), and the script still running now .

    More infos and final response soon!

    Thanks!
    Zimbra Collaboration Server 8.0.0 OSS on Ubuntu 12.04 LTS
    Zimbra Collaboration Server 7.2.0 OSS on Debian 5 & Ubuntu 10.04 LTS

  8. #8
    Senior Member Known ZeXtras Reseller
    Join Date
    Sep 2012
    Posts
    65
    Operation Full Scan Completed



    Thanks !
    d0s0n likes this.
    Zimbra Collaboration Server 8.0.0 OSS on Ubuntu 12.04 LTS
    Zimbra Collaboration Server 7.2.0 OSS on Debian 5 & Ubuntu 10.04 LTS

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •