Memory leak
Results 1 to 6 of 6

Thread: Memory leak

  1. #1
    Active Member
    Join Date
    Jan 2018
    Posts
    11

    Memory leak

    Zextras 2.6.2 seems to be leaking memory rather badly.

    I have upgraded to 8.8.6 GA from 8.7.11 and am still seeing the same gradual march towards Out of Memory crash. Running on Ubuntu 14.04.5 LTS

    The server I am running on has 16GB of RAM and ~13 users active on ZxMobile Activesync. Its not really got that much going on but its gobbling RAM.

    Increasing the mailbox heap size has reduced the crashes from 3-4 times per day to 1 per day.

    Current settings are:

    mailboxd_java_heap_new_size_percent = 30
    mailboxd_java_heap_size = 4096
    mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewRatio=2 -XX:PermSize=196m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true

    These are as per the ZImbra wiki page https://wiki.zimbra.com/wiki/Perform...ge_Deployments



    Code:
    2018-01-23 10:02:40.346:WARN:oejh.HttpParser:qtp1068934215-118177: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@7bb80dd0{r=1,c=false,a=IDLE,uri=null}
    2018-01-23 22:42:36.043:WARN:oejs.ServletHandler:ZeXtras Mobile: Error for /service/extension/zimbrasync
    java.lang.OutOfMemoryError: Java heap space
    	at java.util.Arrays.copyOf(Arrays.java:3236)
    	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    	at com.zextras.utils.FastBase64.encodeToStream(FastBase64.java:371)
    	at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifier$SimpleDataHandler.writeTo(MimeSimplifier.java:126)
    	at javax.mail.internet.PreencodedMimeBodyPart.writeTo(PreencodedMimeBodyPart.java:112)
    	at javax.mail.internet.MimeMultipart.writeTo(MimeMultipart.java:462)
    	at com.sun.mail.handlers.multipart_mixed.writeTo(multipart_mixed.java:103)
    	at javax.activation.ObjectDataContentHandler.writeTo(DataHandler.java:889)
    	at javax.activation.DataHandler.writeTo(DataHandler.java:317)
    	at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
    	at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1773)
    	at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1749)
    	at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifierWithFallback.fixRawMime(MimeSimplifierWithFallback.java:51)
    	at com.zextras.modules.mobile.v2.engine.mime.SimplifierProxyBodyContentEncoder.encodeContentProvider(SimplifierProxyBodyContentEncoder.java:93)
    	at com.zextras.modules.mobile.v2.as.encoders.commons.EmailApplicationDataEncoder.encode(EmailApplicationDataEncoder.java:112)
    	at com.zextras.modules.mobile.v2.as.encoders.commons.SyncEncoder.encode(SyncEncoder.java:173)
    	at com.zextras.modules.mobile.v2.as.encoders.commons.LoggerEncoder.encode(LoggerEncoder.java:111)
    	at com.zextras.modules.mobile.v2.as.events.AddNewEmailEvent.encode(AddNewEmailEvent.java:39)
    	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItem(SyncItemEmitter.java:445)
    	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItems(SyncItemEmitter.java:312)
    	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitChanges(SyncItemEmitter.java:108)
    	at com.zextras.modules.mobile.v2.engine.actions.CollectionSyncAction.execute(CollectionSyncAction.java:121)
    	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
    	at com.zextras.modules.mobile.v2.as.assembler.ProbeAction.execute(ProbeAction.java:46)
    	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
    	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
    	at com.zextras.modules.mobile.v2.engine.actions.StartEndWrapperAction.execute(StartEndWrapperAction.java:31)
    	at com.zextras.modules.mobile.v2.engine.actions.CheckFolderHierarchyAction.execute(CheckFolderHierarchyAction.java:98)
    	at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execAction(InitialExecutor.java:268)
    	at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execute(InitialExecutor.java:211)
    Exception in thread "threads.csv" *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807
    
    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "threads.csv"
    Exception in thread "sql.csv" Exception in thread "ldap.csv" java.lang.OutOfMemoryError: Java heap space
    java.lang.OutOfMemoryError: Java heap space
    Exception in thread "Cron" java.lang.OutOfMemoryError: Java heap space
    
    (more, but forum has size limit)







    As mentioned in other threads, ZxMobile seems very CPU heavy. Reducing http threads & turning off interprocess security has helped as has upgrading from 8.7.11 to 8.8.6 - but for a dozen clients to being pushing the server so hard is frustrating.

    i.e.
    /opt/zimbra/bin/zmprov modifyConfig zimbraHttpNumThreads 100
    /opt/zimbra/bin/zmlocalconfig -e zimbra_require_interprocess_security=0

  2. #2
    Active Member
    Join Date
    Jan 2018
    Posts
    11
    Trace continues:
    Code:
    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=zimbra.us.local:389), maxConnections=50)"
    
    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NioProcessor-550"
    Exception in thread " Activity Timer" java.lang.OutOfMemoryError: Java heap space
    Exception in thread "mailboxd.csv" java.lang.OutOfMemoryError: Java heap space
    2018-01-23 22:50:57.832:WARN:oejs.ServerConnector:qtp1068934215-142-acceptor-0@3c577368-ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}: 
    java.lang.OutOfMemoryError: Java heap space
    2018-01-23 22:58:13.195:WARN:oeji.SelectorManager:qtp1068934215-200293: Exception while notifying connection SslConnection@220a86e3{NEED_UNWRAP,eio=-1/-1,di=-1} -> HttpConnection@1d710bad[DecryptedEndPoint@5a869410{/192.168.1.252:50790<->7072,Open,in,out,-,-,346129/60000,HttpConnection}->SelectChannelEndPoint@2c1747e9{/192.168.1.252:50790<->7072,Open,in,out,-,-,46714/60000,SslConnection}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@4c38b8a6{s=START},c=HttpChannelOverHttp@3af3025c{r=0,c=false,a=IDLE,uri=null}][b=null]
    java.lang.OutOfMemoryError: Java heap space
    OpenJDK 64-Bit Server VM warning: ignoring option PermSize=196m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=350m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
    2018-01-23 23:02:23.703:INFO::main: Logging initialized @1210ms
    JettyMonitor monitoring thread pool qtp1068934215{STOPPED,10<=0<=100,i=0,q=0}
    Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
    Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
    Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
    Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
    Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
    2018-01-23 23:02:25.301:INFO:oejs.SetUIDListener:main: Setting umask=027
    2018-01-23 23:02:25.301:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
    2018-01-23 23:02:25.302:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
    2018-01-23 23:02:25.312:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2473b9ce{HTTP/1.1,[http/1.1]}{localhost:8080}
    2018-01-23 23:02:25.313:INFO:oejs.SetUIDListener:main: Opened ServerConnector@58a90037{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
    2018-01-23 23:02:25.313:INFO:oejs.SetUIDListener:main: Opened ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
    2018-01-23 23:02:25.314:INFO:oejs.SetUIDListener:main: Opened ServerConnector@70a9f84e{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
    2018-01-23 23:02:25.315:INFO:oejs.SetUIDListener:main: Opened ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
    2018-01-23 23:02:25.315:INFO:oejs.SetUIDListener:main: Setting GID=999
    2018-01-23 23:02:25.316:INFO:oejs.SetUIDListener:main: Setting UID=101
    2018-01-23 23:02:25.323:INFO:oejs.Server:main: jetty-9.3.5.v20151012
    2018-01-23 23:02:32.125:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@6e4784bc{/service,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
    Zimbra server process is running as uid=101 euid=101 gid=999 egid=999
    ---------------- TRACE (received) ----------------------
    CONFIG, arg={bind_addr=zimbra.us.local/192.168.1.252}
    --------------------------------------------------------
    ------------------- TRACE (sent) -----------------------
    SET_LOCAL_ADDRESS, arg=zimbra.us.local
    --------------------------------------------------------
    ---------------- TRACE (received) ----------------------
    CONFIG, arg={state_transfer=true, protocol_class=org.jgroups.protocols.pbcast.STATE}
    --------------------------------------------------------
    ------------------- TRACE (sent) -----------------------
    CONNECT_USE_FLUSH, arg=zextras
    --------------------------------------------------------
    
    -------------------------------------------------------------------
    GMS: address=zimbra.us.local, cluster=zextras, physical address=192.168.1.252:8736
    -------------------------------------------------------------------
    ---------------- TRACE (received) ----------------------
    VIEW_CHANGE, arg=[zimbra.us.local|0] (1) [zimbra.us.local]
    --------------------------------------------------------
    ---------------- TRACE (received) ----------------------
    BECOME_SERVER, arg=null
    --------------------------------------------------------
    2018-01-23 23:02:54.292:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@6e4784bc{/service,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/service/,AVAILABLE}{/service}
    2018-01-23 23:02:58.058:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
    2018-01-23 23:02:58.320:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@67a20f67{/,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
    2018-01-23 23:02:59.344:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@67a20f67{/,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/,AVAILABLE}{/zimbra}
    2018-01-23 23:03:02.385:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
    2018-01-23 23:03:02.480:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@57c758ac{/zimbraAdmin,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
    2018-01-23 23:03:03.330:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@57c758ac{/zimbraAdmin,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
    2018-01-23 23:03:03.638:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@a9cd3b1{/zimlet,[file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
    2018-01-23 23:03:03.640:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@a9cd3b1{/zimlet,[file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
    2018-01-23 23:03:03.644:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-01-23
    2018-01-23 23:03:03.645:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
    2018-01-23 23:03:03.687:INFO:oejs.ServerConnector:main: Started ServerConnector@2473b9ce{HTTP/1.1,[http/1.1]}{localhost:8080}
    2018-01-23 23:03:03.693:INFO:oejus.SslContextFactory:main: x509=X509@433362e0(jetty,h=[nepbolt.com.au],w=[nepbolt.com.au]) for SslContextFactory@5e1bfe66(file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/etc/keystore,null)
    2018-01-23 23:03:03.703:INFO:oejs.ServerConnector:main: Started ServerConnector@58a90037{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
    2018-01-23 23:03:03.706:INFO:oejs.ServerConnector:main: Started ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
    2018-01-23 23:03:03.708:INFO:oejs.ServerConnector:main: Started ServerConnector@70a9f84e{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
    2018-01-23 23:03:03.710:INFO:oejs.ServerConnector:main: Started ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
    2018-01-23 23:03:03.710:INFO:oejs.Server:main: Started @41218ms
    2018-01-23 23:05:39.125:WARN:oejh.HttpParser:qtp1068934215-398: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@6020f4{r=1,c=false,a=IDLE,uri=null}
    2018-01-23 23:54:34.784:WARN:oejh.HttpParser:qtp1068934215-6543: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@41c22fcc{r=1,c=false,a=IDLE,uri=null}

  3. #3
    Zextras employee Known ZeXtras Reseller
    Participant

    Join Date
    Aug 2016
    Posts
    79
    Hi djzort!

    It's hard to identify similar issues without detailed information.

    The best option is to execute some
    Code:
    /opt/zimbra/libexec/zmjavawatch
    or
    Code:
    /opt/zimbra/libexec/zmdiaglog -jz
    or (this also also collect a JVM heapdump/coredump )
    Code:
    /opt/zimbra/libexec/zmdiaglog -ajz
    when the server is being exhausted, to identify what java thread is consuming the resources.

    I suggest you collect some of this logs using the above commands and open a support ticket on http://www.zextras.com/support/ in order to better investigate this case.

    Have a nice day,
    Luca
    The Zextras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  4. #4
    Active Member
    Join Date
    Jan 2018
    Posts
    11
    Done, request is #26395

  5. #5
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,343
    Hello everyone!

    I'm glad to inform you that our Dev Team found the cause of this issue, which is tied to the synchronization of "dead" share mountpoints triggering a Jetty bug.

    We'll have a custom Zextras Suite 2.6.2 build ready for anyone affected by this issue within 24 hours, in the meantime if you are expreriencing slowliness or service restarts caused by high resource usage please search your mailbox.log and/or sync.log for synchronization errors and check the relevant account for broken/invalid shares.

    Have a nice day,
    Cine
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website
    # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!

    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery

  6. #6
    Active Member
    Join Date
    Jan 2018
    Posts
    11
    Looking forward to it?

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •