Memory leak
Results 1 to 6 of 6

Thread: Memory leak

  1. 01-25-2018, 12:03 AM #1
    djzort
    djzort is offline
    Active Member
    Join Date
    Jan 2018
    Posts
    17

    Memory leak

    Zextras 2.6.2 seems to be leaking memory rather badly.

    I have upgraded to 8.8.6 GA from 8.7.11 and am still seeing the same gradual march towards Out of Memory crash. Running on Ubuntu 14.04.5 LTS

    The server I am running on has 16GB of RAM and ~13 users active on ZxMobile Activesync. Its not really got that much going on but its gobbling RAM.

    Increasing the mailbox heap size has reduced the crashes from 3-4 times per day to 1 per day.

    Current settings are:

    mailboxd_java_heap_new_size_percent = 30
    mailboxd_java_heap_size = 4096
    mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewRatio=2 -XX:PermSize=196m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true

    These are as per the ZImbra wiki page https://wiki.zimbra.com/wiki/Perform...ge_Deployments



    Code:
    2018-01-23 10:02:40.346:WARN:oejh.HttpParser:qtp1068934215-118177: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@7bb80dd0{r=1,c=false,a=IDLE,uri=null}
2018-01-23 22:42:36.043:WARN:oejs.ServletHandler:ZeXtras Mobile: Error for /service/extension/zimbrasync
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at com.zextras.utils.FastBase64.encodeToStream(FastBase64.java:371)
	at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifier$SimpleDataHandler.writeTo(MimeSimplifier.java:126)
	at javax.mail.internet.PreencodedMimeBodyPart.writeTo(PreencodedMimeBodyPart.java:112)
	at javax.mail.internet.MimeMultipart.writeTo(MimeMultipart.java:462)
	at com.sun.mail.handlers.multipart_mixed.writeTo(multipart_mixed.java:103)
	at javax.activation.ObjectDataContentHandler.writeTo(DataHandler.java:889)
	at javax.activation.DataHandler.writeTo(DataHandler.java:317)
	at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
	at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1773)
	at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1749)
	at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifierWithFallback.fixRawMime(MimeSimplifierWithFallback.java:51)
	at com.zextras.modules.mobile.v2.engine.mime.SimplifierProxyBodyContentEncoder.encodeContentProvider(SimplifierProxyBodyContentEncoder.java:93)
	at com.zextras.modules.mobile.v2.as.encoders.commons.EmailApplicationDataEncoder.encode(EmailApplicationDataEncoder.java:112)
	at com.zextras.modules.mobile.v2.as.encoders.commons.SyncEncoder.encode(SyncEncoder.java:173)
	at com.zextras.modules.mobile.v2.as.encoders.commons.LoggerEncoder.encode(LoggerEncoder.java:111)
	at com.zextras.modules.mobile.v2.as.events.AddNewEmailEvent.encode(AddNewEmailEvent.java:39)
	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItem(SyncItemEmitter.java:445)
	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItems(SyncItemEmitter.java:312)
	at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitChanges(SyncItemEmitter.java:108)
	at com.zextras.modules.mobile.v2.engine.actions.CollectionSyncAction.execute(CollectionSyncAction.java:121)
	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
	at com.zextras.modules.mobile.v2.as.assembler.ProbeAction.execute(ProbeAction.java:46)
	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
	at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
	at com.zextras.modules.mobile.v2.engine.actions.StartEndWrapperAction.execute(StartEndWrapperAction.java:31)
	at com.zextras.modules.mobile.v2.engine.actions.CheckFolderHierarchyAction.execute(CheckFolderHierarchyAction.java:98)
	at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execAction(InitialExecutor.java:268)
	at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execute(InitialExecutor.java:211)
Exception in thread "threads.csv" *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "threads.csv"
Exception in thread "sql.csv" Exception in thread "ldap.csv" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Cron" java.lang.OutOfMemoryError: Java heap space

(more, but forum has size limit)







    As mentioned in other threads, ZxMobile seems very CPU heavy. Reducing http threads & turning off interprocess security has helped as has upgrading from 8.7.11 to 8.8.6 - but for a dozen clients to being pushing the server so hard is frustrating.

    i.e.
    /opt/zimbra/bin/zmprov modifyConfig zimbraHttpNumThreads 100
    /opt/zimbra/bin/zmlocalconfig -e zimbra_require_interprocess_security=0
    Reply With Quote Reply With Quote
  2. 01-25-2018, 12:03 AM #2
    djzort
    djzort is offline
    Active Member
    Join Date
    Jan 2018
    Posts
    17
    Trace continues:
    Code:
    Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=zimbra.us.local:389), maxConnections=50)"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NioProcessor-550"
Exception in thread " Activity Timer" java.lang.OutOfMemoryError: Java heap space
Exception in thread "mailboxd.csv" java.lang.OutOfMemoryError: Java heap space
2018-01-23 22:50:57.832:WARN:oejs.ServerConnector:qtp1068934215-142-acceptor-0@3c577368-ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}: 
java.lang.OutOfMemoryError: Java heap space
2018-01-23 22:58:13.195:WARN:oeji.SelectorManager:qtp1068934215-200293: Exception while notifying connection SslConnection@220a86e3{NEED_UNWRAP,eio=-1/-1,di=-1} -> HttpConnection@1d710bad[DecryptedEndPoint@5a869410{/192.168.1.252:50790<->7072,Open,in,out,-,-,346129/60000,HttpConnection}->SelectChannelEndPoint@2c1747e9{/192.168.1.252:50790<->7072,Open,in,out,-,-,46714/60000,SslConnection}{io=0/0,kio=0,kro=0}][p=HttpParser{s=START,0 of 0},g=HttpGenerator@4c38b8a6{s=START},c=HttpChannelOverHttp@3af3025c{r=0,c=false,a=IDLE,uri=null}][b=null]
java.lang.OutOfMemoryError: Java heap space
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=196m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=350m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-01-23 23:02:23.703:INFO::main: Logging initialized @1210ms
JettyMonitor monitoring thread pool qtp1068934215{STOPPED,10<=0<=100,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-01-23 23:02:25.301:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-01-23 23:02:25.301:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-01-23 23:02:25.302:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-01-23 23:02:25.312:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2473b9ce{HTTP/1.1,[http/1.1]}{localhost:8080}
2018-01-23 23:02:25.313:INFO:oejs.SetUIDListener:main: Opened ServerConnector@58a90037{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-01-23 23:02:25.313:INFO:oejs.SetUIDListener:main: Opened ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-01-23 23:02:25.314:INFO:oejs.SetUIDListener:main: Opened ServerConnector@70a9f84e{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-01-23 23:02:25.315:INFO:oejs.SetUIDListener:main: Opened ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-01-23 23:02:25.315:INFO:oejs.SetUIDListener:main: Setting GID=999
2018-01-23 23:02:25.316:INFO:oejs.SetUIDListener:main: Setting UID=101
2018-01-23 23:02:25.323:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-01-23 23:02:32.125:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@6e4784bc{/service,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=101 euid=101 gid=999 egid=999
---------------- TRACE (received) ----------------------
CONFIG, arg={bind_addr=zimbra.us.local/192.168.1.252}
--------------------------------------------------------
------------------- TRACE (sent) -----------------------
SET_LOCAL_ADDRESS, arg=zimbra.us.local
--------------------------------------------------------
---------------- TRACE (received) ----------------------
CONFIG, arg={state_transfer=true, protocol_class=org.jgroups.protocols.pbcast.STATE}
--------------------------------------------------------
------------------- TRACE (sent) -----------------------
CONNECT_USE_FLUSH, arg=zextras
--------------------------------------------------------

-------------------------------------------------------------------
GMS: address=zimbra.us.local, cluster=zextras, physical address=192.168.1.252:8736
-------------------------------------------------------------------
---------------- TRACE (received) ----------------------
VIEW_CHANGE, arg=[zimbra.us.local|0] (1) [zimbra.us.local]
--------------------------------------------------------
---------------- TRACE (received) ----------------------
BECOME_SERVER, arg=null
--------------------------------------------------------
2018-01-23 23:02:54.292:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@6e4784bc{/service,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/service/,AVAILABLE}{/service}
2018-01-23 23:02:58.058:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-01-23 23:02:58.320:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@67a20f67{/,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-01-23 23:02:59.344:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@67a20f67{/,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-01-23 23:03:02.385:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-01-23 23:03:02.480:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@57c758ac{/zimbraAdmin,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-01-23 23:03:03.330:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@57c758ac{/zimbraAdmin,file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-01-23 23:03:03.638:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@a9cd3b1{/zimlet,[file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-01-23 23:03:03.640:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@a9cd3b1{/zimlet,[file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-01-23 23:03:03.644:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-01-23
2018-01-23 23:03:03.645:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-01-23 23:03:03.687:INFO:oejs.ServerConnector:main: Started ServerConnector@2473b9ce{HTTP/1.1,[http/1.1]}{localhost:8080}
2018-01-23 23:03:03.693:INFO:oejus.SslContextFactory:main: x509=X509@433362e0(jetty,h=[nepbolt.com.au],w=[nepbolt.com.au]) for SslContextFactory@5e1bfe66(file:///opt/zimbra/jetty-distribution-9.3.5.v20151012/etc/keystore,null)
2018-01-23 23:03:03.703:INFO:oejs.ServerConnector:main: Started ServerConnector@58a90037{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-01-23 23:03:03.706:INFO:oejs.ServerConnector:main: Started ServerConnector@74294adb{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-01-23 23:03:03.708:INFO:oejs.ServerConnector:main: Started ServerConnector@70a9f84e{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-01-23 23:03:03.710:INFO:oejs.ServerConnector:main: Started ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-01-23 23:03:03.710:INFO:oejs.Server:main: Started @41218ms
2018-01-23 23:05:39.125:WARN:oejh.HttpParser:qtp1068934215-398: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@6020f4{r=1,c=false,a=IDLE,uri=null}
2018-01-23 23:54:34.784:WARN:oejh.HttpParser:qtp1068934215-6543: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@41c22fcc{r=1,c=false,a=IDLE,uri=null}
    Reply With Quote Reply With Quote
  3. 01-26-2018, 03:46 PM #3
    larcara
    larcara is offline
    Zextras employee Known ZeXtras Reseller
    Participant
    Join Date
    Aug 2016
    Posts
    88
    Hi djzort!

    It's hard to identify similar issues without detailed information.

    The best option is to execute some
    Code:
    /opt/zimbra/libexec/zmjavawatch
    or
    Code:
    /opt/zimbra/libexec/zmdiaglog -jz
    or (this also also collect a JVM heapdump/coredump )
    Code:
    /opt/zimbra/libexec/zmdiaglog -ajz
    when the server is being exhausted, to identify what java thread is consuming the resources.

    I suggest you collect some of this logs using the above commands and open a support ticket on http://www.zextras.com/support/ in order to better investigate this case.

    Have a nice day,
    Luca
    The Zextras Team
    IT Support Team Contact Form
    Sales Team Contact Form

    ZeXtras Website     # ZeXtras Wiki # ZeXtras Store

    Have ZeXtras Suite or ZeXtras Migration Tool been helpful to you?
    Share your experience in the Zimbra Gallery!
    ZeXtras Suite on the Zimbra Gallery
    ZeXtras Migration Tool on the Zimbra Gallery
    Reply With Quote Reply With Quote
  4. 01-27-2018, 05:02 AM #4
    djzort
    djzort is offline
    Active Member
    Join Date
    Jan 2018
    Posts
    17
    Done, request is #26395
    Reply With Quote Reply With Quote
  5. 01-29-2018, 03:15 PM #5
    Cine
    Cine is offline
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,361
    Hello everyone!

    I'm glad to inform you that our Dev Team found the cause of this issue, which is tied to the synchronization of "dead" share mountpoints triggering a Jetty bug.

    We'll have a custom Zextras Suite 2.6.2 build ready for anyone affected by this issue within 24 hours, in the meantime if you are expreriencing slowliness or service restarts caused by high resource usage please search your mailbox.log and/or sync.log for synchronization errors and check the relevant account for broken/invalid shares.

    Have a nice day,
    Cine
    VP of TSE/QA and CCM.

    ZeXtras Website     # ZeXtras Wiki # ZeXtras Store

    Zextras Support Portal
    Reply With Quote Reply With Quote
  6. 02-01-2018, 01:21 PM #6
    djzort
    djzort is offline
    Active Member
    Join Date
    Jan 2018
    Posts
    17
    Looking forward to it?
    Reply With Quote Reply With Quote
« Previous Thread | Next Thread »

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Forum Rules

Copyright © 2016 Zextras, All rights reserved.

Zimbra is a registered trademark of Synacor, Inc.

ActiveSync is a registered Trademark of Microsoft Corporation