Repeatable mailstore/zxChat problem

[zeol]

Folks, today I have spotted a serious problem.
After doing the upgrade to 2.4.0 the HSM stopped working on all mailstores.
Whenever I try to manually run a HSM session I get:

[zimbra@zstore04 backup]$ zmhsm -t
system failure: hsm is only supported on the standard file-per-blob store


2016-10-05 13:44:29,929 INFO [qtp509886383-311053:https://127.0.0.1:7071/service/admin/soap/HsmRequest] [name=zimbra;ip=127.0.0.1;ua=lmc/8.6.0_GA_1200;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: hsm is only supported on the standard file-per-blob store
ExceptionId:qtp509886383-311053:https://127.0.0.1:7071/service/admin...2efa0da83fcfc7
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:260)
at com.zimbra.cs.hsm.BlobMover.<init>(BlobMover.java: 58)
at com.zimbra.cs.hsm.BlobMoverThread.<init>(BlobMover Thread.java:41)
at com.zimbra.cs.hsm.BlobMoverThread.startThread(Blob MoverThread.java:61)
at com.zimbra.cs.service.hsm.Hsm.handleNetworkRequest (Hsm.java:106)
at com.zimbra.cs.service.NetworkDocumentHandler.handl e(NetworkDocumentHandler.java:23)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:581)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:435)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:266)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.jav a:303)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:213)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:707)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1651)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter (RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(Set HeaderFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilte r(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(Gzi pFilter.java:351)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ET agHeaderFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBa lancerFilter.doFilter(ContextPathBasedThreadPoolBa lancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(Zim braQoSFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doF ilter(ZimbraInvalidLoginFilter.java:117)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain (DoSFilter.java:457)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSF ilter.java:326)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSF ilter.java:299)
at org.eclipse.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlet.ServletHandler.doHandle( ServletHandler.java:549)
at org.eclipse.jetty.server.handler.ScopedHandler.han dle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle( SecurityHandler.java:544)
at org.eclipse.jetty.server.session.SessionHandler.do Handle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.do Handle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(S ervletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.do Scope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.do Scope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.han dle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCol lection.handle(ContextHandlerCollection.java:199)
at org.eclipse.jetty.server.handler.HandlerCollection .handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.ha ndle(HandlerWrapper.java:97)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.h andle(RewriteHandler.java:309)
at org.eclipse.jetty.server.handler.DebugHandler.hand le(DebugHandler.java:81)
at org.eclipse.jetty.server.handler.HandlerWrapper.ha ndle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java :462)
at org.eclipse.jetty.server.HttpChannel.handle(HttpCh annel.java:279)
at org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:232)
at org.eclipse.jetty.io.AbstractConnection$2.run(Abst ractConnection.java:534)
at org.eclipse.jetty.util.thread.QueuedThreadPool.run Job(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.r un(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)


I know this is because of ZeXtras because I've remove ZeXtras Suite on one of the mailstores and now I can successfully run HSM.

Please note that I have disabled all modules except for ZxChat.

Is there anything I missed?

[Cine]

Hello zeol!

I'm investigating the issue with the Dev Team, I'll report back to you as soon as possible...

[zeol]

Thanks Cine, I can see you're the most active member of this forum :-)
I have reconfirmed the issue on another mailstore. Whenever I uninstall Zimbra I can run HSM as usual, after installing ZeXtras 2.4.0 I no longer can run HSM.

[zeol]

We are getting full on all our stores so it's very likely I'll be doing emergency downgrade to previous version (2.2.5) to see if it helps.

[zeol]

Extra question.

It looks like upgrading ZeXtras from 2.2.5 to 2.4.0 renewed the trial license
I'm not going to complain but is this expected behaviour ?
Also I've noticed that module ZxPowerStore is NOT enabled on Network edition (according to zxsuite core getlicenseinfo), please check below:

zxsuite core getlicenseinfo

dateStart 2016-10-02 22:56:11 CEST
dateEnd 2016-11-01 22:56:11 CET
expired false
type Trial
accountCount 4084
resourceCount 124
licensedUsers unlimited
isWithinGraceInterval false
modules
ZxBackup
quantity unlimited
licensed true
ZxMobile
quantity unlimited
licensed true
ZxAdmin
quantity unlimited
licensed true
ZxChat
quantity unlimited
licensed true
ZxPowerstore
quantity unlimited
licensed true
enabledOnNE false
serverID 009370d3-9875-460b-a1dd-25cfc55c4132
canRemoveChatBrand true

could this be the reason behind HSM not being able to work properly ?

[Cine]

Hello, Zeol!

The Dev Team is still investigating the issue, so I can't give any final answer at the moment, but the cause of it seems to be related to the recent changes to the Powerstore module indeed...

Have a nice day,
Cine
the Zextras Team

[zeol]

Quick update.
After changing: zxsuite powerstore setProperty ZxPowerstore_EnableHsmOnNe true I was able to use ZxPowerStore to perform HSM operation.
Quite on time as my store was 95% full

Not sure why I have missed this property before.

[Klug]

I had this issue too on a NE platform on which I'm using ZeXtras for other things (chats and export)...
The NE-HSM was not working anymore.

Eventually removed ZeXtras this morning.

[zeol]

I had this issue too on a NE platform on which I'm using ZeXtras for other things (chats and export)...
The NE-HSM was not working anymore.

Eventually removed ZeXtras this morning.

Not a valid solution for me, but thanks for sharing

[Cine]

Hello everyone!

I'm glad to inform you that our Dev Team found and fixed the cause of the zmhsm/zmblobchk errors, which will be included in Zextras Suite 2.4.1 (scheduled for release before the end of this week).

About the topic issue for this thread, we found out that the high number of Zextras Chat requests are not the cause of the connection issues but just another symptom - just like the 502 errors to clients... We're further investigating hunting for the actual cause but this takes a lot of time since we need to go through and possible option and through a lot of log files.

Have a nice day,
Cine
the Zextras Team