high cpu load with zimbra + centos

Hi,

I have installed zimbra on centos6.3, and ran it fine for 2-3 days. Now java process is taking up 100-300% of cpu and zimbra web client and admin interface is not loading up. A restart of zimbra makes things ok for a few mins, and then again the same issue reappears.

The only change i see happened is use of imapsync to sync mails from older server. Whenever imapsync runs, java process starts taking up too much cpu and everything else just slows down. I have shutdown imapsync, but even then the issue of web client and admin interface not loading is still happening.

This is the process which i see getting stuck and consuming resources

zimbra 10253 257 37.0 11461116 2989240 ? Sl 20:37 11:27 \_ /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Xss256k -Xms1971m -Xmx1971m -Xmn492m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar OPTIONS=Server,jsp,jmx,resources,websocket,ext,jta ,plus,rewrite,setuid /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml


Zimbra Version: Release 8.0.2_GA_5569.RHEL6_64_20121210115059 CentOS6_64 FOSS edition

Centos 6.3

No. of Users: 290

Server Config: Dual Quad Core, 8GB RAM (KVM instance), RAID 10 on the base machine.

Any help would be highly appreciated.

Thanks

Hello anandi!

Some sparse questions...
- Is it just the CPU usage spiking up or can you also see RAM/Swap usage or IO performances raising to an alarming level?
- How many IMAPsync processes are you running at the same time?
- Any "abnormal" mail activity? IIRC the most cpu-intensive component is amavisd, so a very high mailflow - e.g: a compromised account is being used to send thousands of spam emails - can cause this, especially on Zimbra 8.0.2 (Bug 79585: Antivirus launched twice on outgoing mails).
- Is ZeXtras Suite installed on the server?

Have a nice day,
Cine

Hi Cine,

Replies to your questions:

1. I don't see much of RAM usage. There is no swap usage. Looking at iotop shows little IO happening.

total used free shared buffers cached
Mem: 7869 3812 4057 0 26 578
-/+ buffers/cache: 3206 4662
Swap: 8190 0 8190

2. No imapsync is running, i shut it down.
3. No abnormal mail activity in progress. Its 10:00pm here, and there are hardly any users. Further, when i restart the server / zimbra, it starts to work for few mins before again causing timeout errors.
4. Yes zextras suite is installed. Since GUI is not working, is there a commandline method to disable it ? This way i can check just to be sure if zextras is not causing issues.

Could it be that zextras backup is backing up / working on the new mails i imported via imapsync ? Just thinking...

Thanks

Looking at /opt/zimbra/log/mailbox.log, i see a lot of sqltrace - slow execution messages:

sqltrace - Slow execution (15435ms): UPDATE mailbox SET item_id_checkpoint = 719, contact_count = 25, change_checkpoint = 3200, size_checkpoint = 29527373, new_messages = 35 WHERE id = 188

Can this be the reason ?

Quote:

---

Originally Posted by anandi

Looking at /opt/zimbra/log/mailbox.log, i see a lot of sqltrace - slow execution messages:

sqltrace - Slow execution (15435ms): UPDATE mailbox SET item_id_checkpoint = 719, contact_count = 25, change_checkpoint = 3200, size_checkpoint = 29527373, new_messages = 35 WHERE id = 188

Can this be the reason ?

---

Hello anandi!

This is very likely to be a side effect, not a cause...

Have a nice day,
Cine

Ok. Any other suggestions to resolve this ? How do i switch off Zextras from command line ?

Quote:

---

Originally Posted by anandi

Ok. Any other suggestions to resolve this ? How do i switch off Zextras from command line ?

---

Hello Anandi!

In order to uninstall the ZeXtras Core simply run the installer with the "-u core" option. This will remove the ZeXtras Core.

However, I'm afraid this is not the cause of the issue... In order to further help you I'd need you to answer the questions I asked a couple of posts back...

Have a nice day,
Cine

Hi Cine,

I already replied to the questions, please see. Let me know if anything else needs to be answered.

Thanks

Hello Anandi!

Visiting the Zimbra Forums I saw your thread: high cpu load with zimbra + centos

I'm really glad you solved the problem and that, as expected, ZeXtras Suite was not the cause...

Have a nice day,
Cine

Hi Cine,

Thanks. Yes i am glad as well that it wasn't zextras causing the extra load. However i am still getting a small error of proxy not found when i go to web client -> contacts -> distribution lists. Any ideas what could be causing it ?

Thanks