Java process high load

[iamcypher]

Hello,

I have a zimbra machine that sends about 3k emails (in and out) and I only noticed now that both cores are used 100% by java process.

I have no idea what may cause this and how to identify the problem. I've check memory (2GB free of 6GB), 1.8TB disk space free, the disk IO is not being overloaded, no network overload, no compromised accounts, no emails in queue. Any ideas what might cause this?

                                                   Ubuntu 12.04 64bit with Linux 3.2.0-29-generic on email

PerCPU    100.0%    75.1%    Load      2-Core    Mem      39.9%           Swap     0.1%
User:     65.4      15.7     1 min:    1.25      Total:   7.8G            Total:   8.0G
Kernel:   34.6      59.4     5 min:    1.65      Used:    5.7G  (3.1G)    Used:    6M
Nice:     0.0       0.0      15 min:   1.70      Free:    2.1G  (4.7G)    Free:    8.0G

Network   Rx/s     Tx/s      Processes 162, 1 running, 161 sleeping, 0 other
eth0:     12Kb     13Kb
eth1:     0b       0b        VIRT   RES    CPU%   MEM%   PID    USER      NI  S  TIME+     IO_R    IO_W    NAME
lo:       58Kb     58Kb      157M   4M     99.8   0.1    11811  zimbra    0   S  37:28.83  A_DENY  A_DENY  java
                             75M    13M    3.4    0.2    21022  paul      0   R  00:10.10  0       0       /usr/bin/python /usr/local/bin/glances
Disk I/O  In/s     Out/s     3.3G   961M   0.7    12.0   17531  zimbra    0   S  06:25.85  A_DENY  A_DENY  java
dm-0:     238K     0         46M    9M     0.5    0.1    16896  zimbra    0   S  00:07.30  A_DENY  A_DENY  /usr/bin/perl /opt/zimbra/libexec/zmlogger
dm-1:     0        0         40M    11M    0.2    0.1    16894  zimbra    0   S  00:03.96  A_DENY  A_DENY  /opt/zimbra/lib
sda5:     238K     0         2.8G   404M   0.2    5.1    17458  zimbra    0   S  02:20.98  A_DENY  A_DENY  mysqld
sdb1:     0        0         38M    7M     0.2    0.1    18263  zimbra    0   S  00:00.22  A_DENY  A_DENY  /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io
sdb2:     0        0         38M    7M     0.2    0.1    18269  zimbra    0   S  00:01.15  A_DENY  A_DENY  /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
sdb5:     0        0         47M    7M     0.2    0.1    18684  root      0   S  00:00.75  A_DENY  A_DENY  /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
sr0:      0        0         24M    2M     0.0    0.0    1      root      0   S  00:01.27  A_DENY  A_DENY  /sbin/init
                             0      0      0.0    0.0    2      root      0   S  00:00.25  A_DENY  A_DENY  kthreadd
Mount     Total    Used      0      0      0.0    0.0    3      root      0   S  00:56.14  A_DENY  A_DENY  ksoftirqd/0
/         1.8T     150.1G    0      0      0.0    0.0    5      root      0   S  00:00.36  A_DENY  A_DENY  kworker/u:0
/boot     228M     121M      0      0      0.0    0.0    6      root      0   S  41:19.50  A_DENY  A_DENY  migration/0
/run      1.6G     256K      0      0      0.0    0.0    7      root      0   S  00:05.22  A_DENY  A_DENY  watchdog/0
                             0      0      0.0    0.0    8      root      0   S  00:00.0   A_DENY  A_DENY  migration/1
                             0      0      0.0    0.0    10     root      0   S  01:38.50  A_DENY  A_DENY  ksoftirqd/1
                             0      0      0.0    0.0    12     root      0   S  00:04.85  A_DENY  A_DENY  watchdog/1
                             0      0      0.0    0.0    13     root      -20 S  00:00.0   A_DENY  A_DENY  cpuset
                             0      0      0.0    0.0    14     root      -20 S  00:00.0   A_DENY  A_DENY  khelper
                             0      0      0.0    0.0    15     root      0   S  00:00.0   A_DENY  A_DENY  kdevtmpfs
                             0      0      0.0    0.0    16     root      -20 S  00:00.0   A_DENY  A_DENY  netns
                             0      0      0.0    0.0    18     root      0   S  00:04.10  A_DENY  A_DENY  sync_supers
                             0      0      0.0    0.0    19     root      0   S  00:00.10  A_DENY  A_DENY  bdi-default
                             0      0      0.0    0.0    20     root      -20 S  00:00.0   A_DENY  A_DENY  kintegrityd
                             0      0      0.0    0.0    21     root      -20 S  00:00.0   A_DENY  A_DENY  kblockd
                             0      0      0.0    0.0    22     root      -20 S  00:00.10  A_DENY  A_DENY  ata_sff
                             0      0      0.0    0.0    23     root      0   S  00:00.0   A_DENY  A_DENY  khubd
                             0      0      0.0    0.0    24     root      -20 S  00:00.0   A_DENY  A_DENY  md
                             0      0      0.0    0.0    25     root      0   S  08:42.41  A_DENY  A_DENY  kworker/1:1
                             0      0      0.0    0.0    26     root      0   S  00:01.22  A_DENY  A_DENY  khungtaskd
                             0      0      0.0    0.0    27     root      0   S  01:05.50  A_DENY  A_DENY  kswapd0
                             0      0      0.0    0.0    28     root      5   S  00:00.0   A_DENY  A_DENY  ksmd
                             0      0      0.0    0.0    29     root      19  S  00:00.0   A_DENY  A_DENY  khugepaged
                             0      0      0.0    0.0    30     root      0   S  00:00.0   A_DENY  A_DENY  fsnotify_mark
                             0      0      0.0    0.0    31     root      0   S  00:00.0   A_DENY  A_DENY  ecryptfs-kthrea

WARNING|CRITICAL logs for CPU|LOAD|MEM (lasts 10 entries)
  2012-12-18 12:07:05 > 2012-12-18 12:08:03 CPU-0 kernel (77.1/82.5/85.8)
  2012-12-18 12:07:03 > 2012-12-18 12:07:05 CPU-1 kernel (82.6/82.6/82.6)
  2012-12-18 12:06:40 > 2012-12-18 12:07:01 CPU-0 kernel (75.2/81.8/85.1)
  2012-12-18 12:06:36 > 2012-12-18 12:06:40 CPU-1 kernel (79.1/80.6/82.1)
  2012-12-18 12:06:05 > 2012-12-18 12:06:11 CPU-1 user (87.6/91.6/96.2)
  2012-12-18 12:06:00 > 2012-12-18 12:06:34 CPU-0 kernel (77.9/82.1/85.5)
  2012-12-18 12:05:50 > 2012-12-18 12:05:58 CPU-1 kernel (79.1/82.2/86.1)
  2012-12-18 12:05:31 > 2012-12-18 12:05:48 CPU-0 kernel (80.2/82.9/85.1)
  2012-12-18 12:05:15 > 2012-12-18 12:05:29 CPU-1 kernel (79.0/82.9/85.6)
  2012-12-18 12:05:13 > 2012-12-18 12:05:15 CPU-1 user (70.9/70.9/70.9)

Press 'h' for help                                                                                                                       2012-12-18 12:08:03

I've run two hours ago a zmcontrol restart command and nothing has changed, is it possible that zextras can cause this?

[Cine]

Hello iamcypher!

It's very unlikely for ZeXtras Suite to be the cause.

Just to be sure, disable ZeXtras Backup's Real Time Scanner and see if the cpu load goes down. If the load does not change - or if the Real Time Scanner is not active - I suggest you to search the Zimbra official forums, as there are quite a lot of users experiencing this kind of issue...

Have a nice day,
Cine
the ZeXtras Team

[iamcypher]

I've already tried this and it went down by 20% then back to 100% and remains still.

Thank you.

[Sylv1]

Hello,

Do you work with a virtual machine ? If yes, wich is the host system ?

[iamcypher]

I've fixed the problem. Basically there was a leftover java process that was eating all the cpu. I ran zmcontrol stop, killed the process, zmcontrol start and sunshine .