Unable to access Zimbra file - FullScan error

[cinmar]

Hi everybody, last week for the first time we received an error notification after a Full Scan. Here is the content of the mail:

- stats -
new accounts: 0
accounts updated: 6
skipped accounts(by COS): 51
item updated: 5203
new metadata: 972
new files: 759
checked items: 921279
backup path: /opt/zimbra/backup/zextras/
skipped items: 3
items/sec: 116.53951
additional notification mails:
Exceptions: item: 10691 on account ******@******
reason:Unable to access zimbra file: lfIElMuflBMVz1UDLDYweBUhb9g=
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:982)
at com.zextras.backup.ZEBAccount.synchronize(ZEBAccou nt.java:1316)
at com.zextras.op.backup.ZEScanOperation.doItemsScan( ZEScanOperation.java:378)
at com.zextras.op.backup.ZEScanOperation.doAllItemsSc an(ZEScanOperation.java:335)
at com.zextras.op.backup.ZEScanOperation.doAccountSca n(ZEScanOperation.java:291)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:72)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:63)
at com.zextras.op.backup.ZEFullScanOperation.doOperat ion(ZEFullScanOperation.java:326)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 74)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:197)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:107)
Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to get content for Message
10691
ExceptionId:ZxBackup Thread:1347865518565:2185a26af0f4ea14
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.mailbox.MailItem.getContentStream(Ma ilItem.java:996)
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:975)
... 10 more
Caused by: java.io.FileNotFoundException: /opt/zimbra/store/0/146/msg/2/10691-20520.msg does not exist
at com.zimbra.cs.store.BlobInputStream.<init>(BlobInp utStream.java:136)
at com.zimbra.cs.store.BlobInputStream.<init>(BlobInp utStream.java:97)
at com.zimbra.cs.store.file.FileBlobStore.getContent( FileBlobStore.java:345)
at com.zimbra.cs.store.file.FileBlobStore.getContent( FileBlobStore.java:339)
at com.zimbra.cs.mailbox.MailItem.getContentStream(Ma ilItem.java:993)
... 11 more
item: 13278 on account ******@******
reason:Unable to access zimbra file: fdyN,9YqLT52KJns3d1f36VsmrQ=
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:982)
at com.zextras.backup.ZEBAccount.synchronize(ZEBAccou nt.java:1316)
at com.zextras.op.backup.ZEScanOperation.doItemsScan( ZEScanOperation.java:378)
at com.zextras.op.backup.ZEScanOperation.doAllItemsSc an(ZEScanOperation.java:335)
at com.zextras.op.backup.ZEScanOperation.doAccountSca n(ZEScanOperation.java:291)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:72)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:63)
at com.zextras.op.backup.ZEFullScanOperation.doOperat ion(ZEFullScanOperation.java:326)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 74)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:197)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:107)
Caused by: com.zimbra.common.service.ServiceException: system failure: Unable to get content for Message
13278
ExceptionId:ZxBackup Thread:1347865734983:2185a26af0f4ea14
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.mailbox.MailItem.getContentStream(Ma ilItem.java:996)
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:975)
... 10 more
Caused by: java.io.FileNotFoundException: /opt/zimbra/store/0/135/msg/3/13278-20425.msg does not exist
at com.zimbra.cs.store.BlobInputStream.<init>(BlobInp utStream.java:136)
at com.zimbra.cs.store.BlobInputStream.<init>(BlobInp utStream.java:97)
at com.zimbra.cs.store.file.FileBlobStore.getContent( FileBlobStore.java:345)
at com.zimbra.cs.store.file.FileBlobStore.getContent( FileBlobStore.java:339)
at com.zimbra.cs.mailbox.MailItem.getContentStream(Ma ilItem.java:993)
... 11 more
item: 10684 on account ******@******
reason:Unable to access zimbra file: N5c4RWC+HpK52SOZiFLTvaZnNaM=
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:982)
at com.zextras.backup.ZEBAccount.synchronize(ZEBAccou nt.java:1316)
at com.zextras.op.backup.ZEScanOperation.doItemsScan( ZEScanOperation.java:378)
at com.zextras.op.backup.ZEScanOperation.doAllItemsSc an(ZEScanOperation.java:335)
at com.zextras.op.backup.ZEScanOperation.doAccountSca n(ZEScanOperation.java:291)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:72)
at com.zextras.op.backup.ZEScanOperation.doDomainScan (ZEScanOperation.java:63)
at com.zextras.op.backup.ZEFullScanOperation.doOperat ion(ZEFullScanOperation.java:326)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 74)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:197)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:107)
Caused by: com.zimbra.cs.mailbox.MailServiceException: No such blob: mailbox=175, item=10684,
change=21740
ExceptionId:ZxBackup Thread:1347870494111:2185a26af0f4ea14
Code:mail.NO_SUCH_BLOB
at com.zimbra.cs.mailbox.MailServiceException.NO_SUCH _BLOB(MailServiceException.java:269)
at com.zimbra.cs.mailbox.MailItem.getBlob(MailItem.ja va:971)
at com.zimbra.cs.mailbox.MailItem.getContentStream(Ma ilItem.java:990)
at com.zextras.backup.ZEBItem.synchronize(ZEBItem.jav a:975)
... 10 more

After this notification, I scheduled another FullScan during the night, and I received another error notification, that refers only to the last object of the first notification (account and item number are the same):

Do you have any clue about this situation? Do you wonder why there's only the last error in the second notification?
Thank you in advance.

[Trantor]

Hello cinmar,
the sort of errors you have described are a normal phenomenon, due to, let's say, «race conditions» between the Zimbra core proper and the ZxBackup module. For instance, in the time between ZxBackup listing the item as one to process and actually processing it Zimbra might have deleted it, to satisfy the request of a client, for instance.
Seeing that happen for the same item repeatedly over time is a symptom of something different, however. The item, for instance, might have become corrupted for some reason ( a system crash or who knows what ), or some stale metadata might be pointing to a non-existent item. Performing a reindexing of the mailbox would be a good first step towards solving the mystery. Should that solve nothing, try to log in that account and use «item: 10684» as the search string, and see what item that is.

Regards,
Trantor

[cinmar]

Re-indexing the mailbox solved the problem (we ran two full scans until today and they didn't report any error). We'll see if this problem occurs again in the future.
Thank you!