Epic Data Corruption In Need Of Help

[iamcypher]

Hello,

I'm still in trial of a zextras backup installation and my raid 1 got degraded and I lost a lot of data. I managed to get a full scan yesterday on a nas device. Also I managed to recover my store and index folder of the live one.

I am reinstalling zimbra right now on a new computer, installing zextras backup.

• Can I restore the data from my zextras backup of my old installation, if I change the folder to the nas share, will he break my backup? Will I be able to restore the data?
• If I have a live copy of store and index and I overwrite the data after the zextras restore, run zmfixperms, should I have the live data up and running?

Thank you for your support.
cypher.

[kral]

Hello,

Hi, I'm one of the developer of ZeXtras.

• Can I restore the data from my zextras backup of my old installation, if I change the folder to the nas share, will he break my backup? Will I be able to restore the data?
• If I have a live copy of store and index and I overwrite the data after the zextras restore, run zmfixperms, should I have the live data up and running?

Yes, absolutely!

I suggest you to mount fullscan data from the NAS (or copy them for better performances) to the new server and run an external import to restore your data.

The store and index directories are not of any help because LDAP data for all users and their information are different on the new server (ex. account and folder IDs are different).

Best regards,
--
kral

[iamcypher]

Thank you for the answer. I need a way of recovering using the store and index folder otherwise I lose a lot of emails. There really is no way at all?

[Cine]

Hello iamcypher,
this is Cine.

If the Real Time Scanner was enabled on your old server, just perform an external import using the old Backup Path as the import path. This will restore all your mailboxes exactly as they were one second before the RAID crash.

Else, the last available backup is the one contained in the NAS... The store and index folders are of no use unless you also have all other Zimbra folders available - these folders contain the actual data but the structure of such data is contained in the LDAP and in the mysql database.

(I edited Kral's post to remove some inconsistencies... Devs are not supposed to explain how their software works )

[iamcypher]

Big problem. I get an exception when I try to restore the backup:

2012-05-12 00:34:45,531 INFO [ZxBackup Thread] [] extensions - begin restoreFromExternalBackup
2012-05-12 00:34:45,616 WARN [ZxBackup Thread] [] extensions - ZxBackup Thread: java.lang.NullPointerException
at com.zextras.backup.ZEBackupServer.getBackup(ZEBack upServer.java:250)
at com.zextras.backup.ZEExternalRestoreOperation.doOp eration(ZEExternalRestoreOperation.java:320)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 81)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:200)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:110)

ZxBackup version 1.2.7 (latest) .

I tried chmodding and chowning to user zimbra and to user root and I keep getting this error. Please advise.

[zimbra@email bin]$ ./zxsuite backup doExternalRestore /restore/nas


server email.censored.com
operationId a2d67a38-9909-4b0f-9fd2-23b0a28a21fa

I tried running as root and as zimbra.

Here are the contents of /restore/nas :

[root@email bin]# ls /restore/nas
accounts backupstat dirtyList items lock_8c6a7eb4-41bb-4947-8ffd-e9f987ecc5b1 server

Here is debug data:

2012-05-12 00:43:11,915 INFO [ZxBackup Thread] [] extensions - ZxBackup Thread: New Operation: External Restore
2012-05-12 00:43:11,915 INFO [ZxBackup Thread] [] extensions - ZeXtras Notifier: Sending notification: This is an automated notification from ZxBackup about External Restore.

Operation External Restore Started.

Operation Id: de6df55b-2fc5-471e-8b36-720bf7cb3133
Operation Host: email.censored.com
Operation Log path: /opt/zimbra/log/zextras_ZxBackup_ExternalRestore_de6df55b-2fc5-471e-8b36-720bf7cb3133.log

- debug info -
-- parameters --
humanDate = last available
notificationeMails =
accountNameList = null
backupPath = /restore/nas
date = null
domainNameList = null
-- context --

2012-05-12 00:43:12,014 INFO [ZxBackup Thread] [] extensions - begin restoreFromExternalBackup
2012-05-12 00:43:12,015 INFO [ZxBackup Thread] [] extensions - ZeXtras Notifier: Sending notification: This is an automated notification from ZxBackup about External Restore.

Operation External Restore failed.

Operation Id: de6df55b-2fc5-471e-8b36-720bf7cb3133
Operation Host: email.censored.com

- exception -
java.lang.NullPointerException
at com.zextras.backup.ZEBackupServer.getBackup(ZEBack upServer.java:250)
at com.zextras.backup.ZEExternalRestoreOperation.doOp eration(ZEExternalRestoreOperation.java:320)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 81)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:200)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:110)

- debug info -
-- parameters --
humanDate = last available
OpStartTime = 1336776192013
notificationeMails =
accountNameList = null
backupPath = /restore/nas
date = null
domainNameList = null
-- context --
completed = 1

2012-05-12 00:43:12,110 WARN [ZxBackup Thread] [] extensions - ZxBackup Thread: java.lang.NullPointerException
at com.zextras.backup.ZEBackupServer.getBackup(ZEBack upServer.java:250)
at com.zextras.backup.ZEExternalRestoreOperation.doOp eration(ZEExternalRestoreOperation.java:320)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 81)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:200)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:110)

[Cine]

Hello iamcypher,

the error you experienced is caused by an I/O error on the "server" subfolder of your backup.

I see 2 possible scenarios:

* The "zimbra" user lacks read permissions on the "server" subfolder (or on the entire /restore/nas/ folder)
* The "server" subfolder is corrupted

Please check if the Zimbra user has read permissions on the /restore/nas/ folder and in all its subfolders...
If permissions are ok, please check the "server" folder itself to see if there is anything abnormal (try running "ls -la /restore/nas/", "ls -la /restore/nas/server/", "touch /restore/nas/server/testfile", "rm /restore/nas/server/testfile" and post the outputs).

Have a nice day,
Cine

[iamcypher]

Hello,

Thank you for your answer. It started restoring now however it's skipping a LOT of emails because he says that they don't exist. My log is being spammed with a lot of restoring items and every 3 or 4 successful restores I get a missing file:

2012-05-12 09:37:53,010 WARN [ZxBackup Thread] [] extensions - Unable to restore item 272, account admin@censored.com(708b0c35-129d-4953-8708-fdfc5deb0b33):
Unknown Exception in restore
at com.zextras.backup.ZEBItem.restore(ZEBItem.java:56 9)
at com.zextras.backup.ZEExternalRestoreOperation.rest oreItemsFromExternal(ZEExternalRestoreOperation.ja va:584)
at com.zextras.backup.ZEExternalRestoreOperation.doOp eration(ZEExternalRestoreOperation.java:474)
at com.zextras.op.ZEOperation.exec(ZEOperation.java:5 81)
at com.zextras.backup.ZEBackupThread.execOperation(ZE BackupThread.java:200)
at com.zextras.backup.ZEBackupThread.run(ZEBackupThre ad.java:110)
Caused by: Unable to restore MailItem
at com.zextras.backup.ZEBItem.restore(ZEBItem.java:55 8)
... 5 more
Caused by: java.io.FileNotFoundException: /restore/nas/items/dm/dmWlDXju+qFJQh8bPNXQ00kkrOI= (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.jav a:120)
at com.zextras.backup.ZEBItem.createZimbraItem(ZEBIte m.java:1224)
at com.zextras.backup.ZEBItem.restore(ZEBItem.java:52 9)
... 5 more

A LOT OF THEM.

Is this possible? I mean how come he is missing so many files? I am sure I copied a verbatim copy of whatever was saved on the nas device.

Does this mean that only a part of my emails will get restored or they are false positive alerts?

Thank You.

Edit: I just checked and the file in the exception /restore/nas/items/dm/dmWlDXju+qFJQh8bPNXQ00kkrOI= exists, belongs to user zimbra (uid 500) and to group zimbra (gid 500) and is has chmod 0777.

Here is the output you requested:

[zimbra@email bin]$ ls -la /restore/nas
total 56
drwxrwxrwx. 5 zimbra zimbra 4096 May 11 21:15 .
drwxr-xr-x. 3 zimbra zimbra 4096 May 12 00:19 ..
drwxrwxrwx. 62 zimbra zimbra 4096 May 11 20:45 accounts
-rwxrwxrwx. 1 zimbra zimbra 353 May 11 20:45 backupstat
-rwxrwxrwx. 1 zimbra zimbra 296 May 11 20:45 dirtyList
drwxrwxrwx. 1446 zimbra zimbra 32768 May 11 21:15 items
-rwxrwxrwx. 1 zimbra zimbra 0 May 11 21:15 lock_8c6a7eb4-41bb-4947-8ffd-e9f987ecc5b1
drwxrwxrwx. 2 zimbra zimbra 4096 May 11 21:15 server
[zimbra@email bin]$ ls -la /restore/nas/server
total 464
drwxrwxrwx. 2 zimbra zimbra 4096 May 11 21:15 .
drwxrwxrwx. 5 zimbra zimbra 4096 May 11 21:15 ..
-rwxrwxrwx. 1 zimbra zimbra 75957 May 11 21:15 backup_10_05_12#11_01_27
-rwxrwxrwx. 1 zimbra zimbra 75957 May 11 21:15 backup_10_05_12#11_01_36
-rwxrwxrwx. 1 zimbra zimbra 75994 May 11 21:15 backup_10_05_12#23_28_27
-rwxrwxrwx. 1 zimbra zimbra 75993 May 11 21:15 backup_11_05_12#00_06_00
-rwxrwxrwx. 1 zimbra zimbra 75993 May 11 21:15 backup_11_05_12#00_06_10
-rwxrwxrwx. 1 zimbra zimbra 75993 May 11 21:15 backup_11_05_12#04_01_12
[zimbra@email bin]$ touch /restore/nas/server/testfile
[zimbra@email bin]$ rm -f /restore/nas/server/testfile

Edit2: I correct myself the file does not exist, there is a small error:

/restore/nas/items/dm/dmWlDXju+qFJQh8bPNXQ00kkrOI=

actually is

/restore/nas/items/dM/dmWlDXju+qFJQh8bPNXQ00kkrOI=

What could possibly cause this error in the zextras indexes?

Edit3: I noticed someone else was having a simillar problem, is there any chances as this is a known problem and there is a fix for it?

Edit4: Result:

http://pastebin.com/B6Jw913t

The result is unacceptable. Please advise.

[Cine]

Hello iamcypher,

we are checking the issue as we speak and we'll provide a solution as soon as possible.

Have a nice day,
Cine

[Cine]

Hello iamcypher.

A quick question: was the NAS mounted as a CIFS share when you performed the Full Scan operation?

[iamcypher]

Hello,

I was using the your advice to store all the backup data on a cifs share and yes the full scan happened on a cifs share not locally. Actually all the backup data was stored on a cifs share. When I did the restore I copied all the data from the cifs share back to the computer.