External import multiple mailbox servers
Results 1 to 8 of 8
Like Tree1Likes
  • 1 Post By Jay

Thread: External import multiple mailbox servers

  1. #1
    j2b
    j2b is offline
    Active Member
    Join Date
    Dec 2012
    Location
    Riga, Latvia
    Posts
    11

    External import multiple mailbox servers

    In case of multi-server Zimbra installation, where multiple Mailbox servers are used, how zeXtras suite backup decides to which server to recover or import particular mailbox? This taking in mind, that originaly there was only one mailbox server. Both servers are connected to server pool for COS.

    And another issue, after initial backup import, there is notification - External Restore failed. Does this mean, that this restore imported almost all data, except with error ones, or it stops operations on first error, thus leaving other data still to be imported after? Or simpler, do we have to solve indicated errors, to completely make data import 100%?

    And another - how should I try to troubleshoot this exception?

    - exception -
    [ExternalRestore] Unknown Exception
    at com.zextras.op.backup.ZEExternalRestoreOperation.d oOperation ( ZEExternalRestoreOperation.java:715 )
    at com.zextras.op.ZEOperation.exec ( ZEOperation.java:593 )
    at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:196 )
    at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:106 )
    Caused by: com.zimbra.common.service.ServiceException: operation sent to wrong host (you want 'mailbox1.domain.com')
    ExceptionId:ZxBackup Thread:1355251147257:7ed7e85d43b34d1d
    Code:service.WRONG_HOST Arghost, STR, "mailbox1.domain.com")
    at com.zimbra.common.service.ServiceException.WRONG_H OST ( ServiceException.java:314 )
    at com.zimbra.cs.mailbox.MailboxManager.lookupAccount WithHostCheck ( MailboxManager.java:362 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccountId ( MailboxManager.java:347 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccountId ( MailboxManager.java:309 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccount ( MailboxManager.java:255 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccount ( MailboxManager.java:214 )
    at com.zextras.op.backup.ZEExternalRestoreOperation.d oOperation ( ZEExternalRestoreOperation.java:578 )
    at com.zextras.op.ZEOperation.exec ( ZEOperation.java:593 )
    at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:196 )
    at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:106 )

    * mailbox1.domain.com - replaced actual mailbox#1 server. Operations introduced on mailbox#2 server.

  2. #2
    CTO ZeXtras Employee d0s0n's Avatar
    Join Date
    Apr 2011
    Posts
    571
    Hi j2b, finally someone who asks questions about multiserver environment

    Quote Originally Posted by j2b View Post
    In case of multi-server Zimbra installation, where multiple Mailbox servers are used, how zeXtras suite backup decides to which server to recover or import particular mailbox? This taking in mind, that originaly there was only one mailbox server. Both servers are connected to server pool for COS.
    The ZxBackup module must have an instance in every mailbox server because it works only with local data. So if you ask to recover a single user-mailbox already present, zxbackup searches on which server is, in ldap and it submits the operation to that server. Else if the account was deleted and you don't know where it was, you need to use the "restore" button on the zxbackup module for each mailbox server giving a specific date in the past.

    Quote Originally Posted by j2b View Post
    And another issue, after initial backup import, there is notification - External Restore failed. Does this mean, that this restore imported almost all data, except with error ones, or it stops operations on first error, thus leaving other data still to be imported after? Or simpler, do we have to solve indicated errors, to completely make data import 100%?
    If you got an "External Restore failed", the operation was interrupted by a critical exception, so we need to find the cause and then the operation can be restarted, it will continue skipping the already imported items.

    Quote Originally Posted by j2b View Post
    And another - how should I try to troubleshoot this exception?

    - exception -
    [ExternalRestore] Unknown Exception
    at com.zextras.op.backup.ZEExternalRestoreOperation.d oOperation ( ZEExternalRestoreOperation.java:715 )
    at com.zextras.op.ZEOperation.exec ( ZEOperation.java:593 )
    at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:196 )
    at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:106 )
    Caused by: com.zimbra.common.service.ServiceException: operation sent to wrong host (you want 'mailbox1.domain.com')
    ExceptionId:ZxBackup Thread:1355251147257:7ed7e85d43b34d1d
    Code:service.WRONG_HOST Arghost, STR, "mailbox1.domain.com")
    at com.zimbra.common.service.ServiceException.WRONG_H OST ( ServiceException.java:314 )
    at com.zimbra.cs.mailbox.MailboxManager.lookupAccount WithHostCheck ( MailboxManager.java:362 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccountId ( MailboxManager.java:347 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccountId ( MailboxManager.java:309 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccount ( MailboxManager.java:255 )
    at com.zimbra.cs.mailbox.MailboxManager.getMailboxByA ccount ( MailboxManager.java:214 )
    at com.zextras.op.backup.ZEExternalRestoreOperation.d oOperation ( ZEExternalRestoreOperation.java:578 )
    at com.zextras.op.ZEOperation.exec ( ZEOperation.java:593 )
    at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:196 )
    at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:106 )

    * mailbox1.domain.com - replaced actual mailbox#1 server. Operations introduced on mailbox#2 server.
    What steps did you follow for the migration?
    On a multiserver enviroment, the export and import process is a bit more delicate, have you followed the multistore wiki page?

    D0s0n
    ZeXtras Website # ZeXtras Wiki # ZeXtras Store

    Head of ZeXtras System Administrators

  3. #3
    j2b
    j2b is offline
    Active Member
    Join Date
    Dec 2012
    Location
    Riga, Latvia
    Posts
    11
    ...finally someone who asks questions about multiserver environment...
    Oh, yes, please. If this topic is very thin, I could provide some extra pages in this section Any way, if I might be of use, you can freely catch me up. And, of course, have to say, that your product solved a week wide problem in 4 hours. Although not as cleanly as I expected. That's why I started to dig deeper I'd definitely advised this product to Zimbra people, unless they feel comfortable and have time to go with CLI and scripts on their own. I have the feeling, that I can pass this burden to you, guys. Any way, result is better and easier.

    The ZxBackup module must have an instance in every mailbox server because it works only with local data. So if you ask to recover a single user-mailbox already present, zxbackup searches on which server is, in ldap and it submits the operation to that server. Else if the account was deleted and you don't know where it was, you need to use the "restore" button on the zxbackup module for each mailbox server giving a specific date in the past.
    OK, what does it mean - "...it submits the operation to that server..."? As from context, I'd assume, that it can submit operation in terms of notice or message to that particular server's process, but that would not be true regarding backup data. Right? At least, if such backup data is not accessible via shared mounted share. Right?

    In our case, there are 2 new mailbox servers, where in this particular situation, separation is for admin/customer purposes, but this is a step forward to implementation of n-customers' mailbox servers. The only difference here for now, is that main ZCS domain "default" is on mb1, all the others - on mb2. I initiated backup via mb2, but for some reason some domains or domain tried to access mb1, which is not clear for me. That's why I started discussion from higher shelf.

    And for initial backup (or actually import of new mailboxes) there are no such domains and mailboxes at all, so ZXB has to decide, where to place these new mailboxes. Or this is done to current mb server, operation is started?

    If you got an "External Restore failed", the operation was interrupted by a critical exception, so we need to find the cause and then the operation can be restarted, it will continue skipping the already imported items.
    It really does not seem, it's working Microsoft file copy way - start copying a couple of gigabytes large folder structure, catch an error on 60%, and all operations are stopped and have to be started from beginning, unless you desire to do manual folder checking I know, there are other tools to do this But still, checking manually, all data seems to be present, regarding e-mail messages, contacts, calendar entries, and briefcase entries. Although missing parts of tags/versions and some other additional data.

    So if this catches "Exteral Restore failed", repeating restore, does it stop on the same place, and does not do anything further?

    Here are another findings, regarding import, which might be connected with this error:
    • Tag list is imported, mails partially are tagged, BUT - taglist does not have any associations with tags on mails. E.g. tags on mails on hover display particular tag ID, which reminds me to think, that there is no connection between this particular tag and taglist. Could be probably due to partial data import.
    • There were some queries in status mails, which could effect import, but issuing query at users search field, returns mails correctly, e.g. items: xxx,yyy.
    • Mail signatures are recovered partially. Still have not got any pattern to troubleshoot this, as well as found any caus. Some mailboxes have signatures imported, some not.
    • Briefcase file/web doc versions missing - only last version imported.


    What steps did you follow for the migration?
    Sh...t, I had a feeling, that I miss something. I can't believe, I missed that wiki. I was going with this one: ZxBackup: Incremental migration with ZeXtras Backup - ZeXtras Suite Wiki

  4. #4
    j2b
    j2b is offline
    Active Member
    Join Date
    Dec 2012
    Location
    Riga, Latvia
    Posts
    11
    In addition to your provided WIKI (ZxBackup: Multistore Informations - ZeXtras Suite Wiki). It seems, that I organically followed mentioned RULES, as all servers work as expected.

    As to the wiki points:
    In a Multistore Environment, Restore on New Account operations ALWAYS create the new account in the Source account's mailbox server.
    Initial data import was done via Admin UI and from mailbox2 server particularly. How does zextras decide, which is Source account's mailbox? No connections to actual source server were done, and new servers have different hostnames.

    If a wrong target server for an operation is chosen, Zimbra automatically proxies the operation request to the right server.
    I rsynced backup files to particular mailbox2 locally mounted disk, which is not shared between mailbox servers. If this requrest is proxied to another mailbox (in case of administration and internal accounts), it should be true, that operation would fail, as mailbox1 in my case has no access to backup source data. Right?

    We have one domain, where there are 2 accounts on both servers, but recovery data does not contain these accounts anyway. Just a note on domain, that could deploy mailboxes on any of mailbox servers.

    Importing multiple accounts of a single domain to different store will break the consistency of ALL the items that are shared from/to a mailbox on a different server.
    I got caught on this, but as it was only one account, it was quite easy to reestablish shares manually. Didn't manage to test doFixShares, although run it once. As a result, other shares within the same mailbox server didn't recover, nor between mailboxes too. Just gave me a list of shares found. Should I expect other action results?

  5. #5
    j2b
    j2b is offline
    Active Member
    Join Date
    Dec 2012
    Location
    Riga, Latvia
    Posts
    11
    Another issue to add - Saved searches not moved.

    I still would ask your assistance or advice on how to troubleshoot import errors, to find out, whether this is connected with backup, or the problem root is somewhere else. (Reminding, that I migrated zcs 7.1.2 to 8.0.1, which implements different new solutions, although have not checked whether particular issues are related with some changes in ZCS.

    Yet still have to add, that not all imported accounts have the same issues consistently.

    Another fast question: I've tried to import backup once more on top of already imported account, which in addition has received new mails and created new objects within new ZCS, and this import didn't overwrite anything new. IS THIS SAFE TO DO AGAIN AND AGAIN, TO TROUBLESHOOT, or I have to count on potential info overwriting or data loss?
    Last edited by j2b; 12-12-2012 at 08:30 PM.

  6. #6
    j2b
    j2b is offline
    Active Member
    Join Date
    Dec 2012
    Location
    Riga, Latvia
    Posts
    11
    Sorry, I misunderstood situation - NO ISSUES WITH SAVED SEARCHES!!! Pardon me again, yet my question remains... The problem turned out that custom created folders on move were truncated, thus leading Saved search to un-existing folder. The question, is which part of software truncated it.

  7. #7
    Jay
    Jay is offline
    Code Master ZeXtras Employee
    Join Date
    May 2011
    Posts
    26
    Quote Originally Posted by j2b View Post
    Oh, yes, please. If this topic is very thin, I could provide some extra pages in this section Any way, if I might be of use, you can freely catch me up. And, of course, have to say, that your product solved a week wide problem in 4 hours. Although not as cleanly as I expected. That's why I started to dig deeper I'd definitely advised this product to Zimbra people, unless they feel comfortable and have time to go with CLI and scripts on their own. I have the feeling, that I can pass this burden to you, guys. Any way, result is better and easier.
    That's our job indeed, you're welcome :-)

    Quote Originally Posted by j2b View Post
    OK, what does it mean - "...it submits the operation to that server..."? As from context, I'd assume, that it can submit operation in terms of notice or message to that particular server's process, but that would not be true regarding backup data. Right? At least, if such backup data is not accessible via shared mounted share. Right?
    Backup data is never sent over network and each operation is handled locally (the mailbox server from where the operation has been started) this means you must launch an import command for each server.
    Whenever an operation is launched to server X and the interested user of that operation is stored in server Y the operation is reidirected to server Y, this is valid for undelete and restore on new operation, this automatic behaviour doesn't affect an import.


    Quote Originally Posted by j2b View Post
    In our case, there are 2 new mailbox servers, where in this particular situation, separation is for admin/customer purposes, but this is a step forward to implementation of n-customers' mailbox servers. The only difference here for now, is that main ZCS domain "default" is on mb1, all the others - on mb2. I initiated backup via mb2, but for some reason some domains or domain tried to access mb1, which is not clear for me. That's why I started discussion from higher shelf.
    "..I initiated backup via mb2.."
    the backup must be initialized separately for each server.

    "..I initiated backup via mb2, but for some reason some domains or domain tried to access mb1.."
    I looked up the exception you posted and it seems that an account you tried to restore was already existing in another server, when it tried to actually write inside that mailbox (taken for granted it was just created in the local server) zimbra gave an exception, you should see a message like this in the import log "Skipping creation of already existing account ...". We did overlook this case, we are discussing right now on what's the best way to handle this particular situation, we'll let you know. As a workaround you can either restore everything but that(or those) accounts and restore that(or those) accounts in mb1, or delete them and restore them locally in mb2.

    Quote Originally Posted by j2b View Post
    And for initial backup (or actually import of new mailboxes) there are no such domains and mailboxes at all, so ZXB has to decide, where to place these new mailboxes. Or this is done to current mb server, operation is started?
    The mailbox is always created locally, you can however import only what you want in the local server by filtering with domains or single accounts.

    Quote Originally Posted by j2b View Post
    It really does not seem, it's working Microsoft file copy way - start copying a couple of gigabytes large folder structure, catch an error on 60%, and all operations are stopped and have to be started from beginning, unless you desire to do manual folder checking : D I know, there are other tools to do this But still, checking manually, all data seems to be present, regarding e-mail messages, contacts, calendar entries, and briefcase entries. Although missing parts of tags/versions and some other additional data.
    The import procedure keeps track of what was imported and it skips every 'item' which was already imported, if the source server adds NEW 'items' those are imported as well, however it doesn't keep track of deletes and item modifications.

    Quote Originally Posted by j2b View Post
    So if this catches "Exteral Restore failed", repeating restore, does it stop on the same place, and does not do anything further?
    Usually if it stops in some place it will keep stopping in the exact same spot, no data will be just skipped like way.

    Quote Originally Posted by j2b View Post
    Here are another findings, regarding import, which might be connected with this error:
    • Tag list is imported, mails partially are tagged, BUT - taglist does not have any associations with tags on mails. E.g. tags on mails on hover display particular tag ID, which reminds me to think, that there is no connection between this particular tag and taglist. Could be probably due to partial data import.
    • There were some queries in status mails, which could effect import, but issuing query at users search field, returns mails correctly, e.g. items: xxx,yyy.
    • Mail signatures are recovered partially. Still have not got any pattern to troubleshoot this, as well as found any caus. Some mailboxes have signatures imported, some not.
    • Briefcase file/web doc versions missing - only last version imported.
    • Tag are heavily modified in zimbra 8, during the migration tags looks horrible, however they will be renamed in the last second
    • "There were some queries in status mails, which could effect import" You could do virtually anything except deletes during import, so import shoudn't be affected.
    • Signatures should always be restored, this looks like a serius issue, could you provide more information? what's missing? were these signatures created in older zimbra version? any information will be helpful.
      (could you pm the output of the command "zxsuite backup getAccountInfo account@domain backup_path /my/export/path" of a problematic account? thanks)
    • Unfortunately versioning is unsupported at the moment (only the last rev is saved), this won't be solved in the near future.


    Jay
    d0s0n likes this.
    ZeXtras Website # ZeXtras Wiki # ZeXtras Store

    Head of ZeXtras Suite Developers

  8. #8
    Jay
    Jay is offline
    Code Master ZeXtras Employee
    Join Date
    May 2011
    Posts
    26
    Quote Originally Posted by j2b View Post
    Initial data import was done via Admin UI and from mailbox2 server particularly. How does zextras decide, which is Source account's mailbox? No connections to actual source server were done, and new servers have different hostnames.
    When you access to the admin UI you must choose which mbox you are using, that is used for restore.

    Quote Originally Posted by j2b View Post
    I rsynced backup files to particular mailbox2 locally mounted disk, which is not shared between mailbox servers. If this requrest is proxied to another mailbox (in case of administration and internal accounts), it should be true, that operation would fail, as mailbox1 in my case has no access to backup source data. Right?
    The operation won't be processed in another server, only the one you select (or you are logged in (CLI)).

    Quote Originally Posted by j2b View Post
    I got caught on this, but as it was only one account, it was quite easy to reestablish shares manually. Didn't manage to test doFixShares, although run it once. As a result, other shares within the same mailbox server didn't recover, nor between mailboxes too. Just gave me a list of shares found. Should I expect other action results?
    Shares should be restored without furher actions when you are doing an import in a single server(or a single import), however to fix shares between servers a doFixShares should be launch in each server AFTER the migration is done.
    Shares are restored in the last import stage, they should be broken until the import is completed.

    Jay
    ZeXtras Website # ZeXtras Wiki # ZeXtras Store

    Head of ZeXtras Suite Developers

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •