Primary Mailstore with HSM keeps growing
Results 1 to 6 of 6
Like Tree4Likes
  • 1 Post By ajcody
  • 1 Post By bhotrock
  • 2 Post By bhotrock

Thread: Primary Mailstore with HSM keeps growing

  1. #1
    Active Member
    Join Date
    Jul 2015
    Posts
    13

    Primary Mailstore with HSM keeps growing

    Hi All,

    I'm wondering if I have a problem, or if I just don't understand something correctly. Here's the situation. When we built a new server, we created separate primary and secondary mailstore disks. We calculated the size of the primary disk based on our complete volume of messages from the old server and thought we'd have enough space to hold several months of mail in the primary.

    (Actually we started with importing all mail from the old server into a Primary that was built to be the long-term Secondary, a much larger disk. Then once the import was complete, we moved the initial Primary to Secondary and set the smaller disk to Primary.)

    Once we re-assigned Primary and Secondary, we started out with HSM settings to keep 30 days on the Primary, then move to compressed Secondary. This worked fine and we began filling the Primary. I don't remember the percentage used on the Primary when we hit 30 days and the HSM started moving blobs to the secondary. I wasn't watching it closely and assumed we'd have plenty of space.

    Over time, we ended up with the Primary filling to 80 then 90%. So I changed the HSM policy to 24 days. A lot of blobs were moved and we dropped our usage of the Primary disk. But then it continued to creep back up to 80-90% again. Dropped the HSM again.. Filled up again... Dropped again... etc....

    Today we had been down to only 12 days in the policy and were filling up again. When I dropped it from 12 days to 10 days just now, we freed up 11% of the space on the Primary. But statistically, it should have moved over 16% since we moved 2 of 12 days.

    We don't seem to be receiving exponentially more email, so it seems that something isn't getting moved from Primary to Secondary. Is there any way to figure out what's happening? HSM Policy is set to move all items (except chats - we don't use chat). The logs indicate that some blobs are being moved every day.

    At this point, if I understand the HSM policy, the Primary mailstore should only have the last 10 days of "anything" stored there. Currently that disk is using 25GB of space. Our secondary, which I think has everything older than 10 days is using 388GB. The secondary IS compressed, and has had deduplication run a few times. But it has several years of email. At the rate of 25GB for 10 days, then a year should consume over 900GB. I know that compression and deduplication (and junk and trash cleanup) will make a difference, but it seems like something is out of whack.

    Trying to look at file ages on the Primary mailstore, we ran this command:
    find /primarymailstore/0/ -type f | wc -l (should give current file count) and the result was 186824.
    find /primarymailstore/0/ -type f -mtime +12 | wc -l (should give count of files modified more than 12 days ago) gave a result of 88281.
    Some of these files with older modified dates go back to August. We shifted to the new server in July, and it was probably some time in August when we moved the original Primary to Secondary and implemented the new smaller Primary disk.

    Sorry this is so long. Just trying to get as much information out there. I'd appreciate any ideas what could be causing this! If I'm not understanding how the HSM policy should be working, let me know that too. :-)
    Thanks!
    Brad

    Zimbra version: 8.6.0, patch 3. Zextras version: 2.0.3 OS: CentOS 6.6
    Last edited by bhotrock; 10-12-2015 at 05:39 PM. Reason: Edited to include version numbers.

  2. #2
    ZeXtras Community Manager ZeXtras Employee Cine's Avatar
    Join Date
    Apr 2011
    Posts
    2,360
    Hello Brad!

    There are a couple of different causes that can lead to the scenario you described:
    • Bad syntax in the HSM policy - could you please post the output of the "zxsuite powerstore getHsmPolicy"?
    • Heavy dumpster use - items in the dumpster are not moved by ZeXtras Powerstore, so if this feature is enabled any deleted item will persist on the Primary volume.
    • Trash and Spam - those folders are not parsed by the HSM policy unless you add the "is:anywhere" clause to it.


    Also, I'd suggest to update to ZeXtras Suite 2.0.4 - there are no substantial changes in the Powestore module but the Devs just told me that there was a very small change in HSM queries - it might not be problem solving, but everything matters

    Have a nice day,
    Cine
    the ZeXtras Team
    Last edited by Cine; 10-13-2015 at 02:39 PM.

  3. #3
    Active Member
    Join Date
    Jul 2015
    Posts
    13
    Thanks for your response Cine!

    Here's the policy from the command: policies
    appointment,task,message,contact,document:before:-10day

    I think most of our COSs do have the dumpster enabled and set to 30 days before purge. So I'll investigate how to shut that off, or change the purge time frame.

    Regarding the trash, spam and dumpster, if those are set to 30 days, does that mean they should still be deleted from the primary mailstore after 30 days, or do they somehow get "orphaned"? Assuming they get deleted anyway at 30 days, I would expect to have messages, appointments, tasks, contacts and documents for the last 10 days, and trash, spam, dumpster for 30 days. Should that be correct?

    I'll try to get Zextras upgraded later today during off-peak time.

    Thanks again for the info!

  4. #4
    ZeXtras USA ZeXtras Reseller ajcody's Avatar
    Join Date
    Aug 2015
    Posts
    5
    Please note, for trash/dumpster is will be days from the change and not the delivery time. I wrote a wiki section on mailbox purge and dumpster here:
    https://wiki.zimbra.com/wiki/Ajcody-...etime_And_More
    Please note the table of contents for that section, it's rather long.

    -Ajcody
    d0s0n likes this.

  5. #5
    Active Member
    Join Date
    Jul 2015
    Posts
    13
    Thanks! Looks like it's time to go dumpster diving. :-)
    ajcody likes this.

  6. #6
    Active Member
    Join Date
    Jul 2015
    Posts
    13
    It looks like we had other dumpster issues. (Which I realize are probably not related to Zextras.) I checked my own account and found items in the dumpster from 3 years ago. I'm sure it's not everything that I've deleted since there were only 585 items. But somehow things aren't all being removed.

    The following parameters were listed in our COS:
    zimbraDumpsterEnabled: TRUE
    zimbraDumpsterPurgeEnabled: TRUE
    zimbraDumpsterUserVisibleAge: 30d
    zimbraMailDumpsterLifetime: 30d

    But even with these set, it doesn't seem that Zimbra was purging after 30 days. Maybe a lot of the old dumpster messages were orphaned with server migrations?

    So, we've gone through and emptied everyone's dumpster, reset the dumpster retention period to 15 days and cleared lots of space on the Primary (and Secondary) mailstores. Now we'll watch what happens after 15 days and see if Zimbra takes care of purging the dumpsters according to the 15 day retention period.

    One thing that still has me a bit concerned is we still have some items on the Primary mailstore that are dated older than 10 (or even 15) days. This is looking at the Primary mailstore immediately after running the script that empties everyone's dumpster. So I'm not sure why there should be any files older than 10 or 11 days old. The good news is that the number of items older than 12 days dropped from 88,281 to only 6,498. I'm just not sure why those 6,498 items are still there.

    Thanks for the great support and pointing us in the right direction! We're in a lot better shape than we were.
    d0s0n and ajcody like this.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •