Looking for a guide on converting to Maildir.
Here are our relevant specs.
sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0
To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant>
All the maildir stuff I can find is postfix oriented. From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
-Jason
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Adam Tauno Williams Sent: Tuesday, January 04, 2011 8:52 To: centos@centos.org Subject: Re: [CentOS] Converting to maildir
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade
this due to
too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is
filling up
too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox
and imap and outlook...
</rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
There are too many modifications to abandon it right now. Besides it is stable as a rock.
From what I can read in procmail man pages, it supports maildir and sendmail uses
procmail as
the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network
The backup server. As one file per mailbox, the backup server is backing up over 25G/hour. These files are not subject to de-duplication. With one message per file only the new messages would get added to the backup size.
What would you use besides Maildir?
access your message store. Cyrus IMAPd will index and filter all your messages for you].
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Tue, 2011-01-04 at 09:05 -0500, Jason Pyeron wrote:
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade
this due to
too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is
filling up
too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox
and imap and outlook...
</rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history
There are too many modifications to abandon it right now. Besides it is stable as a rock.
From what I can read in procmail man pages, it supports maildir and sendmail uses
procmail as
the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network
The backup server. As one file per mailbox, the backup server is backing up over 25G/hour. These files are not subject to de-duplication. With one message per file only the new messages would get added to the backup size. What would you use besides Maildir?
I use Cyrus IMAPd - where external modification of the mailstore is forbidden [or at least very frowned upon]. That way it uses its own internal storage format that can be customized to be efficient. It also means it can keep *consistent* meta-data databases, such as search indexes, which are *IMPOSSIBLE* if other clients are diddling around in the mailstore. These databases add features, performance, and stability. You also get things like delayed expunge and duplicate supression [which can save scads of disk space]. All access to the mailstore is via IMAP or POP. Messages are placed in the mailstore by the MTA (sendmail / postfix) via LMTP - so Cyrus can also run the SIEVE filtering language to provide on-delivery message filtering.
Administrative tools are provided to manipulate the message store in a consistent and reliable way.
access your message store. Cyrus IMAPd will index and filter all your messages for you].
Yup, Cyrus was rock solid for us for years with Thunderbird as the client. We were forced into an Exchange replacement (Scalix) and now Google mail because 'Thunderbird is clunky (read: follows all of the user interface guidelines) and Outlook is cool (read: actually forces overrides on windows standard interface behaviour)' and neither backend is solid. Don't get me started on the huge list of LookOUT! WTFs. I wouldn't trust it for e-mails with my mum. GMail sometimes randomly expunges e-mails because it feels like it. Sometimes when you're actually reading the thing. You can't e-mail yourself.
On Tue, Jan 04, 2011 at 08:52:23AM -0500, Adam Tauno Williams wrote:
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
Many people care about storage format. Mbox is much more slower during operations on it. It's because it's operate on single file, not several of files. Maildir is only slower while opening it. But it depends on number of messages in such a box which is equal to number of descriptors system must open while reading a box.
On Tue, 2011-01-04 at 15:06 +0100, Dominik Zyla wrote:
On Tue, Jan 04, 2011 at 08:52:23AM -0500, Adam Tauno Williams wrote:
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
Mbox is much more slower during operations on it. It's because it's operate on single file,
Correct, but who cares? If the server provides high-performance to the mailbox... why care? Message format storage wars are silly.
On Tue, Jan 04, 2011 at 09:14:57AM -0500, Adam Tauno Williams wrote:
On Tue, 2011-01-04 at 15:06 +0100, Dominik Zyla wrote:
On Tue, Jan 04, 2011 at 08:52:23AM -0500, Adam Tauno Williams wrote:
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
Mbox is much more slower during operations on it. It's because it's operate on single file,
Correct, but who cares? If the server provides high-performance to the mailbox... why care? Message format storage wars are silly.
I agree it's silly. But try to run dozens of maildirs and the same number of mailboxes on the same kind of server. Mboxes would be bottleneck of the entire mail system.
Dominik Zyla wrote:
On Tue, Jan 04, 2011 at 09:14:57AM -0500, Adam Tauno Williams wrote:
On Tue, 2011-01-04 at 15:06 +0100, Dominik Zyla wrote:
<SNIP>
Mbox is much more slower during operations on it. It's because it's operate on single file,
Correct, but who cares? If the server provides high-performance to the mailbox... why care? Message format storage wars are silly.
I agree it's silly. But try to run dozens of maildirs and the same number of mailboxes on the same kind of server. Mboxes would be bottleneck of the entire mail system.
And then there's the problem when your mailtool screws up the formatting of the mbox file....
mark
On 01/04/2011 06:14 AM, Adam Tauno Williams wrote:
On Tue, 2011-01-04 at 15:06 +0100, Dominik Zyla wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
No. They are being eminently practical. mbox format's 'one big file' approach results in significant I/O overhead for update operations, locking complexity (file locks on shared network storage - 'nuff said) and bloat in differential backups.
I have literally tens of gigabytes of email stored on our servers. mbox storage would make backups slower, take significantly more backup storage space and add quite a lot of disk I/O for routine mailbox use as well as slow down email for the end users. It is also more prone to 'one error took out everything' problems.
The idea that "low level/internal details don't matter" is only true when you are so far from your resource limits that they are effectively infinite. The real world often isn't that way.
On Tuesday, January 04, 2011 09:14:57 am Adam Tauno Williams wrote:
On Tue, 2011-01-04 at 15:06 +0100, Dominik Zyla wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
Hmmm, not quite.
When selecting the file system on which to store e-mail, the storage format is significant; it's a 'small number of large files' versus 'large number of small files' issue then, and filesystems differ in their performance between them. Some filesystems slow down with large maildirs; some slow with large mboxes. If you support a hundred or a thousand users, make sure you allocate enough inodes on that mailstore filesystem if you use maildir. For POP-only servers mbox works fine. For IMAP servers where IMAP is the primary access means, not so fine.
In my opinion, maildirs are great for rapidly changing dynamic folders, like the inbox, whereas mboxes are wonderful for archives, where they tend to take less disk space for the same number of messages, and tend to change more slowly. And when you have folders containing hundreds of thousands of e-mails (yes, hundreds of thousands, in one particular archive, I have) where the individual e-mails are quite short, the difference adds up.
In my case, our primary e-mail server is Scalix, so that dictated the storage format. But, honestly, I personally would love to use a PostgreSQL backend so that real concurrent access is possible; I have users with Scalix mail folders that take a long time to rsync simply due to the number of messages (25-30 thousand in the inbox, and they are 'folder clueless' and don't want to throw anything away), and in order to get a consistent backup scalix has to be shut down during the rsync (even if the folder hasn't changed, rsync still has to read all those directory entries, which takes time); an ACID database backend (PostgreSQL, MySQL InnoDB, Oracle, etc) will allow a fully consistent backup to be taken while the database is active. And backup tools for such databases are very mature.
Scalix 11 uses PostgreSQL, but not as the primary mailstore.
On January 4, 2011 07:36:27 am Lamar Owen wrote:
In my case, our primary e-mail server is Scalix, so that dictated the storage format. But, honestly, I personally would love to use a PostgreSQL backend so that real concurrent access is possible;
dbmail with PostgreSQL works really well. I moved a Maildir + Courier setup to it when backups got too painful. It does require a fair bit of RAM, but PostgreSQL compresses text data and dbmail works to dedupe messages and especially attachments.
On Tuesday, January 04, 2011 10:59:21 am Alan Hodgson wrote:
On January 4, 2011 07:36:27 am Lamar Owen wrote:
But, honestly, I personally would love to use a PostgreSQL backend so that real concurrent access is possible;
dbmail with PostgreSQL works really well.
Thanks for the pointer; may look at putting it into test with an alternate Exchange replacement.
On 1/5/2011 9:51 AM, Lamar Owen wrote:
On Tuesday, January 04, 2011 10:59:21 am Alan Hodgson wrote:
On January 4, 2011 07:36:27 am Lamar Owen wrote:
But, honestly, I personally would love to use a PostgreSQL backend so that real concurrent access is possible;
dbmail with PostgreSQL works really well.
Thanks for the pointer; may look at putting it into test with an alternate Exchange replacement.
If you want to avoid dealing with the setup at all, you might look at ClearOS. Comes up running with an ajax-y web interface for administration and what looks to be ldap/postfix/cyrus/kolab and a few other things under the covers for email service. From the user's perspective the only odd thing is the cyrus convention of putting all other folders inside of 'inbox' when you connect with imap.
On 1/4/2011 8:14 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
So how do you suggest solving that problem when it is in fact a problem?
On Tue, 2011-01-04 at 09:56 -0600, Les Mikesell wrote:
On 1/4/2011 8:14 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
So how do you suggest solving that problem when it is in fact a problem?
You're missing the point of my objection.
That the server is slow or difficult to manage [which includes backup] is the issue - not that it uses MBOX [regardless of that its use of MBOX is the root of the *server's* issue].
Thinking of it in terms of messages-storage-format is misguided [I only mention it because I see the MBOX/MH/Maildir/Maildir++/etc... debate frequently]. This represents, IMO, a flawed approach to the problem.
Migration to a new solutions [an IMAP server] that provides better performance / management is the fix. Incidently, it will almost certainly not use MBOX.
If you are 'manually' crawling around in your message-store [the *only* case where you'd actually care much about storage format] indicates something else it wrong [as well].
On 1/4/2011 10:00 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
So how do you suggest solving that problem when it is in fact a problem?
You're missing the point of my objection.
The point doesn't matter. Solving the problem does. And you probably can't solve it without knowing how things work.
That the server is slow or difficult to manage [which includes backup] is the issue - not that it uses MBOX [regardless of that its use of MBOX is the root of the *server's* issue].
Thinking of it in terms of messages-storage-format is misguided [I only mention it because I see the MBOX/MH/Maildir/Maildir++/etc... debate frequently]. This represents, IMO, a flawed approach to the problem.
Yes, you could solve it by ignoring the related physics and throwing infinite resources at it - if you have infinite amounts of money. Or you could do a sysadmin's job and understand the physical constraints and optimize the results you can get from them.
Migration to a new solutions [an IMAP server] that provides better performance / management is the fix. Incidently, it will almost certainly not use MBOX.
Odd that you would mention that, just after saying you shouldn't care... And in fact, some servers (e.g. dovecot) may handle more than one storage format, leaving it up to the admin to choose which is best.
If you are 'manually' crawling around in your message-store [the *only* case where you'd actually care much about storage format] indicates something else it wrong [as well].
Your backup system will most certainly be crawling around your message store.
-----Original Message----- From: Les Mikesell Sent: Tuesday, January 04, 2011 12:28 Subject: Re: [CentOS] Converting to maildir
On 1/4/2011 10:00 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message
storage is
an internal [server's] problem.
So how do you suggest solving that problem when it is in
fact a problem?
You're missing the point of my objection.
The point doesn't matter. Solving the problem does. And you probably can't solve it without knowing how things work.
That the server is slow or difficult to manage [which
includes backup]
is the issue - not that it uses MBOX [regardless of that its use of MBOX is the root of the *server's* issue].
Thinking of it in terms of messages-storage-format is misguided [I only mention it because I see the MBOX/MH/Maildir/Maildir++/etc... debate frequently]. This represents, IMO, a flawed
approach to the problem.
Yes, you could solve it by ignoring the related physics and throwing infinite resources at it - if you have infinite amounts of money. Or you could do a sysadmin's job and understand the physical constraints and optimize the results you can get from them.
Migration to a new solutions [an IMAP server] that provides better performance / management is the fix. Incidently, it will almost certainly not use MBOX.
Odd that you would mention that, just after saying you shouldn't care... And in fact, some servers (e.g. dovecot) may handle more
Thank you. http://wiki.dovecot.org/MailboxFormat/Maildir
I am now going to look for a setup guide of procmail->Maildir->dovecot on RHEL/Centos
than one storage format, leaving it up to the admin to choose which is best.
If you are 'manually' crawling around in your message-store [the *only* case where you'd actually care much about storage format] indicates something else it wrong [as well].
Your backup system will most certainly be crawling around your message store.
-Jason
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
Jason Pyeron wrote:
-----Original Message----- From: Les Mikesell Sent: Tuesday, January 04, 2011 12:28 Subject: Re: [CentOS] Converting to maildir
On 1/4/2011 10:00 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message
storage is
an internal [server's] problem.
So how do you suggest solving that problem when it is in
fact a problem?
You're missing the point of my objection.
The point doesn't matter. Solving the problem does. And you probably can't solve it without knowing how things work.
That the server is slow or difficult to manage [which
includes backup]
is the issue - not that it uses MBOX [regardless of that its use of MBOX is the root of the *server's* issue].
Thinking of it in terms of messages-storage-format is misguided [I only mention it because I see the MBOX/MH/Maildir/Maildir++/etc... debate frequently]. This represents, IMO, a flawed
approach to the problem.
Yes, you could solve it by ignoring the related physics and throwing infinite resources at it - if you have infinite amounts of money. Or you could do a sysadmin's job and understand the physical constraints and optimize the results you can get from them.
Migration to a new solutions [an IMAP server] that provides better performance / management is the fix. Incidently, it will almost certainly not use MBOX.
Odd that you would mention that, just after saying you shouldn't care... And in fact, some servers (e.g. dovecot) may handle more
Thank you. http://wiki.dovecot.org/MailboxFormat/Maildir
I am now going to look for a setup guide of procmail->Maildir->dovecot on RHEL/Centos
I used the centos wiki guides and they are GREAT - thanks to the team that put them together.
than one storage format, leaving it up to the admin to choose which is best.
If you are 'manually' crawling around in your message-store [the *only* case where you'd actually care much about storage format] indicates something else it wrong [as well].
Your backup system will most certainly be crawling around your message store.
-Jason
--
-- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
<snip >
Odd that you would mention that, just after saying you shouldn't care... And in fact, some servers (e.g. dovecot) may handle more
Thank you. http://wiki.dovecot.org/MailboxFormat/Maildir
I am now going to look for a setup guide of procmail->Maildir->dovecot on RHEL/Centos
A good side effect of moving to dovecot from wuimap is a BIG speed increase...
Adam Tauno Williams wrote:
On Tue, 2011-01-04 at 09:56 -0600, Les Mikesell wrote:
On 1/4/2011 8:14 AM, Adam Tauno Williams wrote:
Many people care about storage format.
And they are misguided in doing so. Details of message storage is an internal [server's] problem.
So how do you suggest solving that problem when it is in fact a problem?
You're missing the point of my objection.
That the server is slow or difficult to manage [which includes backup] is the issue - not that it uses MBOX [regardless of that its use of MBOX is the root of the *server's* issue].
Thinking of it in terms of messages-storage-format is misguided [I only mention it because I see the MBOX/MH/Maildir/Maildir++/etc... debate frequently]. This represents, IMO, a flawed approach to the problem.
Migration to a new solutions [an IMAP server] that provides better performance / management is the fix. Incidently, it will almost certainly not use MBOX.
If you are 'manually' crawling around in your message-store [the *only* case where you'd actually care much about storage format] indicates something else it wrong [as well].
At risk of confusing the debate - modern email is now largely HTML with lots of embedded graphics (just love all those base64 encoded bits clogging up the mbox) I made the shift from mbox to maildir about three years ago - my reasoning, let the OS file system worry about where and how to store the stuff - let the mail app worry about what emails I have and how to index. Thus postfix, dovecot (imap only) and related spam tools seem to work fine for my small business. I'm sure the problems only get more involved if one has to support 1,000's of users. Why is it that outlook and thunderbird use mbox type storage for their local storage?? Certainly a pain to manage in today's bloated email world. We haven't seen the end of this problem - it is growing day by day.........really needs some creative solutions before we all drown in the data deluge. Don't forget we also need to be able to search the last x days, weeks, years email's for something?
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I've been happily using and deploying Cyrus based systems for over a decade, so I'll jump in with my $0.02.
Many people care about storage format.
It was mentioned previously that Cyrus IMAPd uses an internal format. Well, it happens that that internal format is really just the mail message itself, one per file. This means that incremental backups are extreamly effective, including for *huge* mailboxes. It also means that if you feel you need to, you can go in as root and read the message directly. Mailbox repairs are done by reparsing the original email.
Cyrus is solid, scalable, fast, portable, and all the other adjectives you'd want from a production mail store. That includes being able to run it in a master/failover HA cluster, BTW.
An observation was also made about large numbers of small files. Yes, you do have to be aware of inode allocations and the behavior of your underlying filesystems with any system that doesn't use monolithic mailboxes. Grep and wc can help you figure out how much that should concern you based on your existing mail store. Cyrus can mitigate the inode and filesystem performance issues by having mailboxes spread across different filesystems. If you run a system with a truly large number of users, then consider Cyrus Murder for horizontal scaleout.
I find that pairing Cyrus IMAPd with sendmail, horde, MailScanner, clamav, and a few assorted antispam mechanisms is a really good combination. (I'm not saying that postfix or whatever is bad, but don't sell sendmail short, either.) That combination can exist on a single host, but doesn't need to. In fact, in a few installations I have multiple sendmails doing remote delivery to cyrus via network-based lmtp (all locked down, of course).
Regarding quotas, cyrus will take care of that for you at the MDA level; you don't need filesystem quotas. (And in fact, because all files are owned by Cyrus, a filesystem-based quota wouldn't work anyway.)
Jason, if you decide to go with Cyrus and need advice on integration with sendmail (since you obviously want to stick with it), drop me a line privately.
Devin
On Jan 4, 2011, at 8:52 AM, Adam Tauno Williams awilliam@whitemice.org wrote:
On Mon, 2011-01-03 at 20:17 -0500, Jason Pyeron wrote:
Looking for a guide on converting to Maildir. Here are our relevant specs. sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0 To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant> All the maildir stuff I can find is postfix oriented.
Because sendmail is rapidly fading into history?
From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
I think the OP said he wanted maildir for backup reasons. With mbox a single new email will mean the whole mbox needs to be backed up, with maildir only that email will need backing up.
-Ross
On Tue, 2011-01-04 at 09:07 -0500, Ross Walker wrote:
On Jan 4, 2011, at 8:52 AM, Adam Tauno Williams awilliam@whitemice.org wrote:
There are numerous IMAP servers that support maildir, and scripts to import MBOX files - that is how I would approach it. [But then I wouldn't use Maildir; I mean, really, who cares what format your messages are in - use IMAP and network access your message store. Cyrus IMAPd will index and filter all your messages for you].
I think the OP said he wanted maildir for backup reasons. With mbox a single new email will mean the whole mbox needs to be backed up, with maildir only that email will need backing up.
+1 The internal format used by Cyrus is one-file-per-message so rsync-ing works very well. But delayed-expunge and clustering works even better.
On Mon, Jan 3, 2011 at 7:17 PM, Jason Pyeron jpyeron@pdinc.us wrote:
Looking for a guide on converting to Maildir.
Here are our relevant specs.
sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0
To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant>
All the maildir stuff I can find is postfix oriented. From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
-Jason
Regardless of the maildir vs mbox argument, I would be seriously examining why you have painted yourself into a corner with your customized sendmail. Eventually, you will have to move on. What are the motivations for the customizations? Do newer or alternate MTAs have added features that can replace those customizations? Postfix can be highly customized through configuration and is not that difficult to learn.
As a migration path, I would separate the MTA (sendmail) and the imap server. Go with cyrus or dovecot on a new machine (virtual?) and use imapsync to move messages to the new box during a maintenance window. As stated in other responses, cyrus has it's own mail storage format with individual files for each message and dovecot supports several formats including maildir. It should not be difficult to have your existing sendmail deliver messages to the new imap store either directly or with a very simple postfix MTA on the imap box. Once mail storage is fixed, you can start working on de-customizing your MTA.
And with regard to backup space, it might be time to suck it up and tell your users that you need to implement mail quotas. How much are you backing up from "Sent" and "Trash" because nobody maintains their mail folders? A quota can be a great tool for teaching basic mail folder housekeeping.
-- Jeff
On Tue, 2011-01-04 at 09:38 -0600, Jeff wrote:
On Mon, Jan 3, 2011 at 7:17 PM, Jason Pyeron jpyeron@pdinc.us wrote: And with regard to backup space, it might be time to suck it up and tell your users that you need to implement mail quotas. How much are you backing up from "Sent" and "Trash" because nobody maintains their mail folders? A quota can be a great tool for teaching basic mail folder housekeeping.
+1 on quotas; they are virtuous even if capacity isn't a constraint - they force users to manage their data.
Regarding sent/trash/SPAM Cyrus IMAPd provides an "expire" annotation that can be applied to folders that will expire messages from the folders older than X number of days [on the server side, user doesn't have to login for this to happen]. For example we expire sent-mail at 365 days, trash at 45 days, and SPAM at 14 days. This helps quite a bit against lazy-user-syndrome.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Adam Tauno Williams Sent: Tuesday, January 04, 2011 10:51 To: centos@centos.org Subject: Re: [CentOS] Converting to maildir
On Tue, 2011-01-04 at 09:38 -0600, Jeff wrote:
On Mon, Jan 3, 2011 at 7:17 PM, Jason Pyeron
jpyeron@pdinc.us wrote:
And with regard to backup space, it might be time to suck it up and tell your users that you need to implement mail quotas. How
much are
you backing up from "Sent" and "Trash" because nobody
maintains their
mail folders? A quota can be a great tool for teaching basic mail folder housekeeping.
+1 on quotas; they are virtuous even if capacity isn't a constraint - they force users to manage their data.
I am sorry if this comes across harsh, but you have no idea about the business objectives, I never asked how do we keep our email size small. In fact I said it was BIG and I had a performance issue.
Regarding sent/trash/SPAM Cyrus IMAPd provides an "expire" annotation that can be applied to folders that will expire messages from the folders older than X number of days [on the server side, user doesn't have to login for this to happen]. For example we expire sent-mail at 365 days, trash at 45 days, and SPAM at 14 days. This helps quite a bit against lazy-user-syndrome.
That is a recipe for fired lazy-sysadmin-syndrom.
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On 1/4/2011 9:38 AM, Jeff wrote:
Here are our relevant specs.
sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0
Regardless of the maildir vs mbox argument, I would be seriously examining why you have painted yourself into a corner with your customized sendmail. Eventually, you will have to move on.
Errr, why??? Sendmail has nothing to do with local deliveries.
What are the motivations for the customizations? Do newer or alternate MTAs have added features that can replace those customizations? Postfix can be highly customized through configuration and is not that difficult to learn.
Why change the part that isn't broken.
As a migration path, I would separate the MTA (sendmail) and the imap server. Go with cyrus or dovecot on a new machine (virtual?) and use imapsync to move messages to the new box during a maintenance window. As stated in other responses, cyrus has it's own mail storage format with individual files for each message and dovecot supports several formats including maildir. It should not be difficult to have your existing sendmail deliver messages to the new imap store either directly or with a very simple postfix MTA on the imap box. Once mail storage is fixed, you can start working on de-customizing your MTA.
Sendmail will let cyrus or procmail or another local delivery agent handle the file format details - and probably already does.
And with regard to backup space, it might be time to suck it up and tell your users that you need to implement mail quotas. How much are you backing up from "Sent" and "Trash" because nobody maintains their mail folders? A quota can be a great tool for teaching basic mail folder housekeeping.
Anything that does sensible incrementals will use much less space with 'file-per-message' formats instead of mbox because the bulk of the messages won't change between runs. But maybe another solution would be to put the backups on a block de-duplicating filesystem like zfs.
On Tue, Jan 04, 2011 at 09:38:47AM -0600, Jeff wrote:
On Mon, Jan 3, 2011 at 7:17 PM, Jason Pyeron jpyeron@pdinc.us wrote:
Looking for a guide on converting to Maildir.
Here are our relevant specs.
sendmail-8.12.11-4.RHEL3.6 (we may not be able to upgrade this due to too many modifications) imap-2002d-14 procmail-3.22-10.el3.centos.0
To a maildir setup...
<rant> I was in a panic today at work because the backup server is filling up too quickly, backing up peoples email. Further it is not backing up often enough. I just lost all of today's email. I hate mbox and imap and outlook... </rant>
All the maildir stuff I can find is postfix oriented. From what I can read in procmail man pages, it supports maildir and sendmail uses procmail as the LDA, hence sendmail "supports" it.
-Jason
And with regard to backup space, it might be time to suck it up and tell your users that you need to implement mail quotas. How much are you backing up from "Sent" and "Trash" because nobody maintains their mail folders? A quota can be a great tool for teaching basic mail folder housekeeping.
I'll suggest to use journaled-quota. In case of some filesystem problems there'll be no need to do quotacheck(8) if you're using ext3.
-
Adam Tauno Williams -
Alan Hodgson -
Benjamin Franz -
Devin Reade -
Dominik Zyla -
Jason Pyeron -
Jeff -
Kevin Thorpe -
Lamar Owen -
Les Mikesell -
m.roth＠5-cent.us -
Rob Kampen -
Ross Walker -
Scott Silva