Centos 4.0 and 4.1 seem to have a memory leak somewhere and I can't find out what is doing it. I think it may be dovecot.
I have installed hundreds of Linux machines, centos/redhat/suse in the past as firewalls and mail relays and use the same/similar methodolgy and config for each.
I have found that since Centos 4.0 and 4.1 came along machines that have run well for years and months, suddenly and systematically start running out of memory after a few weeks/months of uptime.
The symtpom is the load average goes up and up as the machine creeps into swap usage and eventually the machine has to be rebooted....
this only happened with the introduction of Centos 4 and of course that means Dovecot for the IMAP server....
hmmmm......
Anyone else seen this,
P.
On Mon, 3 Oct 2005, Peter Farrow wrote:
this only happened with the introduction of Centos 4 and of course that means Dovecot for the IMAP server....
yes - on a 4 processor setup, I hit this early on. as I recall, I concluded there was a multi-processor threading issue in Dovecot, and so I fell back to the UW imap as a workaround.
- Russ herrold
On Mon, 2005-10-03 at 11:55 -0400, R P Herrold wrote:
On Mon, 3 Oct 2005, Peter Farrow wrote:
this only happened with the introduction of Centos 4 and of course that means Dovecot for the IMAP server....
yes - on a 4 processor setup, I hit this early on. as I recall, I concluded there was a multi-processor threading issue in Dovecot, and so I fell back to the UW imap as a workaround.
---- ignoring of course, cyrus-imapd which is included in the distribution.
Craig
On Mon, 3 Oct 2005, Craig White wrote:
issue in Dovecot, and so I fell back to the UW imap as a workaround.
ignoring of course, cyrus-imapd which is included in the distribution.
yeah - I had an outage and needed to get it up quickly, rather than dink through something new while at a client's site. I knew it had no such issues with SMP
Also, I have used (and participated in the development of) the UW product for years and years in a ISP context, and can accurately build and configure it without conscious effort. ;)
-- Russ Herrold
On Mon, 2005-10-03 at 12:46 -0400, R P Herrold wrote:
On Mon, 3 Oct 2005, Craig White wrote:
issue in Dovecot, and so I fell back to the UW imap as a workaround.
ignoring of course, cyrus-imapd which is included in the distribution.
yeah - I had an outage and needed to get it up quickly, rather than dink through something new while at a client's site. I knew it had no such issues with SMP
Also, I have used (and participated in the development of) the UW product for years and years in a ISP context, and can accurately build and configure it without conscious effort. ;)
---- yeah - understood
for the record, I haven't found cyrus-imapd to be very difficult to configure...and the performance is awesome. But I am also one to go to that which I understand rather than learn new tricks sometimes myself.
Really, the only wrinkle in setting up cyrus is to change the MTA to use lmtp for local delivery and there are plenty of instructions on how to do that in postfix/sendmail/exim.
Craig
On Mon, 2005-10-03 at 12:02, Craig White wrote:
Also, I have used (and participated in the development of) the UW product for years and years in a ISP context, and can accurately build and configure it without conscious effort. ;)
yeah - understood
for the record, I haven't found cyrus-imapd to be very difficult to configure...and the performance is awesome. But I am also one to go to that which I understand rather than learn new tricks sometimes myself.
Really, the only wrinkle in setting up cyrus is to change the MTA to use lmtp for local delivery and there are plenty of instructions on how to do that in postfix/sendmail/exim.
And, of course, going to each user that has gone through the trouble of learning procmail's line-noise syntax and explaining that he'll have to throw that out and start over with sieve.
On Mon, 2005-10-03 at 12:16 -0500, Les Mikesell wrote:
On Mon, 2005-10-03 at 12:02, Craig White wrote:
Also, I have used (and participated in the development of) the UW product for years and years in a ISP context, and can accurately build and configure it without conscious effort. ;)
yeah - understood
for the record, I haven't found cyrus-imapd to be very difficult to configure...and the performance is awesome. But I am also one to go to that which I understand rather than learn new tricks sometimes myself.
Really, the only wrinkle in setting up cyrus is to change the MTA to use lmtp for local delivery and there are plenty of instructions on how to do that in postfix/sendmail/exim.
And, of course, going to each user that has gone through the trouble of learning procmail's line-noise syntax and explaining that he'll have to throw that out and start over with sieve.
---- 1 - there are utilities that will convert procmail scripts to sieve scripts so I am told.
2 - few users are invested in procmail scripts that I have seen
3 - there are such neat auto features of cyrus-imapd that automate script installation on creation of user.
4 - as you point out yourself, procmail's syntax is pretty much impossible for end users and sieve, though slightly less powerful is actually understandable. Put Horde/IMP/Ingo online or websieve online and users can actually create their own server-side filters.
Craig
On 10/3/05, Craig White craigwhite@azapple.com wrote:
2 - few users are invested in procmail scripts that I have seen
You're apparently not looking at the right users. RedHat has been using procmail as the default local delivery agent for a very long time.
Independent of procmail/sieve, you're also assuming that the users never access their mail except via IMAP, and therefore that the usenet-like storage model used by cyrus is not an issue; and, that no existing mailboxes will have to be imported into the cyrus structure.
On Mon, 2005-10-03 at 11:20 -0700, Bart Schaefer wrote:
On 10/3/05, Craig White craigwhite@azapple.com wrote:
2 - few users are invested in procmail scripts that I have seen
You're apparently not looking at the right users. RedHat has been using procmail as the default local delivery agent for a very long time.
--- indeed - right set or wrong set of users is probably determined by one's perspective. ----
Independent of procmail/sieve, you're also assuming that the users never access their mail except via IMAP, and therefore that the usenet-like storage model used by cyrus is not an issue; and, that no existing mailboxes will have to be imported into the cyrus structure.
---- indeed again - the original poster was asking about memory leaks in dovecot and cyrus-imapd is the 'other' imapd in CentOS 4 distribution...which relates to Russ's (?) Herrold's encouragement to use uw-imap. He is a pretty sharp dude and I wanted to needle him on his fall back to uw-imap.
and yes, moving an existing mail store from one type to another does involve some challenges but thankfully, there are scripts around to do that very thing. Likewise, moving the seen state from one imap server to another is painful but can be done.
Of course the real problem I had was blindly using uw-imap all those years in mbox format and when I migrated to cyrus-imapd I couldn't believe the performance gain. Perhaps uw-imap or dovecot with maildir can get fairly close performance wise, which I now know (but didn't way back then), uw-imap is capable of creating/using maildir format for it's mail store.
In the final analysis though, serious consideration needs to be given to the methodology and daemon used for mail delivery for workgroup and larger and the one time efforts of setup and conversion are probably not in the long term meaningful. I found using ldap, postfix and cyrus-imapd eminently workable including mail delivery with aliases.
Craig
On Mon, 3 Oct 2005, Craig White wrote:
And, of course, going to each user that has gone through the trouble of learning procmail's line-noise syntax and explaining that he'll have to throw that out and start over with sieve.
2 - few users are invested in procmail scripts that I have seen
ouch - was not aware I would lose my beloved procmail
[herrold@swampfox .procmail]$ wc -l *c | tail -2 ; pwd 159 zz_filingrc 7262 total /home/herrold/.procmail
4 - as you point out yourself, procmail's syntax is pretty much impossible for end users and sieve, though slightly less powerful is
yeah ... but I have many years invested in those 7k lines of mungeing ;)
-- Russ Herrold
On Mon, 2005-10-03 at 15:10 -0400, R P Herrold wrote:
On Mon, 3 Oct 2005, Craig White wrote:
And, of course, going to each user that has gone through the trouble of learning procmail's line-noise syntax and explaining that he'll have to throw that out and start over with sieve.
2 - few users are invested in procmail scripts that I have seen
ouch - was not aware I would lose my beloved procmail
[herrold@swampfox .procmail]$ wc -l *c | tail -2 ; pwd 159 zz_filingrc 7262 total /home/herrold/.procmail
4 - as you point out yourself, procmail's syntax is pretty much impossible for end users and sieve, though slightly less powerful is
yeah ... but I have many years invested in those 7k lines of mungeing ;)
---- The Procmail way... # fedora-list list :0: * ^TO_fedora-list@redhat.com mail_folder/fedora
The sieve way... # Fedora-list if header :comparator "i;ascii-casemap" :contains "X-BeenThere" "fedora- list@redhat.com" { fileinto "INBOX.fedora"; stop; }
not all that problematic but I appreciate the tendency to not change things that are working. It's just so much faster - even on my 1 user setup at home and incredibly so much faster in multi-client environments.
Craig
On Mon, 2005-10-03 at 14:37, Craig White wrote:
not all that problematic but I appreciate the tendency to not change things that are working. It's just so much faster - even on my 1 user setup at home and incredibly so much faster in multi-client environments.
But you are comparing the speed of uw-imap with mbox format against cyrus and the mbox format is the problem. Procmail knows how to deliver in maildir format and dovecot can access it with some opportunistic indexing to get a big speedup without having to change as much from the users' or administrator's perspectives.
On Mon, 2005-10-03 at 15:31 -0500, Les Mikesell wrote:
On Mon, 2005-10-03 at 14:37, Craig White wrote:
not all that problematic but I appreciate the tendency to not change things that are working. It's just so much faster - even on my 1 user setup at home and incredibly so much faster in multi-client environments.
But you are comparing the speed of uw-imap with mbox format against cyrus and the mbox format is the problem. Procmail knows how to deliver in maildir format and dovecot can access it with some opportunistic indexing to get a big speedup without having to change as much from the users' or administrator's perspectives.
--- absolutely - apparently uw-imap is capable of using maildir too, but I never knew that until I had switched over to cyrus-imapd.
you can use dovecot - I definitely like cyrus-imapd, love sieve, mail quotas, shared folders, autocreate folders/shared folders, autocreate sievescripts, multiple backend stores, etc. Speed is only one of the reasons. As I said, the long view of living with a mail daemon makes things like initial setup and conversion pretty much meaningless.
Craig
On Mon, 2005-10-03 at 10:43, Peter Farrow wrote:
Centos 4.0 and 4.1 seem to have a memory leak somewhere and I can't find out what is doing it. I think it may be dovecot.
I have installed hundreds of Linux machines, centos/redhat/suse in the past as firewalls and mail relays and use the same/similar methodolgy and config for each.
I have found that since Centos 4.0 and 4.1 came along machines that have run well for years and months, suddenly and systematically start running out of memory after a few weeks/months of uptime.
The symtpom is the load average goes up and up as the machine creeps into swap usage and eventually the machine has to be rebooted....
this only happened with the introduction of Centos 4 and of course that means Dovecot for the IMAP server....
hmmmm......
Anyone else seen this,
Have you tried running dovecot under xinetd instead of as a long-running daemon?
On Mon, 2005-10-03 at 16:43 +0100, Peter Farrow wrote:
Centos 4.0 and 4.1 seem to have a memory leak somewhere and I can't find out what is doing it. I think it may be dovecot.
I have installed hundreds of Linux machines, centos/redhat/suse in the past as firewalls and mail relays and use the same/similar methodolgy and config for each.
I have found that since Centos 4.0 and 4.1 came along machines that have run well for years and months, suddenly and systematically start running out of memory after a few weeks/months of uptime.
The symtpom is the load average goes up and up as the machine creeps into swap usage and eventually the machine has to be rebooted....
this only happened with the introduction of Centos 4 and of course that means Dovecot for the IMAP server....
hmmmm......
Anyone else seen this,
There are memory leak problems with all kernels before 2.6.9-11.EL
There may be some issues with that kernel too.
Update 2 for EL4 should be released fairly soon, it will have a newer kernel that addresses some other memory leak issues.
-
Bart Schaefer -
Craig White -
Johnny Hughes -
Les Mikesell -
Peter Farrow -
R P Herrold