Hi all.
We've just migrated our back-end NFS mailstore from FC1 systems to CentOS4 and are experiencing problems/delays with vpopmail and user Maildirs. IMAP is fine, it's just POP access that causes us problems.
Now, NFS mounts successfully, the clients can see /home/vpopmail and serve content as we would expect. IMAP works perfectly and sees new messages as soon as they're delivered into the Maildir, but POP doesn't seem to see new messages until about 90 minutes after it arrived.
Messages can be seen in users' Maildir/new on the filesystem. The timestamps on new messages on the NFS filesystems don't appear to be formatted correctly, I'm guessing this could cause the POP weirdness?
On an NFS client... [root@mailserv2 new]# ls -ltr total 4 -rw------- 1 vpopmail vchkpw 854 Feb 28 2006 1141126119.27119.mailserv1,S=773
Whereas on the NFS mailstore... [root@tempmailstore1a new]# ls -ltr total 4 -rw------- 1 vpopmail vchkpw 854 Feb 28 12:44 1141126119.27119.mailserv1,S=773
'touch'ing the files, either from the client or server doesn't appear to fix the timestamps.
The old FC mailstores were running... kernel 2.4.22-1.2188.nptl nfs-utils-1.0.6-1 DRBD 0.6.12 heartbeat-1.0.4-2.fr.c.1.um.2
The new CentOS mailstores and all their clients are running... kernel 2.6.9-22.0.1.ELsmp nfs-utils-1.0.6-65.EL4 DRBD 0.7.14 heartbeat-1.2.3.cvs.20050927-1.centos4 Qmailrocks 2.2.0
/etc/exports on the mailstores contains... /mnt/drbd/ mailserv1(rw,sync,no_root_squash) /mnt/drbd/ mailserv2(rw,sync,no_root_squash) /mnt/drbd/ mailserv3(rw,sync,no_root_squash) /mnt/drbd/ mailserv4(rw,sync,no_root_squash)
And the NFS mounts on the client mailservers look like...
mailstore1:/mnt/drbd/home /home nfs defaults 0 0
If anyone's seen this or can shed any light it'd be much appreciated.
Will.
On Tuesday 28 February 2006 13:29, Will McDonald wrote:
Hi all.
We've just migrated our back-end NFS mailstore from FC1 systems to CentOS4 and are experiencing problems/delays with vpopmail and user Maildirs. IMAP is fine, it's just POP access that causes us problems.
Now, NFS mounts successfully, the clients can see /home/vpopmail and serve content as we would expect. IMAP works perfectly and sees new messages as soon as they're delivered into the Maildir, but POP doesn't seem to see new messages until about 90 minutes after it arrived.
Messages can be seen in users' Maildir/new on the filesystem. The timestamps on new messages on the NFS filesystems don't appear to be formatted correctly, I'm guessing this could cause the POP weirdness?
On an NFS client... [root@mailserv2 new]# ls -ltr total 4 -rw------- 1 vpopmail vchkpw 854 Feb 28 2006 1141126119.27119.mailserv1,S=773
Whereas on the NFS mailstore... [root@tempmailstore1a new]# ls -ltr total 4 -rw------- 1 vpopmail vchkpw 854 Feb 28 12:44 1141126119.27119.mailserv1,S=773
'touch'ing the files, either from the client or server doesn't appear to fix the timestamps.
The old FC mailstores were running... kernel 2.4.22-1.2188.nptl nfs-utils-1.0.6-1 DRBD 0.6.12 heartbeat-1.0.4-2.fr.c.1.um.2
The new CentOS mailstores and all their clients are running... kernel 2.6.9-22.0.1.ELsmp nfs-utils-1.0.6-65.EL4 DRBD 0.7.14 heartbeat-1.2.3.cvs.20050927-1.centos4 Qmailrocks 2.2.0
/etc/exports on the mailstores contains... /mnt/drbd/ mailserv1(rw,sync,no_root_squash) /mnt/drbd/ mailserv2(rw,sync,no_root_squash) /mnt/drbd/ mailserv3(rw,sync,no_root_squash) /mnt/drbd/ mailserv4(rw,sync,no_root_squash)
And the NFS mounts on the client mailservers look like...
mailstore1:/mnt/drbd/home /home nfs defaults 0 0
If anyone's seen this or can shed any light it'd be much appreciated.
Will. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
IMHO looks more like a POP issue to me although the time representation seems odd. But this could be nfs-client related. What does it look like when you mount a share locally on the server where it is exported ?
I'm thinking locking/timing/uid's/accessrights...
Is there anything in the log about POP ? Is it possible to run in debug-mode to generate some ?
Optional: dovecot imap server will also do pop.
On 28/02/06, Paul Schoonderwoerd paul@pollux-it.nl wrote:
On Tuesday 28 February 2006 13:29, Will McDonald wrote:
IMHO looks more like a POP issue to me although the time representation seems odd. But this could be nfs-client related. What does it look like when you mount a share locally on the server where it is exported ?
I'll have a look at that now. Right, weirdly, exporting to localhost and then mounting that locally on the NFS server, the timestamps look OK, but they're still borked on clients.
[root@tempmailstore1a mnt]# mount -t nfs localhost:/mnt/drbd/home/ /mnt/localnfs/ [root@tempmailstore1a mnt]# cd /mnt/localnfs/vpopmail/domains/${domain.tld}/0/G/dailytest/Maildir/new/ [root@tempmailstore1a new]# ls -ltr total 8 -rw------- 1 vpopmail vchkpw 854 Feb 28 13:40 1141126119.27119.mailserv1,S=773
I'm thinking locking/timing/uid's/accessrights...
The UID/GIDs are identical on the new clients and both old and new servers (Qmail/VPOPMail gets very overwrought if you change UIDs :)).
It could be a locking thing, I noticed this on the nanhat list earlier...
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=167192
Is there anything in the log about POP ? Is it possible to run in debug-mode to generate some ?
POP works just fine in terms of authentication, it's just not seeing new files in the Maildirs until 'x' amount of time has passed, where 'x' is orders of magnitude greater than NFS attribute caching could/should account for.
Optional: dovecot imap server will also do pop.
I dread to think what would be involved in migrating all the Qmail/VPOPmail architecure we have over to something else. It's inherited, and entrenched. :o\
Will.
On Tue, 2006-02-28 at 14:42, Will McDonald wrote:
It could be a locking thing, I noticed this on the nanhat list earlier...
Ahh, that's probably why I'm also experience timing issues after moving my nfs-server from redhat 8 to CentOS 4.2. Konqueror on the client sometimes stalls for 15 seconds before returning the files (haven't investigated this further yet though). Could the pop-server time out on that ? Maybe a strace on the popserver combined with a 'telnet <popserver> 110' and the pop list command could reveil something, or debugmode.
Paul
On 28/02/06, Paul Schoonderwoerd paul@pollux-it.nl wrote:
On Tue, 2006-02-28 at 14:42, Will McDonald wrote:
It could be a locking thing, I noticed this on the nanhat list earlier...
Maybe a strace on the popserver combined with a 'telnet <popserver> 110' and the pop list command could reveil something, or debugmode.
I think we've sorted it. It was... ahem.... unsync-ed clocks on the new mailstore servers. They happened to be out by "about 90 minutes". Gah!
I've been chasing my tail all morning over this one, can't believe it was so obvious!
Will.
On Tue, 2006-02-28 at 15:32, Will McDonald wrote:
I think we've sorted it. It was... ahem.... unsync-ed clocks on the new mailstore servers. They happened to be out by "about 90 minutes". Gah!
I've been chasing my tail all morning over this one, can't believe it was so obvious!
Will.
Hahaha.. moral...always look for the obvious things first (like: if you have a timing issue, look at time). Something I learned in the past but have forgotten again apparently. Anyway, good you solved it.
Paul
-
Paul Schoonderwoerd -
Will McDonald