On Wed, 9 Nov 2005, Colm MacCarthaigh wrote:

On Tue, Nov 08, 2005 at 12:36:04PM +0100, Dag Wieers wrote:

...

I don't know what the problem is. Either the file is empty on the filesystem (corruption on the filesystem level or problems with rsync or NFS) or the web server is handing out empty files under certain conditions.

It's the latter, and the problem has been traced to yet another bug to do with sendfile() and TCP checksum offloading. Basically ... the packets were being sent, but with invalid checksums. So the clients were rejecting the packets.

...

Without access to the system I have no clue whether the filesystem is the problem or it is Apache related. I bet it is Apache's problem as mirrors of Heanet do not seem to have this problem. (This is different than when it was a NFS problem with the same symptoms)

Turned out not to be an Apache problem, as anything which used sendfile() would have been effected. The problem is really in the network cards - and how they handle offloaded checksums when DSCP headers are set.

Thanks Colm, I'm very interested to learn how you figured this out.

Also, are there any tests/monitors to find the same cause in the future ? It could be something that is silently bothering other people as well, and a technical explanation together with Google might make this valuable information.

Kind regards, -- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [all I want is a warm bed and a kind word and unlimited power]