On 6/9/11, Benjamin Franz jfranz@freerun.com wrote:

You should look at running your process using 'ionice -c3 program'. That way it won't starve everything else for I/O cycles. Also, you may want to experiment with using the 'deadline' elevator instead of the default 'cfq' (see http://www.redhat.com/magazine/008jun05/features/schedulers/ and http://www.wlug.org.nz/LinuxIoScheduler). Neither of those would require you to change your hardware out. Also, setting 'noatime' for the mount options for partition holding the files will reduce the number of required I/Os quite a lot.

Thanks for pointing out noatime, I came across in my reading previously but it never sunk in. This experience is definitely going to make sure of that :)

Tthe crawl process is started by another program. crond starts the program, the program starts the email crawl or take other more crucial action depending on situation so I'm unsure if I should run it with ionice since it could potentially cause the more crucial action to lag/slow down.

But I'll give it a try anyway over the weekend when any negative effect has lesser consequences and see if it affects other things.

But yes, in general, distributing your load across more disks should improve your I/O profile.

I'm going with noatime and ionice first to see the impact before I start playing around with the i/o scheduler. If all else fails, then I'll see about requesting for the extra hard disk.

On 6/10/11, Steven Tardy sjt5@its.msstate.edu wrote:

did you set noatime on the host filesystem and/or the VM filesystem? i would think noatime on the VM would provide more benefit than on the host... shrug. now my brain hurts. gee thanks. (:

I was trying it on the host first, thinking that would cut down on half the writes since the host wouldn't have to update the atime everytime the diskfiles are accessed.

But now that you brought it up, I'm wondering if that would had been pointless. If the kernel considers KVM opening the diskfile and holding onto it as a single access, regardless of how many subsequent reads/writes there are, then this wouldn't make a difference would it?

On 6/10/11, Gordon Messmer yinyang@eburg.com wrote:

atime and mtime are updated for *every* read and write operation, not for the open() of the file.

Ok. In any case, the combination of atime and ionice on the cronjob seems to have helped, no locked up in the past 24 hours. But it is a Saturday here so that might just be due to light usage, keeping fingers crossed.

That aside, if you're running KVM I strongly recommend using LVM rather than file-backed VM guests. It's more work to set up, but you'll see drastically better IO performance in the guests. One system that I measured had a write speed of around 8 MB/s for sequential block output on file-backed VMs. LVM backed VMs wrote at around 56 MB/s for sequential block output.

You should *never* used file-backed VMs for production systems.

The irony of it was that I decided to go with qcow2 because I thought that would save overheads from an additional LVM layer but provided snapshot capabilities too :(

Since I don't have enough spare space left on this particular system, I'll probably have to get them to agree to add an extra disk to do the LVM volumes, then figure out how to migrate the VM over from file to raw/partition.

On 6/16/11, Gordon Messmer yinyang@eburg.com wrote:

I read somewhere recently that people were complaining abut LVM overhead and poor performance, but I've never seen any evidence of it. Was there something that made you think that LVM had significant overhead?

Looking at some very sparse notes I made on the decision, I think what tipped the choice was that both qcow2 and lvm added overheads, but lvm was on the whole system i.e. the host has additional processing on every i/o whereas qcow2 overheads was only for guest i/o. More critically my note was the thought as well that it would be easier to move a qcow2 file to another machine/disk if necessary than to move a partition.

On 6/16/11, Gordon Messmer yinyang@eburg.com wrote:

I think you were misinformed, or misled.

That wouldn't be new for me as far as system administration is concerned :D

LVM should not present any noticeable overhead on the host. Using "raw" files to back VMs presents a significant overhead to guests; the host performs all IO through its filesystem. Using "qcow2" files presents even more overhead (probably the most of any configuration) since there are complexities to the qcow2 file itself in addition to the host's filesystem.

I was concerned about qcow2 vs raw as well since it seemed logical that qcow2 would be slower for the added functionality. However there was some site I found that showed that KVM with virtio, turning off host caching (or specifying write-back instead of the default write-through) on the file and doing preallocation on qcow2 files will make qcow2 as fast as raw.

It shouldn't be significantly harder to copy the contents of a partition or LV. The block device is a file. You can read its contents to copy them just as easily as any other file.

Although the combination of ionice and atime seemed to have stopped things from going through the roof, I'll probably still try to convert one of them to LVM and see if that improves things even further.

On 6/10/11, Markus Falb markus.falb@fasel.at wrote:

Yes, but before doing this be sure that your Software does not need atime.

For a brief moment, I had that sinking "Oh No... why didn't I see this earlier" feeling especially since I've already remounted the filesystem with noatime.

Fortunately, so far it seems that everything's still alive and working, keeping fingers crossed :D

On 6/9/2011 1:09 PM, Emmanuel Noobadmin wrote:

On 6/10/11, Markus Falbmarkus.falb@fasel.at wrote:

...

Yes, but before doing this be sure that your Software does not need atime.

For a brief moment, I had that sinking "Oh No... why didn't I see this earlier" feeling especially since I've already remounted the filesystem with noatime.

Fortunately, so far it seems that everything's still alive and working, keeping fingers crossed :D

The last access time is generally not needed, especially for Maildirs. On our setup, Postfix and Dovecot don't care. I always mount as many file systems as possible with 'noatime'.

(Our IMAP Maildir storage is a 4-disk RAID 1+0 array with a few million individual messages across a lot of accounts.)

--On Thursday, June 09, 2011 07:04:24 PM +0200 Rudi Ahlers Rudi@SoftDux.com wrote:

Can one mount the root filesystem with noatime?

Generally speaking, one can mount any of the filesystems with noatime. Whether or not this is a good thing depends on your use. As was previously mentioned, some software (but not a lot) depends on it. The only thing that comes to mind offhand is mail software that uses a single-file monolithic mailbox. (Cyrus IMAPd, for example, uses one file per message, so noatime doesn't affect its behavior).

With noatime, you also (obviously) lose the ability to look at access times. *Once*, in my career, that was useful for doing forensics on a cracked system.

OTOH, it can make a good performance improvement. On SSDs, it's can also help extend the drive's life.

Devin