[CentOS-virt] disk i/o benchmarking inside vm

Sun Oct 23 01:00:38 UTC 2011
Dennis Jacobfeuerborn <dennisml at conversis.de>

On 10/20/2011 08:13 AM, Charles Polisher wrote:
> Mon, Oct 10, 2011, Dennis Jacobfeuerborn wrote:
> <snip />
>> The physical numbers are consistent and what I would expect to see from the
>> sata drives.
>> The guests are minimal centos 6 installations so after booting they have
>> virtually no processes running that could influence the benchmarks in any
>> significant way. The hosts system is installed on /dev/sda so /dev/sd(b|c)
>> are not influenced by the host system either.
>> The entire setup is arranged to make benchmarking mostly reliable.
>> If there were minor temporary fluctuations I would blame some external
>> process but differences of almost 100% that are consistent for the lifetime
>> of the virtual machine do not fit such a scenario.
>
> Hi Dennis,
>
> Did you figure this out?
>
> I'd suggest running something like smartctl -t long to rule out
> underlying issues with a drive.

But that's what puzzles me. On the host side both drives behave exactly 
identical and as expected. It's only in the guests where things get 
strange. I now have created a second disk /dev/vdb with an image on the 
second drive in my first guest so that it has disk from both physical 
drives that I can test in the same VM. These are the results:

seekmark:
/dev/vda:  130 seeks/s
/dev/vdb: 9615 seeks/s

hdparm -t:
/dev/vda:   95 MB/s
/dev/vdb: 1691 MB/s

Running iostat shows no I/O activity when running the tests for /dev/vdb 
which explains the insane numbers. The question is why I get such different 
results when both devices are defined exactly the same way?

In the guest the drive are running using virtio with type=raw and cache=none.
On the host the backing filesystems look exactly the same:

[dennis at nexus ~]$ cat /proc/mounts|grep backup
/dev/sdb3 /mnt/backup01 ext4 
rw,seclabel,relatime,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/sdc3 /mnt/backup02 ext4 
rw,seclabel,relatime,user_xattr,acl,barrier=1,data=ordered 0 0

The drives are identical in every possible way:

[dennis at nexus ~]$ sudo hdparm -i /dev/sdb

/dev/sdb:

  Model=SAMSUNG HD103SJ, FwRev=1AJ100E5, SerialNo=S246J9JB801028
  Config={ Fixed }
  RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
  BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16
  CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
  IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
  PIO modes:  pio0 pio1 pio2 pio3 pio4
  DMA modes:  mdma0 mdma1 mdma2
  UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
  AdvancedPM=yes: disabled (255) WriteCache=enabled
  Drive conforms to: unknown:  ATA/ATAPI-0,1,2,3,4,5,6,7

  * signifies the current active mode

[dennis at nexus ~]$ sudo hdparm -i /dev/sdc

/dev/sdc:

  Model=SAMSUNG HD103SJ, FwRev=1AJ100E5, SerialNo=S246J9JB801029
  Config={ Fixed }
  RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
  BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16
  CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
  IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
  PIO modes:  pio0 pio1 pio2 pio3 pio4
  DMA modes:  mdma0 mdma1 mdma2
  UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
  AdvancedPM=yes: disabled (255) WriteCache=enabled
  Drive conforms to: unknown:  ATA/ATAPI-0,1,2,3,4,5,6,7

  * signifies the current active mode


Regards,
   Dennis