[CentOS-virt] Traffic Accounting & KVM vs Xen

On Thu, 15 Aug 2013, Robert Dinse wrote:

>      I am getting to where we want to offer virtual servers 
> for lease but to do so we need some method of measuring 
> and/or limiting traffic to individual guests.
>
>      I am wondering what others are using for this purpose? 
> I know that you can look at traffic stats on the bridge on 
> the host machine but that information is lost when the 
> machine is rebooted.  I'm wondering if there is any software 
> that databases that information on an ongoing basis and does 
> not lost information across reboots?

At PMman [1], we sample all domU and each dom0, both Xen and 
KVM based, via libvirt methods every five minutes.  We also 
keep snmp derived data, which are out of scope to your 
question.  We use the virsh 'stat' commands ... nodecpustats 
nodememstats domblkstat domifstat dommemstat .  Then, we stuff 
detail into a database.  The use of the intermediate store of 
a database provides a 'looser' coupling' than blocking on 
other methods, so that our control interfaces do not get 
'blocked' when things get SNAFU'd

A sample insert looks like this for drive stats on a domU:

       //insert stats into db
       $_dbQuery = "insert into vm_blk_stats set "
                 . "date = now(), "
                 . "vm_name = '".$_vm_name."', "
                 . "vm_server = '".$_vm_server."', "
                 . "vm_running_id = '".$vm_running_ids[$_vm_name]."', "
                 . "device_index = '".$_blk_index."', "
                 . "device = '".$_blkdev."', "
                 . "rd_req = '".$_blk_stats[$_blk_index]['rd_req'] ."', "
                 . "rd_bytes = '".$_blk_stats[$_blk_index]['rd_bytes'] ."', "
                 . "wr_req = '".$_blk_stats[$_blk_index]['wr_req'] ."', "
                 . "wr_bytes = '".$_blk_stats[$_blk_index]['wr_bytes'] ."'";
       mysql_query($_dbQuery);

and like this for the VM interfaces:

       $_dbQuery = "insert into vm_if_stats set "
                 . "date = now(), "
                 . "vm_name = '".$_vm_name."', "
                 . "vm_server = '".$_vm_server."', "
                 . "vm_running_id = '".$vm_running_ids[$_vm_name]."', "
                 . "device_index = '".$_eth_index."', "
                 . "device = '".$_ethdev."', "
                 . "rx_bytes = '".$_eth_stats[$_eth_index]['rx_bytes'] ."', "
                 . "rx_packets = '".$_eth_stats[$_eth_index]['rx_packets'] ."', "
                 . "rx_errs = '".$_eth_stats[$_eth_index]['rx_errs'] ."', "
                 . "rx_drop = '".$_eth_stats[$_eth_index]['rx_drop'] ."', "
                 . "tx_bytes = '".$_eth_stats[$_eth_index]['tx_bytes'] ."', "
                 . "tx_packets = '".$_eth_stats[$_eth_index]['tx_packets'] ."', "
                 . "tx_errs = '".$_eth_stats[$_eth_index]['tx_errs'] ."', "
                 . "tx_drop = '".$_eth_stats[$_eth_index]['tx_drop'] ."'";
       mysql_query($_dbQuery);

The second is an insert for traffic in and out, per interface 
(which interfaces can 'move around' as to 'name' as to how one 
queries it out via virsh, as VM's come and go)

We have had several tens of million rows active in those 
tables over time, but usually 'age them out' when we get north 
of 20 million into secondary summary tables to keep later 
query performance reasonable

'domifstat' is useful, because we see circumstances where a VM 
is seemingly active, but not moving any network traffic in or 
out (i.e., it has crashed).  We use monitoring of traffic 
stats to detect problems pre-emptively (i.e., before the 
customer calls).  We had an instance of this earlier this week 
after an attack on an httpd of a client VM, which we 
identified.  I got an external monitoring report,and looked 
in.  On the virsh console, it was reporting OOM problems

>      Second question, what are the advantaged and 
> disadvantages of KVM verses Xen?  I played with Xen back 
> when I had CentOS 5, but find KVM easier to work with and 
> not much difference in performance.

We run, offer, and support both, both externally and in our 
developmental labs, but Xen is not the future for people 
following Red Hat, nothwithstanding the CentOS efforts.  Our 
new development effort is KVM focussed

-- Russ herrold

[1] http://www.pmman.com/