Running Centos 5.4 with KVM on a Dell R610 server and I'd like to control which of the four ethernet interfaces are used for specific tasks
My ideal configuration would be
eth0 - Host traffic only, no virtual guests. Used for guest mirroring and management. eth1 - NAT guest traffic only, no address for local machine and in some environments in the same zone as eth0 eth2/3 - Allocated to two different bridge devices which might be in separate network zones.
The configuration of eth2/3 is fairly simple, my issue is restricting any NAT traffic to a specific ethernet devices, and ideally one with no local IP.
Any ideas?
Steve
On Fri, 2009-11-27 at 14:02 +1300, Steven Ellis wrote:
Running Centos 5.4 with KVM on a Dell R610 server and I'd like to control which of the four ethernet interfaces are used for specific tasks
My ideal configuration would be
eth0 - Host traffic only, no virtual guests. Used for guest mirroring and management. eth1 - NAT guest traffic only, no address for local machine and in some environments in the same zone as eth0 eth2/3 - Allocated to two different bridge devices which might be in separate network zones.
The configuration of eth2/3 is fairly simple, my issue is restricting any NAT traffic to a specific ethernet devices, and ideally one with no local IP.
Any ideas?
Steve
So if I have this right, at the basic level you wish to have:
- One interface for Host machine - Multiple interfaces for guest traffic
If your environment supports VLANs (802.1Q), might I suggest a trunk port on eth1 split up into different bridges to have the KVM guests go through to get on different VLANs/address spaces.
This is what I currently do for Xen and it works great. What kind of network setup to you have?
I would also consider this for physical network isolation.
Put your eth0 and eth1 on separate switches and subnets, then work on the firewall tuning between the NICs in the box from there.
I think do that may follow a stronger firewall physical paradigm where you can disconnect networks to help contain situations until resolved rather than throwing rules at your iptables while under stress.
The extra costs of a couple of switches and wiring could get easily offset by your labor time over a few months.
Tait Clarridge wrote:
On Fri, 2009-11-27 at 14:02 +1300, Steven Ellis wrote:
Running Centos 5.4 with KVM on a Dell R610 server and I'd like to control which of the four ethernet interfaces are used for specific tasks
My ideal configuration would be
eth0 - Host traffic only, no virtual guests. Used for guest mirroring and management. eth1 - NAT guest traffic only, no address for local machine and in some environments in the same zone as eth0 eth2/3 - Allocated to two different bridge devices which might be in separate network zones.
The configuration of eth2/3 is fairly simple, my issue is restricting any NAT traffic to a specific ethernet devices, and ideally one with no local IP.
Any ideas?
Steve
So if I have this right, at the basic level you wish to have:
- One interface for Host machine
- Multiple interfaces for guest traffic
If your environment supports VLANs (802.1Q), might I suggest a trunk port on eth1 split up into different bridges to have the KVM guests go through to get on different VLANs/address spaces.
This is what I currently do for Xen and it works great. What kind of network setup to you have?
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On 26/11/09 10:16 PM, "Tait Clarridge" tait@clarridge.ca wrote:
On Fri, 2009-11-27 at 14:02 +1300, Steven Ellis wrote:
Running Centos 5.4 with KVM on a Dell R610 server and I'd like to control which of the four ethernet interfaces are used for specific tasks
My ideal configuration would be
eth0 - Host traffic only, no virtual guests. Used for guest mirroring and management. eth1 - NAT guest traffic only, no address for local machine and in some environments in the same zone as eth0 eth2/3 - Allocated to two different bridge devices which might be in separate network zones.
The configuration of eth2/3 is fairly simple, my issue is restricting any NAT traffic to a specific ethernet devices, and ideally one with no local IP.
Any ideas?
Steve
So if I have this right, at the basic level you wish to have:
- One interface for Host machine
- Multiple interfaces for guest traffic
If your environment supports VLANs (802.1Q), might I suggest a trunk port on eth1 split up into different bridges to have the KVM guests go through to get on different VLANs/address spaces.
This is what I currently do for Xen and it works great. What kind of network setup to you have?
Could you please provide some pointers on how you accomplished this? I've been attempting to set up a similar configuration without success.
Thanks, Kelvin
On Fri, 2009-11-27 at 12:12 -0500, Kelvin Edmison wrote:
On 26/11/09 10:16 PM, "Tait Clarridge" tait@clarridge.ca wrote:
On Fri, 2009-11-27 at 14:02 +1300, Steven Ellis wrote:
<snip>
So if I have this right, at the basic level you wish to have:
- One interface for Host machine
- Multiple interfaces for guest traffic
If your environment supports VLANs (802.1Q), might I suggest a trunk port on eth1 split up into different bridges to have the KVM guests go through to get on different VLANs/address spaces.
This is what I currently do for Xen and it works great. What kind of network setup to you have?
Could you please provide some pointers on how you accomplished this? I've been attempting to set up a similar configuration without success.
Thanks, Kelvin
Sure.
The way we would do it for a 2 NIC box would be:
eth0 - VLAN trunk eth1 - Storage Network VLAN
After you have been given a trunk port with a native vlan (for eth0, allowing you to set an IP for this connection) you can run or script the following:
- Load the VLAN module
modprobe 8021q
- Use vconfig to add VLAN interfaces - This is going to assume that eth0 is your trunk port - eg. vconfig add eth0 <vlan number> - So for VLAN 22 it would be:
vconfig add eth0 22
- Add the bridge for xen to use (requires bridge-utils package) - eg. brctl addbr xenbr<vlan#> - VLAN 22 would be:
brctl addbr xenbr22
- Add the VLAN interface to the bridge - eg. brctl addif xenbr<vlan#> eth0.<vlan#> - VLAN 22 would be:
brctl addif xenbr22 eth0.22
- Bring both the bridge and the VLAN interface up - eg. ifconfig eth0.<vlan#> up - eg. ifconfig xenbr<vlan#> up - VLAN 22 would be:
ifconfig eth0.22 up ifconfig xenbr22 up
If you wanted to script it to run at boot time you can create a simple for loop with a list of VLANs to use.
In this case I want to use VLANS: - 2 - 4 - 6 - 8 - 200 - 120
The script would start as follows:
#!/bin/bash
VLANS="2 4 6 8 200 120" modprobe 8021q
for i in $VLANS do # above commands go here # starting from vconfig add eth0 22
# just change 22 from the examples to $i done
If you have any trouble, let me know.
----- "Tait Clarridge" tait@clarridge.ca wrote:
After you have been given a trunk port with a native vlan (for eth0, allowing you to set an IP for this connection) you can run or script the following:
- Load the VLAN module
[...]
If you wanted to script it to run at boot time you can create a simple for loop with a list of VLANs to use.
The init scripts support VLANs and bridges out of the box. Adding more scripts is not necessary.
I concur that separating the 4 interfaces as the original author suggested is a poor use of resources. If there are any policies in place for physical network isolation for management, it would be a better idea to get a LOM or a separate NIC for out of band management and bond the 4 primary interfaces for general traffic.
On 11/28/2009 12:43 AM, Tait Clarridge wrote:
The init scripts support VLANs and bridges out of the box. Adding more scripts is not necessary.
Interesting, XEN specific scripts? Or CentOS scripts.
Neither. When using centos, vlan related modules are loaded automatically after specifying "VLAN=YES" in sysconfig/network. And vlan/bridge interfaces are configured exactly the same as "normal" interfaces
On Sat, 2009-11-28 at 00:50 +0200, Manuel Wolfshant wrote:
On 11/28/2009 12:43 AM, Tait Clarridge wrote:
The init scripts support VLANs and bridges out of the box. Adding more scripts is not necessary.
Interesting, XEN specific scripts? Or CentOS scripts.
Neither. When using centos, vlan related modules are loaded automatically after specifying "VLAN=YES" in sysconfig/network. And vlan/bridge interfaces are configured exactly the same as "normal" interfaces
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
----- "Tait Clarridge" tait@clarridge.ca wrote:
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
Example bridge interface:
/etc/sysconfig/network-scripts/ifcfg-br10: DEVICE=br10 TYPE=Bridge BOOTPROTO=none ONBOOT=yes DELAY=0 STP=on
Example vlan interface on a bridge (you can do this with physical interfaces, as well):
/etc/sysconfig/network-scripts/ifcfg-bond0.10: DEVICE=bond0.10 BOOTPROTO=none ONBOOT=yes USERCTL=no VLAN=yes BRIDGE=br10
Example bonded slave physical interface:
/etc/sysconfig/network-scripts/ifcfg-eth0: DEVICE=eth0 BOOTPROTO=none HWADDR=xx:xx:xx:xx:xx:xx ONBOOT=yes TYPE=Ethernet MASTER=bond0 SLAVE=yes USERCTL=no
Example master for bonding:
/etc/sysconfig/network-scripts/ifcfg-bond0: DEVICE=bond0 BOOTPROTO=none ONBOOT=yes USERCTL=no
The bonding parameters are set in /etc/modprobe.conf. e.g.:
alias bond0 bonding options bond0 mode=balance-rr miimon=100 updelay=1000
On Fri, 2009-11-27 at 17:14 -0600, Christopher G. Stach II wrote:
----- "Tait Clarridge" tait@clarridge.ca wrote:
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
Example bridge interface:
/etc/sysconfig/network-scripts/ifcfg-br10: DEVICE=br10 TYPE=Bridge BOOTPROTO=none ONBOOT=yes DELAY=0 STP=on
Example vlan interface on a bridge (you can do this with physical interfaces, as well):
/etc/sysconfig/network-scripts/ifcfg-bond0.10: DEVICE=bond0.10 BOOTPROTO=none ONBOOT=yes USERCTL=no VLAN=yes BRIDGE=br10
I do know how to bond interfaces, I was simply asking about the bridging and vlanning from the network init scripts. although it does it by default I think I will stick to my script as it may be useful if not using a redhat based OS.
The script is also nice as I can just add lines to the VLAN= section to add more at boot time and not have to worry about copying and pasting other ones.
But to each their own.
On 11/28/2009 01:05 AM, Tait Clarridge wrote:
On Sat, 2009-11-28 at 00:50 +0200, Manuel Wolfshant wrote:
On 11/28/2009 12:43 AM, Tait Clarridge wrote:
The init scripts support VLANs and bridges out of the box. Adding more scripts is not necessary.
Interesting, XEN specific scripts? Or CentOS scripts.
Neither. When using centos, vlan related modules are loaded automatically after specifying "VLAN=YES" in sysconfig/network. And vlan/bridge interfaces are configured exactly the same as "normal" interfaces
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
Of course you can. It's all well documented and published at www.centos.org/docs
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
Of course you can. It's all well documented and published at www.centos.org/docs
I am well aware of the docs, must have missed that section.
I was relatively new to VLANs when I used the script, and in the script we have it so we can easily change the vlan trunk interface without having to change the name and internals of all the ifcfg scripts.
Just my two cents, I like having very portable scripts instead of doing a mass reconfiguration I can change one line and its all good.
On 11/28/2009 01:24 AM, Tait Clarridge wrote:
Interesting, thanks for the info. Can I specify which VLANs it is going to configure and bridge?
Of course you can. It's all well documented and published at www.centos.org/docs
I am well aware of the docs, must have missed that section.
I was relatively new to VLANs when I used the script, and in the script we have it so we can easily change the vlan trunk interface without having to change the name and internals of all the ifcfg scripts.
Just my two cents, I like having very portable scripts instead of doing a mass reconfiguration I can change one line and its all good.
I am sure that must be very handy when you manage several dozens systems via puppet but as you have said, to each their own. you prefer slackware, I prefer centos.
but as you have said, to each their own. you prefer slackware, I prefer centos.
I never said I use Slackware... anyways. It is nice when rolling out dozens of machines quickly and not having to worry about all the VLAN and bridge init scripts.
Tait Clarridge wrote:
On Fri, 2009-11-27 at 14:02 +1300, Steven Ellis wrote:
Running Centos 5.4 with KVM on a Dell R610 server and I'd like to control which of the four ethernet interfaces are used for specific tasks
My ideal configuration would be
eth0 - Host traffic only, no virtual guests. Used for guest mirroring and management. eth1 - NAT guest traffic only, no address for local machine and in some environments in the same zone as eth0 eth2/3 - Allocated to two different bridge devices which might be in separate network zones.
The configuration of eth2/3 is fairly simple, my issue is restricting any NAT traffic to a specific ethernet devices, and ideally one with no local IP.
Any ideas?
Steve
So if I have this right, at the basic level you wish to have:
- One interface for Host machine
- Multiple interfaces for guest traffic
If your environment supports VLANs (802.1Q), might I suggest a trunk port on eth1 split up into different bridges to have the KVM guests go through to get on different VLANs/address spaces.
This is what I currently do for Xen and it works great. What kind of network setup to you have?
Not quite what I'm after.
My issue is I can't see any way to bring up NAT guests unless they are using a ethernet interface that has a address for the host OS.
eth0 - Host OS Traffic only eth1 - NAT traffic for any guests I don't want to bridge eth2 - Bridge traffic zone 1 eth3 - Bridge traffic zone 2
Setting up eth0/2/3 is ok, but eth1 is the issue.
Steve
-
Ben M. -
Christopher G. Stach II -
Kelvin Edmison -
Manuel Wolfshant -
Steven Ellis -
Tait Clarridge