I was trying to do some performance testing between using iSCSI on the host as a diskfile to a guest vs the VM guest using the iSCSI device directly.
However, in the process of trying to establish a baseline performance figure, I started increasing the MTU settings on the PCI-express NICs with RTL8168B chips.
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
Checking the syslog, I discovered warnings that increasing MTU with this adapter may cause problems.
Searching around, it seems to be a common problem but there doesn't appear to be any clear cut fix, including some suggestions to use a third party driver.
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work?
----- Original Message -----
I was trying to do some performance testing between using iSCSI on the host as a diskfile to a guest vs the VM guest using the iSCSI device directly.
However, in the process of trying to establish a baseline performance figure, I started increasing the MTU settings on the PCI-express NICs with RTL8168B chips.
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
Checking the syslog, I discovered warnings that increasing MTU with this adapter may cause problems.
Searching around, it seems to be a common problem but there doesn't appear to be any clear cut fix, including some suggestions to use a third party driver.
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work?
Realtek NICs are known to be some of the poorest interfaces available. A quality Intel or Broadcom NIC will set you back very little in terms of cost. Just replace it and be done. :)
--Tim
On 06/23/2011 01:28 PM, Tim Nelson wrote:
----- Original Message -----
I was trying to do some performance testing between using iSCSI on the host as a diskfile to a guest vs the VM guest using the iSCSI device directly.
However, in the process of trying to establish a baseline performance figure, I started increasing the MTU settings on the PCI-express NICs with RTL8168B chips.
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
Checking the syslog, I discovered warnings that increasing MTU with this adapter may cause problems.
Searching around, it seems to be a common problem but there doesn't appear to be any clear cut fix, including some suggestions to use a third party driver.
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work?
Realtek NICs are known to be some of the poorest interfaces available. A quality Intel or Broadcom NIC will set you back very little in terms of cost. Just replace it and be done. :)
--Tim
I second this. I switched out for fairly inexpensive Intel Pro/1000CT adapters. They're to be had for ~$30~40 in Canada and work perfectly at 9kb JFs.
Realtek is really built down to cost, and is not viable outside of basic web browsing, imho.
On 6/24/11, Tim Nelson tnelson@rockbochs.com wrote:
Realtek NICs are known to be some of the poorest interfaces available. A quality Intel or Broadcom NIC will set you back very little in terms of cost. Just replace it and be done. :)
I was afraid that might be the case (already had two Intel NICs in the shopping cart to be honest) but was hoping I didn't have to waste time explaining why the plentiful Realtek NICs around cannot be used for testing.
Tim Nelson wrote:
----- Original Message -----
I was trying to do some performance testing between using iSCSI on the host as a diskfile to a guest vs the VM guest using the iSCSI device directly.
However, in the process of trying to establish a baseline performance figure, I started increasing the MTU settings on the PCI-express NICs with RTL8168B chips.
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
Checking the syslog, I discovered warnings that increasing MTU with this adapter may cause problems.
Searching around, it seems to be a common problem but there doesn't appear to be any clear cut fix, including some suggestions to use a third party driver.
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work?
Realtek NICs are known to be some of the poorest interfaces available. A quality Intel or Broadcom NIC will set you back very little in terms of cost. Just replace it and be done. :)
--Tim _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I can not say I had any problems with Realtek NIC's. I operate a small WISP and I am active in StarOS wireless network reouter community. Most of us there agree that when you have problems with NICS like 3Com or even Intel, you can always use Realtek NIC's for X86-PC routers.
Try ElRepo driver and please report if that helps. I would like to know your experience with ElRepo driver.
Ljubomir
----- Original Message -----
I can not say I had any problems with Realtek NIC's. I operate a small WISP and I am active in StarOS wireless network reouter community. Most of us there agree that when you have problems with NICS like 3Com or even Intel, you can always use Realtek NIC's for X86-PC routers.
Try ElRepo driver and please report if that helps. I would like to know your experience with ElRepo driver.
Realtek NICs and drivers are typically quite ubiquitous in that they work on nearly every platform. This is especially true with the RTL-8139(+) series. However, that doesn't make them a quality interface. :)
I'm sure there are plenty of installations where these NICs work great without problems. In fact, now that I think about it, my home router uses two RTL-8189 interfaces on FreeBSD 8.1 without problems. They certainly have a proper spot in the network (low throughput, etc). I just wouldn't want them in my servers, especially not for ones expected of high performance or storage related tasks.
--Tim
On 6/24/11, Ljubomir Ljubojevic office@plnet.rs wrote:
Try ElRepo driver and please report if that helps. I would like to know your experience with ElRepo driver.
The ElRepo driver appears to work, I don't get an error when increasing the MTU but I'll need to solve another problem before I can really test and be sure.
Emmanuel Noobadmin wrote:
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Try driver from ElRepo: http://elrepo.org/tiki/tiki-index.php?page=kmod-r8168
Ljubomir
On Friday, June 24, 2011 01:20 AM, Emmanuel Noobadmin wrote:
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
Yeah, the 8168C goes up to 7k. Some 8168B go up to 6k.
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
I have the crap enabled in Windows and it works there...but on an old server running OpenIndiana the driver refuses to enable jumbo frames for the particular chipset that the board has which is incidentally a 8168B.
Checking the syslog, I discovered warnings that increasing MTU with this adapter may cause problems.
Searching around, it seems to be a common problem but there doesn't appear to be any clear cut fix, including some suggestions to use a third party driver.
Does anybody know of a proven solution or is the Realtek chip itself irrevocably broken/bugged that anything above the default 1500 will simply not work?
Using realtek's own drivers on Windows seem to be okay...not sure about Linux and other platforms.
On 6/24/11, Christopher Chan christopher.chan@bradbury.edu.hk wrote:
On Friday, June 24, 2011 01:20 AM, Emmanuel Noobadmin wrote:
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
Yeah, the 8168C goes up to 7k. Some 8168B go up to 6k.
In cases like this where there are conflicting sources of information regarding the max MTU of a NIC, what would be the correct way to determine the actual max MTU? I figured the 7K limit basically by doing a binary search with the ifconfig mtu commands. But is the this figure simply what the driver will accept for the controller it identified or is that the actual hardware limit?
On Friday, June 24, 2011 02:33 PM, Emmanuel Noobadmin wrote:
On 6/24/11, Christopher Chanchristopher.chan@bradbury.edu.hk wrote:
On Friday, June 24, 2011 01:20 AM, Emmanuel Noobadmin wrote:
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
Yeah, the 8168C goes up to 7k. Some 8168B go up to 6k.
In cases like this where there are conflicting sources of information regarding the max MTU of a NIC, what would be the correct way to determine the actual max MTU? I figured the 7K limit basically by doing a binary search with the ifconfig mtu commands. But is the this figure simply what the driver will accept for the controller it identified or is that the actual hardware limit?
Given that these are the limits from Realtek's own Windows drivers, I'd say they are hardware limits and different from chipset/rev.
Christopher Chan wrote:
On Friday, June 24, 2011 02:33 PM, Emmanuel Noobadmin wrote:
On 6/24/11, Christopher Chanchristopher.chan@bradbury.edu.hk wrote:
On Friday, June 24, 2011 01:20 AM, Emmanuel Noobadmin wrote:
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
Yeah, the 8168C goes up to 7k. Some 8168B go up to 6k.
In cases like this where there are conflicting sources of information regarding the max MTU of a NIC, what would be the correct way to determine the actual max MTU? I figured the 7K limit basically by doing a binary search with the ifconfig mtu commands. But is the this figure simply what the driver will accept for the controller it identified or is that the actual hardware limit?
Given that these are the limits from Realtek's own Windows drivers, I'd say they are hardware limits and different from chipset/rev.
This post: http://ubuntuforums.org/showthread.php?t=1628575 Suggests that Windows and OpenSolaris were pusing 3x more then his current Ubuntu. This would suggest Linux driver problem. That is why I suggested ElRepo driver.
I just checked my server and my desktop. They both have 8168B NIC. Server integrated and desktop has PCI-X if I am not mistaken (I on laptop 35km away) with what seams to be stock kernel driver r8169.
When I first bought Gigabit switch (cheap Intex) they were both plugged next to each other. CentOS -> CentOS NFSv4 large files copy gave me sustained speed of ~250Mbps (~31 MBps), without even touching the settings, which is good for HDD(s) I had back then.
I can not comment on JUMBO frames, never needed them. But it would be nice to see if there is difference with ElRepo drivers.
Ljubomir
On Friday, June 24, 2011 05:06 PM, Ljubomir Ljubojevic wrote:
Christopher Chan wrote:
On Friday, June 24, 2011 02:33 PM, Emmanuel Noobadmin wrote:
On 6/24/11, Christopher Chanchristopher.chan@bradbury.edu.hk wrote:
On Friday, June 24, 2011 01:20 AM, Emmanuel Noobadmin wrote:
First bottleneck was discovering the max MTU allowed on these is 7K instead of 9K but googling seems to indicate that the RTL8168B is only capable of 4K frames.
Yeah, the 8168C goes up to 7k. Some 8168B go up to 6k.
In cases like this where there are conflicting sources of information regarding the max MTU of a NIC, what would be the correct way to determine the actual max MTU? I figured the 7K limit basically by doing a binary search with the ifconfig mtu commands. But is the this figure simply what the driver will accept for the controller it identified or is that the actual hardware limit?
Given that these are the limits from Realtek's own Windows drivers, I'd say they are hardware limits and different from chipset/rev.
This post: http://ubuntuforums.org/showthread.php?t=1628575 Suggests that Windows and OpenSolaris were pusing 3x more then his current Ubuntu. This would suggest Linux driver problem. That is why I suggested ElRepo driver.
No way jumbo frames is enabled on OpenSolaris/OpenIndiana. I run OpenIndiana here with boards that have realtek onboard. They refuse to enable jumbo frame support.
On 06/23/2011 10:20 AM, Emmanuel Noobadmin wrote:
I assumed 4K would still be better than nothing but unfortunately bumping up the MTU to anything else but 1.5K caused the file transfers (using NFS for easy testing), to hang at random points or more accurate slow to a crawl.
I don't know anything specifically about those cards, but you'll see that behavior on any card unless all of the hosts on a broadcast domain are using the same MTU. You need to set all of the devices on a LAN segment, including the router, to the same MTU.
On 6/26/11, Gordon Messmer yinyang@eburg.com wrote:
I don't know anything specifically about those cards, but you'll see that behavior on any card unless all of the hosts on a broadcast domain are using the same MTU. You need to set all of the devices on a LAN segment, including the router, to the same MTU.
Thanks for this note, but it didn't seem to be necessary in my case.
I just finished updating the drivers and running some test, the results were this.
In all cases, default 1500 MTU to 1500MTU = OK for both default and Elrepo drivers.
NFS Host -> NFS Client Default Drivers for both 1500 MTU -> 3000 MTU = fail 3000 MTU -> 3000 MTU = fail
Default Driver -> ElRepo driver 1500 MTU -> 3000 MTU = OK 3000 MTU -> x MTU = fail
ElRepo -> ElRepo driver Both OK
So the MTU problem appears to be firmly pinpointed to the default RTL drivers and both receiving/transmitting NICs must be using the ElRepo drivers to work properly at higher MTU.
Ironically, even at 1500MTU, I was able to hit the full 100+ MB/s speed available on a single Gigabit link so it seems my experiment was a bit futile.
Emmanuel Noobadmin wrote:
Ironically, even at 1500MTU, I was able to hit the full 100+ MB/s speed available on a single Gigabit link so it seems my experiment was a bit futile. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I am glad the "honor" of the Realtek NIC's is defended in the sense that they are not cheap crap, just cheap :-)
Ljubomir
On 6/26/11, Gordon Messmer yinyang@eburg.com wrote:
On 06/25/2011 12:05 PM, Emmanuel Noobadmin wrote:
Default Driver -> ElRepo driver 1500 MTU -> 3000 MTU = OK
That's not going to be reliable. Sooner or later, you'll see the mount hang (any transfer may) if your MTUs don't match.
Thanks for the warning although I wasn't on planning on using mismatched MTU for real operations :) It was for the purpose of determining the effects of using the different MTU that I was doing that.
Even if there wasn't the risk of a hang, for the sake of better performance, it seems that matched MTU are better anyway. The sys time component increased on the receiving side in relation to the receiver's MTU but drops significantly (8~10%) when non-default MTU are matched.
On Jun 25, 2011, at 3:05 PM, Emmanuel Noobadmin centos.admin@gmail.com wrote:
Ironically, even at 1500MTU, I was able to hit the full 100+ MB/s speed available on a single Gigabit link so it seems my experiment was a bit futile.
Jumbo frames don't give higher throughput, but lower CPU.
I find standard frames give better throughput for 4K block sizes like one typically finds with file systems.
You really don't need to go to jumbo frames until you reach 10Gbps speed.
-Ross
On 6/26/11, Ross Walker rswwalker@gmail.com wrote:
Jumbo frames don't give higher throughput, but lower CPU.
I find standard frames give better throughput for 4K block sizes like one typically finds with file systems.
You really don't need to go to jumbo frames until you reach 10Gbps speed.
This was pretty much my conclusion (only useful at higher speeds and for reducing packet processing overheads) after further reading since the experiment.
Now the question is whether the overheads reduction, even at sub-10GBs speeds, may be significant if the host/guest are VMs instead of actual physical machines.
Emmanuel Noobadmin wrote:
Now the question is whether the overheads reduction, even at sub-10GBs speeds, may be significant if the host/guest are VMs instead of actual physical machines.
If you are going to use it on virtual interfaces, I would think it would help, especially if you have greater number of VM's. If virtual interfaces can achieve greater/unlimited speed, this could even have greater impact on throughput itself.
Ljubomir
-
Christopher Chan -
Digimer -
Emmanuel Noobadmin -
Gordon Messmer -
Ljubomir Ljubojevic -
Ross Walker -
Tim Nelson