After successfully getting higher MTU to work on my Realtek NICs, I started testing the impact of higher MTU on file transfers using NFS exported ramdisk to ramdisk.
The results were unexpected. The higher the MTU on the sending NIC, the lower the file transfer speed. I tested by using time cp to copy a 1GB file (In case compression might affect the results, so I dd the test file from the CentOS DVD ISO)
The average of 6 runs per MTU with varying MTU on the receiving NICs MTU 1500 = 116.34 MB/s MTU 3000 = 103.42 MB/s MTU 4000 = 102.52 MB/s
I was actually planning to do 1500, 3000, 4500, 6000 and 7200. However, I was surprised to discover although I had two seemingly identical Realtek PCI-E nics, the sending NIC could only go up to 4K.
I'm wondering, that since Jumbo Frames was supposed to be better for bulk transfers, why am I seeing these results? Is it the ElRepo drivers I used to enable higher MTUs or possibly some kind of oddity with the realtek NICs I am using? Or am I mistaken about the benefits of jumbo frames and that they are only beneficial in specific configurations, perhaps only with higher network speeds and enough packets at 1500 MTU to overwhelm packet handling hardware?
Emmanuel Noobadmin wrote:
I'm wondering, that since Jumbo Frames was supposed to be better for bulk transfers, why am I seeing these results? Is it the ElRepo drivers I used to enable higher MTUs or possibly some kind of oddity with the realtek NICs I am using? Or am I mistaken about the benefits of jumbo frames and that they are only beneficial in specific configurations, perhaps only with higher network speeds and enough packets at 1500 MTU to overwhelm packet handling hardware? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I googled about Jumbo Frames ("jumbo frames impact") and 2 top hits are very interesting: http://sd.wareonearth.com/~phil/jumbo.html http://www.chelsio.com/jumbo_enet_frames.html
I guess if you are not happy with speed provided by Realtek you should definitely buy more expensive NIC's with support of 9000K packets.
Ljubomir
On 06/25/2011 01:13 PM, Emmanuel Noobadmin wrote:
The results were unexpected. The higher the MTU on the sending NIC, the lower the file transfer speed.
What kind of switch are you using? If your switches don't support large packets, you'll also see slowdowns like that.
On 6/26/11, Gordon Messmer yinyang@eburg.com wrote:
On 06/25/2011 01:13 PM, Emmanuel Noobadmin wrote:
The results were unexpected. The higher the MTU on the sending NIC, the lower the file transfer speed.
What kind of switch are you using? If your switches don't support large packets, you'll also see slowdowns like that.
It's a 3Com 2920 now known as HP V1910 switch. Jumbo frames up to 10K is supposed to be supported.
I'm wondering, that since Jumbo Frames was supposed to be better for bulk transfers, why am I seeing these results? Is it the ElRepo drivers I used to enable higher MTUs or possibly some kind of oddity with the realtek NICs I am using? Or am I mistaken about the benefits of jumbo frames and that they are only beneficial in specific configurations, perhaps only with higher network speeds and enough packets at 1500 MTU to overwhelm packet handling hardware?
In short, the entire path from origin to destination must be configured to support jumbo frames. If not, then the devices at various points in the path will attempt to scale and you'll end up getting fragmentation and higher overhead in the devices leading to worse overall performance.
Have you verified this is the case?
Of course, the quality of the driver for your NIC will also play a role.
On 6/26/11, Geoff Galitz geoff@galitz.org wrote:
In short, the entire path from origin to destination must be configured to support jumbo frames. If not, then the devices at various points in the path will attempt to scale and you'll end up getting fragmentation and higher overhead in the devices leading to worse overall performance.
Have you verified this is the case?
I believed so everything along the path supports the MTU size I was testing with. The test path is short, basically NIC -> Lan Cable -> HP 1910 -> Lan Cable -> NIC.
The switch is spec'd to support up to 9K jumbo frame, enabled by default based on a FAQ I found on 3Com/HP when trying to figure out where do I turn it on.
Of course, the quality of the driver for your NIC will also play a role.
This I have no idea which was why I was wondering about the elrepo driver.
-
Emmanuel Noobadmin -
Geoff Galitz -
Gordon Messmer -
Ljubomir Ljubojevic