[CentOS] Bonding performance question

[James Pearson]

lhecking at users.sourceforge.net wrote:
>  I have a Dell server with four bonded, gigabit interfaces. Bonding mode is
>  802.3ad, xmit_hash_policy=layer3+4. When testing this setup with iperf,
>  I never get more than a total of about 3Gbps throughput. Is there anything
>  to tweak to get better throughput? Or am I running into other limits (e.g.
>  was reading about tcp retransmit limits for mode 0).
> 
>  The iperf test was run with iperf -s on the server, and iperf -c server
>  on four clients connected to the same switch.
> 
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.10.15.193 port 5001 connected with 10.10.15.184 port 48588
> [  5] local 10.10.15.193 port 5001 connected with 10.10.15.187 port 49231
> [  6] local 10.10.15.193 port 5001 connected with 10.10.15.188 port 53197
> [  7] local 10.10.15.193 port 5001 connected with 10.10.15.189 port 55309
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
> [  5]  0.0-10.0 sec   872 MBytes   728 Mbits/sec
> [  6]  0.0-10.0 sec   318 MBytes   267 Mbits/sec
> [  7]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
> [  8] local 10.10.15.193 port 5001 connected with 10.10.15.184 port 48589
> [  4] local 10.10.15.193 port 5001 connected with 10.10.15.187 port 49234
> [  5] local 10.10.15.193 port 5001 connected with 10.10.15.188 port 38864
> [  6] local 10.10.15.193 port 5001 connected with 10.10.15.189 port 55311
> [  8]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
> [  4]  0.0-10.0 sec   862 MBytes   721 Mbits/sec
> [  5]  0.0-10.0 sec   322 MBytes   270 Mbits/sec
> [  6]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec

Could it be that with the combinations of IP addresses and port numbers 
used by iperf above, it ends up with only 3 of the links being used?

According to the Linux bonding docs, xmit_hash_policy=layer3+4 uses:

   ((source port XOR dest port) XOR
          ((source IP XOR dest IP) AND 0xffff)
                 modulo slave count

So I guess you could plug in in the above IP addresses and port numbers 
and see if you get 3 unique results?

James Pearson