[CentOS] scheduling differences between CentOS 4 and CentOS 5?

[R P Herrold]

On Mon, 23 May 2011, Mag Gam wrote:

> I would like to confirm Matt's claim. I too experienced larger
> latencies with Centos 5.x compared to 4.x. My application is very
> network sensitive and its easy to prove using lat_tcp.

> Russ,
> I am curious about identifying the problem. What tools do you
> recommend to find where the latency is coming from in the application?

I went through the obvious candidates:
 	system calls
 		(loss of control of when if ever the
 		scheduler decides to let your process run again)
 	polling v select
 		polling is almost always a wrong approach when
 		latency reduction is in play
 		(reading and understanding: man 2 select_tut
 		 is time very well spent)
 	choice of implementation language -- the issue here
 		being if one uses a scripting language, one cannot
 		'see' the time leaks

Doing metrics permits both 'hot spot' analysis, and moves the 
coding from 'guesstimation' to software engineering.  We use 
graphviz, and gnuplot on the plain text 'CSV-style' timings 
files to 'see' outliers and hotspots

Knuth's admonition about premature optimization applies here 
of course

A sensible process might be:
 	Make it work correctly, THEN make it fast

Some people add a precursor step of:
 	make it compile
but this seems to me a less efficient process than simply 
proceeding up with a clean design at the start, and the 
expedient of 'stubbing' out unimplemented portions. Then 
replace the stubs with 'correctly' funcitoning refactorings 
(... I just did this with part of my build tools, writing a 
meta-code outline of what I wanted, and then implementing the 
metacode)

The C++ code of the 'trading-shim' tool (GPLv3+) was produced 
in just this fashion over the last few years, and compared to 
all the competitors in its class, outpaces them all in terms 
of minimal latency .. most of that competition being Java 
based, or in some other scripting language.  The 'shim' runs 
like a scalded dog  ;)

-- Russ herrold