On Mon, 23 May 2011, Mag Gam wrote:
I would like to confirm Matt's claim. I too experienced larger latencies with Centos 5.x compared to 4.x. My application is very network sensitive and its easy to prove using lat_tcp.
Russ, I am curious about identifying the problem. What tools do you recommend to find where the latency is coming from in the application?
I went through the obvious candidates: system calls (loss of control of when if ever the scheduler decides to let your process run again) polling v select polling is almost always a wrong approach when latency reduction is in play (reading and understanding: man 2 select_tut is time very well spent) choice of implementation language -- the issue here being if one uses a scripting language, one cannot 'see' the time leaks
Doing metrics permits both 'hot spot' analysis, and moves the coding from 'guesstimation' to software engineering. We use graphviz, and gnuplot on the plain text 'CSV-style' timings files to 'see' outliers and hotspots
Knuth's admonition about premature optimization applies here of course
A sensible process might be: Make it work correctly, THEN make it fast
Some people add a precursor step of: make it compile but this seems to me a less efficient process than simply proceeding up with a clean design at the start, and the expedient of 'stubbing' out unimplemented portions. Then replace the stubs with 'correctly' funcitoning refactorings (... I just did this with part of my build tools, writing a meta-code outline of what I wanted, and then implementing the metacode)
The C++ code of the 'trading-shim' tool (GPLv3+) was produced in just this fashion over the last few years, and compared to all the competitors in its class, outpaces them all in terms of minimal latency .. most of that competition being Java based, or in some other scripting language. The 'shim' runs like a scalded dog ;)
-- Russ herrold