----------
You might want to think about:
syslog-ng/rsyslog remote logging + syslog-ng/rsyslog master log receiver + splunk
If you find that log messages are getting lost or you need to guarantee that messages arrive you can also consider RELP (supported by rsyslog and possibly by syslog-ng).
I actually have experience with writing these types of tools in perl, and found it is not really that hard to do if you have good in-house devops talent at hand. Management and retention of the all that data is the biggest challenge.