In addition: fluentd 0.12 is now in maintenance phase: https://www.fluentd.org/blog/fluentd-v1.0
"Fluentd v0.12 is maintenance phase
We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development."
On 12/18/2017 04:30 PM, Rich Megginson wrote:
On 12/18/2017 04:18 PM, Rich Megginson wrote:
On 12/18/2017 04:04 PM, Rich Megginson wrote:
On 12/18/2017 02:23 AM, Matthias Runge wrote:
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
I don't really want to do that.
I think the best course of action is to convert the spec files so that they can work both with and without SCL (there are instructions on the SCL package page, and lots of rpm macros to use), then have ruby 2.0 and ruby24 scl versions of all of our packages. There may even be scripts which can convert spec files to be SCL-ized.
We will also need a version of gem2rpm that can generate the correct SCL-ized spec file.
One more thing - perhaps now would be a good time to reconsider the use of fluentd as our log collection agent, and focus some attention on rsyslog, file beat, and fluent-bit?
A quick pros/cons:
Rsyslog
- Pros
- Available in Fedora/EL already (helps with supportability)
- Multi-threaded (should help scalability)
- Experience with performance has been very positive when compared to
fluentd
- Cons
- C coding - high barrier to entry
- upstream community seems not as active as others
- No client cert auth support for elasticsearch
- No support for http input (e.g. for collectd)
Fluent-bit
- Pros
- Written in C, so should be fast - but harder to write plugins
- K8s meta plugin
- Elasticsearch plugin - don’t know about client cert auth, backpressure
- Uses secure_forward protocol so easy to use with existing fluentd
- Cons
- Not shipped or packaged in RH product yet
- Cannot use existing fluentd configuration, nor any existing fluentd
ruby plugins
Filebeat
- Pros
- Supported by Elastic.co - large community
https://www.elastic.co/guide/en/beats/libbeat/5.6/community-beats.html
- Written in Go - best case, performance of C with ease of ruby plugin
development
- Cons
- Not shipped or packaged in RH product yet
- Not sure about client cert auth for elasticsearch - not sure about
backpressure handling either
- No kubernetes metadata (although there was recently an announcement
about this)
- No http input (push - https://github.com/christiangalsterer/httpbeat
will poll http endpoints, not suitable for rhv/collectd use case)
- No support for per-namespace log throttling - might be able to
configure filebeat to do this
I wonder if we can add ruby24 as a "platform" e.g. have a single spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and fluentd-1.0.el7ruby24.x86_64.rpm ?
Matthias
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel