[CentOS-devel] [opstools] How to support fluentd with ruby24?

On 12/18/2017 04:18 PM, Rich Megginson wrote:
> On 12/18/2017 04:04 PM, Rich Megginson wrote:
>> On 12/18/2017 02:23 AM, Matthias Runge wrote:
>>> On 16/12/17 05:51, Rich Megginson wrote:
>>>
>>>>> So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl
>>>>> package, etc. etc.?  That's going to be a non-starter.
>>>> It is the only way.  You cannot run ruby24 and load packages built for
>>>> ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
>>>>
>>>> + exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
>>>> --no-supervisor -vv
>>>> /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in 
>>>>
>>>> `require': incompatible library version -
>>>> /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError)
>>>>          from
>>>> /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in 
>>>>
>>>> `require'
>>>>          from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in 
>>>> `<top
>>>> (required)>'
>>>>
>>>> The gems in /usr/share/gems are installed via RPM and built with 
>>>> ruby 2.0.
>>>>
>>>> It might be possible to perform this conversion at runtime:
>>>>
>>>> - install fluentd from regular ruby 2.0 rpms normally
>>>> - for each gem in /usr/share/gems
>>>>    rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
>>>>
>>>> but this is just piling hacks on top of more hacks
>>>>
>>>> I think we're going to need ruby24 branches for all of our common,
>>>> collectd, and fluentd packages, and we're going to need some sort of
>>>> mass conversion of all of those specs so that we can use them both 
>>>> with
>>>> and without scl.
>>>>
>>> We could also go that route and build ruby 2.4 in opstools (+ 
>>> rebuilding
>>> all ruby-packages).
>>>
>>> Not sure if we really want to do that.
>>
>> I don't really want to do that.
>>
>> I think the best course of action is to convert the spec files so 
>> that they can work both with and without SCL (there are instructions 
>> on the SCL package page, and lots of rpm macros to use), then have 
>> ruby 2.0 and ruby24 scl versions of all of our packages.  There may 
>> even be scripts which can convert spec files to be SCL-ized.
>>
>
> We will also need a version of gem2rpm that can generate the correct 
> SCL-ized spec file.

One more thing - perhaps now would be a good time to reconsider the use 
of fluentd as our log collection agent, and focus some attention on 
rsyslog, file beat, and fluent-bit?

A quick pros/cons:

Rsyslog
- Pros
* Available in Fedora/EL already (helps with supportability)
* Multi-threaded (should help scalability)
* Experience with performance has been very positive when compared to 
fluentd
- Cons
* C coding - high barrier to entry
* upstream community seems not as active as others
* No client cert auth support for elasticsearch
* No support for http input (e.g. for collectd)

Fluent-bit
- Pros
* Written in C, so should be fast - but harder to write plugins
* K8s meta plugin
* Elasticsearch plugin - don’t know about client cert auth, backpressure
* Uses secure_forward protocol so easy to use with existing fluentd
- Cons
* Not shipped or packaged in RH product yet
* Cannot use existing fluentd configuration, nor any existing fluentd 
ruby plugins

Filebeat
- Pros
* Supported by Elastic.co - large community 
https://www.elastic.co/guide/en/beats/libbeat/5.6/community-beats.html
* Written in Go - best case, performance of C with ease of ruby plugin 
development
- Cons
* Not shipped or packaged in RH product yet
* Not sure about client cert auth for elasticsearch - not sure about 
backpressure handling either
* No kubernetes metadata (although there was recently an announcement 
about this)
* No http input (push - https://github.com/christiangalsterer/httpbeat 
will poll http endpoints, not suitable for rhv/collectd use case)
* No support for per-namespace log throttling - might be able to 
configure filebeat to do this

>
>> I wonder if we can add ruby24 as a "platform" e.g. have a single spec 
>> file, and build e.g. fluentd-1.0.el7.x86_64.rpm and 
>> fluentd-1.0.el7ruby24.x86_64.rpm ?
>>
>>>
>>> Matthias
>>
>>
>> _______________________________________________
>> CentOS-devel mailing list
>> CentOS-devel at centos.org
>> https://lists.centos.org/mailman/listinfo/centos-devel
>
>
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel