[CentOS-devel] [opstools] How to support fluentd with ruby24?

In addition: fluentd 0.12 is now in maintenance phase: 
https://www.fluentd.org/blog/fluentd-v1.0

"Fluentd v0.12 is maintenance phase

We continue to update fluentd v0.12 but the main changes are backport 
and security fix. We focus on v1.0 development."


On 12/18/2017 04:30 PM, Rich Megginson wrote:
> On 12/18/2017 04:18 PM, Rich Megginson wrote:
>> On 12/18/2017 04:04 PM, Rich Megginson wrote:
>>> On 12/18/2017 02:23 AM, Matthias Runge wrote:
>>>> On 16/12/17 05:51, Rich Megginson wrote:
>>>>
>>>>>> So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl
>>>>>> package, etc. etc.?  That's going to be a non-starter.
>>>>> It is the only way.  You cannot run ruby24 and load packages built 
>>>>> for
>>>>> ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
>>>>>
>>>>> + exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
>>>>> --no-supervisor -vv
>>>>> /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in 
>>>>>
>>>>> `require': incompatible library version -
>>>>> /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError)
>>>>>          from
>>>>> /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in 
>>>>>
>>>>> `require'
>>>>>          from 
>>>>> /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top
>>>>> (required)>'
>>>>>
>>>>> The gems in /usr/share/gems are installed via RPM and built with 
>>>>> ruby 2.0.
>>>>>
>>>>> It might be possible to perform this conversion at runtime:
>>>>>
>>>>> - install fluentd from regular ruby 2.0 rpms normally
>>>>> - for each gem in /usr/share/gems
>>>>>    rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
>>>>>
>>>>> but this is just piling hacks on top of more hacks
>>>>>
>>>>> I think we're going to need ruby24 branches for all of our common,
>>>>> collectd, and fluentd packages, and we're going to need some sort of
>>>>> mass conversion of all of those specs so that we can use them both 
>>>>> with
>>>>> and without scl.
>>>>>
>>>> We could also go that route and build ruby 2.4 in opstools (+ 
>>>> rebuilding
>>>> all ruby-packages).
>>>>
>>>> Not sure if we really want to do that.
>>>
>>> I don't really want to do that.
>>>
>>> I think the best course of action is to convert the spec files so 
>>> that they can work both with and without SCL (there are instructions 
>>> on the SCL package page, and lots of rpm macros to use), then have 
>>> ruby 2.0 and ruby24 scl versions of all of our packages.  There may 
>>> even be scripts which can convert spec files to be SCL-ized.
>>>
>>
>> We will also need a version of gem2rpm that can generate the correct 
>> SCL-ized spec file.
>
> One more thing - perhaps now would be a good time to reconsider the 
> use of fluentd as our log collection agent, and focus some attention 
> on rsyslog, file beat, and fluent-bit?
>
> A quick pros/cons:
>
> Rsyslog
> - Pros
> * Available in Fedora/EL already (helps with supportability)
> * Multi-threaded (should help scalability)
> * Experience with performance has been very positive when compared to 
> fluentd
> - Cons
> * C coding - high barrier to entry
> * upstream community seems not as active as others
> * No client cert auth support for elasticsearch
> * No support for http input (e.g. for collectd)
>
> Fluent-bit
> - Pros
> * Written in C, so should be fast - but harder to write plugins
> * K8s meta plugin
> * Elasticsearch plugin - don’t know about client cert auth, backpressure
> * Uses secure_forward protocol so easy to use with existing fluentd
> - Cons
> * Not shipped or packaged in RH product yet
> * Cannot use existing fluentd configuration, nor any existing fluentd 
> ruby plugins
>
> Filebeat
> - Pros
> * Supported by Elastic.co - large community 
> https://www.elastic.co/guide/en/beats/libbeat/5.6/community-beats.html
> * Written in Go - best case, performance of C with ease of ruby plugin 
> development
> - Cons
> * Not shipped or packaged in RH product yet
> * Not sure about client cert auth for elasticsearch - not sure about 
> backpressure handling either
> * No kubernetes metadata (although there was recently an announcement 
> about this)
> * No http input (push - https://github.com/christiangalsterer/httpbeat 
> will poll http endpoints, not suitable for rhv/collectd use case)
> * No support for per-namespace log throttling - might be able to 
> configure filebeat to do this
>
>>
>>> I wonder if we can add ruby24 as a "platform" e.g. have a single 
>>> spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and 
>>> fluentd-1.0.el7ruby24.x86_64.rpm ?
>>>
>>>>
>>>> Matthias
>>>
>>>
>>> _______________________________________________
>>> CentOS-devel mailing list
>>> CentOS-devel at centos.org
>>> https://lists.centos.org/mailman/listinfo/centos-devel
>>
>>
>> _______________________________________________
>> CentOS-devel mailing list
>> CentOS-devel at centos.org
>> https://lists.centos.org/mailman/listinfo/centos-devel
>
>
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel