We need to use the ruby24 SCL.
That means ovirt and rdo will have a dependency on ruby24 SCL, which also means the fluentd systemd service will need to know how to run the service using the scl, unless ovirt and rdo are ready to run fluentd containerized, in which case it is as simple as using the centos ruby24 scl base image, which is what openshift is doing: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/D...
ruby24 will be needed at build time for some packages as well as at runtime.
https://www.centos.org/minutes/2017/December/centos-devel.2017-12-13-18.04.l...
On 13/12/17 19:36, Rich Megginson wrote:
We need to use the ruby24 SCL.
That means ovirt and rdo will have a dependency on ruby24 SCL, which also means the fluentd systemd service will need to know how to run the service using the scl, unless ovirt and rdo are ready to run fluentd containerized, in which case it is as simple as using the centos ruby24 scl base image, which is what openshift is doing: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/D...
ruby24 will be needed at build time for some packages as well as at runtime.
https://www.centos.org/minutes/2017/December/centos-devel.2017-12-13-18.04.l...
In the OpenStack world, containers are used from kolla (or very near to that). [1]
The container in kolla itself pull in CentOS-base, centos-opstools, and RDO dependencies.
I'd think we'll need another package (and probably all dependencies) as scl versions for fluentd.
Matthias
[1] https://github.com/openstack/kolla/blob/master/docker/fluentd/Dockerfile.j2
On 12/14/2017 07:02 AM, Matthias Runge wrote:
On 13/12/17 19:36, Rich Megginson wrote:
We need to use the ruby24 SCL.
That means ovirt and rdo will have a dependency on ruby24 SCL, which also means the fluentd systemd service will need to know how to run the service using the scl, unless ovirt and rdo are ready to run fluentd containerized, in which case it is as simple as using the centos ruby24 scl base image, which is what openshift is doing: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/D...
ruby24 will be needed at build time for some packages as well as at runtime.
https://www.centos.org/minutes/2017/December/centos-devel.2017-12-13-18.04.l...
In the OpenStack world, containers are used from kolla (or very near to that). [1]
The container in kolla itself pull in CentOS-base, centos-opstools, and RDO dependencies.
I'd think we'll need another package (and probably all dependencies) as scl versions for fluentd.
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
Matthias
[1] https://github.com/openstack/kolla/blob/master/docker/fluentd/Dockerfile.j2 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/14/2017 08:40 AM, Rich Megginson wrote:
On 12/14/2017 07:02 AM, Matthias Runge wrote:
On 13/12/17 19:36, Rich Megginson wrote:
We need to use the ruby24 SCL.
That means ovirt and rdo will have a dependency on ruby24 SCL, which also means the fluentd systemd service will need to know how to run the service using the scl, unless ovirt and rdo are ready to run fluentd containerized, in which case it is as simple as using the centos ruby24 scl base image, which is what openshift is doing: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/D...
ruby24 will be needed at build time for some packages as well as at runtime.
https://www.centos.org/minutes/2017/December/centos-devel.2017-12-13-18.04.l...
In the OpenStack world, containers are used from kolla (or very near to that). [1]
The container in kolla itself pull in CentOS-base, centos-opstools, and RDO dependencies.
I'd think we'll need another package (and probably all dependencies) as scl versions for fluentd.
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
+ exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd --no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally - for each gem in /usr/share/gems rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
Matthias
[1] https://github.com/openstack/kolla/blob/master/docker/fluentd/Dockerfile.j2
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
Matthias
On 12/18/2017 02:23 AM, Matthias Runge wrote:
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
I don't really want to do that.
I think the best course of action is to convert the spec files so that they can work both with and without SCL (there are instructions on the SCL package page, and lots of rpm macros to use), then have ruby 2.0 and ruby24 scl versions of all of our packages. There may even be scripts which can convert spec files to be SCL-ized.
I wonder if we can add ruby24 as a "platform" e.g. have a single spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and fluentd-1.0.el7ruby24.x86_64.rpm ?
Matthias
On 12/18/2017 04:04 PM, Rich Megginson wrote:
On 12/18/2017 02:23 AM, Matthias Runge wrote:
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
I don't really want to do that.
I think the best course of action is to convert the spec files so that they can work both with and without SCL (there are instructions on the SCL package page, and lots of rpm macros to use), then have ruby 2.0 and ruby24 scl versions of all of our packages. There may even be scripts which can convert spec files to be SCL-ized.
We will also need a version of gem2rpm that can generate the correct SCL-ized spec file.
I wonder if we can add ruby24 as a "platform" e.g. have a single spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and fluentd-1.0.el7ruby24.x86_64.rpm ?
Matthias
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/18/2017 04:18 PM, Rich Megginson wrote:
On 12/18/2017 04:04 PM, Rich Megginson wrote:
On 12/18/2017 02:23 AM, Matthias Runge wrote:
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
I don't really want to do that.
I think the best course of action is to convert the spec files so that they can work both with and without SCL (there are instructions on the SCL package page, and lots of rpm macros to use), then have ruby 2.0 and ruby24 scl versions of all of our packages. There may even be scripts which can convert spec files to be SCL-ized.
We will also need a version of gem2rpm that can generate the correct SCL-ized spec file.
One more thing - perhaps now would be a good time to reconsider the use of fluentd as our log collection agent, and focus some attention on rsyslog, file beat, and fluent-bit?
A quick pros/cons:
Rsyslog - Pros * Available in Fedora/EL already (helps with supportability) * Multi-threaded (should help scalability) * Experience with performance has been very positive when compared to fluentd - Cons * C coding - high barrier to entry * upstream community seems not as active as others * No client cert auth support for elasticsearch * No support for http input (e.g. for collectd)
Fluent-bit - Pros * Written in C, so should be fast - but harder to write plugins * K8s meta plugin * Elasticsearch plugin - don’t know about client cert auth, backpressure * Uses secure_forward protocol so easy to use with existing fluentd - Cons * Not shipped or packaged in RH product yet * Cannot use existing fluentd configuration, nor any existing fluentd ruby plugins
Filebeat - Pros * Supported by Elastic.co - large community https://www.elastic.co/guide/en/beats/libbeat/5.6/community-beats.html * Written in Go - best case, performance of C with ease of ruby plugin development - Cons * Not shipped or packaged in RH product yet * Not sure about client cert auth for elasticsearch - not sure about backpressure handling either * No kubernetes metadata (although there was recently an announcement about this) * No http input (push - https://github.com/christiangalsterer/httpbeat will poll http endpoints, not suitable for rhv/collectd use case) * No support for per-namespace log throttling - might be able to configure filebeat to do this
I wonder if we can add ruby24 as a "platform" e.g. have a single spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and fluentd-1.0.el7ruby24.x86_64.rpm ?
Matthias
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
In addition: fluentd 0.12 is now in maintenance phase: https://www.fluentd.org/blog/fluentd-v1.0
"Fluentd v0.12 is maintenance phase
We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development."
On 12/18/2017 04:30 PM, Rich Megginson wrote:
On 12/18/2017 04:18 PM, Rich Megginson wrote:
On 12/18/2017 04:04 PM, Rich Megginson wrote:
On 12/18/2017 02:23 AM, Matthias Runge wrote:
On 16/12/17 05:51, Rich Megginson wrote:
So a fluentd-ruby24-scl package, a fluent-plugin-add-ruby24-scl package, etc. etc.? That's going to be a non-starter.
It is the only way. You cannot run ruby24 and load packages built for ruby 2.0 - running fluentd on a system which has rh-ruby24 installed:
- exec /opt/rh/rh-ruby24/root/usr/bin/ruby /usr/bin/fluentd
--no-supervisor -vv /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require': incompatible library version - /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so (LoadError) from /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in
`require' from /usr/share/gems/gems/yajl-ruby-1.3.1/lib/yajl.rb:1:in `<top (required)>'
The gems in /usr/share/gems are installed via RPM and built with ruby 2.0.
It might be possible to perform this conversion at runtime:
- install fluentd from regular ruby 2.0 rpms normally
- for each gem in /usr/share/gems
rebuild using ruby24 into /opt/rh/rh-ruby24/root/usr/share
but this is just piling hacks on top of more hacks
I think we're going to need ruby24 branches for all of our common, collectd, and fluentd packages, and we're going to need some sort of mass conversion of all of those specs so that we can use them both with and without scl.
We could also go that route and build ruby 2.4 in opstools (+ rebuilding all ruby-packages).
Not sure if we really want to do that.
I don't really want to do that.
I think the best course of action is to convert the spec files so that they can work both with and without SCL (there are instructions on the SCL package page, and lots of rpm macros to use), then have ruby 2.0 and ruby24 scl versions of all of our packages. There may even be scripts which can convert spec files to be SCL-ized.
We will also need a version of gem2rpm that can generate the correct SCL-ized spec file.
One more thing - perhaps now would be a good time to reconsider the use of fluentd as our log collection agent, and focus some attention on rsyslog, file beat, and fluent-bit?
A quick pros/cons:
Rsyslog
- Pros
- Available in Fedora/EL already (helps with supportability)
- Multi-threaded (should help scalability)
- Experience with performance has been very positive when compared to
fluentd
- Cons
- C coding - high barrier to entry
- upstream community seems not as active as others
- No client cert auth support for elasticsearch
- No support for http input (e.g. for collectd)
Fluent-bit
- Pros
- Written in C, so should be fast - but harder to write plugins
- K8s meta plugin
- Elasticsearch plugin - don’t know about client cert auth, backpressure
- Uses secure_forward protocol so easy to use with existing fluentd
- Cons
- Not shipped or packaged in RH product yet
- Cannot use existing fluentd configuration, nor any existing fluentd
ruby plugins
Filebeat
- Pros
- Supported by Elastic.co - large community
https://www.elastic.co/guide/en/beats/libbeat/5.6/community-beats.html
- Written in Go - best case, performance of C with ease of ruby plugin
development
- Cons
- Not shipped or packaged in RH product yet
- Not sure about client cert auth for elasticsearch - not sure about
backpressure handling either
- No kubernetes metadata (although there was recently an announcement
about this)
- No http input (push - https://github.com/christiangalsterer/httpbeat
will poll http endpoints, not suitable for rhv/collectd use case)
- No support for per-namespace log throttling - might be able to
configure filebeat to do this
I wonder if we can add ruby24 as a "platform" e.g. have a single spec file, and build e.g. fluentd-1.0.el7.x86_64.rpm and fluentd-1.0.el7ruby24.x86_64.rpm ?
Matthias
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 20/12/17 22:03, Rich Megginson wrote:
In addition: fluentd 0.12 is now in maintenance phase: https://www.fluentd.org/blog/fluentd-v1.0
"Fluentd v0.12 is maintenance phase
We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development."
From an enterprise distribution perspective, that's not really bad, no?
At least, they are providing security fixes;
The opposite is marked by other projects, where you're supposed to stay on latest release, which brings new features, requires changes in configs and sometimes even requires data migrations.
From that perspective, I'd prefer to stay on something, which does not
break my install.
Matthias
On 12/20/2017 11:48 PM, Matthias Runge wrote:
On 20/12/17 22:03, Rich Megginson wrote:
In addition: fluentd 0.12 is now in maintenance phase: https://www.fluentd.org/blog/fluentd-v1.0
"Fluentd v0.12 is maintenance phase
We continue to update fluentd v0.12 but the main changes are backport and security fix. We focus on v1.0 development."
From an enterprise distribution perspective, that's not really bad, no?
It is not good. And it makes me wonder how many other ruby gems that fluentd depends on are going to drop support for older versions that work on ruby 2.0, and focus only on ruby 2.4 support . . .
At least, they are providing security fixes;
The opposite is marked by other projects, where you're supposed to stay on latest release, which brings new features, requires changes in configs and sometimes even requires data migrations.
From that perspective, I'd prefer to stay on something, which does not break my install.
I'm not sure I understand - what would that be?
Matthias
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-
Matthias Runge -
Rich Megginson