In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
- I think that the *_secure_forward plugins (which permit using ssl for communication between a fluentd client and a collector) are going to be a necessity for just about everybody, so these should probably be packaged in manner similar to the elasticsearch plugin.
On 08/09/2016 03:41 PM, Lars Kellogg-Stedman wrote:
In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
+1 - as long as the location is configurable. OpenShift typically keeps its pos_file in the log directory (e.g. /var/log) that is mounted from the host.
- I think that the *_secure_forward plugins (which permit using ssl for communication between a fluentd client and a collector) are going to be a necessity for just about everybody, so these should probably be packaged in manner similar to the elasticsearch plugin.
+1 - gem2rpm for starters. But watch out because gem2rpm will sometimes generate too many Requires and BuildRequires which aren't really needed, and will miss some Requires and BuildRequires, and see https://fedoraproject.org/wiki/Packaging:Ruby?rd=Packaging/Ruby
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Wed, Aug 10, 2016 at 07:27:45AM -0600, Rich Megginson wrote:
On 08/09/2016 03:41 PM, Lars Kellogg-Stedman wrote:
In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
+1 - as long as the location is configurable. OpenShift typically keeps its pos_file in the log directory (e.g. /var/log) that is mounted from the host.
- I think that the *_secure_forward plugins (which permit using ssl for communication between a fluentd client and a collector) are going to be a necessity for just about everybody, so these should probably be packaged in manner similar to the elasticsearch plugin.
+1 - gem2rpm for starters. But watch out because gem2rpm will sometimes generate too many Requires and BuildRequires which aren't really needed, and will miss some Requires and BuildRequires, and see https://fedoraproject.org/wiki/Packaging:Ruby?rd=Packaging/Ruby
I'd be happy to see patches/pull requests and to build new packages.
Matthias
On Wed, Aug 10, 2016 at 07:27:45AM -0600, Rich Megginson wrote:
On 08/09/2016 03:41 PM, Lars Kellogg-Stedman wrote:
In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
+1 - as long as the location is configurable. OpenShift typically keeps its pos_file in the log directory (e.g. /var/log) that is mounted from the host.
I don't know about configureable. I mean, I am proposing that the package drops a file into /etc/tmpfiles.d, which would make it a system-wide default, although of course individual <source> blocks could use whatever path they wish.
As the package is distributed, with fluentd running as an unprivileged user, it's not going to have write access to most log directories, so putting the pos files there won't work in most cases without changes to directory permissions.
If everyone is colocating pos files with log files, this proposal doesn't seem all that useful. For now, I'm probably going to tackle this in a puppet module that is consuming fluentd, rather than the package itself.
On 08/11/2016 07:36 AM, Lars Kellogg-Stedman wrote:
On Wed, Aug 10, 2016 at 07:27:45AM -0600, Rich Megginson wrote:
On 08/09/2016 03:41 PM, Lars Kellogg-Stedman wrote:
In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
+1 - as long as the location is configurable. OpenShift typically keeps its pos_file in the log directory (e.g. /var/log) that is mounted from the host.
I don't know about configureable. I mean, I am proposing that the package drops a file into /etc/tmpfiles.d, which would make it a system-wide default, although of course individual <source> blocks could use whatever path they wish.
As the package is distributed, with fluentd running as an unprivileged user, it's not going to have write access to most log directories, so putting the pos files there won't work in most cases without changes to directory permissions.
If everyone is colocating pos files with log files, this proposal doesn't seem all that useful. For now, I'm probably going to tackle this in a puppet module that is consuming fluentd, rather than the package itself.
OpenShift runs fluentd in a privileged pod (for now - soon to be a hostmount-anyuid once the selinux issues are fixed) and bind mounts /var/log, /var/lib/docker (for docker --log-driver=json-file), and /run/log (for docker --log-driver=journald) from the host into the pod/container. It has to keep the pos file outside of the container/pod in the host file system, typically /var/log/*.pos. As long as that is possible with the proposed opstools fluentd packaging, that's fine.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Aug 9, 2016 at 5:41 PM, Lars Kellogg-Stedman lars@redhat.com wrote:
In working with the fluentd packaging from the opstools sig, I think we need to make at least two changes:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
Is /run/fluentd directory persistent across reboots of the machine? Wouldn't the nature of a pos file require that to be useful?
See https://github.com/fluent/fluentd/issues/879 for more information around locating a pos file with a log file in a directory that does not exist.
- I think that the *_secure_forward plugins (which permit using ssl for communication between a fluentd client and a collector) are going to be a necessity for just about everybody, so these should probably be packaged in manner similar to the elasticsearch plugin.
+1
-peter
-- Lars Kellogg-Stedman lars@redhat.com | larsks @ {freenode,twitter,github} Cloud Engineering / OpenStack | http://blog.oddbit.com/
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Wed, Aug 17, 2016 at 08:45:42AM -0400, Peter Portante wrote:
- For type=tail sources, fluentd really wants a pos_file parameter to track the file position. It would be helpful if we created /run/fluentd via /etc/tmpfiles.d as a common location for storing these files.
Is /run/fluentd directory persistent across reboots of the machine? Wouldn't the nature of a pos file require that to be useful?
Yeah, I've settled on /var/cache/fluentd for what I'm doing, rather than using an ephemeral directory.
-
Lars Kellogg-Stedman -
Matthias Runge -
Peter Portante -
Rich Megginson