Hi,
CentOS does not have any equivalent to fedmsg bus [1] at the moment. What is it used for? Services (koji, pkgdb, ...) can send messages about their state and result of tasks they run into the bus to notify other service which can react accordingly. Thus creating a modular infrastructure. E.g. koji send a message about finished build. Agent read the message, trigger a jenkins job and run additional tests on the build. Daily statistics about number of finished /failed builds can be collected. Other application can have their own use cases. Example of bus topology [4]. [5] is another source of some info about fedora infrastructure.
What needs to be done to set up the bus? Basic demo and setup of fedmsg, fedmsg-relay is at [3]. First would be great to setup the bus for koji. Koji runs on CentOS 6. fedpkg package is in epel6 repository [2]. We will be needing fedmsg-hub, fedmsg-relay, koji-hub and koji-plugin parts in the topology at the moment [4]. CentOS koji and Fedora koji are not exacly the same but both APIs are compatible. So maybe take an inspiration from Fedora's koji plugin. In order to set up bus and clients, endpoints has to be set. Setting of endpoints is discussed at [6]. Context of endpoints is described at [3]. Configuration files are shared and must be the same on all machines using the bus. Bus messages can be saved into database [7].
What we can start with? 1) create an instance of centos/fedora for fedmsg to run 2) create/reuse koji plugin (where it can be found?) 3) set/implement koji plugin to send messages for fedmsg
What other application do you have in centos that could use fedmsg?
Once there is a running msg bus, instances set, etc., it will be easier to debug and test.
Ralph, Pierre, what would be the optimal distribution of instances for fedmsg? Is one instance of centos enough? What would be the best first step to set up fedmsg? Where can I found koji-plugin, resp. its implementation?
This is only a discussion at the moment. Any other ideas are appreciated. Other issues and problems will arise while configuring the bus and koji.
[1] https://github.com/fedora-infra/fedmsg/ [2] http://koji.fedoraproject.org/koji/buildinfo?buildID=630367 [3] http://fedmsg.readthedocs.org/en/latest/deployment/ [4] http://fedmsg.readthedocs.org/en/latest/topology/ [5] https://fedoraproject.org/wiki/Messaging_SIG [6] http://fedmsg.readthedocs.org/en/latest/config/#term-endpoints [7] http://fedmsg.readthedocs.org/en/latest/deployment/#store-all-messages
Kind Regards Jan
On Mon, Jun 01, 2015 at 03:25:50PM +0200, Jan Chaloupka wrote:
Hi,
CentOS does not have any equivalent to fedmsg bus [1] at the moment. What is it used for? Services (koji, pkgdb, ...) can send messages about their state and result of tasks they run into the bus to notify other service which can react accordingly. Thus creating a modular infrastructure. E.g. koji send a message about finished build. Agent read the message, trigger a jenkins job and run additional tests on the build. Daily statistics about number of finished /failed builds can be collected. Other application can have their own use cases. Example of bus topology [4]. [5] is another source of some info about fedora infrastructure.
What needs to be done to set up the bus? Basic demo and setup of fedmsg, fedmsg-relay is at [3]. First would be great to setup the bus for koji. Koji runs on CentOS 6. fedpkg package is in epel6 repository [2]. We will be needing fedmsg-hub, fedmsg-relay, koji-hub and koji-plugin parts in the topology at the moment [4]. CentOS koji and Fedora koji are not exacly the same but both APIs are compatible. So maybe take an inspiration from Fedora's koji plugin. In order to set up bus and clients, endpoints has to be set. Setting of endpoints is discussed at [6]. Context of endpoints is described at [3]. Configuration files are shared and must be the same on all machines using the bus. Bus messages can be saved into database [7].
What we can start with?
- create an instance of centos/fedora for fedmsg to run
For now, you may want to create one host on which to run various helper services. Having an instance of the fedmsg-relay daemon is nice for debugging and having an instance of the fedmsg-irc daemon is nice to have some idea of what is happening on the bus before you start eventually doing useful things with the data. See the #fedora-fedmsg channel on freenode for an example of this.
There are more helper services to stand up later if this proves useful: 'datanommer' for storing the history, 'fedmsg-gateway' for broadcasting events to end users, and others.
- create/reuse koji plugin (where it can be found?)
You can take it from Fedora Infrastructure's ansible repo. There are other useful config bits that can be found in there. http://infrastructure.fedoraproject.org/infra/ansible/roles/koji_hub/files/f...
- set/implement koji plugin to send messages for fedmsg
What other application do you have in centos that could use fedmsg?
Once there is a running msg bus, instances set, etc., it will be easier to debug and test.
Ralph, Pierre, what would be the optimal distribution of instances for fedmsg? Is one instance of centos enough? What would be the best first step to set up fedmsg? Where can I found koji-plugin, resp. its implementation?
This is only a discussion at the moment. Any other ideas are appreciated. Other issues and problems will arise while configuring the bus and koji.
I'd be glad to help work on setting this up if it is desirable. Is there an irc channel where centos admins lurk?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Jun 01 14:42, Ralph Bean wrote:
On Mon, Jun 01, 2015 at 03:25:50PM +0200, Jan Chaloupka wrote:
Hi,
CentOS does not have any equivalent to fedmsg bus [1] at the moment. What is it used for? Services (koji, pkgdb, ...) can send messages about their state and result of tasks they run into the bus to notify other service which can react accordingly. Thus creating a modular infrastructure. E.g. koji send a message about finished build. Agent read the message, trigger a jenkins job and run additional tests on the build. Daily statistics about number of finished /failed builds can be collected. Other application can have their own use cases. Example of bus topology [4]. [5] is another source of some info about fedora infrastructure.
What needs to be done to set up the bus? Basic demo and setup of fedmsg, fedmsg-relay is at [3]. First would be great to setup the bus for koji. Koji runs on CentOS 6. fedpkg package is in epel6 repository [2]. We will be needing fedmsg-hub, fedmsg-relay, koji-hub and koji-plugin parts in the topology at the moment [4]. CentOS koji and Fedora koji are not exacly the same but both APIs are compatible. So maybe take an inspiration from Fedora's koji plugin. In order to set up bus and clients, endpoints has to be set. Setting of endpoints is discussed at [6]. Context of endpoints is described at [3]. Configuration files are shared and must be the same on all machines using the bus. Bus messages can be saved into database [7].
What we can start with?
- create an instance of centos/fedora for fedmsg to run
For now, you may want to create one host on which to run various helper services. Having an instance of the fedmsg-relay daemon is nice for debugging and having an instance of the fedmsg-irc daemon is nice to have some idea of what is happening on the bus before you start eventually doing useful things with the data. See the #fedora-fedmsg channel on freenode for an example of this.
There are more helper services to stand up later if this proves useful: 'datanommer' for storing the history, 'fedmsg-gateway' for broadcasting events to end users, and others.
- create/reuse koji plugin (where it can be found?)
You can take it from Fedora Infrastructure's ansible repo. There are other useful config bits that can be found in there. http://infrastructure.fedoraproject.org/infra/ansible/roles/koji_hub/files/f...
- set/implement koji plugin to send messages for fedmsg
What other application do you have in centos that could use fedmsg?
Once there is a running msg bus, instances set, etc., it will be easier to debug and test.
Ralph, Pierre, what would be the optimal distribution of instances for fedmsg? Is one instance of centos enough? What would be the best first step to set up fedmsg? Where can I found koji-plugin, resp. its implementation?
This is only a discussion at the moment. Any other ideas are appreciated. Other issues and problems will arise while configuring the bus and koji.
I'd be glad to help work on setting this up if it is desirable. Is there an irc channel where centos admins lurk?
Thanks for the tips!
This would be a great topic for us to discuss in the next CBS/Infra meeting (Monday 08-Jun 14:00 UTC in #centos-devel), I'll put it on the agenda for anyone who can make it (all are welcome). We should definitely keep this thread going, and I'll try to get a summary posted here after the meeting.
To your question: CentOS admins usually hang out in #centos-devel, feel free to drop in any time.
- --Brian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
at a quick glance fedmsg looks a lot like general message passing services like rabbitmq, zeromq etc.
what was the intention of reinventing the wheel yet another time?
please enlighten me if I'm wrong and fedmsg does something which general message bus systems aren't capable of.
kind regards
Sven
Hi Sven,
On 06/08/2015 06:17 PM, Sven Kieske wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
at a quick glance fedmsg looks a lot like general message passing services like rabbitmq, zeromq etc.
what was the intention of reinventing the wheel yet another time?
please enlighten me if I'm wrong and fedmsg does something which general message bus systems aren't capable of.
fedmsg is a proposition. I am not aware of any other message bus that could suit centos's infrastructure well nor centos's infrastructure itself. The main reason for choosing fedmsg bus is it allows me to listen to it for finished builds in koji and automatically trigger CI jobs in jenkins. If CentOS's guys come with different solution how to handle CI testing for kubernetes, docker and other builds in CentOS, there does not have to be any fedmsg bus.
Regards Jan
kind regards
Sven -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iQGcBAEBCAAGBQJVdcAnAAoJEAq0kGAWDrqlCA4MALMypyITT+ig8U1gIePkJiVn MaNhsKX3HmrbfPIhwdRjgoMNLTCAcbif7vUtL7bdUSEZIVpKALxMaXiL9V0GbYLf qFCmN25plU2L8h8GBS7bjuvVxUJ6lpeUeYgheiNFi9DVMK2/EehXBn3IergJomd2 iVw3ih+X/16Ak9oiE7rfwxt/37Mv28iJdOH0aA6gcQyxbye7Np3AE7HPNVF8dclD v+Tn6+VQFzzaCwcUnGVgwu2ulzMXvLa9ZQGdat6JDAsRi2tY2CyNTydUXduSoISE Ea9gOtX0883LrXclpVN06HmLrEgl6SJOO+iy6LCS8XsnCgh67NBqfH3UlnKlD1PV Yzw4xU+x1IjpAVUmG/ROLw+ofPhdPSRbBieaR4K2NLdmmKCH3XG5oKl5DWY38luN tqwWhwGAffDIG7BaQHOPlkzCHX3AqC8TA554AmaUOFb698cujqbEio+NidJr2u6D ncgjJ3UxwauhZAbjBjN5sSgSGpVd6INYjkUlitJ9Bg== =6uj0 -----END PGP SIGNATURE----- _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
On Mon, Jun 08, 2015 at 06:17:43PM +0200, Sven Kieske wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hi,
at a quick glance fedmsg looks a lot like general message passing services like rabbitmq, zeromq etc.
what was the intention of reinventing the wheel yet another time?
fedmsg is based on zeromq, it's a layer around it making using zeromq easier for developers (both on the producer and consumer side). So in that respect, it does not reinvent any wheel :)
Pierre
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 08.06.2015 19:40, Pierre-Yves Chibon wrote:
fedmsg is based on zeromq, it's a layer around it making using zeromq easier for developers (both on the producer and consumer side). So in that respect, it does not reinvent any wheel :)
thanks for the clarification!
will look deeper into it, might be useful for some other stuff too..
kind regards
Sven
-
Brian Stinson -
Jan Chaloupka -
Pierre-Yves Chibon -
Ralph Bean -
Sven Kieske