I remember once upon a time sending the first email to the Fedora Cloud SIG.
Funny now to find myself sending an email asking about the CentOS Cloud SIG. :)
Would love to find out what's required to get Eucalyptus packages into CentOS, and to build a Eucalyptus spin of CentOS. CentOS is already our platform of choice, all of our dependencies are either in CentOS or EPEL, and we've got an installer that sits as an updates.img on top of CentOS. Want to make sure we know the proper way of integrating into the new community.
--g
Main thing I'm curious about are the common points with other cloud projects.
Reason being, we're looking for SIGs around a broad focus rather than project-specific. So, I know RDO/OpenStack & oVirt are interested, is it possible Eucalyptus has similar need for different kernel or other bits? Or could fit in with that group?
- Karsten -- Karsten Wade | Red Hat OSAS | community.redhat.com | gpg AD0E0C41
Greg DeKoenigsberg greg.dekoenigsberg@gmail.com wrote:
I remember once upon a time sending the first email to the Fedora Cloud SIG.
Funny now to find myself sending an email asking about the CentOS Cloud SIG. :)
Would love to find out what's required to get Eucalyptus packages into CentOS, and to build a Eucalyptus spin of CentOS. CentOS is already our platform of choice, all of our dependencies are either in CentOS or EPEL, and we've got an installer that sits as an updates.img on top of CentOS. Want to make sure we know the proper way of integrating into the new community.
--g
On Wed, Jan 8, 2014 at 5:31 PM, Karsten Wade kwade@redhat.com wrote:
Main thing I'm curious about are the common points with other cloud projects.
Reason being, we're looking for SIGs around a broad focus rather than project-specific. So, I know RDO/OpenStack & oVirt are interested, is it possible Eucalyptus has similar need for different kernel or other bits? Or could fit in with that group?
What need for a different kernel has come up with the RDO/OpenStack & oVirt groups?
In general, I'm very interested in the requirements people have for different kernels, why they need a different kernel, and what version of kernel they believe is suitable. That a general interest on my part, not just specific to a Cloud SIG.
josh
On Wed, Jan 08, 2014 at 08:07:15PM -0500, Josh Boyer wrote:
What need for a different kernel has come up with the RDO/OpenStack
For RDO/OpenStack we needed network namespace support (to support Neutron) that wasn't available in the stock RHEL/CentOS kernel.
On Wed, Jan 8, 2014 at 8:24 PM, Lars Kellogg-Stedman lars@redhat.com wrote:
On Wed, Jan 08, 2014 at 08:07:15PM -0500, Josh Boyer wrote:
What need for a different kernel has come up with the RDO/OpenStack
For RDO/OpenStack we needed network namespace support (to support Neutron) that wasn't available in the stock RHEL/CentOS kernel.
Was that solved with some backporting to the RHEL/CentOS kernel, or did you wind up using an entirely newer kernel version?
josh
On 09.01.2014 12:08, Josh Boyer wrote:
On Wed, Jan 8, 2014 at 8:24 PM, Lars Kellogg-Stedman lars@redhat.com wrote:
On Wed, Jan 08, 2014 at 08:07:15PM -0500, Josh Boyer wrote:
What need for a different kernel has come up with the RDO/OpenStack
For RDO/OpenStack we needed network namespace support (to support Neutron) that wasn't available in the stock RHEL/CentOS kernel.
Was that solved with some backporting to the RHEL/CentOS kernel, or did you wind up using an entirely newer kernel version?
Separate kernels and iproute were/are necessary, they are provided in this repo http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/
AFAIK netns is still unsupported in EL 6.5, iproute can't handle netns for example.
On 01/09/2014 12:56 PM, Nux! wrote:
Separate kernels and iproute were/are necessary, they are provided in this repo http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/
AFAIK netns is still unsupported in EL 6.5, iproute can't handle netns for example.
I havent looked, but do we need a whole kernel for this... cant this just be a kmod ?
On 01/09/2014 12:56 PM, Nux! wrote:
On 09.01.2014 12:08, Josh Boyer wrote:
On Wed, Jan 8, 2014 at 8:24 PM, Lars Kellogg-Stedman lars@redhat.com wrote:
On Wed, Jan 08, 2014 at 08:07:15PM -0500, Josh Boyer wrote:
What need for a different kernel has come up with the RDO/OpenStack
For RDO/OpenStack we needed network namespace support (to support Neutron) that wasn't available in the stock RHEL/CentOS kernel.
Was that solved with some backporting to the RHEL/CentOS kernel, or did you wind up using an entirely newer kernel version?
Also GRE and the newer VXLAN support were added. These were significant additions best managed within a full kernel release. Note since CentOS 6.5 these have been incorporated into the main kernel, but the point still remains that we can benefit from a separate kernel on occasion.
Separate kernels and iproute were/are necessary, they are provided in this repo http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/
AFAIK netns is still unsupported in EL 6.5, iproute can't handle netns for example.
The netns backport is not complete due to kernel ABI constraints. With the older RDO kernel and with the newer CentOS 6.5 kernel you can specify the netns of a link but you can't use 'ip netns'.
The above linked repo is also a good place to browse candidate packages for the cloud sig. Given OpenStack is implemented in python, there are some new and/or backwards incompatible python libraries there. But also there are more general packages like the above discussed kernel for 6.4 based systems and openvswitch packages etc.
Note also that RDO links to the puppetlabs yum repo to incorporate the latest packages from there. There was only a little adjustment required within RDO when puppetlabs updated from puppet 3.3 to 3.4 recently, but this would be another candidate for incorporation within the cloud sig.
thanks, Pádraig.
On Thu, Jan 9, 2014 at 8:49 AM, Pádraig Brady pbrady@redhat.com wrote:
On 01/09/2014 12:56 PM, Nux! wrote:
On 09.01.2014 12:08, Josh Boyer wrote:
On Wed, Jan 8, 2014 at 8:24 PM, Lars Kellogg-Stedman lars@redhat.com wrote:
On Wed, Jan 08, 2014 at 08:07:15PM -0500, Josh Boyer wrote:
What need for a different kernel has come up with the RDO/OpenStack
For RDO/OpenStack we needed network namespace support (to support Neutron) that wasn't available in the stock RHEL/CentOS kernel.
Was that solved with some backporting to the RHEL/CentOS kernel, or did you wind up using an entirely newer kernel version?
Also GRE and the newer VXLAN support were added. These were significant additions best managed within a full kernel release. Note since CentOS 6.5 these have been incorporated into the main kernel, but the point still remains that we can benefit from a separate kernel on occasion.
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
Separate kernels and iproute were/are necessary, they are provided in this repo http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/
AFAIK netns is still unsupported in EL 6.5, iproute can't handle netns for example.
The netns backport is not complete due to kernel ABI constraints. With the older RDO kernel and with the newer CentOS 6.5 kernel you can specify the netns of a link but you can't use 'ip netns'.
Right. I wonder how much kABI is going to play into CentOS variants. Frankly, I hope the answer is "not much".
The above linked repo is also a good place to browse candidate packages for the cloud sig. Given OpenStack is implemented in python, there are some new and/or backwards incompatible python libraries there. But also there are more general packages like the above discussed kernel for 6.4 based systems and openvswitch packages etc.
Good to know, thanks.
josh
On 01/09/2014 02:22 PM, Josh Boyer wrote:
On Thu, Jan 9, 2014 at 8:49 AM, Pádraig Brady pbrady@redhat.com wrote:
Also GRE and the newer VXLAN support were added. These were significant additions best managed within a full kernel release. Note since CentOS 6.5 these have been incorporated into the main kernel, but the point still remains that we can benefit from a separate kernel on occasion.
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
So far it was backports only.
thanks, Pádraig.
Hi,
On 01/09/2014 03:40 PM, Pádraig Brady wrote:
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
So far it was backports only.
Of course, using a later kernel which already has the features available would make it a lot easier to maintain (but would potentially have a bigger impact.
Cheers, Dave.
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
the xen4centos kernel is mainline based 3.x
- KB
On Thu, Jan 9, 2014 at 5:28 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
the xen4centos kernel is mainline based 3.x
Interesting. I'll have a look. Do you know or have pointers to why they chose to go that route?
josh
On 01/10/2014 12:32 PM, Josh Boyer wrote:
On Thu, Jan 9, 2014 at 5:28 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
the xen4centos kernel is mainline based 3.x
Interesting. I'll have a look. Do you know or have pointers to why they chose to go that route?
We tried, quite a bit, to try and get the xen stuff backported into the 2.6.32-EL6 kernel, but given the patch overlap was something we cant control and lack the time and technical depth in kernel code to maintain that longer term ( this was essentially Johnny and me, doing this over and above most other things ), it was just easier to go down that route.
Secondly, there are quite a few anecdotal pieces on dramatic performance improvements in newer kernels, I know atleast one of the top 5 CDN's ( ie carry > 25Gibt/sec ) that recently switched over from the distro centos kernel to a inhouse 3.10 build for 'network perforamnce'.
In quotes, because repeated questions and pokes in various media have failed to get a reasonable, tangiable, technical response out of them beyond 'network perforamance'.
- KB
On Sat, Jan 11, 2014 at 5:51 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/10/2014 12:32 PM, Josh Boyer wrote:
On Thu, Jan 9, 2014 at 5:28 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
the xen4centos kernel is mainline based 3.x
Interesting. I'll have a look. Do you know or have pointers to why they chose to go that route?
We tried, quite a bit, to try and get the xen stuff backported into the 2.6.32-EL6 kernel, but given the patch overlap was something we cant control and lack the time and technical depth in kernel code to maintain that longer term ( this was essentially Johnny and me, doing this over and above most other things ), it was just easier to go down that route.
OK. Those kinds of reasons are what I would expect to drive a divergence from the disto kernel. I'm curious to see how those kinds of situations work out in terms of bug reports and additional maintenance burden.
Secondly, there are quite a few anecdotal pieces on dramatic performance improvements in newer kernels, I know atleast one of the top 5 CDN's ( ie carry > 25Gibt/sec ) that recently switched over from the distro centos kernel to a inhouse 3.10 build for 'network perforamnce'.
In quotes, because repeated questions and pokes in various media have failed to get a reasonable, tangiable, technical response out of them beyond 'network perforamance'.
Interesting. I suppose it's possible but without actually data there's really no way to tell. It might even be that they modified 3.10 themselves to gain that performance and are reluctant to talk about it?
Thanks for the info. I'll keep watching the lists for SIG proposals with interest.
josh
On 01/13/2014 07:41 PM, Josh Boyer wrote:
We tried, quite a bit, to try and get the xen stuff backported into the 2.6.32-EL6 kernel, but given the patch overlap was something we cant control and lack the time and technical depth in kernel code to maintain that longer term ( this was essentially Johnny and me, doing this over and above most other things ), it was just easier to go down that route.
I'd be very interested to see a list of commits and corresponding benchmarks for consideration into RHEL.
OK. Those kinds of reasons are what I would expect to drive a divergence from the disto kernel. I'm curious to see how those kinds of situations work out in terms of bug reports and additional maintenance burden.
Secondly, there are quite a few anecdotal pieces on dramatic performance improvements in newer kernels, I know atleast one of the top 5 CDN's ( ie carry > 25Gibt/sec ) that recently switched over from the distro centos kernel to a inhouse 3.10 build for 'network perforamnce'.
In quotes, because repeated questions and pokes in various media have failed to get a reasonable, tangiable, technical response out of them beyond 'network perforamance'.
Interesting. I suppose it's possible but without actually data there's really no way to tell. It might even be that they modified 3.10 themselves to gain that performance and are reluctant to talk about it?
I suppose you don't have any additional details.
We did a ton of performance work for RHEL6.5 and are beating upstream performance in various benchmarks while we have a minimal gap in a few benchmarks. Some of them track down to a difference in how memory accounting is performed. That is of course assuming a stock kernel without any bypass technology such as DPDK or SR-IOV.
Just bumping to 3.10 or 3.12 is not going to be an universal performance booster fix.
On 01/13/2014 12:41 PM, Josh Boyer wrote:
On Sat, Jan 11, 2014 at 5:51 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/10/2014 12:32 PM, Josh Boyer wrote:
On Thu, Jan 9, 2014 at 5:28 PM, Karanbir Singh mail-lists@karan.org wrote:
On 01/09/2014 02:22 PM, Josh Boyer wrote:
OK, but it's still a kernel based on the RHEL kernel, just with the changes/additions backported to it. It isn't a bump to e.g. the Linux 3.10 stable release or something. Correct?
the xen4centos kernel is mainline based 3.x
Interesting. I'll have a look. Do you know or have pointers to why they chose to go that route?
We tried, quite a bit, to try and get the xen stuff backported into the 2.6.32-EL6 kernel, but given the patch overlap was something we cant control and lack the time and technical depth in kernel code to maintain that longer term ( this was essentially Johnny and me, doing this over and above most other things ), it was just easier to go down that route.
OK. Those kinds of reasons are what I would expect to drive a divergence from the disto kernel. I'm curious to see how those kinds of situations work out in terms of bug reports and additional maintenance burden.
Secondly, there are quite a few anecdotal pieces on dramatic performance improvements in newer kernels, I know atleast one of the top 5 CDN's ( ie carry > 25Gibt/sec ) that recently switched over from the distro centos kernel to a inhouse 3.10 build for 'network perforamnce'.
In quotes, because repeated questions and pokes in various media have failed to get a reasonable, tangiable, technical response out of them beyond 'network perforamance'.
Interesting. I suppose it's possible but without actually data there's really no way to tell. It might even be that they modified 3.10 themselves to gain that performance and are reluctant to talk about it?
Thanks for the info. I'll keep watching the lists for SIG proposals with interest.
Well, in the case of the Xen kernel, we needed things already rolled into the 3.x tree for Xen4 that we could not backport those with our current skill set. We also wanted something that lasts a long time and gets updates, so we settled on the LTS kernels.
We currently have 3.10.x and no kabi support. That means kmods are a PITA, but we were able to roll in the newest upstream (from xen) blktap modules into both the kernel and our Xen RPMS.
We can likely use this kernel in other SIGs, hopefully as is or as a group modify it so it works for the Xen SIG and the other SIGs too. The goal is maintaining one Kernel, not several. The point of doing the LTS one is that kernel.org is rolling back both new hardware support and security updates for a couple of years on the LTS kernels.
Thanks, Johnny Hughes
On 01/13/2014 06:41 PM, Josh Boyer wrote:
OK. Those kinds of reasons are what I would expect to drive a divergence from the disto kernel. I'm curious to see how those kinds of situations work out in terms of bug reports and additional maintenance burden.
We are lucky in that David Vrabel from the xen/citrix kernel group has been helping maintain that stack. The plan is to rebase and stay with kernel.org's LTS as much as possible, so we dont need to own the entire codebase.
- KB
-
Dave Neary -
Greg DeKoenigsberg -
Johnny Hughes -
Josh Boyer -
Karanbir Singh -
Karsten Wade -
Lars Kellogg-Stedman -
Nux! -
Pádraig Brady -
Thomas Graf