Howdy Folks:
Our discussion of the distinctive elements of CentOS’s Aarch64 variant has gotten me pondering. What does an ‘enterprise’ distro bring to the table that makes it desirable over the alternatives available? (By ‘enterprise distro’, I here mean the stream of data originated by the Upstream Provider, as well as those modified and distributed by all the various downstream projects, including CentOS). To my mind, I see two basic categories that are relevant here:
1) ‘Enterprise’ features, by which I mean things like: supports ‘big iron’ systems, supports fault tolerance and scalability, interoperates well with all of the common ‘non-native’ protocols and features found in a typical large computing environment.
2) ‘Stability’ features, by which I mean: has a well-defined infrastructure for distributing code fixes and security updates, has a long support lifetime, manages the ABI carefully so as to ensure that system updates can be applied with good confidence that they will not break existing locally-installed applications.
These two categories are largely orthogonal. I say ‘largely’ because there is a hidden implicit commonality: reliability. Yet the presence of one need not necessarily imply the presence (or even desirability) of the other. For example, a project that uses an ‘embedded’ board (BeagleBoard, etc) to make small, autonomous ‘widget’ type devices really does not care about about the presence of 1), while 2) can be a highly desirable attribute. Category 1) is essentially an *operational* attribute, desirable in certain use cases but superfluous (or even counterproductive) in others, while category 2) addresses *maintenance* issues, critical for long-term service and support of a system but largely irrelevant to the typical day-to-day operation of applications.
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A good plan today is better than a perfect plan tomorrow. | | -- General George Patton | |----------------------------------------------------------------------|
I think "Enterprise" and "Embedded" are diametrically opposed for a start. I cannot think of a single non-embedded distro that doesn't tick all your requirements under 1) 2) generally means no updates unless there's a major bug or security exploit, and fixes for those only get backported - no major updates. Having said that, such backporting is in itself problematic and often introduces bugs. Personally, I run my own built and packaged LT mainline kernels on all my servers, rather than distro provided ones because I find them less problematic and upstream tends to be much more responsive to bug reports unless you are paying very substantial amounts for a support contract that guarantees you actual fixes within an SLA period. And even then, on enterprise distros like CentOS, updates often break things. For example, I've had to disable Xorg updates and mitigate security vulnerabilities in other ways because latest 1.19 completely fails to start up on my system while 1.17 works fine. Older post-1.17 versions would randomly crash out. So just because a distro is "enterprise", it doesn't mean it's more stable in the sense of likely to crash your systems (servers or workstations) less often. The moment you stray off a _very_ well beaten path (handful of models of highly branded servers that sell by the hundreds of thousands that everything gets extensively tested on), you are going to be on your own.
If you are working on your own embedded project on hardware that only a handful of people use on a daily basis, I would say using an enterprise distribution as a base buys you close to nothing. And I say that as one of the original guys that ported EL6 on armv5tel (See: RedSleeve Linux). My motivation is that I'd rather have at least the upstream security fixes long term back-patched, and deal with the porting myself, but while that buys you something in terms of the security aspect, my experience is that it doesn't really buy you all that much in terms of stability.
Gordan
On Fri, Oct 26, 2018 at 5:19 PM Fred Gleason fredg@paravelsystems.com wrote:
Howdy Folks:
Our discussion of the distinctive elements of CentOS’s Aarch64 variant has gotten me pondering. What does an ‘enterprise’ distro bring to the table that makes it desirable over the alternatives available? (By ‘enterprise distro’, I here mean the stream of data originated by the Upstream Provider, as well as those modified and distributed by all the various downstream projects, including CentOS). To my mind, I see two basic categories that are relevant here:
- ‘Enterprise’ features, by which I mean things like: supports ‘big iron’
systems, supports fault tolerance and scalability, interoperates well with all of the common ‘non-native’ protocols and features found in a typical large computing environment.
- ‘Stability’ features, by which I mean: has a well-defined
infrastructure for distributing code fixes and security updates, has a long support lifetime, manages the ABI carefully so as to ensure that system updates can be applied with good confidence that they will not break existing locally-installed applications.
These two categories are largely orthogonal. I say ‘largely’ because there is a hidden implicit commonality: reliability. Yet the presence of one need not necessarily imply the presence (or even desirability) of the other. For example, a project that uses an ‘embedded’ board (BeagleBoard, etc) to make small, autonomous ‘widget’ type devices really does not care about about the presence of 1), while 2) can be a highly desirable attribute. Category
- is essentially an *operational* attribute, desirable in certain use
cases but superfluous (or even counterproductive) in others, while category 2) addresses *maintenance* issues, critical for long-term service and support of a system but largely irrelevant to the typical day-to-day operation of applications.
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A good plan today is better than a perfect plan tomorrow. | | -- General George Patton | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On 10/26/18 12:19 PM, Fred Gleason wrote:
Howdy Folks:
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
There are a number of issues around embedded systems.
They have to work for 10 - 20 years.
They have to be 'safe' for as long as they are working.
At development time, often the most current components are needed (e.g. openSSL 1.1.1, TLS 1.3) This is because, often only patches are done and things still need to work in 10 years.
I still deal with embedded systems that are 8bit processors with 32KB memory/storage. Those need not apply to this discussion.
I have dealt with vendors that say they now charge extra for only 1GB memory, as their current design is 2GB. And they call this an IoT board....
There are many classes of embedded systems. You look at what is being embedded in home control gateways, they either are cloud service based (great for captive customers and monetizing) or self-sufficient for lots of reasons (privacy for one).
So I am here, because I believe in enterprise code for these systems.
On Fri, Oct 26, 2018 at 5:58 PM Robert Moskowitz rgm@htt-consult.com wrote:
On 10/26/18 12:19 PM, Fred Gleason wrote:
Howdy Folks:
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
There are a number of issues around embedded systems.
They have to work for 10 - 20 years.
They have to be 'safe' for as long as they are working.
At development time, often the most current components are needed (e.g. openSSL 1.1.1, TLS 1.3) This is because, often only patches are done and things still need to work in 10 years.
Most embedded devices don't get firmware update after a couple of years if they ever get any in the first place. It's a tragic state of affairs, but that's my experience at least. Unless there is a major issue that is likely to get somebody outright killed, your chances of getting a firmware update after a few years, expecially after the appliance is no longer manufactured, is very close to 0.
I still deal with embedded systems that are 8bit processors with 32KB memory/storage. Those need not apply to this discussion.
I work with guys who write firmware for new devices with such specifications.
I have dealt with vendors that say they now charge extra for only 1GB memory, as their current design is 2GB. And they call this an IoT board....
There are many classes of embedded systems. You look at what is being embedded in home control gateways, they either are cloud service based (great for captive customers and monetizing) or self-sufficient for lots of reasons (privacy for one).
So I am here, because I believe in enterprise code for these systems.
What constitutes "embedded" varies depending on who you ask. 15 yers ago I laughed at a MS developer who programmed "embedded" systems based on XP, which back then to him meant "a device with 'only' 128MB of RAM". My modified ancient WRT56GS WiFi router has 64MB of RAM and a 32-bit MIPS processor - a spec for which I would have had to sell most of my vital organs 20 years go to get it in a monster workstation on my desk.
Unfortunately, as you increase the size and complexity, the probability of problems increases exponentially.
On 10/26/18 1:08 PM, Gordan Bobic wrote:
On Fri, Oct 26, 2018 at 5:58 PM Robert Moskowitz <rgm@htt-consult.com mailto:rgm@htt-consult.com> wrote:
On 10/26/18 12:19 PM, Fred Gleason wrote: > Howdy Folks: > > > All of this to pose the question: is an ‘enterprise’ distro (in the > specific sense meant here) an appropriate long-term choice for an > ‘embedded’ project? Given the stated intention of the Upstream > Provider to support only ARM systems that integrate APCI and comply > with SBSA [Server Base System Architecture] standards in future major > releases (see > https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), > is such a distro an appropriate long-term choice for an ‘embedded’ > project? There are a number of issues around embedded systems. They have to work for 10 - 20 years. They have to be 'safe' for as long as they are working. At development time, often the most current components are needed (e.g. openSSL 1.1.1, TLS 1.3) This is because, often only patches are done and things still need to work in 10 years.Most embedded devices don't get firmware update after a couple of years if they ever get any in the first place. It's a tragic state of affairs, but that's my experience at least. Unless there is a major issue that is likely to get somebody outright killed, your chances of getting a firmware update after a few years, expecially after the appliance is no longer manufactured, is very close to 0.
I work with people that are mandated by law to perform updates for 10 years. Some of these systems have been deployed for over 4 years already and are getting their updates as needed via different strategies.
Some of these companies are firmly in the BSD camp of one flavor or another for licensing reasons. So they say. Some have thousand to millions of systems deployed or planned.
Many vendors simply love the consumer purchase model as they hope they can just wash their hands of any future problems. Make fixes available, but not easy. Hey, it is cheap, go buy a newer model! Look at the attacks against most of the Internet accessible baby monitors. I met the researcher that compromised almost a dozen models (when we last talked).
Most maintain their own distribution services. They like only needing the QA step before releasing an update.
The landscape if varied and full of landmines :)
The current battle ground is the home controller. Apple, Google, and Amazon are fighting for dominance and the rest are, perhaps, bit players.
I still deal with embedded systems that are 8bit processors with 32KB memory/storage. Those need not apply to this discussion.I work with guys who write firmware for new devices with such specifications.
I have dealt with vendors that say they now charge extra for only 1GB memory, as their current design is 2GB. And they call this an IoT board.... There are many classes of embedded systems. You look at what is being embedded in home control gateways, they either are cloud service based (great for captive customers and monetizing) or self-sufficient for lots of reasons (privacy for one). So I am here, because I believe in enterprise code for these systems.What constitutes "embedded" varies depending on who you ask. 15 yers ago I laughed at a MS developer who programmed "embedded" systems based on XP, which back then to him meant "a device with 'only' 128MB of RAM". My modified ancient WRT56GS WiFi router has 64MB of RAM and a 32-bit MIPS processor - a spec for which I would have had to sell most of my vital organs 20 years go to get it in a monster workstation on my desk.
Unfortunately, as you increase the size and complexity, the probability of problems increases exponentially.
Why Apple has very limited partners that can add code at the OS level. I know. I was at Verizon when we wanted to do this and were told categorically NO. Android was willing, but without both, the project died.
Am 26.10.18 um 18:19 schrieb Fred Gleason:
Howdy Folks:
...
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
Hi Fred,
you can run a centos userland on top of almost any kind of firmware(u-boot)/kernel combination (unless the kernel is not too ancient). You can find a small how-to document here [1]. What kind of SBC do you have in mind for your project? Choosing CentOS will give you some kind of stability and long term support. But you should not expect to run your embedded project without the possibility to update your devices. I believe the possibility to manage your devices in a smart way without touching or visiting them is a key success factor. You might visit https://resin.io/, their main focus is about managing changes. They make heavy use of Docker containers, so you might consider to take CentOS a base image for your Docker based development.
Cheers Uli
[1]: https://github.com/umiddelb/aarch64/wiki/Install-CentOS-7-on-your-favourite-...
On Oct 26, 2018, at 15:00, Uli Middelberg uli@middelberg.de wrote:
you can run a centos userland on top of almost any kind of firmware(u-boot)/kernel combination (unless the kernel is not too ancient). You can find a small how-to document here [1].
Thanks Uli! I used to do this sort thing way-back-when [mid1990s], when I’d routinely compile and the install the latest kernel from kernel.org http://kernel.org/ immediately after the installation of Slackware Linux. Kernels (and distros) have gotten a bit more complicated since then...
What kind of SBC do you have in mind for your project?
I have several commercial designs that have already shipped, all based on various models of Raspberry Pi with RedSleeve Linux. The horse has left the stable already as far as those projects are concerned, but RedHat’s stance regarding the future of RHEL on ARM had me concerned regarding future projects. It’s good to be reminded that there are ways of finessing that design decision.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Hi,
On 10/26/2018 11:19 AM, Fred Gleason wrote: (trimming)
All of this to pose the question: is an ‘enterprise’ distro (in the specific sense meant here) an appropriate long-term choice for an ‘embedded’ project? Given the stated intention of the Upstream Provider to support only ARM systems that integrate APCI and comply with SBSA [Server Base System Architecture] standards in future major releases (see https://lists.centos.org/pipermail/arm-dev/2017-October/003120.html), is such a distro an appropriate long-term choice for an ‘embedded’ project?
Why not? SBSA/SBBR, isn't just about big iron servers, its about providing enough standardization and platform abstraction in the Arm ecosystem that you can run the same basic kernel/distro over a wide variety of devices over time.
That seems like it would be useful for embedded machines as well. If for no other reason than to reduce the effort to apply security patches across a wide range of devices.
On Oct 26, 2018, at 15:42, Jeremy Linton jlinton@redhat.com wrote:
Why not? SBSA/SBBR, isn't just about big iron servers, its about providing enough standardization and platform abstraction in the Arm ecosystem that you can run the same basic kernel/distro over a wide variety of devices over time.
That seems like it would be useful for embedded machines as well. If for no other reason than to reduce the effort to apply security patches across a wide range of devices.
To be sure, but none of the current crop of embedded boards seem to support any of those standards. Whether this is because of laziness, ignorance or the need to hit a particular (often very low) price point I am not in a position to determine.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
On Fri, 26 Oct 2018, 23:10 Fred Gleason, fredg@paravelsystems.com wrote:
none of the current crop of embedded boards seem to support any of those standards.
There may be a half way solution you can leverage, at least on some devices - the ones supported by mainline u-boot.
These days u-boot can present EFI compliant environment, or at least enough to satisfy grub-efi. I have this running in DreamPlugs.
Secondary benefit is that you can build a reasonably vanilla multi-SoC kernel and have that booting from grub without having to prepare it for u-boot.
It makes it a lot easier to support multiple devices with a single kernel (just avoid a distro Frankenstein kernel and stick with vanilla mainline).
And once you have the kernel sorted out correctly, the user space will "just work".
Whether this is because of laziness, ignorance or the need to hit a
particular (often very low) price point I am not in a position to determine.
It's lazyness, ignorance, and inertia of historical utterly inconsiderate contempt and unaccountability of SoC manufacturers producing un-mainlineable kernel and boot loader/firmware bodges to get their SoC working without any intent to ever maintain them.
The only sane thing to do is to only ever consider hardware that is well supported by mainline kernels and bootloaders for any project. Custom bodged kernels and bootloaders just aren't worth the effort. If a hardware manufacturer doesn't think it's worth their time to support open standards, it most certainly shouldn't be worth yours. It took me a few years to finally accept that.
Well I wasn't going to pontificate but since it appears I may have triggered this discussion I'll give myself an illusory license. With the backdrop of someone who is probably the least qualified on this list and someone who did their first x86 centos install just a few years ago. I started playing around with arm maybe a little over a year ago as a result I had to seriously bang my head and also have the persistence to work through the pain. Eventually I got everything I wanted to work and am grateful because my knowledge and understanding has tremendously improved. But....
Last week for the first time I installed centos on a arm machine, specifically machiatobin, using a vanilla iso file. The installation was just like on a x86 system and went without a hitch. All of the headache, custom hand crafted human supervised work, gone, poof, no longer required. To me its a game changer.
So without stepping on anyone's toes and after reading Mr. Masters blurb on APCI etc. linked in the beginning of this thread. I find that at first blush it may be worded in a way where it seems to be a ultimatum or antagonistic towards users of centos on arm hardware (i.e. this list). However, I see it more as a message to the hardware folks - that they need to make sure their hardware works with centos and not the other way around. So that we can (emphasis added) seamlessly install centos onto arm hardware and so that we can (emphasis added) run yum install foobar. Not to mention the security and stability/bug implications on the eco system.
I will humbly disagree with the big vs small/embedded argument as what is big or powerful vs what is not is relative, and as we all know these things progress quickly. Nonetheless, you can install centos on a x86 Asus Vivostick, which isn't exactly a power house so why not the same for arm. If it works it works. Furthermore, consistency across platforms will also ideally allow for a single workflow across those platforms.
Absolutely if I have a choice I will go with hardware that makes my workflow easy, consistent and reliable. If I don't have a choice then it just raises the cost of development, to some that doesn't matter to others it shuts them out.
I think I echo what others have already said but then again does original thought really exist or is it collective? I think in the future processors will be liquid and made out of engineered solutions (the liquidy kind).
On Fri, Oct 26, 2018 at 3:48 PM Gordan Bobic gordan@redsleeve.org wrote:
On Fri, 26 Oct 2018, 23:10 Fred Gleason, fredg@paravelsystems.com wrote:
none of the current crop of embedded boards seem to support any of those standards.
There may be a half way solution you can leverage, at least on some devices - the ones supported by mainline u-boot.
These days u-boot can present EFI compliant environment, or at least enough to satisfy grub-efi. I have this running in DreamPlugs.
Secondary benefit is that you can build a reasonably vanilla multi-SoC kernel and have that booting from grub without having to prepare it for u-boot.
It makes it a lot easier to support multiple devices with a single kernel (just avoid a distro Frankenstein kernel and stick with vanilla mainline).
And once you have the kernel sorted out correctly, the user space will "just work".
Whether this is because of laziness, ignorance or the need to hit a
particular (often very low) price point I am not in a position to determine.
It's lazyness, ignorance, and inertia of historical utterly inconsiderate contempt and unaccountability of SoC manufacturers producing un-mainlineable kernel and boot loader/firmware bodges to get their SoC working without any intent to ever maintain them.
The only sane thing to do is to only ever consider hardware that is well supported by mainline kernels and bootloaders for any project. Custom bodged kernels and bootloaders just aren't worth the effort. If a hardware manufacturer doesn't think it's worth their time to support open standards, it most certainly shouldn't be worth yours. It took me a few years to finally accept that.
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
-
Fred Gleason -
Gordan Bobic -
Jeremy Linton -
Robert Moskowitz -
Skorpeo Skorpeo -
Uli Middelberg