[CentOS] Re: Demonizing generic Linux issues as Fedora Core-only issues -- WAS: Hi, Bryan

Sat May 28 16:19:38 UTC 2005
Lamar Owen <lowen at pari.edu>

On Saturday 28 May 2005 01:54, Les Mikesell wrote:
> On Fri, 2005-05-27 at 19:13, Lamar Owen wrote:
> > For the most part, the Red Hat crew is the best in the business.  Or have
> > you never heard of Jakub Jelinek, Alan Cox, Rick van Riel, and many many
> > other of the top upstream developers that are employed in some capacity
> > by Red Hat? 

> I think we've beaten these topics to death, but since it is kind of
> fun if you don't take it too seriously:  Which of these guys knows how
> to make perl do character set conversions correctly better than the perl
> team?

Given time I believe Tom Lane (one of the PostgreSQL core developers and one 
of the key libjpeg developers -- and a Red Hat employee) could do it without 
question.

> Sigh... at this point it is "how many versions does the vendor support"?
> And the issue is that the perl version (among many other things) that
> does a certain operation correctly is only included with a kernel
> version that has features missing and broken.

And what's the solution?  Wait until upstream gets it 'correct' for 1000 
pieces of upstream?  One of the packages is going to always be broken, 
statistically speaking.  You have to draw the line somewhere.

> I don't see how you can call setting up a WAN with many CIPE nodes, then
> finding it unavailable in the next release 'long term stability'.

You then stick with the five year cycle of the release you have installed.  
You don't upgrade.  In fact Red Hat specifically says not to 'upgrade' and to 
carefully roll out version increases with scratch new installs.  Then, if you 
absolutely must have a feature only available in the latest release, you must 
do a cost-benefit analysis to determine which features are most important for 
you.

The five year supported cycle with no in-cycle version upgrades is the long 
term stability I'm talking about; IT Directors (I am one, so I can speak like 
one) want a stable five-year plan; and we do NOT want version upgrades that 
can break things.  Things like OpenSSL's versions, for instance, where the 
ABI is broken between minor versions, even, are unacceptable for mission 
critical servers (which IS the target audience for RHEL!)

The perl code change you mention could have broken many things; Red Hat or any 
other Enterprise Linux vendor must freeze packages at some point to get any 
reasonable amount of QA in;  and you really don't want to mess up the pot in 
a security release.

> > Microsoft-compatible PPP over L2TP over IPsec, which is so easy to set up
> > on the Windows client side it isn't even funny).  That's why for-purpose
> > firewall/VPN appliance Linux dists (SmoothWall and Astaro, for instance)
> > are not using anything but IPsec.  I have a SmoothWall box myself, and it
> > Just Works.

> Can you run it through NAT routers? 

Yes.  NAT traversal is fully supported.

> Whatever happened to the Caos distribuion?  I had some hopes that they
> would combine a very stable kernel/libc combo with very up-to-date apps.

CentOS and cAos are no longer associated.  cAos is at caosity.org and is still 
doing their thing.

> > For instance, suppose you want PostgreSQL 8 instead of the supplied 7.4. 

> So how is it better that when you do get Postgresql 8 in a distribution
> you will likely get a largely untested kernel along with it?  

Just because the corner case that impacts you doesn't work does not mean the 
kernel was largely untested; there were in fact many months of testing behind 
the kernel that is in RHEL4/CentOS4.  Your particular case was not tested; 
did you bother to test the Fedora Core kernels leading up to it, knowing that 
Fedora is the 'testbed' for RHEL?  There's no such thing as a free lunch; 
either you buy your lunch, you make your lunch, or you eat what someone gives 
you.  And beggars cannot be choosers: if the person who gave you your lunch 
uses a seasoning you don't like and you complain, do you think the person is 
going to make all their lunches your way next time?

> Or that 
> you compile your own version and end up with potentially unique problems
> that every user has to solve individually?

When you compile your own you become responsible for what you have compiled.  
This means you lose support from Red Hat, if you have an SLA for RHEL.  This 
means the developers of a free alternative don't have to support your 
version.  You are indeed on your own.  That is the cost of open source, 
unfortunately.  You want support?  Then you work within the framework you 
have purchased or have decided to use.  You don't like the framework?  You 
get a different one, and absorb the costs of the migration.  Basic IT 
business.

> I used to laugh at the people running windows servers because they would
> not even try to run more than one application per machine.  But that was
> when I compiled a lot of stuff by hand.  Now that I've gotten lazy (or
> smart...) and try to stick to binary distributions, I don't laugh any
> more because now my Linux boxes are in exactly the same shape.  I can't
> count on being able to upgrade any single application without breaking
> another one.

That's right.  It's not a Windows disease; it's a server disease, and the two 
items on the balance are version stability and features.  The two are often 
mutually exclusive.

> Fedora breaks everything at once, so it isn't what I want.  I want
> "something like" being able to install a Centos 3.x base, then
> selectively update certain apps to current-fedora level.  A lot of this
> would work with just a source rpm rebuild - but what I'm really after
> is something that once done would still automatically pull updates
> when the selected packages were fixed.

Like Johnny said, then you probably want something like Gentoo or a BSD.

> I think character set conversion is something where the correctness can
> be determined objectively.  And perl on Cento3 and Centos4 will do 2
> different things.  I'll take the perl team's word for it being made
> correct in 5.8.3 and up.

So they do different things.  There are many many things in CentOS4 that will 
do different things from CentOS3; PostgreSQL, for instance, won't upgrade 
between the two versions and you have to do a dump-initdb-restore between 
versions.  When the great Apache divide occurred (1.3 to 2.0) that broke 
things; Sendmail version increases can break things; PHP version upgrades can 
break things; how is the perl example any different?  More to the point: was 
it documented in the release notes?  If not, when the first update came out 
had it been noted?  Was it noted during the development cycle (Fedora Core 
being the development cycle)?  You don't mean you tried to roll out an fully 
unsupported upgrade (Red Hat DOES NOT SUPPORT and WILL NOT support upgrading 
one RHEL version to another!) remotely using the first release of a major 
version increment?  

Well, if there is a critical feature you absolutely must have, then you really 
really have a problem.  I rolled out CentOS4 on a mission critical server: 
but only after I had been using Fedora Core 2 and 3 for a long period of time 
(FC2 and 3 are the development lines leading up to RHEL4, after all, and that 
means the 2.6.x line had gotten right at a year of testing at my site prior 
to rolling CentOS4 out!) on other non-critical but similar servers.  It is 
irresponsible to roll out releases that one has not tested to one's critical 
servers, and in my shop could be grounds for dismissal.

> I have a feeling that the 2.6 kernel will become usable before that
> happens this time around, but maybe by the next time RH rushes out
> a largely broken release there will be viable competition in Ubuntu
> and other distributions that will keep a working kernel under a
> fast release schedule for the apps.

Quite frankly, if RHEL4 were 'largely broken' it would not have been released.  
It is not a perfect release; none are.  But it is a pretty good release and 
meets my needs very well: to the point that CentOS 4 is becoming the standard 
desktop and server Linux for PARI (as well as for me personally).  Just wish 
a SPARC version existed for my cluster of 20 Ultra 30's.

> I don't see how the CIPE author has any reason to care one way or
> another whether any particular distribution includes his package or
> not.  Why should he?

Because his users might care?

> Well, if I had previously included CIPE, promoted it's use, and wanted
> anyone to think the distribution had long term stability...

RHEL3 has long term stability: within RHEL3's line (and RHEL3 is STILL BEING 
SOLD and SUPPORTED for five years from initial release) there is long term 
stability.  Red Hat doesn't support upgrades from RHEL3 to RHEL4, and fully 
documents the lack of CIPE in the release notes, which everyone should read 
before deploying an OS anyway.  This is the way Big IT works, likes to work, 
and needs to work.  You want the stability of a Big IT OS, then you must 
either take what's distributed or put the responsibility on your own 
shoulders.  Or find a distribution that better meets you needs (like Johnny 
said, Gentoo might be your best shot, or, if you like Ubuntu go that way: 
it's Open Source and you're free to choose (that is, after all, what I like 
most about Open Source, is that I am free to choose what meets my needs, and 
can really do a cost-benefit study on various options)).

> Well, now we know how much value they place on maintaining
> interoperability with their own older products. Don't forget that
> when planning for the future.

This is very true.  And is simply one of the costs of doing business; they 
have to target their customers, and for the most part they are doing a great 
job of it.  Yes, they make missteps (caching-nameserver anyone?) but the 
positives are far greater than the negatives, IMHO and for my particular 
purpose.  YMMV.  CIPE wasn't high enough on their customers' lists to 
override their other concerns; they made the executive decision to drop CIPE 
since it was going to be a pain to get it working, and they wanted to go 
another direction anyway.

> Please explain how the CIPE author has any vested interest in this.
> I would have expected Red Hat to try to provide their existing
> users a way to upgrade one site at a time and remain compatible.
> Now I don't expect much at all.

Red Hat makes no secret about their complete non-support of major version RHEL 
upgrades; it's plastered heavily all over their release notes, and, in fact, 
you must supply a kernel command line parameter to get any upgrade 
functionality in RHEL.  This should tell you something; I know it tells me 
'upgrades aren't supported: if they work, you got lucky.'

Further, even with Fedora upgrades aren't really supported (but, then again, 
the whole OS isn't supported).

This is the route Red Hat has chosen to go; for most of their paying customers 
it is the correct behavior.

> > Linus != Red Hat.  This is a Linus issue; not a Red Hat issue.  Thus the
> > subject line.  You can't blame Red Hat for something Linus caused.

> Red Hat made the decisions about if and when to ship it.

And they were the last Enterprise Linux vendor to ship it.  Their customers 
wanted the featureset 2.6 provided, and CIPE wasn't high on their list.

> CIPE doesn't have to play any game, but a version for 2.6 is available
> and has been for some time.  And before you say things had to be in
> Fedora before shipping in RHEL, look at the Mysql version included in
> RHEL4.

MySQL is more important than CIPE to the majority of Red Hat's customers, and 
MySQL 4 was considered a critical feature by Red Hat.  I myself don't use 
MySQL (I use PostgreSQL, which to me is much better in every way).  Red Hat 
was getting beat up for the whole MySQL problem (which was a MySQL AB caused 
problem), and they really had to correct it.  That might very well bite them 
in the future.

> I think I asked before what things are measurably better in 2.6 and
> didn't get any answers.   Are there any?

Sure.  LVM.  IPtables.  ATM support.  SMP. NUMA.  HyperThreading (which really 
isn't enterprise-grade yet, unfortunately).  Sizeably better performance in 
the core LAMP role.  Seriously better performance (nearly twice) in typical 
file server roles.  Red Hat is much much closer to upstream kernels in the 
2.6 line rather than the very heavily patched and nearly 2.6 RHEL3 2.4 
kernels.  Filesystem size not limited to 2TB.  Support for more CPU's in SMP.  
Support for much larger physical memory.  More devices and processes 
available for larger servers.  Better interactive response for desktops.

See http://www.2cpu.com/articles/98_1.html and http://librenix.com/?inode=3972 
for more.  Yes, 2.6 is worth the pain for the gain.  Both of these articles 
were written nearly a year before the release of RHEL4; sounds to me like it 
wasn't an 'untested' kernel.  Also see 
http://www.infoworld.com/infoworld/article/04/01/30/05FElinux_1.html

But I do somewhat sympathize with your position; after all, I still have a 
mission critical server running Red Hat Linux 5.2.  Yes, 5.2.  Long ago in 
the mists of time support stopped; but I have a mission critical binary-only 
application (that will remain nameless) that won't run on anything higher.  
We're attempting to find a replacement, but none exist that is backwards 
compatible to some of our clients; we have the later version for newer 
clients, but keep the old server running for the clients that won't work with 
the newer version.  

The reason?  The app was actually written for Red Hat 4 (I've come full 
circle; I started out with Red Hat Linux 4 and am now, eight years later, 
running a clone of a Red Hat 4 distribution!) and the compat libs needed 
won't work on Red Hat 6 or higher.  There was some work to get it running on 
Red Hat 6.2, but it wasn't stable.  The 5.2 server runs like clockwork; but I 
have to be extremely careful to packport highly critical fixes (2.0.40 
kernel, for instance).  And I can't run any Internet-visible apps on it; too 
old; it's so old it doesn't even have SSH!  So I had to pull in a second 
server to run the Internet-visible stuff.  That was the cost.  This app's 
support of Red Hat 4 was, by the way, one of the reasons I bought Red Hat 
Linux 4.1 in the first place.

Also, I am in a similar position with newer versions of software; in my case, 
there is a repository catering to my needs: KDE-Redhat.  I need the features 
of KDE 3.4 (Kstars in particular) and Rex does a great job keeping up.  But I 
wouldn't deign to ask the core CentOS team to maintain a KDE 3.4 release and 
the over 500MB of needed files (it touches so many things you must download a 
vast array of packages to just get KDE 3.4).  If Rex (Dieter) weren't doing 
it, I would have the difficult choice of whether to do all that work myself 
or do without the features.  In that particular case I would probably switch 
to using Xephem instead of Kstars and maintaining it myself, but even then 
I'd have to look at the decision very carefully.

I also have applications that would like to have Python 2.4 available; but 
CentOS 4 comes with Python 2.3.  When one of the core Python apps I use 
(either GNUradio or Plone/Zope) begins requiring Python 2.4 then I will have 
another choice to make: are the features of the newer versions of GNUradio 
and Plone worth the effort of maintaining my own Python 2.4 installation?  
I'm already maintaining my own fftw3 distribution (GNUradio requires the 
single-precision build, which is not the default), so I can do this (I 
maintained the PostgreSQL RPMs for five years (not anymore; people were 
demanding too much backwards compatibility for me to realistically continue 
volunteering my valuable time!) so I know what I'm doing). But the question 
distills to whether the features are worth it or not.  And I do not expect 
the CentOS team or Red Hat to do any of this for me; if an upgrade breaks 
something I have done then I'm on my own.

I have also done this before with Tcl; I run an application server framework 
called OpenACS at a couple of sites, and it requires Tcl 8.4.6+ built for 
multithreaded operation; this is not the default version nor is it the 
default configure switch.

And I don't use any of the RPMs for Zope; Zope is way too customized at my 
site for that.
-- 
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC  28772
(828)862-5554
www.pari.edu