[CentOS] Spacewalk or Puppet?

[nate]

Karanbir Singh wrote:

> To me, this is a major contribution by some of these tools today -
> spacewalk, puppet, cfengine, chef, bcfg2, slack : all becoming focal
> groups - even if they only address specific use-cases or only address
> certain mindsets / thought process's.

Another aspect of the management tools I forgot to mention is their
own complexity. Most of the enterprise grade stuff is so complicated
you need dedicated trained people to work on it as their sole task,
whether it's something like BMC or HP Openview/sitescope type stuff.

I can only speak for what I've used but at my current company I
was hired after the previous guy left. One of the reasons my company
wanted me was my knowledge of cfengine, which they had deployed but
really nobody knew how to use.

It wasn't easy going into a new enviornment with a five 9 SLA and
dig into their systems with pretty much the only documentation that
I had was a poorly setup cfengine implimentation. I mean their
install process involved *manually* defining classes on the
command line and running the agent to pick up the associated
configurations for those classes. Where as my setup the classes
are always defined and no manual intervention needed. The internal
infrastructure was a total mess, and some of it still is. The
older DNS systems run on windows and weren't setup properly, the
master DNS failed a couple of days ago causing havok on the back
end. I worked around it for a little while they couldn't get the
system back up, then yesterday the main zone for the back end
expired wrecking havok again, and we did an emergency migration
to linux at that point. The guys doing the windows stuff don't know
what they are doing, it's all legacy and hasn't been touched in
years.

So I spent a few months slowly re-writing the entire system, along
with re-doing the entire build system which was just as bad but
fortunately kickstart is a lot easier to work with.

Even today more than a year later there are probably 45-50 systems
out there running on the "old" stuff. There is no good/safe/clean
way to migrate to the new w/o re-installing. So I've been doing
that as opportunities present themselves. Probably 40 of those
systems will be replaced in the next 3-5 months so that will allow
me to almost complete the project..

Maybe when I leave my current company the next person will come
in and re-write everything again, or switch to puppet or
something..

Funny I just realized when I left my previous company I left them
with another guy who was there the whole time who knew CFengine as
well, and he tried to train another guy there before he left
but since all of them have left.. They hired some other guy but
not sure if he's still there and not sure if he ever learned how
things worked, the company has mostly collapsed under multiple
failed business models.

I feel for the folks who are so overworked and stressed out that
they don't have the ability to learn better ways of doing things,
I've been there too. And fortunately am now in a position where I
have the luxury of refusing such positions and tasks now because
it's not worth my time.

As for monitoring automation I hear ya, that would be cool to
have, right now for us it's not much of an issue our growth rate
is pretty small(roughly 400 systems). We have a tier 1 team that
handles most of monitoring setup so I just give them a ticket
and tell them to do it and they do.

I'll be deploying another new edge data center location in a
couple of weeks, 14 servers, 10 bare metal, 4 run ESXi, about
50 instances of CentOS total, with kickstarting over the WAN
I can usually get most everything up in about a day, which
sadly is faster than the network guy takes to setup his two
switches, two load balancers, and two firewalls. Despite the
small server count the hardware is benchmarked to run our
app at about 40,000 transactions a second as a whole, which
is the fastest app by several orders of magnitude that I've
ever worked on anyways. We have more capacity issues on the
load balancers than we have with the servers with such
a small server count.

It certainly was an interesting experience the first time
deploying a data center from remote, we just had 1 really
basic server config used to seed the rest of the network,
everything done via remote management cards in the servers
and remote installations over the WAN. With the exception of
the network stuff which the network guy was on site for
a couple of days to configure.

I have 10x the equipment to manage and can still get things
done faster than him. I could manage all of the network stuff
too without much effort.

If I spent some time I'm sure I could automate some nagios
integration but forget about cacti, lost cause. Maybe OpenNMS at
some point who knows.. At this time automating monitoring
integration isn't a pressing issue. I spend more time writing
custom scripts to query things in the most scalable way I can
than we do adding new things to the monitoring stuff.

Myself I am not holding my breath on any movement or product to
come around and make managing systems especially cross platform
simpler and cheaper. The task is just too complex. The biggest
companies such as Amazon, Google, MS etc have all realized there
is little point in even trying such a thing.

It would only benefit really small companies that are at a growth
point where they don't have enough business to hire people to
standardize on something(or have teams for each thing). And
those companies can't afford the costs involved with some big
new fancy tool to make their lives easier. The bug guys don't
care since they have the teams and stuff to handle it.

Though that won't stop companies from trying..the latest
push of course is to the magical cloud where you only care about
your apps no longer care about the infrastructure.

Maybe someday that will work, I talked to a small company
recently that is investing nearly $1M to in source their
application from the cloud to their own gear(they have never
hosted it themselves) because of issues with the cloud that
they couldn't work around.

good talks though, the most interesting thread I've seen here
in I don't know how long. Even though it was soooooooo off topic
(from what the list is about anyways) :)

nate