[CentOS] /lib/firmware/microcode.dat update on CentOS 6

[Phelps, Matthew]

On Thu, Jan 18, 2018 at 9:01 AM, Johnny Hughes <johnny at centos.org> wrote:

> On 01/18/2018 07:51 AM, Phelps, Matthew wrote:
> > On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at centos.org>
> wrote:
> >
> >> On 01/18/2018 03:41 AM, Pete Biggs wrote:
> >>>
> >>>> Look at:
> >>>>
> >>>> https://t.co/6fT61xgtGH
> >>>>
> >>>> Get the latest microcode.dat file from here:
> >>>>
> >>>> https://t.co/zPwagbeJFY
> >>>>
> >>>> See how to update the microcode from the links at the bottom of this
> >> page:
> >>>>
> >>>> https://t.co/EOgclWdHCw
> >>>>
> >>>> An before anyone asks .. I have no idea why Red Hat chose this path,
> >>>> they did.  It doesn't matter if I (or anyone else) agrees with the
> >>>> decision.  It is what it is.
> >>>>
> >>> **I'm not blaming you.**
> >>>
> >>> But can I just clarify. We have to *manually* install the microcode
> >>> update an EL7 in order to be protected against Spectre? EL6 as well?
> >>>
> >>> Presumably this is to remove RH from the loop and to stop people
> >>> blaming them - i.e. this is between Intel and the customer, it's
> >>> nothing to do with them.
> >>>
> >>
> >> No, this is because at least one major CPU (Intel type 79) is completely
> >> broken by the Intel Microcode Update.  Those machines can't boot after
> >> the microcode rpm is installed.  It impacts at least these processors:
> >>
> >> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
> >> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
> >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
> >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
> >>
> >> There may be others.
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1532283
> >>
> >> That means, it is NOT full-proof install and likely leaves many
> >> servers/machines broken.  I suppose that the decision is, go pack to
> >> what works for all machines (the last known good install) and let admins
> >> work with their hardware vendor because the alternative is breaking
> things.
> >>
> >> This also needs to be fixed with a combination of firmware updates AND
> >> microcode updates.  All of that is outside the expertise of a Linux
> >> vendor and is unique for each processor, chipset and firmware
> combination.
> >>
> >>> What about future microcode updates? They come out reasonably regularly
> >>>  (2 or 3 times a year) - are RH going to absolve themselves from all
> >>> future updates because presumably the next update will also contain the
> >>> Spectre fixes?
> >>>
> >>
> >>
> >> How they handle it in the future I have no way of knowing, but if you
> >> had 20,000 servers with the impacted CPU and you updated and could not
> >> reboot, I would assume that you did not appreciate it.
> >>
> >>> So, before I re-invent the wheel, does anyone have automation scripts
> >>> to do the microcode update? I don't relish the prospect of doing this
> >>> manually on a couple of hundred machines. Is it reasonable to grab the
> >>> microcode_ctl SRPM and create my own updated RPM to do it?
> >>>
> >>
> >> That is what I have found so far with a bit of research.
> >>
> >> This is NOTHING about who to blame and everything about stable, working
> >> updates .. at least it seems so to me.
> >>
> >>
> > So, if we applied the previous microcode update, and all our machines
> > rebooted OK, then we don't need to fallback?
> >
> > Also, do we know if the updated CentOS microcode RPM reverted the
> microcode
> > for *all* Intel CPUs, or just the ones that had issues? In other words,
> if
> > I apply the latest microcode update to our 100+ machines (which all have
> > the previous update, and are OK) will they revert to a vulnerable state?
> >
> >
>
> It reverted for all .. but, your machines may or may not be protected as
> only a subset of machines were updated with the original microcode from
> Intel.
>
> It is your call as to what you install .. but the correct method is to
> install the current microcode_ctl .. and then research your specific
> machine, its CPU, chipset, firmware .. go to the vendor and make sure
> you get all the things necessary to mitigate the issues.  It will be
> different for each CPU vendor (Intel or AMD), each CPU / Chipset combo,
> and even each vendor (Dell may have new firmware for x and y but not z
> models, etc.)
>
> There is no one size fits all update for this issue.
>
>
Johnny,

Thank you. We all appreciate your efforts, and your guidance.


-- 
Matt Phelps
System Administrator, Computation Facility
Harvard - Smithsonian Center for Astrophysics
mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu