On Thu, Jan 18, 2018 at 9:01 AM, Johnny Hughes <johnny at centos.org> wrote: > On 01/18/2018 07:51 AM, Phelps, Matthew wrote: > > On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at centos.org> > wrote: > > > >> On 01/18/2018 03:41 AM, Pete Biggs wrote: > >>> > >>>> Look at: > >>>> > >>>> https://t.co/6fT61xgtGH > >>>> > >>>> Get the latest microcode.dat file from here: > >>>> > >>>> https://t.co/zPwagbeJFY > >>>> > >>>> See how to update the microcode from the links at the bottom of this > >> page: > >>>> > >>>> https://t.co/EOgclWdHCw > >>>> > >>>> An before anyone asks .. I have no idea why Red Hat chose this path, > >>>> they did. It doesn't matter if I (or anyone else) agrees with the > >>>> decision. It is what it is. > >>>> > >>> **I'm not blaming you.** > >>> > >>> But can I just clarify. We have to *manually* install the microcode > >>> update an EL7 in order to be protected against Spectre? EL6 as well? > >>> > >>> Presumably this is to remove RH from the loop and to stop people > >>> blaming them - i.e. this is between Intel and the customer, it's > >>> nothing to do with them. > >>> > >> > >> No, this is because at least one major CPU (Intel type 79) is completely > >> broken by the Intel Microcode Update. Those machines can't boot after > >> the microcode rpm is installed. It impacts at least these processors: > >> > >> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz > >> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz > >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz > >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz > >> > >> There may be others. > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1532283 > >> > >> That means, it is NOT full-proof install and likely leaves many > >> servers/machines broken. I suppose that the decision is, go pack to > >> what works for all machines (the last known good install) and let admins > >> work with their hardware vendor because the alternative is breaking > things. > >> > >> This also needs to be fixed with a combination of firmware updates AND > >> microcode updates. All of that is outside the expertise of a Linux > >> vendor and is unique for each processor, chipset and firmware > combination. > >> > >>> What about future microcode updates? They come out reasonably regularly > >>> (2 or 3 times a year) - are RH going to absolve themselves from all > >>> future updates because presumably the next update will also contain the > >>> Spectre fixes? > >>> > >> > >> > >> How they handle it in the future I have no way of knowing, but if you > >> had 20,000 servers with the impacted CPU and you updated and could not > >> reboot, I would assume that you did not appreciate it. > >> > >>> So, before I re-invent the wheel, does anyone have automation scripts > >>> to do the microcode update? I don't relish the prospect of doing this > >>> manually on a couple of hundred machines. Is it reasonable to grab the > >>> microcode_ctl SRPM and create my own updated RPM to do it? > >>> > >> > >> That is what I have found so far with a bit of research. > >> > >> This is NOTHING about who to blame and everything about stable, working > >> updates .. at least it seems so to me. > >> > >> > > So, if we applied the previous microcode update, and all our machines > > rebooted OK, then we don't need to fallback? > > > > Also, do we know if the updated CentOS microcode RPM reverted the > microcode > > for *all* Intel CPUs, or just the ones that had issues? In other words, > if > > I apply the latest microcode update to our 100+ machines (which all have > > the previous update, and are OK) will they revert to a vulnerable state? > > > > > > It reverted for all .. but, your machines may or may not be protected as > only a subset of machines were updated with the original microcode from > Intel. > > It is your call as to what you install .. but the correct method is to > install the current microcode_ctl .. and then research your specific > machine, its CPU, chipset, firmware .. go to the vendor and make sure > you get all the things necessary to mitigate the issues. It will be > different for each CPU vendor (Intel or AMD), each CPU / Chipset combo, > and even each vendor (Dell may have new firmware for x and y but not z > models, etc.) > > There is no one size fits all update for this issue. > > Johnny, Thank you. We all appreciate your efforts, and your guidance. -- Matt Phelps System Administrator, Computation Facility Harvard - Smithsonian Center for Astrophysics mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu