<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 19, 2017 at 1:32 PM, Jan Chaloupka <span dir="ltr"><<a href="mailto:jchaloup@redhat.com" target="_blank">jchaloup@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div id="gmail-m_-7213588743056765084magicdomid4"><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">Sharing notes from a visit
in IT4Innovations center in Ostrava.<br>
<br>
Supercomputer parameters info are available at [1] - mixture of
Xeon, Xeon Phi and graphic card cores. Two clusters - Anselm
runs RHEL 6, Salomon runs CentOS 6.</span></div>
<div id="gmail-m_-7213588743056765084magicdomid5"><br>
</div>
<div id="gmail-m_-7213588743056765084magicdomid164" class="gmail-m_-7213588743056765084ace-line"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">Each node runs on a
specific piece of a hardware. If a project is built on a builder
of a different hardware, it is built with different flags and
configuration and the resulting binaries are not properly
optimized. Some projects need special features of compilers in
order to run efficiently. Some of the features are available
only in the latest versions of compilers. By the time the latest
versions get into builders, it is usually to late. Or a project
needs to be built with a proprietary compiler that is not
publicly available. Or given each project needs different
version of system libraries in general, the HPC infrastructure
needs to offer various versions of the same library. </span></div>
<div id="gmail-m_-7213588743056765084magicdomid165" class="gmail-m_-7213588743056765084ace-line"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">The software needs to
be available in many flavors. It means the same software to be
built with multiple compilers and multiple versions. Different
versions have different properties/features. Different compilers
accent/provide different optimizations. Thus, a matrix of
software to provide the users with. At the end a software needs
to be built for the end-system architecture so it can use all
the available instructions not to slow down the computation.</span></div>
<div id="gmail-m_-7213588743056765084magicdomid167" class="gmail-m_-7213588743056765084ace-line"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">For that reason (and
many others) all the projects need to be built locally inside
the cluster. That makes most of the binary packages in CentOS
distribution unusable for the HPC use cases.</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> Only usecase for rpm </span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">a</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">s proof of concepting on
devel laptops is useful, not for final deployments.</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc"> Currently, the
EasyBuild project [2] is used as a replacement for rpm based
spec files.</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> </span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">Operators </span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">use Fedora upstream
monitoring tool to monitor latest&greatest software (atm.
~400+ projects). </span></div>
<div id="gmail-m_-7213588743056765084magicdomid8"><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">Infiniband is use</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">d</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> for connection of the
nodes, sometimes issues with 3rd party SW drivers (Bull/Atos
and/or HPE).</span></div>
<div id="gmail-m_-7213588743056765084magicdomid9"><br>
</div>
<div id="gmail-m_-7213588743056765084magicdomid10"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">Other notes:</span></div>
<div id="gmail-m_-7213588743056765084magicdomid11"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- Lot of service
providers are still running on CentOS 6, which blocks upgrade to
CentOS 7</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> -
shutdown of cluster not possible, infra for clusters changed
between RHEL 6 and RHEL 7</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">.</span></div>
<div id="gmail-m_-7213588743056765084magicdomid12"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- Puppet and Ansible
used to deploy clusters (Ansible for disk-free nodes, Puppe</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">t</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc"> for nodes with
disks). Still, each deployment is unique (e.g. unexpected
situations) and thus not fully automated.</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> For Ansible</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc"> part</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">,</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc"> there is no role</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> use</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">d</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> from</span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc"> the Ansible</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z"> Galaxy - just Core
modules and custom roles and playbooks</span></div>
<div id="gmail-m_-7213588743056765084magicdomid68" class="gmail-m_-7213588743056765084ace-line"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- Experiments with
containers as well via Singularity [3] (Docker is not fully
supported on CentOS 6, needs privileged user account)</span></div></div></blockquote><div><br></div><div>even if docker was supported, the kernel on the compute nodes of a cluster will stay fixed and old (due to e.g. infiniband support built-in).</div><div>Won't that break containers in case someone creates a docker image assuming an access to a very recent kernel on the docker host?</div><div><a href="https://forums.docker.com/t/libc-incompatibilities-when-will-they-emerge/9895/4">https://forums.docker.com/t/libc-incompatibilities-when-will-they-emerge/9895/4</a><br></div><div><br></div><div>Marcin</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<div id="gmail-m_-7213588743056765084magicdomid14"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- Demand on packaging
and providing tooling for HPC rather than libraries themselves.
If possible, provide full HPC stack that is upstream and
distribution supported/maintained (including full stack
upgrades). CI/CD supported as well.</span></div>
<div id="gmail-m_-7213588743056765084magicdomid15"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- HPC community is
unfortunately security free</span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z">, security fix deployment
can take several months, dependencies on specific minor
releases or kernel versions. Kernel KABI whitelist should be
advised to 3rd party vendors of drivers to prevent hard version
deps</span></div>
<div id="gmail-m_-7213588743056765084magicdomid16"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">- Each assigned set
of nodes is expected to be vanilla new. Given it takes some
time before a node is rebooted (order of minutes), all the
tooling running inside a node must clean everything a user task
left. Thus, minimize a number of times a node is really
rebooted.</span></div>
<div id="gmail-m_-7213588743056765084magicdomid18"><br>
</div>
<div id="gmail-m_-7213588743056765084magicdomid19"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">[1] </span><span class="gmail-m_-7213588743056765084author-g-312iv59qbixk1uuz122z gmail-m_-7213588743056765084url"><a href="https://docs.it4i.cz/salomon/hardware-overview/" target="_blank">https://docs.it4i.cz/salomon/<wbr>hardware-overview/</a></span></div>
<div id="gmail-m_-7213588743056765084magicdomid20"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">[2] </span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc gmail-m_-7213588743056765084url"><a href="https://github.com/hpcugent/easybuild" target="_blank">https://github.com/hpcugent/<wbr>easybuild</a></span></div>
<div id="gmail-m_-7213588743056765084magicdomid21"><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc">[3] </span><span class="gmail-m_-7213588743056765084author-g-eeoa7mbyz122z3pz122z5kuc gmail-m_-7213588743056765084url"><a href="http://singularity.lbl.gov/" target="_blank">http://singularity.lbl.gov/</a></span></div>
</div>
<br>______________________________<wbr>_________________<br>
CentOS-devel mailing list<br>
<a href="mailto:CentOS-devel@centos.org">CentOS-devel@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-devel" rel="noreferrer" target="_blank">https://lists.centos.org/<wbr>mailman/listinfo/centos-devel</a><br>
<br></blockquote></div><br></div></div>