Hello,
A colleague and I are trying to get an ARM server to boot stateless (i.e. diskless). Has anyone had any success with this? Our primary source of inspiration is the documentation for the Pegasus cluster:
http://web.mst.edu/~vojtat/pegasus/administration.htm
Basically, the idea is to have the kernel load a custom initramfs that decompresses a file system image onto a tmpfs at boot time (along with some other setup), then execute a `switch_root` to said tmpfs.
We have succeeded in getting an x86_64 machine to boot (mostly) diskless, using the disk only for the boot process by creating a GRUB entry. However, when we try this with my ARM (ThunderX) server, the kernel panics during the boot process because it cannot find init. We have also tried passing `init=/init` as a kernel parameter with no luck. The error message is
Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
We looked at the referenced documentation and it's not a ton of help for my specific case. The initramfs environment is based on Busybox, which we compiled on a ThunderX machine (the same one we are trying to boot from, in fact).
We have considered compiling my own kernel with the initramfs built in, but we are trying to avoid doing that. Further, we have tried with the 3.10 and 4.4 kernels on x86, and with 4.5 and 4.14 kernels on aarch64.
Does anyone have any guidance? Also, please let us know if you need more information.
Best regards,
Marc Patton and Seth Meyer ARC-TS - https://arc-ts.umich.edu/ University of Michigan
Hi there,
I built/d a bunch of RHEL, Centos and other 'clones' clusters. First of all, if you need a disk to start a cluster node, well, it's not diskless. Typically, you'd boot a node using PXE. (also, local disks in a diskless environment always causes weird trouble).
After that there are several routes you can take. You can do something straight forward, like use DHCP/tftp or for example use perceus and vnfs for creating your boot images (or something similar). Perceus and vnfs are older, but very manageable, especially on larger scale clusters (dozens/hundreds/thousands) and are still used.
Since you're trying to netboot a RPI, I'd probably keep the initram disk very small and actually use something like perceus and vnfs. (with vnfs you can actually build your whole image, deciding what needs to be on the ramdisk, and what are links to an nfs mount). Using Perceus you'd boot the RPI using a kernel for handling the vnfs image/capsule, and after all of that is taken care of, it switches (kexec) to the kernel you actually want to run.
The stateless part; maybe, but depending on what sort of cluster you want to setup up, you'd start changing "the state" of an image right after boot, before you re-connect, or use different vnfs capsules all together.
It might be easier to start with dhcp/tftp, to get things going. If what you want to accomplish is more elaborate then just bring up a few networked diskless nodes, you probably want to look into something like perceus or so.
If you want to "run" a complete distro, you'd probably have to since the RPI only has 2Gb, and you can't stick a complete filesystem into that.
Ron
On 6/4/19 12:16 PM, Seth Meyer wrote:
Hello,
A colleague and I are trying to get an ARM server to boot stateless (i.e. diskless). Has anyone had any success with this? Our primary source of inspiration is the documentation for the Pegasus cluster:
http://web.mst.edu/~vojtat/pegasus/administration.htm
Basically, the idea is to have the kernel load a custom initramfs that decompresses a file system image onto a tmpfs at boot time (along with some other setup), then execute a `switch_root` to said tmpfs.
We have succeeded in getting an x86_64 machine to boot (mostly) diskless, using the disk only for the boot process by creating a GRUB entry. However, when we try this with my ARM (ThunderX) server, the kernel panics during the boot process because it cannot find init. We have also tried passing `init=/init` as a kernel parameter with no luck. The error message is
Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
We looked at the referenced documentation and it's not a ton of help for my specific case. The initramfs environment is based on Busybox, which we compiled on a ThunderX machine (the same one we are trying to boot from, in fact).
We have considered compiling my own kernel with the initramfs built in, but we are trying to avoid doing that. Further, we have tried with the 3.10 and 4.4 kernels on x86, and with 4.5 and 4.14 kernels on aarch64.
Does anyone have any guidance? Also, please let us know if you need more information.
Best regards,
Marc Patton and Seth Meyer ARC-TS - https://arc-ts.umich.edu/ University of Michigan
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Hello,
I'm not sure if what you call 'diskless' or 'stateless' you also mean a 'non-destructible' filesystem, which was a concept used on early Linux routers.
Anyway, this is what I do.
I'm using CentOS 7 on Raspberry Pi 3 to create Linux routers/gateways for multiple devices (communicating with radio-frequency) on an IoT environment. The Linux system must be capable to survive to power outages at any moment without suffering filesystem corruption. Hence the concept 'non-destructible' filesystem.
I accomplish this by creating a squashfs of the root filesystem with very minimal set of packages. That is, right after the installation of CentOS 7 image, I do a lot of 'yum remove' and 'rpm -e' over multiple packages that I don't need. Once done, I squashfs the root filesystem into a file that occupies no more than 270MB.
With that image I can now create a new microSD with /boot, /data and /root partitions, and copy the squahsfs image directly into /root partition. I use the /data partition as the non-volatile space to save configurations, metric data, notifications, etc. coming from the IoT devices. Then I use Busybox to create a small filesystem that will be used as the initramfs. You can use this tree [1] as a good starting point, or make your own. Don't forget to include the squashfs and overlayfs kernel modules.
In that initramfs I create the /init program which can be a simple shell script. Such init file will be the responsible to setup the system booted on the initramfs to switch to the root filesystem on the squashfs image:
- mount /proc and /sys filesystems. - make /dev with 'mdev -s'. - install the squashfs and overlayfs kernel modules (which won't be installed by default in the current boot with initramfs). - mount the squashfs (i.e: in /ro). - create the tmpfs (no less than 300MB) and mount it (i.e: in /rw). - mount the writable overlayfs combining /ro and /rw. - prepare the switch with 'mount --move' /rw and /ro into the overlay. - and finally 'switch_root /root /sbin/init'.
In the /boot (FAT) partition I place the same Raspberry Pi files that CentOS 7 have in its own /boot. The only change is the line 'initramfs initramfs.gz' appended into the /boot/config.txt file.
Caveats. I'm still using the kernel version 4.9.80-v7.1 because the 4.14.43-v7.1 version panics randomly during boot. It looks like the kernel is unable to detect /dev/mmcblk0 sometimes. I've not tested the latest 4.14.111 yet.
1 - https://github.com/raspberrypi/target_fs
I hope that helps you. Regards.
On 6/4/19 8:16 PM, Seth Meyer wrote:
Hello,
A colleague and I are trying to get an ARM server to boot stateless (i.e. diskless). Has anyone had any success with this? Our primary source of inspiration is the documentation for the Pegasus cluster:
http://web.mst.edu/~vojtat/pegasus/administration.htm
Basically, the idea is to have the kernel load a custom initramfs that decompresses a file system image onto a tmpfs at boot time (along with some other setup), then execute a `switch_root` to said tmpfs.
We have succeeded in getting an x86_64 machine to boot (mostly) diskless, using the disk only for the boot process by creating a GRUB entry. However, when we try this with my ARM (ThunderX) server, the kernel panics during the boot process because it cannot find init. We have also tried passing `init=/init` as a kernel parameter with no luck. The error message is
Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
We looked at the referenced documentation and it's not a ton of help for my specific case. The initramfs environment is based on Busybox, which we compiled on a ThunderX machine (the same one we are trying to boot from, in fact).
We have considered compiling my own kernel with the initramfs built in, but we are trying to avoid doing that. Further, we have tried with the 3.10 and 4.4 kernels on x86, and with 4.5 and 4.14 kernels on aarch64.
Does anyone have any guidance? Also, please let us know if you need more information.
Best regards,
Marc Patton and Seth Meyer ARC-TS - https://arc-ts.umich.edu/ University of Michigan
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Hi Seth, Marc,
So you issues is to provide proper initramfs to kernel (and kernel I guess) to be booted on CN88XX by some variant of net-boot, correct ?
For the ThunderX you may use two options using Boot Manager menu configure boot-entry as one of following options: 1. Add new entry for you NIC (check it by it's MAC) and use PXE-boot, with configuring tftp-contents with grub, kernel and your initramfs which will then do what you need; 2. Add new entry for you NIC (check it by it's MAC) and use tftp-boot (some version of ThunderX FW had that option); in this case you'll need to manually configure IP address of tftpserver and, if I’m not mistaken, proper path to grubaa64.efi image (which should be like <tftp-root>/efi/grubaa64.efi) to let it boot your kernel and initramfs images;
Also while configuring grub at tftp server side for both cases, keep in mind that grub has prebuild ‘prefix’ parameter which could be critical for grub in order to find grub.cfg file and so on.
WBR, Vadim
From: Arm-dev [mailto:arm-dev-bounces@centos.org] On Behalf Of Seth Meyer Sent: Tuesday, June 4, 2019 9:17 PM To: arm-dev@centos.org; Marc Patton pattonmj@umich.edu Subject: [Arm-dev] Stateless CentOS on ARM question
Hello,
A colleague and I are trying to get an ARM server to boot stateless (i.e. diskless). Has anyone had any success with this? Our primary source of inspiration is the documentation for the Pegasus cluster:
http://web.mst.edu/~vojtat/pegasus/administration.htm
Basically, the idea is to have the kernel load a custom initramfs that decompresses a file system image onto a tmpfs at boot time (along with some other setup), then execute a `switch_root` to said tmpfs.
We have succeeded in getting an x86_64 machine to boot (mostly) diskless, using the disk only for the boot process by creating a GRUB entry. However, when we try this with my ARM (ThunderX) server, the kernel panics during the boot process because it cannot find init. We have also tried passing `init=/init` as a kernel parameter with no luck. The error message is
Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
We looked at the referenced documentation and it's not a ton of help for my specific case. The initramfs environment is based on Busybox, which we compiled on a ThunderX machine (the same one we are trying to boot from, in fact).
We have considered compiling my own kernel with the initramfs built in, but we are trying to avoid doing that. Further, we have tried with the 3.10 and 4.4 kernels on x86, and with 4.5 and 4.14 kernels on aarch64.
Does anyone have any guidance? Also, please let us know if you need more information.
Best regards,
Marc Patton and Seth Meyer ARC-TS - https://arc-ts.umich.edu/ University of Michigan
-
Jordi Sanfeliu -
Lomovtsev, Vadim -
R C -
Seth Meyer