It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last time: # grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Wrong architecture binaries, in the chroot'ed folder ?
What do you get with:
file <chroot>/bin/bash
On 26-Jan-19 17:49, Gordan Bobic wrote:
It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last time: # grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Yes, they are "wrong", but not "WRONG!".
As I (possibly poorly) explained, I'm running on an aarch64 machine, but am trying to run armv5tel and armv7hl chroots. I just dug out an old kernel I was using (4.4.72) and I can now chroot just fine.
So it's something that changed in the kernel, but I'm not sure what else it could be other than the page size (which I already checked).
I'm going to build the latest 4.4.172 with my 4.4.72 config and see if that works. If it does, I'll try to make oldconfig that on 4.9.153 and try again, see if that breaks things again. I was just hoping there might be an option I missed in 4.9 that enables/disables downward compatibility.
Gordan
On Sat, Jan 26, 2019 at 5:41 PM Stephan GUILLOUX stephan.guilloux@free.fr wrote:
Wrong architecture binaries, in the chroot'ed folder ?
What do you get with:
file <chroot>/bin/bashOn 26-Jan-19 17:49, Gordan Bobic wrote:
It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last
time:
# grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Hmmm... I'm not that familiar with ARM architectures.
There should be a 32-bits binaries support or compat, at least, to allow armv5 and armv7, no ?
I think I would get the config of working and not working kernels.
Then, sort both to ease comparison ...
That helped me, sometimes back for x86.
On 26-Jan-19 18:44, Gordan Bobic wrote:
Yes, they are "wrong", but not "WRONG!".
As I (possibly poorly) explained, I'm running on an aarch64 machine, but am trying to run armv5tel and armv7hl chroots. I just dug out an old kernel I was using (4.4.72) and I can now chroot just fine.
So it's something that changed in the kernel, but I'm not sure what else it could be other than the page size (which I already checked).
I'm going to build the latest 4.4.172 with my 4.4.72 config and see if that works. If it does, I'll try to make oldconfig that on 4.9.153 and try again, see if that breaks things again. I was just hoping there might be an option I missed in 4.9 that enables/disables downward compatibility.
Gordan
On Sat, Jan 26, 2019 at 5:41 PM Stephan GUILLOUX <stephan.guilloux@free.fr mailto:stephan.guilloux@free.fr> wrote:
Wrong architecture binaries, in the chroot'ed folder ? What do you get with: file <chroot>/bin/bash On 26-Jan-19 17:49, Gordan Bobic wrote: > It's been so long since I last had to do this, that it looks like a > kernel update somewhere along the way broke this for me. :-( > > What happens now: > > # chroot rs7 > chroot: failed to run command ‘/bin/bash’: Exec format error > > I could have sworn that this was all I needed to make this work last time: > # grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 > CONFIG_ARM64_4K_PAGES=y > # CONFIG_ARM64_64K_PAGES is not set > > Unfortunately, now trying to chroot into either hard-float or > soft-float chroot results in the exec format error. > > Looking at time stamps of various things, I had this working with > 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what > changed between 4.4 and 4.9 would be most appreciated. > > What am I missing? > > _______________________________________________ > Arm-dev mailing list > Arm-dev@centos.org <mailto:Arm-dev@centos.org> > https://lists.centos.org/mailman/listinfo/arm-dev _______________________________________________ Arm-dev mailing list Arm-dev@centos.org <mailto:Arm-dev@centos.org> https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Sat, 26 Jan 2019 at 18:45, Gordan Bobic gordan@redsleeve.org wrote:
Yes, they are "wrong", but not "WRONG!".
As I (possibly poorly) explained, I'm running on an aarch64 machine, but am trying to run armv5tel and armv7hl chroots. I just dug out an old kernel I was using (4.4.72) and I can now chroot just fine.
So it's something that changed in the kernel, but I'm not sure what else it could be other than the page size (which I already checked).
I'm going to build the latest 4.4.172 with my 4.4.72 config and see if that works. If it does, I'll try to make oldconfig that on 4.9.153 and try again, see if that breaks things again. I was just hoping there might be an option I missed in 4.9 that enables/disables downward compatibility.
The places I have seen where this breaks before was * it works on only specific aarch64 devices * it works but only after you update the firmware to a newer vendor version (because the kernel expects that to fix something else) * it is a regression and needs to be fixed.
[The first one was where someone remembered it working but forgot they were working on XYZ box and then replicated in ABC and found the arch didn't have that.]
I am expecting it is going to be 3... but I would check on 2 also.
Gordan
On Sat, Jan 26, 2019 at 5:41 PM Stephan GUILLOUX stephan.guilloux@free.fr wrote:
Wrong architecture binaries, in the chroot'ed folder ?
What do you get with:
file <chroot>/bin/bashOn 26-Jan-19 17:49, Gordan Bobic wrote:
It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last
time:
# grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
I did a bit more digging, and rebuilding 4.9.153 with CONFIG_COMPAT and a few related options, had an effect - no longer do I get "exec format error" - now it just outright segfaults. 4.4.172 works perfectly.
I'm using clean mainline kernels with no additional patches.
chrooting to aarch64 chroots works.
On Sat, Jan 26, 2019 at 6:07 PM Stephen John Smoogen smooge@gmail.com wrote:
On Sat, 26 Jan 2019 at 18:45, Gordan Bobic gordan@redsleeve.org wrote:
Yes, they are "wrong", but not "WRONG!".
As I (possibly poorly) explained, I'm running on an aarch64 machine, but am trying to run armv5tel and armv7hl chroots. I just dug out an old kernel I was using (4.4.72) and I can now chroot just fine.
So it's something that changed in the kernel, but I'm not sure what else it could be other than the page size (which I already checked).
I'm going to build the latest 4.4.172 with my 4.4.72 config and see if that works. If it does, I'll try to make oldconfig that on 4.9.153 and try again, see if that breaks things again. I was just hoping there might be an option I missed in 4.9 that enables/disables downward compatibility.
The places I have seen where this breaks before was
- it works on only specific aarch64 devices
- it works but only after you update the firmware to a newer vendor
version (because the kernel expects that to fix something else)
- it is a regression and needs to be fixed.
[The first one was where someone remembered it working but forgot they were working on XYZ box and then replicated in ABC and found the arch didn't have that.]
I am expecting it is going to be 3... but I would check on 2 also.
Gordan
On Sat, Jan 26, 2019 at 5:41 PM Stephan GUILLOUX < stephan.guilloux@free.fr> wrote:
Wrong architecture binaries, in the chroot'ed folder ?
What do you get with:
file <chroot>/bin/bashOn 26-Jan-19 17:49, Gordan Bobic wrote:
It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last
time:
# grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on what changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
-- Stephen J Smoogen.
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Figured it out. Two kernel configuration options were in play:
CONFIG_COMPAT=y CONFIG_LSM_MMAP_MIN_ADDR=32768
Above is what is required to run armv5tel and armv7hl chroots on aarch64. Default for the latter option is 65535, which prevents 32-bit binaries from running. Up and running with 4.9.153 now.
I hope this is useful to someone else.
On Sat, Jan 26, 2019 at 8:08 PM Gordan Bobic gordan@redsleeve.org wrote:
I did a bit more digging, and rebuilding 4.9.153 with CONFIG_COMPAT and a few related options, had an effect - no longer do I get "exec format error"
- now it just outright segfaults.
4.4.172 works perfectly.
I'm using clean mainline kernels with no additional patches.
chrooting to aarch64 chroots works.
On Sat, Jan 26, 2019 at 6:07 PM Stephen John Smoogen smooge@gmail.com wrote:
On Sat, 26 Jan 2019 at 18:45, Gordan Bobic gordan@redsleeve.org wrote:
Yes, they are "wrong", but not "WRONG!".
As I (possibly poorly) explained, I'm running on an aarch64 machine, but am trying to run armv5tel and armv7hl chroots. I just dug out an old kernel I was using (4.4.72) and I can now chroot just fine.
So it's something that changed in the kernel, but I'm not sure what else it could be other than the page size (which I already checked).
I'm going to build the latest 4.4.172 with my 4.4.72 config and see if that works. If it does, I'll try to make oldconfig that on 4.9.153 and try again, see if that breaks things again. I was just hoping there might be an option I missed in 4.9 that enables/disables downward compatibility.
The places I have seen where this breaks before was
- it works on only specific aarch64 devices
- it works but only after you update the firmware to a newer vendor
version (because the kernel expects that to fix something else)
- it is a regression and needs to be fixed.
[The first one was where someone remembered it working but forgot they were working on XYZ box and then replicated in ABC and found the arch didn't have that.]
I am expecting it is going to be 3... but I would check on 2 also.
Gordan
On Sat, Jan 26, 2019 at 5:41 PM Stephan GUILLOUX < stephan.guilloux@free.fr> wrote:
Wrong architecture binaries, in the chroot'ed folder ?
What do you get with:
file <chroot>/bin/bashOn 26-Jan-19 17:49, Gordan Bobic wrote:
It's been so long since I last had to do this, that it looks like a kernel update somewhere along the way broke this for me. :-(
What happens now:
# chroot rs7 chroot: failed to run command ‘/bin/bash’: Exec format error
I could have sworn that this was all I needed to make this work last
time:
# grep 4K_PAGES config-4.9.153-1.el7.centos.aarch64 CONFIG_ARM64_4K_PAGES=y # CONFIG_ARM64_64K_PAGES is not set
Unfortunately, now trying to chroot into either hard-float or soft-float chroot results in the exec format error.
Looking at time stamps of various things, I had this working with 4.4.70, but have since switched to 4.9.x kernels. Any thoughts on
what
changed between 4.4 and 4.9 would be most appreciated.
What am I missing?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
-- Stephen J Smoogen.
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On 27/01/2019 13:18, Gordan Bobic wrote:
Figured it out. Two kernel configuration options were in play:
CONFIG_COMPAT=y CONFIG_LSM_MMAP_MIN_ADDR=32768
Above is what is required to run armv5tel and armv7hl chroots on aarch64. Default for the latter option is 65535, which prevents 32-bit binaries from running. Up and running with 4.9.153 now.
<snip>
For the centos armhfp/armv7hl builds, instead of using chroot'ed env directly on the aarch64 nodes, we just converted those into hypervisors and run armhfp VMs on top.
But, as Stephen also pointed out, it unfortunately only works on some aarch64 cpus and not all of them : I realized that when I wanted to use a thunderX node as replacement for a mustang board and couldn't get any armhfp VM to start : then, after some discussion with ARM specialists, they confirm that, per hardware design, the thunderx (and also thunderx2) nodes can't run 32 bits at all, so no way to get armhfp VMs running on that anymore.
That means that so far, the only boards on which we can still use the "armhfp VMs on top of aarch64 nodes" build process is on mustang or merlin (so basically X-Gene and X-Gene 2)
I've heard also that the new Ampere eMAG should be able to also do that, but unfortunately, haven't verified (no access to such hardware)
On Mon, 28 Jan 2019, 11:15 Fabian Arrotin <arrfab@centos.org wrote:
On 27/01/2019 13:18, Gordan Bobic wrote:
Figured it out. Two kernel configuration options were in play:
CONFIG_COMPAT=y CONFIG_LSM_MMAP_MIN_ADDR=32768
Above is what is required to run armv5tel and armv7hl chroots on aarch64. Default for the latter option is 65535, which prevents 32-bit binaries from running. Up and running with 4.9.153 now.
<snip>
For the centos armhfp/armv7hl builds, instead of using chroot'ed env directly on the aarch64 nodes, we just converted those into hypervisors and run armhfp VMs on top.
Full fat virtualization tax at ~25%+ at full saturation load is too rich for me.
And besides, isn't containerization/docker/kubernettes all the drvops-as-hell buzzword rage these days? ;-)
But, as Stephen also pointed out, it unfortunately only works on some
aarch64 cpus and not all of them : I realized that when I wanted to use a thunderX node as replacement for a mustang board and couldn't get any armhfp VM to start : then, after some discussion with ARM specialists, they confirm that, per hardware design, the thunderx (and also thunderx2) nodes can't run 32 bits at all, so no way to get armhfp VMs running on that anymore.
It's always important to pick the right set of tools for the job. :-)
That means that so far, the only boards on which we can still use the "armhfp VMs on top of aarch64 nodes" build process is on mustang or merlin (so basically X-Gene and X-Gene 2)
Yup, that's what I have.
I've heard also that the new Ampere eMAG should be able to also do that, but unfortunately, haven't verified (no access to such hardware)
My experience is that despite all the song and dance that makes the news when a manufacturer announces a big ARM server, they are either unavailable or ludicrously expensive compared to x86. Gigabyte MP30 AR0/AR1 is to date the only reasonable option I found for anything much bigger than the various Pi and clones (and I like my 128GB of RAM for large scale compile jobs).
-
Fabian Arrotin -
Gordan Bobic -
Stephan GUILLOUX -
Stephen John Smoogen