Hello all,
I noticed many many filesystem corruptions, after raspberry 3B reboot. For instance, with image Raspberry 1804, "yum update and reboot" made my RPM DB corrupted rather often after reboot. We found some other scenarios, but all less easy to reproduce.
Somehow, we came to the conclusion that we had to start some kind of stressing tool, and this how it comes out: - install SD-Card with last CentOS image available (kernel 4.14.82) - create a 4Gb primary partition #4, starting at 2Gb with help of fdisk. - then, grow partition #3 with rootfs-expand. - format partition #4 as EXT4, with default parameters.
In a stress-loop - mount partition #4 - copy huge number of files (/usr/) to partition #4 (cd /usr ; tar -c . ) | (cd /mnt/ ; tar -x) - umount part #4 - fsck on partition #4 The script breaks the loop when FSCK returns status different than 0. Only a couple of iterations are enough to crash partition #4.
I just tried the same with EXT2, and result is identical.
Any idea to make this "system" less vulnerable ?
On 12/13/18 12:34 PM, Stephan Guilloux wrote:
Hello all,
I noticed many many filesystem corruptions, after raspberry 3B reboot. For instance, with image Raspberry 1804, "yum update and reboot" made my RPM DB corrupted rather often after reboot. We found some other scenarios, but all less easy to reproduce.
Somehow, we came to the conclusion that we had to start some kind of stressing tool, and this how it comes out:
- install SD-Card with last CentOS image available (kernel 4.14.82)
- create a 4Gb primary partition #4, starting at 2Gb with help of fdisk.
- then, grow partition #3 with rootfs-expand.
- format partition #4 as EXT4, with default parameters.
In a stress-loop
- mount partition #4
- copy huge number of files (/usr/) to partition #4
(cd /usr ; tar -c . ) | (cd /mnt/ ; tar -x)
- umount part #4
- fsck on partition #4
The script breaks the loop when FSCK returns status different than 0. Only a couple of iterations are enough to crash partition #4.
I just tried the same with EXT2, and result is identical.
Any idea to make this "system" less vulnerable ?
did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has
Le 13/12/2018 à 12:06, Manuel Wolfshant a écrit :
On 12/13/18 12:34 PM, Stephan Guilloux wrote:
Hello all,
I noticed many many filesystem corruptions, after raspberry 3B reboot. For instance, with image Raspberry 1804, "yum update and reboot" made my RPM DB corrupted rather often after reboot. We found some other scenarios, but all less easy to reproduce.
Somehow, we came to the conclusion that we had to start some kind of stressing tool, and this how it comes out:
- install SD-Card with last CentOS image available (kernel 4.14.82)
- create a 4Gb primary partition #4, starting at 2Gb with help of fdisk.
- then, grow partition #3 with rootfs-expand.
- format partition #4 as EXT4, with default parameters.
In a stress-loop
- mount partition #4
- copy huge number of files (/usr/) to partition #4
(cd /usr ; tar -c . ) | (cd /mnt/ ; tar -x)
- umount part #4
- fsck on partition #4
The script breaks the loop when FSCK returns status different than 0. Only a couple of iterations are enough to crash partition #4.
I just tried the same with EXT2, and result is identical.
Any idea to make this "system" less vulnerable ?
did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has
On the SD, I do not use the whole space. I manage to have only 15Gb used, at most, and even less (8Gb) for those tests.
We were also suspecting the SD, so, we tried with different card, but all of the same brand. We have 16 and 32 Gb SD, but the result seems similar for all.
Now stressing with a different brand... Stressing script is running for almost 2h with no PB. I'll let it run for the rest of the afternoon...
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Dec 13, 2018, at 06:06, Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has
++
Been there, done that! After seeing a large number of ‘infant mortalities’ with RaspPi 3+ setups using cheapie ‘no brand’ microSD cards, we made the switch to using only ‘name brand’ cards from reputable manufacturers. Problem solved.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Well... Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-) Something else added to the confusion, was an article on a Linux block layer issue...
OK. Now, at least, we have a "stressing" tool and a stack of RPI in case of doubt on SD :-)
By the way, is there any well known bench mark tool for SD, on CentOS or OpenSource ?
Le 13/12/2018 à 16:19, Fred Gleason a écrit :
On Dec 13, 2018, at 06:06, Manuel Wolfshant <wolfy@nobugconsulting.ro mailto:wolfy@nobugconsulting.ro> wrote:
did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has
++
Been there, done that! After seeing a large number of ‘infant mortalities’ with RaspPi 3+ setups using cheapie ‘no brand’ microSD cards, we made the switch to using only ‘name brand’ cards from reputable manufacturers. Problem solved.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Dec 13, 2018, at 11:54, Stephan Guilloux stephan.guilloux@crisalid.com wrote:
Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-)
Well, remember, we are dealing with flash RAM here. After enough write/erase cycles, it *will* wear out. The goal is to optimize the overall system to get maximum lifetime out of the media. For example, on our production setups we do things like putting ‘/var/log’ on tmpfs (it doesn’t matter for the particular application that the log data evaporates after reboots). That greatly reduces wear on the media, while still allowing log data to be accessed during a session for debug purposes.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | You can never tell which way the train went by looking at the | | tracks. | | -- Kramer's Law | |----------------------------------------------------------------------|
On Thu, 13 Dec 2018, 17:10 Fred Gleason <fredg@paravelsystems.com wrote:
On Dec 13, 2018, at 11:54, Stephan Guilloux stephan.guilloux@crisalid.com wrote:
Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-)
Well, remember, we are dealing with flash RAM here. After enough write/erase cycles, it *will* wear out. The goal is to optimize the overall system to get maximum lifetime out of the media. For example, on our production setups we do things like putting ‘/var/log’ on tmpfs (it doesn’t matter for the particular application that the log data evaporates after reboots). That greatly reduces wear on the media, while still allowing log data to be accessed during a session for debug purposes.
Great advice indeed. Unfortunately, many packages expect directories in /var/log to exist before they start and will break without them. A long time ago I wrote a service called tmpersist that takes care of that, but ultimately it was just left untriaged on RH bugzilla until it rotted - seems I was the only one around who cared.
Make sure /tmp and /var/tmp are also mounted on tmpfs.
On Dec 13, 2018, at 12:22, Gordan Bobic gordan@redsleeve.org wrote:
Great advice indeed. Unfortunately, many packages expect directories in /var/log to exist before they start and will break without them. A long time ago I wrote a service called tmpersist that takes care of that, but ultimately it was just left untriaged on RH bugzilla until it rotted - seems I was the only one around who cared.
IoT (and embedded applications in general) are quite far from the Upstream Provider’s primary envisioned use cases, unfortunately.
Make sure /tmp and /var/tmp are also mounted on tmpfs.
IIRC, they come that way out-of-the-box in the installation image.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | Lazlo's Chinese Relativity Axiom: | | No matter how great your triumphs or how tragic your defeats -- | | approximately one billion Chinese couldn't care less. | |----------------------------------------------------------------------|
On Thu, 13 Dec 2018, 17:32 Fred Gleason <fredg@paravelsystems.com wrote:
On Dec 13, 2018, at 12:22, Gordan Bobic gordan@redsleeve.org wrote:
Great advice indeed. Unfortunately, many packages expect directories in /var/log to exist before they start and will break without them. A long time ago I wrote a service called tmpersist that takes care of that, but ultimately it was just left untriaged on RH bugzilla until it rotted - seems I was the only one around who cared.
IoT (and embedded applications in general) are quite far from the Upstream Provider’s primary envisioned use cases, unfortunately.
Doesn't explain the disinterest from the Fedora crowd. Then again, they (and Linux developers who really should know better) don't care about alignment errors either.
On 12/13/18 12:09 PM, Fred Gleason wrote:
On Dec 13, 2018, at 11:54, Stephan Guilloux <stephan.guilloux@crisalid.com mailto:stephan.guilloux@crisalid.com> wrote:
Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-)
Well, remember, we are dealing with flash RAM here. After enough write/erase cycles, it *will* wear out. The goal is to optimize the overall system to get maximum lifetime out of the media. For example, on our production setups we do things like putting ‘/var/log’ on tmpfs (it doesn’t matter for the particular application that the log data evaporates after reboots). That greatly reduces wear on the media, while still allowing log data to be accessed during a session for debug purposes.
All too often I am looking at contents of /var/log after a system restart to figure out what went wrong.
So that is yet another reason I am sticking with SOCs that have good sata support. So I can run rootfs there...
If there is another, bigger system on the local network, you can always syslog the logs to that. That's what it was designed for.
On Thu, 13 Dec 2018, 18:27 Robert Moskowitz <rgm@htt-consult.com wrote:
On 12/13/18 12:09 PM, Fred Gleason wrote:
On Dec 13, 2018, at 11:54, Stephan Guilloux stephan.guilloux@crisalid.com wrote:
Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-)
Well, remember, we are dealing with flash RAM here. After enough write/erase cycles, it *will* wear out. The goal is to optimize the overall system to get maximum lifetime out of the media. For example, on our production setups we do things like putting ‘/var/log’ on tmpfs (it doesn’t matter for the particular application that the log data evaporates after reboots). That greatly reduces wear on the media, while still allowing log data to be accessed during a session for debug purposes.
All too often I am looking at contents of /var/log after a system restart to figure out what went wrong.
So that is yet another reason I am sticking with SOCs that have good sata support. So I can run rootfs there...
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On 13-Dec-18 18:09, Fred Gleason wrote:
On Dec 13, 2018, at 11:54, Stephan Guilloux <stephan.guilloux@crisalid.com mailto:stephan.guilloux@crisalid.com> wrote:
Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-)
Well, remember, we are dealing with flash RAM here. After enough write/erase cycles, it *will* wear out. The goal is to optimize the overall system to get maximum lifetime out of the media. For example, on our production setups we do things like putting ‘/var/log’ on tmpfs (it doesn’t matter for the particular application that the log data evaporates after reboots). That greatly reduces wear on the media, while still allowing log data to be accessed during a session for debug purposes.
/tmp, /var/log, /var/tmp (and a few others) need TMPFS, yes.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | You can never tell which way the train went by looking at the | | tracks. | | -- Kramer's Law | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
You are likely to find that stress testing will kill most SD cards within a day or two. They simply aren't designed for small write workload. They are designed for large streaming writes (and they are fine for reads, obviously).
"Endurance" models fare much better, but you will still kill then quickly with continuous small random writes.
If you want reasonable performance and longevity out of them, use nilfs2 or ZFS. Otherwise they are only suitable for workloads that are not random-write intensive in the way that traditional UNIX file system workloads are.
On Thu, 13 Dec 2018, 16:54 Stephan Guilloux <stephan.guilloux@crisalid.com wrote:
Well... Stress test still running on reputable brand with no PB, after 6 or 7h.
During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-) Something else added to the confusion, was an article on a Linux block layer issue...
OK. Now, at least, we have a "stressing" tool and a stack of RPI in case of doubt on SD :-)
By the way, is there any well known bench mark tool for SD, on CentOS or OpenSource ?
Le 13/12/2018 à 16:19, Fred Gleason a écrit :
On Dec 13, 2018, at 06:06, Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has
++
Been there, done that! After seeing a large number of ‘infant mortalities’ with RaspPi 3+ setups using cheapie ‘no brand’ microSD cards, we made the switch to using only ‘name brand’ cards from reputable manufacturers. Problem solved.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Arm-dev mailing listArm-dev@centos.orghttps://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On 13-Dec-18 18:16, Gordan Bobic wrote:
You are likely to find that stress testing will kill most SD cards within a day or two. They simply aren't designed for small write workload. They are designed for large streaming writes (and they are fine for reads, obviously).
"Endurance" models fare much better, but you will still kill then quickly with continuous small random writes.
Starting point was exactly this. We saw FS corruption very lately, with very strange effects (especially when you are used to only SATA drives). One of them can be depicted like a ReadOnly FS not so ReadOnly ... Once formatted, the SD was working fine for a while.
So, trying to narrow that down, mixing 3B and 3B+, with many devices, many different HATS, many different stuff, it came to the stress test and mount/copy/umount/fsck scenario, but yes, it took quite some time :-)
The stress test will help us to perform some kind of qualification, for any new SD...
If you want reasonable performance and longevity out of them, use nilfs2 or ZFS. Otherwise they are only suitable for workloads that are not random-write intensive in the way that traditional UNIX file system workloads are.
On Thu, 13 Dec 2018, 16:54 Stephan Guilloux <stephan.guilloux@crisalid.com mailto:stephan.guilloux@crisalid.com wrote:
Well... Stress test still running on reputable brand with no PB, after 6 or 7h. During this stressing session, we found a few broken SD, from another reputable brand. Not all cards, though, but enough to put some confusion... ;-) Something else added to the confusion, was an article on a Linux block layer issue... OK. Now, at least, we have a "stressing" tool and a stack of RPI in case of doubt on SD :-) By the way, is there any well known bench mark tool for SD, on CentOS or OpenSource ? Le 13/12/2018 à 16:19, Fred Gleason a écrit :On Dec 13, 2018, at 06:06, Manuel Wolfshant <wolfy@nobugconsulting.ro <mailto:wolfy@nobugconsulting.ro>> wrote:did this happen with multiple SD-cards ? if not, I strongly suspect that the issue is that particular card , not the OS. it rhymes extremely well with either a faulty card or a card whose firmware was doctored to report a larger size than the one it really has++ Been there, done that! After seeing a large number of ‘infant mortalities’ with RaspPi 3+ setups using cheapie ‘no brand’ microSD cards, we made the switch to using only ‘name brand’ cards from reputable manufacturers. Problem solved. Cheers! |----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------| _______________________________________________ Arm-dev mailing list Arm-dev@centos.org <mailto:Arm-dev@centos.org> https://lists.centos.org/mailman/listinfo/arm-dev_______________________________________________ Arm-dev mailing list Arm-dev@centos.org <mailto:Arm-dev@centos.org> https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Thu, Dec 13, 2018 at 7:59 PM Stephan GUILLOUX stephan.guilloux@free.fr wrote:
On 13-Dec-18 18:16, Gordan Bobic wrote:
You are likely to find that stress testing will kill most SD cards within a day or two. They simply aren't designed for small write workload. They are designed for large streaming writes (and they are fine for reads, obviously).
"Endurance" models fare much better, but you will still kill then quickly with continuous small random writes.
Starting point was exactly this. We saw FS corruption very lately, with very strange effects (especially when you are used to only SATA drives). One of them can be depicted like a ReadOnly FS not so ReadOnly ... Once formatted, the SD was working fine for a while.
So, trying to narrow that down, mixing 3B and 3B+, with many devices, many different HATS, many different stuff, it came to the stress test and mount/copy/umount/fsck scenario, but yes, it took quite some time :-)
The stress test will help us to perform some kind of qualification, for any new SD...
I suggest you only batch test models, rather than testing each end every card in that way. You might find that the stress test uses up most of their useful life. To put it into perspective, even some proper SSDs from reputable brands use NAND made in a process so small that it's only good for 75 erase cycles (e.g. 1TB Crucial BX100 SSDs). It's not necessarily a problem with good flash management (I've had a pair in my main workstation for 3-4 years now in 24/7 regular workstation use, and they are showing something like 2-3% wear), but with the quality of flash management you can expect from a controller small enough to fit into an SD card, that's going to cause problems. Worst case scenario, you could be looking at 100x write amplification to the underlying NAND.
In case, some more info ...
F3 (Fight Fraud Flash) for Linux and MAC OS: https://github.com/AltraMayor/f3%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C... (version 7.0 is available in EPEL) Provides a few tools to check your SD, mainly the SD card size.
Linux can help to read SD info, via a bunch of /sys files: /sys/block/mmcblk0/alignment_offset:0 /sys/block/mmcblk0/capability:50 /sys/block/mmcblk0/dev:179:0 /sys/block/mmcblk0/discard_alignment:0 /sys/block/mmcblk0/ext_range:256 /sys/block/mmcblk0/force_ro:0 /sys/block/mmcblk0/inflight: 0 0 /sys/block/mmcblk0/range:32 /sys/block/mmcblk0/removable:0 /sys/block/mmcblk0/ro:0 /sys/block/mmcblk0/size:249737216 /sys/block/mmcblk0/stat: 536330 94424 116053382 7343060 446532 512290 83512784 532542040 0 13771000 540108990 /sys/block/mmcblk0/uevent:MAJOR=179 /sys/block/mmcblk0/uevent:MINOR=0 /sys/block/mmcblk0/uevent:DEVNAME=mmcblk0 /sys/block/mmcblk0/uevent:DEVTYPE=disk
/sys/block/mmcblk0/device/cid:413432534443495430002d858f01229d /sys/block/mmcblk0/device/csd:400e00325b590000e93f7f800a400063 /sys/block/mmcblk0/device/date:02/2018 /sys/block/mmcblk0/device/dsr:0x404 /sys/block/mmcblk0/device/erase_size:512 /sys/block/mmcblk0/device/fwrev:0x0 /sys/block/mmcblk0/device/hwrev:0x3 /sys/block/mmcblk0/device/manfid:0x000041 /sys/block/mmcblk0/device/name:SDCIT /sys/block/mmcblk0/device/ocr:0x00200000 /sys/block/mmcblk0/device/oemid:0x3432 /sys/block/mmcblk0/device/preferred_erase_size:4194304 /sys/block/mmcblk0/device/scr:0235800201000000 /sys/block/mmcblk0/device/serial:0x002d858f /sys/block/mmcblk0/device/ssr:00000000050000000400900200ab1f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 /sys/block/mmcblk0/device/type:SD /sys/block/mmcblk0/device/uevent:DRIVER=mmcblk /sys/block/mmcblk0/device/uevent:MMC_TYPE=SD /sys/block/mmcblk0/device/uevent:MMC_NAME=SDCIT /sys/block/mmcblk0/device/uevent:MODALIAS=mmc:block
Some details on these fields can be found here: https://www.kernel.org/doc/Documentation/mmc/mmc-dev-attrs.txt
In our case, some of these files helped to identify a few fake SD: - some SD 32Gb built in 2008 (bought this year) !!, - some SD with a dummy serial number, - ...
And below, some other info, mostly for some ID and manufacturers: https://www.cameramemoryspeed.com/sd-memory-card-faq/reading-sd-card-cid-ser...
Le 13/12/2018 à 11:34, Stephan Guilloux a écrit :
Hello all,
I noticed many many filesystem corruptions, after raspberry 3B reboot. For instance, with image Raspberry 1804, "yum update and reboot" made my RPM DB corrupted rather often after reboot. We found some other scenarios, but all less easy to reproduce.
Somehow, we came to the conclusion that we had to start some kind of stressing tool, and this how it comes out:
- install SD-Card with last CentOS image available (kernel 4.14.82)
- create a 4Gb primary partition #4, starting at 2Gb with help of fdisk.
- then, grow partition #3 with rootfs-expand.
- format partition #4 as EXT4, with default parameters.
In a stress-loop
- mount partition #4
- copy huge number of files (/usr/) to partition #4
(cd /usr ; tar -c . ) | (cd /mnt/ ; tar -x)
- umount part #4
- fsck on partition #4
The script breaks the loop when FSCK returns status different than 0. Only a couple of iterations are enough to crash partition #4.
I just tried the same with EXT2, and result is identical.
Any idea to make this "system" less vulnerable ?
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
-
Fred Gleason -
Gordan Bobic -
Manuel Wolfshant -
Robert Moskowitz -
Stephan Guilloux -
Stephan Guilloux