CentOS 6
Hi All:
Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific.
My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis.
The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Does anyone have a suggestion for a better way of doing this?
TIA
Regards, Hugh
On Tue, Apr 21, 2015 at 9:38 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
CentOS 6
My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis.
The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Does anyone have a suggestion for a better way of doing this?
Take a look at 'man tune2fs' and 'man fstab' for modifying the fsck order in your system.
HTH, -- Arun Khan
From: Arun Khan Sent: April 20, 2015 23:49
Take a look at 'man tune2fs' and 'man fstab' for modifying the fsck order in your system.
Thanks but I did look at those and I was not able to find anything that would limit the fsck executions to one per reboot. Changing the order of execution will not address my concern.
Regards, Hugh
On 4/20/2015 9:08 PM, Hugh E Cruickshank wrote:
The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Does anyone have a suggestion for a better way of doing this?
use XFS, no fsck's until/unless something catastrophic happens to the file system.
From: John R Pierce Sent: April 20, 2015 23:58
On 4/20/2015 9:08 PM, Hugh E Cruickshank wrote:
The second idea was to set each filesystem to a different
random count
value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Does anyone have a suggestion for a better way of doing this?
use XFS, no fsck's until/unless something catastrophic happens to the file system.
Thanks. That would avoid the problem on future systems.
Regards, Hugh
On 04/21/2015 06:08 AM, Hugh E Cruickshank wrote:
CentOS 6
Hi All:
Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific.
My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis.
The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Does anyone have a suggestion for a better way of doing this?
TIA
Regards, Hugh
Using "tune2fs -c", set the max-mount-counts to a different prime number for each filesystem. So e.g. instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. This way, three or more fscks on the same boot are quite unlikely.
HTH,
Kay
From: Kay Diederichs Sent: April 21, 2015 03:43
On 04/21/2015 06:08 AM, Hugh E Cruickshank wrote:
The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.
Using "tune2fs -c", set the max-mount-counts to a different prime number for each filesystem. So e.g. instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. This way, three or more fscks on the same boot are quite unlikely.
Thanks but that is not much different then my second idea and does not fully avoid the problem.
Regards, Hugh
On Apr 21, 2015, at 9:50 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
From: Kay Diederichs Sent: April 21, 2015 03:43
instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29.
Thanks but that is not much different then my second idea and does not fully avoid the problem.
You may be missing a key fact of how prime numbers work.
You can only get two or more fscks on a single reboot when the mount count is a multiple of two or more of the max-mount-count values. When those numbers are all prime, the frequency of such occurrences is much lower than when you use purely random values.
With the four values that Kay provided, I calculate a 1.2% chance on average that two or more volumes will need to be checked on the same reboot. If you reboot on average once a month, that means it only happens once every 7 years or so. That means the machine may well be retired before it happens even once, and then only if you reboot that often in the first place.
If you take the same set of values and add one to them to make them all even, and thus composite numbers, the chance rises to 3.3%, or about once every 2.5 years for monthly reboots. Thus, Kay’s solution is actually more than twice as good as your second solution.
Interestingly, the numbers don’t actually have to be prime to take advantage of this property of integers. They just have to be *relatively* prime: http://goo.gl/bQbu5Z
For example, the set {15, 19, 22, 23} isn’t all prime, but is *relatively* prime, since there is no integer other than 1 that evenly divides any pair among the set. This set gives about a 1.5% chance of a 2+ volume collision, nearly as good as Kay’s prime set.
I’ve put the calculator I built to test this up here:
On Apr 21, 2015, at 3:12 PM, Warren Young wyml@etr-usa.com wrote:
With the four values that Kay provided, I calculate a 1.2% chance on average that two or more volumes will need to be checked on the same reboot.
Ooops, forgot to mention one other minor detail:
This calculator gives the chance or a 2+ volume simultaneous check, which is actually a much more stringent case than you ran into. If you have four volumes and are using a relatively prime max-mount-count set, you won’t get a simultaneous check of all four volumes until your reboot count equals the *product* of all values in the set.
With Kay’s prime set, that means it won’t happen until you have rebooted 215,441 times. If your machine lives 10 years, that’s enough to allow for about 60 reboots a day while still having *zero* chance of a 4-volume fsck.
I think it’s fair to say that Kay’s solution (or a relatively-prime set) does actually solve the problem.
From: Warren Young Sent: April 21, 2015 14:13
On Apr 21, 2015, at 9:50 AM, Hugh E Cruickshank wrote:
From: Kay Diederichs Sent: April 21, 2015 03:43
instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to
23, and the
fourth to 29.
Thanks but that is not much different then my second idea
and does not
fully avoid the problem.
You may be missing a key fact of how prime numbers work.
You are right and I stand corrected.
I have done some "what if" testing. Assuming 8 filesystems and weekly reboots over a ten year period...
The random numbers would result in as many as 40 weeks per year where 2, 3 or 4 fscks would be run in the same week depending on the random numbers selected.
Using the prime numbers 7, 11, 13, 17, 19, 23, 29 and 31 the is a maximum of 7 incidents per year of 2 fscks per week and none for 3 or more.
Clearly the prime numbers are better.
Thank, Hugh
On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
I have done some "what if" testing.
Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share?
I’ve updated mine to break out the stats for 3+ volumes instead of just reporting all multi-volume fscks together:
https://gist.github.com/wyoung/7c94967bb635de48d058
Then I rewrote that in C++, since these 8-volume simulations were literally going to take days with the Perl version:
https://gist.github.com/wyoung/966383b4efbb63aafc71
(The Perl version is about 1/5 the speed of the C++ one. This actually isn’t all that bad, considering that it’s an interpreted dynamically-typed language going up against a statically-typed native-code compiled language.)
Using the prime numbers 7, 11, 13, 17, 19, 23, 29 and 31 the is a maximum of 7 incidents per year of 2 fscks per week and none for 3 or more.
This is why I pointed out that you only need *relatively* prime numbers: so that if you decide the largest max-mount-count can’t be over 31, you don’t have to go clear down to 7 in order to find the last prime for 8 volumes.
Using relatively-prime numbers, you can skew the set upwards quite a bit without increasing the largest value. The most efficient set I’ve been able to come up with is:
17, 19, 23, 25, 27, 28, 29, 31
The three composite values (25, 27, and 28) do not share any common factors: 25 uses 5 twice, 27 uses 3 thrice, and 28 uses 7 plus twice 2.
My newer simulators give these results for the chances of a multi-volume fsck with your prime set:
period: ~6.7 billion 2-volume: 8.12% 3-volume: 1.08% 4-volume: 0.09% >5: < 0.01% chance total: 9.3%
My relatively prime improved set gives these results because the set’s median is higher while keeping the same maximum, while also avoiding any reuse of prime factors:
period: ~126.2 billion 2-volume: 0.37% 3-volume: 0.33% 4-volume: 0.02% >5: < 0.01% chance total: 0.7%
See? Number theory *is* useful in real life. :)
From: Warren Young Sent: April 22, 2015 20:46
On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
I have done some "what if" testing.
Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share?
I cobbled something together in OpenEdge ABL. I have uploaded it to
This was intended only for my use so, while the code is relatively clean, it is not documented.
Regards, Hugh
On Apr 23, 2015, at 8:49 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
From: Warren Young Sent: April 22, 2015 20:46
On Apr 22, 2015, at 11:56 AM, Hugh E Cruickshank hugh@forsoft.com wrote:
I have done some "what if" testing.
Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share?
I cobbled something together in OpenEdge ABL. I have uploaded it to
Thanks!
I like how it saves work by taking real world conditions — service life of the machine and reboot frequency — into account. Mine just computes all of the possibilities, even if in practice it would require a machine a million years old to hit them all.
From: Hugh E Cruickshank Sent: April 20, 2015 21:09
Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific.
I have come up with a third idea that would seem to address what I am looking for...
1. Create a file with the list of filesystems in desired order of execution.
2. Create an init.d script that:
a. Sets tune2fs -c 0 on all filesystems. b. Extracts the first filesystem from the file, c. Sets tune2fs -c 1 on it, and d. moves it to the end of the file.
The result is that on each reboot (after the first) only one filesystem will be checked on each boot. The down side is that an fsck will be run on every reboot however this could be mitigated by adding a boot count that would be maintained and checked in the file.
It would appear that I have the beginnings of a workable solution.
Thanks for everyone's suggestions.
Regards, Hugh
-
Arun Khan -
Hugh E Cruickshank -
John R Pierce -
Kay Diederichs -
Warren Young