I have a CentOS-6.2 server in Italy. Every few months the server gets into a strange state and stops working after a power outage. I should say that this does not occur after every power outage; I suspect, but have no real evidence for this, that if the power comes back too quickly then the machine gets confused because it is in the middle of closing down.
If I am in Italy the solution is simple; I just press the power button on the machine until it stops and then switch it on again, and everything works fine.
I guess that the proper solution is to get a UPS. PowerTrust (or Trust) UPS's are available at a reasonable price (€40-€50). But I have one question about this solution. What if the outage lasts longer than the "life" of the UPS, about 10 minutes? I'd actually like the UPS simply to halt the machine in the proper way after say 1 minute; but can this be arranged? (If the machine knew it had gone onto UPS, this could be arranged fairly simply.)
There is one other possibility. There is a Billion modem/router between the ADSL input and the computer. This does not seem to have any problem with power outages. So if it were possible to send a message to the router which would cause the computer to be turned off and on again, that might offer an alternative solution. One problem with this is that the Italian ISP only offers a dynamic IP address. I run ddclient with dyndns on the server to translate this into a fixed domain-name. But this means that any action of this kind would have to be taken before the IP address changes. I think it usually stays the same for several days, but I guess this might not be true if there is an outage at the ISP's site, during a big electric storm.
Any suggestions, advice or experience of this problem gratefully received.
get a ups that has a usb connection. What you can do then is when you loose power after x amount of time the ups software will gracefully shutdown the server automatically. some ups software will even start the server back up once a sufficient charge has been obtained.
On 3/24/2012 7:33 AM, Timothy Murphy wrote:
I have a CentOS-6.2 server in Italy. Every few months the server gets into a strange state and stops working after a power outage. I should say that this does not occur after every power outage; I suspect, but have no real evidence for this, that if the power comes back too quickly then the machine gets confused because it is in the middle of closing down.
If I am in Italy the solution is simple; I just press the power button on the machine until it stops and then switch it on again, and everything works fine.
I guess that the proper solution is to get a UPS. PowerTrust (or Trust) UPS's are available at a reasonable price (€40-€50). But I have one question about this solution. What if the outage lasts longer than the "life" of the UPS, about 10 minutes? I'd actually like the UPS simply to halt the machine in the proper way after say 1 minute; but can this be arranged? (If the machine knew it had gone onto UPS, this could be arranged fairly simply.)
There is one other possibility. There is a Billion modem/router between the ADSL input and the computer. This does not seem to have any problem with power outages. So if it were possible to send a message to the router which would cause the computer to be turned off and on again, that might offer an alternative solution. One problem with this is that the Italian ISP only offers a dynamic IP address. I run ddclient with dyndns on the server to translate this into a fixed domain-name. But this means that any action of this kind would have to be taken before the IP address changes. I think it usually stays the same for several days, but I guess this might not be true if there is an outage at the ISP's site, during a big electric storm.
Any suggestions, advice or experience of this problem gratefully received.
On Sat, Mar 24, 2012 at 7:33 AM, Timothy Murphy gayleard@alice.it wrote:
I have a CentOS-6.2 server in Italy. Every few months the server gets into a strange state and stops working after a power outage. I should say that this does not occur after every power outage; I suspect, but have no real evidence for this, that if the power comes back too quickly then the machine gets confused because it is in the middle of closing down.
If I am in Italy the solution is simple; I just press the power button on the machine until it stops and then switch it on again, and everything works fine.
I guess that the proper solution is to get a UPS. PowerTrust (or Trust) UPS's are available at a reasonable price (€40-€50). But I have one question about this solution. What if the outage lasts longer than the "life" of the UPS, about 10 minutes? I'd actually like the UPS simply to halt the machine in the proper way after say 1 minute; but can this be arranged? (If the machine knew it had gone onto UPS, this could be arranged fairly simply.)
There is one other possibility. There is a Billion modem/router between the ADSL input and the computer. This does not seem to have any problem with power outages. So if it were possible to send a message to the router which would cause the computer to be turned off and on again, that might offer an alternative solution. One problem with this is that the Italian ISP only offers a dynamic IP address. I run ddclient with dyndns on the server to translate this into a fixed domain-name. But this means that any action of this kind would have to be taken before the IP address changes. I think it usually stays the same for several days, but I guess this might not be true if there is an outage at the ISP's site, during a big electric storm.
Any suggestions, advice or experience of this problem gratefully received.
The solution here is to purchase a UPS. I prefer APC, but there are other brands that work great. Most connect with USB, but you might find some that still use serial ports. With the monitoring software loaded, apcupsd for APC, it will shutdown the computer based on the parameters you configure. This can be time elapsed on battery, UPS estimated run time remaining, etc. When the power returns the UPS will wait for time elapsed and minimum battery charge before turning back on. Usually these are adjustable parameters in the UPS firmware.
Additionally if your computer BIOS supports configuring the AC recovery power mode to always on, the computer will start back up. Most default to last state, which doesn't work as well since shutting down will power off the computer, which means that last state is off. There are some workarounds to change the shutdown sequence to just halt and not power off.
Ryan
Ryan Wagoner wrote:
The solution here is to purchase a UPS. I prefer APC, but there are other brands that work great.
...
Additionally if your computer BIOS supports configuring the AC recovery power mode to always on, the computer will start back up. Most default to last state, which doesn't work as well since shutting down will power off the computer, which means that last state is off. There are some workarounds to change the shutdown sequence to just halt and not power off.
Thanks for your very helpful reply. I will get a UPS, as you suggest, and if I can locate an APC I'll get that.
I hadn't really thought about the last part; but of course it is not normally sufficient just to switch the current on, I have to press the button on the computer as well. The computer in question is an HP MicroServer; I will see if it is possible to send it into halt mode.
On Sat, 24 Mar 2012 19:05:20 +0100 Timothy Murphy wrote:
I hadn't really thought about the last part; but of course it is not normally sufficient just to switch the current on, I have to press the button on the computer as well.
I have a number of machines with Intel motherboards that have a bios setting for what happens when power is present. "On" " Off" and "Last State" are the available options. I just select "On".
Ryan Wagoner wrote:
Additionally if your computer BIOS supports configuring the AC recovery power mode to always on, the computer will start back up. Most default to last state, which doesn't work as well since shutting down will power off the computer, which means that last state is off. There are some workarounds to change the shutdown sequence to just halt and not power
Thanks again for your response. I did an experiment which I probably should have done before, to find out exactly what happens if the power is cut off, and then restored. You were completely correct.
My server is an HP MicroServer (running CentOS-6.2). If the machine was running before being cut off then it reboots normally when power is restored. If however it is shutdown before the power is cut then it does not reboot when power is restored. As you say, it always returns to the state it was in when the outage occurs.
I looked in the BIOS, but there is no option on this machine to change that.
So I shall know what to do when I get a UPS. If there is no better setting I can just allow the UPS to become exhausted if the outage lasts longer than the life of the UPS.
On 3/25/2012 2:39 PM, Timothy Murphy wrote:
Ryan Wagoner wrote:
Additionally if your computer BIOS supports configuring the AC recovery power mode to always on, the computer will start back up. Most default to last state, which doesn't work as well since shutting down will power off the computer, which means that last state is off. There are some workarounds to change the shutdown sequence to just halt and not power
Thanks again for your response. I did an experiment which I probably should have done before, to find out exactly what happens if the power is cut off, and then restored. You were completely correct.
My server is an HP MicroServer (running CentOS-6.2). If the machine was running before being cut off then it reboots normally when power is restored. If however it is shutdown before the power is cut then it does not reboot when power is restored. As you say, it always returns to the state it was in when the outage occurs.
I looked in the BIOS, but there is no option on this machine to change that.
So I shall know what to do when I get a UPS. If there is no better setting I can just allow the UPS to become exhausted if the outage lasts longer than the life of the UPS.
in the apc software(or nut worst case) you'll be able to specify that after x minutes it will shutdown. Better to have a clean shutdown and have to hit the power button than a dirty one and risk corrupting your filesystem.
William Warren wrote:
in the apc software(or nut worst case) you'll be able to specify that after x minutes it will shutdown. Better to have a clean shutdown and have to hit the power button than a dirty one and risk corrupting your filesystem.
Unfortunately I won't be there to "hit the power button". If I were there would be no need for UPS.
On Mon, Mar 26, 2012 at 5:30 PM, Timothy Murphy gayleard@alice.it wrote:
William Warren wrote:
in the apc software(or nut worst case) you'll be able to specify that after x minutes it will shutdown. Better to have a clean shutdown and have to hit the power button than a dirty one and risk corrupting your filesystem.
Unfortunately I won't be there to "hit the power button". If I were there would be no need for UPS.
Another approach is to buy a remote power switch. With it, basically you can remotely "power cycle" the server :) It will solve any hang problem just as if you are there :)
Fajar Priyanto wrote:
Another approach is to buy a remote power switch. With it, basically you can remotely "power cycle" the server :) It will solve any hang problem just as if you are there :)
I have actually tried this. I bought a very cheap ($20) device from China which is supposed to switch the power on and off when it receives a GSM message. But to date I haven't been able to get it to work. (The instructions are in a language midway between English and Chinese.)
I've decided UPS is a more reliable solution.
On 03/24/2012 02:33 PM, Timothy Murphy wrote:
Any suggestions, advice or experience of this problem gratefully received.
An advice among all others: - When the machine gets stable, disable FS checks (in fstab, I dont remember what field to set to 0) - Setup the filesystem not to ask for check every N mounts (tune2fs) - Have a "netboot" available when things get really bad, so that you can mount you usual "/" via the netbooted system
Mihamina Rakotomandimby wrote:
An advice among all others:
- When the machine gets stable, disable FS checks (in fstab, I dont
remember what field to set to 0)
- Setup the filesystem not to ask for check every N mounts (tune2fs)
Thanks for your response.
Touch wood, I've found filesystem corruption seems to have become a thing of the past. I assume this is something to do with journaling. My laptop (Fedora-16/KDE) freezes about once a week. I'm not sure why, it doesn't seem to be associated with any application. In any case, I have to stop the machine by pressing the power button. When I re-boot there has not been any problem with the filesystem. As I say, touch wood.
- Have a "netboot" available when things get really bad, so that you can
mount you usual "/" via the netbooted system
Thanks. I'll look into that. Though I don't think that I have any choice except to hope the machine reboots nicely when power returns, since I can't "see" it from a remote location until it has re-booted.
One possibility I've considered is to have a UPS stop the machine, and then use Wake-on-LAN to start it again. But to date I haven't been able to get Wake-on-LAN to work on my HP PowerServer, though it is supposed to be an option.
On 26.3.2012 13:35, Timothy Murphy wrote:
Touch wood, I've found filesystem corruption seems to have become a thing of the past. I assume this is something to do with journaling. My laptop (Fedora-16/KDE) freezes about once a week. I'm not sure why, it doesn't seem to be associated with any application. In any case, I have to stop the machine by pressing the power button. When I re-boot there has not been any problem with the filesystem. As I say, touch wood.
Ok, the filesystem is consistent, but could there be lost data?
On Mon, Mar 26, 2012 at 7:35 PM, Timothy Murphy gayleard@alice.it wrote:
One possibility I've considered is to have a UPS stop the machine, and then use Wake-on-LAN to start it again. But to date I haven't been able to get Wake-on-LAN to work on my HP PowerServer, though it is supposed to be an option.
I see. I think your HP machine also have iLO. You can use it to power on/off remotely too. If I remember correctly earlier you said you have problem with public IP availability. In that case you can setup a gateway with any static IP you have and connect the server from there.
-
Fajar Priyanto -
Frank Cox -
Markus Falb -
Mihamina Rakotomandimby -
Ryan Wagoner -
Timothy Murphy -
William Warren