Hi,
2 websites, hosted on 2 different CentOS 5.7 servers (one being very new, about 3 weeks old) keeps loosing data - but it's more like it's corrupted than being deleted.
For example, a photo would be uploaded last night and today when we checked it, it doesn't show on the website. So we check if the file is on the server, and exists but is 0KB in size. Last night it still worked fine. The photo is 482Kb in size.
The first time this happened we thought it was due to a bug with CentOS 5.7 + EXT4 + quotas (there's a bug open for this) and since the server's console kept giving errors about possible data corruption we thought it would be best if we move everything to a more stable platform. So we brought a new server, setup CentOS 5.7 + ext3 + quotas (which has been working fine on all our servers for a long time) and moved the data across. A few days down the line and I still see this happening.
I'm out of ideas and hope someone could shed some light on the matter. I've checked some suggested search results, but couldn't find any issues with the HDD according to SMART. The servers' both have 4GB RAM and 8Core CPU's.Neither RAM, nor CPU usage is high. Both are setup with RAID10 across 4 entrerprise HDD's, one server has software RAID and the new one hardware RAID. So even when we changed the RAID subsystem it still happens.
From: Rudi Ahlers Rudi@SoftDux.com
For example, a photo would be uploaded last night and today when we checked it, it doesn't show on the website. So we check if the file is on the server, and exists but is 0KB in size. Last night it still worked fine. The photo is 482Kb in size.
Only the size changed to 0 or were some other stats changed too? Maybe try something like: inotifywait -m --format "%T %e %f" --timefmt "%D %T" -r /path/to/dir At least you would know at what time something happens...
JD
On Thu, Dec 15, 2011 at 1:29 PM, John Doe jdmls@yahoo.com wrote:
From: Rudi Ahlers Rudi@SoftDux.com
For example, a photo would be uploaded last night and today when we checked it, it doesn't show on the website. So we check if the file is on the server, and exists but is 0KB in size. Last night it still worked fine. The photo is 482Kb in size.
Only the size changed to 0 or were some other stats changed too? Maybe try something like: inotifywait -m --format "%T %e %f" --timefmt "%D %T" -r /path/to/dir At least you would know at what time something happens...
JD _______________________________________________
John,
Where do I get inotifywait ?
yum what provides "*/inotifywait" didn't return anything
root@mars:[/]$ yum whatprovides "*/inotifywait" Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.highspeedweb.net * extras: centos.omnispring.com * updates: mirrors.igsobe.com Excluding Packages in global exclude list Finished extras/filelists_db
| 206 kB 00:00 updates/filelists_db
| 1.7 MB 00:00 No Matches found
Vreme: 12/16/2011 12:22 AM, Rudi Ahlers piše:
John,
Where do I get inotifywait ?
yum what provides "*/inotifywait" didn't return anything
root@mars:[/]$ yum whatprovides "*/inotifywait" Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile
- base: mirror.highspeedweb.net
- extras: centos.omnispring.com
- updates: mirrors.igsobe.com
Excluding Packages in global exclude list Finished extras/filelists_db
| 206 kB 00:00 updates/filelists_db
| 1.7 MB 00:00 No Matches found
inotify-tools from EPEL?
Cool, thanx :)
On Fri, Dec 16, 2011 at 2:16 AM, Ljubomir Ljubojevic office@plnet.rs wrote:
Vreme: 12/16/2011 12:22 AM, Rudi Ahlers piše:
John,
Where do I get inotifywait ?
yum what provides "*/inotifywait" didn't return anything
root@mars:[/]$ yum whatprovides "*/inotifywait" Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.highspeedweb.net * extras: centos.omnispring.com * updates: mirrors.igsobe.com Excluding Packages in global exclude list Finished extras/filelists_db
| 206 kB 00:00 updates/filelists_db
| 1.7 MB 00:00 No Matches found
inotify-tools from EPEL?
--
Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe
Google is the Mother, Google is the Father, and traceroute is your trusty Spiderman... StarOS, Mikrotik and CentOS/RHEL/Linux consultant _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
From: Rudi Ahlers Rudi@SoftDux.com
Where do I get inotifywait ? yum what provides "*/inotifywait" didn't return anything
Got inotify-tools from repoforge.
JD
Hi Rudi we once had a similar problem on a Web:
This Web had this in particular that its home-page needed to be deleted daily and of course reinstalled immediately. Then, in a new version of the Web it did not need this delete/reinstall cycle any more, so the webadmin just removed the link to the delete script. He did not remove the delete-script itself from the Web. The effect was, that the home-page was still deleted at random times.
What we then found out: We used an internal Search-Engine which crawled this web. And this Search-Engine had not forgotten the link to the delete-script, but invoked it at random times, whenever it crawled the particular Web.
Your case: could it be something similar?
suomi
On 2011-12-15 11:28, Rudi Ahlers wrote:
Hi,
2 websites, hosted on 2 different CentOS 5.7 servers (one being very new, about 3 weeks old) keeps loosing data - but it's more like it's corrupted than being deleted.
For example, a photo would be uploaded last night and today when we checked it, it doesn't show on the website. So we check if the file is on the server, and exists but is 0KB in size. Last night it still worked fine. The photo is 482Kb in size.
The first time this happened we thought it was due to a bug with CentOS 5.7 + EXT4 + quotas (there's a bug open for this) and since the server's console kept giving errors about possible data corruption we thought it would be best if we move everything to a more stable platform. So we brought a new server, setup CentOS 5.7 + ext3 + quotas (which has been working fine on all our servers for a long time) and moved the data across. A few days down the line and I still see this happening.
I'm out of ideas and hope someone could shed some light on the matter. I've checked some suggested search results, but couldn't find any issues with the HDD according to SMART. The servers' both have 4GB RAM and 8Core CPU's.Neither RAM, nor CPU usage is high. Both are setup with RAID10 across 4 entrerprise HDD's, one server has software RAID and the new one hardware RAID. So even when we changed the RAID subsystem it still happens.
On Thu, Dec 15, 2011 at 1:48 PM, anax anax@ayni.com wrote:
Hi Rudi we once had a similar problem on a Web:
This Web had this in particular that its home-page needed to be deleted daily and of course reinstalled immediately. Then, in a new version of the Web it did not need this delete/reinstall cycle any more, so the webadmin just removed the link to the delete script. He did not remove the delete-script itself from the Web. The effect was, that the home-page was still deleted at random times.
What we then found out: We used an internal Search-Engine which crawled this web. And this Search-Engine had not forgotten the link to the delete-script, but invoked it at random times, whenever it crawled the particular Web.
Your case: could it be something similar?
suomi
Thanx for the advice. We found a script which seems to have been modified somehow, probably hacker or a rootkit - don't yet know, but we're looking to see if we can trace it down. I don't want to reveal the code here due to it's uniqueness and I don't want to attrack further attension to the sites by the attack(er).
-
anax -
John Doe -
Ljubomir Ljubojevic -
Rudi Ahlers