Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Thanks in advance, -Ray
My layout is:
#df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 131G 130G 0 100% / /dev/sdc1 271G 156G 102G 61% /home /dev/sdd1 271G 4.5G 253G 2% /home/905 /dev/sda1 99M 29M 66M 31% /boot tmpfs 442M 0 442M 0% /dev/shm
# cat /etc/fstab /dev/VolGroup00/LogVol00 / ext3 defaults 1 1 LABEL=/home /home ext3 defaults 1 2 LABEL=/home/905 /home/905 ext3 defaults 1 2 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/VolGroup00/LogVol01 swap swap defaults 0 0
# cat /etc/mtab /dev/mapper/VolGroup00-LogVol00 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/sdc1 /home ext3 rw 0 0 /dev/sdd1 /home/905 ext3 rw 0 0 /dev/sda1 /boot ext3 rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
On Wed, Nov 26, 2008 at 4:50 PM, Ray Leventhal centos@swhi.net wrote:
Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Thanks in advance, -Ray
My layout is:
#df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 131G 130G 0 100% / /dev/sdc1 271G 156G 102G 61% /home /dev/sdd1 271G 4.5G 253G 2% /home/905 /dev/sda1 99M 29M 66M 31% /boot tmpfs 442M 0 442M 0% /dev/shm
# cat /etc/fstab /dev/VolGroup00/LogVol00 / ext3 defaults 1 1 LABEL=/home /home ext3 defaults 1 2 LABEL=/home/905 /home/905 ext3 defaults 1 2 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/VolGroup00/LogVol01 swap swap defaults 0 0
# cat /etc/mtab /dev/mapper/VolGroup00-LogVol00 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/sdc1 /home ext3 rw 0 0 /dev/sdd1 /home/905 ext3 rw 0 0 /dev/sda1 /boot ext3 rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
Do a search for coredump file (i.e. run updatedb && locate core.*) - they can often fill up the HDD very quicly if something coredumps.
Rudi Ahlers wrote:
On Wed, Nov 26, 2008 at 4:50 PM, Ray Leventhal centos@swhi.net wrote:
Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Thanks in advance, -Ray
<snip>
Do a search for coredump file (i.e. run updatedb && locate core.*) - they can often fill up the HDD very quicly if something coredumps.
Thanks, Rudi...I hadn't thought of that. The results, however, weren't overly helpful for this issue in this case.
I had about 11M of core dumps from March in the samba/cores directory
The big deal here is that the system is choking...any advice on finding what's making it choke and/or relieving the strain on / would be appreciated.
Again, my thanks, -Ray
Ray,
You can find where the data is through the following command:
du -h --max-depth=1
Start in the root (/) and follow the trail.
Succes
Best regards,
Joost Waversveld
Ray Leventhal wrote:
Rudi Ahlers wrote:
On Wed, Nov 26, 2008 at 4:50 PM, Ray Leventhal centos@swhi.net wrote:
Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Thanks in advance, -Ray
<snip>
Do a search for coredump file (i.e. run updatedb && locate core.*) - they can often fill up the HDD very quicly if something coredumps.
Thanks, Rudi...I hadn't thought of that. The results, however, weren't overly helpful for this issue in this case.
I had about 11M of core dumps from March in the samba/cores directory
The big deal here is that the system is choking...any advice on finding what's making it choke and/or relieving the strain on / would be appreciated. Again, my thanks, -Ray _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
You still have 1gig free. Did you do a full install? If you can live with it, uninstall packages you don't need.
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
You still have 1gig free. Did you do a full install? If you can live with it, uninstall packages you don't need.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi Sorin,
It was a nearly-full install and I do connect periodically to do updates. I did a yum clean all before writing to the list, with no significant change to the disk usage.
I will list packages next and see what can be removed safely. I truly don't know what is filling the 130G...this should be more than sufficient as I've loaded CentOS5.x on many systems without this issue.
Thanks again for your input, -Ray
On Wed, Nov 26, 2008 at 5:19 PM, Ray Leventhal centos@swhi.net wrote:
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
You still have 1gig free. Did you do a full install? If you can live with it, uninstall packages you don't need.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi Sorin,
It was a nearly-full install and I do connect periodically to do updates. I did a yum clean all before writing to the list, with no significant change to the disk usage.
I will list packages next and see what can be removed safely. I truly don't know what is filling the 130G...this should be more than sufficient as I've loaded CentOS5.x on many systems without this issue.
Thanks again for your input, -Ray _______________________________________________
Have you rebooted? Maybe there's a glitch in there.
Also, do a search for .ISO files? Maybe, just maybe someone dumped some ISO's on there? Altenatively, if you run du -h ./ | more, you'll be able to see which folder is the biggest, and then goto that folder and see what's in there.
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 4:20 PM:
It was a nearly-full install and I do connect periodically to do updates. I did a yum clean all before writing to the list, with no significant change to the disk usage.
I will list packages next and see what can be removed safely. I truly don't know what is filling the 130G...this should be more than sufficient as I've loaded CentOS5.x on many systems without this issue.
Ok, what about the /home-folder, did it reside on a different disk or on the same as where / is? If it's on /, take a look inside /home, your users may have started downloading pictures, mp3 and stuff. You know the drill. ;-)
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
<snip>
Thanks to all who replied.
/ filled up when my nightly rsync snapshot did something which I'm still looking into.
I run a nightly rsync script to make copies (to an external HDD connected via USB) of user data files:
#backup to USB drive location for /home # /media/bkup is /dev/sdg1 (USB 700GB drive) rsync -av --delete /home/ /media/bkup cd
Well, in /media, there were 2 folders, not just one.../bkup and /bkup_ as well as 2 .lock files. I determined which was the last complete backup and deleted the other... needless to say / space began to increase, but I'm truly puzzled about why a mount point would take up space on / when the media is external.
Anyone with insight into my flawed logic, please let me know :)
Thanks for all help and the ongoing knowledge gained from this list.
-Ray
On Wed, Nov 26, 2008 at 5:50 PM, Ray Leventhal centos@swhi.net wrote:
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
<snip>
Thanks to all who replied.
/ filled up when my nightly rsync snapshot did something which I'm still looking into.
I run a nightly rsync script to make copies (to an external HDD connected via USB) of user data files:
#backup to USB drive location for /home # /media/bkup is /dev/sdg1 (USB 700GB drive) rsync -av --delete /home/ /media/bkup cd
Well, in /media, there were 2 folders, not just one.../bkup and /bkup_ as well as 2 .lock files. I determined which was the last complete backup and deleted the other... needless to say / space began to increase, but I'm truly puzzled about why a mount point would take up space on / when the media is external.
Anyone with insight into my flawed logic, please let me know :)
Thanks for all help and the ongoing knowledge gained from this list.
-Ray _______________________________________________
That all depends on how the USB HDD was mounted. If it wasn't mounted sucesfully, then /media will be treated as just another folder.
How do you mound the HDD?
<snip>
Thanks to all who replied.
/ filled up when my nightly rsync snapshot did something which I'm still looking into.
I run a nightly rsync script to make copies (to an external HDD connected via USB) of user data files:
#backup to USB drive location for /home # /media/bkup is /dev/sdg1 (USB 700GB drive) rsync -av --delete /home/ /media/bkup cd
Well, in /media, there were 2 folders, not just one.../bkup and /bkup_ as well as 2 .lock files. I determined which was the last complete backup and deleted the other... needless to say / space began to increase, but I'm truly puzzled about why a mount point would take up space on / when the media is external.
Anyone with insight into my flawed logic, please let me know :)
Thanks for all help and the ongoing knowledge gained from this list.
-Ray
<snip sig>
Ray,
Perhaps your usb drive became unmounted for some reason.
You might try unmounting your usb drive, and see what is in the mount point directory in the root partition.
For instance, if your usb drive is mounted at /media/bkup, unmount the drive, then cd into /media/bkup and see what's there.
HTH
Monty
Ray Leventhal wrote:
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to the internet, you probably are not running yum at all, so it might not help. Check your temp-directories and clean out as necessary.
<snip>
Thanks to all who replied.
/ filled up when my nightly rsync snapshot did something which I'm still looking into.
I run a nightly rsync script to make copies (to an external HDD connected via USB) of user data files:
#backup to USB drive location for /home # /media/bkup is /dev/sdg1 (USB 700GB drive) rsync -av --delete /home/ /media/bkup cd
Well, in /media, there were 2 folders, not just one.../bkup and /bkup_ as well as 2 .lock files. I determined which was the last complete backup and deleted the other... needless to say / space began to increase, but I'm truly puzzled about why a mount point would take up space on / when the media is external.
Anyone with insight into my flawed logic, please let me know :)
Thanks for all help and the ongoing knowledge gained from this list.
-Ray _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
when you mount an device, you will mount it to an existing directory (/media/bkup in this case).
when the drive is mounted, everything that is written to /media/bkup will be written on the external disk.
When the drive is unmounted, everything that is written to that directory, will be written to.. euhm.. the directory, thus the local file system.
So maybe something happened with the mount of the external disk??
Best regards,
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to
Hi again all,
There was a 3.5hr power outage last night which explains it all. Sadly, I've got some investigation to do about why my *supposed* 5hrs of battery backup didn't last long enough to cover, but the mount point was, in fact, unmounted and so rsync did it's job right into the folder as opposed to the ext. drive.
As always, my sincere thanks to this list and to our CentOS maintainers.
Kindest regards, -Ray
Ray Leventhal wrote:
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to
Hi again all,
There was a 3.5hr power outage last night which explains it all. Sadly, I've got some investigation to do about why my *supposed* 5hrs of battery backup didn't last long enough to cover, but the mount point was, in fact, unmounted and so rsync did it's job right into the folder as opposed to the ext. drive.
As always, my sincere thanks to this list and to our CentOS maintainers.
Kindest regards, -Ray
I noticed the chorus of agreement that your problem was likely a result of failure to mount your backup drive. My backup script, which also uses rsync begins like this, insuring a good mount before shoving bytes around. It's not SUPPOSED to be mounted before the script runs but I test for that, too. (The drive label is OTOT):
#!/bin/bash # Backup using rsync and rotating directories # #Set the Dest. Mount UD=/media/OTOT MS=8388608 # Minimum free space 8GB # # Drive mount logic: # if [ -z $(mount | grep $UD | awk '{ print $3 }') ]; then mount $UD fi if [ -z $(mount | grep $UD | awk '{ print $3 }') ]; then echo "Drive refuses to mount!!" exit 1 fi # # Drive is mounted. Get device name # XDR=$(mount | grep $UD | awk '{ print $1 }')
etc., etc.
Robert wrote:
<snip original stuff>
I noticed the chorus of agreement that your problem was likely a result of failure to mount your backup drive. My backup script, which also uses rsync begins like this, insuring a good mount before shoving bytes around. It's not SUPPOSED to be mounted before the script runs but I test for that, too. (The drive label is OTOT):
#!/bin/bash # Backup using rsync and rotating directories # #Set the Dest. Mount UD=/media/OTOT MS=8388608 # Minimum free space 8GB # # Drive mount logic: # if [ -z $(mount | grep $UD | awk '{ print $3 }') ]; then mount $UD fi if [ -z $(mount | grep $UD | awk '{ print $3 }') ]; then echo "Drive refuses to mount!!" exit 1 fi # # Drive is mounted. Get device name # XDR=$(mount | grep $UD | awk '{ print $1 }')
etc., etc.
Thanks Robert, Ned, William, et al (no offense intended to the many not named here), who replied with the need for logic and for checking that all devices, including the ext HDD, were connected to the UPS.
Re: Logic in the script: no doubt, I will be modifying it to check for valid mount point before starting. As always, I'm learning from my mistakes :)
As for the server having been up, I think what happened was the server and the HDD (which was connected to the UPS) went down before the rsync was scheduled to run. When power was lost, the mount point wasn't properly released. When it came back up and online, the USB drive was mounted to /media/bkup_ (trailing _ provided by the system) which turned out to be a folder on /, not a pointer to the external drive.
Again, thanks for the tips, pointers and sound ideas.
-Ray
On Wed, 2008-11-26 at 11:15 -0500, Ray Leventhal wrote:
Sorin Srbu wrote:
<snip>
Hi again all,
There was a 3.5hr power outage last night which explains it all. Sadly, I've got some investigation to do about why my *supposed* 5hrs of battery backup didn't last long enough to cover, but the mount point was, in fact, unmounted and so rsync did it's job right into the folder as opposed to the ext. drive.
My rsync backup script has a check in it to suppress the backup and issue a message if the media is not mounted.
As always, my sincere thanks to this list and to our CentOS maintainers.
Kindest regards, -Ray
<snip sig stuff>
Ray Leventhal wrote:
There was a 3.5hr power outage last night which explains it all. Sadly, I've got some investigation to do about why my *supposed* 5hrs of battery backup didn't last long enough to cover, but the mount point was, in fact, unmounted and so rsync did it's job right into the folder as opposed to the ext. drive.
Well, the system must have stayed up otherwise it couldn't have done the rsync backup at all. It sounds more like only the ext. usb drive went down which prompts the question - was it connected to the backup battery source?
Ray Leventhal wrote:
Sorin Srbu wrote:
Ray Leventhal <> scribbled on Wednesday, November 26, 2008 3:50 PM:
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Try a yum clean all. That might help. But if it's as you say, not connected to
Hi again all,
There was a 3.5hr power outage last night which explains it all. Sadly, I've got some investigation to do about why my *supposed* 5hrs of battery backup didn't last long enough to cover, but the mount point was, in fact, unmounted and so rsync did it's job right into the folder as opposed to the ext. drive.
You may want to add some logic to your rsync script to check for a properly mounted disk.
On Wed, 2008-11-26 at 09:50 -0500, Ray Leventhal wrote:
Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I suggest "man du ..." with the -x or --one-file-system flag along with the others. That will tell you where the largest stuf is at. Then you can cd into suspect places and do it again to get clues as to the hogs.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
If you need to do that after the above investigation, it will not be hard as pvcreate and lgexpand (?, man not at hand) will allow easy expansion.
<snip>
HTH
Ray Leventhal wrote:
Hi all,
Please pardon my newbie-ness on this issue....I've a / partition which is full (quite suddenly, actually) and I'm not sure how to fix this.
I've searched for uneeded logs, etc in /var/log and /tmp to no avail. The system is CentOS 5.2 and is not connected to the internet, serves as a local LAN server running stock stuff...sendmail, dovecot, apache..nothing strange or special going on.
I have additional HDDs available if growing the partition is in order (would appreciate pointers to that, if applicable), but I'm really stumped as to where the space is being eaten up.
Thanks in advance, -Ray
My layout is:
#df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 131G 130G 0 100% / /dev/sdc1 271G 156G 102G 61% /home /dev/sdd1 271G 4.5G 253G 2% /home/905 /dev/sda1 99M 29M 66M 31% /boot tmpfs 442M 0 442M 0% /dev/shm
# cat /etc/fstab /dev/VolGroup00/LogVol00 / ext3 defaults 1 1 LABEL=/home /home ext3 defaults 1 2 LABEL=/home/905 /home/905 ext3 defaults 1 2 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/VolGroup00/LogVol01 swap swap defaults 0 0
# cat /etc/mtab /dev/mapper/VolGroup00-LogVol00 / ext3 rw 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 devpts /dev/pts devpts rw,gid=5,mode=620 0 0 /dev/sdc1 /home ext3 rw 0 0 /dev/sdd1 /home/905 ext3 rw 0 0 /dev/sda1 /boot ext3 rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
I'm working myself out of one those unhappy situations right now. The way I always start is to pick some arbitrary value of file size, larger than which there will (or should) be a very small number of files and then: # find / -size +2G -print ...which will present me with a list of candidates for removal.