Transition Technology: Ticket #583: tmp files on parrot exceeded inode limit

hours, status, totalhours changed; cc set

chris — Mon, 02 Sep 2013 18:22:44 GMT

cc laura added
hours changed from 0.0 to 0.5
status changed from new to accepted
totalhours changed from 0.0 to 0.5

There is an issue with disk space on /home -- files can't be created:

cd /home
touch test.txt
  touch: cannot touch `test.txt': No space left on device

However df doesn't report a problem:

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2            9.9G  4.1G  5.4G  44% /
tmpfs                1021M     0 1021M   0% /lib/init/rw
udev                  990M   36K  989M   1% /dev
tmpfs                1021M  4.0K 1021M   1% /dev/shm
/dev/xvda3            8.9G  5.1G  3.4G  61% /home

There are a lot of tmp files that can be deleted:

cd /home/recon/tmp
du -h
  1.2G    .
cd /home/movie/tmp/
du -h
 1.1G    .

These seem to be tmp session files that don't get deleted, there are a lot of them:

ls /home/recon/tmp/ | wc -l
294282

So deleting the tmp files that are more than a day old:

find /home/recon/tmp/* -mtime +1 -exec rm {} \;
find /home/movie/tmp/* -mtime +1 -exec rm {} \;

Hopefully this will fix it for now, but more work is needed on this.

hours, totalhours changed

chris — Tue, 03 Sep 2013 10:52:01 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.5 to 0.75

The server had run out of inodes (see https://en.wikipedia.org/wiki/Inode ), not diskspace, there were over 1/2 a million tmp session files:

269711 ./movie/tmp
294284 ./recon/tmp

There is now lots of free inodes:

df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda2            655360   40647  614713    7% /
tmpfs                 261355       4  261351    1% /lib/init/rw
udev                  253188     384  252804    1% /dev
tmpfs                 261355       3  261352    1% /dev/shm
/dev/xvda3            589824   28674  561150    5% /home

I have added a daily cron job to delete all tmp files that are more than 7 days old:

01 06 * * * find /home/*/tmp/* -mtime +7 -exec rm {} \;

I have added a section to the wiki about this, wiki:ParrotServer#tmpfiles

I think that resolves this issue?

hours, totalhours, summary changed

chris — Tue, 03 Sep 2013 12:28:38 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.75 to 1.0
summary changed from Problem with intransition movie site shopping cart to tmp files on parrot exceeded inode limit

For reference the number of inodes on the /home aka /dev/xvda3 partition is 589,824 and this is the limit that was hit:

tune2fs -l /dev/xvda3
...
Filesystem OS type:       Linux
Inode count:              589824
Block count:              2359296
Reserved block count:     117964
Free blocks:              1551253
Free inodes:              561780
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      575
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512

We could consider creating a new partition just for tmp files with a inode count of several million if there is an issue with tmp session files being deleted after a week -- is this a problem, does it cause people to be logged out?

ed — Tue, 03 Sep 2013 12:40:40 GMT

Good work Chris - thanks.

I'm going to ask Laura to have a scan of this ticket - to developer-sense-check it - and see how/if it ties in with the larger piece of work around the whole site and updating it - which would need separate budget from TN...

hours, status, totalhours changed; resolution set

chris — Thu, 12 Sep 2013 09:04:28 GMT

hours changed from 0.0 to 0.5
status changed from accepted to closed
resolution set to fixed
totalhours changed from 1.0 to 1.5

The cronjob failed last night:

From: root@parrot.webarch.net (Cron Daemon)
Date: Thu, 12 Sep 2013 06:01:01 +0100
To: root@parrot.webarch.net
Subject: Cron <root@parrot> find /home/*/tmp/* -mtime +7 -exec rm {} \; (failed)
/bin/sh: find: Argument list too long

It also failed when piping to rm:

find /home/*/tmp/* -mtime +7 | xargs rm -rf
-bash: /usr/bin/find: Argument list too long

And ls fails when trying to count the files:

ls /home/*/tmp/* | wc -l
-bash: /bin/ls: Argument list too long
0

Debian has a package called tmpreaper, so:

aptitude install tmpreaper ; logchange "tmpreaper: installed"

The /etc/tmpreaper.conf file was edited and these values were set:

TMPREAPER_DIRS='/home/*/tmp/.'
TMPREAPER_TIME='7d'

And it was tested on the command line:

/etc/cron.daily/tmpreaper
error: Cannot chdir() to `/home/*/tmp/.' for --protect glob: No such file or directory

So we have to specify the directories without a wild card:

TMPREAPER_DIRS='/home/recon/tmp/. /home/movie/tmp/.'

And it was run again and then the number of files remaining was countable again:

/etc/cron.daily/tmpreaper
ls /home/*/tmp/* | wc -l
42963

And the documentation has been updated, wiki:ParrotServer#tmpfiles