tmp files on parrot exceeded inode limit

[ed]

Users are attempting to purchase a DVD and the cart always zeros out even after attempts to update cart etc.

Laura suggests that this may be a server issue and to start with Chris:

The prob could v likely be on the hosting side, so be worth contacting Chris first (and mention can add to cart all okay when logged in, view cart and alter and get to billing page, so could be a session setting that's changed) -

1. Does the server have disk space left on /tmp. Session data is most likely written there. (sometimes this cart faff issue can happen because of that.)

2. Are the PHP Session settings correct/ have they changed recently? Chris can fix this if needing a tweakette in php.ini or other places in the underworld of those types of files.

3. DB could potentially need check/repair.

Another one is the shop element relies on cookies so if a user has set browser to not accept cookies may poss not work! (but think thats something not to think about here, as I tested as both logged in and logged out with cookies set to work on my browser, logged out got the 'cart empty' faff factor like other users will be getting.)

If Chris can't spot the bug (does sound server side and sessions related from a quick test here, haven't looked at the server logs, but if get a mo later will see if I can), poss good to chat and I can think some more on it, but first thoughts alert me mostly to server side, sometimes a roll back to php 5.2 clicks things back into place (but shouldn't be needed). Be good to see if there is a correlation with any server changes/tweaks happening around same timing that the cart stopped working. Keep me in the loop if it is a server fix-y thing as always interested to hear results on bug fixing on sites.

[chris]

There is an issue with disk space on /home -- files can't be created:

cd /home
touch test.txt
  touch: cannot touch `test.txt': No space left on device

However df doesn't report a problem:

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2            9.9G  4.1G  5.4G  44% /
tmpfs                1021M     0 1021M   0% /lib/init/rw
udev                  990M   36K  989M   1% /dev
tmpfs                1021M  4.0K 1021M   1% /dev/shm
/dev/xvda3            8.9G  5.1G  3.4G  61% /home

There are a lot of tmp files that can be deleted:

cd /home/recon/tmp
du -h
  1.2G    .
cd /home/movie/tmp/
du -h
 1.1G    .

These seem to be tmp session files that don't get deleted, there are a lot of them:

ls /home/recon/tmp/ | wc -l
294282

So deleting the tmp files that are more than a day old:

find /home/recon/tmp/* -mtime +1 -exec rm {} \;
find /home/movie/tmp/* -mtime +1 -exec rm {} \;

Hopefully this will fix it for now, but more work is needed on this.

[chris]

The server had run out of inodes (see ​https://en.wikipedia.org/wiki/Inode ), not diskspace, there were over 1/2 a million tmp session files:

269711 ./movie/tmp
294284 ./recon/tmp

There is now lots of free inodes:

df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda2            655360   40647  614713    7% /
tmpfs                 261355       4  261351    1% /lib/init/rw
udev                  253188     384  252804    1% /dev
tmpfs                 261355       3  261352    1% /dev/shm
/dev/xvda3            589824   28674  561150    5% /home

I have added a daily cron job to delete all tmp files that are more than 7 days old:

01 06 * * * find /home/*/tmp/* -mtime +7 -exec rm {} \;

I have added a section to the wiki about this, wiki:ParrotServer#tmpfiles

I think that resolves this issue?

[chris]

For reference the number of inodes on the /home aka /dev/xvda3 partition is 589,824 and this is the limit that was hit:

tune2fs -l /dev/xvda3

...

Filesystem OS type:       Linux
Inode count:              589824
Block count:              2359296
Reserved block count:     117964
Free blocks:              1551253
Free inodes:              561780
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      575
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512

We could consider creating a new partition just for tmp files with a inode count of several million if there is an issue with tmp session files being deleted after a week -- is this a problem, does it cause people to be logged out?

[ed]

Good work Chris - thanks.

I'm going to ask Laura to have a scan of this ticket - to developer-sense-check it - and see how/if it ties in with the larger piece of work around the whole site and updating it - which would need separate budget from TN...

[chris]

The cronjob failed last night:

From: root@parrot.webarch.net (Cron Daemon)
Date: Thu, 12 Sep 2013 06:01:01 +0100
To: root@parrot.webarch.net
Subject: Cron <root@parrot> find /home/*/tmp/* -mtime +7 -exec rm {} \; (failed)

/bin/sh: find: Argument list too long

It also failed when piping to rm:

find /home/*/tmp/* -mtime +7 | xargs rm -rf
-bash: /usr/bin/find: Argument list too long

And ls fails when trying to count the files:

ls /home/*/tmp/* | wc -l
-bash: /bin/ls: Argument list too long
0

Debian has a package called tmpreaper, so:

aptitude install tmpreaper ; logchange "tmpreaper: installed"

The /etc/tmpreaper.conf file was edited and these values were set:

TMPREAPER_DIRS='/home/*/tmp/.'
TMPREAPER_TIME='7d'

And it was tested on the command line:

/etc/cron.daily/tmpreaper 
error: Cannot chdir() to `/home/*/tmp/.' for --protect glob: No such file or directory

So we have to specify the directories without a wild card:

TMPREAPER_DIRS='/home/recon/tmp/. /home/movie/tmp/.'

And it was run again and then the number of files remaining was countable again:

/etc/cron.daily/tmpreaper 
ls /home/*/tmp/* | wc -l
42963

And the documentation has been updated, wiki:ParrotServer#tmpfiles