Disk space error on parrot for TTT site

[chris]

Email from Laura:

Message appearing at top of ttt website when I just took a look-

Warning: session_start():
open(/home/ttt/tmp/sess_mnme76d2k5s6vopk5u1it126p5, O_RDWR) failed: No
space left on device (28) in
/home/ttt/sites/default/wp-content/plugins/tt-resource-database/participants-database.php
on line 2534

and something in the sidebar -

Warning: call_user_func_array() expects parameter 1 to be a valid 
callback, array must have exactly two members in 
/home/ttt/sites/default/wp-includes/plugin.php on line 199

[chris]

There isn't an issue with overall disk space:

df -h
  Filesystem      Size  Used Avail Use% Mounted on
  rootfs          9.9G  5.2G  4.2G  56% /
  udev             10M     0   10M   0% /dev
  tmpfs           307M  196K  307M   1% /run
  /dev/xvda2      9.9G  5.2G  4.2G  56% /
  tmpfs           5.0M     0  5.0M   0% /run/lock
  tmpfs           818M     0  818M   0% /run/shm
  /dev/xvda3      8.9G  5.9G  2.6G  70% /home

So the issue must be with the quota for the site:

  Disk quotas for user ttt (uid 1007):
    Filesystem                   blocks       soft       hard     inodes     soft     hard
    /dev/xvda3                  1039520          0    6000000       8913        0        0

Looking at the space used in /home:

cd /home
  du -h --max-depth=1
  40M     ./earthin
  16K     ./lost+found
  28K     ./aland
  1.5G    ./tc
  88K     ./chris
  1.6G    ./recondev
  24K     ./jim
  105M    ./moviedev
  1.1G    ./ttt
  1.4G    ./recon
  24K     ./finn
  70M     ./ts
  90M     ./movie
  5.8G    .

So editing the ttt user so it has 3.0GB of space:

edquota -u ttt
  Disk quotas for user ttt (uid 1007):
    Filesystem                   blocks       soft       hard     inodes     soft     hard
    /dev/xvda3                  1039520    3145728          0       8913        0        0

And doing the same for Transition Culture:

edquota -u tc
  Disk quotas for user tc (uid 1011):
    Filesystem                   blocks       soft       hard     inodes     soft     hard
    /dev/xvda3                  1322432    3145728           0     533114        0        0

And giving Recononmy:

Disk quotas for user recon (uid 1006):
  Filesystem                   blocks       soft       hard     inodes     soft     hard
  /dev/xvda3                   935884    3145728          0      24223        0        0

[chris]

The issue is inodes -- we have run out:

df -i | grep home 
  /dev/xvda3     589824 589816      8  100% /home

Identifying where inodes are being used in /home:

cd /home
find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      1 aquota.user
      4 finn
      4 jim
      5 aland
     14 chris
   1833 earthin
   2748 ts
   3073 moviedev
   3122 movie
   8474 ttt
  10644 recondev
  23648 recon
 532858 tc

So this issue is with the Transition Culture site:

cd /home/tc
find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      1 .forward
      2 private
     58 logs
  19166 sites
 513631 tmp

I tried deleting all the files, but there are too many for this:

cd /home/tc
rm -rf tmp/*

So the whole directory was deleted and recreated:

rm -rf /home/tc/tmp
mkdir /home/tc/tmp
chown tc:tc /home/tc/tmp
chmod 700 /home/tc/tmp

And a cronjob was added to delete files older than 7 days in the tmp dirs:

01 01 * * * find /home/*/tmp/* -mtime +7 -type f -delete

So now we are only using 13% of the inodes:

df -i | grep home 
  /dev/xvda3     589824 76020 513804   13% /home

[chris]

Replying to chris:

And a cronjob was added to delete files older than 7 days in the tmp dirs:

01 01 * * * find /home/*/tmp/* -mtime +7 -type f -delete

I should have read the documentation! wiki:ParrotServer#tmpfiles I have now updated that and removed the cronjob.

There are currently 10k tmp files more than a day old:

find /home/*/tmp/* -mtime +1 -type f | wc -l
  10827

[chris]

Don't know how many tmp files there are now:

find /home/*/tmp/* -mtime +1 -type f | wc -l
  -bash: /usr/bin/find: Argument list too long

But there are over 100k inodes on /home:

df -i
  Filesystem     Inodes  IUsed  IFree IUse% Mounted on
  /dev/xvda3     589824 119776 470048   21% /home

Have set tmpreaper to delete after 3 days and have update the docs, at wiki:ParrotServer#tmpfiles

This ticket isn't quite ready to be closed.

[chris]

There are still a huge number of tmp files being created, this isn't a problem as long as old ones are removed:

find /home/*/tmp/*
  -bash: /usr/bin/find: Argument list too long

The numbers of tmp files by site:

/home/reconomy/tmp
ls | wc -l
  156715
cd /home/tc/tmp/
ls | wc -l
  23048
cd /home/ttt/tmp
ls | wc -l
  18699

Total number of inodes:

df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
rootfs         655360  54174 601186    9% /
udev           382011    245 381766    1% /dev
tmpfs          392427    209 392218    1% /run
/dev/xvda2     655360  54174 601186    9% /
tmpfs          392427      5 392422    1% /run/lock
tmpfs          392427      2 392425    1% /run/shm
/dev/xvda3     589824 267775 322049   46% /home

The reason the number of tmp files is so high is that when the reconomy site username was changed, see ticket:718, the /etc/tmpreaper.conf file wasn't updated to reflect this, that has now been fixed.

The old reconomy sites have also been deleted:

rm-user recon
rm-user reconhold

The reconomy tmp files older than a day have been deleted:

find /home/reconomy/tmp/ -mtime +1 -type f -delete

And now we have:

df -i
Filesystem     Inodes IUsed  IFree IUse% Mounted on
rootfs         655360 54077 601283    9% /
udev           382011   245 381766    1% /dev
tmpfs          392427   209 392218    1% /run
/dev/xvda2     655360 54077 601283    9% /
tmpfs          392427     5 392422    1% /run/lock
tmpfs          392427     2 392425    1% /run/shm
/dev/xvda3     589824 97155 492669   17% /home

Now there is loads of room for more WordPress sites!

Closing this as fixed.