Rob is having Image upload issues

[ade]

Hi Paul,
Rob in Totnes is creating a blog post and is getting an error whilst trying to upload an image. Initially I thought it could have been a harddrive full issue, but we can see that we have drive room.
Sam and I have had a quick look to see if anything obvious, but we are now stumbling a bit. We can replicate the issue by trying to upload an image to a blog post, but cannot see where to find an event log to see if any logs are being generated that may aid in a diagnosis. Could you have a look and let us know your thoughts please? Rob is currently waiting to post a blog so is quite urgent.

Many thanks
Ade

[paul]

Hi Ade,

Would you let me know what errors you're receiving or maybe post a screenshot ?

The database logs are disabled on the production site.

Can you reproduce the problem on my stage site?

​https://booker-stage-20141120.transitionnetwork.org
​https://booker-stage-20141120.transitionnetwork.org/admin/reports/dblog

Best, Paul

[chris]

By the way, the Cc field simply needs the Trac username, in this case paul rather than a full email address, there is a list of usernames on the front page of this site to make it easy to find them, wiki:WikiStart#Tracusernames

Drupal logging (dblog) was switched from logging to the database to logging to the server syslog and then switched off entirely, if I remember correctly, on transitionnetwork.org some years ago, the reason might be found in a ticket somewhere or perhaps it was on the Ttech list, sorry I don't recall the exact reason for that decision, it wasn't mine.

[sam]

Hi all.

Paul; Yes I get the same (not descriptive) 'upload failed' error on your test site.

Any progress on this one? Rob is keen to get a new post out and has asked me to check up on progress. I thought we had a workaround (embedding Flickr images temporarily, but actually that won't work because he needs a featured image, which we can't hotlink)

Chris can you confirm that no permissions changes/ maintenance has happened at server level recently that could have impacted image upload?

As an aside Ade & I enabled the dblog module & set it to log the last 100 records. Hopefully this won't break anything! If it does we'll turn it off again.

Thanks

Sam

[sam]

Hi Chris

We had a look at the /files directory which has drwxrwsr -x Is that what you'd expect?

A Google turned this up: ​https://www.drupal.org/node/2177315

Not sure if it's relevant.

Thanks

Sam

[chris]

Good to hear that the problem can be duplicated on the dev server, I'll have a play with uploading images on there and try to track down the issue.

[chris]

I also get "Upload failed" when uploading from this URL ​https://booker-stage-20141120.transitionnetwork.org/imce?app=tinymce|url%40src I don't know how to upload a featured image, the failed uploads I have tried are not logged but I can see that some Paul has tried are.

I can't see anything in the Nginx error log, I can't see anything in the HTTP headers when an upload is attempted:

https://booker-stage-20141120.transitionnetwork.org/imce?jsop=upload&dir=uploaded/u6

POST /imce?jsop=upload&dir=uploaded/u6 HTTP/1.1
Host: booker-stage-20141120.transitionnetwork.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Cookie: SESS-XX-RETACTED-XX; LOGGED_IN=1; has_js=1
Connection: keep-alive
Content-Type: multipart/form-data; boundary=---------------------------1423109404347795154747838767
Content-Length: 52995
-----------------------------1423109404347795154747838767
Content-Disposition: form-data; name="files[imce]"; filename="restricted-boots.jpg"
Content-Type: image/jpeg

ÿØÿà
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 19 Feb 2015 13:39:05 GMT
Content-Type: text/html; charset=utf-8
Vary: Accept-Encoding
x-cookie-domain: .booker-stage-20141120.transitionnetwork.org
Set-Cookie: NoCacheID=POSTBD-XX-RETACTED-XX; expires=Thu, 19-Feb-2015 13:39:19 GMT; path=/; domain=.booker-stage-20141120.transitionnetwork.org
x-accel-expires: 0
Last-Modified: Thu, 19 Feb 2015 13:39:04 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Etag: W/"1424353144"
x-device: normal
x-speed-cache-uid: XX-RETACTED-XX;
x-speed-cache-key: /imce?jsop=upload&dir=uploaded/u6
x-nocache: Method
x-this-proto: https
x-server-name: booker-stage-20141120.transitionnetwork.org
Strict-Transport-Security: max-age=15768000
Content-Encoding: gzip
X-Firefox-Spdy: 3.1

Locating where the staging site is on the server:

 find ./ -name "*booker-stage*" | grep 20141120
./data/disk/tn/clients/tnusers/booker-stage-20141120.transitionnetwork.org
./data/disk/tn/config/server_master/nginx/vhost.d/booker-stage-20141120.transitionnetwork.org
./data/disk/tn/.drush/booker-stage-20141120.transitionnetwork.org.alias.drushrc.php
./data/disk/tn/static/transition-network-d6-34-s001j-booker/sites/www.booker-stage-20141120.transitionnetwork.org
./data/disk/tn/static/transition-network-d6-34-s001j-booker/sites/booker-stage-20141120.transitionnetwork.org
./home/tn.tnusers/.drush/booker-stage-20141120.transitionnetwork.org.alias.drushrc.php
./home/tn.ftp/.drush/booker-stage-20141120.transitionnetwork.org.alias.drushrc.php

Looking in /data/disk/tn/clients/tnusers/booker-stage-20141120.transitionnetwork.org/settings.php we have:

  $conf['file_directory_path'] = 'sites/booker-stage-20141120.transitionnetwork.org/files';
  $conf['file_directory_temp'] = 'sites/booker-stage-20141120.transitionnetwork.org/private/temp';

The files directory permissions:

drwxrwsr-x 977 tn www-data 236K Feb 19 11:44 files/

And the private/temp permissions:

drwxrwsr-x 2 tn www-data 4.0K Feb 19 13:34 temp/

These look OK to me, but making them more permissive for testing:

chmod o+w files
chmod o+w private/temp

Still get "Upload failed" so changing the permissions back:

chmod o-w files
chmod o-w private/temp

I don't know what to suggest I try next apart from looking at what BOA options there are...

[sam]

Just looking at this old thread: ​/trac/ticket/755

I see php.ini was implicated in that one. Is there a total file storage setting in php.ini we are exceeding? Or does it only set per-upload limits?

Groping in the darkness..

Cheers

Sam

[ade]

Things that have changed since the last successful site update (13/02)

DNS has been moved from United to Google
Puffin and Penguin Debian security updates

looking at ​https://booker-

stage-20141120.transitionnetwork.org/imce?app=tinymce|url%40src has the tinymce|url%40src changed at all or could it have been effected by the DNS change?

Adding more wood to the trees...

Ade

[chris]

Replying to sam:

Just looking at this old thread: ​/trac/ticket/755

I see php.ini was implicated in that one. Is there a total file storage setting in php.ini we are exceeding? Or does it only set per-upload limits?

The 100MB max upload is per POST, there are not PHP based file storage limits as far as I'm aware. I have increased the max files per user from 140MB to 250MB.

[chris]

Replying to ade:

Things that have changed since the last successful site update (13/02)

DNS has been moved from United to Google

Email was moved from United to Google, the DNS servers have't changed, they are still with Gandi.

Puffin and Penguin Debian security updates

Perhaps, but BOA has it's own PHP, it doesn't use Debian PHP, I can't think of any update that could have had an impact on Drupal -- this issue does appear to be at an application level to me (Drupal) but it might be a php-fpm issue perhaps, I doubt it is an Nginx issue and the web server simply passes all PHP requests to php-fpm.

[chris]

Two things I can try are running the BOA updater, ​https://github.com/omega8cc/boa/blob/master/docs/UPGRADE.txt (thing might install a new PHP and will result in permissions checks etc being run) and perhaps rebooting the server, but these will result in some downtime, especially the reboot as the BOA firewall takes a long time to load, perhaps 10 mins and there might be an automatic filesystem check which could also take a while.

I'll run the BOA updater now.

[chris]

Following ​https://github.com/omega8cc/boa/blob/master/docs/UPGRADE.txt

cd;wget -q -U iCab http://files.aegir.cc/BOA.sh.txt;bash BOA.sh.txt
  STATUS: Skynet Agent is Active, OK!
  STATUS: You can add the _SKYNET_MODE=OFF line in
  STATUS: /root/.barracuda.cnf to disable me, if needed.
  
  BOA Meta Installer setup completed
  Please check INSTALL.txt and UPGRADE.txt at http://bit.ly/boa-docs
  Bye
  
  [ ok ] Reloading configuration files for periodic command scheduler: cron.

barracuda up-stable
   
  BOA [14:49:33] ==> Skynet Agent v.BOA-2.4.0 welcomes you aboard!
   
  BOA [14:49:36] ==> INFO: UPGRADE
   
  Do you want to proceed with the upgrade? [Y/n] Y
  
  BOA [14:49:48] ==> Aegir on puffin.webarch.net Debian/wheezy x86_64 PHP 5.3
   
  BOA [14:50:13] ==> INFO: Installing extra Drush versions...
  BOA [14:50:21] ==> INFO: Running system packages update...
  BOA [14:51:19] ==> INFO: Upgrading required libraries and tools...
  BOA [14:52:14] ==> INFO: Installed Nginx version nginx/1.7.7, upgrade required
  BOA [14:52:15] ==> INFO: Upgrading Nginx...
  BOA [14:53:43] ==> INFO: Running aptitude full-upgrade...
  BOA [14:58:50] ==> INFO: Installing MySecureShell 1.33...
  BOA [14:59:45] ==> INFO: PHP 5.3.29 rebuild required for MariaDB 5.5.41
  BOA [14:59:45] ==> INFO: Building PHP 5.3.29 from sources, please wait...
  BOA [14:59:57] ==> INFO: Building PHP 5.3.29 part 1/3
  BOA [14:59:59] ==> INFO: Building PHP 5.3.29 part 2/3
  BOA [15:01:56] ==> INFO: Building PHP 5.3.29 part 3/3
  BOA [15:13:08] ==> INFO: Installing Zend OPcache for PHP 5.3.29...
  BOA [15:13:29] ==> INFO: Installing PhpRedis for PHP 5.3.29...
  BOA [15:13:49] ==> INFO: Installing UploadProgress for PHP 5.3.29...
  BOA [15:13:59] ==> INFO: Installing JSMin for PHP 5.3.29...
  BOA [15:14:10] ==> INFO: Installing Imagick for PHP 5.3.29...
  BOA [15:14:38] ==> INFO: Installing MailParse for PHP 5.3.29...
  BOA [15:14:53] ==> INFO: Installing IonCube x86_64 version for PHP...
  BOA [15:14:58] ==> INFO: Installed Redis version 2.8.17, upgrade required
  BOA [15:14:58] ==> INFO: Installing Redis update for Debian/wheezy...
  BOA [15:16:16] ==> INFO: Updating MariaDB and PHP configuration
  BOA [15:16:19] ==> INFO: OS and services upgrade completed
   
  BOA [15:16:19] ==> INFO: Restarting MariaDB server...
  BOA [15:16:36] ==> INFO: Upgrading MariaDB tables if necessary...
   
  Do you want to upgrade Aegir Master Instance? [Y/n] Y
  BOA [15:18:20] ==> INFO: Running Aegir Master Instance upgrade
  BOA [15:18:20] ==> INFO: Running Aegir Master Instance upgrade
  BOA [15:18:59] ==> INFO: Downloading drush mini-7-02-02-2015...
  BOA [15:19:05] ==> INFO: Installing Aegir Provision backend...
  BOA [15:19:05] ==> INFO: Downloading Drush and Provision extensions...
  BOA [15:19:06] ==> INFO: Running hostmaster-migrate, please wait...
  BOA [15:21:39] ==> INFO: Aegir Master Instance upgrade completed
  BOA [15:21:51] ==> INFO: Restarting Redis, PHP-FPM and Nginx...
  BOA [15:21:59] ==> INFO: Restarting MariaDB server...
   
  BOA [15:22:09] ==> INFO: Cleaning up system swap, it may take a moment...
  BOA [15:22:15] ==> CARD: Now charging your credit card for this magic show...
  BOA [15:22:21] ==> CARD: Thank you for entering your correct PIN code...
  BOA [15:22:27] ==> CARD: It will take a moment to process your payment...
  BOA [15:22:36] ==> JOKE: Just kidding !!! Enjoy your Aegir Hosting System :)
   
  BOA [15:22:42] ==> Final post-upgrade cleaning, one moment...
  BOA [15:29:15] ==> BYE!
  
  BARRACUDA upgrade completed
  Bye

barracuda up-stable system 
  
  Preparing the upgrade in silent mode...
  
  NOTE: There will be no progress displayed in the console
  but you will receive an e-mail once the upgrade is complete
  
  You could watch the progress in another window with command:
    tail -f /var/backups/reports/up/barracuda/150219/barracuda-up-150219-1529.log
  or wait until you will see the line: BARRACUDA upgrade completed
  
  Waiting 8 seconds...
  Starting the upgrade in silent mode now...
  
  REPORT: Successful Barracuda upgrade on puffin.webarch.net sent to chris@webarchitects.co.uk
  
  BARRACUDA upgrade completed
  Bye

octopus up-stable all both
  
  load is 4 while maxload is 600
  Octopus upgrade for User /data/disk/tn
  Waiting 4 seconds...
  
  BOA [15:35:10] ==> Skynet Agent v.BOA-2.4.0 welcomes you aboard!
  BOA [15:35:15] ==> Octopus Satellite Instance Upgrade in progress...
   
  BOA [15:35:16] ==> This Octopus URL address is tn.puffin.webarch.net
  BOA [15:35:16] ==> This Octopus PHP version is 5.3
  BOA [15:35:16] ==> This Octopus System is Debian/wheezy x86_64
  
  BOA [15:35:24] ==> UPGRADE A: Shared platforms code v.011 (new) will be created
  BOA [15:36:09] ==> UPGRADE B: Downloading drush mini-7-02-02-2015...
  BOA [15:36:16] ==> UPGRADE B: Installing Aegir Provision backend...
  BOA [15:36:17] ==> UPGRADE B: Downloading Drush and Provision extensions...
  BOA [15:36:28] ==> UPGRADE B: Hostmaster STATUS: Upgrade in progress...
  BOA [15:37:38] ==> UPGRADE B: Enhancing Aegir UI, please wait...
  BOA [15:39:07] ==> UPGRADE A: Aegir Satellite Instance upgrade completed
  
  BOA [15:40:10] ==> UPGRADE C: Shared platforms code v.011 (new) will be created
  BOA [15:40:25] ==> DISTRO: Drupal 7.34.1 P.011 installed
  BOA [15:41:02] ==> DISTRO: OpenAtrium 2.26 7.34.1 [011] installed
  BOA [15:41:02] ==> UPGRADE C: Running Platforms Verify tasks, please wait...
  BOA [15:41:05] ==> UPGRADE A: Platforms installed
  BOA [15:41:05] ==> UPGRADE A: Cleaning up various dot files...
  BOA [15:41:05] ==> UPGRADE A: Adding symlink to the system make_local
  BOA [15:41:05] ==> UPGRADE A: Adding symlink to the system safe_cache_form_clear
  BOA [15:41:05] ==> Final post-upgrade cleaning, one moment...
  BOA [15:43:39] ==> BYE!
  Waiting 4 seconds...
  Done for /data/disk/tn
  
  OCTOPUS upgrade completed
  Bye

bash /var/xdrago/manage_ltd_users.sh 

bash /var/xdrago/daily.sh 
  
  INFO: Daily maintenance start
  nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
  nginx: configuration file /etc/nginx/nginx.conf test is successful
  [ ok ] Reloading (via HUP) Nginx Server...:.
  INFO: Checking BARRACUDA version
  INFO: Version test result: OK
  INFO: Redis server will be restarted in 5 minutes
  nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
  nginx: configuration file /etc/nginx/nginx.conf test is successful
  [ ok ] Reloading (via HUP) Nginx Server...:.
  Stopping redis-server: redis-server.
  redis-server: no process found
  Starting redis-server: redis-server.
  INFO: Redis server restarted OK
  INFO: Daily maintenance complete

The email BOA sent contained:

  BOA [15:30:13] ==> Skynet Agent v.BOA-2.4.0 welcomes you aboard!                                                          
                                                                                                                            
  BOA [15:30:16] ==> INFO: UPGRADE                                                                                          
                                                                                                                            
                                                                                                                            
  BOA [15:30:20] ==> Aegir on puffin.webarch.net Debian/wheezy x86_64 PHP 5.3                                               
                                                                                                                            
  BOA [15:30:33] ==> INFO: Installing extra Drush versions...                                                               
  BOA [15:30:40] ==> INFO: Running system packages update...                                                                
  BOA [15:31:35] ==> INFO: Upgrading required libraries and tools...                                                        
  BOA [15:32:20] ==> INFO: Running aptitude full-upgrade...                                                                 
  BOA [15:33:12] ==> INFO: Updating MariaDB and PHP configuration                                                           
  BOA [15:33:14] ==> INFO: OS and services upgrade completed                                                                
                                                                                                                            
  BOA [15:33:14] ==> INFO: Aegir Master Instance upgrade skipped                                                            
  BOA [15:33:27] ==> INFO: Restarting Redis, PHP-FPM and Nginx...                                                           
  BOA [15:33:36] ==> INFO: Restarting MariaDB server...                                                                     
                                                                                                                            
  BOA [15:33:46] ==> INFO: Cleaning up system swap, it may take a moment...                                                 
  BOA [15:33:52] ==> CARD: Now charging your credit card for this magic show...                                             
  BOA [15:33:58] ==> CARD: Thank you for entering your correct PIN code...                                                  
  BOA [15:34:04] ==> CARD: It will take a moment to process your payment...                                                 
  BOA [15:34:13] ==> JOKE: Just kidding !!! Enjoy your Aegir Hosting System :)                                              
                                                                                                                            
  BOA [15:34:19] ==> Final post-upgrade cleaning, one moment...                                                             
  BOA [15:34:21] ==> BYE!                                                                                                   

So, we now have new versions of MySQL, Ngnix and PHP... I also applied the two fixes for Munin, wiki:PuffinServer#Muninconfigchanges and wiki:PuffinServer#nginxconfigchanges

And I have uploaded an image without an error, see the "restricted boots" image at the top of this article:

• ​https://booker-stage-20141120.transitionnetwork.org/news/2014-04-30/john-bell-bedfordshire-transition-groups-recent-get-together

So I'm none the wiser regarding the exact cause of this problem but it appears to be resolved on the dev site -- can someone test it on the live site please?

[ade]

Many thanks for looking into this Chris, shame we didn't pinpoint the error, but that its back up and running is a massive bonus.

Many thanks to your self and Paul..(sorry for the holiday interruption Paul..)

A