Data loss on save, Cache Router issue?

[jim]

• It seems admin pages are now not saving in some cases.
• #245 - Profile editing bugs
• #260 - Paredes profile issues
• lots of errors in watchdog from the Cache Router module, e.g.:

  Type	cache
  Date	Wednesday, 29 June 2011 - 11:51pm
  User	Transition Admin
  Location	https://www.transitionnetwork.org/admin/settings/piwik
  Referrer	https://www.transitionnetwork.org/admin/settings/piwik
  Message	Cache write error, failed to open file "sites/default/files/filecache/cache/8/variables-87cd8b8808600624d8c590cfc2e6e94b"
  Severity	error
  Hostname	86.184.163.71
  

I think we have a critical data saving issue, probably related to Cache Router not being able to get a write lock for the 'Variables' table's cache file. I've already found and added to the issue at: ​http://drupal.org/node/1152310

Need to disable module on LIVE tomorrow to see if the issues stop, or track down other possible sources. Thought it deserved a ticket as it's possibly losing us entered user content - and probably more.

[ed]

definitely worth a ticket!

NB: we may not have chris today if that's relevant - it's J30 strike day..

[jim]

Don't need Chris for this one.

Cache router now disabled, and lo, the errors hath become quiet verily...

I reckon the update I made in late May from Cache Router RC1 -> RC2 caused all of the above issues we face and more.

I have doubts about the integrity of some of the system, especially stuff done in the last couple of weeks. For example, now seeing errors like:

Location	http://www.transitionnetwork.org/resources
Referrer	
Message	Warning: Invalid argument supplied for foreach() in fivestar_views_value_display_handler() (line 1668 of /web/transitionnetwork.org/www/sites/all/modules/fivestar/fivestar.module).

And that line of code tries to access the site's data schema to connect nodes to their votes

    // Find the VotingAPI tag for this field.
    foreach ($field->query->table_queue[$field->relationship]['join']->extra as $votingapi_setting) {
      if ($votingapi_setting['field'] == 'tag') {
        $tag = $votingapi_setting['value'];
      }
    }

So we may have to re-install VotingAPI and FiveStar?. Will re-save the options forms first to see if that corrects the issue.

But the main thing is: We cannot use CacheRouter? in File mode any more. Varnish/Memcache? use it for different backends so they don't use file caching. I'll continue to add to this ticket as I close down other errors and get any joy on ​http://drupal.org/node/1152310

[ed]

good work that man!

[jim]

Duh.

[jim]

I've removed and re-installed Fivestart and VotingAPI but the errors were still appearing in Watchdog, one for each FiveStar? being shown.

So I've added the PHP code in a field I fixed the issue with in the Group Resources view and it's all cleared up.

Downgrading, but will keep open until something happens on ​http://drupal.org/node/1152310

[jim]

I noted as soon as we re-added CacheRouter? this morning, lots of settings were odd and old, plus Piwik hasnt' been getting any stats for a while.

I've since realised perhaps this is to do with the remaining files in the /sites/default/files/filecache folder being old and/or locked somehow still.

I've just completely wiped (sudo rm -R *) the contents of the folder and all the settings are magically back, and piwik works, as does the new /video page!

I now have even less faith in CR file engine, since when one 'clears all caches' it should wipe these files too... It appears it's not been. That is very dangerous.

I'll continue to keep an eye on things...

[jim]

OK, getting bored of: a) having to keep manually wiping the filecache folder, b) the lack of response to a critical issue by the CR people.

So I'm rolling back to a previous version (RC1) until we're ready to get Varnish going on LIVE.

Will update tonight out of hours.

[chris]

getting bored of: a) having to keep manually wiping the filecache folder ... rolling back to a previous version ... Will update tonight out of hours.

Did you roll back already? If not I could sort a cron job to delete the files that need deleting?

[jim]

Didn't have time or energy to do it last night...

Don't worry about it Chris, the file cache as of RC2 is fundamentally broken, RC1 worked and didn't cause file locking issues. A single process (request) can lock its own files and get confused!

Roll back will happen sometime today/tongight depending on other tickets.

[jim]

RC1 has the same obsolete data appearing issue as RC2.

The sooner we drop the file engine, the better. In the mean time I'll try RC2 back on DEV to compare to RC1 and then, once Chris is ready, push RC2 back to the server.

Next week, obviously.

[ed]

hoping that the varnish thing works...

[jim]

Since #224 is done we're no-longer using cache router's file cache, so this is closed.