Transition Technology: Ticket #630: Archiving Transition Town Totnes site

ed — Thu, 28 Nov 2013 09:49:25 GMT

Hi chris - please estimate :)

hours, totalhours changed

chris — Fri, 29 Nov 2013 12:56:35 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

I have set it off, it won't take long, on wiki:PenguinServer:

cd /web
mkdir -p archive.transitiontowntotnes.org/www/
cd archive.transitiontowntotnes.org
httrack
Welcome to HTTrack Website Copier (Offline Browser) 3.43-9+libhtsjava.so.2
Copyright (C) Xavier Roche and other contributors
To see the option list, enter a blank line or try httrack --help
Enter project name :Transition Town Totnes Archive
Base path (return=/root/websites/) :/web/archive.transitiontowntotnes.org/www/
Enter URLs (separated by commas or blank spaces) :http://archive.transitiontowntotnes.org/
Action:
(enter) 1       Mirror Web Site(s)
        2       Mirror Web Site(s) with Wizard
        3       Just Get Files Indicated
        4       Mirror ALL links in URLs (Multiple Mirror)
        5       Test Links In URLs (Bookmark Test)
        0       Quit
: 1
Proxy (return=none) :
You can define wildcards, like: -*.gif +www.*.com/*.zip -*img_*.zip
Wildcards (return=none) :
You can define additional options, such as recurse level (-r<number>), separed by blank spaces
To see the option list, type help
Additional options (return=none) :
---> Wizard command line: httrack http://archive.transitiontowntotnes.org/  -O "/web/archive.transitiontowntotnes.org/www/Transition Town Totnes Archive"  -%v
Ready to launch the mirror? (Y/n) :Y
WARNING! You are running this program as root!
It might be a good idea to use the -%U option to change the userid:
Example: -%U smith
Mirror launched on Fri, 29 Nov 2013 12:51:01 by HTTrack Website Copier/3.43-9+libhtsjava.so.2 [XR&CO'2010]
mirroring http://archive.transitiontowntotnes.org/ with the wizard help..

I'll check back when it's done to see what we have and set up Nginx to serve it.

attachment set

chris — Sat, 30 Nov 2013 15:48:39 GMT

attachment set to hts-log.txt

HTTrack Log File for Transition Town Totnes Archive 30th Nov 2013

hours, status, totalhours changed

chris — Sat, 30 Nov 2013 15:53:40 GMT

hours changed from 0.0 to 0.25
status changed from new to accepted
totalhours changed from 0.25 to 0.5

HTTrack finally completed:

PANIC! : Too many URLs : >99995 [2870]
Done.
Thanks for using HTTrack!
*

I think there must be some infinite loops.

I have attached the log file, https://trac.transitionnetwork.org/trac/attachment/ticket/630/hts-log.txt the errors I noticed include the print versioiins of pages not being allowed by robots.txt:

13:12:10        Warning:        Link archive.transitiontowntotnes.org/print/2324 not scanned (follow robots meta tag)

And PDF version sof pages generating 500 errors:

05:25:48        Error:  "" (500) after 2 retries at link archive.transitiontowntotnes.org/printpdf/Central/DVDsPresentedByTTT (from archive.transitiontowntotnes.org/Central/DVDsPresentedByTTT)

But I haven't read through the whole log file.

Based in this trial run I expect it would take about 1 to 2 hours to get the site optimised for running HTTrack against it and also for getting the HTTrack command line right and also sorting out the Nginx config to server the site.

Once it has been done for one site we will be in a position to archive other sites quicker.

ed — Mon, 02 Dec 2013 14:18:49 GMT

We've got a £50 budget for archiving TTT...

hours, totalhours changed

chris — Tue, 03 Dec 2013 13:26:43 GMT

hours changed from 0.0 to 1.7
totalhours changed from 0.5 to 2.2

There are also errors for requests like this:

So I have added a Redirect to the .htaccess file:

RedirectPermanent /system/files  http://archive.transitiontowntotnes.org/sites/default/files
RedirectPermanent /content/  http://archive.transitiontowntotnes.org/
RedirectPermanent /drupal/  http://archive.transitiontowntotnes.org/

I have also removed the anon users ability to view the print and pdf versions of documents via http://archive.transitiontowntotnes.org/admin/user/permissions

These pages needed the Input Type to be changed to "Full HTML" for them to render properly:

These needed the "Published" box ticked:

I have enabled DirectoryIndexes here http://archive.transitiontowntotnes.org/sites/default/files/ and I'll rsync across all these files as well to be sure nothing is missing.

There are 404's for old wiki addresses:

http://archive.transitiontowntotnes.org/Building_and_housing/Planning/PlanningConsultationLocalDevelopmentFramework?q=uploads/Main/CorePolicyResponse-Part-V.pdf the file CorePolicyResponse-Part-V.pdf can't be found on the drupal site filesystem, but I found it here http://2011.archive.transitionnetwork.org/Totnes/uploads/Main/CorePolicyResponse-Part-V.pdf

I'll look at sorting out some redirects for these.

Ed, I'm going to go over time wise perhaps on this, what does £50 equate to in terms of time? I'm happy to do some time for free on this if needs be.

hours, totalhours changed

chris — Tue, 03 Dec 2013 14:39:18 GMT

hours changed from 0.0 to 0.5
totalhours changed from 2.2 to 2.7

Actually I can't see a easy fix for the URL that should point to http://2011.archive.transitionnetwork.org/Totnes/ so I'm going to ignore them due to time constraints.

Running HTTrack again:

screen
sudo -i
cd /web/archive.transitiontowntotnes.org/
rm -rf Transition\ Town\ Totnes\ Archive/
httrack http://archive.transitiontowntotnes.org/  -O "/web/archive.transitiontowntotnes.org/www"  -%v

That should do the trick, it'll take several hours to run.

Copying across the files:

cd /web/archive.transitiontowntotnes.org/www/archive.transitiontowntotnes.org/sites/default/files
rsync -av host-a:/home/ttt/archive_public_html/sites/default/files/* .

Nginx config, minus comments

server {
        listen   80;
        server_name archive.transitiontowntotnes.org ttarchive.penguin.webarch.net;
        access_log  /var/log/nginx/ttarchive.access.log;
        error_log   /var/log/nginx/ttarchive.error.log crit;
        root   "/web/archive.transitiontowntotnes.org/www/archive.transitiontowntotnes.org";
        autoindex  on;
        index  index.html;
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        include  gzip;
}
server {
        listen   443;
        server_name archive.transitiontowntotnes.org ttarchive.penguin.webarch.net;
        access_log  /var/log/nginx/ttarchive.ssl_access.log;
        error_log   /var/log/nginx/ttarchive.ssl_error.log debug;
        ssl  on;
        ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;
        ssl_certificate_key  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key;
        ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers  RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;
        root   "/web/archive.transitiontowntotnes.org/www/archive.transitiontowntotnes.org";
        autoindex  on;
        index  index.html;
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
}

The site is now available here: http://ttarchive.penguin.webarch.net/

I have added a link to the site at http://penguin.transitionnetwork.org/

And it's looking good so far:

Bytes saved:    67,75MiB               Links scanned:   251/1229 (+975)
Time:   41min17s                       Files written:   1170
Transfer rate:  25,58KiB/s (20,90KiB/s)Files updated:   0
Active connections:     4              Errors:  8

It should be done in a few hours and then the DNS for archive.transitiontowntotnes.org will need updating.

hours, totalhours changed

chris — Tue, 03 Dec 2013 14:58:44 GMT

hours changed from 0.0 to 0.1
totalhours changed from 2.7 to 2.8

There are some embedded images like this which are 404's:

http://totnes.transitionnetwork.org/sites/default/files/images/ape.thumbnail.jpg

But there are available:

http://ttarchive.penguin.webarch.net/sites/default/files/images/ape.thumbnail.jpg

We can fix this with a DNS update, these need doing:

Point totnes.transitionnetwork.org to 81.95.52.111
Point archive.transitiontowntotnes.org to 81.95.52.111

attachment set

chris — Thu, 05 Dec 2013 12:02:40 GMT

attachment set to hts-log2.txt

HTTrack Log File for Transition Town Totnes Archive 3rd Dec 2013

hours, totalhours changed

chris — Thu, 05 Dec 2013 13:21:21 GMT

hours changed from 0.0 to 1.2
totalhours changed from 2.8 to 4.0

I have attached the log from the last run of HTTrack, https://trac.transitionnetwork.org/trac/attachment/ticket/630/hts-log2.txt

Entries like this:

15:24:56 Error: "Forbidden" (403) at link archive.transitiontowntotnes.org/sites/default/files/Cycliing+at+Hebmbury+Woods.Press+Release.doc?q=system/files/Cycliing+at+Hebmbury+Woods.Press+Release.doc (from archive.transitiontowntotnes.org/central/pressreleases)

Which is for this URL, http://ttarchive.penguin.webarch.net/sites/default/files/Cycliing+at+Hebmbury+Woods.Press+Release.doc is for a file that is at http://ttarchive.penguin.webarch.net/sites/default/files/Cycliing%20at%20Hebmbury%20Woods.Press%20Release.doc

I have done some searching for a Nginx level solution for this but it doesn't look easy to fix at that level.

Also HTTrack has created this file:

http://ttarchive.penguin.webarch.net/sites/default/files/Cycliing+at+Hebmbury+Woods.Press+Releaseb2c2.html

The file that it's linked from:

http://archive.transitiontowntotnes.org/central/pressreleases

Is available from the HTTrack archive here:

http://ttarchive.penguin.webarch.net/central/pressreleases.html

This file needs this HTML:

<a href="../sites/default/files/Cycliing%2bat%2bHebmbury%2bWoods.Press%2bReleaseb2c2.html">Cycle to Hembury Woods </a>

Changing to:

<a href="../sites/default/files/Cycliing%20at%20Hebmbury%20Woods.Press%20Release.doc">Cycle to Hembury Woods </a>

Although we could manually fix these links there isn't the time / budget for this work, there are 337 files ending in .html in the sites/default/files folder and that will be roughly the number of broken links. A very keen researcher could find these files themselves from this index, http://ttarchive.penguin.webarch.net/sites/default/files/

I have also noticed that some pages didn't get HTML ticked, eg:

http://ttarchive.penguin.webarch.net/edap/home.html

I have added this to the Nginx config:

        # serve files without .html
        default_type "text/html";
        try_files  $uri $uri.html $uri/ =404;;

So a old URL like:

http://archive.transitiontowntotnes.org/localfooddirectory/home

Is available at this URL:

http://ttarchive.penguin.webarch.net/localfooddirectory/home

In addition to:

http://ttarchive.penguin.webarch.net/localfooddirectory/home.html

This should dramatically reduce the 404's from existing links to pages on archive.transitiontowntotnes.org

I have added some text to the front page:

<strong>This site is a static archive of the dynamic Drupal site created in December 2013, if you are having problems with some files not being found you can look at the <a href="/sites/default/files/">index of files</a>. See also, the archives of the wiki site from <a href="http://2011.archive.transitionnetwork.org/Totnes/">2011</a> and <a href="http://2010.archive.transitionnetwork.org/Totnes/">2010</a>.</strong>

And copied the edited index.html to home.html.

The archiving failed in the end with:

Too many URLs, giving up..(>100000)
To avoid that: use #L option for more links (example: -#L1000000)

From the times I checked on it's progress I think the massive number of links was caused by the dynamically generated events section of the site.

I have updated the DNS for archive.transitiontowntotnes.org:

dig @dns0.webarchitects.co.uk archive.transitiontowntotnes.org +short
81.95.52.111
dig @dns1.webarchitects.co.uk archive.transitiontowntotnes.org +short
81.95.52.111

And for totnes.transitionnetwork.org, but it hasn't updated yet:

dig @A.DNS.GANDI.NET totnes.transitionnetwork.org +short
81.95.52.103

I think it's now safe to delete the Drupal site and database, sorry this has taken longer than expected, I think this ticket can now be closed, I have also added a link to this ticket from the front page of the archive in case people are desperate to find something.

hours, status, totalhours changed; resolution set

chris — Fri, 06 Dec 2013 10:33:46 GMT

hours changed from 0.0 to 0.3
status changed from accepted to closed
resolution set to fixed
totalhours changed from 4.0 to 4.3

The DNS for http://totnes.transitionnetwork.org/ has updated:

dig @A.DNS.GANDI.NET totnes.transitionnetwork.org +short
81.95.52.111

Regarding the original ticket questions:

Replying to ed:

Please estimate to archive TTT site as per conversation with Ed:

convert to html

Done, 1.5 hours including this comment.

host on Penguin (incl. any likely issues for Penguin doing this)

Done, no issues.

In terms of disk space it was using 1.7GB:

cd /web/archive.transitiontowntotnes.org/
du -h --max-depth=2
1.3G    ./www/archive.transitiontowntotnes.org
397M    ./www/hts-cache
8.0K    ./www/archive.transitiontowntotnes.org_
1.7G    ./www
1.7G    .

I have reduced this to 1.3GB by deleting all the HTTrack files and moving the document root:

cd /web/archive.transitiontowntotnes.org/
du -h --max-depth=1
1.3G    ./www
1.3G    .

There is still lots of space on the server:

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2             40G   16G   22G  42% /

any ongoing maintenance

It's static HTML, the only issue I can think of is if something in the archive needs editing or deleting.

I have deleted the Drupal site off the server it was running on, host-a.ecodis.net but we still have backups of it for now.

I have updated the list of sites / servers here wiki:WikiStart#ServersandWebsites

And added the archive to the list of sites here wiki:PenguinServer#totnes.transitionnetwork.org

I have added the site to the list of sites here http://penguin.transitionnetwork.org/

And for reference here is the Nginx config:

# totnes.transitionnetwork.org virtual server
# http://nginx.org/en/docs/http/ngx_http_core_module.html#server
server {
        # listen for ipv4
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#listen
        #listen   8000;
        listen   80;
        # server name and server aliases
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#server_name
        server_name totnes.transitionnetwork.org archive.transitiontowntotnes.org ttarchive.penguin.webarch.net;
        # logs, error log levels: info | notice | warn | error | crit | alert
        # http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log
        # http://nginx.org/en/docs/ngx_core_module.html#error_log
        access_log  /var/log/nginx/ttarchive.access.log;
        error_log   /var/log/nginx/ttarchive.error.log crit;
        # document root
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#root
        root   "/web/archive.transitiontowntotnes.org/www";
        # serve files without .html
        default_type "text/html";
        try_files  $uri $uri.html $uri/ =404;
        # http://nginx.org/en/docs/http/ngx_http_autoindex_module.html#autoindex
        autoindex  on;
        # document index
        # http://nginx.org/en/docs/http/ngx_http_index_module.html#index
        index  index.html;
        # location match
        # http://nginx.org/en/docs/http/ngx_http_core_module.html#location
        # Prevent access to any files starting with a dot, like .htaccess
        # or text editor temp files
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
        # gzip content
        include  gzip;
}
# totnes.transitionnetwork.org https virtual server
server {
        #listen   4430;
        listen   443;
        server_name totnes.transitionnetwork.org archive.transitiontowntotnes.org ttarchive.penguin.webarch.net;
        access_log  /var/log/nginx/ttarchive.ssl_access.log;
        error_log   /var/log/nginx/ttarchive.ssl_error.log crit;
        ssl  on;
        ssl_certificate  /etc/ssl/transitionnetwork.org/transitionnetwork.org.chained.pem;
        ssl_certificate_key  /etc/ssl/transitionnetwork.org/transitionnetwork.org.key;
        #ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        #ssl_ciphers  RC4-SHA:HIGH:!ADH:!SSLv2:!aNULL;
        ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers  RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;
        root   "/web/archive.transitiontowntotnes.org/www";
        autoindex  on;
        index  index.html;
        # serve files without .html
        default_type "text/html";
        try_files  $uri $uri.html $uri/ =404;
        # Prevent access to any files starting with a dot, like .htaccess
        # or text editor temp files
        location ~ /\. {
                access_log off;
                log_not_found off;
                deny all;
        }
}

chris — Fri, 06 Dec 2013 10:37:11 GMT

Replying to chris:

Replying to ed:

Please estimate to archive TTT site as per conversation with Ed:

convert to html

Done, 1.5 hours including this comment.

Sorry I can't see where I got that figure from, I'm afraid the total time is 4.3 hours.

I said earlier ticket:630#comment:5

Ed, I'm going to go over time wise perhaps on this, what does £50 equate to in terms of time? I'm happy to do some time for free on this if needs be.

ed — Mon, 09 Dec 2013 11:22:37 GMT

£50 is just under two hours of your time I think Chris

chris — Mon, 09 Dec 2013 12:25:34 GMT

Replying to ed:

£50 is just under two hours of your time I think Chris

This ticket has clocked up 4.3 hours so shall we deduct 2.5 hours from my time this month when it comes to billing?

ed — Mon, 09 Dec 2013 14:32:05 GMT

yes please chris - however if we don't hit December's £670 limit, I'm happy to include as much of this as fits

hours, totalhours changed

chris — Fri, 31 Oct 2014 18:02:04 GMT

hours changed from 0.0 to 0.25
totalhours changed from 4.3 to 4.55

Frances at TTT has asked that the archived TTT Drupal site at http://archive.transitiontowntotnes.org/ be excluded from search engine results.

To prevent the site being dropped from the Internet Archive I have allowed their robot but excluded others:

User-agent: ia_archiver
Disallow:
User-agent: *
Disallow: /

See http://archive.transitiontowntotnes.org/robots.txt

For background information see: