Transition Technology: Ticket #706: Upgrade Mediawiki to 1.22.5 and install the new VisualEditor

ed — Wed, 26 Mar 2014 15:54:44 GMT

I am going to start asking for a wiki package with a much more significant WYSIWYG in the next few months. fyi.

chris — Wed, 26 Mar 2014 15:59:50 GMT

Replying to ed:

I am going to start asking for a wiki package with a much more significant WYSIWYG in the next few months. fyi.

We could update to 1.22.5 this week and switch to the new VisualEditor at the same time if you would like.

chris — Wed, 26 Mar 2014 16:03:21 GMT

More on the new WYSIWYG editor here https://www.mediawiki.org/wiki/VisualEditor

Test it here: https://www.mediawiki.org/wiki/VisualEditor:Test

ed — Wed, 26 Mar 2014 16:06:24 GMT

Whatever is available and more friendly for management people like me, and will encourage better documentation from techs, we need it.

Documentation will be in the terms and conditions for future contracts and we will need to offer a *nice* solution. *Nice* from my perspective :)

chris — Wed, 26 Mar 2014 16:11:51 GMT

Replying to ed:

Whatever is available and more friendly for management people like me

Can you test this to see if it is acceptable?

https://www.mediawiki.org/wiki/VisualEditor:Test

hours, totalhours changed

chris — Wed, 26 Mar 2014 16:26:39 GMT

hours changed from 0.0 to 0.25
totalhours changed from 0.0 to 0.25

The VisualEditor project aims to create a reliable rich-text editor for MediaWiki.

https://www.mediawiki.org/wiki/Extension:VisualEditor

This is clearly the one to use and it looks stable enough for us:

Use at your own risk; it's not ready for production deployment except for experts!

Parsoid will be needed:

This will take some time to set up and document (a few hours I guess) but the result would be that we would using the new default WYSIWYG Wikipedia editor and this is clearly going to be the main one for the next few years, all these others will be abandoned in time:

https://www.mediawiki.org/wiki/WYSIWYG_editor

Should I change the title of this ticket to "Upgrade Mediawiki to 1.22.5 and install the new VisualEditor"?

chris — Thu, 27 Mar 2014 18:57:30 GMT

I don't think I'll time tonight to do the 1.19.14 upgrade as it's not yet out.

Tomorrow, if I have an answer to this:

Replying to chris:

Should I change the title of this ticket to "Upgrade Mediawiki to 1.22.5 and install the new VisualEditor"?

I could either upgrade to 1.19.14 or 1.22.4 with the new WYSIWYG editor --Ed which would you like me to do?

ed — Fri, 28 Mar 2014 06:36:46 GMT

Thanks Chris, upgrade to 1.22.4 with the new WYSIWYG editor

summary changed

chris — Fri, 28 Mar 2014 12:57:22 GMT

summary changed from Mediawiki 1.19.14 to Upgrade Mediawiki to 1.22.5 and install the new VisualEditor

hours, totalhours changed

chris — Fri, 28 Mar 2014 14:38:08 GMT

hours changed from 0.0 to 1.75
totalhours changed from 0.25 to 2.0

The existing wiki site is running Mediawiki 1.19.13 and has the following installed extensions:

SyntaxHighlight (Version 1.0.8.10) https://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi "This extension is bundled with MediaWiki 1.21 and above. Thus you do not have to download it again."
Piwik Integration (Version 2.0.0) https://www.mediawiki.org/wiki/Extension:Piwik_Integration
WikiEditor (Version 0.3.1) https://www.mediawiki.org/wiki/Extension:WikiEditor "This extension is bundled with MediaWiki 1.18 and above. Thus you do not have to download it again."

So the only extension that will need manually downloading is Piwik Integration.

Following the upgrade notes, trac:wiki/MediaWiki#Updates

sudo -i
cd /web/wiki.transitionnetwork.org
export MW="1.22.5"
wget http://releases.wikimedia.org/mediawiki/1.22/mediawiki-$MW.tar.gz -O mediawiki-$MW.tar.gz
wget http://releases.wikimedia.org/mediawiki/1.22/mediawiki-$MW.tar.gz.sig -O mediawiki-$MW.tar.gz.sig
  gpg --verify mediawiki-$MW.tar.gz.sig
  gpg: Signature made Fri Mar 28 00:21:11 2014 GMT using RSA key ID 7F901A30
  gpg: Good signature from "Mark A. Hershberger <mah@everybody.org>"
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: 3CEF 8262 806D 3F0B 6BA1  DBDD 7956 EE47 7F90 1A30
tar -zxvf mediawiki-$MW.tar.gz
rsync -av mediawiki-$MW/ www/
chown root:root -R www/
chown -R www-data:www-data www/cache/
chown -R www-data:www-data www/images/
cd www/maintenance/
php update.php
cd -
wget https://github.com/DaSchTour/piwik-mediawiki-extension/archive/master.zip
unzip master.zip
rsync -av piwik-mediawiki-extension-master/ www/extensions/Piwik/
rm mediawiki-$MW.tar.gz mediawiki-$MW.tar.gz.sig
rm -rf mediawiki-$MW
rm -rf piwik-mediawiki-extension-master/
rm master.zip

And now on the https://wiki.transitionnetwork.org/Special:Version page we have:

MediaWiki 1.22.5
SyntaxHighlight (Version 1.0.8.11)
Piwik Integration (Version 2.4.0)
WikiEditor (Version 0.3.1)

Now for the https://www.mediawiki.org/wiki/Extension:VisualEditor install, following https://www.mediawiki.org/wiki/Parsoid/Setup

cd /web/wiki.transitionnetwork.org
git clone https://gerrit.wikimedia.org/r/p/mediawiki/services/parsoid
aptitude install nodejs npm
  Couldn't find any package whose name or description matched "nodejs"
  Couldn't find any package matching "npm".

So, following https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager#backports /etc/apt/sources.list.d/backports.list was created containing:

deb http://ftp.debian.org/debian wheezy-backports main

And following https://github.com/joyent/node/wiki/backports.debian.org

aptitude update
aptitude install nodejs-legacy curl
curl --insecure https://www.npmjs.org/install.sh | bash

Then:

cd /web/wiki.transitionnetwork.org/parsoid
npm install

Then create /web/wiki.transitionnetwork.org/parsoid/api/localsettings.js containing:

/*
 * This is a sample configuration file.
 *
 * Copy this file to localsettings.js and edit that file to fit your needs.
 *
 * Also see the file ParserService.js for more information.
 */
exports.setup = function( parsoidConfig ) {
        // The URL of your MediaWiki API endpoint
        //
        // We pre-define wikipedias as 'enwiki', 'dewiki' etc. Similarly for
        // other projects: 'enwiktionary', 'enwikiquote', 'enwikibooks',
        // 'enwikivoyage' etc.
        //
        // Optionally, you can also pass in a proxy specific to this prefix
        // (overrides defaultAPIProxyURI), or null to disable proxying for
        // this end point.
        //parsoidConfig.setInterwiki( 'localhost', 'http://localhost/w/api.php' );
        parsoidConfig.setInterwiki( 'localhost', 'http://wiki.transitionnetwork.org/api.php' );
        // A default proxy to connect to the API endpoints. Default: undefined
        // (no proxying). Overridden by per-wiki proxy config in setInterwiki.
        // parsoidConfig.defaultAPIProxyURI = 'http://proxy.example.org:8080';
        // Enable debug mode (prints extra debugging messages)
        // parsoidConfig.debug = true;
        // Use the PHP preprocessor to expand templates via the MW API (default true)
        //parsoidConfig.usePHPPreProcessor = false;
        // Use selective serialization (default false)
        parsoidConfig.useSelser = true;
        // Allow cross-domain requests to the API (default '*')
        // Sets Access-Control-Allow-Origin header
        // disable:
        //parsoidConfig.allowCORS = false;
        // restrict:
        parsoidConfig.allowCORS = 'wiki.transitionnetwork.org';
};

The start the server:

cd /web/wiki.transitionnetwork.org/parsoid
export INTERFACE=127.0.0.1
node api/server.js

So it looks like it's working, create a /etc/init.d/pslparsoid file following the example at https://www.mediawiki.org/wiki/Talk:Parsoid

#! /bin/sh
# /etc/init.d/pslparsoid
### BEGIN INIT INFO
# Provides:             pslparsoid
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Start Parsoid daemon
# Description:          Start up Parsoid, Debian run-level script by PSL GbR (draft)
### END INIT INFO
# Some things that run always
#env VCAP_APP_PORT="8000"
#env NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
export VCAP_APP_PORT="8000"
export NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting Parsoid..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js >> nohup.out 2>&1 &
    ;;
  restart)
    echo "Stopping Parsoid..."
    echo "killall -9 node, could do more here..."
    killall -9 node
    ;;
  'reload'|'force-reload')
    echo "Could fail..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js >> nohup.out 2>&1 &
    ;;
  stop)
    echo "Stopping Parsoid..."
    killall -9 node
    ;;
  *)
    echo "Usage: /etc/init.d/pslparsoid {start|stop|restart|reload|force-reload}"
    exit 1
    ;;
esac
exit 0

Install the VisualEditor extension:

cd /web/wiki.transitionnetwork.org
wget https://codeload.github.com/wikimedia/mediawiki-extensions-VisualEditor/legacy.tar.gz/REL1_22 -O VisualEditor.tar.gz
tar -zxvf VisualEditor.tar.gz
mkdir /web/wiki.transitionnetwork.org/www/extensions/VisualEditor
rsync -av wikimedia-mediawiki-extensions-VisualEditor-43e96d7/ www/extensions/VisualEditor/
mv
rm -rf VisualEditor.tar.gz wikimedia-mediawiki-extensions-VisualEditor-43e96d7/

Edit LocalSettings.php and add:

require_once("$IP/extensions/VisualEditor/VisualEditor.php");
// OPTIONAL: Enable VisualEditor in other namespaces
// By default, VE is only enabled in NS_MAIN
//$wgVisualEditorNamespaces[] = NS_PROJECT;
// Enable by default for everybody
$wgDefaultUserOptions['visualeditor-enable'] = 1;
// Don't allow users to disable it
// $wgHiddenPrefs[] = 'visualeditor-enable';
// OPTIONAL: Enable VisualEditor's experimental code features
//$wgVisualEditorEnableExperimentalCode = true;
// URL to the Parsoid instance
// MUST NOT end in a slash due to Parsoid bug
$wgVisualEditorParsoidURL = 'http://localhost:8000';

Start the Parsoid server and set it to run on a reboot:

/etc/init.d/pslparsoid start
update-rc.d pslparsoid defaults

And we now have a new WYSIWYG editor for the Main namespace (most pages):

https://wiki.transitionnetwork.org/index.php?title=Main_Page&veaction=edit

TODO:

Test it
Document it

hours, status, totalhours changed; resolution set

chris — Fri, 28 Mar 2014 23:07:35 GMT

hours changed from 0.0 to 0.25
status changed from new to closed
resolution set to fixed
totalhours changed from 2.0 to 2.25

I have done some testing and the new editor seems to be working fine, I have also updated the wiki:MediaWiki page, the upgrade notes can be updated next time it is upgraded -- this will be a more complex task in the future.

Closing this ticket now.

hours, totalhours changed

chris — Fri, 28 Mar 2014 23:37:41 GMT

hours changed from 0.0 to 0.25
totalhours changed from 2.25 to 2.5

It's worth noting that adding nodejs has increased the memory usage on the server, see:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/memory.html

I have updated the /etc/munin/plugin-conf.d/munin-node files:

[multips_memory]
env.names php5-fpm munin-node nginx mysqld tracd python PassengerHelper PassengerLoggin PassengerWatchd nodejs ruby1.8

To keep track of this here:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/multips_memory.html

There are 11 nodejs processes:

ps -lA | grep nodejs | wc -l
11

This is probably way more than we need, it's using over 500MB of RAM, but I can't find where to set this, we might need to allocate more RAM to the server.

attachment set

chris — Tue, 01 Apr 2014 11:52:24 GMT

attachment set to penguin_2014-04-01_multips_memory-day.png

hours, totalhours changed

chris — Tue, 01 Apr 2014 11:55:03 GMT

hours changed from 0.0 to 0.5
totalhours changed from 2.5 to 3.0

Looking at /web/wiki.transitionnetwork.org/parsoid/api/server.js it's clear why 11 server are started:

 // Start a few more workers than there are cpus visible to the OS,
 // so that we get some degree of parallelism even on single-core
 // systems. A single long-running request would otherwise hold up
 // all concurrent short requests.

We have these options:

cd /web/wiki.transitionnetwork.org/parsoid/api
./server.js -h
  Usage: node ./server.js [-h|-v] [--param[=val]]
  Options:
    -n, --num-workers  [default: 11]
    -c, --config       [default: "/web/wiki.transitionnetwork.org/parsoid/api/localsettings.js"]
    -v, --version      [default: false]
    -h, --help         [default: false]

So /etc/init.d/pslparsoid has been edited to:

#! /bin/sh
# /etc/init.d/pslparsoid
### BEGIN INIT INFO
# Provides:             pslparsoid
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Start Parsoid daemon
# Description:          Start up Parsoid, Debian run-level script by PSL GbR (draft)
### END INIT INFO
# Some things that run always
#env VCAP_APP_PORT="8000"
#env NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
export VCAP_APP_PORT="8000"
export NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
NUM_WORKERS="4"
# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting Parsoid with ${NUM_WORKERS} worker processes..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js -n ${NUM_WORKERS} >> nohup.out 2>&1 &
    ;;
  restart)
    echo "Stopping Parsoid..."
    echo "killall -9 node, could do more here..."
    killall -9 node
    ;;
  'reload'|'force-reload')
    echo "Could fail..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js -n ${NUM_WORKERS} >> nohup.out 2>&1 &
    ;;
  stop)
    echo "Stopping Parsoid..."
    killall -9 node
    ;;
  *)
    echo "Usage: /etc/init.d/pslparsoid {start|stop|restart|reload|force-reload}"
    exit 1
    ;;
esac
exit 0

And it was stopped and restarted and the number of processes checked:

/etc/init.d/pslparsoid stop
/etc/init.d/pslparsoid start
ps -lA | grep nodejs | wc -l
4

We should now see less memory usage overall:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/memory.html

And specifically less used by nodejs:

https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/multips_memory.html

I have done some testing of the new editor on a new https://wiki.transitionnetwork.org/Sandbox page and it seems good to me.

There are now two edit options, "Edit" and "Edit source" -- the "Edit" option is the new editor and the "Edit source" is the old one.

I have added a wiki:MediaWiki#Parsoid section to the documentation.

hours, totalhours changed

chris — Tue, 01 Apr 2014 12:38:26 GMT

hours changed from 0.0 to 0.1
totalhours changed from 3.0 to 3.1

I have just dropped the number of node.js processes from 4 to 2 -- we don't use the Mediawiki site enough to justify more at the moment, this will half the RAM usage.

ed — Tue, 01 Apr 2014 13:52:52 GMT

nice

chris — Tue, 01 Apr 2014 14:08:24 GMT

Replying to ed:

nice

Indeed, it's by far the best HTML WYSIWYG editor I've ever used, the only issue I found was no table support, but I expect that this will be on the todo list.

Now we need to work out how the Transition Network can make more use of their wiki!

ed — Tue, 01 Apr 2014 14:41:09 GMT

atm everyone is all over google docs. make it as good as that and we've a chance.

chris — Tue, 01 Apr 2014 16:58:20 GMT

Replying to ed:

atm everyone is all over google docs. make it as good as that and we've a chance.

Interesting.

Are people doing real-time multi-user editing?

If they are then we should look at setting up pad.transitionnetwork.org running Etherpad Lite:

http://etherpad.org/

It needs node.js and we have just installed this...

https://github.com/ether/etherpad-lite#installation

Are people also using Google for spreadsheets? If they are then Ethercalc at calc.transitionnetwork.org is the answer:

http://ethercalc.net/

It also uses node.js:

http://ethercalc.net/#install

Shall I open a new ticket for this...?

ed — Tue, 01 Apr 2014 19:44:14 GMT

No new tickets for now. TN have an intranet group working on doing some requirements as we type. Needs range from Open Atrium stuff to google docs and Huddle shared docs and groups stuff. Etherpad has the shared editing which is good - imo the killer app for google docs is the commenting - e.g. https://docs.google.com/document/d/1pINZIzr4NhqBS2KYph_llyuQ4ofSVNDYAE21tKAI-yQ/edit