Upgrade Mediawiki to 1.22.5 and install the new VisualEditor

[chris]

From ​MediaWiki-announce:

this is a notice that on Thursday, March 27th between 17:00-18:00 UTC (Thursday, March 27th, 9:00-10:00am PST) we will release security and maintenance updates for current and supported branches of the MediaWiki software. Downloads and patches will be available at that time.

[ed]

I am going to start asking for a wiki package with a much more significant WYSIWYG in the next few months. fyi.

[chris]

Replying to ed:

I am going to start asking for a wiki package with a much more significant WYSIWYG in the next few months. fyi.

We could update to 1.22.5 this week and switch to the new ​VisualEditor at the same time if you would like.

[chris]

More on the new WYSIWYG editor here ​https://www.mediawiki.org/wiki/VisualEditor

Test it here: ​https://www.mediawiki.org/wiki/VisualEditor:Test

[ed]

Whatever is available and more friendly for management people like me, and will encourage better documentation from techs, we need it.

Documentation will be in the terms and conditions for future contracts and we will need to offer a *nice* solution. *Nice* from my perspective :)

[chris]

Replying to ed:

Whatever is available and more friendly for management people like me

Can you test this to see if it is acceptable?

​https://www.mediawiki.org/wiki/VisualEditor:Test

[chris]

The VisualEditor project aims to create a reliable rich-text editor for MediaWiki.

​https://www.mediawiki.org/wiki/Extension:VisualEditor

This is clearly the one to use and it looks stable enough for us:

Use at your own risk; it's not ready for production deployment except for experts!

Parsoid will be needed:

• ​https://www.mediawiki.org/wiki/Parsoid#Getting_started
• ​https://www.mediawiki.org/wiki/Parsoid/Setup

This will take some time to set up and document (a few hours I guess) but the result would be that we would using the new default WYSIWYG Wikipedia editor and this is clearly going to be the main one for the next few years, all these others will be abandoned in time:

• ​https://www.mediawiki.org/wiki/WYSIWYG_editor

Should I change the title of this ticket to "Upgrade Mediawiki to 1.22.5 and install the new VisualEditor"?

[chris]

I don't think I'll time tonight to do the 1.19.14 upgrade as it's not yet out.

Tomorrow, if I have an answer to this:

Replying to chris:

Should I change the title of this ticket to "Upgrade Mediawiki to 1.22.5 and install the new VisualEditor"?

I could either upgrade to 1.19.14 or 1.22.4 with the new WYSIWYG editor --Ed which would you like me to do?

[ed]

Thanks Chris, upgrade to 1.22.4 with the new WYSIWYG editor

[chris]

The existing wiki site is running Mediawiki 1.19.13 and has the following ​installed extensions:

• SyntaxHighlight (Version 1.0.8.10) ​https://www.mediawiki.org/wiki/Extension:SyntaxHighlight_GeSHi "This extension is bundled with MediaWiki 1.21 and above. Thus you do not have to download it again."
• Piwik Integration (Version 2.0.0) ​https://www.mediawiki.org/wiki/Extension:Piwik_Integration
• WikiEditor (Version 0.3.1) ​https://www.mediawiki.org/wiki/Extension:WikiEditor "This extension is bundled with MediaWiki 1.18 and above. Thus you do not have to download it again."

So the only extension that will need manually downloading is Piwik Integration.

Following the upgrade notes, ​trac:wiki/MediaWiki#Updates

sudo -i
cd /web/wiki.transitionnetwork.org
export MW="1.22.5"
wget http://releases.wikimedia.org/mediawiki/1.22/mediawiki-$MW.tar.gz -O mediawiki-$MW.tar.gz
wget http://releases.wikimedia.org/mediawiki/1.22/mediawiki-$MW.tar.gz.sig -O mediawiki-$MW.tar.gz.sig
  gpg --verify mediawiki-$MW.tar.gz.sig 
  gpg: Signature made Fri Mar 28 00:21:11 2014 GMT using RSA key ID 7F901A30
  gpg: Good signature from "Mark A. Hershberger <mah@everybody.org>"
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: 3CEF 8262 806D 3F0B 6BA1  DBDD 7956 EE47 7F90 1A30
tar -zxvf mediawiki-$MW.tar.gz
rsync -av mediawiki-$MW/ www/
chown root:root -R www/
chown -R www-data:www-data www/cache/
chown -R www-data:www-data www/images/
cd www/maintenance/
php update.php 
cd -
wget https://github.com/DaSchTour/piwik-mediawiki-extension/archive/master.zip
unzip master.zip 
rsync -av piwik-mediawiki-extension-master/ www/extensions/Piwik/
rm mediawiki-$MW.tar.gz mediawiki-$MW.tar.gz.sig
rm -rf mediawiki-$MW
rm -rf piwik-mediawiki-extension-master/
rm master.zip 

And now on the ​https://wiki.transitionnetwork.org/Special:Version page we have:

• MediaWiki 1.22.5
• SyntaxHighlight (Version 1.0.8.11)
• Piwik Integration (Version 2.4.0)
• WikiEditor (Version 0.3.1)

Now for the ​https://www.mediawiki.org/wiki/Extension:VisualEditor install, following ​https://www.mediawiki.org/wiki/Parsoid/Setup

cd /web/wiki.transitionnetwork.org
git clone https://gerrit.wikimedia.org/r/p/mediawiki/services/parsoid
aptitude install nodejs npm
  Couldn't find any package whose name or description matched "nodejs"
  Couldn't find any package matching "npm".

So, following ​https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager#backports /etc/apt/sources.list.d/backports.list was created containing:

deb http://ftp.debian.org/debian wheezy-backports main

And following ​https://github.com/joyent/node/wiki/backports.debian.org

aptitude update
aptitude install nodejs-legacy curl
curl --insecure https://www.npmjs.org/install.sh | bash

Then:

cd /web/wiki.transitionnetwork.org/parsoid
npm install

Then create /web/wiki.transitionnetwork.org/parsoid/api/localsettings.js containing:

/*
 * This is a sample configuration file.
 *
 * Copy this file to localsettings.js and edit that file to fit your needs.
 *
 * Also see the file ParserService.js for more information.
 */

exports.setup = function( parsoidConfig ) {
        // The URL of your MediaWiki API endpoint
        //
        // We pre-define wikipedias as 'enwiki', 'dewiki' etc. Similarly for
        // other projects: 'enwiktionary', 'enwikiquote', 'enwikibooks',
        // 'enwikivoyage' etc.
        //
        // Optionally, you can also pass in a proxy specific to this prefix
        // (overrides defaultAPIProxyURI), or null to disable proxying for
        // this end point.
        //parsoidConfig.setInterwiki( 'localhost', 'http://localhost/w/api.php' );
        parsoidConfig.setInterwiki( 'localhost', 'http://wiki.transitionnetwork.org/api.php' );

        // A default proxy to connect to the API endpoints. Default: undefined
        // (no proxying). Overridden by per-wiki proxy config in setInterwiki.
        // parsoidConfig.defaultAPIProxyURI = 'http://proxy.example.org:8080';

        // Enable debug mode (prints extra debugging messages)
        // parsoidConfig.debug = true;

        // Use the PHP preprocessor to expand templates via the MW API (default true)
        //parsoidConfig.usePHPPreProcessor = false;

        // Use selective serialization (default false)
        parsoidConfig.useSelser = true;

        // Allow cross-domain requests to the API (default '*')
        // Sets Access-Control-Allow-Origin header
        // disable:
        //parsoidConfig.allowCORS = false;
        // restrict:
        parsoidConfig.allowCORS = 'wiki.transitionnetwork.org';
};

The start the server:

cd /web/wiki.transitionnetwork.org/parsoid
export INTERFACE=127.0.0.1
node api/server.js

So it looks like it's working, create a /etc/init.d/pslparsoid file following the example at ​https://www.mediawiki.org/wiki/Talk:Parsoid

#! /bin/sh
# /etc/init.d/pslparsoid
### BEGIN INIT INFO
# Provides:             pslparsoid
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Start Parsoid daemon
# Description:          Start up Parsoid, Debian run-level script by PSL GbR (draft)
### END INIT INFO

# Some things that run always
#env VCAP_APP_PORT="8000"
#env NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
export VCAP_APP_PORT="8000"
export NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting Parsoid..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js >> nohup.out 2>&1 &
    ;;
  restart)
    echo "Stopping Parsoid..."
    echo "killall -9 node, could do more here..."
    killall -9 node
    ;;
  'reload'|'force-reload')
    echo "Could fail..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js >> nohup.out 2>&1 &
    ;;
  stop)
    echo "Stopping Parsoid..."
    killall -9 node
    ;;
  *)
    echo "Usage: /etc/init.d/pslparsoid {start|stop|restart|reload|force-reload}"
    exit 1
    ;;
esac

exit 0

Install the VisualEditor extension:

cd /web/wiki.transitionnetwork.org
wget https://codeload.github.com/wikimedia/mediawiki-extensions-VisualEditor/legacy.tar.gz/REL1_22 -O VisualEditor.tar.gz
tar -zxvf VisualEditor.tar.gz 
mkdir /web/wiki.transitionnetwork.org/www/extensions/VisualEditor
rsync -av wikimedia-mediawiki-extensions-VisualEditor-43e96d7/ www/extensions/VisualEditor/
mv 
rm -rf VisualEditor.tar.gz wikimedia-mediawiki-extensions-VisualEditor-43e96d7/

Edit LocalSettings.php and add:

require_once("$IP/extensions/VisualEditor/VisualEditor.php");

// OPTIONAL: Enable VisualEditor in other namespaces
// By default, VE is only enabled in NS_MAIN
//$wgVisualEditorNamespaces[] = NS_PROJECT;

// Enable by default for everybody
$wgDefaultUserOptions['visualeditor-enable'] = 1;

// Don't allow users to disable it
// $wgHiddenPrefs[] = 'visualeditor-enable';

// OPTIONAL: Enable VisualEditor's experimental code features
//$wgVisualEditorEnableExperimentalCode = true;

// URL to the Parsoid instance
// MUST NOT end in a slash due to Parsoid bug
$wgVisualEditorParsoidURL = 'http://localhost:8000';

Start the Parsoid server and set it to run on a reboot:

/etc/init.d/pslparsoid start
update-rc.d pslparsoid defaults 

And we now have a new WYSIWYG editor for the Main namespace (most pages):

• ​https://wiki.transitionnetwork.org/index.php?title=Main_Page&veaction=edit

TODO:

• Test it
• Document it

[chris]

I have done some testing and the new editor seems to be working fine, I have also updated the wiki:MediaWiki page, the upgrade notes can be updated next time it is upgraded -- this will be a more complex task in the future.

Closing this ticket now.

[chris]

It's worth noting that adding nodejs has increased the memory usage on the server, see:

​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/memory.html

I have updated the /etc/munin/plugin-conf.d/munin-node files:

[multips_memory]
env.names php5-fpm munin-node nginx mysqld tracd python PassengerHelper PassengerLoggin PassengerWatchd nodejs ruby1.8

To keep track of this here:

• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/multips_memory.html

There are 11 nodejs processes:

ps -lA | grep nodejs | wc -l
11

This is probably way more than we need, it's using over 500MB of RAM, but I can't find where to set this, we might need to allocate more RAM to the server.

[chris]

Looking at /web/wiki.transitionnetwork.org/parsoid/api/server.js it's clear why 11 server are started:

 // Start a few more workers than there are cpus visible to the OS,
 // so that we get some degree of parallelism even on single-core
 // systems. A single long-running request would otherwise hold up
 // all concurrent short requests.

We have these options:

cd /web/wiki.transitionnetwork.org/parsoid/api
./server.js -h    
  Usage: node ./server.js [-h|-v] [--param[=val]]
  
  Options:
    -n, --num-workers  [default: 11]
    -c, --config       [default: "/web/wiki.transitionnetwork.org/parsoid/api/localsettings.js"]
    -v, --version      [default: false]
    -h, --help         [default: false]

So /etc/init.d/pslparsoid has been edited to:

#! /bin/sh
# /etc/init.d/pslparsoid
### BEGIN INIT INFO
# Provides:             pslparsoid
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Start Parsoid daemon
# Description:          Start up Parsoid, Debian run-level script by PSL GbR (draft)
### END INIT INFO

# Some things that run always
#env VCAP_APP_PORT="8000"
#env NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
export VCAP_APP_PORT="8000"
export NODE_PATH="/web/wiki.transitionnetwork.org/parsoid/node_modules"
NUM_WORKERS="4"

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting Parsoid with ${NUM_WORKERS} worker processes..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js -n ${NUM_WORKERS} >> nohup.out 2>&1 &
    ;;
  restart)
    echo "Stopping Parsoid..."
    echo "killall -9 node, could do more here..."
    killall -9 node
    ;;
  'reload'|'force-reload')
    echo "Could fail..."
    cd "/web/wiki.transitionnetwork.org/parsoid"
    /usr/bin/node /web/wiki.transitionnetwork.org/parsoid/api/server.js -n ${NUM_WORKERS} >> nohup.out 2>&1 &
    ;;
  stop)
    echo "Stopping Parsoid..."
    killall -9 node
    ;;
  *)
    echo "Usage: /etc/init.d/pslparsoid {start|stop|restart|reload|force-reload}"
    exit 1
    ;;
esac

exit 0

And it was stopped and restarted and the number of processes checked:

/etc/init.d/pslparsoid stop
/etc/init.d/pslparsoid start
ps -lA | grep nodejs | wc -l
4

We should now see less memory usage overall:

• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/memory.html

And specifically less used by nodejs:

• ​https://penguin.transitionnetwork.org/munin/transitionnetwork.org/penguin.transitionnetwork.org/multips_memory.html

I have done some testing of the new editor on a new ​https://wiki.transitionnetwork.org/Sandbox page and it seems good to me.

There are now two edit options, "Edit" and "Edit source" -- the "Edit" option is the new editor and the "Edit source" is the old one.

I have added a wiki:MediaWiki#Parsoid section to the documentation.

[chris]

I have just dropped the number of node.js processes from 4 to 2 -- we don't use the Mediawiki site enough to justify more at the moment, this will half the RAM usage.

[ed]

nice

[chris]

Replying to ed:

nice

Indeed, it's by far the best HTML WYSIWYG editor I've ever used, the only issue I found was no table support, but I expect that this will be on the todo list.

Now we need to work out how the Transition Network can make more use of their wiki!

[ed]

atm everyone is all over google docs. make it as good as that and we've a chance.

[chris]

Replying to ed:

atm everyone is all over google docs. make it as good as that and we've a chance.

Interesting.

Are people doing real-time multi-user editing?

If they are then we should look at setting up pad.transitionnetwork.org running Etherpad Lite:

• ​http://etherpad.org/

It needs node.js and we have just installed this...

• ​https://github.com/ether/etherpad-lite#installation

Are people also using Google for spreadsheets? If they are then Ethercalc at calc.transitionnetwork.org is the answer:

• ​http://ethercalc.net/

It also uses node.js:

• ​http://ethercalc.net/#install

Shall I open a new ticket for this...?

[ed]

No new tickets for now. TN have an intranet group working on doing some requirements as we type. Needs range from Open Atrium stuff to google docs and Huddle shared docs and groups stuff. Etherpad has the shared editing which is good - imo the killer app for google docs is the commenting - e.g. ​https://docs.google.com/document/d/1pINZIzr4NhqBS2KYph_llyuQ4ofSVNDYAE21tKAI-yQ/edit