Migrate from SVN to Git

[jim]

SVN smells. Git rules.

OK, so we need to get a publicly available Git repo in place and move to a modern workflow after importing the existing SVN stuff.

Urgent tasks:

• Chris: Tell me where you want to put the master repo? In kiwi:/web or somewhere else?
• Chris: add Trac Git plugin: ​http://trac-hacks.org/wiki/GitPlugin
• Jim: import from svn repo to new git master using attached users mapping and this process: ​http://www.jonmaddox.com/2008/03/05/cleanly-migrate-your-subversion-repository-to-a-git-repository/ (except it's git svn not git-svn per the article)
• Jim & Chris: look into best way to clone the repo over the existing structure without affecting the LIVE site

Then later:

• Jim: look at best workflow practices for importing direct from Drupal.org for core and contrib, plus managing the code in a multi-developer environment.
• Jim: update documentation with new Git workflow
• Chris: once proven to work, remove SVN cruft from sites directories.

The rough plan:

• A new Transition git master repository lives on Kiwi as the SVN repo does today.
• Developers clone it onto their machine, do changes, push/commit them back to master
• DEV and TEST sites use a plain git pull from the master
• LIVE uses tagged releases only.

Comments, issues, ideas?

[jim]

SVN -> Git user mapping

[jim]

Chris: can you check the users.txt file and ensure your email and the list of users is ok please?

[chris]

Chris: Tell me where you want to put the master repo? In kiwi:/web or somewhere else?

How about /web/tech.transitionnetwork.org/git.

Chris: can you check the users.txt file and ensure your email and the list of users is ok please?

Looks fine, the trac / svn users we have though are in /web/tech.transitionnetwork.org/.htpasswd

chris
jim
john
ed
laura
daniel
ben

Though of course this was for using tickets rather then svn, I'll read up about the Git plugin...

[chris]

Hmmm

"As this is for now just a proof of concept implementation"

​http://trac-hacks.org/wiki/GitPlugin

Think I need to first I look at the alternative ways us plugging in git to work out the best.

[jim]

OK I'll set up the repo at kiwi:/web/tech.transitionnetwork.org/git and fill it from SVN in a minute...

Trac: To be honest, Trac integration is a 'nice to have' rather than a requirement so no worries there for now.

[chris]

See that this is the main one ​http://trac-hacks.org/wiki/GitPlugin so the install:

easy_install http://github.com/hvr/trac-git-plugin/tarball/trac-0.11
Downloading http://github.com/hvr/trac-git-plugin/tarball/trac-0.11
Processing trac-0.11
Running hvr-trac-git-plugin-d5a8c77/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3PTCqD/hvr-trac-git-plugin-d5a8c77/egg-dist-tmp-tLiFh2
warning: install_data: setup script did not provide a directory for 'COPYING' -- installing right in 'build/bdist.linux-x86_64/egg'
warning: install_data: setup script did not provide a directory for 'README' -- installing right in 'build/bdist.linux-x86_64/egg'
zip_safe flag not set; analyzing archive contents...
Adding TracGit 0.11.0.2 to easy-install.pth file

Installed /usr/local/lib/python2.6/site-packages/TracGit-0.11.0.2-py2.6.egg
Processing dependencies for TracGit==0.11.0.2
Searching for Trac>=0.11dev,==0.11b1,==0.11b2,==0.11rc1,==0.11rc2,==0.11
Reading http://pypi.python.org/simple/Trac/
Reading http://trac.edgewall.org/
Reading http://trac.edgewall.org/wiki/TracDownload
Reading http://projects.edgewall.com/trac
Reading http://projects.edgewall.com/trac/wiki/TracDownload
Reading http://trac.edgewall.com/
Best match: Trac 0.12.2
Downloading ftp://ftp.edgewall.com/pub/trac/Trac-0.12.2.zip
Processing Trac-0.12.2.zip
Running Trac-0.12.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-OuJMfw/Trac-0.12.2/egg-dist-tmp-R5UUX9
Genshi is needed by Trac setup, pre-installing
Searching for Genshi>=0.6
Reading http://pypi.python.org/simple/Genshi/
Reading http://genshi.edgewall.org/
Reading http://genshi.edgewall.org/wiki/Download
Best match: Genshi 0.6
Downloading http://ftp.edgewall.com/pub/genshi/Genshi-0.6-py2.6.egg
Processing Genshi-0.6-py2.6.egg
Moving Genshi-0.6-py2.6.egg to /tmp/easy_install-OuJMfw/Trac-0.12.2

Installed /tmp/easy_install-OuJMfw/Trac-0.12.2/Genshi-0.6-py2.6.egg
Adding Trac 0.12.2 to easy-install.pth file
Installing trac-admin script to /usr/local/bin
Installing tracd script to /usr/local/bin

Installed /usr/local/lib/python2.6/site-packages/Trac-0.12.2-py2.6.egg
Searching for Genshi>=0.6
Reading http://pypi.python.org/simple/Genshi/
Reading http://genshi.edgewall.org/
Reading http://genshi.edgewall.org/wiki/Download
Best match: Genshi 0.6
Downloading http://ftp.edgewall.com/pub/genshi/Genshi-0.6-py2.6.egg
Processing Genshi-0.6-py2.6.egg
Moving Genshi-0.6-py2.6.egg to /usr/local/lib/python2.6/site-packages
Adding Genshi 0.6 to easy-install.pth file

Installed /usr/local/lib/python2.6/site-packages/Genshi-0.6-py2.6.egg
Finished processing dependencies for TracGit==0.11.0.2

cd /web/tech.transitionnetwork.org/trac/plugins

However this has installed the .egg files in /usr/local/lib/python2.6/site-packages/ and not /web/tech.transitionnetwork.org/trac/plugins.

Changes to /web/tech.transitionnetwork.org/trac/conf/trac.ini:

[trac]
#repository_dir = /web/tech.transitionnetwork.org/svn
#repository_type = svn
repository_dir = /web/tech.transitionnetwork.org/git
repository_type = git

[git]
## let Trac cache meta-data via CachedRepository wrapper; default: false
cached_repository = true

## disable automatic garbage collection for in-memory commit-tree cache; default: false
persistent_cache = true

## length revision sha-sums should be tried to be abbreviated to (must be >= 4 and <= 40); default: 7
shortrev_len = 6

## (0.12.0.3+) minimum length for which hex-strings will be interpreted as commit ids in wiki context; default: 40
wiki_shortrev_len = 7

## executable file name (in case of doubt use absolute path!) of git binary; default: 'git'
git_bin = /usr/bin/git

## (0.12.0.5+) define charset encoding of paths stored within git repository; default: 'utf-8'
#git_fs_encoding = latin1

## (0.12.0.3+) enable reverse mapping of git email addresses to trac user ids; default: false
#trac_user_rlookup = true

## (0.12.0.3+) use git-committer id instead of git-author id as changeset owner; default: true
#use_committer_id = false

## (0.12.0.3+) use git-committer timestamp instead of git-author timestamp as changeset time; default: true
#use_committer_time = false

The errors now in syslog include:

Oct 21 14:09:55 localhost <feff><11>Trac[web_ui] ERROR: Timeline event provider failed: #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/dist-packages/trac/timeline/web_ui.py", line 164, in process_request#012    filters):#012  File "/usr/lib/python2.6/dist-packages/trac/versioncontrol/web_ui/changeset.py", line 818, in get_timeline_events#012    repos = self.env.get_repository(req.authname)#012  File "/usr/lib/python2.6/dist-packages/trac/env.py", line 305, in get_repository#012    return RepositoryManager(self).get_repository(authname)#012  File "/usr/lib/python2.6/dist-packages/trac/versioncontrol/api.py", line 149, in get_repository#012    name=self.repository_type))#012TracError: Unsupported version control system "git": Can't find an appropriate component, maybe the corresponding plugin was not enabled?

Oct 21 14:11:52 localhost <feff><11>Trac[web_ui] ERROR: Timeline event provider failed: #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/dist-packages/trac/timeline/web_ui.py", line 164, in process_request#012    filters):#012  File "/usr/lib/python2.6/dist-packages/trac/versioncontrol/web_ui/changeset.py", line 818, in get_timeline_events#012    repos = self.env.get_repository(req.authname)#012  File "/usr/lib/python2.6/dist-packages/trac/env.py", line 305, in get_repository#012    return RepositoryManager(self).get_repository(authname)#012  File "/usr/lib/python2.6/dist-packages/trac/versioncontrol/api.py", line 149, in get_repository#012    name=self.repository_type))#012TracError: Unsupported version control system "git": Can't find an appropriate component, maybe the corresponding plugin was not enabled?

Oct 21 14:13:03 localhost <feff><11>Trac[loader] ERROR: Skipping "Trac 0.12.2": (version conflict "VersionConflict: (Trac 0.11.7 (/usr/lib/python2.6/dist-packages), Requirement.parse('Trac==0.12.2'))")

I think this is because Trac was initially installed on Lenny with a different version of Python and I think I now need to look at a Trac upgrade to get everything working...

[jim]

OK, on Kiwi I sudo apt-get install git-svn so we have the tool to move from svn to git. I also added Ed to the users.txt I attached... Everyone else is either not involved or on the list.

Then I added /web/tech.transitionnetwork.org/git}} to hold the repos - My plan is to have a git repo for each project, so I added {{{www_tmp (which will be converted to 'www' when I'm done...) Later on I'll do import of be one going in for the sharing engine too. When the PSE stuff starts, it can go in its own repo, and this means if we get a bunch of developers on board we could maybe move it to GitHub? for better mass access... Maybe!

Set up repo in www_tmp with:

sudo git svn init https://tech.transitionnetwork.org/svn/www/ --no-metadata

Told git where to look for users and ran the import:

sudo git config svn.authorsfile ../users.txt
sudo git svn fetch

It's coming in now... more soon.

[jim]

It's pulling in all the releases too, which are all but obsolete... Once it completes I'll strip back the old ones and leave just the current. From here we can move to standard branch and tagging.

[chris]

So, doing a manual install to make sure we have the right version of the git plugin:

cd /web/tech.transitionnetwork.org
git clone git://github.com/hvr/trac-git-plugin.git
cd trac-git-plugin/
git checkout trac-0.11
  Branch trac-0.11 set up to track remote branch trac-0.11 from origin.
  Switched to a new branch 'trac-0.11'
python setup.py bdist_egg
cp dist/TracGit-0.11.0.2-py2.6.egg ../trac/plugins/

But I'm not sure git repo is ready?

Trac errors if this line is uncommented in the config file:

#repository_dir = /web/tech.transitionnetwork.org/git

[chris]

OK, I'll come back to this when the repo is ready.

Half term just started....

[jim]

OK my SSH connection failed so it took a little longer...

Cloned the repo to clean it, looks ok to me:

jim@kiwi:/web/tech.transitionnetwork.org/git/www_tmp$ cd ..
jim@kiwi:/web/tech.transitionnetwork.org/git$ sudo git clone www_tmp/ www
Cloning into www...
done.
jim@kiwi:/web/tech.transitionnetwork.org/git$ cd www
jim@kiwi:/web/tech.transitionnetwork.org/git/www$ ls
CHANGELOG.txt  favicon.ico  INSTALL.mysql.txt  INSTALL.txt      misc           profiles    sites       UPGRADE.txt
COPYRIGHT.txt  includes     INSTALL.pgsql.txt  LICENSE.txt      modules        robots.txt  themes      xmlrpc.php
cron.php       index.php    install.php        MAINTAINERS.txt  PRESSFLOW.txt  scripts     update.php

jim@kiwi:/web/tech.transitionnetwork.org/git/www$ git status
# On branch master
nothing to commit (working directory clean)

Chris: Does the Trac plugin only allow one repo? If so I'll have to pull all the other code into this one and rearrange it...

[chris]

Chris: Does the Trac plugin only allow one repo? If so I'll have to pull all the other code into this one and rearrange it...

Well, we are running with the Debian Trac which is version 0.11 and:

Starting with Trac 0.12, Git Plugin gained support for multi-repository configuration

I'd rather stick with one repo for now if we can, but if it's essential I can also look at upgrading Trac...

[jim]

Sharing engine trunk also imported.

So now in /www/tech.transitionnetwork.org/git we have sharing-engine and www git clones of the SVN trunk folders... What we need now is:

1. to merge these into one mega repo... Or update Trac. Thing is there will be more repos to add in the PSE...
2. to checkout the files and test the ignores etc are working ok. (I can do this on my machine)
3. to do the same but over the top of a functioning website (again, I'll do this on my machine first)
4. update any files Laura sends, test process and document
5. tag/branch for release and do it on LIVE.

Chris, what do you think about 1... I take it you don't want to update Trac (again)?

And any thoughts as to how to do 3 and 5 over the top of LIVE's files without messing anything. Obviously I'll test on my box first...

[jim]

​http://markhansen.co.nz/git-repo-inside-git/ seems to be a good approach to putting a repo in a repo as a 'submodule'.

I'll look into this and the best approach for replacing LIVE's copy without mucking anything up on Monday.

Do you have any ideas or comments Chris?

[chris]

Oops, I forgot to submit this comment on Friday:

Trac 0.12 is in debian testing, but no backport for squeeze, so I could try this approach:

​http://cksum.org/blog/182-how-to-install-trac-0-12-debian-squeeze

for 3. and 5, you could check it out somewhere and then rsync the git version over the svn one and then delete all the .svn directories (I can do the rsync and delete the svn dirs if you want).

I'm going to be really pushed this week due to half term...

[chris]

This trac site is going to be taken off line (by stopping apache) to upgrade it following:

• ​http://trac.edgewall.org/wiki/TracUpgrade
• ​http://cksum.org/blog/182-how-to-install-trac-0-12-debian-squeeze

Shout if this is not OK!

[chris]

That was a bit hairy due to a local python in /usr/local/ ...

The configured repos are here:

​https://tech.transitionnetwork.org/trac/admin/versioncontrol/repository

And the code can be browsed here (see bottom of page):

​https://tech.transitionnetwork.org/trac/browser

After adding each repo there was a message like this:

You should also set up a post-commit hook on the repository to call trac-admin $ENV changeset added "old svn" $REV for each committed changeset.

Where $ENV is /web/tech.transitionnetwork.org/trac -- I haven't done this.

I haven't had a chance to update the documentation yet, but I have added a link to this ticket from wiki:DevelopmentServer#Trac

[chris]

If I'm needed I could do the suggested things to switch from SVN to GIT (see ticket:364#comment:14 ) right now or late tonight, but won't have a chance during the day today.

[jim]

Please do go ahead whenever you have a chance, Chris.

I'll try it all out on my machine later this afternoon and update the docs... Hopefully the process on LIVE will go well and we will meet in the middle.

I estimate my times per ​https://tech.transitionnetwork.org/trac/ticket/364#comment:12 to be about 2 hours on my side.

[jim]

OK I've done some work on this:

• checked out the code on my machine
• partially updated documentation: CodeManagementReleaseProcess

But I've hit some snags pushing my work back:

• Kiwi has all the /web/tech.transitionnetwork.org/git folders owned by Root, which is no good... Chris: is there a 'users' group we can apply so the files can be checked out remotely by mere mortals like me? In the end I did a sudo chown -R jim:www-data * to make the files accessible across SSH, but don't know if that'll work for Laura and others... What's the best move here?
• My knowledge of Git isn't quite up to the task as it stands -- the repo on DEV is now in two parts (www and www_temp) because I'm working on this... Once I have my head around Git, branches and best practice it'll be fine. (it's VERY different from SVN, more than I expected and much more complex than my use of it so far)

So, will continue on this tomorrow so the repo and docs are done, then we can test pulling it onto LIVE.

[chris]

Kiwi has all the /web/tech.transitionnetwork.org/git folders owned by Root, which is no good... Chris: is there a 'users' group we can apply so the files can be checked out remotely by mere mortals like me? In the end I did a sudo chown -R jim:www-data * to make the files accessible across SSH, but don't know if that'll work for Laura and others... What's the best move here?

We could creat a git user and sudo to that user when we want to do git things, would that do?

[jim]

I don't think it would, because we'd get no user history data if we all used git@…... Is that what you're proposing?

[chris]

No, if we still use our current accounts to login but when we want to do any git things we do:

sudo -i
su - git

And then do the git stuff as the git user...

[jim]

Still not sure mate... For example:

git clone USER@dev.transitionnetwork.org:/web/tech.transitionnetwork.org/git/www/

grabs the repo and clones it. The command is run on my PC, and when I want to push changes back, that's where the command runs... Currently 'USER' is 'jim', are you suggesting it will be 'git'? Because if so, that won't work for the reasons I said.

Ultimately some users will indeed be doing stuff directly on Kiwi, but 90% of the time all pushing, updates and branch/tagging will be done locally.

Perhaps we need a quick Skype to iron this and related issues out?

[chris]

Sorry perhaps there is some misunderstanding here, I was suggesting how we could update the files on the server using git and to enable this we would need a user with read right -- ie ssh access to kiwi, and we could use the live user that already exists for this, so things would work like this:

• People update code using their users accounts on their local machines.
• When the live or dev server is to be updated this is done by ssh'ing to the server, sudo'ing to the correct shared user account and then running the git commands.

Alternatively if it's only one person updating the code on the live server you could ssh to it with -A and then pull the updated from the dev server as you.

[jim]

I'm not sure the workflow you suggest is the best... Seems longwinded to have a repo locally, then another in the user's home folder, then the 'master', then whatever's on LIVE...

Can we Skype please? Sorry to be a pest but I have way more questions than answers at this stage.

[chris]

Sure, I'm on Skype now, perhaps I'm misunderstanding how git works and proposing a solution that would be good for subversion via ssh...?

[chris]

Jim and I had a skype chat and I added a gitusers group on the dev server and also deleted all the .svn directories from the live site.

[jim]

And I've done a chown -R jim:gitusers * followed by a sudo chmod -R 664 * in the git folder to allow any member of gitusers (Me, Laura and Chris) to write to all git repo files.

Am now researching the best workflow, then will bring the master repo on at /web/tech.transitionnetwork.org/git/www up to date with my version. Then I'll update the documentation. THEN we'll be ready to apply the work to LIVE...

[ed]

I did a +E ed:gitsoulsoup1 fugwinkle:doobie; followed by a bigGuff cmodc/bob in the mgtCabinet

[jim]

Right, sorry to waste a bit of mine and Chris' time, but I'm just going to put the repo on github.com -- It's an executive decision that avoids plenty of SSH pain.

Doing this will provide us with better infrastructure and avoid the need for more server-level faff.. It'll also mean other developers can comment/add/grab our code in the best of open-source manners. It also means we can rely on great documentation and help on that site.

So the process is now:

• I've set up an account (using mine own one for now, can create a TN GitHub? account if needs be)
• I'll log into LIVE, add a .gitignore file and then initiatilse the repo
• I'll push what's on LIVE to the MASTER version 1 on GitHub?
• I'll check it out on my machine and apply the updated bits/security updates
• I'll push that back and pull it onto DEV.
• Test DEV and document best practice
• Push the updates onto LIVE and finish documentation
• Done...

This should all be done in the next 24 hours. MANY apologies for the delay.

[jim]

LIVE now cloned into new repo, minu sensitive files and uploads thanks to .gitignore file.

Repo lives here (for now) ​https://github.com/i-jk/Transition-Network

So the first 3 bullets from the above comment are done, will now clone onto my machine, set up RELEASE and DEVELOPMENT branches for us all to use, then apply my recent changes and push back to DEVELOPMENT which will then go onto DEV.

The upshot is that root@quince (because of our need to sudo) and jim@my-computer have SSH keys registered for the Transition GitHub? project.

[jim]

Update:

• 'master' branch still holds what was on LIVE. This branch will ultimately be removed.
• 'live' branch is a clone of 'master' and represents LIVE code.
• 'development' branch is 'live' with all module updates and work in #238 is now in Git.
• all pushed to GitHub?

Next:

• Push all this to DEV via the 'development' branch.
• Merge 'development' into 'live'
• delete 'master'
• update documentation, do a git pull of 'live' on LIVE server, run updates, done!
• Documentation!
• done.

BTW, git is FAAAST and good.

[jim]

• master branch deleted.
• development branch now contains all goodness (module/security updates & map clustering)
• DEV server now has development branch on it, updates enabled.

Now updating documentation before merging development with live and pulling onto LIVE server.

[jim]

• 'live' branch now has had 'development' merged in.
• documentation updated: CodeManagementReleaseProcess

So I'm waiting for midnight to do the server update and finish #329 and #238.

[jim]

All done, update went swimmingly and even patched a broken module (​http://drupal.org/node/1327960) and pushed changes back to Git Hub from LIVE...

Low priority to dos:

• There's some work to do on TRAC to use the GitHub? plugin instead -- ​https://github.com/davglass/github-trac
• Put the Sharing Engine into Github too
• Make a PSE repo
• Get Transition Network to get their own Github account and transfer the repos from my account to theirs.

Moving this into PSE so it gets done at the beginning of that project, downgrading.

[chris]

Jim, scroll to the end of this comment for some things I need help from you with.

Python is a bit of mess on the development server because (a) a source version was installed in order to get the GA Piwik import working and (b) a unstable version was installed in order to install a unstable version of Trac in order to get it working with git...

So trying to run step 2. of the github-trac install process, ​https://github.com/davglass/github-trac:

2. Install SimpleJSON: easy_install simplejson

easy_install simplejson
-bash: /usr/local/bin/easy_install: /usr/local/bin/python: bad interpreter: No such file or directory

As /usr/local/bin is in ones PATH before /usr/bin this attempted to use the /usr/local/bin/easy_install so that was moved:

cd /usr/local/bin
mv easy_install easy_install.bak

And it was run again:

easy_install simplejson
Searching for simplejson
Best match: simplejson 2.3.0
Processing simplejson-2.3.0-py2.7.egg
simplejson 2.3.0 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/simplejson-2.3.0-py2.7.egg
Processing dependencies for simplejson
Finished processing dependencies for simplejson

Step 1. Clone the repository:

cd /home/chris
git clone git://github.com/davglass/github-trac.git
 Cloning into github-trac...
 remote: Counting objects: 120, done.
 remote: Compressing objects: 100% (69/69), done.
 remote: Total 120 (delta 55), reused 110 (delta 48)
 Receiving objects: 100% (120/120), 18.71 KiB, done.
 Resolving deltas: 100% (55/55), done.

Step 3. Install the Plugin:

cd github-trac/
sudo /usr/bin/python2.7 setup.py install
running install
Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
/usr/bin/python2.7 -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
creating GithubPlugin.egg-info
writing requirements to GithubPlugin.egg-info/requires.txt
writing GithubPlugin.egg-info/PKG-INFO
writing top-level names to GithubPlugin.egg-info/top_level.txt
writing dependency_links to GithubPlugin.egg-info/dependency_links.txt
writing entry points to GithubPlugin.egg-info/entry_points.txt
writing manifest file 'GithubPlugin.egg-info/SOURCES.txt'
reading manifest file 'GithubPlugin.egg-info/SOURCES.txt'
writing manifest file 'GithubPlugin.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/github
copying github/github.py -> build/lib.linux-x86_64-2.7/github
copying github/hook.py -> build/lib.linux-x86_64-2.7/github
copying github/__init__.py -> build/lib.linux-x86_64-2.7/github
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/github.py -> build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/hook.py -> build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/__init__.py -> build/bdist.linux-x86_64/egg/github
byte-compiling build/bdist.linux-x86_64/egg/github/github.py to github.pyc
byte-compiling build/bdist.linux-x86_64/egg/github/hook.py to hook.pyc
byte-compiling build/bdist.linux-x86_64/egg/github/__init__.py to __init__.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/GithubPlugin-0.4-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing GithubPlugin-0.4-py2.7.egg
creating /usr/local/lib/python2.7/dist-packages/GithubPlugin-0.4-py2.7.egg
Extracting GithubPlugin-0.4-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding GithubPlugin 0.4 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/GithubPlugin-0.4-py2.7.egg
Processing dependencies for GithubPlugin==0.4
Searching for GitPython>=0.1.6
Reading http://pypi.python.org/simple/GitPython/
Reading http://gitorious.org/projects/git-python/
Reading https://github.com/gitpython-developers
Best match: GitPython 0.3.2.RC1
Downloading http://pypi.python.org/packages/source/G/GitPython/GitPython-0.3.2.RC1.tar.gz#md5=849082fe29adc653a3621465213cab96
Processing GitPython-0.3.2.RC1.tar.gz
Running GitPython-0.3.2.RC1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-r4HgSa/GitPython-0.3.2.RC1/egg-dist-tmp-A3Ej2M
warning: no files found matching 'README'
Adding GitPython 0.3.2.RC1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/GitPython-0.3.2.RC1-py2.7.egg
Searching for gitdb>=0.5.1
Reading http://pypi.python.org/simple/gitdb/
Reading https://github.com/gitpython-developers/gitdb
Best match: gitdb 0.5.4
Downloading http://pypi.python.org/packages/source/g/gitdb/gitdb-0.5.4.tar.gz#md5=25353bb8d3ea527ba443dd88cd4e8a1c
Processing gitdb-0.5.4.tar.gz
Running gitdb-0.5.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-h7oXgb/gitdb-0.5.4/egg-dist-tmp-dPjRdM
gitdb/_fun.c:1:20: error: Python.h: No such file or directory
In file included from gitdb/_fun.c:2:
gitdb/_delta_apply.h:3: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
gitdb/_delta_apply.h:4: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
gitdb/_delta_apply.h:6: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘DeltaChunkListType’
gitdb/_fun.c:4: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
gitdb/_fun.c:84: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘py_fun’
gitdb/_fun.c: In function ‘init_perf’:
gitdb/_fun.c:96: error: ‘PyObject’ undeclared (first use in this function)
gitdb/_fun.c:96: error: (Each undeclared identifier is reported only once
gitdb/_fun.c:96: error: for each function it appears in.)
gitdb/_fun.c:96: error: ‘m’ undeclared (first use in this function)
gitdb/_fun.c:98: warning: implicit declaration of function ‘PyType_Ready’
gitdb/_fun.c:98: error: ‘DeltaChunkListType’ undeclared (first use in this function)
gitdb/_fun.c:101: warning: implicit declaration of function ‘Py_InitModule3’
gitdb/_fun.c:101: error: ‘py_fun’ undeclared (first use in this function)
gitdb/_fun.c:101: error: ‘NULL’ undeclared (first use in this function)
gitdb/_fun.c:105: warning: implicit declaration of function ‘Py_INCREF’
gitdb/_fun.c:106: warning: implicit declaration of function ‘PyModule_AddObject’
gitdb/_fun.c:106: error: expected expression before ‘)’ token
Ignored failure when building extensions, pure python modules will be used instead
Adding gitdb 0.5.4 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/gitdb-0.5.4-py2.7-linux-x86_64.egg
Searching for smmap>=0.8.0
Reading http://pypi.python.org/simple/smmap/
Reading https://github.com/Byron/smmap
Best match: smmap 0.8.1
Downloading http://pypi.python.org/packages/source/s/smmap/smmap-0.8.1.tar.gz#md5=a35b64bbb7068b7a3f2e0651b72646b2
Processing smmap-0.8.1.tar.gz
Running smmap-0.8.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-n7qxpd/smmap-0.8.1/egg-dist-tmp-GfCoXO
Adding smmap 0.8.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/smmap-0.8.1-py2.7.egg
Searching for async>=0.6.1
Reading http://pypi.python.org/simple/async/
Reading https://github.com/gitpython-developers/async
Best match: async 0.6.1
Downloading http://pypi.python.org/packages/source/a/async/async-0.6.1.tar.gz#md5=6f0e2ced1fe85f8410b9bde11be08587
Processing async-0.6.1.tar.gz
Running async-0.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-o6gdJb/async-0.6.1/egg-dist-tmp-T0AJm3
mod/zlibmodule.c:7:20: error: Python.h: No such file or directory
mod/zlibmodule.c:60: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Comptype’
mod/zlibmodule.c:61: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Decomptype’
mod/zlibmodule.c:63: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:67: error: expected specifier-qualifier-list before ‘PyObject_HEAD’
mod/zlibmodule.c: In function ‘zlib_error’:
mod/zlibmodule.c:83: warning: implicit declaration of function ‘PyErr_Format’
mod/zlibmodule.c:83: error: ‘ZlibError’ undeclared (first use in this function)
mod/zlibmodule.c:83: error: (Each undeclared identifier is reported only once
mod/zlibmodule.c:83: error: for each function it appears in.)
mod/zlibmodule.c: At top level:
mod/zlibmodule.c:89: error: expected ‘)’ before string constant
mod/zlibmodule.c:94: error: expected ‘)’ before string constant
mod/zlibmodule.c:99: error: expected ‘)’ before ‘*’ token
mod/zlibmodule.c:125: error: expected ‘)’ before string constant
mod/zlibmodule.c:129: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:201: error: expected ‘)’ before string constant
mod/zlibmodule.c:206: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:301: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:340: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c: In function ‘Comp_dealloc’:
mod/zlibmodule.c:379: error: ‘compobject’ has no member named ‘is_initialised’
mod/zlibmodule.c:380: error: ‘compobject’ has no member named ‘zst’
mod/zlibmodule.c:381: warning: implicit declaration of function ‘Py_XDECREF’
mod/zlibmodule.c:381: error: ‘compobject’ has no member named ‘unused_data’
mod/zlibmodule.c:382: error: ‘compobject’ has no member named ‘unconsumed_tail’
mod/zlibmodule.c:383: error: ‘compobject’ has no member named ‘status’
mod/zlibmodule.c:389: warning: implicit declaration of function ‘PyObject_Del’
mod/zlibmodule.c: In function ‘Decomp_dealloc’:
mod/zlibmodule.c:395: error: ‘compobject’ has no member named ‘is_initialised’
mod/zlibmodule.c:396: error: ‘compobject’ has no member named ‘zst’
mod/zlibmodule.c:397: error: ‘compobject’ has no member named ‘unused_data’
mod/zlibmodule.c:398: error: ‘compobject’ has no member named ‘unconsumed_tail’
mod/zlibmodule.c:399: error: ‘compobject’ has no member named ‘status’
mod/zlibmodule.c: At top level:
mod/zlibmodule.c:409: error: expected ‘)’ before string constant
mod/zlibmodule.c:416: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:480: error: expected ‘)’ before string constant
mod/zlibmodule.c:490: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:604: error: expected ‘)’ before string constant
mod/zlibmodule.c:611: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:805: error: expected ‘)’ before string constant
mod/zlibmodule.c:811: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:874: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘comp_methods’
mod/zlibmodule.c:887: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Decomp_methods’
mod/zlibmodule.c:900: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:919: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:945: error: expected ‘)’ before string constant
mod/zlibmodule.c:950: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:968: error: expected ‘)’ before string constant
mod/zlibmodule.c:973: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
mod/zlibmodule.c:991: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘zlib_methods’
mod/zlibmodule.c:1008: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Comptype’
mod/zlibmodule.c:1024: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Decomptype’
mod/zlibmodule.c:1041: error: expected ‘)’ before string constant
mod/zlibmodule.c:1056: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘initzlib’
Ignored failure when building extensions, pure python modules will be used instead
Adding async 0.6.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/async-0.6.1-py2.7-linux-x86_64.egg
Searching for simplejson==2.3.0
Best match: simplejson 2.3.0
Processing simplejson-2.3.0-py2.7.egg
simplejson 2.3.0 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/simplejson-2.3.0-py2.7.egg
Finished processing dependencies for GithubPlugin==0.4

So, try again without /usr/local/bin in the path:

sudo -i
cd /home/chris/github-trac/
export PATH=/usr/sbin:/usr/bin:/sbin:/bin
running install
Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
/usr/bin/python2.7 -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing requirements to GithubPlugin.egg-info/requires.txt
writing GithubPlugin.egg-info/PKG-INFO
writing top-level names to GithubPlugin.egg-info/top_level.txt
writing dependency_links to GithubPlugin.egg-info/dependency_links.txt
writing entry points to GithubPlugin.egg-info/entry_points.txt
reading manifest file 'GithubPlugin.egg-info/SOURCES.txt'
writing manifest file 'GithubPlugin.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/github.py -> build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/hook.py -> build/bdist.linux-x86_64/egg/github
copying build/lib.linux-x86_64-2.7/github/__init__.py -> build/bdist.linux-x86_64/egg/github
byte-compiling build/bdist.linux-x86_64/egg/github/github.py to github.pyc
byte-compiling build/bdist.linux-x86_64/egg/github/hook.py to hook.pyc
byte-compiling build/bdist.linux-x86_64/egg/github/__init__.py to __init__.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying GithubPlugin.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/GithubPlugin-0.4-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing GithubPlugin-0.4-py2.7.egg
removing '/usr/local/lib/python2.7/dist-packages/GithubPlugin-0.4-py2.7.egg' (and everything under it)
creating /usr/local/lib/python2.7/dist-packages/GithubPlugin-0.4-py2.7.egg
Extracting GithubPlugin-0.4-py2.7.egg to /usr/local/lib/python2.7/dist-packages
GithubPlugin 0.4 is already the active version in easy-install.pth

Installed /usr/local/lib/python2.7/dist-packages/GithubPlugin-0.4-py2.7.egg
Processing dependencies for GithubPlugin==0.4
Searching for GitPython==0.3.2.RC1
Best match: GitPython 0.3.2.RC1
Processing GitPython-0.3.2.RC1-py2.7.egg
GitPython 0.3.2.RC1 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/GitPython-0.3.2.RC1-py2.7.egg
Searching for simplejson==2.3.0
Best match: simplejson 2.3.0
Processing simplejson-2.3.0-py2.7.egg
simplejson 2.3.0 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/simplejson-2.3.0-py2.7.egg
Searching for gitdb==0.5.4
Best match: gitdb 0.5.4
Processing gitdb-0.5.4-py2.7-linux-x86_64.egg
gitdb 0.5.4 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/gitdb-0.5.4-py2.7-linux-x86_64.egg
Searching for smmap==0.8.1
Best match: smmap 0.8.1
Processing smmap-0.8.1-py2.7.egg
smmap 0.8.1 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/smmap-0.8.1-py2.7.egg
Searching for async==0.6.1
Best match: async 0.6.1
Processing async-0.6.1-py2.7-linux-x86_64.egg
async 0.6.1 is already the active version in easy-install.pth

Using /usr/local/lib/python2.7/dist-packages/async-0.6.1-py2.7-linux-x86_64.egg
Finished processing dependencies for GithubPlugin==0.4

So perhaps the errors before can be ignored...

4. Configure Trac by editing the /web/tech.transitionnetwork.org/trac/conf/trac.ini:

[components]
#tracext.git.* = enabled
github.* = enabled

[trac]
#repository_dir = /web/tech.transitionnetwork.org/git/.git
#repository_type = git

#[git]
### let Trac cache meta-data via CachedRepository wrapper; default: false
#cached_repository = false
### disable automatic garbage collection for in-memory commit-tree cache; default: false
#persistent_cache = false
### length revision sha-sums should be tried to be abbreviated to (must be >= 4 and <= 40); default: 7
#shortrev_len = 7
### (0.12.0.3+) minimum length for which hex-strings will be interpreted as commit ids in wiki context; default: 40
#wiki_shortrev_len = 40
### executable file name (in case of doubt use absolute path!) of git binary; default: 'git'
#git_bin = /usr/bin/git 
## (0.12.0.5+) define charset encoding of paths stored within git repository; default: 'utf-8'
#git_fs_encoding = utf-8 
## (0.12.0.3+) enable reverse mapping of git email addresses to trac user ids; default: false
#trac_user_rlookup = true
## (0.12.0.3+) use git-committer id instead of git-author id as changeset owner; default: true
#use_committer_id = true
## (0.12.0.3+) use git-committer timestamp instead of git-author timestamp as changeset time; default: true
#use_committer_time = true 

[github]
apitoken = YOUR GITHUB API TOKEN
closestatus = closed (This is optional, defaults to closed
browser = https://github.com/i-jk/Transition-Network 

#Optional - perform a fetch on the local git repository
autofetch = true

So, Jim, can you email me the github apitoken and do this last task:

5. Go to the Admin page for your project on GitHub?. Then select the services tab.

Under the: Post-Receive URLs
Place a link to your Trac repository, in a format like this:

​http://yourdomian.com/projects/projectname/github/APITOKEN

You can get your GitHub? API token from your accounts page.
This is for your security, only those that know the API Token will able to post to this url

[jim]

I've done the Github side of things, but the API key is secret so I've sent you a message via Skype with it in.

[jim]

Moving to maintenance, but leaving in PSE since there's overlap...

To do:

• Put the Sharing Engine into Github too, AND the PSE code when it arrives
• Get Transition Network to get their own Github account and transfer the repos from my account to theirs.

[jim]

See #389 for module updates as a prelude to adding the News Sharing Engine to Git.

[chris]

GitHub Hacked:

Over the weekend, developer Egor Homakov exploited a gaping vulnerability in GitHub that allowed him (or anyone else with basic hacker know-how) to gain administrator access to projects such as Ruby on Rails, Linux, and millions of others. GitHub uses the Ruby on Rails application framework, and Rails has been weak to what's known as a mass-assignment vulnerability for years. Basically, Homakov exploited this vulnerability to add his public key to the Rails project on GitHub, which then meant that GitHub identified him as an administrator of the project. From here, he could effectively do anything, including deleting the entire project from the web; instead, he posted a fairly comical commit. GitHub summarily suspended Homakov, fixed the hole, and, after 'reviewing his activity,' he has been reinstated. Homakov could've gained administrative access to the master branch of any project on GitHub and deleted the history, committed junk, or closed or opened tracker tickets.

​http://it.slashdot.org/story/12/03/05/1243235/github-hacked

Should we reconsider hosting our code with a 3rd party at some point?

[jim]

Meh... we have no important code on GH that doesn't exist elsewhere, like on LIVE, DEV and my machine, and it's all public anyway so no secrets can escape. GH could explode tomorrow and we'd be fine.

I don't see any real risk and, given the thousands of high-profile projects on GH, security will be fixed pretty quickly. I've already audited the 3 ssh keys on my account.

So that leaves the tiny chance a hole is found and a hacker chooses us out of all those other big projects to put some Drupal-hacking code into... Like I said, meh!

---

In other news various updates are in Git pending an update later tonight.

The initial PSE code will be part of it and once we've got it on DEV to test I'll look into Git Submodules to manage all this stuff better.

[jim]

The 'Error with navigation contributor "BrowserModule?"' issue rendered Trac's search functionality broken, so I've disabled the TracGit? stuff by moving the .egg files to
[trac_directory]/plugins-disabled.

Search now works again. We didn't need that anyway, so no loss.

[chris]

Thanks for doing ticket:364#comment:43 -- is this something we should still aim to set up sometime or is it simply not needed?

[chris]

github alternative which looks nice: ​http://sitaramc.github.com/gitolite/

[chris]

I have created an account for myself on github and added ssh keys, I'll try to checkout the code and commit a change to fix this bug later ticket:357#comment:49

[chris]

Following the comprehensive notes here, wiki:CodeManagementReleaseProcess#SoftwareGettingStarted I just tried to checkout the dev code:

git clone -b dev git@github.com:i-jk/Transition-Network.git dev
Cloning into dev...
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I have added my ssh-key(s) following the instructions here ​http://help.github.com/linux-set-up-git/ and it all seems OK:

ssh -T github
  Hi chriscroome! You've successfully authenticated, but GitHub does not provide shell access.

Jim -- do you need to add me to the project or is the problem at my end, any idea? My username is chriscroome.

[chris]

Jim -- you can add me to the git repos if you want, ​https://github.com/chriscroome

Is Trac / Github integration needed? If not we can close this ticket, if it is then it needs some work.

[jim]

Chris you're added.

No other migration needed, just documentation for new Aegir/Git/Drush/Makefile? process to replace: ​https://tech.transitionnetwork.org/trac/wiki/CodeManagementReleaseProcess

Will do documentation over here #488, closing.

[chris]

The wiki frontpage have been updated with links to ​https://github.com/transitionnetwork

I have also created a new wiki:BoaCodeManagement page and removed the front page link to wiki:CodeManagementReleaseProcess -- when a page needs a total re-write I think it's best to simply start a new one and archive the old one.

Now we have Piwik stats for this site it's interesting to note that there are quite a few visits to the trac pages generated from the subversion repo which no longer exists (I didn't migrate it as it's no longer used), I don't think it's worth worrying about these and they should drop out of the google index as the site is re-crawled.

[chris]

Replying to jim:

Chris you're added.

Are you sure, I just see you and Laura ​https://github.com/transitionnetwork?tab=members

I'm not expecting to be committing much, if anything via git, but I'm on for testing and helping with the the documentation and generally trying to get my head around it all.

[jim]

Yep, you're there, you just weren't 'Publicized'... You are now...