Mercurial-Git mirror

ReportLab (and OpenTrack staff) have historically used the Mercurial version control system. However, the main Mercurial hosting company closed down, and GitHub is the home for everything these days. Our deployments on Amazon involve AWS "watching" GitHub.

We therefore set up a repository uder ReportLab's GitHub account containing a mirror of the code. This means that when you push to mercurial, pretty quickly the changes end up in GitHub. This has needed rebuilding and resetting twice - once as Git changed their security policies in summer 2021, and once for no known reason on Dec 1st 2021, so I am documenting it.

Diagnosing problems

If deployments appear to succeed but are not showing up, you can see the date of the code change directly by browsing to e.g.

https://data.opentrack.run/static/revision.txt

This will show a repository id for the change that went live, and the time it happened e.g.

PROD hg id=9ce0f66da5a8+380fac03f7d1+ (deploy-production) @ 20211201U222320

This tells us the code in production was committed on 1st December at 22:23. If this is older than you think, alert ReportLab; Robin, Andy or James can deal with it.

Technical details

This section is to help Andy, James or others follow what Robin has set up.

How it runs and where stuff loves

On ReportLab's app1.reportlab.com server, which hosts the Mercurial repos, there is a cron job running every minutes called $HOME/bin/pull-repos. We also have special local Git repos for each mercurial one we want to mirror. These live in rptlab@app1:~/etc/hg/git-push/REPOS

pull-repos contains a load of timestamp comparing code, and then this:

REPOS="reportlab pyrxp preppy pyfribidi rlextra-examples rlextra-full-tutorial rlPyCairo raceresults"
    for repo in ${REPOS}; do
        $HOME/etc/hg/git-push/spiegel $HOME/etc/hg/git-push/REPOS/$repo-mirror
    done

So, for a bunch of repositories, it executes a command spiegel which knows how to update the local Git repo to reflect the Mercurial changes.

It then pushes the changes to GitHub.

Rebuilding the local Git repo.

This sequence of commands can be used to rebuild a specific local mirror. When we did this on Dec 1st, it all started working again.

src=https://hg.reportlab.com/hg/raceresults
mirror=$(basename "$src")-mirror
dst=git@mrbb-github:rptlab/opentrack.git
git ls-remote "$dst" &>/dev/null && echo OK
grepo="/home/rptlab/etc/hg/git-push/REPOS/$mirror"
echo "$grepo"
rm -r "$grepo"
mkdir -p "$grepo"
git clone --mirror "hg::$src" "$grepo"
cd "$grepo"
git remote set-url --push origin "$dst"
git config --global credential.helper store
git fetch -p origin
git push --mirror

Long term

If we were all confident enough with Git to use it correctly, we arguably should just drop Mercurial and adopt the GitHub repo as our main one. On the other hand, if you mess something up badly on GitHub, we don't have the same freedom to fix things on the server side.