Skip to content

Rosetta (OpenTrack language translation)

We use Django's translation framework, and an app named django-rosetta so users can manage translations (see Django docs and Rosetta docs). This explains the various parts.

This uses a standard Python library and associated Unix C library gettext, installed with apt, brew or whatever. It works quickly with standardised message files for each language, which end in .po (source) or .mo (compiled for runtime use).

Tagging phrases for translations in templates

Our code often has this construct. There is a magic function imported as _ (the shortest legal function name, easy to overlook) which means translate at runtime. You'll see import statements for gettext and gettext_lazy aliasing it to this. The block below will say that the words 'Virtual' and 'Track' should be translated at runtime: 

from django.utils.translation import get_language, gettext_lazy as _

COMP_TYPE_CHOICES = (
    ('VIRTUAL', _('Virtual')),
    ('TRACK', _('Track')),
    # ....etc....
)

gettext_lazy kicks in when the string is evaluated, because if we hard code a list we want it translated when the word Track is about to be displayed, and the users's language is known, not when the code is imported.

In HTML templates django has a { % trans 'Competition' % } tag which would translate the phrase into the desired language.

Updating PO files with new or changed phrases

Be careful to separate two processes: tagging things for translation, and publishing end-user translations. Doing both in the same commit is messy and can temporarily lose phrases!

After any change tagging phrases developers should run

python manage.py collectmessages

This inspects the code and finds all tagged messages, and will generate (or, if they exist, extend with new phrases) the message files for each language. This is our own wrapper around makemessages, BUT it also knows to look in the .prep files to make PDF reports, and only to run in our source and not over all imported libraries. These files are saved (and in version control) as

project/locale/[LANG]/LC_MESSAGES/django.po
project/locale/[LANG]/LC_MESSAGES/djangojs.po

so we have a directory structure with two files per language

The inside of a .po file contains blocks for each phrase, including which source files it should be translated in. For example the translation of "Note" has this, which means that "if you see the tag for "Note" , translate to "Σημ."

#: comp/rml/oris_results.prep:286 comp/rml/oris_results.prep:508
#: comp/rml/oris_results.prep:633 comp/templates/comp/event_detail.html:422
#: comp/templates/comp/event_detail_public.html:365
msgid "Note"
msgstr "Σημ."

After running this command, you will typically see all the files under project/locale have been modified, and a diff will reveal your new phrase with a so-far empty msgstr. These should be committed, and deployed (including to the deploy-trans branch). Once on the translation server, the end user could translate the new phrase.

What translators should do

There is a translation web interface here on a dedicated translation server.

As users work through translating, the PO files on the translation server get edited. Unfortunately for us, this server is an ephemeral AWS Fargate instance which gets blown up on every release, so we needed a way to get the edits off the box. We wrote a hook which pushes them off into Amazon S3, which we can pull down.

To use it, users need to belong to two groups: translators and e.g. for French translators-fr Super users see all languages; normal users, the languages they are given.

  1. django-rosetta has a post-save signal we hook up to. See project/main/apps.py. This copies off the two language files to an area in our private S3 bucket each time someone hits 'save' in Rosetta.

  2. Developers should at least run the command ./fetch_translations each day, if not changing any tags. This will sync down any new or modified language files onto you working copy. If it prints stuff, you will see who is translating on the console. You should then commit and push these.

  3. Our AWS deployment mechanism (bin/post_compile) runs the important python manage.py compilenessages (NOT collectmessages) during deployment, to make the optimised .mo files available for use at runtime.

  4. After deploying the phrases to the translation server had finished, you should go into our S3 file area and delete everything, since the phrases are now versioned.

Here is what you might see after running fetch_tranlations if somebody had translated some French:

(raceresults) andy@Lhotse raceresults % ./fetch_translations                 
download: s3://privatebucket.opentrack.run/rosetta_changes/ot-main-test/project/locale/sv/LC_MESSAGES/django.po to project/locale/sv/LC_MESSAGES/django.po
download: s3://privatebucket.opentrack.run/rosetta_changes/ot-main-test/project/locale/fr/LC_MESSAGES/django.po to project/locale/fr/LC_MESSAGES/django.po
Downloaded modified translations
(raceresults) andy@Lhotse raceresults % git status
M project/locale/fr/LC_MESSAGES/django.po
M project/locale/sv/LC_MESSAGES/django.po

This process sucks, with a half-hour period in which somebody's changes could get lost. Ideally we'd find a way to store phrases in the database while keeping the Rosetta interface or something like it, and not need a separate translation server at all.

Adding a language

Languages can be added by adding the lanuage code (list of codes) to the settings.py LANGS_LOCALES variable. Flag should appear in the header to select new language.

LANGS_LOCALES = (
    ('et', _('Estonian'), 'et_EE'),
    ('en-gb', _('English'), 'en_GB'),
    ('sr-Latn', _('Serbian'), 'sr_RS'),
    ('bs', _('Bosnian'), 'bs_BA'),
    ('fr', _('French'), 'fr_FR'),
    # ('en-mt', _('Maltese English'))
    ('pt', _('Portuguese'), 'pt_PT'),
    ('el', _('Greek'), 'el_GR'),
    ('es', _('Spanish'), 'es_ES'),
    ('be', _('Belarusian'), 'be_BY'),
    ('ru', _('Russian'), 'ru_RU'),
    ('it', _('Italian'), 'it_IT'),
    ('nb', _('Norwegian'), 'nb_NO'),
    ('sv', _('Swedish'), 'sv-SE')
)

Also add here:

COUNTRY_LANGUAGE_MAP = (
    ('EST', 'et'),
    ('PRT', 'pt'),
    ('SRB', 'sr'),
    ('FRA', 'fr'),
    ('GBR', 'en-gb'),
    ('BLR', 'be'),
    ('CYP', 'el'),
    ('GRC', 'el'),
    ('RUS', 'ru'),
    ('BIH', 'bs'),
    ('NOR', 'nb'),
    ('ESP', 'es'),
    ('ITA', 'it'),
    ('SWE', 'sv'),
)

If the language is non-Latin (i.e. does not use our alphabet) you will want to extend these, which affect some parts of our UI to show extra fields for athlete names in local script:

NON_LATIN_LANGUAGES = ['el', 'be', 'ru', 'mk', 'bg', 'ar'] NON_LATIN_COUNTRIES = ['CYP', 'BLR', 'MKD', 'ARE', 'SAU']

Next, create the subdirectory for the new language under project/locale by copying en_GB folder and renaming to the correct language code.