Difference between revisions of "Coding for translation"

From Gramps
Jump to: navigation, search
m (Provide separate strings for masculine and feminine.)
(Changing translated text message in the source code)
(70 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Coding guidelines to enabling easy and correct translation of strings on the User Interface.  
+
Coding guidelines to enable easy and correct translation of strings on the User Interface.  
[[Category:Translators/Categories]][[Category:Developers/Tutorials]]
+
 
[[Category:Developers/General]]
 
 
==Introduction==
 
==Introduction==
In order to consider including a report in the offical GRAMPS release, a report generator must support internationalization. What this means is that the report must support [[Translating GRAMPS|translations]] into different languages. GRAMPS provides support to make this as easy as possible for the developer. For enabling, a language code must be set on ''configure.in'' file into ''ALL_LINGUAS'' section.
+
Gramps has always been internationalized (see
 +
http://gramps-project.org/2006/04/looking-back-over-5-years).
 +
Therefore, all strings meant
 +
for the user should always be flagged for translation.
 +
 
 +
In order to be considered for inclusion in the offical Gramps release, any piece of code must support internationalization. What this means is that the Python module must support [[Translating Gramps|translations]] into different languages. Gramps provides support to make this as easy as possible for the developer. For enabling, a language code must be set on the ''[[Template:Gramps_translations#ALL_LINGUAS |ALL_LINGUAS]]'' section.
  
 
==How to allow translations==
 
==How to allow translations==
GRAMPS provides a simple interface (based on the gettext interface) to mark strings as being translatable. First, import the gettext function from the intl library.
+
Gramps is a fully-internationalized application with translations in many languages. All code which presents text to users must provide for that text to be translated. Fortunately, Gramps provides an extension of [http://docs.python.org/3/library/gettext.html gettext] which makes this fairly painless. First, alias the gettext function from the single localization instance:
  from gen.ggettext import gettext as _
+
  from gramps.gen.const import GRAMPS_LOCALE as glocale
 
+
_ = glocale.translation.gettext
This statement imports the <code>sgettext</code> function under the name of <code>_</code>. This is the function that both marks the strings for translation and performs the actual translation at runtime. Strings that should be translated should be enclosed as an argument to the function.
+
This statement imports the <code>gettext</code> function and aliases it as <code>_</code>. The translation tools treat strings wrapped in _() as translatable and assemble them into catalogs for the translators to work with; by aliasing it to gettext(), we also enable python to retrieve the translation appropriate for the user's locale.
  
 
Example 1:
 
Example 1:
Line 19: Line 23:
 
  print _("Hello world!")
 
  print _("Hello world!")
  
In this example, GRAMPS will attempt to translate the string. If a translation exists, the call to the function will return the translation. If a translation does not exist, the original string is returned.
+
In this example, Gramps will attempt to translate the string. If a translation exists, the call to the function will return the translation. If a translation does not exist, the original string is returned.
 +
 
 +
=== More complicated translations ===
 +
 
 +
In addition to <tt>gettext</tt>, GrampsTranslation offers two more specialized retrieval functions, <tt>ngettext</tt> and <tt>sgettext</tt>.
 +
 
 +
In some strings, it's necessary to specify different translations depending upon the number of an argument. For example,
 +
George Smith and Annie Jones have 1 child
 +
George Smith and Annie Jones have 3 children
 +
 
 +
We'd code that in python as follows:
 +
_ = glocale.translation.ngettext
 +
_(George Smith and Annie Jones have %(num)d child, George Smith and Annie Jones have %(num)d children, n) % {num : n}
 +
 
 +
In other cases, it's necessary to provide a hint to translators, e.g.
 +
_(Remaining names | rest)
 +
 
 +
We're making sure that the translators know that this message id means "what's left" rather than "take a nap". When the file is translated, this is no problem, because the translation doesn't include the hint -- but if the user is working in English, we don't want him to see the hint, so we need to alias _ to sgettext:
 +
_ = glocale.translation.sgettext
 +
 
 +
Often you need to combine them. While <tt>ngettext</tt> and <tt>sgettext</tt> can each handle plain strings, neither can handle the other's strings. Fortunately the <tt>intltool</tt> message extractor is pretty stupid, so any function name that ends in either <tt>_</tt> or <tt>gettext</tt> will work. This will work pretty well:
 +
 
 +
  _ = glocale.translation.gettext
 +
  N_ = glocale.translation.ngettext
 +
  S_ = glocale.translation.sgettext
 +
 
 +
Obviously you would pass the translatable string to the right function.
 +
 
 +
=== Encoding ===
 +
String handling can be a bit tricky in a localized environment so it's important that developers understand Unicode string handling in both versions of the language.
 +
 
 +
This is mostly a problem for Microsoft Windows™: Mac OSX and Linux use UTF8 for just about everything if the locale is set up correctly (and we try to do that when Gramps starts up), so one can get away with a lot of encoding mistakes on those platforms. Windows™ on the other hand uses a slightly modified version of UTF16 for file names and retains the old DOS [http://en.wikipedia.org/wiki/Code_page code page] system for encoding output to cmd.exe. The take-away is that if you need to mess with input or output encoding, be sure to test on both Linux and Windows before deciding that you're done. If you're not set up for multiple-platform testing arrange with someone,, who can test for you on the platform you don't have.
 +
 
 +
====Python 2====
 +
Python 2.7 has two text classes, <tt>[https://docs.python.org/2.7/library/functions.html#str str]</tt> and <tt>[https://docs.python.org/2.7/library/functions.html#unicode unicode]</tt>. <tt>Unicode</tt> objects are encoded in UTF16 internally on most platforms, and most python '''output''' functions will do the right thing with them. One caveat here: passing both <tt>unicodes</tt> and <tt>strs</tt> to <tt>os.path.join()</tt> will return a <tt>str</tt>, so either make sure when constructing a path that all arguments are <tt>unicodes</tt> or convert the result.
 +
 
 +
The bsddb module that ships with Python2 is stupid about paths and requires that they be encoded in the file system encoding. This is handled in gramps/gen/db/write.py with _encode() and independently in a few other places.
 +
 
 +
Strings from the operating system, including environment variables, are a problem on Windows™; The os module uses for input the [http://msdn.microsoft.com/en-us/library/windows/desktop/dd317766%28v=vs.85%29.aspx ANSI API] to the Windows SDK, which interprets the value of the environment variable according to the active code page and produces a <tt>str</tt>, converting any codepoints > 0xff to ? and often misinterpreting those between 0x0f and 0xff if the encoding of the input happens to be something other than the active system codepage. Once this is done it is quite difficult to get non-ASCII pathnames back into a useable form, so gramps/gen/constfunc.py provides a get_env_var() function that uses the [http://msdn.microsoft.com/en-us/library/windows/desktop/dd374081(v=vs.85).aspx Unicode API] to instead. Always use that function to read environment variables which might include non-ASCII characters and avoid using os-module functions for reading paths.
 +
 
 +
By default string constants in Python 2 are <tt>str</tt>.
 +
 
 +
====Python 3====
 +
Python 3 also provides two test classes, <tt>[https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str str]</tt> and <tt>[https://docs.python.org/3/library/stdtypes.html#binary-sequence-types-bytes-bytearray-memoryview bytes]</tt>. In Python 3, <tt>str</tt> is the unicode type and <tt>bytes</tt> is text encoded some other way. Everything pretty much "just works".
 +
 
 +
====Portability Functions and constants====
 +
We've provided a couple of functions in gramps/gen/constfunc.py to ease conversion of <tt>strs</tt> to <tt>unicodes</tt>; these include the necessary tests to portably do the right thing regardless of what's passed to them and according to which version of Python is in use:
 +
* <tt>cuni</tt> is an alias for [https://docs.python.org/2.7/library/functions.html#unicode unicode] in Python 2 and for [https://docs.python.org/3/library/functions.html#func-str str] in Python 3. This has no protective checks so use it with care.
 +
* <tt>conv_to_unicode(string, encoding='utf8')</tt>: This ensures that its return value is a Unicode string which has been converted from a non-Unicode in the <tt>encoding</tt>, which defaults to UTF8 for ease of use with the GUI.
 +
* <tt>get_env_var(string, default=None)</tt>: On Windows™ in Python2, uses the <tt>ctypes</tt> module to invoke the Microsoft Unicode API to read the value of an environment variable and return a Unicode; otherwise returns the value from the <tt>os.environ</tt> array.
 +
There are also two constants:
 +
* <tt>STRTYPE</tt> is an alias for [https://docs.python.org/2.7/library/functions.html#basestring basestring] in Python 2 and for [https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str str] in Python 3. It can be used to test whether an object is a text-type.
 +
* <tt>UNITYPE</tt> is an alias for [https://docs.python.org/2.7/library/functions.html#unicode unicode] in Python 2 and for [https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str str] in Python 3. It can be used to test whether an object is already encoded in Unicode.
 +
 
 +
====For portable string handling on all platforms and for all locales====
 +
* Localized strings returned from gettext, ngettext, etc. are always unicode
 +
* Text files should always be encoded in UTF8. The easy and portable way to do this is to:
 +
*: <pre>import io</pre>
 +
*: <pre>fh = io.open(filepath, mode, encoding='utf8')</pre>
 +
*: where ''mode'' is one of r, rw, r+, or w+. ''Don't open these files in binary mode!'' Pass unicode-type strings to fh.write() and expect the same from fh.read().
 +
* Always read environment variables with <tt>constfuncs.get_env_var()</tt> if there's any chance that it will contain a non-ASCII character.
 +
* Use <tt>from __future__ import unicode_literals</tt> in any source file which might present strings to the user or to the operating system.
 +
*:When creating string literals, '''don't do this:'''
 +
*:<pre>print _(u"Eg, valid values are 12.0154, 50° 52′ 21.92″N")</pre>
 +
*:Because the <tt>u</tt> prefix was removed for Python 3.0-3.2. (It was restored in 3.3 for compatibility with 2.7, but it's not necessary.)
 +
*:Instead, put in the first line of the module
 +
*:<pre># *-* coding: utf-8 *-*</pre>
 +
*:then in the imports section
 +
*:<pre>from __future__ import unicode_literals</pre>
 +
*:which makes all of the literals unicode. '''Make sure that your editor is set up to save utf-8!'''
  
===Into glade file===
+
===Glade files===
  
Just enable the translatable attribute on XML element.
+
Just enable the translatable attribute on an XML element.
  
 
  <property name="label" translatable="yes">_Family:</property>
 
  <property name="label" translatable="yes">_Family:</property>
Line 29: Line 102:
 
  <property name="label" translatable="no">&lt;b&gt; - &lt;/b&gt; </property>
 
  <property name="label" translatable="no">&lt;b&gt; - &lt;/b&gt; </property>
  
===Into addons plugins===
+
====Non ASCII characters====
 +
 
 +
If you plan to use non ASCII characters in a string, that shall be translated,
 +
do not use escape sequences:
 +
 
 +
Eg, valid values are 12.0154, 50<code>&amp;</code>#xB0; 52' 21.92"N
 +
 
 +
use instead:
 +
 
 +
Eg, valid values are 12.0154, 50° 52′ 21.92″N
 +
 
 +
In this case note the unicode characters for deg, min, sec. '''Ensure that your editor is set up to encode the characters in UTF-8!'''
 +
 
 +
====Accessibility====
 +
 
 +
In addition to [http://developer.gnome.org/gtk/2.24/GtkWidget.html#id1294298 accelerators], ''[http://developer.gnome.org/gtk/2.24/GtkWidget.html GtkWidget]'' also support a custom <accessible> element, which supports actions and relations. Properties on the accessible implementation of an object can be set by accessing the internal child "accessible" of a ''[http://developer.gnome.org/gtk/2.24/GtkWidget.html GtkWidget]''. See [http://developer.gnome.org/gtk/2.24/GtkWidget.html#GtkWidget-BUILDER-UI GtkBuilder UI].
 +
 
 +
* Gtk label
 +
 
 +
''A [http://developer.gnome.org/gtk/2.24/GtkLabel.html GtkLabel]'' '''with mnemonic support''' will automaticaly generate accessibility keys on linked ''[http://developer.gnome.org/gtk/2.24/GtkEntry.html GtkEntry]'' and ''UndoableEntry'' fields. Remember that Gramps also uses custom widgets like ''StyledTextEditor'' and ''ValidatableMaskedEntry'', which do not always have relation with a ''GtkLabel''.
  
from TransUtils import get_addon_translator
+
* Toggle buttons and Icons on toolbar
_ = get_addon_translator().gettext
 
  
See [[Addons_Development#Localization|Addon developpement]].
+
Gramps often uses ''[http://developer.gnome.org/gtk/2.24/GtkToggleButton.html GtkToggleButtons]'' and alone ''[http://developer.gnome.org/gtk/2.24/GtkImage.html GtkImage]'' (image without label), this excludes blind people and generates a poor interface for accessibility.
  
==How it works==
+
See [[Accessibility]].
  
[http://www.gnu.org/software/gettext/manual/gettext.html GNU gettext] and [http://live.gnome.org/TranslationProject/DevGuidelines/Localize%20using%20gettext%20and%20intltool Gnome] provide utilities and a [http://www.gnome.org/~malcolm/i18n/build-changes.html translation framework] (''previously [http://gramps.svn.sourceforge.net/viewvc/gramps/branches/maintenance/gramps20/gramps2/src/build_po build_po] and [http://gramps.svn.sourceforge.net/viewvc/gramps/branches/maintenance/gramps20/gramps2/src/get_strings get_strings]''):
+
===Addons===
* [http://www.gnu.org/software/autoconf/manual/gettext/msginit-Invocation.html msginit] will generate a standard gettext header.
 
* intltool-update will manage template and translations.
 
* intltool-extract will extract translation strings on ''.glade'' and ''.xml'' files, by generating files with ''.h'' extension.
 
  
  # Generates a new template (gramps.pot), into ''/po'' directory :
+
External addons often need to provide their own message catalogs. To pull one in, use
intltool-update -p
+
this instead of the usual.
 +
  from gramps.gen.const import import GRAMPS_LOCALE as glocale
 +
_ = glocale.get_addon_translator(__file__).gettext
 +
or if you need more than one retrieval function:
 +
  _translation = glocale.get_addon_translator(__file__)
 +
  _ = _translation.gettext
 +
  S_ = _translation.sgettext
  
* intltool-merge will merge cached translations into .in files
+
The addon translator is another instance of GrampsTranslation, so the rules for creating
 +
translatable strings and for retrieving the translated values are the same as for internal modules.
  
# Merges translated strings into desktop file, ''root'' directory :
+
See [[Addons_Development#Localization|Addons development]] for more details.
intltool-merge -d po/ data/gramps.desktop.in data/gramps.desktop
 
  
# Merges translated strings into xml file, ''root'' directory :
+
==How it works==
intltool-merge -x po/ data/gramps.xml.in data/gramps.xml
 
  
# Merges translated strings into key file, ''root'' directory :
+
We need at least [http://www.gnu.org/software/gettext/manual/gettext.html GNU gettext], then [http://www.gnu.org/software/autoconf/manual/gettext/msginit-Invocation.html msginit] will generate a standard gettext header.
intltool-merge -k po/ data/gramps.keys.in data/gramps.keys
 
  
===Files and directory===
+
Gramps has used different environments according to versions for retrieving strings to translate:
  
There are two stages to getting a translation to work. Translations are stored in a <code>.po</code> file that contains the mappings between the original strings and the translated strings, see [[Translating GRAMPS]].
+
* [[Translation_environment20|2.0.x]]
 +
* [[Translation_environment22|2.2.x to Gramps 3.4.x]]
 +
* [[Translation_environment4|Gramps 4.0.x to master (trunk)]]
  
Translators use a generic file <code>gramps.pot</code> to generate their <code>.po</code> file.
+
There are two stages to getting a translation to work.  
GRAMPS uses a utility that extracts the strings from the source code to build the <code>.po</code> file. This utility (a perl script called by the command <code>make</code>) examines the source files for strings that have been marked as translatable. In the python source, these are the strings enclosed in the <code>_()</code> function calls.
 
  
If you want this script to take your translatable strings into account, you must add your source file path in the file : <code>po/POTFILES.in</code>. For this report example, you should add:
+
===Files and directory===
  
...
+
Translations are stored in a <code>.po</code> file that contains the mappings between the original strings and the translated strings, see [[Translating Gramps]].  
# plugins directory
 
src/plugins/AncestorChart2.py
 
src/plugins/AncestorReport.py
 
...
 
src/plugins/FindDupes.py
 
src/plugins/Leak.py
 
src/plugins/MediaManager.py
 
src/plugins/Myreport.py                # <------
 
src/plugins/NarrativeWeb.py
 
src/plugins/PatchNames.py
 
...
 
  
In this file, the sources are sorted within each directory or category.
+
Translators use a generic file <code>gramps.pot</code> to generate their <code>.po</code> file.
 +
Gramps uses a utility that extracts the strings from the source code to build the <code>.po</code> file. This utility examines the source files for strings that have been marked as translatable. In the python source, these are the strings enclosed in the <code>_()</code> function calls.
  
 
Note that because strings are extracted by a script from the source file, string constants and not variables must be enclosed in the <code>_()</code> call. In the following example, the extraction script will not extract the string.
 
Note that because strings are extracted by a script from the source file, string constants and not variables must be enclosed in the <code>_()</code> call. In the following example, the extraction script will not extract the string.
Line 91: Line 174:
 
At run time, the <code>_()</code> calls will translate the string by looking it up in the translation database (created from the <code>.po</code> files) and returning the translated string.
 
At run time, the <code>_()</code> calls will translate the string by looking it up in the translation database (created from the <code>.po</code> files) and returning the translated string.
  
You can check missing references (not on <code>POTFILES.in</code> and <code>POTFILES.skip</code>) with the command
+
===Add the reference to the file===
/intltool-update -m
+
 
into <code>/po</code> directory.
+
We need to also add a reference to this file for generating the translation template.
 +
 
 +
* [[Translation_environment22#Files_and_directory|2.2.x to Gramps 3.4.x]]
 +
* [[Translation_environment4#Files_and_directory|Gramps 4.0.x, master (trunk)]]
  
==Tips for writing a translatable report==
+
==Tips for writing a translatable Python module==
 
===Use complete sentences===
 
===Use complete sentences===
Don't build up a sentence from phrases. Because a sentence is ordered in a particular way in your language does not mean that it is ordered the same way in another. Providing the entire sentence as a single unit allows the translator to make a meaningful translation.
+
Don't build up a sentence from phrases. Because a sentence is ordered in a particular way in your language does not mean that it is ordered the same way in another. Providing the entire sentence as a single unit allows the translator to make a meaningful translation.  Do not concatenate phrases or terms as they will then show up as separate phrases or terms to be translated and the complete sentence may then show up incorrectly, especially in right-to-left languages (Arabic, Hebrew, etc.).
===Use named %s values===
+
===Use named %s/%d values===
 
Python provides a powerful mechanism that allows the reordering of %s values in a string. A translator may need to rearrange the structure of a sentence, and it may not match the order you chose. For example:
 
Python provides a powerful mechanism that allows the reordering of %s values in a string. A translator may need to rearrange the structure of a sentence, and it may not match the order you chose. For example:
 
  print "%s was born in %s" % ('Joe','Toronto')
 
  print "%s was born in %s" % ('Joe','Toronto')
Line 109: Line 195:
 
             'city' : 'Toronto', 'male_name' : 'Joe'}
 
             'city' : 'Toronto', 'male_name' : 'Joe'}
  
In this case, the order of the %s formatters is not important, since the values will be looked up in the dictionary at run time to resolve the value. The translator can reorder the %s formatters, or even remove them without causing any problems.
+
In this case, the order of the %s formatters is not important, since the values will be looked up in the dictionary at run time to resolve the value. The translator can reorder the %s formatters, or even remove them without causing any problems.
 +
 
 +
Note that Python also allows a variation which some people find easier to read:
 +
print "%(male_name)s was born in %(city)s" % dict(
 +
            city = 'Toronto', male_name = 'Joe')
 +
 
 +
Some languages are using right-to-left text direction. It is important to use named arguments when there is more than one %s/%d value into a translation string.
  
 
===Provide separate strings for masculine and feminine.===
 
===Provide separate strings for masculine and feminine.===
Many languages have the concept of gender, while others don't. A sentence may need to be phrased differently depending on if the subject is male or female. By using the named %s values along with a bit of code, this problem can be solved.
+
Many languages have the concept of gender, while others don't. A sentence may need to be phrased differently depending on whether the subject is male or female. By using the named %s values along with a bit of code, this problem can be solved.
  
 
  if person.getGender() == Person.male:
 
  if person.getGender() == Person.male:
         print _("%{male_name}s was born in %{city}s\n") % {
+
         print _("%(male_name)s was born in %(city)s\n") % {
 
                 'male_name' : name, 'city' : city }
 
                 'male_name' : name, 'city' : city }
 
  else:
 
  else:
         print _("%{female_name}s was born in %{city}s\n") % {
+
         print _("%(female_name)s was born in %(city)s\n") % {
 
                 'female_name' : name, 'city' : city }
 
                 'female_name' : name, 'city' : city }
  
Line 127: Line 219:
 
Plurals are handled differently in various languages. Whilst English or German have a singular and a plural form, other languages like Turkish don't distinguish between plural or singular and there are languages which use different plurals for different numbers, e.g. Polish.
 
Plurals are handled differently in various languages. Whilst English or German have a singular and a plural form, other languages like Turkish don't distinguish between plural or singular and there are languages which use different plurals for different numbers, e.g. Polish.
  
Gramps provides a [[Translating_GRAMPS#Plural_forms|plural forms]] support, useful for locales with multiples plurals according to a number (''often slavic based languages'') or for Asian family languages (''singular = plural'').
+
Gramps provides a [[Translating_Gramps#Plural_forms|plural forms]] support, useful for locales with multiples plurals according to a number (''often slavic based languages'') or for Asian family languages (''singular = plural'').
 +
 
 +
Note, some locales need singular form with [http://en.wikipedia.org/wiki/Plural#Zero zero] and plural form might be also used in this case.
  
 
We need to call module :
 
We need to call module :
Line 134: Line 228:
 
and code like this :
 
and code like this :
  
   ngettext("singular %d", "plural %d" n) %n
+
   ngettext("singular %d", "plural %d", n) %n
  
 
Sample:
 
Sample:
Line 143: Line 237:
 
===Provide a context support.===
 
===Provide a context support.===
  
Translator needs context for a good translation. Keep in mind you can help him/her, by using context on translation string.
+
A translator needs context for a good translation. Keep in mind you can help him/her, by using context on translation string.
  
 
We need to call module :
 
We need to call module :
  from TransUtils import sgettext as _
+
  from gen.ggettext import sgettext as _
 
or
 
or
  from TransUtils import sngettext as _
+
  from gen.ggettext import sngettext as _
(if you use ngettext)
+
(if you use ngettext) # not implemented
  
 
Translation string will use context, but this will be hidden on user interface.
 
Translation string will use context, but this will be hidden on user interface.
 
  _("context|string")
 
  _("context|string")
Translator will see the translation string and an help without loading program.
+
Translator will see the translation string and a help string without loading program.
Program will only display the string in English or with an other locale.
+
Program will only display the string in English or with another locale.
 +
 
 +
===Object classes===
 +
 
 +
Gramps often displays names of primary objects (''Person, Family, Event, etc ...''), for being consistent on displayed strings (also in english!), there is a ''trans_objclass(objclass_str)'' function on TransUtils module.
 +
 
 +
So, when we need to display the primary object name in lower case into a sentence, we can use this function.
 +
 
 +
ex:
 +
from gen.ggettext import sgettext as _
 +
from TransUtils import trans_objclass
 +
 
 +
_("the object|See %s details") % trans_objclass(objclass)
 +
_("the object|Make %s active") % trans_objclass('Person')
 +
 
 +
will display:
 +
 
 +
See ''the person'' details # or See ''the family, the event, etc...'' details
 +
Make ''the person'' active
 +
 
 +
===Genitive form===
 +
 
 +
Genitive (and some other) forms need to modify the name itself into some locales, like Finnish or Swedish.
 +
 
 +
Instead of "free form" text that talks about
 +
e.g.
 +
son '''of %s'''
 +
better would be for example some tabulated format like this:
 +
  son: %s
 +
  daughter: %s
 +
which doesn't require genitive.
 +
 
 +
===Punctuation===
 +
 
 +
Use of commas, semicolons and spacing can be different than into english.
 +
 
 +
''todo''
 +
 
 +
==Changing translated text message in the source code==
 +
One of the severities in our bug tracker is "text", which ranks up as easier than "tweak" and "minor", but more difficult than "trivial". If a bug is concerned with readability or correctness of a text that Gramps outputs, whether in GUI, in a console error message, or in a produced report, then "text" is the severity to use. So why is it more than "trivial"?
 +
 
 +
As described above, any translated text in the source code gets reflected into tens of *.po files, maintained by the translators. So every time you just change it in the source, ALL the translators need to do the translation again. Normally, the translation environment will give a prudent suggestion, but there is still a manual approval step. If you check in the change, the string will not be translated until the translators pick it up.
 +
 
 +
This is why, if what you change is just a couple of spelling mistakes, a missing comma in the middle, or maybe an extra space somewhere in the message, it's a good idea to save the translators' work, by doing a global search and replace of your source message text in the *.po files, and committing these along with your change.
 +
 
 +
For short enough messages, that don't span multiple lines in the *po files, you can do it by executing
 +
perl -pi -e 's/YOUR MESSAGE BEFORE CORRECTION/your message after correction/g;' *.po *.pot
 +
in the po/ directory. Make sure you do a "git diff" and observe the results make sense. (You'll probably have to escape some characters in the regular expression, such as | or .).
 +
 
 +
To make it easier to port your changes across multiple branches, it's a good idea to separate the changes in the source tree from the po/ ones. This way, you'll be able to quickly re-apply the source changes using normal cross-branch porting workflow (such as `git cherry-pick'), and then adjust and re-run the search-and-replace in the po/ on the new branch, because, most likely, it won't reapply due to the differences in the .po layout.
 +
 
 +
{{man note| Note |To stress it again, only do it for text change that didn't change how it is going to be translated. If you'd like your change to be somehow reflected in the translations, let the translators do the work instead.}}
 +
 
 +
==Textual reports==
 +
Since Gramps-3.2 we are able to select the language for textual reports, see [http://www.gramps-project.org/bugs/view.php?id=2371 this feature].  
  
==Textuals reports==
+
For Gramps it was only available on Ancestor report (3.2.x) and detailed reports (3.3.x).
Since Gramps-3.2 we are able to select the language for textuals reports, see [http://www.gramps-project.org/bugs/view.php?id=2371 this feature].  
 
  
Currently only available on Ancestor report.
+
The capability for translated-output was added to some more (gramps core) reports, in the gramps40 branch, before gramps 4.0.0 was released. So more than the "three
 +
original reports" now have had this [https://gramps-project.org/bugs/view.php?id=2371#c33601 feature request implemented].
  
 
For providing this option:
 
For providing this option:
Line 174: Line 322:
 
         self._ = translator.gettext
 
         self._ = translator.gettext
 
         self.__narrator = Narrator(self.database, self.verbose, use_call,  
 
         self.__narrator = Narrator(self.database, self.verbose, use_call,  
                                    empty_date, empty_place,  
+
                                  use_fulldate, empty_date, empty_place,  
 
                                   translator=translator,
 
                                   translator=translator,
 
                                     get_endnote_numbers=self.endnotes)
 
                                     get_endnote_numbers=self.endnotes)
 
         self.__get_date = translator.get_date
 
         self.__get_date = translator.get_date
 +
        self.__get_type = translator.get_type
  
 
  self._("")
 
  self._("")
 
  self.__get_date(event.get_date_object())
 
  self.__get_date(event.get_date_object())
 +
self.__get_type(event.get_type())
 +
 +
[[Category:Translators/Categories]]
 +
[[Category:Developers/Tutorials]]
 +
[[Category:Developers/General]]
 +
[[Category:Addons]]

Revision as of 08:03, 2 October 2014

Coding guidelines to enable easy and correct translation of strings on the User Interface.

Introduction

Gramps has always been internationalized (see http://gramps-project.org/2006/04/looking-back-over-5-years). Therefore, all strings meant for the user should always be flagged for translation.

In order to be considered for inclusion in the offical Gramps release, any piece of code must support internationalization. What this means is that the Python module must support translations into different languages. Gramps provides support to make this as easy as possible for the developer. For enabling, a language code must be set on the ALL_LINGUAS section.

How to allow translations

Gramps is a fully-internationalized application with translations in many languages. All code which presents text to users must provide for that text to be translated. Fortunately, Gramps provides an extension of gettext which makes this fairly painless. First, alias the gettext function from the single localization instance:

from gramps.gen.const import GRAMPS_LOCALE as glocale
_ = glocale.translation.gettext

This statement imports the gettext function and aliases it as _. The translation tools treat strings wrapped in _() as translatable and assemble them into catalogs for the translators to work with; by aliasing it to gettext(), we also enable python to retrieve the translation appropriate for the user's locale.

Example 1:

print "Hello world!"

In this example, the string will always be printed as specified.

Example 1 internationalized:

print _("Hello world!")

In this example, Gramps will attempt to translate the string. If a translation exists, the call to the function will return the translation. If a translation does not exist, the original string is returned.

More complicated translations

In addition to gettext, GrampsTranslation offers two more specialized retrieval functions, ngettext and sgettext.

In some strings, it's necessary to specify different translations depending upon the number of an argument. For example,

George Smith and Annie Jones have 1 child 
George Smith and Annie Jones have 3 children

We'd code that in python as follows:

_ = glocale.translation.ngettext
_(George Smith and Annie Jones have %(num)d child, George Smith and Annie Jones have %(num)d children, n) % {num : n}

In other cases, it's necessary to provide a hint to translators, e.g.

_(Remaining names | rest)

We're making sure that the translators know that this message id means "what's left" rather than "take a nap". When the file is translated, this is no problem, because the translation doesn't include the hint -- but if the user is working in English, we don't want him to see the hint, so we need to alias _ to sgettext:

_ = glocale.translation.sgettext

Often you need to combine them. While ngettext and sgettext can each handle plain strings, neither can handle the other's strings. Fortunately the intltool message extractor is pretty stupid, so any function name that ends in either _ or gettext will work. This will work pretty well:

 _ = glocale.translation.gettext
 N_ = glocale.translation.ngettext
 S_ = glocale.translation.sgettext

Obviously you would pass the translatable string to the right function.

Encoding

String handling can be a bit tricky in a localized environment so it's important that developers understand Unicode string handling in both versions of the language.

This is mostly a problem for Microsoft Windows™: Mac OSX and Linux use UTF8 for just about everything if the locale is set up correctly (and we try to do that when Gramps starts up), so one can get away with a lot of encoding mistakes on those platforms. Windows™ on the other hand uses a slightly modified version of UTF16 for file names and retains the old DOS code page system for encoding output to cmd.exe. The take-away is that if you need to mess with input or output encoding, be sure to test on both Linux and Windows before deciding that you're done. If you're not set up for multiple-platform testing arrange with someone,, who can test for you on the platform you don't have.

Python 2

Python 2.7 has two text classes, str and unicode. Unicode objects are encoded in UTF16 internally on most platforms, and most python output functions will do the right thing with them. One caveat here: passing both unicodes and strs to os.path.join() will return a str, so either make sure when constructing a path that all arguments are unicodes or convert the result.

The bsddb module that ships with Python2 is stupid about paths and requires that they be encoded in the file system encoding. This is handled in gramps/gen/db/write.py with _encode() and independently in a few other places.

Strings from the operating system, including environment variables, are a problem on Windows™; The os module uses for input the ANSI API to the Windows SDK, which interprets the value of the environment variable according to the active code page and produces a str, converting any codepoints > 0xff to ? and often misinterpreting those between 0x0f and 0xff if the encoding of the input happens to be something other than the active system codepage. Once this is done it is quite difficult to get non-ASCII pathnames back into a useable form, so gramps/gen/constfunc.py provides a get_env_var() function that uses the Unicode API to instead. Always use that function to read environment variables which might include non-ASCII characters and avoid using os-module functions for reading paths.

By default string constants in Python 2 are str.

Python 3

Python 3 also provides two test classes, str and bytes. In Python 3, str is the unicode type and bytes is text encoded some other way. Everything pretty much "just works".

Portability Functions and constants

We've provided a couple of functions in gramps/gen/constfunc.py to ease conversion of strs to unicodes; these include the necessary tests to portably do the right thing regardless of what's passed to them and according to which version of Python is in use:

  • cuni is an alias for unicode in Python 2 and for str in Python 3. This has no protective checks so use it with care.
  • conv_to_unicode(string, encoding='utf8'): This ensures that its return value is a Unicode string which has been converted from a non-Unicode in the encoding, which defaults to UTF8 for ease of use with the GUI.
  • get_env_var(string, default=None): On Windows™ in Python2, uses the ctypes module to invoke the Microsoft Unicode API to read the value of an environment variable and return a Unicode; otherwise returns the value from the os.environ array.

There are also two constants:

  • STRTYPE is an alias for basestring in Python 2 and for str in Python 3. It can be used to test whether an object is a text-type.
  • UNITYPE is an alias for unicode in Python 2 and for str in Python 3. It can be used to test whether an object is already encoded in Unicode.

For portable string handling on all platforms and for all locales

  • Localized strings returned from gettext, ngettext, etc. are always unicode
  • Text files should always be encoded in UTF8. The easy and portable way to do this is to:
    import io
    fh = io.open(filepath, mode, encoding='utf8')
    where mode is one of r, rw, r+, or w+. Don't open these files in binary mode! Pass unicode-type strings to fh.write() and expect the same from fh.read().
  • Always read environment variables with constfuncs.get_env_var() if there's any chance that it will contain a non-ASCII character.
  • Use from __future__ import unicode_literals in any source file which might present strings to the user or to the operating system.
    When creating string literals, don't do this:
    print _(u"Eg, valid values are 12.0154, 50° 52′ 21.92″N")
    Because the u prefix was removed for Python 3.0-3.2. (It was restored in 3.3 for compatibility with 2.7, but it's not necessary.)
    Instead, put in the first line of the module
    # *-* coding: utf-8 *-*
    then in the imports section
    from __future__ import unicode_literals
    which makes all of the literals unicode. Make sure that your editor is set up to save utf-8!

Glade files

Just enable the translatable attribute on an XML element.

<property name="label" translatable="yes">_Family:</property>
<property name="tooltip" translatable="yes">Abandon changes and close window</property>
<property name="label" translatable="no"><b> - </b> </property>

Non ASCII characters

If you plan to use non ASCII characters in a string, that shall be translated, do not use escape sequences:

Eg, valid values are 12.0154, 50&#xB0; 52' 21.92"N

use instead:

Eg, valid values are 12.0154, 50° 52′ 21.92″N

In this case note the unicode characters for deg, min, sec. Ensure that your editor is set up to encode the characters in UTF-8!

Accessibility

In addition to accelerators, GtkWidget also support a custom <accessible> element, which supports actions and relations. Properties on the accessible implementation of an object can be set by accessing the internal child "accessible" of a GtkWidget. See GtkBuilder UI.

  • Gtk label

A GtkLabel with mnemonic support will automaticaly generate accessibility keys on linked GtkEntry and UndoableEntry fields. Remember that Gramps also uses custom widgets like StyledTextEditor and ValidatableMaskedEntry, which do not always have relation with a GtkLabel.

  • Toggle buttons and Icons on toolbar

Gramps often uses GtkToggleButtons and alone GtkImage (image without label), this excludes blind people and generates a poor interface for accessibility.

See Accessibility.

Addons

External addons often need to provide their own message catalogs. To pull one in, use this instead of the usual.

from gramps.gen.const import import GRAMPS_LOCALE as glocale
_ = glocale.get_addon_translator(__file__).gettext

or if you need more than one retrieval function:

 _translation = glocale.get_addon_translator(__file__)
 _ = _translation.gettext
 S_ = _translation.sgettext

The addon translator is another instance of GrampsTranslation, so the rules for creating translatable strings and for retrieving the translated values are the same as for internal modules.

See Addons development for more details.

How it works

We need at least GNU gettext, then msginit will generate a standard gettext header.

Gramps has used different environments according to versions for retrieving strings to translate:

There are two stages to getting a translation to work.

Files and directory

Translations are stored in a .po file that contains the mappings between the original strings and the translated strings, see Translating Gramps.

Translators use a generic file gramps.pot to generate their .po file. Gramps uses a utility that extracts the strings from the source code to build the .po file. This utility examines the source files for strings that have been marked as translatable. In the python source, these are the strings enclosed in the _() function calls.

Note that because strings are extracted by a script from the source file, string constants and not variables must be enclosed in the _() call. In the following example, the extraction script will not extract the string.

mystring = "Hello World!"
print _(mystring)

The correct method would be to use one of the following:

mystring = _("Hello World!")
print mystring

At run time, the _() calls will translate the string by looking it up in the translation database (created from the .po files) and returning the translated string.

Add the reference to the file

We need to also add a reference to this file for generating the translation template.

Tips for writing a translatable Python module

Use complete sentences

Don't build up a sentence from phrases. Because a sentence is ordered in a particular way in your language does not mean that it is ordered the same way in another. Providing the entire sentence as a single unit allows the translator to make a meaningful translation. Do not concatenate phrases or terms as they will then show up as separate phrases or terms to be translated and the complete sentence may then show up incorrectly, especially in right-to-left languages (Arabic, Hebrew, etc.).

Use named %s/%d values

Python provides a powerful mechanism that allows the reordering of %s values in a string. A translator may need to rearrange the structure of a sentence, and it may not match the order you chose. For example:

print "%s was born in %s" % ('Joe','Toronto')

In some languages it may make more sense to say:

print "%s is the city in which %s was born" % ('Toronto', 'Joe')

The problem is that this requires a change to the order of the arguments. Python provides a solution for this. By using named operators and dictionaries, we can say:

print "%(male_name)s was born in %(city)s" % {
           'city' : 'Toronto', 'male_name' : 'Joe'}

In this case, the order of the %s formatters is not important, since the values will be looked up in the dictionary at run time to resolve the value. The translator can reorder the %s formatters, or even remove them without causing any problems.

Note that Python also allows a variation which some people find easier to read:

print "%(male_name)s was born in %(city)s" % dict(
           city = 'Toronto', male_name = 'Joe')

Some languages are using right-to-left text direction. It is important to use named arguments when there is more than one %s/%d value into a translation string.

Provide separate strings for masculine and feminine.

Many languages have the concept of gender, while others don't. A sentence may need to be phrased differently depending on whether the subject is male or female. By using the named %s values along with a bit of code, this problem can be solved.

if person.getGender() == Person.male:
       print _("%(male_name)s was born in %(city)s\n") % {
               'male_name' : name, 'city' : city }
else:
       print _("%(female_name)s was born in %(city)s\n") % {
               'female_name' : name, 'city' : city }

This allows languages with gender differences to map nicely into your sentence.

Provide support for plural forms.

Plurals are handled differently in various languages. Whilst English or German have a singular and a plural form, other languages like Turkish don't distinguish between plural or singular and there are languages which use different plurals for different numbers, e.g. Polish.

Gramps provides a plural forms support, useful for locales with multiples plurals according to a number (often slavic based languages) or for Asian family languages (singular = plural).

Note, some locales need singular form with zero and plural form might be also used in this case.

We need to call module :

from gen.ggettext import ngettext

and code like this :

 ngettext("singular %d", "plural %d", n) %n

Sample:

msg = ngettext('Import Complete: %d second',
               'Import Complete: %d seconds', t ) % t

Provide a context support.

A translator needs context for a good translation. Keep in mind you can help him/her, by using context on translation string.

We need to call module :

from gen.ggettext import sgettext as _

or

from gen.ggettext import sngettext as _

(if you use ngettext) # not implemented

Translation string will use context, but this will be hidden on user interface.

_("context|string")

Translator will see the translation string and a help string without loading program. Program will only display the string in English or with another locale.

Object classes

Gramps often displays names of primary objects (Person, Family, Event, etc ...), for being consistent on displayed strings (also in english!), there is a trans_objclass(objclass_str) function on TransUtils module.

So, when we need to display the primary object name in lower case into a sentence, we can use this function.

ex:

from gen.ggettext import sgettext as _
from TransUtils import trans_objclass
_("the object|See %s details") % trans_objclass(objclass)
_("the object|Make %s active") % trans_objclass('Person')

will display:

See the person details # or See the family, the event, etc... details
Make the person active

Genitive form

Genitive (and some other) forms need to modify the name itself into some locales, like Finnish or Swedish.

Instead of "free form" text that talks about e.g.

son of %s

better would be for example some tabulated format like this:

 son: %s
 daughter: %s

which doesn't require genitive.

Punctuation

Use of commas, semicolons and spacing can be different than into english.

todo

Changing translated text message in the source code

One of the severities in our bug tracker is "text", which ranks up as easier than "tweak" and "minor", but more difficult than "trivial". If a bug is concerned with readability or correctness of a text that Gramps outputs, whether in GUI, in a console error message, or in a produced report, then "text" is the severity to use. So why is it more than "trivial"?

As described above, any translated text in the source code gets reflected into tens of *.po files, maintained by the translators. So every time you just change it in the source, ALL the translators need to do the translation again. Normally, the translation environment will give a prudent suggestion, but there is still a manual approval step. If you check in the change, the string will not be translated until the translators pick it up.

This is why, if what you change is just a couple of spelling mistakes, a missing comma in the middle, or maybe an extra space somewhere in the message, it's a good idea to save the translators' work, by doing a global search and replace of your source message text in the *.po files, and committing these along with your change.

For short enough messages, that don't span multiple lines in the *po files, you can do it by executing

perl -pi -e 's/YOUR MESSAGE BEFORE CORRECTION/your message after correction/g;' *.po *.pot 

in the po/ directory. Make sure you do a "git diff" and observe the results make sense. (You'll probably have to escape some characters in the regular expression, such as | or .).

To make it easier to port your changes across multiple branches, it's a good idea to separate the changes in the source tree from the po/ ones. This way, you'll be able to quickly re-apply the source changes using normal cross-branch porting workflow (such as `git cherry-pick'), and then adjust and re-run the search-and-replace in the po/ on the new branch, because, most likely, it won't reapply due to the differences in the .po layout.

Gramps-notes.png
Note

To stress it again, only do it for text change that didn't change how it is going to be translated. If you'd like your change to be somehow reflected in the translations, let the translators do the work instead.

Textual reports

Since Gramps-3.2 we are able to select the language for textual reports, see this feature.

For Gramps it was only available on Ancestor report (3.2.x) and detailed reports (3.3.x).

The capability for translated-output was added to some more (gramps core) reports, in the gramps40 branch, before gramps 4.0.0 was released. So more than the "three original reports" now have had this feature request implemented.

For providing this option:

  1. import EnumeratedListOption
  2. import libtranslate
from gen.plug.menu import EnumeratedListOption 
import TransUtils
from libtranslate import Translator, get_language_string

Sample of code:

language = menu.get_option_by_name('trans').get_value()
       translator = Translator(language)
       self._ = translator.gettext
       self.__narrator = Narrator(self.database, self.verbose, use_call, 
                                  use_fulldate, empty_date, empty_place, 
                                  translator=translator,
                                   get_endnote_numbers=self.endnotes)
       self.__get_date = translator.get_date
       self.__get_type = translator.get_type
self._("")
self.__get_date(event.get_date_object())
self.__get_type(event.get_type())