Chapter 9. Common Functionality

Shell completion means that, similarly as for command names, it is possible to contextually complete command parameters by pressing the Tab key. This allows you to efficiently type in the command line, as well as to quickly remind yourself of options and option parameters without resorting to documentation or browsing the file system.

For example, pressing Tab just after the posieve command will complete sieve names, and Tab after the -s option will complete sieve parameters based on sieves that precede it in the command line. This:

$ posieve s<TAB>

will show all sieves beginning with s, and complete the sieve name once sufficient number of characters have been entered to uniquely determine it, while this:

$ posieve stats -s m<TAB>

will show all parameters to stats beginning with m, and complete one of them after few more characters are typed in.

Various parts of Pology can be configured through the configuration file .pologyrc in the root of user's home directory (~/.pologyrc for short). The configuration file does not have to exist, so you have to create it when you want to configure something for the first time. It must be UTF-8 encoded.

The configuration file is in the INI format, which is composed of sections beginning with a [section] line, and fields of the form field = value within a section. Comments can be written after # character at the beginning of the line. Here is an example of a ~/.pologyrc file:

[global]

[user]
name = Chusslove Illich
original-name = Часлав Илић
email = caslav.ilic@gmx.net
po-editor = Kate

[enchant]
# Autodetection sufficient.

[posieve]
msgfmt-check = yes
param-ondiff/stats = yes

# Project setups follow.

[project-kde]
language = sr
language-team = Serbian
team-email = kde-i18n-sr@kde.org
plural-forms = nplurals=4; plural=n==1 ? ...

This configuration contains five sections: [global], [user], [enchant], [posieve], and [project-kde]. The [global] section set options that have an effect throught Pology, and here it is empty. The [user] section provides some information on the person who uses Pology. The [enchant] section configures the Enchant spell checker wrapper, used by Pology for spell checking. The [posieve] section configures the behavior of the posieve script. The [project-kde] section provides information on a project that the user contributes translation to.

Some details about the configuration file syntax are as follows. Leading and trailing whitespace in section and field names and values is not significant, e.g. foo=bar is same as foo = bar. Percent (%) character is used to expand the value of another field, for example:

rootdir = /path/to/somewhere
datadir = %(rootdir)s/data

where the %(...)s is Python's string interpolation syntax. Importantly, when you need a literal % character within a value (such as in plural-forms field in the previous example), you must repeat it twice, %%. Switch-type fields (msgfmt-check in the previous example) can take any of the following values for the two states: 0, no, false, or off; and 1, yes, true, or on (case is not important).

Sections in the configuration can be of one of four general types:

Internal tool sections are documented together with the respective tools, while sections of other types are described in the following.

When mentioning configuration fields in their documentation and elsewhere, they are referred to as [section]/field. If there is only a fixed number of possible values to a field, this is denoted as [section]/field=[VALUE1|VALUE2|VALUE3|...]; if one of the values is the default, it is prefixed with a star (*).

There are great many places in Pology where you can supply a matching pattern, to select or deselect something. This could be a PO file by its path, a PO message by its msgid, etc. Almost always and by default, this matching pattern will be a regular expression (or regex for short). Regular expressions are a powerful pattern matching language, a fascinating topic in their own right, and they will serve you well in just about any context of searching on computers. The plain text editor that you use probably offers regular expressions in its search dialog, so does your office text processor, and so on.

Actually, the only point of this brief section is to impress the importance and usefulness of regular expressions onto you, in the case that you have not used them yet. The Internet is full of tutorials on regular expressions, so that there is no point in linking any one particular here.

It should be mentioned that different regular expression engines have somewhat different syntax and expressiveness. Pology uses regular expressions from the Python Standard Library, described here: http://docs.python.org/library/re.html (keep in mind that this page is a reference, and not a tutorial, so you should look elsewhere to learn basics of regular expressions).

Pology scripts that can recursively search directory paths for PO files will usually provide several options by which certain files can be included or excluded from processing. The first pair of these options include or exclude files by path:

Especially those PO files which are used at runtime (as opposed to those used for static translation), but others too, are frequently sufficiently identified by their domain name. The domain name is the base name of the installed MO file without the extension, e.g. for /usr/share/locale/sr/LC_MESSAGES/foobar.mo the domain name is foobar. If, in a given translation project, PO files for a given language are all collected under one top directory of that language, their base names are also formed of domain names.^[39] When this is the case, it may be more convenient or safer to match PO files by their domain names instead of paths, which is done by options:

All inclusion and exclusion options can be freely mixed and repeated, with consequent resolution. A file is processed if it matches all inclusion patterns (if any is given) and does not match at least one exclusion pattern (if any is given). The other way around, a file is not processed if does not match at least one inclusion pattern (if any is given) or it matches all exclusion patterns (if any is given).

Sometimes it is convenient to make a temporary or semi-permanent grouping of files, such that the file group can be referenced through a single argument instead of repeating all the files all the time. This is particularly useful when shell piping is not applicable or not comfortable enough. The classic and simple way to group files is by having a file-list file, which contains one file path by line, which a shell command can read to collect files to process.

Many Pology scripts can write and read file-list files. Having scripts write such files automatically is simple enough, just check given script's documentation to see if it has this capability (e.g. the -m option to posieve). More interesting are the special features that you can use when writing a file-list file manually. You would do this for standing categories which are periodically updated, such as a list of PO files ready for release.

For completeness, here is first an example of a basic file-list file:

xray/alpha.po
xray/bravo.po
yankee/charlie.po
yankee/delta.po

As is usual for path arguments to Pology scripts, you can specify both file and directory paths, and directory paths will be searched recursively for PO files (or whatever the file type that the script is processing):

xray/
yankee/
zulu/echo.po
zulu/foxtrot.po

You can add comments by starting the line with hash (#), and have empty lines:

# Translations ready for release.

# Full modules.
xray/
yankee/

# Specific files.
zulu/echo.po
zulu/foxtrot.po

The inclusion-exclusion functionality equivalent to inclusion-exclusion command line options is provided through inclusion-exclusion directives. They are specified by starting the line with colon (:), followed by directive type token, followed by a regular expression. The directives are:

For example, if a whole module should be processed but for one PO file in it, it is easier to list the whole module and exclude that one file, as compared to listing all other files:

# Modules.
xray/
yankee/
# Exclude november.po (in whichever module it is).
:-november

Ordering and position of include-exclude directives is not significant, as they are all applied to all collected files. The semantics of application of multiple directives is the same as that of counterpart command line options.

File-list files are normally fed to Pology scripts with the following option:

In some contexts, Pology scripts color the terminal output for better visual separation and highlighting of important parts of the text. Examples include warning and error messages, data presented as tables and bars, and, importantly, matched segments of the text in search and validation operations.

Output coloring is turned on by default, but sensitive to output destination: the text is colored if the output is to the terminal (using terminal escape sequences), but not if it is piped to a file. Pology scripts provide the following options by which you can influence this behavior:

One of the general aims of Pology is to fit well with other tools typically found in translation workflows based on PO. Although examples of this can be seen throughout the manual, this section gives the overview of integration by the particular supported tool.

While line wrapping of message strings irrelevant to programs that fetch translations from them, it may be significant to the translator, especially when editing the PO file with a plain text editor. Well-wrapped strings make it easier for the translator to follow the text structure, especially in longer messages.

Most Gettext tools (msgmerge, msgcat, msgfilter, etc.) provide options to wrap or not to wrap strings, where wrapping is done on the given column and escaped newlines (\n). Pology can produce this type of wrapping ("basic" wrapping) as well, but it can also wrap on expected visual line breaks in known text markup, e.g. <p> and <br> in HTML ("fine" wrapping). Compare this message in basic wrapping alone:

msgid ""
"<p>These settings control the storage of the corrected images. "
"There are four modes to choose from:</p><p><ul><li><b>Subfolder:</"
"b> The corrected images will be saved in a subfolder under the "
"current album path.</li><li><b>Prefix:</b> A custom prefix will be "
"added to the corrected image.</li><li><b>Suffix:</b> A custom "
"suffix will be added to the corrected image.</li><li><b>Overwrite:</"
"b> All original images will be replaced.</li></ul></p><p>Each of "
"the four modes allows you to add an optional keyword to the image "
"metadata.</p>"
msgstr ""

and in basic and fine wrapping together:

msgid ""
"<p>These settings control the storage of the corrected images. "
"There are four modes to choose from:</p>"
"<p>"
"<ul>"
"<li><b>Subfolder:</b> The corrected images will be saved in a "
"subfolder under the current album path.</li>"
"<li><b>Prefix:</b> A custom prefix will be added to the corrected "
"image.</li>"
"<li><b>Suffix:</b> A custom suffix will be added to the corrected "
"image.</li>"
"<li><b>Overwrite:</b> All original images will be replaced.</li>"
"</ul>"
"</p>"
"<p>Each of the four modes allows you to add an optional keyword "
"to the image metadata.</p>"
msgstr ""

If you are editing the PO file with a dedicated PO editor, it may itself provide finely tuned wrapping and ignore the wrapping in the PO file, in which case Pology's wrapping facilities are superfluous to you^[40]. But a PO editor may also present strings wrapped just as they are in the PO file (and most do!), when Pology's fine wrapping is just as useful as in combination with a plain text editor.

At least for alphabetic languages, the most convenient wrapping may be fine wrapping alone (no basic wrapping), while turning on editor's dynamic (visual) line wrapping. This both makes the text structure easy to follow, and allows editing the translation by logical units (paragraphs, list items) without manually adjusting column breaks or putting up with ugly overlength or mid-broken lines. However, for ideographic languages, editor's dynamic line wrapping may produce bad results, and there basic wrapping might be necessary. In fact, for the moment, for ideographic languages it may be better to pass Pology's wrapping entirely and stick with Gettext's wrapping, since the wrapping algorithm in Gettext is more sophisticated and directly supports ideographic writing systems.

If no wrapping mode is specified when the given PO file is written out, Pology will apply basic wrapping, just as Gettext tools do. There are three general sources from which Pology tools may try to determine the wrapping mode for the given PO file, in decreasing priority: from the command line options, from the PO file's header, and from the user configuration. A tool may or may not provide command line options and configuration fields for wrapping, but PO file headers are always consulted (since this is in Pology's core PO file handling facilities). See the description of the X-Wrapping header field for how to set the wrapping mode in the PO header, and the set-header sieve for how to set this field in many PO files at once.

The PO header is a natural place to provide the information which holds for the PO file as whole. Pology scripts, sieves, and hooks can take into account a number of header fields, when available, to automatically determine some aspects of processing. The fields considered are as follows:

All of the listed header fields may be set manually, when you get to work on the particular PO file. But frequently it is possible to set them automatically, or at least automatically for the first time with later manual corrections where needed. For this you may use the set-header sieve. If PO files are periodically merged by the translation project automation (rather than each translator merging on his own only the PO files which he is about to update), the natural moment to run set-header is just after the merging. If translation is done in summit, you can specify in summit configuration to set header fields on merging.

Pology enables the user to insert special processing elements, called hooks, at many places in the processing chain. Hooks are Python functions with certain prescribed input, output, and behavior. Depending on the exact combination of these three ingredients, there are various hook types. Finally, some hooks can be adapted to a given context through their hook factories. Pology defines many hooks internally, and users can add their own external hooks.

Usage of hooks is best illustrated through examples. Suppose that you want to use the the find-messages sieve to look for a certain word, but the text contains XML-like tags of the form <tagname>...</tagname> which happen to be throwing off your search. Suppose that there exists a hook called remove-xml-tags, in the Pology library module remove, which takes a piece of text as input and returns that piece of text cleared of any XML-like tags. Then you could insert this hook into the search to clear the tags before matching the text, by using the filter: parameter to find-messages:

$ posieve find-messages -s filter:'remove/remove-xml-tags' ...

Here remove/remove-xml-tags is the hook specification, and this is its usual simplest form: the module name, followed by slash, followed by the hook name. (Sometimes it can be only the module name, when the hook function within that module has the same name as the module, but this is rare.) The hook specification was enclosed in single quotes, for the shell to see it as single string; this was not necessary here, but it is a good habit to keep up when adding hooks through command line, because hook specification can get quite involved.

Suppose now that there is a single hook that can remove any kind of markup from the text (not only XML-like tags) called remove-markup, but that it has to be told which markup to remove, by giving it one of the markup type keywords known to Pology. Continuing the previous example, this could be done like this:

$ posieve find-messages -s filter:'remove/remove-markup~"docbook4"' ...

Now the hook specification is remove/remove-markup~"docbook4". Note that outer single quotes in the command line are necessary, as otherwise the shell would strip internal double quotes, which are here integral part of hook specification. remove-markup is actually a hook factory, which produces a hook based on the parameters given after the tilde (~) character. Here "docbook4" is that parameter; why must it be quoted? Because the part after the tilde is passed as argument list to a Python function, and "docbook4" must be of string type, which is in Python denoted by quotes. For a hook factory foo/bar which would take a string and a number, the hook specification would be foo/bar~"qwyx",5. Sometimes a hook factory has default values for some or all of its arguments; in the latter case, if the defaults are sufficient, the part after the tilde in the hook specification can be left empty (e.g. foo/bar~).

Hooks can be language- and project-dependent. Suppose that in your language the letters are sometimes accented, but the accents should be ignored on spell-checking. Then Pology may contain a hook which strips accents from text in your language. If your language code is ll, and the hook is remove-accents in (language-specific) module remove, you could check spelling while ignoring accents using the the check-spell-ec sieve:

$ posieve check-spell-ec -s filter:'ll:remove/remove-accents' ...

The hook specification now also contains the language code separated by colon, as ll:.... If the hook is project-specific instead, it is prefixed with pp%..., where pp is the project identifier and percent sign the separator. If the hook is both language- and project-specific, then the specification is ll:pp%... or pp%ll:....

With all the different heuristic checks and rules that Pology can apply, false positives -- messages proclaimed invalid when they are actually valid -- are inevitable. False positivies are very inconvenient in serious automatic quality control effort. They make it harder for translators to spot real problems, which in turn demotivates them to apply automatic checks at all. If there is one or few dedicated persons in the translation team to tweak and apply automatic checks, they would be particularly hard-hit with this negative feedback. False positives can reduce automatic quality control from a strong normative element in the workflow, to merely advisory "run-if-you-have-the-time" extra.

For this reason, most checks in Pology provide a way for them to be disabled on certain messages, files, or the processing batch, such that it is possible to methodically cancel false positives. From the other side, it is usually possible to run one or few checks on their own, in order to be easier to define and debug. Each checking tool and element documents such functionality, and in the following only some general patterns are described.

The simplest method to disable or enable some checks is "dynamically", for single validation run, through an option to the tool which is being run. For example, the check-rules sieve provides several parameters to select and deselect validation rules which are to be applied. The important point here is that checks in Pology usualy have some sort of a unique identifier, a keyword, by which they can be referred to.

"Static" methods to disable or enable checks are those where the instruction is written down somewhere, in a specific format, and automatically taken into account by the validation tool in subsequent runs. There may be several static methods to disable a certain check, differing in their reach: a group of PO files, single PO file, single message, or even a part of the text in the message. Within one PO file, the following methods are common: