I18N for Package Maintainers

Good afternoon. Our second talk this afternoon is from Mark Harbour, who some people might know as Zugschlus, talking to us about how Debian package maintainers can deal better with translations in their packages. Take it away, Mark. >> Hi everybody. It feels good to be here in Brest, in France. It's the second slot of the conference, so we are still fresh. I hope you will be still fresh when I'm finished with you. I'm going to talk about translations for a package maintainers. Um and I would like to lead you through the technical process of managing a package that has translations and then also mention some of the social aspects of managing translations because you don't only have the need You don't only deal with your users on one side, but you also deal with your upstream and with the translators on the third side, so you find yourself caught between a rock and a hard place sometimes and um I did things wrong a couple of a couple of times and so I wrote documentation and this is the presentation of this documentation. It's all in the wiki, so you can read it all up and you can fix it if I'm wrong. Um my name is Mark Harbour. I'm I go by Zugschlus. I have been a Debian developer since 2001. I work as a freelance consultant in Germany and the other German language countries and I would really like to have a project that has something to do with Debian again for a change. That has not been the case for a while, so I would be glad if there would be some Debian work coming my way. Um The Diplom Informatiker roughly translate to a master in in in informatics, as you can obviously here German is my first language and English is is is just my second one but I do not consider myself target audience for translated software. Everything I have in IT is in English, my notebook, my phone, everything. So I only deal with translations when I'm packaging things because I'm aware that there are people who want their software in German, Korean, Japanese, French, Italian and what we and we have a real truckload of languages in Debian. Um I think that we started translating software around in the middle of the zero years. Um we do translate debconf templates. That's actually the bulk of work that you as Debian developer have to do. Um we also translate program output that usually does your upstream but we have software where we are the upstream ourselves. We translate manual pages. There is a lot of work to do. Um we translate um package description. This is of course wrong in the in the slide. This is meant to be package descriptions and there are probably other translations that I am not aware of. Um What does a package maintainer have to do to be nice to translators? Um we don't want to throw away their work. We don't want to appreciate their work. We do want to appreciate their work and um we don't want to throw away their work. Um To do actual translations, we have copious amounts of documentation. I think this is like five times in the wiki and if you browse the web you will find many documentation about how to do the actual translations. But what we don't have in a very good way is telling the maintainer of the package how to handle that um translations. So when you are a translator, I'm not going to tell you anything new. Um package maintainers may not care much about translations. The package maintainers also can't understand what is in the translation files because they are obviously written in a foreign language. Um on the other side, translators might not care about what they are actually translating. They are just taking a string in English and translating it to it it it to their language and they obviously and often don't see how that translation is going to be to be presented to them. There's nothing we can do about that. Translators might be non-technical persons, so they might come up with weird words for things that we use naturally because it's a recognized technical term. That's an issue that many people have with translations. Also, we don't we we don't have the possibility to deal with that because we cannot influence that for every language. Myself, I tend to ignore the German translation as well because I have everything in English. So, I don't need want to argue with the translators even in my native language. The original author writes usually text in English. I have never seen a software that has its native texts in another language than English. Um then in the software or in the source form, the strings are marked as translatable. And then we have a software that pulls those strings out of the source code. And generates a template file. The translators use that template file from wherever they get it. They make a copy of the template of the template file in their own language and then do the actual translation. And then they submit the translation to the package maintainer. In Debian, this is either the BTS or as also MR. I think that the BTS works better here because translations are complete files. So, you don't need to do fancy patching. Um and as the package maintainer, you only drop the file in the in the correct directory and that's done. So, you don't need the fancy VCS for that. Um of course, you can do that, but um don't do it at the wrong place. The package maintainer then includes the translation file in the package, commit, build, upload the package, and magically the the software presents the correct language to the user and everything and everybody is happy. Now, let's take a look at some code. This is roughly taken from from passwd.c, so we all So, we all have used that program. I hope you have never seen this particular message, but um you can see that um here is the marker that this is a translatable string. This underscore is actually a macro mapped to a function that pulls out the string and at runtime displays looks up the the the correct translation and displays it to the user. In the second case, a message is written to syslog and I want you to notice that this message is not marked as translatable because we want that the the the the error message to message to be in English in the log so that it's easier to process it with software. Then, you use a software called xgettext that um sifts through your sources and generates the template the translation template. The translation template looks basically like this. It has three parts. Um a comment header that has like copyright of the author of the English part and some boilerplate place where the translators put in their data. And there are technical headers where we have version numbers, timestamps, versions, encoding information. And then we have the actual translation. Um please notice that we have message ID. This is the English text to be translated. And we have the message string which is the place where the translator enters their work. And we have comments saying where in the source the string was pulled from. And you also might notice that this no option allowed after names is actually pulled from two points in the source. So, there's only it is only needed to translate a string once even if it's in the source multiple times. This saves the translator time. The translator then sends you their work. They have filled in the comment header with their own copyright which you have to record in Debian copyright. And um hopefully they also enter the technical the technical header um updating um updating the timestamps. Um I have chosen Korean because this obviously looks like a foreign language to us to us Europeans. Um this is the translation and this is what the user gets displayed when the string is going to be printed. So far, so good. So far the the theory. But software changes and string changes and and and strings strings change. What happens when we reword a message or when some new features are added that need new messages. That would invalidate every time what um old translation we we already have, right? Thankfully, no. Um Of course, it can happen that when a string is changes is changed, we don't have the correct translation and the software then falls back to English. So, it might be that the software or even a man page is a wild mixture be- be- between the local language and English. So, the software usually has a threshold when the translated ratio goes below a certain threshold, it the complete translation is no longer used at runtime and everything is is shown in English. At least this is quite ugly. But, thankfully, we do have um a process that allows us to update strings without losing data. Um First, as the package maintainer, you just repeat the extraction process like calling at xgettext, and then you have a program called message merge, where it does like a three-way merge from the old translation, the new templates, and possibly a new translation, and then all those PO files where the actual translation is in get updated automatically. New strings are added with an empty translations. Um strings that are no longer the that are no longer in the software get moved to the very bottom of the file so that the translations for those strings are is not lost. So, existing work is not destroyed. Um when you handle translations in your package, you have to consciously do this update. And then, you might want to ask the translation that that that translators to update the translations. In the technical header of the translation, there is the email address of the individual having done the translation and the and the language team. It's the best idea to email both. We do have software that does this automatically. Um but do not do this too often because this is also the this is always busy work for the translators. So, it is a good time to call for updates of translations when we are approaching when we are approaching a freeze. You can do translation updates even during the the the the freeze period. The release team is pretty is is pretty relaxed about that. Um but don't do it unnecessarily. There are some translators who are very eager to work, so you might get a little avalanche of new translations where where whenever you do an upload. Um sometimes you don't want that because you upload like work in progress or something with some milestone of your development progress. Um and then the translations are bound to be invalidated again when when you continue working. But on the other side, you cannot tell the translators not not translate this other putting the information in the change log and the change log is not a communication means. We are working all in version control systems. Um the translators might do that. Usually, they work from your source package, but I have seen but I have received translations from a Git from things I have pushed to Git as well. Um it is common to update translations in your clean target when you build your package. Um but I think the clean target is actually not supposed to affect the committed parts of the package. So, I think that is the wrong place. Um also when you are using a build mechanism that does not build your package in inside the tree you have checked out but does a working copy like when you have the build area directory um updated templates might never end up in your commit because the copy is made, the package is built, templates are updated, source package is built and then the working copy is thrown away. So you will So So So you So you So you will never see the um updated templates. So um translators who are working from who are working from your git are going to be um um um are going to be confused. um When you commit updated templates too often like you do it automatically, you end up with huge commits and this is going to hurt you when you rebase or merge um or cherry pick. Um so maybe it's a better idea to not do those template updates automatically but on the other hand you will forget doing that when you're preparing an upload. Um I have a checklist for that but I think that we should have that automated. I do not have an idea how to do that. I do not have an idea how to do that. Let's talk a bit of the translators. um The translators are are are often non-technical people. They may not care much about licenses, comments, timestamps, credits. They sometimes only deliver the translation because they might be using software that don't displays or um those headers to you. And frankly I have received for example a translation for package A that had the comment in the header this translation is under the same license as package B. So, if I hadn't paid attention, I would have ended up with a GPL translation in a BSD license package. Nightmare. So, you must pay attention what people are sending you. Um translators may not be around anymore. So, when you send out translation requests, you are sending out like 20, 30 message um email messages to people working on your package. And you will get, I think, about 10 bounces out of that. So, better prepare and see which of your message are reaching their target so that you get so that you end up with current translations. Um in Debian, we have program translations. That's what upstream does for you. So, there is generally nothing not much to do. Um we have debconf translations. Um that's the same just that the PO and pot files are in the Debian directory and there are man page trans man page translations that are in a subdirectory of the directory where the man page sources reside. And if you get a translation, you need to pay close attention what you are putting where. When you put a translation template for debconf in the source code directory, this is going to end up mixed up by message by message merge and you will never get that in a same state again. Um so, be careful to put your PO files in the right place in your package. Um when you change your punctuation and little typos, message merge is supposed to catch that automatically, but it does not always do that. Um you can be really nice to your translators by just doing the same change you did in the actual source in the English part of your PO files so that the translations don't get invalidated. But this is really time-consuming. Um, so do this only if you have too much spare time. And now it's your turn because everything I told you is already in the wiki. And I surely have not touched every side aspect of that because Colin is already holding up that 2-minute warning. Um, so please there is an edit button and improve that webpage. Thank you for your patience. Oh, but by the way, we don't have a Klingon language team. >> All right, does anybody have any questions for Mark? >> So, actually not a question, just a remark or a bit of an experience that I want to share. I guess this is streamed and recorded. Yeah, so I have to be a bit vague, unfortunately. Ask me in private later. So, it's a warning about be careful what the translators actually provide. Um, I'd say if it's from the Debian translation team, I guess I guess you can trust them. But I have a very story first hand where somebody showed up with a translation in a language that you would not understand at all. It's a different different letters to start with. And it contained, um, interesting things, not the actual translation. So, if you do not know who is this translator that's just providing some input, find, in case of doubt, ask somebody else or try some automatic translation on the net and check whether it's really what it's supposed to be. >> I must admit that I never do that. >> Okay, we can probably squeeze in one more. >> Uh >> Okay, you win. >> Um >> [snorts] >> Um For example, uh that Debian Debian use uh report a bug uh he translated a pure pure file. Um At the the same time, the upstream use sh- he also translated the same pure file and the committed to the upstream. >> Mhm. >> How to deal this situation in Debian? >> I think that the best solution for this would not be would not would be to not touch the upstream translations in the Debian package, but ins- but but instead either send the translation upstream so that they so so so so that they so that they can include the translation in the upstream package, making users of other distributions also benefit from the translation, or to ask the translator to submit the the the the translation directly upstream. I would not advise in merging upstream translations with local translations because you don't know what you are merging because you don't understand what's in the translation. >> Uh so pretty quickly just to add to uh what have been said, if you have any doubts about the translation, please send a mail to the Debian/ language mailing lists, and they will be able to tell you if if it came from them or not. So, yeah. >> All right, I think we'll finish there. Uh let's give Mark another round of applause. >> [applause] >> Woo!