TechGist05

This is another post from Juan Rowda, this time co-written with his colleague Olga Pospelova, that provides useful information on this subject. Human evaluation of MT output is key to understanding several fundamental questions about MT technology including: Is the MT engine improving? Do the automated metrics (which MT systems developers use to guide development strategy) correlate with independent human judgments of the same MT output? How difficult/easy is the PEMT output task? What is the fair and reasonable PEMT rate? The difficulty with human evaluation of MT output is closely related to the difficulty associated with any discussion of translation quality in general in the industry. It is challenging to come up with an approach that is consistent over time and across different people. Being scientifically objective is particularly a challenge. So, like much else in MT, estimates have to be made, and some rigor needs to be applied to ensure consistency over...

I have always found Luigi Muzii's aka Il barbaro (The Barbarian) opinions on the "translation industry" interesting and have often felt that his blog never got the attention it deserved. Possibly because he often wrote with frequent references to Roman historical precedence, in what I have been told is an Italian scholarly style, and perhaps because English is not his preferred language to communicate complex thoughts.    His willingness to state the obvious (to common sense) sometimes makes him unpopular, especially with the MT naysayers e.g. "Moreover, translation data - i.e. project data - has a limited lifespan and, at some point in time, it becomes outdated, possibly inaccurate, and definitely irrelevant." I think that many in the "translation industry" still fail to realize that the bulk of the material that they translate;   i.e. the manuals, documen...

This is an important subject and one that needs ongoing examination and continuing study to get to real insight and find greater process efficiency. I hope that this post by Silvio Picinini will trigger further discussion and I also invite others who have information, and opinions to share on this subject, to step forward. In my observation of MT output over time, I have seen that MT produces a greater number of actual errors, but the types of errors most often generated by MT are easy to spot and clean up. Unlike the incorrect or inconsistent terminology and sometimes misunderstood source errors, that may often be hidden in the clear grammar and flow of a human translation.  These human errors are much harder to find, and not as easy to correct without multiple stages of review by independent reviewers.  ----- It is common to see a focus on the errors that machine translation makes. But maybe there are also strengths in MT that can benefit translators and customer...

The issue of rapidly understanding the MT output quality as precisely as possible BEFORE post-editing begins, is an important one. Juan Rowda from eBay provides an interesting way to make this assessment based on linguistic criteria and provides a model that can be further refined and defined by motivated users. Automated metrics like BLEU used by MT system developers provide very little of value for this PEMT effort assessment. This approach is an example of the kinds of insightful tools that can only be developed by linguists who are engaged with a long-term MT project and want to solve problems that can add real value to the MT development and PEMT process. I think it is interesting that somebody outside of the "translation industry" came up with this kind of practical innovation, that can facilitate and greatly enhance efforts in a project involving translation of several hundred million new w...