A Short Guide to Measuring and Comparing Machine Translation Engines
This is an article from the Asia Online November 2011 Newsletter that provides useful advice for meaningful comparisons of MT engines and is authored by Dion Wiggins, CEO of Asia Online. So the next time somebody promises you a BLEU of 60, be skeptical, and make sure you get the proper context and assurances that it was properly done. And if they say they have a BLEU score of 90 you know that you are clearly in the bullshit zone. “What is your BLEU score?” This is the single most irrelevant question relating to translation quality, yet one of the most frequently asked. BLEU scores and other translation quality metrics greatly depend on many factors that must be understood in order for a score to be meaningful. A BLEU score of 20 in some cases can be better than a BLEU score of...