TechGist05

This is a more fleshed-out version of a blog post by Pete Smith and Henry Anderson of the University of Texas at Arlington already published on SDL.com . They describe initial results from a research project they are conducting on MT system quality measurement and related issues.  MT quality measurement, like human translation quality measurement, has been a difficult and challenging subject for both the translation industry and for many MT researchers and systems developers as the most commonly used metric BLEU, is now quite widely understood to be of especially limited value with NMT systems.  Most of the other text-matching NLP scoring measures are just as suspect, and practitioners are reluctant to adopt them as they are either difficult to implement, or the interpretation pitfalls and nuances of these other measures are not well understood. They all can generate a numeric score based on various calculations of Precision and Recall that need to be interpreted with great c...

This is the second post in our posts series on machine translation quality. Again this is a slightly less polished and raw variant of a version published on the SDL site . The first one focused on BLEU scores , which are often improperly used to make decisions on inferred MT quality, where it clearly is not the best metric to draw this inference. The reality of many of these comparisons today is that scores based on publicly available (i.e. not blind) news domain tests are being used by many companies and LSPs to select MT systems which translate IT, customer support, pharma, financial services domain related content. Clearly, this can only result in sub-optimal choices. The use of machine translation (MT) in the translation industry has historically been heavily focused on localization use cases, with the primary intention to improve efficiency, that is, speed up turnaround and reduce unit word cost. Indeed, machine translation post-editing (MTPE) has been instrumental in helping loca...

This is the first in a series of posts discussing various aspects of MT quality from the context of enterprise use and value, where linguistic quality is important, but not the only determinant of suitability in a structured MT technology evaluation process. A cleaner, more polished, and shorter studio version of this post is available here . You can consider this post a first draft, or the live stage performance (stream of consciousness) version. What is BLEU (Bilingual Evaluation Understudy)? As the use of enterprise machine translation expands, it becomes increasingly more important for users and practitioners to understand MT quality issues in a relevant, meaningful, and accurate way. The BLEU score is a string-matching algorithm that provides basic output quality metrics for MT researchers and developers . In this first post, we will review and look more closely at the  BLEU score , which is probably the most widely used MT quality assessment metric in use by MT researchers an...

This is a guest post by Luigi Muzii ( and his unedited post title ). Luigi likes to knock down false idols and speak plainly, sometimes with obscure (to most Americans anyway) Italian literary references. I would characterize this post as an opinion on the lack of honest self-assessment and self-review that pervades the industry (at all levels) and thus slows evolution and progress. Thus, we see that industry groups struggle to promote "the industry", but in fact, the industry is still one that "gets no respect" or is "misunderstood" as we hear at many industry forums.  While change can be uncomfortable, real evolution also results in a higher and better position for some in the internationalization and globalization arena. Efficiency is always valuable, even in the arts. In my view, the companies that solve the most challenging and interesting translation problems today (and thus earn the most money from translation related activity) are all ...