2018 - Human Perception of MT Quality

2001

This paper describes a Machine Translation (MT) evaluation experiment where emphasis is placed on the quality of output and the extent to which it is geared to different users' needs. Adopting a very specific scenario, that of a multilingual international organisation, a clear distinction is made between two user classes: translators and administrators. Whereas the first group requires MT output to be accurate and of good post-editable quality in order to produce a polished translation, the second group primarily needs informative data for carrying out other, non-linguistic tasks, and therefore uses MT more as an information-gathering and gisting tool. During the experiment, MT output of three different systems is compared in order to establish which MT system best serves the organisation's multilingual communication and information needs. This is a comparative usability- and adequacy-oriented evaluation in that it attempts to help such organisations decide which system prod...

Language, Literature, Culture & Integrity, 2020

The paper demonstrates the qualitative evaluation of the English to Indian languages Machine Translation Systems, namely PBSMT and NMT hosted on Google's Translate. This system is popularly known as Rosetta, formerly governed by the Phrase-based approach and is presently governed by the neural module of source and target languages. In this study, a model corpus set of 1k English sentences of cross-domain data has been applied considering various types of verbs as input text to evaluate the output of the online systems in Indian languages. In order to evaluate the output text in a qualitative manner, the Intertranslator Agreement (IA) of three human translators has been considered with their scores on a five-point scale. The scores are calculated by the Fleiss' Kappa statistical measure with regard to comprehensibility and grammaticality on the basis of which error analysis and suggestions have been provided for improvement. In addition, the system has also been quantitatively evaluated on the basis of word error rate and sentence error rate. Furthermore, all the erroneous entities have been analyzed through computational typology. The strategy for evaluation is to evaluate the output text of Indian languages based on the five-point scale with scores that range from 0-4 where 0 refers to incomprehensible or ungrammatical, 1 = little meaning or disfluent, 2 = neutrality, 3 = comprehensible or grammatical and 4 suggests flawless in both cases.

International Journal of Translation , 2005

2021

Along with the development and widespread dissemination of translation by artificial intelligence, it is becoming increasingly important to continuously evaluate and improve its quality and to use it as a tool for the modern translator. In our research, we compared five sentences translated from Armenian into Russian and English by Google Translator, Yandex Translator and two models of the translation system of the Armenian company ‘Avromic’ to find out how effective these translation systems are when working in Armenian. It was necessary to find out how effective it would be to use them as a translation tool and in the learning process by further editing the translation. As there is currently no comprehensive and successful method of human metrics for machine translation, we have developed our own evaluation method and criteria by studying the world's most well-known methods of evaluation for automatic translation. We have used the posteditorial distance evaluation criterion as...

ARPN journal of engineering and applied sciences, 2018

Currently, the high volume of international information exchange involves a wide range of localities. As each locality comes with its own distinctive dialect, the need for an effective means of language translation is becoming more and more apparent. Among the concerns of information professionals is the capacity of an interested party to access web information offered in an unfamiliar language. Classified under the wide field of artificial intelligence, machine translation (MT) is an approach related to natural language processing. The machine translation technique involves the use of software for the conversion of documents or verbalized information from one natural language into another. Of late, a substantial number of procedures have been proposed for the fashioning of an efficient MT system. While these procedures were observed to be capable in certain areas, they were found wanting in others. The objectives of this endeavour are to (a) conduct a thorough investigation on mach...

Translation Studies: Theory and Practice

Along with the development and widespread dissemination of translation by artificial intelligence, it is becoming increasingly important to continuously evaluate and improve its quality and to use it as a tool for the modern translator. In our research, we compared five sentences translated from Armenian into Russian and English by Google Translator, Yandex Translator and two models of the translation system of the Armenian company Avromic to find out how effective these translation systems are when working in Armenian. It was necessary to find out how effective it would be to use them as a translation tool and in the learning process by further editing the translation. As there is currently no comprehensive and successful method of human metrics for machine translation, we have developed our own evaluation method and criteria by studying the world's most well-known methods of evaluation for automatic translation. We have used the post-editorial distance evaluation criterion as ...

Texto Livre: Linguagem e Tecnologia, 2020

Despite fast development of machine translation, the output quality is less than acceptable in certain language pairs. The aim of this paper is to determine the types of errors in machine translation output that cause comprehension problems to potential readers. The study is based on a reading task experiment using eye tracking and a retrospective survey as a complementary method to add more value to the research as eye tracking as a method is considered to be problematic and challenging (O’BRIEN, 2009; ALVES et al., 2009). The cognitive evaluation approach is used in an eye tracking experiment to determine the complexity of the errors in the English–Lithuanian language pair from easiest to hardest as seen by the readers of a machine-translated text. The tested parameters – gaze time and fixation count – demonstrate that a different amount of cognitive effort is required to process different types of errors in machine-translated texts. The current work aims at contributing to other ...

Proceedings of the workshop on Human Language Technology - HLT '93, 1993

This paper reports results of the 1992 Evaluation of machine translation (MT) systems in the DARPA MT initiative and results of a Pre-test to the 1993 Evaluation. The DARPA initiative is unique in that the evaluated systems differ radically in languages translated, theoretical approach to system design, and intended end-user application. In the 1992 suite, a Comprehension Test compared the accuracy and interpretability of system and control outputs; a Quality Panel for each language pair judged the fidelity of translations from each source version. The 1993 suite evaluated adequacy and fluency and investigated three scoring methods.

2006

Abstract: Evaluation of MT evaluation measures is limited by inconsistent human judgment data. Nonetheless, machine translation can be evaluated using the well-known measures precision, recall, and their average, the F-measure. The unigrambased F-measure has significantly higher correlation with human judgments than recently proposed alternatives. More importantly, this standard measure has an intuitive graphical interpretation, which can facilitate insight into how MT systems might be improved.

This paper objects to the current consensus that machine translation (MT) systems are generally inferior to human translation (HT) in terms of translation quality. In our opinion, this belief is erroneous for many reasons, the both most important being a lack of formalism in comparison methods and a certain supineness to recover from past experience. As a side effect, this paper will provide evidence for a much more favorable judgment of the performance of contemporary MT systems. We will present and discuss known methods of automatic MT evaluation, give real world examples of both machine and human translation and finally suggest an universal formal evaluation method to handle both human, as well as MT output in a comparable fashion.