What factors affect machine translation (MT) quality? I’m looking for human, scientific (published research), state-of-the-art, specific reflections, not AI-generated, impressionistic, older, general discussions.
I often hear about the quantity of resources being the crux of the issue. However, my hunch is that language pair, and more precisely language combination (directionality), is also an influencing factor. Say you're translating from Japanese (high-context language) into French (low-context language). In Japanese, you don't need to specify gender, number, etc. In French, you need that information, which means you'll have to make a guess (and take a chance), perform external research, ask the client, etc., but anyway, you probably won't find the answer within the source text (ST). Arguably, a MT system cannot make good decisions in that sort of context. Whereas, if you translate from Spanish into French, most of the information you need for the French target text (TT) can be retrieved directly from the Spanish ST.
When I researched the question in 2017-2018, it was clear from the literature that linguistic distance was a relevant factor in MT quality. For example: "Machine translation (MT) between (closely) related languages is a specific field in the domain of MT which has attracted the attention of several research teams. Nevertheless, it has not attracted as much attention as MT between distant languages. This is, on the one side, due to the fact that speakers of these languages often easily understand each other without switching to the foreign language. […] Another fact is that MT between related languages is less problematic than between distant languages…" (Popović, Arčan & Klubička, 2016, p. 43).
But what now in 2023, soon 2024, with LLMs and recent improvements on NMT? Thank you!
One of the primary factors that affect MT quality is the availability and quality of training data. Machine translation models rely on large amounts of parallel text data, which consists of source language sentences paired with their translations in the target language. The more diverse and extensive this training data is, the better the MT system can learn to generate accurate translations. Insufficient or low-quality training data can lead to poor performance and inaccurate translations.
Another critical factor influencing MT quality is the linguistic complexity and structural differences between languages. Some languages have complex grammatical structures or ambiguous meanings that are challenging for machine translation systems to handle accurately. For instance, word order variations between languages can pose challenges for translating sentences correctly. Languages with a high degree of inflectional morphology or idiomatic expressions also present difficulties for MT systems.
The level of domain expertise also affects MT quality significantly. Machine translation models trained on general-purpose texts may struggle when translating specialized content from specific domains such as legal, medical, or technical fields. These domains often have unique terminologies and sentence structures that require specialized knowledge for accurate translation. Therefore, developing domain-specific machine translation models by incorporating relevant domain-specific training data can greatly enhance translation quality within those domains.
Additionally, the architecture and algorithms used in machine translation systems play a vital role in determining their performance. Different approaches such as rule-based, statistical, or neural machine translation have distinct strengths and weaknesses. Neural machine translation (NMT), based on deep learning techniques, has shown significant improvements over traditional statistical methods due to its ability to capture complex linguistic patterns effectively.
Lastly, post-editing by human translators plays a crucial role in improving MT output quality. While machines excel at processing large volumes of text quickly, they often struggle with nuances, idiomatic expressions, or cultural references. Human translators can refine and correct the machine-generated translations to ensure accuracy and naturalness. Combining the strengths of both human expertise and machine efficiency through post-editing can result in high-quality translations.
In conclusion, several factors influence the quality of machine translation systems. These include the availability and quality of training data, linguistic complexity, domain expertise, system architecture, and post-editing by human translators. By addressing these factors and continuously improving MT models and algorithms, we can enhance the accuracy and reliability of machine translation systems for effective cross-lingual communication.
Reference: Koehn, P. (2010). Statistical Machine Translation. Cam
George Benneh Mensah, thank you for your reply. However, I’m looking for state-of-the-art, specific reflections, not older, general discussions. What you wrote (or copy-pasted?) is very broad and based on a 2010 text. As I explained in my question, when I looked into the matter in 2017-2018, linguistic distance was still said to be a relevant factor. For example:
"Machine translation (MT) between (closely) related languages is a specific field in the domain of MT which has attracted the attention of several research teams. Nevertheless, it has not attracted as much attention as MT between distant languages. This is, on the one side, due to the fact that speakers of these languages often easily understand each other without switching to the foreign language. […] Another fact is that MT between related languages is less problematic than between distant languages…" (Popović, Arčan & Klubička, 2016, p. 43).
But what now in 2023, soon 2024, with LLMs and recent improvements on NMT? Thank you!
Etienne Lehoux-Jobin, the initial response was to put the question into perspective. However, on the hand, based on the context of the question you asked an enhanced form of response is provided for your critique.
In fact, LMs and NMT systems require vast amounts of high-quality parallel corpora to learn from. In 2023 and soon 2024, with the advent of Large Language Models (LLMs), such as GPT-3 or OpenAI's Codex, access to extensive training data will no longer be a major issue. These models can generate synthetic parallel corpora by aligning monolingual data from different languages, thus expanding the available resources for training NMT systems.
Noteworthy had been that traditional statistical machine translation systems struggled to produce accurate translations when faced with specialized domains like legal or medical texts. However, recent improvements in NMT have shown promising results in domain adaptation by fine-tuning pre-trained models on specific domains. For instance, researchers at Facebook AI developed M2M-1000, a multilingual model trained on 2,200 language pairs across various domains. This model achieved remarkable performance across multiple language pairs and domains.
Notwithstanding, LLMs have significantly improved contextual understanding by incorporating large-scale pre-training on diverse text sources like books and websites. This enables them to capture more nuanced meanings and produce more accurate translations. One significant challenge is the lack of linguistic resources for low-resource languages. While LLMs and NMT have made progress in multilingual translation, they heavily rely on data availability. For languages with limited resources, the quality of MT output remains subpar.
Reference:
Facebook AI (2021). M2M-100: The Multilingual Machine Translation Model. Retrieved from https://ai.facebook.com/blog/m2m-100-the-multilingual-machine-translation-model
Jan Buts, thank you so, so much, this is EXACTLY the kind of work I’m looking for! I suppose the list of references will also be of great help! And the following conclusion of the thesis is in line with my own hypothesis:
"The present thesis confirms that there exists an inverse relationship between linguistic distance between languages and neural machine translation quality, thus proving the null hypothesis. The more linguistically distant languages are, the worse machine translation quality gets. This inverse relationship is found to exist across multiple text types and machine translation tools, and even when the number of other error types present in the translations is controlled for." (pp. 71-72)
- Badges
- Science topic