Despite the massive researches in MT, MT still immature yet, that is because a high quality translation is based on full understanding of speech/text, understanding speech is beyond machine's capability. however, for a simple check to get some confidence in the output of an MT software you may do:

1- use Mt to translate your text from source to destination language.

2- use the output text to be translated back to your source language

3- compare your original text with the output from step 2

4- the more similarity you get the higher confidence will be.

I could not understand if you mean the following measures: http://en.wikipedia.org/wiki/Evaluation_of_machine_translation by "formal evaluations". I do not know anything else for the purpose you mentioned.

I often use machine translation to read in English what my friends in China and Vietnam write in their native languages. The translation is often so poor that I cannot even understand the topic of what was written in Mandarin.  

Example: "Spider-Man zoom motor motor launched the."

Given this low accuracy, it is not hard for me to say that I would not trust the translation at all.  :-)

I admit, however, that translating related languages from the same language family may be much more accurate.  These are the situations when your analysis criteria would be valuable.

Ali Hürriyetoglu, thanks for the link. The Wikipedia article you mentioned lists studies that were done for the purposes of evaluating MT system performance, if I'm not mistaken, and moving the field forward (a distinctly system-centered paradigm). There seems to be some incongruity between the upbeat assessments of intelligibility and fidelity of translations and the kind of experience that Michael Marek is describing (thanks for that, Michael). Actual users (esp. monolinguals) can't check fidelity. MT systems don't provide any confidence measures or indication for how much to trust how accurate the translation results might be. Intelligibility (in native languages) seems to be the only parameter users can assess, at least without active knowledge of the second language. Alternatively, the back and forth translation tests can compile errors (as the Wikipedia article stated).  So, have this issue yet been addressed from the point of view of the users' experiences, perceptions, information needs, context of the use of the functionality, ect. (in the user-centered paradigm)? Has anything yet been done to support users' assessment of the system's performance? Don't know if this makes better sense.

Hello Victoria,

I agree with the conclusion of Michael about languages you don't know or that are very different. This does not apply to your case if I read correctly since you presume the user has a passive knowledge of the target language.

In this case, one interesting way to test for the acceptability is to segment your texts in smaller parts up to the phrases. As a professional translator I recommend to use to this effect the corpora-based dictionaries or concordancers wich retrieve similar portions of text such as linguee (http://www.linguee.com/). There is also the possibility to double check phrases in common dictionaries. The most important thing is to double check the acceptability of phrases (their existence) but also the way they are phrased with other phrases, all the way up to constructing sentences and clauses (this is a long process but the only one to make sure the translation is acceptable).

Even though Statistical Machine Translation uses corpora, I would recommend instead to use products that also give their bilingual sources which can also be tested for quality in general. The nature of the sources of the copora material is another criteria to be used to check for quality of translations based on this material.

On Linguee and other concordancers there is also a qualitative function or crowdsourcing evaluation of translation segments that can be very helpful to assess the quality of the given translation. For one segment that has been accepted as a good quality translation, you would not need to make the checks I suggested.

Éric

Thank you, Éric! That's precisely the kinds of strategies (plus resources) that are very helpful to know about. VR

Despite the massive researches in MT, MT still immature yet, that is because a high quality translation is based on full understanding of speech/text, understanding speech is beyond machine's capability. however, for a simple check to get some confidence in the output of an MT software you may do:

1- use Mt to translate your text from source to destination language.

2- use the output text to be translated back to your source language

3- compare your original text with the output from step 2

4- the more similarity you get the higher confidence will be.

Hello!

Ahmad suggestion is good too. The only problem is that the user is never sure if the error is one way only - from translation from language A to B or from backtranslation from language B to A, or both ways. But it is surely another tool to help double-check the correctness of the translation (if not to detect errors per se).

Éric

The suggestion by Ahmad (back translation or round-trip translation: https://en.wikipedia.org/wiki/Translation#Back-translation) is often used in CSCW/CMC prototypes to help improve cross-language communication. It has also been used in a crowd-sourced manner to correct translations with only two groups of monolingual speakers (e.g., http://www.cs.umd.edu/hcil/monotrans/). It does have the limitations Éric  identified.

if the translation machine is a multi-lingual one, such as google translation. You can choose to machine translate the translated version into a third language. For instance, you translate Chinese into English, and then translated the English into German, and then into Russian, and finally back to Chinese to see if the Chinese is still the same as the source language. That helps to some extent. Just an idea you can try out for fun.

MT is not capable to find the context. The same text has different meaning in technical and , say, economic context. Technical: …bare (not insulated) conductor (wire) runs along the train. Common: … bare (naked) conductor (person) runs along the train. – Exhibitionist or so? Similar traps can be found in medical texts. Statistical engine translators (e.g. Google) depend on the share of translations of certain context. Answer to your question - generally I would not trust MT although it does not mean that sometimes the translation is correct.

Andrzej Smolarz, your exhibitionist example is pretty funny! Thanks for the input.

Ahmad, Éric, Scott, I'm not so sure how I feel about the round-trip translation. If one can read the script, it would make sense to check with an authoritative list of some kind (Éric's concordance suggestion). Not sure how well any thesauri-style list would work if the reader can't even segment the text. Perhaps, alignment tools would be better there, but again only if individual tokens (words and phrases) are clear. Mutual comprehensibility/readability of scripts (in target and source languages) is obviously a stumbling block.

Bing Hu, I like a third language idea, but would you choose such drastically different language on purpose? Any reason? Why not stay within the same language family?

Dear Victoria,

Ideally an MT should work on three levels. The denotation level , the pragmatic level and the relevance level. To create an algorithm that caters for all these three different levels we need two things. The right mathematical equation and lots of hard work.

For the time being. knowledge of both languages is the best answer. But in the future we will be able to rely on MT.

Dear Victoria,

I chose Chinese as an example mostly because I am Chinese,LOL.. 

Also, if the two languages are in the same language family, say, German (Source Language)and English(TL), and the user has a passing knowledge of the source language, It would be easier to judge if the MT made a relatively acceptable translation by simply reading the English translation. A drastically different language can better test the flaws of a MT cause the language syntax and pragmatic use of language can be more vast. 

I also like Eric's suggestion about dividing the source language into phrases so that it would be easier for the MT to do the work without context. However, a context free translation would make no sense in the Target language, one can always re-organize the logic of the sentence based on her/his knowledge of her/his mother tongue. 

Victoria,

I would second Ahmad Hassanat & Bing Hu's suggestions.

On the round-trip translation, I did this awhile back when I had to write some letters in languages I didn't know. Complex sentences came back from the round-trip translation as garbage, and I was able to see how to simplify my sentences in order to get a satisfactory translation.

I understand the criticisms of this approach listed on the Wikipedia page, but they don't apply as much to your case, since you are more concerned with features for user feedback, rather than objective evaluation. (In their first example, the user may or may not choose to rephrase based on the system feedback, but the translation would get across. The second example is just a case of an idiom not being properly identified.)

Anyhow, for interactive interface features, I could imagine a system that displays a round-trip translation as the user is typing/speaking or whatever, and this could be used to help guide the user toward overcoming the limitations of the MT system (simpler syntax, avoiding idioms, etc).

Hu's idea adds an extra level of encoding that might serve to constrain the input even more. There is an interesting example of this in a scene in the film Soy Cuba, which was filmed in Spanish and then translated from Spanish to Russian for Soviet audiences. It is dubbed in Russian, but with a man just speaking over all of the Spanish parts. When the film was released in the U.S. (I think Coppola brought it over), they took the Russian script and translated that into English for the English subtitles. But there is a scene where there are some American businessmen in a bar, speaking English. As you read the subtitles, the translation has gone English -> Spanish -> Russian  -> English subtitles, and the differences between what you hear and what's written in the subtitles is rather amusing....

Also, keep in mind that whatever you do, you are ultimately up against Quine's Indeterminacy of Translation problem, and nothing will be perfect.

Translating back and forth is an intuitive method but I do not think you can use it as a formal method. I would recoment you to check MT Quality Estimation methods.

You can read Lucia Specia papers on this topic: http://staffwww.dcs.shef.ac.uk/people/L.Specia/publications.html

And there is a toolkit that can be used to do MT quality estimation: http://staffwww.dcs.shef.ac.uk/people/L.Specia/projects/quest.html

the trouble with google translate is that it tries so hard to answer every query that it will spill out garbage.  i like oleg skopstev;s answer.  check that what you put in is what you get out on the other side.  if you are using rule-based translation you can just say "sorry" if all the rules fail, which is a lot. 

Dear Victoria ,

My article on problems of machine translation could be give some ideas.

good luck

a.