It is an unfortunate combination of both. In my attempts to take great care, time attention (and yes, years and years) in attempting to translate measures, back-translate, evaluate, test, and do it all again (and again) - I am convinced that, no matter the time, care and adaptations (even when your stats look ok) there are some concepts that just do not ever translate - also it may not be the concept but the scaling and references made (e.g., comparing to self, others), cultural tendencies to rate high or low, etc. No wonder most published psychology research is based on about 5% of the global population !!!! (see Arnett, 2008). Yes, it hurts professionally do work outside the US, but I am committed. However, as much care as I take with trying to get a measure right, I have less faith in quantitative measures than ever (especially in sub-saharan Africa) and will always include mixed methods (qualitative). These things (measurement details) get easily brushed aside by authors and editors because they take long and (as I note) even after the time put in may not be feasible, appropriate, or publishable.

I agree with Laura. It is HARD to establish cross-cultural equivalence in quantitative measures of attitudes (or behavior, for that matter). It is relatively EASY to establish a superficial appearance of equivalence. It's hard in part because of the matter of language -- translation--and in part because language comparability (accurate translation) doesn't necessarily speak to the equivalence of the meaning of a concept.

In large cross-national attitude surveys, the original researchers may follow a procedure of translating and backtranslating, which seems satisfactory on the surface but does not guarantee an equivalence of meaning. Researchers who do secondary analysis using large cross-national data sets, such as the World Values Survey, take it for granted that such translation equivalence (using, say, English as the ultimate standard) has been achieved. But do the questions/prompts mean the same thing in different cultures?

I have addressed this in some of my published research. But I also recall a very specific example of how tricky a simple question can be if you fail to put it into context. In this case, in 1991 some American researchers and I were working with Estonian researchers to conduct a social survey in Estonia. We had an inter-generational sample (parents and their children). One question on the survey (in English), which came from prior surveys in English, was "Suppose your father (mother) were to receive an award from the government. How proud would you be?" (This was one in a series of questions aimed at determining how closely identified children were with their own parents.) In this case, our Estonian colleagues could easily TRANSLATE the English question into Estonian. No problem there. But what did the question MEAN in the context of Estonia in 1991? Our colleagues said that their own answer would depend on the meaning of the term "government." "WHICH government," they asked. "The existing Estonian Communist leadership in the still-existing Soviet Union? The government in a soon-to-happen democratic and independent Estonia? The local government, or the country's government?" And so we had to reformulate the question to make it less contextually sensitive. What we ultimately settled on was to replace the word "government" with "community." But we did this only after an extended discussion of the meaning of "community" and what the language and cultural equivalent would be in English and Estonian.

This is a simple example in a political context. But widely accepted and widely used survey scales have analogous problems. There's no simple solution. At the very least researchers who are doing secondary analysis of data collected in cross-cultural contexts need to think about the meaning of the questions. One of my favorite "problem questions," which is used in both the World Values Survey and many other multinational surveys, is one on "national pride": "How proud are you to be [an American]? Another example is the question of how much confidence people have in various political institutions (the president, legislature, police, business leaders, the courts, etc.). What does the word "confidence" mean in different contexts?. I discuss this issue in an unpublished paper (2008) in which I looked at the relation between concern about terrorism and support for civil liberties in the USA and Europe. You can find it on my RG page.

Wonderful discussion. On a basic level, i think many researchers do not even see that there is a problem in taking a research instrument developed in one culture and bringing it to another. Many researchers do not understand--for instance--that the concepts and context are not the same, as Brian Silver so eloquently states above. i discuss a number of these issues in my publications on Ethics in violence Against Women Research and in my book, Interviewing Clients Across cultures. Lots of examples in those pieces.

In my own field, I am endlessly frustrated with colleagues who take an instrument developed in the U.S., for instance, translte it into a variety of languages, and then publish the results comparing the prevalence of a given social problem. Some of the differences are undoubtedly due to translation and other errors. i am similarly frustrated when researchers interview college undergrads about an issue, and then generalize to all of humanity.