Multimodality can be used to build inventories of the semiotic resources, organizing principles, and cultural references that modes make available to people in particular places and times: the actions, materials and artifacts people communicate with. This has included contributions to mapping the semiotic resources of visual communication and colour, gesture and movement, gaze, voice and music, to name a few.

Multimodal studies have also been conducted that set out to understand how semiotic resources are used to articulate discourses across a variety of contexts and media for instance school, workplaces, online environments, textbooks and advertisements. The relationships across and between modes in multimodal texts and interaction are a central area of multimodal research.

Multimodal research makes a significant contribution to research methods for the collection and analysis of digital data and environments within social research. It provides novel methods for the collection and analysis of types of visual data, video data and innovative methods of multimodal transcription and digital data management.

Four core concepts are common across multimodal research: mode, semiotic resource, modal affordance and inter-semiotic relations. Within social semiotics, a mode is understood as an outcome of the cultural shaping of a material through its use in the daily social interaction of people. The semiotic resources of a mode come to display regularities through the ways in which people use them and can be thought of as the connection between representational resources and what people do with them. The term modal affordance refers to the material and the cultural aspects of modes: what it is possible to express and represent easily with a mode. It is a concept connected to both the material as well as the cultural and social historical use of a mode. Modal affordance raises the question of what a mode is ‘best’ for what. This raises the concept of inter-semiotic relationships, and how modes are configured in particular contexts. These four concepts provide the starting point for multimodal analysis.

Dear Martin, did you have a look at K. O'Halloran's research?

Warmly, Encarna

You may sample some ideas from the online journal

http://llt.msu.edu/ as well as http://e-flt.nus.edu.sg/main.htm

I personally find Eva Lam's work interesting as she notice a low achiever of Chinese Immigrant in the mainstream California classroom become conversant in English online.

Lam, Wan Shun Eva (2000). Second Language Literacy and the Design of the Self: A Case Study of a Teenager Writing on the Internet. TESOL Quarterly, 34 (3): 457-483.

Republished as:

Lam, Wan Shun Eva (2009). Second Language Literacy and the Design of the Self in Computer Assisted Language Learning: Critical Concepts in Linguistics Routledge.

You may want to look at the Journal of Multimodal Communication: http://www.degruyter.com/view/j/mc for studies across a variety of contexts.  

Many thanks for your answers to the question posed at the outset which was about 'the major research findings' produced by the multi-modal approach. These have been very helpful and interesting in themselves. The fullest answer was in fact  the very first answer from Muhammad Shoaib. This gave a thorough and comprehensive account of multimodality as a method, for which I am grateful. But it does not really tell us what has been discovered by the application of the method. Instead we are told "Multimodal research makes a significant contribution to research methods". I think that this was really the point of my question. Is multimodality a research programme in itself with its own accumulating sets of findings? Or is it a method that supports work in various other paradigms of the human and social sciences. I followed Encarna's advice and looked up O'Halloran's work and again I found some very sophisticated and innovatory methodological techniques. Eva Lam's work (following the advice of Jyh Wee Sew) certainly looks interesting and offers what I would regard as real findings but does not seem to rely on multimodal methodology as such. So I'm hoping to look through the journal recommended by Srilakshmi Ramakrishnan. It looks interesting. But I still wonder if 'Multmodality' is fundamentally the name of a methodology. Any thoughts?

Martin,

Humbly enough, I think it is a method; probaly, like corpus linguistics.

Warmly, Encarna

I don't really think of multimodality as a methodology; in my discipline (translation studies) we have multimodal analysis which seeks to understand how polysemiotic texts work in translation (though I don't know of any headline findings of the kind you are looking for; the findings that I'm aware of are, unsurprisingly, that the multimodality of multimodal texts is a more or less important factor in their translation). Multimodality, for me, is a feature of textuality which is in itself an object of study rather than a methodology. I hope this makes sense!

Many thanks, Carol. That makes very good sense

Although I would agree with Carol, also from the perspective of translation studies, I would also say that the study of the multimodal features of a texts will determine the methodological approach. In other words, we cannot analyze a multimodal text in the same way we analyze a verbal text. Kress and van Leeuwen, for instance, have developed a framework of analysis, largely based on functional linguistics, aimed at studying the visual component of texts. I would say it is only partially successful. 

Here's a recent title edited by Sigrid Norris

"Multimodality: Critical Concepts in Linguistics," Volumes I-IV 

ISBN: 978-1-13-884550-3 

Attached is the Table of Contents