It depends upon the kind of linguistic theories you subscribe to and the nature of your research. Generally speaking, linguists don't perform experiments but rely upon those within cognitive psychology, neuroscience, etc. A central exception is corpus linguistics:

McEnery, T., & Hardie, A. (2011). Corpus linguistics: method, theory and practice. Cambridge University Press.

Glynn, D., & Fischer, K. (Eds.). (2010). Quantitative methods in cognitive semantics: Corpus-driven approaches (Vol. 46). Walter de Gruyter.

There are some other sources you might be interested in:

Litosseliti, L. (Ed.). (2010). Research methods in linguistics. Continuum.

Gonzalez-Marquez, M. (Ed.). (2007). Methods in cognitive linguistics (Vol. 18). John Benjamins Publishing.

Hi Jiri

Here is some useful sources related to experimental linguistics

-B. MacWhinney, B., Bates, E., & Kliegl, R. (1984) Cue validity and sentence interpretation in English, German, and Italian. Journal of Verbal Learning and Verbal Behavior, 23, 127-150.

-Bard, Ellen Gurman, Dan Robertson, and Antonella Sorace. 1996. Magnitude estimation of linguistic acceptability. Language 72:32-68.

-Cowart, Wayne. 1997. Experimental syntax: Applying objective methods to sentence judgments. Sage.

-Featherston, Sam. 2005a. Magnitude estimation and what it can do for your syntax: Lingua 115.

-Keller, Frank. 2000. Gradience in grammar: Experimental and computational aspects of degrees of grammaticality. Doctoral Dissertation, University of Edinburgh.

-Keller, Frank. 2003. A psychophysical law for linguistic judgments. In Proceedings of the 25th Annual Conference of the Cognitive Science Society .

-Sorace, Antonella, and Frank Keller. 2005. Gradience in linguistic data. Lingua 115:1497-1524.

-Schütze, Carson. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. The University of Chicago Press.

Stevens, Stanley Smith. 1957. On the psychophysical law. Psychological Review 64:153-181.

Can you provide more details on your project?

ad Andrew Messing: Dependency on some "linguistic theory" was exactly what I was trying to avoid.

As for Tony McEnery and Andrew Hardie, that is the corpus linguistics I am familiar with. But the CL is based (in ideal case) on non-manipulative observation (which is I think pure way to do science), while now I need to know, how to do experiments properly. Thank you for the links!

ad Fetita Belkacem Kamel-eddine: also thank you a lot for the inspiration, I'll go throug it.

You are welcome Jiri

If you are interested in experimental paradigms intended to test hypotheses or theories about languages/linguistics, the best place is, in my opinion, the cognitive sciences. The problem is that these run the gamut from philosophical scholarship by e.g., Dennett or Searle to quantum mechanical models of consciousness (not just the most well-known Orch-OR model of Penrose & Hameroff but also e.g., Stapp and even the quantum-like models advocated mostly by Khrennikov and associates).

However, a core has always been language and cognition. So you might check out some of the following:

Stemmer, B., & Whitaker, H. A. (Eds.). (2008). Handbook of the Neuroscience of Language. Academic Press.

Lockyer, L., Bennett, S., Agostinho, S., & Harper, B. (2009). Handbook of research on learning design and learning objects: issues, applications, and technologies (2 volumes). IGI Global

Gertner, Y., Fisher, C., & Eisengart, J. (2006). Learning Words and Rules Abstract Knowledge of Word Order in Early Sentence Comprehension. Psychological Science, 17(8), 684-691.

Perception of motion affects language processing

(http://www.researchgate.net/publication/8112352_Perception_of_motion_affects_language_processing/file/d912f509bdbb2b55f2.pdf)

Perception of Auditory Motion Affects Language Processing

(http://onlinelibrary.wiley.com/doi/10.1207/s15516709cog0000_54/pdf)

Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk

(http://pubman.mpdl.mpg.de/pubman/item/escidoc:1158591:1/component/escidoc:1158590/Workshop_NAACL_2010.pdf?mode=download)

The Magic Spell of Language: Linguistic Categories and Their Perceptual Consequences

(http://dare.ubvu.vu.nl/bitstream/handle/1871/17090/Stapel_Journal?sequence=2)

A note on the last paper: I was using it for a project that was intended to consist of response time tasks judging particular words to set-up the framework for a functional neuroimaging study only to find that the lead author is perhaps the most extreme case of dishonest research practices and fraudulent production of data in the sciences (worse than Hauser). The main model behind the paper is sound, and independent of Stapel (the researcher who faked results for decades), but it is still an important cautionary tale.

Another issue is that the literature on general methods/ reference works on experimental paradigms to linguistics that are from other fields are scanty (reference works on psycholinguistics, behavioral research methods, or neuroimaging paradigms do address linguistics research but are naturally not centered around them.. Mainly, one finds sources such as

Time to Speak: Cognitive and Neural Prerequisites for Time in Language (Language Learning Cognitive Neuroscience Series)

in which the inclusion of research by psychologists, cognitive neuroscientists, social psychologists, etc., provides examples of research on language/linguistics that provide references to or descriptions of research paradigms you may find useful (event-related potential studies, reaction time tests, etc.). Another issue is that most linguists do not have access to neuroimaging technology, cutting out an oft used tool.

Article Perception of motion affects language processing

I think the best way to go includes looking at the most recent high impact papers that use the methods you are interested in, which is why I asked you to provide details on the paradigm or research question.

ad Vitor Zimmerer: yes, that is what we do. But many papers seem to be controversial, so I want to have some broader view, that is why my question is so general.

Why don't you tell us what your paradigm and/or research question is, so that we can help you if we can? Do you work in a secret lab?

Jiri,

your words "Dependency on some "linguistic theory" was exactly what I was trying to avoid" are bait strange: in experimental research, experiments must test an hypothesis, which has to be derived from a theory. So, experimental research cannot be independent of theories, that is even the contrary: theories must guide the experimental process.

By the way, experimenting requires not only methodological skills, but also skills in inferential statistics. Designing an experiment is designing the independent variables (the experimental factors), the task, the measures, and how they will be statistical analyzed. In others words, before conducting the experiment, you need to decide which statistical tests you will carry out for testing your hypotheses.

So I suggest you to look at handbooks in cognitive psychology and on inferential statistics, with a particular interest to mixed and multilevel analyses (they are taking more and more importance in psycholinguistics research).

ad Thierry Olive: I do not contradict that the experiments should test an hypothesis derived from some theory. But in my opinion the method of the testing of the hypothesis can be theory independent. At least in its general sense. So I do not want to rely just on the methodology inherited from our predecessors in the field but to catch some inspiration in others. That is why I have put the question so vaguely – I know how to find "how these tasks are usually done", but I wandered whether some general methodology for the linguistic experimental research exists, or some examples of methodologically well done experimental researches across (even contradictory) linguistic theories can be given.

As for the statistics, yes, we know:-)

That would depend solely on what it is you are trying to determine. I have done great deal of research into schizophrenic speech and to dialect differences between Eastern New England dialects. The methodologies for each are quite different. You can can read the methodology of investigating schizophrenic speech in my article "The Ice Cream Stories:A Study in Normal and Psychotic Narrations," which appeared in The journal "Discourse Processes" vol. 9, 305-328. In that, I specifically lay out how to ascertain valid linguistic methodologies when researching.

You can find lists of references, teaching materials, tutorials, etc. on our webpage for fieldworkers/corpus linguists who want to carry out experiments (and for psycholinguists who want to carry out work in the field): http://experimentalfieldlinguistics.wordpress.com/

I noticed someone mentioned Carson T Schutze, my doctoral adviser recommended one of his papers on the topic of creating linguistic experiments

http://www.linguistics.ucla.edu/people/cschutze/thinking_asking.pdf

@InCollection{Schu05,

author = {Sch{\"u}tze, Carson},

title = {Thinking About What We Are Asking Speakers to Do},

booktitle = {Linguistic Evidence: Empirical, Theoretical and Computational

Perspectives},

OPTcrossref = {},

OPTkey = {},

pages = {457-485},

publisher = {Mouton De Gruyter},

year = {2005},

editor = {Stephan Kepsar and Marga Reis},

note = {Studies in Generative Grammar 85},

}