Dear Ziad,

Scientific Evidence has to be considered part of any research. Thesis supervisors who ignore this important step will end up with a lower quality report.

A few resources:

Davitz, J. R., & Davitz, L. J. (1967). A guide for evaluating research plans in psychology and education. Teachers College Press, Teachers College, Columbia University.

Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2006). Understanding and evaluating research in applied and clinical settings. Psychology Press.

Best regards,

Behrouz

Dear Ziad

Scientific Evidence has to be considered part of any research. I do not understand the statement “Surprisingly, Scientific Evidence is overlooked by many theses supervisors!”

I am not familiar with other research domains, but in my domain we have to consider it all the time whenever we have to explain the behavior. For example if I do not have a scientific theory for the behavior of a system because it is too complex, I will try to formulate a hypothesis and conduct some experiments to prove or disprove the hypothesis. During the process the experimental results that are noted need to be explained based on the known behaviors which are based on past scientific theories that have been put forward by other researchers. We always consider this as an important validation.

The notion of "scientific evidence" is shorthand language. A more accurate notion would be "extremely high statistical probability", which could still be overturned by an even higher statistical probability ..

The outcome of a research thesis which is not based on scientific evidence, are not valid .... ,,,,????

A research thesis goes through many stages :

**Begins just by an idea, in general, by one or more questions: the problem that led us to this search,

**Propose some hypotheses

** Followed by the choice of methods or techniques to conduct this research; and try to find solutions to the initial problem,

**Then, if the experiences and systematic observations confirm this hypothesis, at point you can predict the results of the experiment; these results will be evaluated and validated .

It's just my point of view

Dear Abdalla:

Thank you for your this highly descriptive comment pertaining to Qualitative and Quantitative research approaches. It couldn't be explained in a better way!

Regards,

Ziad

Dear Nageswara:

First, allow me to thank you for your valuable comment!

What I meant by the statement “Surprisingly, Scientific Evidence is overlooked by many theses supervisors!” is that certain research advisors think that the results obtained from research experiments speak for themselves. They totally ignore the validation step implicating that these results did or did not happen by mere chance. This might seem strange to you to come from a researcher, but it happens.

Kind regards,

Ziad

Nice one Abdalla! ;-)

Thanks Ziad. I have not experienced that. In fact I normally insist to my students to explain the behavior. If they cannot provide a rational explanation, it is difficult to accept the experimental behavior. Many a times we suspect the experimentation itself could have gone bad. So if the student is not able to rationally explain then we sometimes repeat the experiment to make sure that the results are valid.

Indeed Nageswara! This is something that needs to be emphasized to students.

In my institution, we have integrated the methodology of research as a course in our undergraduate programs.

Dear Ziad,

Scientific Evidence has to be considered part of any research. Thesis supervisors who ignore this important step will end up with a lower quality report.

A few resources:

Davitz, J. R., & Davitz, L. J. (1967). A guide for evaluating research plans in psychology and education. Teachers College Press, Teachers College, Columbia University.

Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2006). Understanding and evaluating research in applied and clinical settings. Psychology Press.

Best regards,

Behrouz

Couldn't agree more Behrouz!

@Ziad

We have a course (Seminar and methodology of research) in our department as well. I think it is a vital and necessary course for graduate students.

@Behrouz

Great! Certainly such a course has an added value.

@Ziad

I agree that this course has an added value. Actually from this year, this course is mandatory for all graduate students of civil engineering Departments in all Iran's universities.

Research Methods is a mandatory course for all graduate students:

TECH 6292. Research Methods in Technology — 3 hrs.

Research problem and purpose, value of literature review, research proposal elements, quantitative and qualitative research methodologies, measurement of data and statistical methods, reporting of research and evaluating. (Spring)

@Ziad and @Nageswars,

What is the best text book for this course?

@Behrouz

I will get you the name of the book on Monday.

1. Fraenkel, J. R. & Wallen, N. E. (2012). How to Design and Evaluate Research in Education. (8th ed.) New York: McGraw-Hill, ISBN-978-0-07-809785-0. Or

2. Fraenkel, J. R., Wallen, N. E., & Helen Hyun. (2015) How to Design and Evaluate Research in Education, (9th ed.), 704 pages, New York: McGraw-Hill, ISBN-13 9780078110399.

3. American Psychological Association (2010). Publication Manual of the American Psychological Association (6th ed.), Washington, D.C., ISBN: 978-1-4338-0561-5

Research methodology books:

Arora, H. (2011). Research Methodology: a step-by-step guide for beginners. Abhigyan, 29(3), 62-64

John Kuada. (2012). Research methodology: A project guide for university students. Samfundslitteratur.

Ziad, etal. -

Having done a great deal of work in hypothesis testing in the past, it is my opinion that many may not trust statistics for scientific evidence as they should, because such tests are often not implemented in a practical way, yet researchers may think they need them. Too often one p-value is concentrated upon, and because it is a function of sample size, merely changing your sample size may change your conclusions. So I urge more use of confidence intervals (CIs). Some other statisticians may fuss about CIs, saying that many may not interpret them technically correctly. But from a practical point-of-view, I strongly believe that people in need of such information are far more likely to obtain the interpretable information they need from CIs, whenever possible, than from hypothesis testing. If you want to encourage the use of scientific evidence, then good, practical, simply interpretable statistical evidence is very important. - Jim

One of the key problems is that reserchers often interpret "significant results" or low p-values as evidence, what is a severe misconception. Literature is full of examples where p-values are used to "prove" or "disprove" hypotheses (what logically is utter nonsense) and authors claim that they provide evidence for something simply because some tests turned out to be "significant". (I downvoted Nageswara's post because he was taking about proving and disproving hypotheses, what is a weird and harmful mental concept).

The p-value is often denoted as "evidence against the null hypothesis". So far I agree - with some stomachache. Please be very clear about the fact that the distribution of p-values under the null is uniform, so (under the null) getting any p-value is equally likely! - Simply having a low p-value does NOT allow to infere if this p-value came from such a uniform or any other distribution!. But even if one takes this a such a measure of "evidence against the null", still a strong evidence against the null does NOT imply strong evidence *for* the alternative. In fact, for tiny p-values the alternative might be even less likely than the null (and also, the likelhood of the observed data under the alternative may be smaller that the likelihood under the null).

(Empirical) evidence is only given by the data. We may summarize relevant aspects of a large body of data to specify the kind of evidence we like to address. A p-value is clearly no such permissive summary. CIs are far better, since they at least provide an effect size estimate. Bayes factors may also be seen as a measure of evidence ("evidence" as the "force" to refine or precise expectations).

Feel free to google or medline for "evidence and p-values" and similar to get 100s of papers dealing with this issue.

[Edit]

Some papers:

Goodman: Ann Intern Med 1999, 130:995-1013 (two articles)

Goodman: AJPH 1988 Vol 78:1568-1574

Berkson: J Amer Stat Assoc 37 Vol. 219: 325-335

Altman & Blad: BMJ 1995 Vol.311:485

@ James

Thank you for your valuable comment.

Personally, I use the Confidence Interval coined with p-values in my validations, certainly with careful interpretation.

Thanks Jochen for your valuable comment - Certainly, one should not be using values obtained from statistical analyses blindly. Thanks for the tip on "evidence and p-values".

Here is another good resource written by Francis Dane, a relatively new book about evaluating the research thesis results!

http://www.sagepub.com/books/Book234312?subject=W00&productType=Books&course=Course82&sortBy=defaultPubDate%20desc&fs=1

Dear @Ziad, Thanks for the interesting question. It is very important in many scientific fields to write down the hypotheses and improve them or disprove them using some of the statistical evidence and p-value tests. However, this does not work with the research in any research, for example in Mathematics, these mathematical hypotheses can not be applied because Mathematics is exact science.

Thanks Ljubomir for this nice reference book!

Excellent point Abedallah! Certainly, sound discretion is left as a duty for the researcher.

Scientific evidence should be the working tool of any scientist.

Certainly Alexander!

@Behrouz

We use the following titles as references:

- Kothari C.R., Research Methodology Methods and Techniques, Second Edition, Vishwa Prakashan, 2006.

- Malhotra N.K., Basic Marketing Research: Integration of Social Media, Fourth Edition, Pearson, 2012.

- Hejase A.J. and Hejase H.J., Research Methods in Business, Second Edition, Philadelphia, PA, USA: Masadir Inc., 660 pp., 2013.

- Bendat and Piersol. Random Data: Analysis and Measurement Procedures, Wiley Interscience, Fourth Edition, 2010.

- Cooper D.R. and Schindler P.S., Business Research Methods, Eight Edition, Tata McGraw-Hill Co. Ltd., 2006.

- Thill J.V. and Bovée C.L., Excellence in Business Communications, Seventh Edition, Prentice Hall, 2006.

- Hacker D. and Fister B., A Writer’s Reference, Sixth Edition, Bedford/St. Martin’s.

- Lunsford A., Easy Writer, Third Edition, Bedford/St. Martin’s.

- Bassham G., Irwin W., Nardone H., Wallace J.M., Critical Thinking, Second Edition, McGraw Hill, Boston, Toronto, 2004.

Another gooe resource:

Gomm, R., Needham, G., & Bullman, A. (Eds.). (2000). Evaluating research in health and social care. Sage.

Dear Ziad, in the Faculty of Business and Economics, when conducting research we certainly have to plan for the scientific evidence as well manifested by reliability tests and validity tests, and then results are also validated with reported research. Moreover, error analysis is a must to test statistical significance and the right choice of variables and research models. I may guess that if a faculty is not using such tests is really missing a lot and will get surprised if he/she publishes the results.

Hi all, let me share with you my latest experience of a "reviewer" not understanding the reliability and validity tests reported in a paper presented for publication! In a paper I presented to a journal for peer review, and after I got the reviewers' comments, one of several reviewers was having second thoughts about my data "even claiming it is created!". I had to run a whole series of statistical tests (in my answer to him/her) showing all the steps in detail covering both "parametric and non-parametric techniques where applicable" to prove to him the final model suggested by my research team. The write up of the answer to this individual amounted to 13 pages, which although took my time, but taught him/her how to do scientific validation and reliability assessment.

I had fun teaching him/her. He/she was one of 5 peer reviewers team for my paper. Surely, I was successful with all the evidence and following objective, scientific methodology.

Thanks Dear Hussin for sharing with us your valuable thoughts and interesting experience.

Sorry, there is sometimes a temptation to fit the results of the study to

concepts adopted by supervisor.

Dear Ziad

I recently submitted my thesis for examination. My literature review summarised the evidence specific to three studies that comprised my thesis. One examiner commented that I should have gone back to the original studies and not relied on other peoples summations. I thought that providing evidence would strenghten your thesis but it appears in my case it did not.

Dear Peta,

In principle, you are right.

Since a number of information sources is so large, in many cases each source cannot be reviewed. Using general reviews which were written by experts, is a good practice. However, often supervisors are pleased to see their cited articles (or articles of their teachers and scientific friends) in the references of your thesis.

Unfortunately, "political ambitions" often dominate.

Dear Peta:

Using validated results from previous original studies is a good strategy to strengthen the validity of your study. Care should be made, however, not to report what others have reported on original studies, since there is a chance that the reported reports might miss or misinterpret some important points. Thus, if you have access to the original studies do so and use the originals.

Be careful when you want to summarize the evidence from other studies, those referenced studies have to be representatives of the norm and not the exception!

Now, to avoid conflicts with your reviewers/examiners it is best to work with them ahead of time - perhaps in weekly meetings - to get a perception on what they favor and what they don't.

I hope that this helps!

Regards.

Strongly agree with Ziad, the supervisor has a fundamental role to draw a clear direction to the research and how to build the theoretical foundations. many advisers like to see original papers within the literature review specially if it deals with fundamental principles and not the results.

When I read an article, I look at the experimental set up and analysis first. If the authors performed a poor analysis of their data, then I dismiss the article immediately.

I wish that scientists would take a couple classes in statistics, in particular Design of Experiments 1 and 2. In these classes, I have learned about experimental designs that apply to all of the laboratory experiments one can do in chemistry, physics and biology. However, if I used these methods in a lab, the PI's would be mad because they "violate" their assumptions of what is possible. Most scientists think you cannot change more than one thing at a time during an experiment because you can't tell what had the effect. Yet, in High School, we all had to solve systems of equations like:

X +Y + Z = 6

2X -Y +3Z = 6

-X +2Y -Z = 0

I changed everything in all three equations. Yet, I can still figure out that the answer is 1, 2, 3. If I told my HS math teacher, "You can't solve this equation because you changes to many things." he would have failed me.

I would be much happier if scientists:

1) Used the power of a test rather than a P-value of confidence interval.

2) Allowed the data to speak rather than tell the data what is will show. (i.e. if your results, from a powerful test, do not confirm your theory or preconceived model, you are wrong. Not your data.)

3) Used the proper designs and analysis for their data.

4) Made inferences based upon their data and not generalizing their findings outside of their small study.

Far too often I have seen PI's yell at statisticians because the data shows something different than what they wanted. For example, the whole in the ozone layer was found in the mid 70's I believe. Since the scientists involved in the study decided an Ozone conc below a certain value was garbage, they removed it from the analysis. It wasn't until the 80's when statisticians realized that 30%-50% of the data was rejected, that the ozone whole was real and seasonal.

For this reason, If I get results that do not conform to what is "known", or contradict what others have said or the theory behind something, I look at that as an new area of research. If I can't explain why my data says something happened, I turn my attention to that area. Since I use powerful tests in my experiments, I can be fairly certain it really happened and is not a fluke.

Thank you Andrew for your elaborate and interesting comment!

Outcome in the form of a quality publication in a good journal or a patent or a novel theory or practical tool / technique for the benefit of science and / mass people - either at basic level or applied aspects