If you believe that your data is showing equivalence between groups then you should run a test of equivalence. The problem with non-significant results is that they only mean you cannot rule out the null-hypothesis, they do not provide evidence to accept it.

It is surprisingly hard to prove that two or more distributions can be considered equivalent and can often require much higher sample numbers than needed for detecting a moderate difference.

Article Understanding Equivalence and Noninferiority Testing

http://support.minitab.com/en-us/minitab/17/topic-library/basic-statistics-and-graphs/hypothesis-tests/tests-of-means/why-use-an-equivalence-test/

Are you analysing Likert items separately? I mean scores from 1-5 ?

Thanks James! Your reply me a different insight that I never thought of before. I am SPSS user and I am struggling finding a related example. Most of the examples are for other softwares.

Mehmet, Perceptions are measured using compound scale with items ranging from 4 to 7.

So you add (or do some algebra) some single Likert items to find final scores for perceptions. Right?

Your data show small means in a range of 3-3.4 that's why I asked if you used single Likert items to calculate means etc.

Your given data is overall so one cannot interpret. How are means by student groups?

Maybe the students did not give serious answers.

Mehmet, Thanks for your comment.

I used mean to computer scores for perception variables. Unlike most of the cases where responses are concentrated at middle point (neutral/ Neither Agree Nor Disagree) the data set shows bimodal characteristic (that is concentration is on the edges). Perhaps that is the reason I am getting lower mean scores. At the same time, it, kind of, also suggests that students were not completely careless in responding. Nonetheless, this is basic limitation in survey studies as you always get biased results, one way or another.

As far as means by students groups are concerned, had there been any difference in means based upon student groups (gender, age, educational discipline, course grades, overall grades), t-test and ANOVA would have shown it. The non-significant results from the aforementioned tests suggests that the means of student groups are not much different.

However, here directions suggested by James are valuable. We can only reject Null Hypothesis (that student groups are different) using the aforementioned tests, but the same may not be true when the null hypothesis cannot be rejected. We simply cannot deduce that the groups are similar. We need other ways (as suggested by James) to analyze such dataset. Perhaps t-test and ANOVA cannot be used to suggest that groups are similar.

If I interpret correctly, you have a sample size issue. There are four disciplines, and two sexes for a total of 8 treatments. With 115 surveys, that means that you have about 14 or 15 surveys from each group (if evenly distributed). In survey work, this is a really small sample size. If less evenly distributed, then you might have only a few surveys in select groups.

Arguing that the null hypothesis is true is difficult. In part the reason is that it is easy to get this result through bad experimental design, poor laboratory methods, and generally bad scientific method. On the other hand, this outcome is just as useful as "statistically significant" results if one can reasonably rule out bad science. Step 1: Look through the literature where survey methods are used in situations like yours. Make a histogram of the sample size used in each study. You will need to distinguish between the planned sample size (what the authors set out to do), and the actual sample size (what they actually got). You need to have a sample size equal to or greater than the average. Step 2: address any common issues in survey methodology and how your design/approach addressed these issues. This includes relevant transformations and using non-parametric methods for data analysis. Setp 3: If you analyze 100 variables (say 100 preference questions using a 5-point scale) individually (one test for each question), one should expect that the p-values will be uniformly distributed with values ranging from 0 to 1 if the null hypothesis is true. When you plot a histogram of the p-values is this a reasonable conclusion. A roughly equivalent statement is that the number of variables with p-values less than 0.05 equals the number of variables with p-values above 0.95. Another question: given that I have 100 variables, how many statistically significant outcomes should I get by chance alone if the null hypothesis were true?

Ali, as far as I understand you calculated means of some Likert items to calculate perception scores which made them smaller and similar. I think that is also one of the reasons statistical tests cannot find a difference among groups. This is my point.

In SPSS, only from the descriptive statistics, we cannot formulate any conclusion as sometimes there may be difference between mean values of two mutually exclusive groups of respondents but may not be significant. Some time the difference may small but it may be significant. Thus, we need to use the compare mean t-test.

Equivalence in response may be because of some reasons like Students usually do not give natural responses as they know what the researcher wish. Secondly, respondents may be at the same place while providing response. Third, might be only few respondents may have given the response of others too. Fourth, wrong sample selection may also arise such results. Mostly it happens if the information is collected through online survey.

Comparing two bimodal distributions: some thoughts, but these are guesses at things I might try. You could try comparing standard deviations, or skewness values using a randomization or permutation test.

Standard deviations are significantly different if the distance between the two peaks in the bimodal distribution are further apart in one group.

Skewness values are significantly different if there are a few more negative reactions in one group than the other.

You could break the problem in half: is the value of people that "like" or "love" the idea higher in one group or the other. Then ask if the value of people that "dislike" or "despise" the idea lower in one group or the other.

A discriminant analysis: This is a case where you have data from members of several groups. You know what the groups are and you have taken a bunch of measurements (in the form of questions) from these members. You now want to know how well your data identifies/explains group differences.

Rather than t-test (and associated assumptions that the data are normally distributed), a rank based test might work. Kruskal-Wallis or (in SAS) Ryan-Einot-Gabriel-Welsch (REGWQ). The former see: http://www.biostathandbook.com/kruskalwallis.html. The latter see: http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glm_a0000000867.htm.

Thank you Mehmet, Prakash and especially Timothy for your comments.

Mehmet, I agree with you, however, this is how latent variables are computed from the items. :)

Prakash; you have rightly mentioned the issues related to survey studies. We just cannot escape from them in most of the survey designs. We can minimize them though.

Timothy, both of your comments are really enlightening. I must say I knew the data was not normally distributed and yet I stuck to parametric tests - my bad. Should have gone for non-parametric tests. I really like your suggestion of using SD and Skweness values on comparing two bimodel distributions.

Let me run them and see what it turns out to be. :)

Once again, thank you so much everyone.

BR

Ali

Ali, can you add (or do some other algebra) single item scores to get a perception score?

Mehmet, Latent variables are computed by taking average of the items scores. Even if we use other arithmetic functions (For example, summing the scores) I don't think it will change any think as variance will be same, be it mean score or sum score. I can be mistaken here but that is my understanding.

@Ali, your definition is too restrictive by focusing on the mean: https://en.wikipedia.org/wiki/Latent_variable. Latent variables are very useful, but they should be driven by some underlying theory, or developed through multivariate dimension reducing methods (factor analysis or discriminant analysis are two options). You can spend a great deal of time trying random combinations to find significant relationships, but you will have trouble justifying that any significant outcome represents reality rather than a product of chance.

@Timothy, yes, you are absolutely right and I have been naive while defining latent variables. The latent variables should be driven by some underlaying theory or through multivariate dimension reduction method.

And, thank you once again, for refreshing my memory and showing another way to play with my dataset. :)

I've done factor analysis on essentially random variables to test their similarity, by using pseudorandom-number generators in SAS. And yes, you do get a few variables loading highly on the same factor axis, but typically much less than for the biophysical databases that I work with. Any correlations between variables that one finds are best tested with another data set (e.g., different sites or years), to see if the associations are consistent and thus robust.

-Bob Vadas, Jr.

Oops, I've generated pseudorandom numbers for past work, but actually used a random-numbers table for the factor-analysis work that I mentioned.

More what I envisioned is that one goes through a process in data analysis of increasing desperation.

Try 1) I try combining X1 through X9 by simple addition. I tried var1=X1+X2+x3, and var1=X1+x7+X8+X9. I also made up a bunch of other vars that were similar. That did not give the answer that I wanted.

Try 2) I try multiplication: X1*X4, X2*X3*X7, and X4*X5*X6*X8*X9*X10

Try 3) I try ratios

Try 4) I try random transformations (inverse, exponent, roots, log)

Try 5) I am now desperate, so I start with creative functions like var1=log(X1)^cos(X2) and var2=(X1+X2)/(X3*X4)^(1/X5). That last one was close to significant, so maybe I could just tweek it a little. I'll develop the theory to rationalize the significant breakthrough later.

A never ending fishing expidition trying to find latent variables that provide significant models. This is in contrast to something like factor analysis or using existing theory to define latent variables. As Bob suggested, the significant results should be tested with another data set, and should be a suggestion for every research project with novel results.