You may regard one sample as "prior sample" and the other as "current sample". If so regarded, you may combine the two samples using the least-squares method or the law of combination of distribution. This paper may help: Article Two simple and practical methods for combining prior informa...

do you want to test the means of the two datasets? then a t test will be applicable, with the assumption that their variances are the same.

what is your end goal?.

Since the survey was the same you could just merge the two data sets. Best wishes, David Booth

I respectfully disagree with David. It really depends on the comparability of the two sources. These could be very different locations, or the surveys could be from different times, and such factors may systematically affect the responses given in the survey. Actually, each survey location and time (and other factors) should be well considered so that the results will generalize to the intended population. This does not need to be identical for the two surveys. Putting them together will in principle improve the generalizability, but this may be at the cost of precision and also of interpretability, particularily when only few (here: just 2) surveys are combined.

I'd say: yes, it is possible to combine them, but one should include the factor "survey" in the model to account for systematic differences between the survey.

Vikas Ramachandra Thanks for your reply! I aim to use t-test to justify if the two datasets are comarable to jusitfy if it is not problematic to merge them into one dataset. I did conduct t-tests for some demographic variables such as gender, age, years of leadership experience, position, department and industry. However, the first three had significant differences but not the last three. I am now confused about what to conclude?

You trapped into the "absecne of evidence is not evidence of absence" fallacy. Confusion is a common symptom of this.

Jochen Wilhelm Thanks for your input! I do have the 'source of data' dummy variable as a convtrol variable in my model, but I still need to use the t-test to justify if they are comparable in the first place, any idea how I can use t-test to achieve this aim please?

Sorry, you simply cannot use the t-test for this purpose.

You could possibly define a region of negible difference (don't ask me how!) and check if the confidence interval for the difference is entierly inside this region.

The exact sample test can be applied to large sample also though the converse is not true. So that your bigger size (no of elements in addition of two data sets) may greater than (less than) 30 can apply in large (exact) sample test. First, verify there is any significant different between means of samples using t - test. if not, you can merge. This means that these two data are acceptable for consideration as similar.

Dr C. Senthil Murugan Thank you very much for your advice! It really helps! Combining the samples is a must for me, since the samples are siginificantly different on several demographic variables, I justify this decision by controlling those significant variables for the combined samples at the end

It seems that you only listen to what you want to hear. So be it.

+10 for Jochen Wilhelm . As a reviewer, editor, and someone on dissertation committee this issue is one that often leads to rejection of manuscripts/dissertations.

Dare I say once again, PROPER PRIOR PLANNING PREVENTS POOR PERFORMANCE

@ JOCHEN @ DANIEL you are of course correct but I am in salvage mode today. Best wishes, David

According to your replay, I remember a similar situation, (i.e.,) the government make survey for economic status of people in rural, semi urban and urban... etc the samples drawn from these area are significant differents. Finally, we combined those samples for forming final decision of economic status of people.

I thing, this procedure for finding decision from these samples by the government, it may usefull for you.

If possible please try.

Thank you

David Eugene Booth With all due respect, there's no salvage in a sloppy and superficial approach. We know nothing about datasets and simply combining them can introduce serious bias. Assuming Shing Kwan Tam is not doing this just for fun I don't see how the wrong conclusion may provide salvage for Shing and people who may depend on her results and consider them reliable. I understand Shing Kwan Tam may be in an inconvenient position being a young scientist at the beginning of her career and possibly asked to perform an impossible task, but this is exactly the time and place when we should learn that bad research is worse than no research. As Doug Altman said "Why are errors so common? Put simply, much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill-equipped to perform, and nobody stops them. (10.1136/bmj.308.6924.283)". Stopping young scientists may seem excessive, but to educate them why this is wrong (as Jochen Wilhelm did) should be a bare minimum from an expert who decides to provide his/her opinion.

Shing Kwan Tam Are both datasets equally appropriate for what you are trying to do? Do they represent the same population? What was the sampling strategy each time? How was the data collected? No simple test can tell you this, you have to know your data first. If you are absolutely confident you didn't have methodological flaws, the differences could be due to chance. However, if there are no obvious differences this still doesn't mean combining two datasets is a justified approach.

Imagine you wanted to gather information regarding young people eating habits and first conducted your survey in McDonald's. For some reason, you somehow decided you don't have enough participants, however this time you conducted your survey at the stadium of your local Track & Field team (let's say this seemed appropriate and easy at the moment). Both the folks from McDonald's and the athletes may have the same demographic variables, they may be the same age, sex may be equally distributed, they may even go to the same school... yet you have two very different populations.... no result of any test can save you from this and without a good thorough thought experiment on how combining the datasets could generate bias you simply cannot "justify" combining them ("I need a bigger N" is not a valid excuse if it comes at the expense of drawing erroneous conclusions).

I would think of all the possible reasons that may affect your conclusions and then decide whether combining the datasets is possible at all. If they really come from "the same population", you see no way bias may creep in, and you decide to combine the datasets, I'd still be as transparent as possible about this, enlist all possible limitations, definitely keep the "survey" column, and provide raw data.

Best of luck.

Jan

@jan, all I said was she COULD merge the sets. Please don't think I am saying that Covid-19 is a hoax or the equivalent for this question. I'm really not anti science or anything like it. I'll hunker down in my foxhole and stay safe from the barrage.. May you have a great day and keep your powder dry. Best wishes, David Booth

Hi all, thanks a lot for all your inputs! I really appreciate it!

Jan Homolak Jochen Wilhelm yes the decision of using this dataset is not what was planned, and the planning was not perfect at that point because it was the first pilot study I ran. However, same sampling strategy and criteria were used for participants recruitment. Two sources of participants were chosen mainly because of the practical budget and resources issues. As Jochen Wilhelm said, this 'survey' factor does exist in my model to control for the possible biases (thanks for pointing it out!). The t-test is just one of the tests I conducted for my mentioned purpose, as a common practice, there are other tests conducted to justify my decision that it does not cause fatal errors and biases to my result. This t-test method is a quite common technique used in my field. Researchers use t-test to check whether the key variables related to the model have significant differences across multiple samples. It is regarded as the first step to justify if the samples are comparable. If ALL variables are significantly different, it is a definite 'no' for merging the datasets.

Daniel Wright I certainly understand what you meant as my supervisors hold the same views as yours, I can only make the best use of what I have on hand in view of the limitations. No research is perfect, that's why we need to know our data, decisions and research well and address the possible limitations. It's quite true that more works are needed to get to the point where this paper can be accepted for publication.

I think for all of us who went throught PhD journeys would understand that a finished imperfect PhD thesis is better than a hope-to-be-perfect but unfinished thesis. Regardless, all your comments are well-received! Handling different opinions is just part of our lives. Have a great weekend everyone!

A t-test cannot confirm that two samples come from the same population (i.e., the null hypothesis).