It's possible that the lack of response from authors could stem from various reasons, such as time constraints or concerns about data privacy and confidentiality. While openness is indeed valued in the scientific community, there can be legitimate concerns about sharing raw data, especially if it involves sensitive or proprietary information.

You might have better luck finding openly available datasets from reputable sources like research institutions, government agencies, or academic repositories. These datasets often come with clear documentation and could provide the diversity you seek for your book and software.

Once the results of a study are published, the corresponding anonymized (de-identified) data sets should ideally be made available to the public. However, this often does not happen, presumably because (1) researchers often attempt to publish more than one article from a given data set (and want to prevent others from using their data to publish their own articles) and (2) there may be IRB concerns over making data sets openly available without formal IRB approval.

At least some of these concerns are understandable. After all, the researchers spent time and money collecting the data; they don't want others to benefit "for free." They also don't want to get into trouble with the IRB for sharing their data.

Yazen Alawaideh I've consulted some of the few public databases, but they are mostly demographic or weather data, and I need other kinds too.

Christian Geiser Yes, that's understandable sometimes, but often the articles were published more than 5 years ago, so I suppose they have finished their work with the data. Or, often people are asking questions about what to do with their data, and when I offer help and ask them to have a look at the data to understand what's going on, they are suddenly silent... I find that kind of weird.

Christian Gaiser, yes, that makes sense sometimes, but often the articles were published more than 5 years ago, so I assume they have finished their work with the data. Or, people often ask questions about what to do with their data, and when I offer to help and ask them to take a look at the data to understand what's going on, they suddenly go silent... I find this kind of odd."

https://github.com/snwikaij/Data

Christian Geiser , you wrote that "After all, the researchers spent time and money collecting the data; they don't want others to benefit "for free.""

They have a publication, don't they? So they did earn their pay.

Often researchers are paid by the public, they work in public universities, they serve the public. Public money is spent in their institution, their salary, their instruments, their consumables, and their publications (many journals take money!). And after all, the data - the hard stuff that counts most - are kept secret. Even when the publication is in journals that require that the data are made available (as per their Instructions for Authors).

Of course, some human-related (typically medical patient) data may not be made fully available for data protection reasons, if the data would allow for an nearly unambiguous identification of individuals from the combination of subject-related data given (I think this is what you meant with possible IRB concerns). But this applies only for (some) clinical studies where a lot of variables are used or for the combination of extremely rare features.

In my opinion, publication includes publication of data, not just arbitrary summaries and conclusions drawn from it. When the data are not available, then it should not be regarded as a scientific publication. We (the scientific community) should set the bar as high as reasonably achievable, and today this means that the data should be available (ideally following the FAIR principles [https://en.wikipedia.org/wiki/FAIR_data]), either via the supplement or via some dedicated public database.

I have also seen papers where the authors claimed that the data are published because the individual data points were shown in scatter plots. This is not publishing data; the data shown in scatter plots is extremely hard to use, often data points are overlapping, and multivariate relationships can not be recovered at all.

Jochen Wilhelm Yes, this would be ideal (publication of the actual data along with the paper(s)), but the reality looks different. Often, research groups/labs including their doctoral students and postdocs continue to publish out of the same large data bases for years and thus have no interest in making their data sets openly available to the rest of the world. Designing studies and collecting data does cost a lot of time and resources. I can understand that they do not want to give away their data just like that.

I'm not saying this is good but it's a reality. Unless journals start to reject papers that are submitted without the corresponding data sets, I don't think this will change anytime soon.

Creating a database with high-quality data is not trivial and may require many person-years of hard work. Creating such a database and publishing it is itself a valuable scientific contribution. Using these data to generate some hypotheses is comparably simple and publishing such ideas is a scientific contribution as well.

However, I wonder how data from a carefully designed experiment can be used to fuel several publications. Re-using such data for testing independent hypotheses would also be fraud. It's particularly the experimental data that should be published, and no publication of experimental studies with a missing free reference to the underlying data (presented at least in a readable and interpretable format) should be accepted.

For observational studies based on some (unpublished) database, the editor should decide and a statement with reasons for the no-disclosure of the data from the editor should be obligatory.

Jochen Wilhelm Often, observational studies in my field (psychology) contain a huge number of variables, particularly longitudinal studies. For example, there are longitudinal studies in developmental and clinical psychology that look at problem behaviors (etc.) in children or adolescents as rated by multiple reporters (e.g., self, peers, parents, teachers) as well as antecedents, mediators, covariates, and outcome variables. Publishing all aspects and addressing all hypotheses of such a complex study in a single paper is often impossible. As a result, multiple papers are published.

Also, the time and effort that goes into collecting complex data sets like this would not be warranted for just a single paper as long as the scientific community (including funding agencies) counts the number of papers as evidence of scientific productivity. Again, I'm not saying this is a good thing, but it is a reality. When people get funding for studies like this, the funding agencies expect them to publish multiple papers. As a result, people will not share their data bases before they are done publishing out of it.

One solution could be to require publication of partial data bases that contain only the variables that are used in a given paper. That way, researchers would not have access to the other variables in the data sets that the PIs of the studies are using for other papers.

Thank you Christian Geiser for your clarification! I agree that the part of data relevant for the paper should be published as part or (or along with) the paper. I never meant that other data that happens to be part of the same database should be published before it has been used and analyzed in sufficient detail. And yet it might be that the relevant part of the data is not logically extractable, in which case I would certainly understand that the data cannot be published. This is a case that could (should!) be explained by the authors in the "data availability" statement and checked by the editor.

I think a problem that contributes to this poor practice is how researchers are currently rewarded. Currently, this mainly happens by considering their publication record. However, if they would be rewarded by considering how often their dataset has been used by others, I guess this would contribute to making their data openly available.

Overall, a cultural change in our scientific system needs to take place (and for instance in the Netherlands this is currently happening). In particular, we need to move towards open science which values the whole research process and not only the publication at the end...Yes, this requires a lot of work (change in the mindset, infrastructure, awareness, etc). I think we should start with the training of our future academics and of course try to be a role model...

I think it is not necessary when each researcher has other

evidences such as pictures, in vitro and in vivo measurements

, etc to reveal the fact reality of experimental result.

Angel De Souza-Torres Another researcher might obtain different conclusions from the same data. That is the point. It can only be interesting for both parties if people share their data, unless you want to hide something, of course...

many researchers have come to want ready-made data without any effort, and scientific theft has increased, and therefore the reason for not publishing the data is to protect from hackers, some may manipulate and distort the data and thus misuse it.

Sharing is all about one word "Money", in one way or another directly where it is either lost or earned. The old whitepapers shared were during a time of greater moral discipline where theft of intellectual property wasn't a concern. Today, that is not the case where knowledge is the only currency that still retains its value. In the highly competitive marketplace where wisdom gives way to commercialization of that knowledge, we all suffer from the deficit of moral bankruptcy perpetrated by the proverbial few, who ruin the opportunity for the greater majority to benefit from hard work shared.

Researchers may be reluctant to share their data for several reasons:

Chuck A Arize Yes, but they might also have benefits from sharing. Another researcher might find new interesting things in the data.

Researchers may be reluctant to share their data due to concerns about privacy, intellectual property rights, potential misuse, or the effort required to prepare and document the data for sharing.