Our relentless focus on the importance of big data is often misleading. Yes, in some situations, deriving value from data requires having an immense amount of that data. On the other side the key for innovators across enterprise organization is that the size of the data isn’t the most critical factor — having the right data is. Do agree or disagree? Let's have a discussion!
Data science is based on Gaussian (normal) distribution assumptions, e.g. for online marketing, advertising, tracking. Advanced statistical learning methods try to unveil non-deterministic, non-linear and complex bodies of data information, i.e. 'mathematical hermeneutics' of 'chaos' and 'fractality'. It all depends on the right level of interpretation and application.
https://or-zse.hu/hacofe/vol10/erdospal2015.htm
It depends who wants the data:) If the numbers are good or politically right, if everybody is happy with what they see on the bar charts, they don't need to be right. On the other the bigger the data the harder to accept it could be wrong. Because decisions were made on it and it is simply not easy to confess collosal mistakes were made based on bad data (think of state statistics for example). Nobody likes right data:)
Hi Mohammed,
this is a very interesting question. I am by no means an expert but as far as I know, data scientists are indeed interested in data quality and invest a lot of time in screening and cleaning. On the other hand, I have the impression that cleaning is more focused on deleting crap and bringing variables (or "features" as they call it), into a format or shape that fits the assumptions of the analytical method or makes the algorithm easier to converge. In contrast, I don't see that questions about "the meaning of the data", hence, its validity are ever asked. To the contrary, I am often shocked how mindless transformations (e.g., the good old logarithm debate) or rescalings are done without thinking about whether this makes sense from a conceptual point. And the reason for this (again: my subjective perception) that data science completely operates on the observed data level and not--such as traditional scientists--on a conceptual / theoretical level. That's why concepts such as causality only slowly spread in these fields (causality lies beyond the data).
As scientists (unless you have a positivist stance), we see data rather as a mere reflection of some observed or unobserved reality and we are interested not in data patterns per se but on pattern of effects within the reality. From such a perspective, issues such as systematic measurement errors, conceptual gaps and validity add to the issue of poor data quality.
See some papers on this below.
Best,
Holger
- - - - - - - -
Athey, S. (2017). Beyond prediction: Using big data for policy problems. Science, 355(6324), 483-485. doi:10.1126/science.aal4321
Braun, M. T., Kuljanin, G., & DeShon, R. P. (2017). Special considerations for the acquisition and wrangling of big data. Organizational Research Methods, 21(3), 633-659. doi:10.1177/1094428117690235
Clark, W. R., & Golder, M. (2015). Big data, causal inference, and formal theory: Contradictory trends in political science?: Introduction. PS: Political Science & Politics, 48(1), 65-70. doi:10.1017/S1049096514001759
Fan, J., Han, F., & Liu, H. (2014). Challenges of big data analysis. National Science Review, 1(2), 293-314. doi:10.1093/nsr/nwt032
Foster, I., Ghani, R., Jarmin, R. S., Kreuter, F., & Lane, J. (2016). Big data and social science: A practical guide to methods and tools: Chapman and Hall/CRC.
Saha, B., & Srivastava, D. (2014). Data quality: The other face of big data. Paper presented at the Data Engineering (ICDE), 2014 IEEE 30th International Conference on.
Salganik, M. J. (2017). Bit by bit: Social research in the digital age: Princeton University Press.
Wickham, H., & Grolemund, G. (2016). R for data science: Import, tidy, transform, visualize, and model data. Beijing: O'Reilly Media, Inc.
Of course, the right data is important for scientific research. However, in the present era of digitalization of various categories of information and creating various libraries, databases, constantly expanding large data sets stored in database systems, data warehouses and Big Data database systems, it is important to develop techniques and tools for filtering large data sets in those databases data to filter out of terabytes of data only information that is currently needed for the purpose of conducted scientific research in a given field of knowledge, for the purposes of obtaining answers to a given research question and for business needs, eg after connecting these databases to Business Intelligence analytical platforms.
Below are some issues related to Big Data database technologies that can be developed scientifically:
- application of data processing technology in Big Data database systems for modern education 4.0,
- improvement of forecasting of natural, climatic, economic, economic, financial, social etc. phenomena based on analyzing large data sets,
- analysis of sentiment, opinions of citizens, Internet users regarding brand recognition of companies, customer reviews of specific services and products, views on various topics, citizens' worldview based on the analysis of large collections of information downloaded from various websites, from comments downloaded from social media portals,
- analysis of information and marketing services of commercially operating companies that carry out specific analyzes of sentiment, citizens' opinions, Internet users regarding brand recognition, customer reviews of specific services and products etc. on behalf of other companies that purchase specific analytical reports,
- analysis of the possibilities of cooperation, synergy, correlation, conducting interdisciplinary research, connecting Big Data database systems with other information technologies typical for the development of the current fourth technological revolution called Industry 4.0, which include technologies such as: cloud computing, machine learning, Internet of Things, Artificial Intelligence, etc.
It seems to me that the analysis of large collections of information collected in Big Data database systems will have a significant impact on the development of entrepreneurship research.
First of all, the analysis of large data sets Big Data will improve data analytics on cloud computing platforms, the use of computerized Business Intelligence applications as supporting business management processes.
In the context of the above considerations, the following question is also current:
More and more technology and marketing companies acquire information from entries, posts, comments posted by Internet users, including users of social media portals.
The information collected from these sources from these websites is aggregated in Big Data database systems and processed mainly for the purpose of their use in the marketing activity of a given technology company or for the needs of other companies that are its clients.
In the context of the information downloaded in this way and their use for commercial purposes, a question arises regarding the ethics of this activity.
In my opinion to ethically use the discussions, content of entries, posts posted on social media portals, it is necessary to obtain and process this scientific data using the principle of anonymity of aggregation and data processing for the needs of research, use an objective research approach, comply with applicable ethics standards of research and in accordance with applicable law.
In addition, I note the interesting discussion inspired me to the following considerations:
Business entity management processes are more and more often supported by computerized Business Intelligence platforms that facilitate multi-criteria analysis and reporting.
Probably in the future, the business analyst will be supported by artificial intelligence.
It would be a great advance in the field of automation and objectification of multi-criteria economic analyzes of business entities.
Complex, multi-criteria analyzes regarding the verification of large companies' operations require aggregation and analytical processing of large data sets in Big Data database systems.
However, in what direction will technological progress be realized in this field?
In the future, as part of the progressing computerization of analytical processes, it will be possible to implement artificial intelligence to the processes of analyzing large collections of information collected in Big Data database systems.
In view of the above, it is also important to include in the above discussion also the following issue:
The Big Data database technology is finding more and more applications in business.
Multi-criteria processing of huge data sets collected in Big Data database systems allows preparing reports in a relatively short time according to given criteria.
The report development time depends mainly on the computing power of Big Data servers.
In processes of complex economic and financial analyzes, risk management, etc. for the purpose of determining the economic and financial situation of business entities, they are increasingly carried out in computerized analytical platforms of the Business Intelligence type.
Perhaps in the future, artificial intelligence will also be involved in this field of analytics.
In some countries, IT companies have been operating for several years, developing the Big Data database technology for commercial and business purposes.
It is only a matter of time to combine these various analytical and database technologies in computing cloud computing.
In view of the above, the current question is: In which sectors of the economy, in which types of companies and corporations will be the most dynamically developed technologies for analyzing large collections of information in Big Data database systems.
Data science is based on Gaussian (normal) distribution assumptions, e.g. for online marketing, advertising, tracking. Advanced statistical learning methods try to unveil non-deterministic, non-linear and complex bodies of data information, i.e. 'mathematical hermeneutics' of 'chaos' and 'fractality'. It all depends on the right level of interpretation and application.
https://or-zse.hu/hacofe/vol10/erdospal2015.htm
The above discussion inspired me to the following considerations:
An important issue is also the issue of the development of databases on Internet users carried out in Big Data database systems by the largest providers of Internet and information services, by technological internet companies that collect data about web-based users, including social media portals. The scope of data regarding Internet users and the use of these data for marketing purposes is a controversial issue.
Online technology companies that run social media portals such as Facebook, YouTube, WhatsApp, Tweeter, LinkedIn, Instagram, Google+, Snapchat, etc. collect large amounts of data about Internet users from various websites, including mainly users of these social media portals. The Google group collects this type of information about Internet users based on Google web search service and other shared information services, including Google Drive, Gmail, etc. Information on Internet users is collected, among others, by the so-called cookies installed in internet browsers.
The second one (the right data). The problem, however, is how someone can be sure for the collection of the right data? The basic idea matters.
Regards
In addition, large internet technology companies have the ability to combine information downloaded from various Internet sources, from various information services into one avatar, which is an electronic equivalent of a given Internet user and is built from Big Data database systems. For the needs of these considerations, the issue of security of classified data stored in Big Data database systems run by internet technology companies managing social media portals should be added.
There have been frequent thefts made by cybercriminals who, from these Big Data systems, stole classified and sensitive information about hundreds of thousands or millions of users of particular social media portals. The development of social media portals therefore generates an increased risk of theft of data on users of social media portals.
On the other hand, technological internet companies such as Google, Facebook, etc. have the ability to obtain classified, sensitive and personal data from the Internet about the users of websites, including search engines and social media portals who use these information services. These companies in their Big Data database systems are gradually expanding the scope of data related to Internet users and are constantly expanding the avatars mentioned above, ie electronic equivalents of individual Internet users. Therefore, the colloquial wording was created: "If you are not on the Internet, you do not exist."
There is one more issue, namely the above-mentioned activity of internet technology companies expanding their information resources Big Data and orz contained in these resources avatars of individual Internet users, how does this correlate with the legal protection of personal data in individual countries? Therefore, important questions to look for answers through research are as follows: what is the scope of information from which avatars of individual social media portal users are created, which are created and developed in Big Data database systems managed by internet technology companies that run these social portals the media?
In the context of the above considerations, the following question is also current:
Does the development of data processing technology in Big Data database systems and other technologies developed in the field of technological revolution Industry 4.0 generate new business management concepts?
At present, in the age of the technological revolution known as the 4.0 industry, new technology management or Internet-based companies are emerging.
The technological revolution in recent years, known as Industry 4.0, is motivated by the development of the following factors:
Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence.
Added to this is the development of data processing analytics in Business Intelligence enterprises, the development of life science technologies, biotechnology, eco-innovation, energy, medical intelligence, etc.
On the basis of the development of the new technological solutions mentioned in recent years, the processes of innovatively organized analyzes of large information collections gathered in Big Data database systems dynamically develop.
In the context of this problem, many questions arise:
What other technological improvements, innovative organizational, technical and IT solutions will be developed in the future based on the development of the above-mentioned factors?
What kinds of innovations will be able to be created in the coming years, in the future based on the integration of the above-mentioned main determinants of the development of the current technological revolution known as Industry 4.0?
What kind of new categories of added value may be created in the future if the above-mentioned technological solutions are more involved in the creation of biotechnological, ecological, product and other innovations.
Will new technologies be created in this way, with the help of which it will be possible to generate solutions to the problems of excessive exploitation of Earth resources in the process of civilization development?
Do business management processes play a particularly important role in the context of the effective functioning of business entities in currently developing economies based on knowledge, information and technology?
Is e-management a new concept of managing virtual enterprises or rather managing online technology companies?
Are new management concepts of innovative enterprises and start-ups based mainly on knowledge, information, entrepreneurship and creation of innovations?
Does the development of data processing technology in Big Data database systems and other technologies developed in the field of technological revolution Industry 4.0 generates the emergence of new business management concepts?
An important issue in the application of Big Data database systems is the application of these systems in business management support processes.
Large companies, corporations and financial institutions, including banks, are technologically and financially prepared to develop applications for analyzing large data sets collected in Big Data database systems in the process of managing individual categories of banking, financial, including credit, operational, market and IT systems. The main research issue is the study of potential clients, eg potential borrowers in the field of creditworthiness analysis by analyzing data collected in the Big Data database. In the Big Data database system, data on potential clients collected from the Internet from entries, comments, posts contained on social media portals can be collected. If the banks analyze the client's creditworthiness based on sensitive, personal and classified data downloaded from social media portals, then the ethics of this type of analytical process should be considered. This issue of ethics and personal data protection also arises when the insurance company verifies the health of a potential client who submitted an application for life insurance - based on what he writes about himself, he will report his status on social media portals. If the potential client was aware of the fact that he shares personal information with his friends on social media portals only with his friends and the bank, insurance company or other institution uses this secret without using the client's knowledge, then the problem of breaking the basic ethical principles in business arises.
In view of the above, I am asking you the following question: Are you already familiar with the use of Big Data data analysis in the risk management process in corporations and financial institutions?
In the context of the above considerations, the following question is also current:
What kind of research dominate in the analysis of data collected in Big Data database systems?
First of all, as part of the processing of information collected in the Big Data database systems, the sentiment analysis is used very frequently during analytical processes conducted for the needs of scientific research. Sentiment analysis allows you to get an answer to the question, what is, for example, recognizability, what is the awareness, opinion on a specific topic among users of specific Internet portals, websites containing comments from Internet users, and social media portals.
To what extent the Big Data data sentiment analysis from social media portals can be used in forecasting the company's development?
Large amounts of information downloaded from comments, entries, posts from social media portals are processed in Big Data database systems to determine, for example, consumers' awareness of the offer of products and services of specific companies. This type of information is of great importance for the planning of advertising campaigns informing about the mission, idea, product offer, and the usability features of the company's offer. This type of data may be important for forecasting the changing preferences of consumers regarding the offer of specific companies.
In the context of the above considerations, the following question is also current:
Does the technological progress in the field of advanced data processing contribute to the improvement of techniques for conducting forecasts of future processes?
Yes, if you have advanced research tools that help in carrying out prognostic analyzes. For example, tools that allow you to conduct sentiment analyzes on large collections of information downloaded from the Internet and collected in Big Data database systems. If this type of analytics merges with advanced information processing instruments on Business Intelligence analytical platforms, then it is possible to build accurate research tools that enable to carry out precisely completed prediction analyzes of predicted processes for subsequent periods in the future.
Dear Colleagues and Friends from RG
The key aspects and determinants of applications of data processing technologies in Big Data database systems are described in the following publications:
Article APPLICATION OF DATA BASE SYSTEMS BIG DATA AND BUSINESS INTEL...
Article The Big Data technologies as an important factor of electron...
Article The Technological Solutions Big Data and the Importance of B...
I invite you to discussion and cooperation.
Best wishes
Big Data may not in some cases provide the Right Data. However, it is inescapable: from an ocean of data, many factors can be mined. Just think about the theme of language. Only from a huge database can there be translation tools.
You are absolutely right, Mohammed. In ecology, for instance, big data can be associated with variation in sampling methods which can be quite misleading.
Big data requires huge servers processing ( and money as well) such as HPC or Public Cloud tarified with I/0. Only majors company can afford this kind of infra and even sometimes with very limited results considering the financial investments ( Capex/Opex). There is another way of processing named as Smart Data. The idea is only to process with eligible ( ie qualified data) to provide with more accurate results and to avoid the garbage in/garbage out effect
Thank you Olga Bykova, Roberto - Minadeo, Catherine Nohra China and all of you for your insights! Sometimes the right data is big. Occasionally, the right data is small. But for the enterprise organizations the determinant is figuring out what those critical pieces of data are that drive competitive position. Then that's a good target to shoot for. How we can achieve such competitive quality data with in any size of data?
Considering the importance of this topic in medical applications, GDPR is of utmost significance. I believe that in near future all of us would contribute with our personal medical data in order to improve the accuracy of predictions. Sometimes machine learning algorithm learn insignificant features of the set and in the generalization phase tries to search for that, which is the problem need be prevented. So, even when you have so called Big data, the sub-set of which is the right data, you need to be very careful of nuisance data in order to generalize well.
I think the answer to your question varies from situation to situation, but sometimes we do not know what is 'right'.
Hi @Holger,
You mentioned a great point, data cleaning. I agree that data cleaning takes the larger share in most data science project. Recently in my project it took us almost 80% of our total time spent in the project.
This is interesting!
Data need to be quality wether it is small or big it does not matter.
Why do we worry about the size of the data. I believe we need to make the quality first and very most. What you think Mohammed Kemal Milena Cukic Catherine Nohra China Olga Bykova Roberto - Minadeo Olga Bykova ??
I like your positioning. Of course, if I think, for example, of data for online translation of languages, the data needs to be accurate. Quality is key. But the volume of data for such a task is gigantic. Hence Big Data.
Hi Roberto - Minadeo ,
Thank you for your reply. I do agree with you that data with accuracy is quite important for online translation.
It depends on the area of the research. In social science, it's very difficult to predict the right data... Whether, the collected data is right or reliable, it depends on the reliability test. Most interesting thing, if you collect more data, the results of the reliability test show more reliable. But, according to Hair (2012) stated that if the sample size is more than 200, it's enough to carry out any kind of statistical analysis for social science.
Thanks
Experience is the key for the right data collection. I have found that the right data come out from both ways or from none, unless you have done it several times and you know how to discover the hidden right data through the proper window during the right data collection.
Theodore Costopoulos
thank you for sharing your insight. I agree that experience is key for having right data but how about using the right technology? Would that something that can help? Any thought? Theodore Costopoulos Mohammad Mizanur Rahman Dennis Mazur Siddappa Naragatti Khadijah Mohamed Lilianna Wojtynek Roberto - Minadeo Lubaba Yemer Mila Ilieva ?
If you're analyzing general trends that "were" or "are", big data is useful to an extent. If you want more predictive power and a better understanding of causality versus correlation, deep data with key respondents who are aware of the diversity (pending what question/hypothesis you're addressing) is far more powerful and accurate. I use combined big data and deep data (qualitative and quantitative). For example, despite being the only one at my think tank who stated it: I predicted Duterte's win, Brexit, and Trump's win within 1% (none of which were my preferred choices). My think tank colleagues thought I was insane. Deep data also much better for assessments and evaluations - again, choosing the right key respondents and sources of info. I used to run a big polling operation in Afghanistan (e.g., several thousand respondents for the elections), but the deep data (more anthropological) face to face discussions with key key respondents, looking at historical and archaeological data, environment and resource data, long-term demographic data, etc. approaches proved much more accurate with both qualitative and quantitative output; and far superior representation of diversity and nuances that big data just doesn't really capture.
Hi David Kyle Latinis
that sounds very interesting. You don't have any references handy that describe such form of "triangulation" (besides the usual lit on method triangulation)?
One problem is that those face-to-face interviews can only be done with a small group of respondents implying a huge unreliability, doesn't it? Think of a form of "qualitative standard error" which such an approach brings with it....I work in a field in which such interviews provide the basis for quantitative measurement, and the small N-problem always leads to problems (especially when to measure/notice rare events).
Best,
Holger
David Kyle Latinis
Thank you for sharing your insight in this important topic. I do agree that choosing the right key respondents and sources of info is vital for any big data operations.
Ahmed Y. Yassin
Thank you for your comment. Would you mind explaining why you for the right data instead of big data? Thank you for your time in-advance.
First, I agree with David Kyle Latinis about thinking. Second, it is very difficult to estimate (sometimes) your “good idea” is it really good? Small data can make big mistakes in conclusions, and consequently, wrong decisions in patients’ treatment. On the other side, recommendations are to make studies with big data. The most important (for me) is to have good statistician – for several reasons: to make good hypothesis, to make “blitz” – analysis during collecting data (especially if you are collecting huge number of patients, can interrupt further work if data don’t make further differences), and finally, to set the analysis on good way. I prefer big data, I have feeling that from biggest pool of numbers you have better and exactly results. One of the biggest cheats is pre-statistical analysis, by which you can make study with only a few patients (???) and make conclusions? By it, you can make only case reports and/or case series, but you can not make trends and conclusions. Big series is answer.
Big data or right data, it depends on the use case. For instance if you want to make predictive monitoring for IoT and connected objects, you need to train the model with right data and even the right evolution of data during a timeframe (time series database). But for medical purpose, it is clear that large panel of data produces more reliable results
In order to have the right data, at the right time, and in the right way, Big Data has to take action.
Dear Colleagues and Friends from RG, Big Data database technologies are mainly used to collect large collections of information. Without computerized analytical systems such as Data Analytics and / or Business Intelligence, the functionality and usefulness of large sets of information collected in Big Data databases for specific research will be very limited. However, the use of computerized analytical systems such as Data Analytics and / or Business Intelligence in combination with the Internet of Things technology, selective data collection applications on the Internet and other ICT and Industry 4.0 technologies in combination with large sets of information collected on Big Data platforms is becoming an advanced instrumentarium research, which can be useful for conducting various types of scientific research and for the needs of business applications in enterprises and public and financial institutions. thanks to the above-mentioned ICT, Internet, Industry 4.0, Big Data etc. information technologies, it allows to reduce the time and increase the efficiency of conducting analytics on large data sets in order to define, define and characterize right data. I conduct research in this area. I described the above issues in scientific publications that are available on the Research Gate portal. I invite you to cooperation.
Best wishes
Dariusz Prokopowicz
Sometimes both big and right data to conduct accurate results and conclusion. Otherwise, right data is first.
- Badges
- Science topic
- Similar topics
- Econ
- Innovation Economics
- Innovation