I am with Alexander & James on this. First, Nearly, all so-called big-data is "it-is-what-it-is" data (i.e. The analyst has no control over how the data is collected. Under these circumstances the selection process for the sample is non-random & the selection probabilities are typically unknowable. One is often not even certain if the sample is from 1 unique population.) Under such circumstances biases can arise & their effects are not measurable. For something like sales; big-data can be highly applicable & population specific because it represents a sample from one's particular population: your customers. Yet, if it represents health records, one has to wonder whether or not the source for these record misses some parts of the population or over-represents sub-populations. I think solid "statistical thinking" can help big-data analysis more than the other way around.

No, I do not agree with your opinion on this matter.

1-st, what do you mean statistical data? Data can be in different formats: numerical-a set of numbers, video or audio-a set of video or audio files, text- a set of text files, and some other. There is no such term as statistical data. It is better to say statistical analysis of data rater than analysis of statistical data as in your post.

2-nd, and even more importantly, any use of the word 'analysis' is meaningless unless and until you clearly state your question that you want to get answered.

Only if you have a right question you could analyze your data , and your analysis could help you to get the answer to your question. However you will not get the meaningful answer if your question is wrong, or your data do not contain the right information for answering your question. This is typical if data are collected just because they are available without a pre-planned data collection procedure.

3-rd, if you use statistical analysis of your numerical data do you mean getting simply a descriptive statistical summary such as data mean and standard deviation? Getting just descriptive statistical summary is NOT the same as data analysis.

Or you state some statistical hypothesis and get some data samples to accept or reject your hypothesis. Again, NO clear hypothesis/question makes the use of any analysis in general meaningless.

If your data include the entire population for some Big data database then no statistical analysis is needed at all, because the goal of most statistical analysis is inferring properties of the entire population from some random samples from this population, or comparing different populations.

The bottom line: no matter Big data technology or Little data if one does not clearly state the right question / problem the so-called analytics and technology revolution will always be a futile exercise.

Data analysis in big data environment needs to apply a lot of algorithms to reach the right decision in real time data analysis

I think that the basic problem with 'big data' is bias. If certain parts of a population are easier to collect than others, then it makes the problem worse if you just keep collecting more and more of the convenient data. This is not an area in which I've worked, but I suppose that the important work in big data being done today is probably that which addresses this problem.

You might say that the problem is 'representativeness.' Even in a model-based sample, one uses regressor data from the entire population and models for subpopulations as needed. More commonly, for probability/design-based sampling, you strive for representativeness by having each member of a sample represent a number of members of a population which is the inverse of the probability of selection for that member of the sample. Big data may lack a concept of 'representativeness.' - Cheers!

At first glance, and with some interpretation of your question, the response is obviously yes.

Of course the fact that the ability of new database tools (a.k.a big data enablers) to process huge amount of data in very short time helps a lot : with practically no limitation in storage and processing due to cloud as a service (Xaas) solutions, you don't have to worry (unless rare cases) about quantifying the amount and sizes of input in your database, in order to process the data. And clearly this is an huge improvement (and probably a waste of CO2 ;) ). Better, the usability of all those data is improving due to the numbers of available tools and developments (open source or of the shelf) which grow to exploit all those data (data visualization, data matching tools, AI...).

The Big allow us to consider the cases in their whole, and not in samples (and all the bias and approximation they involve, nonetheless their complexity and high level grade understanding). Statistics as long time be a sample oriented science, aiming to resume the "multiple" in the "some". and we abusively call "statistics" anything related to numbers and quantity. That's why, as said by Alexander Kolker if you are referring to Statistics with a big S, big data isn't an improvement... just because Statistics don't need it to improve. but if you refer to data analysis, its the obvious yes :) )

On the other hand, all this easiest way to use data and specially with analytics, put some laziness and candor in the results : it is not because tool on a dataset gives you a result that this result is relevant. Understanding the way those analytics are made is key to avoid correlation and hastiness conclusion.

Despite the fact that you can get all the data you want, the more you'll be selective in the data you want to use, the less you have the chance to make a mistake... the proper of datascientist people is the ability do determine the data set, the process and the accuracy of the results, and despite the easier acces to big data and tools, they are more relevant than never in our companies or research facilities.

Yes, it will definable enhance.

I am with Alexander & James on this. First, Nearly, all so-called big-data is "it-is-what-it-is" data (i.e. The analyst has no control over how the data is collected. Under these circumstances the selection process for the sample is non-random & the selection probabilities are typically unknowable. One is often not even certain if the sample is from 1 unique population.) Under such circumstances biases can arise & their effects are not measurable. For something like sales; big-data can be highly applicable & population specific because it represents a sample from one's particular population: your customers. Yet, if it represents health records, one has to wonder whether or not the source for these record misses some parts of the population or over-represents sub-populations. I think solid "statistical thinking" can help big-data analysis more than the other way around.

So there are different opinions here, saying that of course, big data will help and others saying that big data are biased because the collection is not random. And then there is the opinion that you have to first formulate the question before you can start an analysis to get a meaningful answer.

Particularly the last point is counter anything that is really the gist of big data analytics which is really to let the data define the questions without human intervention and outcome expectations. Probably the best example for this was the breast cancer study from a few years ago that shifted the focus away from the actual tumor to the surrounding tissue for a long time prognosis.

Of course, there may and will always be some bias that trickles into the algorithms but the bias is not related to the data set, or at least, it shouldn't be.

Random vs. non random data sets should also not be an issue because in reality there are no truly random data, so it is more a matter of varying degrees of randomness that needs to be recognized as what it is, in other words any result and outcome of big data can only be applied to the specific data set from which it was derived.

If you take it then to the next level and run big data analytics on the outcome of multiple studies, the bias of the individual studies should actually come out as one of the results but at the same time, there should be also more universally applicable results in the long term.

Michael Schuette

1-st, I totally agree with your statement "...any result and outcome of big data can only be applied to the specific data set from which it was derived."

At the same time it means that prediction of future events and trends within the frame of Big Data using only a huge collection of the past data (what was already happen) is in general not possible (except some specific limited cases in which a stable mostly linear trend can be projected (extrapolated) in the short-time future). Therefore the main paradigm and the real gist of Big Data analytics - prediction of the future in terms of solely huge collection of past data- is not attainable in general (See another discussion on the similar topic here on RG).

2-nd, I totally disagree with your another statement that "...the gist of big data analytics which is really to let the data define the questions without human intervention and outcome expectations".

Perhaps the best illustration that this gist is meaningless is a quotation from The Hitchhiker's Guide to the Galaxy which is a comic science fiction series created by Douglas Adams.

In the first novel, a group of hyper-intelligent beings demanded to learn the Answer to the Ultimate Question of Life, The Universe, and Everything from the supercomputer Deep Thought. It took millions of years of calculations, and the answer produced was 42. However the beings never actually instructed the computer what the Question was. Lacking a real question, it was decided to settle for the out-of-thin-air question "How many roads must a man walk down?".

Do you really support such an approach to Big Data analytics??!

I disagree with your disagreement. Coincidentally, I was tempted to quote the same ASCII code for *. And I do agree that you need to define a frame work or a goal, for example the predictability of long term outcome of cancer therapy. However, the data themselves need to interact with each other to identify commonalities or exceptions and the answer will never be an * as in Hitchhikers Guide to the Galaxy. Big data is all about to expect the unexpected, otherwise, you might just run statistics with the same tunnel vision that has plagued humanity since the beginning of science.

In other words, yes I do support this approach to Big Data and if the result really is an *, then you have to feed more data until you get more detailed results.

Michael Schuette

I'd like to finish this discussion by adding a few additional comments on your last post:

1."..the data themselves need to interact with each other to identify commonalities or exceptions."

If the data interact with each other they are correlated, not truly independent. It means in turn that the information that they carry on is overlapped; thus, additional data points (a number or, say, image or object picture) contain less new information. The number of truly independent data I is always less than the total number of data N by the scaling factor S. If the 1-st order autocorrelation coefficient of data is R1, then I = N*S =(1-R1)/(1+R1). See the attached plot. Even for relatively weakly correlated data with R1~0.3 or so, the actual number of effective data points that carry on some new information drops by a factor of 2. For strongly correlated data R1~0.8-0.9 the number of truly usable data is less than 10% of their total amount. It is the reason why the current neural networks that use BigData are so hungry for more and more (supervised) data, and why they use data so inefficiently. It is always better to get a little amount of accurately measured truly independent data with little noise (good quality data) collected using a pre-planned procedure than a lot of poor quality (noisy and correlated) data collected just because they are available from the various sources.

2. "Big data is all about to expect the unexpected ..." (apparently the unexpected pattern that can be thought of as a new insight or discovery within the data).

The key here is whether that particular pattern of interest was thought of before or after the fact of its observation. If you calculated the probability of a pattern as being small, and it was actually observed, then it might be a new insight in the data. However, if you observed some pattern and now think it is something new (after observation), it is pointless. Indeed, suppose you observed a license plate of a car ahead you ABC123. You calculated before that the chance of this happening is 1 in 26^310^3=17,576,000 but you actually observed this pattern. It is an unexpected pattern. However, if you observed this pattern, and now think that it is the unexpected pattern (after the fact), then it is a piece of trivia.

3. "...you have to feed more data until you get more detailed results."

This is mostly addressed in 1. (see above). Here I would add that using statistical principles will immensely help to BigData paradigm and not the way around.

Statistics is not "the same tunnel vision that has plagued humanity since the beginning of science" as you say. Statistics developed by great scientists and mathematicians over the last two centuries or so is a great asset that must be learned and used no matter BigData or not.

Alexander Kolker

Of course it is more efficient to have a small number of independent data than a large amount of data with unknown dependencies but that is not how things work in real life.

Why would it make a difference if a pattern was thought of before the observation? I am thinking of all kinds of things all the time without observing them but sometimes I will observe patterns that I had thought of and it is not a trivia but a confirmation of my previous train of thought. In some cases, there have been decades in between the two events, for the simple reasons that there were no technical means at the time of even evaluate the subject.

As to point 3, yes, I actually studied statistics and agree that they are a very valuable tool but my comments were pointed towards the abuse of statistics to massage data to fit the narrative, which is an easy thing to do. The same thing is certainly true for Big Data

The above discussion inspired me to the following considerations:

Will the development of Internet data analytics dominate Google?

The technological conglomerate and dominant market on the internet information services market Google already offers a wide range of analytical research tools, including Google Analytics used by scientists and researchers to acquire and analyze data collected on the Internet.

We continue to observe technological progress in this area. However, not all information services were successful in Google and dominated the market. An example of the lack of a spectacular success was the creation of social media portal Google+. On the other hand, Google, by acquiring the YouTube portal, has taken the lead in the field of social media portals. In addition, the information analysis tools are still being developed and the algorithms in the information services already offered are being improved. Google Group is expanding its Big Data database containing huge amounts of data about Internet users. If the technologies of advanced data processing collected in Big Data database systems are enriched with developed Business Intelligence technologies and artificial intelligence, then the Google group may strengthen its dominant position in the scope of offered IT services.

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

Will the development of Internet data analytics dominate Google?

Please reply

I invite you to the discussion

Thank you very much

Best wishes

Micheal, you say you studied statistics but it is plain to me that you do not really understand finite population statistics. I have done a lot of work with big data & one of the greatest challenges that I have had is trying to explain to the analysts that the problem is deeper than they think.

Often, one tries to use data to extend the analysis results outside the data set. In the case of "big data" one has no clue who the sample really represents. Almost all big data includes records that should not be included, records that are missed, some sub-samples are over represented & other are under-represented. That is only the "statistical issues". Typically, the variables available in big data are limited & are rarely the variables that we are really interested in. So without any validation often one assumes they are "proxies" for the variable of interest. Contrary to common opinion, big data frequently has errors & omissions. Most big datasets are totally unedited or validated. With big data, replication is difficult if not impossible & thus the "scientific method" gets thrown out the window.

My experience with some big data analysts is they ignore all these issues & if the results agree with their prior opinions they accept their results. If it does not agree, they keep trying new models or datasets until they get agreement. Not all big data analysis is bad & in theory new things may arise from this analysis but only if one does the analysis in a scientific manner. Pretending that big data makes scientific methods irrelevant usually leads to irrelevant results.

Dariusz, Google collects a complete Census of information of their client base. So as long as one confines the extension of the results to only Google clients, you do not need to worry deeply about statistical issues. Google is a special case & what they do, cannot be generalized to all big data sets. If you are trying to identify clients to sell things (or ideas) to, many big data sets can improve your efficiency at targeting clients encompassed within that data set. Thus for many businesses big data is a gold mine but when you are addressing general social, health or economic questions within a country's population then big data is often useless or takes an immense amount of effort & money to correct the biases in the data.

Jack, I think you are misreading my answer but thanks for elaborating on the details, I 100% agree that every analysis / analytic process is only as good as the data that are put into it.

But with the increased compute power at our hands now, chances are that over time the different biases/preselections will cancel each other out.

Michael, Sorry if I over-reacted & over-stated. Assuming that increasing the record count in big data sets will tend to reduce biases is a common refrain that I have heard but in fact it tends to entrench the biases. Example, I worked in one country where filing income tax returns was voluntary. Paying your taxes was legally required but filing was not. So people who owned no taxes (but maybe had income) tended not to file. Thus, many stay at home wives, handicapped, poor, children, transients, etc. did not file. In any one year about 2/3 rd of the adult population filed & some adults never acquired an income tax identifier during their entire lives. The resulting annual datasets represented a skewed subset of the nation's population. Pooling together all the collected years of this data created an immense dataset that still contained the biases of the annual datasets. Rarely is there a hidden-hand-of-god mechanism making theses biases "average-out". There are many good uses for big datasets but when you use them you must search out these biases & find strategies for removing them or making them irrelevant. Big data tends to give us different ways to look at some issues but often the cost for addressing the biases within the data is high. This significant cost tends to make researchers with limited resources invoke god's help.

Jack, no worries, I know where you are coming from and like you stated, there is never a limit to abuse.

الابداع في طرح الاسئلة يولد التفاعل المفرط

Big data can be tricky. It depends on so many things like who and how the data is collected. It is best practice to use machine collected data like we do in semiconductors. The Ocean CO2 data is another example called SO-CAT. I analysed this data and there exist no correlation between ocean CO2 and troposphere CO2 rise. In conclusion. big data should be machine collected. If not then the analysis is exponentially more difficult. For example: We used to have operator inputted data. We would see differences in the data by shift and by the operator. We had to subtract out these known shifts.

Dave White

Dave, I am curios on why your analysis results in "...no correlation between ocean CO2 and troposphere CO2 rise." It looks like the Henry's law should be at work in this case, i.e. the amount of a given gas dissolved in a liquid (ocean water) is directly proportional to the partial pressure of that gas in equilibrium with the liquid. The troposphere rise of the CO2 amount should be directly proportional to the number of moles of CO2, i.e proportional to its partial pressure. So, one would expect a strong correlation between the troposphere rise of the CO2 amount and its amount dissolved in ocean water (water acidification). Would you please clarify....

Therefore, based on the above considerations, I formulated the following thesis that analyzes based on the processing of data collected in Big Data database systems facilitate the analysis of statistical data. In the context of this thesis, the following questions arise that may be the basis for conducting scientific research in this field:

Is the purpose of conducting analyzes based on the processing of statistical data collected in Big Data database systems more important to have more data and greater computing power of computers or more advanced data processing technologies, multi-faceted analytical models, and Industry 4.0 technologies, including Business intelligence analytical platforms?

Is it possible to prepare reports with the tools of conducting analyzes based on the processing of large statistical data sets collected in Big Data database systems, as well as high computing power of computers and advanced data processing technologies, multi-faceted analytical models, and Industry 4.0 technologies, including Business intelligence analytical platforms analytical forecasts for the development of complex, multi-faceted economic processes, macroeconomic development of the economy, developments on the financial markets?

Are there known, researched, described in the literature examples of multi-faceted, complex macroeconomic prognostic analyzes of the future development of entire national economic systems and / or financial markets based on the processing of large statistical data sets collected in Big Data database systems using advanced data processing technologies, multi-faceted models analytical, Industry 4.0 technologies, including Business Intelligence analytical platforms?

What do you think about this topic?

What is your opinion on this topic?

Please reply

I invite you to discussion

Thank you very much