Very interesting question. My opinion is following (simplified of course): everything we do leaves digital trail... we are spending more and more time on Internet and our life is going bit by bit... all that data can be collected and analyzed with big data systems (data from various sources with different structure in real time) and used for predicting our behavior in the future...

I totally agree with the quote, especially 2):

"Methods for predicting the future: 1) read horoscopes, tea leaves, tarot cards, or crystal balls . . . collectively known as "nutty methods;"

2) put well-researched facts into sophisticated computer . . . commonly referred to as "a complete waste of time."" ~ Scott Adams

A well known example:

In 2008, researchers from Google were claiming that they could predict the flu trend based on millions people’s Google searches. The essential idea, published in a paper in Nature, was that when people are sick with the flu, many search for flu-related information on Google that provides almost instant signals of overall flu prevalence.

However Google Flu trend failed spectacularly missing at the peak of the 2013 flu season by 140 percent. Google then quietly killed the program. See Lazer, D., Wired, 10/01/2015. "WHAT WE CAN LEARN FROM THE EPIC FAILURE OF GOOGLE FLU TRENDS"

See also my answer to your similar question on the forecasting the future economic processes:

Data (big or little) represent the past, i.e. what was already happen. You are trying to predict the future using solely the past. Is it always possible?

In general the future events are NOT defined solely by their past. Only the stable signal (pattern) that can be projected into the future can provide meaningful prediction using a validated mathematical model within the frame of this particular model.

Those data, under the assumption that they are valid and checked by professionals for consistency, can predict the development of your own health. To be more specific, we are currently developing tools and methodologies enabling building reliable models based on AI, machine learning, and related techniques that will help to discover predicting features that precede the development of the diseases.

Those methods can be applied individually and even to the whole societies. We should start to look for markers enabling us to quantify and track the course of the health of societies and whole nations.

This can have two impacts on us: individual and societal. Humanity as such growing so much and fast that we are facing a real danger of a collapse in terms of an abrupt transition of a complex system from one state to the other one, which in turn can disrupt the status quo of the society.

Those issues must be addressed at the international level. We can, when our efforts will be correctly allocated, develop models enabling us to describe and understand the evolution of societies.

It is definitely one of the most challenging spectra of tasks that is humanity currently facing.

In general - yes. But this depends on the type of data and the sensitivity to the methods of collection.

Big Data is usually collected from complex systems

* Human health.

* Stock markets.

* Weather prediction.

* Climate changes.

* Industrial Ecosystems.

* Societies.

* Natural Ecosystems.

the crucial problem is in the way those data are collected. For sure we cannot download all system data in the computer. It is usually done in the form of a selected feature(s) that is continuously measured or sufficiently sampled. Those probes into the system are usually called BigData. It is misleading many researchers.

Actually, they only describe some aspects of the system. Not the whole system. Many people are not aware of this fact. They start right away with those restricted days and implicitly expect that is the whole system.

We must always keep in our mind that the system is far richer than data produces by it! Actually many good papers using AI, machine learning, and similar techniques are not using such reasoning in their research.

I saw too many research papers in medicine that assumed that data are equal to the system under study.

It is wrong, wrong, and once more wrong approach. This leads to incorrect understanding and interpretation of the observed data.

The same problem with phenological observations, for instance. Some data are collected using different observational standards and then analysed all together.

Accordingly, I propose the following research question in this area:

Is technological progress contributing to the improvement of techniques for conducting forecasts of future processes?

Does the technological progress in the field of advanced data processing contribute to the improvement of techniques for conducting forecasts of future processes?

Yes, if you have advanced research tools that help in carrying out prognostic analyzes. For example, tools that allow you to conduct sentiment analyzes on large collections of information downloaded from the Internet and collected in Big Data database systems. If this type of analytics merges with advanced information processing instruments on Business Intelligence analytical platforms, then it is possible to build accurate research tools that enable to carry out precisely completed prediction analyzes of predicted processes for subsequent periods in the future.

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

Is technological progress contributing to the improvement of techniques for conducting forecasts of future processes?

Please reply

I invite you to the discussion

Thank you very much for your response and participation in the discussion

Dear Colleagues and Friends from RG

The key aspects and determinants of applications of data processing technologies in Big Data database systems for predicting future processes are described in the following publications:

Article APPLICATION OF DATA BASE SYSTEMS BIG DATA AND BUSINESS INTEL...

Article The Big Data technologies as an important factor of electron...

Article The Technological Solutions Big Data and the Importance of B...

I invite you to discussion and cooperation.

Best wishes

In the context of the above considerations, the following question is also current:

Are you familiar with the techniques and goals of satellite information gathering for Big Data analysis?

Are you familiar with the techniques, goals and functions of satellite information gathering for the needs of analyzes carried out on large data sets collected in this way information and collected in Big Data database systems?

Can this type of research help in the analysis of key processes in the field of environmental protection, natural ecosystems and biodiversity?

Can this type of research be helpful in accurately identifying complex, multifaceted progressive processes regarding:

- changes in environmental pollution diagnosed in specific areas of the globe?

- precise diagnosis and analysis of the effects of progressive processes of undertakings undertaken in the field of environmental protection, natural ecosystems and biodiversity?

- observation, data collection, investigation and analysis of the ongoing greenhouse gas emission process?

- observation, data collection, investigation and analysis of the ongoing global warming process and the study of the effects of this process diagnosed in specific areas of the globe?

- data collection that could help in forecasting more and more frequently occurring weather anomalies, climate disasters?

- data collection that could help in predicting further natural disasters, for example, developing viral, fungal, bacterial and insect diseases and other pests in areas of arable land?

In my opinion, currently developed techniques of satellite information gathering for the needs of analyzes conducted on large data sets carried out in Big Data database systems may be helpful in the above-mentioned research topics. What other research topics can be implemented in support of satellite information gathering for the needs of analyzes carried out on large data sets made in Big Data database systems?

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

Are you familiar with the techniques and goals of satellite information gathering for Big Data analysis?

Please reply

I invite you to the discussion

Thank you very much

Best wishes

I agree with most of the discussion points presented but I will also say that regardless of wrong projections in terms of big data analysis, if a step by step approach is taken to draw as much as insight as there is available from the data procured and combining them together to dervie usable insights, this approach would be a big welcome one!

At the end of the day, the internet is the voice of the people, undisguised and pure. There may be repetitions, attention seeking attributes , dislodged comments etc but if there is a way to sort the information correctly it can prove to be useful in my opinion.

In view of the above, it is also important to include in the above discussion also the following issue:

How can you improve the process of predictive analysis?

The goal of predictive analysis is to develop predictions for the development of complex, multifaceted processes in various fields of science, industry, economy or other spheres of human activity. In addition, predictive analysis may refer to objectively performing processes such as natural phenomena, climate change, geological, cosmic etc.

Predictive analysis should be based on taking into account in the analytical methodology possible the most modern prognostic models and a large amount of data necessary to perform the most accurate predictive analysis. In this way, the result of the prediction analysis performed will be the least subject to the risk of analytical error, ie an incorrectly designed forecast.

Predictive analysis can be improved by using computerized modern information technologies, which include computing in the cloud of large data sets stored in Big Data database systems. In the predictive analysis, Business Intelligence analytics and other innovative information technologies typical of the current fourth technological revolution, known as Industry 4.0, can also be used.

The current technological revolution known as Industry 4.0 is motivated by the development of the following factors:

Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other advanced data mining technologies. On the basis of the development of the new technological solutions in recent years, dynamically developing processes of innovatively organized analyzes of large information sets stored in Big Data database systems and computing cloud computing for the needs of applications in such areas as: machine learning, Internet of Things, artificial intelligence are dynamically developing. , Business Intelligence.

For the abovementioned application examples, one can add predictive analyzes of subsequent, other fields of application of advanced technologies for the analysis of large data sets such as Medical Intelligence, Life Science, Green Energy, etc. Processing and multi-criteria analysis of large data sets in Big Data database systems is carried out according to V4 concepts, ie Volume (meaning a large number of data), Value (large values of certain parameters of the analyzed information), Velocity (high speed of new information) and Variety (high variety of information).

The advanced information processing and analysis technologies mentioned above are used more and more often for the needs of conducting predictive analyzes concerning, for example, marketing activities of various business entities that advertise their offer on the Internet or analyze the needs in this area reported by other entities, including companies, corporations, institutions financial and public. More and more commercial business entities and financial institutions conduct marketing activities on the Internet, including on social media portals.

More and more public institutions and business entities, including companies, banks and other entities, need to conduct multi-criteria analyzes on large data sets downloaded from the Internet describing the markets on which they operate, as well as contractors and clients with whom they cooperate.

On the other hand, there are already specialized technology companies that offer this type of analytical services, including offering predictive analysis services, develop custom reports, which are the result of multicriteria analyzes of large data sets obtained from various websites and from entries and comments. contained on social media portals based on sentiment analyzes of the content of entries in the comments of Internet users.

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

How can you improve the process of predictive analysis?

Please reply

I invite you to discussion and scientific cooperation

Dear Colleagues and Friends from RG

The key aspects and determinants of the applications of modern computerized information technologies for data processing in Big Data and Business Intelligence database systems for the purpose of conducting predictive analyzes are described in the following publications:

Article APPLICATION OF DATA BASE SYSTEMS BIG DATA AND BUSINESS INTEL...

Article The Big Data technologies as an important factor of electron...

Article The Technological Solutions Big Data and the Importance of B...

Article THE QUESTION OF THE SECURITY OF FACILITATING, COLLECTING AND...

I invite you to discussion and cooperation.

Best wishes

This question was already discussed in the thread following the previous post of this author regarding predicting the future using Big Data. Here the author confuses the terms 'prediction' and 'forecasting' using them interchangeably, such as in the statement"...the result of the prediction analysis performed will be the least subject to the risk of analytical error, i.e. an incorrectly designed forecast." However, prediction is usually reserved for a definitive and specific statement about the value of some variable at the specific time in the future. In contrast, forecasting is a probabilistic statement about a range of the possible future values of the variables.

Next, dynamic systems can be chaotic or stochastic. Chaotic systems can be deterministic and predictable. In this case the value of variable X at a given time can be used to predict the value of X in a long time in the future. Example is Newtonian dynamics that can predict accurately solar eclipse or other events in the universe 1000s years ahead, and others. However, a chaotic system can be deterministic and not predictable in the long run, i.e. the value X can be predicted only for a very short time ahead. The reason for this is high sensitivity to initial conditions that amplifies the unavoidable inaccuracy of the initial conditions. An example is impossibility of a long-term weather forecasting that was originally discovered by the famous Lorenz system in 1963 and is widely known in the literature. The profound fundamental consequence is that regardless of the computer power and amount of collected data (Big Data) the long-term weather forecast as well as forecasting of many complex non-linear economic, geological, etc. systems will never be reliable, although the short-term forecasting accuracy can be somewhat and slowly improved with the increase of computer power. A well-known simple example of a chaotic deterministic but unpredictable system is the non-linear time series equation X(n+1)=A*X(n)[1-X(n)] that behaves totally unpredictable if the numerical coefficient A>=3.95.

In the context of the above discussion, the following question arises:

Will the improvement of Industry 4.0 information technologies, including Big Data analysis, allow for more accurate predictive and predictive analyzes? Will the improvement of forecasting and predictive instruments carried out in this way allow the development of accurate forecasts of the continuation of forecasted climate changes related to the global warming process which is accelerating in subsequent years?

Please reply

Forecasting of climate change has nothing to do with Industry 4.0 information technology.

The challenge is that Science does not understand what causes climate change, can’t explain how it works, can’t predict changes in climate, and has no capability for stopping or reversing climate change.

Who says so? The IPCC itself… the same organization that keeps issuing warnings of imminent disaster in ten or twenty years - predictions that have never come true.

The bottom line: The climate system is a coupled non-linear chaotic system, and therefore the reliable long-term prediction of future climate states is not possible. See the link UN Interagency Panel on Climate Change.

Working Group I: The Scientific Basis

https://archive.ipcc.ch/ipccreports/tar/wg1/501.htm

such as bullet point 4.... "The rapid forcing of a non-linear system has a high prospect of producing surprises."