use spark streaming or kafka platform

Very interesting question. My opinion is following (simplified of course): everything we do leaves digital trail... we are spending more and more time on Internet and our life is going bit by bit... all that data can be collected and analyzed with big data systems (data from various sources with different structure in real time) and used for predicting our behavior in the future...

I totally agree with the quote, especially 2):

"Methods for predicting the future: 1) read horoscopes, tea leaves, tarot cards, or crystal balls . . . collectively known as "nutty methods;"

2) put well-researched facts into sophisticated computer . . . commonly referred to as "a complete waste of time."" ~ Scott Adams

A well known example:

In 2008, researchers from Google were claiming that they could predict the flu trend based on millions people’s Google searches. The essential idea, published in a paper in Nature, was that when people are sick with the flu, many search for flu-related information on Google that provides almost instant signals of overall flu prevalence.

However Google Flu trend failed spectacularly missing at the peak of the 2013 flu season by 140 percent. Google then quietly killed the program. See Lazer, D., Wired, 10/01/2015. "WHAT WE CAN LEARN FROM THE EPIC FAILURE OF GOOGLE FLU TRENDS"

See also my answer to your similar question on the forecasting the future economic processes:

Data (big or little) represent the past, i.e. what was already happen. You are trying to predict the future using solely the past. Is it always possible?

In general the future events are NOT defined solely by their past. Only the stable signal (pattern) that can be projected into the future can provide meaningful prediction using a validated mathematical model within the frame of this particular model.

Those data, under the assumption that they are valid and checked by professionals for consistency, can predict the development of your own health. To be more specific, we are currently developing tools and methodologies enabling building reliable models based on AI, machine learning, and related techniques that will help to discover predicting features that precede the development of the diseases.

Those methods can be applied individually and even to the whole societies. We should start to look for markers enabling us to quantify and track the course of the health of societies and whole nations.

This can have two impacts on us: individual and societal. Humanity as such growing so much and fast that we are facing a real danger of a collapse in terms of an abrupt transition of a complex system from one state to the other one, which in turn can disrupt the status quo of the society.

Those issues must be addressed at the international level. We can, when our efforts will be correctly allocated, develop models enabling us to describe and understand the evolution of societies.

It is definitely one of the most challenging spectra of tasks that is humanity currently facing.

In general - yes. But this depends on the type of data and the sensitivity to the methods of collection.

Big Data is usually collected from complex systems

* Human health.

* Stock markets.

* Weather prediction.

* Climate changes.

* Industrial Ecosystems.

* Societies.

* Natural Ecosystems.

the crucial problem is in the way those data are collected. For sure we cannot download all system data in the computer. It is usually done in the form of a selected feature(s) that is continuously measured or sufficiently sampled. Those probes into the system are usually called BigData. It is misleading many researchers.

Actually, they only describe some aspects of the system. Not the whole system. Many people are not aware of this fact. They start right away with those restricted days and implicitly expect that is the whole system.

We must always keep in our mind that the system is far richer than data produces by it! Actually many good papers using AI, machine learning, and similar techniques are not using such reasoning in their research.

I saw too many research papers in medicine that assumed that data are equal to the system under study.

It is wrong, wrong, and once more wrong approach. This leads to incorrect understanding and interpretation of the observed data.

The same problem with phenological observations, for instance. Some data are collected using different observational standards and then analysed all together.

Accordingly, I propose the following research question in this area:

Is technological progress contributing to the improvement of techniques for conducting forecasts of future processes?

Does the technological progress in the field of advanced data processing contribute to the improvement of techniques for conducting forecasts of future processes?

Yes, if you have advanced research tools that help in carrying out prognostic analyzes. For example, tools that allow you to conduct sentiment analyzes on large collections of information downloaded from the Internet and collected in Big Data database systems. If this type of analytics merges with advanced information processing instruments on Business Intelligence analytical platforms, then it is possible to build accurate research tools that enable to carry out precisely completed prediction analyzes of predicted processes for subsequent periods in the future.

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

Is technological progress contributing to the improvement of techniques for conducting forecasts of future processes?

Please reply

I invite you to the discussion

Thank you very much for your response and participation in the discussion

Dear Colleagues and Friends from RG

The key aspects and determinants of applications of data processing technologies in Big Data database systems for predicting future processes are described in the following publications:

Article APPLICATION OF DATA BASE SYSTEMS BIG DATA AND BUSINESS INTEL...

Article The Big Data technologies as an important factor of electron...

Article The Technological Solutions Big Data and the Importance of B...

I invite you to discussion and cooperation.

Best wishes

In the context of the above considerations, the following question is also current:

Are you familiar with the techniques and goals of satellite information gathering for Big Data analysis?

Are you familiar with the techniques, goals and functions of satellite information gathering for the needs of analyzes carried out on large data sets collected in this way information and collected in Big Data database systems?

Can this type of research help in the analysis of key processes in the field of environmental protection, natural ecosystems and biodiversity?

Can this type of research be helpful in accurately identifying complex, multifaceted progressive processes regarding:

- changes in environmental pollution diagnosed in specific areas of the globe?

- precise diagnosis and analysis of the effects of progressive processes of undertakings undertaken in the field of environmental protection, natural ecosystems and biodiversity?

- observation, data collection, investigation and analysis of the ongoing greenhouse gas emission process?

- observation, data collection, investigation and analysis of the ongoing global warming process and the study of the effects of this process diagnosed in specific areas of the globe?

- data collection that could help in forecasting more and more frequently occurring weather anomalies, climate disasters?

- data collection that could help in predicting further natural disasters, for example, developing viral, fungal, bacterial and insect diseases and other pests in areas of arable land?

In my opinion, currently developed techniques of satellite information gathering for the needs of analyzes conducted on large data sets carried out in Big Data database systems may be helpful in the above-mentioned research topics. What other research topics can be implemented in support of satellite information gathering for the needs of analyzes carried out on large data sets made in Big Data database systems?

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

Are you familiar with the techniques and goals of satellite information gathering for Big Data analysis?

Please reply

I invite you to the discussion

Thank you very much

Best wishes

I agree with most of the discussion points presented but I will also say that regardless of wrong projections in terms of big data analysis, if a step by step approach is taken to draw as much as insight as there is available from the data procured and combining them together to dervie usable insights, this approach would be a big welcome one!

At the end of the day, the internet is the voice of the people, undisguised and pure. There may be repetitions, attention seeking attributes , dislodged comments etc but if there is a way to sort the information correctly it can prove to be useful in my opinion.

In view of the above, it is also important to include in the above discussion also the following issue:

How can you improve the process of predictive analysis?

The goal of predictive analysis is to develop predictions for the development of complex, multifaceted processes in various fields of science, industry, economy or other spheres of human activity. In addition, predictive analysis may refer to objectively performing processes such as natural phenomena, climate change, geological, cosmic etc.

Predictive analysis should be based on taking into account in the analytical methodology possible the most modern prognostic models and a large amount of data necessary to perform the most accurate predictive analysis. In this way, the result of the prediction analysis performed will be the least subject to the risk of analytical error, ie an incorrectly designed forecast.

Predictive analysis can be improved by using computerized modern information technologies, which include computing in the cloud of large data sets stored in Big Data database systems. In the predictive analysis, Business Intelligence analytics and other innovative information technologies typical of the current fourth technological revolution, known as Industry 4.0, can also be used.

The current technological revolution known as Industry 4.0 is motivated by the development of the following factors:

Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other advanced data mining technologies. On the basis of the development of the new technological solutions in recent years, dynamically developing processes of innovatively organized analyzes of large information sets stored in Big Data database systems and computing cloud computing for the needs of applications in such areas as: machine learning, Internet of Things, artificial intelligence are dynamically developing. , Business Intelligence.

For the abovementioned application examples, one can add predictive analyzes of subsequent, other fields of application of advanced technologies for the analysis of large data sets such as Medical Intelligence, Life Science, Green Energy, etc. Processing and multi-criteria analysis of large data sets in Big Data database systems is carried out according to V4 concepts, ie Volume (meaning a large number of data), Value (large values of certain parameters of the analyzed information), Velocity (high speed of new information) and Variety (high variety of information).

The advanced information processing and analysis technologies mentioned above are used more and more often for the needs of conducting predictive analyzes concerning, for example, marketing activities of various business entities that advertise their offer on the Internet or analyze the needs in this area reported by other entities, including companies, corporations, institutions financial and public. More and more commercial business entities and financial institutions conduct marketing activities on the Internet, including on social media portals.

More and more public institutions and business entities, including companies, banks and other entities, need to conduct multi-criteria analyzes on large data sets downloaded from the Internet describing the markets on which they operate, as well as contractors and clients with whom they cooperate.

On the other hand, there are already specialized technology companies that offer this type of analytical services, including offering predictive analysis services, develop custom reports, which are the result of multicriteria analyzes of large data sets obtained from various websites and from entries and comments. contained on social media portals based on sentiment analyzes of the content of entries in the comments of Internet users.

Do you agree with my opinion on this matter?

In view of the above, I am asking you the following question:

How can you improve the process of predictive analysis?

Please reply

I invite you to discussion and scientific cooperation

Dear Colleagues and Friends from RG

The key aspects and determinants of the applications of modern computerized information technologies for data processing in Big Data and Business Intelligence database systems for the purpose of conducting predictive analyzes are described in the following publications:

Article APPLICATION OF DATA BASE SYSTEMS BIG DATA AND BUSINESS INTEL...

Article The Big Data technologies as an important factor of electron...

Article The Technological Solutions Big Data and the Importance of B...

Article THE QUESTION OF THE SECURITY OF FACILITATING, COLLECTING AND...

I invite you to discussion and cooperation.

Best wishes

This question was already discussed in the thread following the previous post of this author regarding predicting the future using Big Data. Here the author confuses the terms 'prediction' and 'forecasting' using them interchangeably, such as in the statement"...the result of the prediction analysis performed will be the least subject to the risk of analytical error, i.e. an incorrectly designed forecast." However, prediction is usually reserved for a definitive and specific statement about the value of some variable at the specific time in the future. In contrast, forecasting is a probabilistic statement about a range of the possible future values of the variables.

Next, dynamic systems can be chaotic or stochastic. Chaotic systems can be deterministic and predictable. In this case the value of variable X at a given time can be used to predict the value of X in a long time in the future. Example is Newtonian dynamics that can predict accurately solar eclipse or other events in the universe 1000s years ahead, and others. However, a chaotic system can be deterministic and not predictable in the long run, i.e. the value X can be predicted only for a very short time ahead. The reason for this is high sensitivity to initial conditions that amplifies the unavoidable inaccuracy of the initial conditions. An example is impossibility of a long-term weather forecasting that was originally discovered by the famous Lorenz system in 1963 and is widely known in the literature. The profound fundamental consequence is that regardless of the computer power and amount of collected data (Big Data) the long-term weather forecast as well as forecasting of many complex non-linear economic, geological, etc. systems will never be reliable, although the short-term forecasting accuracy can be somewhat and slowly improved with the increase of computer power. A well-known simple example of a chaotic deterministic but unpredictable system is the non-linear time series equation X(n+1)=A*X(n)[1-X(n)] that behaves totally unpredictable if the numerical coefficient A>=3.95.

In the context of the above discussion, the following question arises:

Will the improvement of Industry 4.0 information technologies, including Big Data analysis, allow for more accurate predictive and predictive analyzes? Will the improvement of forecasting and predictive instruments carried out in this way allow the development of accurate forecasts of the continuation of forecasted climate changes related to the global warming process which is accelerating in subsequent years?

Please reply

Forecasting of climate change has nothing to do with Industry 4.0 information technology.

The challenge is that Science does not understand what causes climate change, can’t explain how it works, can’t predict changes in climate, and has no capability for stopping or reversing climate change.

Who says so? The IPCC itself… the same organization that keeps issuing warnings of imminent disaster in ten or twenty years - predictions that have never come true.

The bottom line: The climate system is a coupled non-linear chaotic system, and therefore the reliable long-term prediction of future climate states is not possible. See the link UN Interagency Panel on Climate Change.

Working Group I: The Scientific Basis

https://archive.ipcc.ch/ipccreports/tar/wg1/501.htm

such as bullet point 4.... "The rapid forcing of a non-linear system has a high prospect of producing surprises."

use spark streaming or kafka platform

Dear Colleagues and Friends from RG,

The conducted research shows that in recent years the scope of application of Big Data Analytics technology and also other Industry 4.0 technologies in forecasting economic processes has been increasing. In the currently developing fourth technological revolution referred to as Industry 4.0, one of the key technologies that constitute an important element of currently ongoing technological progress are Big Data Analytics analytical and database technologies. Due to the dynamic development of collecting digitized documents on analytical platforms and Big Data Analytics database systems, the importance of data security in these databases is growing, and information risk management systems collected in Big Data Analytics are being improved. Therefore, the systems for managing the risk of data security collected on analytical platforms and Big Data Analytics database systems are being developed and improved. In view of the above, I would like to ask you what particularly important components of the risk management process of information gathered in Big Data Analytics databases are currently being developed, improved and tested? Does anyone of you conduct research in this area?

In view of the above, the dynamic development of the scope of applications of Big Data Analytics and other Industry 4.0 technologies in the processes of forecasting future economic and climate phenomena, geological processes, trends in financial markets, etc. generates many new questions. To answer the question: How accurately can you forecast specific future economic processes? preferred research and prognostic methods should be selected and the necessary, expected level of accuracy of test results, analyzes and quantitative forecasting instruments used should be specified. In the selection of preferred research and prognostic methods, the key issue is to decide whether the analytical instruments used will function only in the field of quantitative sciences or also selected instruments of qualitative research? In my opinion, technological progress in the field of advanced data processing contributes to the improvement of techniques for conducting analyzes of forecasting future processes, especially quantitative techniques of conducted research and analyzes for the needs of forecasting. Progress in this field is determined mainly by the implementation of ICT, Internet and advanced technologies, mainly quantitative data processing typical of the current technological revolution known as Industry 4.0.

The development of advanced data processing technology Industry 4.0 is, among others, determined by the potential applications of these technologies, including primarily the technology of large data analytics carried out on the Big Data Analytics computerized platforms in forecasting business processes. The current technological revolution referred to as Industry 4.0 is determined by the development of the following advanced information processing technologies, which include, among others: Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other advanced data processing technologies Data Mining. If, as part of research and analytical processes, advanced research instruments are available to assist in carrying out prognostic analyzes. For example, tools that allow you to conduct sentiment analysis on large sets of information downloaded from the Internet and stored in Big Data database systems. If this type of analytics is combined with advanced information processing instruments on Business Intelligence analytical platforms, then it is possible to build accurate research tools enabling precise analysis of predictions of anticipated processes to be carried out for future periods in the future, including complex future economic and financial processes.

One of the areas in which the demand for precise forecasting processes is growing rapidly are finance and banking, and above all forecasting of asset valuation trends, including shares, bonds and other securities on stock exchanges. In addition to improving techniques and quantitative instruments for research carried out for the purpose of forecasting regarding multi-faceted macroeconomic and meso-economic processes, prognostic research is increasingly carried out for the purposes of developing microeconomic forecasts for the development of the functioning of specific economic entities. In addition to the field of economics and finance, the improvement of research processes, techniques, instruments and quantitative forecasting methods allows to increase the precision of forecasting future processes that will also take place in other fields of science, civilization development, natural, climate, geological and other processes. To sum up, I say that in recent years the scope of Big Data Analytics and other Industry 4.0 technologies in forecasting business processes has been increasing. I conduct research in this area. The conclusions of the research I published in scientific publications that are available on the Research Gate portal. I invite you to cooperation.

What do you think about this topic?

What is your opinion on this topic?

Please reply.

I invite you to discussion and scientific cooperation.

Thank you very much.

Best wishes.

Dariusz Prokopowicz

Dear Colleagues and Friends from RG, Currently, the issue of the impact of the SARS-CoV-2 Coronavirus pandemic causing Covid-19 disease on many aspects of medicine, economic development, and everyday human functioning has dominated many research processes. This is due, inter alia, to the fact that more and more data in the field of virology knowledge indicates that the SARS-CoV-2 Coronavirus will remain in subsequent years. Unfortunately, however, it is not known whether the new types, strains of this Coronavirus will dominate and replace the SARS-CoV-2 Coronavirus, or whether the SARS-CoV-2 Coronavirus will still dominate in its current form.

These issues will be clarified in a few years and the outcome of this process and the answer to this question will be determined by the following factors: 1. Rate of mutation and formation of new varieties of SARS-CoV-2 Coronavirus significantly different from the SARS-CoV-2 Coronavirus. 2. The development of resistance of the collective human population to the SARS-CoV-2 Coronavirus (similarly to the influenza virus and other infectious diseases caused by viruses). 3. The creation of the SARS-CoV-2 Coronavirus vaccine, its effectiveness and the scale of its use by citizens. For example, ideas about linking vaccinations with the possibility of traveling to other countries are already emerging.

Currently, Big Data Analytics analytical centers are creating specific models for predicting the development of the situation regarding the development of the SARS-CoV-2 pandemic SARS-CoV-2 causing Covid-19 disease in subsequent months and years and the effects of using specific epidemiological and sanitary safety instruments and the use of specific anti-crisis, interventionist socio-crisis policies -economic, their impact on economic processes, the global economy. However, the results of these predictions can be very different depending on the initial assumptions made in analytical processes carried out on Big Data Analytics analytical platforms using modern Industry 4.0 technology solutions.

In connection with the above, I would like to ask the following question:

1. Are papers already published on the use of Big Data Analytics analytical platforms for conducting prognostic analyzes of the development of the SARS-CoV-2 Coronavirus pandemic or other issues that are secondary processes caused by this pandemic?

2. For example, are these types of analytical tools used to predict the impact of the SARS-CoV-2 Coronavirus pandemic on economic processes that will emerge in the coming months?

3. Is there still too little data to carry out this type of analysis, too many unanswered questions about the development determinants of the current pandemic, too short a data collection period, etc.?

What is your opinion on this topic?

Please reply.

I invite you to discussion.

Thank you very much for participating in the discussion.

Greetings,

Dariusz Prokopowicz