How to analyze data distributed in multiple data centers?

How to compare two groups with only two measurements?

Hiiiii everyone! I have an enquiry on statistical analysis. I was looking for many forum and it's still cannot solve my problem. I want to compare means of two groups of data but only with two...

03 March 2021 8,796 3 View

Can we classify HV and LV lines based on line length?

I have dataset which shows the length of power lines. I need to classify the lines based on the line length. Is there a rule to classify the High voltage (HV) and low voltage (LV) lines based on...

03 March 2021 4,116 4 View

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10?

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10? Since, the growth curve from day 2 to 10 won't be like the "Logistic curve" it might not follow logistic...

03 March 2021 1,401 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature)?

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature-based reflection)? Thank you for your time and for...

03 March 2021 2,870 3 View

What degree of missing data in a particular case/participant results in Expectation-Maximisation no longer working?

I'm dealing with a mediation model and am using the PROCESS module in SPSS. Due to SPSS and PROCESS being limited in the imputation methods - being unable to handle multiple imputation - the other...

02 March 2021 4,362 1 View

What analysis to choose with a large neste dataset and clearly skewed or kurtotic distributions?

Hello! I have a dataset of n=3000 nested within 8 countries with approximately 200 or 400 responses in each country. I originally planned to perform multilevel modelling with 4 dependent variables...

02 March 2021 6,865 1 View

Do you know any MR image dataset?

I would like to research on MR images (0.5T and 3T). Can you please suggest some websites that I can download dataset including both 0.5T and 3T MR images? Thank you.

02 March 2021 7,735 3 View

Can NFL theorem be valid in infinite search space in ML?

NFL theorem is valid for algorithms training in fixed training set. However, the general characteristic of algorithms in expanded or open dataset has not been proved yet. Could you show your...

01 March 2021 1,189 3 View

Does anyone know how to plot my station to a section plot on Ocean Data View (ODV)?

I made a vertical section plot with WOA .nc file on ODV, and now I want to plot my stations (csv. file) in it. Does anyone know how to import my points to the section? They are two different...

01 March 2021 3,610 1 View

Tamer Emara

Thank you for your reply. I mean how to process. For example, the naive solution is to collect all the data into one data center then process them. Is there any other method or techniques to solve the problem?

Mohamed AbdElAziz Khamis

https://www.datastax.com/wp-content/uploads/2012/09/WP-DataStax-MultiDC.pdf

https://mapr.com/blog/multi-master-replication-geo-distributed-data/

http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html

https://www.researchgate.net/post/Can_HDFS_operate_across_multiple_datacenters

Ajit kumar Roy

The article in link may be interesting to you

- Using Apache Hadoop: An open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. https://www.tutorialspoint.com/hive/hive_tutorial.pdf https://www.tutorialspoint.com/impala/impala_tutorial.pdf

Ali Rezaee

note that hadoop /spark is not a good fit for geographically distributed data-centers.

Also, as you did not describe the details about the "analysis" you intended to do, the answers will not be helpful.

you should define your use-case and also quality attributes such as response-time constraints before any accurate solution can be offered.

I think wide area data analytics is applicable in your case where data is generated in a geo-distributed fashion.

Thanks Dr. Ajit kumar Roy . That's exactly what I need. I found some papers in the area of wide area data analytics. It is an interesting topic.

Gourav Gupta

Dear Tamer Emara ,

You can use NIFI or Sqoop for importing the data from disparate data islands and process the data using Apache Hive,Apache Pig,Python(Pandas or Numpy),Spark etc.

If you don't want to import the different sources data on Hadoop Ecosystem(HDFS),in that case you have to create a federation layer like Impala,Kudos,Drill etc.

Best regards!

Gouarv Gupta