Now these days 'Big Data' is being considered a vital field of research ( rather I should say it is a buzzword in today's data science research community). This question, I want to float to discuss some real challenges pertaining to Big Data handling for various purposes. So many researchers are viewing this filed merely as extended field of distributed processing by simply dividing the task into small pieces and then agglomerating the results to handle scalability issue( i.e. simple design of map and reduce procedures).
I request respondents to provide references of vital resources, which are posing challenges( theoretical as well as practical aspects) to understand Big Data environment in true sense and to understand difference between distributed environment and big data environment.
Dear Dr. Alam,
Purpose of this question is different, discussions on challanges and related opportunities are expected, in the benefit of bigdata research community.
Regards.
Dear Sanjay Garg,
İ found inreresting articles and sent to you.
https://www.sas.com/resources/asset/five-big-data-challenges-article.pdf
http://www.computerweekly.com/feature/Big-data-and-analytics-a-large-challenge-offering-great-opportunities
Regards, Shafagat
Over ten years, many researchers analyzed the microarray data and could not conclude clear results. I proposed the Matroska feature selection method for microarray data that consists several small linear separable gene sub-spaces and other high-dimension gene spaces that is not linear separable. Therefore, we can easily analyze these small genes sets. Download my papers from RG.
Decision tree is one of the most used algorithm for classification and regression in data mining. To apply DT for large datasets some people from google and some from yahoo has developed following algorithms.
THE PLANET.
research.google.com/pubs/archive/36296.pdf
there are many other references to this topic. in case, if someone is interested to know more on this may reply to this comment .
Dear Sanjay Garg,
Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.
http://www.sas.com/en_us/insights/big-data/what-is-big-data.html
Regards, Shafagat
Apart from Hadoop, Greenplum can also be used for Big Data Analytics.
Machine generated data that is "big" and to handle these data, a new platform is introduced i.e. Splunk which can be used with Hadoop as well as separately in windows and other OS.
You need an end-to-end Data management solution. Hadoop is simply a component of the required solution. What is needed is a Data Lake with auto ingestion of data from structured and unstructured Data sources. Hadoop is considered to be another data source to the Data Lake, some organisations simply use it as storage feeding into the Data lake. For example Telcos store web URLS in Hadoop only.
The is having the wherewithal as discussed to process and access these Big Data for making timely decisions. The newly released text can provide more insight here.
Many thanks,
Debra
https://www.researchgate.net/deref/http%3A%2F%2Fwww.igi-global.com%2Fgateway%2Fchapter%2F179124
Chapter Affordances of Data Science in Agriculture, Manufacturing, a...
- Badges
- Science topic
- Similar topics
- Engineering
- Materials Engineering
- Powders