Due to a recent "BigData" hype, I've recently tried to analyze the possible sources of such data for research projects. The interesting finding is: there does not appear to be such a huge amount of "usable" data in the open space. But I would love to be wrong.
The definition of "usable" in this context is roughly:
What I see is:
- more than half of the internet traffic is the streaming video. Good part of this is porn, then blockbusters (netflix and co) and then "cats and dogs" videos (youtube).
- Next big thing are the photos. The situation appears to be better with photos, and quite a few freely available geo-referenced photos can be found on photo sharing platforms.
- Geo-references tweets and such don't qualify as big data. Same for blogs and such,
- Facebook, google+ & co. have more data, but these platforms are mainly used for sharing of information with friends & family => not open data.
- Environmental open data other than those from the satellites probably does not qualify as big data either. Satellites seem to be OK.
To cut it short, the amount of "useful" open data on the web (see definition of useful above) appears to be quite small. And much of the company-owned data cannot be explored because of the privacy concerns (unless you work for NSA). Besides, the companies such as Google and Facebook base their business on exclusive access to this data.
Does this mean that open data research is essentially about companies digging through the data they somehow collect on their own (+satellites) or am I missing the point?
Have you checked:
Foursquare API: https://developer.foursquare.com
Here API: https://developer.here.com
InfoChimps: http://blog.infochimps.com/2011/08/30/geo-api/
Geocommons: http://geocommons.com
Census: http://www.census.gov/geography.html
Telestrian: http://www.telestrian.com
Good luck in finding more!
Here is something about Earth Observations and Big Data:
http://www.cost.eu/domains_actions/TDP/Actions/TD1403
Everything what is currently collected/kept in national databases and related to climate change issues etc, the databases are large and in principle "open". ESA Sentinel mission starts to add data in huge quantities, and the data analysis still needs ground-based and global in situ measurements.
Denis, you touch an important issue. In the domain I know the quantity of meteorological, hydrological and climatic data are collected by national services worldwide. The World Meteorological Organisation stresses its role as "to broaden and enhance, whenever possible, the free and unrestricted international exchange of such data, in consonance with the needs of the global hydrological community and the requirements for WMO's scientific and technical programmes."
The problem is with "whenever possible", what makes practical access to this data more painful and sometimes impossible, as the latter depends only on a good will of different national administrations.
Even US governmental agencies (see http://www.wmo.int/pages/prog/wcp/documents/private-sector/Sharon_Hays.pdf) makes the difference between :
- freely available data
- readily available data
for no-scientific users.
You opened the Pandora box on this forum, let's see what may emerge from it.
@Ali: not all of these, but some yes (so thx for the hints).
The problem is that they may not fit the definition of the BigData used by EC.
Case "foursquare", which is AFAIK quite big as geo-referenced data goes: 50M users and 1 bilion posts. Sounds like a really big data set, no?
But: as far as I can see, many (all?) of these posts are small geo-located pictures. ca. 80kb per pic.=> we are talking about at most 100kbyte * 10^9 = 10^14 bytes here (100 Tbyte), more likely a factor of magnitude less. (did I miss something?)
Problem: EC is asking for pentabytes. 100Tbyte might qualify as a "small big data set". 10Tbytes would not qualify at all. :-(
Hiya,
just read something that may be of interest, an app being used in India to report trash dumps and the like. while fairly primitiv, sounds effective
Swachh Bharat: http://www.vocativ.com/world/india/swachh-bharat/
not sure if this is big enough for you big boys, but with that number of potential users could get there!
:)
Kathi
Hi Denis,
It is not entirely clear to me how much pre-structuring you are demanding for regarding something 'big data'. The other point is how public is 'public'.
Localized holiday pictures of tourists you can find really a lot, but to make them useful for analysis one mostly needs other datasets as well, plus some cleaning and processing to get a workable dataset.
My hunch is that you can still can find quite some GIS datasets, e.g. geo referenced descriptions of public infrastructure, land use, building stock at EU (Eurostat e.o.) and national levels, also quite some large cities must have something. Also quite some FP7 (and upcoming H2020) studies produce sizeable datasets, that are normally available for 3rd parties (after the project is completed).
For example FMI is involved in the FP7 TOPDAD study, in which we are producing a GIS dataset for climate and socioeconomic scenarios and impacts covering the whole of Europe in 25 x 25 km grid cells. This boils down to ~ 20 000 cells x 12 months (+ whole year) x 19 5-year periods + base period x ~25 variables for 3 RCP-SSP combinations for 17 GCM&RCM output set + ensemble. Otherwise stated 1 scenario set from 1 model contains about 130 million figures (numbers) and the whole collection about 6.6 billion.
Admittedly, various of these datasets are not straightaway accessible, but e.g. require a registration and approval procedure (even though otherwise free of charge). FMI is an example of a large provider of such kind of open-data services and these amounts of data are also really big. See: https://ilmatieteenlaitos.fi/avoin-data (avoin = open) ; some of the clickable topics have english text underneath.
Social media data from Twitter and Brightkite; see the papers
Jiang B. (2015a), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.
Jiang B. and Miao Y. (2014, accepted), The evolution of natural cities from the perspective of location-based social media, The Professional Geographer, xx(xx), xx-xx, DOI: 10.1080/00330124.2014.968886, Preprint: http://arxiv.org/abs/1401.6756
Nighttime image for the entire world; see this paper
Jiang B., Yin J. and Liu Q. (2014, accepted), Zipf’s Law for all the natural cities around the world, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1402.2965
OpenStreetMap for Europe and USA; see the papers
Jiang B. and Jia T. (2011b), Zipf's law for all the natural cities in the United States: a geospatial perspective, International Journal of Geographical Information Science, 25(8), 1269-1281.
Jiang B. and Liu X. (2012), Scaling of geographic space from the perspective of city and field blocks and using volunteered geographic information, International Journal of Geographical Information Science, 26(2), 215-229. Reprinted in Akerkar R. (2013, editor), Big Data Computing, Taylor & Francis: London.
if your data are heavy tailed: http://en.wikipedia.org/wiki/Head/tail_Breaks
Hi,
am currently working on my undergraduate dissertation research looking at the factors that lead to a successful adoption of open data and GIS initiatives for independent businesses or SMEs. I would like to find out what data SMEs use, need and is available and if they could use any open data available to cut costs of using paid-for-data and be more competitive against larger organisations.
I was wondering if anyone can help me in terms of literature and research already available.
Thanks,
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data