well as per your question , you are asking for mere "good data bases" ,

My Suggestion and  my solution ( May be I am wrong in your perspective).

Why don't you  to shift to some other ecosystem like Hadoop where care of voluminous real time data is taken to some extent.( Since currently its the best we have ) .It has been a game-changer in supporting the enormous processing needs of big data; a large data procedure which might take 20 hours of processing time on a centralized relational database system, may only take 3 minutes when distributed across a large Hadoop cluster of commodity servers, all processing in parallel. Talking of Traditional data bases they wont offer efficient solution to be precise owing to your real time data . 

All your queries will be solved if you can change over.

I've been trying several kind of databases to store my sensors data(I work primarily with power station data, high amount of it and very fast) and, as expected, some database cannot fit the needs by themselves. Only Oracle could handle the data and it's not as flexible as I need.

My current setup is a "proxy" script that buffers the data on RAM, while use exception handling techniques (try/catch/finally) to store data on database, avoiding errors(or the collapse of the database) by retrying when needed and taking the time it needs to store data correctly. It have proven to work with Oracle, Mysql, Postgres and even mysql(slower to write).

On my case, as I'll need the data on a relational database for further analysis due to the need to follow the Common Information Model(CIM) model. So, to use a NoSQL solution would increase the computational work(and time).

Perhaps any better approach does exist but I didn't find.

well as per your question , you are asking for mere "good data bases" ,

My Suggestion and  my solution ( May be I am wrong in your perspective).

Why don't you  to shift to some other ecosystem like Hadoop where care of voluminous real time data is taken to some extent.( Since currently its the best we have ) .It has been a game-changer in supporting the enormous processing needs of big data; a large data procedure which might take 20 hours of processing time on a centralized relational database system, may only take 3 minutes when distributed across a large Hadoop cluster of commodity servers, all processing in parallel. Talking of Traditional data bases they wont offer efficient solution to be precise owing to your real time data . 

All your queries will be solved if you can change over.

The only other thing that I can currently think of, other than those added by Paolo and Reshi above, is to add a sieve to your real time data. In other words, only store what you absolutely need to. That, and/or summarise your data and store that, this could be done by simplifying formats, discretising continuous streams, and averaging.

Also perhaps consider a Fuzzy Set Theory approach, if you needed linguistic summaries!

I hope that helps, and feel free to ask more questions.

Daniel Lewis

In our practice within Intel's manufacturing, we take the same approach that Reshi Nawab suggests and direct sensor output to a hadoop cluster.  This makes a lot of sense because to get any use out of lots of sensor data, you are going to require power analytic tools.  Some details here in the attached link.

http://www.intel.com/content/dam/www/program/embedded/internet-of-things/blueprints/iot-business-value-manufacturing-blueprint.pdf

I agree with Daniel John Lewis, I think the better way to go is to reduce/summarize the data before persisting (storing) it.  Sensor or stream data, have the particularity of long period of monotony interrupted by disruptions, normally we are interested on the disruption and not on the monotone part of the data, that can easily be reconstructed using knows patters.    You should try to implement intermediary processes that trait windows of data and only persist the disruption.  Take a look at this post http://wp.me/p1Gouv-3B ; after the data is reduced, you persistent problem will be reduced, and then you can use any of the recommended data persistency approaches.

http://arbime.wordpress.com/2013/03/22/the-streamed-data-management-systems/

Hope this help

I would suggest Mongo DB. This is an highly scalable document oriented database supporting JSON like files. Also supporting mapreduce features for data aggregation. We have good results using this database in our IoT experiments.

I hope this will help you.

Hi,

I would also suggest using a horizontally scalable database, as suggested above. For sensor data you'll probably have a lot of small files, which is not very good for HDFS (http://blog.cloudera.com/blog/2009/02/the-small-files-problem/) so I would use HDFS indirectly through HBase, that would handle file consolidation for you. Besides:

- for time oriented data you can use OpenTSDB http://opentsdb.net/ on top of HBase for data collection, and for using a schema optimized for the access patterns of time oriented data

- for SQL access on HBase you can use Apache Phoenix http://phoenix.apache.org/ that gives you a JDBC driver and other things like secondary indexes

I hope that helps,

Greetings,

Juan

In connection with this, anyone please suggest me the data repositories where sensor data is stored and can be made use of it free for research.

NoSQL solutions are designed for specific scenarios and some would be suitable for sensor data (fast data). Cassandra has been recognized for its ability to handle large number of writes. You also need to consider what you want to do with that data once it is in a database. Querying capabilities provided by NoSQL solutions are very different. The following paper reviews different NoSQL and NewSQL solutions.

Data management in cloud environments: NoSQL and NewSQL data stores (http://www.journalofcloudcomputing.com/content/pdf/2192-113X-2-22.pdf)

Hadoop was mentioned as one of the possibilities; but I would suggest rather going with something that is built on top of Hadoop such as HBase as you would get additional features.

An interesting architecture has been proposed for real-time systems – Lambda architecture (http://lambda-architecture.net/). It is especially suitable for sensor data as it is designed as append only.

As Katarina suggested, a Lambda architecture fits very well. Which NoSQL Data Store to use depends on the type of data and what you want to do with the data (e.g. using Spark). If you have JSON like data you might use MongoDB, but if documents have to be updated frequently, a column-family solution (like Cassandra) might be the better choice because of the immutable data handling (append only as Katarina already mentioned). And within the streaming part (e.g. using Kaffka, Storm) you might decide to filter out/aggregate the data you really need later on in the analysis phase.

Try HDF5! It works like a data matrix or tensor, but stored on a hard drive. I used it to store up to 1TB of data. HDF5 supports transparent compression and works will all programming tools: Matlab, C++, Python, etc.

Thanks so much