I'm mainly working on automotive projects with a focus on location estimation based on GNSS and other sensors within the car. Our main focus is to record sensor data in a framework that allows realtime record & playback in order to do one measureing campain and use the data in "quasi realtime playback" to iteratively improve our algorithms.

For this purpose, the Robot Operating System (ROS) is a good choice. It offers a libraries for a wide varierty of sensor interfaces, allows centralized data handling on so-called ROS nodes and it is possible to run and synchronize multiple instances on multiple machines. In addition, the ROS record function allows to generade bag-files, in which the timestamped data is saved and can be replayed. ROS also offers some data visualisation tools, such as rviz for spatial data. However, this might not be appropriate to simple sensor-output data visualisation. We mostly use python/matplotlib based visualization, which can be interfaced to ROS quite easily.

Another more simple solution is the Lightweight Communications and Marshalling (LCM) framework. This allows decentralized communication between several clients in a network in order to synchronize sensor data output. It is based on UDP Multicast and in contradiction to ROS doesn't need a central control node. It also includes a record&playback function and interface libraries for several programming languages, but apart from that is rather spartanic.

You can look for:

1. Apache Kafka

2. AWS Data Lakes

3. or one can use google spreadsheet as real time sensor storage system with java-script and app-script

Hi sir,

Try it:

Tableau is a trending and market-leading BI tool used to visualize and analyse your real time data in an easily digestible format

Kindly check this link:

https://www.algoworks.com/blog/real-time-data-streaming-tools-and-technologies/

Best wishes..

That is a broad question, it depends on how much data you want to store and analyze at one time. Throw all the data into a data lake, try and follow Kimbal's BUS architecture if you have a lot of sources. If it is just one source of time-series data then just storing it is JSON files in a data lake is good, have a DBMS that the lake feeds for querying and reporting, you also hook your BI tool to the DBMS(like mySQL) preferably Tableau (I have used most of them and Tableau is the best out there) not real competitor. Utilize python, anaconda, Orange, jupyter for ad-hoc analysis... that is the 'best' way to do it. One of the most cost effective also. You can go buy some fancy system that says it does it all, but they never do and then everyone has to learn that one system. If you are moving a lot of data and are an enterprise go ahead and get snowflake or something similar. There are specially designed databases for time-series data, I have never used one, most companies do not have them. I'm researching them now because we might use one.

https://db-engines.com/en/ranking/time+series+dbms

The whole data warehousing, data modeling and implementation process can get quite complex. It is far too easy to go out and buy a bunch of tools and learn that most overlap or are not needed or have steep learning curves. Do not outsource the building of your warehouse you will regret it. Do not go with outdated old tools like Cognos or anything that looks old and clunky or has its own unique language, they will not keep up with your needs, you will be spending money on specialist developers when you just want to know what your sensor are doing and if they are operating within the your tolerance thresholds.

Send me an email if you want to talk further about this. I would need to know some specifics about your needs and system but in general I already mentioned the leading tools and languages. The processes of requirement gathering and building is an art in it self. Go get a trial of tableau and hook it up to a csv of your data and your will just be amazed in a few minutes, python's seaborn library is nice also but it is harder to get into the flow of EDA (exploratory data analysis) when you have to code each viz.