I have large data set(12 GB) in .dat format I have visualize the data for prediction. give me hints
Reduce the dimensionality of data and plot it as one or more 2D graphs. You can use a number of tools such as: Matlab, Paraview (open-souce, http://www.paraview.org/), Mathematica, Wolfram (with ListPlot), Meshlab (also open-source, http://meshlab.sourceforge.net/), PCL (free for commercial and research use, http://pointclouds.org/), etc. Even WEKA (http://www.cs.waikato.ac.nz/ml/weka/) will do. Additionaly, if you need to draw decision trees or any sort of network or hierarchical structure Graphviz (http://www.graphviz.org) is a good free choice.
I dont know how the data was generated because yesterday only I joined as Research Associate. My first work is to load that whole data in .dat format to mysql. the .dat file contains data without the delimiter. I am planning to convert it to .csv and use LOAD DATA LOCAL INFILE in MySQL
Okay - do you know what the data represents e.g. is it transformed video, a financial transaction log, genetic sequence data?
So basically a time-series? Sounds like you'd access the database natively (as well as needing strong visualization), so I'd go for something well-rounded e.g. R or Matlab during data exploration. Both have libraries allowing database access via JDBC, in addition to Finance libraries. Matlab has significant license costs, particularly if you start scaling algorithms; R was not originally designed for horizontal scaling, but is apparently getting better.
I've not had much experience with Python, perhaps someone else could advise?
I will suggest using Pandas (in Python). You can import the .dat file into a dataframe. This will then allow you to view, analyse and manipulate the data as you wish. There are easy to follow tutorials that can take you through this process. (http://wesmckinney.com/blog/?p=647)
We worked with 700 GB OpenStreetMap data to develop new insights into it. The tool we used is head/tail breaks, which enables you to filter out data efficiently and effectively.
Jiang B. (2015), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.
Ma D., Sandberg M., and Jiang B. (2005), Characterizing the heterogeneity of the OpenStreetMap data and community, ISPRS International Journal of Geo-Information, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1503.06091
https://www.researchgate.net/publication/273957178_Characterizing_the_Heterogeneity_of_the_OpenStreetMap_Data_and_Community?ev=prf_pub
https://www.researchgate.net/publication/270634544_Headtail_Breaks_for_Visualization_of_City_Structure_and_Dynamics?ev=prf_pub
Article Characterizing the Heterogeneity of the OpenStreetMap Data a...
Article Head/tail Breaks for Visualization of City Structure and Dynamics
That is not good idea to fully visualize, you should attempt split technique by pagination and show the information according to demand (using filters and queries with limited rows). Other way is through general description using statistic measurement or graph.
- Badges
- Science topic