We download and process observed data (some 20 variables), with various formats, from web, ftp, webservices, .. We load it into MySql and we generate compiled csv files.
So far everything works with python scripts sort of manually, and we want some workflow to orchestrate and visualize the advances, and produce weekly or daily updates to our compiled results,
I see in the web there are dozen tools available, but I would appreciate to get some experiences on scientific or generic workflows, with (web?) visualization, and hopefully easy to configure and monitor and visualize .. Maybe its too much to ask :)
Hi Francisca,
Could you please give some more information about your specific problems?
Managing a database would be possible using SQLAlchemy (http://www.sqlalchemy.org). Parsing your CSV files will probably require customized scripts, because nobody can know your data structure. It could be helpful to use modules like pandas (http://pandas.pydata.org) for working with large data tables. The data could be visualized by using matplotlib (http://matplotlib.org). You could have a look at flask for the construction of a website (http://flask.pocoo.org).
Good luck!
I do a similar thing, collecting data from instruments measuring the particle size distribution, doing some calculation and presenting them on the web and storing the data and results for further reference. I wrote everything myself, calculation in octave, graphics in gnuplot, web in PHP, scripting in bash and perl, everything is started automatically from /etc/crontab. For download I would certainly use wget, in my case the software in the instruments pushes data to a computer to which I can connect via sshfs and grab the data. If you already have the scripts that can do it manually. My PHP code has the right amount of inteligence, it knows where the generated plots reside. According to their names and directories it knows the dates and meaning of the plots. It can thus offer directories and display all images found. All this needs just 126 lines of PHP. (I am thinking about adding jqPlot for its interactivity, the users would be able to choose between static gnuplot images and the interactive ones)
I am not sure if you want to automate the workflow you have or if you want a workflow that evaluates your csv files.
Anyhow, a good start is the pandas modules which is basically written to manipulate data stored in csv files or SQL databases. Pandas can read SQL_queries into a dataframe object for easy manipulation, hence you have a bridge either between SQL pandas or csv pandas. From there you can easily use numpy/scipy for data manipulation.
From what I understand in regards to visualization, its probably best to have a look at the glue module (http://www.glueviz.org/en/stable/).
It allows for an easy comparison of datasets and can be used to monitor changes. I think that is what you are after.
Cheers
Thank you all for the answers, we are using some of the tools you mention for individual tasks (crontab, wget, panda, matplotlib, , but we are looking to automate- orchestrate the existing steps, to be able to launch tasks in parallel, monitor, receive emails if errors occur.. We could do ad-hoc code like Zdenek Wagner, but we were thinking on some structured framework to help :)
In terms of web visualisations, it is easy to make on-the-fly matplotlib visualisations served by Django, or indeed any of the main Python web frameworks, it is something I have done many times.
There is no pre-existing workflow that I know of, but on a fundamental level, if matplotlib can draw it, then Python can serve it on real time.
This link provides a simple example, you can of course use Django to include user defined variables to define your plot conditions.
https://scipy.github.io/old-wiki/pages/Cookbook/Matplotlib/Django.html
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python