Hello everyone,
I have a sequence of files which named like plot_1.txt, plot_2.txt… plot_i.txt… plot_40.txt. The first column of these files is the time. The second column is the value of my interest. All these files have the same first column but different second column. The contents of these file is like:
Time Numbers
1 33
2 127
3 227
4 283
...
Now I want to use Gnuplot to get a graph of Time (first column) vs. mean value of Numbers (second column) and with errorbars on the graph.
Do I have to process these files manually to get the average value together with the Second Deviation of the second column, and then use Gnuplot to plot the file?
If Gnuplot cannot give the mean value of the second column and second deviation directly, and I need to obtain these two values myself. Because there are too many files for manually calculation, could you please give me some advice on how to do this fast and accurate?
Any help would be much appreciated!
Dear Liu,
I don't know it is useful for you right now or not.
For the beginning, there is a command in Gnuplot that is called "stat". For example, you have a file that its name is "myfile1.txt"
In terminal go to that directory and open up Gnuplot and type:
stat "myfile1.txt"
then it shows a lot of information about that file like mean value of each column, standard deviation and a lot of more things.
For inserting error bars, the usual way is to calculate this one:
STD-D/sqrt(number of iteration) or STD-D/sqrt(number of iteration -1)
then you write it in the third column. Personally, I use a simple python code or just a bash script for the value of error bars.
For plotting use sth like this:
plot 'myfile1.txt' u 1:2:3 w errorbars (It has a lot of options)
The main concern is finding an automatic way to do all of this stuff.
I strongly recommend that copy all of your files to a directory then using awk command for deleting unnecessary columns, then doing all this stuff in just a bash script with a simple loop. It's not hard at all. (Of course, you can do all of these in Gnuplot somehow)
At the end, I found the best solution couple of months ago which is using mathplotlib from a bigger module is called pyplot in python that is so great.
I hope this is useful for you.
Dear Dr. Waldemar Koczkodaj,
Do you mean I need to calculate the mean value of the second column of my data file, together with the second deviation, by myself? And I will get a file like:
Time average of Numbers sd of Numbers
1 33 10
2 127 22
3 227 30
4 283 35
……
100 50 6
Then I use this file to plot the relationship of Time vs. Average of Numbers, and with errorbars on the graph?
Could you please give me some more detailed information? And could you please explain what do you mean by “R has loops so it should not be hard to do it”? Sorry, I cannot quite understand that.
Thanks very much for your help!
Dear Liu,
I don't know it is useful for you right now or not.
For the beginning, there is a command in Gnuplot that is called "stat". For example, you have a file that its name is "myfile1.txt"
In terminal go to that directory and open up Gnuplot and type:
stat "myfile1.txt"
then it shows a lot of information about that file like mean value of each column, standard deviation and a lot of more things.
For inserting error bars, the usual way is to calculate this one:
STD-D/sqrt(number of iteration) or STD-D/sqrt(number of iteration -1)
then you write it in the third column. Personally, I use a simple python code or just a bash script for the value of error bars.
For plotting use sth like this:
plot 'myfile1.txt' u 1:2:3 w errorbars (It has a lot of options)
The main concern is finding an automatic way to do all of this stuff.
I strongly recommend that copy all of your files to a directory then using awk command for deleting unnecessary columns, then doing all this stuff in just a bash script with a simple loop. It's not hard at all. (Of course, you can do all of these in Gnuplot somehow)
At the end, I found the best solution couple of months ago which is using mathplotlib from a bigger module is called pyplot in python that is so great.
I hope this is useful for you.
Dear Mohammadreza Niknam Hamidabad,
Thanks very much for your detailed information!
I used a similar way as you described, that is, calculated the mean value and standard deviation of the data and write them to another file, and then plot it using Gnuplot.
However, I know nothing about awk command, could you please give me more information about it? And mathplotlib looks great, maybe I can try it later.
With My Best Regards,
Liu
Dear Wenqiang Liu,
awk command:
Please consider this situation: you have a file let's call it information.txt. This file has 7 columns like this:
line1: John has 3 dollars in his pocket
line2: Jules has 15 dollars in her pocket
line3: Sara has 210 dollars in her pocket
and so on.
You need to extract the amount of money they have. you simply go to that directory and then open up a terminal (in the corresponding path which the information.txt file exists) and type:
awk '{print $3}' information.txt > results.txt
then awk command will print the third column of the information.txt file into the results.txt file (columns are separated by space or/and comma I'm not sure so google it, please. The first column is: John Jules Sara, the second column is: has has has and the third column is the amount of money which they have.
Please remember that this command is so much more powerful than what I described it. You can do a lot of things with this command. Its basic job is processing or analyzing text files—in particular, data files that are organized by lines (rows) and columns.
I hope this is useful for you.
Best regards,
Mohammadreza Niknam Hamidabad
this is a useful link:
https://linuxconfig.org/learning-linux-commands-awk
Dear Mohammadreza Niknam Hamidabad,
That is very helpful. Thank you very much for your help!
With My Best Regards,
Liu
dear Wenqiang Liu ,
the solution is given in two steps:
a. merge the columns of the 'plot.*txt' files in a single file 'merged.txt'
b. use a plot script in the form: plot 'merged.txt' u 1:2:3 with yerrorbars
where, 1 the time, 2 the mean and 3 the standard deviation of your Numbers. As you can see below these values can be computed in gnuplot directly.
In step a, use the following bash script:
#!/bin/bash
cat plot_1.txt > merged.txt
for files in plot_*; do
if [ $files != "plot_1.txt" ]; then
paste -d' ' merged.txt temp.txt && mv temp.txt merged.txt
fi done
in step b, use the appropriate plot command :
n=40
plot "merged.txt" u 1:(sum[col=2:n+1] column(col)/n):(sqrt(sum[col=2:n+1] (column(col)-(sum[col=2:n+1] column(col)/n))**2)/(n-1)) with yerrorbars t "title"
PS. in the bash script i use space as a delimiter, if you use comma replace d' ' with d','
I assume that you can use the gnuplot 'stat' routine, as suggested by Mohammadreza Niknam Hamidabad , but i guess that the double summations inside the plot command look more 'open'.
Dear Anastasios Gotzias
Thanks very much for your reply!
I am not very familiar with bash script. Maybe I should learn some basic knowledge first. I will try that later.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs