Excel is the simplest without learning to program. It is not great for managing or analysing data, but is good enough for many tasks. R is written by statisticians for statisticians, and isn't really very good for data management but is great for analysis, and very powerful. Databases are the best thing for managing data. Free options include MySQL, PostgreSQL and SQLLite, and they all take some work to understand and get working properly.

Most flexible is "R" (http://www.r-project.org/) but it requires a good deal of "activation energy". When this energy is once invested it will be an ever growing fire of enlightment ;-) Grouping of data, tabulations, plotting and analyses are quite easy to do. You can also use R to read the data from different files directly (if you know the format of the files; it is simple for text files).

I second Jochen's recommendation. You can also try with similar programs like JMP (a free starter version is available on the net or PAST (http://folk.uio.no/ohammer/past/multivar.html). They are relatively simpler to "R".

Thanks very much for the suggestions. I have heard of R but thought it might be a little complex for me (I am at the stage of doing computer science 101 on Coursera) but will look into it and JMP and see how difficult they seem (not sure how much activation energy I have in me for this task at the moment). Thanks again for the advice Jochem and Jaysanakar.

Good luck Mark...Jay

Try also ASAP for excel. http://www.asap-utilities.com/. Will help you a lot. Good luck

My students switch to Access- it is WINDOWS only, but it begins rather similar to excel but has immense power. I had a Nurse learning how to use, and within a couple of weeks she starting writing ridiculously complicated queries.

Pro: it's a very very powerful tool, rather decent learning curve.

Con: windows only; not trivial to learn.

After some perusing I am going to take Steven's advice and take a path without too much resistance - I realise excel can do more things than I had assumed. Computer literacy is top of my list these days so I will look at the other programs over the next few months and consider moving into them slowly. Thanks again for all the advice.

Open office Calc is similar to excel more powerful supports on all platforms and you will have option to save data in many formats with its easy interface and it is open source so you don't have to spend money on and software.

i'd suggest to give Sofa statistics ( http://www.sofastatistics.com ) a try. R will be the gold standard to achieve. excel & Co aren't really good tools for scientific computation because their data management capacity is much to low.

Excel is the simplest without learning to program. It is not great for managing or analysing data, but is good enough for many tasks. R is written by statisticians for statisticians, and isn't really very good for data management but is great for analysis, and very powerful. Databases are the best thing for managing data. Free options include MySQL, PostgreSQL and SQLLite, and they all take some work to understand and get working properly.

Statistical software are the best option I guess. I find Prism quite simple to use and it creates great graphs. It's very simple to group values by a given column (treatment, age, sex, date, ...) to compare statistic for example. Not free though...

You can do it in Excel as well, but is not as straightforward: use columns to mark your grouping variables and then you can calculate values (averages for instance) based on the values on those columns. Simple example:

=AVERAGEIF(A2:A51,"1",B2:B51)

And of course you can call R from Excel or import Excel data into R.

Try bioconductor project, from R software, its free. Google it for more information.

I would agree with those above who suggest that Excel can do fairly advanced manipulations. A little bit of VBA and macros can save a lot of copy & paste. I saw this tutorial on excel for archaeology which is interesting too http://www.youtube.com/watch?v=ErpKblAva9E

Unfortunately not free, but I find STATA the best for managing data. However, it also has the downside that you need to learn how to write commands to use it effectively. I would personally never think of relying on Excel to manage my data. If it HAS to be free, then I would go the way of MySQL database....or OpenOffice Base (the database part of Open Office Suit). If you insist on sticking with Microsoft...then MS Access (even standalone) is probably the best choice--as it does all you need and doesn't need a lot of learning... and STATA if you are a professional and have some time to get to grips with how it works. I highly recommend NOT using a spreadsheet! Spreadsheets will lead to errors and management headaches, duplicated data, copy and paste errors, and errors with conversions, etc.

I agree that a generalized statistical software could be a good solution, because it contains many already defined functions to summarize your data.

I would suggest ADaMSoft, a free and open source software that can, also, read directly your Excel file and that can manipulate them with a nice graphical user interface.

More info on the software at:

http://adamsoft.sourceforge.net

Regards,

Marco

I agree with Jonathan that sql databases are far better suited to managing large datasets. Many software packages can work with sql databases and can do anything from simple to highly complex data manipulation. Things like grouping and sorting are the bread-and-butter of sql, it's exactly what it was designed for. However, the learning curve for sql is somewhat high, so if your datasets aren't huge, you could also stick with excel and use freely-available scripts (or write your own!) to do the work.

For a completely free solution with maximum power, extensibility, a large resource base of users, and the lowest learning curve, I agree with Fabio. I've looked at the LIMS and ADaMsoft solutions suggested above and could not recommend these.

Well, MicrOsiris is free and through grouping and internal sorting might be useful and can read in your Excel files.

At the risk of always saying the same thing, if your data set has fewer than 100 variables, MYSTAT, the student version of SYSTAT (v 12 vs. the current version 13) with fewer high-end statistical features is the equivalent of a full commercial statistical package. It is menu driven, but also like most statitical package has a programming script language based loosely on the old computer language BASIC. I believe the download includes PDFs of the manuals as well as the help files; it is worth printing out the relevant sections to learn the script language if the drop downs aren't enought. Like most of the similar menu-driven programs (SPSS is the most similar) if you use the menus, the syntax is saved in a log, which is one way to learn the programming language. It may not be quite as powerful as SAS, probably the most data manipulation tools of a stats package, but it handles all the basic tasks. It can open and save Excel and many other formats, although probably not the latest versions for some software, such as Stata and SPSS. You can find it at www.systat.com.

In my opinion a great future is out there for the open source project RapidMiner. The more people using it, the more it get tuned to our individual research questions.

There are some suggestions for drag and drop tools. My suggestions to this kind of tools: avoid them. You may use drag and drop tools to learn code if it produces code and you can see the code. But I really recommend to write codes.

For example: I wrote a comment to your question. Later, when I wanted to read other peoples comments, I realized that my comment was doubled and in different places. So I deleted one. What happend? Both were deleted. I don't know what happend and as long I don't have the code of my drag and drop-delete action and the data model of ResearchGate I'll never know what happend.

To put it simple: For data management use and learn an opensource sql-database. If you have only few records maybe it is not necessary to install a complex sql-tool. For example, you could directly use R (and learning R-code). For data management you can also use sql in R using library sqldf. (Obviously, sql is such an advantages language that even SAS implented it). Late you will also do some analysis and R is one of the top analysis tools (for me the best) and also open source. Later on you want to write reports: learn Latex. For R there is a library called Sweave which produces Latex code which produces nice pdf files. Later you can automize the repetitve things you have (analysis, reportings). Maybe, if you have huge data to manage you may consider an sql-databasee which can be accessed by R. Combined with a script language (Perl, Phyton or Ruby), autohotkey (in Windows world), a revision control system (git, subversion, CVS), reference manager (Zotero, Mendelej, etc.) you can develop you'r own working environment (a process which never ends and which you can use in every situation where you have to handle with data). I mentioned already in an other place that one of the big advantages of open source tools (especially R) is that there is a huge community which may already solved the problems you encounter or are happy to solve them.

As Ioannis and others mentioned, Excel is not an approriate tool for data management. It hinders to learn writing codes. VBA is in my oppinion much more difficult than SQL. I like to use Excel (or their oo-versions) as documentation tool and data surfing tool (using Pivot-table) but for analysis and data preperation definitivly not.

Dear Mark,

I found this question by searching the solution of the questions as you faced, like I worked in Excel for management like Sort, Pivot, Combine data sets corresponding ID

But matter is that lot of errors conduct this time and so much time consuming

Did you find your question solution?

If you found then I would like to request you to share some ideas with me

email: [email protected]

Hello Mark Horowitz,

I had to manage a lot of biological data in my PhD. We developed Pillar Science (https://pillar.science) as a way to organize and analyze easily research data. You can simply select how you want to organize your data (by treatment, by date, by sample etc..) and automatically create publication-quality charts and advanced statistical analyses.

• https://pillar.science

I hope that helps!