A list is a dynamically allocated structure where each list element may be any object. A data.frame is in fact a list with the restriction that each list element must be a vector, and some methods defined that let you access it like a matrix. Any data stored in a data.frame can be (and is) stored in a list, but it is not the case that every list can be represented as a data.frame.
An array is a vector with a dimension attribute, so I will speak only of vectors. A vector is an enumerated collection of items of the same type. That is, you can have a vector of integers, or a vector of characters, or logicals, numeric, etc. But you can not have a vector v where the ith element v[i] is a character and the jth element v[j] is a double, nor any other combination of types. Because an array is a vector, the same restriction applies to arrays.
And now we come back to data.frames, which we often think of and treat like arrays. But, a data.frame, df, is a list of vectors, v_i, and so it inherits some of the flexibility of lists. In particular, a data.frame looks very much like an array, but each column is a list element in the data.frame. Since the columns are separate objects in data frames, it's possible to have the ith colum contain a character vector and the jth column a logical vector, and so on. Thus, a data.frame is a convenient way to store mixed data in R.
The additional flexibility comes with the usual costs associated with lists. That is a list takes up more space in memory compared to a vector with the same data. And, it takes time proportional to the number of elements in a list to access a given element and constant time when stored as a vector. A more thorough discussion of the cost benefit analysis of linked lists vs vectors/arrays can be found in most introductory computer science text books.
A list is a dynamically allocated structure where each list element may be any object. A data.frame is in fact a list with the restriction that each list element must be a vector, and some methods defined that let you access it like a matrix. Any data stored in a data.frame can be (and is) stored in a list, but it is not the case that every list can be represented as a data.frame.
An array is a vector with a dimension attribute, so I will speak only of vectors. A vector is an enumerated collection of items of the same type. That is, you can have a vector of integers, or a vector of characters, or logicals, numeric, etc. But you can not have a vector v where the ith element v[i] is a character and the jth element v[j] is a double, nor any other combination of types. Because an array is a vector, the same restriction applies to arrays.
And now we come back to data.frames, which we often think of and treat like arrays. But, a data.frame, df, is a list of vectors, v_i, and so it inherits some of the flexibility of lists. In particular, a data.frame looks very much like an array, but each column is a list element in the data.frame. Since the columns are separate objects in data frames, it's possible to have the ith colum contain a character vector and the jth column a logical vector, and so on. Thus, a data.frame is a convenient way to store mixed data in R.
The additional flexibility comes with the usual costs associated with lists. That is a list takes up more space in memory compared to a vector with the same data. And, it takes time proportional to the number of elements in a list to access a given element and constant time when stored as a vector. A more thorough discussion of the cost benefit analysis of linked lists vs vectors/arrays can be found in most introductory computer science text books.
For large data sets package data.table can help speed up your code. A data.table is mostly compatible with a data.frame, but avoids as much as possible copying your data when you manipulate it (it uses references) and keys can be set to make access to individual rows faster. This is not a native R data structure, but it's use is becoming quite popular. If you use data.table be aware that copying by reference instead of actual copying can alter the semantics of your code, so be careful when using them.
Normal syntax works mostly as would be expected based on how data frames work, data table's special syntax can be used to make your code more compact, and faster.
I fully agree with all of Shane's very thorough comments. I will just add two points: (1) data frames are indeed lists of vectors. But they have the restriction that all the vectors must be of the same length. There is no such restriction on list elements. (2) List elements can themselves be lists, or any other R object, for that matter.
I use lists extensively in my work. There's definitely a significant learning curve involved in using them effectively, but it is well worth the effort. You will find yourself using the powerful lapply, sapply and mapply on lists to perform all kinds of manipulations in single lines of code that would otherwise require cumbersome and much slower (sometimes nested) loops.
One last piece of advice, though: use the simplest data structure that will get the job done, as Shane implied.
Lists are the most versatile in terms of the elements that they can hold. A single list can hold many different data types. However, a list can be a lit bit of a pain to perform operations on. Many functions in R operate on matrices, data frames, or vectors, but don't work on lists.