First of all, I rather doubt about the Conant's Rule that you mentioned in the question. Any literature for it?

Secondly, too much data? do you mean the number of samples or the number of dimensions (features, variables)? The size of the data set is depend on the problems you are trying to solve. An extreme example , assume that the input dimension of a problem is infinite, then the number of samples you need is also infinite.

In practice, the size of the data set is often limited by some problems such as computational cost and overfitting.

Zhenyu Wang -- Conant's Rule is very well known in multivariate statistics, where every cell of the matrix is 'required' to have at least 5 entries in order to allow for adequate sample size ... I'll dig back into my stacks and get the proper attribution, but you can easily convince yourself of its validity by playing with a small dataset based on a single binary variable, and then build up.

Regards,

Doug

As an example, take the Internet Bubble Burst in the stock market -- would you use a dataset comprised of the activity of the NYSE since its inception, in order to investigate a 6 month period around 2000? Where would your dataset start and end, and why? And if you *did* utilize the entire recorded history of the NYSE, what kind of effect would you expect that to have on the ability to recognize the significance of events that are known, historically, to be noteworthy? 'Smoothing' of effects is a well-recognized artifact of larger data 'windows' -- compare the trace of a 5-day (one trading week) moving average in the NYSE to that of a 20-day (one trading month) to the 100- and 300-day moving averages: spikes that to a day-trader were highly significant in either the 5- or the 20-day traces, are completely smoothed away by the effects of accumulating so much more data to bury them in. So the question is not simply 'how much data do I use?', but also 'how much data, in that set, do I investigate in a single instance-at-a-time, my sampling window?'.

Very good question indeed, in my opinion there are two basic reasons why the data can be too much:

1) Overfitting: this is the most common and well known one: too many variables will generate models that work for simple algebraic reasons with no conection with reality. This is obvious in the case of trivial errors (for two ponts a straight line passes by definition with r=1, for three points a parabola) but even for less trivial errors (chance correlations by the fact that a p < 0.05 is almost for sure when you have 100 descriptors) and not-so-trivial points (indeterminacy of the model when the regressors are each other correlated).

2) Scaling problems: too wide a range of variation will generate aspecific correlations between variables, by the way the fact of a very diffeent history and consequently societal constructs in Italy coming from the North-South gradient will make any pair of varaibles measured along this gradient t o correlate, so the number of olive trees (more frequent in the South) will be strictly negatively correlated with the number of theatres (more frequent in the North) , the presence of a global structuring of the analized data set (much more probable in large data sets having inside a lot of heterogeneous situations, i.e. units coming from (unknown) different poplations) provokes a structuring (i.e. clustering) of the big data sets generating a lot of correlations between variables that can be erroneously considered as 'native' or generating from the similar meaning of the variables while, on the contrary, they are only an image of the co-existence of heterogeneous groups inside the same data set.

While type 1) pathologies can be quantified in an easy way, type 2) are very difficult to check and estimate.

my favorite example of overfitting with lots of data is Ptolemy's geocentric hypercycle method for describing the orbits of the planets (and sun and moon) - it was far more accurate for predicting positions of celestial bodies than copernicus' heliocentric model for decades (until kepler changed the orbits into ellipses) - the better model is the one that explains more, not just the one that predicts more accurately

my opinion is there cannot be too much data; there can only be too wide a range of data, or too weak a condition on the acceptability of data (all of the above examples fit into these categories) - if more data causes a problem, then you need to examine your assumptions about what data is relevant and how to collect it (this is not always easy; for example, in pattern recognition problems, there is the phenomenon that choosing too many features for making pattern discriminations can lead to very poor distinction results; a judicious choice of more or less independent features is needed)

for conant's rule, check out the paper by don foley from the early 1970's - it was also about the number of samples a discrimination problem required for reliability:

Donald H. Foley,

Considerations of Sample and Feature Size,

IEEE Transactions on Information Theory, Vol.18, Issue 5, pp.618-626 (Sep 1972)

I think the question may need to be specified further. My thoughts on the question are as follows:

1. The right data is more important than the amount of the data. For a certain problem we are trying to solve (or analysis by data), we can divide all data into 3 parts: the relevant data, the irrelevant data and the redundant data, where the irrelevant data provides no useful information to the problem solving, the redundant data are those which provide no more information than the current data.

2. Using more data may help us to include more useful information, but may also decrease the percentage of useful information, which often mislead the model to give incorrect conclusions.

OK, back to the question here. I think we are talking about the amount of the relevant data, which means we assume all the variables or features we used are relevant to the problem, then we discuss about the amount of the samples. Otherwise, 500 x the overall-state-set-size of irrelevant/redundant data is useless. But how do we know the variables we used are relevant or enough. For example, in cancer modeling, we often see some genes or factors effects on some of the people, but look no effect at all on some other people.

3. Back to the example of stock market data, in fact, whether the price of a stock inthe next minute is related to its historically price (no matter 5 mins, 5 days, 5 months or 500 months) is also arguable. “In a world where observations are drawn from a non-ergodic stochastic environment, past data cannot provide any reliable information about future probability distributions. -- see (Davidson 2002: 187) and (Davidson 2002: 187)”

4. Finding a statistical significance is just a part of the story, doesn't always mean we have a right answer to problem. A statistical significance based on a incorrect data set or a incorrect assumption is useless.

"It is easy to lie with statistics. It is hard to tell the truth without it.--by Andrejs Dunkels".

5. About the data mass problem, I think the difficulty is not about the amount of the data we need to use, is about how to identify what is the right data for our problem from a mass of data. What we need is the right assumptions, plus a good understanding of the problems. Otherwise, it is easy to be cheated by the large data, as it can bring you anything you want.

Please correct me, if I am wrong :).

Regards,

Zhenyu

Alessandro --

Many thanks for bringing up 'overfitting', makes perfect sense; 'scaling' is going to take a bit more thought, in this particular instance, but is a wonderful seed, again my thanks!

Christopher --

First off: my sincere appreciation for having run down that citation!

Next: "...there cannot be too much data; there can only be too wide a range of data, or too weak a condition on the acceptability of data ... if more data causes a problem, then you need to examine your assumptions about what data is relevant and how to collect it"

By 'range' do you mean the number of categories that the data is mapped into? I.e., instead of recording the exact-to-sensitivity-limits values of a real variable, it is mapped into, say, 5 ordered categories ... or 10, or 100, or 1000, and the higher the number of categories, the wider the range?

I'd very much appreciate an example of what you would consider 'unacceptable data'; could this be associated with something like 'dependability of source'? I would have expected that kind of situation to be handled by some kind of veracity-weighting factor, along with the judicious use of some form of Dempster-Shafer combination-of-experts incorporating evidence from other sources. So I think I don't understand how that works ... an example would help tremendously, thanks in advance!

"...what data is relevant ..." is, of course, exactly what I'm trying to figure out how to determine; I almost wonder if this isn't something that can only be told in retrospect?

"...how to collect it." -- as in 'how many data points are generated in a given support interval'? This will actually be a very significant consideration going forward (collection of new data), but how can I apply it to the determination of how to sift data already collected and waiting to be processed? Possibly some form of homomorphic simplfication/compaction, reducing the dataset complexity but (unavoidably) also resulting in information loss ...

Wonderful ideas, however! Many thanks, and again -- greatly appreciate the citation.

Zhenyu --

"...divide all data into 3 parts: the relevant data, the irrelevant data and the redundant data..."

Absolutely! The 'redundant' I can determine ... but 'relevant' and 'irrelevant', as I previously wondered, seems to imply that I already know what I'm looking for; and, worse, is it possible for 'relevancy' to, like quark-color, change states? I.e., data that has been relevant for a time, becomes (for some not-yet-known reason, maybe?) irrelevant?

"...Using more data may help us to include more useful information, but may also decrease the percentage of useful information..."

Are you suggesting 'noise'? Would this, then, fall into the relevant-vs-irrelevant category?

"...500 x the overall-state-set-size ..." --- *5* x, otherwise we'd be absolutely *swimming* in data, and wouldn't THAT be a shame!

"... how do we know the variables we used are relevant or enough ..."

By whether we find meaningful and repeatable results; if not, then we widen-or-otherwise-modifiy our 'information neighborhood' -- replace variables, include factors heretofore considered irrelevant and/or random, contemplate 'emergent behavior' ... damn, *I* don't know!?! Pull out the Ouija board! I did that once ... didn't help, mind you, but I was out of ideas at that point.

"For example, in cancer modeling, we often see some genes or factors effects on some of the people, but look no effect at all on some other people."

Well, of course in as rich an environment as anything having to do with DNA, and all of the kinds of things that are being shown to be able to get in there and muck around with it ... damn, you picked a hard field, Zhenyu. Very best of luck.

"...whether the price of a stock in next minute is related to its historically price (no matter 5 mins, 5 days, 5 months or 500 months) is also arguable ..."

I'm not claiming that this is anything to put your money on, mind you, but when I ran an analysis out to 2 trading weeks (10 days), and considered all possible relationiships among those 10 closing prices on the behavior of the 10th day, all the way down to 'everything else completely disconnected', the most accurate predictor was the behavior of the day before ... take that away and everything fell to pieces, it was the behavioral kernel. Go figure. Again -- I'm NOT a financial advisor, I don't even PLAY one on TV!

"Finding a statistical significance is just a part of the story, doesn't always mean we have a right answer to problem"

Absolutely!

"About the data mass problem, I think the difficulty is not about the amount of the data we need to use, is about how to identify what is the right data for our problem from a mass of data. What we need is the right assumptions, plus a good understanding of the problems. Otherwise, it is easy to be cheated by the large data, as it can bring you anything you want."

Wonderful summation. Thank you.

Regards to all,

Doug

Providing more than a certain amount of information, you will overload the communications channel beyond the available bandwidth; i thik this would be the most critical limit. Also, a consideration that came to my mind derives from an article by R. Ackoff (Management MIS information systems; Management Sciences dic 1967, v14, n 4)

Thank you, Sr. Molino ... however, I'm not sure that channel capacity is the limiting factor in the scenario I'm envisioning. Consider again the situation where you have available to you the complete NYSE dataset, from its inception, and you are interested in its behavior around the time of the Internet Bubble collapse; would you use the entire dataset or, if not, where would you draw the line saying "anything prior to this point will not significantly impact the analysis", and how would you justify it? If I, instead, chose to use the entire history of the NYSE, what kind of differences in our analyzes would you expect to see, and why? It would seem, on reflection, that analyzes obviously suffer from having too little data on the one hand, and, I'm assuming, too much on the other, so there would seem to be some kind of 'sweet spot' in the size of the dataset to be used for analysis ... how can this be characterized? What underlying aspects of the system would drive this? It would seem to have something to do with systemic variability, but my feeling is that that is only the beginning.

Regards,

Doug

Channel capacity would be the ultimate barrier (as I see it). However, in you case, I would consider cost/benefit analysis. That is: what is the cost of getting, processing, cleaning, analyzing, etc. of an additional unit of information, vs the benefit in the refinement of your results derived from this additional information. Attached is a diagram showing this idea.

Here are some provisional answers

i didn't exactly run down that citation - i worked with the author way back then (it was not called conant's law then either, but that might originally be a term from statistics instead of pattern recognition)

i am not thinking about the number of categories foro range, but the disparity of the data (more like scope) - for example, if i were trying to measure something about density profiles of sphere-like objects in nature, too wide a range could be to include raindrops, soap bubbles, rocky planets, hailstones, walnuts, and apples - in this case, there are obvious categories, but that is not always true (especially when the only data recorded does not define the categories

many collection processes throw out what they consider to be outliers, without a corresponding sensitivity analysis to verify or refute their effect on the final outcome - for me, though acceptability is more than that - it is your definition of what data you want to consider, and it certainly also includes source credibility

basically, every analysis that has any design decisions in it should be accompanied by a sensitivity analysis to show how much (or whether) that decision actually affects the results - then you can decide which decisions are irrelevant and which ones matter so much that further analysis is needed

in this sense, some conclusions cannot be drawn until after some of the analyses are completed, and some analyses cannot be designed until others have completed

hope this helps

Hi all!

very interesting question and discussion!

I am by no mean an expert here,just guessing, but bare with me please =)

.. About the question of which data window to use from NYSE

So I would guess that the answer depends on the question what you want to find out - like, future long-term trend or reasons for the Internet Bubble collapse thingy (to be honest, I am ignorant of this event or NY stock in general..)

I would then intuitively go with reciprocity : how long of a trend you want to observe, at least that long time window you use. That's the lower limit.

Doug, you mentioned that you did some analysis about correlation between the results on the 10th day and the values of the previous day? So 10th day is one day, and your lower limit is that one day.

Now for the upper limit, couldn't you do something like sensitivity analysis (sorry, i don't know if this is an exact term). But you test the strength of a correlation and prediction power when you include more and more data (more and more days in this case), and at some point it shouldn't be relevant anymore? More data inclucion shouldn't yield in more precise or more accurate predictions.

Does this make any sense?

I am curious because I will also (soon) deal with some accumulated-over-time data, and I am also confused with the precission/magnifying glass (should I look at the daily-basis or smoothed values?) and the importance (10 days or 10 years worth a data?)

Thank you again, Sr. Molino ... I understand your point about channel capacity, and very much appreciate your presentation of the cost/benefit curves -- that type of analysis is always relevant to an investigation.

Greetings, Nina, and welcome! Instead of what you call 'reciprocity' (as much prior data as the trend to be observed), look into what I have referred to (as I'd been taught) as "Conant's Rule" :: 5 x total_state_set; e.g., if you have 3 variables for which you recognize 2, 4 and 6 states each, then the total_state_set would be 2*4*6 == 48, and the Rule than says that you should observe no fewer than 240 data points. The reasoning behind this rule, from multivariate statistics, is that it allows 5 chances for each of the possible states to occur, and thus sets a lower bound on the amount of data that needs to be taken.

But what about the other end? It seems clear to me that it is possible to swamp an otherwise significant event by burying it in an avalanche of data, smoothing over its trace in the wash and the noise. But how to tell when that limit is being reached?

I'm looking for some kind of measure that will help me determine when my data window becomes too large, when I've reached a level of sufficiency and any more would be counter-productive. I can approach it like this: I can construct a significant event then continue to add data until the significance of the event falls into insignificance. But I would have thought that some kind of measure would have bee developed that addresses this, and expresses a maximum data size that does not need to be exceeded. If not, then how do you know how much data to gather? Is the size of your data set by chance, convenience or cost? Is it enough to simply exceed the absolute minimum, and then it's a matter of cost/benefit as to how much more you'll gather?

Well. Time to go play some more with the data and let it talk to me.

Regards,

Doug

I am no statistician but neural network analysis of large data sets ought to have a view on the issue of “how much data is too much”. I bet it is the issue of computing resources and the statistical fit required for the result to be useful. One approach to reducing the likelihood of bogus results and improving fit is to (in Chemical Engineering), use relationships previously derived to group the data before performing the analysis. E.G. relationships developed according to the Buckingham PI theory of dimensionless groups. They are essentially logical groups of factors and it has been found that neural net analysis often improves, if the data is pregrouped.

Thanks, Gary -- I hadn't run across the theory of dimensionless groups, but looking at it I think i see what's going on. Neat.

I agree that the amount of computing power readily available is going to drive a realistic determination; I was just wondering if there was something more tailored to fundamental characteristics of the basic variables, rather than parameters of the investigation ... something akin to how the effective minimum was determined -- by looking at the minimum acceptable variability over the cross product of the state sets.

Another way to look at it is to take the minimum amount of data necessary to resolve the effect (effect size) you are trying to observe, let's call it x.Then add to this number the number of experiments that are likely to fail (i.e. subjects dropping out, equipment failure...), let's call that %fail and you should get the ideal dataset size.

So you would use #Data = x + x * %fail

Of course, some unforeseen issues could arise in which case you could increase your %fail number just to be sure. Anything above this number might be considered too much data in that you would be throwing money on new data that are necessary to answer your scientific question, and you might start overfitting.

I hope this helps.

Remi

Depends on the purpose and method of analysis