Firstly, I guess much more details need to be known to help you accurately, such as how these methods were implemented, what parameters and termination conditions were used.

The initial root cause that comes to my mind is that with smaller data sizes there's simply not enough guidance for the learning model (especially with a higher number and complexity of features). This will enforce more iterations, and thus running time, for reaching convergence. However, as I mentioned, more needs to be known to form a robust explaining theory. But, you might want to investigate this by looking at the learning curves.

Hope this helps.

Remember that the "big O" notation always refers to the *worst-case* behaviour, not the "average" or "typical" behaviour. So, ask yourself how many computations the algorithm would take when faced with an especially difficult instance of a given size.

I appreciate it Sir Abdipoor,

It's good to see we're on the same page about this matter.

I've used R2 criterion throughout the iteration process, however, for all the tests I've used R2=0.95 to be met.

For data sizes of 30 samples the model even couldn't run successfully. I increased the data to 50 samples and finally the models were running good, but the R2 was about 0.2, and as i increased the data size further, the R2 of the interactions was getting better and closer to the limit.

All in all, for lower data size infinite time was required to reach R2=0.95 but for bigger data, only a fraction of seconds was needed

Sir Letchford,

Thanks for your response,

I am aware of that definition, but the point is, this matter has been raised by the peer reviewers of my paper, so I'm skeptical to report the reality not writing something to simply justify the situation.

I'm thinking could this results be reliable?

Concisely, have you seen such occurrence throughout your experience?

For further clarification please refer to my previous response to Mr Abdipoor

Appreciate your time

Yes, it can happen that, in practice, an algorithm runs faster on a bigger instance. However, that is not what the "big O" notation measures.

I understand, now I'm positive that it's practically possible.

In fact as it can be seen from the figure i shared, in the last stage of data size the timing tends to slightly increase, and in my eyes, i suppose if we extend the size fo data extraction further than this one, the timing is likely to increase which is in line with what big o notation talks about

Anyway, thank you very much

Stay good

Alan Zandie

The question itself is a little odd, in the sense that we rarely talk about "worst case" runtime performance in learning systems. Typically, once you've learned something and deployed the system, the runtime is constant O(1). Even while you are learning something, you typically are learning against data of which all the elements are the same size (e.g., everything has 100 features), which again is not a situation in which Big O is appropriate.

Based on your replies to the above questions, it looks like you are looking at "what is the number of training data elements do I need to achieve a goal accuracy". This question is more appropriately addressed by Valient's (~'86) learning theory "probably approximately correct".

So, in short, unless your reviewer wants to know how processing time changes based on the number of features each training element contains, Big O is not relevant.

One last point, worst case performance is typically done looking at the algorithm itself, and not the data ingested. So, for example, if the radial basis function were calculated based on every combination of each data element ingested so far, then the runtime would grow exponentially (even in the best case).

Thanks Mr Allen,

I understand your point,

So, in conclusion, big o notation is working once all the features are given, and the data samples vary.

Say, if the features are sorted in different rows and samples are given through columns, increasing the columns each time can be used for big o notation.

Anyway, I don't want to dwell on the big o notation primarily, cuz i believe it fails to understand my case as it's discussing the runtime versus features.

I have clarified the results in a detail manner in my paper, and i linked the plunge in runtime by increase in data features to the complexity of the structure (hydrocyclone) itself, and the easiness of models in comprehending the target under bigger number of given inputs.

Thanks for your time