I think there are some information criteria that give a measure of the quality of the model order. For example: Akaike Information Criterion (AIC). There is no exact criteria. However, they may provide useful information. They attempt to balance the complexity (or length) of the model against how well the model fits the data.
Thank you Chacón and Professor Giannella, the interest of you made me look for a POSSIBLE solution.
I searched on Marple's* book which I have here at home and in some texts on the web and finally I reached an algorithm that calculates the curve formed by error values depending on the model order.
I do not know if the code is 100%, but the final figure shows the famous knee, it can be noticed the trade-off between the error and the model order.
Basically used the AIC
AIC [p] = log (fpe) + 2 * order / N; where fpe is the final predicted error and N is the signal length.
I'm attaching a test code with a simulated signal (simple sinusoid) to show the code. I'm also attaching a figure with a fpe curve for a given RRi signal during rest.
*S. Lawrence Marple Jr. Digital Spectral Analysis with Applications. Prentice-Hall, 1987
Which model order do you choose in the figure above?
I was also wondering a better criteria to find the best trade-off from that curve than just visual decision. Maybe a variation smaller than 5% in the successive "fpe" values.