If you don’t have a test set then you need to use these classes in an application and get the performance metric of the application. So think “Why do you need these classes?” Then find the application that can be evaluated by feeding your classes to it.

You can use 5-fold cross validation to validate your classification results.

Article INVARIANT TO VOLUMES OF DATA, A FUZZY MULTICLASS GENERALIZAT...

How small is small? The number of examples is not the only important figure. The number of variables is critical too.

And following Gérard Dreyfus answer, there exist also methods to increase the data size. See, for example, these two publications:

Article A New Method to Generate Artificial Frames Using the Empiric...

Article A Novel Deep Learning Approach with Data Augmentation to Cla...

Hope it helps!

k-fold cross validation techniques, kappa static analysis

Labib Sadek Terrissa

The use of modern Machine Learning methods may (wrongly) suggest that this problem is new, but I believe that the best answers for your question were provided almost a century ago. Please consider reading the second chapter of the book 'The Design of Experiments', by R. A. Fisher (THE PRINCIPLES OF EXPERIMENTATION, ILLUSTRATED BY A PSYCHO-PHYSICAL EXPERIMENT). Although it was written in 1935, I believe that if you mentally replace the lady-tasting-tea with your Machine Learning algorithm "tasting" your medical data, the two problems become analogous, and the effects of small samples (small data bases) can be properly tackled with old statistical approaches, as explained in that book.

Generally four measures or metrics used for classification task.

Accuracy , Precision, recall , F-Measure.

I recommend using several different approaches to implementing cluster analysis. Then compare these clustering results themselves. Those dendrograms that occur most often in the results of clustering are more likely to be closer to the truth. Try also the Eidos system. It make a good reasonable from the point of view of experts, the results.

The Eidos-X++ system differs from other artificial intelligence systems in the following parameters:

- was developed in a universal setting, independent of the subject area. Therefore, it is universal and can be applied in many subject areas (http://lc.kubagro.ru/aidos/index.htm);

- is in full open free access (http://lc.kubagro.ru/aidos/_Aidos-X.htm), and with the relevant source texts (http://lc.kubagro.ru/__AIDOS-X.txt);

- is one of the first domestic systems of artificial intelligence of the personal level, i.e. it does not take special training in the field of technologies of artificial intelligence from the user (there is an act of introduction of system "Eidos" of 1987) (http://lc.kubagro.ru/aidos/aidos02/PR-4.htm);

- provides stable identification in a comparable form of strengh and direction of cause-effect relationships in interdependent incomplete noisy (nonlinear) data of very large dimension of numerical and non-numerical nature, measured in different types of scales (nominal, ordinal and numerical) and in different units of measurement (i.e. does not impose strict requirements to the data that can not be performed, and processes the data that is);

- contains a large number of local (supplied with the installation) and cloud educational and scientific applications (currently 31 and 154, respectively) (http://lc.kubagro.ru/aidos/Presentation_Aidos-online.pdf);

- provides multilingual interface support in 44 languages. Language databases are included in the installation and can be replenished automatically;

- supports on-line environment of knowledge accumulation and is widely used all over the world (http://aidos.byethost5.com/map5.php);

- the most time-consuming computationally, the operations of the synthesis models and implements recognition by using graphic processing unit (GPU) that some tasks can only support up to the solution of these tasks is several thousand times that really provides intelligent processing of big data, big information and big knowledge;

- provides transformation of the initial empirical data into information, and its knowledge and using this knowledge in the solution of classification problems, decision support and research of the subject area by studying its system-cognitive model, generating a very large number of tabular and graphical output forms (development of cognitive graphics), many of which have no analogues in other systems (examples of forms can be found in: http://lc.kubagro.ru/aidos/aidos18_LLS/aidos18_LLS.pdf);

- well imitates the style of human thinking: gives the results of the analysis, understandable to experts on the basis of their experience, intuition and professional competence.