In my opinion, for your task (only 43 records for 8 features with some missing data !) it is impossible to use regular ML methods, such as SVM, NN, Logistic Regression, etc. Try to use Decision Tree , they work well with categorical attributes.

In my opinion, for your task (only 43 records for 8 features with some missing data !) it is impossible to use regular ML methods, such as SVM, NN, Logistic Regression, etc. Try to use Decision Tree , they work well with categorical attributes.

Hi Lubna,

Please find the answers below.

1- Which theme is better for analysis, with more attributes (2) or with less attributes (1) ?

Having large number of instances is preferable in general cases.

2- Noting that i want to predict 2 things students MSc AGPA and Each Msc. Course Grade]

Do you mean that you have two features (classes) to predict? If so, then you have a multi-label learning process. At this stage, you need to apply algorithms that can handle this sort of learning (multi-target regression).

3- For the last 7 students, since they dont have the last 2 courses records, I am planning to make all the 43 records with six cources and i will delete the last 2 courses of the old batches. Whats your advice?

if this will not affect on your research's goal, then go with it.

4- Another question, in substituting missing records, what is better for small datasets, i.e. to substituion missings with mean or median? or what is the most reliable way to substitute with for smal size dataset? [Link to a papers related to that will be much appreciated]

In general, for a normal (symmetric) data distributions, the mean can be used, while skewed data distribution should employ the median. The literature has many references about that, I recommend you finding them yourself.

5- I know that there is no classifiers that could be suggested for a certain datasets, but since i am going to predict ordinal and continues numbers, and based on your experiance which classification algorithims are willing to handle small size datasets? and which descriptive statistics or visualizationtechnique that can tell me this classification algorithim is best for your case?

See the answer of the second question. (Note that ordinal features = numeric features.)

6- Last, for the scores of the prediction accuracy, how to judge if its good acuracy rate or not? some are using the probabilities of occurance of that grade number, but what if the number is continuase not ordinal ?

For classification problems (nominal class), the higher accuracy the better (above 70% is considered to be reasonable). However, accuracy alone is not enough, you need to consider another evaluation metrics like ROC, Kappa statistic, etc. With regression problems, accuracy can not be used, we basically rely on RMSE and correlation coefficient, for instance.

HTH.

Dr. Samer Sarsam

I agree with my colleague Sergey, and Samer.

Best regards

Thank you all for your valuable participation.

Dr. Sergey,

How about running DT , ANN, & Multi regression And select the most accurate one? Y

That could be tested for predicting the course grade (A,B,C, or F). But I think for the AGPA only Multi regression and ANN can be tested for that. What do you think?

I am concerning about that because before starting any analysis I want to understand and be able to explain why I used this algorithm not others.

Also I need to know your opinion about the attached image for the 2 type of dataset arrangements. Please let me know your feedback about it. And for the rest of you don’t mind.

( btw. The data set missing sells were about 7 cells.)

Dr. Samer,

thank you for answering all questions in very clear manner.

Regarding your answer for the second question, kindly note that I have 2 different aims, the first is to predict students final AGPA ( continues value), and the Second is to predict each course grade including dissertation (letters ABCF or numbers 1,2,3,4) . In dataset arrangement 1 I won’t be able to see which course , course grade, or instructor that could affect the prediction of the last aforementioned grades and in the second one, it will be hard to point to course or instructors as an indicators for students grades which I am Interested to point to it as well.

I am trying to make my project more utilizable by my institution. So please reelaborate on point 2 based of the aforementioned facts.

Another questions, when you said instances, do you mean the number of records in each variable or the number of students and their corresponding records?

And would that mean you prefer less attributes with more instances ? Like dataset structure 1 in the attachment.

Dear Aparna,

thank you for the advice. But what do you mean by the catch ? Do you mean single input variable and single out put? if yes , I don’t think so, as ANN can accept multi input to produce an out put. If no, please clarify more.

Dear all,

Any visualizatiin technique, rather than the heat-map (which work well in showing the Course grade indicators for AGPA and Dissertation courses for dataset in arrangement 2) or correlation plot, that help in getting information about the data set and attributes relationship?

Noting that hat yes there is a sckewness in dataset but there is no outliers or extreme values.

No worries Lubna. You're welcome.

• Regarding your answer for the second question, kindly note that I have 2 different aims, the first is to predict students final AGPA ( continues value), and the Second is to predict each course grade including dissertation (letters ABCF or numbers 1,2,3,4). In dataset arrangement 1 I won’t be able to see which course , course grade, or instructor that could affect the prediction of the last aforementioned grades and in the second one, it will be hard to point to course or instructors as an indicators for students grades which I am Interested to point to it as well. I am trying to make my project more utilizable by my institution. So please reelaborate on point 2 based of the aforementioned facts.

Well, if you are about to predict continues data using, then this is considered to be a regression problem. In this sense, you can use algorithms like Linear regression, MLP, SVM for regression, and so on. For the sake of evaluation, RMSE and correlation coefficient can be considered. In contrast, if your class label that you need to predict belongs to the nominal type (letters ABCF, in your case), this is a classification problem, where algorithms like Naive bayes, SVM, and RandomForest (and many other algorithms) can be used. To evaluate this sort of learning, accuracy, ROC, and Kappa statistic, for instance, can be utilized though. The easy way to deal with those 2 different learning types (i.e., classification/regression) is to do an independent prediction process by considering one class in each learning process (either nominal or numeric). (Note that, cross-validation, for example, is recommended to be applied for evaluating the overall learning process.)

• Another questions, when you said instances, do you mean the number of records in each variable or the number of students and their corresponding records? And would that mean you prefer less attributes with more instances ? Like dataset structure 1 in the attachment.

Instance = example/record/ tuple

Attribute = feature/variable

Cheers,

Dr. Samer Sarsam

You could try cross-validation, or create synthetic data points. But as Sergey pointed out, that is a really small data set. For such a small data set I'm not sure you will get a good representation with ML techniques.

I think it is a very tricky issue. You are right in the sense that she needs 2 datasets. However, using ANN can work for these smaller datasets. Reason being it is easy to categorise it and come up with targets. The real question now is, you are trying to predict AGPA, now are you representing the data/pattern of it as just student Id and the AGPA or are you doing it as a spread?

If you are doing it as just Id and AGPA as patterns then you will struggle with the last records. however, now if you exclude those missing records and make a pattern out of the available individual module scores. You can make the AGPA as the output target and the pattern will now consist of the scores for individual courses or modules as the case may be. You can then use that to train you ANN and then see how close it outputs. Then being student records, I am sure you will not find it hard to come up with arbitrary student scores that make up respective AGPAs n test with it. This isn't a sacrosanct final. Just a good preliminary way to go.

There is also another way you can try it, since you have incomplete data, if the first method doesn't work then send me a message and I will point you in the right direction for the second.

Thank you all for the reply.

Actually, i decided to convert all attributes that i am goining to predict to letters ( nominal ) instead of continues numbers.

I understand your point Peter, but I’ll try.

Yes Ahmed & Mohammed, i think I’ll try it with the Decision Tree ( not sure if C4.5 or C5.0 ), then i'll compare between the accuracy of both, plus, i will see which variables were chosen ones from the decision tree results (most correlated), and i will use them with the most accurate model algorithm to predict the Dissertation marks ( Nominal attribute).

That will be done using the dataset 2.

Then i will use the dataset1 to predict each course marks to see if that could be predictable with acceptable accuracy rate or not, also using the aforementioned method.

But at the beginning, for visualization, i will use the heat map and correlation plot for dataset2, and correlation plot only for dataset1, unless you could suggest others for better visualize the relationship between variables.

Thanks Samer for the reply.

I think for validation i'll use: accuracy, precision, & recall.

Thank you all again for the valuable information.

In my opinion, it is very good idea - to convert all attributes to letters instead of continious and to use Decision Tree. May be, you will have some problem how to select optimal boundaries for gates, but it isn't very hard problem. Good luck!

Lubana,

Thus use of precision and recall is not a very good choice. You could replace them with either F-measure or PRC area.

Cheers,

Dr. Samer Sarsam

Dear Samer, sorry, but I don't agree with you. To use F_1, F_2 or some else (F_k) ? How can we select value of k - Recall in k times more important than Precision? In my opinion, more realistic to use one parameter as restriction and to optimize another parameter. For example, Recall/Sensitivity should be more than 90% (we can loss maximum 1 of 10 positive answers) and to maximize Precision (or Specificity).

Dear Sergey,

Good to hear from you after our last discussion; where I answered your question that was about missing values imputation.

First of all, precision, recall, and F-measure are valid approach to evaluate machine learning algorithm, you can read the literature since they are well-known evaluation metrics.

Second, in the current discussion, I am not supporting the use of both precision and recall; this is Lubana not me :-)

If computing the precision_k and recall_k are fine for you, then computing the F_k becomes even simpler because the f-measure in its formula considers both the precision and the recall. See this link:

https://en.wikipedia.org/wiki/F1_score

F-measure is recommended in many cases and application because precision and recall are often in tension, see this link:

https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall

thus, to make one valid decision, F-measure is recommended (this answers your current question: How can we select value of k - Recall in k times more important than Precision?).

Anyway, computing the F-measure (same thing goes with both precision and recall) in k-fold cross-validation is not a big deal, as F-measure is calculated for each of the k test folds (or r \times k test folds, if k-fold cross-validation is repeated r times), and these values are then averaged.

One last thing, sorry can not trust your approach. You need to provide us with references from the literature to support your claim.

Cheers,

Dr. Samer Sarsam

Dear Samer, thanks a lot for your detailed answer. Unfortunately, I don't know, what are precision_k and recall_k - perhaps, they are new metrics. Early I have worked only with regular precision and recall and calculated F_k measure, based on them.

I fully agree with you, that theoretically to use measure F_k to perform tuning and select meta-parameters is good approach, but practically? How is it really possible to select appropriate value of k ? Consider, e.g., task of early diagnostics of cancer. Realistic approach - to restrict Sensitivity (= Recall) by some fixed value (to loss less than some % of patients with real cancer) and to maximize Specifity, i.e. to minimize unless biopsy. It isn't my own approach, it is well-known approach to solve multi-criteria tasks - to restrict some criteria and to optimize one. Regards, Sergey.

I recommend Decision tree, or random forest. As your data size is small you can use KNN (however, make it your last option)

Dear Sergey,

Thanks for your valuable contribution.

The approach that you are referring to (I call it your approach because till now you did not provide its name although it is well-known--there are many famous approaches, so we give them names to differentiate them, e.g., cross-validation) needs to show that it is working perfectly on several applications.

Regarding the use of F-measure, I am mainly concerned on its practical implementation (not only theoretical) as it shows a balanced decision without having any sort of confusion with Precision and Recall.

Cheers,

Dr. Samer Sarsam

Dear Lubna

You can try using projection pursuit which is already used for classification purpose. please refer to this article "Support Vector Machine based Exploratory

Projection Pursuit Optimization for User face identification (DOI:10.1109/ICTRC.2015.7156487)"

Regards

Dear Samer, thanks a lot for your answer about F-measure. In my opinion, it was useful discussion. If I will have some free time, perhaps, I will write some discussion-post about this on RG. Regards, Sergey.

The link below may be helpful for you

https://stats.stackexchange.com/questions/174305/which-classifiers-or-anomaly-detectors-work-well-with-little-data?noredirect=1&lq=1

I Article An empirical study of cost-sensitive learning in cultural modeling

Thank you Mahesh for the link, actually it was interesting since its my first time to be introuced with Data augmentation. It was really interesting question and discussion.

Thank you Vector , and thank you Natalia.

Actually i decided to add to the aforementioned steps the logistic regression algorithims.

And for the accuracy check i will use the confusion matrix.

For those who were experianced with R, please respond to my private message.

Thanks & Best Regards,

Lubna

http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

Lubna,

I have already replied your private message (you will find the command there though).

Sometimes, if you don't get fast reply from me, this because I am busy, as there are many another stuff to do. That is the only reason.

Cheers,

Dr. Samer Sarsam

Dear Samer, I once more have read your answers and now see, that it was confusion (miss-understanding) between our answers. For k you mean amount of folds on the k-fold cross-validation, and I mean measure of relative importance on the F-k expression:

F_k = (1 + k)/(k/recall + 1/precision).

So, my question was, certainly, not about how to select amount of folds, but rather how to select measure of relative importance of recall and precision. Regards, Sergey.

Hi Lubna,

I think that the number of attributes should be dependent of your Unit of Analysis. If the student is your unit of analysis, then the second format in which each row represents a student would be appropiate. If your Unit of Analysis is the course, then you should go with the first format. As you want to predict for both, you could try building two separate models: one for students MSc AGPA and another model to predict the Course Grade.

As others have suggested, with such a small dataset you could try Decision Tree or Logistic Regression to predict the students MSc AGPA (Classification) and Decision Tree of Linear Regression to predict the Course Grade. You could get very simple models with high interpretability. In addition, from these models you can derive the importance / influence of the variables.

For evaluating the models, as you have a small sample, you could try Leave-one-out cross-validation. This will be useful to evaluate how your model will perform on new data instances without using too few examples (as you would be evaluating the model on one instance and training with the rest).

Hope this helps and wish you success and good luck on your analysis!

Dear Sergey,

I see. Anyway, thanks for bringing this issue out.

To make things much clear, I would appreciate whether you could highlight first the source of this formula:

F_k = (1 + k)/(k/recall + 1/precision)

Cheers,

Dr. Samer Sarsam

Dear Samer, I will search source and will answer you ASAP. Regards, Sergey.

For the missing data, you can use the average value for the missing data. Again for the classifier, you can use decision tree or simple knn

You can you use regression to predict missing values. Just select the order or regression is it linear or polynomial and avoid overfitting.

Dear Samer, answer for your question ( ..." source of this formula:

F_k = (1 + k)/(k/recall + 1/precision) "... ) is below. See. please, expression for F_beta. Regards, Sergey.

https://en.wikipedia.org/wiki/F1_score

Dear Sergey,

Thanks for your reply.

Well, based on the link you referred to lastly, the the harmonic mean or so-called F-measure (or F1 score) has to reach its best value at 1( perfect precision and recall ). This led me to recommend the use of F-measure (or F1 score) instead of using precision and recall.

The main formula of F1 score is: 2*[(precision * recall)/(precision + recall)]

The rate of the type II error is denoted by the Greek letter β (beta). For more information about F_β (measures the effectiveness of retrieval with respect to a user who attaches β times as much importance to recall as precision) please have a look at this link:

http://www.dcs.gla.ac.uk/Keith/Preface.html

Cheers,

Dr. Samer Sarsam

How we can decide about the perfect number of the hidden layers in ANN, as per my understanding multilayer NN is better for small dataset size.

But how much is the hidden layers or how to decide about it, still I can’t figure it out.

Any idea?

Hi Lubna,

There is no rule of thumb to find out how many hidden layers you need. You have to use methods like cross-validation to examine the performance of the classifier on the test set. Usually people tend to use one hidden layer for simple tasks, but recent research in deep neural network architectures show that many hidden layers can be fruitful for some applications like face recognition.

Cheers,

Dr. Samer Sarsam

I am asking that, as I tried different layers and the accuracy differed.

Accuracy is not the only metric you need to consider. I encourage you to read some articles and see how people have tackled such issue.

Cheers,

Dr. Samer Sarsam

Here is a little something for You (possibly):

http://lc.kubagro.ru/aidos/Presentation_Aidos-online.pdf

http://lc.kubagro.ru/

http://lc.kubagro.ru/aidos/_Aidos-X.htm

Существует много систем искусственного интеллекта. Универсальная когнитивная аналитическая система «Эйдос-Х++» отличается от них следующими параметрами:

- разработана в универсальной постановке, не зависящей от предметной области. Поэтому она является универсальной и может быть применена во многих предметных областях (http://lc.kubagro.ru/aidos/index.htm);

- находится в полном открытом бесплатном доступе (http://lc.kubagro.ru/aidos/_Aidos-X.htm), причем с актуальными исходными текстами (http://lc.kubagro.ru/__AIDOS-X.txt);

- является одной из первых отечественных систем искусственного интеллекта персонального уровня, т.е. она не требует от пользователя специальной подготовки в области технологий искусственного интеллекта (есть акт внедрения системы «Эйдос» 1987 года) (http: //lc.kubagro.ru/aidos/aidos02/PR-4.htm);

- обеспечивает устойчивое выявление в сопоставимой форме силы и направления причинно-следственных зависимостей в неполных зашумленных взаимозависимых (нелинейных) данных очень большой размерности числовой и не числовой природы, измеряемых в различных типах шкал (номинальных, порядковых и числовых) и в различных единицах измерения;

- содержит большое количество локальных (поставляемых с инсталляцией) и облачных учебных и научных приложений (в настоящее время их около 30 и 128, соответственно) (http://lc.kubagro.ru/aidos/Presentation ...online.pdf);

- обеспечивает мультиязычную поддержку интерфейса на 44 языках. Языковые базы входят в инсталляцию и могут пополняться в автоматическом режиме;

- поддерживает on-line среду накопления знаний и широко используется во всем мире (http://aidos.byethost5.com/map3.php).

There are many artificial intelligence systems. Universal cognitive analytical system "Eidos-x++" differs from them in the following parameters:

- developed in a universal setting, independent of the subject area. Therefore, it is universal and can be applied in many subject areas (http://lc.kubagro.ru/aidos/index.htm);

- is in full open free access (http://lc.kubagro.ru/aidos/_Aidos-X.htm), and with the actual source texts (http://lc.kubagro.ru/__AIDOS-X.txt);

- is one of the first domestic systems of artificial intelligence of the personal level, i.e. it does not demand from the user special training in the field of technologies of artificial intelligence (there is an act of introduction of the Eidos system of 1987) (http://lc.kubagro.ru/aidos/aidos02/PR-4.htm);

- provides a stable identification in a comparable form of force and direction of cause-effect relationships in incomplete noisy interdependent (nonlinear) data of very large dimensions of numerical and non-numerical nature, measured in different types of scales (nominal, ordinal and numerical) and in different units of measurement;

- contains a large number of on-premises (supplied with the installation) and cloud-based learning and scientific applications (currently there are about 30 and 128, respectively) (http://lc.kubagro.ru/aidos/Presentation Oh?.. online.pdf);

- provides multi-language interface support in 44 languages. Language databases are included in the installation and can be replenished automatically;

- supports on-line environment of knowledge accumulation and is widely used all over the world (http://aidos.byethost5.com/map3.php).

Dear Lubna,

My paper on Educational Data Mining may be helpful for you. "Educational Data Mining and Analysis of Students' Academic Performance Using WEKA". If you have any issues, you may contact me.

Dear Lubna,

I recommend to apply ATOVIC system:

Article Amended fused TOPSIS-VIKOR for classification (ATOVIC) appli...

look at this:

http://lc.kubagro.ru/aidos/_Aidos-X.htm

http://lc.kubagro.ru/aidos/Presentation_Aidos-online.pdf

Classification algorithm: C5.0. It also handles missing values

right Sourabh Shastri

You can use the Bayesian algorithm, it has advantages in this kind of problem solving.

Naive Bayes and Bayesian TAN are also the good options for classifying small datasets.

Festim Halili !

Article Informational Neurobayesian Approach to Neural Networks Trai...

ATOVIC system

Article Amended fused TOPSIS-VIKOR for classification (ATOVIC) appli...