You might be interested in this Python module:

https://github.com/navicto/Discretization-MDLPC

I have no personal experience withis package and cannot tell details. Good luck.

I've followed the advice from Jiří Kroc . My goal is to figure out how to use MDLP_Discretizer to process a dataset from a csv.

The program that needs to process a csv file. This file needs to be converted into a dataset. The example that I am working with comes from the popular python tutorial with the iris data set. I am trying to replace datasets.load_iris() with a method to read the csv 'A1-md.csv'.

Expected: The program process the csv and loads the data.

Actual:

The program fails with

```

Traceback (most recent call last):

File ".\e_disc.py", line 43, in

main()

File ".\e_disc.py", line 27, in main

discretizer.fit(X_train, y_train)

File "D:\Data Mining\assignment_1\entropy_disc_test\MDLP.py", line 58, in fit

if len(self._col_idx) != self._data_raw.shape[1]: # some columns will not be discretized

IndexError: tuple index out of range

```

A working example looks like this

```

import numpy as np

from sklearn import datasets

from sklearn.model_selection import train_test_split

from MDLP import MDLP_Discretizer

def main():

######### USE-CASE EXAMPLE #############

#read dataset

dataset = datasets.load_iris()

print(dataset)

X, y = dataset['data'], dataset['target']

feature_names, class_names = dataset['feature_names'], dataset['target_names']

numeric_features = np.arange(X.shape[1]) # all fetures in this dataset are numeric. These will be discretized

print(f'numeric_feature type {type(numeric_features)}')

# Split between training and test

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

# #Initialize discretizer object and fit to training data

discretizer = MDLP_Discretizer(features=numeric_features)

discretizer.fit(X_train, y_train)

X_train_discretized = discretizer.transform(X_train)

#apply same discretization to test set

X_test_discretized = discretizer.transform(X_test)

#Print a slice of original and discretized data

print('Original dataset:\n%s' % str(X_train[0:5]))

print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

#see how feature 0 was discretized

print('Feature: %s' % feature_names[0])

print('Interval cut-points: %s' % str(discretizer._cuts[0]))

print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':

main()

```

My implementation looks like this

```

import numpy as np

import pandas as pd

from sklearn import datasets

from sklearn.model_selection import train_test_split

from MDLP import MDLP_Discretizer

def main():

######### USE-CASE EXAMPLE #############

#read dataset

dataset = pd.read_csv('A1-dm.csv')

print(dataset)

X, y = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()

print(dataset)

feature_names, class_names = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()

numeric_features = X # all fetures in this dataset are numeric. These will be discretized

print(numeric_features)

print(f'numeric_feature type {type(numeric_features)}')

# #Split between training and test

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

print(numeric_features)

# #Initialize discretizer object and fit to training data

discretizer = MDLP_Discretizer(features=numeric_features)

discretizer.fit(X_train, y_train)

X_train_discretized = discretizer.transform(X_train)

# #apply same discretization to test set

X_test_discretized = discretizer.transform(X_test)

# #Print a slice of original and discretized data

print('Original dataset:\n%s' % str(X_train[0:5]))

print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

# #see how feature 0 was discretized

print('Feature: %s' % feature_names[0])

print('Interval cut-points: %s' % str(discretizer._cuts[0]))

print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':

main()

```

Any help understanding how to convert my dataset

```

A1,A2,A3,Class

2,0.4631338,1.5,3

8,0.7460648,3.0,3

6,0.264391038,2.5,2

5,0.4406713,2.3,1

2,0.410438159,1.5,3

2,0.302901816,1.5,2

6,0.275869396,2.5,3

8,0.084782428,3.0,3

2,0.53226533,1.5,2

```

Into a form that is similar to the iris data set would be greatly appreciated. Thank you.

Dear colleague, according to what you described, you need to convert a text file into an another text file that will have exactly the same format as required by the processing program. You split your task to two steps: data preprocessing and running the data evaluation by the given software.

Hence:

1) find the format of data required by the evaluation program,

2) compare it with the format of data you have,

3) write a conversion program in Python

4) run the converted file

5) you can pipe the conversion program and your program using shell commands

Thank you Jiří Kroc , I was able to make it past that issue. Now I am facing an issue with the MDLP_Discretizer method in the example.

Expected: The program process the csv and loads the data.

Actual:

The program fails with

```

Traceback (most recent call last):

File ".\e_disc.py", line 43, in

main()

File ".\e_disc.py", line 27, in main

discretizer.fit(X_train, y_train)

File "D:\Data Mining\assignment_1\entropy_disc_test\MDLP.py", line 58, in fit

if len(self._col_idx) != self._data_raw.shape[1]: # some columns will not be discretized

IndexError: tuple index out of range

```

A working example looks like this

```

import numpy as np

from sklearn import datasets

from sklearn.model_selection import train_test_split

from MDLP import MDLP_Discretizer

def main():

######### USE-CASE EXAMPLE #############

#read dataset

dataset = datasets.load_iris()

print(dataset)

X, y = dataset['data'], dataset['target']

feature_names, class_names = dataset['feature_names'], dataset['target_names']

numeric_features = np.arange(X.shape[1]) # all fetures in this dataset are numeric. These will be discretized

print(f'numeric_feature type {type(numeric_features)}')

# Split between training and test

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

# #Initialize discretizer object and fit to training data

discretizer = MDLP_Discretizer(features=numeric_features)

discretizer.fit(X_train, y_train)

X_train_discretized = discretizer.transform(X_train)

#apply same discretization to test set

X_test_discretized = discretizer.transform(X_test)

#Print a slice of original and discretized data

print('Original dataset:\n%s' % str(X_train[0:5]))

print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

#see how feature 0 was discretized

print('Feature: %s' % feature_names[0])

print('Interval cut-points: %s' % str(discretizer._cuts[0]))

print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':

main()

```

My implementation looks like this

```

import numpy as np

import pandas as pd

from sklearn import datasets

from sklearn.model_selection import train_test_split

from MDLP import MDLP_Discretizer

def main():

######### USE-CASE EXAMPLE #############

#read dataset

dataset = pd.read_csv('A1-dm.csv')

print(dataset)

X, y = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()

print(dataset)

feature_names, class_names = dataset['A1'].to_numpy(), dataset['Class'].to_numpy()

numeric_features = X # all fetures in this dataset are numeric. These will be discretized

print(numeric_features)

print(f'numeric_feature type {type(numeric_features)}')

# #Split between training and test

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

print(numeric_features)

# #Initialize discretizer object and fit to training data

discretizer = MDLP_Discretizer(features=numeric_features)

discretizer.fit(X_train, y_train)

X_train_discretized = discretizer.transform(X_train)

# #apply same discretization to test set

X_test_discretized = discretizer.transform(X_test)

# #Print a slice of original and discretized data

print('Original dataset:\n%s' % str(X_train[0:5]))

print('Discretized dataset:\n%s' % str(X_train_discretized[0:5]))

# #see how feature 0 was discretized

print('Feature: %s' % feature_names[0])

print('Interval cut-points: %s' % str(discretizer._cuts[0]))

print('Bin descriptions: %s' % str(discretizer._bin_descriptions[0]))

if __name__ == '__main__':

main()

```

It is difficult to code off computer using RG. Let us try it. Try to print from working example those data and check what type and shape they have explicitly. There are methods for this in python. It is possible that the code expect different type to what it gets.

print(dataset)

X, y = dataset['data'], dataset['target']

Do the same for your code. The problem might be hidden in it. It is just my guess without reading the code and knowing exactly what it does.

By the way, is this method doing something similar to GINI index? What is the purpose to use this method on your data.

I forgot to mention the error report: "IndexError: tuple index out of range". It tells you that the number of expected and given data differs. Hence, you should do what was already said above.

Jiří Kroc {'data': array([[5.1, 3.5, 1.4, 0.2],

[4.9, 3. , 1.4, 0.2],

[4.7, 3.2, 1.3, 0.2],

[4.6, 3.1, 1.5, 0.2],

[5. , 3.6, 1.4, 0.2],

[5.4, 3.9, 1.7, 0.4],

[4.6, 3.4, 1.4, 0.3],

[5. , 3.4, 1.5, 0.2],

[4.4, 2.9, 1.4, 0.2],

[4.9, 3.1, 1.5, 0.1],

[5.4, 3.7, 1.5, 0.2],

[4.8, 3.4, 1.6, 0.2],

[4.8, 3. , 1.4, 0.1],

[4.3, 3. , 1.1, 0.1],

[5.8, 4. , 1.2, 0.2],

[5.7, 4.4, 1.5, 0.4],

[5.4, 3.9, 1.3, 0.4],

[5.1, 3.5, 1.4, 0.3],

[5.7, 3.8, 1.7, 0.3],

[5.1, 3.8, 1.5, 0.3],

[5.4, 3.4, 1.7, 0.2],

[5.1, 3.7, 1.5, 0.4],

[4.6, 3.6, 1. , 0.2],

[5.1, 3.3, 1.7, 0.5],

[4.8, 3.4, 1.9, 0.2],

[5. , 3. , 1.6, 0.2],

[5. , 3.4, 1.6, 0.4],

[5.2, 3.5, 1.5, 0.2],

[5.2, 3.4, 1.4, 0.2],

[4.7, 3.2, 1.6, 0.2],

[4.8, 3.1, 1.6, 0.2],

[5.4, 3.4, 1.5, 0.4],

[5.2, 4.1, 1.5, 0.1],

[5.5, 4.2, 1.4, 0.2],

[4.9, 3.1, 1.5, 0.2],

[5. , 3.2, 1.2, 0.2],

[5.5, 3.5, 1.3, 0.2],

[4.9, 3.6, 1.4, 0.1],

[4.4, 3. , 1.3, 0.2],

[5.1, 3.4, 1.5, 0.2],

[5. , 3.5, 1.3, 0.3],

[4.5, 2.3, 1.3, 0.3],

[4.4, 3.2, 1.3, 0.2],

[5. , 3.5, 1.6, 0.6],

[5.1, 3.8, 1.9, 0.4],

[4.8, 3. , 1.4, 0.3],

[5.1, 3.8, 1.6, 0.2],

[4.6, 3.2, 1.4, 0.2],

[5.3, 3.7, 1.5, 0.2],

[5. , 3.3, 1.4, 0.2],

[7. , 3.2, 4.7, 1.4],

[6.4, 3.2, 4.5, 1.5],

[6.9, 3.1, 4.9, 1.5],

[5.5, 2.3, 4. , 1.3],

[6.5, 2.8, 4.6, 1.5],

[5.7, 2.8, 4.5, 1.3],

[6.3, 3.3, 4.7, 1.6],

[4.9, 2.4, 3.3, 1. ],

[6.6, 2.9, 4.6, 1.3],

[5.2, 2.7, 3.9, 1.4],

[5. , 2. , 3.5, 1. ],

[5.9, 3. , 4.2, 1.5],

[6. , 2.2, 4. , 1. ],

[6.1, 2.9, 4.7, 1.4],

[5.6, 2.9, 3.6, 1.3],

[6.7, 3.1, 4.4, 1.4],

[5.6, 3. , 4.5, 1.5],

[5.8, 2.7, 4.1, 1. ],

[6.2, 2.2, 4.5, 1.5],

[5.6, 2.5, 3.9, 1.1],

[5.9, 3.2, 4.8, 1.8],

[6.1, 2.8, 4. , 1.3],

[6.3, 2.5, 4.9, 1.5],

[6.1, 2.8, 4.7, 1.2],

[6.4, 2.9, 4.3, 1.3],

[6.6, 3. , 4.4, 1.4],

[6.8, 2.8, 4.8, 1.4],

[6.7, 3. , 5. , 1.7],

[6. , 2.9, 4.5, 1.5],

[5.7, 2.6, 3.5, 1. ],

[5.5, 2.4, 3.8, 1.1],

[5.5, 2.4, 3.7, 1. ],

[5.8, 2.7, 3.9, 1.2],

[6. , 2.7, 5.1, 1.6],

[5.4, 3. , 4.5, 1.5],

[6. , 3.4, 4.5, 1.6],

[6.7, 3.1, 4.7, 1.5],

[6.3, 2.3, 4.4, 1.3],

[5.6, 3. , 4.1, 1.3],

[5.5, 2.5, 4. , 1.3],

[5.5, 2.6, 4.4, 1.2],

[6.1, 3. , 4.6, 1.4],

[5.8, 2.6, 4. , 1.2],

[5. , 2.3, 3.3, 1. ],

[5.6, 2.7, 4.2, 1.3],

[5.7, 3. , 4.2, 1.2],

[5.7, 2.9, 4.2, 1.3],

[6.2, 2.9, 4.3, 1.3],

[5.1, 2.5, 3. , 1.1],

[5.7, 2.8, 4.1, 1.3],

[6.3, 3.3, 6. , 2.5],

[5.8, 2.7, 5.1, 1.9],

[7.1, 3. , 5.9, 2.1],

[6.3, 2.9, 5.6, 1.8],

[6.5, 3. , 5.8, 2.2],

[7.6, 3. , 6.6, 2.1],

[4.9, 2.5, 4.5, 1.7],

[7.3, 2.9, 6.3, 1.8],

[6.7, 2.5, 5.8, 1.8],

[7.2, 3.6, 6.1, 2.5],

[6.5, 3.2, 5.1, 2. ],

[6.4, 2.7, 5.3, 1.9],

[6.8, 3. , 5.5, 2.1],

[5.7, 2.5, 5. , 2. ],

[5.8, 2.8, 5.1, 2.4],

[6.4, 3.2, 5.3, 2.3],

[6.5, 3. , 5.5, 1.8],

[7.7, 3.8, 6.7, 2.2],

[7.7, 2.6, 6.9, 2.3],

[6. , 2.2, 5. , 1.5],

[6.9, 3.2, 5.7, 2.3],

[5.6, 2.8, 4.9, 2. ],

[7.7, 2.8, 6.7, 2. ],

[6.3, 2.7, 4.9, 1.8],

[6.7, 3.3, 5.7, 2.1],

[7.2, 3.2, 6. , 1.8],

[6.2, 2.8, 4.8, 1.8],

[6.1, 3. , 4.9, 1.8],

[6.4, 2.8, 5.6, 2.1],

[7.2, 3. , 5.8, 1.6],

[7.4, 2.8, 6.1, 1.9],

[7.9, 3.8, 6.4, 2. ],

[6.4, 2.8, 5.6, 2.2],

[6.3, 2.8, 5.1, 1.5],

[6.1, 2.6, 5.6, 1.4],

[7.7, 3. , 6.1, 2.3],

[6.3, 3.4, 5.6, 2.4],

[6.4, 3.1, 5.5, 1.8],

[6. , 3. , 4.8, 1.8],

[6.9, 3.1, 5.4, 2.1],

[6.7, 3.1, 5.6, 2.4],

[6.9, 3.1, 5.1, 2.3],

[5.8, 2.7, 5.1, 1.9],

[6.8, 3.2, 5.9, 2.3],

[6.7, 3.3, 5.7, 2.5],

[6.7, 3. , 5.2, 2.3],

[6.3, 2.5, 5. , 1.9],

[6.5, 3. , 5.2, 2. ],

[6.2, 3.4, 5.4, 2.3],

[5.9, 3. , 5.1, 1.8]]), 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,

1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,

1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,

2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,

2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]), 'frame': None, 'target_names': array(['setosa', 'versicolor', 'virginica'], dtype='

From your answer it is unclear what data you shared. It is not possible to proceed in this way furder.

Jiří Kroc I've created a solution that discretizes the dataset based on the criteria stated above. This breaks the dataset up into two partitions s1 and s2. According to my results s1 is discretized as follows

```

******************************************************

Entropy Discretization STARTED

The unique values for dataset s["Class"] are Int64Index([2, 3, 1], dtype='int64')

Using threshold 2

s1 after spitting

A1 A2 A3 Class

3 5 0.440671 2.3 1

9 8 0.070035 2.9 1

14 2 0.868410 1.5 1

29 6 0.587487 2.6 1

34 8 0.652936 3.0 1

38 8 0.181508 3.0 1

45 8 0.953230 3.0 1

54 7 0.737604 2.7 1

68 5 0.187475 2.2 1

70 5 0.511385 2.3 1

71 8 0.688134 3.0 1

73 2 0.054908 1.5 1

87 8 0.461797 3.0 1

90 2 0.756518 1.5 1

91 2 0.761448 1.5 1

93 5 0.858036 2.3 1

94 5 0.306459 2.2 1

98 5 0.692804 2.2 1

******************

s2 after spitting

A1 A2 A3 Class

0 2 0.463134 1.5 3

1 8 0.746065 3.0 3

2 6 0.264391 2.5 2

4 2 0.410438 1.5 3

5 2 0.302902 1.5 2

.. .. ... ... ...

92 5 0.775842 2.3 2

95 5 0.844920 2.2 2

96 5 0.428071 2.2 2

97 5 0.356044 2.2 3

99 5 0.815400 2.2 3

[82 rows x 4 columns]

******************

******************

calculating maxf

maxf 3

******************

******************

calculating minf

maxf 1

******************

```