Hi,
I have some text data that has details of customer name, address etc,..along with some other inforamtion which are not required and I would like to extract the required data and place it in the web page in appropriate fields.
Please let me know which classification holds good and the different types of rules and any file conversion involved in this?
If your dataset is limited to a finite set of customer attributes and with limited scope for regression, you can settle for using an appropriate regex to classify the data.
if your dataset would grow exponentially or if you have covariant dependencies (like the format of customer address changing depending on the country of the customer etc) which increases regression, then you can adopt a simple SVM based approach.
Once you've classified the data, it would be straight-forward to match it to fields on the web page.
Normally if your data is very huge and may have so many dimensions and features, you apply feature selection and dimension reduction techniques. Once you find appropriate and required fields are selected using feature selection methods then you apply classification either SVM (support vector machine) or any kind of classification method
Kishore,
My data set is having address of the same country only difference is it might not be in the same order like for eg:
Data set1: Name, additional data,account number, customer address, zip code, additional inf, phone number,
Data set2: Additional data, name, additional inf, additional inf, address, zip, additional inf, phone number, account number.
If this is how it varies for each data set then can I use reg ex for this and can do the work or do I have to go for classification algorithm for this.
One more thing is my data sets might not be more than 2k as of now.
Please clarify me on this.
Jie,
I am not very sure but the problem is my data sets will be in HTML format that comes from external source and I took some dummy HTML file and extracted data with in the tags.
If each file I extract will be in different order then I am not sure how to loop through each file and apply reg ex on those.
Do you have any dummy data sets that will be generated with random order in HTML format with customer name, address, phone number...etc.
Jie,
Can you please suggest any sites where I can get some files or data sets so that I can work on extracting data based on reg ex alone. I am using beautful soup for this task as of now.
Sree,
This would be a factor of the nature of the pattern. Taking a sample as below,
Datset 1: Sree Sindhusruthi 12456 A12872 60 Abe Street Chicago IL 901-123-2456
Dataset 2: 5439 Sree Sindhustruthi A23098 924-167-1234 324 XYZ 90 Bce Street Chicago IL
if all the variations in patterns are known for all datasets, then it would be strightforward to apply regex.
if the variation in patterns are totally random (although I would then question the source of the data and the dataset generation process, as entirely random sequences for a data like "customer data" is very unusual), then a linear approach is - you could do a space delimited sequential split, and examine as below -
If di is a datapoint, and F(d) represents a known pattern (eg., an "address")
F(d)= F(di, di+1, di+2...di+c) -> You examine if a function of successive datapoints result in any of the known patterns identified by F(d), if yes, you mark them and proceed with di+c+1.
Given that the datasets may not be more than 2k, an SVM approach may not really be necessary, and one of the above approaches should provide you results within a reasonable response time.
Kishore,
Please correct my understanding if I am wrong.
F1(d) = {set of data sets } # that satisfies F1 regex pattern
F2(d) = {set of data sets } # that satisfies F2 regex pattern
And each data set has to be cheked against each reg ex patterns and what ever fits I can extract data accordingly.
- Badges
- Science topic