Hi

Take a look at the OCR software Tesseract which originated from HP and later was improved by Google, and I believe, used to scan lots and lots of documents.

https://github.com/tesseract-ocr/tesseract/wiki

https://sourceforge.net/projects/tesseract-ocr/

In any case, Tesseract is one of the better OCR engines and not that hard to use.

Last - OCR will never be 100% correct. If you want to improve the hitrate you probably need to add some sort of deep learning mechanism to recognize that the likelihood for a particular word or phrase really is a slightly different word. There are several such engines available e.g. in Python.

Hope this helps

Claes

I don't see where Big Data fits here. 

For conversion from PDF image to Excel try out this link :

https://pdf.wondershare.com/pdf-converting-tips/pdf-image-to-excel.html

Hi Dr. Fredo, I am just starting to do my research in Big Data in construction. I feel really interested in the Neural Network and Deep Learning. You gave me a really interesting idea on the part of structuring data from my raw data. Do you know which way we can structuring the unstructured data, in general? I am looking for some good article about structuring raw data, especially unstructured in construction industry

Thank you very much Dr. Ahmed, this is a very useful link. Actually, I also think that Big Data may not fit those type of scanning data. However, I really consider in some way about the potential application for the future. Construction is a highly conservative industry. Can we considered a relative large amount of unstructured data like PDFs in multiple forms are potential for Big Data analytics?

Hi

I have no experience with deep learning. I want to get into this subject as well. It seems that Tesseract is a mini-AI as it has been trained to recognize certain patterns.

Enclosed is a Matlab code snippet on how to execute Tesseract. 

Structuring. I can only venture a guess as I have never done this myself. I imagine you need to classify the data first to distinguish apples from pears. To do so, I imagine you must identify typical patterns in your PDF data and process these according to appropriate rules that you define before you dump data into a database.  Regardless on what classification you choose, there will always be logic holes, i.e. cases where the classification does not work, so keep flexibility in mind when setting up rules and executing.

Then again, there are lots of people that have barked up this tree long before us, take a look here

https://en.wikipedia.org/wiki/Unstructured_data

https://en.wikipedia.org/wiki/Data_structure

and remember the Pareto rule - you get 80% at 20% of the effort. Move beyond this and you need much larger effort and still never will reach 100%.

Have fun

/Claes

Hi Anh,

You may consider sampling the data before generating pdfs. When I come across such situations, I use python pandas sampling function, to sample the data and perform analysis.

Thanks

Ravikant

Hi Ravikant, can you say more about sampling data relating with Python pandas. I am not major in computer science, so I know very little about Python. Thanks you very much

Hi Anh,

There is nothing particularly specific to python here.  The problem is that when the data set is large (~1 million ), so it there are too many points on the plot to see a trend. Plus, it takes a while to plot. 

Therefore you want to sub-sample say  ~10000 randomly distributed data points. Also, this should be reproducible, so the same sequence of random numbers is generated in each run.

You can find libraries for such sampling in R, python or any other data analysis tools.

Hope this helps!

Thanks

Ravikant