For instance, layout of the sentences (i.e. knowing that a specific sentence is a bullet and it is correlated to another sentence that is stating the scope of the bullets). Moreover, lots of NLP parsers break if the mechanism delivers broken sentences (i.e. in a way that regex could not know if there is indeed a breakline in the source document and therefore unable to clean the text). This type of broken sentences also may disturb when obtaining embeddings, since it takes into account neighbor words.
Marcio,
Other problems which I have faced to supplement the ones that you have already given are:
- Quotes within text
- e.g. and i.e. how to deal with them
- Footnotes
Some of these are relevant in some cases that need to be taken into account for context and in others they are worthless. Since I am dealing with a particular type of document I have several rules in place on whether to include them in the parsing step or exclude them altogether.
Other challenges that I have faced when parsing engineering texts is whether to include measurements or not in a parse. Some of them give tolerances that will put out of context the text in other they are just example measures that can be obtained.
Also depending on the field knowing when to isolate segments can be really tricky. Scientific text is less challenging in this respect than novels in which the structure itself may even shift with the text progression.
Hi,
In relation to the text cleaning part of your question, we can make a list, containing your example and Arturo's:
1. Line feeds ("\n") could be invisible and they break the parsing of the sentence
2. Different types of brackets, you name them ( ), [ ], { }, of course it depends
3. Encoding, sometimes there are characters inside the text which should be set before, even one single comma (,) can stop a whole system
4. Some languages (e.g. Java, PHP) are confused with single quote sometimes. It is better to use regex like style for them
5. A mixture of above points is also possible, for example a foreign name inside an English text with an encoding and single quote inside it
6. My mentor does not agree that a really too long sentence could confuse the parser but I have seen it :) The question is though how long a really long sentence is which I cannot answer
So I suggest in order to start the text extraction, following could be beneficial:
1. Visual inspection of a sample of text to see if there are unwanted items like comments in brackets and the like
2. Setting encoding correctly or neutralizing the text, if possible, e.g. by converting them to plain text in Notepad, Gedit, Nano, etc. (We are not talking about big data)
3. Step #2 does not remove line feeds. If system needs to process a single sentence at a time, there may remained line feeds.
3. Replacing single quotes, etc with their regex
Marcio Ferreira Moreno ,
Another problem is assuming well constructed text. An example is doing a check for quotes where the text (either by OCR or user omission) does not include the closing or opening quotes. This turns extremely interesting in CSV's!
Another example is abbreviations at the end of a sentence where the user did not capitalize the word at the beginning of the next sentence.
While this are errors in the text, the point that I learned is not to assume well behaved text. While the problem is obvious, its solution is not.
- Badges
- Science topic
- Similar topics
- Vectorization