How can I find out whether I should split the compounds and search or not? Eg: if the compound is research fellow and I perform compound splitting, the results may be too noisy. And in some other cases we may get appropriate results only when we perform compound splitting.
The word "research" is a common word; "fellow" is less likely. The two together are even less likely. So start with the compound, then split selectively.
@Ian Kennedy
Do i need to maintain a corpora of common words, so that when i see each and every word of the compound i can analyze and decide splitting should occur or not.
what if i am domain specific? I can maintain a separate ontology of compounds and use it. If i get some compounds other than in my ontology i may do splitting.. how do you people rate this idea? I would be happy to hear both open domain and specific domain
Yes. It might help to also store the probability of that word occurring. You can use a rough, fast measure like 0..255
It would be more helpful if you derived your corpus and their probabilities from your domain.
Your corpus and probability could also contain phrases. As you suggest, unknown phrases could be split.
Your approach is like giving Google a phrase in quotes, getting nothing then repeating without quotes.
A follow-up to the contribution of Lionel Nicolas: it might help to test whether the compound parts form a collocation. If so the compound's meaning goes beyond the composition of the meanings of its parts and thus it should be treated as a single lexical unit. In [1] several statistical methods for collocation extraction are described.
[1] http://nlp.stanford.edu/fsnlp/promo/colloc.pdf
Aarthi,
This is a link to Google compound word breaker algorithm/process: http://www.google.com/patents/US7447627. The process is much like what others on the forum has discussed, they break the component words/terms apart to test if the are in a lexicon of terms and prepare to search on the words separately first to see if searching for the compound term finds documents online that are similar to those returned when looking for each component term. If the algorithm finds enough similarity, it gives more weight to the compound term. If the compound term doesn't exist, it adds it to its lexicon of terms with an index of links, a relevance ranking/weight and a reference to the two separate component terms' links for future relevance checking/ranking.
Because compound terms tend to have less ambiguity or separate meanings than single words/terms, they usually (not always) increase contextual awareness in information retrieval. It sounds like the lexicon you're using doesn't have compound terms. Are you building a dictionary/index for the compound terms so that they have their own associated semantic tags/classifiers as well as links to documents or corpora containing the compound terms for future classification and machine learning?
http://www.google.com/patents/US7447627
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs