Hi Nirmal,

As per my understanding, the question you have put here, is related to Text Mining but not to Natural Language Processing. Though I am not good at this topic but I feel worth sharing some suggestions that might help you out.

Idea is, the declarative sentence that is required to search, should be broken out in keywords and then apply SQL query using SELECT query with AND condition for all these keywords to find a closest sentence.

E.g. I am good OR I am doing good are two sentences same in meaning and one of this is to be searched. Keywords from this sentence could be selected as I, am and good.

Chances are you find similar sentences in database based on a match in meaning. But on other hand, it could be mere another sentence that just matches because of match of all keywords.

Hopefully, this gives you some direction!!

Hi Dhaneshwar,

Thank you for your reply. Here i am facing the problem in selecting the keyword. I want my algorithm to do it. I shall give you an example.

My database contains a sentence :

Gear failure at 4th speed

Now i have two sentences which states :

1)System failure at 4th speed.

2)Gear failure at 5th speed.

An SQL query might show that both sentences are correct matches with the sentence in the database because it was able to find exactly 4 words matches. But our intuition shows us that only the first sentence is correct. How do i make my algorithm make it recognise this ??

I believe that this question is a part of NLP. I kindly request you to respond to this post

Thank you in advance

Nirmal

This looks more like text mining. You could look at RapidMiner 5 (http://rapid-i.com/content/view/193/74/lang,en/) which has a free version and includes some text mining tools including a system for testing text blocks for similarlity and automatically grouping them by content. It connects to some database systems.

Nirmal, You are right, even it is difficult for us as being humans to point out right keywords so it must be more difficult if same task is assigned to a computer program to highlight keywords from sentence that can be used to search similar sentences in database.

However, I am daring to add that we can design an algorithm that look for specific words such as noun, adjective and verbs (adjective only if they are not followed by noun), and that is also a difficult task (One of my friends designed such program for their final year project and shared his experience with me).

Otherwise, I second @Geoffrey Stephenson's advice to look at existing programs or APIs.

@Geoffrey : Thank you for your answer. I will definitely look about your suggestion and get back to you on this.

@Dhaneshwar : Thank you Mr.Dhaneshwar. Can you direct me to resources where i can find the algorithm that can be used to look for specific words like noun, adjective and verbs and make a decision based on that ? It would be really helpful for me

This is completely out of my area of expertise and I hate to play kill joy here, but if someone had the answer to this question they would publish it in Nature or Science.

Having said that, parts of speech would not solve your problem with the example you are giving and both sentences would have the same score no matter what since they are grammatically identical (I am not a native speaker, though).

In this particular example Levenshtein distance would solve the problem but that would also not generalize well.

Here you would need more than mere grammar or text mining, here you need semantics and real understanding something that is far from being solved as far as I know.

I hope I am wrong and this is an easy task in NLP as it would help with a text mining task we are facing currently. I'll, therefore, be following the development of this question.

In general NLP cannot answer these questions generally and reliably. In your example, the relevance of DB sentence one is based on the that fact that a gear failure is a member of the class of system failures. If you can enumerate all such class memberships (or find some sort of way to induce them automatically), then great, but you will still be nowhere near solving the problem generally. I think Jens was on target: if you find a general solution to this problem, send it in to Nature. It's AI-hard.

You might take a look at "Situation Theory" developed by Barwise and others. Keith Devlin seems to be the most recent proponent. You'll find work around this topic at CSLI (Center for the Study of Language and Information).

Books: Logic and Information, InfoSense: Turning Information into Knowledge, Language at Work: Analyzing Communication Breakdown in the Workplace to Inform Systems Design. Duska Rosenberg co-authored the last of these. If I recall correctly, the first is a semi-popularized treatment, the second is more theoretical, and the last moves through an application of the theory to a specific problem. Between the three there is quite a bit of repetition.

Essentially, Situation Theory attempts to mathematize knowledge in situ using the language of sets, relations, properties, etc. Using it for natural language is a narrowing of its full potential, but still a major component in my opinion. I really like the treatment and that they specifically include context as part of the analysis. The major problem is that it is rather unweildy. You, of course, would first need to put your declarative sentences into Situation Theory form. I've run out of steam rather quickly doing this because I find it tedious (partly because I am still learning to understand the mathematical approach). However, I can see how one could create an interface to a database that would make this much easier. With a bit of experience and talent, one might even find a decent way to automate most of the work, especially for declarative sentences. I hope to return to Situation Theory once I've further developed the mathematical and programing skills needed.

@Joe Byproduct: Thank you for your answer, I am a novice in this field and i am searching for answers. Currently i am learning the basics of NLP. Hope i could get some concrete idea to develop such a system.

@Thomas Sandidge: Thank you for your reply. I am glad that u have introduced me to such a pertinent topic. I would definitely look into this.

@Nirmal, Did you find correct algorithm to compare two sentences  ?