Hello everyone,
What are the best methods/tools to extract keywords from a tweet? Removal of stop words, word stemming and word frequency seem to be insufficient. According to previous research, the restriction of 140 characters makes it difficult for traditional Natural Language Processing tools.
Applying an efficient method/tool in the following example would give something like this:
Example: "Tesla just released solar roof prices, and they're cheaper than expected".
Keywords: "Tesla", "Solar roof".
Thanks.
One of the approaches can be the usage semantic based NLP services and libraries, that are analyzing the tweet and returning concepts, entities, keywords, categories, sentiment, emotion, relations, and semantic roles. Some of those services are free, and others are commercial.
- DBpedia Spotlight, this is an annotation service based on DBpedia.http://demo.dbpedia-spotlight.org/
- Open Calais from Thomson Reuters. http://www.opencalais.com/. There is a demo site so you can test it. http://www.opencalais.com/opencalais-demo/
- Natural Language Understanding service from IBM Bluemix Watson services. This service is the continuation of Alchemy API and is one of the best services. https://www.ibm.com/watson/developercloud/natural-language-understanding.html
- FRED: Machine Reading for the Semantic Web, it is a service that can to parse natural language text in 48 different languages and transform it into linked data. http://wit.istc.cnr.it/stlab-tools/fred. And you can read the paper about FRED here: http://semantic-web-journal.org/system/files/swj1379.pdf
- You can also look at this project presented in the paper "Extracting knowledge from text using SHELDON, a Semantic Holistic framEwork for LinkeD ONtology data". http://speroni.web.cs.unibo.it/publications/reforgiato-recupero-2015-extracting-knowledge-from.pdf
I think it is realistic to assume that in
colloquial speech one has to deal with
a set of phrases rather than complete
sentences. Any natural language parser
can provide a list of phrases.
Why have you chosen Tesla and solar roof?
I assume you have a set of principles
in mind which could be put into an
algorithm.
Tesla is a company. You could have a
list of company names in a database.
Or more general, Tesla is a proper
noun. You could mark all proper nouns
in a tweet as keywords.
Then, why solar roof, why not solar
roof prices as keywords? Perhaps because
you have a rule in mind like: if it is
a compound noun, use all of the list
except the head of the noun phrase
as a keyword.
In the end, I think, you don't even
need a natural language parser. You can
do it with a dictionary of word
categories.
Regards,
Joachim
hi, now in 2020 what's your answer to your question ?
Sami Belkacem
Amirhosein Bodaghi I think DBpedia Spotlight [1] is the best one. It is not compromised by the short and informal nature of tweets. It includes also named entity recognition. There is a Python library to use it.
[1]: https://www.dbpedia-spotlight.org/demo/
Sami Belkacem but here :
https://www.dbpedia-spotlight.org/faq
it says it can't be used for tweets since it is highly sensitive to the text structure.
I don't agree, I have tried it myself on tweets. The keywords given by DBpedia are relevant in general, you just need to find the right confidence value. You can try it out on some tweets and you will see by yourself if the results are satisfying.
- Badges
- Science method
- Similar topics
- Teaching