Hi,
I'm working on a data that each sentence is in separate rows in dataframe. I want to determine word frequency in each row with a word list that I have created. Then I would like to add these values to my dataframe as a new variable in R.
For example:
Let's say, I have a list of words that contains apple and banana. And I have three independent sentences in rows:
1. X x x apple x x banana x apple.
2. X apple x x x x.
3. X x babana x apple x.
Now I want to calculate how many times the words in my list have been repeated. As a result, the new column I want to create is:
1. = 3
2. = 1
3. = 2
Thanks in advance.
Hi Fatih,
you will work with textmining. In this context you have to deal with the three topics: tokenization, stemming and tolower.
From your question it is not clear in which kind of data (text/sentences) you want to look for your words an if you want to look also for terms e.g. banana tree
A very basic way:
clean your keywords: singular, all lower, charachter.
###Keywords
Dear Fatih Bayrak,
You can use the function table() to count frequencies. For example, suppose that
r1
Hi Fatih,
I would transform the data frame into a character matrix, then use the lapply() function to search each row for any one of the key words.
for(i in 1:dim(Data)[2]){Data[,i]
Mohamed I. Riffi, Florian Jansen, Thank you for your attention, but the data I'm working on is not as small as the example I've given above. Data consists of thousands rows with different sentences. Therefore, I cannot do analysis one by one using the "
Dear Fatih Bayrak,
always try to provide minimal examples of your data. Most often it is not as obvious as you might think and for sure it will be the most time consuming part of possible answers. I remember you wrote about lists?
If your data is really a data.frame, is each sentence stored in a single cell, or every word?
@ Fatih Bayrak
Even if your df has millions of rows, you can start with a simple function that counts the frequency of the specific words you want in a row, where df is the name of your data.frame. Then apply this function to the df using the for loop. For example,
for(i in 1:nrow(df){
f(df[ , i])
}
where f is the function that counts the frequencies of the specific words in each row of the database.
Frequency is statistical equation which determine the number of response on its corresponding scale and can be used by this formula fx multiply by X divide the size of given population.
Hi Fatih Bayrak , you mentioned that you wanted an R package to identify frequency; tidytext - https://www.tidytextmining.com/ , does exactly that, include code which will give frequency counts across a dataframe. From your initial description, your data should already be amenable to the code in this package.
You would first unnest_tokens(), which would take your 1 sentence = 1 row, form and turn it into a tibble, with the sentence numbers and words as variables.
Next, you the count() function on this new tibble, which will return a word count. If you want frequencies, divide this count by the total number.
You will also want to use the tolower() function mentioned by Stefan Krottenthaler , although I believe several of the functions in the tidytext package perform this by default.
Good luck!
I've always used grepl with a combination of how I'm trying to find. In addition, someone already mentioned - you lemmatize the tokens if necessary, match and then go back to the original table. Not sure if this is helpful, but you might have to go deeper into NLTK and parse each sentence as a bag of words, lemmatize, match and go back to original sentence. Good luck!
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R