Recently, when I have to solve a sentiment analysis problem the dataset contains a lot of impurities and missing data. I tried to clean data but it takes lot of time. While text itself contains more symbols which is inconsistent with text. How can we remove with less time?
Dear Vrushali Lanjewar
Data cleasing step really takes a lot of time. We can say about 70% of our time spent on the whole process of collecting, processing, analyzing and presenting the results comes from this step.
I believe this problem will be solved in two ways:
#1 Using tricks and calling a function to make it easier to see impurities and remove it, like:
- Identifying outliers
- // Incomplete data
- // Duplicate data
- // Punctuations and emojis
- Identifying and removing stopwords (the same idea in Word Clouds projects)
and so on.
#2 Automate data cleaning
You can use Python and its Pandas Dataframe library to do this and then import your CSV file into a database such as PostgreSQL.
The following address below demonstrates this step-by-step in a complete way that I could not provide here:
https://towardsdatascience.com/automating-data-science-tasks-on-python-to-save-hours-2a1ad7a0311f
In addition, you can also choose to develop an unsupervised machine learning algorithm and make a relation to your input data.
As for the first way, you will have more control over the quality of your data, however you won't save as much time as with the second way (automated) which although it is much better, we should be careful how we use it, because it can consider things shouldn't be in the project if we are not careful in its configuration.
Thank you Daniel Henrique Pereira but is there any tool to remove it before using data for training or validation purposes.
https://stackoverflow.com/questions/64719706/cleaning-twitter-data-pandas-python
https://www.mygreatlearning.com/blog/data-cleaning-in-python/
Hope it helps
there are many function used to clear dataset such as fitering, on hot encoding ,transformation and normalization to vonvert data into binary to make it easier to treat with less time
there are many function used to clear dataset such as fitering, on hot encoding ,transformation and normalization to vonvert data into binary to make it easier to treat with less time
there are many function used to clear dataset such as fitering, on hot encoding ,transformation and normalization to vonvert data into binary to make it easier to treat with less time
- Badges
- Science topic