The term data harvesting, or web scraping, has always been a concern for website operators and data publishers. Data harvesting is a process where a small script, also known as a malicious bot, is used to automatically extract large amount of data from websites and use it for other purposes. As a cheap and easy way to collect online data, the technique is often used without permission to steal website information such as text, photos, email addresses, and contact lists.

One method of data harvesting targets databases in particular. The script finds a way to cycle through the records of a database and then download each and every record in the database.

Aside from obvious consequence of data loss, data harvesting can also be detrimental to businesses in other ways:

• Poor SEO Ranking: If your website content is scraped, reproduced and used on other sites, this will significantly affect the SEO ranking and performance for your website on search engines.
• Decreased Website Speed: When used repeatedly, scraping attacks can lower the performance of your websites and affect the user experience.
• Lost Market Advantages: Your competitors may use data harvesting to scrape valuable information such as customer lists to gather intelligence about your business.
• I think if you refer it for research then concentrate on techniques rather than tools.

There are also a lot of other Data Mining techniques but these seven are considered more frequently used by peoples.

• Statistics
• Clustering
• Visualization
• Decision Tree
• Association Rules
• Neural Networks
• Classification

There are two tasks you have to do.

First one is to scrap the web content according to your need and based on the harvested data you have to perform mining techniques.

So for web scraping this may be useful to you. https://goo.gl/iEeHss

And for Mining techniques Weka tool is very popular but you can also try some other tools like https://goo.gl/qKumZM

The term data harvesting, or web scraping, has always been a concern for website operators and data publishers. Data harvesting is a process where a small script, also known as a malicious bot, is used to automatically extract large amount of data from websites and use it for other purposes. As a cheap and easy way to collect online data, the technique is often used without permission to steal website information such as text, photos, email addresses, and contact lists.

One method of data harvesting targets databases in particular. The script finds a way to cycle through the records of a database and then download each and every record in the database.

Aside from obvious consequence of data loss, data harvesting can also be detrimental to businesses in other ways:

• Poor SEO Ranking: If your website content is scraped, reproduced and used on other sites, this will significantly affect the SEO ranking and performance for your website on search engines.
• Decreased Website Speed: When used repeatedly, scraping attacks can lower the performance of your websites and affect the user experience.
• Lost Market Advantages: Your competitors may use data harvesting to scrape valuable information such as customer lists to gather intelligence about your business.
• I think if you refer it for research then concentrate on techniques rather than tools.

There are also a lot of other Data Mining techniques but these seven are considered more frequently used by peoples.

• Statistics
• Clustering
• Visualization
• Decision Tree
• Association Rules
• Neural Networks
• Classification

If the web site offers an API, like Twitter or Facebook do, you can collect data directly from within your Java, Python or other application. If there is no API, you can use a web scraper as mentioned before.

you should develop your own program which will crawl relevant information from internet structured text. You can use any computer language along with regular expression to identify relevant information from structured text.

Hello Venkatesh! I'm using for last 6 month only two tools (to gane data that i need): https://www.webscraper.io/ , https://www.mydataprovider.com/ . As i know, some of them are made with focus on web, other for shop etc. So it's just my recommendation. Good luck!

https://www.searchtechnologies.com/blog/web-data-mining-tools-techniques