I would like to parse a webpage and extract meaningful content from it. By meaningful, I mean the content (text only) that the user wants to see in that particular page (data excluding ads, banners, comments etc.) I want to ensure that when a user saves a page, the data that he wanted to read is saved, and nothing else.
In short, I need to build an application which works just like Readability. ( http://www.readability.com ) I need to take this useful content of the web page and store it in a separate file. I don't really know how to go about it.
I don't want to use API's that need me to connect to the internet and fetch data from their servers as the process of data extraction needs to be done offline.
There are two methods that I could think of:
1. Use a machine learning based algorithm
2. Develop a web scraper that could satisfactorily remove all clutter from web pages.
Is there an existing tool that does this? I came across the boilerpipe library ( http://code.google.com/p/boilerpipe/ ) but didn't use it. Has anybody used it? Does it give satisfactory results? Are there any other tools, particularly written in java which do this kind of web scraping?
If I need to build my own tool to do this, what would you guys suggest to go about it?
Since I'd need to clean up messy or incomplete HTML before I begin its parsing, I'd use a tool like Tidy ( http://www.w3.org/People/Raggett/tidy/ ) or Beautiful Soup ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/ ) to do the job.
But I don't know how to extract content after this step.
Thanks a lot!
I'll suggest giving boilerpipe a try. It seems like exactly doing what you want. Even if you don't want to use it, you should read the article mentioned in its page. boilerpipe's classification method and related work mentioned will give you a pretty good idea about what you are up against.
Another option may be installing Readability's Firefox plugin and then inspecting its code to see how it gets the job done:
http://www.instructables.com/id/Extract-Firefox-Addon-Source-Code/
I'll suggest also using browser plugin AdBlock's database (i.e. a bunch of regexp's which match ad URL's) to filter out the ads in your pages:
https://easylist.adblockplus.org/en/
If you need this on a particular site only, you won't need such complex methods. You will be able to extract just what you want after some inspection of the HTML and CSS of content pages and using a DOM library like BeautifulSoup.
As you mentioned, the best way is to develop a web scraper yourself! In this way, you can customize it to fit the problem. In JAVA, I have used IOUtils (Apache Commons IO) API to extract the URL and found it very useful. The rest is dealing with the strings and finding the exact information that you are looking for.
1. Use Regular expressions
2. overlap 2 existing pages in order to extract it's structure.
You could try KNIME workflows; see knime.org. There are many nodes for text processing. Without much knowledge of KNIME you can build a workflow for extracting for instance email addresses from web pages.
Take a look at RIDIRE-CPI (https://github.com/lablita/ridire-cpi): it's a Java software to build a corpus from web. It's quite a big project but inside it you can find a web crawler (Heritrix), an HTML cleaner (in fact Readability is used) and other tools to manage text data.
https://github.com/lablita/ridire-cpi
You may want to look at the following:
Jaunt - http://jaunt-api.com/
Jsoup - http://jsoup.org/
SimpleScrape - http://simplescrape.sourceforge.net/
Hi,
i think this is a current topic for web crawling. For my extractions, i used Python with Selenium library. I think it is a good solution to make an own tool for the full extraction of dynamic web pages.
https://pypi.python.org/pypi/selenium
In the old times, when none of these solutions were even immagined, we have used tools built by our minds. My entire phd thesis is based on that ;)
Hello,
I specialize in public data gathering (web harvesting) from open access websites by programming a web-crawler. The data can later be used for statistical or content analysis. For example, my recent collection was data from booking.com and tripadvisor.com with information about reviews, ratings and prices along with the accompanying data such as geographical region, addresses, and many more. The data comes out in a form that is easily converted to SPSS or Excel format.
Technically, any website or social network can be a source of data. Please feel free to contact me if you are interested as I am open for research collaboration.
Sincerely,
Evgeny
Well, I have tried Octoparse (Bazhuayu in Chinese) before to extract the information I wanted online and found it really easy to use. With little programming knowledge, I find this web scraping tool quite suitable for guys like me. It provides AD blocking as you mentioned. If you are interested, you could have a try.
www.octoparse.com/?qu
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining