We want to write our own search engine to find specific regular expressions of interest on the web.
However, we don't want to really have to trawl the whole web, as that would take many years on the computer!
I want to be able to access preexisting web indexing, in order to run our customized search engine front-end.
Has anyone pulled off anything like this before, and can anyone recommend where to find accessible web indexes?
It would be great if it were possible to tap into Google's indexes for research purposes, but if they don't allow that are there any others out there that can be used?
To find that gem you are looking for does indeed require a full traversal (spider crawl) of the Web.
However, you can use the Google API.
Crawling might be the best approach in your case, because that way you can preselect a number of targeted web locations and sites linked from and to the pages on those targeted set of domains. Of course, creating this set of locations will certainly be a difficult job. Especially if you do not have clear targets, but actually target the whole Web as a definite collection of data which you wish to search and match your regular expressions against.
Depending on what you want to do exactly, you may get away with just using an n-gram corpus rather than a crawl. These basically just count occurrences of phrases.
Google released a 5-gram (phrases up to 5 words long) corpus in 2006 that is described here:
http://googleresearch.blogspot.com.au/2006/08/all-our-n-gram-are-belong-to-you.html
and available here:
http://catalog.ldc.upenn.edu/LDC2006T13
There is a newer V2 version floating around somewhere, but I'm a bit confused about whether this was build from web pages or books.
If you absolutely need full content of web pages, then the ClueWeb corpora might be appropriate. Just bear in mind that even the 2009 version is 25 terabytes.
Clueweb 2012: http://boston.lti.cs.cmu.edu/clueweb12
Clueweb 2009: http://lemurproject.org/clueweb09.php/
- Badges
- Science topic
- Similar topics
- Computer Science