I know that there are others, my gradstudents know of them. Have to wait till September to ask them, I seem to recall that they mentioned bootcat or

webbootcat  or something like that hope it helps

chris

there is common crawl as well good luck

Hi,

is there a specific usecase you want to solve? Depending on this, it may be easier to recommend something good.

I know for example of yacy, which contains a crawler and has a very open API to get a lot of informations from the crawled sites. It is actually a distributed Search Engine, so it may be overkill and maybe does not even provide you with the data you are really interested in.

If you are interested in more specific stuff, and you have some programming experience, you can use things like the PHP Goutte project, or the more basic Guzzle one. Similar should be available in other programming languages.

http://yacy.de/en/index.html

https://github.com/FriendsOfPHP/Goutte

http://docs.guzzlephp.org/en/latest/

https://www.import.io/ is pretty good at data extraction

You can use https://www.screamingfrog.co.uk/seo-spider/