it can be used in part(s), but not substantially plagiarized, for educational purposes. see: https://www.eff.org/document/fair-use-presentation-one-pager

I think this recent case might be of interest to you in relation to your query:

http://www.ahrecs.com/resources/publicly-available-data-thousands-okcupid-users-pulled-copyright-claim-retraction-watch-alison-mccook-may-2016

http://retractionwatch.com/2016/05/16/publicly-available-data-on-thousands-of-okcupid-users-pulled-over-copyright-claim/

Thanks! That is helpful. My data wouldn't be about humans though, it's about exotic pets sold online. So, I wonder if the same type of argument can be made.

Possibly. Maybe it would be worth a conversation with legal team at your University?

I scraped the OKCupid data mentioned above, as have many others done before. Database data cannot be copyrighted, so it would be difficult to see how it could be protected. One could argue that the profile text data is copyrightable, which is debatable. Still, the other data is definitely not copyrightable. See https://en.wikipedia.org/wiki/Sui_generis_database_right#United_States

I do not think that in general the copyright issue is relevant (whether there is a copyright depends on the country where the website is registered, pay attention to that--as the Wikipedia link at Emil's answer suggests). Rather, you should worry about the possibility that the website owners could say that you are slowing down their website while scraping. This is important, because it might generate money loss for them and thus represent a legal case against you (money loss may represent a legal case in any western country). You can come up with ways to legally circumvent this issue, though; for example, you could scrape data over several days, little by little, while avoiding to keep continuously logged in on their website.

First you should check the websites "robots.txt" to scrape it according with their scraping policy. Just enter "www.NAME.DOMAIN/robots.txt" in your browser and google for explanation of the codes.

It is legal to scrape websites. The legal cases are scarce and dealt with on a case-by-case basis since the law does not directly address web scraping. Indirectly web scraping is addressed by copyright infringement law (see the wiki link for more info). What the legal cases have in common is that the users of web scraping were indicted if by using the scraped they hurt the websites business/profitability. Bottom line seems to be that if you are scraping data for academic purposes you have nothing to fear.

https://en.wikipedia.org/wiki/Web_scraping#Legal_issues

Why not? If the source quoted correctly, I find no problem. Jiri

it is not illegal, it may break copyright claim, and it may be unethical. I do it quite frequently for my researc, the association of internet researchers ethics guidelines will help.

Dear Oliver C. Stringham ,

I think it is not illegal, but you need to make the data anonymous for publishing because you don't have permission from the specific website to disclose its identity in your research.Also, as a researcher collecting the scraped data from the web, your crawler should respect the rules specified in the 'robots.txt' files of the websites it is scarping.

Any update on this issue? Please share if you have any more recent news or experience. Thanks

If the data is public, you can scrap it, taking into consideration the problems mentioned by Luca Fumarco. This apply to the open, public (anyone can view) content in social networks and in websites. However, if you are required a login and password and the /robots.txt have disable permissions to scrap data, I guess permission is needed. Moreover, in the case of sensitive data (such as children images or information, or the type of data I imagine Emil scraped from OKCupid, I guess there might be ethical concerns. I would feel more comfortable asking permissions even if my sample is reduced. I don't think is fair that you extract data without engaging users just because is technically possible. That's only my feeling.

Thank you Juliana, I totally agree with you.

Francesca Dal Mas, Juliana Elisa Raffaghelli, Mahdieh Zabihimayvan

It is necessary to check both robots.txt and rules of publications in the website. For instance, in https://ssarherps.org/cndb/#c25ha2UmbG9vc2U9dHJ1ZQ there are a lot of reptiles images publicly available, however, it is prohibited to use them for commercial and research use without direct permission (https://calphotos.berkeley.edu/use.html).

Best regards,

Reza Sadeghi

Thank you, we did, and all seems to be fine. 🙏

I am wep scraping some newspaper website as well, and I've never thought about this copyright issue. THX

Dear Oliver, it is perfectly legal if you use an API. If you don't use an API, but a software that directly scrape the data from the raw html (like beautifulsoup, rvest, selenium, etc.), it is also legal if you don't vulnerate sensitive data or private data. I have scraped data from public websites in different ocassions, without any legal problem.

It is legal as long as you don't manipulate the data which will ultimately change the scenario for your research. Still, make sure that you check their robots.txt file once to manually check which paths of their website do they allow or disallow for crawling.

For Example, IMDB has the following robots.txt file

Hello All,

Just to weigh in on this debate. I think it also worth mentioning that make continuous calls to a website does put extra pressure on their service (which costs them money) and in the cases of small website might cause them to run slow or even crash. So to avoid this in addition to the points above regarding robot.txt files and checking terms of service/use it is also worth only scraping if a) there is no API, b) only scraping information you need for your research project and c) inserting pauses in your scrapers call to the website (super simple in r using Sys.sleep(time)).

Aside from all the mentioned facts, I believe we can read texts available on websites or watch their images and get insights. I don't believe using my computer instead of my brain to get insights would be that different. As far as we don't use their publicly available data directly, but just use them to have a better understanding of different concepts.