Dear Community , in order to analyze a component website and its features ; I want to extract some data from the website , it's first time to hear about the web scraping, can you guide me please , how to do it using python for competition intelligence?
thank you
Wassim Diai ,
A web scraper is usually divided into the following components
- The get /post routine- this part carries out the actual extraction of the webpage
- A link/image extractor- this basically scans the web page to extract further pages to crawl. This part of the scraper can also contain domain filtering (so the crawler remains within the domain that it is crawling and also the depth.
- Data structure- A data structure such as a stack or queue is used to store the pages to be crawled
- Behaviour control- Routine to read the robots.txt file to control the 'aggressiveness' of the crawling.
if you want to get started in programming a crawler look for tutorials such as [1] for further details
Regards
[1]https://towardsdatascience.com/how-to-web-scrape-with-python-in-4-minutes-bc49186a8460
This tutorial link maybe useful if you use R
"Tutorial: Web Scraping in R with rvest – Dataquest" https://www.dataquest.io/blog/web-scraping-in-r-rvest/
Dear Wassim Diai
I suggest that you try as many tutorials as you can find since web scraping can be a bit tricky at times. Even more difficult is the fact that websites vary so much in terms of layout. Basic knowledge of mark up languages like HTML and XML, however, can be extremely helpful. When it comes to Competitive Intelligence, it is no different; you just need to select the website and make your script accordingly. You should also check the website's policies to make sure scrapping is allowed. This can be done by appending the "/robots.txt" file to the URL. Please see below if it helps. Thanks
https://www.edureka.co/blog/web-scraping-with-python/
https://towardsdatascience.com/data-analytics-with-python-by-web-scraping-illustration-with-cia-world-factbook-abbdaa687a84
https://bigdata-madesimple.com/complete-guide-web-scraping-python/
The web data scraping process
- Badges
- Science topic