Wassim Diai ,

A web scraper is usually divided into the following components

• The get /post routine- this part carries out the actual extraction of the webpage
• A link/image extractor- this basically scans the web page to extract further pages to crawl. This part of the scraper can also contain domain filtering (so the crawler remains within the domain that it is crawling and also the depth.
• Data structure- A data structure such as a stack or queue is used to store the pages to be crawled
• Behaviour control- Routine to read the robots.txt file to control the 'aggressiveness' of the crawling.

if you want to get started in programming a crawler look for tutorials such as [1] for further details

Regards

[1]https://towardsdatascience.com/how-to-web-scrape-with-python-in-4-minutes-bc49186a8460

This tutorial link maybe useful if you use R

"Tutorial: Web Scraping in R with rvest – Dataquest" https://www.dataquest.io/blog/web-scraping-in-r-rvest/

Dear Wassim Diai

I suggest that you try as many tutorials as you can find since web scraping can be a bit tricky at times. Even more difficult is the fact that websites vary so much in terms of layout. Basic knowledge of mark up languages like HTML and XML, however, can be extremely helpful. When it comes to Competitive Intelligence, it is no different; you just need to select the website and make your script accordingly. You should also check the website's policies to make sure scrapping is allowed. This can be done by appending the "/robots.txt" file to the URL. Please see below if it helps. Thanks

https://www.edureka.co/blog/web-scraping-with-python/

https://towardsdatascience.com/data-analytics-with-python-by-web-scraping-illustration-with-cia-world-factbook-abbdaa687a84

https://bigdata-madesimple.com/complete-guide-web-scraping-python/

The web data scraping process