Lorena Gongang
Tech journey with Lorena

Follow

Tech journey with Lorena

Follow
Things to consider before working on a web scraping project

Photo by Markus Spiske on Unsplash

Things to consider before working on a web scraping project

Lorena Gongang's photo
Lorena Gongang
·Apr 25, 2022·

2 min read

Play this article

Table of contents

  • Is it legal to web scrap websites?
  • Look at the robots.txt file.
  • Consider the quantity and the speed of extraction

I had to admit that I did things in an entire reversely way. I firstly build a web scraping project without questioning myself about the points to consider before jumping into it.

This article aims to provide information about what to know before working on a web scraping project from my experience. In addition, I also did some research on the topic.

The first question I asked myself was: is this thing called web scraping I'm doing legal? After some research, I found that scraping websites where no account is required for access or where the user has made the data public was not problematic.

So now, it is clear that you can scrap data from websites if there are public.

Look at the robots.txt file.

Websites use the Robots.txt file to define if or how a bot should crawl or scrape the website. The file is accessible using the URL of the website by doing example.com/robots.txt.

When working on a web scraping project, you can access it to find if the website allows web scraping.

Consider the quantity and the speed of extraction

Don't do excessive web scraping. It is essential to web scrap data with a time interval between two extractions. Extracting too fast like a robot without pause can cause the website to block you.

DISCLAIMER: You may still get blocked if the server detects you are trying to scrape large amounts of data with your script;

Conclusion

Here is some information to consider before working on a web scraping project.

 
Share this