Scrapy Proxy Integration

This guide may be outdated. For an up-to-date guide please see our documentation.

What is Scrapy?

Scrapy is a Python framework for web crawling and scraping, which allows users to extract structured data from websites. It is open-source, fast, and extensible. Scrapy can be used for various purposes, such as data mining, monitoring, and automated testing.

Get proxies for Scrapy

Scrapy integration with Bright Data proxies

Open your preferred IDE and start a new scrapy project, type in the command line :

      scrapy startproject <project_name>

This will create a new folder with the project name, within the folder open a python file.

Go to your Bright Data Control Panel and clicking the ‘Proxies & Scraping Infra’ icon
Create a new proxy zone by clicking ‘Add’, choosing a network type, configuring the proxy, and clicking save
Under your proxy-zone’s ‘Access parameters’ tab, you will find your ‘USERNAME’ and ‘PASSWORD’ values.
In your scrapy spider code file, within the request’s meta parameter set the ‘proxy’ value to be the following, using the ‘USERNAME’ and ‘PASSWORD’ values from before: “http://USERNAME:[email protected]:33335”
For Example:

      import scrapy



class BrightdatascrapyexampleSpider(scrapy.Spider):

   name = "BrightDataScrapyExample"



  def start_requests(self):

       request = scrapy.Request(url="http://example.com",callback=self.parse)

       request.meta['proxy'] = "http://USERNAME:[email protected]:33335"

       yield request



   def parse(self, response):

       print(response.body)

Then run the following command in your command line :

      scrapy runspider <Pythonfilename.py>

How To Use Bright Data Proxy Manger With Scrapy

Create a proxy zone same as in the direct integration above
Install the Proxy Manager
Click ‘add new port’ and configure it for your use case
In your Scrapy spider code file, within the request’s meta parameter set the ‘proxy’ value to be the following: “http://IP:PORTNUMBER”
The local host IP is 127.0.0.1 – this is the value you need to use if the proxy manager is installed on your machine. If the proxy manager is installed on an external server, input that server’s IP address
The port created in the Proxy Manager is 24XXX, for example, 24000 – the default first port number
For example:

      import scrapy



class BrightdatascrapyexampleSpider(scrapy.Spider):

   name = "BrightDataScrapyExample"



   def start_requests(self):

       request = scrapy.Request(url="http://example.com",callback=self.parse)

       request.meta['proxy'] = "http://127.0.0.1:24000"

       yield request



   def parse(self, response):

       print(response.body)

⚠️Important note: If you are using Bright Data’s Residential Proxies, Web Unlocker or SERP API, you need to install an SSL certificate to enable end-to-end secure connections to your target website(s). This is a simple process, see https://docs.brightdata.com/general/account/ssl-certificate#installation-of-the-ssl-certificate for instructions.