Writing a Proxy Server Rotation Script in Python

A rotating proxy server is an IP rotation service with a set of IPs, usually rotating multiple times a day. Rotating proxy server comes in handy in web scraping and anonymous web browsing as it can bypass IP bans and rate limits.

Recently, I have used ProxyMesh proxy server for a project. ProxyMesh offers 15 proxy servers, each denoting a specific location (such as us-dc) with 10 IP addresses rotating twice per day.

However, the free trial allows you to have one proxy server at a time. This means that if you are working with a server or CDN that strictly throttles IPs, you have to change the proxy server manually to keep rotating between the 15 proxy servers.

Fortunately, ProxyMesh provides an API that can be used to add and delete the proxy servers in user dashboard. Once a proxy server is assigned to the user, it can be fetched and used. All we need is a script that rotates between the proxy servers, deleting and adding them as required.

A Proxy Rotation Script in Python:

I have written a script that would fetch existing proxy server from ProxyMesh, deletes it, and sets a new proxy server. The new proxy server can either be set randomly or by using specific index, from the list of proxy servers provided by ProxyMesh.

Prerequisites:

An account with ProxyMesh, either free trial or paid. The user name and password will be used in the proxy rotation script.

self.user = ""
self.password = ""

Usage:

Setting the Proxy Server

from rotatingproxy import RotatingProxy

rproxy = RotatingProxy()

The proxy server can either be set randomly or selected from an available list of proxy servers. The active proxy server is saved in a text file which can be accessed as required.

# select a random proxy server
rproxy.set_proxy(israndom="r") 

# select proxy server with index=1 from the list of proxy servers.
rproxy.set_proxy(proxy_num=1) 

Accessing the Proxy Server

def get_proxy_from_file():
    
    # fetches proxy from proxy.txt
    with open("proxy.txt", "r") as f:
        return loads(f.read())

proxy = get_proxy_from_file()

The proxy can now be used with Python Requests or any other library or module that requires a proxy:

import requests
response = requests.get("url-to-fetch", proxies=proxy)

The proxy server rotation script is available at GitHub [link here].

If you don’t want to get into the hassle of manually configuring and managing the proxies, here’s an alternative.

Kashif Aziz

Python Developer in Karachi, Pakistan. Interested in Web Scraping | Data Mining | Web Bots | Python Development using Django. [Check my Portfolio here]

If you found the above post interesting, join my list to get updated when the next one comes up.

For consulting assignments, contact me here with your requirements.