{"id":1636,"date":"2024-03-18T01:53:24","date_gmt":"2024-03-18T01:53:24","guid":{"rendered":"https:\/\/adyouri.com\/?p=1636"},"modified":"2024-03-20T03:24:11","modified_gmt":"2024-03-20T03:24:11","slug":"selenium-proxy-firefox","status":"publish","type":"post","link":"https:\/\/adyouri.com\/selenium-proxy-firefox","title":{"rendered":"How to Use a Proxy in Selenium v4 with Firefox &amp; Python (2024)"},"content":{"rendered":"\n<p>Have you been struggling to get a proxy to work in Selenium while using the Firefox web driver?<\/p>\n\n\n\n<p>Rest assured, in this tutorial, I will give you Python Selenium code that actually works for proxying requests in Selenium with a Firefox web driver.<\/p>\n\n\n\n<p>Selenium is a browser automation tool that is used to build bots, test web applications and scrape data from the internet. Its&nbsp;<a href=\"https:\/\/adyouri.com\/python-web-scraping-libraries#4-selenium\">Python library<\/a>&nbsp;is a popular choice for web automation and web scraping.<\/p>\n\n\n\n<p>If you\u2019re not sure what\u2019s a proxy, check out my&nbsp;<a href=\"https:\/\/adyouri.com\/what-is-a-web-scraping-proxy\">What is a Web Scraping Proxy?<\/a>&nbsp;guide.<\/p>\n\n\n\n<p>Without further ado, here is a clear, step-by-step guide on how to use a proxy in Selenium v4 with Firefox in Python.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"768\" height=\"512\" src=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Firefox-Selenium-Proxy-1.png\" alt=\"How to use a proxy in selenium v4 with Firefox\" class=\"wp-image-1637\" srcset=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Firefox-Selenium-Proxy-1.png 768w, https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Firefox-Selenium-Proxy-1-300x200.png 300w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure>\n\n\n\n<p><strong>Note: <\/strong>If you are using the Chrome webdriver, checkout my <a href=\"https:\/\/adyouri.com\/selenium-proxy-python\">How to Use a Proxy in Selenium v4 with Chrome<\/a> tutorial.<\/p>\n\n\n<style>.kb-table-of-content-nav.kb-table-of-content-id1636_65028b-7d .kb-table-of-content-wrap{padding-top:var(--global-kb-spacing-sm, 1.5rem);padding-right:var(--global-kb-spacing-sm, 1.5rem);padding-bottom:var(--global-kb-spacing-sm, 1.5rem);padding-left:var(--global-kb-spacing-sm, 1.5rem);}.kb-table-of-content-nav.kb-table-of-content-id1636_65028b-7d .kb-table-of-contents-title-wrap{padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;}.kb-table-of-content-nav.kb-table-of-content-id1636_65028b-7d .kb-table-of-contents-title{font-weight:regular;font-style:normal;}.kb-table-of-content-nav.kb-table-of-content-id1636_65028b-7d .kb-table-of-content-wrap .kb-table-of-content-list{font-weight:regular;font-style:normal;margin-top:var(--global-kb-spacing-sm, 1.5rem);margin-right:0px;margin-bottom:0px;margin-left:0px;}<\/style>\n\n\n<h2 class=\"wp-block-heading\" id=\"how-to-use-a-proxy-in-selenium-with-firefox\">How to Use a Proxy in Selenium v4 with Firefox<\/h2>\n\n\n\n<p>For experienced developers with no time for explanations, here is the Python code you need to use a Proxy in Selenium with the Firefox driver:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import time\nfrom selenium import webdriver\nfrom selenium.webdriver.firefox.options import Options\nfrom selenium.webdriver.common.by import By\n\noptions = Options()\noptions.set_preference(\"network.proxy.type\", 1)\noptions.set_preference(\"network.proxy.http\", \"207.2.120.19\")\noptions.set_preference(\"network.proxy.http_port\", 80)\n\ndriver = webdriver.Firefox(\n    options=options\n)\n\ndriver.get(\"http:\/\/httpbin.org\/ip\")\n\nprint(driver.find_element(By.TAG_NAME, \"body\").text)\n\ntime.sleep(5)\ndriver.close()\n<\/code><\/pre>\n\n\n\n<p>The output should have your proxy server\u2019s IP address instead of your local machine\u2019s IP:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"708\" height=\"454\" src=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/carbon-1.png\" alt=\"selenium proxy in firefox\" class=\"wp-image-1595\" srcset=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/carbon-1.png 708w, https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/carbon-1-300x192.png 300w\" sizes=\"auto, (max-width: 708px) 100vw, 708px\" \/><\/figure>\n\n\n\n<p>That\u2019s it! You are now using a proxy in Selenium with a Firefox web driver. If the code above is unclear, or you can\u2019t get it to work, then carefully read the full detailed guide below.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-01-\u2013-installing-selenium\">Step 01 \u2013 Installing Selenium<\/h2>\n\n\n\n<p>First, make sure you have Firefox installed on your system.<\/p>\n\n\n\n<p>Next, create a new Python folder called&nbsp;<code>selenium-proxy<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mkdir selenium-proxy<\/code><\/pre>\n\n\n\n<p>Create a new Python virtual environment inside it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python -m venv env<\/code><\/pre>\n\n\n\n<p>Activate the virtual environment:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>source env\/bin\/activate<\/code><\/pre>\n\n\n\n<p>Install Selenium:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install selenium<\/code><\/pre>\n\n\n\n<p>Next, add a file called&nbsp;<code>scraper.py<\/code>&nbsp;and then add the following code to it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from selenium import webdriver\n\ndriver = webdriver.Firefox()\ndriver.get(\"https:\/\/www.python.org\")\n\nprint(driver.title)\n\nprint(driver.current_url)\n\ndriver.close()<\/code><\/pre>\n\n\n\n<p>Here\u2019s a brief explanation of each part:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><code>from selenium import webdriver<\/code>: This line imports the Selenium module which provides tools for automating web browsers.<\/li>\n\n\n\n<li><code>driver = webdriver.Firefox()<\/code>: This line creates an instance of the Firefox web driver, which allows Python to control the Firefox web browser.<\/li>\n\n\n\n<li><code>driver.get(\"https:\/\/www.python.org\")<\/code>: This instructs the web driver to navigate to the URL \u201c<a href=\"https:\/\/www.python.org\/\" target=\"_blank\" rel=\"noopener\">https:\/\/www.python.org<\/a>\u201d.<\/li>\n\n\n\n<li><code>print(driver.title)<\/code>: This prints the title of the web page currently loaded in the browser (in this case, the Python website).<\/li>\n\n\n\n<li><code>print(driver.current_url)<\/code>: This prints the current URL of the web page loaded in the browser.<\/li>\n\n\n\n<li><code>driver.close()<\/code>: This closes the browser window opened by the web driver.<\/li>\n<\/ol>\n\n\n\n<p>Now run&nbsp;<code>scraper.py<\/code>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python scraper.py<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"output\">Output<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>Welcome to Python.org\nhttps:&#47;&#47;www.python.org\/<\/code><\/pre>\n\n\n\n<p>The preceding output indicates your Selenium Firefox environment is working properly. If you are facing issues, check that you\u2019ve correctly set up your environment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-02-\u2013-using-a-proxy-with-selenium-in-firefox\">Step 02 \u2013 Using a Proxy with Selenium in Firefox<\/h2>\n\n\n\n<p>With Selenium and Firefox successfully set up. Let\u2019s add a proxy.<\/p>\n\n\n\n<p>Inside your&nbsp;<code>selenium-proxy<\/code>&nbsp;project folder, create a new file called&nbsp;<code>proxy.py<\/code>, and add the following code to it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import time\nfrom selenium import webdriver\nfrom selenium.webdriver.firefox.options import Options\nfrom selenium.webdriver.common.by import By\n\noptions = Options()\noptions.set_preference(\"network.proxy.type\", 1)\noptions.set_preference(\"network.proxy.http\", \"207.2.120.19\")\noptions.set_preference(\"network.proxy.http_port\", 80)\n\ndriver = webdriver.Firefox(\n    options=options\n)\n\ndriver.get(\"http:\/\/httpbin.org\/ip\")\n\nprint(driver.find_element(By.TAG_NAME, \"body\").text)\n\ntime.sleep(5)\ndriver.close()<\/code><\/pre>\n\n\n\n<p>Here is a breakdown of the preceding code:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><code>import time<\/code>: This imports the time module, which is used to add time delays in the script.<\/li>\n\n\n\n<li><code>from selenium import webdriver<\/code>: This imports the webdriver module from the Selenium library, which provides tools for automating web browsers.<\/li>\n\n\n\n<li><code>from selenium.webdriver.firefox.options import Options<\/code>: This imports the Options class from the Selenium Firefox module, which allows customizing browser options.<\/li>\n\n\n\n<li><code>from selenium.webdriver.common.by import By<\/code>: This imports the By class from the Selenium library, which provides mechanisms for selecting elements in the web page.<\/li>\n\n\n\n<li><code>options = Options()<\/code>: This creates an instance of the Options class, which is used to set browser preferences.<\/li>\n\n\n\n<li><code>options.set_preference(\"network.proxy.type\", 1)<\/code>: This sets the browser\u2019s network proxy type to use a manual proxy configuration.<\/li>\n\n\n\n<li><code>options.set_preference(\"network.proxy.http\", \"207.2.120.19\")<\/code>: This sets the HTTP proxy address to \u201c207.2.120.19\u201d.<\/li>\n\n\n\n<li><code>options.set_preference(\"network.proxy.http_port\", 80)<\/code>: This sets the HTTP proxy port to 80.<\/li>\n\n\n\n<li><code>driver = webdriver.Firefox(options=options)<\/code>: This initializes a new instance of the Firefox web driver with the custom options set.<\/li>\n\n\n\n<li><code>driver.get(\"http:\/\/httpbin.org\/ip\")<\/code>: This instructs the browser to navigate to the URL \u201c<a href=\"http:\/\/httpbin.org\/ip\" target=\"_blank\" rel=\"noopener\">http:\/\/httpbin.org\/ip<\/a>\u201d, which is a website that returns information about the client\u2019s IP address.<\/li>\n\n\n\n<li><code>print(driver.find_element(By.TAG_NAME, \"body\").text)<\/code>: This finds the body element of the web page and prints its text content, which in this case should contain information about the client\u2019s IP address.<\/li>\n\n\n\n<li><code>time.sleep(5)<\/code>: This pauses the script execution for 5 seconds, allowing time for the web page to load and display its content.<\/li>\n\n\n\n<li><code>driver.close()<\/code>: This closes the browser window opened by the web driver.<\/li>\n<\/ol>\n\n\n\n<p>The output should be as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"origin\": \"207.2.120.19\"\n}<\/code><\/pre>\n\n\n\n<p>To use an SSL Proxy with Selenium in Firefox, implement the following options:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>options.set_preference(\"network.proxy.type\", 1)\noptions.set_preference(\"network.proxy.ssl\", SSL_IP)\noptions.set_preference(\"network.proxy.ssl_port\", SSL_PORT)<\/code><\/pre>\n\n\n\n<p>To use proxy authentication in Firefox, check out the next step.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-03-\u2013-selenium-proxy-authentication-with-python-in-firefox\">Step 03 \u2013 Selenium Proxy Authentication with Python in Firefox<\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"768\" height=\"512\" src=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Selenium-Proxy-Auth-1.png\" alt=\"How to use Proxy Authentication in Selenium v4 with Firefox\" class=\"wp-image-1598\" srcset=\"https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Selenium-Proxy-Auth-1.png 768w, https:\/\/adyouri.com\/wp-content\/uploads\/2024\/03\/Selenium-Proxy-Auth-1-300x200.png 300w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure>\n\n\n\n<p>If you want to use a proxy that requires authentication with a username and password. The simplest way to achieve this is by using the&nbsp;<a href=\"https:\/\/pypi.org\/project\/selenium-wire\/\" target=\"_blank\" rel=\"noopener\"><code>selenium-wire<\/code><\/a>&nbsp;extension. Selenium Wire makes it really easy to use both unauthenticated and authenticated proxies in Selenium.<\/p>\n\n\n\n<p>To install Selenium Wire:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install selenium-wire<\/code><\/pre>\n\n\n\n<p>To use proxy authentication with Selenium Wire in Firefox, you can just pass the username and password of your proxy server in the&nbsp;<code>options<\/code>&nbsp;dictionary like so:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import time\nfrom seleniumwire import webdriver\n\noptions = {\n'proxy': {\n    'http': 'http:\/\/USERNAME:PASSWORD@PROXY_SERVER:PORT',\n    'https': 'https:\/\/USERNAME:PASSWORD@PROXY_SERVER:PORT',\n    'no_proxy': 'localhost,127.0.0.1,dev_server:8080'\n    }\n}\n\ndriver = webdriver.Firefox(seleniumwire_options=options)\n\ndriver.get('http:\/\/httpbin.org\/ip')\ntime.sleep(5)\ndriver.close()<\/code><\/pre>\n\n\n\n<p>If you\u2019re proxy uses the&nbsp;<code>Bearer<\/code>&nbsp;scheme, you can pass your authorization token in a&nbsp;<code>Proxy-Authorization<\/code>&nbsp;header using the&nbsp;<code>custom_authorization<\/code>&nbsp;option:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>options = {\n    'proxy': {\n        'https': 'https:\/\/192.168.10.100:8888',  # No username or password used\n        'custom_authorization': 'Bearer mytoken123'  # Custom Proxy-Authorization header value\n    }\n}<\/code><\/pre>\n\n\n\n<p>For more on how to use proxies, check out the&nbsp;<a href=\"https:\/\/pypi.org\/project\/selenium-wire\/#proxies\" target=\"_blank\" rel=\"noopener\">Selenium Wire Documentation<\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"sources\">Sources<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/selenium-python.readthedocs.io\/api.html#module-selenium.webdriver.common.proxy\" target=\"_blank\" rel=\"noopener\">Python Selenium Documentation<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pypi.org\/project\/selenium-wire\/#proxies\" target=\"_blank\" rel=\"noopener\">Selenium Wire Documentation<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n\n<p>Congrats, you now know how to use a proxy in Python Selenium v4.0 with a Firefox web driver. You also learned how to use proxy authentication in Selenium for proxy servers protected with a username and password.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"faq-selenium-firefox-and-proxies\">FAQ: Selenium, Firefox, and Proxies<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"why-is-using-proxies-important-in-web-scraping-with-selenium-and-firefox\">Why is using proxies important in web scraping with Selenium and Firefox?<\/h3>\n\n\n\n<p>Proxies are essential in web scraping to mask your real IP address and avoid being blocked or rate-limited by websites.<\/p>\n\n\n\n<p>By rotating through a pool of proxies, you can distribute requests across multiple IP addresses, mitigating the risk of detection and ensuring uninterrupted scraping operations.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"can-i-rotate-proxies-automatically-while-scraping-with-selenium-and-firefox\">Can I rotate proxies automatically while scraping with Selenium and Firefox?<\/h3>\n\n\n\n<p>Yes, you can rotate proxies automatically by managing a pool of proxies and switching between them in your Selenium script.<\/p>\n\n\n\n<p>Implementing a proxy rotation mechanism involves periodically selecting a new proxy from the pool and updating the Firefox options accordingly. This approach helps distribute requests evenly and reduces the likelihood of IP bans or detection.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"are-there-any-limitations-or-considerations-when-using-proxies-with-selenium-and-firefox-for-web-scraping\">Are there any limitations or considerations when using proxies with Selenium and Firefox for web scraping?<\/h3>\n\n\n\n<p>While proxies provide anonymity and flexibility in web scraping, there are some considerations to keep in mind.<\/p>\n\n\n\n<p>Firstly, free proxies may be unreliable or slow, affecting scraping performance. Additionally, websites may detect and block proxy IP addresses, requiring frequent updates to your proxy pool. Also, using proxies may incur additional costs, especially when utilizing premium or rotating proxy services.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Have you been struggling to get a proxy to work in Selenium while using the Firefox web driver? Rest assured, in this tutorial, I will give you Python Selenium code that actually works for proxying requests in Selenium with a Firefox web driver. Selenium is a browser automation tool that is used to build bots,&#8230;<\/p>\n","protected":false},"author":1,"featured_media":1637,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"footnotes":""},"categories":[17,19],"tags":[18],"class_list":["post-1636","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-api-tutorials","category-selenium","tag-proxy"],"_links":{"self":[{"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/posts\/1636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/comments?post=1636"}],"version-history":[{"count":3,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/posts\/1636\/revisions"}],"predecessor-version":[{"id":1644,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/posts\/1636\/revisions\/1644"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/media\/1637"}],"wp:attachment":[{"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/media?parent=1636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/categories?post=1636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/adyouri.com\/wp-json\/wp\/v2\/tags?post=1636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}