{"@attributes":{"version":"2.0"},"channel":{"title":"ScrapingBee \u2013 The Best Web Scraping API","link":"https:\/\/www.scrapingbee.com\/","description":"Recent content on ScrapingBee \u2013 The Best Web Scraping API","generator":"Hugo","language":"en-us","lastBuildDate":"Fri, 17 Apr 2026 00:00:00 +0000","item":[{"title":"How to handle infinite scroll pages in C#","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-handle-infinite-scroll-pages-in-c\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-handle-infinite-scroll-pages-in-c\/","description":"

Nowadays, most websites use different methods and techniques to decrease the load and data served to their clients\u2019 devices. One of these techniques is the infinite scroll.<\/p>\n

In this tutorial, we will see how we can scrape infinite scroll<\/a> web pages using a\u00a0js_scenario<\/a>, specifically the\u00a0scroll_y<\/code>\u00a0and\u00a0scroll_x<\/code>\u00a0features. And we will use\u00a0this page<\/a>\u00a0as a demo. Only 9 boxes are loaded when we first open the page, but as soon as we scroll to the end of it, we will load 9 more, and that will keep happening each time we scroll to the bottom of the page.<\/p>"},{"title":"Adding items to an eCommerce shopping cart","link":"https:\/\/www.scrapingbee.com\/tutorials\/adding-items-to-an-ecommerce-shopping-cart\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/adding-items-to-an-ecommerce-shopping-cart\/","description":"

Here is a quick tutorial on how you may add items to a shopping cart on eCommerce websites using ScrapingBee API via a JS scenario on Python.<\/p>\n

1. You would need to identify any CSS selector that uniquely identifies the button or 'add to cart' element you wish to click. This can be done via the inspect element option on any browser, more details can be found on this tutorial:
https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-css-selectors-using-chrome\/<\/a><\/p>"},{"title":"Data extraction in C#","link":"https:\/\/www.scrapingbee.com\/tutorials\/data-extraction-in-c\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/data-extraction-in-c\/","description":"

One of the most important features of ScrapingBee, is the ability to extract exact data without need to post-process the request\u2019s content using external libraries.<\/p>\n

We can use this feature by specifying an additional parameter with the name\u00a0extract_rules<\/code>. We specify the label of elements we want to extract, their CSS Selectors and ScrapingBee will do the rest!<\/p>\n

Let\u2019s say that we want to extract the title & the subtitle of the\u00a0data extraction documentation page<\/a>. Their CSS selectors are\u00a0h1<\/code>\u00a0and\u00a0span.text-[20px]<\/code>\u00a0respectively. To make sure that they\u2019re the correct ones, you can use the JavaScript function:\u00a0document.querySelector("CSS_SELECTOR")<\/code>\u00a0in that page\u2019s developer tool\u2019s console.<\/p>"},{"title":"How to extract content from a Shadow DOM","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-content-from-a-shadow-dom\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-content-from-a-shadow-dom\/","description":"

Certain websites may hide all of their page content inside a shadow root, which makes scraping them quite challenging. This is because most scrapers cannot directly access HTML content embedded within a shadow root. Here is a guide on how you can extract such data via ScrapingBee.<\/p>\n

\n

We will use a quite popular site as an example: www.msn.com<\/a>
If you inspect any article on this page, let\u2019s use this one<\/a>. You can see that all of its contents are inside a shadow root:
\"\"<\/p>"},{"title":"How to extract curl requests from Chrome","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-chrome\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-chrome\/","description":"

    \n
  1. Open the\u00a0Network<\/a>\u00a0tab in the\u00a0DevTools<\/a><\/li>\n
  2. Right click (or Ctrl-click) a request<\/li>\n
  3. Click "Copy" \u2192 "Copy as cURL"<\/li>\n
  4. You can now paste it in the relevant\u00a0curl converter<\/a>\u00a0to translate it in the language you want<\/li>\n<\/ol>\n\"Screenshot\"\/"},{"title":"How to extract curl requests from Firefox","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-firefox\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-firefox\/","description":"
      \n
    1. Open the\u00a0Network Monitor<\/a>\u00a0tab in the\u00a0Developer Tools<\/a><\/li>\n
    2. Right click (or Ctrl-click) a request<\/li>\n
    3. Click "Copy" \u2192 "Copy as cURL"<\/li>\n
    4. You can now paste it in the relevant\u00a0curl converter<\/a>\u00a0to translate it in the language you want<\/li>\n<\/ol>\n\"Screenshot\"\/"},{"title":"How to extract curl requests from Safari","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-safari\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-extract-curl-requests-from-safari\/","description":"
        \n
      1. Open the\u00a0Network<\/a>\u00a0tab in the\u00a0Developer Tools<\/a><\/li>\n
      2. Right click (or Ctrl-click or two-finger click) a request<\/li>\n
      3. Click "Copy as cURL" in the dropdown menu<\/li>\n
      4. You can now paste it in the relevant\u00a0curl converter<\/a>\u00a0to translate it in the language you want<\/li>\n<\/ol>\n\"Screenshot\"\/"},{"title":"How to remove any element from the HTML response","link":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-remove-any-element-from-the-html-response\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/how-to-remove-any-element-from-the-html-response\/","description":"

        Sometimes you may need to remove specific HTML elements from the page's content, either to get cleaner results for your data extraction rules<\/a>, or to simply delete unnecessary content from your response.<\/p>\n

        To achieve that using ScrapingBee, you can use aJavaScript Scenario<\/a>, with an evaluate instruction and execute this custom JS code:<\/p>\n

        document.querySelectorAll("ELEMENT-CSS-SELECTOR").forEach(function(e){e.remove();});\u200b\n<\/code><\/pre>
        For example, to remove all of the <style> elements from the response, you can use this JavaScript Scenario:<\/p>"},{"title":"Scrolling and loading more content via a JS scenario","link":"https:\/\/www.scrapingbee.com\/tutorials\/scrolling-and-loading-more-content-via-a-js-scenario\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/scrolling-and-loading-more-content-via-a-js-scenario\/","description":"
        Certain websites may require you to scroll in order to load more results on the page or within a specific element. 

        \"\"

        This is a quick guide on how to achieve different scrolling behaviors using JavaScript scenario.
        *Note that the JavaScript Scenario has a maximum execution time limit of 40 seconds. Requests exceeding this limit will result in a timeout:<\/strong>         https:\/\/www.scrapingbee.com\/documentation\/js-scenario\/#timeout<\/strong><\/a><\/p>\n
        \n
        1. Scrolling a Specific Element<\/h3>\n
        Some page elements, such as tables or graphs, may contain content that only becomes visible after scrolling. 
        \"\"<\/p>"},{"title":"Scrolling via page API","link":"https:\/\/www.scrapingbee.com\/tutorials\/scrolling-via-page-api\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/scrolling-via-page-api\/","description":"
        Some pages load more content only after you click \u201cLoad more results\u201d or scroll and wait. In reality, the page often fetches additional results from its own API. If ScrapingBee can\u2019t load those results, you can target the site\u2019s API URL directly. 

        Here\u2019s how to do that using this URL as an example:         https:\/\/www.reuters.com\/technology<\/a>
        \"\"
        *Note that the JavaScript Scenario has a maximum execution time limit of 40 seconds. Requests exceeding this limit will result in a timeout:<\/strong>         https:\/\/www.scrapingbee.com\/documentation\/js-scenario\/#timeout<\/strong><\/a><\/p>"},{"title":"Make concurrent requests in C#","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-c\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-c\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        using System;\nusing System.IO;\nusing System.Net;\nusing System.Web;\nusing System.Threading;\n\nnamespace test {\n class test{\n\n private static string BASE_URL = "https:\/\/app.scrapingbee.com\/api\/v1\/?";\n private static string API_KEY = "YOUR-API-KEY";\n\n public static string Get(string uri)\n {\n HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);\n request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;\n\n using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())\n using(Stream stream = response.GetResponseStream())\n using(StreamReader reader = new StreamReader(stream))\n {\n return reader.ReadToEnd();\n }\n }\n\n public static bool Scrape(string uri, string path) {\n Console.WriteLine("Scraping " + uri);\n var query = HttpUtility.ParseQueryString(string.Empty);\n query["api_key"] = API_KEY;\n query["url"] = uri;\n string queryString = query.ToString(); \/\/ Transforming the URL queries to string\n\n string output = Get(BASE_URL+queryString); \/\/ Make the request\n try {\n using (StreamWriter sw = File.CreateText(path))\n {\n sw.Write(output);\n }\n return true;\n } catch {return false;}\n }\n\n public static void Main(string[] args) {\n Thread thread1 = new Thread(() => Scrape("https:\/\/scrapingbee.com\/blog", ".\/scrapingbeeBlog.html"));\n Thread thread2 = new Thread(() => Scrape("https:\/\/scrapingbee.com\/documentation", ".\/scrapingbeeDocumentation.html"));\n thread1.Start();\n thread2.Start();\n\n }\n }\n}\n<\/code><\/pre>"},{"title":"Make concurrent requests in Go","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-go\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-go\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        Making concurrent requests in GoLang is as easy as adding a \u201cgo\u201d keyword before our scraping functions! The code below will make two concurrent requests to ScrapingBee\u2019s pages, and save the content in an HTML file.<\/p>"},{"title":"Make concurrent requests in NodeJS","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-nodejs\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-nodejs\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        Making concurrent requests in NodeJS is very straightforward using Cluster module. The code below will make two concurrent requests to ScrapingBee\u2019s pages, and save the content in an HTML file.<\/p>"},{"title":"Make concurrent requests in PHP","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-php\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-php\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        Making concurrent requests in PHP is as easy as creating threads for our scraping functions! The code below will make two concurrent requests to ScrapingBee\u2019s pages and display the HTML content of each page:<\/p>"},{"title":"Make concurrent requests in Python","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-python\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-python\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        import<\/span> concurrent.futures\n<\/span><\/span>import<\/span> time\n<\/span><\/span>from<\/span> scrapingbee import<\/span> ScrapingBeeClient # Importing SPB's client<\/span>\n<\/span><\/span>client =<\/span> ScrapingBeeClient(api_key=<\/span>'YOUR-API-KEY'<\/span>) # Initialize the client with your API Key, and using screenshot_full_page parameter to take a screenshot!<\/span>\n<\/span><\/span>\n<\/span><\/span>MAX_RETRIES =<\/span> 5<\/span> # Setting the maximum number of retries if we have failed requests to 5.<\/span>\n<\/span><\/span>MAX_THREADS =<\/span> 4<\/span>\n<\/span><\/span>urls =<\/span> ["http:\/\/scrapingbee.com\/blog"<\/span>, "http:\/\/reddit.com\/"<\/span>]\n<\/span><\/span>\n<\/span><\/span>def<\/span> scrape<\/span>(url):\n<\/span><\/span> for<\/span> _ in<\/span> range(MAX_RETRIES):\n<\/span><\/span> response =<\/span> client.<\/span>get(url, params=<\/span>{'screenshot'<\/span>: True<\/span>}) # Scrape!<\/span>\n<\/span><\/span>\n<\/span><\/span> if<\/span> response.<\/span>ok: # If we get a successful request<\/span>\n<\/span><\/span> with<\/span> open(".\/"<\/span>+<\/span>str(time.<\/span>time())+<\/span>"screenshot.png"<\/span>, "wb"<\/span>) as<\/span> f:\n<\/span><\/span> f.<\/span>write(response.<\/span>content) # Save the screenshot in the file "screenshot.png"<\/span>\n<\/span><\/span> break<\/span> # Then get out of the retry loop<\/span>\n<\/span><\/span> else<\/span>: # If we get a failed request, then we continue the loop<\/span>\n<\/span><\/span> print(response.<\/span>content)\n<\/span><\/span>\n<\/span><\/span>with<\/span> concurrent.<\/span>futures.<\/span>ThreadPoolExecutor(max_workers=<\/span>MAX_THREADS) as<\/span> executor:\n<\/span><\/span> executor.<\/span>map(scrape, urls)\n<\/span><\/span><\/code><\/pre><\/div>"},{"title":"Make concurrent requests in Ruby","link":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-ruby\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/make-concurrent-requests-in-ruby\/","description":"
        Our API is designed to allow you to have multiple concurrent scraping operations. That means you can speed up scraping for hundreds, thousands or even millions of pages per day, depending on your plan.<\/p>\n
        The more concurrent requests limit you have the more calls you can have active in parallel, and the faster you can scrape.<\/p>\n
        Making concurrent requests in Ruby is as easy as creating threads for our scraping functions! The code below will make two concurrent requests to ScrapingBee\u2019s pages and display the HTML content of each page:<\/p>"},{"title":"Retry failed requests in C#","link":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-c\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-c\/","description":"
        For most websites, your first requests will always be successful, however, it\u2019s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won\u2019t charge you for the request.<\/p>\n
        In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:<\/p>\n
        using System;\nusing System.IO;\nusing System.Net;\nusing System.Web;\nusing System.Collections.Generic;\n\nnamespace test {\n class test{\n\n private static string BASE_URL = @"https:\/\/app.scrapingbee.com\/api\/v1\/?";\n private static string API_KEY = "YOUR-API-KEY";\n\n public static Dictionary<string, dynamic> Get(string uri)\n {\n HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);\n request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;\n\n using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())\n using(Stream stream = response.GetResponseStream())\n using(StreamReader reader = new StreamReader(stream))\n {\n Dictionary<string, dynamic> OutputList = new Dictionary<string, dynamic>();\n OutputList.Add("StatusCode", response.StatusCode);\n OutputList.Add("Response", reader.ReadToEnd());\n return OutputList;\n }\n }\n\n public static void Main(string[] args) {\n\n var query = HttpUtility.ParseQueryString(string.Empty);\n query["api_key"] = API_KEY;\n query["url"] = @"https:\/\/scrapingbee.com\/blog";\n string queryString = query.ToString(); \/\/ Transforming the URL queries to string\n\n const int MAX_RETRIES = 5; \/\/ Set the maximum number of retries we're looking to execute\n\n for (int i = 0; i < MAX_RETRIES; i++) {\n try {\n\n var output = Get(BASE_URL+queryString); \/\/ Make the request\n var StatusCode = output["StatusCode"];\n var content = output["Response"];\n\n if (StatusCode == HttpStatusCode.OK) { \/\/ If the response is 200\/OK\n string path = @".\/ScrapingBeeBlog.html"; \/\/ Output file\n \/\/ Create a file to write to.\n using (StreamWriter sw = File.CreateText(path))\n {\n sw.Write(output);\n }\n Console.WriteLine("Done!");\n break;\n } else {\n Console.WriteLine("Failed request; Status code: " + StatusCode);\n Console.WriteLine("Retrying...");\n }\n\n } catch (Exception ex) {\n Console.WriteLine("An error has occured:" + ex.Message);\n Console.WriteLine("Retrying...");\n }\n\n }\n\n }\n }\n}\n<\/code><\/pre>"},{"title":"Retry failed requests in Go","link":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-go\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-go\/","description":"
        For most websites, your first requests will always be successful, however, it\u2019s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won\u2019t charge you for the request.<\/p>\n
        In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:<\/p>\n
        package main\n\nimport (\n "fmt"\n "io"\n "net\/http"\n "os"\n)\n\nconst API_KEY = "YOUR-API-KEY"\nconst SCRAPINGBEE_URL = "https:\/\/app.scrapingbee.com\/api\/v1"\n\nfunc save_page_to_html(target_url string, file_path string) (interface{}, error) { \/\/ Using sync.Waitgroup to wait for goroutines to finish\n\n req, err := http.NewRequest("GET", SCRAPINGBEE_URL, nil)\n if err != nil {\n return nil, fmt.Errorf("Failed to build the request: %s", err)\n }\n\n q := req.URL.Query()\n q.Add("api_key", API_KEY)\n q.Add("url", target_url)\n req.URL.RawQuery = q.Encode()\n\n client := &http.Client{}\n resp, err := client.Do(req)\n if err != nil {\n return nil, fmt.Errorf("Failed to request ScrapingBee: %s", err)\n }\n defer resp.Body.Close()\n\n if resp.StatusCode != http.StatusOK {\n return nil, fmt.Errorf("Error request response with status code %d", resp.StatusCode)\n }\n\n bodyBytes, err := io.ReadAll(resp.Body)\n\n file, err := os.Create(file_path)\n if err != nil {\n return nil, fmt.Errorf("Couldn't create the file ", err)\n }\n\n l, err := file.Write(bodyBytes) \/\/ Write content to the file.\n if err != nil {\n file.Close()\n return nil, fmt.Errorf("Couldn't write content to the file ", err)\n }\n err = file.Close()\n if err != nil {\n return nil, fmt.Errorf("Couldn't close the file ", err)\n }\n\n return l, nil\n}\n\nfunc main() {\n\n MAX_RETRIES := 5 \/\/ Set a maximum number of retries\n\n target_url := "https:\/\/www.scrapingbee.com"\n\n for i := 0; i < MAX_RETRIES; i++ {\n saved_screenshot, err := save_page_to_html(target_url,".\/scrapingbee.html")\n if err != nil {\n fmt.Println(err)\n fmt.Println("Retrying...")\n } else {\n fmt.Println("Done!", saved_screenshot)\n break\n }\n }\n\n}\n<\/code><\/pre>"},{"title":"Retry failed requests in PHP","link":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-php\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-php\/","description":"
        For most websites, your first requests will always be successful, however, it\u2019s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won\u2019t charge you for the request.<\/p>\n
        In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:<\/p>\n
        <?php\n\n\/\/ Get cURL resource\n$ch = curl_init();\n\n\/\/ Set base url & API key\n$BASE_URL = "https:\/\/app.scrapingbee.com\/api\/v1\/?";\n$API_KEY = "YOUR-API-KEY";\n\n\/\/ Set max retries:\n$MAX_RETRIES = 5;\n\n\/\/ Set parameters\n$parameters = array(\n 'api_key' => $API_KEY,\n 'url' => 'https:\/\/www.scrapingbee.com' \/\/ The URL to scrape\n);\n\/\/ Building the URL query\n$query = http_build_query($parameters);\n\n\/\/ Set the URL for cURL\ncurl_setopt($ch, CURLOPT_URL, $BASE_URL.$query);\n\n\/\/ Set method\ncurl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');\n\n\/\/ Return the transfer as a string\ncurl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);\n\nfor ($i = 0; $i < $MAX_RETRIES; $i++) {\n\n \/\/ Send the request and save response to $response\n $response = curl_exec($ch);\n\n \/\/ Stop if fails\n if (!$response) {\n die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));\n }\n\n $status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);\n echo 'HTTP Status Code: ' . $status_code . PHP_EOL;\n\n \/\/ If it's a successful request (200 or 404 status code):\n if (in_array($status_code, array(200, 404))) {\n echo 'Response Body: ' . $response . PHP_EOL;\n break;\n } else {\n echo 'Retrying...';\n }\n\n}\n\n\/\/ Close curl resource to free up system resources\ncurl_close($ch);\n?>\n<\/code><\/pre>"},{"title":"Retry failed requests in Python","link":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-python\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-python\/","description":"
        For most websites, your first requests will always be successful, however, it\u2019s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won\u2019t charge you for the request.<\/p>\n
        In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:<\/p>\n
        from<\/span> scrapingbee import<\/span> ScrapingBeeClient # Importing SPB's client<\/span>\n<\/span><\/span>client =<\/span> ScrapingBeeClient(api_key=<\/span>'YOUR-API-KEY'<\/span>) # Initialize the client with your API Key, and using screenshot_full_page parameter to take a screenshot!<\/span>\n<\/span><\/span>\n<\/span><\/span>MAX_RETRIES =<\/span> 5<\/span> # Setting the maximum number of retries if we have failed requests to 5.<\/span>\n<\/span><\/span>\n<\/span><\/span>for<\/span> _ in<\/span> range(MAX_RETRIES):\n<\/span><\/span> response =<\/span> client.<\/span>get("http:\/\/scrapingbee.com\/blog"<\/span>, params=<\/span>{'screenshot'<\/span>: True<\/span>}) # Scrape!<\/span>\n<\/span><\/span>\n<\/span><\/span> if<\/span> response.<\/span>ok: # If we get a successful request<\/span>\n<\/span><\/span> with<\/span> open(".\/screenshot.png"<\/span>, "wb"<\/span>) as<\/span> f:\n<\/span><\/span> f.<\/span>write(response.<\/span>content) # Save the screenshot in the file "screenshot.png"<\/span>\n<\/span><\/span> break<\/span> # Then get out of the retry loop<\/span>\n<\/span><\/span> else<\/span>: # If we get a failed request, then we continue the loop<\/span>\n<\/span><\/span> print(response.<\/span>content)\n<\/span><\/span><\/code><\/pre><\/div>"},{"title":"Retry failed requests in Ruby","link":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-ruby\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/retry-failed-requests-in-ruby\/","description":"
        For most websites, your first requests will always be successful, however, it\u2019s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won\u2019t charge you for the request.<\/p>\n
        In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:<\/p>\n
        require 'net\/http'\nrequire 'net\/https'\nrequire 'addressable\/uri'\n\n# Classic (GET)\ndef send_request(user_url)\n uri = Addressable::URI.parse("https:\/\/app.scrapingbee.com\/api\/v1\/")\n api_key = "YOUR-API-KEY"\n uri.query_values = {\n 'api_key' => api_key,\n 'url' => user_url\n }\n uri = URI(uri)\n\n # Create client\n http = Net::HTTP.new(uri.host, uri.port)\n http.use_ssl = true\n http.verify_mode = OpenSSL::SSL::VERIFY_PEER\n\n # Create Request\n req = Net::HTTP::Get.new(uri)\n\n # Fetch Request\n res = http.request(req)\n\n # Return Response\n return res\nrescue StandardError => e\n puts "HTTP Request failed (#{ e.message })"\nend\n\nmax_retries = 5\nfor a in 1..max_retries do\n request = send_request("https:\/\/scrapingbee.com")\n if not [404, 200].include?(request.code)\n puts "Request failed - Status Code: #{ request.code }"\n puts "Retrying..."\n else\n puts "Successful request - Status Code: #{ request.code }"\n puts request.body\n break\n end\nend\n<\/code><\/pre>
        \u00a0<\/p>"},{"title":"Getting started with ScrapingBee and C#","link":"https:\/\/www.scrapingbee.com\/tutorials\/getting-started-with-scrapingbee-and-c\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/www.scrapingbee.com\/tutorials\/getting-started-with-scrapingbee-and-c\/","description":"
        In this tutorial, we will see how you can use ScrapingBee\u2019s API with C#, and use it to scrape web pages. As such, we will cover these topics:<\/p>\n