switching from requests to selenium #72
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
PR Overview
This PR migrates the scraping functionality from the requests-based implementation to a selenium/undetected-chromedriver setup for improved resilience and compatibility with modern search pages. Key changes include:
- Introducing selenium calls and helper methods (_init_chromedriver, _send_chromedriver_request, _conduct_chromedriver_search) in the SearchEngine class.
- Removing or commenting out requests-specific header and session code.
- Updating documentation and dependency lists to reflect the switch to selenium.
- Adding a basic selenium test and updating demo scripts and the README.
Reviewed Changes
| File | Description |
|---|---|
| WebSearcher/searchers.py | Added selenium integrations, removed unused requests session/headers. |
| tests/selenium_test.py | Introduced an example test for selenium-driven search. |
| pyproject.toml | Updated dependencies to include undetected-chromedriver and selenium. |
| README.md | Updated instructions and notes to reflect the new selenium-based approach. |
| scripts/demo_search.py | Updated to launch chromedriver before initiating search. |
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
WebSearcher/searchers.py:169
- [nitpick] The variable name 'ai_button' is misleading since it holds a boolean rather than an element; consider renaming it to 'ai_expand_available' or similar for clarity.
ai_button = self._check_ai_expand()
| self.log.exception(f'SERP | Timeout error | {self.serp_id}') | ||
| except Exception: | ||
| self._send_chromedriver_request() | ||
| except: |
There was a problem hiding this comment.
Avoid using a bare except; specify the exception type (e.g. Exception) to prevent masking unexpected errors.
Suggested change
| except: | |
| except Exception: |
Owner
|
Merge munging notes:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Begin migrating library from requests to selenium. Added selenium calls (built on
undetected-chromedriver) in_conduct_chromedriver_searchand_send_chromedriver_request. Also added an additional step to launch chromedriver to facilitate debugging. test/selenium_test.py contains a functioning example call.TODO: fix broken poetry command, include chromedriver update/path instructions, remove requests-specific headers