Skip to content

Adding take a screenshot func for images of screen elements in mobile#4222

Merged
Maheshkale447 merged 8 commits into
Releases/Betafrom
Add-Appium-Operation
Jun 19, 2025
Merged

Adding take a screenshot func for images of screen elements in mobile#4222
Maheshkale447 merged 8 commits into
Releases/Betafrom
Add-Appium-Operation

Conversation

@shai1712

@shai1712 shai1712 commented May 22, 2025

Copy link
Copy Markdown
Collaborator

Thank you for your contribution.
Before submitting this PR, please make sure:

  • PR description and commit message should describe the changes done in this PR
  • Verify the PR is pointing to correct branch i.e. Release or Beta branch if the code fix is for specific release , else point it to master
  • Latest Code from master or specific release branch is merged to your branch
  • No unwanted\commented\junk code is included
  • No new warning upon build solution
  • Code Summary\Comments are added to my code which explains what my code is doing
  • Existing unit test cases are passed
  • New Unit tests are added for your development
  • Sanity Tests are successfully executed for New and Existing Functionality
  • Verify that changes are compatible with all relevant browsers and platforms.
  • After creating pull request there should not be any conflicts
  • Resolve all Codacy comments
  • Builds and checks are passed before PR is sent for review
  • Resolve code review comments
  • Update the Help Library document to match any feature changes

Summary by CodeRabbit

  • New Features
    • Added the ability to capture and display screenshots of individual UI elements when viewing visible controls in mobile automation.
    • Enhanced biometric simulation with a new cancel option for improved interaction control.

@coderabbitai

coderabbitai Bot commented May 22, 2025

Copy link
Copy Markdown
Contributor

Walkthrough

The update enhances the GenericAppiumDriver by capturing full-screen screenshots and generating cropped images for individual UI elements when retrieving visible controls. It restructures biometric simulation handling and introduces private methods for image processing: TakeElementScreenShot, GetLocationAndSizeOfElement, and ScreenshotToImage.

Changes

File(s) Change Summary
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Refactored biometric simulation switch case; added element-level screenshot cropping in GetVisibleControls; introduced private methods for image processing: TakeElementScreenShot, GetLocationAndSizeOfElement, and ScreenshotToImage.

Sequence Diagram(s)

sequenceDiagram
    participant TestRunner
    participant GenericAppiumDriver
    participant AppiumDriver
    participant UIElement

    TestRunner->>GenericAppiumDriver: GetVisibleControls()
    GenericAppiumDriver->>AppiumDriver: GetPageSource()
    GenericAppiumDriver->>AppiumDriver: GetScreenshot()
    loop For each visible element
        GenericAppiumDriver->>AppiumDriver: FindElementByXPath()
        GenericAppiumDriver->>UIElement: GetLocationAndSize()
        GenericAppiumDriver->>GenericAppiumDriver: TakeElementScreenShot(element)
        GenericAppiumDriver->>UIElement: Assign cropped screenshot (base64)
    end
    GenericAppiumDriver-->>TestRunner: Return list of controls with screenshots
Loading

Possibly related PRs

Suggested reviewers

  • MeniKadosh1
  • Maheshkale447

Poem

A snapshot here, a crop just right,
UI elements shining bright.
With code and care, a picture's born,
Of every button, field, and form.
Now each control can proudly show
Its image—thanks to code we grow!
🐰✨

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate Unit Tests
  • Create PR with Unit Tests
  • Post Copyable Unit Tests in Comment
  • Commit Unit Tests in branch Add-Appium-Operation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai auto-generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai or @coderabbitai title anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🔭 Outside diff range comments (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

4448-4461: 🛠️ Refactor suggestion

remoteFilePath variable name is misleading & missing directory-existence check

remoteFilePath actually points to a local file (concatenated with LocalFolderPath).
Rename to avoid confusion and ensure the target directory exists before WriteAllBytes:

- var remoteFilePath = Path.Combine(LocalFolderPath, Path.GetFileName(DeviceFilePath));
+ var localFilePath = Path.Combine(LocalFolderPath, Path.GetFileName(DeviceFilePath));
+ Directory.CreateDirectory(Path.GetDirectoryName(localFilePath));
 ...
- File.WriteAllBytes(remoteFilePath, FileContent);
+ File.WriteAllBytes(localFilePath, FileContent);

If the directory is missing the current implementation throws DirectoryNotFoundException.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ad06927 and c1d827f.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (5 hunks)

Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated
Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated
@prashelke
prashelke changed the base branch from master to Releases/Beta June 18, 2025 13:41
@prashelke
prashelke requested a review from Maheshkale447 June 18, 2025 13:47

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (2)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (2)

2568-2590: fullScreenshot is still never disposed – unmanaged handle leak re-introduced

You moved the Screenshot field to a class-level variable and keep a reference after GetVisibleControls returns.
OpenQA.Selenium.Screenshot wraps an internal Bitmap/GDI handle; keeping many instances alive → GDI exhaustion and crashes on long POM scans.

The fix that was suggested in the previous review is still relevant:

- Screenshot fullScreenshot = null;
+ // Capture once inside GetVisibleControls and wrap with using
+ using Screenshot fullScreenshot = ((ITakesScreenshot)Driver).GetScreenshot();

Or cache one Bitmap (already converted) and Dispose() it in a finally block.


2660-2693: Bitmap produced for EVERY element → O(n²) memory/CPU

TakeElementScreenShot converts the same Screenshot to a Bitmap inside a using block for every single element.
On pages with hundreds of nodes this multiplies memory & CPU needlessly.

Reuse a single Bitmap:

- EI.ScreenShotImage = TakeElementScreenShot(EI, fullScreenshot);
+ // Convert once
+ using var fullImage = ScreenshotToImage(fullScreenshot);
+ ...
+ EI.ScreenShotImage = TakeElementScreenShot(EI, fullImage);

Then change TakeElementScreenShot to accept a Bitmap instead of a Screenshot.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c1d827f and 25f058a.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (5 hunks)

Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

♻️ Duplicate comments (2)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (2)

2589-2589: Memory-safety concern: ScreenshotToImage returns a Bitmap backed by a disposed stream

using var fullImage = ScreenshotToImage(...); disposes fullImage correctly, but the helper still instantiates the Bitmap from a MemoryStream that is disposed inside the helper. The resulting bitmap keeps a reference to that stream and will trigger “A generic error occurred in GDI+” on the first access.

See past review at lines 2721-2727 – issue not yet fixed.


2721-2727: ScreenshotToImage still disposes the backing stream too early

Returning new Bitmap(ms) inside a using disposes ms, leaving the bitmap unusable later.

-return new Bitmap(ms);
+var bmp = new Bitmap(ms);
+return (Bitmap)bmp.Clone();    // decouple from stream

or load via Image.FromStream(ms, true, true) then clone.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 25f058a and d0c4851.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (5 hunks)

Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated
Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated
Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🔭 Outside diff range comments (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

70-78: Remove unused System.Linq.Expressions import

The namespace isn’t referenced anywhere in this file; keeping it triggers CS8019 and adds needless compile time.
Just delete the directive.

♻️ Duplicate comments (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

2692-2703: Clamp & simplify cropping logic

cropRect can still point outside fullImage when the element is partially off-screen, throwing GDI+ errors.
Before creating the rectangle, clamp X/Y/Width/Height to the bitmap size, then use the cheaper clone API:

-Rectangle cropRect = new Rectangle(cropX, cropY, cropWidth, cropHeight);
-using (Bitmap elementImage = new Bitmap(cropRect.Width, cropRect.Height))
-{
-    using (Graphics g = Graphics.FromImage(elementImage))
-    {
-        g.DrawImage(fullImage, new Rectangle(0,0,cropRect.Width,cropRect.Height), cropRect, GraphicsUnit.Pixel);
-    }
+int x = Math.Max(0, cropX);
+int y = Math.Max(0, cropY);
+int w = Math.Min(cropWidth,  fullImage.Width  - x);
+int h = Math.Min(cropHeight, fullImage.Height - y);
+if (w <= 0 || h <= 0) return null;
+Rectangle cropRect = new Rectangle(x, y, w, h);
+using (Bitmap elementImage = fullImage.Clone(cropRect, fullImage.PixelFormat))
 {
@@
 }

This removes the extra Graphics allocation and guards against out-of-range coordinates.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d0c4851 and aa8ee90.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (10 hunks)
🧰 Additional context used
🧠 Learnings (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

undefined

<retrieved_learning>
Learnt from: prashelke
PR: #4232
File: Ginger/GingerCoreNET/ActionsLib/UI/VisualTesting/VRTAnalyzer.cs:219-226
Timestamp: 2025-06-16T10:13:01.319Z
Learning: In VRTAnalyzer.cs for the GenericAppiumDriver class, the mAct.IsFullPageScreenshot flag is not applicable and should not be used to control screenshot behavior. GenericAppiumDriver always uses CaptureFullPageCroppedScreenshot() method regardless of the IsFullPageScreenshot flag setting.
</retrieved_learning>

Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

2669-2716: Add bounds clamping to prevent exceptions.

The cropping logic should validate that the crop rectangle doesn't exceed the full image bounds to prevent ArgumentException when elements are partially off-screen.

 Rectangle cropRect = new Rectangle(cropX, cropY, cropWidth, cropHeight);
+
+// Clamp bounds to prevent exceptions  
+int x = Math.Max(0, cropX);
+int y = Math.Max(0, cropY);
+int w = Math.Max(0, Math.Min(cropWidth, fullImage.Width - x));
+int h = Math.Max(0, Math.Min(cropHeight, fullImage.Height - y));
+
+if (w <= 0 || h <= 0)
+    return null;
+
+Rectangle cropRect = new Rectangle(x, y, w, h);
📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aa8ee90 and 5824573.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (10 hunks)
🧰 Additional context used
🧠 Learnings (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (1)

undefined

<retrieved_learning>
Learnt from: prashelke
PR: #4232
File: Ginger/GingerCoreNET/ActionsLib/UI/VisualTesting/VRTAnalyzer.cs:219-226
Timestamp: 2025-06-16T10:13:01.319Z
Learning: In VRTAnalyzer.cs for the GenericAppiumDriver class, the mAct.IsFullPageScreenshot flag is not applicable and should not be used to control screenshot behavior. GenericAppiumDriver always uses CaptureFullPageCroppedScreenshot() method regardless of the IsFullPageScreenshot flag setting.
</retrieved_learning>

🔇 Additional comments (4)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (4)

1300-1326: LGTM! Improved biometrics simulation structure.

The restructured switch statement with explicit cases for Success, Failure, and Cancel provides better readability and maintainability compared to the previous implementation.


2589-2589: Excellent performance optimization.

Taking the screenshot once and converting to Bitmap upfront prevents repeated conversions for each element, significantly improving performance during POM discovery.


2760-2767: LGTM! Proper bitmap cloning implementation.

The ScreenshotToImage method correctly clones the bitmap to avoid disposal issues with the underlying MemoryStream. This prevents potential GDI+ errors.


2645-2656: Make XPath check case-insensitive.

The XPath checks using Contains("android") and Contains("XCUIElement") are case-sensitive, which may cause elements with differently cased strings to be skipped.

-if ((EI.XPath).Contains("android") || (EI.XPath).Contains("XCUIElement"))
+if (EI.XPath.Contains("android", StringComparison.OrdinalIgnoreCase) ||
+    EI.XPath.Contains("XCUIElement", StringComparison.OrdinalIgnoreCase))
⛔ Skipped due to learnings
Learnt from: prashelke
PR: Ginger-Automation/Ginger#4222
File: Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs:2645-2656
Timestamp: 2025-06-19T05:08:45.170Z
Learning: In the Ginger mobile automation framework (GenericAppiumDriver.cs), XPath checks for "android" and "XCUIElement" should remain case-sensitive as currently implemented. The team prefers not to make these checks case-insensitive.

Comment thread Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5824573 and 256fee8.

📒 Files selected for processing (1)
  • Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (9 hunks)
🧰 Additional context used
🧠 Learnings (1)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (2)

undefined

<retrieved_learning>
Learnt from: prashelke
PR: #4232
File: Ginger/GingerCoreNET/ActionsLib/UI/VisualTesting/VRTAnalyzer.cs:219-226
Timestamp: 2025-06-16T10:13:01.319Z
Learning: In VRTAnalyzer.cs for the GenericAppiumDriver class, the mAct.IsFullPageScreenshot flag is not applicable and should not be used to control screenshot behavior. GenericAppiumDriver always uses CaptureFullPageCroppedScreenshot() method regardless of the IsFullPageScreenshot flag setting.
</retrieved_learning>

<retrieved_learning>
Learnt from: prashelke
PR: #4222
File: Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs:2645-2656
Timestamp: 2025-06-19T05:08:45.170Z
Learning: In the Ginger mobile automation framework (GenericAppiumDriver.cs), XPath checks for "android" and "XCUIElement" should remain case-sensitive as currently implemented. The team prefers not to make these checks case-insensitive.
</retrieved_learning>

🔇 Additional comments (3)
Ginger/GingerCoreNET/Drivers/CoreDrivers/Mobile/Appium/GenericAppiumDriver.cs (3)

2588-2589: 👍 Full-page image is captured only once – good optimisation

Capturing the screenshot once and re-using the Bitmap removes hundreds of device round-trips. Nice catch.


2644-2656: Graceful fallback when element screenshot fails

The new try/catch prevents a single bad node from killing the learn loop – good defensive coding.


2759-2766: Good: bitmap completely detached from the stream

Cloning after new Bitmap(ms) avoids the classic GDI+ “generic error” caused by a disposed stream. Nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants