Skip to content

ultralytics 8.3.187 SAM2: Add SAM2DynamicInteractivePredictor support few-shot inference#21232

Merged
glenn-jocher merged 155 commits intoultralytics:mainfrom
ShuaiLYU:add_SAM2DynamicInteractivePredictor
Aug 27, 2025
Merged

ultralytics 8.3.187 SAM2: Add SAM2DynamicInteractivePredictor support few-shot inference#21232
glenn-jocher merged 155 commits intoultralytics:mainfrom
ShuaiLYU:add_SAM2DynamicInteractivePredictor

Conversation

@ShuaiLYU
Copy link
Copy Markdown
Contributor

@ShuaiLYU ShuaiLYU commented Jun 29, 2025

I have read the CLA Document and I sign the CLA

Add SAM2DynamicInteractivePredictor for Dynamic Memory-based Segmentation

📋 Description

This PR introduces SAM2DynamicInteractivePredictor, a new interactive predictor that extends pre-trained SAM2 capabilities with dynamic memory management for multi-object segmentation tasks.

✨ Key Features

  • Dynamic Memory Management: Maintains object memory across multiple images for consistent tracking
  • Interactive Segmentation: Supports bbox-based prompt
  • Flexible Object Tracking: Configurable maximum object number with automatic memory updates
  • Seamless Integration: Built on existing SAM2 architecture with enhanced functionality

🔧 Changes Made

  • Added SAM2DynamicInteractivePredictor class in ultralytics.models.sam
  • Implemented memory-based object tracking across image sequences
  • Added support for conditional memory updates based on user input insead of the previous frames.

Application Scenarios:

  • The system supports semi - automatic annotation with a small amount of human prompts, greatly reducing the workload of traditional annotation.
  • In object detection tasks, an online detection model can be constructed through a small number of image annotations. It can continuously be improved through human feedback, enabling universal object presence/absence detection and online updates.
    n more suitable for advanced annotation and video analysis tasks!**

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Adds a new training-free SAM 2 mode for dynamic, interactive segmentation and tracking across images/videos, plus related fixes and docs updates. 🚀

📊 Key Changes

  • New SAM2DynamicInteractivePredictor
    • Real-time prompt updates (boxes/points/masks) during inference 🖱️
    • Continual learning: refine existing objects over time ♻️
    • Cross-image memory for consistent tracking across independent images 🧠
    • Non-overlapping mask handling and confidence-based filtering in results
    • Public API export and reference docs added
  • Improved SAM 2 internals
    • Safer mask-as-output path and refactored track step flow
    • Corrected feature ordering in get_im_features for SAM 2
    • Robust mask prompt preprocessing (letterbox + nearest-neighbor, dtype fixes)
    • Postprocess now filters masks by confidence
  • Docs
    • New “Dynamic Interactive Segment and Track” guide with examples
    • Reference page for SAM2DynamicInteractivePredictor
    • Author metadata update
  • Version bump: 8.3.187

🎯 Purpose & Impact

  • Faster interactive workflows: Add or refine objects on-the-fly without retraining, ideal for video annotation, editing, and surveillance. ⚡
  • Better tracking across frames and images via a managed memory bank, improving consistency and robustness. 🎯
  • Easier adoption: Simple API to add prompts and update memory; works with pre-trained SAM 2 models. ✅
  • More reliable outputs: Confidence filtering and non-overlap options yield cleaner masks. 🧼

Quick start:

from ultralytics.models.sam import SAM2DynamicInteractivePredictor

overrides = dict(model="sam2_t.pt", imgsz=1024, task="segment", mode="predict", conf=0.01)
predictor = SAM2DynamicInteractivePredictor(overrides=overrides, max_obj_num=10)

# Add an object by box prompt and update memory
predictor.inference(img="image1.jpg", bboxes=[[100,100,200,200]], obj_ids=[1], update_memory=True)

# Track the object in a new image
results = predictor(source="image2.jpg")

See the Ultralytics Docs for details.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jun 29, 2025

All Contributors have signed the CLA. ✅
Posted by the CLA Assistant Lite bot.

@UltralyticsAssistant UltralyticsAssistant added enhancement New feature or request python Pull requests that update python code segment Instance Segmentation issues, PR's labels Jun 29, 2025
@UltralyticsAssistant
Copy link
Copy Markdown
Member

👋 Hello @ShuaiLYU, thank you for submitting an ultralytics/ultralytics 🚀 PR! To ensure a seamless integration of your work, please review the following checklist:

  • Define a Purpose: Please confirm your PR description clearly explains the purpose and scope of your contribution. If possible, link to any relevant issues for added context. Ensure your commit messages are clear and concise.
  • Synchronize with Source: Make sure your PR is up to date with the main branch of ultralytics/ultralytics. If your branch is behind, use the 'Update branch' button or run git pull and git merge main locally.
  • Ensure CI Checks Pass: Check that all Ultralytics Continuous Integration (CI) checks have passed. If there are any failures, please resolve them.
  • Update Documentation: If your PR introduces new features or changes existing functionality, update the relevant documentation accordingly.
  • Add Tests: Where applicable, include or update tests to ensure your changes are covered and passing.
  • Sign the CLA: If this is your first Ultralytics contribution, please sign our Contributor License Agreement (CLA) by commenting "I have read the CLA Document and I sign the CLA" in this PR.
  • Minimize Changes: Please limit your changes to only what's necessary for your addition or bug fix. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

For further details, you can refer to our Contributing Guide. If your PR addresses a bug and you haven't already, please provide a minimum reproducible example (MRE) so we can verify the fix.

This is an automated response 🛠️—an Ultralytics engineer will review your contribution and assist you soon. Thank you for helping improve Ultralytics! 🚀✨

@codecov
Copy link
Copy Markdown

codecov bot commented Jun 29, 2025

Codecov Report

❌ Patch coverage is 16.94915% with 98 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ultralytics/models/sam/predict.py 14.67% 93 Missing ⚠️
ultralytics/models/sam/modules/sam.py 16.66% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ShuaiLYU ShuaiLYU force-pushed the add_SAM2DynamicInteractivePredictor branch 3 times, most recently from 1907b82 to d6f8b56 Compare June 29, 2025 11:24
@ShuaiLYU
Copy link
Copy Markdown
Contributor Author

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I sign the CLA

You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

I have read the CLA Document and I sign the CLA

@ShuaiLYU ShuaiLYU force-pushed the add_SAM2DynamicInteractivePredictor branch 2 times, most recently from 89f69d7 to df2f278 Compare June 29, 2025 11:39
@ShuaiLYU
Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I sign the CLA

@ShuaiLYU ShuaiLYU force-pushed the add_SAM2DynamicInteractivePredictor branch 4 times, most recently from d05205e to 9a87e7e Compare June 29, 2025 12:39
@ShuaiLYU ShuaiLYU force-pushed the add_SAM2DynamicInteractivePredictor branch from d565a29 to 3d8f554 Compare June 29, 2025 15:29
@ShuaiLYU ShuaiLYU force-pushed the add_SAM2DynamicInteractivePredictor branch from 5b5444f to 1cb488c Compare August 5, 2025 13:04
@Laughing-q Laughing-q force-pushed the add_SAM2DynamicInteractivePredictor branch from b142c20 to 68aa0d2 Compare August 21, 2025 07:28
@Laughing-q Laughing-q changed the title add SAM2DynamicInteractivePredictor SAM2: Add SAM2DynamicInteractivePredictor support few-shot inference Aug 21, 2025
@Laughing-q Laughing-q changed the title SAM2: Add SAM2DynamicInteractivePredictor support few-shot inference ultralytics 8.3.187 SAM2: Add SAM2DynamicInteractivePredictor support few-shot inference Aug 26, 2025
Comment thread docs/mkdocs_github_authors.yaml
@glenn-jocher glenn-jocher merged commit a1ed237 into ultralytics:main Aug 27, 2025
16 checks passed
@UltralyticsAssistant
Copy link
Copy Markdown
Member

Absolutely thrilled to see this merged — huge thanks @ShuaiLYU, @Laughing-q, and @glenn-jocher! “Great things are done by a series of small things brought together.” — Vincent van Gogh. Your collective effort delivers a training-free, dynamic SAM 2 experience that makes interactive segmentation and cross-image tracking faster, cleaner, and more intuitive for real-world annotation, editing, and surveillance workflows.

Quick start:

from ultralytics.models.sam import SAM2DynamicInteractivePredictor

overrides = dict(model="sam2_t.pt", imgsz=1024, task="segment", mode="predict", conf=0.01)
predictor = SAM2DynamicInteractivePredictor(overrides=overrides, max_obj_num=10)

# Add an object via box prompt and update memory
predictor.inference(img="image1.jpg", bboxes=[[100, 100, 200, 200]], obj_ids=[1], update_memory=True)

# Track it in a new image
results = predictor(source="image2.jpg")

See the Ultralytics Docs for details: https://docs.ultralytics.com

Brilliant work pushing the boundaries of dynamic, interactive segmentation — this will be a game-changer for our users. 🚀

@Laughing-q
Copy link
Copy Markdown
Member

Laughing-q commented Aug 27, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request python Pull requests that update python code segment Instance Segmentation issues, PR's

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants