Jianwei Yang (@jw2yang4ai) / X

Jianwei Yang

481 posts

Jianwei Yang

@jw2yang4ai

Omni @xAI. Core contributor of Project Florence, Phi-3V, Omniparser; (Co-)Inventor of FocalNet, SEEM, SoM, DeepStack (in Qwen3VL) and Magma.

Bellevue, WA

Joined July 2016

Jianwei Yang
@jw2yang4ai
Feb 14, 2025
We will release our project, code, model and training data next week after the long weekend. Stay tuned!
Microsoft Research
@MSFTResearch
Feb 14, 2025
Learn about Magma, a new multimodal agentic foundation model designed for UI navigation in digital environments and robotics manipulation in physical settings with @jw2yang4ai at Microsoft Research Forum on February 25. Register and watch here: msft.it/6017UqcPl
42K
Jianwei Yang
@jw2yang4ai
Nov 1, 2022
Excited to share our FocalNets achieve new SoTA 64.3(test)/64.2(val) on COCO object detection, w/ 3x smaller model/data size than prior arts! It is the first ATTENTION-FREE arch ever in the last 2yrs beating ALL attention-based models on this leaderboard! github.com/microsoft/Foca…
This post is unavailable.
Jianwei Yang
@jw2yang4ai
Jun 16, 2025
Life Update: Now that I have finished the presentation of last @MSFTResearch project Magma at @CVPR, I am excited to share that I have joined @AIatMeta as a research scientist to further push forward the boundary of multimodal foundation models! I have always been passionate
30K
Jianwei Yang
@jw2yang4ai
Jul 11, 2023
🔥We are releasing Semantic-SAM! It jointly learns on 12M images from 7 datasets, and delivers a strong and unique model that produces and recognizes masks of diverse granularities w/ one click! Check out the paper, demo, and imagine your applications! github.com/UX-Decoder/Sem…
This post is unavailable.
89K
Jianwei Yang
@jw2yang4ai
Apr 28, 2023
We are delighted to release the inference code for SEEM! With one line command, you can now play with SEEM locally! We have seen a lot of cool demos with the fantastic SAM, now you can integrate SEEM to build demos as well! Please check out the code here! github.com/UX-Decoder/Seg…
AK
@_akhaliq
Apr 14, 2023
Segment Everything Everywhere All at Once abs: arxiv.org/abs/2304.06718 @Gradio demo: 36771ee9c49a4631.gradio.app
56K
Jianwei Yang
@jw2yang4ai
Jul 2, 2021
We are releasing Focal Transformer! A new self-attention mechanism with fine-grain attention surrounding but gradually coarser-grain attention far away. By modeling the short and long-range visual dependencies in a focal manner, we are delivering new SoTA on COCO and ADE20K!
AK
@_akhaliq
Jul 2, 2021
Focal Self-attention for Local-Global Interactions in Vision Transformers pdf: arxiv.org/pdf/2107.00641… largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation
Jianwei Yang
@jw2yang4ai
Apr 14, 2023
🔥Check out our SEEM! A new image segmentation interface with a single model supporting prompts like text, points, boxes, scribbles and even ref images! It is versatile, compositional, interactive, and semantic-aware! Code: shorturl.at/bpuKV Demo: shorturl.at/apsV0
AK
@_akhaliq
Apr 14, 2023
Segment Everything Everywhere All at Once abs: arxiv.org/abs/2304.06718 @Gradio demo: 36771ee9c49a4631.gradio.app
67K
Jianwei Yang
@jw2yang4ai
Nov 8, 2023
With the stunning GPT-4V API, we finally built a demo that connects SoM with GPT-4V! It helps you recognize the objects in images, instructs you how to use tools, and even guides you to navigate in the living space! A lot of opportunity ahead! Try it here: github.com/microsoft/SoM
00:00
This post is unavailable.
99K
Jianwei Yang
@jw2yang4ai
Jun 4, 2022
Checkout out our @Gradio demo (huggingface.co/spaces/jw2yang…) on how UniCL works for "zero-shot" image recognition! [1/n]
This post is unavailable.
Jianwei Yang
@jw2yang4ai
Apr 17, 2023
Honestly, I really do not understand why this should be an arXiv paper, this has been shown in our instruct X-Decoder demo, and recently grounding SAM. I am confused and also want to advocate we are doing researches, let engineering people do the engineer!
AK
@_akhaliq
Apr 17, 2023
Inpaint Anything: Segment Anything Meets Image Inpainting abs: arxiv.org/abs/2304.06790 github: github.com/geekyutao/Inpa…
113K
Jianwei Yang
@jw2yang4ai
Feb 19, 2025
Thanks for featuring our work! @arankomatsuzaki. 🔥Today we are thrilled to announce our MSR flagship project Magma! This is a fully open-sourced project. We will roll out all the stuff: code, model and training data through the following days. Check out our full work here:
Aran Komatsuzaki
@arankomatsuzaki
Feb 19, 2025
Microsoft presents: Magma: A Foundation Model for Multimodal AI Agents - SotA on UI navigation and robotic manipulation tasks - Pretrained on a large dataset annotated with Set-of-Mark (SoM) for action grounding and Trace-of-Mark (ToM) for action planning.
36K
Jianwei Yang
@jw2yang4ai
Sep 29, 2021
Focal Attention is accepted by NeurIPS as Spotlight! This could not happen without the efforts of all authors! Got some nice suggestions from reviewers and ACs, and of course a common concern about the runtime. Don’t worry, we are working hard to accelerate our Focal!
This post is unavailable.
Jianwei Yang
@jw2yang4ai
Nov 10, 2023
Inspired by our simple SoM prompting, a number of projects are developing automatic web navigators using GPT-4V! So mind-blowing! Can't wait to have a deeper integration! GPT-4V-ACT: github.com/ddupont808/GPT… GPT-V-on-Web: github.com/Jiayi-Pan/GPT-… vimGPT: github.com/ishan0102/vimG…
02:17
Jianwei Yang
@jw2yang4ai
Nov 8, 2023
With the stunning GPT-4V API, we finally built a demo that connects SoM with GPT-4V! It helps you recognize the objects in images, instructs you how to use tools, and even guides you to navigate in the living space! A lot of opportunity ahead! Try it here: github.com/microsoft/SoM
43K
Jianwei Yang
@jw2yang4ai
Mar 4, 2023
Accepted by #CVPR2023! X-Decoder is the FIRST generalist decoder that supports all segmentation tasks (ins/sem/pano/ref) in OPEN VOCABULARY, both inter- AND intra-image VL tasks, and even helps instruct image inpainting/editing! New demo below and more at huggingface.co/xdecoder!
00:00
01:13
AK
@_akhaliq
Dec 22, 2022
X-Decoder: Generalized Decoding for Pixel, Image and Language Hugging Face demo: huggingface.co/spaces/xdecode… abs: arxiv.org/abs/2212.11270 project page: x-decoder-vl.github.io github: github.com/microsoft/X-De…
52K