feat: HSAP platform v2 — modular navigation, quality review, audit log, world model simulation
Major changes: - New frontend (platform/web/): Vite + React 18 + TypeScript + Tailwind - 4-module navigation: 数据送标 / 模型管理 / 车队管理 / 系统管理 - Data catalog with charts (DMS/ADAS/Lane 3-tab view) - Quality review workflow (标注质检): Good/Fine/Bad scoring with auto-advance - Audit enhancements: batch operations, rejection categories, Feishu notifications - Operation audit log (操作日志) - World model simulation studio (仿真工坊) - Dataset version management with snapshots and diff - ADAS 7-class dataset integration (138K images organized + compressed) - User management with Feishu integration and pagination - CRUD/search/filter on all pages, card layout redesign - PIL-optimized image overlay rendering - Auto-snapshot on build, in_review workflow stage - Removed embedded algorithm code (now in workspace)
This commit is contained in:
364
algorithms/dms_yolo/code.embedded.bak/docs/en/models/fast-sam.md
Normal file
364
algorithms/dms_yolo/code.embedded.bak/docs/en/models/fast-sam.md
Normal file
@@ -0,0 +1,364 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover FastSAM, a real-time CNN-based solution for segmenting any object in an image. Efficient, competitive, and ideal for various vision tasks.
|
||||
keywords: FastSAM, Fast Segment Anything Model, Ultralytics, real-time segmentation, CNN, YOLOv8-seg, object segmentation, image processing, computer vision
|
||||
---
|
||||
|
||||
# Fast Segment Anything Model (FastSAM)
|
||||
|
||||
The Fast Segment Anything Model (FastSAM) is a novel, real-time CNN-based solution for the Segment Anything task. This task is designed to segment any object within an image based on various possible user interaction prompts. FastSAM significantly reduces computational demands while maintaining competitive performance, making it a practical choice for a variety of vision tasks.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/F7db-EHhxss"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Object Tracking using FastSAM with Ultralytics
|
||||
</p>
|
||||
|
||||
## Model Architecture
|
||||
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
FastSAM is designed to address the limitations of the [Segment Anything Model (SAM)](sam.md), a heavy [Transformer](https://www.ultralytics.com/glossary/transformer) model with substantial computational resource requirements. The FastSAM decouples the segment anything task into two sequential stages: all-[instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation) and prompt-guided selection. The first stage uses [YOLOv8-seg](../tasks/segment.md) to produce the segmentation masks of all instances in the image. In the second stage, it outputs the region-of-interest corresponding to the prompt.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Real-time Solution:** By leveraging the computational efficiency of CNNs, FastSAM provides a real-time solution for the segment anything task, making it valuable for industrial applications that require quick results.
|
||||
|
||||
2. **Efficiency and Performance:** FastSAM offers a significant reduction in computational and resource demands without compromising on performance quality. It achieves comparable performance to SAM but with drastically reduced computational resources, enabling real-time application.
|
||||
|
||||
3. **Prompt-guided Segmentation:** FastSAM can segment any object within an image guided by various possible user interaction prompts, providing flexibility and adaptability in different scenarios.
|
||||
|
||||
4. **Based on YOLOv8-seg:** FastSAM is based on [YOLOv8-seg](../tasks/segment.md), an object detector equipped with an instance segmentation branch. This allows it to effectively produce the segmentation masks of all instances in an image.
|
||||
|
||||
5. **Competitive Results on Benchmarks:** On the object proposal task on MS COCO, FastSAM achieves high scores at a significantly faster speed than [SAM](sam.md) on a single NVIDIA RTX 3090, demonstrating its efficiency and capability.
|
||||
|
||||
6. **Practical Applications:** The proposed approach provides a new, practical solution for a large number of vision tasks at a really high speed, tens or hundreds of times faster than current methods.
|
||||
|
||||
7. **Model Compression Feasibility:** FastSAM demonstrates the feasibility of a path that can significantly reduce the computational effort by introducing an artificial prior to the structure, thus opening new possibilities for large model architecture for general vision tasks.
|
||||
|
||||
## Available Models, Supported Tasks, and Operating Modes
|
||||
|
||||
This table presents the available models with their specific pretrained weights, the tasks they support, and their compatibility with different operating modes like [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md), indicated by ✅ emojis for supported modes and ❌ emojis for unsupported modes.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ---------- | ------------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| FastSAM-s | [FastSAM-s.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/FastSAM-s.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ✅ |
|
||||
| FastSAM-x | [FastSAM-x.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/FastSAM-x.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ✅ |
|
||||
|
||||
## FastSAM Comparison vs YOLO
|
||||
|
||||
Here we compare Meta's SAM 2 models, including the smallest SAM2-t variant, with Ultralytics smallest segmentation model, [YOLO11n-seg](../tasks/segment.md):
|
||||
|
||||
| Model | Size<br><sup>(MB)</sup> | Parameters<br><sup>(M)</sup> | Speed (CPU)<br><sup>(ms/im)</sup> |
|
||||
| ---------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------- | --------------------------------- |
|
||||
| [Meta SAM-b](sam.md) | 375 | 93.7 | 49401 |
|
||||
| [Meta SAM2-b](sam-2.md) | 162 | 80.8 | 31901 |
|
||||
| [Meta SAM2-t](sam-2.md) | 78.1 | 38.9 | 25997 |
|
||||
| [MobileSAM](mobile-sam.md) | 40.7 | 10.1 | 25381 |
|
||||
| [FastSAM-s](fast-sam.md) with YOLOv8 [backbone](https://www.ultralytics.com/glossary/backbone) | 23.7 | 11.8 | 55.9 |
|
||||
| Ultralytics [YOLOv8n-seg](yolov8.md) | **6.7** (11.7x smaller) | **3.4** (11.4x less) | **24.5** (1061x faster) |
|
||||
| Ultralytics [YOLO11n-seg](yolo11.md) | **5.9** (13.2x smaller) | **2.9** (13.4x less) | **30.1** (864x faster) |
|
||||
|
||||
This comparison demonstrates the substantial differences in model sizes and speeds between SAM variants and YOLO segmentation models. While SAM provides unique automatic segmentation capabilities, YOLO models, particularly YOLOv8n-seg and YOLO11n-seg, are significantly smaller, faster, and more computationally efficient.
|
||||
|
||||
Tests run on a 2025 Apple M4 Pro with 24GB of RAM using `torch==2.6.0` and `ultralytics==8.3.90`. To reproduce this test:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, SAM, YOLO, FastSAM
|
||||
|
||||
# Profile SAM2-t, SAM2-b, SAM-b, MobileSAM
|
||||
for file in ["sam_b.pt", "sam2_b.pt", "sam2_t.pt", "mobile_sam.pt"]:
|
||||
model = SAM(file)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile FastSAM-s
|
||||
model = FastSAM("FastSAM-s.pt")
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile YOLO models
|
||||
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt"]:
|
||||
model = YOLO(file_name)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
The FastSAM models are easy to integrate into your Python applications. Ultralytics provides user-friendly Python API and CLI commands to streamline development.
|
||||
|
||||
### Predict Usage
|
||||
|
||||
To perform [object detection](https://www.ultralytics.com/glossary/object-detection) on an image, use the `predict` method as shown below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
|
||||
# Define an inference source
|
||||
source = "path/to/bus.jpg"
|
||||
|
||||
# Create a FastSAM model
|
||||
model = FastSAM("FastSAM-s.pt") # or FastSAM-x.pt
|
||||
|
||||
# Run inference on an image
|
||||
everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)
|
||||
|
||||
# Run inference with bboxes prompt
|
||||
results = model(source, bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Run inference with points prompt
|
||||
results = model(source, points=[[200, 200]], labels=[1])
|
||||
|
||||
# Run inference with texts prompt
|
||||
results = model(source, texts="a photo of a dog")
|
||||
|
||||
# Run inference with bboxes and points and texts prompt at the same time
|
||||
results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a FastSAM model and segment everything with it
|
||||
yolo segment predict model=FastSAM-s.pt source=path/to/bus.jpg imgsz=640
|
||||
```
|
||||
|
||||
This snippet demonstrates the simplicity of loading a pretrained model and running a prediction on an image.
|
||||
|
||||
!!! example "FastSAMPredictor example"
|
||||
|
||||
This way you can run inference on image and get all the segment `results` once and run prompts inference multiple times without running inference multiple times.
|
||||
|
||||
=== "Prompt inference"
|
||||
|
||||
```python
|
||||
from ultralytics.models.fastsam import FastSAMPredictor
|
||||
|
||||
# Create FastSAMPredictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", model="FastSAM-s.pt", save=False, imgsz=1024)
|
||||
predictor = FastSAMPredictor(overrides=overrides)
|
||||
|
||||
# Segment everything
|
||||
everything_results = predictor("ultralytics/assets/bus.jpg")
|
||||
|
||||
# Prompt inference
|
||||
bbox_results = predictor.prompt(everything_results, bboxes=[[200, 200, 300, 300]])
|
||||
point_results = predictor.prompt(everything_results, points=[200, 200])
|
||||
text_results = predictor.prompt(everything_results, texts="a photo of a dog")
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
All the returned `results` in above examples are [Results](../modes/predict.md#working-with-results) object which allows access predicted masks and source image easily.
|
||||
|
||||
### Val Usage
|
||||
|
||||
Validation of the model on a dataset can be done as follows:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
|
||||
# Create a FastSAM model
|
||||
model = FastSAM("FastSAM-s.pt") # or FastSAM-x.pt
|
||||
|
||||
# Validate the model
|
||||
results = model.val(data="coco8-seg.yaml")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a FastSAM model and validate it on the COCO8 example dataset at image size 640
|
||||
yolo segment val model=FastSAM-s.pt data=coco8.yaml imgsz=640
|
||||
```
|
||||
|
||||
Please note that FastSAM only supports detection and segmentation of a single class of object. This means it will recognize and segment all objects as the same class. Therefore, when preparing the dataset, you need to convert all object category IDs to 0.
|
||||
|
||||
### Track Usage
|
||||
|
||||
To perform object tracking on an image, use the `track` method as shown below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
|
||||
# Create a FastSAM model
|
||||
model = FastSAM("FastSAM-s.pt") # or FastSAM-x.pt
|
||||
|
||||
# Track with a FastSAM model on a video
|
||||
results = model.track(source="path/to/video.mp4", imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo segment track model=FastSAM-s.pt source="path/to/video.mp4" imgsz=640
|
||||
```
|
||||
|
||||
## FastSAM official Usage
|
||||
|
||||
FastSAM is also available directly from the [https://github.com/CASIA-IVA-Lab/FastSAM](https://github.com/CASIA-LMC-Lab/FastSAM) repository. Here is a brief overview of the typical steps you might take to use FastSAM:
|
||||
|
||||
### Installation
|
||||
|
||||
1. Clone the FastSAM repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/CASIA-IVA-Lab/FastSAM.git
|
||||
```
|
||||
|
||||
2. Create and activate a Conda environment with Python 3.9:
|
||||
|
||||
```bash
|
||||
conda create -n FastSAM python=3.9
|
||||
conda activate FastSAM
|
||||
```
|
||||
|
||||
3. Navigate to the cloned repository and install the required packages:
|
||||
|
||||
```bash
|
||||
cd FastSAM
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
4. Install the CLIP model:
|
||||
```bash
|
||||
pip install git+https://github.com/ultralytics/CLIP.git
|
||||
```
|
||||
|
||||
### Example Usage
|
||||
|
||||
1. Download a [model checkpoint](https://drive.google.com/file/d/1m1sjY4ihXBU1fZXdQ-Xdj-mDltW-2Rqv/view?usp=sharing).
|
||||
|
||||
2. Use FastSAM for inference. Example commands:
|
||||
- Segment everything in an image:
|
||||
|
||||
```bash
|
||||
python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg
|
||||
```
|
||||
|
||||
- Segment specific objects using text prompt:
|
||||
|
||||
```bash
|
||||
python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --text_prompt "the yellow dog"
|
||||
```
|
||||
|
||||
- Segment objects within a [bounding box](https://www.ultralytics.com/glossary/bounding-box) (provide box coordinates in xywh format):
|
||||
|
||||
```bash
|
||||
python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --box_prompt "[570,200,230,400]"
|
||||
```
|
||||
|
||||
- Segment objects near specific points:
|
||||
```bash
|
||||
python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --point_prompt "[[520,360],[620,300]]" --point_label "[1,0]"
|
||||
```
|
||||
|
||||
Additionally, you can try FastSAM through the CASIA-IVA-Lab [Colab demo](https://colab.research.google.com/drive/1oX14f6IneGGw612WgVlAiy91UHwFAvr9?usp=sharing).
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the FastSAM authors for their significant contributions in the field of real-time instance segmentation:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{zhao2023fast,
|
||||
title={Fast Segment Anything},
|
||||
author={Xu Zhao and Wenchao Ding and Yongqi An and Yinglong Du and Tao Yu and Min Li and Ming Tang and Jinqiao Wang},
|
||||
year={2023},
|
||||
eprint={2306.12156},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
The original FastSAM paper can be found on [arXiv](https://arxiv.org/abs/2306.12156). The authors have made their work publicly available, and the codebase can be accessed on [GitHub](https://github.com/CASIA-LMC-Lab/FastSAM). We appreciate their efforts in advancing the field and making their work accessible to the broader community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is FastSAM and how does it differ from SAM?
|
||||
|
||||
FastSAM, short for Fast Segment Anything Model, is a real-time [convolutional neural network](https://www.ultralytics.com/glossary/convolutional-neural-network-cnn) (CNN)-based solution designed to reduce computational demands while maintaining high performance in object segmentation tasks. Unlike the Segment Anything Model (SAM), which uses a heavier Transformer-based architecture, FastSAM leverages [Ultralytics YOLOv8-seg](../tasks/segment.md) for efficient instance segmentation in two stages: all-instance segmentation followed by prompt-guided selection.
|
||||
|
||||
### How does FastSAM achieve real-time segmentation performance?
|
||||
|
||||
FastSAM achieves real-time segmentation by decoupling the segmentation task into all-instance segmentation with YOLOv8-seg and prompt-guided selection stages. By utilizing the computational efficiency of CNNs, FastSAM offers significant reductions in computational and resource demands while maintaining competitive performance. This dual-stage approach enables FastSAM to deliver fast and efficient segmentation suitable for applications requiring quick results.
|
||||
|
||||
### What are the practical applications of FastSAM?
|
||||
|
||||
FastSAM is practical for a variety of [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks that require real-time segmentation performance. Applications include:
|
||||
|
||||
- Industrial automation for quality control and assurance
|
||||
- Real-time video analysis for security and surveillance
|
||||
- [Autonomous vehicles](https://www.ultralytics.com/glossary/autonomous-vehicles) for object detection and segmentation
|
||||
- Medical imaging for precise and quick segmentation tasks
|
||||
|
||||
Its ability to handle various user interaction prompts makes FastSAM adaptable and flexible for diverse scenarios.
|
||||
|
||||
### How do I use the FastSAM model for inference in Python?
|
||||
|
||||
To use FastSAM for inference in Python, you can follow the example below:
|
||||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
|
||||
# Define an inference source
|
||||
source = "path/to/bus.jpg"
|
||||
|
||||
# Create a FastSAM model
|
||||
model = FastSAM("FastSAM-s.pt") # or FastSAM-x.pt
|
||||
|
||||
# Run inference on an image
|
||||
everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)
|
||||
|
||||
# Run inference with bboxes prompt
|
||||
results = model(source, bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Run inference with points prompt
|
||||
results = model(source, points=[[200, 200]], labels=[1])
|
||||
|
||||
# Run inference with texts prompt
|
||||
results = model(source, texts="a photo of a dog")
|
||||
|
||||
# Run inference with bboxes and points and texts prompt at the same time
|
||||
results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")
|
||||
```
|
||||
|
||||
For more details on inference methods, check the [Predict Usage](#predict-usage) section of the documentation.
|
||||
|
||||
### What types of prompts does FastSAM support for segmentation tasks?
|
||||
|
||||
FastSAM supports multiple prompt types for guiding the segmentation tasks:
|
||||
|
||||
- **Everything Prompt**: Generates segmentation for all visible objects.
|
||||
- **Bounding Box (BBox) Prompt**: Segments objects within a specified bounding box.
|
||||
- **Text Prompt**: Uses a descriptive text to segment objects matching the description.
|
||||
- **Point Prompt**: Segments objects near specific user-defined points.
|
||||
|
||||
This flexibility allows FastSAM to adapt to a wide range of user interaction scenarios, enhancing its utility across different applications. For more information on using these prompts, refer to the [Key Features](#key-features) section.
|
||||
148
algorithms/dms_yolo/code.embedded.bak/docs/en/models/index.md
Normal file
148
algorithms/dms_yolo/code.embedded.bak/docs/en/models/index.md
Normal file
@@ -0,0 +1,148 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover a variety of models supported by Ultralytics, including YOLOv3 to YOLO11, NAS, SAM, and RT-DETR for detection, segmentation, and more.
|
||||
keywords: Ultralytics, supported models, YOLOv3, YOLOv4, YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLOv9, YOLOv10, YOLO11, SAM, SAM2, SAM3, MobileSAM, FastSAM, YOLO-NAS, RT-DETR, YOLO-World, object detection, image segmentation, classification, pose estimation, multi-object tracking
|
||||
---
|
||||
|
||||
# Models Supported by Ultralytics
|
||||
|
||||
Welcome to Ultralytics' model documentation! We offer support for a wide range of models, each tailored to specific tasks like [object detection](../tasks/detect.md), [instance segmentation](../tasks/segment.md), [image classification](../tasks/classify.md), [pose estimation](../tasks/pose.md), and [multi-object tracking](../modes/track.md). If you're interested in contributing your model architecture to Ultralytics, check out our [Contributing Guide](../help/contributing.md).
|
||||
|
||||
|
||||
|
||||
## Featured Models
|
||||
|
||||
Here are some of the key models supported:
|
||||
|
||||
1. **[YOLOv3](yolov3.md)**: The third iteration of the YOLO model family, originally by Joseph Redmon, known for its efficient real-time object detection capabilities.
|
||||
2. **[YOLOv4](yolov4.md)**: A darknet-native update to YOLOv3, released by Alexey Bochkovskiy in 2020.
|
||||
3. **[YOLOv5](yolov5.md)**: An improved version of the YOLO architecture by Ultralytics, offering better performance and speed trade-offs compared to previous versions.
|
||||
4. **[YOLOv6](yolov6.md)**: Released by [Meituan](https://www.meituan.com/) in 2022, and in use in many of the company's autonomous delivery robots.
|
||||
5. **[YOLOv7](yolov7.md)**: Updated YOLO models released in 2022 by the authors of YOLOv4. Only inference is supported.
|
||||
6. **[YOLOv8](yolov8.md)**: A versatile model featuring enhanced capabilities such as [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), pose/keypoints estimation, and classification.
|
||||
7. **[YOLOv9](yolov9.md)**: An experimental model trained on the Ultralytics [YOLOv5](yolov5.md) codebase implementing Programmable Gradient Information (PGI).
|
||||
8. **[YOLOv10](yolov10.md)**: By Tsinghua University, featuring NMS-free training and efficiency-accuracy driven architecture, delivering state-of-the-art performance and latency.
|
||||
9. **[YOLO11](yolo11.md)**: Ultralytics' YOLO models delivering high performance across multiple tasks including detection, segmentation, pose estimation, tracking, and classification.
|
||||
10. **[YOLO26](yolo26.md) 🚀 NEW**: Ultralytics' **latest** next-generation YOLO model optimized for edge deployment with end-to-end NMS-free inference.
|
||||
11. **[Segment Anything Model (SAM)](sam.md)**: Meta's original Segment Anything Model (SAM).
|
||||
12. **[Segment Anything Model 2 (SAM2)](sam-2.md)**: The next generation of Meta's Segment Anything Model for videos and images.
|
||||
13. **[Segment Anything Model 3 (SAM3)](sam-3.md) 🚀 NEW**: Meta's third generation Segment Anything Model with Promptable Concept Segmentation for text and image exemplar-based segmentation.
|
||||
14. **[Mobile Segment Anything Model (MobileSAM)](mobile-sam.md)**: MobileSAM for mobile applications, by Kyung Hee University.
|
||||
15. **[Fast Segment Anything Model (FastSAM)](fast-sam.md)**: FastSAM by Image & Video Analysis Group, Institute of Automation, Chinese Academy of Sciences.
|
||||
16. **[YOLO-NAS](yolo-nas.md)**: YOLO [Neural Architecture Search](https://www.ultralytics.com/glossary/neural-architecture-search-nas) (NAS) Models.
|
||||
17. **[Real-Time Detection Transformers (RT-DETR)](rtdetr.md)**: Baidu's PaddlePaddle real-time Detection [Transformer](https://www.ultralytics.com/glossary/transformer) (RT-DETR) models.
|
||||
18. **[YOLO-World](yolo-world.md)**: Real-time Open Vocabulary Object Detection models from Tencent AI Lab.
|
||||
19. **[YOLOE](yoloe.md)**: An improved open-vocabulary object detector that maintains YOLO's real-time performance while detecting arbitrary classes beyond its training data.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/MWq1UxqTClU?si=nHAW-lYDzrz68jR0"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Run Ultralytics YOLO models in just a few lines of code.
|
||||
</p>
|
||||
|
||||
## Getting Started: Usage Examples
|
||||
|
||||
This example provides simple YOLO training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
Note the below example spotlights YOLO11 [Detect](../tasks/detect.md) models for [object detection](https://www.ultralytics.com/glossary/object-detection). For additional supported tasks see the [Segment](../tasks/segment.md), [Classify](../tasks/classify.md) and [Pose](../tasks/pose.md) docs.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()`, `SAM()`, `NAS()` and `RTDETR()` classes to create a model instance in Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLO26n model
|
||||
model = YOLO("yolo26n.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLO26n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO26n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolo26n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLO26n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolo26n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Contributing New Models
|
||||
|
||||
Interested in contributing your model to Ultralytics? Great! We're always open to expanding our model portfolio.
|
||||
|
||||
1. **Fork the Repository**: Start by forking the [Ultralytics GitHub repository](https://github.com/ultralytics/ultralytics).
|
||||
|
||||
2. **Clone Your Fork**: Clone your fork to your local machine and create a new branch to work on.
|
||||
|
||||
3. **Implement Your Model**: Add your model following the coding standards and guidelines provided in our [Contributing Guide](../help/contributing.md).
|
||||
|
||||
4. **Test Thoroughly**: Make sure to test your model rigorously, both in isolation and as part of the pipeline.
|
||||
|
||||
5. **Create a Pull Request**: Once you're satisfied with your model, create a pull request to the main repository for review.
|
||||
|
||||
6. **Code Review & Merging**: After review, if your model meets our criteria, it will be merged into the main repository.
|
||||
|
||||
For detailed steps, consult our [Contributing Guide](../help/contributing.md).
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is the latest Ultralytics YOLO model?
|
||||
|
||||
The latest Ultralytics YOLO model is [YOLO26](yolo26.md), released in January 2026. YOLO26 features end-to-end NMS-free inference, optimized edge deployment, and supports all five tasks (detection, segmentation, classification, pose estimation, and OBB) plus open-vocabulary versions. For stable production workloads, both YOLO26 and [YOLO11](yolo11.md) are recommended choices.
|
||||
|
||||
### How can I train a YOLO model on custom data?
|
||||
|
||||
Training a YOLO model on custom data can be easily accomplished using Ultralytics' libraries. Here's a quick example:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a YOLO model
|
||||
model = YOLO("yolo26n.pt") # or any other YOLO model
|
||||
|
||||
# Train the model on custom dataset
|
||||
results = model.train(data="custom_data.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo train model=yolo26n.pt data='custom_data.yaml' epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For more detailed instructions, visit the [Train](../modes/train.md) documentation page.
|
||||
|
||||
### Which YOLO versions are supported by Ultralytics?
|
||||
|
||||
Ultralytics supports a comprehensive range of YOLO (You Only Look Once) versions from YOLOv3 to YOLO11, along with models like YOLO-NAS, SAM, and RT-DETR. Each version is optimized for various tasks such as detection, segmentation, and classification. For detailed information on each model, refer to the [Models Supported by Ultralytics](../models/index.md) documentation.
|
||||
|
||||
### Why should I use Ultralytics Platform for [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) projects?
|
||||
|
||||
[Ultralytics Platform](../platform/index.md) provides a no-code, end-to-end platform for training, deploying, and managing YOLO models. It simplifies complex workflows, enabling users to focus on model performance and application. The HUB also offers [cloud training capabilities](../platform/train/cloud-training.md), comprehensive dataset management, and user-friendly interfaces for both beginners and experienced developers.
|
||||
|
||||
### What types of tasks can Ultralytics YOLO models perform?
|
||||
|
||||
Ultralytics YOLO models are versatile and can perform tasks including object detection, instance segmentation, classification, pose estimation, and oriented object detection (OBB). The latest model, [YOLO26](yolo26.md), supports all five tasks plus open-vocabulary detection. For details on specific tasks, refer to the [Task pages](../tasks/index.md).
|
||||
@@ -0,0 +1,247 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover MobileSAM, a lightweight and fast image segmentation model for mobile and edge applications. Compare its performance with SAM and YOLO models.
|
||||
keywords: MobileSAM, image segmentation, lightweight model, fast segmentation, mobile applications, SAM, Tiny-ViT, YOLO, Ultralytics
|
||||
---
|
||||
|
||||
|
||||
|
||||
# Mobile Segment Anything (MobileSAM)
|
||||
|
||||
MobileSAM is a compact, efficient image segmentation model purpose-built for mobile and edge devices. Designed to bring the power of Meta's Segment Anything Model ([SAM](sam.md)) to environments with limited compute, MobileSAM delivers near-instant segmentation while maintaining compatibility with the original SAM pipeline. Whether you're developing real-time applications or lightweight deployments, MobileSAM provides impressive segmentation results with a fraction of the size and speed requirements of its predecessors.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/yXQPLMrNX2s"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Run Inference with MobileSAM using Ultralytics | Step-by-Step Guide 🎉
|
||||
</p>
|
||||
|
||||
MobileSAM has been adopted in a variety of projects, including [Grounding-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything), [AnyLabeling](https://github.com/vietanhdev/anylabeling), and [Segment Anything in 3D](https://github.com/Jumpat/SegmentAnythingin3D).
|
||||
|
||||
MobileSAM was trained on a single GPU using a 100k image dataset (1% of the original images) in less than a day. The training code will be released in the future.
|
||||
|
||||
## Available Models, Supported Tasks, and Operating Modes
|
||||
|
||||
The table below outlines the available MobileSAM model, its pretrained weights, supported tasks, and compatibility with different operating modes such as [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). Supported modes are indicated by ✅ and unsupported modes by ❌.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ---------- | --------------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| MobileSAM | [mobile_sam.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/mobile_sam.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
|
||||
## MobileSAM Comparison vs YOLO
|
||||
|
||||
The following comparison highlights the differences between Meta's SAM variants, MobileSAM, and Ultralytics' smallest segmentation models, including [YOLO11n-seg](../models/yolo11.md):
|
||||
|
||||
| Model | Size<br><sup>(MB)</sup> | Parameters<br><sup>(M)</sup> | Speed (CPU)<br><sup>(ms/im)</sup> |
|
||||
| ------------------------------------------------------------------------------- | ----------------------- | ---------------------------- | --------------------------------- |
|
||||
| Meta SAM-b | 375 | 93.7 | 49401 |
|
||||
| Meta SAM2-b | 162 | 80.8 | 31901 |
|
||||
| Meta SAM2-t | 78.1 | 38.9 | 25997 |
|
||||
| MobileSAM | 40.7 | 10.1 | 25381 |
|
||||
| FastSAM-s with YOLOv8 [backbone](https://www.ultralytics.com/glossary/backbone) | 23.7 | 11.8 | 55.9 |
|
||||
| Ultralytics YOLOv8n-seg | **6.7** (11.7x smaller) | **3.4** (11.4x less) | **24.5** (1061x faster) |
|
||||
| Ultralytics YOLO11n-seg | **5.9** (13.2x smaller) | **2.9** (13.4x less) | **30.1** (864x faster) |
|
||||
|
||||
This comparison demonstrates the substantial differences in model size and speed between SAM variants and YOLO segmentation models. While SAM models offer unique automatic segmentation capabilities, YOLO models—especially YOLOv8n-seg and YOLO11n-seg—are significantly smaller, faster, and more computationally efficient.
|
||||
|
||||
Tests were conducted on a 2025 Apple M4 Pro with 24GB RAM using `torch==2.6.0` and `ultralytics==8.3.90`. To reproduce these results:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, SAM, YOLO, FastSAM
|
||||
|
||||
# Profile SAM2-t, SAM2-b, SAM-b, MobileSAM
|
||||
for file in ["sam_b.pt", "sam2_b.pt", "sam2_t.pt", "mobile_sam.pt"]:
|
||||
model = SAM(file)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile FastSAM-s
|
||||
model = FastSAM("FastSAM-s.pt")
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile YOLO models
|
||||
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt"]:
|
||||
model = YOLO(file_name)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
```
|
||||
|
||||
## Adapting from SAM to MobileSAM
|
||||
|
||||
MobileSAM retains the same pipeline as the original [SAM](sam.md), including pre-processing, post-processing, and all interfaces. This means you can transition from SAM to MobileSAM with minimal changes to your workflow.
|
||||
|
||||
The key difference is the image encoder: MobileSAM replaces the original ViT-H encoder (632M parameters) with a much smaller Tiny-ViT encoder (5M parameters). On a single GPU, MobileSAM processes an image in about 12ms (8ms for the encoder, 4ms for the mask decoder).
|
||||
|
||||
### ViT-Based Image Encoder Comparison
|
||||
|
||||
| Image Encoder | Original SAM | MobileSAM |
|
||||
| ------------- | ------------ | --------- |
|
||||
| Parameters | 611M | 5M |
|
||||
| Speed | 452ms | 8ms |
|
||||
|
||||
### Prompt-Guided Mask Decoder
|
||||
|
||||
| Mask Decoder | Original SAM | MobileSAM |
|
||||
| ------------ | ------------ | --------- |
|
||||
| Parameters | 3.876M | 3.876M |
|
||||
| Speed | 4ms | 4ms |
|
||||
|
||||
### Whole Pipeline Comparison
|
||||
|
||||
| Whole Pipeline (Enc+Dec) | Original SAM | MobileSAM |
|
||||
| ------------------------ | ------------ | --------- |
|
||||
| Parameters | 615M | 9.66M |
|
||||
| Speed | 456ms | 12ms |
|
||||
|
||||
The performance of MobileSAM and the original SAM is illustrated below using both point and box prompts.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
MobileSAM is approximately 7 times smaller and 5 times faster than FastSAM. For further details, visit the [MobileSAM project page](https://github.com/ChaoningZhang/MobileSAM).
|
||||
|
||||
## Testing MobileSAM in Ultralytics
|
||||
|
||||
Just like the original [SAM](sam.md), Ultralytics provides a simple interface for testing MobileSAM, supporting both Point and Box prompts.
|
||||
|
||||
### Model Download
|
||||
|
||||
Download the MobileSAM pretrained weights from [Ultralytics assets](https://github.com/ultralytics/assets/releases/download/v8.4.0/mobile_sam.pt).
|
||||
|
||||
### Point Prompt
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load the model
|
||||
model = SAM("mobile_sam.pt")
|
||||
|
||||
# Predict a segment based on a single point prompt
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[900, 370], labels=[1])
|
||||
|
||||
# Predict multiple segments based on multiple points prompt
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Predict a segment based on multiple points prompt per object
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
|
||||
|
||||
# Predict a segment using both positive and negative prompts.
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
### Box Prompt
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load the model
|
||||
model = SAM("mobile_sam.pt")
|
||||
|
||||
# Predict a segment based on a single point prompt
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[900, 370], labels=[1])
|
||||
|
||||
# Predict multiple segments based on multiple points prompt
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Predict a segment based on multiple points prompt per object
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
|
||||
|
||||
# Predict a segment using both positive and negative prompts.
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
Both `MobileSAM` and `SAM` share the same API. For more usage details, see the [SAM documentation](sam.md).
|
||||
|
||||
### Automatically Build Segmentation Datasets Using a Detection Model
|
||||
|
||||
To automatically annotate your dataset with the Ultralytics framework, use the `auto_annotate` function as shown below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.data.annotator import auto_annotate
|
||||
|
||||
auto_annotate(data="path/to/images", det_model="yolo11x.pt", sam_model="mobile_sam.pt")
|
||||
```
|
||||
|
||||
{% include "macros/sam-auto-annotate.md" %}
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If MobileSAM is helpful in your research or development, please consider citing the following paper:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{mobile_sam,
|
||||
title={Faster Segment Anything: Towards Lightweight SAM for Mobile Applications},
|
||||
author={Zhang, Chaoning and Han, Dongshen and Qiao, Yu and Kim, Jung Uk and Bae, Sung Ho and Lee, Seungkyu and Hong, Choong Seon},
|
||||
journal={arXiv preprint arXiv:2306.14289},
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
Read the full [MobileSAM paper on arXiv](https://arxiv.org/pdf/2306.14289).
|
||||
|
||||
## FAQ
|
||||
|
||||
### What Is MobileSAM and How Does It Differ from the Original SAM Model?
|
||||
|
||||
MobileSAM is a lightweight, fast [image segmentation](https://www.ultralytics.com/glossary/image-segmentation) model optimized for mobile and edge applications. It maintains the same pipeline as the original SAM but replaces the large ViT-H encoder (632M parameters) with a compact Tiny-ViT encoder (5M parameters). This results in MobileSAM being about 5 times smaller and 7 times faster than the original SAM, operating at roughly 12ms per image versus SAM's 456ms. Explore more about MobileSAM's implementation on the [MobileSAM GitHub repository](https://github.com/ChaoningZhang/MobileSAM).
|
||||
|
||||
### How Can I Test MobileSAM Using Ultralytics?
|
||||
|
||||
Testing MobileSAM in Ultralytics is straightforward. You can use Point and Box prompts to predict segments. For example, using a Point prompt:
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load the model
|
||||
model = SAM("mobile_sam.pt")
|
||||
|
||||
# Predict a segment based on a point prompt
|
||||
model.predict("ultralytics/assets/zidane.jpg", points=[900, 370], labels=[1])
|
||||
```
|
||||
|
||||
For more details, see the [Testing MobileSAM in Ultralytics](#testing-mobilesam-in-ultralytics) section.
|
||||
|
||||
### Why Should I Use MobileSAM for My Mobile Application?
|
||||
|
||||
MobileSAM is ideal for mobile and edge applications due to its lightweight design and rapid inference speed. Compared to the original SAM, MobileSAM is about 5 times smaller and 7 times faster, making it suitable for real-time segmentation on devices with limited computational resources. Its efficiency enables mobile devices to perform [real-time image segmentation](https://www.ultralytics.com/glossary/real-time-inference) without significant latency. Additionally, MobileSAM supports [Inference mode](../modes/predict.md) optimized for mobile performance.
|
||||
|
||||
### How Was MobileSAM Trained, and Is the Training Code Available?
|
||||
|
||||
MobileSAM was trained on a single GPU with a 100k image dataset (1% of the original images) in under a day. While the training code will be released in the future, you can currently access pretrained weights and implementation details from the [MobileSAM GitHub repository](https://github.com/ChaoningZhang/MobileSAM).
|
||||
|
||||
### What Are the Primary Use Cases for MobileSAM?
|
||||
|
||||
MobileSAM is designed for fast, efficient image segmentation in mobile and edge environments. Primary use cases include:
|
||||
|
||||
- **Real-time [object detection and segmentation](https://www.ultralytics.com/glossary/object-detection)** for mobile apps
|
||||
- **Low-latency image processing** on devices with limited compute
|
||||
- **Integration in AI-powered mobile applications** for augmented reality (AR), analytics, and more
|
||||
|
||||
For more details on use cases and performance, see [Adapting from SAM to MobileSAM](#adapting-from-sam-to-mobilesam) and the [Ultralytics blog on MobileSAM applications](https://www.ultralytics.com/blog/applications-of-meta-ai-segment-anything-model-2-sam-2).
|
||||
189
algorithms/dms_yolo/code.embedded.bak/docs/en/models/rtdetr.md
Normal file
189
algorithms/dms_yolo/code.embedded.bak/docs/en/models/rtdetr.md
Normal file
@@ -0,0 +1,189 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore Baidu's RT-DETR, a Vision Transformer-based real-time object detector offering high accuracy and adaptable inference speed. Learn more with Ultralytics.
|
||||
keywords: RT-DETR, Baidu, Vision Transformer, real-time object detection, PaddlePaddle, Ultralytics, pretrained models, AI, machine learning, computer vision
|
||||
---
|
||||
|
||||
# Baidu's RT-DETR: A Vision [Transformer](https://www.ultralytics.com/glossary/transformer)-Based Real-Time Object Detector
|
||||
|
||||
## Overview
|
||||
|
||||
Real-Time Detection Transformer (RT-DETR), developed by Baidu, is a cutting-edge end-to-end object detector that provides real-time performance while maintaining high [accuracy](https://www.ultralytics.com/glossary/accuracy). It is based on the idea of DETR (the NMS-free framework), meanwhile introducing conv-based [backbone](https://www.ultralytics.com/glossary/backbone) and an efficient hybrid encoder to gain real-time speed. RT-DETR efficiently processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. The model is highly adaptable, supporting flexible adjustment of inference speed using different decoder layers without retraining. RT-DETR excels on accelerated backends like CUDA with TensorRT, outperforming many other real-time object detectors.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/i70ecEGB1ro"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Use Baidu's RT-DETR for Object Detection | Inference and Benchmarking with Ultralytics 🚀
|
||||
</p>
|
||||
|
||||
 **Overview of Baidu's RT-DETR.** The RT-DETR model architecture diagram shows the last three stages of the backbone {S3, S4, S5} as the input to the encoder. The efficient hybrid encoder transforms multiscale features into a sequence of image features through intrascale feature interaction (AIFI) and cross-scale feature-fusion module (CCFM). The IoU-aware query selection is employed to select a fixed number of image features to serve as initial object queries for the decoder. Finally, the decoder with auxiliary prediction heads iteratively optimizes object queries to generate boxes and confidence scores ([source](https://arxiv.org/pdf/2304.08069)).
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Efficient Hybrid Encoder:** Baidu's RT-DETR uses an efficient hybrid encoder that processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. This unique Vision Transformers-based design reduces computational costs and allows for real-time [object detection](https://www.ultralytics.com/glossary/object-detection).
|
||||
- **IoU-aware Query Selection:** Baidu's RT-DETR improves object query initialization by utilizing IoU-aware query selection. This allows the model to focus on the most relevant objects in the scene, enhancing the detection accuracy.
|
||||
- **Adaptable Inference Speed:** Baidu's RT-DETR supports flexible adjustments of inference speed by using different decoder layers without the need for retraining. This adaptability facilitates practical application in various real-time object detection scenarios.
|
||||
- **NMS-Free Framework:** Based on DETR, RT-DETR eliminates the need for [non-maximum suppression](https://www.ultralytics.com/glossary/non-maximum-suppression-nms) post-processing, simplifying the detection pipeline and potentially improving efficiency.
|
||||
- **Anchor-Free Detection:** As an [anchor-free detector](https://www.ultralytics.com/glossary/anchor-free-detectors), RT-DETR simplifies the detection process and may improve generalization across different datasets.
|
||||
|
||||
## Pretrained Models
|
||||
|
||||
The Ultralytics Python API provides pretrained PaddlePaddle RT-DETR models with different scales:
|
||||
|
||||
- RT-DETR-L: 53.0% AP on COCO val2017, 114 FPS on T4 GPU
|
||||
- RT-DETR-X: 54.8% AP on COCO val2017, 74 FPS on T4 GPU
|
||||
|
||||
Additionally, Baidu has released RTDETRv2 in July 2024, which further improves upon the original architecture with enhanced performance metrics.
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["RTDETRv2"]'></canvas>
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This example provides simple RT-DETR training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import RTDETR
|
||||
|
||||
# Load a COCO-pretrained RT-DETR-l model
|
||||
model = RTDETR("rtdetr-l.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the RT-DETR-l model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained RT-DETR-l model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=rtdetr-l.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained RT-DETR-l model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=rtdetr-l.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
This table presents the model types, the specific pretrained weights, the tasks supported by each model, and the various modes ([Train](../modes/train.md) , [Val](../modes/val.md), [Predict](../modes/predict.md), [Export](../modes/export.md)) that are supported, indicated by ✅ emojis.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| RT-DETR Large | [rtdetr-l.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/rtdetr-l.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| RT-DETR Extra-Large | [rtdetr-x.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/rtdetr-x.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
## Ideal Use Cases
|
||||
|
||||
RT-DETR is particularly well-suited for applications requiring both high accuracy and real-time performance:
|
||||
|
||||
- **Autonomous Driving**: For reliable environmental perception in self-driving systems where both speed and accuracy are critical. [Learn more about AI in self-driving cars](https://www.ultralytics.com/solutions/ai-in-automotive).
|
||||
- **Advanced Robotics**: Enabling robots to perform complex tasks requiring accurate object recognition and interaction in dynamic environments. [Explore AI's role in robotics](https://www.ultralytics.com/blog/from-algorithms-to-automation-ais-role-in-robotics).
|
||||
- **Medical Imaging**: For applications in healthcare where precision in object detection can be crucial for diagnostics. [Discover AI in healthcare](https://www.ultralytics.com/solutions/ai-in-healthcare).
|
||||
- **Surveillance Systems**: For security applications requiring real-time monitoring with high detection accuracy. [Learn about security alarm systems](https://docs.ultralytics.com/guides/security-alarm-system/).
|
||||
- **Satellite Image Analysis**: For detailed analysis of high-resolution imagery where global context understanding is important. [Read about computer vision in satellite imagery](https://www.ultralytics.com/blog/using-computer-vision-to-analyze-satellite-imagery).
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If you use Baidu's RT-DETR in your research or development work, please cite the [original paper](https://arxiv.org/abs/2304.08069):
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{lv2023detrs,
|
||||
title={DETRs Beat YOLOs on Real-time Object Detection},
|
||||
author={Wenyu Lv and Shangliang Xu and Yian Zhao and Guanzhong Wang and Jinman Wei and Cheng Cui and Yuning Du and Qingqing Dang and Yi Liu},
|
||||
year={2023},
|
||||
eprint={2304.08069},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
For RTDETRv2, you can cite the [2024 paper](https://arxiv.org/abs/2407.17140):
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{lv2024rtdetrv2,
|
||||
title={RTDETRv2: All-in-One Detection Transformer Beats YOLO and DINO},
|
||||
author={Wenyu Lv and Yian Zhao and Qinyao Chang and Kui Huang and Guanzhong Wang and Yi Liu},
|
||||
year={2024},
|
||||
eprint={2407.17140},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
We would like to acknowledge Baidu and the [PaddlePaddle](https://github.com/PaddlePaddle/PaddleDetection) team for creating and maintaining this valuable resource for the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) community. Their contribution to the field with the development of the Vision Transformers-based real-time object detector, RT-DETR, is greatly appreciated.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is Baidu's RT-DETR model and how does it work?
|
||||
|
||||
Baidu's RT-DETR (Real-Time Detection Transformer) is an advanced real-time object detector built upon the Vision Transformer architecture. It efficiently processes multiscale features by decoupling intra-scale interaction and cross-scale fusion through its efficient hybrid encoder. By employing IoU-aware query selection, the model focuses on the most relevant objects, enhancing detection accuracy. Its adaptable inference speed, achieved by adjusting decoder layers without retraining, makes RT-DETR suitable for various real-time object detection scenarios. Learn more about RT-DETR features in the [RT-DETR Arxiv paper](https://arxiv.org/pdf/2304.08069).
|
||||
|
||||
### How can I use the pretrained RT-DETR models provided by Ultralytics?
|
||||
|
||||
You can leverage Ultralytics Python API to use pretrained PaddlePaddle RT-DETR models. For instance, to load an RT-DETR-l model pretrained on COCO val2017 and achieve high FPS on T4 GPU, you can utilize the following example:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import RTDETR
|
||||
|
||||
# Load a COCO-pretrained RT-DETR-l model
|
||||
model = RTDETR("rtdetr-l.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the RT-DETR-l model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained RT-DETR-l model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=rtdetr-l.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained RT-DETR-l model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=rtdetr-l.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
### Why should I choose Baidu's RT-DETR over other real-time object detectors?
|
||||
|
||||
Baidu's RT-DETR stands out due to its efficient hybrid encoder and IoU-aware query selection, which drastically reduce computational costs while maintaining high accuracy. Its unique ability to adjust inference speed by using different decoder layers without retraining adds significant flexibility. This makes it particularly advantageous for applications requiring real-time performance on accelerated backends like CUDA with TensorRT, outclassing many other real-time object detectors. The [transformer architecture](https://www.ultralytics.com/glossary/transformer) also provides better global context understanding compared to traditional CNN-based detectors.
|
||||
|
||||
### How does RT-DETR support adaptable inference speed for different real-time applications?
|
||||
|
||||
Baidu's RT-DETR allows flexible adjustments of inference speed by using different decoder layers without requiring retraining. This adaptability is crucial for scaling performance across various real-time object detection tasks. Whether you need faster processing for lower [precision](https://www.ultralytics.com/glossary/precision) needs or slower, more accurate detections, RT-DETR can be tailored to meet your specific requirements. This feature is particularly valuable when deploying models across devices with varying computational capabilities.
|
||||
|
||||
### Can I use RT-DETR models with other Ultralytics modes, such as training, validation, and export?
|
||||
|
||||
Yes, RT-DETR models are compatible with various Ultralytics modes including training, validation, prediction, and export. You can refer to the respective documentation for detailed instructions on how to utilize these modes: [Train](../modes/train.md), [Val](../modes/val.md), [Predict](../modes/predict.md), and [Export](../modes/export.md). This ensures a comprehensive workflow for developing and deploying your object detection solutions. The Ultralytics framework provides a consistent API across different model architectures, making it easy to work with RT-DETR models.
|
||||
483
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam-2.md
Normal file
483
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam-2.md
Normal file
@@ -0,0 +1,483 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover SAM 2, the next generation of Meta's Segment Anything Model, supporting real-time promptable segmentation in both images and videos with state-of-the-art performance. Learn about its key features, datasets, and how to use it.
|
||||
keywords: SAM 2, SAM 2.1, Segment Anything, video segmentation, image segmentation, promptable segmentation, zero-shot performance, SA-V dataset, Ultralytics, real-time segmentation, AI, machine learning
|
||||
---
|
||||
|
||||
# SAM 2: Segment Anything Model 2
|
||||
|
||||
!!! note "SAM Evolution"
|
||||
|
||||
SAM 2 builds upon the original [SAM](sam.md) with video segmentation capabilities. For Promptable Concept Segmentation with text and image exemplar prompts, see [SAM 3](sam-3.md).
|
||||
|
||||
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/inference-with-meta-sam-and-sam2-using-ultralytics-python-package.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Inference with Segment Anything 2 In Colab"></a>
|
||||
|
||||
SAM 2, the successor to Meta's [Segment Anything Model (SAM)](sam.md), is a cutting-edge tool designed for comprehensive object segmentation in both images and videos. It excels in handling complex visual data through a unified, promptable model architecture that supports real-time processing and zero-shot generalization.
|
||||
|
||||
|
||||
|
||||
## Key Features
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/yXQPLMrNX2s"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Run Inference with Meta's SAM2 using Ultralytics | Step-by-Step Guide 🎉
|
||||
</p>
|
||||
|
||||
### Unified Model Architecture
|
||||
|
||||
SAM 2 combines the capabilities of image and video segmentation in a single model. This unification simplifies deployment and allows for consistent performance across different media types. It leverages a flexible prompt-based interface, enabling users to specify objects of interest through various prompt types, such as points, bounding boxes, or masks.
|
||||
|
||||
### Real-Time Performance
|
||||
|
||||
The model achieves real-time inference speeds, processing approximately 44 frames per second. This makes SAM 2 suitable for applications requiring immediate feedback, such as video editing and augmented reality.
|
||||
|
||||
### Zero-Shot Generalization
|
||||
|
||||
SAM 2 can segment objects it has never encountered before, demonstrating strong zero-shot generalization. This is particularly useful in diverse or evolving visual domains where pre-defined categories may not cover all possible objects.
|
||||
|
||||
### Interactive Refinement
|
||||
|
||||
Users can iteratively refine the segmentation results by providing additional prompts, allowing for precise control over the output. This interactivity is essential for fine-tuning results in applications like video annotation or medical imaging.
|
||||
|
||||
### Advanced Handling of Visual Challenges
|
||||
|
||||
SAM 2 includes mechanisms to manage common video segmentation challenges, such as object occlusion and reappearance. It uses a sophisticated memory mechanism to keep track of objects across frames, ensuring continuity even when objects are temporarily obscured or exit and re-enter the scene.
|
||||
|
||||
For a deeper understanding of SAM 2's architecture and capabilities, explore the [SAM 2 research paper](https://arxiv.org/abs/2408.00714).
|
||||
|
||||
## Performance and Technical Details
|
||||
|
||||
SAM 2 sets a new benchmark in the field, outperforming previous models on various metrics:
|
||||
|
||||
| Metric | SAM 2 | Previous SOTA |
|
||||
| ------------------------------------------------------------------------------------------ | ------------- | ------------- |
|
||||
| **Interactive Video Segmentation** | **Best** | - |
|
||||
| **Human Interactions Required** | **3x fewer** | Baseline |
|
||||
| **[Image Segmentation](https://www.ultralytics.com/glossary/image-segmentation) Accuracy** | **Improved** | SAM |
|
||||
| **Inference Speed** | **6x faster** | SAM |
|
||||
|
||||
## Model Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
- **Image and Video Encoder**: Utilizes a [transformer](https://www.ultralytics.com/glossary/transformer)-based architecture to extract high-level features from both images and video frames. This component is responsible for understanding the visual content at each timestep.
|
||||
- **Prompt Encoder**: Processes user-provided prompts (points, boxes, masks) to guide the segmentation task. This allows SAM 2 to adapt to user input and target specific objects within a scene.
|
||||
- **Memory Mechanism**: Includes a memory encoder, memory bank, and memory attention module. These components collectively store and utilize information from past frames, enabling the model to maintain consistent [object tracking](https://www.ultralytics.com/glossary/object-tracking) over time.
|
||||
- **Mask Decoder**: Generates the final segmentation masks based on the encoded image features and prompts. In video, it also uses memory context to ensure accurate tracking across frames.
|
||||
|
||||
|
||||
|
||||
### Memory Mechanism and Occlusion Handling
|
||||
|
||||
The memory mechanism allows SAM 2 to handle temporal dependencies and occlusions in video data. As objects move and interact, SAM 2 records their features in a memory bank. When an object becomes occluded, the model can rely on this memory to predict its position and appearance when it reappears. The occlusion head specifically handles scenarios where objects are not visible, predicting the likelihood of an object being occluded.
|
||||
|
||||
### Multi-Mask Ambiguity Resolution
|
||||
|
||||
In situations with ambiguity (e.g., overlapping objects), SAM 2 can generate multiple mask predictions. This feature is crucial for accurately representing complex scenes where a single mask might not sufficiently describe the scene's nuances.
|
||||
|
||||
## SA-V Dataset
|
||||
|
||||
The SA-V dataset, developed for SAM 2's training, is one of the largest and most diverse video segmentation datasets available. It includes:
|
||||
|
||||
- **51,000+ Videos**: Captured across 47 countries, providing a wide range of real-world scenarios.
|
||||
- **600,000+ Mask Annotations**: Detailed spatio-temporal mask annotations, referred to as "masklets," covering whole objects and parts.
|
||||
- **Dataset Scale**: It features 4.5 times more videos and 53 times more annotations than previous largest datasets, offering unprecedented diversity and complexity.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
### Video Object Segmentation
|
||||
|
||||
SAM 2 has demonstrated superior performance across major video segmentation benchmarks:
|
||||
|
||||
| Dataset | J&F | J | F |
|
||||
| --------------- | ---- | ---- | ---- |
|
||||
| **DAVIS 2017** | 82.5 | 79.8 | 85.2 |
|
||||
| **YouTube-VOS** | 81.2 | 78.9 | 83.5 |
|
||||
|
||||
### Interactive Segmentation
|
||||
|
||||
In interactive segmentation tasks, SAM 2 shows significant efficiency and accuracy:
|
||||
|
||||
| Dataset | NoC@90 | AUC |
|
||||
| --------------------- | ------ | ----- |
|
||||
| **DAVIS Interactive** | 1.54 | 0.872 |
|
||||
|
||||
## Installation
|
||||
|
||||
To install SAM 2, use the following command. All SAM 2 models will automatically download on first use.
|
||||
|
||||
```bash
|
||||
pip install ultralytics
|
||||
```
|
||||
|
||||
## How to Use SAM 2: Versatility in Image and Video Segmentation
|
||||
|
||||
The following table details the available SAM 2 models, their pretrained weights, supported tasks, and compatibility with different operating modes like [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md).
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| SAM 2 tiny | [sam2_t.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2_t.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2 small | [sam2_s.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2_s.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2 base | [sam2_b.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2_b.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2 large | [sam2_l.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2_l.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2.1 tiny | [sam2.1_t.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2.1_t.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2.1 small | [sam2.1_s.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2.1_s.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2.1 base | [sam2.1_b.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2.1_b.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM 2.1 large | [sam2.1_l.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam2.1_l.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
|
||||
### SAM 2 Prediction Examples
|
||||
|
||||
SAM 2 can be utilized across a broad spectrum of tasks, including real-time video editing, medical imaging, and autonomous systems. Its ability to segment both static and dynamic visual data makes it a versatile tool for researchers and developers.
|
||||
|
||||
#### Segment with Prompts
|
||||
|
||||
!!! example "Segment with Prompts"
|
||||
|
||||
Use prompts to segment specific objects in images or videos.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam2.1_b.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Run inference with bboxes prompt
|
||||
results = model("path/to/image.jpg", bboxes=[100, 100, 200, 200])
|
||||
|
||||
# Run inference with single point
|
||||
results = model(points=[900, 370], labels=[1])
|
||||
|
||||
# Run inference with multiple points
|
||||
results = model(points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Run inference with multiple points prompt per object
|
||||
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
|
||||
|
||||
# Run inference with negative points prompt
|
||||
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
#### Segment Everything
|
||||
|
||||
!!! example "Segment Everything"
|
||||
|
||||
Segment the entire image or video content without specific prompts.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam2.1_b.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Run inference
|
||||
model("path/to/video.mp4")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Run inference with a SAM 2 model
|
||||
yolo predict model=sam2.1_b.pt source=path/to/video.mp4
|
||||
```
|
||||
|
||||
#### Segment Video and Track objects
|
||||
|
||||
!!! example "Segment Video"
|
||||
|
||||
Segment the entire video content with specific prompts and track objects.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM2VideoPredictor
|
||||
|
||||
# Create SAM2VideoPredictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", imgsz=1024, model="sam2_b.pt")
|
||||
predictor = SAM2VideoPredictor(overrides=overrides)
|
||||
|
||||
# Run inference with single point
|
||||
results = predictor(source="test.mp4", points=[920, 470], labels=[1])
|
||||
|
||||
# Run inference with multiple points
|
||||
results = predictor(source="test.mp4", points=[[920, 470], [909, 138]], labels=[1, 1])
|
||||
|
||||
# Run inference with multiple points prompt per object
|
||||
results = predictor(source="test.mp4", points=[[[920, 470], [909, 138]]], labels=[[1, 1]])
|
||||
|
||||
# Run inference with negative points prompt
|
||||
results = predictor(source="test.mp4", points=[[[920, 470], [909, 138]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
- This example demonstrates how SAM 2 can be used to segment the entire content of an image or video if no prompts (bboxes/points/masks) are provided.
|
||||
|
||||
## Dynamic Interactive Segment and Track
|
||||
|
||||
SAM2DynamicInteractivePredictor is an advanced training-free extension of SAM2 that enables dynamic interaction with multiple frames and continual learning capabilities. This predictor supports real-time prompt updates and memory management for improved tracking performance across a sequence of images. Compared to the original SAM2, SAM2DynamicInteractivePredictor rebuilds the inference flow to make the best use of pretrained SAM2 models without requiring additional training.
|
||||
|
||||
|
||||
|
||||
### Key Features
|
||||
|
||||
It offers three significant enhancements:
|
||||
|
||||
1. **Dynamic Interactive**: Add new prompts for merging/untracked new instances in following frames anytime during video processing
|
||||
2. **Continual Learning**: Add new prompts for existing instances to improve the model performance over time
|
||||
3. **Independent Multi-Image Support**: Process multiple independent images (not necessarily from a video sequence) with memory sharing and cross-image object tracking
|
||||
|
||||
### Core Capabilities
|
||||
|
||||
- **Prompt Flexibility**: Accepts bounding boxes, points, and masks as prompts
|
||||
- **Memory Bank Management**: Maintains a dynamic memory bank to store object states across frames
|
||||
- **Multi-Object Tracking**: Supports tracking multiple objects simultaneously with individual object IDs
|
||||
- **Real-Time Updates**: Allows adding new prompts during inference without reprocessing previous frames
|
||||
- **Independent Image Processing**: Process standalone images with shared memory context for cross-image object consistency
|
||||
|
||||
!!! example "Dynamic Object Addition"
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM2DynamicInteractivePredictor
|
||||
|
||||
# Create SAM2DynamicInteractivePredictor
|
||||
overrides = dict(conf=0.01, task="segment", mode="predict", imgsz=1024, model="sam2_t.pt", save=False)
|
||||
predictor = SAM2DynamicInteractivePredictor(overrides=overrides, max_obj_num=10)
|
||||
|
||||
# Define a category by box prompt
|
||||
predictor(source="image1.jpg", bboxes=[[100, 100, 200, 200]], obj_ids=[0], update_memory=True)
|
||||
|
||||
# Detect this particular object in a new image
|
||||
results = predictor(source="image2.jpg")
|
||||
|
||||
# Add new category with a new object ID
|
||||
results = predictor(
|
||||
source="image4.jpg",
|
||||
bboxes=[[300, 300, 400, 400]], # New object
|
||||
obj_ids=[1], # New object ID
|
||||
update_memory=True, # Add to memory
|
||||
)
|
||||
# Perform inference
|
||||
results = predictor(source="image5.jpg")
|
||||
|
||||
# Add refinement prompts to the same category to boost performance
|
||||
# This helps when object appearance changes significantly
|
||||
results = predictor(
|
||||
source="image6.jpg",
|
||||
points=[[150, 150]], # Refinement point
|
||||
labels=[1], # Positive point
|
||||
obj_ids=[1], # Same object ID
|
||||
update_memory=True, # Update memory with new information
|
||||
)
|
||||
# Perform inference on new image
|
||||
results = predictor(source="image7.jpg")
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The `SAM2DynamicInteractivePredictor` is designed to work with SAM2 models, and support adding/refining categories by all the [box/point/mask prompts](#sam-2-prediction-examples) natively that SAM2 supports. It is particularly useful for scenarios where objects appear or change over time, such as in video annotation or interactive editing tasks.
|
||||
|
||||
#### Arguments
|
||||
|
||||
| Name | Default Value | Data Type | Description |
|
||||
| --------------- | ------------- | ----------- | ------------------------------------------- |
|
||||
| `max_obj_num` | `3` | `int` | The preset maximum number of categories |
|
||||
| `update_memory` | `False` | `bool` | Whether to update memory with new prompts |
|
||||
| `obj_ids` | `None` | `List[int]` | List of object IDs corresponding to prompts |
|
||||
|
||||
### Use Cases
|
||||
|
||||
`SAM2DynamicInteractivePredictor` is ideal for:
|
||||
|
||||
- **Video annotation workflows** where new objects appear during the sequence
|
||||
- **Interactive video editing** requiring real-time object addition and refinement
|
||||
- **Surveillance applications** with dynamic object tracking needs
|
||||
- **Medical imaging** for tracking anatomical structures across time series
|
||||
- **Autonomous systems** requiring adaptive object detection and tracking
|
||||
- **Multi-image datasets** for consistent object segmentation across independent images
|
||||
- **Image collection analysis** where objects need to be tracked across different scenes
|
||||
- **Cross-domain segmentation** leveraging memory from diverse image contexts
|
||||
- **Semi-automatic annotation** for efficient dataset creation with minimal manual intervention
|
||||
|
||||
## SAM 2 Comparison vs YOLO
|
||||
|
||||
Here we compare Meta's SAM 2 models, including the smallest SAM2-t variant, with Ultralytics smallest segmentation model, [YOLO11n-seg](../tasks/segment.md):
|
||||
|
||||
| Model | Size<br><sup>(MB)</sup> | Parameters<br><sup>(M)</sup> | Speed (CPU)<br><sup>(ms/im)</sup> |
|
||||
| ---------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------- | --------------------------------- |
|
||||
| [Meta SAM-b](sam.md) | 375 | 93.7 | 49401 |
|
||||
| Meta SAM2-b | 162 | 80.8 | 31901 |
|
||||
| Meta SAM2-t | 78.1 | 38.9 | 25997 |
|
||||
| [MobileSAM](mobile-sam.md) | 40.7 | 10.1 | 25381 |
|
||||
| [FastSAM-s](fast-sam.md) with YOLOv8 [backbone](https://www.ultralytics.com/glossary/backbone) | 23.7 | 11.8 | 55.9 |
|
||||
| Ultralytics [YOLOv8n-seg](yolov8.md) | **6.7** (11.7x smaller) | **3.4** (11.4x less) | **24.5** (1061x faster) |
|
||||
| Ultralytics [YOLO11n-seg](yolo11.md) | **5.9** (13.2x smaller) | **2.9** (13.4x less) | **30.1** (864x faster) |
|
||||
|
||||
This comparison demonstrates the substantial differences in model sizes and speeds between SAM variants and YOLO segmentation models. While SAM provides unique automatic segmentation capabilities, YOLO models, particularly YOLOv8n-seg and YOLO11n-seg, are significantly smaller, faster, and more computationally efficient.
|
||||
|
||||
Tests run on a 2025 Apple M4 Pro with 24GB of RAM using `torch==2.6.0` and `ultralytics==8.3.90`. To reproduce this test:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, SAM, YOLO, FastSAM
|
||||
|
||||
# Profile SAM2-t, SAM2-b, SAM-b, MobileSAM
|
||||
for file in ["sam_b.pt", "sam2_b.pt", "sam2_t.pt", "mobile_sam.pt"]:
|
||||
model = SAM(file)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile FastSAM-s
|
||||
model = FastSAM("FastSAM-s.pt")
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile YOLO models
|
||||
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt"]:
|
||||
model = YOLO(file_name)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
```
|
||||
|
||||
## Auto-Annotation: Efficient Dataset Creation
|
||||
|
||||
Auto-annotation is a powerful feature of SAM 2, enabling users to generate segmentation datasets quickly and accurately by leveraging pretrained models. This capability is particularly useful for creating large, high-quality datasets without extensive manual effort.
|
||||
|
||||
### How to Auto-Annotate with SAM 2
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/M7xWw4Iodhg"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Auto Annotation with Meta's Segment Anything 2 Model using Ultralytics | Data Labeling
|
||||
</p>
|
||||
|
||||
To auto-annotate your dataset using SAM 2, follow this example:
|
||||
|
||||
!!! example "Auto-Annotation Example"
|
||||
|
||||
```python
|
||||
from ultralytics.data.annotator import auto_annotate
|
||||
|
||||
auto_annotate(data="path/to/images", det_model="yolo11x.pt", sam_model="sam2_b.pt")
|
||||
```
|
||||
|
||||
{% include "macros/sam-auto-annotate.md" %}
|
||||
|
||||
This function facilitates the rapid creation of high-quality segmentation datasets, ideal for researchers and developers aiming to accelerate their projects.
|
||||
|
||||
## Limitations
|
||||
|
||||
Despite its strengths, SAM 2 has certain limitations:
|
||||
|
||||
- **Tracking Stability**: SAM 2 may lose track of objects during extended sequences or significant viewpoint changes.
|
||||
- **Object Confusion**: The model can sometimes confuse similar-looking objects, particularly in crowded scenes.
|
||||
- **Efficiency with Multiple Objects**: Segmentation efficiency decreases when processing multiple objects simultaneously due to the lack of inter-object communication.
|
||||
- **Detail [Accuracy](https://www.ultralytics.com/glossary/accuracy)**: May miss fine details, especially with fast-moving objects. Additional prompts can partially address this issue, but temporal smoothness is not guaranteed.
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If SAM 2 is a crucial part of your research or development work, please cite it using the following reference:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{ravi2024sam2,
|
||||
title={SAM 2: Segment Anything in Images and Videos},
|
||||
author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
|
||||
journal={arXiv preprint},
|
||||
year={2024}
|
||||
}
|
||||
```
|
||||
|
||||
We extend our gratitude to Meta AI for their contributions to the AI community with this groundbreaking model and dataset.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is SAM 2 and how does it improve upon the original Segment Anything Model (SAM)?
|
||||
|
||||
SAM 2, the successor to Meta's [Segment Anything Model (SAM)](sam.md), is a cutting-edge tool designed for comprehensive object segmentation in both images and videos. It excels in handling complex visual data through a unified, promptable model architecture that supports real-time processing and zero-shot generalization. SAM 2 offers several improvements over the original SAM, including:
|
||||
|
||||
- **Unified Model Architecture**: Combines image and video segmentation capabilities in a single model.
|
||||
- **Real-Time Performance**: Processes approximately 44 frames per second, making it suitable for applications requiring immediate feedback.
|
||||
- **Zero-Shot Generalization**: Segments objects it has never encountered before, useful in diverse visual domains.
|
||||
- **Interactive Refinement**: Allows users to iteratively refine segmentation results by providing additional prompts.
|
||||
- **Advanced Handling of Visual Challenges**: Manages common video segmentation challenges like object occlusion and reappearance.
|
||||
|
||||
For more details on SAM 2's architecture and capabilities, explore the [SAM 2 research paper](https://arxiv.org/abs/2408.00714).
|
||||
|
||||
### How can I use SAM 2 for real-time video segmentation?
|
||||
|
||||
SAM 2 can be utilized for real-time video segmentation by leveraging its promptable interface and real-time inference capabilities. Here's a basic example:
|
||||
|
||||
!!! example "Segment with Prompts"
|
||||
|
||||
Use prompts to segment specific objects in images or videos.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam2_b.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Segment with bounding box prompt
|
||||
results = model("path/to/image.jpg", bboxes=[100, 100, 200, 200])
|
||||
|
||||
# Segment with point prompt
|
||||
results = model("path/to/image.jpg", points=[150, 150], labels=[1])
|
||||
```
|
||||
|
||||
For more comprehensive usage, refer to the [How to Use SAM 2](#how-to-use-sam-2-versatility-in-image-and-video-segmentation) section.
|
||||
|
||||
### What datasets are used to train SAM 2, and how do they enhance its performance?
|
||||
|
||||
SAM 2 is trained on the SA-V dataset, one of the largest and most diverse video segmentation datasets available. The SA-V dataset includes:
|
||||
|
||||
- **51,000+ Videos**: Captured across 47 countries, providing a wide range of real-world scenarios.
|
||||
- **600,000+ Mask Annotations**: Detailed spatio-temporal mask annotations, referred to as "masklets," covering whole objects and parts.
|
||||
- **Dataset Scale**: Features 4.5 times more videos and 53 times more annotations than previous largest datasets, offering unprecedented diversity and complexity.
|
||||
|
||||
This extensive dataset allows SAM 2 to achieve superior performance across major video segmentation benchmarks and enhances its zero-shot generalization capabilities. For more information, see the [SA-V Dataset](#sa-v-dataset) section.
|
||||
|
||||
### How does SAM 2 handle occlusions and object reappearances in video segmentation?
|
||||
|
||||
SAM 2 includes a sophisticated memory mechanism to manage temporal dependencies and occlusions in video data. The memory mechanism consists of:
|
||||
|
||||
- **Memory Encoder and Memory Bank**: Stores features from past frames.
|
||||
- **Memory Attention Module**: Utilizes stored information to maintain consistent object tracking over time.
|
||||
- **Occlusion Head**: Specifically handles scenarios where objects are not visible, predicting the likelihood of an object being occluded.
|
||||
|
||||
This mechanism ensures continuity even when objects are temporarily obscured or exit and re-enter the scene. For more details, refer to the [Memory Mechanism and Occlusion Handling](#memory-mechanism-and-occlusion-handling) section.
|
||||
|
||||
### How does SAM 2 compare to other segmentation models like YOLO11?
|
||||
|
||||
SAM 2 models, such as Meta's SAM2-t and SAM2-b, offer powerful zero-shot segmentation capabilities but are significantly larger and slower compared to YOLO11 models. For instance, YOLO11n-seg is approximately **13 times smaller** and over **860 times faster** than SAM2-b. While SAM 2 excels in versatile, prompt-based, and zero-shot segmentation scenarios, YOLO11 is optimized for speed, efficiency, and real-time applications, making it better suited for deployment in resource-constrained environments.
|
||||
648
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam-3.md
Normal file
648
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam-3.md
Normal file
@@ -0,0 +1,648 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover SAM 3, Meta's next evolution of the Segment Anything Model, introducing Promptable Concept Segmentation with text and image exemplar prompts for detecting all instances of visual concepts across images and videos.
|
||||
keywords: SAM 3, Segment Anything 3, SAM3, SAM-3, video segmentation, image segmentation, concept segmentation, promptable AI, SA-Co dataset, Meta, Ultralytics, computer vision, AI, machine learning, open vocabulary
|
||||
---
|
||||
|
||||
# SAM 3: Segment Anything with Concepts
|
||||
|
||||
!!! success "Now Available in Ultralytics"
|
||||
|
||||
SAM 3 is fully integrated into the Ultralytics package as of **version 8.3.237** ([PR #22897](https://github.com/ultralytics/ultralytics/pull/22897)). Install or upgrade with `pip install -U ultralytics` to access all SAM 3 features including text-based concept segmentation, image exemplar prompts, and video tracking.
|
||||
|
||||
|
||||
|
||||
**SAM 3** (Segment Anything Model 3) is Meta's released foundation model for **Promptable Concept Segmentation (PCS)**. Building upon [SAM 2](sam-2.md), SAM 3 introduces a fundamentally new capability: detecting, segmenting, and tracking **all instances** of a visual concept specified by text prompts, image exemplars, or both. Unlike previous SAM versions that segment single objects per prompt, SAM 3 can find and segment every occurrence of a concept appearing anywhere in images or videos, aligning with open-vocabulary goals in modern [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation).
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/BE2Nu3edyOo"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Use Meta Segment Anything 3 with Ultralytics | Text-Prompt Segmentation on Images & Videos
|
||||
</p>
|
||||
|
||||
SAM 3 is now fully integrated into the `ultralytics` package, providing native support for concept segmentation with text prompts, image exemplar prompts, and video tracking capabilities.
|
||||
|
||||
## Overview
|
||||
|
||||
SAM 3 achieves a **2× performance gain** over existing systems in Promptable Concept Segmentation while maintaining and improving SAM 2's capabilities for interactive [visual segmentation](../tasks/segment.md). The model excels at open-vocabulary segmentation, allowing users to specify concepts using simple noun phrases (e.g., "yellow school bus", "striped cat") or by providing example images of the target object. These capabilities complement production-ready pipelines that rely on streamlined [predict](../modes/predict.md) and [track](../modes/track.md) workflows.
|
||||
|
||||
|
||||
|
||||
### What is Promptable Concept Segmentation (PCS)?
|
||||
|
||||
The PCS task takes a **concept prompt** as input and returns segmentation masks with unique identities for **all matching object instances**. Concept prompts can be:
|
||||
|
||||
- **Text**: Simple noun phrases like "red apple" or "person wearing a hat", similar to [zero-shot learning](https://www.ultralytics.com/glossary/zero-shot-learning)
|
||||
- **Image exemplars**: Bounding boxes around example objects (positive or negative) for fast generalization
|
||||
- **Combined**: Both text and image exemplars together for precise control
|
||||
|
||||
This differs from traditional visual prompts (points, boxes, masks) which segment only a single specific object instance, as popularized by the original [SAM family](../models/sam.md).
|
||||
|
||||
### Key Performance Metrics
|
||||
|
||||
| Metric | SAM 3 Achievement |
|
||||
| ------------------------------ | -------------------------------------------------------- |
|
||||
| **LVIS Zero-Shot Mask AP** | **47.0** (vs previous best 38.5, +22% improvement) |
|
||||
| **SA-Co Benchmark** | **2× better** than existing systems |
|
||||
| **Inference Speed (H200 GPU)** | **30 ms** per image with 100+ detected objects |
|
||||
| **Video Performance** | Near real-time for ~5 concurrent objects |
|
||||
| **MOSEv2 VOS Benchmark** | **60.1 J&F** (+25.5% over SAM 2.1, +17% over prior SOTA) |
|
||||
| **Interactive Refinement** | **+18.6 CGF1** improvement after 3 exemplar prompts |
|
||||
| **Human Performance Gap** | Achieves **88%** of estimated lower bound on SA-Co/Gold |
|
||||
|
||||
For context on model metrics and trade-offs in production, see [model evaluation insights](../guides/model-evaluation-insights.md) and [YOLO performance metrics](../guides/yolo-performance-metrics.md).
|
||||
|
||||
## Architecture
|
||||
|
||||
SAM 3 consists of a **detector** and **tracker** that share a Perception Encoder (PE) vision backbone. This decoupled design avoids task conflicts while enabling both image-level detection and video-level tracking, with an interface compatible with Ultralytics [Python usage](../usage/python.md) and [CLI usage](../usage/cli.md).
|
||||
|
||||
### Core Components
|
||||
|
||||
- **Detector**: [DETR-based architecture](rtdetr.md) for image-level concept detection
|
||||
- Text encoder for noun phrase prompts
|
||||
- Exemplar encoder for image-based prompts
|
||||
- Fusion encoder to condition image features on prompts
|
||||
- Novel **presence head** that decouples recognition ("what") from localization ("where")
|
||||
- Mask head for generating instance segmentation masks
|
||||
|
||||
- **Tracker**: Memory-based video segmentation inherited from [SAM 2](sam-2.md)
|
||||
- Prompt encoder, mask decoder, memory encoder
|
||||
- Memory bank for storing object appearance across frames
|
||||
- Temporal disambiguation aided by techniques like a [Kalman filter](../reference/trackers/utils/kalman_filter.md) in multi-object settings
|
||||
|
||||
- **Presence Token**: A learned global token that predicts whether the target concept is present in the image/frame, improving detection by separating recognition from localization.
|
||||
|
||||
|
||||
|
||||
### Key Innovations
|
||||
|
||||
1. **Decoupled Recognition and Localization**: The presence head predicts concept presence globally, while proposal queries focus only on localization, avoiding conflicting objectives.
|
||||
2. **Unified Concept and Visual Prompts**: Supports both PCS (concept prompts) and PVS (visual prompts like SAM 2's clicks/boxes) in a single model.
|
||||
3. **Interactive Exemplar Refinement**: Users can add positive or negative image exemplars to iteratively refine results, with the model generalizing to similar objects rather than just correcting individual instances.
|
||||
4. **Temporal Disambiguation**: Uses masklet detection scores and periodic re-prompting to handle occlusions, crowded scenes, and tracking failures in video, aligning with [instance segmentation and tracking](../guides/instance-segmentation-and-tracking.md) best practices.
|
||||
|
||||
## SA-Co Dataset
|
||||
|
||||
SAM 3 is trained on **Segment Anything with Concepts (SA-Co)**, Meta's largest and most diverse segmentation dataset to date, expanding beyond common benchmarks like [COCO](../datasets/detect/coco.md) and [LVIS](../datasets/detect/lvis.md).
|
||||
|
||||
### Training Data
|
||||
|
||||
| Dataset Component | Description | Scale |
|
||||
| ----------------- | ---------------------------------------------------------------- | --------------------------------------- |
|
||||
| **SA-Co/HQ** | High-quality human-annotated image data from 4-phase data engine | 5.2M images, 4M unique noun phrases |
|
||||
| **SA-Co/SYN** | Synthetic dataset labeled by AI without human involvement | 38M noun phrases, 1.4B masks |
|
||||
| **SA-Co/EXT** | 15 external datasets enriched with hard negatives | Varies by source |
|
||||
| **SA-Co/VIDEO** | Video annotations with temporal tracking | 52.5K videos, 24.8K unique noun phrases |
|
||||
|
||||
### Benchmark Data
|
||||
|
||||
The **SA-Co evaluation benchmark** contains **214K unique phrases** across **126K images and videos**, providing over **50× more concepts** than existing benchmarks. It includes:
|
||||
|
||||
- **SA-Co/Gold**: 7 domains, triple-annotated for measuring human performance bounds
|
||||
- **SA-Co/Silver**: 10 domains, single human annotation
|
||||
- **SA-Co/Bronze** and **SA-Co/Bio**: 9 existing datasets adapted for concept segmentation
|
||||
- **SA-Co/VEval**: Video benchmark with 3 domains (SA-V, YT-Temporal-1B, SmartGlasses)
|
||||
|
||||
### Data Engine Innovations
|
||||
|
||||
SAM 3's scalable human- and model-in-the-loop data engine achieves **2× annotation throughput** through:
|
||||
|
||||
1. **AI Annotators**: [Llama](https://arxiv.org/abs/2302.13971) based models propose diverse noun phrases including hard negatives
|
||||
2. **AI Verifiers**: Fine-tuned [multimodal LLMs](https://ai.google.dev/gemini-api/docs) verify mask quality and exhaustivity at near-human performance
|
||||
3. **Active Mining**: Focuses human effort on challenging failure cases where AI struggles
|
||||
4. **Ontology-Driven**: Leverages a large ontology grounded in [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page) for concept coverage
|
||||
|
||||
## Installation
|
||||
|
||||
SAM 3 is available in Ultralytics **version 8.3.237** and later. Install or upgrade with:
|
||||
|
||||
```bash
|
||||
pip install -U ultralytics
|
||||
```
|
||||
|
||||
!!! warning "SAM 3 Model Weights Required"
|
||||
|
||||
Unlike other Ultralytics models, SAM 3 weights (`sam3.pt`) are **not automatically downloaded**. You must first request access for the model weights on the [SAM 3 model page on Hugging Face](https://huggingface.co/facebook/sam3) and then, once approved, download the [`sam3.pt` file](https://huggingface.co/facebook/sam3/resolve/main/sam3.pt?download=true). Place the downloaded `sam3.pt` file in your working directory or specify the full path when loading the model.
|
||||
|
||||
!!! warning "`TypeError: 'SimpleTokenizer' object is not callable`"
|
||||
|
||||
If you get the above error during prediction, it means you have the incorrect `clip` package installed. Install the correct `clip` package by running the following:
|
||||
```bash
|
||||
pip uninstall clip -y
|
||||
pip install git+https://github.com/ultralytics/CLIP.git
|
||||
```
|
||||
|
||||
## How to Use SAM 3: Versatility in Concept Segmentation
|
||||
|
||||
SAM 3 supports both Promptable Concept Segmentation (PCS) and Promptable Visual Segmentation (PVS) tasks through different predictor interfaces:
|
||||
|
||||
### Supported Tasks and Models
|
||||
|
||||
| Task Type | Prompt Types | Output |
|
||||
| ------------------------------ | ------------------------------------------ | ------------------------------------------- |
|
||||
| **Concept Segmentation (PCS)** | Text (noun phrases), image exemplars | All instances matching the concept |
|
||||
| **Visual Segmentation (PVS)** | Points, boxes, masks | Single object instance (SAM 2 style) |
|
||||
| **Interactive Refinement** | Add/remove exemplars or clicks iteratively | Refined segmentation with improved accuracy |
|
||||
|
||||
### Concept Segmentation Examples
|
||||
|
||||
#### Segment with Text Prompts
|
||||
|
||||
!!! example "Text-based Concept Segmentation"
|
||||
|
||||
Find and segment all instances of a concept using a text description. Text prompts require the `SAM3SemanticPredictor` interface.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM3SemanticPredictor
|
||||
|
||||
# Initialize predictor with configuration
|
||||
overrides = dict(
|
||||
conf=0.25,
|
||||
task="segment",
|
||||
mode="predict",
|
||||
model="sam3.pt",
|
||||
half=True, # Use FP16 for faster inference
|
||||
save=True,
|
||||
)
|
||||
predictor = SAM3SemanticPredictor(overrides=overrides)
|
||||
|
||||
# Set image once for multiple queries
|
||||
predictor.set_image("path/to/image.jpg")
|
||||
|
||||
# Query with multiple text prompts
|
||||
results = predictor(text=["person", "bus", "glasses"])
|
||||
|
||||
# Works with descriptive phrases
|
||||
results = predictor(text=["person with red cloth", "person with blue cloth"])
|
||||
|
||||
# Query with a single concept
|
||||
results = predictor(text=["a person"])
|
||||
```
|
||||
|
||||
#### Segment with Image Exemplars
|
||||
|
||||
!!! example "Image Exemplar-based Segmentation"
|
||||
|
||||
Use bounding boxes as visual prompts to find all similar instances. This also requires `SAM3SemanticPredictor` for concept-based matching.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM3SemanticPredictor
|
||||
|
||||
# Initialize predictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", model="sam3.pt", half=True, save=True)
|
||||
predictor = SAM3SemanticPredictor(overrides=overrides)
|
||||
|
||||
# Set image
|
||||
predictor.set_image("path/to/image.jpg")
|
||||
|
||||
# Provide bounding box examples to segment similar objects
|
||||
results = predictor(bboxes=[[480.0, 290.0, 590.0, 650.0]])
|
||||
|
||||
# Multiple bounding boxes for different concepts
|
||||
results = predictor(bboxes=[[539, 599, 589, 639], [343, 267, 499, 662]])
|
||||
```
|
||||
|
||||
#### Feature-based Inference for Efficiency
|
||||
|
||||
!!! example "Reusing Image Features for Multiple Queries"
|
||||
|
||||
Extract image features once and reuse them for multiple segmentation queries to improve efficiency.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
|
||||
from ultralytics.models.sam import SAM3SemanticPredictor
|
||||
from ultralytics.utils.plotting import Annotator, colors
|
||||
|
||||
# Initialize predictors
|
||||
overrides = dict(conf=0.50, task="segment", mode="predict", model="sam3.pt", verbose=False)
|
||||
predictor = SAM3SemanticPredictor(overrides=overrides)
|
||||
predictor2 = SAM3SemanticPredictor(overrides=overrides)
|
||||
|
||||
# Extract features from the first predictor
|
||||
source = "path/to/image.jpg"
|
||||
predictor.set_image(source)
|
||||
src_shape = cv2.imread(source).shape[:2]
|
||||
|
||||
# Setup second predictor and reuse features
|
||||
predictor2.setup_model()
|
||||
|
||||
# Perform inference using shared features with text prompt
|
||||
masks, boxes = predictor2.inference_features(predictor.features, src_shape=src_shape, text=["person"])
|
||||
|
||||
# Perform inference using shared features with bounding box prompt
|
||||
masks, boxes = predictor2.inference_features(predictor.features, src_shape=src_shape, bboxes=[[439, 437, 524, 709]])
|
||||
|
||||
# Visualize results
|
||||
if masks is not None:
|
||||
masks, boxes = masks.cpu().numpy(), boxes.cpu().numpy()
|
||||
im = cv2.imread(source)
|
||||
annotator = Annotator(im, pil=False)
|
||||
annotator.masks(masks, [colors(x, True) for x in range(len(masks))])
|
||||
|
||||
cv2.imshow("result", annotator.result())
|
||||
cv2.waitKey(0)
|
||||
```
|
||||
|
||||
### Video Concept Segmentation
|
||||
|
||||
#### Track Concepts Across Video with Bounding Boxes
|
||||
|
||||
!!! example "Video Tracking with Visual Prompts"
|
||||
|
||||
Detect and track object instances across video frames using bounding box prompts.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM3VideoPredictor
|
||||
|
||||
# Create video predictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", model="sam3.pt", half=True)
|
||||
predictor = SAM3VideoPredictor(overrides=overrides)
|
||||
|
||||
# Track objects using bounding box prompts
|
||||
results = predictor(source="path/to/video.mp4", bboxes=[[706.5, 442.5, 905.25, 555], [598, 635, 725, 750]], stream=True)
|
||||
|
||||
# Process and display results
|
||||
for r in results:
|
||||
r.show() # Display frame with segmentation masks
|
||||
```
|
||||
|
||||
#### Track Concepts with Text Prompts
|
||||
|
||||
!!! example "Video Tracking with Semantic Queries"
|
||||
|
||||
Track all instances of concepts specified by text across video frames.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import SAM3VideoSemanticPredictor
|
||||
|
||||
# Initialize semantic video predictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", imgsz=640, model="sam3.pt", half=True, save=True)
|
||||
predictor = SAM3VideoSemanticPredictor(overrides=overrides)
|
||||
|
||||
# Track concepts using text prompts
|
||||
results = predictor(source="path/to/video.mp4", text=["person", "bicycle"], stream=True)
|
||||
|
||||
# Process results
|
||||
for r in results:
|
||||
r.show() # Display frame with tracked objects
|
||||
|
||||
# Alternative: Track with bounding box prompts
|
||||
results = predictor(
|
||||
source="path/to/video.mp4",
|
||||
bboxes=[[864, 383, 975, 620], [705, 229, 782, 402]],
|
||||
labels=[1, 1], # Positive labels
|
||||
stream=True,
|
||||
)
|
||||
```
|
||||
|
||||
### Visual Prompts (SAM 2 Compatibility)
|
||||
|
||||
SAM 3 maintains full backward compatibility with SAM 2's visual prompting for single-object segmentation:
|
||||
|
||||
!!! example "SAM 2 Style Visual Prompts"
|
||||
|
||||
The basic `SAM` interface behaves exactly like SAM 2, segmenting only the specific area indicated by visual prompts (points, boxes, or masks).
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
model = SAM("sam3.pt")
|
||||
|
||||
# Single point prompt - segments object at specific location
|
||||
results = model.predict(source="path/to/image.jpg", points=[900, 370], labels=[1])
|
||||
results[0].show()
|
||||
|
||||
# Multiple points - segments single object with multiple point hints
|
||||
results = model.predict(source="path/to/image.jpg", points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Box prompt - segments object within bounding box
|
||||
results = model.predict(source="path/to/image.jpg", bboxes=[100, 150, 300, 400])
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
!!! warning "Visual Prompts vs Concept Segmentation"
|
||||
|
||||
Using `SAM("sam3.pt")` with visual prompts (points/boxes/masks) will segment **only the specific object** at that location, just like SAM 2. To segment **all instances of a concept**, use `SAM3SemanticPredictor` with text or exemplar prompts as shown above.
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
### Image Segmentation
|
||||
|
||||
SAM 3 achieves state-of-the-art results across multiple benchmarks, including real-world datasets like [LVIS](../datasets/detect/lvis.md) and [COCO for segmentation](../datasets/segment/coco.md):
|
||||
|
||||
| Benchmark | Metric | SAM 3 | Previous Best | Improvement |
|
||||
| ----------------------------- | ------- | -------- | ------------- | ----------- |
|
||||
| **LVIS (zero-shot)** | Mask AP | **47.0** | 38.5 | +22.1% |
|
||||
| **SA-Co/Gold** | CGF1 | **65.0** | 34.3 (OWLv2) | +89.5% |
|
||||
| **COCO (zero-shot)** | Box AP | **53.5** | 52.2 (T-Rex2) | +2.5% |
|
||||
| **ADE-847 (semantic seg)** | mIoU | **14.7** | 9.2 (APE-D) | +59.8% |
|
||||
| **PascalConcept-59** | mIoU | **59.4** | 58.5 (APE-D) | +1.5% |
|
||||
| **Cityscapes (semantic seg)** | mIoU | **65.1** | 44.2 (APE-D) | +47.3% |
|
||||
|
||||
Explore dataset options for quick experimentation in [Ultralytics datasets](../datasets/index.md).
|
||||
|
||||
### Video Segmentation Performance
|
||||
|
||||
SAM 3 shows significant improvements over SAM 2 and prior state-of-the-art across video benchmarks such as [DAVIS 2017](https://davischallenge.org/) and [YouTube-VOS](https://youtube-vos.org/):
|
||||
|
||||
| Benchmark | Metric | SAM 3 | SAM 2.1 L | Improvement |
|
||||
| -------------- | ------ | -------- | --------- | ----------- |
|
||||
| **MOSEv2** | J&F | **60.1** | 47.9 | +25.5% |
|
||||
| **DAVIS 2017** | J&F | **92.0** | 90.7 | +1.4% |
|
||||
| **LVOSv2** | J&F | **88.2** | 79.6 | +10.8% |
|
||||
| **SA-V** | J&F | **84.6** | 78.4 | +7.9% |
|
||||
| **YTVOS19** | J&F | **89.6** | 89.3 | +0.3% |
|
||||
|
||||
### Few-Shot Adaptation
|
||||
|
||||
SAM 3 excels at adapting to new domains with minimal examples, relevant for [data-centric AI](https://www.ultralytics.com/glossary/data-centric-ai) workflows:
|
||||
|
||||
| Benchmark | 0-shot AP | 10-shot AP | Previous Best (10-shot) |
|
||||
| ------------ | --------- | ---------- | ----------------------- |
|
||||
| **ODinW13** | 59.9 | **71.6** | 67.9 (gDino1.5-Pro) |
|
||||
| **RF100-VL** | 14.3 | **35.7** | 33.7 (gDino-T) |
|
||||
|
||||
### Interactive Refinement Effectiveness
|
||||
|
||||
SAM 3's concept-based prompting with exemplars converges much faster than visual prompting:
|
||||
|
||||
| Prompts Added | CGF1 Score | Gain vs Text-Only | Gain vs PVS Baseline |
|
||||
| ------------- | ---------- | ----------------- | -------------------- |
|
||||
| Text only | 46.4 | baseline | baseline |
|
||||
| +1 exemplar | 57.6 | +11.2 | +6.7 |
|
||||
| +2 exemplars | 62.2 | +15.8 | +9.7 |
|
||||
| +3 exemplars | **65.0** | **+18.6** | **+11.2** |
|
||||
| +4 exemplars | 65.7 | +19.3 | +11.5 (plateau) |
|
||||
|
||||
### Object Counting Accuracy
|
||||
|
||||
SAM 3 provides accurate counting by segmenting all instances, a common requirement in [object counting](../guides/object-counting.md):
|
||||
|
||||
| Benchmark | Accuracy | MAE | vs Best MLLM |
|
||||
| --------------- | --------- | ---- | ------------------ |
|
||||
| **CountBench** | **95.6%** | 0.11 | 92.4% (Gemini 2.5) |
|
||||
| **PixMo-Count** | **87.3%** | 0.22 | 88.8% (Molmo-72B) |
|
||||
|
||||
## SAM 3 vs SAM 2 vs YOLO Comparison
|
||||
|
||||
Here we compare SAM 3's capabilities with SAM 2 and [YOLO11](../models/yolo11.md) models:
|
||||
|
||||
| Capability | SAM 3 | SAM 2 | YOLO11n-seg |
|
||||
| ---------------------------- | ------------------------------------- | -------------------- | ------------------ |
|
||||
| **Concept Segmentation** | ✅ All instances from text/exemplars | ❌ Not supported | ❌ Not supported |
|
||||
| **Visual Segmentation** | ✅ Single instance (SAM 2 compatible) | ✅ Single instance | ✅ All instances |
|
||||
| **Zero-shot Capability** | ✅ Open vocabulary | ✅ Geometric prompts | ❌ Closed set |
|
||||
| **Interactive Refinement** | ✅ Exemplars + clicks | ✅ Clicks only | ❌ Not supported |
|
||||
| **Video Tracking** | ✅ Multi-object with identities | ✅ Multi-object | ✅ Multi-object |
|
||||
| **LVIS Mask AP (zero-shot)** | **47.0** | N/A | N/A |
|
||||
| **MOSEv2 J&F** | **60.1** | 47.9 | N/A |
|
||||
| **Inference Speed (H200)** | **30 ms** (100+ objects) | ~23 ms (per object) | **2-3 ms** (image) |
|
||||
| **Model Size** | 3.4GB | 162 MB (base) | **5.9 MB** |
|
||||
|
||||
**Key Takeaways**:
|
||||
|
||||
- **SAM 3**: Best for open-vocabulary concept segmentation, finding all instances of a concept with text or exemplar prompts
|
||||
- **SAM 2**: Best for interactive single-object segmentation in images and videos with geometric prompts
|
||||
- **YOLO11**: Best for real-time, high-speed segmentation in resource-constrained deployments using efficient [export pipelines](../modes/export.md) like [ONNX](../integrations/onnx.md) and [TensorRT](../integrations/tensorrt.md)
|
||||
|
||||
## Evaluation Metrics
|
||||
|
||||
SAM 3 introduces new metrics designed for the PCS task, complementing familiar measures like [F1 score](https://www.ultralytics.com/glossary/f1-score), [precision](https://www.ultralytics.com/glossary/precision), and [recall](https://www.ultralytics.com/glossary/recall).
|
||||
|
||||
### Classification-Gated F1 (CGF1)
|
||||
|
||||
The primary metric combining localization and classification:
|
||||
|
||||
**CGF1 = 100 × pmF1 × IL_MCC**
|
||||
|
||||
Where:
|
||||
|
||||
- **pmF1** (Positive Macro F1): Measures localization quality on positive examples
|
||||
- **IL_MCC** (Image-Level Matthews Correlation Coefficient): Measures binary classification accuracy ("is the concept present?")
|
||||
|
||||
### Why These Metrics?
|
||||
|
||||
Traditional AP metrics don't account for calibration, making models difficult to use in practice. By evaluating only predictions above 0.5 confidence, SAM 3's metrics enforce good calibration and mimic real-world usage patterns in interactive [predict](../modes/predict.md) and [track](../modes/track.md) loops.
|
||||
|
||||
## Key Ablations and Insights
|
||||
|
||||
### Impact of Presence Head
|
||||
|
||||
The presence head decouples recognition from localization, providing significant improvements:
|
||||
|
||||
| Configuration | CGF1 | IL_MCC | pmF1 |
|
||||
| ----------------- | -------- | -------- | -------- |
|
||||
| Without presence | 57.6 | 0.77 | 74.7 |
|
||||
| **With presence** | **63.3** | **0.82** | **77.1** |
|
||||
|
||||
The presence head provides a **+5.7 CGF1 boost** (+9.9%), primarily improving recognition ability (IL_MCC +6.5%).
|
||||
|
||||
### Effect of Hard Negatives
|
||||
|
||||
| Hard Negatives/Image | CGF1 | IL_MCC | pmF1 |
|
||||
| -------------------- | -------- | -------- | -------- |
|
||||
| 0 | 31.8 | 0.44 | 70.2 |
|
||||
| 5 | 44.8 | 0.62 | 71.9 |
|
||||
| **30** | **49.2** | **0.68** | **72.3** |
|
||||
|
||||
Hard negatives are crucial for open-vocabulary recognition, improving IL_MCC by **54.5%** (0.44 → 0.68).
|
||||
|
||||
### Training Data Scaling
|
||||
|
||||
| Data Sources | CGF1 | IL_MCC | pmF1 |
|
||||
| -------------------- | -------- | -------- | -------- |
|
||||
| External only | 30.9 | 0.46 | 66.3 |
|
||||
| External + Synthetic | 39.7 | 0.57 | 70.6 |
|
||||
| External + HQ | 51.8 | 0.71 | 73.2 |
|
||||
| **All three** | **54.3** | **0.74** | **73.5** |
|
||||
|
||||
High-quality human annotations provide large gains over synthetic or external data alone. For background on data quality practices, see [data collection and annotation](../guides/data-collection-and-annotation.md).
|
||||
|
||||
## Applications
|
||||
|
||||
SAM 3's concept segmentation capability enables new use cases:
|
||||
|
||||
- **Content Moderation**: Find all instances of specific content types across media libraries
|
||||
- **E-commerce**: Segment all products of a certain type in catalog images, supporting [auto-annotation](../guides/preprocessing_annotated_data.md)
|
||||
- **Medical Imaging**: Identify all occurrences of specific tissue types or abnormalities
|
||||
- **Autonomous Systems**: Track all instances of traffic signs, pedestrians, or vehicles by category
|
||||
- **Video Analytics**: Count and track all people wearing specific clothing or performing actions
|
||||
- **Dataset Annotation**: Rapidly annotate all instances of rare object categories
|
||||
- **Scientific Research**: Quantify and analyze all specimens matching specific criteria
|
||||
|
||||
## SAM 3 Agent: Extended Language Reasoning
|
||||
|
||||
SAM 3 can be combined with Multimodal Large Language Models (MLLMs) to handle complex queries requiring reasoning, similar in spirit to open-vocabulary systems like [OWLv2](https://arxiv.org/abs/2306.09683) and [T-Rex](https://arxiv.org/abs/2401.03533).
|
||||
|
||||
### Performance on Reasoning Tasks
|
||||
|
||||
| Benchmark | Metric | SAM 3 Agent (Gemini 2.5 Pro) | Previous Best |
|
||||
| -------------------------- | ------ | ---------------------------- | ------------- |
|
||||
| **ReasonSeg (validation)** | gIoU | **76.0** | 65.0 (SoTA) |
|
||||
| **ReasonSeg (test)** | gIoU | **73.8** | 61.3 (SoTA) |
|
||||
| **OmniLabel (validation)** | AP | **46.7** | 36.5 (REAL) |
|
||||
| **RefCOCO+** | Acc | **91.2** | 89.3 (LISA) |
|
||||
|
||||
### Example Complex Queries
|
||||
|
||||
SAM 3 Agent can handle queries requiring reasoning:
|
||||
|
||||
- "People sitting down but not holding a gift box in their hands"
|
||||
- "The dog closest to the camera that is not wearing a collar"
|
||||
- "Red objects larger than the person's hand"
|
||||
|
||||
The MLLM proposes simple noun phrase queries to SAM 3, analyzes returned masks, and iterates until satisfied.
|
||||
|
||||
## Limitations
|
||||
|
||||
While SAM 3 represents a major advancement, it has certain limitations:
|
||||
|
||||
- **Phrase Complexity**: Best suited for simple noun phrases; long referring expressions or complex reasoning may require MLLM integration
|
||||
- **Ambiguity Handling**: Some concepts remain inherently ambiguous (e.g., "small window", "cozy room")
|
||||
- **Computational Requirements**: Larger and slower than specialized detection models like [YOLO](../models/yolo26.md)
|
||||
- **Vocabulary Scope**: Focused on atomic visual concepts; compositional reasoning is limited without MLLM assistance
|
||||
- **Rare Concepts**: Performance may degrade on extremely rare or fine-grained concepts not well-represented in training data
|
||||
|
||||
## Citation
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@inproceedings{sam3_2025,
|
||||
title = {SAM 3: Segment Anything with Concepts},
|
||||
author = {Anonymous authors},
|
||||
booktitle = {Submitted to ICLR 2026},
|
||||
year = {2025},
|
||||
url = {https://openreview.net/forum?id=r35clVtGzw},
|
||||
note = {Paper ID: 4183, under double-blind review}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
### When Was SAM 3 Released?
|
||||
|
||||
SAM 3 was released by Meta on **November 20th, 2025** and is fully integrated into Ultralytics as of **version 8.3.237** ([PR #22897](https://github.com/ultralytics/ultralytics/pull/22897)). Full support is available for [predict mode](../modes/predict.md) and [track mode](../modes/track.md).
|
||||
|
||||
### Is SAM 3 Integrated Into Ultralytics?
|
||||
|
||||
Yes! SAM 3 is fully integrated into the Ultralytics Python package, including concept segmentation, SAM 2–style visual prompts, and multi-object video tracking.
|
||||
|
||||
### What Is Promptable Concept Segmentation (PCS)?
|
||||
|
||||
PCS is a new task introduced in SAM 3 that segments **all instances** of a visual concept in an image or video. Unlike traditional segmentation that targets a specific object instance, PCS finds every occurrence of a category. For example:
|
||||
|
||||
- **Text prompt**: "yellow school bus" → segments all yellow school buses in the scene
|
||||
- **Image exemplar**: Box around one dog → segments all dogs in the image
|
||||
- **Combined**: "striped cat" + exemplar box → segments all striped cats matching the example
|
||||
|
||||
See related background on [object detection](https://www.ultralytics.com/glossary/object-detection) and [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation).
|
||||
|
||||
### How Does SAM 3 Differ From SAM 2?
|
||||
|
||||
| Feature | SAM 2 | SAM 3 |
|
||||
| -------------------------- | ----------------------------- | ------------------------------------- |
|
||||
| **Task** | Single object per prompt | All instances of a concept |
|
||||
| **Prompt Types** | Points, boxes, masks | + Text phrases, image exemplars |
|
||||
| **Detection Capability** | Requires external detector | Built-in open-vocabulary detector |
|
||||
| **Recognition** | Geometry-based only | Text and visual recognition |
|
||||
| **Architecture** | Tracker only | Detector + Tracker with presence head |
|
||||
| **Zero-Shot Performance** | N/A (requires visual prompts) | 47.0 AP on LVIS, 2× better on SA-Co |
|
||||
| **Interactive Refinement** | Clicks only | Clicks + exemplar generalization |
|
||||
|
||||
SAM 3 maintains backward compatibility with [SAM 2](sam-2.md) visual prompting while adding concept-based capabilities.
|
||||
|
||||
### What datasets are used to train SAM 3?
|
||||
|
||||
SAM 3 is trained on the **Segment Anything with Concepts (SA-Co)** dataset:
|
||||
|
||||
**Training Data**:
|
||||
|
||||
- **5.2M images** with **4M unique noun phrases** (SA-Co/HQ) - high-quality human annotations
|
||||
- **52.5K videos** with **24.8K unique noun phrases** (SA-Co/VIDEO)
|
||||
- **1.4B synthetic masks** across **38M noun phrases** (SA-Co/SYN)
|
||||
- **15 external datasets** enriched with hard negatives (SA-Co/EXT)
|
||||
|
||||
**Benchmark Data**:
|
||||
|
||||
- **214K unique concepts** across **126K images/videos**
|
||||
- **50× more concepts** than existing benchmarks (e.g., LVIS has ~4K concepts)
|
||||
- Triple annotation on SA-Co/Gold for measuring human performance bounds
|
||||
|
||||
This massive scale and diversity enables SAM 3's superior zero-shot generalization across open-vocabulary concepts.
|
||||
|
||||
### How does SAM 3 compare to YOLO11 for segmentation?
|
||||
|
||||
SAM 3 and YOLO11 serve different use cases:
|
||||
|
||||
**SAM 3 Advantages**:
|
||||
|
||||
- **Open-vocabulary**: Segments any concept via text prompts without training
|
||||
- **Zero-shot**: Works on new categories immediately
|
||||
- **Interactive**: Exemplar-based refinement generalizes to similar objects
|
||||
- **Concept-based**: Automatically finds all instances of a category
|
||||
- **Accuracy**: 47.0 AP on LVIS zero-shot instance segmentation
|
||||
|
||||
**YOLO11 Advantages**:
|
||||
|
||||
- **Speed**: 10-15× faster inference (2-3ms vs 30ms per image)
|
||||
- **Efficiency**: 576× smaller models (5.9MB vs 3.4GB)
|
||||
- **Resource-friendly**: Runs on edge devices and mobile
|
||||
- **Real-time**: Optimized for production deployments
|
||||
|
||||
**Recommendation**:
|
||||
|
||||
- Use **SAM 3** for flexible, open-vocabulary segmentation where you need to find all instances of concepts described by text or examples
|
||||
- Use **YOLO11** for high-speed, production deployments where categories are known in advance
|
||||
- Use **SAM 2** for interactive single-object segmentation with geometric prompts
|
||||
|
||||
### Can SAM 3 handle complex language queries?
|
||||
|
||||
SAM 3 is designed for simple noun phrases (e.g., "red apple", "person wearing hat"). For complex queries requiring reasoning, combine SAM 3 with an MLLM as **SAM 3 Agent**:
|
||||
|
||||
**Simple queries (native SAM 3)**:
|
||||
|
||||
- "yellow school bus"
|
||||
- "striped cat"
|
||||
- "person wearing red hat"
|
||||
|
||||
**Complex queries (SAM 3 Agent with MLLM)**:
|
||||
|
||||
- "People sitting down but not holding a gift box"
|
||||
- "The dog closest to the camera without a collar"
|
||||
- "Red objects larger than the person's hand"
|
||||
|
||||
SAM 3 Agent achieves **76.0 gIoU** on ReasonSeg validation (vs 65.0 previous best, +16.9% improvement) by combining SAM 3's segmentation with MLLM reasoning capabilities.
|
||||
|
||||
### How accurate is SAM 3 compared to human performance?
|
||||
|
||||
On the SA-Co/Gold benchmark with triple human annotation:
|
||||
|
||||
- **Human lower bound**: 74.2 CGF1 (most conservative annotator)
|
||||
- **SAM 3 performance**: 65.0 CGF1
|
||||
- **Achievement**: **88%** of estimated human lower bound
|
||||
- **Human upper bound**: 81.4 CGF1 (most liberal annotator)
|
||||
|
||||
SAM 3 achieves strong performance approaching human-level accuracy on open-vocabulary concept segmentation, with the gap primarily on ambiguous or subjective concepts (e.g., "small window", "cozy room").
|
||||
304
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam.md
Normal file
304
algorithms/dms_yolo/code.embedded.bak/docs/en/models/sam.md
Normal file
@@ -0,0 +1,304 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore the revolutionary Segment Anything Model (SAM) for promptable image segmentation with zero-shot performance. Discover key features, datasets, and usage tips.
|
||||
keywords: Segment Anything, SAM, image segmentation, promptable segmentation, zero-shot performance, SA-1B dataset, advanced architecture, auto-annotation, Ultralytics, pretrained models, instance segmentation, computer vision, AI, machine learning
|
||||
---
|
||||
|
||||
# Segment Anything Model (SAM)
|
||||
|
||||
!!! note "SAM Evolution"
|
||||
|
||||
This is the original SAM model from Meta. For improved capabilities, see [SAM 2](sam-2.md) for video segmentation or [SAM 3](sam-3.md) for Promptable Concept Segmentation with text and image exemplar prompts.
|
||||
|
||||
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/inference-with-meta-sam-and-sam2-using-ultralytics-python-package.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="How to use Segment Anything In Colab"></a>
|
||||
|
||||
Welcome to the frontier of [image segmentation](https://www.ultralytics.com/glossary/image-segmentation) with the Segment Anything Model, or SAM. This revolutionary model has changed the game by introducing promptable image segmentation with real-time performance, setting new standards in the field.
|
||||
|
||||
## Introduction to SAM: The Segment Anything Model
|
||||
|
||||
The Segment Anything Model, or SAM, is a cutting-edge image segmentation model that allows for promptable segmentation, providing unparalleled versatility in image analysis tasks. SAM forms the heart of the Segment Anything initiative, a groundbreaking project that introduces a novel model, task, and dataset for image segmentation.
|
||||
|
||||
SAM's advanced design allows it to adapt to new image distributions and tasks without prior knowledge, a feature known as zero-shot transfer. Trained on the expansive [SA-1B dataset](https://ai.meta.com/datasets/segment-anything/), which contains more than 1 billion masks spread over 11 million carefully curated images, SAM has displayed impressive zero-shot performance, surpassing previous fully supervised results in many cases.
|
||||
|
||||
 **SA-1B Example images.** Dataset images overlaid masks from the newly introduced SA-1B dataset. SA-1B contains 11M diverse, high-resolution, licensed, and privacy protecting images and 1.1B high-quality segmentation masks. These masks were annotated fully automatically by SAM, and as verified by human ratings and numerous experiments, are of high quality and diversity. Images are grouped by number of masks per image for visualization (there are ∼100 masks per image on average).
|
||||
|
||||
## Key Features of the Segment Anything Model (SAM)
|
||||
|
||||
- **Promptable Segmentation Task:** SAM was designed with a promptable segmentation task in mind, allowing it to generate valid segmentation masks from any given prompt, such as spatial or text clues identifying an object.
|
||||
- **Advanced Architecture:** The Segment Anything Model employs a powerful image encoder, a prompt encoder, and a lightweight mask decoder. This unique architecture enables flexible prompting, real-time mask computation, and ambiguity awareness in segmentation tasks.
|
||||
- **The SA-1B Dataset:** Introduced by the Segment Anything project, the SA-1B dataset features over 1 billion masks on 11 million images. As the largest segmentation dataset to date, it provides SAM with a diverse and large-scale training data source.
|
||||
- **Zero-Shot Performance:** SAM displays outstanding zero-shot performance across various segmentation tasks, making it a ready-to-use tool for diverse applications with minimal need for [prompt engineering](https://www.ultralytics.com/glossary/prompt-engineering).
|
||||
|
||||
For an in-depth look at the Segment Anything Model and the SA-1B dataset, please visit the [Segment Anything GitHub](https://github.com/facebookresearch/segment-anything) and check out the research paper [Segment Anything](https://arxiv.org/abs/2304.02643).
|
||||
|
||||
## Available Models, Supported Tasks, and Operating Modes
|
||||
|
||||
This table presents the available models with their specific pretrained weights, the tasks they support, and their compatibility with different operating modes like [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md), indicated by ✅ emojis for supported modes and ❌ emojis for unsupported modes.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ---------- | ----------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| SAM base | [sam_b.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam_b.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
| SAM large | [sam_l.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/sam_l.pt) | [Instance Segmentation](../tasks/segment.md) | ✅ | ❌ | ❌ | ❌ |
|
||||
|
||||
## How to Use SAM: Versatility and Power in Image Segmentation
|
||||
|
||||
The Segment Anything Model can be employed for a multitude of downstream tasks that go beyond its training data. This includes edge detection, object proposal generation, [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), and preliminary text-to-mask prediction. With prompt engineering, SAM can swiftly adapt to new tasks and data distributions in a zero-shot manner, establishing it as a versatile and potent tool for all your image segmentation needs.
|
||||
|
||||
### SAM prediction example
|
||||
|
||||
!!! example "Segment with prompts"
|
||||
|
||||
Segment image with given prompts.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam_b.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Run inference with bboxes prompt
|
||||
results = model("ultralytics/assets/zidane.jpg", bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Run inference with single point
|
||||
results = model(points=[900, 370], labels=[1])
|
||||
|
||||
# Run inference with multiple points
|
||||
results = model(points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Run inference with multiple points prompt per object
|
||||
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
|
||||
|
||||
# Run inference with negative points prompt
|
||||
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
!!! example "Segment everything"
|
||||
|
||||
Segment the whole image.
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam_b.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Run inference
|
||||
model("path/to/image.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Run inference with a SAM model
|
||||
yolo predict model=sam_b.pt source=path/to/image.jpg
|
||||
```
|
||||
|
||||
- The logic here is to segment the whole image if you don't pass any prompts (bboxes/points/masks).
|
||||
|
||||
!!! example "SAMPredictor example"
|
||||
|
||||
This way you can set image once and run prompts inference multiple times without running image encoder multiple times.
|
||||
|
||||
=== "Prompt inference"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import Predictor as SAMPredictor
|
||||
|
||||
# Create SAMPredictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", imgsz=1024, model="mobile_sam.pt")
|
||||
predictor = SAMPredictor(overrides=overrides)
|
||||
|
||||
# Set image
|
||||
predictor.set_image("ultralytics/assets/zidane.jpg") # set with image file
|
||||
predictor.set_image(cv2.imread("ultralytics/assets/zidane.jpg")) # set with np.ndarray
|
||||
results = predictor(bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Run inference with single point prompt
|
||||
results = predictor(points=[900, 370], labels=[1])
|
||||
|
||||
# Run inference with multiple points prompt
|
||||
results = predictor(points=[[400, 370], [900, 370]], labels=[1, 1])
|
||||
|
||||
# Run inference with negative points prompt
|
||||
results = predictor(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
|
||||
# Reset image
|
||||
predictor.reset_image()
|
||||
```
|
||||
|
||||
Segment everything with additional args.
|
||||
|
||||
=== "Segment everything"
|
||||
|
||||
```python
|
||||
from ultralytics.models.sam import Predictor as SAMPredictor
|
||||
|
||||
# Create SAMPredictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", imgsz=1024, model="mobile_sam.pt")
|
||||
predictor = SAMPredictor(overrides=overrides)
|
||||
|
||||
# Segment with additional args
|
||||
results = predictor(source="ultralytics/assets/zidane.jpg", crop_n_layers=1, points_stride=64)
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
All the returned `results` in above examples are [Results](../modes/predict.md#working-with-results) object which allows access predicted masks and source image easily.
|
||||
|
||||
- More additional args for `Segment everything` see [`Predictor/generate` Reference](../reference/models/sam/predict.md).
|
||||
|
||||
## SAM Comparison vs YOLO
|
||||
|
||||
Here we compare Meta's SAM-b model with Ultralytics smallest segmentation model, [YOLO11n-seg](../tasks/segment.md):
|
||||
|
||||
| Model | Size<br><sup>(MB)</sup> | Parameters<br><sup>(M)</sup> | Speed (CPU)<br><sup>(ms/im)</sup> |
|
||||
| ---------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------- | --------------------------------- |
|
||||
| [Meta SAM-b](sam.md) | 375 | 93.7 | 49401 |
|
||||
| [MobileSAM](mobile-sam.md) | 40.7 | 10.1 | 25381 |
|
||||
| [FastSAM-s](fast-sam.md) with YOLOv8 [backbone](https://www.ultralytics.com/glossary/backbone) | 23.7 | 11.8 | 55.9 |
|
||||
| Ultralytics [YOLOv8n-seg](yolov8.md) | **6.7** (11.7x smaller) | **3.4** (11.4x less) | **24.5** (1061x faster) |
|
||||
| Ultralytics [YOLO11n-seg](yolo11.md) | **5.9** (13.2x smaller) | **2.9** (13.4x less) | **30.1** (864x faster) |
|
||||
|
||||
This comparison demonstrates the substantial differences in model sizes and speeds between SAM variants and YOLO segmentation models. While SAM provides unique automatic segmentation capabilities, YOLO models, particularly YOLOv8n-seg and YOLO11n-seg, are significantly smaller, faster, and more computationally efficient.
|
||||
|
||||
Tests run on a 2025 Apple M4 Pro with 24GB of RAM using `torch==2.6.0` and `ultralytics==8.3.90`. To reproduce this test:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, SAM, YOLO, FastSAM
|
||||
|
||||
# Profile SAM2-t, SAM2-b, SAM-b, MobileSAM
|
||||
for file in ["sam_b.pt", "sam2_b.pt", "sam2_t.pt", "mobile_sam.pt"]:
|
||||
model = SAM(file)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile FastSAM-s
|
||||
model = FastSAM("FastSAM-s.pt")
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
|
||||
# Profile YOLO models
|
||||
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt"]:
|
||||
model = YOLO(file_name)
|
||||
model.info()
|
||||
model(ASSETS)
|
||||
```
|
||||
|
||||
## Auto-Annotation: A Quick Path to Segmentation Datasets
|
||||
|
||||
Auto-annotation is a key feature of SAM, allowing users to generate a [segmentation dataset](../datasets/segment/index.md) using a pretrained detection model. This feature enables rapid and accurate annotation of a large number of images, bypassing the need for time-consuming manual labeling.
|
||||
|
||||
### Generate Your Segmentation Dataset Using a Detection Model
|
||||
|
||||
To auto-annotate your dataset with the Ultralytics framework, use the `auto_annotate` function as shown below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.data.annotator import auto_annotate
|
||||
|
||||
auto_annotate(data="path/to/images", det_model="yolo26x.pt", sam_model="sam_b.pt")
|
||||
```
|
||||
|
||||
{% include "macros/sam-auto-annotate.md" %}
|
||||
|
||||
The `auto_annotate` function takes the path to your images, with optional arguments for specifying the pretrained detection and SAM segmentation models, the device to run the models on, and the output directory for saving the annotated results.
|
||||
|
||||
Auto-annotation with pretrained models can dramatically cut down the time and effort required for creating high-quality segmentation datasets. This feature is especially beneficial for researchers and developers dealing with large image collections, as it allows them to focus on model development and evaluation rather than manual annotation.
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If you find SAM useful in your research or development work, please consider citing our paper:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{kirillov2023segment,
|
||||
title={Segment Anything},
|
||||
author={Alexander Kirillov and Eric Mintun and Nikhila Ravi and Hanzi Mao and Chloe Rolland and Laura Gustafson and Tete Xiao and Spencer Whitehead and Alexander C. Berg and Wan-Yen Lo and Piotr Dollár and Ross Girshick},
|
||||
year={2023},
|
||||
eprint={2304.02643},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
We would like to express our gratitude to Meta AI for creating and maintaining this valuable resource for the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is the Segment Anything Model (SAM) by Ultralytics?
|
||||
|
||||
The Segment Anything Model (SAM) by Ultralytics is a revolutionary image segmentation model designed for promptable segmentation tasks. It leverages advanced architecture, including image and prompt encoders combined with a lightweight mask decoder, to generate high-quality segmentation masks from various prompts such as spatial or text cues. Trained on the expansive [SA-1B dataset](https://ai.meta.com/datasets/segment-anything/), SAM excels in zero-shot performance, adapting to new image distributions and tasks without prior knowledge.
|
||||
|
||||
### How can I use the Segment Anything Model (SAM) for image segmentation?
|
||||
|
||||
You can use the Segment Anything Model (SAM) for image segmentation by running inference with various prompts such as bounding boxes or points. Here's an example using Python:
|
||||
|
||||
```python
|
||||
from ultralytics import SAM
|
||||
|
||||
# Load a model
|
||||
model = SAM("sam_b.pt")
|
||||
|
||||
# Segment with bounding box prompt
|
||||
model("ultralytics/assets/zidane.jpg", bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Segment with points prompt
|
||||
model("ultralytics/assets/zidane.jpg", points=[900, 370], labels=[1])
|
||||
|
||||
# Segment with multiple points prompt
|
||||
model("ultralytics/assets/zidane.jpg", points=[[400, 370], [900, 370]], labels=[[1, 1]])
|
||||
|
||||
# Segment with multiple points prompt per object
|
||||
model("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
|
||||
|
||||
# Segment with negative points prompt.
|
||||
model("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
|
||||
```
|
||||
|
||||
Alternatively, you can run inference with SAM in the command line interface (CLI):
|
||||
|
||||
```bash
|
||||
yolo predict model=sam_b.pt source=path/to/image.jpg
|
||||
```
|
||||
|
||||
For more detailed usage instructions, visit the [Segmentation section](#sam-prediction-example).
|
||||
|
||||
### How do SAM and YOLO models compare in terms of performance?
|
||||
|
||||
Compared to YOLO models, SAM variants like SAM-b, SAM2-t, MobileSAM, and FastSAM-s are typically larger and slower but offer unique zero-shot segmentation capabilities. For example, Ultralytics [YOLOv8n-seg](../tasks/segment.md) is **11.7x smaller** and **1069x faster** than Meta's original SAM-b model, highlighting YOLO's significant advantage in speed and efficiency. Similarly, the newer [YOLO11n-seg](../tasks/segment.md) provides even smaller size and maintains impressive inference speed. This makes YOLO models ideal for applications requiring rapid, lightweight, and computationally efficient segmentation, while SAM models excel in flexible, promptable, and zero-shot segmentation tasks.
|
||||
|
||||
### How can I auto-annotate my dataset using SAM?
|
||||
|
||||
Ultralytics' SAM offers an auto-annotation feature that allows generating segmentation datasets using a pretrained detection model. Here's an example in Python:
|
||||
|
||||
```python
|
||||
from ultralytics.data.annotator import auto_annotate
|
||||
|
||||
auto_annotate(data="path/to/images", det_model="yolov8x.pt", sam_model="sam_b.pt")
|
||||
```
|
||||
|
||||
This function takes the path to your images and optional arguments for pretrained detection and SAM segmentation models, along with device and output directory specifications. For a complete guide, see [Auto-Annotation](#auto-annotation-a-quick-path-to-segmentation-datasets).
|
||||
|
||||
### What datasets are used to train the Segment Anything Model (SAM)?
|
||||
|
||||
SAM is trained on the extensive [SA-1B dataset](https://ai.meta.com/datasets/segment-anything/) which comprises over 1 billion masks across 11 million images. SA-1B is the largest segmentation dataset to date, providing high-quality and diverse [training data](https://www.ultralytics.com/glossary/training-data), ensuring impressive zero-shot performance in varied segmentation tasks. For more details, visit the [Dataset section](#key-features-of-the-segment-anything-model-sam).
|
||||
173
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo-nas.md
Normal file
173
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo-nas.md
Normal file
@@ -0,0 +1,173 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLO-NAS by Deci AI - a state-of-the-art object detection model with quantization support. Explore features, pretrained models, and implementation examples.
|
||||
keywords: YOLO-NAS, Deci AI, object detection, deep learning, Neural Architecture Search, Ultralytics, Python API, YOLO model, SuperGradients, pretrained models, quantization, AutoNAC
|
||||
---
|
||||
|
||||
# YOLO-NAS
|
||||
|
||||
!!! note "Important Update"
|
||||
|
||||
Please note that [Deci](https://www.linkedin.com/company/deciai/), the original creators of YOLO-NAS, have been acquired by NVIDIA. As a result, these models are no longer actively maintained by Deci. Ultralytics continues to support the usage of these models, but no further updates from the original team are expected.
|
||||
|
||||
## Overview
|
||||
|
||||
Developed by Deci AI, YOLO-NAS is a groundbreaking object detection foundational model. It is the product of advanced [Neural Architecture Search](https://www.ultralytics.com/glossary/neural-architecture-search-nas) technology, meticulously designed to address the limitations of previous YOLO models. With significant improvements in quantization support and [accuracy](https://www.ultralytics.com/glossary/accuracy)-latency trade-offs, YOLO-NAS represents a major leap in object detection.
|
||||
|
||||
 **Overview of YOLO-NAS.** YOLO-NAS employs quantization-aware blocks and selective quantization for optimal performance. The model, when converted to its INT8 quantized version, experiences a minimal precision drop, a significant improvement over other models. These advancements culminate in a superior architecture with unprecedented object detection capabilities and outstanding performance.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Quantization-Friendly Basic Block:** YOLO-NAS introduces a new basic block that is friendly to quantization, addressing one of the significant limitations of previous YOLO models.
|
||||
- **Sophisticated Training and Quantization:** YOLO-NAS leverages advanced training schemes and post-training quantization to enhance performance.
|
||||
- **AutoNAC Optimization and Pre-training:** YOLO-NAS utilizes AutoNAC optimization and is pretrained on prominent datasets such as COCO, Objects365, and Roboflow 100. This pre-training makes it extremely suitable for downstream object detection tasks in production environments.
|
||||
|
||||
## Pretrained Models
|
||||
|
||||
Experience the power of next-generation object detection with the pretrained YOLO-NAS models provided by Ultralytics. These models are designed to deliver top-notch performance in terms of both speed and accuracy. Choose from a variety of options tailored to your specific needs:
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
| Model | mAP | Latency (ms) |
|
||||
| ---------------- | ----- | ------------ |
|
||||
| YOLO-NAS S | 47.5 | 3.21 |
|
||||
| YOLO-NAS M | 51.55 | 5.85 |
|
||||
| YOLO-NAS L | 52.22 | 7.87 |
|
||||
| YOLO-NAS S INT-8 | 47.03 | 2.36 |
|
||||
| YOLO-NAS M INT-8 | 51.0 | 3.78 |
|
||||
| YOLO-NAS L INT-8 | 52.1 | 4.78 |
|
||||
|
||||
Each model variant is designed to offer a balance between [Mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP) and latency, helping you optimize your object detection tasks for both performance and speed.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
Ultralytics has made YOLO-NAS models easy to integrate into your Python applications via our `ultralytics` python package. The package provides a user-friendly Python API to streamline the process.
|
||||
|
||||
The following examples show how to use YOLO-NAS models with the `ultralytics` package for inference and validation:
|
||||
|
||||
### Inference and Validation Examples
|
||||
|
||||
In this example we validate YOLO-NAS-s on the COCO8 dataset.
|
||||
|
||||
!!! example
|
||||
|
||||
This example provides simple inference and validation code for YOLO-NAS. For handling inference results see [Predict](../modes/predict.md) mode. For using YOLO-NAS with additional modes see [Val](../modes/val.md) and [Export](../modes/export.md). YOLO-NAS on the `ultralytics` package does not support training.
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models files can be passed to the `NAS()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import NAS
|
||||
|
||||
# Load a COCO-pretrained YOLO-NAS-s model
|
||||
model = NAS("yolo_nas_s.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Validate the model on the COCO8 example dataset
|
||||
results = model.val(data="coco8.yaml")
|
||||
|
||||
# Run inference with the YOLO-NAS-s model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO-NAS-s model and validate it's performance on the COCO8 example dataset
|
||||
yolo val model=yolo_nas_s.pt data=coco8.yaml
|
||||
|
||||
# Load a COCO-pretrained YOLO-NAS-s model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolo_nas_s.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
We offer three variants of the YOLO-NAS models: Small (s), Medium (m), and Large (l). Each variant is designed to cater to different computational and performance needs:
|
||||
|
||||
- **YOLO-NAS-s**: Optimized for environments where computational resources are limited but efficiency is key.
|
||||
- **YOLO-NAS-m**: Offers a balanced approach, suitable for general-purpose [object detection](https://www.ultralytics.com/glossary/object-detection) with higher accuracy.
|
||||
- **YOLO-NAS-l**: Tailored for scenarios requiring the highest accuracy, where computational resources are less of a constraint.
|
||||
|
||||
Below is a detailed overview of each model, including links to their pretrained weights, the tasks they support, and their compatibility with different operating modes.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| ---------- | --------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLO-NAS-s | [yolo_nas_s.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_s.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ❌ | ✅ |
|
||||
| YOLO-NAS-m | [yolo_nas_m.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_m.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ❌ | ✅ |
|
||||
| YOLO-NAS-l | [yolo_nas_l.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_l.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ❌ | ✅ |
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If you employ YOLO-NAS in your research or development work, please cite SuperGradients:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{supergradients,
|
||||
doi = {10.5281/ZENODO.7789328},
|
||||
url = {https://zenodo.org/records/7789328},
|
||||
author = {Aharon, Shay and {Louis-Dupont} and {Ofri Masad} and Yurkova, Kate and {Lotem Fridman} and {Lkdci} and Khvedchenya, Eugene and Rubin, Ran and Bagrov, Natan and Tymchenko, Borys and Keren, Tomer and Zhilko, Alexander and {Eran-Deci}},
|
||||
title = {Super-Gradients},
|
||||
publisher = {GitHub},
|
||||
journal = {GitHub repository},
|
||||
year = {2021},
|
||||
}
|
||||
```
|
||||
|
||||
We express our gratitude to Deci AI's [SuperGradients](https://github.com/Deci-AI/super-gradients/) team for their efforts in creating and maintaining this valuable resource for the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) community. We believe YOLO-NAS, with its innovative architecture and superior object detection capabilities, will become a critical tool for developers and researchers alike.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is YOLO-NAS and how does it improve over previous YOLO models?
|
||||
|
||||
YOLO-NAS, developed by Deci AI, is a state-of-the-art object detection model leveraging advanced Neural Architecture Search (NAS) technology. It addresses the limitations of previous YOLO models by introducing features like quantization-friendly basic blocks and sophisticated training schemes. This results in significant improvements in performance, particularly in environments with limited computational resources. YOLO-NAS also supports quantization, maintaining high accuracy even when converted to its INT8 version, enhancing its suitability for production environments. For more details, see the [Overview](#overview) section.
|
||||
|
||||
### How can I integrate YOLO-NAS models into my Python application?
|
||||
|
||||
You can easily integrate YOLO-NAS models into your Python application using the `ultralytics` package. Here's a simple example of how to load a pretrained YOLO-NAS model and perform inference:
|
||||
|
||||
```python
|
||||
from ultralytics import NAS
|
||||
|
||||
# Load a COCO-pretrained YOLO-NAS-s model
|
||||
model = NAS("yolo_nas_s.pt")
|
||||
|
||||
# Validate the model on the COCO8 example dataset
|
||||
results = model.val(data="coco8.yaml")
|
||||
|
||||
# Run inference with the YOLO-NAS-s model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
For more information, refer to the [Inference and Validation Examples](#inference-and-validation-examples).
|
||||
|
||||
### What are the key features of YOLO-NAS and why should I consider using it?
|
||||
|
||||
YOLO-NAS introduces several key features that make it a superior choice for object detection tasks:
|
||||
|
||||
- **Quantization-Friendly Basic Block:** Enhanced architecture that improves model performance with minimal [precision](https://www.ultralytics.com/glossary/precision) drop post quantization.
|
||||
- **Sophisticated Training and Quantization:** Employs advanced training schemes and post-training quantization techniques.
|
||||
- **AutoNAC Optimization and Pre-training:** Utilizes AutoNAC optimization and is pretrained on prominent datasets like COCO, Objects365, and Roboflow 100.
|
||||
|
||||
These features contribute to its high accuracy, efficient performance, and suitability for deployment in production environments. Learn more in the [Key Features](#key-features) section.
|
||||
|
||||
### Which tasks and modes are supported by YOLO-NAS models?
|
||||
|
||||
YOLO-NAS models support various object detection tasks and modes such as inference, validation, and export. They do not support training. The supported models include YOLO-NAS-s, YOLO-NAS-m, and YOLO-NAS-l, each tailored to different computational capacities and performance needs. For a detailed overview, refer to the [Supported Tasks and Modes](#supported-tasks-and-modes) section.
|
||||
|
||||
### Are there pretrained YOLO-NAS models available and how do I access them?
|
||||
|
||||
Yes, Ultralytics provides pretrained YOLO-NAS models that you can access directly. These models are pretrained on datasets like COCO, ensuring high performance in terms of both speed and accuracy. You can download these models using the links provided in the [Pretrained Models](#pretrained-models) section. Here are some examples:
|
||||
|
||||
- [YOLO-NAS-s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_s.pt)
|
||||
- [YOLO-NAS-m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_m.pt)
|
||||
- [YOLO-NAS-l](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo_nas_l.pt)
|
||||
@@ -0,0 +1,480 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore the YOLO-World Model for efficient, real-time open-vocabulary object detection using Ultralytics YOLOv8 advancements. Achieve top performance with minimal computation.
|
||||
keywords: YOLO-World, Ultralytics, open-vocabulary detection, YOLOv8, real-time object detection, machine learning, computer vision, AI, deep learning, model training
|
||||
---
|
||||
|
||||
# YOLO-World Model
|
||||
|
||||
The YOLO-World Model introduces an advanced, real-time [Ultralytics](https://www.ultralytics.com/) [YOLOv8](yolov8.md)-based approach for Open-Vocabulary Detection tasks. This innovation enables the detection of any object within an image based on descriptive texts. By significantly lowering computational demands while preserving competitive performance, YOLO-World emerges as a versatile tool for numerous vision-based applications.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/cfTKj96TjSE"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> YOLO World training workflow on custom dataset
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
YOLO-World tackles the challenges faced by traditional Open-Vocabulary detection models, which often rely on cumbersome [Transformer](https://www.ultralytics.com/glossary/transformer) models requiring extensive computational resources. These models' dependence on pre-defined object categories also restricts their utility in dynamic scenarios. YOLO-World revitalizes the YOLOv8 framework with open-vocabulary detection capabilities, employing vision-[language modeling](https://www.ultralytics.com/glossary/language-modeling) and pre-training on expansive datasets to excel at identifying a broad array of objects in zero-shot scenarios with unmatched efficiency.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Real-time Solution:** Harnessing the computational speed of CNNs, YOLO-World delivers a swift open-vocabulary detection solution, catering to industries in need of immediate results.
|
||||
|
||||
2. **Efficiency and Performance:** YOLO-World slashes computational and resource requirements without sacrificing performance, offering a robust alternative to models like SAM but at a fraction of the computational cost, enabling real-time applications.
|
||||
|
||||
3. **Inference with Offline Vocabulary:** YOLO-World introduces a "prompt-then-detect" strategy, employing an offline vocabulary to enhance efficiency further. This approach enables the use of custom prompts computed apriori, including captions or categories, to be encoded and stored as offline vocabulary embeddings, streamlining the detection process.
|
||||
|
||||
4. **Powered by YOLOv8:** Built upon [Ultralytics YOLOv8](yolov8.md), YOLO-World leverages the latest advancements in real-time object detection to facilitate open-vocabulary detection with unparalleled accuracy and speed.
|
||||
|
||||
5. **Benchmark Excellence:** YOLO-World outperforms existing open-vocabulary detectors, including MDETR and GLIP series, in terms of speed and efficiency on standard benchmarks, showcasing YOLOv8's superior capability on a single NVIDIA V100 GPU.
|
||||
|
||||
6. **Versatile Applications:** YOLO-World's innovative approach unlocks new possibilities for a multitude of vision tasks, delivering speed improvements by orders of magnitude over existing methods.
|
||||
|
||||
## Available Models, Supported Tasks, and Operating Modes
|
||||
|
||||
This section details the models available with their specific pretrained weights, the tasks they support, and their compatibility with various operating modes such as [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md), denoted by ✅ for supported modes and ❌ for unsupported modes.
|
||||
|
||||
!!! note
|
||||
|
||||
All the YOLOv8-World weights have been directly migrated from the official [YOLO-World](https://github.com/AILab-CVC/YOLO-World) repository, highlighting their excellent contributions.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv8s-world | [yolov8s-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8s-worldv2 | [yolov8s-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8m-world | [yolov8m-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8m-worldv2 | [yolov8m-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8l-world | [yolov8l-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8l-worldv2 | [yolov8l-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8x-world | [yolov8x-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8x-worldv2 | [yolov8x-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
## Zero-shot Transfer on COCO Dataset
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
| Model Type | mAP | mAP50 | mAP75 |
|
||||
| --------------- | ---- | ----- | ----- |
|
||||
| yolov8s-world | 37.4 | 52.0 | 40.6 |
|
||||
| yolov8s-worldv2 | 37.7 | 52.2 | 41.0 |
|
||||
| yolov8m-world | 42.0 | 57.0 | 45.6 |
|
||||
| yolov8m-worldv2 | 43.0 | 58.4 | 46.8 |
|
||||
| yolov8l-world | 45.7 | 61.3 | 49.8 |
|
||||
| yolov8l-worldv2 | 45.8 | 61.3 | 49.8 |
|
||||
| yolov8x-world | 47.0 | 63.0 | 51.2 |
|
||||
| yolov8x-worldv2 | 47.1 | 62.8 | 51.4 |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
The YOLO-World models are easy to integrate into your Python applications. Ultralytics provides user-friendly [Python API](../usage/python.md) and [CLI commands](../usage/cli.md) to streamline development.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/sWEm3dIGKU8"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> YOLO-World Model Usage examples with Ultralytics | Open Vocab, Prompt-Free & others 🚀
|
||||
</p>
|
||||
|
||||
### Train Usage
|
||||
|
||||
!!! tip
|
||||
|
||||
We strongly recommend to use `yolov8-worldv2` model for custom training, because it supports deterministic training and also easy to export other formats i.e onnx/tensorrt.
|
||||
|
||||
[Object detection](https://www.ultralytics.com/glossary/object-detection) is straightforward with the `train` method, as illustrated below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLOWorld()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
|
||||
# Load a pretrained YOLOv8s-worldv2 model
|
||||
model = YOLOWorld("yolov8s-worldv2.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLO-World model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a pretrained YOLOv8s-worldv2 model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov8s-worldv2.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
### Predict Usage
|
||||
|
||||
Object detection is straightforward with the `predict` method, as illustrated below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
|
||||
# Initialize a YOLO-World model
|
||||
model = YOLOWorld("yolov8s-world.pt") # or select yolov8m/l-world.pt for different sizes
|
||||
|
||||
# Execute inference with the YOLOv8s-world model on the specified image
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Perform object detection using a YOLO-World model
|
||||
yolo predict model=yolov8s-world.pt source=path/to/image.jpg imgsz=640
|
||||
```
|
||||
|
||||
This snippet demonstrates the simplicity of loading a pretrained model and running a prediction on an image.
|
||||
|
||||
### Val Usage
|
||||
|
||||
Model validation on a dataset is streamlined as follows:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Create a YOLO-World model
|
||||
model = YOLO("yolov8s-world.pt") # or select yolov8m/l-world.pt for different sizes
|
||||
|
||||
# Conduct model validation on the COCO8 example dataset
|
||||
metrics = model.val(data="coco8.yaml")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Validate a YOLO-World model on the COCO8 dataset with a specified image size
|
||||
yolo val model=yolov8s-world.pt data=coco8.yaml imgsz=640
|
||||
```
|
||||
|
||||
### Track Usage
|
||||
|
||||
[Object tracking](https://www.ultralytics.com/glossary/object-tracking) with YOLO-World model on a video/images is streamlined as follows:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Create a YOLO-World model
|
||||
model = YOLO("yolov8s-world.pt") # or select yolov8m/l-world.pt for different sizes
|
||||
|
||||
# Track with a YOLO-World model on a video
|
||||
results = model.track(source="path/to/video.mp4")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Track with a YOLO-World model on the video with a specified image size
|
||||
yolo track model=yolov8s-world.pt imgsz=640 source="path/to/video.mp4"
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The YOLO-World models provided by Ultralytics come pre-configured with [COCO dataset](../datasets/detect/coco.md) categories as part of their offline vocabulary, enhancing efficiency for immediate application. This integration allows the YOLOv8-World models to directly recognize and predict the 80 standard categories defined in the COCO dataset without requiring additional setup or customization.
|
||||
|
||||
### Set prompts
|
||||
|
||||
|
||||
|
||||
The YOLO-World framework allows for the dynamic specification of classes through custom prompts, empowering users to tailor the model to their specific needs **without retraining**. This feature is particularly useful for adapting the model to new domains or specific tasks that were not originally part of the [training data](https://www.ultralytics.com/glossary/training-data). By setting custom prompts, users can essentially guide the model's focus towards objects of interest, enhancing the relevance and [accuracy](https://www.ultralytics.com/glossary/accuracy) of the detection results.
|
||||
|
||||
For instance, if your application only requires detecting 'person' and 'bus' objects, you can specify these classes directly:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Custom Inference Prompts"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Initialize a YOLO-World model
|
||||
model = YOLO("yolov8s-world.pt") # or choose yolov8m/l-world.pt
|
||||
|
||||
# Define custom classes
|
||||
model.set_classes(["person", "bus"])
|
||||
|
||||
# Execute prediction for specified categories on an image
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
!!! note "Background Class"
|
||||
|
||||
Some users have found that appending an empty string `""` as a background class can improve detection performance in certain scenarios. This behavior appears to be scenario-dependent and the exact mechanism is not fully understood:
|
||||
|
||||
```python
|
||||
model.set_classes(["person", "bus", ""])
|
||||
```
|
||||
|
||||
You can also save a model after setting custom classes. By doing this you create a version of the YOLO-World model that is specialized for your specific use case. This process embeds your custom class definitions directly into the model file, making the model ready to use with your specified classes without further adjustments. Follow these steps to save and load your custom YOLO-World model:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Persisting Models with Custom Vocabulary"
|
||||
|
||||
First load a YOLO-World model, set custom classes for it and save it:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Initialize a YOLO-World model
|
||||
model = YOLO("yolov8s-world.pt") # or select yolov8m/l-world.pt
|
||||
|
||||
# Define custom classes
|
||||
model.set_classes(["person", "bus"])
|
||||
|
||||
# Save the model with the defined offline vocabulary
|
||||
model.save("custom_yolov8s.pt")
|
||||
```
|
||||
|
||||
After saving, the custom_yolov8s.pt model behaves like any other pretrained YOLOv8 model but with a key difference: it is now optimized to detect only the classes you have defined. This customization can significantly improve detection performance and efficiency for your specific application scenarios.
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load your custom model
|
||||
model = YOLO("custom_yolov8s.pt")
|
||||
|
||||
# Run inference to detect your custom classes
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
### Benefits of Saving with Custom Vocabulary
|
||||
|
||||
- **Efficiency**: Streamlines the detection process by focusing on relevant objects, reducing computational overhead and speeding up inference.
|
||||
- **Flexibility**: Allows for easy adaptation of the model to new or niche detection tasks without the need for extensive retraining or data collection.
|
||||
- **Simplicity**: Simplifies deployment by eliminating the need to repeatedly specify custom classes at runtime, making the model directly usable with its embedded vocabulary.
|
||||
- **Performance**: Enhances detection accuracy for specified classes by focusing the model's attention and resources on recognizing the defined objects.
|
||||
|
||||
This approach provides a powerful means of customizing state-of-the-art [object detection](../tasks/detect.md) models for specific tasks, making advanced AI more accessible and applicable to a broader range of practical applications.
|
||||
|
||||
## Reproduce official results from scratch(Experimental)
|
||||
|
||||
### Prepare datasets
|
||||
|
||||
- Train data
|
||||
|
||||
| Dataset | Type | Samples | Boxes | Annotation Files |
|
||||
| ----------------------------------------------------------------- | ----------------------------------------------------------- | ------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| [Objects365v1](https://opendatalab.com/OpenDataLab/Objects365_v1) | Detection | 609k | 9621k | [objects365_train.json](https://opendatalab.com/OpenDataLab/Objects365_v1) |
|
||||
| [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html) | [Grounding](https://www.ultralytics.com/glossary/grounding) | 621k | 3681k | [final_mixed_train_no_coco.json](https://huggingface.co/GLIPModel/GLIP/blob/main/mdetr_annotations/final_mixed_train_no_coco.json) |
|
||||
| [Flickr30k](https://shannon.cs.illinois.edu/DenotationGraph/) | Grounding | 149k | 641k | [final_flickr_separateGT_train.json](https://huggingface.co/GLIPModel/GLIP/blob/main/mdetr_annotations/final_flickr_separateGT_train.json) |
|
||||
|
||||
- Val data
|
||||
|
||||
| Dataset | Type | Annotation Files |
|
||||
| ------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------ |
|
||||
| [LVIS minival](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/lvis.yaml) | Detection | [minival.txt](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/lvis.yaml) |
|
||||
|
||||
### Launch training from scratch
|
||||
|
||||
!!! note
|
||||
|
||||
`WorldTrainerFromScratch` is highly customized to allow training yolo-world models on both detection datasets and grounding datasets simultaneously. More details please checkout [ultralytics.model.yolo.world.train_world.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/yolo/world/train_world.py).
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
|
||||
|
||||
# Option 1: Use Python dictionary
|
||||
data = dict(
|
||||
train=dict(
|
||||
yolo_data=["Objects365.yaml"],
|
||||
grounding_data=[
|
||||
dict(
|
||||
img_path="flickr30k/images",
|
||||
json_file="flickr30k/final_flickr_separateGT_train.json",
|
||||
),
|
||||
dict(
|
||||
img_path="GQA/images",
|
||||
json_file="GQA/final_mixed_train_no_coco.json",
|
||||
),
|
||||
],
|
||||
),
|
||||
val=dict(yolo_data=["lvis.yaml"]),
|
||||
)
|
||||
|
||||
# Option 2: Use YAML file (yolo_world_data.yaml)
|
||||
# train:
|
||||
# yolo_data:
|
||||
# - Objects365.yaml
|
||||
# grounding_data:
|
||||
# - img_path: flickr/full_images/
|
||||
# json_file: flickr/annotations/final_flickr_separateGT_train_segm.json
|
||||
# - img_path: mixed_grounding/gqa/images
|
||||
# json_file: mixed_grounding/annotations/final_mixed_train_no_coco_segm.json
|
||||
# val:
|
||||
# yolo_data:
|
||||
# - lvis.yaml
|
||||
|
||||
model = YOLOWorld("yolov8s-worldv2.yaml")
|
||||
model.train(
|
||||
data=data, # or data="yolo_world_data.yaml" if using YAML file
|
||||
batch=128,
|
||||
epochs=100,
|
||||
trainer=WorldTrainerFromScratch,
|
||||
)
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We extend our gratitude to the [Tencent AILab Computer Vision Center](https://www.tencent.com/) for their pioneering work in real-time open-vocabulary object detection with YOLO-World:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{cheng2024yolow,
|
||||
title={YOLO-World: Real-Time Open-Vocabulary Object Detection},
|
||||
author={Cheng, Tianheng and Song, Lin and Ge, Yixiao and Liu, Wenyu and Wang, Xinggang and Shan, Ying},
|
||||
journal={arXiv preprint arXiv:2401.17270},
|
||||
year={2024}
|
||||
}
|
||||
```
|
||||
|
||||
For further reading, the original YOLO-World paper is available on [arXiv](https://arxiv.org/pdf/2401.17270v2). The project's source code and additional resources can be accessed via their [GitHub repository](https://github.com/AILab-CVC/YOLO-World). We appreciate their commitment to advancing the field and sharing their valuable insights with the community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is the YOLO-World model and how does it work?
|
||||
|
||||
The YOLO-World model is an advanced, real-time object detection approach based on the [Ultralytics YOLOv8](yolov8.md) framework. It excels in Open-Vocabulary Detection tasks by identifying objects within an image based on descriptive texts. Using vision-language modeling and pre-training on large datasets, YOLO-World achieves high efficiency and performance with significantly reduced computational demands, making it ideal for real-time applications across various industries.
|
||||
|
||||
### How does YOLO-World handle inference with custom prompts?
|
||||
|
||||
YOLO-World supports a "prompt-then-detect" strategy, which utilizes an offline vocabulary to enhance efficiency. Custom prompts like captions or specific object categories are pre-encoded and stored as offline vocabulary [embeddings](https://www.ultralytics.com/glossary/embeddings). This approach streamlines the detection process without the need for retraining. You can dynamically set these prompts within the model to tailor it to specific detection tasks, as shown below:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
|
||||
# Initialize a YOLO-World model
|
||||
model = YOLOWorld("yolov8s-world.pt")
|
||||
|
||||
# Define custom classes
|
||||
model.set_classes(["person", "bus"])
|
||||
|
||||
# Execute prediction on an image
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
### Why should I choose YOLO-World over traditional Open-Vocabulary detection models?
|
||||
|
||||
YOLO-World provides several advantages over traditional Open-Vocabulary detection models:
|
||||
|
||||
- **Real-Time Performance:** It leverages the computational speed of CNNs to offer quick, efficient detection.
|
||||
- **Efficiency and Low Resource Requirement:** YOLO-World maintains high performance while significantly reducing computational and resource demands.
|
||||
- **Customizable Prompts:** The model supports dynamic prompt setting, allowing users to specify custom detection classes without retraining.
|
||||
- **Benchmark Excellence:** It outperforms other open-vocabulary detectors like MDETR and GLIP in both speed and efficiency on standard benchmarks.
|
||||
|
||||
### How do I train a YOLO-World model on my dataset?
|
||||
|
||||
Training a YOLO-World model on your dataset is straightforward through the provided Python API or CLI commands. Here's how to start training using Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
|
||||
# Load a pretrained YOLOv8s-worldv2 model
|
||||
model = YOLOWorld("yolov8s-worldv2.pt")
|
||||
|
||||
# Train the model on the COCO8 dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
Or using CLI:
|
||||
|
||||
```bash
|
||||
yolo train model=yolov8s-worldv2.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
### What are the available pretrained YOLO-World models and their supported tasks?
|
||||
|
||||
Ultralytics offers multiple pretrained YOLO-World models supporting various tasks and operating modes:
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv8s-world | [yolov8s-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8s-worldv2 | [yolov8s-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8m-world | [yolov8m-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8m-worldv2 | [yolov8m-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8l-world | [yolov8l-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8l-worldv2 | [yolov8l-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8x-world | [yolov8x-world.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-world.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ❌ |
|
||||
| YOLOv8x-worldv2 | [yolov8x-worldv2.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-worldv2.pt) | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
### How do I reproduce the official results of YOLO-World from scratch?
|
||||
|
||||
To reproduce the official results from scratch, you need to prepare the datasets and launch the training using the provided code. The training procedure involves creating a data dictionary and running the `train` method with a custom trainer:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLOWorld
|
||||
from ultralytics.models.yolo.world.train_world import WorldTrainerFromScratch
|
||||
|
||||
data = {
|
||||
"train": {
|
||||
"yolo_data": ["Objects365.yaml"],
|
||||
"grounding_data": [
|
||||
{
|
||||
"img_path": "flickr30k/images",
|
||||
"json_file": "flickr30k/final_flickr_separateGT_train.json",
|
||||
},
|
||||
{
|
||||
"img_path": "GQA/images",
|
||||
"json_file": "GQA/final_mixed_train_no_coco.json",
|
||||
},
|
||||
],
|
||||
},
|
||||
"val": {"yolo_data": ["lvis.yaml"]},
|
||||
}
|
||||
|
||||
model = YOLOWorld("yolov8s-worldv2.yaml")
|
||||
model.train(data=data, batch=128, epochs=100, trainer=WorldTrainerFromScratch)
|
||||
```
|
||||
249
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo11.md
Normal file
249
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo11.md
Normal file
@@ -0,0 +1,249 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLO11, an advancement in real-time object detection, offering excellent accuracy and efficiency for diverse computer vision tasks.
|
||||
keywords: YOLO11, real-time object detection, YOLO series, Ultralytics, computer vision, AI, machine learning, deep learning
|
||||
---
|
||||
|
||||
# Ultralytics YOLO11
|
||||
|
||||
## Overview
|
||||
|
||||
YOLO11 was released by Ultralytics on September 10, 2024, delivering excellent [accuracy](https://www.ultralytics.com/glossary/accuracy), speed, and efficiency. Building upon the impressive advancements of previous YOLO versions, YOLO11 introduces significant improvements in architecture and training methods, making it a versatile choice for a wide range of [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks. For the latest Ultralytics model with end-to-end NMS-free inference and optimized edge deployment, see [YOLO26](yolo26.md).
|
||||
|
||||
|
||||
|
||||
<div style="text-align: center">
|
||||
<audio controls preload="none" style="width:100%; max-width:1920px;">
|
||||
<source src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/Ultralytics-YOLO11-podcast-notebook.LM.mp3" type="audio/mpeg">
|
||||
Your browser does not support the audio element.
|
||||
</audio>
|
||||
<p>Ultralytics YOLO11 🚀 Podcast generated by <a href="https://notebooklm.google/">NotebookLM</a></p>
|
||||
</div>
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/-JXwa-WlkU8"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Use Ultralytics YOLO11 for Object Detection and Tracking | How to Benchmark | YOLO11 RELEASED🚀
|
||||
</p>
|
||||
|
||||
!!! tip "Try on Ultralytics Platform"
|
||||
|
||||
Explore and run YOLO11 models directly on [Ultralytics Platform](https://platform.ultralytics.com/ultralytics/yolo11).
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Enhanced Feature Extraction:** YOLO11 employs an improved [backbone](https://www.ultralytics.com/glossary/backbone) and neck architecture, which enhances [feature extraction](https://www.ultralytics.com/glossary/feature-extraction) capabilities for more precise object detection and complex task performance.
|
||||
- **Optimized for Efficiency and Speed:** YOLO11 introduces refined architectural designs and optimized training pipelines, delivering faster processing speeds and maintaining an optimal balance between accuracy and performance.
|
||||
- **Greater Accuracy with Fewer Parameters:** With advancements in model design, YOLO11m achieves a higher [mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP) on the COCO dataset while using 22% fewer parameters than YOLOv8m, making it computationally efficient without compromising accuracy.
|
||||
- **Adaptability Across Environments:** YOLO11 can be seamlessly deployed across various environments, including edge devices, cloud platforms, and systems supporting NVIDIA GPUs, ensuring maximum flexibility.
|
||||
- **Broad Range of Supported Tasks:** Whether it's object detection, instance segmentation, image classification, pose estimation, or oriented object detection (OBB), YOLO11 is designed to cater to a diverse set of computer vision challenges.
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
YOLO11 builds upon the versatile model range established by earlier Ultralytics YOLO releases, offering enhanced support across various computer vision tasks:
|
||||
|
||||
| Model | Filenames | Task | Inference | Validation | Training | Export |
|
||||
| ----------- | ----------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLO11 | `yolo11n.pt` `yolo11s.pt` `yolo11m.pt` `yolo11l.pt` `yolo11x.pt` | [Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO11-seg | `yolo11n-seg.pt` `yolo11s-seg.pt` `yolo11m-seg.pt` `yolo11l-seg.pt` `yolo11x-seg.pt` | [Instance Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO11-pose | `yolo11n-pose.pt` `yolo11s-pose.pt` `yolo11m-pose.pt` `yolo11l-pose.pt` `yolo11x-pose.pt` | [Pose/Keypoints](../tasks/pose.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO11-obb | `yolo11n-obb.pt` `yolo11s-obb.pt` `yolo11m-obb.pt` `yolo11l-obb.pt` `yolo11x-obb.pt` | [Oriented Detection](../tasks/obb.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO11-cls | `yolo11n-cls.pt` `yolo11s-cls.pt` `yolo11m-cls.pt` `yolo11l-cls.pt` `yolo11x-cls.pt` | [Classification](../tasks/classify.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides an overview of the YOLO11 model variants, showcasing their applicability in specific tasks and compatibility with operational modes such as Inference, Validation, Training, and Export. This flexibility makes YOLO11 suitable for a wide range of applications in computer vision, from real-time detection to complex segmentation tasks.
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLO11"]'></canvas>
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
See [Detection Docs](../tasks/detect.md) for usage examples with these models trained on [COCO](../datasets/detect/coco.md), which include 80 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT10<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| ------------------------------------------------------------------------------------ | --------------------- | -------------------- | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLO11n](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n.pt) | 640 | 39.5 | 56.1 ± 0.8 | 1.5 ± 0.0 | 2.6 | 6.5 |
|
||||
| [YOLO11s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s.pt) | 640 | 47.0 | 90.0 ± 1.2 | 2.5 ± 0.0 | 9.4 | 21.5 |
|
||||
| [YOLO11m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11m.pt) | 640 | 51.5 | 183.2 ± 2.0 | 4.7 ± 0.1 | 20.1 | 68.0 |
|
||||
| [YOLO11l](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11l.pt) | 640 | 53.4 | 238.6 ± 1.4 | 6.2 ± 0.1 | 25.3 | 86.9 |
|
||||
| [YOLO11x](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11x.pt) | 640 | 54.7 | 462.8 ± 6.7 | 11.3 ± 0.2 | 56.9 | 194.9 |
|
||||
|
||||
=== "Segmentation (COCO)"
|
||||
|
||||
See [Segmentation Docs](../tasks/segment.md) for usage examples with these models trained on [COCO](../datasets/segment/coco.md), which include 80 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>box<br>50-95</sup> | mAP<sup>mask<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT10<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| -------------------------------------------------------------------------------------------- | --------------------- | -------------------- | --------------------- | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLO11n-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n-seg.pt) | 640 | 38.9 | 32.0 | 65.9 ± 1.1 | 1.8 ± 0.0 | 2.9 | 9.7 |
|
||||
| [YOLO11s-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s-seg.pt) | 640 | 46.6 | 37.8 | 117.6 ± 4.9 | 2.9 ± 0.0 | 10.1 | 33.0 |
|
||||
| [YOLO11m-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11m-seg.pt) | 640 | 51.5 | 41.5 | 281.6 ± 1.2 | 6.3 ± 0.1 | 22.4 | 113.2 |
|
||||
| [YOLO11l-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11l-seg.pt) | 640 | 53.4 | 42.9 | 344.2 ± 3.2 | 7.8 ± 0.2 | 27.6 | 132.2 |
|
||||
| [YOLO11x-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11x-seg.pt) | 640 | 54.7 | 43.8 | 664.5 ± 3.2 | 15.8 ± 0.7 | 62.1 | 296.4 |
|
||||
|
||||
=== "Classification (ImageNet)"
|
||||
|
||||
See [Classification Docs](../tasks/classify.md) for usage examples with these models trained on [ImageNet](../datasets/classify/imagenet.md), which include 1000 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | acc<br><sup>top1</sup> | acc<br><sup>top5</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT10<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B) at 224</sup> |
|
||||
| -------------------------------------------------------------------------------------------- | --------------------- | ---------------- | ---------------- | ------------------------------ | ----------------------------------- | ------------------ | ------------------------ |
|
||||
| [YOLO11n-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n-cls.pt) | 224 | 70.0 | 89.4 | 5.0 ± 0.3 | 1.1 ± 0.0 | 2.8 | 0.5 |
|
||||
| [YOLO11s-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s-cls.pt) | 224 | 75.4 | 92.7 | 7.9 ± 0.2 | 1.3 ± 0.0 | 6.7 | 1.6 |
|
||||
| [YOLO11m-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11m-cls.pt) | 224 | 77.3 | 93.9 | 17.2 ± 0.4 | 2.0 ± 0.0 | 11.6 | 4.9 |
|
||||
| [YOLO11l-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11l-cls.pt) | 224 | 78.3 | 94.3 | 23.2 ± 0.3 | 2.8 ± 0.0 | 14.1 | 6.2 |
|
||||
| [YOLO11x-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11x-cls.pt) | 224 | 79.5 | 94.9 | 41.4 ± 0.9 | 3.8 ± 0.0 | 29.6 | 13.6 |
|
||||
|
||||
=== "Pose (COCO)"
|
||||
|
||||
See [Pose Estimation Docs](../tasks/pose.md) for usage examples with these models trained on [COCO](../datasets/pose/coco.md), which include 1 pretrained class, 'person'.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>pose<br>50-95</sup> | mAP<sup>pose<br>50</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT10<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| ---------------------------------------------------------------------------------------------- | --------------------- | --------------------- | ------------------ | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLO11n-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n-pose.pt) | 640 | 50.0 | 81.0 | 52.4 ± 0.5 | 1.7 ± 0.0 | 2.9 | 7.4 |
|
||||
| [YOLO11s-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s-pose.pt) | 640 | 58.9 | 86.3 | 90.5 ± 0.6 | 2.6 ± 0.0 | 9.9 | 23.1 |
|
||||
| [YOLO11m-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11m-pose.pt) | 640 | 64.9 | 89.4 | 187.3 ± 0.8 | 4.9 ± 0.1 | 20.9 | 71.4 |
|
||||
| [YOLO11l-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11l-pose.pt) | 640 | 66.1 | 89.9 | 247.7 ± 1.1 | 6.4 ± 0.1 | 26.1 | 90.3 |
|
||||
| [YOLO11x-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11x-pose.pt) | 640 | 69.5 | 91.1 | 488.0 ± 13.9 | 12.1 ± 0.2 | 58.8 | 202.8 |
|
||||
|
||||
=== "OBB (DOTAv1)"
|
||||
|
||||
See [Oriented Detection Docs](../tasks/obb.md) for usage examples with these models trained on [DOTAv1](../datasets/obb/dota-v2.md#dota-v10), which include 15 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>test<br>50</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT10<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| -------------------------------------------------------------------------------------------- | --------------------- | ------------------ | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLO11n-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n-obb.pt) | 1024 | 78.4 | 117.6 ± 0.8 | 4.4 ± 0.0 | 2.7 | 16.8 |
|
||||
| [YOLO11s-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s-obb.pt) | 1024 | 79.5 | 219.4 ± 4.0 | 5.1 ± 0.0 | 9.7 | 57.1 |
|
||||
| [YOLO11m-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11m-obb.pt) | 1024 | 80.9 | 562.8 ± 2.9 | 10.1 ± 0.4 | 20.9 | 182.8 |
|
||||
| [YOLO11l-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11l-obb.pt) | 1024 | 81.0 | 712.5 ± 5.0 | 13.5 ± 0.6 | 26.1 | 231.2 |
|
||||
| [YOLO11x-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11x-obb.pt) | 1024 | 81.3 | 1408.6 ± 7.7 | 28.6 ± 1.0 | 58.8 | 519.1 |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This section provides simple YOLO11 training and inference examples. For full documentation on these and other [modes](../modes/index.md), see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md), and [Export](../modes/export.md) docs pages.
|
||||
|
||||
Note that the example below is for YOLO11 [Detect](../tasks/detect.md) models for [object detection](https://www.ultralytics.com/glossary/object-detection). For additional supported tasks, see the [Segment](../tasks/segment.md), [Classify](../tasks/classify.md), [OBB](../tasks/obb.md), and [Pose](../tasks/pose.md) docs.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLO11n model
|
||||
model = YOLO("yolo11n.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLO11n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO11n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolo11n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLO11n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolo11n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
!!! tip "Ultralytics YOLO11 Publication"
|
||||
|
||||
Ultralytics has not published a formal research paper for YOLO11 due to the rapidly evolving nature of the models. We focus on advancing the technology and making it easier to use, rather than producing static documentation. For the most up-to-date information on YOLO architecture, features, and usage, please refer to our [GitHub repository](https://github.com/ultralytics/ultralytics) and [documentation](https://docs.ultralytics.com/).
|
||||
|
||||
If you use YOLO11 or any other software from this repository in your work, please cite it using the following format:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@software{yolo11_ultralytics,
|
||||
author = {Glenn Jocher and Jing Qiu},
|
||||
title = {Ultralytics YOLO11},
|
||||
version = {11.0.0},
|
||||
year = {2024},
|
||||
url = {https://github.com/ultralytics/ultralytics},
|
||||
orcid = {0000-0001-5950-6979, 0000-0003-3783-7069},
|
||||
license = {AGPL-3.0}
|
||||
}
|
||||
```
|
||||
|
||||
Please note that the DOI is pending and will be added to the citation once it is available. YOLO11 models are provided under [AGPL-3.0](https://github.com/ultralytics/ultralytics/blob/main/LICENSE) and [Enterprise](https://www.ultralytics.com/license) licenses.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What are the key improvements in Ultralytics YOLO11 compared to YOLOv8?
|
||||
|
||||
Ultralytics YOLO11 introduces several significant advancements over YOLOv8. Key improvements include:
|
||||
|
||||
- **Enhanced Feature Extraction:** YOLO11 employs an improved backbone and neck architecture, enhancing [feature extraction](https://www.ultralytics.com/glossary/feature-extraction) capabilities for more precise object detection.
|
||||
- **Optimized Efficiency and Speed:** Refined architectural designs and optimized training pipelines deliver faster processing speeds while maintaining a balance between accuracy and performance.
|
||||
- **Greater Accuracy with Fewer Parameters:** YOLO11m achieves higher mean Average [Precision](https://www.ultralytics.com/glossary/precision) (mAP) on the COCO dataset with 22% fewer parameters than YOLOv8m, making it computationally efficient without compromising accuracy.
|
||||
- **Adaptability Across Environments:** YOLO11 can be deployed across various environments, including edge devices, cloud platforms, and systems supporting NVIDIA GPUs.
|
||||
- **Broad Range of Supported Tasks:** YOLO11 supports diverse computer vision tasks such as object detection, [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), image classification, pose estimation, and oriented object detection (OBB).
|
||||
|
||||
### How do I train a YOLO11 model for object detection?
|
||||
|
||||
Training a YOLO11 model for object detection can be done using Python or CLI commands. Below are examples for both methods:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLO11n model
|
||||
model = YOLO("yolo11n.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO11n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolo11n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For more detailed instructions, refer to the [Train](../modes/train.md) documentation.
|
||||
|
||||
### What tasks can YOLO11 models perform?
|
||||
|
||||
YOLO11 models are versatile and support a wide range of computer vision tasks, including:
|
||||
|
||||
- **Object Detection:** Identifying and locating objects within an image.
|
||||
- **Instance Segmentation:** Detecting objects and delineating their boundaries.
|
||||
- **[Image Classification](https://www.ultralytics.com/glossary/image-classification):** Categorizing images into predefined classes.
|
||||
- **Pose Estimation:** Detecting and tracking keypoints on human bodies.
|
||||
- **Oriented Object Detection (OBB):** Detecting objects with rotation for higher precision.
|
||||
|
||||
For more information on each task, see the [Detection](../tasks/detect.md), [Instance Segmentation](../tasks/segment.md), [Classification](../tasks/classify.md), [Pose Estimation](../tasks/pose.md), and [Oriented Detection](../tasks/obb.md) documentation.
|
||||
|
||||
### How does YOLO11 achieve greater accuracy with fewer parameters?
|
||||
|
||||
YOLO11 achieves greater accuracy with fewer parameters through advancements in model design and optimization techniques. The improved architecture allows for efficient feature extraction and processing, resulting in higher mean Average Precision (mAP) on datasets like COCO while using 22% fewer parameters than YOLOv8m. This makes YOLO11 computationally efficient without compromising on accuracy, making it suitable for deployment on resource-constrained devices.
|
||||
|
||||
### Can YOLO11 be deployed on edge devices?
|
||||
|
||||
Yes, YOLO11 is designed for adaptability across various environments, including edge devices. Its optimized architecture and efficient processing capabilities make it suitable for deployment on edge devices, cloud platforms, and systems supporting NVIDIA GPUs. This flexibility ensures that YOLO11 can be used in diverse applications, from real-time detection on mobile devices to complex segmentation tasks in cloud environments. For more details on deployment options, refer to the [Export](../modes/export.md) documentation.
|
||||
188
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo12.md
Normal file
188
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo12.md
Normal file
@@ -0,0 +1,188 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLO12, featuring groundbreaking attention-centric architecture for state-of-the-art object detection with unmatched accuracy and efficiency.
|
||||
keywords: YOLO12, attention-centric object detection, YOLO series, Ultralytics, computer vision, AI, machine learning, deep learning
|
||||
---
|
||||
|
||||
# YOLO12: Attention-Centric Object Detection
|
||||
|
||||
## Overview
|
||||
|
||||
YOLO12, released in early 2025, introduces an attention-centric architecture that departs from the traditional CNN-based approaches used in previous YOLO models, yet retains the real-time inference speed essential for many applications. This model achieves high object detection accuracy through novel methodological innovations in attention mechanisms and overall network architecture, while maintaining real-time performance. Despite those advantages, YOLO12 remains a community-driven release that may exhibit training instability, elevated memory consumption, and slower CPU throughput due to its heavy attention blocks, so Ultralytics recommends [YOLO11](yolo11.md) or [YOLO26](yolo26.md) for most production workloads.
|
||||
|
||||
!!! note "Community Model"
|
||||
|
||||
YOLO12 is maintained primarily for benchmarking and research. If you need stable training, predictable memory usage, and optimized CPU inference, choose [YOLO11](yolo11.md) or [YOLO26](yolo26.md) for deployment.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/mcqTxD-FD5M"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Use YOLO12 for Object Detection with the Ultralytics Package | Is YOLO12 Fast or Slow? 🚀
|
||||
</p>
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Area Attention Mechanism**: A new self-attention approach that processes large receptive fields efficiently. It divides [feature maps](https://www.ultralytics.com/glossary/feature-maps) into _l_ equal-sized regions (defaulting to 4), either horizontally or vertically, avoiding complex operations and maintaining a large effective receptive field. This significantly reduces computational cost compared to standard self-attention.
|
||||
- **Residual Efficient Layer Aggregation Networks (R-ELAN)**: An improved feature aggregation module based on ELAN, designed to address optimization challenges, especially in larger-scale attention-centric models. R-ELAN introduces:
|
||||
- Block-level residual connections with scaling (similar to layer scaling).
|
||||
- A redesigned feature aggregation method creating a bottleneck-like structure.
|
||||
- **Optimized Attention Architecture**: YOLO12 streamlines the standard attention mechanism for greater efficiency and compatibility with the YOLO framework. This includes:
|
||||
- Using FlashAttention to minimize memory access overhead.
|
||||
- Removing positional encoding for a cleaner and faster model.
|
||||
- Adjusting the MLP ratio (from the typical 4 to 1.2 or 2) to better balance computation between attention and feed-forward layers.
|
||||
- Reducing the depth of stacked blocks for improved optimization.
|
||||
- Leveraging convolution operations (where appropriate) for their computational efficiency.
|
||||
- Adding a 7x7 separable convolution (the "position perceiver") to the attention mechanism to implicitly encode positional information.
|
||||
- **Comprehensive Task Support**: YOLO12 supports a range of core computer vision tasks: object detection, [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), [image classification](https://www.ultralytics.com/glossary/image-classification), pose estimation, and oriented object detection (OBB).
|
||||
- **Enhanced Efficiency**: Achieves higher accuracy with fewer parameters compared to many prior models, demonstrating an improved balance between speed and accuracy.
|
||||
- **Flexible Deployment**: Designed for deployment across diverse platforms, from edge devices to cloud infrastructure.
|
||||
|
||||
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
YOLO12 supports a variety of computer vision tasks. The table below shows task support and the operational modes (Inference, Validation, Training, and Export) enabled for each:
|
||||
|
||||
| Model Type | Task | Inference | Validation | Training | Export |
|
||||
| -------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| [YOLO12](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/12/yolo12.yaml) | [Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| [YOLO12-seg](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/12/yolo12-seg.yaml) | [Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| [YOLO12-pose](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/12/yolo12-pose.yaml) | [Pose](../tasks/pose.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| [YOLO12-cls](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/12/yolo12-cls.yaml) | [Classification](../tasks/classify.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| [YOLO12-obb](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/12/yolo12-obb.yaml) | [OBB](../tasks/obb.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
YOLO12 demonstrates significant [accuracy](https://www.ultralytics.com/glossary/accuracy) improvements across all model scales, with some trade-offs in speed compared to the _fastest_ prior YOLO models. Below are quantitative results for [object detection](https://www.ultralytics.com/glossary/object-detection) on the COCO validation dataset:
|
||||
|
||||
### Detection Performance (COCO val2017)
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>T4 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> | Comparison<br><sup>(mAP/Speed)</sup> |
|
||||
| ------------------------------------------------------------------------------------ | --------------------- | -------------------- | ------------------------------ | --------------------------------- | ------------------ | ----------------- | ------------------------------ |
|
||||
| [YOLO12n](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo12n.pt) | 640 | 40.6 | - | 1.64 | 2.6 | 6.5 | +2.1%/-9% (vs. YOLOv10n) |
|
||||
| [YOLO12s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo12s.pt) | 640 | 48.0 | - | 2.61 | 9.3 | 21.4 | +0.1%/+42% (vs. RT-DETRv2) |
|
||||
| [YOLO12m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo12m.pt) | 640 | 52.5 | - | 4.86 | 20.2 | 67.5 | +1.0%/-3% (vs. YOLO11m) |
|
||||
| [YOLO12l](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo12l.pt) | 640 | 53.7 | - | 6.77 | 26.4 | 88.9 | +0.4%/-8% (vs. YOLO11l) |
|
||||
| [YOLO12x](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo12x.pt) | 640 | 55.2 | - | 11.79 | 59.1 | 199.0 | +0.6%/-4% (vs. YOLO11x) |
|
||||
|
||||
- Inference speed measured on an NVIDIA T4 GPU with TensorRT FP16 [precision](https://www.ultralytics.com/glossary/precision).
|
||||
- Comparisons show the relative improvement in mAP and the percentage change in speed (positive indicates faster; negative indicates slower). Comparisons are made against published results for YOLOv10, YOLO11, and RT-DETR where available.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This section provides examples for training and inference with YOLO12. For more comprehensive documentation on these and other modes (including [Validation](../modes/val.md) and [Export](../modes/export.md)), consult the dedicated [Predict](../modes/predict.md) and [Train](../modes/train.md) pages.
|
||||
|
||||
The examples below focus on YOLO12 [Detect](../tasks/detect.md) models (for object detection). For other supported tasks (segmentation, classification, oriented object detection, and pose estimation), refer to the respective task-specific documentation: [Segment](../tasks/segment.md), [Classify](../tasks/classify.md), [OBB](../tasks/obb.md), and [Pose](../tasks/pose.md).
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
Pretrained `*.pt` models (using [PyTorch](https://pytorch.org/)) and configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLO12n model
|
||||
model = YOLO("yolo12n.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLO12n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
Command Line Interface (CLI) commands are also available:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO12n model and train on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolo12n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLO12n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolo12n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Key Improvements
|
||||
|
||||
1. **Enhanced [Feature Extraction](https://www.ultralytics.com/glossary/feature-extraction)**:
|
||||
- **Area Attention**: Efficiently handles large [receptive fields](https://www.ultralytics.com/glossary/receptive-field), reducing computational cost.
|
||||
- **Optimized Balance**: Improved balance between attention and feed-forward network computations.
|
||||
- **R-ELAN**: Enhances feature aggregation using the R-ELAN architecture.
|
||||
|
||||
2. **Optimization Innovations**:
|
||||
- **Residual Connections**: Introduces residual connections with scaling to stabilize training, especially in larger models.
|
||||
- **Refined Feature Integration**: Implements an improved method for feature integration within R-ELAN.
|
||||
- **FlashAttention**: Incorporates FlashAttention to reduce memory access overhead.
|
||||
|
||||
3. **Architectural Efficiency**:
|
||||
- **Reduced Parameters**: Achieves a lower parameter count while maintaining or improving accuracy compared to many previous models.
|
||||
- **Streamlined Attention**: Uses a simplified attention implementation, avoiding positional encoding.
|
||||
- **Optimized MLP Ratios**: Adjusts MLP ratios to more effectively allocate computational resources.
|
||||
|
||||
## Requirements
|
||||
|
||||
The Ultralytics YOLO12 implementation, by default, _does not require_ FlashAttention. However, FlashAttention can be optionally compiled and used with YOLO12. To compile FlashAttention, one of the following NVIDIA GPUs is needed:
|
||||
|
||||
- [Turing GPUs](<https://en.wikipedia.org/wiki/Turing_(microarchitecture)>) (e.g., T4, Quadro RTX series)
|
||||
- [Ampere GPUs](<https://en.wikipedia.org/wiki/Ampere_(microarchitecture)>) (e.g., RTX30 series, A30/40/100)
|
||||
- [Ada Lovelace GPUs](https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture/) (e.g., RTX40 series)
|
||||
- [Hopper GPUs](https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/) (e.g., H100/H200)
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If you use YOLO12 in your research, please cite the original work by [University at Buffalo](https://www.buffalo.edu/) and the [University of Chinese Academy of Sciences](https://english.ucas.ac.cn/):
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{tian2025yolo12,
|
||||
title={YOLO12: Attention-Centric Real-Time Object Detectors},
|
||||
author={Tian, Yunjie and Ye, Qixiang and Doermann, David},
|
||||
journal={arXiv preprint arXiv:2502.12524},
|
||||
year={2025}
|
||||
}
|
||||
|
||||
@software{yolo12,
|
||||
author = {Tian, Yunjie and Ye, Qixiang and Doermann, David},
|
||||
title = {YOLO12: Attention-Centric Real-Time Object Detectors},
|
||||
year = {2025},
|
||||
url = {https://github.com/sunsmarterjie/yolov12},
|
||||
license = {AGPL-3.0}
|
||||
}
|
||||
```
|
||||
|
||||
## FAQ
|
||||
|
||||
### How does YOLO12 achieve real-time object detection while maintaining high accuracy?
|
||||
|
||||
YOLO12 incorporates several key innovations to balance speed and accuracy. The Area [Attention mechanism](https://www.ultralytics.com/glossary/attention-mechanism) efficiently processes large receptive fields, reducing computational cost compared to standard self-attention. The Residual Efficient Layer Aggregation Networks (R-ELAN) improve feature aggregation, addressing optimization challenges in larger attention-centric models. Optimized Attention Architecture, including the use of FlashAttention and removal of positional encoding, further enhances efficiency. These features allow YOLO12 to achieve state-of-the-art accuracy while maintaining the real-time inference speed crucial for many applications.
|
||||
|
||||
### What [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks does YOLO12 support?
|
||||
|
||||
YOLO12 is a versatile model that supports a wide range of core computer vision tasks. It excels in object [detection](../tasks/detect.md), instance [segmentation](../tasks/segment.md), image [classification](../tasks/classify.md), [pose estimation](../tasks/pose.md), and oriented object detection (OBB) ([see details](../tasks/obb.md)). This comprehensive task support makes YOLO12 a powerful tool for diverse applications, from [robotics](https://www.ultralytics.com/glossary/robotics) and autonomous driving to medical imaging and industrial inspection. Each of these tasks can be performed in Inference, Validation, Training, and Export modes.
|
||||
|
||||
### How does YOLO12 compare to other YOLO models and competitors like RT-DETR?
|
||||
|
||||
YOLO12 demonstrates significant accuracy improvements across all model scales compared to prior YOLO models like YOLOv10 and YOLO11, with some trade-offs in speed compared to the _fastest_ prior models. For example, YOLO12n achieves a +2.1% mAP improvement over YOLOv10n and +1.2% over YOLO11n on the COCO val2017 dataset. Compared to models like [RT-DETR](https://docs.ultralytics.com/models/rtdetr/), YOLO12s offers a +1.5% mAP improvement and a substantial +42% speed increase. These metrics highlight YOLO12's strong balance between accuracy and efficiency. See the [performance metrics section](#performance-metrics) for detailed comparisons.
|
||||
|
||||
### What are the hardware requirements for running YOLO12, especially for using FlashAttention?
|
||||
|
||||
By default, the Ultralytics YOLO12 implementation does _not_ require FlashAttention. However, FlashAttention can be optionally compiled and used with YOLO12 to minimize memory access overhead. To compile FlashAttention, one of the following NVIDIA GPUs is needed: Turing GPUs (e.g., T4, Quadro RTX series), Ampere GPUs (e.g., RTX30 series, A30/40/100), Ada Lovelace GPUs (e.g., RTX40 series), or Hopper GPUs (e.g., H100/H200). This flexibility allows users to leverage FlashAttention's benefits when hardware resources permit.
|
||||
|
||||
### Where can I find usage examples and more detailed documentation for YOLO12?
|
||||
|
||||
This page provides basic [usage examples](#usage-examples) for training and inference. For comprehensive documentation on these and other modes, including [Validation](../modes/val.md) and [Export](../modes/export.md), consult the dedicated [Predict](../modes/predict.md) and [Train](../modes/train.md) pages. For task-specific information (segmentation, classification, oriented object detection, and pose estimation), refer to the respective documentation: [Segment](../tasks/segment.md), [Classify](../tasks/classify.md), [OBB](../tasks/obb.md), and [Pose](../tasks/pose.md). These resources provide in-depth guidance for effectively utilizing YOLO12 in various scenarios.
|
||||
383
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo26.md
Normal file
383
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolo26.md
Normal file
@@ -0,0 +1,383 @@
|
||||
---
|
||||
comments: true
|
||||
description: YOLO26 from Ultralytics delivers faster, simpler, end-to-end NMS-free object detection optimized for edge and low-power devices.
|
||||
keywords: YOLO26, Ultralytics YOLO, object detection, end-to-end NMS-free, simplified architecture, computer vision, AI, machine learning, edge AI, low power devices, quantization, real-time inference
|
||||
---
|
||||
|
||||
# Ultralytics YOLO26
|
||||
|
||||
## Overview
|
||||
|
||||
[Ultralytics](https://www.ultralytics.com/) YOLO26 is the latest evolution in the YOLO series of real-time object detectors, engineered from the ground up for **edge and low-power devices**. It introduces a streamlined design that removes unnecessary complexity while integrating targeted innovations to deliver faster, lighter, and more accessible deployment.
|
||||
|
||||
|
||||
|
||||
!!! tip "Try on Ultralytics Platform"
|
||||
|
||||
Explore and run YOLO26 models directly on [Ultralytics Platform](https://platform.ultralytics.com/ultralytics/yolo26).
|
||||
|
||||
The architecture of YOLO26 is guided by three core principles:
|
||||
|
||||
- **Simplicity:** YOLO26 is a **native end-to-end model**, producing predictions directly without the need for non-maximum suppression (NMS). By eliminating this post-processing step, inference becomes faster, lighter, and easier to deploy in real-world systems. This breakthrough approach was first pioneered in [YOLOv10](../models/yolov10.md) by Ao Wang at Tsinghua University and has been further advanced in YOLO26.
|
||||
- **Deployment Efficiency:** The end-to-end design cuts out an entire stage of the pipeline, dramatically simplifying integration, reducing latency, and making deployment more robust across diverse environments.
|
||||
- **Training Innovation:** YOLO26 introduces the **MuSGD optimizer**, a hybrid of [SGD](https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html) and [Muon](https://arxiv.org/abs/2502.16982) — inspired by Moonshot AI's [Kimi K2](https://www.kimi.com/) breakthroughs in LLM training. This optimizer brings enhanced stability and faster convergence, transferring optimization advances from language models into computer vision.
|
||||
- **Task-Specific Optimizations:** YOLO26 introduces targeted improvements for specialized tasks, including semantic segmentation loss and multi-scale proto modules for **Segmentation**, Residual Log-Likelihood Estimation (RLE) for high-precision **Pose** estimation, and optimized decoding with angle loss to resolve boundary issues in **OBB**.
|
||||
|
||||
Together, these innovations deliver a model family that achieves higher accuracy on small objects, provides seamless deployment, and runs **up to 43% faster on CPUs** — making YOLO26 one of the most practical and deployable YOLO models to date for resource-constrained environments.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **DFL Removal**
|
||||
The Distribution Focal Loss (DFL) module, while effective, often complicated export and limited hardware compatibility. YOLO26 removes DFL entirely, simplifying inference and broadening support for **edge and low-power devices**.
|
||||
|
||||
- **End-to-End NMS-Free Inference**
|
||||
Unlike traditional detectors that rely on NMS as a separate post-processing step, YOLO26 is **natively end-to-end**. Predictions are generated directly, reducing latency and making integration into production systems faster, lighter, and more reliable.
|
||||
|
||||
- **ProgLoss + STAL**
|
||||
Improved loss functions increase detection accuracy, with notable improvements in **small-object recognition**, a critical requirement for IoT, robotics, aerial imagery, and other edge applications.
|
||||
|
||||
- **MuSGD Optimizer**
|
||||
A new hybrid optimizer that combines [SGD](https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html) with [Muon](https://arxiv.org/abs/2502.16982). Inspired by Moonshot AI's [Kimi K2](https://www.kimi.com/), MuSGD introduces advanced optimization methods from LLM training into computer vision, enabling more stable training and faster convergence.
|
||||
|
||||
- **Up to 43% Faster CPU Inference**
|
||||
Specifically optimized for edge computing, YOLO26 delivers significantly faster CPU inference, ensuring real-time performance on devices without GPUs.
|
||||
|
||||
- **Instance Segmentation Enhancements**
|
||||
Introduces semantic segmentation loss to improve model convergence and an upgraded proto module that leverages multi-scale information for superior mask quality.
|
||||
|
||||
- **Precision Pose Estimation**
|
||||
Integrates [Residual Log-Likelihood Estimation](https://arxiv.org/abs/2107.11291) (RLE) for more accurate keypoint localization and optimizes the decoding process for increased inference speed.
|
||||
|
||||
- **Refined OBB Decoding**
|
||||
Introduces a specialized angle loss to improve detection accuracy for square-shaped objects and optimizes OBB decoding to resolve boundary discontinuity issues.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
YOLO26 builds upon the versatile model range established by earlier Ultralytics YOLO releases, offering enhanced support across various computer vision tasks:
|
||||
|
||||
| Model | Filenames | Task | Inference | Validation | Training | Export |
|
||||
| ----------- | ----------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLO26 | `yolo26n.pt` `yolo26s.pt` `yolo26m.pt` `yolo26l.pt` `yolo26x.pt` | [Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO26-seg | `yolo26n-seg.pt` `yolo26s-seg.pt` `yolo26m-seg.pt` `yolo26l-seg.pt` `yolo26x-seg.pt` | [Instance Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO26-pose | `yolo26n-pose.pt` `yolo26s-pose.pt` `yolo26m-pose.pt` `yolo26l-pose.pt` `yolo26x-pose.pt` | [Pose/Keypoints](../tasks/pose.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO26-obb | `yolo26n-obb.pt` `yolo26s-obb.pt` `yolo26m-obb.pt` `yolo26l-obb.pt` `yolo26x-obb.pt` | [Oriented Detection](../tasks/obb.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLO26-cls | `yolo26n-cls.pt` `yolo26s-cls.pt` `yolo26m-cls.pt` `yolo26l-cls.pt` `yolo26x-cls.pt` | [Classification](../tasks/classify.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This unified framework ensures YOLO26 is applicable across real-time detection, segmentation, classification, pose estimation, and oriented object detection — all with training, validation, inference, and export support.
|
||||
|
||||
---
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
See [Detection Docs](../tasks/detect.md) for usage examples with these models trained on [COCO](../datasets/detect/coco.md), which include 80 pretrained classes.
|
||||
|
||||
--8<-- "docs/macros/yolo-det-perf.md"
|
||||
|
||||
=== "Segmentation (COCO)"
|
||||
|
||||
See [Segmentation Docs](../tasks/segment.md) for usage examples with these models trained on [COCO](../datasets/segment/coco.md), which include 80 pretrained classes.
|
||||
|
||||
--8<-- "docs/macros/yolo-seg-perf.md"
|
||||
|
||||
=== "Classification (ImageNet)"
|
||||
|
||||
See [Classification Docs](../tasks/classify.md) for usage examples with these models trained on [ImageNet](../datasets/classify/imagenet.md), which include 1000 pretrained classes.
|
||||
|
||||
--8<-- "docs/macros/yolo-cls-perf.md"
|
||||
|
||||
=== "Pose (COCO)"
|
||||
|
||||
See [Pose Estimation Docs](../tasks/pose.md) for usage examples with these models trained on [COCO](../datasets/pose/coco.md), which include 1 pretrained class, 'person'.
|
||||
|
||||
--8<-- "docs/macros/yolo-pose-perf.md"
|
||||
|
||||
=== "OBB (DOTAv1)"
|
||||
|
||||
See [Oriented Detection Docs](../tasks/obb.md) for usage examples with these models trained on [DOTAv1](../datasets/obb/dota-v2.md#dota-v10), which include 15 pretrained classes.
|
||||
|
||||
--8<-- "docs/macros/yolo-obb-perf.md"
|
||||
|
||||
---
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This section provides simple YOLO26 training and inference examples. For full documentation on these and other [modes](../modes/index.md), see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md), and [Export](../modes/export.md) docs pages.
|
||||
|
||||
Note that the example below is for YOLO26 [Detect](../tasks/detect.md) models for [object detection](https://www.ultralytics.com/glossary/object-detection). For additional supported tasks, see the [Segment](../tasks/segment.md), [Classify](../tasks/classify.md), [OBB](../tasks/obb.md), and [Pose](../tasks/pose.md) docs.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLO26n model
|
||||
model = YOLO("yolo26n.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLO26n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLO26n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolo26n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLO26n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolo26n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
!!! note "Dual-Head Architecture"
|
||||
|
||||
YOLO26 features a **dual-head architecture** that provides flexibility for different deployment scenarios:
|
||||
|
||||
- **One-to-One Head (Default)**: Produces end-to-end predictions without NMS, outputting `(N, 300, 6)` with a maximum of 300 detections per image. This head is optimized for fast inference and simplified deployment.
|
||||
- **One-to-Many Head**: Generates traditional YOLO outputs requiring NMS post-processing, outputting `(N, nc + 4, 8400)` where `nc` is the number of classes. This head typically achieves slightly higher accuracy at the cost of additional processing.
|
||||
|
||||
You can switch between heads during export, prediction, or validation:
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
model = YOLO("yolo26n.pt")
|
||||
|
||||
# Use one-to-one head (default, no NMS required)
|
||||
results = model.predict("image.jpg") # inference
|
||||
metrics = model.val(data="coco.yaml") # validation
|
||||
model.export(format="onnx") # export
|
||||
|
||||
# Use one-to-many head (requires NMS)
|
||||
results = model.predict("image.jpg", end2end=False) # inference
|
||||
metrics = model.val(data="coco.yaml", end2end=False) # validation
|
||||
model.export(format="onnx", end2end=False) # export
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Use one-to-one head (default, no NMS required)
|
||||
yolo predict model=yolo26n.pt source=image.jpg
|
||||
yolo val model=yolo26n.pt data=coco.yaml
|
||||
yolo export model=yolo26n.pt format=onnx
|
||||
|
||||
# Use one-to-many head (requires NMS)
|
||||
yolo predict model=yolo26n.pt source=image.jpg end2end=False
|
||||
yolo val model=yolo26n.pt data=coco.yaml end2end=False
|
||||
yolo export model=yolo26n.pt format=onnx end2end=False
|
||||
```
|
||||
|
||||
The choice depends on your deployment requirements: use the one-to-one head for maximum speed and simplicity, or the one-to-many head when accuracy is the top priority.
|
||||
|
||||
## YOLOE-26: Open-Vocabulary Instance Segmentation
|
||||
|
||||
YOLOE-26 integrates the high-performance YOLO26 architecture with the open-vocabulary capabilities of the [YOLOE](yoloe.md) series. It enables real-time detection and segmentation of any object class using **text prompts**, **visual prompts**, or a **prompt-free mode** for zero-shot inference, effectively removing the constraints of fixed-category training.
|
||||
|
||||
By leveraging YOLO26's **NMS-free, end-to-end design**, YOLOE-26 delivers fast open-world inference. This makes it a powerful solution for edge applications in dynamic environments where the objects of interest represent a broad and evolving vocabulary.
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Text/Visual Prompts"
|
||||
|
||||
See [YOLOE Docs](./yoloe.md) for usage examples with these models trained on [Objects365v1](https://opendatalab.com/OpenDataLab/Objects365_v1), [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html) and [Flickr30k](https://shannon.cs.illinois.edu/DenotationGraph/) datasets.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | Prompt Type | mAP<sup>minival<br>50-95(e2e)</sup> | mAP<sup>minival<br>50-95</sup> | mAP<sub>r</sub> | mAP<sub>c</sub> | mAP<sub>f</sub> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|---------------|-----------------------------|-------------|-------------------------------------|----------------------------|-----------------|-----------------|-----------------|--------------------------|-------------------------|
|
||||
| [YOLOE-26n-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26n-seg.pt) | 640 | Text/Visual | 23.7 / 20.9 | 24.7 / 21.9 | 20.5 / 17.6 | 24.1 / 22.3 | 26.1 / 22.4 | 4.8 | 6.0 |
|
||||
| [YOLOE-26s-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26s-seg.pt) | 640 | Text/Visual | 29.9 / 27.1 | 30.8 / 28.6 | 23.9 / 25.1 | 29.6 / 27.8 | 33.0 / 29.9 | 13.1 | 21.7 |
|
||||
| [YOLOE-26m-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26m-seg.pt) | 640 | Text/Visual | 35.4 / 31.3 | 35.4 / 33.9 | 31.1 / 33.4 | 34.7 / 34.0 | 36.9 / 33.8 | 27.9 | 70.1 |
|
||||
| [YOLOE-26l-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26l-seg.pt) | 640 | Text/Visual | 36.8 / 33.7 | 37.8 / 36.3 | 35.1 / 37.6 | 37.6 / 36.2 | 38.5 / 36.1 | 32.3 | 88.3 |
|
||||
| [YOLOE-26x-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26x-seg.pt) | 640 | Text/Visual | 39.5 / 36.2 | 40.6 / 38.5 | 37.4 / 35.3 | 40.9 / 38.8 | 41.0 / 38.8 | 69.9 | 196.7 |
|
||||
|
||||
|
||||
=== "Prompt-free"
|
||||
|
||||
See [YOLOE Docs](./yoloe.md) for usage examples with these models trained on [Objects365v1](https://opendatalab.com/OpenDataLab/Objects365_v1), [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html) and [Flickr30k](https://shannon.cs.illinois.edu/DenotationGraph/) datasets.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>minival<br>50-95(e2e)</sup> | mAP<sup>minival<br>50(e2e)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|------------------|-----------------------------|-------------------------------------|------------------------------|--------------------------|-------------------------|
|
||||
| [YOLOE-26n-seg-pf](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26n-seg-pf.pt) | 640 | 16.6 | 22.7 | 6.5 | 15.8 |
|
||||
| [YOLOE-26s-seg-pf](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26s-seg-pf.pt) | 640 | 21.4 | 28.6 | 16.2 | 35.5 |
|
||||
| [YOLOE-26m-seg-pf](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26m-seg-pf.pt) | 640 | 25.7 | 33.6 | 36.2 | 122.1 |
|
||||
| [YOLOE-26l-seg-pf](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26l-seg-pf.pt) | 640 | 27.2 | 35.4 | 40.6 | 140.4 |
|
||||
| [YOLOE-26x-seg-pf](https://github.com/ultralytics/assets/releases/download/v8.4.0/yoloe-26x-seg-pf.pt) | 640 | 29.9 | 38.7 | 86.3 | 314.4 |
|
||||
|
||||
### Usage Example
|
||||
|
||||
YOLOE-26 supports both text-based and visual prompting. Using prompts is straightforward—just pass them through the `predict` method as shown below:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Text Prompt"
|
||||
|
||||
Text prompts allow you to specify the classes that you wish to detect through textual descriptions. The following code shows how you can use YOLOE-26 to detect people and buses in an image:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Initialize model
|
||||
model = YOLO("yoloe-26l-seg.pt") # or select yoloe-26s/m-seg.pt for different sizes
|
||||
|
||||
# Set text prompt to detect person and bus. You only need to do this once after you load the model.
|
||||
names = ["person", "bus"]
|
||||
model.set_classes(names, model.get_text_pe(names))
|
||||
|
||||
# Run detection on the given image
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
=== "Visual Prompt"
|
||||
|
||||
Visual prompts allow you to guide the model by showing it visual examples of the target classes, rather than describing them in text.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
from ultralytics import YOLO
|
||||
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor
|
||||
|
||||
# Initialize model
|
||||
model = YOLO("yoloe-26l-seg.pt")
|
||||
|
||||
# Define visual prompts using bounding boxes and their corresponding class IDs.
|
||||
# Each box highlights an example of the object you want the model to detect.
|
||||
visual_prompts = dict(
|
||||
bboxes=np.array(
|
||||
[
|
||||
[221.52, 405.8, 344.98, 857.54], # Box enclosing person
|
||||
[120, 425, 160, 445], # Box enclosing glasses
|
||||
],
|
||||
),
|
||||
cls=np.array(
|
||||
[
|
||||
0, # ID to be assigned for person
|
||||
1, # ID to be assigned for glasses
|
||||
]
|
||||
),
|
||||
)
|
||||
|
||||
# Run inference on an image, using the provided visual prompts as guidance
|
||||
results = model.predict(
|
||||
"ultralytics/assets/bus.jpg",
|
||||
visual_prompts=visual_prompts,
|
||||
predictor=YOLOEVPSegPredictor,
|
||||
)
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
=== "Prompt free"
|
||||
|
||||
YOLOE-26 includes prompt-free variants that come with a built-in vocabulary. These models don't require any prompts and work like traditional YOLO models. Instead of relying on user-provided labels or visual examples, they detect objects from a [predefined list of 4,585 classes](https://github.com/xinyu1205/recognize-anything/blob/main/ram/data/ram_tag_list.txt) based on the tag set used by the [Recognize Anything Model Plus (RAM++)](https://arxiv.org/abs/2310.15200).
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Initialize model
|
||||
model = YOLO("yoloe-26l-seg-pf.pt")
|
||||
|
||||
# Run prediction. No prompts required.
|
||||
results = model.predict("path/to/image.jpg")
|
||||
|
||||
# Show results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
For a deep dive into prompting techniques, training from scratch, and full usage examples, visit the **[YOLOE Documentation](yoloe.md)**.
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
!!! tip "Ultralytics YOLO26 Publication"
|
||||
|
||||
Ultralytics has not published a formal research paper for YOLO26 due to the rapidly evolving nature of the models. Instead, we focus on delivering cutting-edge models and making them easy to use. For the latest updates on YOLO features, architectures, and usage, visit our [GitHub repository](https://github.com/ultralytics/ultralytics) and [documentation](https://docs.ultralytics.com/).
|
||||
|
||||
If you use YOLO26 or other Ultralytics software in your work, please cite it as:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@software{yolo26_ultralytics,
|
||||
author = {Glenn Jocher and Jing Qiu},
|
||||
title = {Ultralytics YOLO26},
|
||||
version = {26.0.0},
|
||||
year = {2026},
|
||||
url = {https://github.com/ultralytics/ultralytics},
|
||||
orcid = {0000-0001-5950-6979, 0000-0003-3783-7069},
|
||||
license = {AGPL-3.0}
|
||||
}
|
||||
```
|
||||
|
||||
DOI pending. YOLO26 is available under [AGPL-3.0](https://github.com/ultralytics/ultralytics/blob/main/LICENSE) and [Enterprise](https://www.ultralytics.com/license) licenses.
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
### What are the key improvements in YOLO26 compared to YOLO11?
|
||||
|
||||
- **DFL Removal**: Simplifies export and expands edge compatibility
|
||||
- **End-to-End NMS-Free Inference**: Eliminates NMS for faster, simpler deployment
|
||||
- **ProgLoss + STAL**: Boosts accuracy, especially on small objects
|
||||
- **MuSGD Optimizer**: Combines SGD and Muon (inspired by Moonshot's Kimi K2) for more stable, efficient training
|
||||
- **Up to 43% Faster CPU Inference**: Major performance gains for CPU-only devices
|
||||
|
||||
### What tasks does YOLO26 support?
|
||||
|
||||
YOLO26 is a **unified model family**, providing end-to-end support for multiple computer vision tasks:
|
||||
|
||||
- [Object Detection](../tasks/detect.md)
|
||||
- [Instance Segmentation](../tasks/segment.md)
|
||||
- [Image Classification](../tasks/classify.md)
|
||||
- [Pose Estimation](../tasks/pose.md)
|
||||
- [Oriented Object Detection (OBB)](../tasks/obb.md)
|
||||
|
||||
Each size variant (n, s, m, l, x) supports all tasks, plus open-vocabulary versions via [YOLOE-26](#yoloe-26-open-vocabulary-instance-segmentation).
|
||||
|
||||
### Why is YOLO26 optimized for edge deployment?
|
||||
|
||||
YOLO26 delivers **state-of-the-art edge performance** with:
|
||||
|
||||
- Up to 43% faster CPU inference
|
||||
- Reduced model size and memory footprint
|
||||
- Architecture simplified for compatibility (no DFL, no NMS)
|
||||
- Flexible export formats including TensorRT, ONNX, CoreML, TFLite, and OpenVINO
|
||||
|
||||
### How do I get started with YOLO26?
|
||||
|
||||
YOLO26 models were released on January 14, 2026, and are available for download. Install or update the `ultralytics` package and load a model:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a pretrained YOLO26 nano model
|
||||
model = YOLO("yolo26n.pt")
|
||||
|
||||
# Run inference on an image
|
||||
results = model("image.jpg")
|
||||
```
|
||||
|
||||
See the [Usage Examples](#usage-examples) section for training, validation, and export instructions.
|
||||
1010
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yoloe.md
Normal file
1010
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yoloe.md
Normal file
File diff suppressed because it is too large
Load Diff
312
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov10.md
Normal file
312
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov10.md
Normal file
@@ -0,0 +1,312 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLOv10 for real-time object detection, eliminating NMS and boosting efficiency. Achieve top performance with a low computational cost.
|
||||
keywords: YOLOv10, real-time object detection, NMS-free, deep learning, Tsinghua University, Ultralytics, machine learning, neural networks, performance optimization
|
||||
---
|
||||
|
||||
# YOLOv10: Real-Time End-to-End Object Detection
|
||||
|
||||
YOLOv10, released in May 2024 and built on the [Ultralytics](https://www.ultralytics.com/) [Python package](https://pypi.org/project/ultralytics/) by researchers at [Tsinghua University](https://www.tsinghua.edu.cn/en/), introduces a new approach to real-time object detection, addressing both the post-processing and model architecture deficiencies found in previous YOLO versions. By eliminating non-maximum suppression (NMS) and optimizing various model components, YOLOv10 achieved excellent performance with significantly reduced computational overhead at its time of release. Its NMS-free end-to-end design pioneered an approach that has been further developed in [YOLO26](yolo26.md).
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/_gRqR-miFPE"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Train YOLOv10 on SKU-110k Dataset using Ultralytics | Retail Dataset
|
||||
</p>
|
||||
|
||||
## Overview
|
||||
|
||||
Real-time object detection aims to accurately predict object categories and positions in images with low latency. The YOLO series has been at the forefront of this research due to its balance between performance and efficiency. However, reliance on NMS and architectural inefficiencies have hindered optimal performance. YOLOv10 addresses these issues by introducing [consistent dual assignments](https://arxiv.org/abs/2405.14458) for NMS-free training and a holistic efficiency-accuracy driven model design strategy.
|
||||
|
||||
### Architecture
|
||||
|
||||
The architecture of YOLOv10 builds upon the strengths of previous YOLO models while introducing several key innovations. The model architecture consists of the following components:
|
||||
|
||||
1. **[Backbone](https://www.ultralytics.com/glossary/backbone)**: Responsible for [feature extraction](https://www.ultralytics.com/glossary/feature-extraction), the backbone in YOLOv10 uses an enhanced version of CSPNet (Cross Stage Partial Network) to improve gradient flow and reduce computational redundancy.
|
||||
2. **Neck**: The neck is designed to aggregate features from different scales and passes them to the head. It includes PAN (Path Aggregation Network) layers for effective multiscale feature fusion.
|
||||
3. **One-to-Many Head**: Generates multiple predictions per object during training to provide rich supervisory signals and improve learning accuracy.
|
||||
4. **One-to-One Head**: Generates a single best prediction per object during inference to eliminate the need for NMS, thereby reducing latency and improving efficiency.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **NMS-Free Training**: Utilizes consistent dual assignments to eliminate the need for NMS, reducing [inference latency](https://www.ultralytics.com/glossary/inference-latency).
|
||||
2. **Holistic Model Design**: Comprehensive optimization of various components from both efficiency and accuracy perspectives, including lightweight classification heads, spatial-channel decoupled down sampling, and rank-guided block design.
|
||||
3. **Enhanced Model Capabilities**: Incorporates large-kernel [convolutions](https://www.ultralytics.com/glossary/convolution) and partial self-attention modules to improve performance without significant computational cost.
|
||||
|
||||
## Model Variants
|
||||
|
||||
YOLOv10 comes in various model scales to cater to different application needs:
|
||||
|
||||
- **YOLOv10n**: Nano version for extremely resource-constrained environments.
|
||||
- **YOLOv10s**: Small version balancing speed and accuracy.
|
||||
- **YOLOv10m**: Medium version for general-purpose use.
|
||||
- **YOLOv10b**: Balanced version with increased width for higher accuracy.
|
||||
- **YOLOv10l**: Large version for higher accuracy at the cost of increased computational resources.
|
||||
- **YOLOv10x**: Extra-large version for maximum accuracy and performance.
|
||||
|
||||
## Performance
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv10"]'></canvas>
|
||||
|
||||
YOLOv10 outperforms previous YOLO versions and other state-of-the-art models in terms of accuracy and efficiency. For example, YOLOv10s is 1.8x faster than [RT-DETR-R18](../models/rtdetr.md) with similar AP on the COCO dataset, and YOLOv10b has 46% less latency and 25% fewer parameters than [YOLOv9-C](../models/yolov9.md) with the same performance.
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
Latency measured with TensorRT FP16 on T4 GPU.
|
||||
|
||||
| Model | Input Size | AP<sup>val</sup> | FLOPs (G) | Latency (ms) |
|
||||
| ------------- | ---------- | ---------------- | --------- | ------------ |
|
||||
| [YOLOv10n][1] | 640 | 38.5 | **6.7** | **1.84** |
|
||||
| [YOLOv10s][2] | 640 | 46.3 | 21.6 | 2.49 |
|
||||
| [YOLOv10m][3] | 640 | 51.1 | 59.1 | 4.74 |
|
||||
| [YOLOv10b][4] | 640 | 52.5 | 92.0 | 5.74 |
|
||||
| [YOLOv10l][5] | 640 | 53.2 | 120.3 | 7.28 |
|
||||
| [YOLOv10x][6] | 640 | **54.4** | 160.4 | 10.70 |
|
||||
|
||||
## Methodology
|
||||
|
||||
### Consistent Dual Assignments for NMS-Free Training
|
||||
|
||||
YOLOv10 employs dual label assignments, combining one-to-many and one-to-one strategies during training to ensure rich supervision and efficient end-to-end deployment. The consistent matching metric aligns the supervision between both strategies, enhancing the quality of predictions during [inference](../modes/predict.md).
|
||||
|
||||
### Holistic Efficiency-[Accuracy](https://www.ultralytics.com/glossary/accuracy) Driven Model Design
|
||||
|
||||
#### Efficiency Enhancements
|
||||
|
||||
1. **Lightweight Classification Head**: Reduces the computational overhead of the classification head by using depth-wise separable convolutions.
|
||||
2. **Spatial-Channel Decoupled Down sampling**: Decouples spatial reduction and channel modulation to minimize information loss and computational cost.
|
||||
3. **Rank-Guided Block Design**: Adapts block design based on intrinsic stage redundancy, ensuring optimal parameter utilization.
|
||||
|
||||
#### Accuracy Enhancements
|
||||
|
||||
1. **Large-Kernel Convolution**: Enlarges the [receptive field](https://www.ultralytics.com/glossary/receptive-field) to enhance feature extraction capability.
|
||||
2. **Partial Self-Attention (PSA)**: Incorporates self-attention modules to improve global representation learning with minimal overhead.
|
||||
|
||||
## Experiments and Results
|
||||
|
||||
YOLOv10 has been extensively tested on standard benchmarks like [COCO](../datasets/detect/coco.md), demonstrating superior performance and efficiency. The model achieves state-of-the-art results across different variants, showcasing significant improvements in latency and accuracy compared to previous versions and other contemporary detectors.
|
||||
|
||||
## Comparisons
|
||||
|
||||
|
||||
|
||||
Compared to other state-of-the-art detectors:
|
||||
|
||||
- YOLOv10s / x are 1.8× / 1.3× faster than RT-DETR-R18 / R101 with similar accuracy
|
||||
- YOLOv10b has 25% fewer parameters and 46% lower latency than YOLOv9-C at same accuracy
|
||||
- YOLOv10l / x outperform [YOLOv8l / x](../models/yolov8.md) by 0.3 AP / 0.5 AP with 1.8× / 2.3× fewer parameters
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
Here is a detailed comparison of YOLOv10 variants with other state-of-the-art models:
|
||||
|
||||
| Model | Params<br><sup>(M)</sup> | FLOPs<br><sup>(G)</sup> | mAP<sup>val<br>50-95</sup> | Latency<br><sup>(ms)</sup> | Latency-forward<br><sup>(ms)</sup> |
|
||||
| ----------------- | ------------------ | ----------------- | -------------------- | -------------------- | ---------------------------- |
|
||||
| YOLOv6-3.0-N | 4.7 | 11.4 | 37.0 | 2.69 | **1.76** |
|
||||
| Gold-YOLO-N | 5.6 | 12.1 | **39.6** | 2.92 | 1.82 |
|
||||
| YOLOv8n | 3.2 | 8.7 | 37.3 | 6.16 | 1.77 |
|
||||
| **[YOLOv10n][1]** | **2.3** | **6.7** | 39.5 | **1.84** | 1.79 |
|
||||
| | | | | | |
|
||||
| YOLOv6-3.0-S | 18.5 | 45.3 | 44.3 | 3.42 | 2.35 |
|
||||
| Gold-YOLO-S | 21.5 | 46.0 | 45.4 | 3.82 | 2.73 |
|
||||
| YOLOv8s | 11.2 | 28.6 | 44.9 | 7.07 | **2.33** |
|
||||
| **[YOLOv10s][2]** | **7.2** | **21.6** | **46.8** | **2.49** | 2.39 |
|
||||
| | | | | | |
|
||||
| RT-DETR-R18 | 20.0 | 60.0 | 46.5 | **4.58** | **4.49** |
|
||||
| YOLOv6-3.0-M | 34.9 | 85.8 | 49.1 | 5.63 | 4.56 |
|
||||
| Gold-YOLO-M | 41.3 | 87.5 | 49.8 | 6.38 | 5.45 |
|
||||
| YOLOv8m | 25.9 | 78.9 | 50.6 | 9.50 | 5.09 |
|
||||
| **[YOLOv10m][3]** | **15.4** | **59.1** | **51.3** | 4.74 | 4.63 |
|
||||
| | | | | | |
|
||||
| YOLOv6-3.0-L | 59.6 | 150.7 | 51.8 | 9.02 | 7.90 |
|
||||
| Gold-YOLO-L | 75.1 | 151.7 | 51.8 | 10.65 | 9.78 |
|
||||
| YOLOv8l | 43.7 | 165.2 | 52.9 | 12.39 | 8.06 |
|
||||
| RT-DETR-R50 | 42.0 | 136.0 | 53.1 | 9.20 | 9.07 |
|
||||
| **[YOLOv10l][5]** | **24.4** | **120.3** | **53.4** | **7.28** | **7.21** |
|
||||
| | | | | | |
|
||||
| YOLOv8x | 68.2 | 257.8 | 53.9 | 16.86 | 12.83 |
|
||||
| RT-DETR-R101 | 76.0 | 259.0 | 54.3 | 13.71 | 13.58 |
|
||||
| **[YOLOv10x][6]** | **29.5** | **160.4** | **54.4** | **10.70** | **10.60** |
|
||||
|
||||
[1]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10n.pt
|
||||
[2]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10s.pt
|
||||
[3]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10m.pt
|
||||
[4]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10b.pt
|
||||
[5]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10l.pt
|
||||
[6]: https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov10x.pt
|
||||
|
||||
## Usage Examples
|
||||
|
||||
For predicting new images with YOLOv10:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a pretrained YOLOv10n model
|
||||
model = YOLO("yolov10n.pt")
|
||||
|
||||
# Perform object detection on an image
|
||||
results = model("image.jpg")
|
||||
|
||||
# Display the results
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv10n model and run inference on the 'bus.jpg' image
|
||||
yolo detect predict model=yolov10n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
For training YOLOv10 on a custom dataset:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load YOLOv10n model from scratch
|
||||
model = YOLO("yolov10n.yaml")
|
||||
|
||||
# Train the model
|
||||
model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Build a YOLOv10n model from scratch and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov10n.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Build a YOLOv10n model from scratch and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov10n.yaml source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
The YOLOv10 model series offers a range of models, each optimized for high-performance [Object Detection](../tasks/detect.md). These models cater to varying computational needs and accuracy requirements, making them versatile for a wide array of applications.
|
||||
|
||||
| Model | Filenames | Tasks | Inference | Validation | Training | Export |
|
||||
| ------- | --------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv10 | `yolov10n.pt` `yolov10s.pt` `yolov10m.pt` `yolov10l.pt` `yolov10x.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
## Exporting YOLOv10
|
||||
|
||||
Due to the new operations introduced with YOLOv10, not all export formats provided by Ultralytics are currently supported. The following table outlines which formats have been successfully converted using Ultralytics for YOLOv10. Feel free to open a pull request if you're able to [provide a contribution change](../help/contributing.md) for adding export support of additional formats for YOLOv10.
|
||||
|
||||
| Export Format | Export Support | Exported Model Inference | Notes |
|
||||
| ------------------------------------------------- | -------------- | ------------------------ | -------------------------------------------------------------------------------------- |
|
||||
| [TorchScript](../integrations/torchscript.md) | ✅ | ✅ | Standard [PyTorch](https://www.ultralytics.com/glossary/pytorch) model format. |
|
||||
| [ONNX](../integrations/onnx.md) | ✅ | ✅ | Widely supported for deployment. |
|
||||
| [OpenVINO](../integrations/openvino.md) | ✅ | ✅ | Optimized for Intel hardware. |
|
||||
| [TensorRT](../integrations/tensorrt.md) | ✅ | ✅ | Optimized for NVIDIA GPUs. |
|
||||
| [CoreML](../integrations/coreml.md) | ✅ | ✅ | Limited to Apple devices. |
|
||||
| [TF SavedModel](../integrations/tf-savedmodel.md) | ✅ | ✅ | [TensorFlow](https://www.ultralytics.com/glossary/tensorflow)'s standard model format. |
|
||||
| [TF GraphDef](../integrations/tf-graphdef.md) | ✅ | ✅ | Legacy TensorFlow format. |
|
||||
| [TF Lite](../integrations/tflite.md) | ✅ | ✅ | Optimized for mobile and embedded. |
|
||||
| [TF Edge TPU](../integrations/edge-tpu.md) | ✅ | ✅ | Specific to Google's Edge TPU devices. |
|
||||
| [TF.js](../integrations/tfjs.md) | ✅ | ✅ | JavaScript environment for browser use. |
|
||||
| [PaddlePaddle](../integrations/paddlepaddle.md) | ❌ | ❌ | Popular in China; less global support. |
|
||||
| [NCNN](../integrations/ncnn.md) | ✅ | ❌ | Layer `torch.topk` not exists or registered |
|
||||
|
||||
## Conclusion
|
||||
|
||||
YOLOv10 set a new standard in real-time object detection at its release by addressing the shortcomings of previous YOLO versions and incorporating innovative design strategies. Its NMS-free approach pioneered end-to-end object detection in the YOLO family. For the latest Ultralytics model with improved performance and NMS-free inference, see [YOLO26](yolo26.md).
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the YOLOv10 authors from [Tsinghua University](https://www.tsinghua.edu.cn/en/) for their extensive research and significant contributions to the [Ultralytics](https://www.ultralytics.com/) framework:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{THU-MIGyolov10,
|
||||
title={YOLOv10: Real-Time End-to-End Object Detection},
|
||||
author={Ao Wang, Hui Chen, Lihao Liu, et al.},
|
||||
journal={arXiv preprint arXiv:2405.14458},
|
||||
year={2024},
|
||||
institution={Tsinghua University},
|
||||
license = {AGPL-3.0}
|
||||
}
|
||||
```
|
||||
|
||||
For detailed implementation, architectural innovations, and experimental results, please refer to the YOLOv10 [research paper](https://arxiv.org/pdf/2405.14458) and [GitHub repository](https://github.com/THU-MIG/yolov10) by the Tsinghua University team.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is YOLOv10 and how does it differ from previous YOLO versions?
|
||||
|
||||
YOLOv10, developed by researchers at [Tsinghua University](https://www.tsinghua.edu.cn/en/), introduces several key innovations to real-time object detection. It eliminates the need for non-maximum suppression (NMS) by employing consistent dual assignments during training and optimized model components for superior performance with reduced computational overhead. For more details on its architecture and key features, check out the [YOLOv10 overview](#overview) section.
|
||||
|
||||
### How can I get started with running inference using YOLOv10?
|
||||
|
||||
For easy inference, you can use the Ultralytics YOLO Python library or the command line interface (CLI). Below are examples of predicting new images using YOLOv10:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load the pretrained YOLOv10n model
|
||||
model = YOLO("yolov10n.pt")
|
||||
results = model("image.jpg")
|
||||
results[0].show()
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo detect predict model=yolov10n.pt source=path/to/image.jpg
|
||||
```
|
||||
|
||||
For more usage examples, visit our [Usage Examples](#usage-examples) section.
|
||||
|
||||
### Which model variants does YOLOv10 offer and what are their use cases?
|
||||
|
||||
YOLOv10 offers several model variants to cater to different use cases:
|
||||
|
||||
- **YOLOv10n**: Suitable for extremely resource-constrained environments
|
||||
- **YOLOv10s**: Balances speed and accuracy
|
||||
- **YOLOv10m**: General-purpose use
|
||||
- **YOLOv10b**: Higher accuracy with increased width
|
||||
- **YOLOv10l**: High accuracy at the cost of computational resources
|
||||
- **YOLOv10x**: Maximum accuracy and performance
|
||||
|
||||
Each variant is designed for different computational needs and accuracy requirements, making them versatile for a variety of applications. Explore the [Model Variants](#model-variants) section for more information.
|
||||
|
||||
### How does the NMS-free approach in YOLOv10 improve performance?
|
||||
|
||||
YOLOv10 eliminates the need for non-maximum suppression (NMS) during inference by employing consistent dual assignments for training. This approach reduces inference latency and enhances prediction efficiency. The architecture also includes a one-to-one head for inference, ensuring that each object gets a single best prediction. For a detailed explanation, see the [Consistent Dual Assignments for NMS-Free Training](#consistent-dual-assignments-for-nms-free-training) section.
|
||||
|
||||
### Where can I find the export options for YOLOv10 models?
|
||||
|
||||
YOLOv10 supports several export formats, including TorchScript, ONNX, OpenVINO, and TensorRT. However, not all export formats provided by Ultralytics are currently supported for YOLOv10 due to its new operations. For details on the supported formats and instructions on exporting, visit the [Exporting YOLOv10](#exporting-yolov10) section.
|
||||
|
||||
### What are the performance benchmarks for YOLOv10 models?
|
||||
|
||||
YOLOv10 outperforms previous YOLO versions and other state-of-the-art models in both accuracy and efficiency. For example, YOLOv10s is 1.8x faster than RT-DETR-R18 with a similar AP on the COCO dataset. YOLOv10b shows 46% less latency and 25% fewer parameters than YOLOv9-C with the same performance. Detailed benchmarks can be found in the [Comparisons](#comparisons) section.
|
||||
181
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov3.md
Normal file
181
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov3.md
Normal file
@@ -0,0 +1,181 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLOv3 and its variants YOLOv3-Ultralytics and YOLOv3u. Learn about their features, implementations, and support for object detection tasks.
|
||||
keywords: YOLOv3, YOLOv3-Ultralytics, YOLOv3u, object detection, Ultralytics, computer vision, AI models, deep learning
|
||||
---
|
||||
|
||||
# YOLOv3, and YOLOv3u
|
||||
|
||||
## Overview
|
||||
|
||||
This document presents an overview of three closely related object detection models, namely [YOLOv3](https://pjreddie.com/darknet/yolo/), [YOLOv3-Ultralytics](https://github.com/ultralytics/yolov3), and [YOLOv3u](https://github.com/ultralytics/ultralytics).
|
||||
|
||||
1. **YOLOv3:** This is the third version of the You Only Look Once (YOLO) object detection algorithm. Originally developed by Joseph Redmon, YOLOv3 improved on its predecessors by introducing features such as multiscale predictions and three different sizes of detection kernels.
|
||||
|
||||
2. **YOLOv3u:** This is an updated version of YOLOv3-Ultralytics that incorporates the anchor-free, objectness-free split head used in YOLOv8 models. YOLOv3u maintains the same [backbone](https://www.ultralytics.com/glossary/backbone) and neck architecture as YOLOv3 but with the updated [detection head](https://www.ultralytics.com/glossary/detection-head) from YOLOv8.
|
||||
|
||||
|
||||
|
||||
## Key Features
|
||||
|
||||
- **YOLOv3:** Introduced the use of three different scales for detection, leveraging three different sizes of detection kernels: 13x13, 26x26, and 52x52. This significantly improved detection accuracy for objects of different sizes. Additionally, YOLOv3 added features such as multi-label predictions for each [bounding box](https://www.ultralytics.com/glossary/bounding-box) and a better feature extractor network.
|
||||
|
||||
- **YOLOv3u:** This updated model incorporates the anchor-free, objectness-free split head from YOLOv8. By eliminating the need for pre-defined anchor boxes and objectness scores, this detection head design can improve the model's ability to detect objects of varying sizes and shapes. This makes YOLOv3u more robust and accurate for object detection tasks.
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
YOLOv3 is designed specifically for [object detection](https://www.ultralytics.com/glossary/object-detection) tasks. Ultralytics supports three variants of YOLOv3: `yolov3u`, `yolov3-tinyu` and `yolov3-sppu`. The `u` in the name signifies that these utilize the anchor-free head of YOLOv8, unlike their original architecture which is anchor-based. These models are renowned for their effectiveness in various real-world scenarios, balancing accuracy and speed. Each variant offers unique features and optimizations, making them suitable for a range of applications.
|
||||
|
||||
All three models support a comprehensive set of modes, ensuring versatility in various stages of [model deployment](https://www.ultralytics.com/glossary/model-deployment) and development. These modes include [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md), providing users with a complete toolkit for effective object detection.
|
||||
|
||||
| Model Type | Pretrained Weights | Tasks Supported | Inference | Validation | Training | Export |
|
||||
| -------------- | ------------------ | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv3(u) | `yolov3u.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv3-Tiny(u) | `yolov3-tinyu.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv3u-SPP(u) | `yolov3-sppu.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides an at-a-glance view of the capabilities of each YOLOv3 variant, highlighting their versatility and suitability for various tasks and operational modes in object detection workflows.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This example provides simple YOLOv3 training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv3u model
|
||||
model = YOLO("yolov3u.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLOv3u model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv3u model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov3u.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLOv3u model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov3u.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
If you use YOLOv3 in your research, please cite the original YOLO papers and the Ultralytics YOLOv3 repository:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{redmon2018yolov3,
|
||||
title={YOLOv3: An Incremental Improvement},
|
||||
author={Redmon, Joseph and Farhadi, Ali},
|
||||
journal={arXiv preprint arXiv:1804.02767},
|
||||
year={2018}
|
||||
}
|
||||
```
|
||||
|
||||
Thank you to Joseph Redmon and Ali Farhadi for developing the original YOLOv3.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What are the differences between YOLOv3, YOLOv3-Ultralytics, and YOLOv3u?
|
||||
|
||||
YOLOv3 is the third iteration of the YOLO (You Only Look Once) [object detection](https://www.ultralytics.com/glossary/object-detection) algorithm developed by Joseph Redmon, known for its balance of [accuracy](https://www.ultralytics.com/glossary/accuracy) and speed, utilizing three different scales (13x13, 26x26, and 52x52) for detections. YOLOv3-Ultralytics is Ultralytics' adaptation of YOLOv3 that adds support for more pretrained models and facilitates easier model customization. YOLOv3u is an upgraded variant of YOLOv3-Ultralytics, integrating the anchor-free, objectness-free split head from YOLOv8, improving detection robustness and accuracy for various object sizes. For more details on the variants, refer to the [YOLOv3 series](https://github.com/ultralytics/yolov3).
|
||||
|
||||
### How can I train a YOLOv3 model using Ultralytics?
|
||||
|
||||
Training a YOLOv3 model with Ultralytics is straightforward. You can train the model using either Python or CLI:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv3u model
|
||||
model = YOLO("yolov3u.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv3u model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov3u.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For more comprehensive training options and guidelines, visit our [Train mode documentation](../modes/train.md).
|
||||
|
||||
### What makes YOLOv3u more accurate for object detection tasks?
|
||||
|
||||
YOLOv3u improves upon YOLOv3 and YOLOv3-Ultralytics by incorporating the anchor-free, objectness-free split head used in YOLOv8 models. This upgrade eliminates the need for pre-defined anchor boxes and objectness scores, enhancing its capability to detect objects of varying sizes and shapes more precisely. This makes YOLOv3u a better choice for complex and diverse object detection tasks. For more information, refer to the [Key Features](#key-features) section.
|
||||
|
||||
### How can I use YOLOv3 models for inference?
|
||||
|
||||
You can perform inference using YOLOv3 models by either Python scripts or CLI commands:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv3u model
|
||||
model = YOLO("yolov3u.pt")
|
||||
|
||||
# Run inference with the YOLOv3u model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv3u model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov3u.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
Refer to the [Inference mode documentation](../modes/predict.md) for more details on running YOLO models.
|
||||
|
||||
### What tasks are supported by YOLOv3 and its variants?
|
||||
|
||||
YOLOv3, YOLOv3-Tiny and YOLOv3-SPP primarily support object detection tasks. These models can be used for various stages of model deployment and development, such as Inference, Validation, Training, and Export. For a comprehensive set of tasks supported and more in-depth details, visit our [Object Detection tasks documentation](../tasks/detect.md).
|
||||
|
||||
### Where can I find resources to cite YOLOv3 in my research?
|
||||
|
||||
If you use YOLOv3 in your research, please cite the original YOLO papers and the Ultralytics YOLOv3 repository. Example BibTeX citation:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{redmon2018yolov3,
|
||||
title={YOLOv3: An Incremental Improvement},
|
||||
author={Redmon, Joseph and Farhadi, Ali},
|
||||
journal={arXiv preprint arXiv:1804.02767},
|
||||
year={2018}
|
||||
}
|
||||
```
|
||||
|
||||
For more citation details, refer to the [Citations and Acknowledgments](#citations-and-acknowledgments) section.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore YOLOv4, a state-of-the-art real-time object detection model by Alexey Bochkovskiy. Discover its architecture, features, and performance.
|
||||
keywords: YOLOv4, object detection, real-time detection, Alexey Bochkovskiy, neural networks, machine learning, computer vision
|
||||
---
|
||||
|
||||
# YOLOv4: High-Speed and Precise Object Detection
|
||||
|
||||
Welcome to the Ultralytics documentation page for YOLOv4, a state-of-the-art, real-time object detector launched in 2020 by Alexey Bochkovskiy at [https://github.com/AlexeyAB/darknet](https://github.com/AlexeyAB/darknet). YOLOv4 is designed to provide the optimal balance between speed and accuracy, making it an excellent choice for many applications.
|
||||
|
||||
 **YOLOv4 architecture diagram**. Showcasing the intricate network design of YOLOv4, including the backbone, neck, and head components, and their interconnected layers for optimal real-time object detection.
|
||||
|
||||
## Introduction
|
||||
|
||||
YOLOv4 stands for You Only Look Once version 4. It is a real-time object detection model developed to address the limitations of previous YOLO versions like [YOLOv3](yolov3.md) and other object detection models. Unlike other [convolutional neural network](https://www.ultralytics.com/glossary/convolutional-neural-network-cnn) (CNN) based object detectors, YOLOv4 is not only applicable for recommendation systems but also for standalone process management and human input reduction. Its operation on conventional graphics processing units (GPUs) allows for mass usage at an affordable price, and it is designed to work in real-time on a conventional GPU while requiring only one such GPU for training.
|
||||
|
||||
## Architecture
|
||||
|
||||
YOLOv4 makes use of several innovative features that work together to optimize its performance. These include Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), Cross mini-[Batch Normalization](https://www.ultralytics.com/glossary/batch-normalization) (CmBN), Self-adversarial-training (SAT), Mish-activation, Mosaic [data augmentation](https://www.ultralytics.com/glossary/data-augmentation), DropBlock [regularization](https://www.ultralytics.com/glossary/regularization), and CIoU loss. These features are combined to achieve state-of-the-art results.
|
||||
|
||||
A typical object detector is composed of several parts including the input, the [backbone](https://www.ultralytics.com/glossary/backbone), the neck, and the head. The backbone of YOLOv4 is pretrained on [ImageNet](https://docs.ultralytics.com/datasets/classify/imagenet/) and is used to predict classes and [bounding boxes](https://www.ultralytics.com/glossary/bounding-box) of objects. The backbone could be from several models including VGG, ResNet, ResNeXt, or DenseNet. The neck part of the detector is used to collect [feature maps](https://www.ultralytics.com/glossary/feature-maps) from different stages and usually includes several bottom-up paths and several top-down paths. The head part is what is used to make the final object detections and classifications.
|
||||
|
||||
## Bag of Freebies
|
||||
|
||||
YOLOv4 also makes use of methods known as "bag of freebies," which are techniques that improve the [accuracy](https://www.ultralytics.com/glossary/accuracy) of the model during training without increasing the cost of inference. [Data augmentation](https://www.ultralytics.com/blog/the-ultimate-guide-to-data-augmentation-in-2025) is a common bag of freebies technique used in [object detection](https://www.ultralytics.com/glossary/object-detection), which increases the variability of the input images to improve the robustness of the model. Some examples of data augmentation include photometric distortions (adjusting the brightness, contrast, hue, saturation, and noise of an image) and geometric distortions (adding random scaling, cropping, flipping, and rotating). These techniques help the model to generalize better to different types of images.
|
||||
|
||||
## Features and Performance
|
||||
|
||||
YOLOv4 is designed for optimal speed and accuracy in object detection. The architecture of YOLOv4 includes CSPDarknet53 as the backbone, PANet as the neck, and YOLOv3 as the [detection head](https://www.ultralytics.com/glossary/detection-head). This design allows YOLOv4 to perform object detection at an impressive speed, making it suitable for real-time applications. YOLOv4 also excels in accuracy, achieving state-of-the-art results in object detection benchmarks like [COCO](https://docs.ultralytics.com/datasets/detect/coco/).
|
||||
|
||||
When compared to other models in the YOLO family, such as [YOLOv5](https://docs.ultralytics.com/models/yolov5/) and [YOLOv7](https://docs.ultralytics.com/models/yolov7/), YOLOv4 maintains a strong position in the balance between speed and accuracy. While newer models may offer certain advantages, YOLOv4's architectural innovations continue to make it relevant for many applications requiring real-time performance.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
As of the time of writing, Ultralytics does not currently support YOLOv4 models. Therefore, any users interested in using YOLOv4 will need to refer directly to the YOLOv4 GitHub repository for installation and usage instructions.
|
||||
|
||||
Here is a brief overview of the typical steps you might take to use YOLOv4:
|
||||
|
||||
1. Visit the YOLOv4 GitHub repository: [https://github.com/AlexeyAB/darknet](https://github.com/AlexeyAB/darknet).
|
||||
|
||||
2. Follow the instructions provided in the README file for installation. This typically involves cloning the repository, installing necessary dependencies, and setting up any necessary environment variables.
|
||||
|
||||
3. Once installation is complete, you can train and use the model as per the usage instructions provided in the repository. This usually involves preparing your dataset, configuring the model parameters, training the model, and then using the trained model to perform object detection.
|
||||
|
||||
Please note that the specific steps may vary depending on your specific use case and the current state of the YOLOv4 repository. Therefore, it is strongly recommended to refer directly to the instructions provided in the YOLOv4 GitHub repository.
|
||||
|
||||
We regret any inconvenience this may cause and will strive to update this document with usage examples for Ultralytics once support for YOLOv4 is implemented.
|
||||
|
||||
## Conclusion
|
||||
|
||||
YOLOv4 is a powerful and efficient object detection model that strikes a balance between speed and accuracy. Its use of unique features and bag of freebies techniques during training allows it to perform excellently in real-time object detection tasks. YOLOv4 can be trained and used by anyone with a conventional GPU, making it accessible and practical for a wide range of applications including [surveillance systems](https://www.ultralytics.com/blog/shattering-the-surveillance-status-quo-with-vision-ai), [autonomous vehicles](https://www.ultralytics.com/solutions/ai-in-automotive), and [industrial automation](https://www.ultralytics.com/blog/improving-manufacturing-with-computer-vision).
|
||||
|
||||
For those looking to implement object detection in their projects, YOLOv4 remains a strong contender, especially when real-time performance is a priority. While Ultralytics currently focuses on supporting newer YOLO versions like [YOLOv8](https://docs.ultralytics.com/models/yolov8/) and [YOLO11](https://docs.ultralytics.com/models/yolo11/), the architectural innovations introduced in YOLOv4 have influenced the development of these later models.
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the YOLOv4 authors for their significant contributions in the field of real-time object detection:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{bochkovskiy2020yolov4,
|
||||
title={YOLOv4: Optimal Speed and Accuracy of Object Detection},
|
||||
author={Alexey Bochkovskiy and Chien-Yao Wang and Hong-Yuan Mark Liao},
|
||||
year={2020},
|
||||
eprint={2004.10934},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
The original YOLOv4 paper can be found on [arXiv](https://arxiv.org/abs/2004.10934). The authors have made their work publicly available, and the codebase can be accessed on [GitHub](https://github.com/AlexeyAB/darknet). We appreciate their efforts in advancing the field and making their work accessible to the broader community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is YOLOv4 and why should I use it for [object detection](https://www.ultralytics.com/glossary/object-detection)?
|
||||
|
||||
YOLOv4, which stands for "You Only Look Once version 4," is a state-of-the-art real-time object detection model developed by Alexey Bochkovskiy in 2020. It achieves an optimal balance between speed and [accuracy](https://www.ultralytics.com/glossary/accuracy), making it highly suitable for real-time applications. YOLOv4's architecture incorporates several innovative features like Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), and Self-adversarial-training (SAT), among others, to achieve state-of-the-art results. If you're looking for a high-performance model that operates efficiently on conventional GPUs, YOLOv4 is an excellent choice.
|
||||
|
||||
### How does the architecture of YOLOv4 enhance its performance?
|
||||
|
||||
The architecture of YOLOv4 includes several key components: the [backbone](https://www.ultralytics.com/glossary/backbone), the neck, and the head. The backbone, which can be models like VGG, ResNet, or CSPDarknet53, is pretrained to predict classes and bounding boxes. The neck, utilizing PANet, connects [feature maps](https://www.ultralytics.com/glossary/feature-maps) from different stages for comprehensive data extraction. Finally, the head, which uses configurations from YOLOv3, makes the final object detections. YOLOv4 also employs "bag of freebies" techniques like mosaic data augmentation and DropBlock regularization, further optimizing its speed and accuracy.
|
||||
|
||||
### What are "bag of freebies" in the context of YOLOv4?
|
||||
|
||||
"Bag of freebies" refers to methods that improve the training accuracy of YOLOv4 without increasing the cost of inference. These techniques include various forms of data augmentation like photometric distortions (adjusting brightness, contrast, etc.) and geometric distortions (scaling, cropping, flipping, rotating). By increasing the variability of the input images, these augmentations help YOLOv4 generalize better to different types of images, thereby improving its robustness and accuracy without compromising its real-time performance.
|
||||
|
||||
### Why is YOLOv4 considered suitable for real-time object detection on conventional GPUs?
|
||||
|
||||
YOLOv4 is designed to optimize both speed and accuracy, making it ideal for real-time object detection tasks that require quick and reliable performance. It operates efficiently on conventional GPUs, needing only one for both training and inference. This makes it accessible and practical for various applications ranging from [recommendation systems](https://www.ultralytics.com/glossary/recommendation-system) to standalone process management, thereby reducing the need for extensive hardware setups and making it a cost-effective solution for real-time object detection.
|
||||
|
||||
### How can I get started with YOLOv4 if Ultralytics does not currently support it?
|
||||
|
||||
To get started with YOLOv4, you should visit the official [YOLOv4 GitHub repository](https://github.com/AlexeyAB/darknet). Follow the installation instructions provided in the README file, which typically include cloning the repository, installing dependencies, and setting up environment variables. Once installed, you can train the model by preparing your dataset, configuring the model parameters, and following the usage instructions provided. Since Ultralytics does not currently support YOLOv4, it is recommended to refer directly to the YOLOv4 GitHub for the most up-to-date and detailed guidance.
|
||||
179
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov5.md
Normal file
179
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov5.md
Normal file
@@ -0,0 +1,179 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore YOLOv5u, an advanced object detection model with optimized accuracy-speed tradeoff, featuring anchor-free Ultralytics head and various pretrained models.
|
||||
keywords: YOLOv5, YOLOv5u, object detection, Ultralytics, anchor-free, pretrained models, accuracy, speed, real-time detection
|
||||
---
|
||||
|
||||
# Ultralytics YOLOv5
|
||||
|
||||
## Overview
|
||||
|
||||
YOLOv5u represents an advancement in [object detection](https://www.ultralytics.com/glossary/object-detection) methodologies. Originating from the foundational architecture of the [YOLOv5](https://github.com/ultralytics/yolov5) model developed by Ultralytics, YOLOv5u integrates the anchor-free, objectness-free split head, a feature previously introduced in the [YOLOv8](yolov8.md) models. This adaptation refines the model's architecture, leading to an improved accuracy-speed tradeoff in object detection tasks. Given the empirical results and its derived features, YOLOv5u provides an efficient alternative for those seeking robust solutions in both research and practical applications.
|
||||
|
||||
|
||||
|
||||
!!! warning "YOLOv5 models trained by [ultralytics/yolov5](https://github.com/ultralytics/yolov5) are not compatible with [ultralytics/ultralytics](https://github.com/ultralytics/ultralytics) library"
|
||||
|
||||
Ultralytics provides an [anchor-free variant of YOLOv5 model](#performance-metrics). Models trained with the [original YOLOv5 repo](https://github.com/ultralytics/yolov5) cannot be used with the Ultralytics library.
|
||||
|
||||
!!! tip "Try on Ultralytics Platform"
|
||||
|
||||
Explore and run YOLOv5 models directly on [Ultralytics Platform](https://platform.ultralytics.com/ultralytics/yolov5).
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Anchor-free Split Ultralytics Head:** Traditional object detection models rely on predefined anchor boxes to predict object locations. However, YOLOv5u modernizes this approach. By adopting an anchor-free split Ultralytics head, it ensures a more flexible and adaptive detection mechanism, consequently enhancing the performance in diverse scenarios.
|
||||
|
||||
- **Optimized Accuracy-Speed Tradeoff:** Speed and accuracy often pull in opposite directions. But YOLOv5u challenges this tradeoff. It offers a calibrated balance, ensuring real-time detections without compromising on accuracy. This feature is particularly invaluable for applications that demand swift responses, such as [autonomous vehicles](https://www.ultralytics.com/glossary/autonomous-vehicles), [robotics](https://www.ultralytics.com/glossary/robotics), and real-time video analytics.
|
||||
|
||||
- **Variety of Pretrained Models:** Understanding that different tasks require different toolsets, YOLOv5u provides a plethora of pretrained models. Whether you're focusing on Inference, Validation, or Training, there's a tailor-made model awaiting you. This variety ensures you're not just using a one-size-fits-all solution, but a model specifically fine-tuned for your unique challenge.
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
The YOLOv5u models, with various pretrained weights, excel in [Object Detection](../tasks/detect.md) tasks. They support a comprehensive range of modes, making them suitable for diverse applications, from development to deployment.
|
||||
|
||||
| Model Type | Pretrained Weights | Task | Inference | Validation | Training | Export |
|
||||
| ---------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv5u | `yolov5nu`, `yolov5su`, `yolov5mu`, `yolov5lu`, `yolov5xu`, `yolov5n6u`, `yolov5s6u`, `yolov5m6u`, `yolov5l6u`, `yolov5x6u` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides a detailed overview of the YOLOv5u model variants, highlighting their applicability in object detection tasks and support for various operational modes such as [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). This comprehensive support ensures that users can fully leverage the capabilities of YOLOv5u models in a wide range of object detection scenarios.
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv5"]'></canvas>
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
See [Detection Docs](../tasks/detect.md) for usage examples with these models trained on [COCO](../datasets/detect/coco.md), which include 80 pretrained classes.
|
||||
|
||||
| Model | YAML | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|-----------------------|----------------------|--------------------------------|-------------------------------------|--------------------|-------------------|
|
||||
| [yolov5nu.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5nu.pt) | [yolov5n.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5.yaml) | 640 | 34.3 | 73.6 | 1.06 | 2.6 | 7.7 |
|
||||
| [yolov5su.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5su.pt) | [yolov5s.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5.yaml) | 640 | 43.0 | 120.7 | 1.27 | 9.1 | 24.0 |
|
||||
| [yolov5mu.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5mu.pt) | [yolov5m.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5.yaml) | 640 | 49.0 | 233.9 | 1.86 | 25.1 | 64.2 |
|
||||
| [yolov5lu.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5lu.pt) | [yolov5l.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5.yaml) | 640 | 52.2 | 408.4 | 2.50 | 53.2 | 135.0 |
|
||||
| [yolov5xu.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5xu.pt) | [yolov5x.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5.yaml) | 640 | 53.2 | 763.2 | 3.81 | 97.2 | 246.4 |
|
||||
| | | | | | | | |
|
||||
| [yolov5n6u.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5n6u.pt) | [yolov5n6.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5-p6.yaml) | 1280 | 42.1 | 211.0 | 1.83 | 4.3 | 7.8 |
|
||||
| [yolov5s6u.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5s6u.pt) | [yolov5s6.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5-p6.yaml) | 1280 | 48.6 | 422.6 | 2.34 | 15.3 | 24.6 |
|
||||
| [yolov5m6u.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5m6u.pt) | [yolov5m6.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5-p6.yaml) | 1280 | 53.6 | 810.9 | 4.36 | 41.2 | 65.7 |
|
||||
| [yolov5l6u.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5l6u.pt) | [yolov5l6.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5-p6.yaml) | 1280 | 55.7 | 1470.9 | 5.47 | 86.1 | 137.4 |
|
||||
| [yolov5x6u.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov5x6u.pt) | [yolov5x6.yaml](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v5/yolov5-p6.yaml) | 1280 | 56.8 | 2436.5 | 8.98 | 155.4 | 250.7 |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This example provides simple YOLOv5 training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv5n model
|
||||
model = YOLO("yolov5n.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLOv5n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv5n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov5n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLOv5n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov5n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
!!! tip "Ultralytics YOLOv5 Publication"
|
||||
|
||||
Ultralytics has not published a formal research paper for YOLOv5 due to the rapidly evolving nature of the models. We focus on advancing the technology and making it easier to use, rather than producing static documentation. For the most up-to-date information on YOLO architecture, features, and usage, please refer to our [GitHub repository](https://github.com/ultralytics/ultralytics) and [documentation](https://docs.ultralytics.com/).
|
||||
|
||||
If you use YOLOv5 or YOLOv5u in your research, please cite the Ultralytics YOLOv5 repository as follows:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@software{yolov5,
|
||||
title = {Ultralytics YOLOv5},
|
||||
author = {Glenn Jocher},
|
||||
year = {2020},
|
||||
version = {7.0},
|
||||
license = {AGPL-3.0},
|
||||
url = {https://github.com/ultralytics/yolov5},
|
||||
doi = {10.5281/zenodo.3908559},
|
||||
orcid = {0000-0001-5950-6979}
|
||||
}
|
||||
```
|
||||
|
||||
Please note that YOLOv5 models are provided under [AGPL-3.0](https://github.com/ultralytics/ultralytics/blob/main/LICENSE) and [Enterprise](https://www.ultralytics.com/license) licenses.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is Ultralytics YOLOv5u and how does it differ from YOLOv5?
|
||||
|
||||
Ultralytics YOLOv5u is an advanced version of YOLOv5, integrating the anchor-free, objectness-free split head that enhances the [accuracy](https://www.ultralytics.com/glossary/accuracy)-speed tradeoff for real-time object detection tasks. Unlike the traditional YOLOv5, YOLOv5u adopts an anchor-free detection mechanism, making it more flexible and adaptive in diverse scenarios. For more detailed information on its features, you can refer to the [YOLOv5 Overview](#overview).
|
||||
|
||||
### How does the anchor-free Ultralytics head improve object detection performance in YOLOv5u?
|
||||
|
||||
The anchor-free Ultralytics head in YOLOv5u improves object detection performance by eliminating the dependency on predefined anchor boxes. This results in a more flexible and adaptive detection mechanism that can handle various object sizes and shapes with greater efficiency. This enhancement directly contributes to a balanced tradeoff between accuracy and speed, making YOLOv5u suitable for real-time applications. Learn more about its architecture in the [Key Features](#key-features) section.
|
||||
|
||||
### Can I use pretrained YOLOv5u models for different tasks and modes?
|
||||
|
||||
Yes, you can use pretrained YOLOv5u models for various tasks such as [Object Detection](../tasks/detect.md). These models support multiple modes, including [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). This flexibility allows users to leverage the capabilities of YOLOv5u models across different operational requirements. For a detailed overview, check the [Supported Tasks and Modes](#supported-tasks-and-modes) section.
|
||||
|
||||
### How do the performance metrics of YOLOv5u models compare on different platforms?
|
||||
|
||||
The performance metrics of YOLOv5u models vary depending on the platform and hardware used. For example, the YOLOv5nu model achieves a 34.3 mAP on COCO dataset with a speed of 73.6 ms on CPU (ONNX) and 1.06 ms on A100 TensorRT. Detailed performance metrics for different YOLOv5u models can be found in the [Performance Metrics](#performance-metrics) section, which provides a comprehensive comparison across various devices.
|
||||
|
||||
### How can I train a YOLOv5u model using the Ultralytics Python API?
|
||||
|
||||
You can train a YOLOv5u model by loading a pretrained model and running the training command with your dataset. Here's a quick example:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv5n model
|
||||
model = YOLO("yolov5n.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv5n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov5n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For more detailed instructions, visit the [Usage Examples](#usage-examples) section.
|
||||
167
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov6.md
Normal file
167
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov6.md
Normal file
@@ -0,0 +1,167 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore Meituan YOLOv6, a top-tier object detector balancing speed and accuracy. Learn about its unique features and performance metrics on Ultralytics Docs.
|
||||
keywords: Meituan YOLOv6, object detection, real-time applications, BiC module, Anchor-Aided Training, COCO dataset, high-performance models, Ultralytics Docs
|
||||
---
|
||||
|
||||
# Meituan YOLOv6
|
||||
|
||||
## Overview
|
||||
|
||||
[Meituan](https://www.meituan.com/) YOLOv6, released in 2022, offers a strong balance between speed and accuracy, making it a popular choice for real-time applications. This model introduces several notable enhancements on its architecture and training scheme, including the implementation of a Bi-directional Concatenation (BiC) module, an anchor-aided training (AAT) strategy, and an improved [backbone](https://www.ultralytics.com/glossary/backbone) and neck design for high accuracy on the [COCO dataset](https://docs.ultralytics.com/datasets/detect/coco/).
|
||||
|
||||
|
||||
 **Overview of YOLOv6.** Model architecture diagram showing the redesigned network components and training strategies that have led to significant performance improvements. (a) The neck of YOLOv6 (N and S are shown). Note for M/L, RepBlocks is replaced with CSPStackRep. (b) The structure of a BiC module. (c) A SimCSPSPPF block. ([source](https://arxiv.org/pdf/2301.05586)).
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Bidirectional Concatenation (BiC) Module:** YOLOv6 introduces a BiC module in the neck of the detector, enhancing localization signals and delivering performance gains with negligible speed degradation.
|
||||
- **Anchor-Aided Training (AAT) Strategy:** This model proposes AAT to enjoy the benefits of both [anchor-based](https://www.ultralytics.com/glossary/anchor-based-detectors) and [anchor-free](https://www.ultralytics.com/glossary/anchor-free-detectors) paradigms without compromising inference efficiency.
|
||||
- **Enhanced Backbone and Neck Design:** By deepening YOLOv6 to include another stage in the backbone and neck, this model achieved strong performance on the COCO dataset at high-resolution input when released.
|
||||
- **Self-Distillation Strategy:** A new self-distillation strategy is implemented to boost the performance of smaller models of YOLOv6, enhancing the auxiliary regression branch during training and removing it at inference to avoid a marked speed decline.
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv6-3.0"]'></canvas>
|
||||
|
||||
YOLOv6 provides various pretrained models with different scales:
|
||||
|
||||
- YOLOv6-N: 37.5% AP on COCO val2017 at 1187 FPS with NVIDIA T4 GPU.
|
||||
- YOLOv6-S: 45.0% AP at 484 FPS.
|
||||
- YOLOv6-M: 50.0% AP at 226 FPS.
|
||||
- YOLOv6-L: 52.8% AP at 116 FPS.
|
||||
- YOLOv6-L6: State-of-the-art accuracy in real-time.
|
||||
|
||||
YOLOv6 also provides quantized models for different [precisions](https://www.ultralytics.com/glossary/precision) and models optimized for mobile platforms.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This example provides simple YOLOv6 training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
YOLOv6 `*.yaml` files can be passed to the `YOLO()` class to build the corresponding model in Python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Build a YOLOv6n model from scratch
|
||||
model = YOLO("yolov6n.yaml")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLOv6n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Build a YOLOv6n model from scratch and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov6n.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Build a YOLOv6n model from scratch and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov6n.yaml source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
The YOLOv6 series offers a range of models, each optimized for high-performance [Object Detection](../tasks/detect.md). These models cater to varying computational needs and [accuracy](https://www.ultralytics.com/glossary/accuracy) requirements, making them versatile for a wide array of applications.
|
||||
|
||||
| Model | Filenames | Tasks | Inference | Validation | Training | Export |
|
||||
| -------- | -------------- | -------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv6-N | `yolov6n.yaml` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv6-S | `yolov6s.yaml` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv6-M | `yolov6m.yaml` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv6-L | `yolov6l.yaml` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv6-X | `yolov6x.yaml` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides a detailed overview of the YOLOv6 model variants, highlighting their capabilities in [object detection](https://www.ultralytics.com/glossary/object-detection) tasks and their compatibility with various operational modes such as [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). This comprehensive support ensures that users can fully leverage the capabilities of YOLOv6 models in a broad range of object detection scenarios.
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the authors for their significant contributions in the field of real-time object detection:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@misc{li2023yolov6,
|
||||
title={YOLOv6 v3.0: A Full-Scale Reloading},
|
||||
author={Chuyi Li and Lulu Li and Yifei Geng and Hongliang Jiang and Meng Cheng and Bo Zhang and Zaidan Ke and Xiaoming Xu and Xiangxiang Chu},
|
||||
year={2023},
|
||||
eprint={2301.05586},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
The original YOLOv6 paper can be found on [arXiv](https://arxiv.org/abs/2301.05586). The authors have made their work publicly available, and the codebase can be accessed on [GitHub](https://github.com/meituan/YOLOv6). We appreciate their efforts in advancing the field and making their work accessible to the broader community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is Meituan YOLOv6 and what makes it unique?
|
||||
|
||||
Meituan YOLOv6, released in 2022, is an object detector that balances speed and accuracy, designed for real-time applications. It features notable architectural enhancements like the Bi-directional Concatenation (BiC) module and an Anchor-Aided Training (AAT) strategy. These innovations provide substantial performance gains with minimal speed degradation, making YOLOv6 a competitive choice for object detection tasks.
|
||||
|
||||
### How does the Bi-directional Concatenation (BiC) Module in YOLOv6 improve performance?
|
||||
|
||||
The Bi-directional Concatenation (BiC) module in YOLOv6 enhances localization signals in the detector's neck, delivering performance improvements with negligible speed impact. This module effectively combines different [feature maps](https://www.ultralytics.com/glossary/feature-maps), increasing the model's ability to detect objects accurately. For more details on YOLOv6's features, refer to the [Key Features](#key-features) section.
|
||||
|
||||
### How can I train a YOLOv6 model using Ultralytics?
|
||||
|
||||
You can train a YOLOv6 model using Ultralytics with simple Python or CLI commands. For instance:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Build a YOLOv6n model from scratch
|
||||
model = YOLO("yolov6n.yaml")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo train model=yolov6n.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For more information, visit the [Train](../modes/train.md) page.
|
||||
|
||||
### What are the different versions of YOLOv6 and their performance metrics?
|
||||
|
||||
YOLOv6 offers multiple versions, each optimized for different performance requirements:
|
||||
|
||||
- YOLOv6-N: 37.5% AP at 1187 FPS
|
||||
- YOLOv6-S: 45.0% AP at 484 FPS
|
||||
- YOLOv6-M: 50.0% AP at 226 FPS
|
||||
- YOLOv6-L: 52.8% AP at 116 FPS
|
||||
- YOLOv6-L6: State-of-the-art accuracy in real-time scenarios
|
||||
|
||||
These models are evaluated on the COCO dataset using an NVIDIA T4 GPU. For more on performance metrics, see the [Performance Metrics](#performance-metrics) section.
|
||||
|
||||
### How does the Anchor-Aided Training (AAT) strategy benefit YOLOv6?
|
||||
|
||||
Anchor-Aided Training (AAT) in YOLOv6 combines elements of anchor-based and anchor-free approaches, enhancing the model's detection capabilities without compromising inference efficiency. This strategy leverages anchors during training to improve [bounding box](https://www.ultralytics.com/glossary/bounding-box) predictions, making YOLOv6 effective in diverse object detection tasks.
|
||||
|
||||
### Which operational modes are supported by YOLOv6 models in Ultralytics?
|
||||
|
||||
YOLOv6 supports various operational modes including Inference, Validation, Training, and Export. This flexibility allows users to fully exploit the model's capabilities in different scenarios. Check out the [Supported Tasks and Modes](#supported-tasks-and-modes) section for a detailed overview of each mode.
|
||||
410
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov7.md
Normal file
410
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov7.md
Normal file
@@ -0,0 +1,410 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover YOLOv7, the breakthrough real-time object detector with top speed and accuracy. Learn about key features, usage, and performance metrics.
|
||||
keywords: YOLOv7, real-time object detection, Ultralytics, AI, computer vision, model training, object detector
|
||||
---
|
||||
|
||||
# YOLOv7: Trainable Bag-of-Freebies
|
||||
|
||||
YOLOv7, released in July 2022, was a significant advancement in real-time object detection at its time of release. It achieved 56.8% AP on GPU V100, setting new benchmarks when introduced. YOLOv7 outperformed contemporary object detectors such as YOLOR, YOLOX, Scaled-YOLOv4, and YOLOv5 in speed and [accuracy](https://www.ultralytics.com/glossary/accuracy). The model is trained on the MS COCO dataset from scratch without using any other datasets or pretrained weights. Source code for YOLOv7 is available on GitHub. Note that newer models like [YOLO11](yolo11.md) and [YOLO26](yolo26.md) have since achieved higher accuracy with improved efficiency.
|
||||
|
||||
|
||||
|
||||
## Comparison of SOTA object detectors
|
||||
|
||||
From the results in the YOLO comparison table we know that the proposed method has the best speed-accuracy trade-off comprehensively. If we compare YOLOv7-tiny-SiLU with YOLOv5-N (r6.1), our method is 127 fps faster and 10.7% more accurate on AP. In addition, YOLOv7 has 51.4% AP at frame rate of 161 fps, while PPYOLOE-L with the same AP has only 78 fps frame rate. In terms of parameter usage, YOLOv7 is 41% less than PPYOLOE-L.
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv7"]'></canvas>
|
||||
|
||||
If we compare YOLOv7-X with 114 fps inference speed to YOLOv5-L (r6.1) with 99 fps inference speed, YOLOv7-X can improve AP by 3.9%. If YOLOv7-X is compared with YOLOv5-X (r6.1) of similar scale, the inference speed of YOLOv7-X is 31 fps faster. In addition, in terms the amount of parameters and computation, YOLOv7-X reduces 22% of parameters and 8% of computation compared to YOLOv5-X (r6.1), but improves AP by 2.2% ([Source](https://arxiv.org/pdf/2207.02696)).
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
| Model | Params<br><sup>(M)</sup> | FLOPs<br><sup>(G)</sup> | Size<br><sup>(pixels)</sup> | FPS | AP<sup>test / val<br>50-95</sup> | AP<sup>test<br>50</sup> | AP<sup>test<br>75</sup> | AP<sup>test<br>S</sup> | AP<sup>test<br>M</sup> | AP<sup>test<br>L</sup> |
|
||||
| --------------------- | ------------------ | ----------------- | --------------------- | ------- | -------------------------- | ----------------- | ----------------- | ---------------- | ---------------- | ---------------- |
|
||||
| [YOLOX-S][1] | **9.0** | **26.8** | 640 | **102** | 40.5% / 40.5% | - | - | - | - | - |
|
||||
| [YOLOX-M][1] | 25.3 | 73.8 | 640 | 81 | 47.2% / 46.9% | - | - | - | - | - |
|
||||
| [YOLOX-L][1] | 54.2 | 155.6 | 640 | 69 | 50.1% / 49.7% | - | - | - | - | - |
|
||||
| [YOLOX-X][1] | 99.1 | 281.9 | 640 | 58 | **51.5% / 51.1%** | - | - | - | - | - |
|
||||
| | | | | | | | | | | |
|
||||
| [PPYOLOE-S][2] | **7.9** | **17.4** | 640 | **208** | 43.1% / 42.7% | 60.5% | 46.6% | 23.2% | 46.4% | 56.9% |
|
||||
| [PPYOLOE-M][2] | 23.4 | 49.9 | 640 | 123 | 48.9% / 48.6% | 66.5% | 53.0% | 28.6% | 52.9% | 63.8% |
|
||||
| [PPYOLOE-L][2] | 52.2 | 110.1 | 640 | 78 | 51.4% / 50.9% | 68.9% | 55.6% | 31.4% | 55.3% | 66.1% |
|
||||
| [PPYOLOE-X][2] | 98.4 | 206.6 | 640 | 45 | **52.2% / 51.9%** | **69.9%** | **56.5%** | **33.3%** | **56.3%** | **66.4%** |
|
||||
| | | | | | | | | | | |
|
||||
| [YOLOv5-N (r6.1)][3] | **1.9** | **4.5** | 640 | **159** | - / 28.0% | - | - | - | - | - |
|
||||
| [YOLOv5-S (r6.1)][3] | 7.2 | 16.5 | 640 | 156 | - / 37.4% | - | - | - | - | - |
|
||||
| [YOLOv5-M (r6.1)][3] | 21.2 | 49.0 | 640 | 122 | - / 45.4% | - | - | - | - | - |
|
||||
| [YOLOv5-L (r6.1)][3] | 46.5 | 109.1 | 640 | 99 | - / 49.0% | - | - | - | - | - |
|
||||
| [YOLOv5-X (r6.1)][3] | 86.7 | 205.7 | 640 | 83 | - / **50.7%** | - | - | - | - | - |
|
||||
| | | | | | | | | | | |
|
||||
| [YOLOR-CSP][4] | 52.9 | 120.4 | 640 | 106 | 51.1% / 50.8% | 69.6% | 55.7% | 31.7% | 55.3% | 64.7% |
|
||||
| [YOLOR-CSP-X][4] | 96.9 | 226.8 | 640 | 87 | 53.0% / 52.7% | 71.4% | 57.9% | 33.7% | 57.1% | 66.8% |
|
||||
| [YOLOv7-tiny-SiLU][5] | **6.2** | **13.8** | 640 | **286** | 38.7% / 38.7% | 56.7% | 41.7% | 18.8% | 42.4% | 51.9% |
|
||||
| [YOLOv7][5] | 36.9 | 104.7 | 640 | 161 | 51.4% / 51.2% | 69.7% | 55.9% | 31.8% | 55.5% | 65.0% |
|
||||
| [YOLOv7-X][5] | 71.3 | 189.9 | 640 | 114 | **53.1% / 52.9%** | **71.2%** | **57.8%** | **33.8%** | **57.1%** | **67.4%** |
|
||||
| | | | | | | | | | | |
|
||||
| [YOLOv5-N6 (r6.1)][3] | **3.2** | **18.4** | 1280 | **123** | - / 36.0% | - | - | - | - | - |
|
||||
| [YOLOv5-S6 (r6.1)][3] | 12.6 | 67.2 | 1280 | 122 | - / 44.8% | - | - | - | - | - |
|
||||
| [YOLOv5-M6 (r6.1)][3] | 35.7 | 200.0 | 1280 | 90 | - / 51.3% | - | - | - | - | - |
|
||||
| [YOLOv5-L6 (r6.1)][3] | 76.8 | 445.6 | 1280 | 63 | - / 53.7% | - | - | - | - | - |
|
||||
| [YOLOv5-X6 (r6.1)][3] | 140.7 | 839.2 | 1280 | 38 | - / **55.0%** | - | - | - | - | - |
|
||||
| | | | | | | | | | | |
|
||||
| [YOLOR-P6][4] | **37.2** | **325.6** | 1280 | **76** | 53.9% / 53.5% | 71.4% | 58.9% | 36.1% | 57.7% | 65.6% |
|
||||
| [YOLOR-W6][4] | 79.8 | 453.2 | 1280 | 66 | 55.2% / 54.8% | 72.7% | 60.5% | 37.7% | 59.1% | 67.1% |
|
||||
| [YOLOR-E6][4] | 115.8 | 683.2 | 1280 | 45 | 55.8% / 55.7% | 73.4% | 61.1% | 38.4% | 59.7% | 67.7% |
|
||||
| [YOLOR-D6][4] | 151.7 | 935.6 | 1280 | 34 | **56.5% / 56.1%** | **74.1%** | **61.9%** | **38.9%** | **60.4%** | **68.7%** |
|
||||
| | | | | | | | | | | |
|
||||
| [YOLOv7-W6][5] | **70.4** | **360.0** | 1280 | **84** | 54.9% / 54.6% | 72.6% | 60.1% | 37.3% | 58.7% | 67.1% |
|
||||
| [YOLOv7-E6][5] | 97.2 | 515.2 | 1280 | 56 | 56.0% / 55.9% | 73.5% | 61.2% | 38.0% | 59.9% | 68.4% |
|
||||
| [YOLOv7-D6][5] | 154.7 | 806.8 | 1280 | 44 | 56.6% / 56.3% | 74.0% | 61.8% | 38.8% | 60.1% | 69.5% |
|
||||
| [YOLOv7-E6E][5] | 151.7 | 843.2 | 1280 | 36 | **56.8% / 56.8%** | **74.4%** | **62.1%** | **39.3%** | **60.5%** | **69.0%** |
|
||||
|
||||
[1]: https://github.com/Megvii-BaseDetection/YOLOX
|
||||
[2]: https://github.com/PaddlePaddle/PaddleDetection
|
||||
[3]: https://github.com/ultralytics/yolov5
|
||||
[4]: https://github.com/WongKinYiu/yolor
|
||||
[5]: https://github.com/WongKinYiu/yolov7
|
||||
|
||||
## Overview
|
||||
|
||||
Real-time object detection is an important component in many [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) systems, including multi-[object tracking](https://www.ultralytics.com/glossary/object-tracking), autonomous driving, [robotics](https://www.ultralytics.com/glossary/robotics), and [medical image analysis](https://www.ultralytics.com/glossary/medical-image-analysis). In recent years, real-time object detection development has focused on designing efficient architectures and improving the inference speed of various CPUs, GPUs, and neural processing units (NPUs). YOLOv7 supports both mobile GPU and GPU devices, from the edge to the cloud.
|
||||
|
||||
Unlike traditional real-time object detectors that focus on architecture optimization, YOLOv7 introduces a focus on the optimization of the training process. This includes modules and optimization methods designed to improve the accuracy of object detection without increasing the inference cost, a concept known as the "trainable bag-of-freebies".
|
||||
|
||||
## Key Features
|
||||
|
||||
YOLOv7 introduces several key features:
|
||||
|
||||
1. **Model Re-parameterization**: YOLOv7 proposes a planned re-parameterized model, which is a strategy applicable to layers in different networks with the concept of gradient propagation path.
|
||||
|
||||
2. **Dynamic Label Assignment**: The training of the model with multiple output layers presents a new issue: "How to assign dynamic targets for the outputs of different branches?" To solve this problem, YOLOv7 introduces a new label assignment method called coarse-to-fine lead guided label assignment.
|
||||
|
||||
3. **Extended and Compound Scaling**: YOLOv7 proposes "extend" and "compound scaling" methods for the real-time object detector that can effectively utilize parameters and computation.
|
||||
|
||||
4. **Efficiency**: The method proposed by YOLOv7 can effectively reduce about 40% parameters and 50% computation of state-of-the-art real-time object detector, and has faster inference speed and higher detection accuracy.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
As of the time of writing, Ultralytics only supports ONNX and TensorRT inference for YOLOv7.
|
||||
|
||||
### ONNX Export
|
||||
|
||||
To use YOLOv7 ONNX model with Ultralytics:
|
||||
|
||||
1. (Optional) Install Ultralytics and export an ONNX model to have the required dependencies automatically installed:
|
||||
|
||||
```bash
|
||||
pip install ultralytics
|
||||
yolo export model=yolo26n.pt format=onnx
|
||||
```
|
||||
|
||||
2. Export the desired YOLOv7 model by using the exporter in the [YOLOv7 repo](https://github.com/WongKinYiu/yolov7):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/WongKinYiu/yolov7
|
||||
cd yolov7
|
||||
python export.py --weights yolov7-tiny.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --max-wh 640
|
||||
```
|
||||
|
||||
3. Modify the ONNX model graph to be compatible with Ultralytics using the following script:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import onnx
|
||||
from onnx import helper, numpy_helper
|
||||
|
||||
# Load the ONNX model
|
||||
model_path = "yolov7/yolov7-tiny.onnx" # Replace with your model path
|
||||
model = onnx.load(model_path)
|
||||
graph = model.graph
|
||||
|
||||
# Fix input shape to batch size 1
|
||||
input_shape = graph.input[0].type.tensor_type.shape
|
||||
input_shape.dim[0].dim_value = 1
|
||||
|
||||
# Define the output of the original model
|
||||
original_output_name = graph.output[0].name
|
||||
|
||||
# Create slicing nodes
|
||||
sliced_output_name = f"{original_output_name}_sliced"
|
||||
|
||||
# Define initializers for slicing (remove the first value)
|
||||
start = numpy_helper.from_array(np.array([1], dtype=np.int64), name="slice_start")
|
||||
end = numpy_helper.from_array(np.array([7], dtype=np.int64), name="slice_end")
|
||||
axes = numpy_helper.from_array(np.array([1], dtype=np.int64), name="slice_axes")
|
||||
steps = numpy_helper.from_array(np.array([1], dtype=np.int64), name="slice_steps")
|
||||
|
||||
graph.initializer.extend([start, end, axes, steps])
|
||||
|
||||
slice_node = helper.make_node(
|
||||
"Slice",
|
||||
inputs=[original_output_name, "slice_start", "slice_end", "slice_axes", "slice_steps"],
|
||||
outputs=[sliced_output_name],
|
||||
name="SliceNode",
|
||||
)
|
||||
graph.node.append(slice_node)
|
||||
|
||||
# Define segment slicing
|
||||
seg1_start = numpy_helper.from_array(np.array([0], dtype=np.int64), name="seg1_start")
|
||||
seg1_end = numpy_helper.from_array(np.array([4], dtype=np.int64), name="seg1_end")
|
||||
seg2_start = numpy_helper.from_array(np.array([4], dtype=np.int64), name="seg2_start")
|
||||
seg2_end = numpy_helper.from_array(np.array([5], dtype=np.int64), name="seg2_end")
|
||||
seg3_start = numpy_helper.from_array(np.array([5], dtype=np.int64), name="seg3_start")
|
||||
seg3_end = numpy_helper.from_array(np.array([6], dtype=np.int64), name="seg3_end")
|
||||
|
||||
graph.initializer.extend([seg1_start, seg1_end, seg2_start, seg2_end, seg3_start, seg3_end])
|
||||
|
||||
# Create intermediate tensors for segments
|
||||
segment_1_name = f"{sliced_output_name}_segment1"
|
||||
segment_2_name = f"{sliced_output_name}_segment2"
|
||||
segment_3_name = f"{sliced_output_name}_segment3"
|
||||
|
||||
# Add segment slicing nodes
|
||||
graph.node.extend(
|
||||
[
|
||||
helper.make_node(
|
||||
"Slice",
|
||||
inputs=[sliced_output_name, "seg1_start", "seg1_end", "slice_axes", "slice_steps"],
|
||||
outputs=[segment_1_name],
|
||||
name="SliceSegment1",
|
||||
),
|
||||
helper.make_node(
|
||||
"Slice",
|
||||
inputs=[sliced_output_name, "seg2_start", "seg2_end", "slice_axes", "slice_steps"],
|
||||
outputs=[segment_2_name],
|
||||
name="SliceSegment2",
|
||||
),
|
||||
helper.make_node(
|
||||
"Slice",
|
||||
inputs=[sliced_output_name, "seg3_start", "seg3_end", "slice_axes", "slice_steps"],
|
||||
outputs=[segment_3_name],
|
||||
name="SliceSegment3",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
# Concatenate the segments
|
||||
concat_output_name = f"{sliced_output_name}_concat"
|
||||
concat_node = helper.make_node(
|
||||
"Concat",
|
||||
inputs=[segment_1_name, segment_3_name, segment_2_name],
|
||||
outputs=[concat_output_name],
|
||||
axis=1,
|
||||
name="ConcatSwapped",
|
||||
)
|
||||
graph.node.append(concat_node)
|
||||
|
||||
# Reshape to [1, -1, 6]
|
||||
reshape_shape = numpy_helper.from_array(np.array([1, -1, 6], dtype=np.int64), name="reshape_shape")
|
||||
graph.initializer.append(reshape_shape)
|
||||
|
||||
final_output_name = f"{concat_output_name}_batched"
|
||||
reshape_node = helper.make_node(
|
||||
"Reshape",
|
||||
inputs=[concat_output_name, "reshape_shape"],
|
||||
outputs=[final_output_name],
|
||||
name="AddBatchDimension",
|
||||
)
|
||||
graph.node.append(reshape_node)
|
||||
|
||||
# Get the shape of the reshaped tensor
|
||||
shape_node_name = f"{final_output_name}_shape"
|
||||
shape_node = helper.make_node(
|
||||
"Shape",
|
||||
inputs=[final_output_name],
|
||||
outputs=[shape_node_name],
|
||||
name="GetShapeDim",
|
||||
)
|
||||
graph.node.append(shape_node)
|
||||
|
||||
# Extract the second dimension
|
||||
dim_1_index = numpy_helper.from_array(np.array([1], dtype=np.int64), name="dim_1_index")
|
||||
graph.initializer.append(dim_1_index)
|
||||
|
||||
second_dim_name = f"{final_output_name}_dim1"
|
||||
gather_node = helper.make_node(
|
||||
"Gather",
|
||||
inputs=[shape_node_name, "dim_1_index"],
|
||||
outputs=[second_dim_name],
|
||||
name="GatherSecondDim",
|
||||
)
|
||||
graph.node.append(gather_node)
|
||||
|
||||
# Subtract from 100 to determine how many values to pad
|
||||
target_size = numpy_helper.from_array(np.array([100], dtype=np.int64), name="target_size")
|
||||
graph.initializer.append(target_size)
|
||||
|
||||
pad_size_name = f"{second_dim_name}_padsize"
|
||||
sub_node = helper.make_node(
|
||||
"Sub",
|
||||
inputs=["target_size", second_dim_name],
|
||||
outputs=[pad_size_name],
|
||||
name="CalculatePadSize",
|
||||
)
|
||||
graph.node.append(sub_node)
|
||||
|
||||
# Build the [2, 3] pad array:
|
||||
# 1st row -> [0, 0, 0] (no padding at the start of any dim)
|
||||
# 2nd row -> [0, pad_size, 0] (pad only at the end of the second dim)
|
||||
pad_starts = numpy_helper.from_array(np.array([0, 0, 0], dtype=np.int64), name="pad_starts")
|
||||
graph.initializer.append(pad_starts)
|
||||
|
||||
zero_scalar = numpy_helper.from_array(np.array([0], dtype=np.int64), name="zero_scalar")
|
||||
graph.initializer.append(zero_scalar)
|
||||
|
||||
pad_ends_name = "pad_ends"
|
||||
concat_pad_ends_node = helper.make_node(
|
||||
"Concat",
|
||||
inputs=["zero_scalar", pad_size_name, "zero_scalar"],
|
||||
outputs=[pad_ends_name],
|
||||
axis=0,
|
||||
name="ConcatPadEnds",
|
||||
)
|
||||
graph.node.append(concat_pad_ends_node)
|
||||
|
||||
pad_values_name = "pad_values"
|
||||
concat_pad_node = helper.make_node(
|
||||
"Concat",
|
||||
inputs=["pad_starts", pad_ends_name],
|
||||
outputs=[pad_values_name],
|
||||
axis=0,
|
||||
name="ConcatPadStartsEnds",
|
||||
)
|
||||
graph.node.append(concat_pad_node)
|
||||
|
||||
# Create Pad operator to pad with zeros
|
||||
pad_output_name = f"{final_output_name}_padded"
|
||||
pad_constant_value = numpy_helper.from_array(
|
||||
np.array([0.0], dtype=np.float32),
|
||||
name="pad_constant_value",
|
||||
)
|
||||
graph.initializer.append(pad_constant_value)
|
||||
|
||||
pad_node = helper.make_node(
|
||||
"Pad",
|
||||
inputs=[final_output_name, pad_values_name, "pad_constant_value"],
|
||||
outputs=[pad_output_name],
|
||||
mode="constant",
|
||||
name="PadToFixedSize",
|
||||
)
|
||||
graph.node.append(pad_node)
|
||||
|
||||
# Update the graph's final output to [1, 100, 6]
|
||||
new_output_type = onnx.helper.make_tensor_type_proto(
|
||||
elem_type=graph.output[0].type.tensor_type.elem_type, shape=[1, 100, 6]
|
||||
)
|
||||
new_output = onnx.helper.make_value_info(name=pad_output_name, type_proto=new_output_type)
|
||||
|
||||
# Replace the old output with the new one
|
||||
graph.output.pop()
|
||||
graph.output.extend([new_output])
|
||||
|
||||
# Save the modified model
|
||||
onnx.save(model, "yolov7-ultralytics.onnx")
|
||||
```
|
||||
|
||||
4. You can then load the modified ONNX model and run inference with it in Ultralytics normally:
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, YOLO
|
||||
|
||||
model = YOLO("yolov7-ultralytics.onnx", task="detect")
|
||||
|
||||
results = model(ASSETS / "bus.jpg")
|
||||
```
|
||||
|
||||
### TensorRT Export
|
||||
|
||||
1. Follow steps 1-2 in the [ONNX Export](#onnx-export) section.
|
||||
|
||||
2. Install the `TensorRT` Python package:
|
||||
|
||||
```bash
|
||||
pip install tensorrt
|
||||
```
|
||||
|
||||
3. Run the following script to convert the modified ONNX model to TensorRT engine:
|
||||
|
||||
```python
|
||||
from ultralytics.utils.export import export_engine
|
||||
|
||||
export_engine("yolov7-ultralytics.onnx", half=True)
|
||||
```
|
||||
|
||||
4. Load and run the model in Ultralytics:
|
||||
|
||||
```python
|
||||
from ultralytics import ASSETS, YOLO
|
||||
|
||||
model = YOLO("yolov7-ultralytics.engine", task="detect")
|
||||
|
||||
results = model(ASSETS / "bus.jpg")
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the YOLOv7 authors for their significant contributions in the field of real-time object detection:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{wang2022yolov7,
|
||||
title={YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors},
|
||||
author={Wang, Chien-Yao and Bochkovskiy, Alexey and Liao, Hong-Yuan Mark},
|
||||
journal={arXiv preprint arXiv:2207.02696},
|
||||
year={2022}
|
||||
}
|
||||
```
|
||||
|
||||
The original YOLOv7 paper can be found on [arXiv](https://arxiv.org/pdf/2207.02696). The authors have made their work publicly available, and the codebase can be accessed on [GitHub](https://github.com/WongKinYiu/yolov7). We appreciate their efforts in advancing the field and making their work accessible to the broader community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is YOLOv7 and why is it considered a breakthrough in real-time [object detection](https://www.ultralytics.com/glossary/object-detection)?
|
||||
|
||||
YOLOv7, released in July 2022, was a significant real-time object detection model that achieved excellent speed and accuracy at its time of release. It surpassed contemporary models such as YOLOX, YOLOv5, and PPYOLOE in both parameters usage and inference speed. YOLOv7's distinguishing features include its model re-parameterization and dynamic label assignment, which optimize its performance without increasing inference costs. For more technical details about its architecture and comparison metrics with other state-of-the-art object detectors, refer to the [YOLOv7 paper](https://arxiv.org/pdf/2207.02696).
|
||||
|
||||
### How does YOLOv7 improve on previous YOLO models like YOLOv4 and YOLOv5?
|
||||
|
||||
YOLOv7 introduces several innovations, including model re-parameterization and dynamic label assignment, which enhance the training process and improve inference accuracy. Compared to YOLOv5, YOLOv7 significantly boosts speed and accuracy. For instance, YOLOv7-X improves accuracy by 2.2% and reduces parameters by 22% compared to YOLOv5-X. Detailed comparisons can be found in the performance table [YOLOv7 comparison with SOTA object detectors](#comparison-of-sota-object-detectors).
|
||||
|
||||
### Can I use YOLOv7 with Ultralytics tools and platforms?
|
||||
|
||||
As of now, Ultralytics only supports YOLOv7 ONNX and TensorRT inference. To run the ONNX and TensorRT exported version of YOLOv7 with Ultralytics, check the [Usage Examples](#usage-examples) section.
|
||||
|
||||
### How do I train a custom YOLOv7 model using my dataset?
|
||||
|
||||
To install and train a custom YOLOv7 model, follow these steps:
|
||||
|
||||
1. Clone the YOLOv7 repository:
|
||||
```bash
|
||||
git clone https://github.com/WongKinYiu/yolov7
|
||||
```
|
||||
2. Navigate to the cloned directory and install dependencies:
|
||||
```bash
|
||||
cd yolov7
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
3. Prepare your dataset and configure the model parameters according to the [usage instructions](https://github.com/WongKinYiu/yolov7) provided in the repository.
|
||||
For further guidance, visit the YOLOv7 GitHub repository for the latest information and updates.
|
||||
|
||||
4. After training, you can export the model to ONNX or TensorRT for use in Ultralytics as shown in [Usage Examples](#usage-examples).
|
||||
|
||||
### What are the key features and optimizations introduced in YOLOv7?
|
||||
|
||||
YOLOv7 offers several key features that revolutionize real-time object detection:
|
||||
|
||||
- **Model Re-parameterization**: Enhances the model's performance by optimizing gradient propagation paths.
|
||||
- **Dynamic Label Assignment**: Uses a coarse-to-fine lead guided method to assign dynamic targets for outputs across different branches, improving accuracy.
|
||||
- **Extended and Compound Scaling**: Efficiently utilizes parameters and computation to scale the model for various real-time applications.
|
||||
- **Efficiency**: Reduces parameter count by 40% and computation by 50% compared to other state-of-the-art models while achieving faster inference speeds.
|
||||
|
||||
For further details on these features, see the [YOLOv7 Overview](#overview) section.
|
||||
262
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov8.md
Normal file
262
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov8.md
Normal file
@@ -0,0 +1,262 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover Ultralytics YOLOv8, an advancement in real-time object detection, optimizing performance with an array of pretrained models for diverse tasks.
|
||||
keywords: YOLOv8, real-time object detection, YOLO series, Ultralytics, computer vision, advanced object detection, AI, machine learning, deep learning
|
||||
---
|
||||
|
||||
# Explore Ultralytics YOLOv8
|
||||
|
||||
## Overview
|
||||
|
||||
YOLOv8 was released by Ultralytics on January 10, 2023, offering cutting-edge performance in terms of accuracy and speed. Building upon the advancements of previous YOLO versions, YOLOv8 introduced new features and optimizations that make it an ideal choice for various [object detection](https://www.ultralytics.com/blog/a-guide-to-deep-dive-into-object-detection-in-2025) tasks in a wide range of applications.
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Na0HvJ4hkk0"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Ultralytics YOLOv8 Model Overview
|
||||
</p>
|
||||
|
||||
!!! tip "Try on Ultralytics Platform"
|
||||
|
||||
Explore and run YOLOv8 models directly on [Ultralytics Platform](https://platform.ultralytics.com/ultralytics/yolov8).
|
||||
|
||||
## Key Features of YOLOv8
|
||||
|
||||
- **Advanced Backbone and Neck Architectures:** YOLOv8 employs state-of-the-art backbone and neck architectures, resulting in improved [feature extraction](https://www.ultralytics.com/glossary/feature-extraction) and [object detection](https://www.ultralytics.com/glossary/object-detection) performance.
|
||||
- **Anchor-free Split Ultralytics Head:** YOLOv8 adopts an anchor-free split Ultralytics head, which contributes to better accuracy and a more efficient detection process compared to anchor-based approaches.
|
||||
- **Optimized Accuracy-Speed Tradeoff:** With a focus on maintaining an optimal balance between accuracy and speed, YOLOv8 is suitable for real-time object detection tasks in diverse application areas.
|
||||
- **Variety of Pretrained Models:** YOLOv8 offers a range of pretrained models to cater to various tasks and performance requirements, making it easier to find the right model for your specific use case.
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
The YOLOv8 series offers a diverse range of models, each specialized for specific tasks in computer vision. These models are designed to cater to various requirements, from object detection to more complex tasks like [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), pose/keypoints detection, oriented object detection, and classification.
|
||||
|
||||
Each variant of the YOLOv8 series is optimized for its respective task, ensuring high performance and accuracy. Additionally, these models are compatible with various operational modes including [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md), facilitating their use in different stages of deployment and development.
|
||||
|
||||
| Model | Filenames | Task | Inference | Validation | Training | Export |
|
||||
| ----------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv8 | `yolov8n.pt` `yolov8s.pt` `yolov8m.pt` `yolov8l.pt` `yolov8x.pt` | [Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8-seg | `yolov8n-seg.pt` `yolov8s-seg.pt` `yolov8m-seg.pt` `yolov8l-seg.pt` `yolov8x-seg.pt` | [Instance Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8-pose | `yolov8n-pose.pt` `yolov8s-pose.pt` `yolov8m-pose.pt` `yolov8l-pose.pt` `yolov8x-pose.pt` `yolov8x-pose-p6.pt` | [Pose/Keypoints](../tasks/pose.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8-obb | `yolov8n-obb.pt` `yolov8s-obb.pt` `yolov8m-obb.pt` `yolov8l-obb.pt` `yolov8x-obb.pt` | [Oriented Detection](../tasks/obb.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv8-cls | `yolov8n-cls.pt` `yolov8s-cls.pt` `yolov8m-cls.pt` `yolov8l-cls.pt` `yolov8x-cls.pt` | [Classification](../tasks/classify.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides an overview of the YOLOv8 model variants, highlighting their applicability in specific tasks and their compatibility with various operational modes such as Inference, Validation, Training, and Export. It showcases the versatility and robustness of the YOLOv8 series, making them suitable for a variety of applications in [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv).
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv8"]'></canvas>
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
See [Detection Docs](../tasks/detect.md) for usage examples with these models trained on [COCO](../datasets/detect/coco.md), which include 80 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| ------------------------------------------------------------------------------------ | --------------------- | -------------------- | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n.pt) | 640 | 37.3 | 80.4 | 0.99 | 3.2 | 8.7 |
|
||||
| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s.pt) | 640 | 44.9 | 128.4 | 1.20 | 11.2 | 28.6 |
|
||||
| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m.pt) | 640 | 50.2 | 234.7 | 1.83 | 25.9 | 78.9 |
|
||||
| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l.pt) | 640 | 52.9 | 375.2 | 2.39 | 43.7 | 165.2 |
|
||||
| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x.pt) | 640 | 53.9 | 479.1 | 3.53 | 68.2 | 257.8 |
|
||||
|
||||
=== "Detection (Open Images V7)"
|
||||
|
||||
See [Detection Docs](../tasks/detect.md) for usage examples with these models trained on [Open Image V7](../datasets/detect/open-images-v7.md), which include 600 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| ----------------------------------------------------------------------------------------- | --------------------- | -------------------- | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n-oiv7.pt) | 640 | 18.4 | 142.4 | 1.21 | 3.5 | 10.5 |
|
||||
| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-oiv7.pt) | 640 | 27.7 | 183.1 | 1.40 | 11.4 | 29.7 |
|
||||
| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-oiv7.pt) | 640 | 33.6 | 408.5 | 2.26 | 26.2 | 80.6 |
|
||||
| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-oiv7.pt) | 640 | 34.9 | 596.9 | 2.43 | 44.1 | 167.4 |
|
||||
| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-oiv7.pt) | 640 | 36.3 | 860.6 | 3.56 | 68.7 | 260.6 |
|
||||
|
||||
=== "Segmentation (COCO)"
|
||||
|
||||
See [Segmentation Docs](../tasks/segment.md) for usage examples with these models trained on [COCO](../datasets/segment/coco.md), which include 80 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>box<br>50-95</sup> | mAP<sup>mask<br>50-95</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| -------------------------------------------------------------------------------------------- | --------------------- | -------------------- | --------------------- | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLOv8n-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n-seg.pt) | 640 | 36.7 | 30.5 | 96.1 | 1.21 | 3.4 | 12.6 |
|
||||
| [YOLOv8s-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-seg.pt) | 640 | 44.6 | 36.8 | 155.7 | 1.47 | 11.8 | 42.6 |
|
||||
| [YOLOv8m-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-seg.pt) | 640 | 49.9 | 40.8 | 317.0 | 2.18 | 27.3 | 110.2 |
|
||||
| [YOLOv8l-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-seg.pt) | 640 | 52.3 | 42.6 | 572.4 | 2.79 | 46.0 | 220.5 |
|
||||
| [YOLOv8x-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-seg.pt) | 640 | 53.4 | 43.4 | 712.1 | 4.02 | 71.8 | 344.1 |
|
||||
|
||||
=== "Classification (ImageNet)"
|
||||
|
||||
See [Classification Docs](../tasks/classify.md) for usage examples with these models trained on [ImageNet](../datasets/classify/imagenet.md), which include 1000 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | acc<br><sup>top1</sup> | acc<br><sup>top5</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B) at 224</sup> |
|
||||
| -------------------------------------------------------------------------------------------- | --------------------- | ---------------- | ---------------- | ------------------------------ | ----------------------------------- | ------------------ | ------------------------ |
|
||||
| [YOLOv8n-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n-cls.pt) | 224 | 69.0 | 88.3 | 12.9 | 0.31 | 2.7 | 0.5 |
|
||||
| [YOLOv8s-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-cls.pt) | 224 | 73.8 | 91.7 | 23.4 | 0.35 | 6.4 | 1.7 |
|
||||
| [YOLOv8m-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-cls.pt) | 224 | 76.8 | 93.5 | 85.4 | 0.62 | 17.0 | 5.3 |
|
||||
| [YOLOv8l-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-cls.pt) | 224 | 76.8 | 93.5 | 163.0 | 0.87 | 37.5 | 12.3 |
|
||||
| [YOLOv8x-cls](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-cls.pt) | 224 | 79.0 | 94.6 | 232.0 | 1.01 | 57.4 | 19.0 |
|
||||
|
||||
=== "Pose (COCO)"
|
||||
|
||||
See [Pose Estimation Docs](../tasks/pose.md) for usage examples with these models trained on [COCO](../datasets/pose/coco.md), which include 1 pretrained class, 'person'.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>pose<br>50-95</sup> | mAP<sup>pose<br>50</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
| ---------------------------------------------------------------------------------------------------- | --------------------- | --------------------- | ------------------ | ------------------------------ | ----------------------------------- | ------------------ | ----------------- |
|
||||
| [YOLOv8n-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n-pose.pt) | 640 | 50.4 | 80.1 | 131.8 | 1.18 | 3.3 | 9.2 |
|
||||
| [YOLOv8s-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-pose.pt) | 640 | 60.0 | 86.2 | 233.2 | 1.42 | 11.6 | 30.2 |
|
||||
| [YOLOv8m-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-pose.pt) | 640 | 65.0 | 88.8 | 456.3 | 2.00 | 26.4 | 81.0 |
|
||||
| [YOLOv8l-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-pose.pt) | 640 | 67.6 | 90.0 | 784.5 | 2.59 | 44.4 | 168.6 |
|
||||
| [YOLOv8x-pose](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-pose.pt) | 640 | 69.2 | 90.2 | 1607.1 | 3.73 | 69.4 | 263.2 |
|
||||
| [YOLOv8x-pose-p6](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-pose-p6.pt) | 1280 | 71.6 | 91.2 | 4088.7 | 10.04 | 99.1 | 1066.4 |
|
||||
|
||||
=== "OBB (DOTAv1)"
|
||||
|
||||
See [Oriented Detection Docs](../tasks/obb.md) for usage examples with these models trained on [DOTAv1](../datasets/obb/dota-v2.md#dota-v10), which include 15 pretrained classes.
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>test<br>50</sup> | Speed<br><sup>CPU ONNX<br>(ms)</sup> | Speed<br><sup>A100 TensorRT<br>(ms)</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|----------------------------------------------------------------------------------------------|-----------------------| -------------------- | -------------------------------- | ------------------------------------- | -------------------- | ----------------- |
|
||||
| [YOLOv8n-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8n-obb.pt) | 1024 | 78.0 | 204.77 | 3.57 | 3.1 | 23.3 |
|
||||
| [YOLOv8s-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8s-obb.pt) | 1024 | 79.5 | 424.88 | 4.07 | 11.4 | 76.3 |
|
||||
| [YOLOv8m-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8m-obb.pt) | 1024 | 80.5 | 763.48 | 7.61 | 26.4 | 208.6 |
|
||||
| [YOLOv8l-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8l-obb.pt) | 1024 | 80.7 | 1278.42 | 11.83 | 44.5 | 433.8 |
|
||||
| [YOLOv8x-obb](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov8x-obb.pt) | 1024 | 81.36 | 1759.10 | 13.23 | 69.5 | 676.7 |
|
||||
|
||||
## YOLOv8 Usage Examples
|
||||
|
||||
This example provides simple YOLOv8 training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
Note the below example is for YOLOv8 [Detect](../tasks/detect.md) models for object detection. For additional supported tasks see the [Segment](../tasks/segment.md), [Classify](../tasks/classify.md), [OBB](../tasks/obb.md) docs and [Pose](../tasks/pose.md) docs.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv8n model
|
||||
model = YOLO("yolov8n.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLOv8n model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Load a COCO-pretrained YOLOv8n model and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov8n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Load a COCO-pretrained YOLOv8n model and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov8n.pt source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
!!! tip "Ultralytics YOLOv8 Publication"
|
||||
|
||||
Ultralytics has not published a formal research paper for YOLOv8 due to the rapidly evolving nature of the models. We focus on advancing the technology and making it easier to use, rather than producing static documentation. For the most up-to-date information on YOLO architecture, features, and usage, please refer to our [GitHub repository](https://github.com/ultralytics/ultralytics) and [documentation](https://docs.ultralytics.com/).
|
||||
|
||||
If you use the YOLOv8 model or any other software from this repository in your work, please cite it using the following format:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@software{yolov8_ultralytics,
|
||||
author = {Glenn Jocher and Ayush Chaurasia and Jing Qiu},
|
||||
title = {Ultralytics YOLOv8},
|
||||
version = {8.0.0},
|
||||
year = {2023},
|
||||
url = {https://github.com/ultralytics/ultralytics},
|
||||
orcid = {0000-0001-5950-6979, 0000-0002-7603-6750, 0000-0003-3783-7069},
|
||||
license = {AGPL-3.0}
|
||||
}
|
||||
```
|
||||
|
||||
Please note that the DOI is pending and will be added to the citation once it is available. YOLOv8 models are provided under [AGPL-3.0](https://github.com/ultralytics/ultralytics/blob/main/LICENSE) and [Enterprise](https://www.ultralytics.com/license) licenses.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is YOLOv8 and how does it differ from previous YOLO versions?
|
||||
|
||||
YOLOv8 is designed to improve real-time object detection performance with advanced features. Unlike earlier versions, YOLOv8 incorporates an **anchor-free split Ultralytics head**, state-of-the-art [backbone](https://www.ultralytics.com/glossary/backbone) and neck architectures, and offers optimized [accuracy](https://www.ultralytics.com/glossary/accuracy)-speed tradeoff, making it ideal for diverse applications. For more details, check the [Overview](#overview) and [Key Features](#key-features-of-yolov8) sections.
|
||||
|
||||
### How can I use YOLOv8 for different computer vision tasks?
|
||||
|
||||
YOLOv8 supports a wide range of computer vision tasks, including object detection, instance segmentation, pose/keypoints detection, oriented object detection, and classification. Each model variant is optimized for its specific task and compatible with various operational modes like [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). Refer to the [Supported Tasks and Modes](#supported-tasks-and-modes) section for more information.
|
||||
|
||||
### What are the performance metrics for YOLOv8 models?
|
||||
|
||||
YOLOv8 models achieve state-of-the-art performance across various benchmarking datasets. For instance, the YOLOv8n model achieves a mAP (mean Average Precision) of 37.3 on the COCO dataset and a speed of 0.99 ms on A100 TensorRT. Detailed performance metrics for each model variant across different tasks and datasets can be found in the [Performance Metrics](#performance-metrics) section.
|
||||
|
||||
### How do I train a YOLOv8 model?
|
||||
|
||||
Training a YOLOv8 model can be done using either Python or CLI. Below are examples for training a model using a COCO-pretrained YOLOv8 model on the COCO8 dataset for 100 [epochs](https://www.ultralytics.com/glossary/epoch):
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a COCO-pretrained YOLOv8n model
|
||||
model = YOLO("yolov8n.pt")
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo train model=yolov8n.pt data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
For further details, visit the [Training](../modes/train.md) documentation.
|
||||
|
||||
### Can I benchmark YOLOv8 models for performance?
|
||||
|
||||
Yes, YOLOv8 models can be benchmarked for performance in terms of speed and accuracy across various export formats. You can use PyTorch, ONNX, TensorRT, and more for benchmarking. Below are example commands for benchmarking using Python and CLI:
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics.utils.benchmarks import benchmark
|
||||
|
||||
# Benchmark on GPU
|
||||
benchmark(model="yolov8n.pt", data="coco8.yaml", imgsz=640, half=False, device=0)
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
yolo benchmark model=yolov8n.pt data='coco8.yaml' imgsz=640 half=False device=0
|
||||
```
|
||||
|
||||
For additional information, check the [Performance Metrics](#performance-metrics) section.
|
||||
241
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov9.md
Normal file
241
algorithms/dms_yolo/code.embedded.bak/docs/en/models/yolov9.md
Normal file
@@ -0,0 +1,241 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore YOLOv9, a leap in real-time object detection, featuring innovations like PGI and GELAN, and achieving new benchmarks in efficiency and accuracy.
|
||||
keywords: YOLOv9, object detection, real-time, PGI, GELAN, deep learning, MS COCO, AI, neural networks, model efficiency, accuracy, Ultralytics
|
||||
---
|
||||
|
||||
# YOLOv9: A Leap Forward in [Object Detection](https://www.ultralytics.com/glossary/object-detection) Technology
|
||||
|
||||
YOLOv9 marks a significant advancement in real-time object detection, introducing groundbreaking techniques such as Programmable Gradient Information (PGI) and the Generalized Efficient Layer Aggregation Network (GELAN). This model demonstrates remarkable improvements in efficiency, accuracy, and adaptability, setting new benchmarks on the MS COCO dataset. The YOLOv9 project, while developed by a separate open-source team, builds upon the robust codebase provided by [Ultralytics](https://www.ultralytics.com/) [YOLOv5](yolov5.md), showcasing the collaborative spirit of the AI research community.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/ZF7EAodHn1U"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> YOLOv9 Training on Custom Data using Ultralytics | Industrial Package Dataset
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
## Introduction to YOLOv9
|
||||
|
||||
In the quest for optimal real-time object detection, YOLOv9 stands out with its innovative approach to overcoming information loss challenges inherent in deep [neural networks](https://www.ultralytics.com/glossary/neural-network-nn). By integrating PGI and the versatile GELAN architecture, YOLOv9 not only enhances the model's learning capacity but also ensures the retention of crucial information throughout the detection process, thereby achieving exceptional accuracy and performance.
|
||||
|
||||
## Core Innovations of YOLOv9
|
||||
|
||||
YOLOv9's advancements are deeply rooted in addressing the challenges posed by information loss in deep neural networks. The Information Bottleneck Principle and the innovative use of Reversible Functions are central to its design, ensuring YOLOv9 maintains high efficiency and accuracy.
|
||||
|
||||
### Information Bottleneck Principle
|
||||
|
||||
The Information Bottleneck Principle reveals a fundamental challenge in deep learning: as data passes through successive layers of a network, the potential for information loss increases. This phenomenon is mathematically represented as:
|
||||
|
||||
```python
|
||||
I(X, X) >= I(X, f_theta(X)) >= I(X, g_phi(f_theta(X)))
|
||||
```
|
||||
|
||||
where `I` denotes mutual information, and `f` and `g` represent transformation functions with parameters `theta` and `phi`, respectively. YOLOv9 counters this challenge by implementing Programmable Gradient Information (PGI), which aids in preserving essential data across the network's depth, ensuring more reliable gradient generation and, consequently, better model convergence and performance.
|
||||
|
||||
### Reversible Functions
|
||||
|
||||
The concept of Reversible Functions is another cornerstone of YOLOv9's design. A function is deemed reversible if it can be inverted without any loss of information, as expressed by:
|
||||
|
||||
```python
|
||||
X = v_zeta(r_psi(X))
|
||||
```
|
||||
|
||||
with `psi` and `zeta` as parameters for the reversible and its inverse function, respectively. This property is crucial for [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) architectures, as it allows the network to retain a complete information flow, thereby enabling more accurate updates to the model's parameters. YOLOv9 incorporates reversible functions within its architecture to mitigate the risk of information degradation, especially in deeper layers, ensuring the preservation of critical data for object detection tasks.
|
||||
|
||||
### Impact on Lightweight Models
|
||||
|
||||
Addressing information loss is particularly vital for lightweight models, which are often under-parameterized and prone to losing significant information during the feedforward process. YOLOv9's architecture, through the use of PGI and reversible functions, ensures that even with a streamlined model, the essential information required for accurate object detection is retained and effectively utilized.
|
||||
|
||||
### Programmable Gradient Information (PGI)
|
||||
|
||||
PGI is a novel concept introduced in YOLOv9 to combat the information bottleneck problem, ensuring the preservation of essential data across deep network layers. This allows for the generation of reliable gradients, facilitating accurate model updates and improving the overall detection performance.
|
||||
|
||||
### Generalized Efficient Layer Aggregation Network (GELAN)
|
||||
|
||||
GELAN represents a strategic architectural advancement, enabling YOLOv9 to achieve superior parameter utilization and computational efficiency. Its design allows for flexible integration of various computational blocks, making YOLOv9 adaptable to a wide range of applications without sacrificing speed or accuracy.
|
||||
|
||||
|
||||
|
||||
## YOLOv9 Benchmarks
|
||||
|
||||
Benchmarking in YOLOv9 using [Ultralytics](https://docs.ultralytics.com/modes/benchmark/) involves evaluating the performance of your trained and validated model in real-world scenarios. This process includes:
|
||||
|
||||
- **Performance Evaluation:** Assessing the model's speed and accuracy.
|
||||
- **Export Formats:** Testing the model across different export formats to ensure it meets the necessary standards and performs well in various environments.
|
||||
- **Framework Support:** Providing a comprehensive framework within Ultralytics YOLOv8 to facilitate these assessments and ensure consistent and reliable results.
|
||||
|
||||
By benchmarking, you can ensure that your model not only performs well in controlled testing environments but also maintains high performance in practical, real-world applications.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/ziJR01lKnio"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> How to Benchmark the YOLOv9 Model Using the Ultralytics Python Package
|
||||
</p>
|
||||
|
||||
## Performance on MS COCO Dataset
|
||||
|
||||
<script async src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script defer src="../../javascript/benchmark.js"></script>
|
||||
|
||||
<canvas id="modelComparisonChart" width="1024" height="400" active-models='["YOLOv9"]'></canvas>
|
||||
|
||||
The performance of YOLOv9 on the [COCO dataset](../datasets/detect/coco.md) exemplifies its significant advancements in real-time object detection, setting new benchmarks across various model sizes. Table 1 presents a comprehensive comparison of state-of-the-art real-time object detectors, illustrating YOLOv9's superior efficiency and [accuracy](https://www.ultralytics.com/glossary/accuracy).
|
||||
|
||||
!!! tip "Performance"
|
||||
|
||||
=== "Detection (COCO)"
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>val<br>50-95</sup> | mAP<sup>val<br>50</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|---------------------------------------------------------------------------------------|-----------------------|----------------------|-------------------|--------------------|-------------------|
|
||||
| [YOLOv9t](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9t.pt) | 640 | 38.3 | 53.1 | 2.0 | 7.7 |
|
||||
| [YOLOv9s](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9s.pt) | 640 | 46.8 | 63.4 | 7.2 | 26.7 |
|
||||
| [YOLOv9m](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9m.pt) | 640 | 51.4 | 68.1 | 20.1 | 76.8 |
|
||||
| [YOLOv9c](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9c.pt) | 640 | 53.0 | 70.2 | 25.5 | 102.8 |
|
||||
| [YOLOv9e](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9e.pt) | 640 | 55.6 | 72.8 | 58.1 | 192.5 |
|
||||
|
||||
=== "Segmentation (COCO)"
|
||||
|
||||
| Model | size<br><sup>(pixels)</sup> | mAP<sup>box<br>50-95</sup> | mAP<sup>mask<br>50-95</sup> | params<br><sup>(M)</sup> | FLOPs<br><sup>(B)</sup> |
|
||||
|-----------------------------------------------------------------------------------------------|-----------------------|----------------------|-----------------------|--------------------|-------------------|
|
||||
| [YOLOv9c-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9c-seg.pt) | 640 | 52.4 | 42.2 | 27.9 | 159.4 |
|
||||
| [YOLOv9e-seg](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolov9e-seg.pt) | 640 | 55.1 | 44.3 | 60.5 | 248.4 |
|
||||
|
||||
YOLOv9's iterations, ranging from the tiny `t` variant to the extensive `e` model, demonstrate improvements not only in accuracy (mAP metrics) but also in efficiency with a reduced number of parameters and computational needs (FLOPs). This table underscores YOLOv9's ability to deliver high [precision](https://www.ultralytics.com/glossary/precision) while maintaining or reducing the computational overhead compared to prior versions and competing models.
|
||||
|
||||
Comparatively, YOLOv9 exhibits remarkable gains:
|
||||
|
||||
- **Lightweight Models**: YOLOv9s surpasses the YOLO MS-S in parameter efficiency and computational load while achieving an improvement of 0.4∼0.6% in AP.
|
||||
- **Medium to Large Models**: YOLOv9m and YOLOv9e show notable advancements in balancing the trade-off between model complexity and detection performance, offering significant reductions in parameters and computations against the backdrop of improved accuracy.
|
||||
|
||||
The YOLOv9c model, in particular, highlights the effectiveness of the architecture's optimizations. It operates with 42% fewer parameters and 21% less computational demand than YOLOv7 AF, yet it achieves comparable accuracy, demonstrating YOLOv9's significant efficiency improvements. Furthermore, the YOLOv9e model sets a new standard for large models, with 15% fewer parameters and 25% less computational need than [YOLOv8x](yolov8.md), alongside an incremental 1.7% improvement in AP.
|
||||
|
||||
These results showcase YOLOv9's strategic advancements in model design, emphasizing its enhanced efficiency without compromising on the precision essential for real-time object detection tasks. The model not only pushes the boundaries of performance metrics but also emphasizes the importance of computational efficiency, making it a pivotal development in the field of [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv).
|
||||
|
||||
## Conclusion
|
||||
|
||||
YOLOv9, released in February 2024, represented a pivotal development in real-time object detection, offering significant improvements in terms of efficiency, accuracy, and adaptability. By addressing critical challenges through innovative solutions like PGI and GELAN, YOLOv9 set new benchmarks at its time of release. While newer models like [YOLO11](yolo11.md) and [YOLO26](yolo26.md) have since been released with additional improvements, YOLOv9's architectural innovations continue to influence the field.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
This example provides simple YOLOv9 training and inference examples. For full documentation on these and other [modes](../modes/index.md) see the [Predict](../modes/predict.md), [Train](../modes/train.md), [Val](../modes/val.md) and [Export](../modes/export.md) docs pages.
|
||||
|
||||
!!! example
|
||||
|
||||
=== "Python"
|
||||
|
||||
[PyTorch](https://www.ultralytics.com/glossary/pytorch) pretrained `*.pt` models as well as configuration `*.yaml` files can be passed to the `YOLO()` class to create a model instance in python:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Build a YOLOv9c model from scratch
|
||||
model = YOLO("yolov9c.yaml")
|
||||
|
||||
# Build a YOLOv9c model from pretrained weight
|
||||
model = YOLO("yolov9c.pt")
|
||||
|
||||
# Display model information (optional)
|
||||
model.info()
|
||||
|
||||
# Train the model on the COCO8 example dataset for 100 epochs
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
|
||||
# Run inference with the YOLOv9c model on the 'bus.jpg' image
|
||||
results = model("path/to/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
CLI commands are available to directly run the models:
|
||||
|
||||
```bash
|
||||
# Build a YOLOv9c model from scratch and train it on the COCO8 example dataset for 100 epochs
|
||||
yolo train model=yolov9c.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
|
||||
# Build a YOLOv9c model from scratch and run inference on the 'bus.jpg' image
|
||||
yolo predict model=yolov9c.yaml source=path/to/bus.jpg
|
||||
```
|
||||
|
||||
## Supported Tasks and Modes
|
||||
|
||||
The YOLOv9 series offers a range of models, each optimized for high-performance [Object Detection](../tasks/detect.md). These models cater to varying computational needs and accuracy requirements, making them versatile for a wide array of applications.
|
||||
|
||||
| Model | Filenames | Tasks | Inference | Validation | Training | Export |
|
||||
| ---------- | ---------------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ |
|
||||
| YOLOv9 | `yolov9t.pt` `yolov9s.pt` `yolov9m.pt` `yolov9c.pt` `yolov9e.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
| YOLOv9-seg | `yolov9c-seg.pt` `yolov9e-seg.pt` | [Instance Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
This table provides a detailed overview of the YOLOv9 model variants, highlighting their capabilities in object detection tasks and their compatibility with various operational modes such as [Inference](../modes/predict.md), [Validation](../modes/val.md), [Training](../modes/train.md), and [Export](../modes/export.md). This comprehensive support ensures that users can fully leverage the capabilities of YOLOv9 models in a broad range of object detection scenarios.
|
||||
|
||||
!!! note
|
||||
|
||||
Training YOLOv9 models will require _more_ resources **and** take longer than the equivalent sized [YOLOv8 model](yolov8.md).
|
||||
|
||||
## Citations and Acknowledgments
|
||||
|
||||
We would like to acknowledge the YOLOv9 authors for their significant contributions in the field of real-time object detection:
|
||||
|
||||
!!! quote ""
|
||||
|
||||
=== "BibTeX"
|
||||
|
||||
```bibtex
|
||||
@article{wang2024yolov9,
|
||||
title={YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information},
|
||||
author={Wang, Chien-Yao and Liao, Hong-Yuan Mark},
|
||||
booktitle={arXiv preprint arXiv:2402.13616},
|
||||
year={2024}
|
||||
}
|
||||
```
|
||||
|
||||
The original YOLOv9 paper can be found on [arXiv](https://arxiv.org/pdf/2402.13616). The authors have made their work publicly available, and the codebase can be accessed on [GitHub](https://github.com/WongKinYiu/yolov9). We appreciate their efforts in advancing the field and making their work accessible to the broader community.
|
||||
|
||||
## FAQ
|
||||
|
||||
### What innovations does YOLOv9 introduce for real-time object detection?
|
||||
|
||||
YOLOv9 introduces groundbreaking techniques such as Programmable Gradient Information (PGI) and the Generalized Efficient Layer Aggregation Network (GELAN). These innovations address information loss challenges in deep neural networks, ensuring high efficiency, accuracy, and adaptability. PGI preserves essential data across network layers, while GELAN optimizes parameter utilization and computational efficiency. Learn more about [YOLOv9's core innovations](#core-innovations-of-yolov9) that set new benchmarks on the MS COCO dataset.
|
||||
|
||||
### How does YOLOv9 perform on the MS COCO dataset compared to other models?
|
||||
|
||||
YOLOv9 outperforms state-of-the-art real-time object detectors by achieving higher accuracy and efficiency. On the [COCO dataset](../datasets/detect/coco.md), YOLOv9 models exhibit superior mAP scores across various sizes while maintaining or reducing computational overhead. For instance, YOLOv9c achieves comparable accuracy with 42% fewer parameters and 21% less computational demand than YOLOv7 AF. Explore [performance comparisons](#performance-on-ms-coco-dataset) for detailed metrics.
|
||||
|
||||
### How can I train a YOLOv9 model using Python and CLI?
|
||||
|
||||
You can train a YOLOv9 model using both Python and CLI commands. For Python, instantiate a model using the `YOLO` class and call the `train` method:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Build a YOLOv9c model from pretrained weights and train
|
||||
model = YOLO("yolov9c.pt")
|
||||
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)
|
||||
```
|
||||
|
||||
For CLI training, execute:
|
||||
|
||||
```bash
|
||||
yolo train model=yolov9c.yaml data=coco8.yaml epochs=100 imgsz=640
|
||||
```
|
||||
|
||||
Learn more about [usage examples](#usage-examples) for training and inference.
|
||||
|
||||
### What are the advantages of using Ultralytics YOLOv9 for lightweight models?
|
||||
|
||||
YOLOv9 is designed to mitigate information loss, which is particularly important for lightweight models often prone to losing significant information. By integrating Programmable Gradient Information (PGI) and reversible functions, YOLOv9 ensures essential data retention, enhancing the model's accuracy and efficiency. This makes it highly suitable for applications requiring compact models with high performance. For more details, explore the section on [YOLOv9's impact on lightweight models](#impact-on-lightweight-models).
|
||||
|
||||
### What tasks and modes does YOLOv9 support?
|
||||
|
||||
YOLOv9 supports various tasks including object detection and [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation). It is compatible with multiple operational modes such as inference, validation, training, and export. This versatility makes YOLOv9 adaptable to diverse real-time computer vision applications. Refer to the [supported tasks and modes](#supported-tasks-and-modes) section for more information.
|
||||
Reference in New Issue
Block a user