ADAS/HSAP
2
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: HSAP platform v2 — modular navigation, quality review, audit log, world model simulation

Browse Source 
Major changes:
- New frontend (platform/web/): Vite + React 18 + TypeScript + Tailwind
- 4-module navigation: 数据送标 / 模型管理 / 车队管理 / 系统管理
- Data catalog with charts (DMS/ADAS/Lane 3-tab view)
- Quality review workflow (标注质检): Good/Fine/Bad scoring with auto-advance
- Audit enhancements: batch operations, rejection categories, Feishu notifications
- Operation audit log (操作日志)
- World model simulation studio (仿真工坊)
- Dataset version management with snapshots and diff
- ADAS 7-class dataset integration (138K images organized + compressed)
- User management with Feishu integration and pagination
- CRUD/search/filter on all pages, card layout redesign
- PIL-optimized image overlay rendering
- Auto-snapshot on build, in_review workflow stage
- Removed embedded algorithm code (now in workspace)
This commit is contained in:
Chengfang Lu
2026-06-03 11:40:21 +08:00
parent 7c43b44c57
commit e72bc061c5
5487 changed files with 979207 additions and 6197 deletions
Download Patch File Download Diff File Expand all files Collapse all files

330
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/analytics.md Normal file
View File

@@ -0,0 +1,330 @@
---
comments: true
description: Learn to create line graphs, bar plots, and pie charts using Python with guided instructions and code snippets. Maximize your data visualization skills!
keywords: Ultralytics, YOLO26, data visualization, line graphs, bar plots, pie charts, Python, analytics, tutorial, guide
---
# Analytics using Ultralytics YOLO26
## Introduction
This guide provides a comprehensive overview of three fundamental types of [data visualizations](https://www.ultralytics.com/glossary/data-visualization): line graphs, bar plots, and pie charts. Each section includes step-by-step instructions and code snippets on how to create these visualizations using Python.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/tVuLIMt4DMY"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to generate Analytical Graphs using Ultralytics | Line Graphs, Bar Plots, Area and Pie Charts
</p>
### Visual Samples
| Line Graph | Bar Plot | Pie Chart |
| :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: |
| ![YOLO analytics line graph for object tracking](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/analytics-line-graph.avif) | ![YOLO analytics bar plot for detection counts](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/analytics-bar-plot.avif) | ![YOLO analytics pie chart for class distribution](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/analytics-pie-chart.avif) |
### Why Graphs are Important
- Line graphs are ideal for tracking changes over short and long periods and for comparing changes for multiple groups over the same period.
- Bar plots, on the other hand, are suitable for comparing quantities across different categories and showing relationships between a category and its numerical value.
- Lastly, pie charts are effective for illustrating proportions among categories and showing parts of a whole.
!!! example "Analytics using Ultralytics YOLO"
=== "CLI"
```bash
yolo solutions analytics show=True
# Pass the source
yolo solutions analytics source="path/to/video.mp4"
# Generate the pie chart
yolo solutions analytics analytics_type="pie" show=True
# Generate the bar plots
yolo solutions analytics analytics_type="bar" show=True
# Generate the area plots
yolo solutions analytics analytics_type="area" show=True
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
out = cv2.VideoWriter(
"analytics_output.avi",
cv2.VideoWriter_fourcc(*"MJPG"),
fps,
(1280, 720), # this is fixed
)
# Initialize analytics object
analytics = solutions.Analytics(
show=True, # display the output
analytics_type="line", # pass the analytics type, could be "pie", "bar" or "area".
model="yolo26n.pt", # path to the YOLO26 model file
# classes=[0, 2], # display analytics for specific detection classes
)
# Process video
frame_count = 0
while cap.isOpened():
success, im0 = cap.read()
if success:
frame_count += 1
results = analytics(im0, frame_count) # update analytics graph every frame
# print(results) # access the output
out.write(results.plot_im) # write the video file
else:
break
cap.release()
out.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `Analytics` Arguments
Here's a table outlining the Analytics arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "analytics_type"]) }}
You can also leverage different [`track`](../modes/track.md) arguments in the `Analytics` solution.
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the following visualization arguments are supported:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width"]) }}
## Conclusion
Understanding when and how to use different types of visualizations is crucial for effective data analysis. Line graphs, bar plots, and pie charts are fundamental tools that can help you convey your data's story more clearly and effectively. The Ultralytics YOLO26 Analytics solution provides a streamlined way to generate these visualizations from your [object detection](https://www.ultralytics.com/glossary/object-detection) and tracking results, making it easier to extract meaningful insights from your visual data.
## FAQ
### How do I create a line graph using Ultralytics YOLO26 Analytics?
To create a line graph using Ultralytics YOLO26 Analytics, follow these steps:
1. Load a YOLO26 model and open your video file.
2. Initialize the `Analytics` class with the type set to "line."
3. Iterate through video frames, updating the line graph with relevant data, such as object counts per frame.
4. Save the output video displaying the line graph.
Example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
out = cv2.VideoWriter(
"ultralytics_analytics.avi",
cv2.VideoWriter_fourcc(*"MJPG"),
fps,
(1280, 720), # this is fixed
)
analytics = solutions.Analytics(
analytics_type="line",
show=True,
)
frame_count = 0
while cap.isOpened():
success, im0 = cap.read()
if success:
frame_count += 1
results = analytics(im0, frame_count) # update analytics graph every frame
out.write(results.plot_im) # write the video file
else:
break
cap.release()
out.release()
cv2.destroyAllWindows()
```
For further details on configuring the `Analytics` class, visit the [Analytics using Ultralytics YOLO26](#analytics-using-ultralytics-yolo26) section.
### What are the benefits of using Ultralytics YOLO26 for creating bar plots?
Using Ultralytics YOLO26 for creating bar plots offers several benefits:
1. **Real-time Data Visualization**: Seamlessly integrate [object detection](https://www.ultralytics.com/glossary/object-detection) results into bar plots for dynamic updates.
2. **Ease of Use**: Simple API and functions make it straightforward to implement and visualize data.
3. **Customization**: Customize titles, labels, colors, and more to fit your specific requirements.
4. **Efficiency**: Efficiently handle large amounts of data and update plots in real-time during video processing.
Use the following example to generate a bar plot:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
out = cv2.VideoWriter(
"ultralytics_analytics.avi",
cv2.VideoWriter_fourcc(*"MJPG"),
fps,
(1280, 720), # this is fixed
)
analytics = solutions.Analytics(
analytics_type="bar",
show=True,
)
frame_count = 0
while cap.isOpened():
success, im0 = cap.read()
if success:
frame_count += 1
results = analytics(im0, frame_count) # update analytics graph every frame
out.write(results.plot_im) # write the video file
else:
break
cap.release()
out.release()
cv2.destroyAllWindows()
```
To learn more, visit the [Bar Plot](#visual-samples) section in the guide.
### Why should I use Ultralytics YOLO26 for creating pie charts in my data visualization projects?
Ultralytics YOLO26 is an excellent choice for creating pie charts because:
1. **Integration with Object Detection**: Directly integrate object detection results into pie charts for immediate insights.
2. **User-Friendly API**: Simple to set up and use with minimal code.
3. **Customizable**: Various customization options for colors, labels, and more.
4. **Real-time Updates**: Handle and visualize data in real-time, which is ideal for video analytics projects.
Here's a quick example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
out = cv2.VideoWriter(
"ultralytics_analytics.avi",
cv2.VideoWriter_fourcc(*"MJPG"),
fps,
(1280, 720), # this is fixed
)
analytics = solutions.Analytics(
analytics_type="pie",
show=True,
)
frame_count = 0
while cap.isOpened():
success, im0 = cap.read()
if success:
frame_count += 1
results = analytics(im0, frame_count) # update analytics graph every frame
out.write(results.plot_im) # write the video file
else:
break
cap.release()
out.release()
cv2.destroyAllWindows()
```
For more information, refer to the [Pie Chart](#visual-samples) section in the guide.
### Can Ultralytics YOLO26 be used to track objects and dynamically update visualizations?
Yes, Ultralytics YOLO26 can be used to track objects and dynamically update visualizations. It supports tracking multiple objects in real-time and can update various visualizations like line graphs, bar plots, and pie charts based on the tracked objects' data.
Example for tracking and updating a line graph:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
out = cv2.VideoWriter(
"ultralytics_analytics.avi",
cv2.VideoWriter_fourcc(*"MJPG"),
fps,
(1280, 720), # this is fixed
)
analytics = solutions.Analytics(
analytics_type="line",
show=True,
)
frame_count = 0
while cap.isOpened():
success, im0 = cap.read()
if success:
frame_count += 1
results = analytics(im0, frame_count) # update analytics graph every frame
out.write(results.plot_im) # write the video file
else:
break
cap.release()
out.release()
cv2.destroyAllWindows()
```
To learn about the complete functionality, see the [Tracking](../modes/track.md) section.
### What makes Ultralytics YOLO26 different from other object detection solutions like [OpenCV](https://www.ultralytics.com/glossary/opencv) and [TensorFlow](https://www.ultralytics.com/glossary/tensorflow)?
Ultralytics YOLO26 stands out from other object detection solutions like OpenCV and TensorFlow for multiple reasons:
1. **State-of-the-art [Accuracy](https://www.ultralytics.com/glossary/accuracy)**: YOLO26 provides superior accuracy in object detection, segmentation, and classification tasks.
2. **Ease of Use**: User-friendly API allows for quick implementation and integration without extensive coding.
3. **Real-time Performance**: Optimized for high-speed inference, suitable for real-time applications.
4. **Diverse Applications**: Supports various tasks including multi-object tracking, custom model training, and exporting to different formats like ONNX, TensorRT, and CoreML.
5. **Comprehensive Documentation**: Extensive [documentation](https://docs.ultralytics.com/) and [blog resources](https://www.ultralytics.com/blog) to guide users through every step.
For more detailed comparisons and use cases, explore our [Ultralytics Blog](https://www.ultralytics.com/blog/ai-use-cases-transforming-your-future).

227
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/azureml-quickstart.md Normal file
View File

@@ -0,0 +1,227 @@
---
comments: true
description: Learn how to run YOLO26 on AzureML. Quickstart instructions for terminal and notebooks to harness Azure's cloud computing for efficient model training.
keywords: YOLO26, AzureML, machine learning, cloud computing, quickstart, terminal, notebooks, model training, Python SDK, AI, Ultralytics
---
# YOLO26 🚀 on AzureML
## What is Azure?
[Azure](https://azure.microsoft.com/) is Microsoft's [cloud computing](https://www.ultralytics.com/glossary/cloud-computing) platform, designed to help organizations move their workloads to the cloud from on-premises data centers. With the full spectrum of cloud services including those for computing, databases, analytics, [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml), and networking, users can pick and choose from these services to develop and scale new applications, or run existing applications, in the public cloud.
## What is Azure Machine Learning (AzureML)?
Azure Machine Learning, commonly referred to as AzureML, is a fully managed cloud service that enables data scientists and developers to efficiently embed predictive analytics into their applications, helping organizations use massive data sets and bring all the benefits of the cloud to machine learning. AzureML offers a variety of services and capabilities aimed at making machine learning accessible, easy to use, and scalable. It provides capabilities like automated machine learning, drag-and-drop model training, as well as a robust Python SDK so that developers can make the most out of their machine learning models.
## How Does AzureML Benefit YOLO Users?
For users of YOLO (You Only Look Once), AzureML provides a robust, scalable, and efficient platform to both train and deploy machine learning models. Whether you are looking to run quick prototypes or scale up to handle more extensive data, AzureML's flexible and user-friendly environment offers various tools and services to fit your needs. You can leverage AzureML to:
- Easily manage large datasets and computational resources for training.
- Utilize built-in tools for data preprocessing, feature selection, and model training.
- Collaborate more efficiently with capabilities for MLOps (Machine Learning Operations), including but not limited to monitoring, auditing, and versioning of models and data.
In the subsequent sections, you will find a quickstart guide detailing how to run YOLO26 object detection models using AzureML, either from a compute terminal or a notebook.
## Prerequisites
Before you can get started, make sure you have access to an AzureML workspace. If you don't have one, you can create a new [AzureML workspace](https://learn.microsoft.com/azure/machine-learning/concept-workspace?view=azureml-api-2) by following Azure's official documentation. This workspace acts as a centralized place to manage all AzureML resources.
## Create a compute instance
From your AzureML workspace, select Compute > Compute instances > New, select the instance with the resources you need.
<p align="center">
<img width="1280" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/create-compute-arrow.avif" alt="Create Azure Compute Instance">
</p>
## Quickstart from Terminal
Start your compute and open a Terminal:
<p align="center">
<img width="480" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/open-terminal.avif" alt="Open Terminal">
</p>
### Create virtualenv
Create a conda virtual environment with your preferred Python version and install pip in it. Python 3.13.1 currently has dependency issues in AzureML, so use Python 3.12 instead.
```bash
conda create --name yolo26env -y python=3.12
conda activate yolo26env
conda install pip -y
```
Install the required dependencies:
```bash
cd ultralytics
pip install -r requirements.txt
pip install ultralytics
pip install onnx
```
### Perform YOLO26 tasks
Predict:
```bash
yolo predict model=yolo26n.pt source='https://ultralytics.com/images/bus.jpg'
```
Train a detection model for 10 [epochs](https://www.ultralytics.com/glossary/epoch) with an initial learning_rate of 0.01:
```bash
yolo train data=coco8.yaml model=yolo26n.pt epochs=10 lr0=0.01
```
You can find more [instructions to use the Ultralytics CLI here](../quickstart.md#use-ultralytics-with-cli).
## Quickstart from a Notebook
### Create a new IPython kernel
Open the compute Terminal.
<p align="center">
<img width="480" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/open-terminal.avif" alt="Open Terminal">
</p>
From your compute terminal, create a new ipykernel using Python 3.12 that will be used by your notebook to manage dependencies:
```bash
conda create --name yolo26env -y python=3.12
conda activate yolo26env
conda install pip -y
conda install ipykernel -y
python -m ipykernel install --user --name yolo26env --display-name "yolo26env"
```
Close your terminal and create a new notebook. From your notebook, select the newly created kernel.
Then open a notebook cell and install the required dependencies:
```bash
%%bash
source activate yolo26env
cd ultralytics
pip install -r requirements.txt
pip install ultralytics
pip install onnx
```
Note that you need to run `source activate yolo26env` in every `%%bash` cell to ensure the cell uses the intended environment.
Run some predictions using the [Ultralytics CLI](../quickstart.md#use-ultralytics-with-cli):
```bash
%%bash
source activate yolo26env
yolo predict model=yolo26n.pt source='https://ultralytics.com/images/bus.jpg'
```
Or with the [Ultralytics Python interface](../quickstart.md#use-ultralytics-with-python), for example to train the model:
```python
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt") # load an official YOLO26n model
# Use the model
model.train(data="coco8.yaml", epochs=3) # train the model
metrics = model.val() # evaluate model performance on the validation set
results = model("https://ultralytics.com/images/bus.jpg") # predict on an image
path = model.export(format="onnx") # export the model to ONNX format
```
You can use either the Ultralytics CLI or Python interface for running YOLO26 tasks, as described in the terminal section above.
By following these steps, you should be able to get YOLO26 running quickly on AzureML for quick trials. For more advanced uses, you may refer to the full AzureML documentation linked at the beginning of this guide.
## Explore More with AzureML
This guide serves as an introduction to get you up and running with YOLO26 on AzureML. However, it only scratches the surface of what AzureML can offer. To delve deeper and unlock the full potential of AzureML for your machine learning projects, consider exploring the following resources:
- [Create a Data Asset](https://learn.microsoft.com/azure/machine-learning/how-to-create-data-assets): Learn how to set up and manage your data assets effectively within the AzureML environment.
- [Initiate an AzureML Job](https://learn.microsoft.com/azure/machine-learning/how-to-train-model): Get a comprehensive understanding of how to kickstart your machine learning training jobs on AzureML.
- [Register a Model](https://learn.microsoft.com/azure/machine-learning/how-to-manage-models): Familiarize yourself with model management practices including registration, versioning, and deployment.
- [Train YOLO26 with AzureML Python SDK](https://medium.com/@ouphi/how-to-train-the-yolov8-model-with-azure-machine-learning-python-sdk-8268696be8ba): Explore a step-by-step guide on using the AzureML Python SDK to train your YOLO26 models.
- [Train YOLO26 with AzureML CLI](https://medium.com/@ouphi/how-to-train-the-yolov8-model-with-azureml-and-the-az-cli-73d3c870ba8e): Discover how to utilize the command-line interface for streamlined training and management of YOLO26 models on AzureML.
## FAQ
### How do I run YOLO26 on AzureML for model training?
Running YOLO26 on AzureML for model training involves several steps:
1. **Create a Compute Instance**: From your AzureML workspace, navigate to Compute > Compute instances > New, and select the required instance.
2. **Set Up the Environment**: Start your compute instance, open a terminal, and create a Conda environment. Set your Python version (Python 3.13.1 is not supported yet):
```bash
conda create --name yolo26env -y python=3.12
conda activate yolo26env
conda install pip -y
pip install ultralytics onnx
```
3. **Run YOLO26 Tasks**: Use the Ultralytics CLI to train your model:
```bash
yolo train data=coco8.yaml model=yolo26n.pt epochs=10 lr0=0.01
```
For more details, you can refer to the [instructions to use the Ultralytics CLI](../quickstart.md#use-ultralytics-with-cli).
### What are the benefits of using AzureML for YOLO26 training?
AzureML provides a robust and efficient ecosystem for training YOLO26 models:
- **Scalability**: Easily scale your compute resources as your data and model complexity grows.
- **MLOps Integration**: Utilize features like versioning, monitoring, and auditing to streamline ML operations.
- **Collaboration**: Share and manage resources within teams, enhancing collaborative workflows.
These advantages make AzureML an ideal platform for projects ranging from quick prototypes to large-scale deployments. For more tips, check out [AzureML Jobs](https://learn.microsoft.com/azure/machine-learning/how-to-train-model).
### How do I troubleshoot common issues when running YOLO26 on AzureML?
Troubleshooting common issues with YOLO26 on AzureML can involve the following steps:
- **Dependency Issues**: Ensure all required packages are installed. Refer to the `requirements.txt` file for dependencies.
- **Environment Setup**: Verify that your conda environment is correctly activated before running commands.
- **Resource Allocation**: Make sure your compute instances have sufficient resources to handle the training workload.
For additional guidance, review our [YOLO Common Issues](https://docs.ultralytics.com/guides/yolo-common-issues/) documentation.
### Can I use both the Ultralytics CLI and Python interface on AzureML?
Yes, AzureML allows you to use both the Ultralytics CLI and the Python interface seamlessly:
- **CLI**: Ideal for quick tasks and running standard scripts directly from the terminal.
```bash
yolo predict model=yolo26n.pt source='https://ultralytics.com/images/bus.jpg'
```
- **Python Interface**: Useful for more complex tasks requiring custom coding and integration within notebooks.
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
model.train(data="coco8.yaml", epochs=3)
```
For step-by-step instructions, refer to the [CLI quickstart guide](../quickstart.md#use-ultralytics-with-cli) and the [Python quickstart guide](../quickstart.md#use-ultralytics-with-python).
### What is the advantage of using Ultralytics YOLO26 over other [object detection](https://www.ultralytics.com/glossary/object-detection) models?
Ultralytics YOLO26 offers several unique advantages over competing object detection models:
- **Speed**: Faster inference and training times compared to models like Faster R-CNN and SSD.
- **[Accuracy](https://www.ultralytics.com/glossary/accuracy)**: High accuracy in detection tasks with features like anchor-free design and enhanced augmentation strategies.
- **Ease of Use**: Intuitive API and CLI for quick setup, making it accessible both to beginners and experts.
To explore more about YOLO26's features, visit the [Ultralytics YOLO](https://www.ultralytics.com/yolo) page for detailed insights.

193
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/conda-quickstart.md Normal file
View File

@@ -0,0 +1,193 @@
---
comments: true
description: Learn to set up a Conda environment for Ultralytics projects. Follow our comprehensive guide for easy installation and initialization.
keywords: Ultralytics, Conda, setup, installation, environment, guide, machine learning, data science
---
# Conda Quickstart Guide for Ultralytics
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-conda-package-visual.avif" alt="Ultralytics Conda Package Visual">
</p>
This guide provides a comprehensive introduction to setting up a Conda environment for your Ultralytics projects. Conda is an open-source package and environment management system that offers an excellent alternative to pip for installing packages and dependencies. Its isolated environments make it particularly well-suited for data science and [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) endeavors. For more details, visit the Ultralytics Conda package on [Anaconda](https://anaconda.org/conda-forge/ultralytics) and check out the Ultralytics feedstock repository for package updates on [GitHub](https://github.com/conda-forge/ultralytics-feedstock/).
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/ultralytics?logo=condaforge)](https://anaconda.org/conda-forge/ultralytics)
[![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/ultralytics.svg)](https://anaconda.org/conda-forge/ultralytics)
[![Conda Recipe](https://img.shields.io/badge/recipe-ultralytics-green.svg)](https://anaconda.org/conda-forge/ultralytics)
[![Conda Platforms](https://img.shields.io/conda/pn/conda-forge/ultralytics.svg)](https://anaconda.org/conda-forge/ultralytics)
## What You Will Learn
- Setting up a Conda environment
- Installing Ultralytics via Conda
- Initializing Ultralytics in your environment
- Using Ultralytics Docker images with Conda
---
## Prerequisites
- You should have Anaconda or Miniconda installed on your system. If not, download and install it from [Anaconda](https://www.anaconda.com/) or [Miniconda](https://www.anaconda.com/docs/main).
---
## Setting up a Conda Environment
First, let's create a new Conda environment. Open your terminal and run the following command:
```bash
conda create --name ultralytics-env python=3.11 -y
```
Activate the new environment:
```bash
conda activate ultralytics-env
```
---
## Installing Ultralytics
You can install the Ultralytics package from the conda-forge channel. Execute the following command:
```bash
conda install -c conda-forge ultralytics
```
### Note on CUDA Environment
If you're working in a CUDA-enabled environment, it's a good practice to install `ultralytics`, `pytorch`, and `pytorch-cuda` together to resolve any conflicts:
```bash
conda install -c pytorch -c nvidia -c conda-forge pytorch torchvision pytorch-cuda=11.8 ultralytics
```
---
## Using Ultralytics
With Ultralytics installed, you can now start using its robust features for [object detection](https://www.ultralytics.com/glossary/object-detection), [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), and more. For example, to predict an image, you can run:
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt") # initialize model
results = model("path/to/image.jpg") # perform inference
results[0].show() # display results for the first image
```
---
## Ultralytics Conda Docker Image
If you prefer using Docker, Ultralytics offers Docker images with a Conda environment included. You can pull these images from [DockerHub](https://hub.docker.com/r/ultralytics/ultralytics).
Pull the latest Ultralytics image:
```bash
# Set image name as a variable
t=ultralytics/ultralytics:latest-conda
# Pull the latest Ultralytics image from Docker Hub
sudo docker pull $t
```
Run the image:
```bash
# Run the Ultralytics image in a container with GPU support
sudo docker run -it --ipc=host --runtime=nvidia --gpus all $t # all GPUs
sudo docker run -it --ipc=host --runtime=nvidia --gpus '"device=2,3"' $t # specify GPUs
```
## Speeding Up Installation with Libmamba
If you're looking to [speed up the package installation](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community) process in Conda, you can opt to use `libmamba`, a fast, cross-platform, and dependency-aware package manager that serves as an alternative solver to Conda's default.
### How to Enable Libmamba
To enable `libmamba` as the solver for Conda, you can perform the following steps:
1. First, install the `conda-libmamba-solver` package. This can be skipped if your Conda version is 4.11 or above, as `libmamba` is included by default.
```bash
conda install conda-libmamba-solver
```
2. Next, configure Conda to use `libmamba` as the solver:
```bash
conda config --set solver libmamba
```
And that's it! Your Conda installation will now use `libmamba` as the solver, which should result in a faster package installation process.
---
You have successfully set up a Conda environment, installed the Ultralytics package, and are now ready to explore its features. For more advanced tutorials and examples, see the [Ultralytics documentation](../index.md).
## FAQ
### What is the process for setting up a Conda environment for Ultralytics projects?
Setting up a Conda environment for Ultralytics projects is straightforward and ensures smooth package management. First, create a new Conda environment using the following command:
```bash
conda create --name ultralytics-env python=3.11 -y
```
Then, activate the new environment with:
```bash
conda activate ultralytics-env
```
Finally, install Ultralytics from the conda-forge channel:
```bash
conda install -c conda-forge ultralytics
```
### Why should I use Conda over pip for managing dependencies in Ultralytics projects?
Conda is a robust package and environment management system that offers several advantages over pip. It manages dependencies efficiently and ensures that all necessary libraries are compatible. Conda's isolated environments prevent conflicts between packages, which is crucial in data science and machine learning projects. Additionally, Conda supports binary package distribution, speeding up the installation process.
### Can I use Ultralytics YOLO in a CUDA-enabled environment for faster performance?
Yes, you can enhance performance by utilizing a CUDA-enabled environment. Ensure that you install `ultralytics`, `pytorch`, and `pytorch-cuda` together to avoid conflicts:
```bash
conda install -c pytorch -c nvidia -c conda-forge pytorch torchvision pytorch-cuda=11.8 ultralytics
```
This setup enables GPU acceleration, crucial for intensive tasks like [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) model training and inference. For more information, visit the [Ultralytics installation guide](../quickstart.md).
### What are the benefits of using Ultralytics Docker images with a Conda environment?
Using Ultralytics Docker images ensures a consistent and reproducible environment, eliminating "it works on my machine" issues. These images include a pre-configured Conda environment, simplifying the setup process. You can pull and run the latest Ultralytics Docker image with the following commands:
```bash
sudo docker pull ultralytics/ultralytics:latest-conda
sudo docker run -it --ipc=host --runtime=nvidia --gpus all ultralytics/ultralytics:latest-conda # all GPUs
sudo docker run -it --ipc=host --runtime=nvidia --gpus '"device=2,3"' ultralytics/ultralytics:latest-conda # specify GPUs
```
This approach is ideal for deploying applications in production or running complex workflows without manual configuration. Learn more about [Ultralytics Conda Docker Image](../quickstart.md).
### How can I speed up Conda package installation in my Ultralytics environment?
You can speed up the package installation process by using `libmamba`, a fast dependency solver for Conda. First, install the `conda-libmamba-solver` package:
```bash
conda install conda-libmamba-solver
```
Then configure Conda to use `libmamba` as the solver:
```bash
conda config --set solver libmamba
```
This setup provides faster and more efficient package management. For more tips on optimizing your environment, read about [libmamba installation](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community).

287
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/coral-edge-tpu-on-raspberry-pi.md Normal file
View File

@@ -0,0 +1,287 @@
---
comments: true
description: Learn how to boost your Raspberry Pi's ML performance using Coral Edge TPU with Ultralytics YOLO26. Follow our detailed setup and installation guide.
keywords: Coral Edge TPU, Raspberry Pi, YOLO26, Ultralytics, TensorFlow Lite, ML inference, machine learning, AI, installation guide, setup tutorial
---
# Coral Edge TPU on a Raspberry Pi with Ultralytics YOLO26 🚀
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/edge-tpu-usb-accelerator-and-pi.avif" alt="Raspberry Pi with Edge TPU accelerator">
</p>
## What is a Coral Edge TPU?
The Coral Edge TPU is a compact device that adds an Edge TPU coprocessor to your system. It enables low-power, high-performance ML inference for [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) Lite models. Read more at the [Coral Edge TPU home page](https://developers.google.com/coral).
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/w4yHORvDBw0"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Run Inference on Raspberry Pi using Google Coral Edge TPU
</p>
## Boost Raspberry Pi Model Performance with Coral Edge TPU
Many people want to run their models on an embedded or mobile device such as a Raspberry Pi, since they are very power efficient and can be used in many different applications. However, the inference performance on these devices is usually poor even when using formats like [ONNX](../integrations/onnx.md) or [OpenVINO](../integrations/openvino.md). The Coral Edge TPU is a great solution to this problem, since it can be used with a Raspberry Pi and accelerate inference performance greatly.
## Edge TPU on Raspberry Pi with TensorFlow Lite (New)⭐
The [existing guide](https://gweb-coral-full.uc.r.appspot.com/docs/accelerator/get-started/) by Coral on how to use the Edge TPU with a Raspberry Pi is outdated, and the current Coral Edge TPU runtime builds do not work with the current TensorFlow Lite runtime versions anymore. In addition to that, Google seems to have completely abandoned the Coral project, and there have not been any updates between 2021 and 2025. This guide will show you how to get the Edge TPU working with the latest versions of the TensorFlow Lite runtime and an updated Coral Edge TPU runtime on a Raspberry Pi single board computer (SBC).
## Prerequisites
- [Raspberry Pi 4B](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/) (2GB or more recommended) or [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) (Recommended)
- [Raspberry Pi OS](https://www.raspberrypi.com/software/) Bullseye/Bookworm (64-bit) with desktop (Recommended)
- [Coral USB Accelerator](https://developers.google.com/coral)
- A non-ARM based platform for exporting an Ultralytics [PyTorch](https://www.ultralytics.com/glossary/pytorch) model
## Installation Walkthrough
This guide assumes that you already have a working Raspberry Pi OS install and have installed `ultralytics` and all dependencies. To get `ultralytics` installed, visit the [quickstart guide](../quickstart.md) to get set up before continuing here.
### Installing the Edge TPU runtime
First, we need to install the Edge TPU runtime. There are many different versions available, so you need to choose the right version for your operating system.
The high-frequency version runs the Edge TPU at a higher clock speed, which improves performance. However, it might result in Edge TPU thermal throttling, so it is recommended to have some sort of cooling mechanism in place.
| Raspberry Pi OS | High frequency mode | Version to download |
| --------------- | :-----------------: | ------------------------------------------ |
| Bullseye 32bit | No | `libedgetpu1-std_ ... .bullseye_armhf.deb` |
| Bullseye 64bit | No | `libedgetpu1-std_ ... .bullseye_arm64.deb` |
| Bullseye 32bit | Yes | `libedgetpu1-max_ ... .bullseye_armhf.deb` |
| Bullseye 64bit | Yes | `libedgetpu1-max_ ... .bullseye_arm64.deb` |
| Bookworm 32bit | No | `libedgetpu1-std_ ... .bookworm_armhf.deb` |
| Bookworm 64bit | No | `libedgetpu1-std_ ... .bookworm_arm64.deb` |
| Bookworm 32bit | Yes | `libedgetpu1-max_ ... .bookworm_armhf.deb` |
| Bookworm 64bit | Yes | `libedgetpu1-max_ ... .bookworm_arm64.deb` |
[Download the latest version from here](https://github.com/feranick/libedgetpu/releases).
After downloading the file, you can install it with the following command:
```bash
sudo dpkg -i path/to/package.deb
```
After installing the runtime, plug your Coral Edge TPU into a USB 3.0 port on the Raspberry Pi so the new `udev` rule can take effect.
???+ warning "Important"
If you already have the Coral Edge TPU runtime installed, uninstall it using the following command.
```bash
# If you installed the standard version
sudo apt remove libedgetpu1-std
# If you installed the high-frequency version
sudo apt remove libedgetpu1-max
```
## Export to Edge TPU
To use the Edge TPU, you need to convert your model into a compatible format. It is recommended that you run export on Google Colab, x86_64 Linux machine, using the official [Ultralytics Docker container](docker-quickstart.md), or using [Ultralytics Platform](../platform/quickstart.md), since the Edge TPU compiler is not available on ARM. See the [Export Mode](../modes/export.md) for the available arguments.
!!! example "Exporting the model"
=== "Python"
```python
from ultralytics import YOLO
# Load a model
model = YOLO("path/to/model.pt") # Load an official model or custom model
# Export the model
model.export(format="edgetpu")
```
=== "CLI"
```bash
yolo export model=path/to/model.pt format=edgetpu # Export an official model or custom model
```
The exported model will be saved in the `<model_name>_saved_model/` folder with the name `<model_name>_full_integer_quant_edgetpu.tflite`. Make sure the file name ends with the `_edgetpu.tflite` suffix; otherwise, Ultralytics will not detect that you're using an Edge TPU model.
## Running the model
Before you can actually run the model, you will need to install the correct libraries.
If you already have TensorFlow installed, uninstall it with the following command:
```bash
pip uninstall tensorflow tensorflow-aarch64
```
Then install or update `tflite-runtime`:
```bash
pip install -U tflite-runtime
```
Now you can run inference using the following code:
!!! example "Running the model"
=== "Python"
```python
from ultralytics import YOLO
# Load a model
model = YOLO("path/to/<model_name>_full_integer_quant_edgetpu.tflite") # Load an official model or custom model
# Run Prediction
model.predict("path/to/source.png")
```
=== "CLI"
```bash
yolo predict model=path/to/MODEL_NAME_full_integer_quant_edgetpu.tflite source=path/to/source.png # Load an official model or custom model
```
Find comprehensive information on the [Predict](../modes/predict.md) page for full prediction mode details.
!!! note "Inference with multiple Edge TPUs"
If you have multiple Edge TPUs, you can use the following code to select a specific TPU.
=== "Python"
```python
from ultralytics import YOLO
# Load a model
model = YOLO("path/to/<model_name>_full_integer_quant_edgetpu.tflite") # Load an official model or custom model
# Run Prediction
model.predict("path/to/source.png") # Inference defaults to the first TPU
model.predict("path/to/source.png", device="tpu:0") # Select the first TPU
model.predict("path/to/source.png", device="tpu:1") # Select the second TPU
```
## Benchmarks
!!! tip "Benchmarks"
Tested with Raspberry Pi OS Bookworm 64-bit and a USB Coral Edge TPU.
!!! note
Shown is the inference time, pre-/postprocessing is not included.
=== "Raspberry Pi 4B 2GB"
| Image Size | Model | Standard Inference Time (ms) | High-Frequency Inference Time (ms) |
|------------|---------|------------------------------|------------------------------------|
| 320 | YOLOv8n | 32.2 | 26.7 |
| 320 | YOLOv8s | 47.1 | 39.8 |
| 512 | YOLOv8n | 73.5 | 60.7 |
| 512 | YOLOv8s | 149.6 | 125.3 |
=== "Raspberry Pi 5 8GB"
| Image Size | Model | Standard Inference Time (ms) | High Frequency Inference Time (ms) |
|------------|---------|------------------------------|------------------------------------|
| 320 | YOLOv8n | 22.2 | 16.7 |
| 320 | YOLOv8s | 40.1 | 32.2 |
| 512 | YOLOv8n | 53.5 | 41.6 |
| 512 | YOLOv8s | 132.0 | 103.3 |
On average:
- The Raspberry Pi 5 is 22% faster with the standard mode than the Raspberry Pi 4B.
- The Raspberry Pi 5 is 30.2% faster with the high-frequency mode than the Raspberry Pi 4B.
- The high-frequency mode is 28.4% faster than the standard mode.
## FAQ
### What is a Coral Edge TPU and how does it enhance Raspberry Pi's performance with Ultralytics YOLO26?
The Coral Edge TPU is a compact device designed to add an Edge TPU coprocessor to your system. This coprocessor enables low-power, high-performance [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) inference, particularly optimized for TensorFlow Lite models. When using a Raspberry Pi, the Edge TPU accelerates ML model inference, significantly boosting performance, especially for Ultralytics YOLO26 models. You can read more about the Coral Edge TPU on their [home page](https://developers.google.com/coral).
### How do I install the Coral Edge TPU runtime on a Raspberry Pi?
To install the Coral Edge TPU runtime on your Raspberry Pi, download the appropriate `.deb` package for your Raspberry Pi OS version from [this link](https://github.com/feranick/libedgetpu/releases). Once downloaded, use the following command to install it:
```bash
sudo dpkg -i path/to/package.deb
```
Make sure to uninstall any previous Coral Edge TPU runtime versions by following the steps outlined in the [Installation Walkthrough](#installation-walkthrough) section.
### Can I export my Ultralytics YOLO26 model to be compatible with Coral Edge TPU?
Yes, you can export your Ultralytics YOLO26 model to be compatible with the Coral Edge TPU. It is recommended to perform the export on Google Colab, an x86_64 Linux machine, or using the [Ultralytics Docker container](docker-quickstart.md). You can also use [Ultralytics Platform](../platform/quickstart.md) for exporting. Here is how you can export your model using Python and CLI:
!!! example "Exporting the model"
=== "Python"
```python
from ultralytics import YOLO
# Load a model
model = YOLO("path/to/model.pt") # Load an official model or custom model
# Export the model
model.export(format="edgetpu")
```
=== "CLI"
```bash
yolo export model=path/to/model.pt format=edgetpu # Export an official model or custom model
```
For more information, refer to the [Export Mode](../modes/export.md) documentation.
### What should I do if TensorFlow is already installed on my Raspberry Pi, but I want to use tflite-runtime instead?
If you have TensorFlow installed on your Raspberry Pi and need to switch to `tflite-runtime`, you'll need to uninstall TensorFlow first using:
```bash
pip uninstall tensorflow tensorflow-aarch64
```
Then, install or update `tflite-runtime` with the following command:
```bash
pip install -U tflite-runtime
```
For detailed instructions, refer to the [Running the Model](#running-the-model) section.
### How do I run inference with an exported YOLO26 model on a Raspberry Pi using the Coral Edge TPU?
After exporting your YOLO26 model to an Edge TPU-compatible format, you can run inference using the following code snippets:
!!! example "Running the model"
=== "Python"
```python
from ultralytics import YOLO
# Load a model
model = YOLO("path/to/edgetpu_model.tflite") # Load an official model or custom model
# Run Prediction
model.predict("path/to/source.png")
```
=== "CLI"
```bash
yolo predict model=path/to/edgetpu_model.tflite source=path/to/source.png # Load an official model or custom model
```
Comprehensive details on full prediction mode features can be found on the [Predict Page](../modes/predict.md).

208
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/data-collection-and-annotation.md Normal file
View File

@@ -0,0 +1,208 @@
---
comments: true
description: Data collection and annotation are vital steps in any computer vision project. Explore the tools, techniques, and best practices for collecting and annotating data.
keywords: What is Data Annotation, Data Annotation Tools, Annotating Data, Avoiding Bias in Data Collection, Ethical Data Collection, Annotation Strategies
---
# Data Collection and Annotation Strategies for Computer Vision
## Introduction
The key to success in any [computer vision project](./steps-of-a-cv-project.md) starts with effective data collection and annotation strategies. The quality of the data directly impacts model performance, so it's important to understand the best practices related to data collection and data annotation.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/iBk6S-PHwS0"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Build Effective Data Collection and Annotation Strategies for Computer Vision 🚀
</p>
Every consideration regarding the data should closely align with [your project's goals](./defining-project-goals.md). Changes in your annotation strategies could shift the project's focus or effectiveness and vice versa. With this in mind, let's take a closer look at the best ways to approach data collection and annotation.
## Setting Up Classes and Collecting Data
Collecting images and video for a computer vision project involves defining the number of classes, sourcing data, and considering ethical implications. Before you start gathering your data, you need to be clear about:
### Choosing the Right Classes for Your Project
One of the first questions when starting a computer vision project is how many classes to include. You need to determine the class membership, which involves the different categories or labels that you want your model to recognize and differentiate. The number of classes should be determined by the specific goals of your project.
For example, if you want to monitor traffic, your classes might include "car," "truck," "bus," "motorcycle," and "bicycle." On the other hand, for tracking items in a store, your classes could be "fruits," "vegetables," "beverages," and "snacks." Defining classes based on your project goals helps keep your dataset relevant and focused.
When you define your classes, another important distinction to make is whether to choose coarse or fine class counts. 'Count' refers to the number of distinct classes you are interested in. This decision influences the granularity of your data and the complexity of your model. Here are the considerations for each approach:
- **Coarse Class-Count**: These are broader, more inclusive categories, such as "vehicle" and "non-vehicle." They simplify annotation and require fewer computational resources but provide less detailed information, potentially limiting the model's effectiveness in complex scenarios.
- **Fine Class-Count**: More categories with finer distinctions, such as "sedan," "SUV," "pickup truck," and "motorcycle." They capture more detailed information, improving model accuracy and performance. However, they are more time-consuming and labor-intensive to annotate and require more computational resources.
Starting with more specific classes can be very helpful, especially in complex projects where details are important. More specific classes let you collect more detailed data, gain deeper insights, and establish clearer distinctions between categories. Not only does it improve the accuracy of the model, but it also makes it easier to adjust the model later if needed, saving both time and resources.
### Sources of Data
You can use public datasets or gather your own custom data. Public datasets like those on [Kaggle](https://www.kaggle.com/datasets) and [Google Dataset Search Engine](https://datasetsearch.research.google.com/) offer well-annotated, standardized data, making them great starting points for training and validating models.
Custom data collection, on the other hand, allows you to customize your dataset to your specific needs. You might capture images and videos with cameras or drones, scrape the web for images, or use existing internal data from your organization. Custom data gives you more control over its quality and relevance. Combining both public and custom data sources helps create a diverse and comprehensive dataset.
### Avoiding Bias in Data Collection
Bias occurs when certain groups or scenarios are underrepresented or overrepresented in your dataset. It leads to a model that performs well on some data but poorly on others. It's crucial to avoid [bias in AI](https://www.ultralytics.com/glossary/bias-in-ai) so that your computer vision model can perform well in a variety of scenarios.
Here is how you can avoid bias while collecting data:
- **Diverse Sources**: Collect data from many sources to capture different perspectives and scenarios.
- **Balanced Representation**: Include balanced representation from all relevant groups. For example, consider different ages, genders, and ethnicities.
- **Continuous Monitoring**: Regularly review and update your dataset to identify and address any emerging biases.
- **Bias Mitigation Techniques**: Use methods like oversampling underrepresented classes, [data augmentation](https://www.ultralytics.com/glossary/data-augmentation), and fairness-aware algorithms.
Following these practices helps create a more robust and fair model that can generalize well in real-world applications.
## What is Data Annotation?
Data annotation is the process of labeling data to make it usable for training [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) models. In computer vision, this means labeling images or videos with the information that a model needs to learn from. Without properly annotated data, models cannot accurately learn the relationships between inputs and outputs.
### Types of Data Annotation
Depending on the specific requirements of a [computer vision task](../tasks/index.md), there are different types of data annotation. Here are some examples:
- **Bounding Boxes**: Rectangular boxes drawn around objects in an image, used primarily for object detection tasks. These boxes are defined by their top-left and bottom-right coordinates.
- **Polygons**: Detailed outlines for objects, allowing for more precise annotation than bounding boxes. Polygons are used in tasks like [instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation), where the shape of the object is important.
- **Masks**: Binary masks where each pixel is either part of an object or the background. Masks are used in [semantic segmentation](https://www.ultralytics.com/glossary/semantic-segmentation) tasks to provide pixel-level detail.
- **Keypoints**: Specific points marked within an image to identify locations of interest. Keypoints are used in tasks like [pose estimation](../tasks/pose.md) and facial landmark detection.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/types-of-data-annotation.avif" alt="Data annotation types including bounding boxes, polygons, and masks">
</p>
### Common Annotation Formats
After selecting a type of annotation, it's important to choose the appropriate format for storing and sharing annotations.
Commonly used formats include [COCO](../datasets/detect/coco.md), which supports various annotation types like [object detection](https://www.ultralytics.com/glossary/object-detection), keypoint detection, stuff segmentation, [panoptic segmentation](https://www.ultralytics.com/glossary/panoptic-segmentation), and image captioning, stored in JSON. [Pascal VOC](../datasets/detect/voc.md) uses XML files and is popular for object detection tasks. YOLO, on the other hand, creates a .txt file for each image, containing annotations like object class, coordinates, height, and width, making it suitable for object detection.
### Techniques of Annotation
Now, assuming you've chosen a type of annotation and format, it's time to establish clear and objective labeling rules. These rules are like a roadmap for consistency and [accuracy](https://www.ultralytics.com/glossary/accuracy) throughout the annotation process. Key aspects of these rules include:
- **Clarity and Detail**: Make sure your instructions are clear. Use examples and illustrations to show what's expected.
- **Consistency**: Keep your annotations uniform. Set standard criteria for annotating different types of data, so all annotations follow the same rules.
- **Reducing Bias**: Stay neutral. Train yourself to be objective and minimize personal biases to ensure fair annotations.
- **Efficiency**: Work smarter, not harder. Use tools and workflows that automate repetitive tasks, making the annotation process faster and more efficient.
Regularly reviewing and updating your labeling rules will help keep your annotations accurate, consistent, and aligned with your project goals.
### Popular Annotation Tools
Let's say you are ready to annotate now. There are several open-source tools available to help streamline the data annotation process. Here are some useful open annotation tools:
- **[Label Studio](https://github.com/HumanSignal/label-studio)**: A flexible tool that supports a wide range of annotation tasks and includes features for managing projects and quality control.
- **[CVAT](https://github.com/cvat-ai/cvat)**: A powerful tool that supports various annotation formats and customizable workflows, making it suitable for complex projects.
- **[Labelme](https://github.com/wkentaro/labelme)**: A simple and easy-to-use tool that allows for quick annotation of images with polygons, making it ideal for straightforward tasks.
- **[LabelImg](https://github.com/HumanSignal/labelImg)**: An easy-to-use graphical image annotation tool that's particularly good for creating bounding box annotations in YOLO format.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/labelme-instance-segmentation-annotation.avif" alt="LabelMe annotation tool for instance segmentation">
</p>
These open-source tools are budget-friendly and provide a range of features to meet different annotation needs.
### Some More Things to Consider Before Annotating Data
Before you dive into annotating your data, there are a few more things to keep in mind. You should be aware of accuracy, [precision](https://www.ultralytics.com/glossary/precision), outliers, and quality control to avoid labeling your data in a counterproductive manner.
#### Understanding Accuracy and Precision
It's important to understand the difference between accuracy and precision and how it relates to annotation. Accuracy refers to how close the annotated data is to the true values. It helps us measure how closely the labels reflect real-world scenarios. Precision indicates the consistency of annotations. It checks if you are giving the same label to the same object or feature throughout the dataset. High accuracy and precision lead to better-trained models by reducing noise and improving the model's ability to generalize from the [training data](https://www.ultralytics.com/glossary/training-data).
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/example-of-precision.avif" alt="Accuracy vs precision comparison for data annotation">
</p>
#### Identifying Outliers
Outliers are data points that deviate quite a bit from other observations in the dataset. With respect to annotations, an outlier could be an incorrectly labeled image or an annotation that doesn't fit with the rest of the dataset. Outliers are concerning because they can distort the model's learning process, leading to inaccurate predictions and poor generalization.
You can use various methods to detect and correct outliers:
- **Statistical Techniques**: To detect outliers in numerical features like pixel values, [bounding box](https://www.ultralytics.com/glossary/bounding-box) coordinates, or object sizes, you can use methods such as box plots, histograms, or z-scores.
- **Visual Techniques**: To spot anomalies in categorical features like object classes, colors, or shapes, use visual methods like plotting images, labels, or heat maps.
- **Algorithmic Methods**: Use tools like clustering (e.g., K-means clustering, [DBSCAN](https://www.ultralytics.com/glossary/dbscan-density-based-spatial-clustering-of-applications-with-noise)) and [anomaly detection](https://www.ultralytics.com/glossary/anomaly-detection) algorithms to identify outliers based on data distribution patterns.
#### Quality Control of Annotated Data
Just like other technical projects, quality control is a must for annotated data. It is a good practice to regularly check annotations to make sure they are accurate and consistent. This can be done in a few different ways:
- Reviewing samples of annotated data
- Using automated tools to spot common errors
- Having another person double-check the annotations
If you are working with multiple people, consistency between different annotators is important. Good inter-annotator agreement means that the guidelines are clear and everyone is following them the same way. It keeps everyone on the same page and the annotations consistent.
While reviewing, if you find errors, correct them and update the guidelines to avoid future mistakes. Provide feedback to annotators and offer regular training to help reduce errors. Having a strong process for handling errors keeps your dataset accurate and reliable.
## Efficient Data Labeling Strategies
To make the process of data labeling smoother and more effective, consider implementing these strategies:
- **Clear Annotation Guidelines**: Provide detailed instructions with examples to ensure all annotators interpret tasks consistently. For instance, when labeling birds, specify whether to include the entire bird or just specific parts.
- **Regular Quality Checks**: Set benchmarks and use specific metrics to review work, maintaining high standards through continuous feedback.
- **Use Pre-annotation Tools**: Many modern annotation platforms offer AI-assisted pre-annotation features that can significantly speed up the process by automatically generating initial annotations that humans can then refine.
- **Implement Active Learning**: This approach prioritizes labeling the most informative samples first, which can reduce the total number of annotations needed while maintaining model performance.
- **Batch Processing**: Group similar images together for annotation to maintain consistency and improve efficiency.
These strategies can help maintain high-quality annotations while reducing the time and resources required for the labeling process.
## Share Your Thoughts with the Community
Bouncing your ideas and queries off other [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) enthusiasts can help accelerate your projects. Here are some great ways to learn, troubleshoot, and network:
### Where to Find Help and Support
- **GitHub Issues:** Visit the YOLO26 GitHub repository and use the [Issues tab](https://github.com/ultralytics/ultralytics/issues) to raise questions, report bugs, and suggest features. The community and maintainers are there to help with any issues you face.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to connect with other users and developers, get support, share knowledge, and brainstorm ideas.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Refer to the [official YOLO26 documentation](./index.md) for thorough guides and valuable insights on numerous computer vision tasks and projects.
## Conclusion
By following the best practices for collecting and annotating data, avoiding bias, and using the right tools and techniques, you can significantly improve your model's performance. Engaging with the community and using available resources will keep you informed and help you troubleshoot issues effectively. Remember, quality data is the foundation of a successful project, and the right strategies will help you build robust and reliable models.
## FAQ
### What is the best way to avoid bias in data collection for computer vision projects?
Avoiding bias in data collection ensures that your computer vision model performs well across various scenarios. To minimize bias, consider collecting data from diverse sources to capture different perspectives and scenarios. Ensure balanced representation among all relevant groups, such as different ages, genders, and ethnicities. Regularly review and update your dataset to identify and address any emerging biases. Techniques such as oversampling underrepresented classes, data augmentation, and fairness-aware algorithms can also help mitigate bias. By employing these strategies, you maintain a robust and fair dataset that enhances your model's generalization capability.
### How can I ensure high consistency and accuracy in data annotation?
Ensuring high consistency and accuracy in data annotation involves establishing clear and objective labeling guidelines. Your instructions should be detailed, with examples and illustrations to clarify expectations. Consistency is achieved by setting standard criteria for annotating various data types, ensuring all annotations follow the same rules. To reduce personal biases, train annotators to stay neutral and objective. Regular reviews and updates of labeling rules help maintain accuracy and alignment with project goals. Using automated tools to check for consistency and getting feedback from other annotators also contribute to maintaining high-quality annotations.
### How many images do I need for training Ultralytics YOLO models?
For effective [transfer learning](https://www.ultralytics.com/glossary/transfer-learning) and object detection with Ultralytics YOLO models, start with a minimum of a few hundred annotated objects per class. If training for just one class, begin with at least 100 annotated images and train for approximately 100 [epochs](https://www.ultralytics.com/glossary/epoch). More complex tasks might require thousands of images per class to achieve high reliability and performance. Quality annotations are crucial, so ensure your data collection and annotation processes are rigorous and aligned with your project's specific goals. Explore detailed training strategies in the [YOLO26 training guide](../modes/train.md).
### What are some popular tools for data annotation?
Several popular open-source tools can streamline the data annotation process:
- **[Label Studio](https://github.com/HumanSignal/label-studio)**: A flexible tool supporting various annotation tasks, project management, and quality control features.
- **[CVAT](https://www.cvat.ai/)**: Offers multiple annotation formats and customizable workflows, making it suitable for complex projects.
- **[Labelme](https://github.com/wkentaro/labelme)**: Ideal for quick and straightforward image annotation with polygons.
- **[LabelImg](https://github.com/HumanSignal/labelImg)**: Perfect for creating bounding box annotations in YOLO format with a simple interface.
These tools can help enhance the efficiency and accuracy of your annotation workflows. For extensive feature lists and guides, refer to our [data annotation tools documentation](../datasets/index.md).
### What types of data annotation are commonly used in computer vision?
Different types of data annotation cater to various computer vision tasks:
- **Bounding Boxes**: Used primarily for object detection, these are rectangular boxes around objects in an image.
- **Polygons**: Provide more precise object outlines suitable for instance segmentation tasks.
- **Masks**: Offer pixel-level detail, used in semantic segmentation to differentiate objects from the background.
- **Keypoints**: Identify specific points of interest within an image, useful for tasks like pose estimation and facial landmark detection.
Selecting the appropriate annotation type depends on your project's requirements. Learn more about how to implement these annotations and their formats in our [data annotation guide](#what-is-data-annotation).

433
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/deepstream-nvidia-jetson.md Normal file
View File

@@ -0,0 +1,433 @@
---
comments: true
description: Learn how to deploy Ultralytics YOLO26 on NVIDIA Jetson devices using TensorRT and DeepStream SDK. Explore performance benchmarks and maximize AI capabilities.
keywords: Ultralytics, YOLO26, NVIDIA Jetson, JetPack, AI deployment, embedded systems, deep learning, TensorRT, DeepStream SDK, computer vision
---
# Ultralytics YOLO26 on NVIDIA Jetson using DeepStream SDK and TensorRT
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/hvGqrVT2wPg"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to use Ultralytics YOLO26 models with NVIDIA Deepstream on Jetson Orin NX 🚀
</p>
This comprehensive guide provides a detailed walkthrough for deploying Ultralytics YOLO26 on [NVIDIA Jetson](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/) devices using DeepStream SDK and TensorRT. Here we use TensorRT to maximize the inference performance on the Jetson platform.
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/deepstream-nvidia-jetson.avif" alt="NVIDIA DeepStream SDK on Jetson platform">
!!! note
This guide has been tested with [NVIDIA Jetson Orin Nano Super Developer Kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit) running the latest stable JetPack release of [JP6.1](https://developer.nvidia.com/embedded/jetpack-sdk-61),
[Seeed Studio reComputer J4012](https://www.seeedstudio.com/reComputer-J4012-p-5586.html) which is based on NVIDIA Jetson Orin NX 16GB running JetPack release of [JP5.1.3](https://developer.nvidia.com/embedded/jetpack-sdk-513) and [Seeed Studio reComputer J1020 v2](https://www.seeedstudio.com/reComputer-J1020-v2-p-5498.html) which is based on NVIDIA Jetson Nano 4GB running JetPack release of [JP4.6.4](https://developer.nvidia.com/jetpack-sdk-464). It is expected to work across all the NVIDIA Jetson hardware lineup including latest and legacy.
## What is NVIDIA DeepStream?
[NVIDIA's DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) is a complete streaming analytics toolkit based on GStreamer for AI-based multi-sensor processing, video, audio, and image understanding. It's ideal for vision AI developers, software partners, startups, and OEMs building IVA (Intelligent Video Analytics) apps and services. You can now create stream-processing pipelines that incorporate [neural networks](https://www.ultralytics.com/glossary/neural-network-nn) and other complex processing tasks like tracking, video encoding/decoding, and video rendering. These pipelines enable real-time analytics on video, image, and sensor data. DeepStream's multi-platform support gives you a faster, easier way to develop vision AI applications and services on-premise, at the edge, and in the cloud.
## Prerequisites
Before you start to follow this guide:
- Visit our documentation, [Quick Start Guide: NVIDIA Jetson with Ultralytics YOLO26](nvidia-jetson.md) to set up your NVIDIA Jetson device with Ultralytics YOLO26
- Install [DeepStream SDK](https://developer.nvidia.com/deepstream-getting-started) according to the JetPack version
- For JetPack 4.6.4, install [DeepStream 6.0.1](https://docs.nvidia.com/metropolis/deepstream/6.0.1/dev-guide/text/DS_Quickstart.html)
- For JetPack 5.1.3, install [DeepStream 6.3](https://docs.nvidia.com/metropolis/deepstream/6.3/dev-guide/text/DS_Quickstart.html)
- For JetPack 6.1, install [DeepStream 7.1](https://docs.nvidia.com/metropolis/deepstream/7.0/dev-guide/text/DS_Overview.html)
!!! tip
In this guide we have used the Debian package method of installing DeepStream SDK to the Jetson device. You can also visit the [DeepStream SDK on Jetson (Archived)](https://developer.nvidia.com/embedded/deepstream-on-jetson-downloads-archived) to access legacy versions of DeepStream.
## DeepStream Configuration for YOLO26
Here we are using [marcoslucianops/DeepStream-Yolo](https://github.com/marcoslucianops/DeepStream-Yolo) GitHub repository which includes NVIDIA DeepStream SDK support for YOLO models. We appreciate the efforts of marcoslucianops for his contributions!
1. Install Ultralytics with necessary dependencies
```bash
cd ~
pip install -U pip
git clone https://github.com/ultralytics/ultralytics
cd ultralytics
pip install -e ".[export]" onnxslim
```
2. Clone the DeepStream-Yolo repository
```bash
cd ~
git clone https://github.com/marcoslucianops/DeepStream-Yolo
```
3. Copy the `export_yolo26.py` file from `DeepStream-Yolo/utils` directory to the `ultralytics` folder
```bash
cp ~/DeepStream-Yolo/utils/export_yolo26.py ~/ultralytics
cd ultralytics
```
4. Download Ultralytics YOLO26 detection model (.pt) of your choice from [YOLO26 releases](https://github.com/ultralytics/assets/releases). Here we use [yolo26s.pt](https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26s.pt).
```bash
wget https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo26s.pt
```
!!! note
You can also use a [custom-trained YOLO26 model](https://docs.ultralytics.com/modes/train/).
5. Convert model to ONNX
```bash
python3 export_yolo26.py -w yolo26s.pt
```
!!! note "Pass the below arguments to the above command"
For DeepStream 5.1, remove the `--dynamic` arg and use `opset` 12 or lower. The default `opset` is 17.
```bash
--opset 12
```
To change the inference size (default: 640)
```bash
-s SIZE
--size SIZE
-s HEIGHT WIDTH
--size HEIGHT WIDTH
```
Example for 1280:
```bash
-s 1280
or
-s 1280 1280
```
To simplify the ONNX model (DeepStream >= 6.0)
```bash
--simplify
```
To use dynamic batch-size (DeepStream >= 6.1)
```bash
--dynamic
```
To use static batch-size (example for batch-size = 4)
```bash
--batch 4
```
6. Copy the generated `.onnx` model file and `labels.txt` file to the `DeepStream-Yolo` folder
```bash
cp yolo26s.pt.onnx labels.txt ~/DeepStream-Yolo
cd ~/DeepStream-Yolo
```
7. Set the CUDA version according to the JetPack version installed
For JetPack 4.6.4:
```bash
export CUDA_VER=10.2
```
For JetPack 5.1.3:
```bash
export CUDA_VER=11.4
```
For JetPack 6.1:
```bash
export CUDA_VER=12.6
```
8. Compile the library
```bash
make -C nvdsinfer_custom_impl_Yolo clean && make -C nvdsinfer_custom_impl_Yolo
```
9. Edit the `config_infer_primary_yolo26.txt` file according to your model (for YOLO26s with 80 classes)
```bash
[property]
...
onnx-file=yolo26s.pt.onnx
...
num-detected-classes=80
...
```
10. Edit the `deepstream_app_config` file
```bash
...
[primary-gie]
...
config-file=config_infer_primary_yolo26.txt
```
11. You can also change the video source in `deepstream_app_config` file. Here, a default video file is loaded
```bash
...
[source0]
...
uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
```
### Run Inference
```bash
deepstream-app -c deepstream_app_config.txt
```
!!! note
It will take a long time to generate the TensorRT engine file before starting the inference. So please be patient.
<div align=center><img width=1000 src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/yolov8-with-deepstream.avif" alt="YOLO26 with deepstream"></div>
!!! tip
If you want to convert the model to FP16 precision, simply set `model-engine-file=model_b1_gpu0_fp16.engine` and `network-mode=2` inside `config_infer_primary_yolo26.txt`
## INT8 Calibration
If you want to use INT8 precision for inference, you need to follow the steps below:
!!! note
Currently INT8 does not work with TensorRT 10.x. This section of the guide has been tested with TensorRT 8.x which is expected to work.
1. Set `OPENCV` environment variable
```bash
export OPENCV=1
```
2. Compile the library
```bash
make -C nvdsinfer_custom_impl_Yolo clean && make -C nvdsinfer_custom_impl_Yolo
```
3. For COCO dataset, download the [val2017](http://images.cocodataset.org/zips/val2017.zip), extract, and move to `DeepStream-Yolo` folder
4. Make a new directory for calibration images
```bash
mkdir calibration
```
5. Run the following to select 1000 random images from COCO dataset to run calibration
```bash
for jpg in $(ls -1 val2017/*.jpg | sort -R | head -1000); do
cp ${jpg} calibration/
done
```
!!! note
NVIDIA recommends at least 500 images to get a good [accuracy](https://www.ultralytics.com/glossary/accuracy). On this example, 1000 images are chosen to get better accuracy (more images = more accuracy). You can set it from **head -1000**. For example, for 2000 images, **head -2000**. This process can take a long time.
6. Create the `calibration.txt` file with all selected images
```bash
realpath calibration/*jpg > calibration.txt
```
7. Set environment variables
```bash
export INT8_CALIB_IMG_PATH=calibration.txt
export INT8_CALIB_BATCH_SIZE=1
```
!!! note
Higher INT8_CALIB_BATCH_SIZE values will result in more accuracy and faster calibration speed. Set it according to your GPU memory.
8. Update the `config_infer_primary_yolo26.txt` file
From
```bash
...
model-engine-file=model_b1_gpu0_fp32.engine
#int8-calib-file=calib.table
...
network-mode=0
...
```
To
```bash
...
model-engine-file=model_b1_gpu0_int8.engine
int8-calib-file=calib.table
...
network-mode=1
...
```
### Run Inference
```bash
deepstream-app -c deepstream_app_config.txt
```
## MultiStream Setup
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/wWmXKIteRLA"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Run Multiple Streams with DeepStream SDK on Jetson Nano using Ultralytics YOLO26 🎉
</p>
To set up multiple streams under a single DeepStream application, make the following changes to the `deepstream_app_config.txt` file:
1. Change the rows and columns to build a grid display according to the number of streams you want to have. For example, for 4 streams, we can add 2 rows and 2 columns.
```bash
[tiled-display]
rows=2
columns=2
```
2. Set `num-sources=4` and add the `uri` entries for all four streams.
```bash
[source0]
enable=1
type=3
uri=path/to/video1.jpg
uri=path/to/video2.jpg
uri=path/to/video3.jpg
uri=path/to/video4.jpg
num-sources=4
```
### Run Inference
```bash
deepstream-app -c deepstream_app_config.txt
```
<div align=center><img width=1000 src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/multistream-setup.avif" alt="DeepStream multi-camera streaming configuration"></div>
## Benchmark Results
The following benchmarks summarizes how YOLO26 models perform at different TensorRT precision levels with an input size of 640x640 on NVIDIA Jetson Orin NX 16GB.
### Comparison Chart
<div align=center><img width=1000 src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-deepstream-benchmarks.avif" alt="NVIDIA Jetson DeepStream performance benchmarks"></div>
### Detailed Comparison Table
!!! tip "Performance"
=== "YOLO11n"
| Format | Status | Inference time (ms/im) |
|-----------------|--------|------------------------|
| TensorRT (FP32) | ✅ | 8.64 |
| TensorRT (FP16) | ✅ | 5.27 |
| TensorRT (INT8) | ✅ | 4.54 |
=== "YOLO11s"
| Format | Status | Inference time (ms/im) |
|-----------------|--------|------------------------|
| TensorRT (FP32) | ✅ | 14.53 |
| TensorRT (FP16) | ✅ | 7.91 |
| TensorRT (INT8) | ✅ | 6.05 |
=== "YOLO11m"
| Format | Status | Inference time (ms/im) |
|-----------------|--------|------------------------|
| TensorRT (FP32) | ✅ | 32.05 |
| TensorRT (FP16) | ✅ | 15.55 |
| TensorRT (INT8) | ✅ | 10.43 |
=== "YOLO11l"
| Format | Status | Inference time (ms/im) |
|-----------------|--------|------------------------|
| TensorRT (FP32) | ✅ | 39.68 |
| TensorRT (FP16) | ✅ | 19.88 |
| TensorRT (INT8) | ✅ | 13.64 |
=== "YOLO11x"
| Format | Status | Inference time (ms/im) |
|-----------------|--------|------------------------|
| TensorRT (FP32) | ✅ | 80.65 |
| TensorRT (FP16) | ✅ | 39.06 |
| TensorRT (INT8) | ✅ | 22.83 |
## Acknowledgments
This guide was initially created by our friends at Seeed Studio, Lakshantha and Elaine.
## FAQ
### How do I set up Ultralytics YOLO26 on an NVIDIA Jetson device?
To set up Ultralytics YOLO26 on an [NVIDIA Jetson](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/) device, you first need to install the [DeepStream SDK](https://developer.nvidia.com/deepstream-getting-started) compatible with your JetPack version. Follow the step-by-step guide in our [Quick Start Guide](nvidia-jetson.md) to configure your NVIDIA Jetson for YOLO26 deployment.
### What is the benefit of using TensorRT with YOLO26 on NVIDIA Jetson?
Using TensorRT with YOLO26 optimizes the model for inference, significantly reducing latency and improving throughput on NVIDIA Jetson devices. TensorRT provides high-performance, low-latency [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) inference through layer fusion, precision calibration, and kernel auto-tuning. This leads to faster and more efficient execution, particularly useful for real-time applications like video analytics and autonomous machines.
### Can I run Ultralytics YOLO26 with DeepStream SDK across different NVIDIA Jetson hardware?
Yes, the guide for deploying Ultralytics YOLO26 with the DeepStream SDK and TensorRT is compatible across the entire NVIDIA Jetson lineup. This includes devices like the Jetson Orin NX 16GB with [JetPack 5.1.3](https://developer.nvidia.com/embedded/jetpack-sdk-513) and the Jetson Nano 4GB with [JetPack 4.6.4](https://developer.nvidia.com/jetpack-sdk-464). Refer to the section [DeepStream Configuration for YOLO26](#deepstream-configuration-for-yolo26) for detailed steps.
### How can I convert a YOLO26 model to ONNX for DeepStream?
To convert a YOLO26 model to ONNX format for deployment with DeepStream, use the `utils/export_yolo26.py` script from the [DeepStream-Yolo](https://github.com/marcoslucianops/DeepStream-Yolo) repository.
Here's an example command:
```bash
python3 utils/export_yolo26.py -w yolo26s.pt --opset 12 --simplify
```
For more details on model conversion, check out our [model export section](../modes/export.md).
### What are the performance benchmarks for YOLO on NVIDIA Jetson Orin NX?
The performance of YOLO26 models on NVIDIA Jetson Orin NX 16GB varies based on TensorRT precision levels. For example, YOLO26s models achieve:
- **FP32 Precision**: 14.6 ms/im, 68.5 FPS
- **FP16 Precision**: 7.94 ms/im, 126 FPS
- **INT8 Precision**: 5.95 ms/im, 168 FPS
These benchmarks underscore the efficiency and capability of using TensorRT-optimized YOLO26 models on NVIDIA Jetson hardware. For further details, see our [Benchmark Results](#benchmark-results) section.

187
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/defining-project-goals.md Normal file
View File

@@ -0,0 +1,187 @@
---
comments: true
description: Learn how to define clear goals and objectives for your computer vision project with our practical guide. Includes tips on problem statements, measurable objectives, and key decisions.
keywords: computer vision, project planning, problem statement, measurable objectives, dataset preparation, model selection, YOLO26, Ultralytics
---
# A Practical Guide for Defining Your Computer Vision Project
## Introduction
The first step in any computer vision project is defining what you want to achieve. It's crucial to have a clear roadmap from the start, which includes everything from data collection to deploying your model.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/q1tXfShvbAw"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to define Computer Vision Project's Goal | Problem Statement and VisionAI Tasks Connection 🚀
</p>
If you need a quick refresher on the basics of a computer vision project, take a moment to read our guide on [the key steps in a computer vision project](./steps-of-a-cv-project.md). It will give you a solid overview of the whole process. Once you're caught up, come back here to dive into how exactly you can define and refine the goals for your project.
Now, let's get to the heart of defining a clear problem statement for your project and exploring the key decisions you'll need to make along the way.
## Defining A Clear Problem Statement
Setting clear goals and objectives for your project is the first big step toward finding the most effective solutions. Let's understand how you can clearly define your project's problem statement:
- **Identify the Core Issue:** Pinpoint the specific challenge your computer vision project aims to solve.
- **Determine the Scope:** Define the boundaries of your problem.
- **Consider End Users and Stakeholders:** Identify who will be affected by the solution.
- **Analyze Project Requirements and Constraints:** Assess available resources (time, budget, personnel) and identify any technical or regulatory constraints.
### Example of a Business Problem Statement
Let's walk through an example.
Consider a computer vision project where you want to [estimate the speed of vehicles](./speed-estimation.md) on a highway. The core issue is that current speed monitoring methods are inefficient and error-prone due to outdated radar systems and manual processes. The project aims to develop a real-time computer vision system that can replace legacy [speed estimation](https://www.ultralytics.com/blog/ultralytics-yolov8-for-speed-estimation-in-computer-vision-projects) systems.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/speed-estimation-using-yolov8.avif" alt="Speed Estimation Using YOLO26">
</p>
Primary users include traffic management authorities and law enforcement, while secondary stakeholders are highway planners and the public benefiting from safer roads. Key requirements involve evaluating budget, time, and personnel, as well as addressing technical needs like high-resolution cameras and real-time data processing. Additionally, regulatory constraints on privacy and [data security](https://www.ultralytics.com/glossary/data-security) must be considered.
### Setting Measurable Objectives
Setting measurable objectives is key to the success of a computer vision project. These goals should be clear, achievable, and time-bound.
For example, if you are developing a system to estimate vehicle speeds on a highway, you could consider the following measurable objectives:
- To achieve at least 95% [accuracy](https://www.ultralytics.com/glossary/accuracy) in speed detection within six months, using a dataset of 10,000 vehicle images.
- The system should be able to process real-time video feeds at 30 frames per second with minimal delay.
By setting specific and quantifiable goals, you can effectively track progress, identify areas for improvement, and ensure the project stays on course.
## The Connection Between The Problem Statement and The Computer Vision Tasks
Your problem statement helps you conceptualize which computer vision task can solve your issue.
For example, if your problem is monitoring vehicle speeds on a highway, the relevant computer vision task is object tracking. [Object tracking](../modes/track.md) is suitable because it allows the system to continuously follow each vehicle in the video feed, which is crucial for accurately calculating their speeds.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/example-of-object-tracking.avif" alt="YOLO vehicle tracking on highway">
</p>
Other tasks, like [object detection](../tasks/detect.md), are not suitable as they don't provide continuous location or movement information. Once you've identified the appropriate computer vision task, it guides several critical aspects of your project, like model selection, dataset preparation, and model training approaches.
## Which Comes First: Model Selection, Dataset Preparation, or Model Training Approach?
The order of model selection, dataset preparation, and training approach depends on the specifics of your project. Here are a few tips to help you decide:
- **Clear Understanding of the Problem**: If your problem and objectives are well-defined, start with model selection. Then, prepare your dataset and decide on the training approach based on the model's requirements.
- **Example**: Start by selecting a model for a traffic monitoring system that estimates vehicle speeds. Choose an object tracking model, gather and annotate highway videos, and then train the model with techniques for real-time video processing.
- **Unique or Limited Data**: If your project is constrained by unique or limited data, begin with dataset preparation. For instance, if you have a rare dataset of medical images, annotate and prepare the data first. Then, select a model that performs well on such data, followed by choosing a suitable training approach.
- **Example**: Prepare the data first for a facial recognition system with a small dataset. Annotate it, then select a model that works well with limited data, such as a pretrained model for [transfer learning](https://www.ultralytics.com/glossary/transfer-learning). Finally, decide on a training approach, including [data augmentation](https://www.ultralytics.com/glossary/data-augmentation), to expand the dataset.
- **Need for Experimentation**: In projects where experimentation is crucial, start with the training approach. This is common in research projects where you might initially test different training techniques. Refine your model selection after identifying a promising method and prepare the dataset based on your findings.
- **Example**: In a project exploring new methods for detecting manufacturing defects, start with experimenting on a small data subset. Once you find a promising technique, select a model tailored to those findings and prepare a comprehensive dataset.
## Common Discussion Points in the Community
Next, let's look at a few common discussion points in the community regarding computer vision tasks and project planning.
### What Are the Different Computer Vision Tasks?
The most popular computer vision tasks include [image classification](https://www.ultralytics.com/glossary/image-classification), [object detection](https://www.ultralytics.com/glossary/object-detection), and [image segmentation](https://www.ultralytics.com/glossary/image-segmentation).
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/image-classification-vs-object-detection-vs-image-segmentation.avif" alt="Classification vs detection vs segmentation comparison">
</p>
For a detailed explanation of various tasks, please take a look at the Ultralytics Docs page on [YOLO26 Tasks](../tasks/index.md).
### Can a Pretrained Model Remember Classes It Knew Before Custom Training?
No, pretrained models don't "remember" classes in the traditional sense. They learn patterns from massive datasets, and during custom training (fine-tuning), these patterns are adjusted for your specific task. The model's capacity is limited, and focusing on new information can overwrite some previous learnings.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/overview-of-transfer-learning.avif" alt="Transfer learning from pretrained to custom model">
</p>
If you want to use the classes the model was pretrained on, a practical approach is to use two models: one retains the original performance, and the other is fine-tuned for your specific task. This way, you can combine the outputs of both models. There are other options like freezing layers, using the pretrained model as a feature extractor, and task-specific branching, but these are more complex solutions and require more expertise.
### How Do Deployment Options Affect My Computer Vision Project?
[Model deployment options](./model-deployment-options.md) critically impact the performance of your computer vision project. For instance, the deployment environment must handle the computational load of your model. Here are some practical examples:
- **Edge Devices**: Deploying on edge devices like smartphones or IoT devices requires lightweight models due to their limited computational resources. Example technologies include [TensorFlow Lite](../integrations/tflite.md) and [ONNX Runtime](../integrations/onnx.md), which are optimized for such environments.
- **Cloud Servers**: Cloud deployments can handle more complex models with larger computational demands. Cloud platforms like [AWS](../integrations/amazon-sagemaker.md), Google Cloud, and Azure offer robust hardware options that can scale based on the project's needs.
- **On-Premise Servers**: For scenarios requiring high [data privacy](https://www.ultralytics.com/glossary/data-privacy) and security, deploying on-premise might be necessary. This involves significant upfront hardware investment but allows full control over the data and infrastructure.
- **Hybrid Solutions**: Some projects might benefit from a hybrid approach, where some processing is done on the edge, while more complex analyses are offloaded to the cloud. This can balance performance needs with cost and latency considerations.
Each deployment option offers different benefits and challenges, and the choice depends on specific project requirements like performance, cost, and security.
## Connecting with the Community
Connecting with other computer vision enthusiasts can be incredibly helpful for your projects by providing support, solutions, and new ideas. Here are some great ways to learn, troubleshoot, and network:
### Community Support Channels
- **GitHub Issues:** Head over to the YOLO26 GitHub repository. You can use the [Issues tab](https://github.com/ultralytics/ultralytics/issues) to raise questions, report bugs, and suggest features. The community and maintainers can assist with specific problems you encounter.
- **Ultralytics Discord Server:** Become part of the [Ultralytics Discord server](https://discord.com/invite/ultralytics). Connect with fellow users and developers, seek support, exchange knowledge, and discuss ideas.
### Comprehensive Guides and Documentation
- **Ultralytics YOLO26 Documentation:** Explore the [official YOLO26 documentation](./index.md) for in-depth guides and valuable tips on various computer vision tasks and projects.
## Conclusion
Defining a clear problem and setting measurable goals is key to a successful computer vision project. We've highlighted the importance of being clear and focused from the start. Having specific goals helps avoid oversight. Also, staying connected with others in the community through platforms like [GitHub](https://github.com/ultralytics/ultralytics) or [Discord](https://discord.com/invite/ultralytics) is important for learning and staying current. In short, good planning and engaging with the community are a huge part of successful computer vision projects.
## FAQ
### How do I define a clear problem statement for my Ultralytics computer vision project?
To define a clear problem statement for your Ultralytics computer vision project, follow these steps:
1. **Identify the Core Issue:** Pinpoint the specific challenge your project aims to solve.
2. **Determine the Scope:** Clearly outline the boundaries of your problem.
3. **Consider End Users and Stakeholders:** Identify who will be affected by your solution.
4. **Analyze Project Requirements and Constraints:** Assess available resources and any technical or regulatory limitations.
Providing a well-defined problem statement ensures that the project remains focused and aligned with your objectives. For a detailed guide, refer to our [practical guide](#defining-a-clear-problem-statement).
### Why should I use Ultralytics YOLO26 for speed estimation in my computer vision project?
Ultralytics YOLO26 is ideal for speed estimation because of its real-time object tracking capabilities, high accuracy, and robust performance in detecting and monitoring vehicle speeds. It overcomes inefficiencies and inaccuracies of traditional radar systems by leveraging cutting-edge computer vision technology. Check out our blog on [speed estimation using YOLO26](https://www.ultralytics.com/blog/ultralytics-yolov8-for-speed-estimation-in-computer-vision-projects) for more insights and practical examples.
### How do I set effective measurable objectives for my computer vision project with Ultralytics YOLO26?
Set effective and measurable objectives using the SMART criteria:
- **Specific:** Define clear and detailed goals.
- **Measurable:** Ensure objectives are quantifiable.
- **Achievable:** Set realistic targets within your capabilities.
- **Relevant:** Align objectives with your overall project goals.
- **Time-bound:** Set deadlines for each objective.
For example, "Achieve 95% accuracy in speed detection within six months using a 10,000 vehicle image dataset." This approach helps track progress and identifies areas for improvement. Read more about [setting measurable objectives](#setting-measurable-objectives).
### How do deployment options affect the performance of my Ultralytics YOLO models?
Deployment options critically impact the performance of your Ultralytics YOLO models. Here are key options:
- **Edge Devices:** Use lightweight models like [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) Lite or ONNX Runtime for deployment on devices with limited resources.
- **Cloud Servers:** Utilize robust cloud platforms like AWS, Google Cloud, or Azure for handling complex models.
- **On-Premise Servers:** High data privacy and security needs may require on-premise deployments.
- **Hybrid Solutions:** Combine edge and cloud approaches for balanced performance and cost-efficiency.
For more information, refer to our [detailed guide on model deployment options](./model-deployment-options.md).
### What are the most common challenges in defining the problem for a computer vision project with Ultralytics?
Common challenges include:
- Vague or overly broad problem statements.
- Unrealistic objectives.
- Lack of stakeholder alignment.
- Insufficient understanding of technical constraints.
- Underestimating data requirements.
Address these challenges through thorough initial research, clear communication with stakeholders, and iterative refinement of the problem statement and objectives. Learn more about these challenges in our [Computer Vision Project guide](steps-of-a-cv-project.md).

161
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/distance-calculation.md Normal file
View File

@@ -0,0 +1,161 @@
---
comments: true
description: Learn how to calculate distances between objects using Ultralytics YOLO26 for accurate spatial positioning and scene understanding.
keywords: Ultralytics, YOLO26, distance calculation, computer vision, object tracking, spatial positioning
---
# Distance Calculation using Ultralytics YOLO26
## What is Distance Calculation?
Measuring the gap between two objects is known as distance calculation within a specified space. In the case of [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics), the [bounding box](https://www.ultralytics.com/glossary/bounding-box) centroid is employed to calculate the distance for bounding boxes highlighted by the user.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Oe0vmsvnY74"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to estimate distance between detected objects with Ultralytics YOLO in Pixels 🚀
</p>
## Visuals
| Distance Calculation using Ultralytics YOLO26 |
| :----------------------------------------------------------------------------------------------------------------------------: |
| ![Ultralytics YOLO26 Distance Calculation](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/distance-calculation.avif) |
## Advantages of Distance Calculation
- **Localization [Precision](https://www.ultralytics.com/glossary/precision):** Enhances accurate spatial positioning in [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks.
- **Size Estimation:** Allows estimation of object size for better contextual understanding.
- **Scene Understanding:** Improves 3D scene comprehension for better decision-making in applications like [autonomous vehicles](https://www.ultralytics.com/glossary/autonomous-vehicles) and surveillance systems.
- **Collision Avoidance:** Enables systems to detect potential collisions by monitoring distances between moving objects.
- **Spatial Analysis:** Facilitates analysis of object relationships and interactions within the monitored environment.
???+ tip "Distance Calculation"
- Click any two bounding boxes with the left mouse button to calculate distance.
- Use the right mouse button to delete all drawn points.
- Left-click anywhere in the frame to add new points.
???+ warning "Distance is an estimate"
Distance is an estimate and may not be fully accurate because it is calculated using 2D data,
which lacks depth information.
!!! example "Distance Calculation using Ultralytics YOLO"
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("distance_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize distance calculation object
distancecalculator = solutions.DistanceCalculation(
model="yolo26n.pt", # path to the YOLO26 model file.
show=True, # display the output
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = distancecalculator(im0)
print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `DistanceCalculation()` Arguments
Here's a table with the `DistanceCalculation` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model"]) }}
You can also make use of various `track` arguments in the `DistanceCalculation` solution.
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization arguments are available:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## Implementation Details
The `DistanceCalculation` class works by tracking objects across video frames and calculating the Euclidean distance between the centroids of selected bounding boxes. When you click on two objects, the solution:
1. Extracts the centroids (center points) of the selected bounding boxes
2. Calculates the Euclidean distance between these centroids in pixels
3. Displays the distance on the frame with a connecting line between the objects
The implementation uses the `mouse_event_for_distance` method to handle mouse interactions, allowing users to select objects and clear selections as needed. The `process` method handles the frame-by-frame processing, tracking objects, and calculating distances.
## Applications
Distance calculation with YOLO26 has numerous practical applications:
- **Retail Analytics:** Measure customer proximity to products and analyze store layout effectiveness
- **Industrial Safety:** Monitor safe distances between workers and machinery
- **Traffic Management:** Analyze vehicle spacing and detect tailgating
- **Sports Analysis:** Calculate distances between players, the ball, and key field positions
- **Healthcare:** Ensure proper distancing in waiting areas and monitor patient movement
- **Robotics:** Enable robots to maintain appropriate distances from obstacles and people
## FAQ
### How do I calculate distances between objects using Ultralytics YOLO26?
To calculate distances between objects using [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics), you need to identify the bounding box centroids of the detected objects. This process involves initializing the `DistanceCalculation` class from Ultralytics' `solutions` module and using the model's tracking outputs to calculate the distances.
### What are the advantages of using distance calculation with Ultralytics YOLO26?
Using distance calculation with Ultralytics YOLO26 offers several advantages:
- **Localization Precision:** Provides accurate spatial positioning for objects.
- **Size Estimation:** Helps estimate physical sizes, contributing to better contextual understanding.
- **Scene Understanding:** Enhances 3D scene comprehension, aiding improved decision-making in applications like autonomous driving and surveillance.
- **Real-time Processing:** Performs calculations on-the-fly, making it suitable for live video analysis.
- **Integration Capabilities:** Works seamlessly with other YOLO26 solutions like [object tracking](../modes/track.md) and [speed estimation](speed-estimation.md).
### Can I perform distance calculation in real-time video streams with Ultralytics YOLO26?
Yes, you can perform distance calculation in real-time video streams with Ultralytics YOLO26. The process involves capturing video frames using [OpenCV](https://www.ultralytics.com/glossary/opencv), running YOLO26 [object detection](https://www.ultralytics.com/glossary/object-detection), and using the `DistanceCalculation` class to calculate distances between objects in successive frames. For a detailed implementation, see the [video stream example](#distance-calculation-using-ultralytics-yolo26).
### How do I delete points drawn during distance calculation using Ultralytics YOLO26?
To delete points drawn during distance calculation with Ultralytics YOLO26, you can use a right mouse click. This action will clear all the points you have drawn. For more details, refer to the note section under the [distance calculation example](#distance-calculation-using-ultralytics-yolo26).
### What are the key arguments for initializing the DistanceCalculation class in Ultralytics YOLO26?
The key arguments for initializing the `DistanceCalculation` class in Ultralytics YOLO26 include:
- `model`: Path to the YOLO26 model file.
- `tracker`: Tracking algorithm to use (default is 'botsort.yaml').
- `conf`: Confidence threshold for detections.
- `show`: Flag to display the output.
For an exhaustive list and default values, see the [arguments of DistanceCalculation](#distancecalculation-arguments).

359
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/docker-quickstart.md Normal file
View File

@@ -0,0 +1,359 @@
---
comments: true
description: Learn to effortlessly set up Ultralytics in Docker, from installation to running with CPU/GPU support. Follow our comprehensive guide for seamless container experience.
keywords: Ultralytics, Docker, Quickstart Guide, CPU support, GPU support, NVIDIA Docker, NVIDIA Container Toolkit, container setup, Docker environment, Docker Hub, Ultralytics projects
---
# Docker Quickstart Guide for Ultralytics
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-docker-package-visual.avif" alt="Ultralytics Docker Package Visual">
</p>
This guide serves as a comprehensive introduction to setting up a Docker environment for your Ultralytics projects. [Docker](https://www.docker.com/) is a platform for developing, shipping, and running applications in containers. It is particularly beneficial for ensuring that the software will always run the same, regardless of where it's deployed. For more details, visit the Ultralytics Docker repository on [Docker Hub](https://hub.docker.com/r/ultralytics/ultralytics).
[![Docker Image Version](https://img.shields.io/docker/v/ultralytics/ultralytics?sort=semver&logo=docker)](https://hub.docker.com/r/ultralytics/ultralytics)
[![Docker Pulls](https://img.shields.io/docker/pulls/ultralytics/ultralytics)](https://hub.docker.com/r/ultralytics/ultralytics)
## What You Will Learn
- Setting up Docker with NVIDIA support
- Installing Ultralytics Docker images
- Running Ultralytics in a Docker container with CPU or GPU support
- Using a display server with Docker to show Ultralytics detection results
- Mounting local directories into the container
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/IYWQZvtOy_Q"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Get started with Docker | Usage of Ultralytics Python Package inside Docker live demo 🎉
</p>
---
## Prerequisites
- Make sure Docker is installed on your system. If not, you can download and install it from [Docker's website](https://www.docker.com/products/docker-desktop/).
- Ensure that your system has an NVIDIA GPU and NVIDIA drivers are installed.
- If you are using NVIDIA Jetson devices, ensure that you have the appropriate JetPack version installed. Refer to the [NVIDIA Jetson guide](https://docs.ultralytics.com/guides/nvidia-jetson/) for more details.
---
## Setting up Docker with NVIDIA Support
First, verify that the NVIDIA drivers are properly installed by running:
```bash
nvidia-smi
```
### Installing NVIDIA Container Toolkit
Now, let's install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html) to enable GPU support in Docker containers:
=== "Ubuntu/Debian"
```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```
Update the package lists and install the nvidia-container-toolkit package:
```bash
sudo apt-get update
```
Install the latest version of `nvidia-container-toolkit`:
```bash
sudo apt-get install -y nvidia-container-toolkit \
nvidia-container-toolkit-base libnvidia-container-tools \
libnvidia-container1
```
??? info "Optional: Install specific version of nvidia-container-toolkit"
Optionally, you can install a specific version of the nvidia-container-toolkit by setting the `NVIDIA_CONTAINER_TOOLKIT_VERSION` environment variable:
```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
sudo apt-get install -y \
nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```
```bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
=== "RHEL/CentOS/Fedora/Amazon Linux"
```bash
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
| sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
```
Update the package lists and install the nvidia-container-toolkit package:
```bash
sudo dnf clean expire-cache
sudo dnf check-update
```
```bash
sudo dnf install \
nvidia-container-toolkit \
nvidia-container-toolkit-base \
libnvidia-container-tools \
libnvidia-container1
```
??? info "Optional: Install specific version of nvidia-container-toolkit"
Optionally, you can install a specific version of the nvidia-container-toolkit by setting the `NVIDIA_CONTAINER_TOOLKIT_VERSION` environment variable:
```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
sudo dnf install -y \
nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```
```bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
### Verify NVIDIA Runtime with Docker
Run `docker info | grep -i runtime` to ensure that `nvidia` appears in the list of runtimes:
```bash
docker info | grep -i runtime
```
---
## Installing Ultralytics Docker Images
Ultralytics offers several Docker images optimized for various platforms and use-cases:
- **Dockerfile:** GPU image, ideal for training.
- **Dockerfile-arm64:** For ARM64 architecture, suitable for devices like [Raspberry Pi](raspberry-pi.md).
- **Dockerfile-cpu:** CPU-only version for inference and non-GPU environments.
- **Dockerfile-jetson-jetpack4:** Optimized for [NVIDIA Jetson](https://docs.ultralytics.com/guides/nvidia-jetson/) devices running [NVIDIA JetPack 4](https://developer.nvidia.com/embedded/jetpack-sdk-461).
- **Dockerfile-jetson-jetpack5:** Optimized for [NVIDIA Jetson](https://docs.ultralytics.com/guides/nvidia-jetson/) devices running [NVIDIA JetPack 5](https://developer.nvidia.com/embedded/jetpack-sdk-512).
- **Dockerfile-jetson-jetpack6:** Optimized for [NVIDIA Jetson](https://docs.ultralytics.com/guides/nvidia-jetson/) devices running [NVIDIA JetPack 6](https://developer.nvidia.com/embedded/jetpack-sdk-61).
- **Dockerfile-jupyter:** For interactive development using JupyterLab in the browser.
- **Dockerfile-python:** Minimal Python environment for lightweight applications.
- **Dockerfile-conda:** Includes [Miniconda3](https://www.anaconda.com/docs/main) and Ultralytics package installed via Conda.
To pull the latest image:
```bash
# Set image name as a variable
t=ultralytics/ultralytics:latest
# Pull the latest Ultralytics image from Docker Hub
sudo docker pull $t
```
---
## Running Ultralytics in Docker Container
Here's how to execute the Ultralytics Docker container:
### Using only the CPU
```bash
# Run without GPU
sudo docker run -it --ipc=host $t
```
### Using GPUs
```bash
# Run with all GPUs
sudo docker run -it --ipc=host --runtime=nvidia --gpus all $t
# Run specifying which GPUs to use
sudo docker run -it --ipc=host --runtime=nvidia --gpus '"device=2,3"' $t
```
The `-it` flag assigns a pseudo-TTY and keeps stdin open, allowing you to interact with the container. The `--ipc=host` flag enables sharing of host's IPC namespace, essential for sharing memory between processes. The `--gpus` flag allows the container to access the host's GPUs.
### Note on File Accessibility
To work with files on your local machine within the container, you can use Docker volumes:
```bash
# Mount a local directory into the container
sudo docker run -it --ipc=host --runtime=nvidia --gpus all -v /path/on/host:/path/in/container $t
```
Replace `/path/on/host` with the directory path on your local machine and `/path/in/container` with the desired path inside the Docker container.
### Persisting Training Outputs
Training outputs save to `/ultralytics/runs/<task>/<name>/` inside the container by default. Without mounting a host directory, outputs are lost when the container is removed.
To persist training outputs:
```bash
# Recommended: mount workspace and specify project path
sudo docker run --rm -it -v "$(pwd)":/w -w /w ultralytics/ultralytics:latest \
yolo train model=yolo26n.pt data=coco8.yaml project=/w/runs
```
This saves all training outputs to `./runs` on your host machine.
## Run graphical user interface (GUI) applications in a Docker Container
!!! danger "Highly Experimental - User Assumes All Risk"
The following instructions are experimental. Sharing a X11 socket with a Docker container poses potential security risks. Therefore, it's recommended to test this solution only in a controlled environment. For more information, refer to these resources on how to use `xhost`<sup>[(1)](http://users.stat.umn.edu/~geyer/secure.html)[(2)](https://linux.die.net/man/1/xhost)</sup>.
Docker is primarily used to containerize background applications and CLI programs, but it can also run graphical programs. In the Linux world, two main graphic servers handle graphical display: [X11](https://www.x.org/wiki/) (also known as the X Window System) and [Wayland](<https://en.wikipedia.org/wiki/Wayland_(protocol)>). Before starting, it's essential to determine which graphics server you are currently using. Run this command to find out:
```bash
env | grep -E -i 'x11|xorg|wayland'
```
Setup and configuration of an X11 or Wayland display server is outside the scope of this guide. If the above command returns nothing, then you'll need to start by getting either working for your system before continuing.
### Running a Docker Container with a GUI
!!! example
??? info "Use GPUs"
If you're using [GPUs](#using-gpus), you can add the `--gpus all` flag to the command.
??? info "Docker runtime flag"
If your Docker installation does not use the `nvidia` runtime by default, you can add the `--runtime=nvidia` flag to the command.
=== "X11"
If you're using X11, you can run the following command to allow the Docker container to access the X11 socket:
```bash
xhost +local:docker && docker run -e DISPLAY=$DISPLAY \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v ~/.Xauthority:/root/.Xauthority \
-it --ipc=host $t
```
This command sets the `DISPLAY` environment variable to the host's display, mounts the X11 socket, and maps the `.Xauthority` file to the container. The `xhost +local:docker` command allows the Docker container to access the X11 server.
=== "Wayland"
For Wayland, use the following command:
```bash
xhost +local:docker && docker run -e DISPLAY=$DISPLAY \
-v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/tmp/$WAYLAND_DISPLAY \
--net=host -it --ipc=host $t
```
This command sets the `DISPLAY` environment variable to the host's display, mounts the Wayland socket, and allows the Docker container to access the Wayland server.
### Using Docker with a GUI
Now you can display graphical applications inside your Docker container. For example, you can run the following [CLI command](../usage/cli.md) to visualize the [predictions](../modes/predict.md) from a [YOLO26 model](../models/yolo26.md):
```bash
yolo predict model=yolo26n.pt show=True
```
??? info "Testing"
A simple way to validate that the Docker group has access to the X11 server is to run a container with a GUI program like [`xclock`](https://www.x.org/archive/X11R6.8.1/doc/xclock.1.html) or [`xeyes`](https://www.x.org/releases/X11R7.5/doc/man/man1/xeyes.1.html). Alternatively, you can also install these programs in the Ultralytics Docker container to test the access to the X11 server of your GNU-Linux display server. If you run into any problems, consider setting the environment variable `-e QT_DEBUG_PLUGINS=1`. Setting this environment variable enables the output of debugging information, aiding in the troubleshooting process.
### When finished with Docker GUI
!!! warning "Revoke access"
In both cases, don't forget to revoke access from the Docker group when you're done.
```bash
xhost -local:docker
```
??? question "Want to view image results directly in the Terminal?"
Refer to the following guide on [viewing the image results using a terminal](./view-results-in-terminal.md)
---
You are now set up to use Ultralytics with Docker and ready to take advantage of its capabilities. For alternative installation methods, see the [Ultralytics quickstart documentation](../quickstart.md).
## FAQ
### How do I set up Ultralytics with Docker?
To set up Ultralytics with Docker, first ensure that Docker is installed on your system. If you have an NVIDIA GPU, install the [NVIDIA Container Toolkit](#installing-nvidia-container-toolkit) to enable GPU support. Then, pull the latest Ultralytics Docker image from Docker Hub using the following command:
```bash
sudo docker pull ultralytics/ultralytics:latest
```
For detailed steps, refer to our Docker Quickstart Guide.
### What are the benefits of using Ultralytics Docker images for machine learning projects?
Using Ultralytics Docker images ensures a consistent environment across different machines, replicating the same software and dependencies. This is particularly useful for [collaborating across teams](https://www.ultralytics.com/blog/how-ultralytics-integration-can-enhance-your-workflow), running models on various hardware, and maintaining reproducibility. For GPU-based training, Ultralytics provides optimized Docker images such as `Dockerfile` for general GPU usage and `Dockerfile-jetson` for NVIDIA Jetson devices. Explore [Ultralytics Docker Hub](https://hub.docker.com/r/ultralytics/ultralytics) for more details.
### How can I run Ultralytics YOLO in a Docker container with GPU support?
First, ensure that the [NVIDIA Container Toolkit](#installing-nvidia-container-toolkit) is installed and configured. Then, use the following command to run Ultralytics YOLO with GPU support:
```bash
sudo docker run -it --ipc=host --runtime=nvidia --gpus all ultralytics/ultralytics:latest # all GPUs
```
This command sets up a Docker container with GPU access. For additional details, see the Docker Quickstart Guide.
### How do I visualize YOLO prediction results in a Docker container with a display server?
To visualize YOLO prediction results with a GUI in a Docker container, you need to allow Docker to access your display server. For systems running X11, the command is:
```bash
xhost +local:docker && docker run -e DISPLAY=$DISPLAY \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v ~/.Xauthority:/root/.Xauthority \
-it --ipc=host ultralytics/ultralytics:latest
```
For systems running Wayland, use:
```bash
xhost +local:docker && docker run -e DISPLAY=$DISPLAY \
-v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/tmp/$WAYLAND_DISPLAY \
--net=host -it --ipc=host ultralytics/ultralytics:latest
```
More information can be found in the [Run graphical user interface (GUI) applications in a Docker Container](#run-graphical-user-interface-gui-applications-in-a-docker-container) section.
### Can I mount local directories into the Ultralytics Docker container?
Yes, you can mount local directories into the Ultralytics Docker container using the `-v` flag:
```bash
sudo docker run -it --ipc=host --runtime=nvidia --gpus all -v /path/on/host:/path/in/container ultralytics/ultralytics:latest
```
Replace `/path/on/host` with the directory on your local machine and `/path/in/container` with the desired path inside the container. This setup allows you to work with your local files within the container. For more information, refer to the [Note on File Accessibility](#note-on-file-accessibility) section.

219
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/heatmaps.md Normal file
View File

@@ -0,0 +1,219 @@
---
comments: true
description: Transform complex data into insightful heatmaps using Ultralytics YOLO26. Discover patterns, trends, and anomalies with vibrant visualizations.
keywords: Ultralytics, YOLO26, heatmaps, data visualization, data analysis, complex data, patterns, trends, anomalies
---
# Advanced [Data Visualization](https://www.ultralytics.com/glossary/data-visualization): Heatmaps using Ultralytics YOLO26 🚀
## Introduction to Heatmaps
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-generate-heatmaps-using-ultralytics-yolo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Heatmaps In Colab"></a>
A heatmap generated with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) transforms complex data into a vibrant, color-coded matrix. This visual tool employs a spectrum of colors to represent varying data values, where warmer hues indicate higher intensities and cooler tones signify lower values. Heatmaps excel in visualizing intricate data patterns, correlations, and anomalies, offering an accessible and engaging approach to data interpretation across diverse domains.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/4ezde5-nZZw"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Heatmaps using Ultralytics YOLO26
</p>
## Why Choose Heatmaps for Data Analysis?
- **Intuitive Data Distribution Visualization:** Heatmaps simplify the comprehension of data concentration and distribution, converting complex datasets into easy-to-understand visual formats.
- **Efficient Pattern Detection:** By visualizing data in heatmap format, it becomes easier to spot trends, clusters, and outliers, facilitating quicker analysis and insights.
- **Enhanced Spatial Analysis and Decision-Making:** Heatmaps are instrumental in illustrating spatial relationships, aiding in decision-making processes in sectors such as business intelligence, environmental studies, and urban planning.
## Real World Applications
| Transportation | Retail |
| :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: |
| ![Ultralytics YOLO26 Transportation Heatmap](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-yolov8-transportation-heatmap.avif) | ![Ultralytics YOLO26 Retail Heatmap](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-yolov8-retail-heatmap.avif) |
| Ultralytics YOLO26 Transportation Heatmap | Ultralytics YOLO26 Retail Heatmap |
!!! example "Heatmaps using Ultralytics YOLO"
=== "CLI"
```bash
# Run a heatmap example
yolo solutions heatmap show=True
# Pass a source video
yolo solutions heatmap source="path/to/video.mp4"
# Pass a custom colormap
yolo solutions heatmap colormap=cv2.COLORMAP_INFERNO
# Heatmaps + object counting
yolo solutions heatmap region="[(20, 400), (1080, 400), (1080, 360), (20, 360)]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("heatmap_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# For object counting with heatmap, you can pass region points.
# region_points = [(20, 400), (1080, 400)] # line points
# region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360)] # rectangle region
# region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360), (20, 400)] # polygon points
# Initialize heatmap object
heatmap = solutions.Heatmap(
show=True, # display the output
model="yolo26n.pt", # path to the YOLO26 model file
colormap=cv2.COLORMAP_PARULA, # colormap of heatmap
# region=region_points, # object counting with heatmaps, you can pass region_points
# classes=[0, 2], # generate heatmap for specific classes, e.g., person and car.
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = heatmap(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `Heatmap()` Arguments
Here's a table with the `Heatmap` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "colormap", "show_in", "show_out", "region"]) }}
You can also apply different `track` arguments in the `Heatmap` solution.
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the supported visualization arguments are listed below:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
#### Heatmap COLORMAPs
| Colormap Name | Description |
| ------------------------------- | -------------------------------------- |
| `cv::COLORMAP_AUTUMN` | Autumn color map |
| `cv::COLORMAP_BONE` | Bone color map |
| `cv::COLORMAP_JET` | Jet color map |
| `cv::COLORMAP_WINTER` | Winter color map |
| `cv::COLORMAP_RAINBOW` | Rainbow color map |
| `cv::COLORMAP_OCEAN` | Ocean color map |
| `cv::COLORMAP_SUMMER` | Summer color map |
| `cv::COLORMAP_SPRING` | Spring color map |
| `cv::COLORMAP_COOL` | Cool color map |
| `cv::COLORMAP_HSV` | HSV (Hue, Saturation, Value) color map |
| `cv::COLORMAP_PINK` | Pink color map |
| `cv::COLORMAP_HOT` | Hot color map |
| `cv::COLORMAP_PARULA` | Parula color map |
| `cv::COLORMAP_MAGMA` | Magma color map |
| `cv::COLORMAP_INFERNO` | Inferno color map |
| `cv::COLORMAP_PLASMA` | Plasma color map |
| `cv::COLORMAP_VIRIDIS` | Viridis color map |
| `cv::COLORMAP_CIVIDIS` | Cividis color map |
| `cv::COLORMAP_TWILIGHT` | Twilight color map |
| `cv::COLORMAP_TWILIGHT_SHIFTED` | Shifted Twilight color map |
| `cv::COLORMAP_TURBO` | Turbo color map |
| `cv::COLORMAP_DEEPGREEN` | Deep Green color map |
These colormaps are commonly used for visualizing data with different color representations.
## How Heatmaps Work in Ultralytics YOLO26
The [Heatmap solution](../reference/solutions/heatmap.md) in Ultralytics YOLO26 extends the [ObjectCounter](../reference/solutions/object_counter.md) class to generate and visualize movement patterns in video streams. When initialized, the solution creates a blank heatmap layer that gets updated as objects move through the frame.
For each detected object, the solution:
1. Tracks the object across frames using YOLO26's tracking capabilities
2. Updates the heatmap intensity at the object's location
3. Applies a selected colormap to visualize the intensity values
4. Overlays the colored heatmap on the original frame
The result is a dynamic visualization that builds up over time, revealing traffic patterns, crowd movements, or other spatial behaviors in your video data.
## FAQ
### How does Ultralytics YOLO26 generate heatmaps and what are their benefits?
Ultralytics YOLO26 generates heatmaps by transforming complex data into a color-coded matrix where different hues represent data intensities. Heatmaps make it easier to visualize patterns, correlations, and anomalies in the data. Warmer hues indicate higher values, while cooler tones represent lower values. The primary benefits include intuitive visualization of data distribution, efficient pattern detection, and enhanced spatial analysis for decision-making. For more details and configuration options, refer to the [Heatmap Configuration](#heatmap-arguments) section.
### Can I use Ultralytics YOLO26 to perform object tracking and generate a heatmap simultaneously?
Yes, Ultralytics YOLO26 supports object tracking and heatmap generation concurrently. This can be achieved through its `Heatmap` solution integrated with object tracking models. To do so, you need to initialize the heatmap object and use YOLO26's tracking capabilities. Here's a simple example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
heatmap = solutions.Heatmap(colormap=cv2.COLORMAP_PARULA, show=True, model="yolo26n.pt")
while cap.isOpened():
success, im0 = cap.read()
if not success:
break
results = heatmap(im0)
cap.release()
cv2.destroyAllWindows()
```
For further guidance, check the [Tracking Mode](../modes/track.md) page.
### What makes Ultralytics YOLO26 heatmaps different from other data visualization tools like those from [OpenCV](https://www.ultralytics.com/glossary/opencv) or Matplotlib?
Ultralytics YOLO26 heatmaps are specifically designed for integration with its [object detection](https://www.ultralytics.com/glossary/object-detection) and tracking models, providing an end-to-end solution for real-time data analysis. Unlike generic visualization tools like OpenCV or Matplotlib, YOLO26 heatmaps are optimized for performance and automated processing, supporting features like persistent tracking, decay factor adjustment, and real-time video overlay. For more information on YOLO26's unique features, visit the [Ultralytics YOLO26 Introduction](https://www.ultralytics.com/blog/introducing-ultralytics-yolov8).
### How can I visualize only specific object classes in heatmaps using Ultralytics YOLO26?
You can visualize specific object classes by specifying the desired classes in the `track()` method of the YOLO model. For instance, if you only want to visualize cars and persons (assuming their class indices are 0 and 2), you can set the `classes` parameter accordingly.
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
heatmap = solutions.Heatmap(show=True, model="yolo26n.pt", classes=[0, 2])
while cap.isOpened():
success, im0 = cap.read()
if not success:
break
results = heatmap(im0)
cap.release()
cv2.destroyAllWindows()
```
### Why should businesses choose Ultralytics YOLO26 for heatmap generation in data analysis?
Ultralytics YOLO26 offers seamless integration of advanced object detection and real-time heatmap generation, making it an ideal choice for businesses looking to visualize data more effectively. The key advantages include intuitive data distribution visualization, efficient pattern detection, and enhanced spatial analysis for better decision-making. Additionally, YOLO26's cutting-edge features such as persistent tracking, customizable colormaps, and support for various export formats make it superior to other tools like [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) and OpenCV for comprehensive data analysis. Learn more about business applications at [Ultralytics Plans](https://www.ultralytics.com/plans).

347
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/hyperparameter-tuning.md Normal file
View File

@@ -0,0 +1,347 @@
---
comments: true
description: Master hyperparameter tuning for Ultralytics YOLO to optimize model performance with our comprehensive guide. Elevate your machine learning models today!
keywords: Ultralytics YOLO, hyperparameter tuning, machine learning, model optimization, genetic algorithms, learning rate, batch size, epochs
---
# Ultralytics YOLO [Hyperparameter Tuning](https://www.ultralytics.com/glossary/hyperparameter-tuning) Guide
## Introduction
Hyperparameter tuning is not just a one-time setup but an iterative process aimed at optimizing the [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) model's performance metrics, such as accuracy, precision, and recall. In the context of Ultralytics YOLO, these hyperparameters could range from learning rate to architectural details, such as the number of layers or types of activation functions used.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/j0MOGKBqx7E"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Tune Hyperparameters for Better Model Performance 🚀
</p>
### What are Hyperparameters?
Hyperparameters are high-level, structural settings for the algorithm. They are set prior to the training phase and remain constant during it. Here are some commonly tuned hyperparameters in Ultralytics YOLO:
- **Learning Rate** `lr0`: Determines the step size at each iteration while moving towards a minimum in the [loss function](https://www.ultralytics.com/glossary/loss-function).
- **[Batch Size](https://www.ultralytics.com/glossary/batch-size)** `batch`: Number of images processed simultaneously in a forward pass.
- **Number of [Epochs](https://www.ultralytics.com/glossary/epoch)** `epochs`: An epoch is one complete forward and backward pass of all the training examples.
- **Architecture Specifics**: Such as channel counts, number of layers, types of activation functions, etc.
<p align="center">
<img width="640" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/hyperparameter-tuning-visual.avif" alt="Hyperparameter optimization search space visualization">
</p>
For a full list of augmentation hyperparameters used in YOLO26 please refer to the [configurations page](../usage/cfg.md#augmentation-settings).
### Genetic Evolution and Mutation
Ultralytics YOLO uses [genetic algorithms](https://en.wikipedia.org/wiki/Genetic_algorithm) to optimize hyperparameters. Genetic algorithms are inspired by the mechanism of natural selection and genetics.
- **Mutation**: In the context of Ultralytics YOLO, mutation helps in locally searching the hyperparameter space by applying small, random changes to existing hyperparameters, producing new candidates for evaluation.
- **Crossover**: Although crossover is a popular genetic algorithm technique, it is not currently used in Ultralytics YOLO for hyperparameter tuning. The focus is mainly on mutation for generating new hyperparameter sets.
## Preparing for Hyperparameter Tuning
Before you begin the tuning process, it's important to:
1. **Identify the Metrics**: Determine the metrics you will use to evaluate the model's performance. This could be AP50, F1-score, or others.
2. **Set the Tuning Budget**: Define how much computational resources you're willing to allocate. Hyperparameter tuning can be computationally intensive.
## Steps Involved
### Initialize Hyperparameters
Start with a reasonable set of initial hyperparameters. This could either be the default hyperparameters set by Ultralytics YOLO or something based on your domain knowledge or previous experiments.
### Mutate Hyperparameters
Use the `_mutate` method to produce a new set of hyperparameters based on the existing set. The [Tuner class](https://docs.ultralytics.com/reference/engine/tuner/) handles this process automatically.
### Train Model
Training is performed using the mutated set of hyperparameters. The training performance is then assessed using your chosen metrics.
### Evaluate Model
Use metrics like AP50, F1-score, or custom metrics to evaluate the model's performance. The [evaluation process](https://docs.ultralytics.com/modes/val/) helps determine if the current hyperparameters are better than previous ones.
### Log Results
It's crucial to log both the performance metrics and the corresponding hyperparameters for future reference. Ultralytics YOLO automatically saves these results in CSV format.
### Repeat
The process is repeated until either the set number of iterations is reached or the performance metric is satisfactory. Each iteration builds upon the knowledge gained from previous runs.
## Default Search Space Description
The following table lists the default search space parameters for hyperparameter tuning in YOLO26. Each parameter has a specific value range defined by a tuple `(min, max)`.
| Parameter | Type | Value Range | Description |
| ----------------- | ------- | -------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `lr0` | `float` | `(1e-5, 1e-1)` | Initial learning rate at the start of training. Lower values provide more stable training but slower convergence |
| `lrf` | `float` | `(0.01, 1.0)` | Final learning rate factor as a fraction of lr0. Controls how much the learning rate decreases during training |
| `momentum` | `float` | `(0.6, 0.98)` | SGD momentum factor. Higher values help maintain consistent gradient direction and can speed up convergence |
| `weight_decay` | `float` | `(0.0, 0.001)` | L2 regularization factor to prevent overfitting. Larger values enforce stronger regularization |
| `warmup_epochs` | `float` | `(0.0, 5.0)` | Number of epochs for linear learning rate warmup. Helps prevent early training instability |
| `warmup_momentum` | `float` | `(0.0, 0.95)` | Initial momentum during warmup phase. Gradually increases to the final momentum value |
| `box` | `float` | `(0.02, 0.2)` | Bounding box loss weight in the total loss function. Balances box regression vs classification |
| `cls` | `float` | `(0.2, 4.0)` | Classification loss weight in the total loss function. Higher values emphasize correct class prediction |
| `dfl` | `float` | `(0.4, 6.0)` | DFL (Distribution Focal Loss) weight in the total loss function. Higher values emphasize precise bounding box localization |
| `hsv_h` | `float` | `(0.0, 0.1)` | Random hue augmentation range in HSV color space. Helps model generalize across color variations |
| `hsv_s` | `float` | `(0.0, 0.9)` | Random saturation augmentation range in HSV space. Simulates different lighting conditions |
| `hsv_v` | `float` | `(0.0, 0.9)` | Random value (brightness) augmentation range. Helps model handle different exposure levels |
| `degrees` | `float` | `(0.0, 45.0)` | Maximum rotation augmentation in degrees. Helps model become invariant to object orientation |
| `translate` | `float` | `(0.0, 0.9)` | Maximum translation augmentation as fraction of image size. Improves robustness to object position |
| `scale` | `float` | `(0.0, 0.9)` | Random scaling augmentation range. Helps model detect objects at different sizes |
| `shear` | `float` | `(0.0, 10.0)` | Maximum shear augmentation in degrees. Adds perspective-like distortions to training images |
| `perspective` | `float` | `(0.0, 0.001)` | Random perspective augmentation range. Simulates different viewing angles |
| `flipud` | `float` | `(0.0, 1.0)` | Probability of vertical image flip during training. Useful for overhead/aerial imagery |
| `fliplr` | `float` | `(0.0, 1.0)` | Probability of horizontal image flip. Helps model become invariant to object direction |
| `bgr` | `float` | `(0.0, 1.0)` | Probability of using BGR augmentation, which swaps color channels. Can help with color invariance |
| `mosaic` | `float` | `(0.0, 1.0)` | Probability of using mosaic augmentation, which combines 4 images. Especially useful for small object detection |
| `mixup` | `float` | `(0.0, 1.0)` | Probability of using mixup augmentation, which blends two images. Can improve model robustness |
| `copy_paste` | `float` | `(0.0, 1.0)` | Probability of using copy-paste augmentation. Helps improve instance segmentation performance |
| `close_mosaic` | `int` | `(0, 10)` | Disables mosaic in the last N epochs to stabilize training before completion. |
## Custom Search Space Example
Here's how to define a search space and use the `model.tune()` method to utilize the `Tuner` class for hyperparameter tuning of YOLO26n on COCO8 for 30 epochs with an AdamW optimizer and skipping plotting, checkpointing and validation other than on final epoch for faster Tuning.
!!! warning
This example is for **demonstration** only. Hyperparameters derived from short or small-scale tuning runs are rarely optimal for real-world training. In practice, tuning should be performed under settings similar to full training — including comparable datasets, epochs, and augmentations — to ensure reliable and transferable results. Quick tuning may bias parameters toward faster convergence or short-term validation gains that do not generalize.
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Initialize the YOLO model
model = YOLO("yolo26n.pt")
# Define search space
search_space = {
"lr0": (1e-5, 1e-1),
"degrees": (0.0, 45.0),
}
# Tune hyperparameters on COCO8 for 30 epochs
model.tune(
data="coco8.yaml",
epochs=30,
iterations=300,
optimizer="AdamW",
space=search_space,
plots=False,
save=False,
val=False,
)
```
## Resuming an Interrupted Hyperparameter Tuning Session
You can resume an interrupted hyperparameter tuning session by passing `resume=True`. You can optionally pass the directory `name` used under `runs/{task}` to resume. Otherwise, it would resume the last interrupted session. You also need to provide all the previous training arguments including `data`, `epochs`, `iterations` and `space`.
!!! example "Using `resume=True` with `model.tune()`"
```python
from ultralytics import YOLO
# Define a YOLO model
model = YOLO("yolo26n.pt")
# Define search space
search_space = {
"lr0": (1e-5, 1e-1),
"degrees": (0.0, 45.0),
}
# Resume previous run
results = model.tune(data="coco8.yaml", epochs=50, iterations=300, space=search_space, resume=True)
# Resume tuning run with name 'tune_exp'
results = model.tune(data="coco8.yaml", epochs=50, iterations=300, space=search_space, name="tune_exp", resume=True)
```
## Results
After you've successfully completed the hyperparameter tuning process, you will obtain several files and directories that encapsulate the results of the tuning. The following describes each:
### File Structure
Here's what the directory structure of the results will look like. Training directories like `train1/` contain individual tuning iterations, i.e., one model trained with one set of hyperparameters. The `tune/` directory contains tuning results from all the individual model trainings:
```plaintext
runs/
└── detect/
├── train1/
├── train2/
├── ...
└── tune/
├── best_hyperparameters.yaml
├── best_fitness.png
├── tune_results.csv
├── tune_scatter_plots.png
└── weights/
├── last.pt
└── best.pt
```
### File Descriptions
#### best_hyperparameters.yaml
This YAML file contains the best-performing hyperparameters found during the tuning process. You can use this file to initialize future trainings with these optimized settings.
- **Format**: YAML
- **Usage**: Hyperparameter results
- **Example**:
```yaml
# 558/900 iterations complete ✅ (45536.81s)
# Results saved to /usr/src/ultralytics/runs/detect/tune
# Best fitness=0.64297 observed at iteration 498
# Best fitness metrics are {'metrics/precision(B)': 0.87247, 'metrics/recall(B)': 0.71387, 'metrics/mAP50(B)': 0.79106, 'metrics/mAP50-95(B)': 0.62651, 'val/box_loss': 2.79884, 'val/cls_loss': 2.72386, 'val/dfl_loss': 0.68503, 'fitness': 0.64297}
# Best fitness model is /usr/src/ultralytics/runs/detect/train498
# Best fitness hyperparameters are printed below.
lr0: 0.00269
lrf: 0.00288
momentum: 0.73375
weight_decay: 0.00015
warmup_epochs: 1.22935
warmup_momentum: 0.1525
box: 18.27875
cls: 1.32899
dfl: 0.56016
hsv_h: 0.01148
hsv_s: 0.53554
hsv_v: 0.13636
degrees: 0.0
translate: 0.12431
scale: 0.07643
shear: 0.0
perspective: 0.0
flipud: 0.0
fliplr: 0.08631
mosaic: 0.42551
mixup: 0.0
copy_paste: 0.0
```
#### best_fitness.png
This is a plot displaying fitness (typically a performance metric like AP50) against the number of iterations. It helps you visualize how well the genetic algorithm performed over time.
- **Format**: PNG
- **Usage**: Performance visualization
<p align="center">
<img width="640" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/best-fitness.avif" alt="Hyperparameter Tuning Fitness vs Iteration">
</p>
#### tune_results.csv
A CSV file containing detailed results of each iteration during the tuning. Each row in the file represents one iteration, and it includes metrics like fitness score, [precision](https://www.ultralytics.com/glossary/precision), [recall](https://www.ultralytics.com/glossary/recall), as well as the hyperparameters used.
- **Format**: CSV
- **Usage**: Per-iteration results tracking.
- **Example**:
```csv
fitness,lr0,lrf,momentum,weight_decay,warmup_epochs,warmup_momentum,box,cls,dfl,hsv_h,hsv_s,hsv_v,degrees,translate,scale,shear,perspective,flipud,fliplr,mosaic,mixup,copy_paste
0.05021,0.01,0.01,0.937,0.0005,3.0,0.8,7.5,0.5,1.5,0.015,0.7,0.4,0.0,0.1,0.5,0.0,0.0,0.0,0.5,1.0,0.0,0.0
0.07217,0.01003,0.00967,0.93897,0.00049,2.79757,0.81075,7.5,0.50746,1.44826,0.01503,0.72948,0.40658,0.0,0.0987,0.4922,0.0,0.0,0.0,0.49729,1.0,0.0,0.0
0.06584,0.01003,0.00855,0.91009,0.00073,3.42176,0.95,8.64301,0.54594,1.72261,0.01503,0.59179,0.40658,0.0,0.0987,0.46955,0.0,0.0,0.0,0.49729,0.80187,0.0,0.0
```
#### tune_scatter_plots.png
This file contains scatter plots generated from `tune_results.csv`, helping you visualize relationships between different hyperparameters and performance metrics. Note that hyperparameters initialized to 0 will not be tuned, such as `degrees` and `shear` below.
- **Format**: PNG
- **Usage**: Exploratory data analysis
<p align="center">
<img width="1000" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/tune-scatter-plots.avif" alt="Hyperparameter tuning results scatter plot analysis">
</p>
#### weights/
This directory contains the saved [PyTorch](https://www.ultralytics.com/glossary/pytorch) models for the last and the best iterations during the hyperparameter tuning process.
- **`last.pt`**: The last.pt are the weights from the last epoch of training.
- **`best.pt`**: The best.pt weights for the iteration that achieved the best fitness score.
Using these results, you can make more informed decisions for your future model trainings and analyses. Feel free to consult these artifacts to understand how well your model performed and how you might improve it further.
## Conclusion
The hyperparameter tuning process in Ultralytics YOLO is simplified yet powerful, thanks to its genetic algorithm-based approach focused on mutation. Following the steps outlined in this guide will assist you in systematically tuning your model to achieve better performance.
### Further Reading
1. [Hyperparameter Optimization in Wikipedia](https://en.wikipedia.org/wiki/Hyperparameter_optimization)
2. [YOLOv5 Hyperparameter Evolution Guide](../yolov5/tutorials/hyperparameter_evolution.md)
3. [Efficient Hyperparameter Tuning with Ray Tune and YOLO26](../integrations/ray-tune.md)
For deeper insights, you can explore the [`Tuner` class](https://docs.ultralytics.com/reference/engine/tuner/) source code and accompanying documentation. Should you have any questions, feature requests, or need further assistance, feel free to reach out to us on [GitHub](https://github.com/ultralytics/ultralytics/issues/new/choose) or [Discord](https://discord.com/invite/ultralytics).
## FAQ
### How do I optimize the [learning rate](https://www.ultralytics.com/glossary/learning-rate) for Ultralytics YOLO during hyperparameter tuning?
To optimize the learning rate for Ultralytics YOLO, start by setting an initial learning rate using the `lr0` parameter. Common values range from `0.001` to `0.01`. During the hyperparameter tuning process, this value will be mutated to find the optimal setting. You can utilize the `model.tune()` method to automate this process. For example:
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Initialize the YOLO model
model = YOLO("yolo26n.pt")
# Tune hyperparameters on COCO8 for 30 epochs
model.tune(data="coco8.yaml", epochs=30, iterations=300, optimizer="AdamW", plots=False, save=False, val=False)
```
For more details, check the [Ultralytics YOLO configuration page](../usage/cfg.md#augmentation-settings).
### What are the benefits of using genetic algorithms for hyperparameter tuning in YOLO26?
Genetic algorithms in Ultralytics YOLO26 provide a robust method for exploring the hyperparameter space, leading to highly optimized model performance. Key benefits include:
- **Efficient Search**: Genetic algorithms like mutation can quickly explore a large set of hyperparameters.
- **Avoiding Local Minima**: By introducing randomness, they help in avoiding local minima, ensuring better global optimization.
- **Performance Metrics**: They adapt based on performance metrics such as AP50 and F1-score.
To see how genetic algorithms can optimize hyperparameters, check out the [hyperparameter evolution guide](../yolov5/tutorials/hyperparameter_evolution.md).
### How long does the hyperparameter tuning process take for Ultralytics YOLO?
The time required for hyperparameter tuning with Ultralytics YOLO largely depends on several factors such as the size of the dataset, the complexity of the model architecture, the number of iterations, and the computational resources available. For instance, tuning YOLO26n on a dataset like COCO8 for 30 epochs might take several hours to days, depending on the hardware.
To effectively manage tuning time, define a clear tuning budget beforehand ([internal section link](#preparing-for-hyperparameter-tuning)). This helps in balancing resource allocation and optimization goals.
### What metrics should I use to evaluate model performance during hyperparameter tuning in YOLO?
When evaluating model performance during hyperparameter tuning in YOLO, you can use several key metrics:
- **AP50**: The average precision at IoU threshold of 0.50.
- **F1-Score**: The harmonic mean of precision and recall.
- **Precision and Recall**: Individual metrics indicating the model's [accuracy](https://www.ultralytics.com/glossary/accuracy) in identifying true positives versus false positives and false negatives.
These metrics help you understand different aspects of your model's performance. Refer to the [Ultralytics YOLO performance metrics](../guides/yolo-performance-metrics.md) guide for a comprehensive overview.
### Can I use Ray Tune for advanced hyperparameter optimization with YOLO26?
Yes, Ultralytics YOLO26 integrates with [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) for advanced hyperparameter optimization. Ray Tune offers sophisticated search algorithms like Bayesian Optimization and Hyperband, along with parallel execution capabilities to speed up the tuning process.
To use Ray Tune with YOLO26, simply set the `use_ray=True` parameter in your `model.tune()` method call. For more details and examples, check out the [Ray Tune integration guide](../integrations/ray-tune.md).

106
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/index.md Normal file
View File

@@ -0,0 +1,106 @@
---
comments: true
description: Master YOLO with Ultralytics tutorials covering training, deployment and optimization. Find solutions, improve metrics, and deploy with ease.
keywords: Ultralytics, YOLO, tutorials, guides, object detection, deep learning, PyTorch, training, deployment, optimization, computer vision
---
# Comprehensive Tutorials for Ultralytics YOLO
Welcome to Ultralytics' YOLO Guides. Our comprehensive tutorials cover various aspects of the YOLO [object detection](https://www.ultralytics.com/glossary/object-detection) model, ranging from training and prediction to deployment. Built on [PyTorch](https://www.ultralytics.com/glossary/pytorch), YOLO stands out for its exceptional speed and [accuracy](https://www.ultralytics.com/glossary/accuracy) in real-time object detection tasks.
Whether you're a beginner or an expert in [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl), our tutorials offer valuable insights into the implementation and optimization of YOLO for your [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) projects.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/96NkhsV-W1U"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Ultralytics YOLO26 Guides Overview
</p>
## Guides
Here's a compilation of in-depth guides to help you master different aspects of Ultralytics YOLO.
- [A Guide on Model Testing](model-testing.md): A thorough guide on testing your computer vision models in realistic settings. Learn how to verify accuracy, reliability, and performance in line with project goals.
- [AzureML Quickstart](azureml-quickstart.md): Get up and running with Ultralytics YOLO models on Microsoft's Azure [Machine Learning](https://www.ultralytics.com/glossary/machine-learning-ml) platform. Learn how to train, deploy, and scale your object detection projects in the cloud.
- [Best Practices for Model Deployment](model-deployment-practices.md): Walk through tips and best practices for efficiently deploying models in computer vision projects, with a focus on optimization, troubleshooting, and security.
- [Conda Quickstart](conda-quickstart.md): Step-by-step guide to setting up a [Conda](https://anaconda.org/conda-forge/ultralytics) environment for Ultralytics. Learn how to install and start using the Ultralytics package efficiently with Conda.
- [Data Collection and Annotation](data-collection-and-annotation.md): Explore the tools, techniques, and best practices for collecting and annotating data to create high-quality inputs for your computer vision models.
- [DeepStream on NVIDIA Jetson](deepstream-nvidia-jetson.md): Quickstart guide for deploying YOLO models on NVIDIA Jetson devices using DeepStream and TensorRT.
- [Defining A Computer Vision Project's Goals](defining-project-goals.md): Walk through how to effectively define clear and measurable goals for your computer vision project. Learn the importance of a well-defined problem statement and how it creates a roadmap for your project.
- [Docker Quickstart](docker-quickstart.md): Complete guide to setting up and using Ultralytics YOLO models with [Docker](https://hub.docker.com/r/ultralytics/ultralytics). Learn how to install Docker, manage GPU support, and run YOLO models in isolated containers for consistent development and deployment.
- [Edge TPU on Raspberry Pi](coral-edge-tpu-on-raspberry-pi.md): [Google Edge TPU](https://developers.google.com/coral) accelerates YOLO inference on [Raspberry Pi](https://www.raspberrypi.com/).
- [Hyperparameter Tuning](hyperparameter-tuning.md): Discover how to optimize your YOLO models by fine-tuning hyperparameters using the Tuner class and genetic evolution algorithms.
- [Insights on Model Evaluation and Fine-Tuning](model-evaluation-insights.md): Gain insights into the strategies and best practices for evaluating and fine-tuning your computer vision models. Learn about the iterative process of refining models to achieve optimal results.
- [Isolating Segmentation Objects](isolating-segmentation-objects.md): Step-by-step recipe and explanation on how to extract and/or isolate objects from images using Ultralytics Segmentation.
- [K-Fold Cross Validation](kfold-cross-validation.md): Learn how to improve model generalization using K-Fold cross-validation technique.
- [Maintaining Your Computer Vision Model](model-monitoring-and-maintenance.md): Understand the key practices for monitoring, maintaining, and documenting computer vision models to guarantee accuracy, spot anomalies, and mitigate data drift.
- [Model Deployment Options](model-deployment-options.md): Overview of YOLO [model deployment](https://www.ultralytics.com/glossary/model-deployment) formats like ONNX, OpenVINO, and TensorRT, with pros and cons for each to inform your deployment strategy.
- [Model YAML Configuration Guide](model-yaml-config.md): A comprehensive deep dive into Ultralytics' model architecture definitions. Explore the YAML format, understand the module resolution system, and learn how to integrate custom modules seamlessly.
- [NVIDIA DGX Spark](nvidia-dgx-spark.md): Quickstart guide for deploying YOLO models on NVIDIA DGX Spark devices.
- [NVIDIA Jetson](nvidia-jetson.md): Quickstart guide for deploying YOLO models on NVIDIA Jetson devices.
- [OpenVINO Latency vs Throughput Modes](optimizing-openvino-latency-vs-throughput-modes.md): Learn latency and throughput optimization techniques for peak YOLO inference performance.
- [Preprocessing Annotated Data](preprocessing_annotated_data.md): Learn about preprocessing and augmenting image data in computer vision projects using YOLO26, including normalization, dataset augmentation, splitting, and exploratory data analysis (EDA).
- [Raspberry Pi](raspberry-pi.md): Quickstart tutorial to run YOLO models on the latest Raspberry Pi hardware.
- [ROS Quickstart](ros-quickstart.md): Learn how to integrate YOLO with the Robot Operating System (ROS) for real-time object detection in robotics applications, including Point Cloud and Depth images.
- [SAHI Tiled Inference](sahi-tiled-inference.md): Comprehensive guide on leveraging SAHI's sliced inference capabilities with YOLO26 for object detection in high-resolution images.
- [Steps of a Computer Vision Project](steps-of-a-cv-project.md): Learn about the key steps involved in a computer vision project, including defining goals, selecting models, preparing data, and evaluating results.
- [Tips for Model Training](model-training-tips.md): Explore tips on optimizing [batch sizes](https://www.ultralytics.com/glossary/batch-size), using [mixed precision](https://www.ultralytics.com/glossary/mixed-precision), applying pretrained weights, and more to make training your computer vision model a breeze.
- [Triton Inference Server Integration](triton-inference-server.md): Dive into the integration of Ultralytics YOLO26 with NVIDIA's Triton Inference Server for scalable and efficient deep learning inference deployments.
- [Vertex AI Deployment with Docker](vertex-ai-deployment-with-docker.md): Streamlined guide to containerizing YOLO models with Docker and deploying them on Google Cloud Vertex AI—covering build, push, autoscaling, and monitoring.
- [View Inference Images in a Terminal](view-results-in-terminal.md): Use VSCode's integrated terminal to view inference results when using Remote Tunnel or SSH sessions.
- [YOLO Common Issues](yolo-common-issues.md) ⭐ RECOMMENDED: Practical solutions and troubleshooting tips to the most frequently encountered issues when working with Ultralytics YOLO models.
- [YOLO Data Augmentation](yolo-data-augmentation.md): Master the complete range of data augmentation techniques in YOLO, from basic transformations to advanced strategies for improving model robustness and performance.
- [YOLO Performance Metrics](yolo-performance-metrics.md) ⭐ ESSENTIAL: Understand the key metrics like mAP, IoU, and [F1 score](https://www.ultralytics.com/glossary/f1-score) used to evaluate the performance of your YOLO models. Includes practical examples and tips on how to improve detection accuracy and speed.
- [YOLO Thread-Safe Inference](yolo-thread-safe-inference.md): Guidelines for performing inference with YOLO models in a thread-safe manner. Learn the importance of thread safety and best practices to prevent race conditions and ensure consistent predictions.
## Contribute to Our Guides
We welcome contributions from the community! If you've mastered a particular aspect of Ultralytics YOLO that's not yet covered in our guides, we encourage you to share your expertise. Writing a guide is a great way to give back to the community and help us make our documentation more comprehensive and user-friendly.
To get started, please read our [Contributing Guide](../help/contributing.md) for guidelines on how to open a Pull Request (PR). We look forward to your contributions.
## FAQ
### How do I train a custom object detection model using Ultralytics YOLO?
Training a custom object detection model with Ultralytics YOLO is straightforward. Start by preparing your dataset in the correct format and installing the Ultralytics package. Use the following code to initiate training:
!!! example
=== "Python"
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt") # Load a pretrained YOLO model
model.train(data="path/to/dataset.yaml", epochs=50) # Train on custom dataset
```
=== "CLI"
```bash
yolo task=detect mode=train model=yolo26n.pt data=path/to/dataset.yaml epochs=50
```
For detailed dataset formatting and additional options, refer to our [Tips for Model Training](model-training-tips.md) guide.
### What performance metrics should I use to evaluate my YOLO model?
Evaluating your YOLO model performance is crucial to understanding its efficacy. Key metrics include [Mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP), [Intersection over Union](https://www.ultralytics.com/glossary/intersection-over-union-iou) (IoU), and F1 score. These metrics help assess the accuracy and [precision](https://www.ultralytics.com/glossary/precision) of object detection tasks. You can learn more about these metrics and how to improve your model in our [YOLO Performance Metrics](yolo-performance-metrics.md) guide.
### Why should I use Ultralytics Platform for my computer vision projects?
Ultralytics Platform is a no-code platform that simplifies managing, training, and deploying YOLO models. It supports seamless integration, real-time tracking, and cloud training, making it ideal for both beginners and professionals. Discover more about its features and how it can streamline your workflow with our [Ultralytics Platform](https://docs.ultralytics.com/platform/) quickstart guide.
### What are the common issues faced during YOLO model training, and how can I resolve them?
Common issues during YOLO model training include data formatting errors, model architecture mismatches, and insufficient [training data](https://www.ultralytics.com/glossary/training-data). To address these, ensure your dataset is correctly formatted, check for compatible model versions, and augment your training data. For a comprehensive list of solutions, refer to our [YOLO Common Issues](yolo-common-issues.md) guide.
### How can I deploy my YOLO model for real-time object detection on edge devices?
Deploying YOLO models on edge devices like NVIDIA Jetson and Raspberry Pi requires converting the model to a compatible format such as TensorRT or TFLite. Follow our step-by-step guides for [NVIDIA Jetson](nvidia-jetson.md) and [Raspberry Pi](raspberry-pi.md) deployments to get started with real-time object detection on edge hardware. These guides will walk you through installation, configuration, and performance optimization.

185
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/instance-segmentation-and-tracking.md Normal file
View File

@@ -0,0 +1,185 @@
---
comments: true
description: Master instance segmentation and tracking with Ultralytics YOLO26. Learn techniques for precise object identification and tracking.
keywords: instance segmentation, tracking, YOLO26, Ultralytics, object detection, machine learning, computer vision, python
---
# Instance Segmentation and Tracking using Ultralytics YOLO26 🚀
## What is Instance Segmentation?
[Instance segmentation](https://www.ultralytics.com/glossary/instance-segmentation) is a computer vision task that involves identifying and outlining individual objects in an image at the pixel level. Unlike [semantic segmentation](https://www.ultralytics.com/glossary/semantic-segmentation) which only classifies pixels by category, instance segmentation uniquely labels and precisely delineates each object instance, making it crucial for applications requiring detailed spatial understanding like medical imaging, autonomous driving, and industrial automation.
[Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) provides powerful instance segmentation capabilities that enable precise object boundary detection while maintaining the speed and efficiency YOLO models are known for.
There are two types of instance segmentation tracking available in the Ultralytics package:
- **Instance Segmentation with Class Objects:** Each class object is assigned a unique color for clear visual separation.
- **Instance Segmentation with Object Tracks:** Every track is represented by a distinct color, facilitating easy identification and tracking across video frames.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/75G_S1Ngji8"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Instance Segmentation with Object Tracking using Ultralytics YOLO26
</p>
## Samples
| Instance Segmentation | Instance Segmentation + Object Tracking |
| :-----------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Ultralytics Instance Segmentation](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-instance-segmentation.avif) | ![Ultralytics Instance Segmentation with Object Tracking](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-instance-segmentation-object-tracking.avif) |
| Ultralytics Instance Segmentation 😍 | Ultralytics Instance Segmentation with Object Tracking 🔥 |
!!! example "Instance segmentation using Ultralytics YOLO"
=== "CLI"
```bash
# Instance segmentation using Ultralytics YOLO26
yolo solutions isegment show=True
# Pass a source video
yolo solutions isegment source="path/to/video.mp4"
# Monitor the specific classes
yolo solutions isegment classes="[0, 5]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("isegment_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize instance segmentation object
isegment = solutions.InstanceSegmentation(
show=True, # display the output
model="yolo26n-seg.pt", # model="yolo26n-seg.pt" for object segmentation using YOLO26.
# classes=[0, 2], # segment specific classes, e.g., person and car with the pretrained model.
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or video processing has been successfully completed.")
break
results = isegment(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `InstanceSegmentation` Arguments
Here's a table with the `InstanceSegmentation` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "region"]) }}
You can also take advantage of `track` arguments within the `InstanceSegmentation` solution:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization arguments are available:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## Applications of Instance Segmentation
Instance segmentation with YOLO26 has numerous real-world applications across various industries:
### Waste Management and Recycling
YOLO26 can be used in [waste management facilities](https://www.ultralytics.com/blog/simplifying-e-waste-management-with-ai-innovations) to identify and sort different types of materials. The model can segment plastic waste, cardboard, metal, and other recyclables with high precision, enabling automated sorting systems to process waste more efficiently. This is particularly valuable considering that only about 10% of the 7 billion tonnes of plastic waste generated globally gets recycled.
### Autonomous Vehicles
In [self-driving cars](https://www.ultralytics.com/solutions/ai-in-automotive), instance segmentation helps identify and track pedestrians, vehicles, traffic signs, and other road elements at the pixel level. This precise understanding of the environment is crucial for navigation and safety decisions. YOLO26's real-time performance makes it ideal for these time-sensitive applications.
### Medical Imaging
Instance segmentation can identify and outline tumors, organs, or cellular structures in medical scans. YOLO26's ability to precisely delineate object boundaries makes it valuable for [medical diagnostics](https://www.ultralytics.com/blog/ai-and-radiology-a-new-era-of-precision-and-efficiency) and treatment planning.
### Construction Site Monitoring
At construction sites, instance segmentation can track heavy machinery, workers, and materials. This helps ensure safety by monitoring equipment positions and detecting when workers enter hazardous areas, while also optimizing workflow and resource allocation.
## Note
For any inquiries, feel free to post your questions in the [Ultralytics Issue Section](https://github.com/ultralytics/ultralytics/issues/new/choose) or the discussion section mentioned below.
## FAQ
### How do I perform instance segmentation using Ultralytics YOLO26?
To perform instance segmentation using Ultralytics YOLO26, initialize the YOLO model with a segmentation version of YOLO26 and process video frames through it. Here's a simplified code example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("instance-segmentation.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init InstanceSegmentation
isegment = solutions.InstanceSegmentation(
show=True, # display the output
model="yolo26n-seg.pt", # model="yolo26n-seg.pt" for object segmentation using YOLO26.
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = isegment(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
```
Learn more about instance segmentation in the [Ultralytics YOLO26 guide](https://docs.ultralytics.com/tasks/segment/).
### What is the difference between instance segmentation and object tracking in Ultralytics YOLO26?
Instance segmentation identifies and outlines individual objects within an image, giving each object a unique label and mask. Object tracking extends this by assigning consistent IDs to objects across video frames, facilitating continuous tracking of the same objects over time. When combined, as in YOLO26's implementation, you get powerful capabilities for analyzing object movement and behavior in videos while maintaining precise boundary information.
### Why should I use Ultralytics YOLO26 for instance segmentation and tracking over other models like Mask R-CNN or Faster R-CNN?
Ultralytics YOLO26 offers real-time performance, superior [accuracy](https://www.ultralytics.com/glossary/accuracy), and ease of use compared to other models like Mask R-CNN or Faster R-CNN. YOLO26 processes images in a single pass (one-stage detection), making it significantly faster while maintaining high precision. It also provides seamless integration with [Ultralytics Platform](https://platform.ultralytics.com), allowing users to manage models, datasets, and training pipelines efficiently. For applications requiring both speed and accuracy, YOLO26 provides an optimal balance.
### Are there any datasets provided by Ultralytics suitable for training YOLO26 models for instance segmentation and tracking?
Yes, Ultralytics offers several datasets suitable for training YOLO26 models for instance segmentation, including [COCO-Seg](https://docs.ultralytics.com/datasets/segment/coco/), [COCO8-Seg](https://docs.ultralytics.com/datasets/segment/coco8-seg/) (a smaller subset for quick testing), [Package-Seg](https://docs.ultralytics.com/datasets/segment/package-seg/), and [Crack-Seg](https://docs.ultralytics.com/datasets/segment/crack-seg/). These datasets come with pixel-level annotations needed for instance segmentation tasks. For more specialized applications, you can also create custom datasets following the Ultralytics format. Complete dataset information and usage instructions can be found in the [Ultralytics Datasets documentation](https://docs.ultralytics.com/datasets/).

396
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/isolating-segmentation-objects.md Normal file
View File

@@ -0,0 +1,396 @@
---
comments: true
description: Learn to extract isolated objects from inference results using Ultralytics Predict Mode. Step-by-step guide for segmentation object isolation.
keywords: Ultralytics, segmentation, object isolation, Predict Mode, YOLO26, machine learning, object detection, binary mask, image processing
---
# Isolating Segmentation Objects
After performing the [Segment Task](../tasks/segment.md), it's sometimes desirable to extract the isolated objects from the inference results. This guide provides a generic recipe on how to accomplish this using the Ultralytics [Predict Mode](../modes/predict.md).
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/5HBB5IBuJ6c"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Remove Background and Isolate Objects with Ultralytics YOLO Segmentation & OpenCV in Python 🚀
</p>
## Recipe Walkthrough
1. See the [Ultralytics Quickstart Installation section](../quickstart.md) for a quick walkthrough on installing the required libraries.
***
2. Load a model and run `predict()` method on a source.
```python
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n-seg.pt")
# Run inference
results = model.predict()
```
!!! question "No Prediction Arguments?"
Without specifying a source, the example images from the library will be used:
```
'ultralytics/assets/bus.jpg'
'ultralytics/assets/zidane.jpg'
```
This is helpful for rapid testing with the `predict()` method.
For additional information about Segmentation Models, visit the [Segment Task](../tasks/segment.md#models) page. To learn more about `predict()` method, see [Predict Mode](../modes/predict.md) section of the Documentation.
***
3. Now iterate over the results and the contours. For workflows that want to save an image to file, the source image `base-name` and the detection `class-label` are retrieved for later use (optional).
```{ .py .annotate }
from pathlib import Path
import numpy as np
# (2) Iterate detection results (helpful for multiple images)
for r in results:
img = np.copy(r.orig_img)
img_name = Path(r.path).stem # source image base-name
# Iterate each object contour (multiple detections)
for ci, c in enumerate(r):
# (1) Get detection class name
label = c.names[c.boxes.cls.tolist().pop()]
```
1. To learn more about working with detection results, see [Boxes Section for Predict Mode](../modes/predict.md#boxes).
2. To learn more about `predict()` results see [Working with Results for Predict Mode](../modes/predict.md#working-with-results)
??? info "For-Loop"
A single image will only iterate the first loop once. A single image with only a single detection will iterate each loop _only_ once.
***
4. Start with generating a binary mask from the source image and then draw a filled contour onto the mask. This will allow the object to be isolated from the other parts of the image. An example from `bus.jpg` for one of the detected `person` class objects is shown on the right.
![Binary Mask Image](https://github.com/ultralytics/ultralytics/assets/62214284/59bce684-fdda-4b17-8104-0b4b51149aca){ width="240", align="right" }
```{ .py .annotate }
import cv2
# Create binary mask
b_mask = np.zeros(img.shape[:2], np.uint8)
# (1) Extract contour result
contour = c.masks.xy.pop()
# (2) Changing the type
contour = contour.astype(np.int32)
# (3) Reshaping
contour = contour.reshape(-1, 1, 2)
# Draw contour onto mask
_ = cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)
```
1. For more info on `c.masks.xy` see [Masks Section from Predict Mode](../modes/predict.md#masks).
2. Here the values are cast into `np.int32` for compatibility with `drawContours()` function from [OpenCV](https://www.ultralytics.com/glossary/opencv).
3. The OpenCV `drawContours()` function expects contours to have a shape of `[N, 1, 2]` expand section below for more details.
<details>
<summary> Expand to understand what is happening when defining the <code>contour</code> variable.</summary>
<p>
- `c.masks.xy` :: Provides the coordinates of the mask contour points in the format `(x, y)`. For more details, refer to the [Masks Section from Predict Mode](../modes/predict.md#masks).
- `.pop()` :: As `masks.xy` is a list containing a single element, this element is extracted using the `pop()` method.
- `.astype(np.int32)` :: Using `masks.xy` will return with a data type of `float32`, but this won't be compatible with the OpenCV `drawContours()` function, so this will change the data type to `int32` for compatibility.
- `.reshape(-1, 1, 2)` :: Reformats the data into the required shape of `[N, 1, 2]` where `N` is the number of contour points, with each point represented by a single entry `1`, and the entry is composed of `2` values. The `-1` denotes that the number of values along this dimension is flexible.
</details>
<p></p>
<details>
<summary> Expand for an explanation of the <code>drawContours()</code> configuration.</summary>
<p>
- Encapsulating the `contour` variable within square brackets, `[contour]`, was found to effectively generate the desired contour mask during testing.
- The value `-1` specified for the `drawContours()` parameter instructs the function to draw all contours present in the image.
- The `tuple` `(255, 255, 255)` represents the color white, which is the desired color for drawing the contour in this binary mask.
- The addition of `cv2.FILLED` will color all pixels enclosed by the contour boundary the same, in this case, all enclosed pixels will be white.
- See [OpenCV Documentation on `drawContours()`](https://docs.opencv.org/4.8.0/d6/d6e/group__imgproc__draw.html#ga746c0625f1781f1ffc9056259103edbc) for more information.
</details>
<p></p>
***
5. Next there are 2 options for how to move forward with the image from this point and a subsequent option for each.
### Object Isolation Options
!!! example
=== "Black Background Pixels"
```python
# Create 3-channel mask
mask3ch = cv2.cvtColor(b_mask, cv2.COLOR_GRAY2BGR)
# Isolate object with binary mask
isolated = cv2.bitwise_and(mask3ch, img)
```
??? question "How does this work?"
- First, the binary mask is first converted from a single-channel image to a three-channel image. This conversion is necessary for the subsequent step where the mask and the original image are combined. Both images must have the same number of channels to be compatible with the blending operation.
- The original image and the three-channel binary mask are merged using the OpenCV function `bitwise_and()`. This operation retains <u>only</u> pixel values that are greater than zero `(> 0)` from both images. Since the mask pixels are greater than zero `(> 0)` <u>only</u> within the contour region, the pixels remaining from the original image are those that overlap with the contour.
### Isolate with Black Pixels: Sub-options
??? info "Full-size Image"
There are no additional steps required if keeping full size image.
<figure markdown>
![Example Full size Isolated Object Image Black Background](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/full-size-isolated-object-black-background.avif){ width=240 }
<figcaption>Example full-size output</figcaption>
</figure>
??? info "Cropped object Image"
Additional steps required to crop image to only include object region.
![Example Crop Isolated Object Image Black Background](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/example-crop-isolated-object-image-black-background.avif){ align="right" }
```{ .py .annotate }
# (1) Bounding box coordinates
x1, y1, x2, y2 = c.boxes.xyxy.cpu().numpy().squeeze().astype(np.int32)
# Crop image to object region
iso_crop = isolated[y1:y2, x1:x2]
```
1. For more information on [bounding box](https://www.ultralytics.com/glossary/bounding-box) results, see [Boxes Section from Predict Mode](../modes/predict.md#boxes)
??? question "What does this code do?"
- The `c.boxes.xyxy.cpu().numpy()` call retrieves the bounding boxes as a NumPy array in the `xyxy` format, where `xmin`, `ymin`, `xmax`, and `ymax` represent the coordinates of the bounding box rectangle. See [Boxes Section from Predict Mode](../modes/predict.md#boxes) for more details.
- The `squeeze()` operation removes any unnecessary dimensions from the NumPy array, ensuring it has the expected shape.
- Converting the coordinate values using `.astype(np.int32)` changes the box coordinates data type from `float32` to `int32`, making them compatible for image cropping using index slices.
- Finally, the bounding box region is cropped from the image using index slicing. The bounds are defined by the `[ymin:ymax, xmin:xmax]` coordinates of the detection bounding box.
=== "Transparent Background Pixels"
```python
# Isolate object with transparent background (when saved as PNG)
isolated = np.dstack([img, b_mask])
```
??? question "How does this work?"
- Using the NumPy `dstack()` function (array stacking along depth-axis) in conjunction with the binary mask generated, will create an image with four channels. This allows for all pixels outside of the object contour to be transparent when saving as a `PNG` file.
### Isolate with Transparent Pixels: Sub-options
??? info "Full-size Image"
There are no additional steps required if keeping full size image.
<figure markdown>
![Example Full size Isolated Object Image No Background](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/example-full-size-isolated-object-image-no-background.avif){ width=240 }
<figcaption>Example full-size output + transparent background</figcaption>
</figure>
??? info "Cropped object Image"
Additional steps required to crop image to only include object region.
![Example Crop Isolated Object Image No Background](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/example-crop-isolated-object-image-no-background.avif){ align="right" }
```{ .py .annotate }
# (1) Bounding box coordinates
x1, y1, x2, y2 = c.boxes.xyxy.cpu().numpy().squeeze().astype(np.int32)
# Crop image to object region
iso_crop = isolated[y1:y2, x1:x2]
```
1. For more information on bounding box results, see [Boxes Section from Predict Mode](../modes/predict.md#boxes)
??? question "What does this code do?"
- When using `c.boxes.xyxy.cpu().numpy()`, the bounding boxes are returned as a NumPy array, using the `xyxy` box coordinates format, which correspond to the points `xmin, ymin, xmax, ymax` for the bounding box (rectangle), see [Boxes Section from Predict Mode](../modes/predict.md#boxes) for more information.
- Adding `squeeze()` ensures that any extraneous dimensions are removed from the NumPy array.
- Converting the coordinate values using `.astype(np.int32)` changes the box coordinates data type from `float32` to `int32` which will be compatible when cropping the image using index slices.
- Finally the image region for the bounding box is cropped using index slicing, where the bounds are set using the `[ymin:ymax, xmin:xmax]` coordinates of the detection bounding box.
??? question "What if I want the cropped object **including** the background?"
This is a built-in feature for the Ultralytics library. See the `save_crop` argument for [Predict Mode Inference Arguments](../modes/predict.md#inference-arguments) for details.
***
6. <u>What to do next is entirely left to you as the developer.</u> A basic example of one possible next step (saving the image to file for future use) is shown.
- **NOTE:** this step is optional and can be skipped if not required for your specific use case.
??? example "Example Final Step"
```python
# Save isolated object to file
_ = cv2.imwrite(f"{img_name}_{label}-{ci}.png", iso_crop)
```
- In this example, the `img_name` is the base-name of the source image file, `label` is the detected class-name, and `ci` is the index of the [object detection](https://www.ultralytics.com/glossary/object-detection) (in case of multiple instances with the same class name).
## Full Example code
Here, all steps from the previous section are combined into a single block of code. For repeated use, it would be optimal to define a function to do some or all commands contained in the `for`-loops, but that is an exercise left to the reader.
```{ .py .annotate }
from pathlib import Path
import cv2
import numpy as np
from ultralytics import YOLO
m = YOLO("yolo26n-seg.pt") # (4)!
res = m.predict(source="path/to/image.jpg") # (3)!
# Iterate detection results (5)
for r in res:
img = np.copy(r.orig_img)
img_name = Path(r.path).stem
# Iterate each object contour (6)
for ci, c in enumerate(r):
label = c.names[c.boxes.cls.tolist().pop()]
b_mask = np.zeros(img.shape[:2], np.uint8)
# Create contour mask (1)
contour = c.masks.xy.pop().astype(np.int32).reshape(-1, 1, 2)
_ = cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)
# Choose one:
# OPTION-1: Isolate object with black background
mask3ch = cv2.cvtColor(b_mask, cv2.COLOR_GRAY2BGR)
isolated = cv2.bitwise_and(mask3ch, img)
# OPTION-2: Isolate object with transparent background (when saved as PNG)
isolated = np.dstack([img, b_mask])
# OPTIONAL: detection crop (from either OPT1 or OPT2)
x1, y1, x2, y2 = c.boxes.xyxy.cpu().numpy().squeeze().astype(np.int32)
iso_crop = isolated[y1:y2, x1:x2]
# Add your custom post-processing here (2)
```
1. The line populating `contour` is combined into a single line here, where it was split to multiple above.
2. {==What goes here is up to you!==}
3. See [Predict Mode](../modes/predict.md) for additional information.
4. See [Segment Task](../tasks/segment.md#models) for more information.
5. Learn more about [Working with Results](../modes/predict.md#working-with-results)
6. Learn more about [Segmentation Mask Results](../modes/predict.md#masks)
## FAQ
### How do I isolate objects using Ultralytics YOLO26 for segmentation tasks?
To isolate objects using Ultralytics YOLO26, follow these steps:
1. **Load the model and run inference:**
```python
from ultralytics import YOLO
model = YOLO("yolo26n-seg.pt")
results = model.predict(source="path/to/your/image.jpg")
```
2. **Generate a binary mask and draw contours:**
```python
import cv2
import numpy as np
img = np.copy(results[0].orig_img)
b_mask = np.zeros(img.shape[:2], np.uint8)
contour = results[0].masks.xy[0].astype(np.int32).reshape(-1, 1, 2)
cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)
```
3. **Isolate the object using the binary mask:**
```python
mask3ch = cv2.cvtColor(b_mask, cv2.COLOR_GRAY2BGR)
isolated = cv2.bitwise_and(mask3ch, img)
```
Refer to the guide on [Predict Mode](../modes/predict.md) and the [Segment Task](../tasks/segment.md) for more information.
### What options are available for saving the isolated objects after segmentation?
Ultralytics YOLO26 offers two main options for saving isolated objects:
1. **With a Black Background:**
```python
mask3ch = cv2.cvtColor(b_mask, cv2.COLOR_GRAY2BGR)
isolated = cv2.bitwise_and(mask3ch, img)
```
2. **With a Transparent Background:**
```python
isolated = np.dstack([img, b_mask])
```
For further details, visit the [Predict Mode](../modes/predict.md) section.
### How can I crop isolated objects to their bounding boxes using Ultralytics YOLO26?
To crop isolated objects to their bounding boxes:
1. **Retrieve bounding box coordinates:**
```python
x1, y1, x2, y2 = results[0].boxes.xyxy[0].cpu().numpy().astype(np.int32)
```
2. **Crop the isolated image:**
```python
iso_crop = isolated[y1:y2, x1:x2]
```
Learn more about bounding box results in the [Predict Mode](../modes/predict.md#boxes) documentation.
### Why should I use Ultralytics YOLO26 for object isolation in segmentation tasks?
Ultralytics YOLO26 provides:
- **High-speed** real-time object detection and segmentation.
- **Accurate bounding box and mask generation** for precise object isolation.
- **Comprehensive documentation** and easy-to-use API for efficient development.
Explore the benefits of using YOLO in the [Segment Task documentation](../tasks/segment.md).
### Can I save isolated objects including the background using Ultralytics YOLO26?
Yes, this is a built-in feature in Ultralytics YOLO26. Use the `save_crop` argument in the `predict()` method. For example:
```python
results = model.predict(source="path/to/your/image.jpg", save_crop=True)
```
Read more about the `save_crop` argument in the [Predict Mode Inference Arguments](../modes/predict.md#inference-arguments) section.

324
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/kfold-cross-validation.md Normal file
View File

@@ -0,0 +1,324 @@
---
comments: true
description: Learn to implement K-Fold Cross Validation for object detection datasets using Ultralytics YOLO. Improve your model's reliability and robustness.
keywords: Ultralytics, YOLO, K-Fold Cross Validation, object detection, sklearn, pandas, PyYAML, machine learning, dataset split
---
# K-Fold Cross Validation with Ultralytics
## Introduction
This comprehensive guide illustrates the implementation of K-Fold Cross Validation for [object detection](https://www.ultralytics.com/glossary/object-detection) datasets within the Ultralytics ecosystem. We'll leverage the YOLO detection format and key Python libraries such as sklearn, pandas, and PyYAML to guide you through the necessary setup, the process of generating feature vectors, and the execution of a K-Fold dataset split.
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/k-fold-cross-validation-overview.avif" alt="K-fold cross validation data splitting">
</p>
Whether your project involves the Fruit Detection dataset or a custom data source, this tutorial aims to help you comprehend and apply K-Fold Cross Validation to bolster the reliability and robustness of your [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) models. While we're applying `k=5` folds for this tutorial, keep in mind that the optimal number of folds can vary depending on your dataset and the specifics of your project.
Let's get started.
## Setup
- Your annotations should be in the [YOLO detection format](../datasets/detect/index.md).
- This guide assumes that annotation files are locally available.
- For our demonstration, we use the [Fruit Detection](https://www.kaggle.com/datasets/lakshaytyagi01/fruit-detection/code) dataset.
- This dataset contains a total of 8479 images.
- It includes 6 class labels, each with its total instance counts listed below.
| Class Label | Instance Count |
| :---------- | :------------: |
| Apple | 7049 |
| Grapes | 7202 |
| Pineapple | 1613 |
| Orange | 15549 |
| Banana | 3536 |
| Watermelon | 1976 |
- Necessary Python packages include:
- `ultralytics`
- `sklearn`
- `pandas`
- `pyyaml`
- This tutorial operates with `k=5` folds. However, you should determine the best number of folds for your specific dataset.
1. Initiate a new Python virtual environment (`venv`) for your project and activate it. Use `pip` (or your preferred package manager) to install:
- The Ultralytics library: `pip install -U ultralytics`. Alternatively, you can clone the official [repo](https://github.com/ultralytics/ultralytics).
- Scikit-learn, pandas, and PyYAML: `pip install -U scikit-learn pandas pyyaml`.
2. Verify that your annotations are in the [YOLO detection format](../datasets/detect/index.md).
- For this tutorial, all annotation files are found in the `Fruit-Detection/labels` directory.
## Generating Feature Vectors for Object Detection Dataset
1. Start by creating a new `example.py` Python file for the steps below.
2. Proceed to retrieve all label files for your dataset.
```python
from pathlib import Path
dataset_path = Path("./Fruit-detection") # replace with 'path/to/dataset' for your custom data
labels = sorted(dataset_path.rglob("*labels/*.txt")) # all data in 'labels'
```
3. Now, read the contents of the dataset YAML file and extract the indices of the class labels.
```python
import yaml
yaml_file = "path/to/data.yaml" # your data YAML with data directories and names dictionary
with open(yaml_file, encoding="utf8") as y:
classes = yaml.safe_load(y)["names"]
cls_idx = sorted(classes.keys())
```
4. Initialize an empty `pandas` DataFrame.
```python
import pandas as pd
index = [label.stem for label in labels] # uses base filename as ID (no extension)
labels_df = pd.DataFrame([], columns=cls_idx, index=index)
```
5. Count the instances of each class-label present in the annotation files.
```python
from collections import Counter
for label in labels:
lbl_counter = Counter()
with open(label) as lf:
lines = lf.readlines()
for line in lines:
# classes for YOLO label uses integer at first position of each line
lbl_counter[int(line.split(" ", 1)[0])] += 1
labels_df.loc[label.stem] = lbl_counter
labels_df = labels_df.fillna(0.0) # replace `nan` values with `0.0`
```
6. The following is a sample view of the populated DataFrame:
```
0 1 2 3 4 5
'0000a16e4b057580_jpg.rf.00ab48988370f64f5ca8ea4...' 0.0 0.0 0.0 0.0 0.0 7.0
'0000a16e4b057580_jpg.rf.7e6dce029fb67f01eb19aa7...' 0.0 0.0 0.0 0.0 0.0 7.0
'0000a16e4b057580_jpg.rf.bc4d31cdcbe229dd022957a...' 0.0 0.0 0.0 0.0 0.0 7.0
'00020ebf74c4881c_jpg.rf.508192a0a97aa6c4a3b6882...' 0.0 0.0 0.0 1.0 0.0 0.0
'00020ebf74c4881c_jpg.rf.5af192a2254c8ecc4188a25...' 0.0 0.0 0.0 1.0 0.0 0.0
... ... ... ... ... ... ...
'ff4cd45896de38be_jpg.rf.c4b5e967ca10c7ced3b9e97...' 0.0 0.0 0.0 0.0 0.0 2.0
'ff4cd45896de38be_jpg.rf.ea4c1d37d2884b3e3cbce08...' 0.0 0.0 0.0 0.0 0.0 2.0
'ff5fd9c3c624b7dc_jpg.rf.bb519feaa36fc4bf630a033...' 1.0 0.0 0.0 0.0 0.0 0.0
'ff5fd9c3c624b7dc_jpg.rf.f0751c9c3aa4519ea3c9d6a...' 1.0 0.0 0.0 0.0 0.0 0.0
'fffe28b31f2a70d4_jpg.rf.7ea16bd637ba0711c53b540...' 0.0 6.0 0.0 0.0 0.0 0.0
```
The rows index the label files, each corresponding to an image in your dataset, and the columns correspond to your class-label indices. Each row represents a pseudo feature-vector, with the count of each class-label present in your dataset. This data structure enables the application of [K-Fold Cross Validation](https://www.ultralytics.com/glossary/cross-validation) to an object detection dataset.
## K-Fold Dataset Split
1. Now we will use the `KFold` class from `sklearn.model_selection` to generate `k` splits of the dataset.
- Important:
- Setting `shuffle=True` ensures a randomized distribution of classes in your splits.
- By setting `random_state=M` where `M` is a chosen integer, you can obtain repeatable results.
```python
import random
from sklearn.model_selection import KFold
random.seed(0) # for reproducibility
ksplit = 5
kf = KFold(n_splits=ksplit, shuffle=True, random_state=20) # setting random_state for repeatable results
kfolds = list(kf.split(labels_df))
```
2. The dataset has now been split into `k` folds, each having a list of `train` and `val` indices. We will construct a DataFrame to display these results more clearly.
```python
folds = [f"split_{n}" for n in range(1, ksplit + 1)]
folds_df = pd.DataFrame(index=index, columns=folds)
for i, (train, val) in enumerate(kfolds, start=1):
folds_df[f"split_{i}"].loc[labels_df.iloc[train].index] = "train"
folds_df[f"split_{i}"].loc[labels_df.iloc[val].index] = "val"
```
3. Now we will calculate the distribution of class labels for each fold as a ratio of the classes present in `val` to those present in `train`.
```python
fold_lbl_distrb = pd.DataFrame(index=folds, columns=cls_idx)
for n, (train_indices, val_indices) in enumerate(kfolds, start=1):
train_totals = labels_df.iloc[train_indices].sum()
val_totals = labels_df.iloc[val_indices].sum()
# To avoid division by zero, we add a small value (1E-7) to the denominator
ratio = val_totals / (train_totals + 1e-7)
fold_lbl_distrb.loc[f"split_{n}"] = ratio
```
The ideal scenario is for all class ratios to be reasonably similar for each split and across classes. This, however, will be subject to the specifics of your dataset.
4. Next, we create the directories and dataset YAML files for each split.
```python
import datetime
supported_extensions = [".jpg", ".jpeg", ".png"]
# Initialize an empty list to store image file paths
images = []
# Loop through supported extensions and gather image files
for ext in supported_extensions:
images.extend(sorted((dataset_path / "images").rglob(f"*{ext}")))
# Create the necessary directories and dataset YAML files
save_path = Path(dataset_path / f"{datetime.date.today().isoformat()}_{ksplit}-Fold_Cross-val")
save_path.mkdir(parents=True, exist_ok=True)
ds_yamls = []
for split in folds_df.columns:
# Create directories
split_dir = save_path / split
split_dir.mkdir(parents=True, exist_ok=True)
(split_dir / "train" / "images").mkdir(parents=True, exist_ok=True)
(split_dir / "train" / "labels").mkdir(parents=True, exist_ok=True)
(split_dir / "val" / "images").mkdir(parents=True, exist_ok=True)
(split_dir / "val" / "labels").mkdir(parents=True, exist_ok=True)
# Create dataset YAML files
dataset_yaml = split_dir / f"{split}_dataset.yaml"
ds_yamls.append(dataset_yaml)
with open(dataset_yaml, "w") as ds_y:
yaml.safe_dump(
{
"path": split_dir.as_posix(),
"train": "train",
"val": "val",
"names": classes,
},
ds_y,
)
```
5. Lastly, copy images and labels into the respective directory ('train' or 'val') for each split.
- **NOTE:** The time required for this portion of the code will vary based on the size of your dataset and your system hardware.
```python
import shutil
from tqdm import tqdm
for image, label in tqdm(zip(images, labels), total=len(images), desc="Copying files"):
for split, k_split in folds_df.loc[image.stem].items():
# Destination directory
img_to_path = save_path / split / k_split / "images"
lbl_to_path = save_path / split / k_split / "labels"
# Copy image and label files to new directory (SamefileError if file already exists)
shutil.copy(image, img_to_path / image.name)
shutil.copy(label, lbl_to_path / label.name)
```
## Save Records (Optional)
Optionally, you can save the records of the K-Fold split and label distribution DataFrames as CSV files for future reference.
```python
folds_df.to_csv(save_path / "kfold_datasplit.csv")
fold_lbl_distrb.to_csv(save_path / "kfold_label_distribution.csv")
```
## Train YOLO using K-Fold Data Splits
1. First, load the YOLO model.
```python
from ultralytics import YOLO
weights_path = "path/to/weights.pt" # use yolo26n.pt for a small model
model = YOLO(weights_path, task="detect")
```
2. Next, iterate over the dataset YAML files to run training. The results will be saved to a directory specified by the `project` and `name` arguments. By default, this directory is 'runs/detect/train#' where # is an integer index.
```python
results = {}
# Define your additional arguments here
batch = 16
project = "kfold_demo"
epochs = 100
for k, dataset_yaml in enumerate(ds_yamls):
model = YOLO(weights_path, task="detect")
results[k] = model.train(
data=dataset_yaml, epochs=epochs, batch=batch, project=project, name=f"fold_{k + 1}"
) # include any additional train arguments
```
3. You can also use [Ultralytics data.utils.autosplit](https://docs.ultralytics.com/reference/data/utils/) function for automatic dataset splitting:
```python
from ultralytics.data.split import autosplit
# Automatically split dataset into train/val/test
autosplit(path="path/to/images", weights=(0.8, 0.2, 0.0), annotated_only=True)
```
## Conclusion
In this guide, we have explored the process of using K-Fold cross-validation for training the YOLO object detection model. We learned how to split our dataset into K partitions, ensuring a balanced class distribution across the different folds.
We also explored the procedure for creating report DataFrames to visualize the data splits and label distributions across these splits, providing us a clear insight into the structure of our training and validation sets.
Optionally, we saved our records for future reference, which could be particularly useful in large-scale projects or when troubleshooting model performance.
Finally, we implemented the actual model training using each split in a loop, saving our training results for further analysis and comparison.
This technique of K-Fold cross-validation is a robust way of making the most out of your available data, and it helps to ensure that your model performance is reliable and consistent across different data subsets. This results in a more generalizable and reliable model that is less likely to [overfit](https://www.ultralytics.com/glossary/overfitting) to specific data patterns.
Remember that although we used YOLO in this guide, these steps are mostly transferable to other machine learning models. Understanding these steps allows you to apply cross-validation effectively in your own machine learning projects.
## FAQ
### What is K-Fold Cross Validation and why is it useful in object detection?
K-Fold Cross Validation is a technique where the dataset is divided into 'k' subsets (folds) to evaluate model performance more reliably. Each fold serves as both training and [validation data](https://www.ultralytics.com/glossary/validation-data). In the context of object detection, using K-Fold Cross Validation helps to ensure your Ultralytics YOLO model's performance is robust and generalizable across different data splits, enhancing its reliability. For detailed instructions on setting up K-Fold Cross Validation with Ultralytics YOLO, refer to [K-Fold Cross Validation with Ultralytics](#introduction).
### How do I implement K-Fold Cross Validation using Ultralytics YOLO?
To implement K-Fold Cross Validation with Ultralytics YOLO, you need to follow these steps:
1. Verify annotations are in the [YOLO detection format](../datasets/detect/index.md).
2. Use Python libraries like `sklearn`, `pandas`, and `pyyaml`.
3. Create feature vectors from your dataset.
4. Split your dataset using `KFold` from `sklearn.model_selection`.
5. Train the YOLO model on each split.
For a comprehensive guide, see the [K-Fold Dataset Split](#k-fold-dataset-split) section in our documentation.
### Why should I use Ultralytics YOLO for object detection?
Ultralytics YOLO offers state-of-the-art, real-time object detection with high [accuracy](https://www.ultralytics.com/glossary/accuracy) and efficiency. It's versatile, supporting multiple [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks such as detection, segmentation, and classification. Additionally, it integrates seamlessly with tools like [Ultralytics Platform](https://docs.ultralytics.com/platform/) for no-code model training and deployment. For more details, explore the benefits and features on our [Ultralytics YOLO page](https://www.ultralytics.com/yolo).
### How can I ensure my annotations are in the correct format for Ultralytics YOLO?
Your annotations should follow the YOLO detection format. Each annotation file must list the object class, alongside its [bounding box](https://www.ultralytics.com/glossary/bounding-box) coordinates in the image. The YOLO format ensures streamlined and standardized data processing for training object detection models. For more information on proper annotation formatting, visit the [YOLO detection format guide](../datasets/detect/index.md).
### Can I use K-Fold Cross Validation with custom datasets other than Fruit Detection?
Yes, you can use K-Fold Cross Validation with any custom dataset as long as the annotations are in the YOLO detection format. Replace the dataset paths and class labels with those specific to your custom dataset. This flexibility ensures that any object detection project can benefit from robust model evaluation using K-Fold Cross Validation. For a practical example, review our [Generating Feature Vectors](#generating-feature-vectors-for-object-detection-dataset) section.

317
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-deployment-options.md Normal file
View File

@@ -0,0 +1,317 @@
---
comments: true
description: Learn about YOLO26's diverse deployment options to maximize your model's performance. Explore PyTorch, TensorRT, OpenVINO, TF Lite, and more!
keywords: YOLO26, deployment options, export formats, PyTorch, TensorRT, OpenVINO, TF Lite, machine learning, model deployment
---
# Comparative Analysis of YOLO26 Deployment Options
## Introduction
You've come a long way on your journey with YOLO26. You've diligently collected data, meticulously annotated it, and put in the hours to train and rigorously evaluate your custom YOLO26 model. Now, it's time to put your model to work for your specific application, use case, or project. But there's a critical decision that stands before you: how to export and deploy your model effectively.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/QkCsj2SvZc4"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Choose the Best Ultralytics YOLO26 Deployment Format for Your Project | TensorRT | OpenVINO 🚀
</p>
This guide walks you through YOLO26's deployment options and the essential factors to consider to choose the right option for your project.
## How to Select the Right Deployment Option for Your YOLO26 Model
When it's time to deploy your YOLO26 model, selecting a suitable export format is very important. As outlined in the [Ultralytics YOLO26 Modes documentation](../modes/export.md#usage-examples), the `model.export()` function allows you to convert your trained model into a variety of formats tailored to diverse environments and performance requirements.
The ideal format depends on your model's intended operational context, balancing speed, hardware constraints, and ease of integration. In the following section, we'll take a closer look at each export option, understanding when to choose each one.
## YOLO26's Deployment Options
Let's walk through the different YOLO26 deployment options. For a detailed walkthrough of the export process, visit the [Ultralytics documentation page on exporting](../modes/export.md).
### PyTorch
PyTorch is an open-source machine learning library widely used for applications in [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) and [artificial intelligence](https://www.ultralytics.com/glossary/artificial-intelligence-ai). It provides a high level of flexibility and speed, which has made it a favorite among researchers and developers.
- **Performance Benchmarks**: PyTorch is known for its ease of use and flexibility, which may result in a slight trade-off in raw performance when compared to other frameworks that are more specialized and optimized.
- **Compatibility and Integration**: Offers excellent compatibility with various data science and machine learning libraries in Python.
- **Community Support and Ecosystem**: One of the most vibrant communities, with extensive resources for learning and troubleshooting.
- **Case Studies**: Commonly used in research prototypes, many academic papers reference models deployed in PyTorch.
- **Maintenance and Updates**: Regular updates with active development and support for new features.
- **Security Considerations**: Regular patches for security issues, but security is largely dependent on the overall environment it's deployed in.
- **Hardware Acceleration**: Supports CUDA for GPU acceleration, essential for speeding up model training and inference.
### TorchScript
TorchScript extends PyTorch's capabilities by allowing the exportation of models to be run in a C++ runtime environment. This makes it suitable for production environments where Python is unavailable.
- **Performance Benchmarks**: Can offer improved performance over native PyTorch, especially in production environments.
- **Compatibility and Integration**: Designed for seamless transition from PyTorch to C++ production environments, though some advanced features might not translate perfectly.
- **Community Support and Ecosystem**: Benefits from PyTorch's large community but has a narrower scope of specialized developers.
- **Case Studies**: Widely used in industry settings where Python's performance overhead is a bottleneck.
- **Maintenance and Updates**: Maintained alongside PyTorch with consistent updates.
- **Security Considerations**: Offers improved security by enabling the running of models in environments without full Python installations.
- **Hardware Acceleration**: Inherits PyTorch's CUDA support, ensuring efficient GPU utilization.
### ONNX
The Open [Neural Network](https://www.ultralytics.com/glossary/neural-network-nn) Exchange (ONNX) is a format that allows for model interoperability across different frameworks, which can be critical when deploying to various platforms.
- **Performance Benchmarks**: ONNX models may experience a variable performance depending on the specific runtime they are deployed on.
- **Compatibility and Integration**: High interoperability across multiple platforms and hardware due to its framework-agnostic nature.
- **Community Support and Ecosystem**: Supported by many organizations, leading to a broad ecosystem and a variety of tools for optimization.
- **Case Studies**: Frequently used to move models between different machine learning frameworks, demonstrating its flexibility.
- **Maintenance and Updates**: As an open standard, ONNX is regularly updated to support new operations and models.
- **Security Considerations**: As with any cross-platform tool, it's essential to ensure secure practices in the conversion and deployment pipeline.
- **Hardware Acceleration**: With ONNX Runtime, models can leverage various hardware optimizations.
### OpenVINO
OpenVINO is an Intel toolkit designed to facilitate the deployment of deep learning models across Intel hardware, enhancing performance and speed.
- **Performance Benchmarks**: Specifically optimized for Intel CPUs, GPUs, and VPUs, offering significant performance boosts on compatible hardware.
- **Compatibility and Integration**: Works best within the Intel ecosystem but also supports a range of other platforms.
- **Community Support and Ecosystem**: Backed by Intel, with a solid user base especially in the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) domain.
- **Case Studies**: Often utilized in IoT and [edge computing](https://www.ultralytics.com/glossary/edge-computing) scenarios where Intel hardware is prevalent.
- **Maintenance and Updates**: Intel regularly updates OpenVINO to support the latest deep learning models and Intel hardware.
- **Security Considerations**: Provides robust security features suitable for deployment in sensitive applications.
- **Hardware Acceleration**: Tailored for acceleration on Intel hardware, leveraging dedicated instruction sets and hardware features.
For more details on deployment using OpenVINO, refer to the Ultralytics Integration documentation: [Intel OpenVINO Export](../integrations/openvino.md).
### TensorRT
TensorRT is a high-performance deep learning inference optimizer and runtime from NVIDIA, ideal for applications needing speed and efficiency.
- **Performance Benchmarks**: Delivers top-tier performance on NVIDIA GPUs with support for high-speed inference.
- **Compatibility and Integration**: Best suited for NVIDIA hardware, with limited support outside this environment.
- **Community Support and Ecosystem**: Strong support network through NVIDIA's developer forums and documentation.
- **Case Studies**: Widely adopted in industries requiring real-time inference on video and image data.
- **Maintenance and Updates**: NVIDIA maintains TensorRT with frequent updates to enhance performance and support new GPU architectures.
- **Security Considerations**: Like many NVIDIA products, it has a strong emphasis on security, but specifics depend on the deployment environment.
- **Hardware Acceleration**: Exclusively designed for NVIDIA GPUs, providing deep optimization and acceleration.
For more information on TensorRT deployment, check out the [TensorRT integration guide](../integrations/tensorrt.md).
### CoreML
CoreML is Apple's machine learning framework, optimized for on-device performance in the Apple ecosystem, including iOS, macOS, watchOS, and tvOS.
- **Performance Benchmarks**: Optimized for on-device performance on Apple hardware with minimal battery usage.
- **Compatibility and Integration**: Exclusively for Apple's ecosystem, providing a streamlined workflow for iOS and macOS applications.
- **Community Support and Ecosystem**: Strong support from Apple and a dedicated developer community, with extensive documentation and tools.
- **Case Studies**: Commonly used in applications that require on-device machine learning capabilities on Apple products.
- **Maintenance and Updates**: Regularly updated by Apple to support the latest machine learning advancements and Apple hardware.
- **Security Considerations**: Benefits from Apple's focus on user privacy and [data security](https://www.ultralytics.com/glossary/data-security).
- **Hardware Acceleration**: Takes full advantage of Apple's neural engine and GPU for accelerated machine learning tasks.
### TF SavedModel
TF SavedModel is TensorFlow's format for saving and serving machine learning models, particularly suited for scalable server environments.
- **Performance Benchmarks**: Offers scalable performance in server environments, especially when used with TensorFlow Serving.
- **Compatibility and Integration**: Wide compatibility across TensorFlow's ecosystem, including cloud and enterprise server deployments.
- **Community Support and Ecosystem**: Large community support due to TensorFlow's popularity, with a vast array of tools for deployment and optimization.
- **Case Studies**: Extensively used in production environments for serving deep learning models at scale.
- **Maintenance and Updates**: Supported by Google and the TensorFlow community, ensuring regular updates and new features.
- **Security Considerations**: Deployment using TensorFlow Serving includes robust security features for enterprise-grade applications.
- **Hardware Acceleration**: Supports various hardware accelerations through TensorFlow's backends.
### TF GraphDef
TF GraphDef is a TensorFlow format that represents the model as a graph, which is beneficial for environments where a static computation graph is required.
- **Performance Benchmarks**: Provides stable performance for static computation graphs, with a focus on consistency and reliability.
- **Compatibility and Integration**: Easily integrates within TensorFlow's infrastructure but less flexible compared to SavedModel.
- **Community Support and Ecosystem**: Good support from TensorFlow's ecosystem, with many resources available for optimizing static graphs.
- **Case Studies**: Useful in scenarios where a static graph is necessary, such as in certain embedded systems.
- **Maintenance and Updates**: Regular updates alongside TensorFlow's core updates.
- **Security Considerations**: Ensures safe deployment with TensorFlow's established security practices.
- **Hardware Acceleration**: Can utilize TensorFlow's hardware acceleration options, though not as flexible as SavedModel.
Learn more about TF GraphDef in our [TF GraphDef integration guide](../integrations/tf-graphdef.md).
### TF Lite
TF Lite is TensorFlow's solution for mobile and embedded device machine learning, providing a lightweight library for on-device inference.
- **Performance Benchmarks**: Designed for speed and efficiency on mobile and embedded devices.
- **Compatibility and Integration**: Can be used on a wide range of devices due to its lightweight nature.
- **Community Support and Ecosystem**: Backed by Google, it has a robust community and a growing number of resources for developers.
- **Case Studies**: Popular in mobile applications that require on-device inference with minimal footprint.
- **Maintenance and Updates**: Regularly updated to include the latest features and optimizations for mobile devices.
- **Security Considerations**: Provides a secure environment for running models on end-user devices.
- **Hardware Acceleration**: Supports a variety of hardware acceleration options, including GPU and DSP.
### TF Edge TPU
TF Edge TPU is designed for high-speed, efficient computing on Google's Edge TPU hardware, perfect for IoT devices requiring real-time processing.
- **Performance Benchmarks**: Specifically optimized for high-speed, efficient computing on Google's Edge TPU hardware.
- **Compatibility and Integration**: Works exclusively with TensorFlow Lite models on Edge TPU devices.
- **Community Support and Ecosystem**: Growing support with resources provided by Google and third-party developers.
- **Case Studies**: Used in IoT devices and applications that require real-time processing with low latency.
- **Maintenance and Updates**: Continually improved upon to leverage the capabilities of new Edge TPU hardware releases.
- **Security Considerations**: Integrates with Google's robust security for IoT and edge devices.
- **Hardware Acceleration**: Custom-designed to take full advantage of Google Coral devices.
### TF.js
TensorFlow.js (TF.js) is a library that brings machine learning capabilities directly to the browser, offering a new realm of possibilities for web developers and users alike. It allows for the integration of machine learning models in web applications without the need for back-end infrastructure.
- **Performance Benchmarks**: Enables machine learning directly in the browser with reasonable performance, depending on the client device.
- **Compatibility and Integration**: High compatibility with web technologies, allowing for easy integration into web applications.
- **Community Support and Ecosystem**: Support from a community of web and Node.js developers, with a variety of tools for deploying ML models in browsers.
- **Case Studies**: Ideal for interactive web applications that benefit from client-side machine learning without the need for server-side processing.
- **Maintenance and Updates**: Maintained by the TensorFlow team with contributions from the open-source community.
- **Security Considerations**: Runs within the browser's secure context, utilizing the security model of the web platform.
- **Hardware Acceleration**: Performance can be enhanced with web-based APIs that access hardware acceleration like WebGL.
### PaddlePaddle
PaddlePaddle is an open-source deep learning framework developed by Baidu. It is designed to be both efficient for researchers and easy to use for developers. It's particularly popular in China and offers specialized support for Chinese language processing.
- **Performance Benchmarks**: Offers competitive performance with a focus on ease of use and scalability.
- **Compatibility and Integration**: Well-integrated within Baidu's ecosystem and supports a wide range of applications.
- **Community Support and Ecosystem**: While the community is smaller globally, it's rapidly growing, especially in China.
- **Case Studies**: Commonly used in Chinese markets and by developers looking for alternatives to other major frameworks.
- **Maintenance and Updates**: Regularly updated with a focus on serving Chinese language AI applications and services.
- **Security Considerations**: Emphasizes [data privacy](https://www.ultralytics.com/glossary/data-privacy) and security, catering to Chinese data governance standards.
- **Hardware Acceleration**: Supports various hardware accelerations, including Baidu's own Kunlun chips.
### MNN
MNN is a highly efficient and lightweight deep learning framework. It supports inference and training of deep learning models and has industry-leading performance for inference and training on-device. In addition, MNN is also used on embedded devices, such as IoT.
- **Performance Benchmarks**: High-performance for mobile devices with excellent optimization for ARM systems.
- **Compatibility and Integration**: Works well with mobile and embedded ARM systems and X86-64 CPU architectures.
- **Community Support and Ecosystem**: Supported by the mobile and embedded machine learning community.
- **Case Studies**: Ideal for applications requiring efficient performance on mobile systems.
- **Maintenance and Updates**: Regularly maintained to ensure high performance on mobile devices.
- **Security Considerations**: Provides on-device security advantages by keeping data local.
- **Hardware Acceleration**: Optimized for ARM CPUs and GPUs for maximum efficiency.
### NCNN
NCNN is a high-performance neural network inference framework optimized for the mobile platform. It stands out for its lightweight nature and efficiency, making it particularly well-suited for mobile and embedded devices where resources are limited.
- **Performance Benchmarks**: Highly optimized for mobile platforms, offering efficient inference on ARM-based devices.
- **Compatibility and Integration**: Suitable for applications on mobile phones and embedded systems with ARM architecture.
- **Community Support and Ecosystem**: Supported by a niche but active community focused on mobile and embedded ML applications.
- **Case Studies**: Favored for mobile applications where efficiency and speed are critical on Android and other ARM-based systems.
- **Maintenance and Updates**: Continuously improved to maintain high performance on a range of ARM devices.
- **Security Considerations**: Focuses on running locally on the device, leveraging the inherent security of on-device processing.
- **Hardware Acceleration**: Tailored for ARM CPUs and GPUs, with specific optimizations for these architectures.
## Comparative Analysis of YOLO26 Deployment Options
The following table provides a snapshot of the various deployment options available for YOLO26 models, helping you to assess which may best fit your project needs based on several critical criteria. For an in-depth look at each deployment option's format, please see the [Ultralytics documentation page on export formats](../modes/export.md#export-formats).
| Deployment Option | Performance Benchmarks | Compatibility and Integration | Community Support and Ecosystem | Case Studies | Maintenance and Updates | Security Considerations | Hardware Acceleration |
| ----------------- | ----------------------------------------------- | ---------------------------------------------- | --------------------------------------------- | ------------------------------------------ | ---------------------------------------------- | ------------------------------------------------- | ---------------------------------- |
| PyTorch | Good flexibility; may trade off raw performance | Excellent with Python libraries | Extensive resources and community | Research and prototypes | Regular, active development | Dependent on deployment environment | CUDA support for GPU acceleration |
| TorchScript | Better for production than PyTorch | Smooth transition from PyTorch to C++ | Specialized but narrower than PyTorch | Industry where Python is a bottleneck | Consistent updates with PyTorch | Improved security without full Python | Inherits CUDA support from PyTorch |
| ONNX | Variable depending on runtime | High across different frameworks | Broad ecosystem, supported by many orgs | Flexibility across ML frameworks | Regular updates for new operations | Ensure secure conversion and deployment practices | Various hardware optimizations |
| OpenVINO | Optimized for Intel hardware | Best within Intel ecosystem | Solid in computer vision domain | IoT and edge with Intel hardware | Regular updates for Intel hardware | Robust features for sensitive applications | Tailored for Intel hardware |
| TensorRT | Top-tier on NVIDIA GPUs | Best for NVIDIA hardware | Strong network through NVIDIA | Real-time video and image inference | Frequent updates for new GPUs | Emphasis on security | Designed for NVIDIA GPUs |
| CoreML | Optimized for on-device Apple hardware | Exclusive to Apple ecosystem | Strong Apple and developer support | On-device ML on Apple products | Regular Apple updates | Focus on privacy and security | Apple neural engine and GPU |
| TF SavedModel | Scalable in server environments | Wide compatibility in TensorFlow ecosystem | Large support due to TensorFlow popularity | Serving models at scale | Regular updates by Google and community | Robust features for enterprise | Various hardware accelerations |
| TF GraphDef | Stable for static computation graphs | Integrates well with TensorFlow infrastructure | Resources for optimizing static graphs | Scenarios requiring static graphs | Updates alongside TensorFlow core | Established TensorFlow security practices | TensorFlow acceleration options |
| TF Lite | Speed and efficiency on mobile/embedded | Wide range of device support | Robust community, Google backed | Mobile applications with minimal footprint | Latest features for mobile | Secure environment on end-user devices | GPU and DSP among others |
| TF Edge TPU | Optimized for Google's Edge TPU hardware | Exclusive to Edge TPU devices | Growing with Google and third-party resources | IoT devices requiring real-time processing | Improvements for new Edge TPU hardware | Google's robust IoT security | Custom-designed for Google Coral |
| TF.js | Reasonable in-browser performance | High with web technologies | Web and Node.js developers support | Interactive web applications | TensorFlow team and community contributions | Web platform security model | Enhanced with WebGL and other APIs |
| PaddlePaddle | Competitive, easy to use and scalable | Baidu ecosystem, wide application support | Rapidly growing, especially in China | Chinese market and language processing | Focus on Chinese AI applications | Emphasizes data privacy and security | Including Baidu's Kunlun chips |
| MNN | High-performance for mobile devices. | Mobile and embedded ARM systems and X86-64 CPU | Mobile/embedded ML community | Mobile systems efficiency | High performance maintenance on Mobile Devices | On-device security advantages | ARM CPUs and GPUs optimizations |
| NCNN | Optimized for mobile ARM-based devices | Mobile and embedded ARM systems | Niche but active mobile/embedded ML community | Android and ARM systems efficiency | High performance maintenance on ARM | On-device security advantages | ARM CPUs and GPUs optimizations |
This comparative analysis gives you a high-level overview. For deployment, it's essential to consider the specific requirements and constraints of your project, and consult the detailed documentation and resources available for each option.
## Community and Support
When you're getting started with YOLO26, having a helpful community and support can make a significant impact. Here's how to connect with others who share your interests and get the assistance you need.
### Engage with the Broader Community
- **GitHub Discussions:** The [YOLO26 repository on GitHub](https://github.com/ultralytics/ultralytics) has a "Discussions" section where you can ask questions, report issues, and suggest improvements.
- **Ultralytics Discord Server:** Ultralytics has a [Discord server](https://discord.com/invite/ultralytics) where you can interact with other users and developers.
### Official Documentation and Resources
- **Ultralytics YOLO26 Docs:** The [official documentation](../index.md) provides a comprehensive overview of YOLO26, along with guides on installation, usage, and troubleshooting.
These resources will help you tackle challenges and stay updated on the latest trends and best practices in the YOLO26 community.
## Conclusion
In this guide, we've explored the different deployment options for YOLO26. We've also discussed the important factors to consider when making your choice. These options allow you to customize your model for various environments and performance requirements, making it suitable for real-world applications.
Don't forget that the YOLO26 and [Ultralytics community](https://github.com/orgs/ultralytics/discussions) is a valuable source of help. Connect with other developers and experts to learn unique tips and solutions you might not find in regular documentation. Keep seeking knowledge, exploring new ideas, and sharing your experiences.
## FAQ
### What are the deployment options available for YOLO26 on different hardware platforms?
Ultralytics YOLO26 supports various deployment formats, each designed for specific environments and hardware platforms. Key formats include:
- **PyTorch** for research and prototyping, with excellent Python integration.
- **TorchScript** for production environments where Python is unavailable.
- **ONNX** for cross-platform compatibility and hardware acceleration.
- **OpenVINO** for optimized performance on Intel hardware.
- **TensorRT** for high-speed inference on NVIDIA GPUs.
Each format has unique advantages. For a detailed walkthrough, see our [export process documentation](../modes/export.md#usage-examples).
### How do I improve the inference speed of my YOLO26 model on an Intel CPU?
To enhance inference speed on Intel CPUs, you can deploy your YOLO26 model using Intel's OpenVINO toolkit. OpenVINO offers significant performance boosts by optimizing models to leverage Intel hardware efficiently.
1. Convert your YOLO26 model to the OpenVINO format using the `model.export()` function.
2. Follow the detailed setup guide in the [Intel OpenVINO Export documentation](../integrations/openvino.md).
For more insights, check out our [blog post](https://www.ultralytics.com/blog/achieve-faster-inference-speeds-ultralytics-yolov8-openvino).
### Can I deploy YOLO26 models on mobile devices?
Yes, YOLO26 models can be deployed on mobile devices using [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) Lite (TF Lite) for both Android and iOS platforms. TF Lite is designed for mobile and embedded devices, providing efficient on-device inference.
!!! example
=== "Python"
```python
# Export command for TFLite format
model.export(format="tflite")
```
=== "CLI"
```bash
# CLI command for TFLite export
yolo export --format tflite
```
For more details on deploying models to mobile, refer to our [TF Lite integration guide](../integrations/tflite.md).
### What factors should I consider when choosing a deployment format for my YOLO26 model?
When choosing a deployment format for YOLO26, consider the following factors:
- **Performance**: Some formats like TensorRT provide exceptional speeds on NVIDIA GPUs, while OpenVINO is optimized for Intel hardware.
- **Compatibility**: ONNX offers broad compatibility across different platforms.
- **Ease of Integration**: Formats like CoreML or TF Lite are tailored for specific ecosystems like iOS and Android, respectively.
- **Community Support**: Formats like [PyTorch](https://www.ultralytics.com/glossary/pytorch) and TensorFlow have extensive community resources and support.
For a comparative analysis, refer to our [export formats documentation](../modes/export.md#export-formats).
### How can I deploy YOLO26 models in a web application?
To deploy YOLO26 models in a web application, you can use TensorFlow.js (TF.js), which allows for running [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) models directly in the browser. This approach eliminates the need for backend infrastructure and provides real-time performance.
1. Export the YOLO26 model to the TF.js format.
2. Integrate the exported model into your web application.
For step-by-step instructions, refer to our guide on [TensorFlow.js integration](../integrations/tfjs.md).

206
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-deployment-practices.md Normal file
View File

@@ -0,0 +1,206 @@
---
comments: true
description: Learn essential tips, insights, and best practices for deploying computer vision models with a focus on efficiency, optimization, troubleshooting, and maintaining security.
keywords: Model Deployment, Machine Learning Model Deployment, ML Model Deployment, AI Model Deployment, How to Deploy a Machine Learning Model, How to Deploy ML Models
---
# Best Practices for [Model Deployment](https://www.ultralytics.com/glossary/model-deployment)
## Introduction
Model deployment is the [step in a computer vision project](./steps-of-a-cv-project.md) that brings a model from the development phase into a real-world application. There are various [model deployment options](./model-deployment-options.md): cloud deployment offers scalability and ease of access, edge deployment reduces latency by bringing the model closer to the data source, and local deployment ensures privacy and control. Choosing the right strategy depends on your application's needs, balancing speed, security, and scalability.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Tt_35YnQ9uk"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Optimize and Deploy AI Models: Best Practices, Troubleshooting, and Security Considerations
</p>
It's also important to follow best practices when deploying a model because deployment can significantly impact the effectiveness and reliability of the model's performance. In this guide, we'll focus on how to make sure that your model deployment is smooth, efficient, and secure.
## Model Deployment Options
Oftentimes, once a model is [trained](./model-training-tips.md), [evaluated](./model-evaluation-insights.md), and [tested](./model-testing.md), it needs to be converted into specific formats to be deployed effectively in various environments, such as cloud, edge, or local devices.
With YOLO26, you can [export your model to various formats](../modes/export.md) depending on your deployment needs. For instance, [exporting YOLO26 to ONNX](../integrations/onnx.md) is straightforward and ideal for transferring models between frameworks. To explore more integration options and ensure a smooth deployment across different environments, visit our [model integration hub](../integrations/index.md).
### Choosing a Deployment Environment
Choosing where to deploy your [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) model depends on multiple factors. Different environments have unique benefits and challenges, so it's essential to pick the one that best fits your needs.
#### Cloud Deployment
Cloud deployment is great for applications that need to scale up quickly and handle large amounts of data. Platforms like AWS, [Google Cloud](../yolov5/environments/google_cloud_quickstart_tutorial.md), and Azure make it easy to manage your models from training to deployment. They offer services like [AWS SageMaker](../integrations/amazon-sagemaker.md), Google AI Platform, and [Azure Machine Learning](./azureml-quickstart.md) to help you throughout the process.
However, using the cloud can be expensive, especially with high data usage, and you might face latency issues if your users are far from the data centers. To manage costs and performance, it's important to optimize resource use and ensure compliance with [data privacy](https://www.ultralytics.com/glossary/data-privacy) rules.
#### Edge Deployment
Edge deployment works well for applications needing real-time responses and low latency, particularly in places with limited or no internet access. Deploying models on edge devices like smartphones or IoT gadgets ensures fast processing and keeps data local, which enhances privacy. Deploying on edge also saves bandwidth due to reduced data sent to the cloud.
However, edge devices often have limited processing power, so you'll need to optimize your models. Tools like [TensorFlow Lite](../integrations/tflite.md) and [NVIDIA Jetson](./nvidia-jetson.md) can help. Despite the benefits, maintaining and updating many devices can be challenging.
#### Local Deployment
Local Deployment is best when data privacy is critical or when there's unreliable or no internet access. Running models on local servers or desktops gives you full control and keeps your data secure. It can also reduce latency if the server is near the user.
However, scaling locally can be tough, and maintenance can be time-consuming. Using tools like [Docker](./docker-quickstart.md) for containerization and Kubernetes for management can help make local deployments more efficient. Regular updates and maintenance are necessary to keep everything running smoothly.
## Containerization for Streamlined Deployment
Containerization is a powerful approach that packages your model and all its dependencies into a standardized unit called a container. This technique ensures consistent performance across different environments and simplifies the deployment process.
### Benefits of Using Docker for Model Deployment
[Docker](./docker-quickstart.md) has become the industry standard for containerization in machine learning deployments for several reasons:
- **Environment Consistency**: Docker containers encapsulate your model and all its dependencies, eliminating the "it works on my machine" problem by ensuring consistent behavior across development, testing, and production environments.
- **Isolation**: Containers isolate applications from one another, preventing conflicts between different software versions or libraries.
- **Portability**: Docker containers can run on any system that supports Docker, making it easy to deploy your models across different platforms without modification.
- **Scalability**: Containers can be easily scaled up or down based on demand, and orchestration tools like Kubernetes can automate this process.
- **Version Control**: Docker images can be versioned, allowing you to track changes and roll back to previous versions if needed.
### Implementing Docker for YOLO26 Deployment
To containerize your YOLO26 model, you can create a Dockerfile that specifies all the necessary dependencies and configurations. Here's a basic example:
```dockerfile
FROM ultralytics/ultralytics:latest
WORKDIR /app
# Copy your model and any additional files
COPY ./models/yolo26.pt /app/models/
COPY ./scripts /app/scripts/
# Set up any environment variables
ENV MODEL_PATH=/app/models/yolo26.pt
# Command to run when the container starts
CMD ["python", "/app/scripts/predict.py"]
```
This approach ensures that your model deployment is reproducible and consistent across different environments, significantly reducing the "works on my machine" problem that often plagues deployment processes.
## Model Optimization Techniques
Optimizing your computer vision model helps it runs efficiently, especially when deploying in environments with limited resources like edge devices. Here are some key techniques for optimizing your model.
### Model Pruning
Pruning reduces the size of the model by removing weights that contribute little to the final output. It makes the model smaller and faster without significantly affecting accuracy. Pruning involves identifying and eliminating unnecessary parameters, resulting in a lighter model that requires less computational power. It is particularly useful for deploying models on devices with limited resources.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/model-pruning-overview.avif" alt="Neural network pruning workflow">
</p>
### Model Quantization
Quantization converts the model's weights and activations from high [precision](https://www.ultralytics.com/glossary/precision) (like 32-bit floats) to lower precision (like 8-bit integers). By reducing the model size, it speeds up inference. Quantization-aware training (QAT) is a method where the model is trained with quantization in mind, preserving accuracy better than post-training quantization. By handling quantization during the training phase, the model learns to adjust to lower precision, maintaining performance while reducing computational demands.
<p align="center">
<img width="100%" src="https://miro.medium.com/v2/resize:fit:1032/format:webp/1*Jlq_cyLvRdmp_K5jCd3LkA.png" alt="Model quantization reducing precision for efficiency">
</p>
### Knowledge Distillation
Knowledge distillation involves training a smaller, simpler model (the student) to mimic the outputs of a larger, more complex model (the teacher). The student model learns to approximate the teacher's predictions, resulting in a compact model that retains much of the teacher's [accuracy](https://www.ultralytics.com/glossary/accuracy). This technique is beneficial for creating efficient models suitable for deployment on edge devices with constrained resources.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/knowledge-distillation-overview.avif" alt="Knowledge distillation training process">
</p>
## Troubleshooting Deployment Issues
You may face challenges while deploying your computer vision models, but understanding common problems and solutions can make the process smoother. Here are some general troubleshooting tips and best practices to help you navigate deployment issues.
### Your Model is Less Accurate After Deployment
Experiencing a drop in your model's accuracy after deployment can be frustrating. This issue can stem from various factors. Here are some steps to help you identify and resolve the problem:
- **Check Data Consistency:** Check that the data your model is processing post-deployment is consistent with the data it was trained on. Differences in data distribution, quality, or format can significantly impact performance.
- **Validate Preprocessing Steps:** Verify that all preprocessing steps applied during training are also applied consistently during deployment. This includes resizing images, normalizing pixel values, and other data transformations.
- **Evaluate the Model's Environment:** Ensure that the hardware and software configurations used during deployment match those used during training. Differences in libraries, versions, and hardware capabilities can introduce discrepancies.
- **Monitor Model Inference:** Log inputs and outputs at various stages of the inference pipeline to detect any anomalies. It can help identify issues like data corruption or improper handling of model outputs.
- **Review Model Export and Conversion:** Re-export the model and make sure that the conversion process maintains the integrity of the model weights and architecture.
- **Test with a Controlled Dataset:** Deploy the model in a test environment with a dataset you control and compare the results with the training phase. You can identify if the issue is with the deployment environment or the data.
When deploying YOLO26, several factors can affect model accuracy. Converting models to formats like [TensorRT](../integrations/tensorrt.md) involves optimizations such as weight quantization and layer fusion, which can cause minor precision losses. Using FP16 (half-precision) instead of FP32 (full-precision) can speed up inference but may introduce numerical precision errors. Also, hardware constraints, like those on the [Jetson Nano](./nvidia-jetson.md), with lower CUDA core counts and reduced memory bandwidth, can impact performance.
### Inferences Are Taking Longer Than You Expected
When deploying [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) models, it's important that they run efficiently. If inferences are taking longer than expected, it can affect the user experience and the effectiveness of your application. Here are some steps to help you identify and resolve the problem:
- **Implement Warm-Up Runs**: Initial runs often include setup overhead, which can skew latency measurements. Perform a few warm-up inferences before measuring latency. Excluding these initial runs provides a more accurate measurement of the model's performance.
- **Optimize the Inference Engine:** Double-check that the inference engine is fully optimized for your specific GPU architecture. Use the latest drivers and software versions tailored to your hardware to ensure maximum performance and compatibility.
- **Use Asynchronous Processing:** Asynchronous processing can help manage workloads more efficiently. Use asynchronous processing techniques to handle multiple inferences concurrently, which can help distribute the load and reduce wait times.
- **Profile the Inference Pipeline:** Identifying bottlenecks in the inference pipeline can help pinpoint the source of delays. Use profiling tools to analyze each step of the inference process, identifying and addressing any stages that cause significant delays, such as inefficient layers or data transfer issues.
- **Use Appropriate Precision:** Using higher precision than necessary can slow down inference times. Experiment with using lower precision, such as FP16 (half-precision), instead of FP32 (full-precision). While FP16 can reduce inference time, also keep in mind that it can impact model accuracy.
If you are facing this issue while deploying YOLO26, consider that YOLO26 offers [various model sizes](../models/yolo26.md), such as YOLO26n (nano) for devices with lower memory capacity and YOLO26x (extra-large) for more powerful GPUs. Choosing the right model variant for your hardware can help balance memory usage and processing time.
Also keep in mind that the size of the input images directly impacts memory usage and processing time. Lower resolutions reduce memory usage and speed up inference, while higher resolutions improve accuracy but require more memory and processing power.
## Security Considerations in Model Deployment
Another important aspect of deployment is security. The security of your deployed models is critical to protect sensitive data and intellectual property. Here are some best practices you can follow related to secure model deployment.
### Secure Data Transmission
Making sure data sent between clients and servers is secure is very important to prevent it from being intercepted or accessed by unauthorized parties. You can use encryption protocols like TLS (Transport Layer Security) to encrypt data while it's being transmitted. Even if someone intercepts the data, they won't be able to read it. You can also use end-to-end encryption that protects the data all the way from the source to the destination, so no one in between can access it.
### Access Controls
It's essential to control who can access your model and its data to prevent unauthorized use. Use strong authentication methods to verify the identity of users or systems trying to access the model, and consider adding extra security with multi-factor authentication (MFA). Set up role-based access control (RBAC) to assign permissions based on user roles so that people only have access to what they need. Keep detailed audit logs to track all access and changes to the model and its data, and regularly review these logs to spot any suspicious activity.
### Model Obfuscation
Protecting your model from being reverse-engineered or misuse can be done through model obfuscation. It involves encrypting model parameters, such as weights and biases in [neural networks](https://www.ultralytics.com/glossary/neural-network-nn), to make it difficult for unauthorized individuals to understand or alter the model. You can also obfuscate the model's architecture by renaming layers and parameters or adding dummy layers, making it harder for attackers to reverse-engineer it. You can also serve the model in a secure environment, like a secure enclave or using a trusted execution environment (TEE), can provide an extra layer of protection during inference.
## Share Ideas With Your Peers
Being part of a community of computer vision enthusiasts can help you solve problems and learn faster. Here are some ways to connect, get help, and share ideas.
### Community Resources
- **GitHub Issues:** Explore the [YOLO26 GitHub repository](https://github.com/ultralytics/ultralytics/issues) and use the Issues tab to ask questions, report bugs, and suggest new features. The community and maintainers are very active and ready to help.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to chat with other users and developers, get support, and share your experiences.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Visit the [official YOLO26 documentation](./index.md) for detailed guides and helpful tips on various computer vision projects.
Using these resources will help you solve challenges and stay up-to-date with the latest trends and practices in the computer vision community.
## Conclusion and Next Steps
We walked through some best practices to follow when deploying computer vision models. By securing data, controlling access, and obfuscating model details, you can protect sensitive information while keeping your models running smoothly. We also discussed how to address common issues like reduced accuracy and slow inferences using strategies such as warm-up runs, optimizing engines, asynchronous processing, profiling pipelines, and choosing the right precision.
After deploying your model, the next step would be monitoring, maintaining, and documenting your application. Regular monitoring helps catch and fix issues quickly, maintenance keeps your models up-to-date and functional, and good documentation tracks all changes and updates. These steps will help you achieve the [goals of your computer vision project](./defining-project-goals.md).
## FAQ
### What are the best practices for deploying a machine learning model using Ultralytics YOLO26?
Deploying a machine learning model, particularly with Ultralytics YOLO26, involves several best practices to ensure efficiency and reliability. First, choose the deployment environment that suits your needs—cloud, edge, or local. Optimize your model through techniques like [pruning, quantization, and knowledge distillation](#model-optimization-techniques) for efficient deployment in resource-constrained environments. Consider using [containerization with Docker](#containerization-for-streamlined-deployment) to ensure consistency across different environments. Lastly, ensure data consistency and preprocessing steps align with the training phase to maintain performance. You can also refer to [model deployment options](./model-deployment-options.md) for more detailed guidelines.
### How can I troubleshoot common deployment issues with Ultralytics YOLO26 models?
Troubleshooting deployment issues can be broken down into a few key steps. If your model's accuracy drops after deployment, check for data consistency, validate preprocessing steps, and ensure the hardware/software environment matches what you used during training. For slow inference times, perform warm-up runs, optimize your inference engine, use asynchronous processing, and profile your inference pipeline. Refer to [troubleshooting deployment issues](#troubleshooting-deployment-issues) for a detailed guide on these best practices.
### How does Ultralytics YOLO26 optimization enhance model performance on edge devices?
Optimizing Ultralytics YOLO26 models for edge devices involves using techniques like pruning to reduce the model size, quantization to convert weights to lower precision, and knowledge distillation to train smaller models that mimic larger ones. These techniques ensure the model runs efficiently on devices with limited computational power. Tools like [TensorFlow Lite](../integrations/tflite.md) and [NVIDIA Jetson](./nvidia-jetson.md) are particularly useful for these optimizations. Learn more about these techniques in our section on [model optimization](#model-optimization-techniques).
### What are the security considerations for deploying machine learning models with Ultralytics YOLO26?
Security is paramount when deploying machine learning models. Ensure secure data transmission using encryption protocols like TLS. Implement robust access controls, including strong authentication and role-based access control (RBAC). Model obfuscation techniques, such as encrypting model parameters and serving models in a secure environment like a trusted execution environment (TEE), offer additional protection. For detailed practices, refer to [security considerations](#security-considerations-in-model-deployment).
### How do I choose the right deployment environment for my Ultralytics YOLO26 model?
Selecting the optimal deployment environment for your Ultralytics YOLO26 model depends on your application's specific needs. Cloud deployment offers scalability and ease of access, making it ideal for applications with high data volumes. Edge deployment is best for low-latency applications requiring real-time responses, using tools like [TensorFlow Lite](../integrations/tflite.md). Local deployment suits scenarios needing stringent data privacy and control. For a comprehensive overview of each environment, check out our section on [choosing a deployment environment](#choosing-a-deployment-environment).

199
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-evaluation-insights.md Normal file
View File

@@ -0,0 +1,199 @@
---
comments: true
description: Explore the most effective ways to assess and refine YOLO26 models for better performance. Learn about evaluation metrics, fine-tuning processes, and how to customize your model for specific needs.
keywords: Model Evaluation, Machine Learning Model Evaluation, Fine Tuning Machine Learning, Fine Tune Model, Evaluating Models, Model Fine Tuning, How to Fine Tune a Model
---
# Insights on Model Evaluation and Fine-Tuning
## Introduction
Once you've [trained](./model-training-tips.md) your computer vision model, evaluating and refining it to perform optimally is essential. Just training your model isn't enough. You need to make sure that your model is accurate, efficient, and fulfills the [objective](./defining-project-goals.md) of your computer vision project. By evaluating and fine-tuning your model, you can identify weaknesses, improve its accuracy, and boost overall performance.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/-aYO-6VaDrw"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Insights into Model Evaluation and Fine-Tuning | Tips for Improving Mean Average Precision
</p>
In this guide, we share insights on model evaluation and fine-tuning to make this [step of a computer vision project](./steps-of-a-cv-project.md) more approachable. We discuss how to understand evaluation metrics and implement fine-tuning techniques, giving you the knowledge to elevate your model's capabilities.
## Evaluating Model Performance Using Metrics
Evaluating how well a model performs helps us understand how effectively it works. Various metrics are used to measure performance. These [performance metrics](./yolo-performance-metrics.md) provide clear, numerical insights that can guide improvements toward making sure the model meets its intended goals. Let's take a closer look at a few key metrics.
### Confidence Score
The confidence score represents the model's certainty that a detected object belongs to a particular class. It ranges from 0 to 1, with higher scores indicating greater confidence. The confidence score helps filter predictions; only detections with confidence scores above a specified threshold are considered valid.
_Quick Tip:_ When running inferences, if you aren't seeing any predictions, and you've checked everything else, try lowering the confidence score. Sometimes, the threshold is too high, causing the model to ignore valid predictions. Lowering the score allows the model to consider more possibilities. This might not meet your project goals, but it's a good way to see what the model can do and decide how to fine-tune it.
### Intersection over Union
[Intersection over Union](https://www.ultralytics.com/glossary/intersection-over-union-iou) (IoU) is a metric in [object detection](https://www.ultralytics.com/glossary/object-detection) that measures how well the predicted [bounding box](https://www.ultralytics.com/glossary/bounding-box) overlaps with the ground truth bounding box. IoU values range from 0 to 1, where one stands for a perfect match. IoU is essential because it measures how closely the predicted boundaries match the actual object boundaries.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/intersection-over-union-overview.avif" alt="Intersection over Union Overview">
</p>
### Mean Average Precision
[Mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP) is a way to measure how well an object detection model performs. It looks at the precision of detecting each object class, averages these scores, and gives an overall number that shows how accurately the model can identify and classify objects.
Let's focus on two specific mAP metrics:
- *mAP@.5:* Measures the average precision at a single IoU (Intersection over Union) threshold of 0.5. This metric checks if the model can correctly find objects with a looser [accuracy](https://www.ultralytics.com/glossary/accuracy) requirement. It focuses on whether the object is roughly in the right place, not needing perfect placement. It helps see if the model is generally good at spotting objects.
- *mAP@.5:.95:* Averages the mAP values calculated at multiple IoU thresholds, from 0.5 to 0.95 in 0.05 increments. This metric is more detailed and strict. It gives a fuller picture of how accurately the model can find objects at different levels of strictness and is especially useful for applications that need precise object detection.
Other mAP metrics include mAP@0.75, which uses a stricter IoU threshold of 0.75, and mAP@small, medium, and large, which evaluate precision across objects of different sizes.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/mean-average-precision-overview.avif" alt="Mean average precision mAP metric">
</p>
## Evaluating YOLO26 Model Performance
With respect to YOLO26, you can use the [validation mode](../modes/val.md) to evaluate the model. Also, be sure to take a look at our guide that goes in-depth into [YOLO26 performance metrics](./yolo-performance-metrics.md) and how they can be interpreted.
### Common Community Questions
When evaluating your YOLO26 model, you might run into a few hiccups. Based on common community questions, here are some tips to help you get the most out of your YOLO26 model:
#### Handling Variable Image Sizes
Evaluating your YOLO26 model with images of different sizes can help you understand its performance on diverse datasets. Using the `rect=true` validation parameter, YOLO26 adjusts the network's stride for each batch based on the image sizes, allowing the model to handle rectangular images without forcing them to a single size.
The `imgsz` validation parameter sets the maximum dimension for image resizing, which is 640 by default. You can adjust this based on your dataset's maximum dimensions and the GPU memory available. Even with `imgsz` set, `rect=true` lets the model manage varying image sizes effectively by dynamically adjusting the stride.
#### Accessing YOLO26 Metrics
If you want to get a deeper understanding of your YOLO26 model's performance, you can easily access specific evaluation metrics with a few lines of Python code. The code snippet below will let you load your model, run an evaluation, and print out various metrics that show how well your model is doing.
!!! example "Usage"
=== "Python"
```python
from ultralytics import YOLO
# Load the model
model = YOLO("yolo26n.pt")
# Run the evaluation
results = model.val(data="coco8.yaml")
# Print specific metrics
print("Class indices with average precision:", results.ap_class_index)
print("Average precision for all classes:", results.box.all_ap)
print("Average precision:", results.box.ap)
print("Average precision at IoU=0.50:", results.box.ap50)
print("Class indices for average precision:", results.box.ap_class_index)
print("Class-specific results:", results.box.class_result)
print("F1 score:", results.box.f1)
print("F1 score curve:", results.box.f1_curve)
print("Overall fitness score:", results.box.fitness)
print("Mean average precision:", results.box.map)
print("Mean average precision at IoU=0.50:", results.box.map50)
print("Mean average precision at IoU=0.75:", results.box.map75)
print("Mean average precision for different IoU thresholds:", results.box.maps)
print("Mean results for different metrics:", results.box.mean_results)
print("Mean precision:", results.box.mp)
print("Mean recall:", results.box.mr)
print("Precision:", results.box.p)
print("Precision curve:", results.box.p_curve)
print("Precision values:", results.box.prec_values)
print("Specific precision metrics:", results.box.px)
print("Recall:", results.box.r)
print("Recall curve:", results.box.r_curve)
```
The results object also includes speed metrics like preprocess time, inference time, loss, and postprocess time. By analyzing these metrics, you can fine-tune and optimize your YOLO26 model for better performance, making it more effective for your specific use case.
## How Does Fine-Tuning Work?
Fine-tuning involves taking a pretrained model and adjusting its parameters to improve performance on a specific task or dataset. The process, also known as model retraining, allows the model to better understand and predict outcomes for the specific data it will encounter in real-world applications. You can retrain your model based on your model evaluation to achieve optimal results.
## Tips for Fine-Tuning Your Model
Fine-tuning a model means paying close attention to several vital parameters and techniques to achieve optimal performance. Here are some essential tips to guide you through the process.
### Starting With a Higher Learning Rate
Usually, during the initial training [epochs](https://www.ultralytics.com/glossary/epoch), the learning rate starts low and gradually increases to stabilize the training process. However, since your model has already learned some features from the previous dataset, starting with a higher [learning rate](https://www.ultralytics.com/glossary/learning-rate) right away can be more beneficial.
When evaluating your YOLO26 model, you can set the `warmup_epochs` validation parameter to `warmup_epochs=0` to prevent the learning rate from starting too low. By following this process, the training will continue from the provided weights, adjusting to the nuances of your new data.
### Image Tiling for Small Objects
Image tiling can improve detection accuracy for small objects. By dividing larger images into smaller segments, such as splitting 1280x1280 images into multiple 640x640 segments, you maintain the original resolution, and the model can learn from high-resolution fragments. When using YOLO26, make sure to adjust your labels for these new segments correctly.
## Engage with the Community
Sharing your ideas and questions with other [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) enthusiasts can inspire creative solutions to roadblocks in your projects. Here are some excellent ways to learn, troubleshoot, and connect.
### Finding Help and Support
- **GitHub Issues:** Explore the YOLO26 GitHub repository and use the [Issues tab](https://github.com/ultralytics/ultralytics/issues) to ask questions, report bugs, and suggest features. The community and maintainers are available to assist with any issues you encounter.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to connect with other users and developers, get support, share knowledge, and brainstorm ideas.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Check out the [official YOLO26 documentation](./index.md) for comprehensive guides and valuable insights on various computer vision tasks and projects.
## Final Thoughts
Evaluating and fine-tuning your computer vision model are important steps for successful [model deployment](https://www.ultralytics.com/glossary/model-deployment). These steps help make sure that your model is accurate, efficient, and suited to your overall application. The key to training the best model possible is continuous experimentation and learning. Don't hesitate to tweak parameters, try new techniques, and explore different datasets. Keep experimenting and pushing the boundaries of what's possible!
## FAQ
### What are the key metrics for evaluating YOLO26 model performance?
To evaluate YOLO26 model performance, important metrics include Confidence Score, Intersection over Union (IoU), and Mean Average Precision (mAP). Confidence Score measures the model's certainty for each detected object class. IoU evaluates how well the predicted bounding box overlaps with the ground truth. Mean Average Precision (mAP) aggregates precision scores across classes, with mAP@.5 and mAP@.5:.95 being two common types for varying IoU thresholds. Learn more about these metrics in our [YOLO26 performance metrics guide](./yolo-performance-metrics.md).
### How can I fine-tune a pretrained YOLO26 model for my specific dataset?
Fine-tuning a pretrained YOLO26 model involves adjusting its parameters to improve performance on a specific task or dataset. Start by evaluating your model using metrics, then set a higher initial learning rate by adjusting the `warmup_epochs` parameter to 0 for immediate stability. Use parameters like `rect=true` for handling varied image sizes effectively. For more detailed guidance, refer to our section on [fine-tuning YOLO26 models](#how-does-fine-tuning-work).
### How can I handle variable image sizes when evaluating my YOLO26 model?
To handle variable image sizes during evaluation, use the `rect=true` parameter in YOLO26, which adjusts the network's stride for each batch based on image sizes. The `imgsz` parameter sets the maximum dimension for image resizing, defaulting to 640. Adjust `imgsz` to suit your dataset and GPU memory. For more details, visit our [section on handling variable image sizes](#handling-variable-image-sizes).
### What practical steps can I take to improve mean average precision for my YOLO26 model?
Improving mean average precision (mAP) for a YOLO26 model involves several steps:
1. **Tuning Hyperparameters**: Experiment with different learning rates, [batch sizes](https://www.ultralytics.com/glossary/batch-size), and image augmentations.
2. **[Data Augmentation](https://www.ultralytics.com/glossary/data-augmentation)**: Use techniques like Mosaic and MixUp to create diverse training samples.
3. **Image Tiling**: Split larger images into smaller tiles to improve detection accuracy for small objects.
Refer to our detailed guide on [model fine-tuning](#tips-for-fine-tuning-your-model) for specific strategies.
### How do I access YOLO26 model evaluation metrics in Python?
You can access YOLO26 model evaluation metrics using Python with the following steps:
!!! example "Usage"
=== "Python"
```python
from ultralytics import YOLO
# Load the model
model = YOLO("yolo26n.pt")
# Run the evaluation
results = model.val(data="coco8.yaml")
# Print specific metrics
print("Class indices with average precision:", results.ap_class_index)
print("Average precision for all classes:", results.box.all_ap)
print("Mean average precision at IoU=0.50:", results.box.map50)
print("Mean recall:", results.box.mr)
```
Analyzing these metrics helps fine-tune and optimize your YOLO26 model. For a deeper dive, check out our guide on [YOLO26 metrics](../modes/val.md).

183
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-monitoring-and-maintenance.md Normal file
View File

@@ -0,0 +1,183 @@
---
comments: true
description: Understand the key practices for monitoring, maintaining, and documenting computer vision models to guarantee accuracy, spot anomalies, and mitigate data drift.
keywords: Computer Vision Models, AI Model Monitoring, Data Drift Detection, Anomaly Detection in AI, Model Maintenance
---
# Maintaining Your Computer Vision Models After Deployment
## Introduction
If you are here, we can assume you've completed many [steps in your computer vision project](./steps-of-a-cv-project.md): from [gathering requirements](./defining-project-goals.md), [annotating data](./data-collection-and-annotation.md), and [training the model](./model-training-tips.md) to finally [deploying](./model-deployment-practices.md) it. Your application is now running in production, but your project doesn't end here. The most important part of a computer vision project is making sure your model continues to fulfill your [project's objectives](./defining-project-goals.md) over time, and that's where monitoring, maintaining, and documenting your computer vision model enters the picture.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/zCupPHqSLTI"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Maintain Computer Vision Models after Deployment | Data Drift Detection
</p>
In this guide, we'll take a closer look at how you can maintain your computer vision models after deployment. We'll explore how model monitoring can help you catch problems early on, how to keep your model accurate and up-to-date, and why documentation is important for troubleshooting.
## Model Monitoring is Key
Keeping a close eye on your deployed computer vision models is essential. Without proper monitoring, models can lose accuracy. A common issue is data distribution shift or [data drift](https://www.ultralytics.com/glossary/data-drift), where the data the model encounters changes from what it was trained on. When the model has to make predictions on data it doesn't recognize, it can lead to misinterpretations and poor performance. Outliers, or unusual data points, can also throw off the model's accuracy.
Regular model monitoring helps developers track the [model's performance](./model-evaluation-insights.md), spot anomalies, and quickly address problems like data drift. It also helps manage resources by indicating when updates are needed, avoiding expensive overhauls, and keeping the model relevant.
### Best Practices for Model Monitoring
Here are some best practices to keep in mind while monitoring your computer vision model in production:
- **Track Performance Regularly**: Continuously monitor the model's performance to detect changes over time.
- **Double-Check the Data Quality**: Check for missing values or anomalies in the data.
- **Use Diverse Data Sources**: Monitor data from various sources to get a comprehensive view of the model's performance.
- **Combine Monitoring Techniques**: Use a mix of drift detection algorithms and rule-based approaches to identify a wide range of issues.
- **Monitor Inputs and Outputs**: Keep an eye on both the data the model processes and the results it produces to make sure everything is functioning correctly.
- **Set Up Alerts**: Implement alerts for unusual behavior, such as performance drops, to be able to make quick corrective actions.
### Tools for AI Model Monitoring
You can use automated monitoring tools to make it easier to monitor models after deployment. Many tools offer real-time insights and alerting capabilities. Here are some examples of open-source model monitoring tools that can work together:
- **[Prometheus](https://prometheus.io/)**: Prometheus is an open-source monitoring tool that collects and stores metrics for detailed performance tracking. It integrates easily with Kubernetes and Docker, collecting data at set intervals and storing it in a time-series database. Prometheus can also scrape HTTP endpoints to gather real-time metrics. Collected data can be queried using the PromQL language.
- **[Grafana](https://grafana.com/)**: Grafana is an open-source [data visualization](https://www.ultralytics.com/glossary/data-visualization) and monitoring tool that allows you to query, visualize, alert on, and understand your metrics no matter where they are stored. It works well with Prometheus and offers advanced data visualization features. You can create custom dashboards to show important metrics for your computer vision models, like inference latency, error rates, and resource usage. Grafana turns collected data into easy-to-read dashboards with line graphs, heat maps, and histograms. It also supports alerts, which can be sent through channels like Slack to quickly notify teams of any issues.
- **[Evidently AI](https://www.evidentlyai.com/)**: Evidently AI is an open-source tool designed for monitoring and debugging [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) models in production. It generates interactive reports from pandas DataFrames, helping analyze machine learning models. Evidently AI can detect data drift, model performance degradation, and other issues that may arise with your deployed models.
The three tools introduced above, Evidently AI, Prometheus, and Grafana, can work together seamlessly as a fully open-source ML monitoring solution that is ready for production. Evidently AI is used to collect and calculate metrics, Prometheus stores these metrics, and Grafana displays them and sets up alerts. While there are many other tools available, this setup is an exciting open-source option that provides robust capabilities for [model monitoring](https://www.ultralytics.com/glossary/model-monitoring) and maintaining your models.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/evidently-prometheus-grafana-monitoring-tools.avif" alt="Overview of Open Source Model Monitoring Tools">
</p>
### Anomaly Detection and Alert Systems
An anomaly is any data point or pattern that deviates quite a bit from what is expected. With respect to [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) models, anomalies can be images that are very different from the ones the model was trained on. These unexpected images can be signs of issues like changes in data distribution, outliers, or behaviors that might reduce model performance. Setting up alert systems to detect these anomalies is an important part of model monitoring.
By setting standard performance levels and limits for key metrics, you can catch problems early. When performance goes outside these limits, alerts are triggered, prompting quick fixes. Regularly updating and retraining models with new data keeps them relevant and accurate as the data changes.
#### Things to Keep in Mind When Configuring Thresholds and Alerts
When you are setting up your alert systems, keep these best practices in mind:
- **Standardized Alerts**: Use consistent tools and formats for all alerts, such as email or messaging apps like Slack. Standardization makes it easier for you to quickly understand and respond to alerts.
- **Include Expected Behavior**: Alert messages should clearly state what went wrong, what was expected, and the timeframe evaluated. It helps you gauge the urgency and context of the alert.
- **Configurable Alerts**: Make alerts easily configurable to adapt to changing conditions. Allow yourself to edit thresholds, snooze, disable, or acknowledge alerts.
### Data Drift Detection
Data drift detection is a concept that helps identify when the statistical properties of the input data change over time, which can degrade model performance. Before you decide to retrain or adjust your models, this technique helps spot that there is an issue. Data drift deals with changes in the overall data landscape over time, while [anomaly detection](https://www.ultralytics.com/glossary/anomaly-detection) focuses on identifying rare or unexpected data points that may require immediate attention.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/data-drift-detection-overview.avif" alt="Data drift detection monitoring pipeline">
</p>
Here are several methods to detect data drift:
**Continuous Monitoring**: Regularly monitor the model's input data and outputs for signs of drift. Track key metrics and compare them against historical data to identify significant changes.
**Statistical Techniques**: Use methods like the Kolmogorov-Smirnov test or Population Stability Index (PSI) to detect changes in data distributions. These tests compare the distribution of new data with the [training data](https://www.ultralytics.com/glossary/training-data) to identify significant differences.
**Feature Drift**: Monitor individual features for drift. Sometimes, the overall data distribution may remain stable, but individual features may drift. Identifying which features are drifting helps in fine-tuning the retraining process.
## Model Maintenance
Model maintenance is crucial to keep computer vision models accurate and relevant over time. Model maintenance involves regularly updating and retraining models, addressing data drift, and ensuring the model stays relevant as data and environments change. You might be wondering how model maintenance differs from model monitoring. Monitoring is about watching the model's performance in real time to catch issues early. Maintenance, on the other hand, is about fixing these issues.
### Regular Updates and Re-training
Once a model is deployed, while monitoring, you may notice changes in data patterns or performance, indicating model drift. Regular updates and re-training become essential parts of model maintenance to ensure the model can handle new patterns and scenarios. There are a few techniques you can use based on how your data is changing.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/computer-vision-model-drift-overview.avif" alt="Computer vision model drift causes">
</p>
For example, if the data is changing gradually over time, incremental learning is a good approach. Incremental learning involves updating the model with new data without completely retraining it from scratch, saving computational resources and time. However, if the data has changed drastically, a periodic full re-training might be a better option to ensure the model does not [overfit](https://www.ultralytics.com/glossary/overfitting) on the new data while losing track of older patterns.
Regardless of the method, validation and testing are a must after updates. It is important to validate the model on a separate [test dataset](./model-testing.md) to check for performance improvements or degradation.
### Deciding When to Retrain Your Model
The frequency of retraining your computer vision model depends on data changes and model performance. Retrain your model whenever you observe a significant performance drop or detect data drift. Regular evaluations can help determine the right retraining schedule by testing the model against new data. Monitoring performance metrics and data patterns lets you decide if your model needs more frequent updates to maintain [accuracy](https://www.ultralytics.com/glossary/accuracy).
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/when-to-retrain-overview.avif" alt="When to retrain ML models flowchart">
</p>
## Documentation
Documenting a computer vision project makes it easier to understand, reproduce, and collaborate on. Good documentation covers model architecture, [hyperparameters](https://www.ultralytics.com/glossary/hyperparameter-tuning), datasets, evaluation metrics, and more. It provides transparency, helping team members and stakeholders understand what has been done and why. Documentation also aids in troubleshooting, maintenance, and future enhancements by providing a clear reference of past decisions and methods.
### Key Elements to Document
These are some of the key elements that should be included in project documentation:
- **[Project Overview](./steps-of-a-cv-project.md)**: Provide a high-level summary of the project, including the problem statement, solution approach, expected outcomes, and project scope. Explain the role of computer vision in addressing the problem and outline the stages and deliverables.
- **Model Architecture**: Detail the structure and design of the model, including its components, layers, and connections. Explain the chosen hyperparameters and the rationale behind these choices.
- **[Data Preparation](./data-collection-and-annotation.md)**: Describe the data sources, types, formats, sizes, and preprocessing steps. Discuss data quality, reliability, and any transformations applied before training the model.
- **[Training Process](./model-training-tips.md)**: Document the training procedure, including the datasets used, training parameters, and [loss functions](https://www.ultralytics.com/glossary/loss-function). Explain how the model was trained and any challenges encountered during training.
- **[Evaluation Metrics](./model-evaluation-insights.md)**: Specify the metrics used to evaluate the model's performance, such as accuracy, [precision](https://www.ultralytics.com/glossary/precision), [recall](https://www.ultralytics.com/glossary/recall), and [F1-score](https://www.ultralytics.com/glossary/f1-score). Include performance results and an analysis of these metrics.
- **[Deployment Steps](./model-deployment-options.md)**: Outline the steps taken to deploy the model, including the tools and platforms used, deployment configurations, and any specific challenges or considerations.
- **Monitoring and Maintenance Procedure**: Provide a detailed plan for monitoring the model's performance post-deployment. Include methods for detecting and addressing data and model drift, and describe the process for regular updates and retraining.
### Tools for Documentation
There are many options when it comes to documenting AI projects, with open-source tools being particularly popular. Two of these are [Jupyter Notebooks](https://docs.ultralytics.com/integrations/jupyterlab/) and MkDocs. Jupyter Notebooks allow you to create interactive documents with embedded code, visualizations, and text, making them ideal for sharing experiments and analyses. MkDocs is a static site generator that is easy to set up and deploy and is perfect for creating and hosting project documentation online.
## Connect with the Community
Joining a community of computer vision enthusiasts can help you solve problems and learn more quickly. Here are some ways to connect, get support, and share ideas.
### Community Resources
- **GitHub Issues:** Check out the [YOLO26 GitHub repository](https://github.com/ultralytics/ultralytics/issues) and use the Issues tab to ask questions, report bugs, and suggest new features. The community and maintainers are highly active and supportive.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to chat with other users and developers, get support, and share your experiences.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Visit the [official YOLO26 documentation](./index.md) for detailed guides and helpful tips on various computer vision projects.
Using these resources will help you solve challenges and stay up-to-date with the latest trends and practices in the computer vision community.
## Key Takeaways
We covered key tips for monitoring, maintaining, and documenting your computer vision models. Regular updates and re-training help the model adapt to new data patterns. Detecting and fixing data drift helps your model stay accurate. Continuous monitoring catches issues early, and good documentation makes collaboration and future updates easier. Following these steps will help your computer vision project stay successful and effective over time.
## FAQ
### How do I monitor the performance of my deployed computer vision model?
Monitoring the performance of your deployed computer vision model is crucial to ensure its accuracy and reliability over time. You can use tools like [Prometheus](https://prometheus.io/), [Grafana](https://grafana.com/), and [Evidently AI](https://www.evidentlyai.com/) to track key metrics, detect anomalies, and identify data drift. Regularly monitor inputs and outputs, set up alerts for unusual behavior, and use diverse data sources to get a comprehensive view of your model's performance. For more details, check out our section on [Model Monitoring](#model-monitoring-is-key).
### What are the best practices for maintaining computer vision models after deployment?
Maintaining computer vision models involves regular updates, retraining, and monitoring to ensure continued accuracy and relevance. Best practices include:
- **Continuous Monitoring**: Track performance metrics and data quality regularly.
- **Data Drift Detection**: Use statistical techniques to identify changes in data distributions.
- **Regular Updates and Retraining**: Implement incremental learning or periodic full retraining based on data changes.
- **Documentation**: Maintain detailed documentation of model architecture, training processes, and evaluation metrics. For more insights, visit our [Model Maintenance](#model-maintenance) section.
### Why is data drift detection important for AI models?
Data drift detection is essential because it helps identify when the statistical properties of the input data change over time, which can degrade model performance. Techniques like continuous monitoring, statistical tests (e.g., Kolmogorov-Smirnov test), and feature drift analysis can help spot issues early. Addressing data drift ensures that your model remains accurate and relevant in changing environments. Learn more about data drift detection in our [Data Drift Detection](#data-drift-detection) section.
### What tools can I use for anomaly detection in computer vision models?
For anomaly detection in computer vision models, tools like [Prometheus](https://prometheus.io/), [Grafana](https://grafana.com/), and [Evidently AI](https://www.evidentlyai.com/) are highly effective. These tools can help you set up alert systems to detect unusual data points or patterns that deviate from expected behavior. Configurable alerts and standardized messages can help you respond quickly to potential issues. Explore more in our [Anomaly Detection and Alert Systems](#anomaly-detection-and-alert-systems) section.
### How can I document my computer vision project effectively?
Effective documentation of a computer vision project should include:
- **Project Overview**: High-level summary, problem statement, and solution approach.
- **Model Architecture**: Details of the model structure, components, and hyperparameters.
- **Data Preparation**: Information on data sources, preprocessing steps, and transformations.
- **Training Process**: Description of the training procedure, datasets used, and challenges encountered.
- **Evaluation Metrics**: Metrics used for performance evaluation and analysis.
- **Deployment Steps**: Steps taken for [model deployment](https://www.ultralytics.com/glossary/model-deployment) and any specific challenges.
- **Monitoring and Maintenance Procedure**: Plan for ongoing monitoring and maintenance. For more comprehensive guidelines, refer to our [Documentation](#documentation) section.

211
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-testing.md Normal file
View File

@@ -0,0 +1,211 @@
---
comments: true
description: Explore effective methods for testing computer vision models to make sure they are reliable, perform well, and are ready to be deployed.
keywords: Overfitting and Underfitting in Machine Learning, Model Testing, Data Leakage Machine Learning, Testing a Model, Testing Machine Learning Models, How to Test AI Models
---
# A Guide on Model Testing
## Introduction
After [training](./model-training-tips.md) and [evaluating](./model-evaluation-insights.md) your model, it's time to test it. Model testing involves assessing how well it performs in real-world scenarios. Testing considers factors like accuracy, reliability, fairness, and how easy it is to understand the model's decisions. The goal is to make sure the model performs as intended, delivers the expected results, and fits into the [overall objective of your application](./defining-project-goals.md) or project.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/SyyCUvxw9BM"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Test Machine Learning Models | Avoid Data Leakage in Computer Vision 🚀
</p>
Model testing is quite similar to model evaluation, but they are two distinct [steps in a computer vision project](./steps-of-a-cv-project.md). Model evaluation involves metrics and plots to assess the model's accuracy. On the other hand, model testing checks if the model's learned behavior is the same as expectations. In this guide, we'll explore strategies for testing your [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) models.
## Model Testing Vs. Model Evaluation
First, let's understand the difference between model evaluation and testing with an example.
Suppose you have trained a computer vision model to recognize cats and dogs, and you want to deploy this model at a pet store to monitor the animals. During the model evaluation phase, you use a labeled dataset to calculate metrics like accuracy, [precision](https://www.ultralytics.com/glossary/precision), [recall](https://www.ultralytics.com/glossary/recall), and F1 score. For instance, the model might have an accuracy of 98% in distinguishing between cats and dogs in a given dataset.
After evaluation, you test the model using images from a pet store to see how well it identifies cats and dogs in more varied and realistic conditions. You check if it can correctly label cats and dogs when they are moving, in different lighting conditions, or partially obscured by objects like toys or furniture. Model testing checks that the model behaves as expected outside the controlled evaluation environment.
## Preparing for Model Testing
Computer vision models learn from datasets by detecting patterns, making predictions, and evaluating their performance. These [datasets](./preprocessing_annotated_data.md) are usually divided into training and testing sets to simulate real-world conditions. [Training data](https://www.ultralytics.com/glossary/training-data) teaches the model while testing data verifies its accuracy.
Here are two points to keep in mind before testing your model:
- **Realistic Representation:** The previously unseen testing data should be similar to the data that the model will have to handle when deployed. This helps get a realistic understanding of the model's capabilities.
- **Sufficient Size:** The size of the testing dataset needs to be large enough to provide reliable insights into how well the model performs.
## Testing Your Computer Vision Model
Here are the key steps to take to test your computer vision model and understand its performance.
- **Run Predictions:** Use the model to make predictions on the test dataset.
- **Compare Predictions:** Check how well the model's predictions match the actual labels (ground truth).
- **Calculate Performance Metrics:** [Compute metrics](./yolo-performance-metrics.md) like accuracy, precision, recall, and F1 score to understand the model's strengths and weaknesses. Testing focuses on how these metrics reflect real-world performance.
- **Visualize Results:** Create visual aids like confusion matrices and ROC curves. These help you spot specific areas where the model might not be performing well in practical applications.
Next, the testing results can be analyzed:
- **Misclassified Images:** Identify and review images that the model misclassified to understand where it is going wrong.
- **Error Analysis:** Perform a thorough error analysis to understand the types of errors (e.g., false positives vs. false negatives) and their potential causes.
- **Bias and Fairness:** Check for any biases in the model's predictions. Ensure that the model performs equally well across different subsets of the data, especially if it includes sensitive attributes like race, gender, or age.
## Testing Your YOLO26 Model
To test your YOLO26 model, you can use the validation mode. It's a straightforward way to understand the model's strengths and areas that need improvement. Also, you'll need to format your test dataset correctly for YOLO26. For more details on how to use the validation mode, check out the [Model Validation](../modes/val.md) docs page.
## Using YOLO26 to Predict on Multiple Test Images
If you want to test your trained YOLO26 model on multiple images stored in a folder, you can easily do so in one go. Instead of using the validation mode, which is typically used to evaluate model performance on a validation set and provide detailed metrics, you might just want to see predictions on all images in your test set. For this, you can use the [prediction mode](../modes/predict.md).
### Difference Between Validation and Prediction Modes
- **[Validation Mode](../modes/val.md):** Used to evaluate the model's performance by comparing predictions against known labels (ground truth). It provides detailed metrics such as accuracy, precision, recall, and F1 score.
- **[Prediction Mode](../modes/predict.md):** Used to run the model on new, unseen data to generate predictions. It does not provide detailed performance metrics but allows you to see how the model performs on real-world images.
## Running YOLO26 Predictions Without Custom Training
If you are interested in testing the basic YOLO26 model to understand whether it can be used for your application without custom training, you can use the prediction mode. While the model is pretrained on datasets like COCO, running predictions on your own dataset can give you a quick sense of how well it might perform in your specific context.
## Overfitting and [Underfitting](https://www.ultralytics.com/glossary/underfitting) in [Machine Learning](https://www.ultralytics.com/glossary/machine-learning-ml)
When testing a machine learning model, especially in computer vision, it's important to watch out for overfitting and underfitting. These issues can significantly affect how well your model works with new data.
### Overfitting
Overfitting happens when your model learns the training data too well, including the noise and details that don't generalize to new data. In computer vision, this means your model might do great with training images but struggle with new ones.
#### Signs of Overfitting
- **High Training Accuracy, Low Validation Accuracy:** If your model performs very well on training data but poorly on validation or [test data](https://www.ultralytics.com/glossary/test-data), it's likely overfitting.
- **Visual Inspection:** Sometimes, you can see overfitting if your model is too sensitive to minor changes or irrelevant details in images.
### Underfitting
Underfitting occurs when your model can't capture the underlying patterns in the data. In computer vision, an underfitted model might not even recognize objects correctly in the training images.
#### Signs of Underfitting
- **Low Training Accuracy:** If your model can't achieve high accuracy on the training set, it might be underfitting.
- **Visual Mis-classification:** Consistent failure to recognize obvious features or objects suggests underfitting.
### Balancing Overfitting and Underfitting
The key is to find a balance between overfitting and underfitting. Ideally, a model should perform well on both training and validation datasets. Regularly monitoring your model's performance through metrics and visual inspections, along with applying the right strategies, can help you achieve the best results.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/overfitting-underfitting-appropriate-fitting.avif" alt="Overfitting vs underfitting visualization">
</p>
## Data Leakage in Computer Vision and How to Avoid It
While testing your model, something important to keep in mind is data leakage. Data leakage happens when information from outside the training dataset accidentally gets used to train the model. The model may seem very accurate during training, but it won't perform well on new, unseen data when data leakage occurs.
### Why Data Leakage Happens
Data leakage can be tricky to spot and often comes from hidden biases in the training data. Here are some common ways it can happen in computer vision:
- **Camera Bias:** Different angles, lighting, shadows, and camera movements can introduce unwanted patterns.
- **Overlay Bias:** Logos, timestamps, or other overlays in images can mislead the model.
- **Font and Object Bias:** Specific fonts or objects that frequently appear in certain classes can skew the model's learning.
- **Spatial Bias:** Imbalances in foreground-background, [bounding box](https://www.ultralytics.com/glossary/bounding-box) distributions, and object locations can affect training.
- **Label and Domain Bias:** Incorrect labels or shifts in data types can lead to leakage.
### Detecting Data Leakage
To find data leakage, you can:
- **Check Performance:** If the model's results are surprisingly good, it might be leaking.
- **Look at Feature Importance:** If one feature is much more important than others, it could indicate leakage.
- **Visual Inspection:** Double-check that the model's decisions make sense intuitively.
- **Verify Data Separation:** Make sure data was divided correctly before any processing.
### Avoiding Data Leakage
To prevent data leakage, use a diverse dataset with images or videos from different cameras and environments. Carefully review your data and check that there are no hidden biases, such as all positive samples being taken at a specific time of day. Avoiding data leakage will help make your computer vision models more reliable and effective in real-world situations.
## What Comes After Model Testing
After testing your model, the next steps depend on the results. If your model performs well, you can deploy it into a real-world environment. If the results aren't satisfactory, you'll need to make improvements. This might involve analyzing errors, [gathering more data](./data-collection-and-annotation.md), improving data quality, [adjusting hyperparameters](./hyperparameter-tuning.md), and retraining the model.
## Join the AI Conversation
Becoming part of a community of computer vision enthusiasts can aid in solving problems and learning more efficiently. Here are some ways to connect, seek help, and share your thoughts.
### Community Resources
- **GitHub Issues:** Explore the [YOLO26 GitHub repository](https://github.com/ultralytics/ultralytics/issues) and use the Issues tab to ask questions, report bugs, and suggest new features. The community and maintainers are very active and ready to help.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to chat with other users and developers, get support, and share your experiences.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Check out the [official YOLO26 documentation](./index.md) for detailed guides and helpful tips on various computer vision projects.
These resources will help you navigate challenges and remain updated on the latest trends and practices within the computer vision community.
## In Summary
Building trustworthy computer vision models relies on rigorous model testing. By testing the model with previously unseen data, we can analyze it and spot weaknesses like [overfitting](https://www.ultralytics.com/glossary/overfitting) and data leakage. Addressing these issues before deployment helps the model perform well in real-world applications. It's important to remember that model testing is just as crucial as model evaluation in guaranteeing the model's long-term success and effectiveness.
## FAQ
### What are the key differences between model evaluation and model testing in computer vision?
Model evaluation and model testing are distinct steps in a computer vision project. Model evaluation involves using a labeled dataset to compute metrics such as [accuracy](https://www.ultralytics.com/glossary/accuracy), precision, recall, and [F1 score](https://www.ultralytics.com/glossary/f1-score), providing insights into the model's performance with a controlled dataset. Model testing, on the other hand, assesses the model's performance in real-world scenarios by applying it to new, unseen data, ensuring the model's learned behavior aligns with expectations outside the evaluation environment. For a detailed guide, refer to the [steps in a computer vision project](./steps-of-a-cv-project.md).
### How can I test my Ultralytics YOLO26 model on multiple images?
To test your Ultralytics YOLO26 model on multiple images, you can use the [prediction mode](../modes/predict.md). This mode allows you to run the model on new, unseen data to generate predictions without providing detailed metrics. This is ideal for real-world performance testing on larger image sets stored in a folder. For evaluating performance metrics, use the [validation mode](../modes/val.md) instead.
### What should I do if my computer vision model shows signs of overfitting or underfitting?
To address **overfitting**:
- [Regularization](https://www.ultralytics.com/glossary/regularization) techniques like dropout.
- Increase the size of the training dataset.
- Simplify the model architecture.
To address **underfitting**:
- Use a more complex model.
- Provide more relevant features.
- Increase training iterations or [epochs](https://www.ultralytics.com/glossary/epoch).
Review misclassified images, perform thorough error analysis, and regularly track performance metrics to maintain a balance. For more information on these concepts, explore our section on [Overfitting and Underfitting](#overfitting-and-underfitting-in-machine-learning).
### How can I detect and avoid data leakage in computer vision?
To detect data leakage:
- Verify that the testing performance is not unusually high.
- Check feature importance for unexpected insights.
- Intuitively review model decisions.
- Ensure correct data division before processing.
To avoid data leakage:
- Use diverse datasets with various environments.
- Carefully review data for hidden biases.
- Ensure no overlapping information between training and testing sets.
For detailed strategies on preventing data leakage, refer to our section on [Data Leakage in Computer Vision](#data-leakage-in-computer-vision-and-how-to-avoid-it).
### What steps should I take after testing my computer vision model?
Post-testing, if the model performance meets the project goals, proceed with deployment. If the results are unsatisfactory, consider:
- Error analysis.
- Gathering more diverse and high-quality data.
- [Hyperparameter tuning](https://www.ultralytics.com/glossary/hyperparameter-tuning).
- Retraining the model.
Gain insights from the [Model Testing Vs. Model Evaluation](#model-testing-vs-model-evaluation) section to refine and enhance model effectiveness in real-world applications.
### How do I run YOLO26 predictions without custom training?
You can run predictions using the pretrained YOLO26 model on your dataset to see if it suits your application needs. Utilize the [prediction mode](../modes/predict.md) to get a quick sense of performance results without diving into custom training.

204
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-training-tips.md Normal file
View File

@@ -0,0 +1,204 @@
---
comments: true
description: Learn best practices for training computer vision models, including batch size optimization, mixed precision training, early stopping, and optimizer selection for improved efficiency and accuracy.
keywords: Model Training Machine Learning, AI Model Training, Number of Epochs, How to Train a Model in Machine Learning, Machine Learning Best Practices, What is Model Training
---
# Machine Learning Best Practices and Tips for Model Training
## Introduction
One of the most important steps when working on a [computer vision project](./steps-of-a-cv-project.md) is model training. Before reaching this step, you need to [define your goals](./defining-project-goals.md) and [collect and annotate your data](./data-collection-and-annotation.md). After [preprocessing the data](./preprocessing_annotated_data.md) to make sure it is clean and consistent, you can move on to training your model.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/GIrFEoR5PoU"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Model Training Tips | How to Handle Large Datasets | Batch Size, GPU Utilization and <a href="https://www.ultralytics.com/glossary/mixed-precision">Mixed Precision</a>
</p>
So, what is [model training](../modes/train.md)? Model training is the process of teaching your model to recognize visual patterns and make predictions based on your data. It directly impacts the performance and accuracy of your application. In this guide, we'll cover best practices, optimization techniques, and troubleshooting tips to help you train your computer vision models effectively.
## How to Train a Machine Learning Model
A computer vision model is trained by adjusting its internal parameters to minimize errors. Initially, the model is fed a large set of labeled images. It makes predictions about what is in these images, and the predictions are compared to the actual labels or contents to calculate errors. These errors show how far off the model's predictions are from the true values.
During training, the model iteratively makes predictions, calculates errors, and updates its parameters through a process called [backpropagation](https://www.ultralytics.com/glossary/backpropagation). In this process, the model adjusts its internal parameters (weights and biases) to reduce the errors. By repeating this cycle many times, the model gradually improves its accuracy. Over time, it learns to recognize complex patterns such as shapes, colors, and textures.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/backpropagation-diagram.avif" alt="What is Backpropagation?">
</p>
This learning process makes it possible for the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) model to perform various [tasks](../tasks/index.md), including [object detection](../tasks/detect.md), [instance segmentation](../tasks/segment.md), and [image classification](../tasks/classify.md). The ultimate goal is to create a model that can generalize its learning to new, unseen images so that it can accurately understand visual data in real-world applications.
Now that we know what is happening behind the scenes when we train a model, let's look at points to consider when training a model.
## Training on Large Datasets
There are a few different aspects to think about when you are planning on using a large dataset to train a model. For example, you can adjust the batch size, control the GPU utilization, choose to use multiscale training, etc. Let's walk through each of these options in detail.
### Batch Size and GPU Utilization
When training models on large datasets, efficiently utilizing your GPU is key. Batch size is an important factor. It is the number of data samples that a machine learning model processes in a single training iteration.
Using the maximum batch size supported by your GPU, you can fully take advantage of its capabilities and reduce the time model training takes. However, you want to avoid running out of GPU memory. If you encounter memory errors, reduce the batch size incrementally until the model trains smoothly.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Gxl6Bbpcxs0"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Use Batch Inference with Ultralytics YOLO26 | Speed Up Object Detection in Python 🎉
</p>
With respect to YOLO26, you can set the `batch_size` parameter in the [training configuration](../modes/train.md) to match your GPU capacity. Also, setting `batch=-1` in your training script will automatically determine the [batch size](https://www.ultralytics.com/glossary/batch-size) that can be efficiently processed based on your device's capabilities. By fine-tuning the batch size, you can make the most of your GPU resources and improve the overall training process.
### Subset Training
Subset training is a smart strategy that involves training your model on a smaller set of data that represents the larger dataset. It can save time and resources, especially during initial model development and testing. If you are running short on time or experimenting with different model configurations, subset training is a good option.
When it comes to YOLO26, you can easily implement subset training by using the `fraction` parameter. This parameter lets you specify what fraction of your dataset to use for training. For example, setting `fraction=0.1` will train your model on 10% of the data. You can use this technique for quick iterations and tuning your model before committing to training a model using a full dataset. Subset training helps you make rapid progress and identify potential issues early on.
### Multi-scale Training
Multiscale training is a technique that improves your model's ability to generalize by training it on images of varying sizes. Your model can learn to detect objects at different scales and distances and become more robust.
For example, when you train YOLO26, you can enable multiscale training by setting the `scale` parameter. This parameter adjusts the size of training images by a specified factor, simulating objects at different distances. For example, setting `scale=0.5` randomly zooms training images by a factor between 0.5 and 1.5 during training. Configuring this parameter allows your model to experience a variety of image scales and improve its detection capabilities across different object sizes and scenarios.
Ultralytics also supports image-size multi-scale training via the `multi_scale` parameter. Unlike `scale`, which zooms images and then pads/crops back to `imgsz`, `multi_scale` changes `imgsz` itself each batch (rounded to the model stride). For example, with `imgsz=640` and `multi_scale=0.25`, the training size is sampled from 480 up to 800 in stride steps (e.g., 480, 512, 544, ..., 800), while `multi_scale=0.0` keeps a fixed size.
### Caching
Caching is an important technique to improve the efficiency of training machine learning models. By storing preprocessed images in memory, caching reduces the time the GPU spends waiting for data to be loaded from the disk. The model can continuously receive data without delays caused by disk I/O operations.
Caching can be controlled when training YOLO26 using the `cache` parameter:
- _`cache=True`_: Stores dataset images in RAM, providing the fastest access speed but at the cost of increased memory usage.
- _`cache='disk'`_: Stores the images on disk, slower than RAM but faster than loading fresh data each time.
- _`cache=False`_: Disables caching, relying entirely on disk I/O, which is the slowest option.
### Mixed Precision Training
Mixed precision training uses both 16-bit (FP16) and 32-bit (FP32) floating-point types. The strengths of both FP16 and FP32 are leveraged by using FP16 for faster computation and FP32 to maintain precision where needed. Most of the [neural network](https://www.ultralytics.com/glossary/neural-network-nn)'s operations are done in FP16 to benefit from faster computation and lower memory usage. However, a master copy of the model's weights is kept in FP32 to ensure accuracy during the weight update steps. You can handle larger models or larger batch sizes within the same hardware constraints.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/mixed-precision-training-overview.avif" alt="Mixed precision FP16 training benefits">
</p>
To implement mixed precision training, you'll need to modify your training scripts and ensure your hardware (like GPUs) supports it. Many modern [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) frameworks, such as [PyTorch](https://www.ultralytics.com/glossary/pytorch) and [TensorFlow](https://www.ultralytics.com/glossary/tensorflow), offer built-in support for mixed precision.
Mixed precision training is straightforward when working with YOLO26. You can use the `amp` flag in your training configuration. Setting `amp=True` enables Automatic Mixed Precision (AMP) training. Mixed precision training is a simple yet effective way to optimize your model training process.
### Pretrained Weights
Using pretrained weights is a smart way to speed up your model's training process. Pretrained weights come from models already trained on large datasets, giving your model a head start. [Transfer learning](https://www.ultralytics.com/glossary/transfer-learning) adapts pretrained models to new, related tasks. Fine-tuning a pretrained model involves starting with these weights and then continuing training on your specific dataset. This method of training results in faster training times and often better performance because the model starts with a solid understanding of basic features.
The `pretrained` parameter makes transfer learning easy with YOLO26. Setting `pretrained=True` will use default pretrained weights, or you can specify a path to a custom pretrained model. Using pretrained weights and transfer learning effectively boosts your model's capabilities and reduces training costs.
### Other Techniques to Consider When Handling a Large Dataset
There are a couple of other techniques to consider when handling a large dataset:
- **[Learning Rate](https://www.ultralytics.com/glossary/learning-rate) Schedulers**: Implementing learning rate schedulers dynamically adjusts the learning rate during training. A well-tuned learning rate can prevent the model from overshooting minima and improve stability. When training YOLO26, the `lrf` parameter helps manage learning rate scheduling by setting the final learning rate as a fraction of the initial rate.
- **Distributed Training**: For handling large datasets, distributed training can be a game-changer. You can reduce the training time by spreading the training workload across multiple GPUs or machines. This approach is particularly valuable for enterprise-scale projects with substantial computational resources.
## The Number of Epochs To Train For
When training a model, an [epoch](https://www.ultralytics.com/glossary/epoch) refers to one complete pass through the entire training dataset. During an epoch, the model processes each example in the training set once and updates its parameters based on the learning algorithm. Multiple epochs are usually needed to allow the model to learn and refine its parameters over time.
A common question that comes up is how to determine the number of epochs to train the model for. A good starting point is 300 epochs. If the model overfits early, you can reduce the number of epochs. If [overfitting](https://www.ultralytics.com/glossary/overfitting) does not occur after 300 epochs, you can extend the training to 600, 1200, or more epochs.
However, the ideal number of epochs can vary based on your dataset's size and project goals. Larger datasets might require more epochs for the model to learn effectively, while smaller datasets might need fewer epochs to avoid overfitting. With respect to YOLO26, you can set the `epochs` parameter in your training script.
## Early Stopping
Early stopping is a valuable technique for optimizing model training. By monitoring validation performance, you can halt training once the model stops improving. You can save computational resources and prevent overfitting.
The process involves setting a patience parameter that determines how many epochs to wait for an improvement in validation metrics before stopping training. If the model's performance does not improve within these epochs, training is stopped to avoid wasting time and resources.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/early-stopping-overview.avif" alt="Early stopping to prevent model overfitting">
</p>
For YOLO26, you can enable early stopping by setting the patience parameter in your training configuration. For example, `patience=5` means training will stop if there's no improvement in validation metrics for 5 consecutive epochs. Using this method ensures the training process remains efficient and achieves optimal performance without excessive computation.
## Choosing Between Cloud and Local Training
There are two options for training your model: cloud training and local training.
Cloud training offers scalability and powerful hardware and is ideal for handling large datasets and complex models. Platforms like [Google Cloud](https://cloud.google.com/), [AWS](https://aws.amazon.com/), and [Azure](https://azure.microsoft.com/) provide on-demand access to high-performance GPUs and TPUs, speeding up training times and enabling experiments with larger models. However, cloud training can be expensive, especially for long periods, and data transfer can add to costs and latency.
Local training provides greater control and customization, letting you tailor your environment to specific needs and avoid ongoing cloud costs. It can be more economical for long-term projects, and since your data stays on-premises, it's more secure. However, local hardware may have resource limitations and require maintenance, which can lead to longer training times for large models.
## Selecting an Optimizer
An optimizer is an algorithm that adjusts the weights of your neural network to minimize the [loss function](https://www.ultralytics.com/glossary/loss-function), which measures how well the model is performing. In simpler terms, the optimizer helps the model learn by tweaking its parameters to reduce errors. Choosing the right optimizer directly affects how quickly and accurately the model learns.
You can also fine-tune optimizer parameters to improve model performance. Adjusting the learning rate sets the size of the steps when updating parameters. For stability, you might start with a moderate learning rate and gradually decrease it over time to improve long-term learning. Additionally, setting the momentum determines how much influence past updates have on current updates. A common value for momentum is around 0.9. It generally provides a good balance.
### Common Optimizers
Different optimizers have various strengths and weaknesses. Let's take a glimpse at a few common optimizers.
- **SGD (Stochastic Gradient Descent)**:
- Updates model parameters using the gradient of the loss function with respect to the parameters.
- Simple and efficient but can be slow to converge and might get stuck in local minima.
- **[Adam](https://www.ultralytics.com/glossary/adam-optimizer) (Adaptive Moment Estimation)**:
- Combines the benefits of both SGD with momentum and RMSProp.
- Adjusts the learning rate for each parameter based on estimates of the first and second moments of the gradients.
- Well-suited for noisy data and sparse gradients.
- Efficient and generally requires less tuning, making it a recommended optimizer for YOLO26.
- **RMSProp (Root Mean Square Propagation)**:
- Adjusts the learning rate for each parameter by dividing the gradient by a running average of the magnitudes of recent gradients.
- Helps in handling the vanishing gradient problem and is effective for [recurrent neural networks](https://www.ultralytics.com/glossary/recurrent-neural-network-rnn).
For YOLO26, the `optimizer` parameter lets you choose from various optimizers, including SGD, Adam, AdamW, NAdam, RAdam, and RMSProp, or you can set it to `auto` for automatic selection based on model configuration.
## Connecting with the Community
Being part of a community of computer vision enthusiasts can help you solve problems and learn faster. Here are some ways to connect, get help, and share ideas.
### Community Resources
- **GitHub Issues:** Visit the [YOLO26 GitHub repository](https://github.com/ultralytics/ultralytics/issues) and use the Issues tab to ask questions, report bugs, and suggest new features. The community and maintainers are very active and ready to help.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to chat with other users and developers, get support, and share your experiences.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Check out the [official YOLO26 documentation](./index.md) for detailed guides and helpful tips on various computer vision projects.
Using these resources will help you solve challenges and stay up-to-date with the latest trends and practices in the computer vision community.
## Key Takeaways
Training computer vision models involves following good practices, optimizing your strategies, and solving problems as they arise. Techniques like adjusting batch sizes, mixed [precision](https://www.ultralytics.com/glossary/precision) training, and starting with pretrained weights can make your models work better and train faster. Methods like subset training and early stopping help you save time and resources. Staying connected with the community and keeping up with new trends will help you keep improving your model training skills.
## FAQ
### How can I improve GPU utilization when training a large dataset with Ultralytics YOLO?
To improve GPU utilization, set the `batch_size` parameter in your training configuration to the maximum size supported by your GPU. This ensures that you make full use of the GPU's capabilities, reducing training time. If you encounter memory errors, incrementally reduce the batch size until training runs smoothly. For YOLO26, setting `batch=-1` in your training script will automatically determine the optimal batch size for efficient processing. For further information, refer to the [training configuration](../modes/train.md).
### What is mixed precision training, and how do I enable it in YOLO26?
Mixed precision training utilizes both 16-bit (FP16) and 32-bit (FP32) floating-point types to balance computational speed and precision. This approach speeds up training and reduces memory usage without sacrificing model [accuracy](https://www.ultralytics.com/glossary/accuracy). To enable mixed precision training in YOLO26, set the `amp` parameter to `True` in your training configuration. This activates Automatic Mixed Precision (AMP) training. For more details on this optimization technique, see the [training configuration](../modes/train.md).
### How does multiscale training enhance YOLO26 model performance?
Multiscale training enhances model performance by training on images of varying sizes, allowing the model to better generalize across different scales and distances. In YOLO26, you can enable multiscale training by setting the `scale` parameter in the training configuration. For example, `scale=0.5` samples a zoom factor between 0.5 and 1.5, then pads/crops back to `imgsz`. This technique simulates objects at different distances, making the model more robust across various scenarios. For settings and more details, check out the [training configuration](../modes/train.md).
### How can I use pretrained weights to speed up training in YOLO26?
Using pretrained weights can greatly accelerate training and enhance model accuracy by leveraging a model already familiar with foundational visual features. In YOLO26, simply set the `pretrained` parameter to `True` or provide a path to your custom pretrained weights in the training configuration. This method, called transfer learning, allows models trained on large datasets to be effectively adapted to your specific application. Learn more about how to use pretrained weights and their benefits in the [training configuration guide](../modes/train.md).
### What is the recommended number of epochs for training a model, and how do I set this in YOLO26?
The number of epochs refers to the complete passes through the training dataset during model training. A typical starting point is 300 epochs. If your model overfits early, you can reduce the number. Alternatively, if overfitting isn't observed, you might extend training to 600, 1200, or more epochs. To set this in YOLO26, use the `epochs` parameter in your training script. For additional advice on determining the ideal number of epochs, refer to this section on [number of epochs](#the-number-of-epochs-to-train-for).

483
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/model-yaml-config.md Normal file
View File

@@ -0,0 +1,483 @@
---
comments: true
description: Learn how to structure and customize model architectures using Ultralytics YAML configuration files. Master module definitions, connections, and scaling parameters.
keywords: Ultralytics, YOLO, model architecture, YAML configuration, neural networks, deep learning, backbone, head, modules, custom models
---
# Model YAML Configuration Guide
The model YAML configuration file serves as the architectural blueprint for Ultralytics neural networks. It defines how layers connect, what parameters each module uses, and how the entire network scales across different model sizes.
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/yaml-configuration-guide.avif" alt="Model YAML configuration workflow.">
## Configuration Structure
Model YAML files are organized into three main sections that work together to define the architecture.
### Parameters Section
The **parameters** section specifies the model's global characteristics and scaling behavior:
```yaml
# Parameters
nc: 80 # number of classes
scales: # compound scaling constants [depth, width, max_channels]
n: [0.50, 0.25, 1024] # nano: shallow layers, narrow channels
s: [0.50, 0.50, 1024] # small: shallow depth, standard width
m: [0.50, 1.00, 512] # medium: moderate depth, full width
l: [1.00, 1.00, 512] # large: full depth and width
x: [1.00, 1.50, 512] # extra-large: maximum performance
kpt_shape: [17, 3] # pose models only
```
- `nc` sets the number of classes the model predicts.
- `scales` define compound scaling factors that adjust model depth, width, and maximum channels to produce different size variants (nano through extra-large).
- `kpt_shape` applies to pose models. It can be `[N, 2]` for `(x, y)` keypoints or `[N, 3]` for `(x, y, visibility)`.
!!! tip "Reduce redundancy with `scales`"
The `scales` parameter lets you generate multiple model sizes from a single base YAML. For instance, when you load `yolo26n.yaml`, Ultralytics reads the base `yolo26.yaml` and applies the `n` scaling factors (`depth=0.50`, `width=0.25`) to build the nano variant.
!!! note "`nc` and `kpt_shape` are dataset-dependent"
If your dataset specifies a different `nc` or `kpt_shape`, Ultralytics will automatically override the model config at runtime to match the dataset YAML.
### Backbone and Head Architecture
The model architecture consists of backbone (feature extraction) and head (task-specific) sections:
```yaml
backbone:
# [from, repeats, module, args]
- [-1, 1, Conv, [64, 3, 2]] # 0: Initial convolution
- [-1, 1, Conv, [128, 3, 2]] # 1: Downsample
- [-1, 3, C2f, [128, True]] # 2: Feature processing
head:
- [-1, 1, nn.Upsample, [None, 2, nearest]] # 6: Upsample
- [[-1, 2], 1, Concat, [1]] # 7: Skip connection
- [-1, 3, C2f, [256]] # 8: Process features
- [[8], 1, Detect, [nc]] # 9: Detection layer
```
## Layer Specification Format
Every layer follows the consistent pattern: **`[from, repeats, module, args]`**
| Component | Purpose | Examples |
| ----------- | --------------------- | --------------------------------------------------------- |
| **from** | Input connections | `-1` (previous), `6` (layer 6), `[4, 6, 8]` (multi-input) |
| **repeats** | Number of repetitions | `1` (single), `3` (repeat 3 times) |
| **module** | Module type | `Conv`, `C2f`, `TorchVision`, `Detect` |
| **args** | Module arguments | `[64, 3, 2]` (channels, kernel, stride) |
### Connection Patterns
The `from` field creates flexible data flow patterns throughout your network:
=== "Sequential Flow"
```yaml
- [-1, 1, Conv, [64, 3, 2]] # Takes input from previous layer
```
=== "Skip Connections"
```yaml
- [[-1, 6], 1, Concat, [1]] # Combines current layer with layer 6
```
=== "Multi-Input Fusion"
```yaml
- [[4, 6, 8], 1, Detect, [nc]] # Detection head using 3 feature scales
```
!!! note "Layer Indexing"
Layers are indexed starting from 0. Negative indices reference previous layers (`-1` = previous layer), while positive indices reference specific layers by their position.
### Module Repetition
The `repeats` parameter creates deeper network sections:
```yaml
- [-1, 3, C2f, [128, True]] # Creates 3 consecutive C2f blocks
- [-1, 1, Conv, [64, 3, 2]] # Single convolution layer
```
The actual repetition count gets multiplied by the depth scaling factor from your model size configuration.
## Available Modules
Modules are organized by functionality and defined in the [Ultralytics modules directory](https://github.com/ultralytics/ultralytics/tree/main/ultralytics/nn/modules). The following tables show commonly used modules by category, with many more available in the source code:
### Basic Operations
| Module | Purpose | Source | Arguments |
| ------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------- | --------------------------------------- |
| `Conv` | Convolution + BatchNorm + Activation | [conv.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/conv.py) | `[out_ch, kernel, stride, pad, groups]` |
| `nn.Upsample` | Spatial upsampling | [PyTorch](https://docs.pytorch.org/docs/stable/generated/torch.nn.Upsample.html) | `[size, scale_factor, mode]` |
| `nn.Identity` | Pass-through operation | [PyTorch](https://docs.pytorch.org/docs/stable/generated/torch.nn.Identity.html) | `[]` |
### Composite Blocks
| Module | Purpose | Source | Arguments |
| -------- | ---------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------- |
| `C2f` | CSP bottleneck with 2 convolutions | [block.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/block.py) | `[out_ch, shortcut, expansion]` |
| `SPPF` | Spatial Pyramid Pooling (fast) | [block.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/block.py) | `[out_ch, kernel_size]` |
| `Concat` | Channel-wise concatenation | [conv.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/conv.py) | `[dimension]` |
### Specialized Modules
| Module | Purpose | Source | Arguments |
| ------------- | --------------------------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------- |
| `TorchVision` | Load any torchvision model | [block.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/block.py) | `[out_ch, model_name, weights, unwrap, truncate, split]` |
| `Index` | Extract specific tensor from list | [block.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/block.py) | `[out_ch, index]` |
| `Detect` | YOLO detection head | [head.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/head.py) | `[nc, anchors, ch]` |
!!! info "Complete Module List"
This represents a subset of available modules. For the full list of modules and their parameters, explore the [modules directory](https://github.com/ultralytics/ultralytics/tree/main/ultralytics/nn/modules).
## Advanced Features
### TorchVision Integration
The TorchVision module enables seamless integration of any [TorchVision model](https://docs.pytorch.org/vision/stable/models.html) as a backbone:
=== "Python"
```python
from ultralytics import YOLO
# Model with ConvNeXt backbone
model = YOLO("convnext_backbone.yaml")
results = model.train(data="coco8.yaml", epochs=100)
```
=== "YAML Configuration"
```yaml
backbone:
- [-1, 1, TorchVision, [768, convnext_tiny, DEFAULT, True, 2, False]]
head:
- [-1, 1, Classify, [nc]]
```
**Parameter Breakdown:**
- `768`: Expected output channels
- `convnext_tiny`: Model architecture ([available models](https://docs.pytorch.org/vision/stable/models.html))
- `DEFAULT`: Use pretrained weights
- `True`: Remove classification head
- `2`: Truncate last 2 layers
- `False`: Return single tensor (not list)
!!! tip "Multi-Scale Features"
Set the last parameter to `True` to get intermediate feature maps for multi-scale detection.
### Index Module for Feature Selection
When using models that output multiple feature maps, the Index module selects specific outputs:
```yaml
backbone:
- [-1, 1, TorchVision, [768, convnext_tiny, DEFAULT, True, 2, True]] # Multi-output
head:
- [0, 1, Index, [192, 4]] # Select 4th feature map (192 channels)
- [0, 1, Index, [384, 6]] # Select 6th feature map (384 channels)
- [0, 1, Index, [768, 8]] # Select 8th feature map (768 channels)
- [[1, 2, 3], 1, Detect, [nc]] # Multi-scale detection
```
## Module Resolution System
Understanding how Ultralytics locates and imports modules is crucial for customization:
### Module Lookup Process
Ultralytics uses a three-tier system in [`parse_model`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/tasks.py):
```python
# Core resolution logic
m = getattr(torch.nn, m[3:]) if "nn." in m else getattr(torchvision.ops, m[4:]) if "ops." in m else globals()[m]
```
1. **PyTorch modules**: Names starting with `'nn.'` → `torch.nn` namespace
2. **TorchVision operations**: Names starting with `'ops.'` → `torchvision.ops` namespace
3. **Ultralytics modules**: All other names → global namespace via imports
### Module Import Chain
Standard modules become available through imports in [`tasks.py`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/tasks.py):
```python
from ultralytics.nn.modules import ( # noqa: F401
SPPF,
C2f,
Conv,
Detect,
# ... many more modules
Index,
TorchVision,
)
```
## Custom Module Integration
### Source Code Modification
Modifying the source code is the most versatile way to integrate your custom modules, but it can be tricky. To define and use a custom module, follow these steps:
1. **Install Ultralytics in development mode** using the Git clone method from the [Quickstart guide](https://docs.ultralytics.com/quickstart/#git-clone).
2. **Define your module** in [`ultralytics/nn/modules/block.py`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/block.py):
```python
class CustomBlock(nn.Module):
"""Custom block with Conv-BatchNorm-ReLU sequence."""
def __init__(self, c1, c2):
"""Initialize CustomBlock with input and output channels."""
super().__init__()
self.layers = nn.Sequential(nn.Conv2d(c1, c2, 3, 1, 1), nn.BatchNorm2d(c2), nn.ReLU())
def forward(self, x):
"""Forward pass through the block."""
return self.layers(x)
```
3. **Expose your module at the package level** in [`ultralytics/nn/modules/__init__.py`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/modules/__init__.py):
```python
from .block import CustomBlock # noqa makes CustomBlock available as ultralytics.nn.modules.CustomBlock
```
4. **Add to imports** in [`ultralytics/nn/tasks.py`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/tasks.py):
```python
from ultralytics.nn.modules import CustomBlock # noqa
```
5. **Handle special arguments** (if needed) inside [`parse_model()`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/tasks.py) in `ultralytics/nn/tasks.py`:
```python
# Add this condition in the parse_model() function
if m is CustomBlock:
c1, c2 = ch[f], args[0] # input channels, output channels
args = [c1, c2, *args[1:]]
```
6. **Use the module** in your model YAML:
```yaml
# custom_model.yaml
nc: 1
backbone:
- [-1, 1, CustomBlock, [64]]
head:
- [-1, 1, Classify, [nc]]
```
7. **Check FLOPs** to ensure the forward pass works:
```python
from ultralytics import YOLO
model = YOLO("custom_model.yaml", task="classify")
model.info() # should print non-zero FLOPs if working
```
## Example Configurations
### Basic Detection Model
```yaml
# Simple YOLO detection model
nc: 80
scales:
n: [0.33, 0.25, 1024]
backbone:
- [-1, 1, Conv, [64, 3, 2]] # 0-P1/2
- [-1, 1, Conv, [128, 3, 2]] # 1-P2/4
- [-1, 3, C2f, [128, True]] # 2
- [-1, 1, Conv, [256, 3, 2]] # 3-P3/8
- [-1, 6, C2f, [256, True]] # 4
- [-1, 1, SPPF, [256, 5]] # 5
head:
- [-1, 1, Conv, [256, 3, 1]] # 6
- [[6], 1, Detect, [nc]] # 7
```
### TorchVision Backbone Model
```yaml
# ConvNeXt backbone with YOLO head
nc: 80
backbone:
- [-1, 1, TorchVision, [768, convnext_tiny, DEFAULT, True, 2, True]]
head:
- [0, 1, Index, [192, 4]] # P3 features
- [0, 1, Index, [384, 6]] # P4 features
- [0, 1, Index, [768, 8]] # P5 features
- [[1, 2, 3], 1, Detect, [nc]] # Multi-scale detection
```
### Classification Model
```yaml
# Simple classification model
nc: 1000
backbone:
- [-1, 1, Conv, [64, 7, 2, 3]]
- [-1, 1, nn.MaxPool2d, [3, 2, 1]]
- [-1, 4, C2f, [64, True]]
- [-1, 1, Conv, [128, 3, 2]]
- [-1, 8, C2f, [128, True]]
- [-1, 1, nn.AdaptiveAvgPool2d, [1]]
head:
- [-1, 1, Classify, [nc]]
```
## Best Practices
### Architecture Design Tips
**Start Simple**: Begin with proven architectures before customizing. Use existing YOLO configurations as templates and modify incrementally rather than building from scratch.
**Test Incrementally**: Validate each modification step-by-step. Add one custom module at a time and verify it works before proceeding to the next change.
**Monitor Channels**: Ensure channel dimensions match between connected layers. The output channels (`c2`) of one layer must match the input channels (`c1`) of the next layer in the sequence.
**Use Skip Connections**: Leverage feature reuse with `[[-1, N], 1, Concat, [1]]` patterns. These connections help with gradient flow and allow the model to combine features from different scales.
**Scale Appropriately**: Choose model scales based on your computational constraints. Use nano (`n`) for edge devices, small (`s`) for balanced performance, and larger scales (`m`, `l`, `x`) for maximum accuracy.
### Performance Considerations
**Depth vs Width**: Deep networks capture complex hierarchical features through multiple transformation layers, while wide networks process more information in parallel at each layer. Balance these based on your task complexity.
**Skip Connections**: Improve gradient flow during training and enable feature reuse throughout the network. They're particularly important in deeper architectures to prevent vanishing gradients.
**Bottleneck Blocks**: Reduce computational cost while maintaining model expressiveness. Modules like `C2f` use fewer parameters than standard convolutions while preserving feature learning capacity.
**Multi-Scale Features**: Essential for detecting objects at different sizes in the same image. Use Feature Pyramid Network (FPN) patterns with multiple detection heads at different scales.
## Troubleshooting
### Common Issues
| Problem | Cause | Solution |
| ----------------------------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `KeyError: 'ModuleName'` | Module not imported | Add to [`tasks.py`](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/nn/tasks.py) imports |
| Channel dimension mismatch | Incorrect `args` specification | Verify input/output channel compatibility |
| `AttributeError: 'int' object has no attribute` | Wrong argument type | Check module documentation for correct argument types |
| Model fails to build | Invalid `from` reference | Ensure referenced layers exist |
### Debugging Tips
When developing custom architectures, systematic debugging helps identify issues early:
**Use Identity Head for Testing**
Replace complex heads with `nn.Identity` to isolate backbone issues:
```yaml
nc: 1
backbone:
- [-1, 1, CustomBlock, [64]]
head:
- [-1, 1, nn.Identity, []] # Pass-through for debugging
```
This allows direct inspection of backbone outputs:
```python
import torch
from ultralytics import YOLO
model = YOLO("debug_model.yaml")
output = model.model(torch.randn(1, 3, 640, 640))
print(f"Output shape: {output.shape}") # Should match expected dimensions
```
**Model Architecture Inspection**
Checking the FLOPs count and printing out each layer can also help debug issues with your custom model config. FLOPs count should be non-zero for a valid model. If it's zero, then there's likely an issue with the forward pass. Running a simple forward pass should show the exact error being encountered.
```python
from ultralytics import YOLO
# Build model with verbose output to see layer details
model = YOLO("debug_model.yaml", verbose=True)
# Check model FLOPs. Failed forward pass causes 0 FLOPs.
model.info()
# Inspect individual layers
for i, layer in enumerate(model.model.model):
print(f"Layer {i}: {layer}")
```
**Step-by-Step Validation**
1. **Start minimal**: Test with simplest possible architecture first
2. **Add incrementally**: Build complexity layer by layer
3. **Check dimensions**: Verify channel and spatial size compatibility
4. **Validate scaling**: Test with different model scales (`n`, `s`, `m`)
## FAQ
### How do I change the number of classes in my model?
Set the `nc` parameter at the top of your YAML file to match your dataset's number of classes.
```yaml
nc: 5 # 5 classes
```
### Can I use a custom backbone in my model YAML?
Yes. You can use any supported module, including TorchVision backbones, or define your own custom module and import it as described in [Custom Module Integration](#custom-module-integration).
### How do I scale my model for different sizes (nano, small, medium, etc.)?
Use the [`scales` section](#parameters-section) in your YAML to define scaling factors for depth, width, and max channels. The model will automatically apply these when you load the base YAML file with the scale appended to the filename (e.g., `yolo26n.yaml`).
### What does the `[from, repeats, module, args]` format mean?
This format specifies how each layer is constructed:
- `from`: input source(s)
- `repeats`: number of times to repeat the module
- `module`: the layer type
- `args`: arguments for the module
### How do I troubleshoot channel mismatch errors?
Check that the output channels of one layer match the expected input channels of the next. Use `print(model.model.model)` to inspect your model's architecture.
### Where can I find a list of available modules and their arguments?
Check the source code in the [`ultralytics/nn/modules` directory](https://github.com/ultralytics/ultralytics/tree/main/ultralytics/nn/modules) for all available modules and their arguments.
### How do I add a custom module to my YAML configuration?
Define your module in the source code, import it as shown in [Source Code Modification](#source-code-modification), and reference it by name in your YAML file.
### Can I use pretrained weights with a custom YAML?
Yes, you can use `model.load("path/to/weights")` to load weights from a pretrained checkpoint. However, only weights for layers that match would load successfully.
### How do I validate my model configuration?
Use `model.info()` to check whether FLOPs count is non-zero. A valid model should show non-zero FLOPs count. If it's zero, follow the suggestions in [Debugging Tips](#debugging-tips) to find the issue.

407
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/nvidia-dgx-spark.md Normal file
View File

@@ -0,0 +1,407 @@
---
comments: true
description: Learn to deploy Ultralytics YOLO26 on NVIDIA DGX Spark with our detailed guide. Explore performance benchmarks and maximize AI capabilities on this compact desktop AI supercomputer.
keywords: Ultralytics, YOLO26, NVIDIA DGX Spark, AI deployment, performance benchmarks, deep learning, TensorRT, computer vision, GB10 Grace Blackwell
---
# Quick Start Guide: NVIDIA DGX Spark with Ultralytics YOLO26
This comprehensive guide provides a detailed walkthrough for deploying Ultralytics YOLO26 on [NVIDIA DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/), NVIDIA's compact desktop AI supercomputer. Additionally, it showcases performance benchmarks to demonstrate the capabilities of YOLO26 on this powerful system.
<p align="center">
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/nvidia-dgx-spark.avif" alt="NVIDIA DGX Spark AI workstation overview">
</p>
!!! note
This guide has been tested with NVIDIA DGX Spark Founders Edition running DGX OS based on Ubuntu. It is expected to work with the latest DGX OS releases.
## What is NVIDIA DGX Spark?
NVIDIA DGX Spark is a compact desktop AI supercomputer powered by the NVIDIA GB10 Grace Blackwell Superchip. It delivers up to 1 petaFLOP of AI computing performance with FP4 precision, making it ideal for developers, researchers, and data scientists who need powerful AI capabilities in a desktop form factor.
### Key Specifications
| Specification | Details |
| ---------------- | --------------------------------------------------------------------------------------- |
| AI Performance | Up to 1 PFLOP (FP4) |
| GPU | NVIDIA Blackwell Architecture with 5th Generation Tensor Cores, 4th Generation RT Cores |
| CPU | 20-core Arm processor (10 Cortex-X925 + 10 Cortex-A725) |
| Memory | 128 GB LPDDR5x unified system memory, 256-bit interface, 4266 MHz, 273 GB/s bandwidth |
| Storage | 1 TB or 4 TB NVMe M.2 with self-encryption |
| Network | 1x RJ-45 (10 GbE), ConnectX-7 Smart NIC, Wi-Fi 7, Bluetooth 5.4 |
| Connectivity | 4x USB Type-C, 1x HDMI 2.1a, HDMI multichannel audio |
| Video Processing | 1x NVENC, 1x NVDEC |
### DGX OS
[NVIDIA DGX OS](https://docs.nvidia.com/dgx/dgx-os-7-user-guide/introduction.html) is a customized Linux distribution that provides a stable, tested, and supported operating system foundation for running AI, machine learning, and analytics applications on DGX systems. It includes:
- A robust Linux foundation optimized for AI workloads
- Pre-configured drivers and system settings for NVIDIA hardware
- Security updates and system maintenance capabilities
- Compatibility with the broader NVIDIA software ecosystem
DGX OS follows a regular release schedule with updates typically provided twice per year (around February and August), with additional security patches provided between major releases.
### DGX Dashboard
DGX Spark comes with a built-in [DGX Dashboard](https://docs.nvidia.com/dgx/dgx-spark/dgx-dashboard.html) that provides:
- **Real-time System Monitoring**: Overview of the system's current operational metrics
- **System Updates**: Ability to apply updates directly from the dashboard
- **System Settings**: Change device name and other configurations
- **Integrated JupyterLab**: Access local Jupyter Notebooks for development
<p align="center">
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/nvidia-dgx-dashboard.avif" alt="NVIDIA DGX management dashboard interface">
</p>
#### Accessing the Dashboard
=== "Locally"
Click the "Show Apps" button in the bottom left corner of the Ubuntu desktop, then select "DGX Dashboard" to open it in your browser.
=== "Remotely via SSH"
```bash
# Open an SSH tunnel
ssh -L 11000:localhost:11000 username@spark-abcd.local
# Then open in browser
# http://localhost:11000
```
=== "Remotely via NVIDIA Sync"
After connecting with NVIDIA Sync, click the "DGX Dashboard" button to open the dashboard at `http://localhost:11000`.
!!! tip "Integrated JupyterLab"
The dashboard includes an integrated JupyterLab instance that automatically creates a virtual environment and installs recommended packages when started. Each user account is assigned a dedicated port for JupyterLab access.
## Quick Start with Docker
The fastest way to get started with Ultralytics YOLO26 on NVIDIA DGX Spark is to run with pre-built docker images. The same Docker image that supports Jetson AGX Thor (JetPack 7.0) works on DGX Spark with DGX OS.
```bash
t=ultralytics/ultralytics:latest-nvidia-arm64
sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia --gpus all $t
```
After this is done, skip to [Use TensorRT on NVIDIA DGX Spark section](#use-tensorrt-on-nvidia-dgx-spark).
## Start with Native Installation
For a native installation without Docker, follow these steps.
### Install Ultralytics Package
Here we will install Ultralytics package on DGX Spark with optional dependencies so that we can export the [PyTorch](https://www.ultralytics.com/glossary/pytorch) models to other different formats. We will mainly focus on [NVIDIA TensorRT exports](../integrations/tensorrt.md) because TensorRT will make sure we can get the maximum performance out of the DGX Spark.
1. Update packages list, install pip and upgrade to latest
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install `ultralytics` pip package with optional dependencies
```bash
pip install ultralytics[export]
```
3. Reboot the device
```bash
sudo reboot
```
### Install PyTorch and Torchvision
The above ultralytics installation will install Torch and Torchvision. However, these packages installed via pip may not be fully optimized for the DGX Spark's ARM64 architecture with CUDA 13. Therefore, we recommend installing the CUDA 13 compatible versions:
```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
```
!!! info
When running PyTorch 2.9.1 on NVIDIA DGX Spark, you may encounter the following `UserWarning` when initializing CUDA (e.g. running `yolo checks`, `yolo predict`, etc.):
```text
UserWarning: Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)
```
This warning can be safely ignored. To address this permanently, a fix has been submitted in PyTorch PR [#164590](https://github.com/pytorch/pytorch/pull/164590) which will be included in the PyTorch 2.10 release.
### Install `onnxruntime-gpu`
The [onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu/) package hosted in PyPI does not have `aarch64` binaries for ARM64 systems. So we need to manually install this package. This package is needed for some of the exports.
Here we will download and install `onnxruntime-gpu 1.24.0` with `Python3.12` support.
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.24.0-cp312-cp312-linux_aarch64.whl
```
## Use TensorRT on NVIDIA DGX Spark
Among all the model export formats supported by Ultralytics, TensorRT offers the highest inference performance on NVIDIA DGX Spark, making it our top recommendation for deployments. For setup instructions and advanced usage, see our [dedicated TensorRT integration guide](../integrations/tensorrt.md).
### Convert Model to TensorRT and Run Inference
The YOLO26n model in PyTorch format is converted to TensorRT to run inference with the exported model.
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Export the model to TensorRT
model.export(format="engine") # creates 'yolo26n.engine'
# Load the exported TensorRT model
trt_model = YOLO("yolo26n.engine")
# Run inference
results = trt_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO26n PyTorch model to TensorRT format
yolo export model=yolo26n.pt format=engine # creates 'yolo26n.engine'
# Run inference with the exported model
yolo predict model=yolo26n.engine source='https://ultralytics.com/images/bus.jpg'
```
!!! note
Visit the [Export page](../modes/export.md#arguments) to access additional arguments when exporting models to different model formats
## NVIDIA DGX Spark YOLO11 Benchmarks
YOLO11 benchmarks were run by the Ultralytics team on multiple model formats measuring speed and [accuracy](https://www.ultralytics.com/glossary/accuracy): PyTorch, TorchScript, ONNX, OpenVINO, TensorRT, TF SavedModel, TF GraphDef, TF Lite, MNN, NCNN, ExecuTorch. Benchmarks were run on NVIDIA DGX Spark at FP32 [precision](https://www.ultralytics.com/glossary/precision) with default input image size of 640.
### Detailed Comparison Table
The below table represents the benchmark results for five different models (YOLO11n, YOLO11s, YOLO11m, YOLO11l, YOLO11x) across multiple formats, giving us the status, size, mAP50-95(B) metric, and inference time for each combination.
!!! tip "Performance"
=== "YOLO11n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.4 | 0.5071 | 2.67 |
| TorchScript | ✅ | 10.5 | 0.5083 | 2.62 |
| ONNX | ✅ | 10.2 | 0.5074 | 5.92 |
| OpenVINO | ✅ | 10.4 | 0.5058 | 14.95 |
| TensorRT (FP32) | ✅ | 12.8 | 0.5085 | 1.95 |
| TensorRT (FP16) | ✅ | 7.0 | 0.5068 | 1.01 |
| TensorRT (INT8) | ✅ | 18.6 | 0.4880 | 1.62 |
| TF SavedModel | ✅ | 25.7 | 0.5076 | 36.39 |
| TF GraphDef | ✅ | 10.3 | 0.5076 | 41.06 |
| TF Lite | ✅ | 10.3 | 0.5075 | 64.36 |
| MNN | ✅ | 10.1 | 0.5075 | 12.14 |
| NCNN | ✅ | 10.2 | 0.5041 | 12.31 |
| ExecuTorch | ✅ | 10.2 | 0.5075 | 27.61 |
=== "YOLO11s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 18.4 | 0.5767 | 5.38 |
| TorchScript | ✅ | 36.5 | 0.5781 | 5.48 |
| ONNX | ✅ | 36.3 | 0.5784 | 8.17 |
| OpenVINO | ✅ | 36.4 | 0.5809 | 27.12 |
| TensorRT (FP32) | ✅ | 39.8 | 0.5783 | 3.59 |
| TensorRT (FP16) | ✅ | 20.1 | 0.5800 | 1.85 |
| TensorRT (INT8) | ✅ | 17.5 | 0.5664 | 1.88 |
| TF SavedModel | ✅ | 90.8 | 0.5782 | 66.63 |
| TF GraphDef | ✅ | 36.3 | 0.5782 | 71.67 |
| TF Lite | ✅ | 36.3 | 0.5782 | 187.36 |
| MNN | ✅ | 36.2 | 0.5775 | 27.05 |
| NCNN | ✅ | 36.2 | 0.5806 | 26.26 |
| ExecuTorch | ✅ | 36.2 | 0.5782 | 54.73 |
=== "YOLO11m"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 38.8 | 0.6254 | 11.14 |
| TorchScript | ✅ | 77.3 | 0.6304 | 12.00 |
| ONNX | ✅ | 76.9 | 0.6304 | 13.83 |
| OpenVINO | ✅ | 77.1 | 0.6284 | 62.44 |
| TensorRT (FP32) | ✅ | 79.9 | 0.6305 | 6.96 |
| TensorRT (FP16) | ✅ | 40.6 | 0.6313 | 3.14 |
| TensorRT (INT8) | ✅ | 26.6 | 0.6204 | 3.30 |
| TF SavedModel | ✅ | 192.4 | 0.6306 | 139.85 |
| TF GraphDef | ✅ | 76.9 | 0.6306 | 146.76 |
| TF Lite | ✅ | 76.9 | 0.6306 | 568.18 |
| MNN | ✅ | 76.8 | 0.6306 | 67.67 |
| NCNN | ✅ | 76.8 | 0.6308 | 60.49 |
| ExecuTorch | ✅ | 76.9 | 0.6306 | 120.37 |
=== "YOLO11l"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 49.0 | 0.6366 | 13.95 |
| TorchScript | ✅ | 97.6 | 0.6399 | 15.67 |
| ONNX | ✅ | 97.0 | 0.6399 | 16.62 |
| OpenVINO | ✅ | 97.3 | 0.6377 | 78.80 |
| TensorRT (FP32) | ✅ | 99.2 | 0.6407 | 8.86 |
| TensorRT (FP16) | ✅ | 50.8 | 0.6350 | 3.85 |
| TensorRT (INT8) | ✅ | 32.5 | 0.6224 | 4.52 |
| TF SavedModel | ✅ | 242.7 | 0.6409 | 187.45 |
| TF GraphDef | ✅ | 97.0 | 0.6409 | 193.92 |
| TF Lite | ✅ | 97.0 | 0.6409 | 728.61 |
| MNN | ✅ | 96.9 | 0.6369 | 85.21 |
| NCNN | ✅ | 96.9 | 0.6373 | 77.62 |
| ExecuTorch | ✅ | 97.0 | 0.6409 | 153.56 |
=== "YOLO11x"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 109.3 | 0.6992 | 23.19 |
| TorchScript | ✅ | 218.1 | 0.6900 | 25.75 |
| ONNX | ✅ | 217.5 | 0.6900 | 27.43 |
| OpenVINO | ✅ | 217.8 | 0.6872 | 149.44 |
| TensorRT (FP32) | ✅ | 222.7 | 0.6902 | 13.87 |
| TensorRT (FP16) | ✅ | 111.1 | 0.6883 | 6.19 |
| TensorRT (INT8) | ✅ | 62.9 | 0.6793 | 6.62 |
| TF SavedModel | ✅ | 543.9 | 0.6900 | 335.10 |
| TF GraphDef | ✅ | 217.5 | 0.6900 | 348.86 |
| TF Lite | ✅ | 217.5 | 0.6900 | 1578.66 |
| MNN | ✅ | 217.3 | 0.6874 | 168.95 |
| NCNN | ✅ | 217.4 | 0.6901 | 132.13 |
| ExecuTorch | ✅ | 217.4 | 0.6900 | 297.17 |
Benchmarked with Ultralytics 8.3.249
## Reproduce Our Results
To reproduce the above Ultralytics benchmarks on all export [formats](../modes/export.md) run this code:
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Benchmark YOLO26n speed and accuracy on the COCO128 dataset for all export formats
results = model.benchmark(data="coco128.yaml", imgsz=640)
```
=== "CLI"
```bash
# Benchmark YOLO26n speed and accuracy on the COCO128 dataset for all export formats
yolo benchmark model=yolo26n.pt data=coco128.yaml imgsz=640
```
Note that benchmarking results might vary based on the exact hardware and software configuration of a system, as well as the current workload of the system at the time the benchmarks are run. For the most reliable results, use a dataset with a large number of images, e.g., `data='coco.yaml'` (5000 val images).
## Best Practices for NVIDIA DGX Spark
When using NVIDIA DGX Spark, there are a couple of best practices to follow in order to enable maximum performance running YOLO26.
1. **Monitor System Performance**
Use NVIDIA's monitoring tools to track GPU and CPU utilization:
```bash
nvidia-smi
```
2. **Optimize Memory Usage**
With 128GB of unified memory, DGX Spark can handle large batch sizes and models. Consider increasing batch size for improved throughput:
```python
from ultralytics import YOLO
model = YOLO("yolo26n.engine")
results = model.predict(source="path/to/images", batch=16)
```
3. **Use TensorRT with FP16 or INT8**
For best performance, export models with FP16 or INT8 precision:
```bash
yolo export model=yolo26n.pt format=engine half=True # FP16
yolo export model=yolo26n.pt format=engine int8=True # INT8
```
## System Updates (Founders Edition)
Keeping your DGX Spark Founders Edition up to date is crucial for performance and security. NVIDIA provides two primary methods for updating the system OS, drivers, and firmware.
### Using DGX Dashboard (Recommended)
The [DGX Dashboard](https://docs.nvidia.com/dgx/dgx-spark/dgx-dashboard.html) is the recommended way to perform system updates ensuring compatibility. It allows you to:
- View available system updates
- Install security patches and system updates
- Manage NVIDIA driver and firmware updates
### Manual System Updates
For advanced users, updates can be performed manually via terminal:
```bash
sudo apt update
sudo apt dist-upgrade
sudo fwupdmgr refresh
sudo fwupdmgr upgrade
sudo reboot
```
!!! warning
Ensure your system is connected to a stable power source and you have backed up critical data before performing updates.
## Next Steps
For further learning and support, see the [Ultralytics YOLO26 Docs](../index.md).
## FAQ
### How do I deploy Ultralytics YOLO26 on NVIDIA DGX Spark?
Deploying Ultralytics YOLO26 on NVIDIA DGX Spark is straightforward. You can use the pre-built Docker image for quick setup or manually install the required packages. Detailed steps for each approach can be found in sections [Quick Start with Docker](#quick-start-with-docker) and [Start with Native Installation](#start-with-native-installation).
### What performance can I expect from YOLO26 on NVIDIA DGX Spark?
YOLO26 models deliver excellent performance on DGX Spark thanks to the GB10 Grace Blackwell Superchip. The TensorRT format provides the best inference performance. Check the [Detailed Comparison Table](#detailed-comparison-table) section for specific benchmark results across different model sizes and formats.
### Why should I use TensorRT for YOLO26 on DGX Spark?
TensorRT is highly recommended for deploying YOLO26 models on DGX Spark due to its optimal performance. It accelerates inference by leveraging the Blackwell GPU capabilities, ensuring maximum efficiency and speed. Learn more in the [Use TensorRT on NVIDIA DGX Spark](#use-tensorrt-on-nvidia-dgx-spark) section.
### How does DGX Spark compare to Jetson devices for YOLO26?
DGX Spark offers significantly more compute power than Jetson devices with up to 1 PFLOP of AI performance and 128GB unified memory, compared to Jetson AGX Thor's 2070 TFLOPS and 128GB memory. DGX Spark is designed as a desktop AI supercomputer, while Jetson devices are embedded systems optimized for edge deployment.
### Can I use the same Docker image for DGX Spark and Jetson AGX Thor?
Yes! The `ultralytics/ultralytics:latest-nvidia-arm64` Docker image supports both NVIDIA DGX Spark (with DGX OS) and Jetson AGX Thor (with JetPack 7.0), as both use ARM64 architecture with CUDA 13 and similar software stacks.

898
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/nvidia-jetson.md Normal file
View File

@@ -0,0 +1,898 @@
---
comments: true
description: Learn to deploy Ultralytics YOLO26 on NVIDIA Jetson devices with our detailed guide. Explore performance benchmarks and maximize AI capabilities.
keywords: Ultralytics, YOLO26, NVIDIA Jetson, JetPack, AI deployment, performance benchmarks, embedded systems, deep learning, TensorRT, computer vision
---
# Quick Start Guide: NVIDIA Jetson with Ultralytics YOLO26
This comprehensive guide provides a detailed walkthrough for deploying Ultralytics YOLO26 on [NVIDIA Jetson](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/) devices. Additionally, it showcases performance benchmarks to demonstrate the capabilities of YOLO26 on these small and powerful devices.
!!! tip "New product support"
We have updated this guide with the latest [NVIDIA Jetson AGX Thor Developer Kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor) which delivers up to 2070 FP4 TFLOPS of AI compute and 128 GB of memory with power configurable between 40 W and 130 W. It delivers over 7.5x higher AI compute than NVIDIA Jetson AGX Orin, with 3.5x better energy efficiency to seamlessly run the most popular AI models.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/BPYkGt3odNk"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to use Ultralytics YOLO26 on NVIDIA Jetson Devices
</p>
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/nvidia-jetson-ecosystem.avif" alt="NVIDIA Jetson Ecosystem">
!!! note
This guide has been tested with [NVIDIA Jetson AGX Thor Developer Kit (Jetson T5000)](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor) running the latest stable JetPack release of [JP7.0](https://developer.nvidia.com/embedded/jetpack/downloads), [NVIDIA Jetson AGX Orin Developer Kit (64GB)](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin) running JetPack release of [JP6.2](https://developer.nvidia.com/embedded/jetpack-sdk-62), [NVIDIA Jetson Orin Nano Super Developer Kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit) running JetPack release of [JP6.1](https://developer.nvidia.com/embedded/jetpack-sdk-61), [Seeed Studio reComputer J4012](https://www.seeedstudio.com/reComputer-J4012-p-5586.html) which is based on NVIDIA Jetson Orin NX 16GB running JetPack release of [JP6.0](https://developer.nvidia.com/embedded/jetpack-sdk-60)/ JetPack release of [JP5.1.3](https://developer.nvidia.com/embedded/jetpack-sdk-513) and [Seeed Studio reComputer J1020 v2](https://www.seeedstudio.com/reComputer-J1020-v2-p-5498.html) which is based on NVIDIA Jetson Nano 4GB running JetPack release of [JP4.6.1](https://developer.nvidia.com/embedded/jetpack-sdk-461). It is expected to work across all the NVIDIA Jetson hardware lineup, including the latest and legacy devices.
## What is NVIDIA Jetson?
NVIDIA Jetson is a series of embedded computing boards designed to bring accelerated AI (artificial intelligence) computing to edge devices. These compact and powerful devices are built around NVIDIA's GPU architecture and can run complex AI algorithms and [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) models directly on the device, without relying on [cloud computing](https://www.ultralytics.com/glossary/cloud-computing) resources. Jetson boards are often used in robotics, autonomous vehicles, industrial automation, and other applications where AI inference needs to be performed locally with low latency and high efficiency. Additionally, these boards are based on the ARM64 architecture and run at lower power compared to traditional GPU computing devices.
## NVIDIA Jetson Series Comparison
[NVIDIA Jetson AGX Thor](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/) is the latest iteration of the NVIDIA Jetson family based on NVIDIA Blackwell architecture which brings drastically improved AI performance when compared to the previous generations. The table below compares a few of the Jetson devices in the ecosystem.
| | Jetson AGX Thor(T5000) | Jetson AGX Orin 64GB | Jetson Orin NX 16GB | Jetson Orin Nano Super | Jetson AGX Xavier | Jetson Xavier NX | Jetson Nano |
| ----------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------- |
| AI Performance | 2070 TFLOPS | 275 TOPS | 100 TOPS | 67 TOPS | 32 TOPS | 21 TOPS | 472 GFLOPS |
| GPU | 2560-core NVIDIA Blackwell architecture GPU with 96 Tensor Cores | 2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores | 1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores | 1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores | 512-core NVIDIA Volta architecture GPU with 64 Tensor Cores | 384-core NVIDIA Volta™ architecture GPU with 48 Tensor Cores | 128-core NVIDIA Maxwell™ architecture GPU |
| GPU Max Frequency | 1.57 GHz | 1.3 GHz | 918 MHz | 1020 MHz | 1377 MHz | 1100 MHz | 921MHz |
| CPU | 14-core Arm® Neoverse®-V3AE 64-bit CPU 1MB L2 + 16MB L3 | 12-core NVIDIA Arm® Cortex A78AE v8.2 64-bit CPU 3MB L2 + 6MB L3 | 8-core NVIDIA Arm® Cortex A78AE v8.2 64-bit CPU 2MB L2 + 4MB L3 | 6-core Arm® Cortex®-A78AE v8.2 64-bit CPU 1.5MB L2 + 4MB L3 | 8-core NVIDIA Carmel Arm®v8.2 64-bit CPU 8MB L2 + 4MB L3 | 6-core NVIDIA Carmel Arm®v8.2 64-bit CPU 6MB L2 + 4MB L3 | Quad-Core Arm® Cortex®-A57 MPCore processor |
| CPU Max Frequency | 2.6 GHz | 2.2 GHz | 2.0 GHz | 1.7 GHz | 2.2 GHz | 1.9 GHz | 1.43GHz |
| Memory | 128GB 256-bit LPDDR5X 273GB/s | 64GB 256-bit LPDDR5 204.8GB/s | 16GB 128-bit LPDDR5 102.4GB/s | 8GB 128-bit LPDDR5 102 GB/s | 32GB 256-bit LPDDR4x 136.5GB/s | 8GB 128-bit LPDDR4x 59.7GB/s | 4GB 64-bit LPDDR4 25.6GB/s |
For a more detailed comparison table, please visit the **Compare Specifications** section of [official NVIDIA Jetson page](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems).
## What is NVIDIA JetPack?
[NVIDIA JetPack SDK](https://developer.nvidia.com/embedded/jetpack) powering the Jetson modules is the most comprehensive solution and provides full development environment for building end-to-end accelerated AI applications and shortens time to market. JetPack includes Jetson Linux with bootloader, Linux kernel, Ubuntu desktop environment, and a complete set of libraries for acceleration of GPU computing, multimedia, graphics, and [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv). It also includes samples, documentation, and developer tools for both host computer and developer kit, and supports higher level SDKs such as [DeepStream](https://docs.ultralytics.com/guides/deepstream-nvidia-jetson/) for streaming video analytics, Isaac for robotics, and Riva for conversational AI.
## Flash JetPack to NVIDIA Jetson
The first step after getting your hands on an NVIDIA Jetson device is to flash NVIDIA JetPack to the device. There are several different ways of flashing NVIDIA Jetson devices.
1. If you own an official NVIDIA Development Kit such as the Jetson AGX Thor Developer Kit, you can [download an image and prepare a bootable USB stick to flash JetPack to the included SSD](https://docs.nvidia.com/jetson/agx-thor-devkit/user-guide/latest/quick_start.html).
2. If you own an official NVIDIA Development Kit such as the Jetson Orin Nano Developer Kit, you can [download an image and prepare an SD card with JetPack for booting the device](https://developer.nvidia.com/embedded/learn/get-started-jetson-orin-nano-devkit).
3. If you own any other NVIDIA Development Kit, you can [flash JetPack to the device using SDK Manager](https://docs.nvidia.com/sdk-manager/install-with-sdkm-jetson/index.html).
4. If you own a Seeed Studio reComputer J4012 device, you can [flash JetPack to the included SSD](https://wiki.seeedstudio.com/reComputer_J4012_Flash_Jetpack/) and if you own a Seeed Studio reComputer J1020 v2 device, you can [flash JetPack to the eMMC/ SSD](https://wiki.seeedstudio.com/reComputer_J2021_J202_Flash_Jetpack/).
5. If you own any other third-party device powered by the NVIDIA Jetson module, it is recommended to follow [command-line flashing](https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/IN/QuickStart.html).
!!! note
For methods 1, 4 and 5 above, after flashing the system and booting the device, please enter "sudo apt update && sudo apt install nvidia-jetpack -y" on the device terminal to install all the remaining JetPack components needed.
## JetPack Support Based on Jetson Device
The below table highlights NVIDIA JetPack versions supported by different NVIDIA Jetson devices.
| | JetPack 4 | JetPack 5 | JetPack 6 | JetPack 7 |
| ----------------- | --------- | --------- | --------- | --------- |
| Jetson Nano | ✅ | ❌ | ❌ | ❌ |
| Jetson TX2 | ✅ | ❌ | ❌ | ❌ |
| Jetson Xavier NX | ✅ | ✅ | ❌ | ❌ |
| Jetson AGX Xavier | ✅ | ✅ | ❌ | ❌ |
| Jetson AGX Orin | ❌ | ✅ | ✅ | ❌ |
| Jetson Orin NX | ❌ | ✅ | ✅ | ❌ |
| Jetson Orin Nano | ❌ | ✅ | ✅ | ❌ |
| Jetson AGX Thor | ❌ | ❌ | ❌ | ✅ |
## Quick Start with Docker
The fastest way to get started with Ultralytics YOLO26 on NVIDIA Jetson is to run with pre-built docker images for Jetson. Refer to the table above and choose the JetPack version according to the Jetson device you own.
=== "JetPack 4"
```bash
t=ultralytics/ultralytics:latest-jetson-jetpack4
sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia $t
```
=== "JetPack 5"
```bash
t=ultralytics/ultralytics:latest-jetson-jetpack5
sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia $t
```
=== "JetPack 6"
```bash
t=ultralytics/ultralytics:latest-jetson-jetpack6
sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia $t
```
=== "JetPack 7"
```bash
t=ultralytics/ultralytics:latest-nvidia-arm64
sudo docker pull $t && sudo docker run -it --ipc=host --runtime=nvidia $t
```
After this is done, skip to [Use TensorRT on NVIDIA Jetson section](#use-tensorrt-on-nvidia-jetson).
## Start with Native Installation
For a native installation without Docker, please refer to the steps below.
### Run on JetPack 7.0
#### Install Ultralytics Package
Here we will install Ultralytics package on the Jetson with optional dependencies so that we can export the [PyTorch](https://www.ultralytics.com/glossary/pytorch) models to other different formats. We will mainly focus on [NVIDIA TensorRT exports](../integrations/tensorrt.md) because TensorRT will make sure we can get the maximum performance out of the Jetson devices.
1. Update packages list, install pip and upgrade to latest
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install `ultralytics` pip package with optional dependencies
```bash
pip install ultralytics[export]
```
3. Reboot the device
```bash
sudo reboot
```
#### Install PyTorch and Torchvision
The above ultralytics installation will install Torch and Torchvision. However, these 2 packages installed via pip are not compatible to run on Jetson AGX Thor which comes with JetPack 7.0 and CUDA 13. Therefore, we need to manually install them.
Install `torch` and `torchvision` according to JP7.0
```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
```
#### Install `onnxruntime-gpu`
The [onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu/) package hosted in PyPI does not have `aarch64` binaries for the Jetson. So we need to manually install this package. This package is needed for some of the exports.
Here we will download and install `onnxruntime-gpu 1.24.0` with `Python3.12` support.
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.24.0-cp312-cp312-linux_aarch64.whl
```
### Run on JetPack 6.1
#### Install Ultralytics Package
Here we will install Ultralytics package on the Jetson with optional dependencies so that we can export the [PyTorch](https://www.ultralytics.com/glossary/pytorch) models to other different formats. We will mainly focus on [NVIDIA TensorRT exports](../integrations/tensorrt.md) because TensorRT will make sure we can get the maximum performance out of the Jetson devices.
1. Update packages list, install pip and upgrade to latest
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install `ultralytics` pip package with optional dependencies
```bash
pip install ultralytics[export]
```
3. Reboot the device
```bash
sudo reboot
```
#### Install PyTorch and Torchvision
The above ultralytics installation will install Torch and Torchvision. However, these two packages installed via pip are not compatible with the Jetson platform, which is based on ARM64 architecture. Therefore, we need to manually install a pre-built PyTorch pip wheel and compile or install Torchvision from source.
Install `torch 2.5.0` and `torchvision 0.20` according to JP6.1
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.5.0a0+872d972e41.nv24.08-cp310-cp310-linux_aarch64.whl
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torchvision-0.20.0a0+afc54f7-cp310-cp310-linux_aarch64.whl
```
!!! note
Visit the [PyTorch for Jetson page](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048) to access all different versions of PyTorch for different JetPack versions. For a more detailed list on the PyTorch, Torchvision compatibility, visit the [PyTorch and Torchvision compatibility page](https://github.com/pytorch/vision).
Install [`cuSPARSELt`](https://developer.nvidia.com/cusparselt-downloads?target_os=Linux&target_arch=aarch64-jetson&Compilation=Native&Distribution=Ubuntu&target_version=22.04&target_type=deb_network) to fix a dependency issue with `torch 2.5.0`
```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install libcusparselt0 libcusparselt-dev
```
#### Install `onnxruntime-gpu`
The [onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu/) package hosted in PyPI does not have `aarch64` binaries for the Jetson. So we need to manually install this package. This package is needed for some of the exports.
You can find all available `onnxruntime-gpu` packages—organized by JetPack version, Python version, and other compatibility details—in the [Jetson Zoo ONNX Runtime compatibility matrix](https://elinux.org/Jetson_Zoo#ONNX_Runtime).
For **JetPack 6** with `Python 3.10` support, you can install `onnxruntime-gpu 1.23.0`:
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.23.0-cp310-cp310-linux_aarch64.whl
```
Alternatively, for `onnxruntime-gpu 1.20.0`:
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl
```
### Run on JetPack 5.1.2
#### Install Ultralytics Package
Here we will install Ultralytics package on the Jetson with optional dependencies so that we can export the PyTorch models to other different formats. We will mainly focus on [NVIDIA TensorRT exports](../integrations/tensorrt.md) because TensorRT will make sure we can get the maximum performance out of the Jetson devices.
1. Update packages list, install pip and upgrade to latest
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install `ultralytics` pip package with optional dependencies
```bash
pip install ultralytics[export]
```
3. Reboot the device
```bash
sudo reboot
```
#### Install PyTorch and Torchvision
The above ultralytics installation will install Torch and Torchvision. However, these two packages installed via pip are not compatible with the Jetson platform, which is based on ARM64 architecture. Therefore, we need to manually install a pre-built PyTorch pip wheel and compile or install Torchvision from source.
1. Uninstall currently installed PyTorch and Torchvision
```bash
pip uninstall torch torchvision
```
2. Install `torch 2.2.0` and `torchvision 0.17.2` according to JP5.1.2
```bash
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torch-2.2.0-cp38-cp38-linux_aarch64.whl
pip install https://github.com/ultralytics/assets/releases/download/v0.0.0/torchvision-0.17.2+c1d70fe-cp38-cp38-linux_aarch64.whl
```
!!! note
Visit the [PyTorch for Jetson page](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048) to access all different versions of PyTorch for different JetPack versions. For a more detailed list on the PyTorch, Torchvision compatibility, visit the [PyTorch and Torchvision compatibility page](https://github.com/pytorch/vision).
#### Install `onnxruntime-gpu`
The [onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu/) package hosted in PyPI does not have `aarch64` binaries for the Jetson. So we need to manually install this package. This package is needed for some of the exports.
You can find all available `onnxruntime-gpu` packages—organized by JetPack version, Python version, and other compatibility details—in the [Jetson Zoo ONNX Runtime compatibility matrix](https://elinux.org/Jetson_Zoo#ONNX_Runtime). Here we will download and install `onnxruntime-gpu 1.17.0` with `Python3.8` support.
```bash
wget https://nvidia.box.com/shared/static/zostg6agm00fb6t5uisw51qi6kpcuwzd.whl -O onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl
pip install onnxruntime_gpu-1.17.0-cp38-cp38-linux_aarch64.whl
```
!!! note
`onnxruntime-gpu` will automatically revert back the numpy version to latest. So we need to reinstall numpy to `1.23.5` to fix an issue by executing:
`pip install numpy==1.23.5`
## Use TensorRT on NVIDIA Jetson
Among all the model export formats supported by Ultralytics, TensorRT offers the highest inference performance on NVIDIA Jetson devices, making it our top recommendation for Jetson deployments. For setup instructions and advanced usage, see our [dedicated TensorRT integration guide](../integrations/tensorrt.md).
### Convert Model to TensorRT and Run Inference
The YOLO26n model in PyTorch format is converted to TensorRT to run inference with the exported model.
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Export the model to TensorRT
model.export(format="engine") # creates 'yolo26n.engine'
# Load the exported TensorRT model
trt_model = YOLO("yolo26n.engine")
# Run inference
results = trt_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO26n PyTorch model to TensorRT format
yolo export model=yolo26n.pt format=engine # creates 'yolo26n.engine'
# Run inference with the exported model
yolo predict model=yolo26n.engine source='https://ultralytics.com/images/bus.jpg'
```
!!! note
Visit the [Export page](../modes/export.md#arguments) to access additional arguments when exporting models to different model formats
### Use NVIDIA Deep Learning Accelerator (DLA)
[NVIDIA Deep Learning Accelerator (DLA)](https://developer.nvidia.com/deep-learning-accelerator) is a specialized hardware component built into NVIDIA Jetson devices that optimizes deep learning inference for energy efficiency and performance. By offloading tasks from the GPU (freeing it up for more intensive processes), DLA enables models to run with lower power consumption while maintaining high throughput, ideal for embedded systems and real-time AI applications.
The following Jetson devices are equipped with DLA hardware:
| Jetson Device | DLA Cores | DLA Max Frequency |
| ------------------------ | --------- | ----------------- |
| Jetson AGX Orin Series | 2 | 1.6 GHz |
| Jetson Orin NX 16GB | 2 | 614 MHz |
| Jetson Orin NX 8GB | 1 | 614 MHz |
| Jetson AGX Xavier Series | 2 | 1.4 GHz |
| Jetson Xavier NX Series | 2 | 1.1 GHz |
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Export the model to TensorRT with DLA enabled (only works with FP16 or INT8)
model.export(format="engine", device="dla:0", half=True) # dla:0 or dla:1 corresponds to the DLA cores
# Load the exported TensorRT model
trt_model = YOLO("yolo26n.engine")
# Run inference
results = trt_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO26n PyTorch model to TensorRT format with DLA enabled (only works with FP16 or INT8)
# Once DLA core number is specified at export, it will use the same core at inference
yolo export model=yolo26n.pt format=engine device="dla:0" half=True # dla:0 or dla:1 corresponds to the DLA cores
# Run inference with the exported model on the DLA
yolo predict model=yolo26n.engine source='https://ultralytics.com/images/bus.jpg'
```
!!! note
When using DLA exports, some layers may not be supported to run on DLA and will fall back to the GPU for execution. This fallback can introduce additional latency and impact the overall inference performance. Therefore, DLA is not primarily designed to reduce inference latency compared to TensorRT running entirely on the GPU. Instead, its primary purpose is to increase throughput and improve energy efficiency.
## NVIDIA Jetson YOLO11/ YOLO26 Benchmarks
YOLO11/ YOLO26 benchmarks were run by the Ultralytics team on 11 different model formats measuring speed and [accuracy](https://www.ultralytics.com/glossary/accuracy): PyTorch, TorchScript, ONNX, OpenVINO, TensorRT, TF SavedModel, TF GraphDef, TF Lite, MNN, NCNN, ExecuTorch. Benchmarks were run on NVIDIA Jetson AGX Thor Developer Kit, NVIDIA Jetson AGX Orin Developer Kit (64GB), NVIDIA Jetson Orin Nano Super Developer Kit and Seeed Studio reComputer J4012 powered by Jetson Orin NX 16GB device at FP32 [precision](https://www.ultralytics.com/glossary/precision) with default input image size of 640.
### Comparison Charts
Even though all model exports work on NVIDIA Jetson, we have only included **PyTorch, TorchScript, TensorRT** for the comparison chart below because they make use of the GPU on the Jetson and are guaranteed to produce the best results. All the other exports only utilize the CPU and the performance is not as good as the above three. You can find benchmarks for all exports in the section after this chart.
#### NVIDIA Jetson AGX Thor Developer Kit
<figure style="text-align: center;">
<img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-agx-thor-benchmarks-coco128.avif" alt="Jetson AGX Thor Benchmarks">
<figcaption style="font-style: italic; color: gray;">Benchmarked with Ultralytics 8.3.226</figcaption>
</figure>
#### NVIDIA Jetson AGX Orin Developer Kit (64GB)
<figure style="text-align: center;">
<img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-agx-orin-benchmarks-coco128.avif" alt="Jetson AGX Orin Benchmarks">
<figcaption style="font-style: italic; color: gray;">Benchmarked with Ultralytics 8.3.157</figcaption>
</figure>
#### NVIDIA Jetson Orin Nano Super Developer Kit
<figure style="text-align: center;">
<img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-orin-nano-super-benchmarks-coco128.avif" alt="Jetson Orin Nano Super Benchmarks">
<figcaption style="font-style: italic; color: gray;">Benchmarked with Ultralytics 8.3.157</figcaption>
</figure>
#### NVIDIA Jetson Orin NX 16GB
<figure style="text-align: center;">
<img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-orin-nx-16-benchmarks-coco128.avif" alt="Jetson Orin NX 16GB Benchmarks">
<figcaption style="font-style: italic; color: gray;">Benchmarked with Ultralytics 8.3.157</figcaption>
</figure>
### Detailed Comparison Tables
The below table represents the benchmark results for five different models (YOLO11n, YOLO11s, YOLO11m, YOLO11l, YOLO11x) across 11 different formats (PyTorch, TorchScript, ONNX, OpenVINO, TensorRT, TF SavedModel, TF GraphDef, TF Lite, MNN, NCNN, ExecuTorch), giving us the status, size, mAP50-95(B) metric, and inference time for each combination.
#### NVIDIA Jetson AGX Thor Developer Kit
!!! tip "Performance"
=== "YOLO26n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.3 | 0.4798 | 7.39 |
| TorchScript | ✅ | 9.8 | 0.4789 | 4.21 |
| ONNX | ✅ | 9.5 | 0.4767 | 6.58 |
| OpenVINO | ✅ | 10.1 | 0.4794 | 17.50 |
| TensorRT (FP32) | ✅ | 13.9 | 0.4791 | 1.90 |
| TensorRT (FP16) | ✅ | 7.6 | 0.4797 | 1.39 |
| TensorRT (INT8) | ✅ | 6.5 | 0.4273 | 1.52 |
| TF SavedModel | ✅ | 25.7 | 0.4764 | 47.24 |
| TF GraphDef | ✅ | 9.5 | 0.4764 | 45.98 |
| TF Lite | ✅ | 9.9 | 0.4764 | 182.04 |
| MNN | ✅ | 9.4 | 0.4784 | 21.83 |
=== "YOLO26s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 19.5 | 0.5738 | 7.99 |
| TorchScript | ✅ | 36.8 | 0.5664 | 6.01 |
| ONNX | ✅ | 36.5 | 0.5666 | 9.31 |
| OpenVINO | ✅ | 38.5 | 0.5656 | 35.56 |
| TensorRT (FP32) | ✅ | 38.9 | 0.5664 | 2.95 |
| TensorRT (FP16) | ✅ | 21.0 | 0.5650 | 1.77 |
| TensorRT (INT8) | ✅ | 13.5 | 0.5010 | 1.75 |
| TF SavedModel | ✅ | 96.6 | 0.5665 | 88.87 |
| TF GraphDef | ✅ | 36.5 | 0.5665 | 89.20 |
| TF Lite | ✅ | 36.9 | 0.5665 | 604.25 |
| MNN | ✅ | 36.4 | 0.5651 | 53.75 |
=== "YOLO26m"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 42.2 | 0.6237 | 10.76 |
| TorchScript | ✅ | 78.5 | 0.6217 | 10.57 |
| ONNX | ✅ | 78.2 | 0.6211 | 14.91 |
| OpenVINO | ✅ | 82.2 | 0.6204 | 86.27 |
| TensorRT (FP32) | ✅ | 82.2 | 0.6230 | 5.56 |
| TensorRT (FP16) | ✅ | 41.6 | 0.6209 | 2.58 |
| TensorRT (INT8) | ✅ | 24.3 | 0.5595 | 2.49 |
| TF SavedModel | ✅ | 205.8 | 0.6229 | 200.96 |
| TF GraphDef | ✅ | 78.2 | 0.6229 | 203.00 |
| TF Lite | ✅ | 78.6 | 0.6229 | 1867.12 |
| MNN | ✅ | 78.0 | 0.6176 | 142.00 |
=== "YOLO26l"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 50.7 | 0.6258 | 13.34 |
| TorchScript | ✅ | 95.5 | 0.6248 | 13.86 |
| ONNX | ✅ | 95.0 | 0.6247 | 18.44 |
| OpenVINO | ✅ | 99.9 | 0.6238 | 106.67 |
| TensorRT (FP32) | ✅ | 99.0 | 0.6249 | 6.74 |
| TensorRT (FP16) | ✅ | 50.3 | 0.6243 | 3.34 |
| TensorRT (INT8) | ✅ | 29.0 | 0.5708 | 3.24 |
| TF SavedModel | ✅ | 250.0 | 0.6245 | 259.74 |
| TF GraphDef | ✅ | 95.0 | 0.6245 | 263.42 |
| TF Lite | ✅ | 95.4 | 0.6245 | 2367.83 |
| MNN | ✅ | 94.8 | 0.6272 | 174.39 |
=== "YOLO26x"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 113.2 | 0.6565 | 20.92 |
| TorchScript | ✅ | 213.5 | 0.6595 | 21.76 |
| ONNX | ✅ | 212.9 | 0.6590 | 26.72 |
| OpenVINO | ✅ | 223.6 | 0.6620 | 205.27 |
| TensorRT (FP32) | ✅ | 217.2 | 0.6593 | 12.29 |
| TensorRT (FP16) | ✅ | 112.1 | 0.6611 | 5.16 |
| TensorRT (INT8) | ✅ | 58.9 | 0.5222 | 4.72 |
| TF SavedModel | ✅ | 559.2 | 0.6593 | 498.85 |
| TF GraphDef | ✅ | 213.0 | 0.6593 | 507.43 |
| TF Lite | ✅ | 213.3 | 0.6593 | 5134.22 |
| MNN | ✅ | 212.8 | 0.6625 | 347.84 |
Benchmarked with Ultralytics 8.4.7
!!! note
Inference time does not include pre/ post-processing.
#### NVIDIA Jetson AGX Orin Developer Kit (64GB)
!!! tip "Performance"
=== "YOLO11n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.4 | 0.5101 | 9.40 |
| TorchScript | ✅ | 10.5 | 0.5083 | 11.00 |
| ONNX | ✅ | 10.2 | 0.5077 | 48.32 |
| OpenVINO | ✅ | 10.4 | 0.5058 | 27.24 |
| TensorRT (FP32) | ✅ | 12.1 | 0.5085 | 3.93 |
| TensorRT (FP16) | ✅ | 8.3 | 0.5063 | 2.55 |
| TensorRT (INT8) | ✅ | 5.4 | 0.4719 | 2.18 |
| TF SavedModel | ✅ | 25.9 | 0.5077 | 66.87 |
| TF GraphDef | ✅ | 10.3 | 0.5077 | 65.68 |
| TF Lite | ✅ | 10.3 | 0.5077 | 272.92 |
| MNN | ✅ | 10.1 | 0.5059 | 36.33 |
| NCNN | ✅ | 10.2 | 0.5031 | 28.51 |
=== "YOLO11s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 18.4 | 0.5783 | 12.10 |
| TorchScript | ✅ | 36.5 | 0.5782 | 11.01 |
| ONNX | ✅ | 36.3 | 0.5782 | 107.54 |
| OpenVINO | ✅ | 36.4 | 0.5810 | 55.03 |
| TensorRT (FP32) | ✅ | 38.1 | 0.5781 | 6.52 |
| TensorRT (FP16) | ✅ | 21.4 | 0.5803 | 3.65 |
| TensorRT (INT8) | ✅ | 12.1 | 0.5735 | 2.81 |
| TF SavedModel | ✅ | 91.0 | 0.5782 | 132.73 |
| TF GraphDef | ✅ | 36.4 | 0.5782 | 134.96 |
| TF Lite | ✅ | 36.3 | 0.5782 | 798.21 |
| MNN | ✅ | 36.2 | 0.5777 | 82.35 |
| NCNN | ✅ | 36.2 | 0.5784 | 56.07 |
=== "YOLO11m"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 38.8 | 0.6265 | 22.20 |
| TorchScript | ✅ | 77.3 | 0.6307 | 21.47 |
| ONNX | ✅ | 76.9 | 0.6307 | 270.89 |
| OpenVINO | ✅ | 77.1 | 0.6284 | 129.10 |
| TensorRT (FP32) | ✅ | 78.8 | 0.6306 | 12.53 |
| TensorRT (FP16) | ✅ | 41.9 | 0.6305 | 6.25 |
| TensorRT (INT8) | ✅ | 23.2 | 0.6291 | 4.69 |
| TF SavedModel | ✅ | 192.7 | 0.6307 | 299.95 |
| TF GraphDef | ✅ | 77.1 | 0.6307 | 310.58 |
| TF Lite | ✅ | 77.0 | 0.6307 | 2400.54 |
| MNN | ✅ | 76.8 | 0.6308 | 213.56 |
| NCNN | ✅ | 76.8 | 0.6284 | 141.18 |
=== "YOLO11l"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 49.0 | 0.6364 | 27.70 |
| TorchScript | ✅ | 97.6 | 0.6399 | 27.94 |
| ONNX | ✅ | 97.0 | 0.6409 | 345.47 |
| OpenVINO | ✅ | 97.3 | 0.6378 | 161.93 |
| TensorRT (FP32) | ✅ | 99.1 | 0.6406 | 16.11 |
| TensorRT (FP16) | ✅ | 52.6 | 0.6376 | 8.08 |
| TensorRT (INT8) | ✅ | 30.8 | 0.6208 | 6.12 |
| TF SavedModel | ✅ | 243.1 | 0.6409 | 390.78 |
| TF GraphDef | ✅ | 97.2 | 0.6409 | 398.76 |
| TF Lite | ✅ | 97.1 | 0.6409 | 3037.05 |
| MNN | ✅ | 96.9 | 0.6372 | 265.46 |
| NCNN | ✅ | 96.9 | 0.6364 | 179.68 |
=== "YOLO11x"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 109.3 | 0.7005 | 44.40 |
| TorchScript | ✅ | 218.1 | 0.6898 | 47.49 |
| ONNX | ✅ | 217.5 | 0.6900 | 682.98 |
| OpenVINO | ✅ | 217.8 | 0.6876 | 298.15 |
| TensorRT (FP32) | ✅ | 219.6 | 0.6904 | 28.50 |
| TensorRT (FP16) | ✅ | 112.2 | 0.6887 | 13.55 |
| TensorRT (INT8) | ✅ | 60.0 | 0.6574 | 9.40 |
| TF SavedModel | ✅ | 544.3 | 0.6900 | 749.85 |
| TF GraphDef | ✅ | 217.7 | 0.6900 | 753.86 |
| TF Lite | ✅ | 217.6 | 0.6900 | 6603.27 |
| MNN | ✅ | 217.3 | 0.6868 | 519.77 |
| NCNN | ✅ | 217.3 | 0.6849 | 298.58 |
Benchmarked with Ultralytics 8.3.157
!!! note
Inference time does not include pre/ post-processing.
#### NVIDIA Jetson Orin Nano Super Developer Kit
!!! tip "Performance"
=== "YOLO11n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.4 | 0.5101 | 13.70 |
| TorchScript | ✅ | 10.5 | 0.5082 | 13.69 |
| ONNX | ✅ | 10.2 | 0.5081 | 14.47 |
| OpenVINO | ✅ | 10.4 | 0.5058 | 56.66 |
| TensorRT (FP32) | ✅ | 12.0 | 0.5081 | 7.44 |
| TensorRT (FP16) | ✅ | 8.2 | 0.5061 | 4.53 |
| TensorRT (INT8) | ✅ | 5.4 | 0.4825 | 3.70 |
| TF SavedModel | ✅ | 25.9 | 0.5077 | 116.23 |
| TF GraphDef | ✅ | 10.3 | 0.5077 | 114.92 |
| TF Lite | ✅ | 10.3 | 0.5077 | 340.75 |
| MNN | ✅ | 10.1 | 0.5059 | 76.26 |
| NCNN | ✅ | 10.2 | 0.5031 | 45.03 |
=== "YOLO11s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 18.4 | 0.5790 | 20.90 |
| TorchScript | ✅ | 36.5 | 0.5781 | 21.22 |
| ONNX | ✅ | 36.3 | 0.5781 | 25.07 |
| OpenVINO | ✅ | 36.4 | 0.5810 | 122.98 |
| TensorRT (FP32) | ✅ | 37.9 | 0.5783 | 13.02 |
| TensorRT (FP16) | ✅ | 21.8 | 0.5779 | 6.93 |
| TensorRT (INT8) | ✅ | 12.2 | 0.5735 | 5.08 |
| TF SavedModel | ✅ | 91.0 | 0.5782 | 250.65 |
| TF GraphDef | ✅ | 36.4 | 0.5782 | 252.69 |
| TF Lite | ✅ | 36.3 | 0.5782 | 998.68 |
| MNN | ✅ | 36.2 | 0.5781 | 188.01 |
| NCNN | ✅ | 36.2 | 0.5784 | 101.37 |
=== "YOLO11m"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 38.8 | 0.6266 | 46.50 |
| TorchScript | ✅ | 77.3 | 0.6307 | 47.95 |
| ONNX | ✅ | 76.9 | 0.6307 | 53.06 |
| OpenVINO | ✅ | 77.1 | 0.6284 | 301.63 |
| TensorRT (FP32) | ✅ | 78.8 | 0.6305 | 27.86 |
| TensorRT (FP16) | ✅ | 41.7 | 0.6309 | 13.50 |
| TensorRT (INT8) | ✅ | 23.2 | 0.6291 | 9.12 |
| TF SavedModel | ✅ | 192.7 | 0.6307 | 622.24 |
| TF GraphDef | ✅ | 77.1 | 0.6307 | 628.74 |
| TF Lite | ✅ | 77.0 | 0.6307 | 2997.93 |
| MNN | ✅ | 76.8 | 0.6299 | 509.96 |
| NCNN | ✅ | 76.8 | 0.6284 | 292.99 |
=== "YOLO11l"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 49.0 | 0.6364 | 56.50 |
| TorchScript | ✅ | 97.6 | 0.6409 | 62.51 |
| ONNX | ✅ | 97.0 | 0.6399 | 68.35 |
| OpenVINO | ✅ | 97.3 | 0.6378 | 376.03 |
| TensorRT (FP32) | ✅ | 99.2 | 0.6396 | 35.59 |
| TensorRT (FP16) | ✅ | 52.1 | 0.6361 | 17.48 |
| TensorRT (INT8) | ✅ | 30.9 | 0.6207 | 11.87 |
| TF SavedModel | ✅ | 243.1 | 0.6409 | 807.47 |
| TF GraphDef | ✅ | 97.2 | 0.6409 | 822.88 |
| TF Lite | ✅ | 97.1 | 0.6409 | 3792.23 |
| MNN | ✅ | 96.9 | 0.6372 | 631.16 |
| NCNN | ✅ | 96.9 | 0.6364 | 350.46 |
=== "YOLO11x"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 109.3 | 0.7005 | 90.00 |
| TorchScript | ✅ | 218.1 | 0.6901 | 113.40 |
| ONNX | ✅ | 217.5 | 0.6901 | 122.94 |
| OpenVINO | ✅ | 217.8 | 0.6876 | 713.1 |
| TensorRT (FP32) | ✅ | 219.5 | 0.6904 | 66.93 |
| TensorRT (FP16) | ✅ | 112.2 | 0.6892 | 32.58 |
| TensorRT (INT8) | ✅ | 61.5 | 0.6612 | 19.90 |
| TF SavedModel | ✅ | 544.3 | 0.6900 | 1605.4 |
| TF GraphDef | ✅ | 217.8 | 0.6900 | 2961.8 |
| TF Lite | ✅ | 217.6 | 0.6900 | 8234.86 |
| MNN | ✅ | 217.3 | 0.6893 | 1254.18 |
| NCNN | ✅ | 217.3 | 0.6849 | 725.50 |
Benchmarked with Ultralytics 8.3.157
!!! note
Inference time does not include pre/ post-processing.
#### NVIDIA Jetson Orin NX 16GB
!!! tip "Performance"
=== "YOLO11n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.4 | 0.5101 | 12.90 |
| TorchScript | ✅ | 10.5 | 0.5082 | 13.17 |
| ONNX | ✅ | 10.2 | 0.5081 | 15.43 |
| OpenVINO | ✅ | 10.4 | 0.5058 | 39.80 |
| TensorRT (FP32) | ✅ | 11.8 | 0.5081 | 7.94 |
| TensorRT (FP16) | ✅ | 8.1 | 0.5085 | 4.73 |
| TensorRT (INT8) | ✅ | 5.4 | 0.4786 | 3.90 |
| TF SavedModel | ✅ | 25.9 | 0.5077 | 88.48 |
| TF GraphDef | ✅ | 10.3 | 0.5077 | 86.67 |
| TF Lite | ✅ | 10.3 | 0.5077 | 302.55 |
| MNN | ✅ | 10.1 | 0.5059 | 52.73 |
| NCNN | ✅ | 10.2 | 0.5031 | 32.04 |
=== "YOLO11s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 18.4 | 0.5790 | 21.70 |
| TorchScript | ✅ | 36.5 | 0.5781 | 22.71 |
| ONNX | ✅ | 36.3 | 0.5781 | 26.49 |
| OpenVINO | ✅ | 36.4 | 0.5810 | 84.73 |
| TensorRT (FP32) | ✅ | 37.8 | 0.5783 | 13.77 |
| TensorRT (FP16) | ✅ | 21.2 | 0.5796 | 7.31 |
| TensorRT (INT8) | ✅ | 12.0 | 0.5735 | 5.33 |
| TF SavedModel | ✅ | 91.0 | 0.5782 | 185.06 |
| TF GraphDef | ✅ | 36.4 | 0.5782 | 186.45 |
| TF Lite | ✅ | 36.3 | 0.5782 | 882.58 |
| MNN | ✅ | 36.2 | 0.5775 | 126.36 |
| NCNN | ✅ | 36.2 | 0.5784 | 66.73 |
=== "YOLO11m"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 38.8 | 0.6266 | 45.00 |
| TorchScript | ✅ | 77.3 | 0.6307 | 51.87 |
| ONNX | ✅ | 76.9 | 0.6307 | 56.00 |
| OpenVINO | ✅ | 77.1 | 0.6284 | 202.69 |
| TensorRT (FP32) | ✅ | 78.7 | 0.6305 | 30.38 |
| TensorRT (FP16) | ✅ | 41.8 | 0.6302 | 14.48 |
| TensorRT (INT8) | ✅ | 23.2 | 0.6291 | 9.74 |
| TF SavedModel | ✅ | 192.7 | 0.6307 | 445.58 |
| TF GraphDef | ✅ | 77.1 | 0.6307 | 460.94 |
| TF Lite | ✅ | 77.0 | 0.6307 | 2653.65 |
| MNN | ✅ | 76.8 | 0.6308 | 339.38 |
| NCNN | ✅ | 76.8 | 0.6284 | 187.64 |
=== "YOLO11l"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 49.0 | 0.6364 | 56.60 |
| TorchScript | ✅ | 97.6 | 0.6409 | 66.72 |
| ONNX | ✅ | 97.0 | 0.6399 | 71.92 |
| OpenVINO | ✅ | 97.3 | 0.6378 | 254.17 |
| TensorRT (FP32) | ✅ | 99.2 | 0.6406 | 38.89 |
| TensorRT (FP16) | ✅ | 51.9 | 0.6363 | 18.59 |
| TensorRT (INT8) | ✅ | 30.9 | 0.6207 | 12.60 |
| TF SavedModel | ✅ | 243.1 | 0.6409 | 575.98 |
| TF GraphDef | ✅ | 97.2 | 0.6409 | 583.79 |
| TF Lite | ✅ | 97.1 | 0.6409 | 3353.41 |
| MNN | ✅ | 96.9 | 0.6367 | 421.33 |
| NCNN | ✅ | 96.9 | 0.6364 | 228.26 |
=== "YOLO11x"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|-----------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 109.3 | 0.7005 | 98.50 |
| TorchScript | ✅ | 218.1 | 0.6901 | 123.03 |
| ONNX | ✅ | 217.5 | 0.6901 | 129.55 |
| OpenVINO | ✅ | 217.8 | 0.6876 | 483.44 |
| TensorRT (FP32) | ✅ | 219.6 | 0.6904 | 75.92 |
| TensorRT (FP16) | ✅ | 112.1 | 0.6885 | 35.78 |
| TensorRT (INT8) | ✅ | 61.6 | 0.6592 | 21.60 |
| TF SavedModel | ✅ | 544.3 | 0.6900 | 1120.43 |
| TF GraphDef | ✅ | 217.7 | 0.6900 | 1172.35 |
| TF Lite | ✅ | 217.6 | 0.6900 | 7283.63 |
| MNN | ✅ | 217.3 | 0.6877 | 840.16 |
| NCNN | ✅ | 217.3 | 0.6849 | 474.41 |
Benchmarked with Ultralytics 8.3.157
!!! note
Inference time does not include pre/ post-processing.
[Explore more benchmarking efforts by Seeed Studio](https://www.seeedstudio.com/blog/2023/03/30/yolov8-performance-benchmarks-on-nvidia-jetson-devices/) running on different versions of NVIDIA Jetson hardware.
## Reproduce Our Results
To reproduce the above Ultralytics benchmarks on all export [formats](../modes/export.md) run this code:
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO11n PyTorch model
model = YOLO("yolo11n.pt")
# Benchmark YOLO11n speed and accuracy on the COCO128 dataset for all export formats
results = model.benchmark(data="coco128.yaml", imgsz=640)
```
=== "CLI"
```bash
# Benchmark YOLO11n speed and accuracy on the COCO128 dataset for all export formats
yolo benchmark model=yolo11n.pt data=coco128.yaml imgsz=640
```
Note that benchmarking results might vary based on the exact hardware and software configuration of a system, as well as the current workload of the system at the time the benchmarks are run. For the most reliable results, use a dataset with a large number of images, e.g., `data='coco.yaml'` (5000 val images).
## Best Practices when using NVIDIA Jetson
When using NVIDIA Jetson, there are a couple of best practices to follow in order to enable maximum performance on the NVIDIA Jetson running YOLO26.
1. Enable MAX Power Mode
Enabling MAX Power Mode on the Jetson will make sure all CPU, GPU cores are turned on.
```bash
sudo nvpmodel -m 0
```
2. Enable Jetson Clocks
Enabling Jetson Clocks will make sure all CPU, GPU cores are clocked at their maximum frequency.
```bash
sudo jetson_clocks
```
3. Install Jetson Stats Application
We can use jetson stats application to monitor the temperatures of the system components and check other system details such as view CPU, GPU, RAM utilization, change power modes, set to max clocks, check JetPack information
```bash
sudo apt update
sudo pip install jetson-stats
sudo reboot
jtop
```
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/jetson-stats-application.avif" alt="Jetson Stats">
## Next Steps
For further learning and support, see the [Ultralytics YOLO26 Docs](../index.md).
## FAQ
### How do I deploy Ultralytics YOLO26 on NVIDIA Jetson devices?
Deploying Ultralytics YOLO26 on NVIDIA Jetson devices is a straightforward process. First, flash your Jetson device with the NVIDIA JetPack SDK. Then, either use a pre-built Docker image for quick setup or manually install the required packages. Detailed steps for each approach can be found in sections [Quick Start with Docker](#quick-start-with-docker) and [Start with Native Installation](#start-with-native-installation).
### What performance benchmarks can I expect from YOLO11 models on NVIDIA Jetson devices?
YOLO11 models have been benchmarked on various NVIDIA Jetson devices showing significant performance improvements. For example, the TensorRT format delivers the best inference performance. The table in the [Detailed Comparison Tables](#detailed-comparison-tables) section provides a comprehensive view of performance metrics like mAP50-95 and inference time across different model formats.
### Why should I use TensorRT for deploying YOLO26 on NVIDIA Jetson?
TensorRT is highly recommended for deploying YOLO26 models on NVIDIA Jetson due to its optimal performance. It accelerates inference by leveraging the Jetson's GPU capabilities, ensuring maximum efficiency and speed. Learn more about how to convert to TensorRT and run inference in the [Use TensorRT on NVIDIA Jetson](#use-tensorrt-on-nvidia-jetson) section.
### How can I install PyTorch and Torchvision on NVIDIA Jetson?
To install PyTorch and Torchvision on NVIDIA Jetson, first uninstall any existing versions that may have been installed via pip. Then, manually install the compatible PyTorch and Torchvision versions for the Jetson's ARM64 architecture. Detailed instructions for this process are provided in the [Install PyTorch and Torchvision](#install-pytorch-and-torchvision) section.
### What are the best practices for maximizing performance on NVIDIA Jetson when using YOLO26?
To maximize performance on NVIDIA Jetson with YOLO26, follow these best practices:
1. Enable MAX Power Mode to utilize all CPU and GPU cores.
2. Enable Jetson Clocks to run all cores at their maximum frequency.
3. Install the Jetson Stats application for monitoring system metrics.
For commands and additional details, refer to the [Best Practices when using NVIDIA Jetson](#best-practices-when-using-nvidia-jetson) section.

187
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/object-blurring.md Normal file
View File

@@ -0,0 +1,187 @@
---
comments: true
description: Learn how to use Ultralytics YOLO26 for real-time object blurring to enhance privacy and focus in your images and videos.
keywords: YOLO26, object blurring, real-time processing, privacy protection, image manipulation, video editing, Ultralytics
---
# Object Blurring using Ultralytics YOLO26 🚀
## What is Object Blurring?
Object blurring with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves applying a blurring effect to specific detected objects in an image or video. This can be achieved using the YOLO26 model capabilities to identify and manipulate objects within a given scene.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/J1BaCqytBmA"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Object Blurring using Ultralytics YOLO26
</p>
## Advantages of Object Blurring
- **Privacy Protection**: Object blurring is an effective tool for safeguarding privacy by concealing sensitive or personally identifiable information in images or videos.
- **Selective Focus**: YOLO26 allows for selective blurring, enabling users to target specific objects, ensuring a balance between privacy and retaining relevant visual information.
- **Real-time Processing**: YOLO26's efficiency enables object blurring in real-time, making it suitable for applications requiring on-the-fly privacy enhancements in dynamic environments.
- **Regulatory Compliance**: Helps organizations comply with data protection regulations like GDPR by anonymizing identifiable information in visual content.
- **Content Moderation**: Useful for blurring inappropriate or sensitive content in media platforms while preserving the overall context.
!!! example "Object Blurring using Ultralytics YOLO"
=== "CLI"
```bash
# Blur the objects
yolo solutions blur show=True
# Pass a source video
yolo solutions blur source="path/to/video.mp4"
# Blur the specific classes
yolo solutions blur classes="[0, 5]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("object_blurring_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize object blurrer
blurrer = solutions.ObjectBlurrer(
show=True, # display the output
model="yolo26n.pt", # model for object blurring, e.g., yolo26m.pt
# line_width=2, # width of bounding box.
# classes=[0, 2], # blur specific classes, e.g., person and car with the COCO pretrained model.
# blur_ratio=0.5, # adjust percentage of blur intensity, value in range 0.1 - 1.0
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = blurrer(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `ObjectBlurrer` Arguments
Here's a table with the `ObjectBlurrer` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "line_width", "blur_ratio"]) }}
The `ObjectBlurrer` solution also supports a range of `track` arguments:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization arguments can be used:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## Real-World Applications
### Privacy Protection in Surveillance
[Security cameras](https://www.ultralytics.com/blog/the-cutting-edge-world-of-ai-security-cameras) and surveillance systems can use YOLO26 to automatically blur faces, license plates, or other identifying information while still capturing important activity. This helps maintain security while respecting privacy rights in public spaces.
### Healthcare Data Anonymization
In [medical imaging](https://www.ultralytics.com/blog/ai-and-radiology-a-new-era-of-precision-and-efficiency), patient information often appears in scans or photos. YOLO26 can detect and blur this information to comply with regulations like HIPAA when sharing medical data for research or educational purposes.
### Document Redaction
When sharing documents that contain sensitive information, YOLO26 can automatically detect and blur specific elements like signatures, account numbers, or personal details, streamlining the redaction process while maintaining document integrity.
### Media and Content Creation
Content creators can use YOLO26 to blur brand logos, copyrighted material, or inappropriate content in videos and images, helping avoid legal issues while preserving the overall content quality.
## FAQ
### What is object blurring with Ultralytics YOLO26?
Object blurring with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves automatically detecting and applying a blurring effect to specific objects in images or videos. This technique enhances privacy by concealing sensitive information while retaining relevant visual data. YOLO26's real-time processing capabilities make it suitable for applications requiring immediate privacy protection and selective focus adjustments.
### How can I implement real-time object blurring using YOLO26?
To implement real-time object blurring with YOLO26, follow the provided Python example. This involves using YOLO26 for [object detection](https://www.ultralytics.com/glossary/object-detection) and OpenCV for applying the blur effect. Here's a simplified version:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
# Video writer
video_writer = cv2.VideoWriter("object_blurring_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init ObjectBlurrer
blurrer = solutions.ObjectBlurrer(
show=True, # display the output
model="yolo26n.pt", # model="yolo26n-obb.pt" for object blurring using YOLO26 OBB model.
blur_ratio=0.5, # set blur percentage, e.g., 0.7 for 70% blur on detected objects
# line_width=2, # width of bounding box.
# classes=[0, 2], # count specific classes, e.g., person and car with the COCO pretrained model.
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = blurrer(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
```
### What are the benefits of using Ultralytics YOLO26 for object blurring?
Ultralytics YOLO26 offers several advantages for object blurring:
- **Privacy Protection**: Effectively obscure sensitive or identifiable information.
- **Selective Focus**: Target specific objects for blurring, maintaining essential visual content.
- **Real-time Processing**: Execute object blurring efficiently in dynamic environments, suitable for instant privacy enhancements.
- **Customizable Intensity**: Adjust the blur ratio to balance privacy needs with visual context.
- **Class-Specific Blurring**: Selectively blur only certain types of objects while leaving others visible.
For more detailed applications, check the [advantages of object blurring section](#advantages-of-object-blurring).
### Can I use Ultralytics YOLO26 to blur faces in a video for privacy reasons?
Yes, Ultralytics YOLO26 can be configured to detect and blur faces in videos to protect privacy. By training or using a pretrained model to specifically recognize faces, the detection results can be processed with [OpenCV](https://www.ultralytics.com/glossary/opencv) to apply a blur effect. Refer to our guide on [object detection with YOLO26](https://docs.ultralytics.com/models/yolo26/) and modify the code to target face detection.
### How does YOLO26 compare to other object detection models like Faster R-CNN for object blurring?
Ultralytics YOLO26 typically outperforms models like Faster R-CNN in terms of speed, making it more suitable for real-time applications. While both models offer accurate detection, YOLO26's architecture is optimized for rapid inference, which is critical for tasks like real-time object blurring. Learn more about the technical differences and performance metrics in our [YOLO26 documentation](https://docs.ultralytics.com/models/yolo26/).

235
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/object-counting.md Normal file
View File

@@ -0,0 +1,235 @@
---
comments: true
description: Learn to accurately identify and count objects in real-time using Ultralytics YOLO26 for applications like crowd analysis and surveillance.
keywords: object counting, YOLO26, Ultralytics, real-time object detection, AI, deep learning, object tracking, crowd analysis, surveillance, resource optimization
---
# Object Counting using Ultralytics YOLO26
## What is Object Counting?
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-count-the-objects-using-ultralytics-yolo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Object Counting In Colab"></a>
Object counting with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves accurate identification and counting of specific objects in videos and camera streams. YOLO26 excels in real-time applications, providing efficient and precise object counting for various scenarios like crowd analysis and surveillance, thanks to its state-of-the-art algorithms and [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) capabilities.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/vKcD44GkSF8"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Perform Real-Time Object Counting with Ultralytics YOLO26 🍏
</p>
## Advantages of Object Counting
- **Resource Optimization:** Object counting facilitates efficient resource management by providing accurate counts, optimizing resource allocation in applications like [inventory management](https://docs.ultralytics.com/guides/analytics/).
- **Enhanced Security:** Object counting enhances security and surveillance by accurately tracking and counting entities, aiding in proactive [threat detection](https://docs.ultralytics.com/guides/security-alarm-system/).
- **Informed Decision-Making:** Object counting offers valuable insights for decision-making, optimizing processes in retail, [traffic management](https://www.ultralytics.com/blog/ai-in-traffic-management-from-congestion-to-coordination), and various other domains.
## Real World Applications
| Logistics | Aquaculture |
| :------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Conveyor Belt Packets Counting Using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/conveyor-belt-packets-counting.avif) | ![Fish Counting in Sea using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/fish-counting-in-sea-using-ultralytics-yolov8.avif) |
| Conveyor Belt Packets Counting Using Ultralytics YOLO26 | Fish Counting in Sea using Ultralytics YOLO26 |
!!! example "Object Counting using Ultralytics YOLO"
=== "CLI"
```bash
# Run a counting example
yolo solutions count show=True
# Pass a source video
yolo solutions count source="path/to/video.mp4"
# Pass region coordinates
yolo solutions count region="[(20, 400), (1080, 400), (1080, 360), (20, 360)]"
```
The `region` argument accepts either two points (for a line) or a polygon with three or more points. Define the coordinates in the order they should be connected so the counter knows exactly where entries and exits occur.
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# region_points = [(20, 400), (1080, 400)] # line counting
region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360)] # rectangular region
# region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360), (20, 400)] # polygon region
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("object_counting_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize object counter object
counter = solutions.ObjectCounter(
show=True, # display the output
region=region_points, # pass region points
model="yolo26n.pt", # model="yolo26n-obb.pt" for object counting with OBB model.
# classes=[0, 2], # count specific classes, e.g., person and car with the COCO pretrained model.
# tracker="botsort.yaml", # choose trackers, e.g., "bytetrack.yaml"
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = counter(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `ObjectCounter` Arguments
Here's a table with the `ObjectCounter` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "show_in", "show_out", "region"]) }}
The `ObjectCounter` solution allows the use of several `track` arguments:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the visualization arguments listed below are supported:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## FAQ
### How do I count objects in a video using Ultralytics YOLO26?
To count objects in a video using Ultralytics YOLO26, you can follow these steps:
1. Import the necessary libraries (`cv2`, `ultralytics`).
2. Define the counting region (e.g., a polygon, line, etc.).
3. Set up the video capture and initialize the object counter.
4. Process each frame to track objects and count them within the defined region.
Here's a simple example for counting in a region:
```python
import cv2
from ultralytics import solutions
def count_objects_in_region(video_path, output_video_path, model_path):
"""Count objects in a specific region within a video."""
cap = cv2.VideoCapture(video_path)
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter(output_video_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360)]
counter = solutions.ObjectCounter(show=True, region=region_points, model=model_path)
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = counter(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
count_objects_in_region("path/to/video.mp4", "output_video.avi", "yolo26n.pt")
```
For more advanced configurations and options, check out the [RegionCounter solution](https://docs.ultralytics.com/guides/region-counting/) for counting objects in multiple regions simultaneously.
### What are the advantages of using Ultralytics YOLO26 for object counting?
Using Ultralytics YOLO26 for object counting offers several advantages:
1. **Resource Optimization:** It facilitates efficient resource management by providing accurate counts, helping optimize resource allocation in industries like [inventory management](https://www.ultralytics.com/blog/ai-for-smarter-retail-inventory-management).
2. **Enhanced Security:** It enhances security and surveillance by accurately tracking and counting entities, aiding in proactive threat detection and [security systems](https://docs.ultralytics.com/guides/security-alarm-system/).
3. **Informed Decision-Making:** It offers valuable insights for decision-making, optimizing processes in domains like retail, traffic management, and more.
4. **Real-time Processing:** YOLO26's architecture enables [real-time inference](https://www.ultralytics.com/glossary/real-time-inference), making it suitable for live video streams and time-sensitive applications.
For implementation examples and practical applications, explore the [TrackZone solution](https://docs.ultralytics.com/guides/trackzone/) for tracking objects in specific zones.
### How can I count specific classes of objects using Ultralytics YOLO26?
To count specific classes of objects using Ultralytics YOLO26, you need to specify the classes you are interested in during the tracking phase. Below is a Python example:
```python
import cv2
from ultralytics import solutions
def count_specific_classes(video_path, output_video_path, model_path, classes_to_count):
"""Count specific classes of objects in a video."""
cap = cv2.VideoCapture(video_path)
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter(output_video_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
line_points = [(20, 400), (1080, 400)]
counter = solutions.ObjectCounter(show=True, region=line_points, model=model_path, classes=classes_to_count)
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = counter(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
count_specific_classes("path/to/video.mp4", "output_specific_classes.avi", "yolo26n.pt", [0, 2])
```
In this example, `classes_to_count=[0, 2]` means it counts objects of class `0` and `2` (e.g., person and car in the COCO dataset). You can find more information about class indices in the [COCO dataset documentation](https://docs.ultralytics.com/datasets/detect/coco/).
### Why should I use YOLO26 over other [object detection](https://www.ultralytics.com/glossary/object-detection) models for real-time applications?
Ultralytics YOLO26 provides several advantages over other object detection models like [Faster R-CNN](https://docs.ultralytics.com/compare/yolo26-vs-efficientdet/), SSD, and previous YOLO versions:
1. **Speed and Efficiency:** YOLO26 offers real-time processing capabilities, making it ideal for applications requiring high-speed inference, such as surveillance and [autonomous driving](https://www.ultralytics.com/blog/ai-in-self-driving-cars).
2. **[Accuracy](https://www.ultralytics.com/glossary/accuracy):** It provides state-of-the-art accuracy for object detection and tracking tasks, reducing the number of false positives and improving overall system reliability.
3. **Ease of Integration:** YOLO26 offers seamless integration with various platforms and devices, including mobile and [edge devices](https://docs.ultralytics.com/guides/nvidia-jetson/), which is crucial for modern AI applications.
4. **Flexibility:** Supports various tasks like object detection, [segmentation](https://docs.ultralytics.com/tasks/segment/), and tracking with configurable models to meet specific use-case requirements.
Check out Ultralytics [YOLO26 Documentation](https://docs.ultralytics.com/models/yolo26/) for a deeper dive into its features and performance comparisons.
### Can I use YOLO26 for advanced applications like crowd analysis and traffic management?
Yes, Ultralytics YOLO26 is perfectly suited for advanced applications like crowd analysis and traffic management due to its real-time detection capabilities, scalability, and integration flexibility. Its advanced features allow for high-accuracy object tracking, counting, and classification in dynamic environments. Example use cases include:
- **Crowd Analysis:** Monitor and manage large gatherings, ensuring safety and optimizing crowd flow with [region-based counting](https://docs.ultralytics.com/guides/region-counting/).
- **Traffic Management:** Track and count vehicles, analyze traffic patterns, and manage congestion in real-time with [speed estimation](https://docs.ultralytics.com/guides/speed-estimation/) capabilities.
- **Retail Analytics:** Analyze customer movement patterns and product interactions to optimize store layouts and improve customer experience.
- **Industrial Automation:** Count products on conveyor belts and monitor production lines for quality control and efficiency improvements.
For more specialized applications, explore [Ultralytics Solutions](https://docs.ultralytics.com/solutions/) for a comprehensive set of tools designed for real-world computer vision challenges.

121
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/object-cropping.md Normal file
View File

@@ -0,0 +1,121 @@
---
comments: true
description: Learn how to crop and extract objects using Ultralytics YOLO26 for focused analysis, reduced data volume, and enhanced precision.
keywords: Ultralytics, YOLO26, object cropping, object detection, image processing, video analysis, AI, machine learning
---
# Object Cropping using Ultralytics YOLO26
## What is Object Cropping?
Object cropping with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves isolating and extracting specific detected objects from an image or video. The YOLO26 model capabilities are utilized to accurately identify and delineate objects, enabling precise cropping for further analysis or manipulation.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/J1BaCqytBmA"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Object Cropping using Ultralytics YOLO
</p>
## Advantages of Object Cropping
- **Focused Analysis**: YOLO26 facilitates targeted object cropping, allowing for in-depth examination or processing of individual items within a scene.
- **Reduced Data Volume**: By extracting only relevant objects, object cropping helps in minimizing data size, making it efficient for storage, transmission, or subsequent computational tasks.
- **Enhanced Precision**: YOLO26's [object detection](https://www.ultralytics.com/glossary/object-detection) [accuracy](https://www.ultralytics.com/glossary/accuracy) ensures that the cropped objects maintain their spatial relationships, preserving the integrity of the visual information for detailed analysis.
## Visuals
| Airport Luggage |
| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Conveyor Belt at Airport Suitcases Cropping using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/suitcases-cropping-airport-conveyor-belt.avif) |
| Suitcases Cropping at airport conveyor belt using Ultralytics YOLO26 |
!!! example "Object Cropping using Ultralytics YOLO"
=== "CLI"
```bash
# Crop the objects
yolo solutions crop show=True
# Pass a source video
yolo solutions crop source="path/to/video.mp4"
# Crop specific classes
yolo solutions crop classes="[0, 2]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Initialize object cropper
cropper = solutions.ObjectCropper(
show=True, # display the output
model="yolo26n.pt", # model for object cropping, e.g., yolo26x.pt.
classes=[0, 2], # crop specific classes such as person and car with the COCO pretrained model.
# conf=0.5, # adjust confidence threshold for the objects.
# crop_dir="cropped-detections", # set the directory name for cropped detections
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = cropper(im0)
# print(results) # access the output
cap.release()
cv2.destroyAllWindows() # destroy all opened windows
```
When you provide the optional `crop_dir` argument, every cropped object is written to that folder with filenames that include the source image name and class. This makes it easy to inspect detections or build downstream datasets without writing extra code.
### `ObjectCropper` Arguments
Here's a table with the `ObjectCropper` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "crop_dir"]) }}
Moreover, the following visualization arguments are available for use:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width"]) }}
## FAQ
### What is object cropping in Ultralytics YOLO26 and how does it work?
Object cropping using [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics) involves isolating and extracting specific objects from an image or video based on YOLO26's detection capabilities. This process allows for focused analysis, reduced data volume, and enhanced [precision](https://www.ultralytics.com/glossary/precision) by leveraging YOLO26 to identify objects with high accuracy and crop them accordingly. For an in-depth tutorial, refer to the [object cropping example](#object-cropping-using-ultralytics-yolo26).
### Why should I use Ultralytics YOLO26 for object cropping over other solutions?
Ultralytics YOLO26 stands out due to its precision, speed, and ease of use. It allows detailed and accurate object detection and cropping, essential for [focused analysis](#advantages-of-object-cropping) and applications needing high data integrity. Moreover, YOLO26 integrates seamlessly with tools like [OpenVINO](../integrations/openvino.md) and [TensorRT](../integrations/tensorrt.md) for deployments requiring real-time capabilities and optimization on diverse hardware. Explore the benefits in the [guide on model export](../modes/export.md).
### How can I reduce the data volume of my dataset using object cropping?
By using Ultralytics YOLO26 to crop only relevant objects from your images or videos, you can significantly reduce the data size, making it more efficient for storage and processing. This process involves training the model to detect specific objects and then using the results to crop and save these portions only. For more information on exploiting Ultralytics YOLO26's capabilities, visit our [quickstart guide](../quickstart.md).
### Can I use Ultralytics YOLO26 for real-time video analysis and object cropping?
Yes, Ultralytics YOLO26 can process real-time video feeds to detect and crop objects dynamically. The model's high-speed inference capabilities make it ideal for real-time applications such as [surveillance](security-alarm-system.md), sports analysis, and automated inspection systems. Check out the [tracking](../modes/track.md) and [prediction modes](../modes/predict.md) to understand how to implement real-time processing.
### What are the hardware requirements for efficiently running YOLO26 for object cropping?
Ultralytics YOLO26 is optimized for both CPU and GPU environments, but to achieve optimal performance, especially for real-time or high-volume inference, a dedicated GPU (e.g., NVIDIA Tesla, RTX series) is recommended. For deployment on lightweight devices, consider using [CoreML](../integrations/coreml.md) for iOS or [TFLite](../integrations/tflite.md) for Android. More details on supported devices and formats can be found in our [model deployment options](../guides/model-deployment-options.md).

158
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/optimizing-openvino-latency-vs-throughput-modes.md Normal file
View File

@@ -0,0 +1,158 @@
---
comments: true
description: Discover how to enhance Ultralytics YOLO model performance using Intel's OpenVINO toolkit. Boost latency and throughput efficiently.
keywords: Ultralytics YOLO, OpenVINO optimization, deep learning, model inference, throughput optimization, latency optimization, AI deployment, Intel's OpenVINO, performance tuning
---
# OpenVINO Inference Optimization for YOLO
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/openvino-ecosystem.avif" alt="OpenVINO Ecosystem">
## Introduction
When deploying [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) models, particularly those for [object detection](https://www.ultralytics.com/glossary/object-detection) such as Ultralytics YOLO models, achieving optimal performance is crucial. This guide delves into leveraging [Intel's OpenVINO toolkit](https://docs.ultralytics.com/integrations/openvino/) to optimize inference, focusing on latency and throughput. Whether you're working on consumer-grade applications or large-scale deployments, understanding and applying these optimization strategies will ensure your models run efficiently on various devices.
## Optimizing for Latency
Latency optimization is vital for applications requiring immediate response from a single model given a single input, typical in consumer scenarios. The goal is to minimize the delay between input and inference result. However, achieving low latency involves careful consideration, especially when running concurrent inferences or managing multiple models.
### Key Strategies for Latency Optimization:
- **Single Inference per Device:** The simplest way to achieve low latency is by limiting to one inference at a time per device. Additional concurrency often leads to increased latency.
- **Leveraging Sub-Devices:** Devices like multi-socket CPUs or multi-tile GPUs can execute multiple requests with minimal latency increase by utilizing their internal sub-devices.
- **OpenVINO Performance Hints:** Utilizing OpenVINO's `ov::hint::PerformanceMode::LATENCY` for the `ov::hint::performance_mode` property during model compilation simplifies performance tuning, offering a device-agnostic and future-proof approach.
### Managing First-Inference Latency:
- **Model Caching:** To mitigate model load and compile times impacting latency, use model caching where possible. For scenarios where caching isn't viable, CPUs generally offer the fastest model load times.
- **Model Mapping vs. Reading:** To reduce load times, OpenVINO replaced model reading with mapping. However, if the model is on a removable or network drive, consider using `ov::enable_mmap(false)` to switch back to reading.
- **AUTO Device Selection:** This mode begins inference on the CPU, shifting to an accelerator once ready, seamlessly reducing first-inference latency.
## Optimizing for Throughput
Throughput optimization is crucial for scenarios serving numerous inference requests simultaneously, maximizing [resource utilization](https://www.ultralytics.com/blog/measuring-ai-performance-to-weigh-the-impact-of-your-innovations) without significantly sacrificing individual request performance.
### Approaches to Throughput Optimization:
1. **OpenVINO Performance Hints:** A high-level, future-proof method to enhance throughput across devices using performance hints.
```python
import openvino.properties.hint as hints
config = {hints.performance_mode: hints.PerformanceMode.THROUGHPUT}
compiled_model = core.compile_model(model, "GPU", config)
```
2. **Explicit Batching and Streams:** A more granular approach involving explicit batching and the use of streams for advanced performance tuning.
### Designing Throughput-Oriented Applications:
To maximize throughput, applications should:
- Process inputs in parallel, making full use of the device's capabilities.
- Decompose data flow into concurrent inference requests, scheduled for parallel execution.
- Utilize the Async API with callbacks to maintain efficiency and avoid device starvation.
### Multi-Device Execution:
OpenVINO's multi-device mode simplifies scaling throughput by automatically balancing inference requests across devices without requiring application-level device management.
## Real-World Performance Gains
Implementing OpenVINO optimizations with Ultralytics YOLO models can yield significant performance improvements. As demonstrated in [benchmarks](https://docs.ultralytics.com/integrations/openvino/#openvino-yolov8-benchmarks), users can experience up to 3x faster inference speeds on Intel CPUs, with even greater accelerations possible across Intel's hardware spectrum including integrated GPUs, dedicated GPUs, and VPUs.
For example, when running YOLOv8 models on Intel Xeon CPUs, the OpenVINO-optimized versions consistently outperform their PyTorch counterparts in terms of inference time per image, without compromising on [accuracy](https://www.ultralytics.com/glossary/accuracy).
## Practical Implementation
To export and optimize your Ultralytics YOLO model for OpenVINO, you can use the [export](https://docs.ultralytics.com/modes/export/) functionality:
```python
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Export the model to OpenVINO format
model.export(format="openvino", half=True) # Export with FP16 precision
```
After exporting, you can run inference with the optimized model:
```python
# Load the OpenVINO model
ov_model = YOLO("yolo26n_openvino_model/")
# Run inference with performance hints for latency
results = ov_model("path/to/image.jpg", verbose=True)
```
## Conclusion
Optimizing Ultralytics YOLO models for latency and throughput with OpenVINO can significantly enhance your application's performance. By carefully applying the strategies outlined in this guide, developers can ensure their models run efficiently, meeting the demands of various deployment scenarios. Remember, the choice between optimizing for latency or throughput depends on your specific application needs and the characteristics of the deployment environment.
For more detailed technical information and the latest updates, refer to the [OpenVINO documentation](https://docs.openvino.ai/2024/index.html) and [Ultralytics YOLO repository](https://github.com/ultralytics/ultralytics). These resources provide in-depth guides, tutorials, and community support to help you get the most out of your deep learning models.
---
Ensuring your models achieve optimal performance is not just about tweaking configurations; it's about understanding your application's needs and making informed decisions. Whether you're optimizing for [real-time responses](https://www.ultralytics.com/blog/real-time-inferences-in-vision-ai-solutions-are-making-an-impact) or maximizing throughput for large-scale processing, the combination of Ultralytics YOLO models and OpenVINO offers a powerful toolkit for developers to deploy high-performance AI solutions.
## FAQ
### How do I optimize Ultralytics YOLO models for low latency using OpenVINO?
Optimizing Ultralytics YOLO models for low latency involves several key strategies:
1. **Single Inference per Device:** Limit inferences to one at a time per device to minimize delays.
2. **Leveraging Sub-Devices:** Utilize devices like multi-socket CPUs or multi-tile GPUs which can handle multiple requests with minimal latency increase.
3. **OpenVINO Performance Hints:** Use OpenVINO's `ov::hint::PerformanceMode::LATENCY` during model compilation for simplified, device-agnostic tuning.
For more practical tips on optimizing latency, check out the [Latency Optimization section](#optimizing-for-latency) of our guide.
### Why should I use OpenVINO for optimizing Ultralytics YOLO throughput?
OpenVINO enhances Ultralytics YOLO model throughput by maximizing device resource utilization without sacrificing performance. Key benefits include:
- **Performance Hints:** Simple, high-level performance tuning across devices.
- **Explicit Batching and Streams:** Fine-tuning for advanced performance.
- **Multi-Device Execution:** Automated inference load balancing, easing application-level management.
Example configuration:
```python
import openvino.properties.hint as hints
config = {hints.performance_mode: hints.PerformanceMode.THROUGHPUT}
compiled_model = core.compile_model(model, "GPU", config)
```
Learn more about throughput optimization in the [Throughput Optimization section](#optimizing-for-throughput) of our detailed guide.
### What is the best practice for reducing first-inference latency in OpenVINO?
To reduce first-inference latency, consider these practices:
1. **Model Caching:** Use model caching to decrease load and compile times.
2. **Model Mapping vs. Reading:** Use mapping (`ov::enable_mmap(true)`) by default but switch to reading (`ov::enable_mmap(false)`) if the model is on a removable or network drive.
3. **AUTO Device Selection:** Utilize AUTO mode to start with CPU inference and transition to an accelerator seamlessly.
For detailed strategies on managing first-inference latency, refer to the [Managing First-Inference Latency section](#managing-first-inference-latency).
### How do I balance optimizing for latency and throughput with Ultralytics YOLO and OpenVINO?
Balancing latency and throughput optimization requires understanding your application needs:
- **Latency Optimization:** Ideal for real-time applications requiring immediate responses (e.g., consumer-grade apps).
- **Throughput Optimization:** Best for scenarios with many concurrent inferences, maximizing resource use (e.g., large-scale deployments).
Using OpenVINO's high-level performance hints and multi-device modes can help strike the right balance. Choose the appropriate [OpenVINO Performance hints](https://docs.ultralytics.com/integrations/openvino/#openvino-performance-hints) based on your specific requirements.
### Can I use Ultralytics YOLO models with other AI frameworks besides OpenVINO?
Yes, Ultralytics YOLO models are highly versatile and can be integrated with various AI frameworks. Options include:
- **TensorRT:** For NVIDIA GPU optimization, follow the [TensorRT integration guide](https://docs.ultralytics.com/integrations/tensorrt/).
- **CoreML:** For Apple devices, refer to our [CoreML export instructions](https://docs.ultralytics.com/integrations/coreml/).
- **[TensorFlow](https://www.ultralytics.com/glossary/tensorflow).js:** For web and Node.js apps, see the [TF.js conversion guide](https://docs.ultralytics.com/integrations/tfjs/).
Explore more integrations on the [Ultralytics Integrations page](https://docs.ultralytics.com/integrations/).

161
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/parking-management.md Normal file
View File

@@ -0,0 +1,161 @@
---
comments: true
description: Optimize parking spaces and enhance safety with Ultralytics YOLO26. Explore real-time vehicle detection and smart parking solutions.
keywords: parking management, YOLO26, Ultralytics, vehicle detection, real-time tracking, parking lot optimization, smart parking
---
# Parking Management using Ultralytics YOLO26 🚀
## What is Parking Management System?
Parking management with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) ensures efficient and safe parking by organizing spaces and monitoring availability. YOLO26 can improve parking lot management through real-time vehicle detection, and insights into parking occupancy.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/WwXnljc7ZUM"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Implement Parking Management Using Ultralytics YOLO 🚀
</p>
## Advantages of Parking Management System
- **Efficiency**: Parking lot management optimizes the use of parking spaces and reduces congestion.
- **Safety and Security**: Parking management using YOLO26 improves the safety of both people and vehicles through surveillance and security measures.
- **Reduced Emissions**: Parking management using YOLO26 manages traffic flow to minimize idle time and emissions in parking lots.
## Real World Applications
| Parking Management System | Parking Management System |
| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Parking lots Analytics Using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/parking-management-aerial-view-ultralytics-yolov8.avif) | ![Parking management top view using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/parking-management-top-view-ultralytics-yolov8.avif) |
| Parking management Aerial View using Ultralytics YOLO26 | Parking management Top View using Ultralytics YOLO26 |
## Parking Management System Code Workflow
??? note "Points selection is now easy"
Choosing parking points is a critical and complex task in parking management systems. Ultralytics streamlines this process by providing a tool "Parking slots annotator" that lets you define parking lot areas, which can be utilized later for additional processing.
**Step-1:** Capture a frame from the video or camera stream where you want to manage the parking lot.
**Step-2:** Use the provided code to launch a graphical interface, where you can select an image and start outlining parking regions by mouse click to create polygons.
!!! example "Parking slots annotator Ultralytics YOLO"
??? note "Additional step for installing `tkinter`"
Generally, `tkinter` comes pre-packaged with Python. However, if it did not, you can install it using the highlighted steps:
- **Linux**: (Debian/Ubuntu): `sudo apt install python3-tk`
- **Fedora**: `sudo dnf install python3-tkinter`
- **Arch**: `sudo pacman -S tk`
- **Windows**: Reinstall Python and enable the checkbox `tcl/tk and IDLE` on **Optional Features** during installation
- **MacOS**: Reinstall Python from [https://www.python.org/downloads/macos/](https://www.python.org/downloads/macos/) or `brew install python-tk`
=== "Python"
```python
from ultralytics import solutions
solutions.ParkingPtsSelection()
```
**Step-3:** After defining the parking areas with polygons, click `save` to store a JSON file with the data in your working directory.
![Ultralytics YOLO26 Points Selection Demo](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-yolov8-points-selection-demo.avif)
**Step-4:** You can now utilize the provided code for parking management with Ultralytics YOLO.
!!! example "Parking Management using Ultralytics YOLO"
=== "Python"
```python
import cv2
from ultralytics import solutions
# Video capture
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("parking management.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize parking management object
parkingmanager = solutions.ParkingManagement(
model="yolo26n.pt", # path to model file
json_file="bounding_boxes.json", # path to parking annotations file
)
while cap.isOpened():
ret, im0 = cap.read()
if not ret:
break
results = parkingmanager(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `ParkingManagement` Arguments
Here's a table with the `ParkingManagement` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "json_file"]) }}
The `ParkingManagement` solution allows the use of several `track` parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization options are supported:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width"]) }}
## FAQ
### How does Ultralytics YOLO26 enhance parking management systems?
Ultralytics YOLO26 greatly enhances parking management systems by providing **real-time vehicle detection** and monitoring. This results in optimized usage of parking spaces, reduced congestion, and improved safety through continuous surveillance. The [Parking Management System](https://github.com/ultralytics/ultralytics) enables efficient traffic flow, minimizing idle times and emissions in parking lots, thereby contributing to environmental sustainability. For further details, refer to the [parking management code workflow](#parking-management-system-code-workflow).
### What are the benefits of using Ultralytics YOLO26 for smart parking?
Using Ultralytics YOLO26 for smart parking yields numerous benefits:
- **Efficiency**: Optimizes the use of parking spaces and decreases congestion.
- **Safety and Security**: Enhances surveillance and ensures the safety of vehicles and pedestrians.
- **Environmental Impact**: Helps reduce emissions by minimizing vehicle idle times. Explore more benefits in the [Advantages of Parking Management System section](#advantages-of-parking-management-system).
### How can I define parking spaces using Ultralytics YOLO26?
Defining parking spaces is straightforward with Ultralytics YOLO26:
1. Capture a frame from a video or camera stream.
2. Use the provided code to launch a GUI for selecting an image and drawing polygons to define parking spaces.
3. Save the labeled data in JSON format for further processing. For comprehensive instructions, check the selection of points section above.
### Can I customize the YOLO26 model for specific parking management needs?
Yes, Ultralytics YOLO26 allows customization for specific parking management needs. You can adjust parameters such as the **occupied and available region colors**, margins for text display, and much more. Utilizing the `ParkingManagement` class's [arguments](#parkingmanagement-arguments), you can tailor the model to suit your particular requirements, ensuring maximum efficiency and effectiveness.
### What are some real-world applications of Ultralytics YOLO26 in parking lot management?
Ultralytics YOLO26 is utilized in various real-world applications for parking lot management, including:
- **Parking Space Detection**: Accurately identifying available and occupied spaces.
- **Surveillance**: Enhancing security through real-time monitoring.
- **Traffic Flow Management**: Reducing idle times and congestion with efficient traffic handling. Images showcasing these applications can be found in [real-world applications](#real-world-applications).

187
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/preprocessing_annotated_data.md Normal file
View File

@@ -0,0 +1,187 @@
---
comments: true
description: Learn essential data preprocessing techniques for annotated computer vision data, including resizing, normalizing, augmenting, and splitting datasets for optimal model training.
keywords: data preprocessing, computer vision, image resizing, normalization, data augmentation, training dataset, validation dataset, test dataset, YOLO26
---
# Data Preprocessing Techniques for Annotated [Computer Vision](https://www.ultralytics.com/glossary/computer-vision-cv) Data
## Introduction
After you've defined your computer vision [project's goals](./defining-project-goals.md) and [collected and annotated data](./data-collection-and-annotation.md), the next step is to preprocess annotated data and prepare it for model training. Clean and consistent data are vital to creating a model that performs well.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/E_d7xuY4hEk"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Use Data Preprocessing and Augmentation to Improve Model Accuracy in Real-World Scenarios 🚀
</p>
Preprocessing is a step in the [computer vision project workflow](./steps-of-a-cv-project.md) that includes resizing images, normalizing pixel values, augmenting the dataset, and splitting the data into training, validation, and test sets. Let's explore the essential techniques and best practices for cleaning your data!
## Importance of Data Preprocessing
We are already collecting and annotating our data carefully with multiple considerations in mind. Then, what makes data preprocessing so important to a computer vision project? Well, data preprocessing is all about getting your data into a suitable format for training that reduces the computational load and helps improve model performance. Here are some common issues in raw data that preprocessing addresses:
- **Noise**: Irrelevant or random variations in data.
- **Inconsistency**: Variations in image sizes, formats, and quality.
- **Imbalance**: Unequal distribution of classes or categories in the dataset.
## Data Preprocessing Techniques
One of the first and foremost steps in data preprocessing is resizing. Some models are designed to handle variable input sizes, but many models require a consistent input size. Resizing images makes them uniform and reduces computational complexity.
### Resizing Images
You can resize your images using the following methods:
- **Bilinear Interpolation**: Smooths pixel values by taking a weighted average of the four nearest pixel values.
- **Nearest Neighbor**: Assigns the nearest pixel value without averaging, leading to a blocky image but faster computation.
To make resizing a simpler task, you can use the following tools:
- **[OpenCV](https://www.ultralytics.com/glossary/opencv)**: A popular computer vision library with extensive functions for image processing.
- **PIL (Pillow)**: A Python Imaging Library for opening, manipulating, and saving image files.
With respect to YOLO26, the 'imgsz' parameter during [model training](../modes/train.md) allows for flexible input sizes. When set to a specific size, such as 640, the model will resize input images so their largest dimension is 640 pixels while maintaining the original aspect ratio.
By evaluating your model's and dataset's specific needs, you can determine whether resizing is a necessary preprocessing step or if your model can efficiently handle images of varying sizes.
### Normalizing Pixel Values
Another preprocessing technique is normalization. Normalization scales the pixel values to a standard range, which helps in faster convergence during training and improves model performance. Here are some common normalization techniques:
- **Min-Max Scaling**: Scales pixel values to a range of 0 to 1.
- **Z-Score Normalization**: Scales pixel values based on their mean and standard deviation.
With respect to YOLO26, normalization is seamlessly handled as part of its preprocessing pipeline during model training. YOLO26 automatically performs several preprocessing steps, including conversion to RGB, scaling pixel values to the range [0, 1], and normalization using predefined mean and standard deviation values.
### Splitting the Dataset
Once you've cleaned the data, you are ready to split the dataset. Splitting the data into training, validation, and test sets is done to ensure that the model can be evaluated on unseen data to assess its generalization performance. A common split is 70% for training, 20% for validation, and 10% for testing. There are various tools and libraries that you can use to split your data like [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) or [TensorFlow](https://www.ultralytics.com/glossary/tensorflow).
Consider the following when splitting your dataset:
- **Maintaining Data Distribution**: Ensure that the data distribution of classes is maintained across training, validation, and test sets.
- **Avoiding Data Leakage**: Typically, data augmentation is done after the dataset is split. Data augmentation and any other preprocessing should only be applied to the training set to prevent information from the validation or test sets from influencing the model training.
- **Balancing Classes**: For imbalanced datasets, consider techniques such as oversampling the minority class or under-sampling the majority class within the training set.
### What is Data Augmentation?
The most commonly discussed data preprocessing step is data augmentation. Data augmentation artificially increases the size of the dataset by creating modified versions of images. By augmenting your data, you can reduce [overfitting](https://www.ultralytics.com/glossary/overfitting) and improve model generalization.
Here are some other benefits of data augmentation:
- **Creates a More Robust Dataset**: Data augmentation can make the model more robust to variations and distortions in the input data. This includes changes in lighting, orientation, and scale.
- **Cost-Effective**: Data augmentation is a cost-effective way to increase the amount of [training data](https://www.ultralytics.com/glossary/training-data) without collecting and labeling new data.
- **Better Use of Data**: Every available data point is used to its maximum potential by creating new variations
#### Data Augmentation Methods
Common augmentation techniques include flipping, rotation, scaling, and color adjustments. Several libraries, such as [Albumentations](../integrations/albumentations.md), Imgaug, and TensorFlow's ImageDataGenerator, can generate these augmentations.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/overview-of-data-augmentations.avif" alt="Overview of Data Augmentations">
</p>
With respect to YOLO26, you can [augment your custom dataset](../modes/train.md) by modifying the dataset configuration file, a .yaml file. In this file, you can add an augmentation section with parameters that specify how you want to augment your data.
The [Ultralytics YOLO26 repository](https://github.com/ultralytics/ultralytics/tree/main) supports a wide range of data augmentations. You can apply various transformations such as:
- Random Crops
- Flipping: Images can be flipped horizontally or vertically.
- Rotation: Images can be rotated by specific angles.
- Distortion
Also, you can adjust the intensity of these augmentation techniques through specific parameters to generate more data variety.
## A Case Study of Preprocessing
Consider a project aimed at developing a model to detect and classify different types of vehicles in traffic images using YOLO26. We've collected traffic images and annotated them with bounding boxes and labels.
Here's what each step of preprocessing would look like for this project:
- **Resizing Images**: Since YOLO26 handles flexible input sizes and performs resizing automatically, manual resizing is not required. The model will adjust the image size according to the specified 'imgsz' parameter during training.
- **Normalizing Pixel Values**: YOLO26 automatically normalizes pixel values to a range of 0 to 1 during preprocessing, so it's not required.
- **Splitting the Dataset**: Divide the dataset into training (70%), validation (20%), and test (10%) sets using tools like scikit-learn.
- **[Data Augmentation](https://www.ultralytics.com/glossary/data-augmentation)**: Modify the dataset configuration file (.yaml) to include data augmentation techniques such as random crops, horizontal flips, and brightness adjustments.
These steps make sure the dataset is prepared without any potential issues and is ready for Exploratory Data Analysis (EDA).
## Exploratory Data Analysis Techniques
After preprocessing and augmenting your dataset, the next step is to gain insights through Exploratory Data Analysis. EDA uses statistical techniques and visualization tools to understand the patterns and distributions in your data. You can identify issues like class imbalances or outliers and make informed decisions about further data preprocessing or model training adjustments.
### Statistical EDA Techniques
Statistical techniques often begin with calculating basic metrics such as mean, median, standard deviation, and range. These metrics provide a quick overview of your image dataset's properties, such as pixel intensity distributions. Understanding these basic statistics helps you grasp the overall quality and characteristics of your data, allowing you to spot any irregularities early on.
### Visual EDA Techniques
Visualizations are key in EDA for image datasets. For example, class imbalance analysis is another vital aspect of EDA. It helps determine if certain classes are underrepresented in your dataset. Visualizing the distribution of different image classes or categories using bar charts can quickly reveal any imbalances. Similarly, outliers can be identified using visualization tools like box plots, which highlight anomalies in pixel intensity or feature distributions. Outlier detection prevents unusual data points from skewing your results.
Common tools for visualizations include:
- **Histograms and Box Plots**: Useful for understanding the distribution of pixel values and identifying outliers.
- **Scatter Plots**: Helpful for exploring relationships between image features or annotations.
- **Heatmaps**: Effective for visualizing the distribution of pixel intensities or the spatial distribution of annotated features within images.
### Using Ultralytics Explorer for EDA
!!! warning "Community Note"
As of **`ultralytics>=8.3.10`**, Ultralytics Explorer support is deprecated. Similar (and expanded) dataset exploration features are available in [Ultralytics Platform](https://platform.ultralytics.com/).
For a more advanced approach to EDA, you can use the Ultralytics Explorer tool. It offers robust capabilities for exploring computer vision datasets. By supporting semantic search, SQL queries, and vector similarity search, the tool makes it easy to analyze and understand your data. With Ultralytics Explorer, you can create [embeddings](https://www.ultralytics.com/glossary/embeddings) for your dataset to find similar images, run SQL queries for detailed analysis, and perform semantic searches, all through a user-friendly graphical interface.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-explorer-openai-integration.avif" alt="Overview of Ultralytics Explorer">
</p>
## Reach Out and Connect
Having discussions about your project with other computer vision enthusiasts can give you new ideas from different perspectives. Here are some great ways to learn, troubleshoot, and network:
### Channels to Connect with the Community
- **GitHub Issues:** Visit the YOLO26 GitHub repository and use the [Issues tab](https://github.com/ultralytics/ultralytics/issues) to raise questions, report bugs, and suggest features. The community and maintainers are there to help with any issues you face.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to connect with other users and developers, get support, share knowledge, and brainstorm ideas.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Refer to the [official YOLO26 documentation](./index.md) for thorough guides and valuable insights on numerous computer vision tasks and projects.
## Your Dataset Is Ready!
Properly resized, normalized, and augmented data improves model performance by reducing noise and improving generalization. By following the preprocessing techniques and best practices outlined in this guide, you can create a solid dataset. With your preprocessed dataset ready, you can confidently proceed to the next steps in your project.
## FAQ
### What is the importance of data preprocessing in computer vision projects?
Data preprocessing is essential in computer vision projects because it ensures that the data is clean, consistent, and in a format that is optimal for model training. By addressing issues such as noise, inconsistency, and imbalance in raw data, preprocessing steps like resizing, normalization, augmentation, and dataset splitting help reduce computational load and improve model performance. For more details, visit the [steps of a computer vision project](../guides/steps-of-a-cv-project.md).
### How can I use Ultralytics YOLO for data augmentation?
For data augmentation with Ultralytics YOLO26, you need to modify the dataset configuration file (.yaml). In this file, you can specify various augmentation techniques such as random crops, horizontal flips, and brightness adjustments. This can be effectively done using the training configurations [explained here](../modes/train.md). Data augmentation helps create a more robust dataset, reduce [overfitting](https://www.ultralytics.com/glossary/overfitting), and improve model generalization.
### What are the best data normalization techniques for computer vision data?
Normalization scales pixel values to a standard range for faster convergence and improved performance during training. Common techniques include:
- **Min-Max Scaling**: Scales pixel values to a range of 0 to 1.
- **Z-Score Normalization**: Scales pixel values based on their mean and standard deviation.
For YOLO26, normalization is handled automatically, including conversion to RGB and pixel value scaling. Learn more about it in the [model training section](../modes/train.md).
### How should I split my annotated dataset for training?
To split your dataset, a common practice is to divide it into 70% for training, 20% for validation, and 10% for testing. It is important to maintain the data distribution of classes across these splits and avoid data leakage by performing augmentation only on the training set. Use tools like scikit-learn or [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) for efficient dataset splitting. See the detailed guide on [dataset preparation](../guides/data-collection-and-annotation.md).
### Can I handle varying image sizes in YOLO26 without manual resizing?
Yes, Ultralytics YOLO26 can handle varying image sizes through the 'imgsz' parameter during model training. This parameter ensures that images are resized so their largest dimension matches the specified size (e.g., 640 pixels), while maintaining the aspect ratio. For more flexible input handling and automatic adjustments, check the [model training section](../modes/train.md).

210
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/queue-management.md Normal file
View File

@@ -0,0 +1,210 @@
---
comments: true
description: Learn how to manage and optimize queues using Ultralytics YOLO26 to reduce wait times and increase efficiency in various real-world applications.
keywords: queue management, YOLO26, Ultralytics, reduce wait times, efficiency, customer satisfaction, retail, airports, healthcare, banks
---
# Queue Management using Ultralytics YOLO26 🚀
## What is Queue Management?
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-monitor-objects-in-queue-using-queue-management-solution.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Queue Management In Colab"></a>
Queue management using [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves organizing and controlling lines of people or vehicles to reduce wait times and enhance efficiency. It's about optimizing queues to improve customer satisfaction and system performance in various settings like retail, banks, airports, and healthcare facilities.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Gxr9SpYPLh0"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Build a Queue Management System with Ultralytics YOLO | Retail, Bank & Crowd Use Cases 🚀
</p>
## Advantages of Queue Management
- **Reduced Waiting Times:** Queue management systems efficiently organize queues, minimizing wait times for customers. This leads to improved satisfaction levels as customers spend less time waiting and more time engaging with products or services.
- **Increased Efficiency:** Implementing queue management allows businesses to allocate resources more effectively. By analyzing queue data and optimizing staff deployment, businesses can streamline operations, reduce costs, and improve overall productivity.
- **Real-time Insights:** YOLO26-powered queue management provides instant data on queue lengths and wait times, enabling managers to make informed decisions quickly.
- **Enhanced Customer Experience:** By reducing frustration associated with long waits, businesses can significantly improve customer satisfaction and loyalty.
## Real World Applications
| Logistics | Retail |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Queue management at airport ticket counter using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/queue-management-airport-ticket-counter-ultralytics-yolov8.avif) | ![Queue monitoring in crowd using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/queue-monitoring-crowd-ultralytics-yolov8.avif) |
| Queue management at airport ticket counter Using Ultralytics YOLO26 | Queue monitoring in crowd Ultralytics YOLO26 |
!!! example "Queue Management using Ultralytics YOLO"
=== "CLI"
```bash
# Run a queue example
yolo solutions queue show=True
# Pass a source video
yolo solutions queue source="path/to/video.mp4"
# Pass queue coordinates
yolo solutions queue region="[(20, 400), (1080, 400), (1080, 360), (20, 360)]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("queue_management.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Define queue points
queue_region = [(20, 400), (1080, 400), (1080, 360), (20, 360)] # region points
# queue_region = [(20, 400), (1080, 400), (1080, 360), (20, 360), (20, 400)] # polygon points
# Initialize queue manager object
queuemanager = solutions.QueueManager(
show=True, # display the output
model="yolo26n.pt", # path to the YOLO26 model file
region=queue_region, # pass queue region points
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = queuemanager(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `QueueManager` Arguments
Here's a table with the `QueueManager` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "region"]) }}
The `QueueManagement` solution also support some `track` arguments:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the following visualization parameters are available:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## Implementation Strategies
When implementing queue management with YOLO26, consider these best practices:
1. **Strategic Camera Placement:** Position cameras to capture the entire queue area without obstructions.
2. **Define Appropriate Queue Regions:** Carefully set queue boundaries based on the physical layout of your space.
3. **Adjust Detection Confidence:** Fine-tune the confidence threshold based on lighting conditions and crowd density.
4. **Integrate with Existing Systems:** Connect your queue management solution with digital signage or staff notification systems for automated responses.
## FAQ
### How can I use Ultralytics YOLO26 for real-time queue management?
To use Ultralytics YOLO26 for real-time queue management, you can follow these steps:
1. Load the YOLO26 model with `YOLO("yolo26n.pt")`.
2. Capture the video feed using `cv2.VideoCapture`.
3. Define the region of interest (ROI) for queue management.
4. Process frames to detect objects and manage queues.
Here's a minimal example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
queue_region = [(20, 400), (1080, 400), (1080, 360), (20, 360)]
queuemanager = solutions.QueueManager(
model="yolo26n.pt",
region=queue_region,
line_width=3,
show=True,
)
while cap.isOpened():
success, im0 = cap.read()
if success:
results = queuemanager(im0)
cap.release()
cv2.destroyAllWindows()
```
Leveraging [Ultralytics Platform](https://docs.ultralytics.com/platform/) can streamline this process by providing a user-friendly platform for deploying and managing your queue management solution.
### What are the key advantages of using Ultralytics YOLO26 for queue management?
Using Ultralytics YOLO26 for queue management offers several benefits:
- **Plummeting Waiting Times:** Efficiently organizes queues, reducing customer wait times and boosting satisfaction.
- **Enhancing Efficiency:** Analyzes queue data to optimize staff deployment and operations, thereby reducing costs.
- **Real-time Alerts:** Provides real-time notifications for long queues, enabling quick intervention.
- **Scalability:** Easily scalable across different environments like retail, airports, and healthcare.
For more details, explore our [Queue Management](https://docs.ultralytics.com/reference/solutions/queue_management/) solutions.
### Why should I choose Ultralytics YOLO26 over competitors like [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) or Detectron2 for queue management?
Ultralytics YOLO26 has several advantages over TensorFlow and Detectron2 for queue management:
- **Real-time Performance:** YOLO26 is known for its real-time detection capabilities, offering faster processing speeds.
- **Ease of Use:** Ultralytics provides a user-friendly experience, from training to deployment, via [Ultralytics Platform](https://docs.ultralytics.com/platform/).
- **Pretrained Models:** Access to a range of pretrained models, minimizing the time needed for setup.
- **Community Support:** Extensive documentation and active community support make problem-solving easier.
Learn how to get started with [Ultralytics YOLO](https://docs.ultralytics.com/quickstart/).
### Can Ultralytics YOLO26 handle multiple types of queues, such as in airports and retail?
Yes, Ultralytics YOLO26 can manage various types of queues, including those in airports and retail environments. By configuring the QueueManager with specific regions and settings, YOLO26 can adapt to different queue layouts and densities.
Example for airports:
```python
queue_region_airport = [(50, 600), (1200, 600), (1200, 550), (50, 550)]
queue_airport = solutions.QueueManager(
model="yolo26n.pt",
region=queue_region_airport,
line_width=3,
)
```
For more information on diverse applications, check out our [Real World Applications](#real-world-applications) section.
### What are some real-world applications of Ultralytics YOLO26 in queue management?
Ultralytics YOLO26 is used in various real-world applications for queue management:
- **Retail:** Monitors checkout lines to reduce wait times and improve customer satisfaction.
- **Airports:** Manages queues at ticket counters and security checkpoints for a smoother passenger experience.
- **Healthcare:** Optimizes patient flow in clinics and hospitals.
- **Banks:** Enhances customer service by managing queues efficiently in banks.
Check our [blog on real-world queue management](https://www.ultralytics.com/blog/a-look-at-real-time-queue-monitoring-enabled-by-computer-vision) to learn more about how computer vision is transforming queue monitoring across industries.

499
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/raspberry-pi.md Normal file
View File

@@ -0,0 +1,499 @@
---
comments: true
description: Learn how to deploy Ultralytics YOLO26 on Raspberry Pi with our comprehensive guide. Get performance benchmarks, setup instructions, and best practices.
keywords: Ultralytics, YOLO26, Raspberry Pi, setup, guide, benchmarks, computer vision, object detection, NCNN, Docker, camera modules
---
# Quick Start Guide: Raspberry Pi with Ultralytics YOLO26
This comprehensive guide provides a detailed walkthrough for deploying Ultralytics YOLO26 on [Raspberry Pi](https://www.raspberrypi.com/) devices. Additionally, it showcases performance benchmarks to demonstrate the capabilities of YOLO26 on these small and powerful devices.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/yul4gq_LrOI"
title="Introducing Raspberry Pi 5" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Raspberry Pi 5 updates and improvements.
</p>
!!! note
This guide has been tested with Raspberry Pi 4 and Raspberry Pi 5 running the latest [Raspberry Pi OS Bookworm (Debian 12)](https://www.raspberrypi.com/software/operating-systems/). Using this guide for older Raspberry Pi devices such as the Raspberry Pi 3 is expected to work as long as the same Raspberry Pi OS Bookworm is installed.
## What is Raspberry Pi?
Raspberry Pi is a small, affordable, single-board computer. It has become popular for a wide range of projects and applications, from hobbyist home automation to industrial uses. Raspberry Pi boards are capable of running a variety of operating systems, and they offer GPIO (General Purpose Input/Output) pins that allow for easy integration with sensors, actuators, and other hardware components. They come in different models with varying specifications, but they all share the same basic design philosophy of being low-cost, compact, and versatile.
## Raspberry Pi Series Comparison
| | Raspberry Pi 3 | Raspberry Pi 4 | Raspberry Pi 5 |
| ----------------- | -------------------------------------- | -------------------------------------- | -------------------------------------- |
| CPU | Broadcom BCM2837, Cortex-A53 64Bit SoC | Broadcom BCM2711, Cortex-A72 64Bit SoC | Broadcom BCM2712, Cortex-A76 64Bit SoC |
| CPU Max Frequency | 1.4GHz | 1.8GHz | 2.4GHz |
| GPU | Videocore IV | Videocore VI | VideoCore VII |
| GPU Max Frequency | 400Mhz | 500Mhz | 800Mhz |
| Memory | 1GB LPDDR2 SDRAM | 1GB, 2GB, 4GB, 8GB LPDDR4-3200 SDRAM | 4GB, 8GB LPDDR4X-4267 SDRAM |
| PCIe | N/A | N/A | 1xPCIe 2.0 Interface |
| Max Power Draw | 2.5A@5V | 3A@5V | 5A@5V (PD enabled) |
## What is Raspberry Pi OS?
[Raspberry Pi OS](https://www.raspberrypi.com/software/) (formerly known as Raspbian) is a Unix-like operating system based on the Debian GNU/Linux distribution for the Raspberry Pi family of compact single-board computers distributed by the Raspberry Pi Foundation. Raspberry Pi OS is highly optimized for the Raspberry Pi with ARM CPUs and uses a modified LXDE desktop environment with the Openbox stacking window manager. Raspberry Pi OS is under active development, with an emphasis on improving the stability and performance of as many Debian packages as possible on Raspberry Pi.
## Flash Raspberry Pi OS to Raspberry Pi
The first thing to do after getting your hands on a Raspberry Pi is to flash a micro-SD card with Raspberry Pi OS, insert into the device and boot into the OS. Follow along with detailed [Getting Started Documentation by Raspberry Pi](https://www.raspberrypi.com/documentation/computers/getting-started.html) to prepare your device for first use.
## Set Up Ultralytics
There are two ways of setting up Ultralytics package on Raspberry Pi to build your next [Computer Vision](https://www.ultralytics.com/glossary/computer-vision-cv) project. You can use either of them.
- [Start with Docker](#start-with-docker)
- [Start without Docker](#start-without-docker)
### Start with Docker
The fastest way to get started with Ultralytics YOLO26 on Raspberry Pi is to run with pre-built docker image for Raspberry Pi.
Execute the below command to pull the Docker container and run on Raspberry Pi. This is based on [arm64v8/debian](https://hub.docker.com/r/arm64v8/debian) docker image which contains Debian 12 (Bookworm) in a Python3 environment.
```bash
t=ultralytics/ultralytics:latest-arm64
sudo docker pull $t && sudo docker run -it --ipc=host $t
```
After this is done, skip to [Use NCNN on Raspberry Pi section](#use-ncnn-on-raspberry-pi).
### Start without Docker
#### Install Ultralytics Package
Here we will install Ultralytics package on the Raspberry Pi with optional dependencies so that we can export the [PyTorch](https://www.ultralytics.com/glossary/pytorch) models to other different formats.
1. Update packages list, install pip and upgrade to latest
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install `ultralytics` pip package with optional dependencies
```bash
pip install ultralytics[export]
```
3. Reboot the device
```bash
sudo reboot
```
## Use NCNN on Raspberry Pi
Out of all the model export formats supported by Ultralytics, [NCNN](https://docs.ultralytics.com/integrations/ncnn/) delivers the best inference performance when working with Raspberry Pi devices because NCNN is highly optimized for mobile/ embedded platforms (such as ARM architecture).
## Convert Model to NCNN and Run Inference
The YOLO26n model in PyTorch format is converted to NCNN to run inference with the exported model.
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Export the model to NCNN format
model.export(format="ncnn") # creates 'yolo26n_ncnn_model'
# Load the exported NCNN model
ncnn_model = YOLO("yolo26n_ncnn_model")
# Run inference
results = ncnn_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO26n PyTorch model to NCNN format
yolo export model=yolo26n.pt format=ncnn # creates 'yolo26n_ncnn_model'
# Run inference with the exported model
yolo predict model='yolo26n_ncnn_model' source='https://ultralytics.com/images/bus.jpg'
```
!!! tip
For more details about supported export options, visit the [Ultralytics documentation page on deployment options](https://docs.ultralytics.com/guides/model-deployment-options/).
## Raspberry Pi 5 YOLO26 Benchmarks
YOLO26 benchmarks were run by the Ultralytics team on ten different model formats measuring speed and [accuracy](https://www.ultralytics.com/glossary/accuracy): PyTorch, TorchScript, ONNX, OpenVINO, TF SavedModel, TF GraphDef, TF Lite, MNN, NCNN, ExecuTorch. Benchmarks were run on a Raspberry Pi 5 at FP32 [precision](https://www.ultralytics.com/glossary/precision) with default input image size of 640.
### Comparison Chart
We have only included benchmarks for YOLO26n and YOLO26s models because other model sizes are too big to run on the Raspberry Pis and do not offer decent performance.
<figure style="text-align: center;">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/raspberry-pi-yolo26-benchmarks.avif" alt="YOLO26 benchmarks on RPi 5">
<figcaption style="font-style: italic; color: gray;">Benchmarked with Ultralytics 8.4.1</figcaption>
</figure>
### Detailed Comparison Table
The below table represents the benchmark results for two different models (YOLO26n, YOLO26s) across ten different formats (PyTorch, TorchScript, ONNX, OpenVINO, TF SavedModel, TF GraphDef, TF Lite, MNN, NCNN, ExecuTorch), running on a Raspberry Pi 5, giving us the status, size, mAP50-95(B) metric, and inference time for each combination.
!!! tip "Performance"
=== "YOLO26n"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|---------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 5.3 | 0.4798 | 302.15 |
| TorchScript | ✅ | 9.8 | 0.4764 | 357.58 |
| ONNX | ✅ | 9.5 | 0.4764 | 130.33 |
| OpenVINO | ✅ | 9.6 | 0.4818 | 70.74 |
| TF SavedModel | ✅ | 24.6 | 0.4764 | 213.58 |
| TF GraphDef | ✅ | 9.5 | 0.4764 | 213.5 |
| TF Lite | ✅ | 9.9 | 0.4764 | 251.41 |
| MNN | ✅ | 9.4 | 0.4784 | 90.89 |
| NCNN | ✅ | 9.4 | 0.4805 | 67.69 |
| ExecuTorch | ✅ | 9.4 | 0.4764 | 148.36 |
=== "YOLO26s"
| Format | Status | Size on disk (MB) | mAP50-95(B) | Inference time (ms/im) |
|---------------|--------|-------------------|-------------|------------------------|
| PyTorch | ✅ | 19.5 | 0.5740 | 836.54 |
| TorchScript | ✅ | 36.8 | 0.5665 | 1032.25 |
| ONNX | ✅ | 36.5 | 0.5665 | 351.96 |
| OpenVINO | ✅ | 36.7 | 0.5654 | 158.6 |
| TF SavedModel | ✅ | 92.2 | 0.5665 | 507.6 |
| TF GraphDef | ✅ | 36.5 | 0.5665 | 525.64 |
| TF Lite | ✅ | 36.9 | 0.5665 | 805.3 |
| MNN | ✅ | 36.4 | 0.5644 | 236.47 |
| NCNN | ✅ | 36.4 | 0.5697 | 168.47 |
| ExecuTorch | ✅ | 36.5 | 0.5665 | 388.72 |
Benchmarked with Ultralytics 8.4.1
!!! note
Inference time does not include pre/ post-processing.
## Reproduce Our Results
To reproduce the above Ultralytics benchmarks on all [export formats](../modes/export.md), run this code:
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Benchmark YOLO26n speed and accuracy on the COCO128 dataset for all export formats
results = model.benchmark(data="coco128.yaml", imgsz=640)
```
=== "CLI"
```bash
# Benchmark YOLO26n speed and accuracy on the COCO128 dataset for all export formats
yolo benchmark model=yolo26n.pt data=coco128.yaml imgsz=640
```
Note that benchmarking results might vary based on the exact hardware and software configuration of a system, as well as the current workload of the system at the time the benchmarks are run. For the most reliable results, use a dataset with a large number of images, e.g., `data='coco.yaml'` (5000 val images).
## Use Raspberry Pi Camera
When using Raspberry Pi for Computer Vision projects, it can be essential to grab real-time video feeds to perform inference. The onboard MIPI CSI connector on the Raspberry Pi allows you to connect official Raspberry PI camera modules. In this guide, we have used a [Raspberry Pi Camera Module 3](https://www.raspberrypi.com/products/camera-module-3/) to grab the video feeds and perform inference using YOLO26 models.
!!! tip
Learn more about the [different camera modules offered by Raspberry Pi](https://www.raspberrypi.com/documentation/accessories/camera.html) and also [how to get started with the Raspberry Pi camera modules](https://www.raspberrypi.com/documentation/computers/camera_software.html#introducing-the-raspberry-pi-cameras).
!!! note
Raspberry Pi 5 uses smaller CSI connectors than the Raspberry Pi 4 (15-pin vs 22-pin), so you will need a [15-pin to 22-pin adapter cable](https://www.raspberrypi.com/products/camera-cable/) to connect to a Raspberry Pi Camera.
### Test the Camera
Execute the following command after connecting the camera to the Raspberry Pi. You should see a live video feed from the camera for about 5 seconds.
```bash
rpicam-hello
```
!!! tip
Learn more about [`rpicam-hello` usage on official Raspberry Pi documentation](https://www.raspberrypi.com/documentation/computers/camera_software.html#rpicam-hello)
### Inference with Camera
There are 2 methods of using the Raspberry Pi Camera to run inference on YOLO26 models.
!!! usage
=== "Method 1"
We can use `picamera2` which comes pre-installed with Raspberry Pi OS to access the camera and run inference on YOLO26 models.
!!! example
=== "Python"
```python
import cv2
from picamera2 import Picamera2
from ultralytics import YOLO
# Initialize the Picamera2
picam2 = Picamera2()
picam2.preview_configuration.main.size = (1280, 720)
picam2.preview_configuration.main.format = "RGB888"
picam2.preview_configuration.align()
picam2.configure("preview")
picam2.start()
# Load the YOLO26 model
model = YOLO("yolo26n.pt")
while True:
# Capture frame-by-frame
frame = picam2.capture_array()
# Run YOLO26 inference on the frame
results = model(frame)
# Visualize the results on the frame
annotated_frame = results[0].plot()
# Display the resulting frame
cv2.imshow("Camera", annotated_frame)
# Break the loop if 'q' is pressed
if cv2.waitKey(1) == ord("q"):
break
# Release resources and close windows
cv2.destroyAllWindows()
```
=== "Method 2"
We need to initiate a TCP stream with `rpicam-vid` from the connected camera so that we can use this stream URL as an input when we are inferencing later. Execute the following command to start the TCP stream.
```bash
rpicam-vid -n -t 0 --inline --listen -o tcp://127.0.0.1:8888
```
Learn more about [`rpicam-vid` usage on official Raspberry Pi documentation](https://www.raspberrypi.com/documentation/computers/camera_software.html#rpicam-vid)
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Run inference
results = model("tcp://127.0.0.1:8888")
```
=== "CLI"
```bash
yolo predict model=yolo26n.pt source="tcp://127.0.0.1:8888"
```
!!! tip
Check our document on [Inference Sources](https://docs.ultralytics.com/modes/predict/#inference-sources) if you want to change the image/video input type
## Best Practices when using Raspberry Pi
There are a couple of best practices to follow in order to enable maximum performance on Raspberry Pis running YOLO26.
1. Use an SSD
When using Raspberry Pi for 24x7 continued usage, it is recommended to use an SSD for the system because an SD card will not be able to withstand continuous writes and might get broken. With the onboard PCIe connector on the Raspberry Pi 5, now you can connect SSDs using an adapter such as the [NVMe Base for Raspberry Pi 5](https://shop.pimoroni.com/products/nvme-base).
2. Flash without GUI
When flashing Raspberry Pi OS, you can choose to not install the Desktop environment (Raspberry Pi OS Lite) and this can save a bit of RAM on the device, leaving more space for computer vision processing.
3. Overclock Raspberry Pi
If you want a little boost in performance while running Ultralytics YOLO26 models on Raspberry Pi 5, you can overclock the CPU from its base 2.4GHz to 2.9GHz and the GPU from 800MHz to 1GHz. If the system becomes unstable or crashes, reduce the overclock values by 100MHz increments. Ensure proper cooling is in place, as overclocking increases heat generation and may lead to thermal throttling.
a. Upgrade the software
```bash
sudo apt update && sudo apt dist-upgrade
```
b. Open to edit the configuration file
```bash
sudo nano /boot/firmware/config.txt
```
c. Add the following lines at the bottom
```bash
arm_freq=3000
gpu_freq=1000
force_turbo=1
```
d. Save and exit by pressing CTRL + X, then Y, and hit ENTER
e. Reboot the Raspberry Pi
## Next Steps
You have successfully set up YOLO on your Raspberry Pi. For further learning and support, visit [Ultralytics YOLO26 Docs](../index.md) and [Kashmir World Foundation](https://www.kashmirworldfoundation.org/).
## Acknowledgments and Citations
This guide was initially created by Daan Eeltink for Kashmir World Foundation, an organization dedicated to the use of YOLO for the conservation of endangered species. We acknowledge their pioneering work and educational focus in the realm of object detection technologies.
For more information about Kashmir World Foundation's activities, you can visit their [website](https://www.kashmirworldfoundation.org/).
## FAQ
### How do I set up Ultralytics YOLO26 on a Raspberry Pi without using Docker?
To set up Ultralytics YOLO26 on a Raspberry Pi without Docker, follow these steps:
1. Update the package list and install `pip`:
```bash
sudo apt update
sudo apt install python3-pip -y
pip install -U pip
```
2. Install the Ultralytics package with optional dependencies:
```bash
pip install ultralytics[export]
```
3. Reboot the device to apply changes:
```bash
sudo reboot
```
For detailed instructions, refer to the [Start without Docker](#start-without-docker) section.
### Why should I use Ultralytics YOLO26's NCNN format on Raspberry Pi for AI tasks?
Ultralytics YOLO26's NCNN format is highly optimized for mobile and embedded platforms, making it ideal for running AI tasks on Raspberry Pi devices. NCNN maximizes inference performance by leveraging ARM architecture, providing faster and more efficient processing compared to other formats. For more details on supported export options, visit the [Ultralytics documentation page on deployment options](https://docs.ultralytics.com/guides/model-deployment-options/).
### How can I convert a YOLO26 model to NCNN format for use on Raspberry Pi?
You can convert a PyTorch YOLO26 model to NCNN format using either Python or CLI commands:
!!! example
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO26n PyTorch model
model = YOLO("yolo26n.pt")
# Export the model to NCNN format
model.export(format="ncnn") # creates 'yolo26n_ncnn_model'
# Load the exported NCNN model
ncnn_model = YOLO("yolo26n_ncnn_model")
# Run inference
results = ncnn_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO26n PyTorch model to NCNN format
yolo export model=yolo26n.pt format=ncnn # creates 'yolo26n_ncnn_model'
# Run inference with the exported model
yolo predict model='yolo26n_ncnn_model' source='https://ultralytics.com/images/bus.jpg'
```
For more details, see the [Use NCNN on Raspberry Pi](#use-ncnn-on-raspberry-pi) section.
### What are the hardware differences between Raspberry Pi 4 and Raspberry Pi 5 relevant to running YOLO26?
Key differences include:
- **CPU**: Raspberry Pi 4 uses Broadcom BCM2711, Cortex-A72 64-bit SoC, while Raspberry Pi 5 uses Broadcom BCM2712, Cortex-A76 64-bit SoC.
- **Max CPU Frequency**: Raspberry Pi 4 has a max frequency of 1.8GHz, whereas Raspberry Pi 5 reaches 2.4GHz.
- **Memory**: Raspberry Pi 4 offers up to 8GB of LPDDR4-3200 SDRAM, while Raspberry Pi 5 features LPDDR4X-4267 SDRAM, available in 4GB and 8GB variants.
These enhancements contribute to better performance benchmarks for YOLO26 models on Raspberry Pi 5 compared to Raspberry Pi 4. Refer to the [Raspberry Pi Series Comparison](#raspberry-pi-series-comparison) table for more details.
### How can I set up a Raspberry Pi Camera Module to work with Ultralytics YOLO26?
There are two methods to set up a Raspberry Pi Camera for YOLO26 inference:
1. **Using `picamera2`**:
```python
import cv2
from picamera2 import Picamera2
from ultralytics import YOLO
picam2 = Picamera2()
picam2.preview_configuration.main.size = (1280, 720)
picam2.preview_configuration.main.format = "RGB888"
picam2.preview_configuration.align()
picam2.configure("preview")
picam2.start()
model = YOLO("yolo26n.pt")
while True:
frame = picam2.capture_array()
results = model(frame)
annotated_frame = results[0].plot()
cv2.imshow("Camera", annotated_frame)
if cv2.waitKey(1) == ord("q"):
break
cv2.destroyAllWindows()
```
2. **Using a TCP Stream**:
```bash
rpicam-vid -n -t 0 --inline --listen -o tcp://127.0.0.1:8888
```
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
results = model("tcp://127.0.0.1:8888")
```
For detailed setup instructions, visit the [Inference with Camera](#inference-with-camera) section.

156
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/region-counting.md Normal file
View File

@@ -0,0 +1,156 @@
---
comments: true
description: Learn how to use Ultralytics YOLO26 for precise object counting in specified regions, enhancing efficiency across various applications.
keywords: object counting, regions, YOLO26, computer vision, Ultralytics, efficiency, accuracy, automation, real-time, applications, surveillance, monitoring
---
# Object Counting in Different Regions using Ultralytics YOLO 🚀
## What is Object Counting in Regions?
[Object counting](../guides/object-counting.md) in regions with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) involves precisely determining the number of objects within specified areas using advanced [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv). This approach is valuable for optimizing processes, enhancing security, and improving efficiency in various applications.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/mzLfC13ISF4"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Object Counting in Different Regions using Ultralytics YOLO26 | Ultralytics Solutions 🚀
</p>
## Advantages of Object Counting in Regions
- **[Precision](https://www.ultralytics.com/glossary/precision) and Accuracy:** Object counting in regions with advanced computer vision ensures precise and accurate counts, minimizing errors often associated with manual counting.
- **Efficiency Improvement:** Automated object counting enhances operational efficiency, providing real-time results and streamlining processes across different applications.
- **Versatility and Application:** The versatility of object counting in regions makes it applicable across various domains, from manufacturing and surveillance to traffic monitoring, contributing to its widespread utility and effectiveness.
## Real World Applications
| Retail | Market Streets |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![People Counting in Different Region using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/people-counting-different-region-ultralytics-yolov8.avif) | ![Crowd Counting in Different Region using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/crowd-counting-different-region-ultralytics-yolov8.avif) |
| People Counting in Different Region using Ultralytics YOLO26 | Crowd Counting in Different Region using Ultralytics YOLO26 |
## Usage Examples
!!! example "Region counting using Ultralytics YOLO"
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Pass region as list
# region_points = [(20, 400), (1080, 400), (1080, 360), (20, 360)]
# Pass region as dictionary
region_points = {
"region-01": [(50, 50), (250, 50), (250, 250), (50, 250)],
"region-02": [(640, 640), (780, 640), (780, 720), (640, 720)],
}
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("region_counting.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize region counter object
regioncounter = solutions.RegionCounter(
show=True, # display the frame
region=region_points, # pass region points
model="yolo26n.pt", # model for counting in regions, e.g., yolo26s.pt
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = regioncounter(im0)
# print(results) # access the output
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
!!! tip "Ultralytics Example Code"
The Ultralytics region counting module is available in our [examples section](https://github.com/ultralytics/ultralytics/blob/main/examples/YOLOv8-Region-Counter/yolov8_region_counter.py). You can explore this example for code customization and modify it to suit your specific use case.
### `RegionCounter` Arguments
Here's a table with the `RegionCounter` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "region"]) }}
The `RegionCounter` solution enables the use of object tracking parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the following visualization settings are supported:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## FAQ
### What is object counting in specified regions using Ultralytics YOLO26?
Object counting in specified regions with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics) involves detecting and tallying the number of objects within defined areas using advanced computer vision. This precise method enhances efficiency and [accuracy](https://www.ultralytics.com/glossary/accuracy) across various applications like manufacturing, surveillance, and traffic monitoring.
### How do I run the region based object counting script with Ultralytics YOLO26?
Follow these steps to run object counting in Ultralytics YOLO26:
1. Clone the Ultralytics repository and navigate to the directory:
```bash
git clone https://github.com/ultralytics/ultralytics
cd ultralytics/examples/YOLOv8-Region-Counter
```
2. Execute the region counting script:
```bash
python yolov8_region_counter.py --source "path/to/video.mp4" --save-img
```
For more options, visit the [Usage Examples](#usage-examples) section.
### Why should I use Ultralytics YOLO26 for object counting in regions?
Using Ultralytics YOLO26 for object counting in regions offers several advantages:
1. **Real-time Processing:** YOLO26's architecture enables fast inference, making it ideal for applications requiring immediate counting results.
2. **Flexible Region Definition:** The solution allows you to define multiple custom regions as polygons, rectangles, or lines to suit your specific monitoring needs.
3. **Multi-class Support:** Count different object types simultaneously within the same regions, providing comprehensive analytics.
4. **Integration Capabilities:** Easily integrate with existing systems through the Ultralytics Python API or command-line interface.
Explore deeper benefits in the [Advantages](#advantages-of-object-counting-in-regions) section.
### What are some real-world applications of object counting in regions?
Object counting with Ultralytics YOLO26 can be applied to numerous real-world scenarios:
- **Retail Analytics:** Count customers in different store sections to optimize layout and staffing.
- **Traffic Management:** Monitor vehicle flow in specific road segments or intersections.
- **Manufacturing:** Track products moving through different production zones.
- **Warehouse Operations:** Count inventory items in designated storage areas.
- **Public Safety:** Monitor crowd density in specific zones during events.
Explore more examples in the [Real World Applications](#real-world-applications) section and the [TrackZone](../guides/trackzone.md) solution for additional zone-based monitoring capabilities.

629
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/ros-quickstart.md Normal file
View File

@@ -0,0 +1,629 @@
---
comments: true
description: Learn to integrate Ultralytics YOLO with your robot running ROS Noetic, utilizing RGB images, depth images, and point clouds for efficient object detection, segmentation, and enhanced robotic perception.
keywords: Ultralytics, YOLO, object detection, deep learning, machine learning, guide, ROS, Robot Operating System, robotics, ROS Noetic, Python, Ubuntu, simulation, visualization, communication, middleware, hardware abstraction, tools, utilities, ecosystem, Noetic Ninjemys, autonomous vehicle, AMV
---
# ROS (Robot Operating System) quickstart guide
<p align="center"> <iframe src="https://player.vimeo.com/video/639236696?h=740f412ce5" width="640" height="360" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe></p>
<p align="center"><a href="https://vimeo.com/639236696">ROS Introduction (captioned)</a> from <a href="https://vimeo.com/osrfoundation">Open Robotics</a> on <a href="https://vimeo.com/">Vimeo</a>.</p>
## What is ROS?
The [Robot Operating System (ROS)](https://www.ros.org/) is an open-source framework widely used in robotics research and industry. ROS provides a collection of [libraries and tools](https://www.ros.org/blog/ecosystem/) to help developers create robot applications. ROS is designed to work with various [robotic platforms](https://robots.ros.org/), making it a flexible and powerful tool for roboticists.
### Key Features of ROS
1. **Modular Architecture**: ROS has a modular architecture, allowing developers to build complex systems by combining smaller, reusable components called [nodes](https://wiki.ros.org/ROS/Tutorials/UnderstandingNodes). Each node typically performs a specific function, and nodes communicate with each other using messages over [topics](https://wiki.ros.org/ROS/Tutorials/UnderstandingTopics) or [services](https://wiki.ros.org/ROS/Tutorials/UnderstandingServicesParams).
2. **Communication Middleware**: ROS offers a robust communication infrastructure that supports inter-process communication and distributed computing. This is achieved through a publish-subscribe model for data streams (topics) and a request-reply model for service calls.
3. **Hardware Abstraction**: ROS provides a layer of abstraction over the hardware, enabling developers to write device-agnostic code. This allows the same code to be used with different hardware setups, facilitating easier integration and experimentation.
4. **Tools and Utilities**: ROS comes with a rich set of tools and utilities for visualization, debugging, and simulation. For instance, RViz is used for visualizing sensor data and robot state information, while Gazebo provides a powerful simulation environment for testing algorithms and robot designs.
5. **Extensive Ecosystem**: The ROS ecosystem is vast and continually growing, with numerous packages available for different robotic applications, including navigation, manipulation, perception, and more. The community actively contributes to the development and maintenance of these packages.
???+ note "Evolution of ROS Versions"
Since its development in 2007, ROS has evolved through [multiple versions](https://wiki.ros.org/Distributions), each introducing new features and improvements to meet the growing needs of the robotics community. The development of ROS can be categorized into two main series: ROS 1 and ROS 2. This guide focuses on the Long Term Support (LTS) version of ROS 1, known as ROS Noetic Ninjemys, the code should also work with earlier versions.
### ROS 1 vs. ROS 2
While ROS 1 provided a solid foundation for robotic development, ROS 2 addresses its shortcomings by offering:
- **Real-time Performance**: Improved support for real-time systems and deterministic behavior.
- **Security**: Enhanced security features for safe and reliable operation in various environments.
- **Scalability**: Better support for multi-robot systems and large-scale deployments.
- **Cross-platform Support**: Expanded compatibility with various operating systems beyond Linux, including Windows and macOS.
- **Flexible Communication**: Use of DDS for more flexible and efficient inter-process communication.
### ROS Messages and Topics
In ROS, communication between nodes is facilitated through [messages](https://wiki.ros.org/Messages) and [topics](https://wiki.ros.org/Topics). A message is a data structure that defines the information exchanged between nodes, while a topic is a named channel over which messages are sent and received. Nodes can publish messages to a topic or subscribe to messages from a topic, enabling them to communicate with each other. This publish-subscribe model allows for asynchronous communication and decoupling between nodes. Each sensor or actuator in a robotic system typically publishes data to a topic, which can then be consumed by other nodes for processing or control. For the purpose of this guide, we will focus on Image, Depth and PointCloud messages and camera topics.
## Setting Up Ultralytics YOLO with ROS
This guide has been tested using [this ROS environment](https://github.com/ambitious-octopus/rosbot_ros/tree/noetic), which is a fork of the [ROSbot ROS repository](https://github.com/husarion/rosbot_ros). This environment includes the Ultralytics YOLO package, a Docker container for easy setup, comprehensive ROS packages, and Gazebo worlds for rapid testing. It is designed to work with the [Husarion ROSbot 2 PRO](https://husarion.com/manuals/rosbot/). The code examples provided will work in any ROS Noetic/Melodic environment, including both simulation and real-world.
<p align="center">
<img width="50%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/husarion-rosbot-2-pro.avif" alt="Husarion ROSbot 2 PRO autonomous robot platform">
</p>
### Dependencies Installation
Apart from the ROS environment, you will need to install the following dependencies:
- **[ROS Numpy package](https://github.com/eric-wieser/ros_numpy)**: This is required for fast conversion between ROS Image messages and numpy arrays.
```bash
pip install ros_numpy
```
- **Ultralytics package**:
```bash
pip install ultralytics
```
## Use Ultralytics with ROS `sensor_msgs/Image`
The `sensor_msgs/Image` [message type](https://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html) is commonly used in ROS for representing image data. It contains fields for encoding, height, width, and pixel data, making it suitable for transmitting images captured by cameras or other sensors. Image messages are widely used in robotic applications for tasks such as visual perception, [object detection](https://www.ultralytics.com/glossary/object-detection), and navigation.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/detection-segmentation-ros-gazebo.avif" alt="Detection and Segmentation in ROS Gazebo">
</p>
### Image Step-by-Step Usage
The following code snippet demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topics for [detection](../tasks/detect.md) and [segmentation](../tasks/segment.md).
First, import the necessary libraries and instantiate two models: one for [segmentation](../tasks/segment.md) and one for [detection](../tasks/detect.md). Initialize a ROS node (with the name `ultralytics`) to enable communication with the ROS master. To ensure a stable connection, we include a brief pause, giving the node sufficient time to establish the connection before proceeding.
```python
import time
import rospy
from ultralytics import YOLO
detection_model = YOLO("yolo26m.pt")
segmentation_model = YOLO("yolo26m-seg.pt")
rospy.init_node("ultralytics")
time.sleep(1)
```
Initialize two ROS topics: one for [detection](../tasks/detect.md) and one for [segmentation](../tasks/segment.md). These topics will be used to publish the annotated images, making them accessible for further processing. The communication between nodes is facilitated using `sensor_msgs/Image` messages.
```python
from sensor_msgs.msg import Image
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5)
```
Finally, create a subscriber that listens to messages on the `/camera/color/image_raw` topic and calls a callback function for each new message. This callback function receives messages of type `sensor_msgs/Image`, converts them into a numpy array using `ros_numpy`, processes the images with the previously instantiated YOLO models, annotates the images, and then publishes them back to the respective topics: `/ultralytics/detection/image` for detection and `/ultralytics/segmentation/image` for segmentation.
```python
import ros_numpy
def callback(data):
"""Callback function to process image and publish annotated images."""
array = ros_numpy.numpify(data)
if det_image_pub.get_num_connections():
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
if seg_image_pub.get_num_connections():
seg_result = segmentation_model(array)
seg_annotated = seg_result[0].plot(show=False)
seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
```
??? example "Complete code"
```python
import time
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from ultralytics import YOLO
detection_model = YOLO("yolo26m.pt")
segmentation_model = YOLO("yolo26m-seg.pt")
rospy.init_node("ultralytics")
time.sleep(1)
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5)
def callback(data):
"""Callback function to process image and publish annotated images."""
array = ros_numpy.numpify(data)
if det_image_pub.get_num_connections():
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
if seg_image_pub.get_num_connections():
seg_result = segmentation_model(array)
seg_annotated = seg_result[0].plot(show=False)
seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
```
???+ tip "Debugging"
Debugging ROS (Robot Operating System) nodes can be challenging due to the system's distributed nature. Several tools can assist with this process:
1. `rostopic echo <TOPIC-NAME>` : This command allows you to view messages published on a specific topic, helping you inspect the data flow.
2. `rostopic list`: Use this command to list all available topics in the ROS system, giving you an overview of the active data streams.
3. `rqt_graph`: This visualization tool displays the communication graph between nodes, providing insights into how nodes are interconnected and how they interact.
4. For more complex visualizations, such as 3D representations, you can use [RViz](https://wiki.ros.org/rviz). RViz (ROS Visualization) is a powerful 3D visualization tool for ROS. It allows you to visualize the state of your robot and its environment in real-time. With RViz, you can view sensor data (e.g., `sensor_msgs/Image`), robot model states, and various other types of information, making it easier to debug and understand the behavior of your robotic system.
### Publish Detected Classes with `std_msgs/String`
Standard ROS messages also include `std_msgs/String` messages. In many applications, it is not necessary to republish the entire annotated image; instead, only the classes present in the robot's view are needed. The following example demonstrates how to use `std_msgs/String` [messages](https://docs.ros.org/en/noetic/api/std_msgs/html/msg/String.html) to republish the detected classes on the `/ultralytics/detection/classes` topic. These messages are more lightweight and provide essential information, making them valuable for various applications.
#### Example Use Case
Consider a warehouse robot equipped with a camera and object [detection model](../tasks/detect.md). Instead of sending large annotated images over the network, the robot can publish a list of detected classes as `std_msgs/String` messages. For instance, when the robot detects objects like "box", "pallet" and "forklift" it publishes these classes to the `/ultralytics/detection/classes` topic. This information can then be used by a central monitoring system to track the inventory in real-time, optimize the robot's path planning to avoid obstacles, or trigger specific actions such as picking up a detected box. This approach reduces the bandwidth required for communication and focuses on transmitting critical data.
### String Step-by-Step Usage
This example demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topic `/ultralytics/detection/classes` using `std_msgs/String` messages. The `ros_numpy` package is used to convert the ROS Image message to a numpy array for processing with YOLO.
```python
import time
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from std_msgs.msg import String
from ultralytics import YOLO
detection_model = YOLO("yolo26m.pt")
rospy.init_node("ultralytics")
time.sleep(1)
classes_pub = rospy.Publisher("/ultralytics/detection/classes", String, queue_size=5)
def callback(data):
"""Callback function to process image and publish detected classes."""
array = ros_numpy.numpify(data)
if classes_pub.get_num_connections():
det_result = detection_model(array)
classes = det_result[0].boxes.cls.cpu().numpy().astype(int)
names = [det_result[0].names[i] for i in classes]
classes_pub.publish(String(data=str(names)))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
```
## Use Ultralytics with ROS Depth Images
In addition to RGB images, ROS supports [depth images](https://en.wikipedia.org/wiki/Depth_map), which provide information about the distance of objects from the camera. Depth images are crucial for robotic applications such as obstacle avoidance, 3D mapping, and localization.
A depth image is an image where each pixel represents the distance from the camera to an object. Unlike RGB images that capture color, depth images capture spatial information, enabling robots to perceive the 3D structure of their environment.
!!! tip "Obtaining Depth Images"
Depth images can be obtained using various sensors:
1. [Stereo Cameras](https://en.wikipedia.org/wiki/Stereo_camera): Use two cameras to calculate depth based on image disparity.
2. [Time-of-Flight (ToF) Cameras](https://en.wikipedia.org/wiki/Time-of-flight_camera): Measure the time light takes to return from an object.
3. [Structured Light Sensors](https://en.wikipedia.org/wiki/Structured-light_3D_scanner): Project a pattern and measure its deformation on surfaces.
### Using YOLO with Depth Images
In ROS, depth images are represented by the `sensor_msgs/Image` message type, which includes fields for encoding, height, width, and pixel data. The encoding field for depth images often uses a format like "16UC1", indicating a 16-bit unsigned integer per pixel, where each value represents the distance to the object. Depth images are commonly used in conjunction with RGB images to provide a more comprehensive view of the environment.
Using YOLO, it is possible to extract and combine information from both RGB and depth images. For instance, YOLO can detect objects within an RGB image, and this detection can be used to pinpoint corresponding regions in the depth image. This allows for the extraction of precise depth information for detected objects, enhancing the robot's ability to understand its environment in three dimensions.
!!! warning "RGB-D Cameras"
When working with depth images, it is essential to ensure that the RGB and depth images are correctly aligned. RGB-D cameras, such as the [Intel RealSense](https://realsenseai.com/) series, provide synchronized RGB and depth images, making it easier to combine information from both sources. If using separate RGB and depth cameras, it is crucial to calibrate them to ensure accurate alignment.
#### Depth Step-by-Step Usage
In this example, we use YOLO to segment an image and apply the extracted mask to segment the object in the depth image. This allows us to determine the distance of each pixel of the object of interest from the camera's focal center. By obtaining this distance information, we can calculate the distance between the camera and the specific object in the scene. Begin by importing the necessary libraries, creating a ROS node, and instantiating a segmentation model and a ROS topic.
```python
import time
import rospy
from std_msgs.msg import String
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolo26m-seg.pt")
classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5)
```
Next, define a callback function that processes the incoming depth image message. The function waits for the depth image and RGB image messages, converts them into numpy arrays, and applies the segmentation model to the RGB image. It then extracts the segmentation mask for each detected object and calculates the average distance of the object from the camera using the depth image. Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`. Finally, it publishes the detected objects along with their average distances to the `/ultralytics/detection/distance` topic.
```python
import numpy as np
import ros_numpy
from sensor_msgs.msg import Image
def callback(data):
"""Callback function to process depth image and RGB image."""
image = rospy.wait_for_message("/camera/color/image_raw", Image)
image = ros_numpy.numpify(image)
depth = ros_numpy.numpify(data)
result = segmentation_model(image)
all_objects = []
for index, cls in enumerate(result[0].boxes.cls):
class_index = int(cls.cpu().numpy())
name = result[0].names[class_index]
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
obj = depth[mask == 1]
obj = obj[~np.isnan(obj)]
avg_distance = np.mean(obj) if len(obj) else np.inf
all_objects.append(f"{name}: {avg_distance:.2f}m")
classes_pub.publish(String(data=str(all_objects)))
rospy.Subscriber("/camera/depth/image_raw", Image, callback)
while True:
rospy.spin()
```
??? example "Complete code"
```python
import time
import numpy as np
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from std_msgs.msg import String
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolo26m-seg.pt")
classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5)
def callback(data):
"""Callback function to process depth image and RGB image."""
image = rospy.wait_for_message("/camera/color/image_raw", Image)
image = ros_numpy.numpify(image)
depth = ros_numpy.numpify(data)
result = segmentation_model(image)
all_objects = []
for index, cls in enumerate(result[0].boxes.cls):
class_index = int(cls.cpu().numpy())
name = result[0].names[class_index]
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
obj = depth[mask == 1]
obj = obj[~np.isnan(obj)]
avg_distance = np.mean(obj) if len(obj) else np.inf
all_objects.append(f"{name}: {avg_distance:.2f}m")
classes_pub.publish(String(data=str(all_objects)))
rospy.Subscriber("/camera/depth/image_raw", Image, callback)
while True:
rospy.spin()
```
## Use Ultralytics with ROS `sensor_msgs/PointCloud2`
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/detection-segmentation-ros-gazebo-1.avif" alt="Detection and Segmentation in ROS Gazebo">
</p>
The `sensor_msgs/PointCloud2` [message type](https://docs.ros.org/en/api/sensor_msgs/html/msg/PointCloud2.html) is a data structure used in ROS to represent 3D point cloud data. This message type is integral to robotic applications, enabling tasks such as 3D mapping, object recognition, and localization.
A point cloud is a collection of data points defined within a three-dimensional coordinate system. These data points represent the external surface of an object or a scene, captured via 3D scanning technologies. Each point in the cloud has `X`, `Y`, and `Z` coordinates, which correspond to its position in space, and may also include additional information such as color and intensity.
!!! warning "Reference frame"
When working with `sensor_msgs/PointCloud2`, it's essential to consider the reference frame of the sensor from which the point cloud data was acquired. The point cloud is initially captured in the sensor's reference frame. You can determine this reference frame by listening to the `/tf_static` topic. However, depending on your specific application requirements, you might need to convert the point cloud into another reference frame. This transformation can be achieved using the `tf2_ros` package, which provides tools for managing coordinate frames and transforming data between them.
!!! tip "Obtaining Point clouds"
Point Clouds can be obtained using various sensors:
1. **LIDAR (Light Detection and Ranging)**: Uses laser pulses to measure distances to objects and create high-[precision](https://www.ultralytics.com/glossary/precision) 3D maps.
2. **Depth Cameras**: Capture depth information for each pixel, allowing for 3D reconstruction of the scene.
3. **Stereo Cameras**: Utilize two or more cameras to obtain depth information through triangulation.
4. **Structured Light Scanners**: Project a known pattern onto a surface and measure the deformation to calculate depth.
### Using YOLO with Point Clouds
To integrate YOLO with `sensor_msgs/PointCloud2` type messages, we can employ a method similar to the one used for depth maps. By leveraging the color information embedded in the point cloud, we can extract a 2D image, perform segmentation on this image using YOLO, and then apply the resulting mask to the three-dimensional points to isolate the 3D object of interest.
For handling point clouds, we recommend using Open3D (`pip install open3d`), a user-friendly Python library. Open3D provides robust tools for managing point cloud data structures, visualizing them, and executing complex operations seamlessly. This library can significantly simplify the process and enhance our ability to manipulate and analyze point clouds in conjunction with YOLO-based segmentation.
#### Point Clouds Step-by-Step Usage
Import the necessary libraries and instantiate the YOLO model for segmentation.
```python
import time
import rospy
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolo26m-seg.pt")
```
Create a function `pointcloud2_to_array`, which transforms a `sensor_msgs/PointCloud2` message into two numpy arrays. The `sensor_msgs/PointCloud2` messages contain `n` points based on the `width` and `height` of the acquired image. For instance, a `480 x 640` image will have `307,200` points. Each point includes three spatial coordinates (`xyz`) and the corresponding color in `RGB` format. These can be considered as two separate channels of information.
The function returns the `xyz` coordinates and `RGB` values in the format of the original camera resolution (`width x height`). Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`.
```python
import numpy as np
import ros_numpy
def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple:
"""Convert a ROS PointCloud2 message to a numpy array.
Args:
pointcloud2 (PointCloud2): the PointCloud2 message
Returns:
(tuple): tuple containing (xyz, rgb)
"""
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
nan_rows = np.isnan(xyz).all(axis=2)
xyz[nan_rows] = [0, 0, 0]
rgb[nan_rows] = [0, 0, 0]
return xyz, rgb
```
Next, subscribe to the `/camera/depth/points` topic to receive the point cloud message and convert the `sensor_msgs/PointCloud2` message into numpy arrays containing the XYZ coordinates and RGB values (using the `pointcloud2_to_array` function). Process the RGB image using the YOLO model to extract segmented objects. For each detected object, extract the segmentation mask and apply it to both the RGB image and the XYZ coordinates to isolate the object in 3D space.
Processing the mask is straightforward since it consists of binary values, with `1` indicating the presence of the object and `0` indicating the absence. To apply the mask, simply multiply the original channels by the mask. This operation effectively isolates the object of interest within the image. Finally, create an Open3D point cloud object and visualize the segmented object in 3D space with associated colors.
```python
import sys
import open3d as o3d
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((ros_cloud.height * ros_cloud.width, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((ros_cloud.height * ros_cloud.width, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
```
??? example "Complete code"
```python
import sys
import time
import numpy as np
import open3d as o3d
import ros_numpy
import rospy
from sensor_msgs.msg import PointCloud2
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolo26m-seg.pt")
def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple:
"""Convert a ROS PointCloud2 message to a numpy array.
Args:
pointcloud2 (PointCloud2): the PointCloud2 message
Returns:
(tuple): tuple containing (xyz, rgb)
"""
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
nan_rows = np.isnan(xyz).all(axis=2)
xyz[nan_rows] = [0, 0, 0]
rgb[nan_rows] = [0, 0, 0]
return xyz, rgb
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((ros_cloud.height * ros_cloud.width, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((ros_cloud.height * ros_cloud.width, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
```
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/point-cloud-segmentation-ultralytics.avif" alt="Point Cloud Segmentation with Ultralytics ">
</p>
## FAQ
### What is the Robot Operating System (ROS)?
The [Robot Operating System (ROS)](https://www.ros.org/) is an open-source framework commonly used in robotics to help developers create robust robot applications. It provides a collection of [libraries and tools](https://www.ros.org/blog/ecosystem/) for building and interfacing with robotic systems, enabling easier development of complex applications. ROS supports communication between nodes using messages over [topics](https://wiki.ros.org/ROS/Tutorials/UnderstandingTopics) or [services](https://wiki.ros.org/ROS/Tutorials/UnderstandingServicesParams).
### How do I integrate Ultralytics YOLO with ROS for real-time object detection?
Integrating Ultralytics YOLO with ROS involves setting up a ROS environment and using YOLO for processing sensor data. Begin by installing the required dependencies like `ros_numpy` and Ultralytics YOLO:
```bash
pip install ros_numpy ultralytics
```
Next, create a ROS node and subscribe to an [image topic](../tasks/detect.md) to process the incoming data. Here is a minimal example:
```python
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from ultralytics import YOLO
detection_model = YOLO("yolo26m.pt")
rospy.init_node("ultralytics")
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
def callback(data):
array = ros_numpy.numpify(data)
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
rospy.spin()
```
### What are ROS topics and how are they used in Ultralytics YOLO?
ROS topics facilitate communication between nodes in a ROS network by using a publish-subscribe model. A topic is a named channel that nodes use to send and receive messages asynchronously. In the context of Ultralytics YOLO, you can make a node subscribe to an image topic, process the images using YOLO for tasks like [detection](https://docs.ultralytics.com/tasks/detect/) or [segmentation](https://docs.ultralytics.com/tasks/segment/), and publish outcomes to new topics.
For example, subscribe to a camera topic and process the incoming image for detection:
```python
rospy.Subscriber("/camera/color/image_raw", Image, callback)
```
### Why use depth images with Ultralytics YOLO in ROS?
Depth images in ROS, represented by `sensor_msgs/Image`, provide the distance of objects from the camera, crucial for tasks like obstacle avoidance, 3D mapping, and localization. By [using depth information](https://en.wikipedia.org/wiki/Depth_map) along with RGB images, robots can better understand their 3D environment.
With YOLO, you can extract [segmentation masks](https://www.ultralytics.com/glossary/image-segmentation) from RGB images and apply these masks to depth images to obtain precise 3D object information, improving the robot's ability to navigate and interact with its surroundings.
### How can I visualize 3D point clouds with YOLO in ROS?
To visualize 3D point clouds in ROS with YOLO:
1. Convert `sensor_msgs/PointCloud2` messages to numpy arrays.
2. Use YOLO to segment RGB images.
3. Apply the segmentation mask to the point cloud.
Here's an example using [Open3D](https://www.open3d.org/) for visualization:
```python
import sys
import open3d as o3d
import ros_numpy
import rospy
from sensor_msgs.msg import PointCloud2
from ultralytics import YOLO
rospy.init_node("ultralytics")
segmentation_model = YOLO("yolo26m-seg.pt")
def pointcloud2_to_array(pointcloud2):
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
return xyz, rgb
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((-1, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((-1, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
```
This approach provides a 3D visualization of segmented objects, useful for tasks like navigation and manipulation in [robotics applications](https://docs.ultralytics.com/guides/steps-of-a-cv-project/).

300
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/sahi-tiled-inference.md Normal file
View File

@@ -0,0 +1,300 @@
---
comments: true
description: Learn how to implement YOLO26 with SAHI for sliced inference. Optimize memory usage and enhance detection accuracy for large-scale applications.
keywords: YOLO26, SAHI, Sliced Inference, Object Detection, Ultralytics, High-resolution Images, Computational Efficiency, Integration Guide
---
# Ultralytics Docs: Using YOLO26 with SAHI for Sliced Inference
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-use-ultralytics-yolo-with-sahi.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open SAHI for Sliced Inference In Colab"></a>
Welcome to the Ultralytics documentation on how to use YOLO26 with [SAHI](https://github.com/obss/sahi) (Slicing Aided Hyper Inference). This comprehensive guide aims to furnish you with all the essential knowledge you'll need to implement SAHI alongside YOLO26. We'll deep-dive into what SAHI is, why sliced inference is critical for large-scale applications, and how to integrate these functionalities with YOLO26 for enhanced [object detection](https://www.ultralytics.com/glossary/object-detection) performance.
<p align="center">
<img width="1024" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/sahi-sliced-inference-overview.avif" alt="SAHI tiled inference for small objects">
</p>
## Introduction to SAHI
SAHI (Slicing Aided Hyper Inference) is an innovative library designed to optimize object detection algorithms for large-scale and high-resolution imagery. Its core functionality lies in partitioning images into manageable slices, running object detection on each slice, and then stitching the results back together. SAHI is compatible with a range of object detection models, including the YOLO series, thereby offering flexibility while ensuring optimized use of computational resources.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/ILqMBah5ZvI"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Inference with SAHI (Slicing Aided Hyper Inference) using Ultralytics YOLO26
</p>
### Key Features of SAHI
- **Seamless Integration**: SAHI integrates effortlessly with YOLO models, meaning you can start slicing and detecting without a lot of code modification.
- **Resource Efficiency**: By breaking down large images into smaller parts, SAHI optimizes the memory usage, allowing you to run high-quality detection on hardware with limited resources.
- **High [Accuracy](https://www.ultralytics.com/glossary/accuracy)**: SAHI maintains the detection accuracy by employing smart algorithms to merge overlapping detection boxes during the stitching process.
## What is Sliced Inference?
Sliced Inference refers to the practice of subdividing a large or high-resolution image into smaller segments (slices), conducting object detection on these slices, and then recompiling the slices to reconstruct the object locations on the original image. This technique is invaluable in scenarios where computational resources are limited or when working with extremely high-resolution images that could otherwise lead to memory issues.
### Benefits of Sliced Inference
- **Reduced Computational Burden**: Smaller image slices are faster to process, and they consume less memory, enabling smoother operation on lower-end hardware.
- **Preserved Detection Quality**: Since each slice is treated independently, there is no reduction in the quality of object detection, provided the slices are large enough to capture the objects of interest.
- **Enhanced Scalability**: The technique allows for object detection to be more easily scaled across different sizes and resolutions of images, making it ideal for a wide range of applications from satellite imagery to medical diagnostics.
<table border="0">
<tr>
<th>YOLO26 without SAHI</th>
<th>YOLO26 with SAHI</th>
</tr>
<tr>
<td><img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/yolov8-without-sahi.avif" alt="YOLO26 without SAHI" width="640"></td>
<td><img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/yolov8-with-sahi.avif" alt="YOLO26 with SAHI" width="640"></td>
</tr>
</table>
## Installation and Preparation
### Installation
To get started, install the latest versions of SAHI and Ultralytics:
```bash
pip install -U ultralytics sahi
```
### Import Modules and Download Resources
Here's how to import the necessary modules and download a YOLO26 model and some test images:
```python
from sahi.utils.file import download_from_url
from sahi.utils.ultralytics import download_yolo26n_model
# Download YOLO26 model
model_path = "models/yolo26n.pt"
download_yolo26n_model(model_path)
# Download test images
download_from_url(
"https://raw.githubusercontent.com/obss/sahi/main/demo/demo_data/small-vehicles1.jpeg",
"demo_data/small-vehicles1.jpeg",
)
download_from_url(
"https://raw.githubusercontent.com/obss/sahi/main/demo/demo_data/terrain2.png",
"demo_data/terrain2.png",
)
```
## Standard Inference with YOLO26
### Instantiate the Model
You can instantiate a YOLO26 model for object detection like this:
```python
from sahi import AutoDetectionModel
detection_model = AutoDetectionModel.from_pretrained(
model_type="ultralytics",
model_path=model_path,
confidence_threshold=0.3,
device="cpu", # or 'cuda:0'
)
```
### Perform Standard Prediction
Perform standard inference using an image path or a numpy image.
```python
from sahi.predict import get_prediction
from sahi.utils.cv import read_image
# With an image path
result = get_prediction("demo_data/small-vehicles1.jpeg", detection_model)
# With a numpy image
result_with_np_image = get_prediction(read_image("demo_data/small-vehicles1.jpeg"), detection_model)
```
### Visualize Results
Export and visualize the predicted bounding boxes and masks:
```python
from IPython.display import Image
result.export_visuals(export_dir="demo_data/")
Image("demo_data/prediction_visual.png")
```
## Sliced Inference with YOLO26
Perform sliced inference by specifying the slice dimensions and overlap ratios:
```python
from sahi.predict import get_sliced_prediction
result = get_sliced_prediction(
"demo_data/small-vehicles1.jpeg",
detection_model,
slice_height=256,
slice_width=256,
overlap_height_ratio=0.2,
overlap_width_ratio=0.2,
)
```
## Handling Prediction Results
SAHI provides a `PredictionResult` object, which can be converted into various annotation formats:
```python
# Access the object prediction list
object_prediction_list = result.object_prediction_list
# Convert to COCO annotation, COCO prediction, imantics, and fiftyone formats
result.to_coco_annotations()[:3]
result.to_coco_predictions(image_id=1)[:3]
result.to_imantics_annotations()[:3]
result.to_fiftyone_detections()[:3]
```
## Batch Prediction
For batch prediction on a directory of images:
```python
from sahi.predict import predict
predict(
model_type="ultralytics",
model_path="path/to/yolo26n.pt",
model_device="cpu", # or 'cuda:0'
model_confidence_threshold=0.4,
source="path/to/dir",
slice_height=256,
slice_width=256,
overlap_height_ratio=0.2,
overlap_width_ratio=0.2,
)
```
You are now ready to use YOLO26 with SAHI for both standard and sliced inference.
## Citations and Acknowledgments
If you use SAHI in your research or development work, please cite the original SAHI paper and acknowledge the authors:
!!! quote ""
=== "BibTeX"
```bibtex
@article{akyon2022sahi,
title={Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection},
author={Akyon, Fatih Cagatay and Altinuc, Sinan Onur and Temizel, Alptekin},
journal={2022 IEEE International Conference on Image Processing (ICIP)},
doi={10.1109/ICIP46576.2022.9897990},
pages={966-970},
year={2022}
}
```
We extend our thanks to the SAHI research group for creating and maintaining this invaluable resource for the [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) community. For more information about SAHI and its creators, visit the [SAHI GitHub repository](https://github.com/obss/sahi).
## FAQ
### How can I integrate YOLO26 with SAHI for sliced inference in object detection?
Integrating Ultralytics YOLO26 with SAHI (Slicing Aided Hyper Inference) for sliced inference optimizes your object detection tasks on high-resolution images by partitioning them into manageable slices. This approach improves memory usage and ensures high detection accuracy. To get started, you need to install the ultralytics and sahi libraries:
```bash
pip install -U ultralytics sahi
```
Then, download a YOLO26 model and test images:
```python
from sahi.utils.file import download_from_url
from sahi.utils.ultralytics import download_yolo26n_model
# Download YOLO26 model
model_path = "models/yolo26n.pt"
download_yolo26n_model(model_path)
# Download test images
download_from_url(
"https://raw.githubusercontent.com/obss/sahi/main/demo/demo_data/small-vehicles1.jpeg",
"demo_data/small-vehicles1.jpeg",
)
```
For more detailed instructions, refer to our [Sliced Inference guide](#sliced-inference-with-yolo26).
### Why should I use SAHI with YOLO26 for object detection on large images?
Using SAHI with Ultralytics YOLO26 for object detection on large images offers several benefits:
- **Reduced Computational Burden**: Smaller slices are faster to process and consume less memory, making it feasible to run high-quality detections on hardware with limited resources.
- **Maintained Detection Accuracy**: SAHI uses intelligent algorithms to merge overlapping boxes, preserving the detection quality.
- **Enhanced Scalability**: By scaling object detection tasks across different image sizes and resolutions, SAHI becomes ideal for various applications, such as satellite imagery analysis and medical diagnostics.
Learn more about the [benefits of sliced inference](#benefits-of-sliced-inference) in our documentation.
### Can I visualize prediction results when using YOLO26 with SAHI?
Yes, you can visualize prediction results when using YOLO26 with SAHI. Here's how you can export and visualize the results:
```python
from IPython.display import Image
result.export_visuals(export_dir="demo_data/")
Image("demo_data/prediction_visual.png")
```
This command will save the visualized predictions to the specified directory, and you can then load the image to view it in your notebook or application. For a detailed guide, check out the [Standard Inference section](#visualize-results).
### What features does SAHI offer for improving YOLO26 object detection?
SAHI (Slicing Aided Hyper Inference) offers several features that complement Ultralytics YOLO26 for object detection:
- **Seamless Integration**: SAHI easily integrates with YOLO models, requiring minimal code adjustments.
- **Resource Efficiency**: It partitions large images into smaller slices, which optimizes memory usage and speed.
- **High Accuracy**: By effectively merging overlapping detection boxes during the stitching process, SAHI maintains high detection accuracy.
For a deeper understanding, read about SAHI's [key features](#key-features-of-sahi).
### How do I handle large-scale inference projects using YOLO26 and SAHI?
To handle large-scale inference projects using YOLO26 and SAHI, follow these best practices:
1. **Install Required Libraries**: Ensure that you have the latest versions of ultralytics and sahi.
2. **Configure Sliced Inference**: Determine the optimal slice dimensions and overlap ratios for your specific project.
3. **Run Batch Predictions**: Use SAHI's capabilities to perform batch predictions on a directory of images, which improves efficiency.
Example for batch prediction:
```python
from sahi.predict import predict
predict(
model_type="ultralytics",
model_path="path/to/yolo26n.pt",
model_device="cpu", # or 'cuda:0'
model_confidence_threshold=0.4,
source="path/to/dir",
slice_height=256,
slice_width=256,
overlap_height_ratio=0.2,
overlap_width_ratio=0.2,
)
```
For more detailed steps, visit our section on [Batch Prediction](#batch-prediction).

137
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/security-alarm-system.md Normal file
View File

@@ -0,0 +1,137 @@
---
comments: true
description: Enhance your security with real-time object detection using Ultralytics YOLO26. Reduce false positives and integrate seamlessly with existing systems.
keywords: YOLO26, Security Alarm System, real-time object detection, Ultralytics, computer vision, integration, false positives
---
# Security Alarm System Project Using Ultralytics YOLO26
<img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/security-alarm-system-ultralytics-yolov8.avif" alt="AI-powered security alarm system with object detection">
The Security Alarm System Project utilizing Ultralytics YOLO26 integrates advanced [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) capabilities to enhance security measures. YOLO26, developed by Ultralytics, provides real-time [object detection](https://www.ultralytics.com/glossary/object-detection), allowing the system to identify and respond to potential security threats promptly. This project offers several advantages:
- **Real-time Detection:** YOLO26's efficiency enables the Security Alarm System to detect and respond to security incidents in real-time, minimizing response time.
- **[Accuracy](https://www.ultralytics.com/glossary/accuracy):** YOLO26 is known for its accuracy in object detection, reducing false positives and enhancing the reliability of the security alarm system.
- **Integration Capabilities:** The project can be seamlessly integrated with existing security infrastructure, providing an upgraded layer of intelligent surveillance.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/DTjtBnSK2fY"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Security Alarm System with Ultralytics YOLO26 + Solutions <a href="https://www.ultralytics.com/glossary/object-detection">Object Detection</a>
</p>
???+ note
App Password Generation is necessary
- Navigate to [App Password Generator](https://myaccount.google.com/apppasswords), designate an app name such as "security project," and obtain a 16-digit password. Copy this password and paste it into the designated `password` field in the code below.
!!! example "Security Alarm System using Ultralytics YOLO"
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("security_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
from_email = "abc@gmail.com" # the sender email address
password = "---- ---- ---- ----" # 16-digits password generated via: https://myaccount.google.com/apppasswords
to_email = "xyz@gmail.com" # the receiver email address
# Initialize security alarm object
securityalarm = solutions.SecurityAlarm(
show=True, # display the output
model="yolo26n.pt", # e.g., yolo26s.pt, yolo26m.pt
records=1, # total detections count to send an email
)
securityalarm.authenticate(from_email, password, to_email) # authenticate the email server
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or video processing has been successfully completed.")
break
results = securityalarm(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
When you run the code, you will receive a single email notification if any object is detected. The notification is sent immediately, not repeatedly. You can customize the code to suit your project requirements.
#### Email Received Sample
<img width="256" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/email-received-sample.avif" alt="Security alert email notification example">
### `SecurityAlarm` Arguments
Here's a table with the `SecurityAlarm` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "records"]) }}
The `SecurityAlarm` solution supports a variety of `track` parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization settings are available:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## How It Works
The Security Alarm System uses [object tracking](https://docs.ultralytics.com/modes/track/) to monitor video feeds and detect potential security threats. When the system detects objects that exceed the specified threshold (set by the `records` parameter), it automatically sends an email notification with an image attachment showing the detected objects.
The system leverages the [SecurityAlarm class](https://docs.ultralytics.com/reference/solutions/security_alarm/) which provides methods to:
1. Process frames and extract object detections
2. Annotate frames with bounding boxes around detected objects
3. Send email notifications when detection thresholds are exceeded
This implementation is ideal for home security, retail surveillance, and other monitoring applications where immediate notification of detected objects is critical.
## FAQ
### How does Ultralytics YOLO26 improve the accuracy of a security alarm system?
Ultralytics YOLO26 enhances security alarm systems by delivering high-accuracy, real-time object detection. Its advanced algorithms significantly reduce false positives, ensuring that the system only responds to genuine threats. This increased reliability can be seamlessly integrated with existing security infrastructure, upgrading the overall surveillance quality.
### Can I integrate Ultralytics YOLO26 with my existing security infrastructure?
Yes, Ultralytics YOLO26 can be seamlessly integrated with your existing security infrastructure. The system supports various modes and provides flexibility for customization, allowing you to enhance your existing setup with advanced object detection capabilities. For detailed instructions on integrating YOLO26 in your projects, visit the [integration section](https://docs.ultralytics.com/integrations/).
### What are the storage requirements for running Ultralytics YOLO26?
Running Ultralytics YOLO26 on a standard setup typically requires around 5GB of free disk space. This includes space for storing the YOLO26 model and any additional dependencies. For cloud-based solutions, [Ultralytics Platform](https://docs.ultralytics.com/platform/) offers efficient project management and dataset handling, which can optimize storage needs. Learn more about the [Pro Plan](../platform/account/billing.md) for enhanced features including extended storage.
### What makes Ultralytics YOLO26 different from other object detection models like Faster R-CNN or SSD?
Ultralytics YOLO26 provides an edge over models like Faster R-CNN or SSD with its real-time detection capabilities and higher accuracy. Its unique architecture allows it to process images much faster without compromising on [precision](https://www.ultralytics.com/glossary/precision), making it ideal for time-sensitive applications like security alarm systems. For a comprehensive comparison of object detection models, you can explore our [guide](https://docs.ultralytics.com/models/).
### How can I reduce the frequency of false positives in my security system using Ultralytics YOLO26?
To reduce false positives, ensure your Ultralytics YOLO26 model is adequately trained with a diverse and well-annotated dataset. Fine-tuning hyperparameters and regularly updating the model with new data can significantly improve detection accuracy. Detailed [hyperparameter tuning](https://www.ultralytics.com/glossary/hyperparameter-tuning) techniques can be found in our [hyperparameter tuning guide](../guides/hyperparameter-tuning.md).

169
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/similarity-search.md Normal file
View File

@@ -0,0 +1,169 @@
---
comments: true
description: Build a semantic image search web app using OpenAI CLIP, Meta FAISS, and Flask. Learn how to embed images and retrieve them using natural language.
keywords: CLIP, FAISS, Flask, semantic search, image retrieval, OpenAI, Ultralytics, tutorial, computer vision, web app
---
# Semantic Image Search with OpenAI CLIP and Meta FAISS
## Introduction
This guide walks you through building a **semantic image search** engine using [OpenAI CLIP](https://openai.com/blog/clip), [Meta FAISS](https://github.com/facebookresearch/faiss), and [Flask](https://flask.palletsprojects.com/en/stable/). By combining CLIP's powerful visual-language embeddings with FAISS's efficient nearest-neighbor search, you can create a fully functional web interface where you can retrieve relevant images using natural language queries.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/zplKRlX3sLg"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How Similarity Search Works | Visual Search Using OpenAI CLIP, META FAISS and Ultralytics Package 🎉
</p>
## Semantic Image Search Visual Preview
![Flask webpage with semantic search results overview](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/flask-ui.avif)
## How It Works
- **CLIP** uses a vision encoder (e.g., ResNet or ViT) for images and a text encoder (Transformer-based) for language to project both into the same multimodal embedding space. This allows for direct comparison between text and images using [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).
- **FAISS** (Facebook AI Similarity Search) builds an index of the image embeddings and enables fast, scalable retrieval of the closest vectors to a given query.
- **Flask** provides a simple web interface to submit natural language queries and display semantically matched images from the index.
This architecture supports zero-shot search, meaning you don't need labels or categories, just image data and a good prompt.
!!! example "Semantic Image Search using Ultralytics Python package"
??? note "Image Path Warning"
If you're using your own images, make sure to provide an absolute path to the image directory. Otherwise, the images may not appear on the webpage due to Flask's file serving limitations.
=== "Python"
```python
from ultralytics import solutions
app = solutions.SearchApp(
# data = "path/to/img/directory" # Optional, build search engine with your own images
device="cpu" # configure the device for processing, e.g., "cpu" or "cuda"
)
app.run(debug=False) # You can also use `debug=True` argument for testing
```
## `VisualAISearch` class
This class performs all the backend operations:
- Loads or builds a FAISS index from local images.
- Extracts image and text [embeddings](https://platform.openai.com/docs/guides/embeddings) using CLIP.
- Performs similarity search using cosine similarity.
!!! example "Similar Images Search"
??? note "Image Path Warning"
If you're using your own images, make sure to provide an absolute path to the image directory. Otherwise, the images may not appear on the webpage due to Flask's file serving limitations.
=== "Python"
```python
from ultralytics import solutions
searcher = solutions.VisualAISearch(
# data = "path/to/img/directory" # Optional, build search engine with your own images
device="cuda" # configure the device for processing, e.g., "cpu" or "cuda"
)
results = searcher("a dog sitting on a bench")
# Ranked Results:
# - 000000546829.jpg | Similarity: 0.3269
# - 000000549220.jpg | Similarity: 0.2899
# - 000000517069.jpg | Similarity: 0.2761
# - 000000029393.jpg | Similarity: 0.2742
# - 000000534270.jpg | Similarity: 0.2680
```
## `VisualAISearch` Parameters
The table below outlines the available parameters for `VisualAISearch`:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["data"]) }}
{% from "macros/track-args.md" import param_table %}
{{ param_table(["device"]) }}
## Advantages of Semantic Image Search with CLIP and FAISS
Building your own semantic image search system with CLIP and FAISS provides several compelling advantages:
1. **Zero-Shot Capabilities**: You don't need to train the model on your specific dataset. CLIP's zero-shot learning lets you perform search queries on any image dataset using free-form natural language, saving both time and resources.
2. **Human-Like Understanding**: Unlike keyword-based search engines, CLIP understands semantic context. It can retrieve images based on abstract, emotional, or relational queries like "a happy child in nature" or "a futuristic city skyline at night".
![OpenAI Clip image retrieval workflow](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/clip-image-retrieval.avif)
3. **No Need for Labels or Metadata**: Traditional image search systems require carefully labeled data. This approach only needs raw images. CLIP generates embeddings without needing any manual annotation.
4. **Flexible and Scalable Search**: FAISS enables fast nearest-neighbor search even with large-scale datasets. It's optimized for speed and memory, allowing real-time response even with thousands (or millions) of embeddings.
![Meta FAISS embedding vectors building workflow](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/faiss-indexing-workflow.avif)
5. **Cross-Domain Applications**: Whether you're building a personal photo archive, a creative inspiration tool, a product search engine, or even an art recommendation system, this stack adapts to diverse domains with minimal tweaking.
## FAQ
### How does CLIP understand both images and text?
[CLIP](https://github.com/openai/CLIP) (Contrastive Language Image Pretraining) is a model developed by [OpenAI](https://openai.com/) that learns to connect visual and linguistic information. It's trained on a massive dataset of images paired with natural language captions. This training allows it to map both images and text into a shared embedding space, so you can compare them directly using vector similarity.
### Why is CLIP considered so powerful for AI tasks?
What makes CLIP stand out is its ability to generalize. Instead of being trained just for specific labels or tasks, it learns from natural language itself. This allows it to handle flexible queries like “a man riding a jet ski” or “a surreal dreamscape,” making it useful for everything from classification to creative semantic search, without retraining.
### What exactly does FAISS do in this project (Semantic Search)?
[FAISS](https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/) (Facebook AI Similarity Search) is a toolkit that helps you search through high-dimensional vectors very efficiently. Once CLIP turns your images into embeddings, FAISS makes it fast and easy to find the closest matches to a text query, perfect for real-time image retrieval.
### Why use the [Ultralytics](https://www.ultralytics.com/) [Python package](https://github.com/ultralytics/ultralytics/) if CLIP and FAISS are from OpenAI and Meta?
While CLIP and FAISS are developed by OpenAI and Meta respectively, the [Ultralytics Python package](https://pypi.org/project/ultralytics/) simplifies their integration into a complete semantic image search pipeline in a 2-lines workflow that just works:
!!! example "Similar Images Search"
=== "Python"
```python
from ultralytics import solutions
searcher = solutions.VisualAISearch(
# data = "path/to/img/directory" # Optional, build search engine with your own images
device="cuda" # configure the device for processing, e.g., "cpu" or "cuda"
)
results = searcher("a dog sitting on a bench")
# Ranked Results:
# - 000000546829.jpg | Similarity: 0.3269
# - 000000549220.jpg | Similarity: 0.2899
# - 000000517069.jpg | Similarity: 0.2761
# - 000000029393.jpg | Similarity: 0.2742
# - 000000534270.jpg | Similarity: 0.2680
```
This high-level implementation handles:
- CLIP-based image and text embedding generation.
- FAISS index creation and management.
- Efficient semantic search with cosine similarity.
- Directory-based image loading and [visualization](https://www.ultralytics.com/glossary/data-visualization).
### Can I customize the frontend of this app?
Yes. The current setup uses Flask with a basic HTML frontend, but you can replace it with your own HTML or build a more dynamic UI with React, Vue, or another frontend framework. Flask can serve as the backend API for your custom interface.
### Is it possible to search through videos instead of static images?
Not directly. A simple workaround is to extract individual frames from your videos (e.g., one every second), treat them as standalone images, and feed those into the system. This way, the search engine can semantically index visual moments from your videos.

187
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/speed-estimation.md Normal file
View File

@@ -0,0 +1,187 @@
---
comments: true
description: Learn how to estimate object speed using Ultralytics YOLO26 for applications in traffic control, autonomous navigation, and surveillance.
keywords: Ultralytics YOLO26, speed estimation, object tracking, computer vision, traffic control, autonomous navigation, surveillance, security
---
# Speed Estimation using Ultralytics YOLO26 🚀
## What is Speed Estimation?
[Speed estimation](https://www.ultralytics.com/blog/ultralytics-yolov8-for-speed-estimation-in-computer-vision-projects) is the process of calculating the rate of movement of an object within a given context, often employed in [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) applications. Using [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) you can now calculate the speed of objects using [object tracking](../modes/track.md) alongside distance and time data, crucial for tasks like traffic monitoring and surveillance. The accuracy of speed estimation directly influences the efficiency and reliability of various applications, making it a key component in the advancement of intelligent systems and real-time decision-making processes.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/rCggzXRRSRo"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Speed Estimation using Ultralytics YOLO26
</p>
!!! tip "Check Out Our Blog"
For deeper insights into speed estimation, check out our blog post: [Ultralytics YOLO for Speed Estimation in Computer Vision Projects](https://www.ultralytics.com/blog/ultralytics-yolov8-for-speed-estimation-in-computer-vision-projects)
## Advantages of Speed Estimation
- **Efficient Traffic Control:** Accurate speed estimation aids in managing traffic flow, enhancing safety, and reducing congestion on roadways.
- **Precise Autonomous Navigation:** In autonomous systems like [self-driving cars](https://www.ultralytics.com/solutions/ai-in-automotive), reliable speed estimation ensures safe and accurate vehicle navigation.
- **Enhanced Surveillance Security:** Speed estimation in surveillance analytics helps identify unusual behaviors or potential threats, improving the effectiveness of security measures.
## Real World Applications
| Transportation | Transportation |
| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Speed Estimation on Road using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/speed-estimation-on-road-using-ultralytics-yolov8.avif) | ![Speed Estimation on Bridge using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/speed-estimation-on-bridge-using-ultralytics-yolov8.avif) |
| Speed Estimation on Road using Ultralytics YOLO26 | Speed Estimation on Bridge using Ultralytics YOLO26 |
???+ warning "Speed is an Estimate"
Speed will be an estimate and may not be completely accurate. Additionally, the estimation can vary on camera specifications and related factors.
!!! example "Speed Estimation using Ultralytics YOLO"
=== "CLI"
```bash
# Run a speed example
yolo solutions speed show=True
# Pass a source video
yolo solutions speed source="path/to/video.mp4"
# Adjust meter per pixel value based on camera configuration
yolo solutions speed meter_per_pixel=0.05
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("speed_management.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize speed estimation object
speedestimator = solutions.SpeedEstimator(
show=True, # display the output
model="yolo26n.pt", # path to the YOLO26 model file.
fps=fps, # adjust speed based on frame per second
# max_speed=120, # cap speed to a max value (km/h) to avoid outliers
# max_hist=5, # minimum frames object tracked before computing speed
# meter_per_pixel=0.05, # highly depends on the camera configuration
# classes=[0, 2], # estimate speed of specific classes.
# line_width=2, # adjust the line width for bounding boxes
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = speedestimator(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `SpeedEstimator` Arguments
Here's a table with the `SpeedEstimator` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "fps", "max_hist", "meter_per_pixel", "max_speed"]) }}
The `SpeedEstimator` solution allows the use of `track` parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the following visualization options are supported:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## FAQ
### How do I estimate object speed using Ultralytics YOLO26?
Estimating object speed with Ultralytics YOLO26 involves combining [object detection](https://www.ultralytics.com/glossary/object-detection) and tracking techniques. First, you need to detect objects in each frame using the YOLO26 model. Then, track these objects across frames to calculate their movement over time. Finally, use the distance traveled by the object between frames and the frame rate to estimate its speed.
**Example**:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("speed_estimation.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize SpeedEstimator
speedestimator = solutions.SpeedEstimator(
model="yolo26n.pt",
show=True,
)
while cap.isOpened():
success, im0 = cap.read()
if not success:
break
results = speedestimator(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
```
For more details, refer to our [official blog post](https://www.ultralytics.com/blog/ultralytics-yolov8-for-speed-estimation-in-computer-vision-projects).
### What are the benefits of using Ultralytics YOLO26 for speed estimation in traffic management?
Using Ultralytics YOLO26 for speed estimation offers significant advantages in traffic management:
- **Enhanced Safety**: Accurately estimate vehicle speeds to detect over-speeding and improve road safety.
- **Real-Time Monitoring**: Benefit from YOLO26's real-time object detection capability to monitor traffic flow and congestion effectively.
- **Scalability**: Deploy the model on various hardware setups, from [edge devices](https://docs.ultralytics.com/guides/nvidia-jetson/) to servers, ensuring flexible and scalable solutions for large-scale implementations.
For more applications, see [advantages of speed estimation](#advantages-of-speed-estimation).
### Can YOLO26 be integrated with other AI frameworks like [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) or [PyTorch](https://www.ultralytics.com/glossary/pytorch)?
Yes, YOLO26 can be integrated with other AI frameworks like TensorFlow and PyTorch. Ultralytics provides support for exporting YOLO26 models to various formats like [ONNX](../integrations/onnx.md), [TensorRT](../integrations/tensorrt.md), and [CoreML](../integrations/coreml.md), ensuring smooth interoperability with other ML frameworks.
To export a YOLO26 model to ONNX format:
```bash
yolo export model=yolo26n.pt format=onnx
```
Learn more about exporting models in our [guide on export](../modes/export.md).
### How accurate is the speed estimation using Ultralytics YOLO26?
The [accuracy](https://www.ultralytics.com/glossary/accuracy) of speed estimation using Ultralytics YOLO26 depends on several factors, including the quality of the object tracking, the resolution and frame rate of the video, and environmental variables. While the speed estimator provides reliable estimates, it may not be 100% accurate due to variances in frame processing speed and object occlusion.
**Note**: Always consider margin of error and validate the estimates with ground truth data when possible.
For further accuracy improvement tips, check the [Arguments `SpeedEstimator` section](#speedestimator-arguments).

237
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/steps-of-a-cv-project.md Normal file
View File

@@ -0,0 +1,237 @@
---
comments: true
description: Discover essential steps for launching a successful computer vision project, from defining goals to model deployment and maintenance.
keywords: Computer Vision, AI, Object Detection, Image Classification, Instance Segmentation, Data Annotation, Model Training, Model Evaluation, Model Deployment
---
# Understanding the Key Steps in a Computer Vision Project
## Introduction
Computer vision is a subfield of [artificial intelligence](https://www.ultralytics.com/glossary/artificial-intelligence-ai) (AI) that helps computers see and understand the world like humans do. It processes and analyzes images or videos to extract information, recognize patterns, and make decisions based on that data.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/CfbHwPG01cE"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Do <a href="https://www.ultralytics.com/glossary/computer-vision-cv">Computer Vision</a> Projects | A Step-by-Step Guide
</p>
Computer vision techniques like [object detection](../tasks/detect.md), [image classification](../tasks/classify.md), and [instance segmentation](../tasks/segment.md) can be applied across various industries, from [autonomous driving](https://www.ultralytics.com/solutions/ai-in-automotive) to [medical imaging](https://www.ultralytics.com/solutions/ai-in-healthcare) to gain valuable insights.
Working on your own computer vision projects is a great way to understand and learn more about computer vision. However, a computer vision project can consist of many steps, and it might seem confusing at first. By the end of this guide, you'll be familiar with the steps involved in a computer vision project. We'll walk through everything from the beginning to the end of a project, explaining why each part is important.
## An Overview of a Computer Vision Project
Before discussing the details of each step involved in a computer vision project, let's look at the overall process. If you started a computer vision project today, you'd take the following steps:
- Your first priority would be to understand your project's requirements.
- Then, you'd collect and accurately label the images that will help train your model.
- Next, you'd clean your data and apply augmentation techniques to prepare it for model training.
- After model training, you'd thoroughly test and evaluate your model to make sure it performs consistently under different conditions.
- Finally, you'd deploy your model into the real world and update it based on new insights and feedback.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/five-stages-of-ml-development-lifecycle.avif" alt="Computer Vision Project Steps Overview">
</p>
Now that we know what to expect, let's dive right into the steps and get your project moving forward.
## Step 1: Defining Your Project's Goals
The first step in any computer vision project is clearly defining the problem you're trying to solve. Knowing the end goal helps you start to build a solution. This is especially true when it comes to computer vision because your project's objective will directly affect which computer vision task you need to focus on.
Here are some examples of project objectives and the computer vision tasks that can be used to reach these objectives:
- **Objective:** To develop a system that can monitor and manage the flow of different vehicle types on highways, improving traffic management and safety.
- **Computer Vision Task:** Object detection is ideal for traffic monitoring because it efficiently locates and identifies multiple vehicles. It is less computationally demanding than image segmentation, which provides unnecessary detail for this task, ensuring faster, real-time analysis.
- **Objective:** To develop a tool that assists radiologists by providing precise, pixel-level outlines of tumors in medical imaging scans.
- **Computer Vision Task:** Image segmentation is suitable for medical imaging because it provides accurate and detailed boundaries of tumors that are crucial for assessing size, shape, and treatment planning.
- **Objective:** To create a digital system that categorizes various documents (e.g., invoices, receipts, legal paperwork) to improve organizational efficiency and document retrieval.
- **Computer Vision Task:** [Image classification](https://www.ultralytics.com/glossary/image-classification) is ideal here as it handles one document at a time, without needing to consider the document's position in the image. This approach simplifies and accelerates the sorting process.
### Step 1.5: Selecting the Right Model and Training Approach
After understanding the project objective and suitable computer vision tasks, an essential part of defining the project goal is [selecting the right model](../models/index.md) and training approach.
Depending on the objective, you might choose to select the model first or after seeing what data you are able to collect in Step 2. For example, suppose your project is highly dependent on the availability of specific types of data. In that case, it may be more practical to gather and analyze the data first before selecting a model. On the other hand, if you have a clear understanding of the model requirements, you can choose the model first and then collect data that fits those specifications.
Choosing between training from scratch or using [transfer learning](https://www.ultralytics.com/glossary/transfer-learning) affects how you prepare your data. Training from scratch requires a diverse dataset to build the model's understanding from the ground up. Transfer learning, on the other hand, allows you to use a pretrained model and adapt it with a smaller, more specific dataset. Also, choosing a specific model to train will determine how you need to prepare your data, such as resizing images or adding annotations, according to the model's specific requirements.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/training-from-scratch-vs-transfer-learning.avif" alt="Training From Scratch Vs. Using Transfer Learning">
</p>
Note: When choosing a model, consider its [deployment](./model-deployment-options.md) to ensure compatibility and performance. For example, lightweight models are ideal for [edge computing](https://www.ultralytics.com/glossary/edge-computing) due to their efficiency on resource-constrained devices. To learn more about the key points related to defining your project, read [our guide](./defining-project-goals.md) on defining your project's goals and selecting the right model.
Before getting into the hands-on work of a computer vision project, it's important to have a clear understanding of these details. Double-check that you've considered the following before moving on to Step 2:
- Clearly define the problem you're trying to solve.
- Determine the end goal of your project.
- Identify the specific computer vision task needed (e.g., object detection, image classification, image segmentation).
- Decide whether to train a model from scratch or use transfer learning.
- Select the appropriate model for your task and deployment needs.
## Step 2: Data Collection and Data Annotation
The quality of your computer vision models depends on the quality of your dataset. You can either collect images from the internet, take your own pictures, or use pre-existing datasets. Here are some great resources for downloading high-quality datasets: [Google Dataset Search Engine](https://datasetsearch.research.google.com/), [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/), and [Kaggle Datasets](https://www.kaggle.com/datasets).
Some libraries, like Ultralytics, provide [built-in support for various datasets](../datasets/index.md), making it easier to get started with high-quality data. These libraries often include utilities for using popular datasets seamlessly, which can save you a lot of time and effort in the initial stages of your project.
However, if you choose to collect images or take your own pictures, you'll need to annotate your data. Data annotation is the process of labeling your data to impart knowledge to your model. The type of data annotation you'll work with depends on your specific computer vision technique. Here are some examples:
- **Image Classification:** You'll label the entire image as a single class.
- **[Object Detection](https://www.ultralytics.com/glossary/object-detection):** You'll draw bounding boxes around each object in the image and label each box.
- **[Image Segmentation](https://www.ultralytics.com/glossary/image-segmentation):** You'll label each pixel in the image according to the object it belongs to, creating detailed object boundaries.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/different-types-of-image-annotation.avif" alt="Bounding box, polygon, and keypoint annotations">
</p>
[Data collection and annotation](./data-collection-and-annotation.md) can be a time-consuming manual effort. Annotation tools can help make this process easier. Here are some useful open annotation tools: [LabeI Studio](https://github.com/HumanSignal/label-studio), [CVAT](https://github.com/cvat-ai/cvat), and [Labelme](https://github.com/wkentaro/labelme).
## Step 3: Data Augmentation and Splitting Your Dataset
After collecting and annotating your image data, it's important to first split your dataset into training, validation, and test sets before performing [data augmentation](https://www.ultralytics.com/glossary/data-augmentation). Splitting your dataset before augmentation is crucial to test and validate your model on original, unaltered data. It helps accurately assess how well the model generalizes to new, unseen data.
Here's how to split your data:
- **Training Set:** It is the largest portion of your data, typically 70-80% of the total, used to train your model.
- **Validation Set:** Usually around 10-15% of your data; this set is used to tune hyperparameters and validate the model during training, helping to prevent [overfitting](https://www.ultralytics.com/glossary/overfitting).
- **Test Set:** The remaining 10-15% of your data is set aside as the test set. It is used to evaluate the model's performance on unseen data after training is complete.
After splitting your data, you can perform data augmentation by applying transformations like rotating, scaling, and flipping images to artificially increase the size of your dataset. Data augmentation makes your model more robust to variations and improves its performance on unseen images.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/examples-of-data-augmentations.avif" alt="Data augmentation examples">
</p>
Libraries like [OpenCV](https://www.ultralytics.com/glossary/opencv), [Albumentations](../integrations/albumentations.md), and [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) offer flexible augmentation functions that you can use. Additionally, some libraries, such as Ultralytics, have [built-in augmentation settings](../modes/train.md) directly within its model training function, simplifying the process.
To understand your data better, you can use tools like [Matplotlib](https://matplotlib.org/) or [Seaborn](https://seaborn.pydata.org/) to visualize the images and analyze their distribution and characteristics. Visualizing your data helps identify patterns, anomalies, and the effectiveness of your augmentation techniques. You can also use [Ultralytics Explorer](../datasets/explorer/index.md), a tool for exploring computer vision datasets with semantic search, SQL queries, and vector similarity search.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-1.avif" alt="The Ultralytics Explorer Tool">
</p>
By properly [understanding, splitting, and augmenting your data](./preprocessing_annotated_data.md), you can develop a well-trained, validated, and tested model that performs well in real-world applications.
## Step 4: Model Training
Once your dataset is ready for training, you can focus on setting up the necessary environment, managing your datasets, and training your model.
First, you'll need to make sure your environment is configured correctly. Typically, this includes the following:
- Installing essential libraries and frameworks like TensorFlow, [PyTorch](https://www.ultralytics.com/glossary/pytorch), or [Ultralytics](../quickstart.md).
- If you are using a GPU, installing libraries like CUDA and cuDNN will help enable GPU acceleration and speed up the training process.
Then, you can load your training and validation datasets into your environment. Normalize and preprocess the data through resizing, format conversion, or augmentation. With your model selected, configure the layers and specify hyperparameters. Compile the model by setting the [loss function](https://www.ultralytics.com/glossary/loss-function), optimizer, and performance metrics.
Libraries like Ultralytics simplify the training process. You can [start training](../modes/train.md) by feeding data into the model with minimal code. These libraries handle weight adjustments, [backpropagation](https://www.ultralytics.com/glossary/backpropagation), and validation automatically. They also offer tools to monitor progress and adjust hyperparameters easily. After training, save the model and its weights with a few commands.
It's important to keep in mind that proper dataset management is vital for efficient training. Use version control for datasets to track changes and ensure reproducibility. Tools like [DVC (Data Version Control)](../integrations/dvc.md) can help manage large datasets.
## Step 5: Model Evaluation and Model Finetuning
It's important to assess your model's performance using various metrics and refine it to improve [accuracy](https://www.ultralytics.com/glossary/accuracy). [Evaluating](../modes/val.md) helps identify areas where the model excels and where it may need improvement. [Fine-tuning](https://www.ultralytics.com/glossary/fine-tuning) ensures the model is optimized for the best possible performance.
- **[Performance Metrics](./yolo-performance-metrics.md):** Use metrics like accuracy, [precision](https://www.ultralytics.com/glossary/precision), [recall](https://www.ultralytics.com/glossary/recall), and F1-score to evaluate your model's performance. These metrics provide insights into how well your model is making predictions.
- **[Hyperparameter Tuning](./hyperparameter-tuning.md):** Adjust hyperparameters to optimize model performance. Techniques like grid search or random search can help find the best hyperparameter values.
- **Fine-Tuning:** Make small adjustments to the model architecture or training process to enhance performance. This might involve tweaking [learning rates](https://www.ultralytics.com/glossary/learning-rate), [batch sizes](https://www.ultralytics.com/glossary/batch-size), or other model parameters.
For a deeper understanding of model evaluation and fine-tuning techniques, check out our [model evaluation insights guide](./model-evaluation-insights.md).
## Step 6: Model Testing
In this step, you can make sure that your model performs well on completely unseen data, confirming its readiness for deployment. The difference between model testing and model evaluation is that it focuses on verifying the final model's performance rather than iteratively improving it.
It's important to thoroughly test and debug any common issues that may arise. Test your model on a separate test dataset that was not used during training or validation. This dataset should represent real-world scenarios to ensure the model's performance is consistent and reliable.
Also, address common problems such as overfitting, [underfitting](https://www.ultralytics.com/glossary/underfitting), and data leakage. Use techniques like [cross-validation](https://www.ultralytics.com/glossary/cross-validation) and [anomaly detection](https://www.ultralytics.com/glossary/anomaly-detection) to identify and fix these issues. For comprehensive testing strategies, refer to our [model testing guide](./model-testing.md).
## Step 7: Model Deployment
Once your model has been thoroughly tested, it's time to deploy it. [Model deployment](https://www.ultralytics.com/glossary/model-deployment) involves making your model available for use in a production environment. Here are the steps to deploy a computer vision model:
- **Setting Up the Environment:** Configure the necessary infrastructure for your chosen deployment option, whether it's cloud-based (AWS, Google Cloud, Azure) or edge-based (local devices, IoT).
- **[Exporting the Model](../modes/export.md):** Export your model to the appropriate format (e.g., ONNX, TensorRT, CoreML for YOLO26) to ensure compatibility with your deployment platform.
- **Deploying the Model:** Deploy the model by setting up APIs or endpoints and integrating it with your application.
- **Ensuring Scalability:** Implement load balancers, auto-scaling groups, and monitoring tools to manage resources and handle increasing data and user requests.
For more detailed guidance on deployment strategies and best practices, check out our [model deployment practices guide](./model-deployment-practices.md).
## Step 8: Monitoring, Maintenance, and Documentation
Once your model is deployed, it's important to continuously monitor its performance, maintain it to handle any issues, and document the entire process for future reference and improvements.
Monitoring tools can help you track key performance indicators (KPIs) and detect anomalies or drops in accuracy. By monitoring the model, you can be aware of model drift, where the model's performance declines over time due to changes in the input data. Periodically retrain the model with updated data to maintain accuracy and relevance.
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/model-monitoring-maintenance-loop.avif" alt="Model monitoring and maintenance lifecycle">
</p>
In addition to monitoring and maintenance, documentation is also key. Thoroughly document the entire process, including model architecture, training procedures, hyperparameters, data preprocessing steps, and any changes made during deployment and maintenance. Good documentation ensures reproducibility and makes future updates or troubleshooting easier. By effectively [monitoring, maintaining, and documenting your model](./model-monitoring-and-maintenance.md), you can ensure it remains accurate, reliable, and easy to manage over its lifecycle.
## Engaging with the Community
Connecting with a community of computer vision enthusiasts can help you tackle any issues you face while working on your computer vision project with confidence. Here are some ways to learn, troubleshoot, and network effectively.
### Community Resources
- **GitHub Issues:** Check out the [YOLO26 GitHub repository](https://github.com/ultralytics/ultralytics/issues) and use the Issues tab to ask questions, report bugs, and suggest new features. The active community and maintainers are there to help with specific issues.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to interact with other users and developers, get support, and share insights.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Explore the [official YOLO26 documentation](./index.md) for detailed guides with helpful tips on different computer vision tasks and projects.
Using these resources will help you overcome challenges and stay updated with the latest trends and best practices in the computer vision community.
## Next Steps
Taking on a computer vision project can be exciting and rewarding. By following the steps in this guide, you can build a solid foundation for success. Each step is crucial for developing a solution that meets your objectives and works well in real-world scenarios. As you gain experience, you'll discover advanced techniques and tools to improve your projects.
## FAQ
### How do I choose the right computer vision task for my project?
Choosing the right computer vision task depends on your project's end goal. For instance, if you want to monitor traffic, **object detection** is suitable as it can locate and identify multiple vehicle types in real-time. For medical imaging, **image segmentation** is ideal for providing detailed boundaries of tumors, aiding in diagnosis and treatment planning. Learn more about specific tasks like [object detection](../tasks/detect.md), [image classification](../tasks/classify.md), and [instance segmentation](../tasks/segment.md).
### Why is data annotation crucial in computer vision projects?
Data annotation is vital for teaching your model to recognize patterns. The type of annotation varies with the task:
- **Image Classification**: Entire image labeled as a single class.
- **Object Detection**: Bounding boxes drawn around objects.
- **Image Segmentation**: Each pixel labeled according to the object it belongs to.
Tools like [Label Studio](https://github.com/HumanSignal/label-studio), [CVAT](https://github.com/cvat-ai/cvat), and [Labelme](https://github.com/wkentaro/labelme) can assist in this process. For more details, refer to our [data collection and annotation guide](./data-collection-and-annotation.md).
### What steps should I follow to augment and split my dataset effectively?
Splitting your dataset before augmentation helps validate model performance on original, unaltered data. Follow these steps:
- **Training Set**: 70-80% of your data.
- **Validation Set**: 10-15% for [hyperparameter tuning](https://www.ultralytics.com/glossary/hyperparameter-tuning).
- **Test Set**: Remaining 10-15% for final evaluation.
After splitting, apply data augmentation techniques like rotation, scaling, and flipping to increase dataset diversity. Libraries such as [Albumentations](../integrations/albumentations.md) and OpenCV can help. Ultralytics also offers [built-in augmentation settings](../modes/train.md) for convenience.
### How can I export my trained computer vision model for deployment?
Exporting your model ensures compatibility with different deployment platforms. Ultralytics provides multiple formats, including [ONNX](../integrations/onnx.md), [TensorRT](../integrations/tensorrt.md), and [CoreML](../integrations/coreml.md). To export your YOLO26 model, follow this guide:
- Use the `export` function with the desired format parameter.
- Ensure the exported model fits the specifications of your deployment environment (e.g., edge devices, cloud).
For more information, check out the [model export guide](../modes/export.md).
### What are the best practices for monitoring and maintaining a deployed computer vision model?
Continuous monitoring and maintenance are essential for a model's long-term success. Implement tools for tracking Key Performance Indicators (KPIs) and detecting anomalies. Regularly retrain the model with updated data to counteract model drift. Document the entire process, including model architecture, hyperparameters, and changes, to ensure reproducibility and ease of future updates. Learn more in our [monitoring and maintenance guide](./model-monitoring-and-maintenance.md).

178
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/streamlit-live-inference.md Normal file
View File

@@ -0,0 +1,178 @@
---
comments: true
description: Learn how to set up a real-time object detection application using Streamlit and Ultralytics YOLO26. Follow this step-by-step guide to implement webcam-based object detection.
keywords: Streamlit, YOLO26, Real-time Object Detection, Streamlit Application, YOLO26 Streamlit Tutorial, Webcam Object Detection
---
# Live Inference with Streamlit Application using Ultralytics YOLO26
## Introduction
Streamlit makes it simple to build and deploy interactive web applications. Combining this with Ultralytics YOLO26 allows for real-time [object detection](https://www.ultralytics.com/glossary/object-detection) and analysis directly in your browser. YOLO26's high accuracy and speed ensure seamless performance for live video streams, making it ideal for applications in security, retail, and beyond.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/1H9ktpHUUB8"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Use Streamlit with Ultralytics for Real-Time <a href="https://www.ultralytics.com/glossary/computer-vision-cv">Computer Vision</a> in Your Browser
</p>
| Aquaculture | Animal Husbandry |
| :-----------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: |
| ![Fish Detection using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/fish-detection-ultralytics-yolov8.avif) | ![Animals Detection using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/animals-detection-yolov8.avif) |
| Fish Detection using Ultralytics YOLO26 | Animals Detection using Ultralytics YOLO26 |
## Advantages of Live Inference
- **Seamless Real-Time Object Detection**: Streamlit combined with YOLO26 enables real-time object detection directly from your webcam feed. This allows for immediate analysis and insights, making it ideal for [applications requiring instant feedback](https://docs.ultralytics.com/modes/predict/).
- **User-Friendly Deployment**: Streamlit's interactive interface makes it easy to deploy and use the application without extensive technical knowledge. Users can start live inference with a simple click, enhancing accessibility and usability.
- **Efficient Resource Utilization**: YOLO26's optimized algorithms ensure high-speed processing with minimal computational resources. This efficiency allows for smooth and reliable webcam inference even on standard hardware, making advanced [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) accessible to a wider audience.
## Streamlit Application Code
!!! tip "Ultralytics Installation"
Before you start building the application, ensure you have the Ultralytics Python package installed.
```bash
pip install ultralytics
```
!!! example "Inference using Streamlit with Ultralytics YOLO"
=== "CLI"
```bash
yolo solutions inference
yolo solutions inference model="path/to/model.pt"
```
These commands launch the default Streamlit interface that ships with Ultralytics. Use `yolo solutions inference --help` to view additional flags such as `source`, `conf`, or `persist` if you want to customize the experience without editing Python code.
=== "Python"
```python
from ultralytics import solutions
inf = solutions.Inference(
model="yolo26n.pt", # you can use any model that Ultralytics supports, e.g., YOLO26, or a custom-trained model
)
inf.inference()
# Make sure to run the file using command `streamlit run path/to/file.py`
```
This will launch the Streamlit application in your default web browser. You will see the main title, subtitle, and the sidebar with configuration options. Select your desired YOLO26 model, set the confidence and [NMS thresholds](https://www.ultralytics.com/glossary/non-maximum-suppression-nms), and click the "Start" button to begin the real-time object detection.
## How It Works
Under the hood, the Streamlit application uses the [Ultralytics solutions module](https://docs.ultralytics.com/reference/solutions/streamlit_inference/) to create an interactive interface. When you start the inference, the application:
1. Captures video from your webcam or uploaded video file
2. Processes each frame through the YOLO26 model
3. Applies object detection with your specified confidence and IoU thresholds
4. Displays both the original and annotated frames in real-time
5. Optionally enables object tracking if selected
The application provides a clean, user-friendly interface with controls to adjust model parameters and start/stop inference at any time.
## Conclusion
By following this guide, you have successfully created a real-time object detection application using Streamlit and Ultralytics YOLO26. This application allows you to experience the power of YOLO26 in detecting objects through your webcam, with a user-friendly interface and the ability to stop the video stream at any time.
For further enhancements, you can explore adding more features such as recording the video stream, saving the annotated frames, or integrating with other [computer vision libraries](https://www.ultralytics.com/blog/exploring-vision-ai-frameworks-tensorflow-pytorch-and-opencv).
## Share Your Thoughts with the Community
Engage with the community to learn more, troubleshoot issues, and share your projects:
### Where to Find Help and Support
- **GitHub Issues:** Visit the [Ultralytics GitHub repository](https://github.com/ultralytics/ultralytics/issues) to raise questions, report bugs, and suggest features.
- **Ultralytics Discord Server:** Join the [Ultralytics Discord server](https://discord.com/invite/ultralytics) to connect with other users and developers, get support, share knowledge, and brainstorm ideas.
### Official Documentation
- **Ultralytics YOLO26 Documentation:** Refer to the [official YOLO26 documentation](https://docs.ultralytics.com/) for comprehensive guides and insights on various computer vision tasks and projects.
## FAQ
### How can I set up a real-time object detection application using Streamlit and Ultralytics YOLO26?
Setting up a real-time object detection application with Streamlit and Ultralytics YOLO26 is straightforward. First, ensure you have the Ultralytics Python package installed using:
```bash
pip install ultralytics
```
Then, you can create a basic Streamlit application to run live inference:
!!! example "Streamlit Application"
=== "Python"
```python
from ultralytics import solutions
inf = solutions.Inference(
model="yolo26n.pt", # you can use any model that Ultralytics supports, e.g., YOLO26, YOLOv10
)
inf.inference()
# Make sure to run the file using command `streamlit run path/to/file.py`
```
=== "CLI"
```bash
yolo solutions inference
```
For more details on the practical setup, refer to the [Streamlit Application Code section](#streamlit-application-code) of the documentation.
### What are the main advantages of using Ultralytics YOLO26 with Streamlit for real-time object detection?
Using Ultralytics YOLO26 with Streamlit for real-time object detection offers several advantages:
- **Seamless Real-Time Detection**: Achieve high-[accuracy](https://www.ultralytics.com/glossary/accuracy), real-time object detection directly from webcam feeds.
- **User-Friendly Interface**: Streamlit's intuitive interface allows easy use and deployment without extensive technical knowledge.
- **Resource Efficiency**: YOLO26's optimized algorithms ensure high-speed processing with minimal computational resources.
Learn more about these benefits in the [Advantages of Live Inference section](#advantages-of-live-inference).
### How do I deploy a Streamlit object detection application in my web browser?
After coding your Streamlit application integrating Ultralytics YOLO26, you can deploy it by running:
```bash
streamlit run path/to/file.py
```
This command will launch the application in your default web browser, enabling you to select YOLO26 models, set confidence and NMS thresholds, and start real-time object detection with a simple click. For a detailed guide, refer to the [Streamlit Application Code](#streamlit-application-code) section.
### What are some use cases for real-time object detection using Streamlit and Ultralytics YOLO26?
Real-time object detection using Streamlit and Ultralytics YOLO26 can be applied in various sectors:
- **Security**: Real-time monitoring for unauthorized access and [security alarm systems](https://docs.ultralytics.com/guides/security-alarm-system/).
- **Retail**: Customer counting, shelf management, and [inventory tracking](https://www.ultralytics.com/blog/from-shelves-to-sales-exploring-yolov8s-impact-on-inventory-management).
- **Wildlife and Agriculture**: Monitoring animals and crop conditions for [conservation efforts](https://www.ultralytics.com/blog/ai-in-wildlife-conservation).
For more in-depth use cases and examples, explore [Ultralytics Solutions](https://docs.ultralytics.com/solutions/).
### How does Ultralytics YOLO26 compare to other object detection models like YOLOv5 and RCNNs?
Ultralytics YOLO26 provides several enhancements over prior models like YOLOv5 and RCNNs:
- **Higher Speed and Accuracy**: Improved performance for real-time applications.
- **Ease of Use**: Simplified interfaces and deployment.
- **Resource Efficiency**: Optimized for better speed with minimal computational requirements.
For a comprehensive comparison, check [Ultralytics YOLO26 Documentation](https://docs.ultralytics.com/models/yolo26/) and related blog posts discussing model performance.

180
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/trackzone.md Normal file
View File

@@ -0,0 +1,180 @@
---
comments: true
description: Discover how TrackZone leverages Ultralytics YOLO26 to precisely track objects within specific zones, enabling real-time insights for crowd analysis, surveillance, and targeted monitoring.
keywords: TrackZone, object tracking, YOLO26, Ultralytics, real-time object detection, AI, deep learning, crowd analysis, surveillance, zone-based tracking, resource optimization
---
# TrackZone using Ultralytics YOLO26
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-track-the-objects-in-zone-using-ultralytics-yolo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open TrackZone In Colab"></a>
## What is TrackZone?
TrackZone specializes in monitoring objects within designated areas of a frame instead of the whole frame. Built on [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/), it integrates object detection and tracking specifically within zones for videos and live camera feeds. YOLO26's advanced algorithms and [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) technologies make it a perfect choice for real-time use cases, offering precise and efficient object tracking in applications like crowd monitoring and surveillance.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/SMSJvjUG1ko"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Track Objects in Region using Ultralytics YOLO26 | TrackZone 🚀
</p>
## Advantages of Object Tracking in Zones (TrackZone)
- **Targeted Analysis:** Tracking objects within specific zones allows for more focused insights, enabling precise monitoring and analysis of areas of interest, such as entry points or restricted zones.
- **Improved Efficiency:** By narrowing the tracking scope to defined zones, TrackZone reduces computational overhead, ensuring faster processing and optimal performance.
- **Enhanced Security:** Zonal tracking improves surveillance by monitoring critical areas, aiding in the early detection of unusual activity or security breaches.
- **Scalable Solutions:** The ability to focus on specific zones makes TrackZone adaptable to various scenarios, from retail spaces to industrial settings, ensuring seamless integration and scalability.
## Real World Applications
| Agriculture | Transportation |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ![Plants Tracking in Field Using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/plants-tracking-in-zone-using-ultralytics-yolo11.avif) | ![Vehicles Tracking on Road using Ultralytics YOLO26](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/vehicle-tracking-in-zone-using-ultralytics-yolo11.avif) |
| Plants Tracking in Field Using Ultralytics YOLO26 | Vehicles Tracking on Road using Ultralytics YOLO26 |
!!! example "TrackZone using Ultralytics YOLO"
=== "CLI"
```bash
# Run a trackzone example
yolo solutions trackzone show=True
# Pass a source video
yolo solutions trackzone source="path/to/video.mp4" show=True
# Pass region coordinates
yolo solutions trackzone show=True region="[(150, 150), (1130, 150), (1130, 570), (150, 570)]"
```
TrackZone relies on the `region` list to know which part of the frame to monitor. Define the polygon to match the physical zone you care about (doors, gates, etc.), and keep `show=True` enabled while configuring so you can verify the overlay aligns with the video feed.
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Define region points
region_points = [(150, 150), (1130, 150), (1130, 570), (150, 570)]
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("trackzone_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init trackzone (object tracking in zones, not complete frame)
trackzone = solutions.TrackZone(
show=True, # display the output
region=region_points, # pass region points
model="yolo26n.pt", # use any model that Ultralytics supports, e.g., YOLOv9, YOLOv10
# line_width=2, # adjust the line width for bounding boxes and text display
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = trackzone(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the video file
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### `TrackZone` Arguments
Here's a table with the `TrackZone` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "region"]) }}
The TrackZone solution includes support for `track` parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Moreover, the following visualization options are available:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## FAQ
### How do I track objects in a specific area or zone of a video frame using Ultralytics YOLO26?
Tracking objects in a defined area or zone of a video frame is straightforward with Ultralytics YOLO26. Simply use the command provided below to initiate tracking. This approach ensures efficient analysis and accurate results, making it ideal for applications like surveillance, crowd management, or any scenario requiring zonal tracking.
```bash
yolo solutions trackzone source="path/to/video.mp4" show=True
```
### How can I use TrackZone in Python with Ultralytics YOLO26?
With just a few lines of code, you can set up object tracking in specific zones, making it easy to integrate into your projects.
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
# Define region points
region_points = [(150, 150), (1130, 150), (1130, 570), (150, 570)]
# Video writer
video_writer = cv2.VideoWriter("object_counting_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init trackzone (object tracking in zones, not complete frame)
trackzone = solutions.TrackZone(
show=True, # display the output
region=region_points, # pass region points
model="yolo26n.pt",
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or video processing has been successfully completed.")
break
results = trackzone(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
```
### How do I configure the zone points for video processing using Ultralytics TrackZone?
Configuring zone points for video processing with Ultralytics TrackZone is simple and customizable. You can directly define and adjust the zones through a Python script, allowing precise control over the areas you want to monitor.
```python
# Define region points
region_points = [(150, 150), (1130, 150), (1130, 570), (150, 570)]
# Initialize trackzone
trackzone = solutions.TrackZone(
show=True, # display the output
region=region_points, # pass region points
)
```

357
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/triton-inference-server.md Normal file
View File

@@ -0,0 +1,357 @@
---
comments: true
description: Learn how to integrate Ultralytics YOLO26 with NVIDIA Triton Inference Server for scalable, high-performance AI model deployment.
keywords: Triton Inference Server, YOLO26, Ultralytics, NVIDIA, deep learning, AI model deployment, ONNX, scalable inference
---
# Triton Inference Server with Ultralytics YOLO26
The [Triton Inference Server](https://developer.nvidia.com/dynamo) (formerly known as TensorRT Inference Server) is an open-source software solution developed by NVIDIA. It provides a cloud inference solution optimized for NVIDIA GPUs. Triton simplifies the deployment of AI models at scale in production. Integrating Ultralytics YOLO26 with Triton Inference Server allows you to deploy scalable, high-performance [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) inference workloads. This guide provides steps to set up and test the integration.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/NQDtfSi5QF4"
title="Getting Started with NVIDIA Triton Inference Server" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Getting Started with NVIDIA Triton Inference Server.
</p>
## What is Triton Inference Server?
Triton Inference Server is designed to deploy a variety of AI models in production. It supports a wide range of deep learning and [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) frameworks, including TensorFlow, [PyTorch](https://www.ultralytics.com/glossary/pytorch), ONNX Runtime, and many others. Its primary use cases are:
- Serving multiple models from a single server instance
- Dynamic model loading and unloading without server restart
- Ensemble inference, allowing multiple models to be used together to achieve results
- Model versioning for A/B testing and rolling updates
## Key Benefits of Triton Inference Server
Using Triton Inference Server with Ultralytics YOLO26 provides several advantages:
- **Automatic batching**: Groups multiple AI requests together before processing them, reducing latency and improving inference speed
- **Kubernetes integration**: Cloud-native design works seamlessly with Kubernetes for managing and scaling AI applications
- **Hardware-specific optimizations**: Takes full advantage of NVIDIA GPUs for maximum performance
- **Framework flexibility**: Supports multiple AI frameworks including TensorFlow, PyTorch, ONNX, and TensorRT
- **Open-source and customizable**: Can be modified to fit specific needs, ensuring flexibility for various AI applications
## Prerequisites
Ensure you have the following prerequisites before proceeding:
- Docker installed on your machine
- Install `tritonclient`:
```bash
pip install tritonclient[all]
```
## Exporting YOLO26 to ONNX Format
Before deploying the model on Triton, it must be exported to the ONNX format. ONNX (Open Neural Network Exchange) is a format that allows models to be transferred between different deep learning frameworks. Use the `export` function from the `YOLO` class:
```python
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt") # load an official model
# Retrieve metadata during export. Metadata needs to be added to config.pbtxt. See next section.
metadata = []
def export_cb(exporter):
metadata.append(exporter.metadata)
model.add_callback("on_export_end", export_cb)
# Export the model
onnx_file = model.export(format="onnx", dynamic=True)
```
## Setting Up Triton Model Repository
The Triton Model Repository is a storage location where Triton can access and load models.
1. Create the necessary directory structure:
```python
from pathlib import Path
# Define paths
model_name = "yolo"
triton_repo_path = Path("tmp") / "triton_repo"
triton_model_path = triton_repo_path / model_name
# Create directories
(triton_model_path / "1").mkdir(parents=True, exist_ok=True)
```
2. Move the exported ONNX model to the Triton repository:
```python
from pathlib import Path
# Move ONNX model to Triton Model path
Path(onnx_file).rename(triton_model_path / "1" / "model.onnx")
# Create config file
(triton_model_path / "config.pbtxt").touch()
data = """
# Add metadata
parameters {
key: "metadata"
value {
string_value: "%s"
}
}
# (Optional) Enable TensorRT for GPU inference
# First run will be slow due to TensorRT engine conversion
optimization {
execution_accelerators {
gpu_execution_accelerator {
name: "tensorrt"
parameters {
key: "precision_mode"
value: "FP16"
}
parameters {
key: "max_workspace_size_bytes"
value: "3221225472"
}
parameters {
key: "trt_engine_cache_enable"
value: "1"
}
parameters {
key: "trt_engine_cache_path"
value: "/models/yolo/1"
}
}
}
}
""" % metadata[0] # noqa
with open(triton_model_path / "config.pbtxt", "w") as f:
f.write(data)
```
## Running Triton Inference Server
Run the Triton Inference Server using Docker:
```python
import contextlib
import subprocess
import time
from tritonclient.http import InferenceServerClient
# Define image https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver
tag = "nvcr.io/nvidia/tritonserver:24.09-py3" # 8.57 GB
# Pull the image
subprocess.call(f"docker pull {tag}", shell=True)
# Run the Triton server and capture the container ID
container_id = (
subprocess.check_output(
f"docker run -d --rm --runtime=nvidia --gpus 0 -v {triton_repo_path}:/models -p 8000:8000 {tag} tritonserver --model-repository=/models",
shell=True,
)
.decode("utf-8")
.strip()
)
# Wait for the Triton server to start
triton_client = InferenceServerClient(url="localhost:8000", verbose=False, ssl=False)
# Wait until model is ready
for _ in range(10):
with contextlib.suppress(Exception):
assert triton_client.is_model_ready(model_name)
break
time.sleep(1)
```
Then run inference using the Triton Server model:
```python
from ultralytics import YOLO
# Load the Triton Server model
model = YOLO("http://localhost:8000/yolo", task="detect")
# Run inference on the server
results = model("path/to/image.jpg")
```
Cleanup the container:
```python
# Kill and remove the container at the end of the test
subprocess.call(f"docker kill {container_id}", shell=True)
```
## TensorRT Optimization (Optional)
For even greater performance, you can use [TensorRT](https://docs.ultralytics.com/integrations/tensorrt/) with Triton Inference Server. TensorRT is a high-performance deep learning optimizer built specifically for NVIDIA GPUs that can significantly increase inference speed.
Key benefits of using TensorRT with Triton include:
- Up to 36x faster inference compared to unoptimized models
- Hardware-specific optimizations for maximum GPU utilization
- Support for reduced precision formats (INT8, FP16) while maintaining accuracy
- Layer fusion to reduce computational overhead
To use TensorRT directly, you can export your YOLO26 model to TensorRT format:
```python
from ultralytics import YOLO
# Load the YOLO26 model
model = YOLO("yolo26n.pt")
# Export the model to TensorRT format
model.export(format="engine") # creates 'yolo26n.engine'
```
For more information on TensorRT optimization, see the [TensorRT integration guide](https://docs.ultralytics.com/integrations/tensorrt/).
---
By following the above steps, you can deploy and run Ultralytics YOLO26 models efficiently on Triton Inference Server, providing a scalable and high-performance solution for deep learning inference tasks. If you face any issues or have further queries, refer to the [official Triton documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html) or reach out to the Ultralytics community for support.
## FAQ
### How do I set up Ultralytics YOLO26 with NVIDIA Triton Inference Server?
Setting up [Ultralytics YOLO26](../models/yolo26.md) with [NVIDIA Triton Inference Server](https://developer.nvidia.com/dynamo) involves a few key steps:
1. **Export YOLO26 to ONNX format**:
```python
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt") # load an official model
# Export the model to ONNX format
onnx_file = model.export(format="onnx", dynamic=True)
```
2. **Set up Triton Model Repository**:
```python
from pathlib import Path
# Define paths
model_name = "yolo"
triton_repo_path = Path("tmp") / "triton_repo"
triton_model_path = triton_repo_path / model_name
# Create directories
(triton_model_path / "1").mkdir(parents=True, exist_ok=True)
Path(onnx_file).rename(triton_model_path / "1" / "model.onnx")
(triton_model_path / "config.pbtxt").touch()
```
3. **Run the Triton Server**:
```python
import contextlib
import subprocess
import time
from tritonclient.http import InferenceServerClient
# Define image https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver
tag = "nvcr.io/nvidia/tritonserver:24.09-py3"
subprocess.call(f"docker pull {tag}", shell=True)
container_id = (
subprocess.check_output(
f"docker run -d --rm --runtime=nvidia --gpus 0 -v {triton_repo_path}:/models -p 8000:8000 {tag} tritonserver --model-repository=/models",
shell=True,
)
.decode("utf-8")
.strip()
)
triton_client = InferenceServerClient(url="localhost:8000", verbose=False, ssl=False)
for _ in range(10):
with contextlib.suppress(Exception):
assert triton_client.is_model_ready(model_name)
break
time.sleep(1)
```
This setup can help you efficiently deploy YOLO26 models at scale on Triton Inference Server for high-performance AI model inference.
### What benefits does using Ultralytics YOLO26 with NVIDIA Triton Inference Server offer?
Integrating Ultralytics YOLO26 with [NVIDIA Triton Inference Server](https://developer.nvidia.com/dynamo) provides several advantages:
- **Scalable AI Inference**: Triton allows serving multiple models from a single server instance, supporting dynamic model loading and unloading, making it highly scalable for diverse AI workloads.
- **High Performance**: Optimized for NVIDIA GPUs, Triton Inference Server ensures high-speed inference operations, perfect for real-time applications such as [object detection](https://www.ultralytics.com/glossary/object-detection).
- **Ensemble and Model Versioning**: Triton's ensemble mode enables combining multiple models to improve results, and its model versioning supports A/B testing and rolling updates.
- **Automatic Batching**: Triton automatically groups multiple inference requests together, significantly improving throughput and reducing latency.
- **Simplified Deployment**: Gradual optimization of AI workflows without requiring complete system overhauls, making it easier to scale efficiently.
For detailed instructions on setting up and running YOLO26 with Triton, you can refer to the [setup guide](#setting-up-triton-model-repository).
### Why should I export my YOLO26 model to ONNX format before using Triton Inference Server?
Using ONNX (Open Neural Network Exchange) format for your Ultralytics YOLO26 model before deploying it on [NVIDIA Triton Inference Server](https://developer.nvidia.com/dynamo) offers several key benefits:
- **Interoperability**: ONNX format supports transfer between different deep learning frameworks (such as PyTorch, TensorFlow), ensuring broader compatibility.
- **Optimization**: Many deployment environments, including Triton, optimize for ONNX, enabling faster inference and better performance.
- **Ease of Deployment**: ONNX is widely supported across frameworks and platforms, simplifying the deployment process in various operating systems and hardware configurations.
- **Framework Independence**: Once converted to ONNX, your model is no longer tied to its original framework, making it more portable.
- **Standardization**: ONNX provides a standardized representation that helps overcome compatibility issues between different AI frameworks.
To export your model, use:
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
onnx_file = model.export(format="onnx", dynamic=True)
```
You can follow the steps in the [ONNX integration guide](https://docs.ultralytics.com/integrations/onnx/) to complete the process.
### Can I run inference using the Ultralytics YOLO26 model on Triton Inference Server?
Yes, you can run inference using the Ultralytics YOLO26 model on [NVIDIA Triton Inference Server](https://developer.nvidia.com/dynamo). Once your model is set up in the Triton Model Repository and the server is running, you can load and run inference on your model as follows:
```python
from ultralytics import YOLO
# Load the Triton Server model
model = YOLO("http://localhost:8000/yolo", task="detect")
# Run inference on the server
results = model("path/to/image.jpg")
```
This approach allows you to leverage Triton's optimizations while using the familiar Ultralytics YOLO interface. For an in-depth guide on setting up and running Triton Server with YOLO26, refer to the [running triton inference server](#running-triton-inference-server) section.
### How does Ultralytics YOLO26 compare to TensorFlow and PyTorch models for deployment?
[Ultralytics YOLO26](../models/yolo26.md) offers several unique advantages compared to [TensorFlow](https://www.ultralytics.com/glossary/tensorflow) and PyTorch models for deployment:
- **Real-time Performance**: Optimized for real-time object detection tasks, YOLO26 provides state-of-the-art [accuracy](https://www.ultralytics.com/glossary/accuracy) and speed, making it ideal for applications requiring live video analytics.
- **Ease of Use**: YOLO26 integrates seamlessly with Triton Inference Server and supports diverse export formats (ONNX, TensorRT, CoreML), making it flexible for various deployment scenarios.
- **Advanced Features**: YOLO26 includes features like dynamic model loading, model versioning, and ensemble inference, which are crucial for scalable and reliable AI deployments.
- **Simplified API**: The Ultralytics API provides a consistent interface across different deployment targets, reducing the learning curve and development time.
- **Edge Optimization**: YOLO26 models are designed with edge deployment in mind, offering excellent performance even on resource-constrained devices.
For more details, compare the deployment options in the [model export guide](../modes/export.md).

598
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/vertex-ai-deployment-with-docker.md Normal file
View File

@@ -0,0 +1,598 @@
---
comments: true
description: Learn how to deploy pretrained YOLO26 models on Google Cloud Vertex AI using Docker containers and FastAPI for scalable inference with complete control over preprocessing and postprocessing.
keywords: YOLO26, Vertex AI, Docker, FastAPI, deployment, container, GCP, Artifact Registry, Ultralytics, cloud deployment
---
# Deploy a pretrained YOLO model with Ultralytics on Vertex AI for inference
This guide will show you how to containerize a pretrained YOLO26 model with Ultralytics, build a FastAPI inference server for it, and deploy the model with inference server on Google Cloud Vertex AI. The example implementation will cover the object detection use case for YOLO26, but the same principles will apply for using [other YOLO modes](../modes/index.md).
Before we start, you will need to create a Google Cloud Platform (GCP) project. You get $300 in GCP credits to use for free as a new user, and this amount is enough to test a running setup that you can later extend for any other YOLO26 use case, including training, or batch and streaming inference.
## What you will learn
1. Create an inference backend for Ultralytics YOLO26 model using FastAPI.
2. Create a GCP Artifact Registry repository to store your Docker image.
3. Build and push the Docker image with the model to Artifact Registry.
4. Import your model in Vertex AI.
5. Create a Vertex AI endpoint and deploy the model.
!!! tip "Why deploy a containerized model?"
- **Full model control with Ultralytics**: You can use custom inference logic with complete control over preprocessing, postprocessing, and response formatting.
- **Vertex AI handles the rest**: It auto-scales, yet gives flexibility in configuring compute resources, memory, and GPU configurations.
- **Native GCP integrations and security**: Seamless setup with Cloud Storage, BigQuery, Cloud Functions, VPC controls, IAM policies, and audit logs.
## Prerequisites
1. Install [Docker](https://docs.docker.com/engine/install/) on your machine.
2. Install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) and [authenticate for using the gcloud CLI](https://cloud.google.com/docs/authentication/gcloud).
3. It is highly recommended that you go through the [Docker Quickstart Guide for Ultralytics](https://docs.ultralytics.com/guides/docker-quickstart/), because you will need to extend one of the official Ultralytics Docker images while following this guide.
## 1. Create an inference backend with FastAPI
First, you need to create a FastAPI application that will serve the YOLO26 model inference requests. This application will handle the model loading, image preprocessing, and inference (prediction) logic.
### Vertex AI Compliance Fundamentals
Vertex AI expects your container to implement two specific endpoints:
1. **Health** endpoint (`/health`): Must return HTTP status `200 OK` when service is ready.
2. **Predict** endpoint (`/predict`): Accepts structured prediction requests with **base64-encoded** images and optional parameters. [Payload size limits](https://docs.cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type) apply depending on the endpoint type.
Request payloads for the `/predict` endpoint should follow this JSON structure:
```json
{
"instances": [{ "image": "base64_encoded_image" }],
"parameters": { "confidence": 0.5 }
}
```
### Project folder structure
The bulk of our build will be happening inside the Docker container, and Ultralytics will also load a pretrained YOLO26 model, so you can keep the local folder structure simple:
```txt
YOUR_PROJECT/
├── src/
│ ├── __init__.py
│ ├── app.py # Core YOLO26 inference logic
│ └── main.py # FastAPI inference server
├── tests/
├── .env # Environment variables for local development
├── Dockerfile # Container configuration
├── LICENSE # AGPL-3.0 License
└── pyproject.toml # Python dependencies and project config
```
!!! note "Important license note"
Ultralytics YOLO26 models and framework are licensed under AGPL-3.0, which has important compliance requirements. Make sure to read the Ultralytics docs on [how to comply with the license terms](../help/contributing.md#how-to-comply-with-agpl-30).
### Create pyproject.toml with dependencies
To conveniently manage your project, create a `pyproject.toml` file with the following dependencies:
```toml
[project]
name = "YOUR_PROJECT_NAME"
version = "0.0.1"
description = "YOUR_PROJECT_DESCRIPTION"
requires-python = ">=3.10,<3.13"
dependencies = [
"ultralytics>=8.3.0",
"fastapi[all]>=0.89.1",
"uvicorn[standard]>=0.20.0",
"pillow>=9.0.0",
]
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
```
- `uvicorn` will be used to run the FastAPI server.
- `pillow` will be used for image processing, but you are not limited to PIL images only — Ultralytics supports [many other formats](../modes/predict.md#inference-sources).
### Create inference logic with Ultralytics YOLO26
Now that you have the project structure and dependencies set up, you can implement the core YOLO26 inference logic. Create a `src/app.py` file that will handle model loading, image processing, and prediction, using Ultralytics Python API.
```python
# src/app.py
from ultralytics import YOLO
# Model initialization and readiness state
model_yolo = None
_model_ready = False
def _initialize_model():
"""Initialize the YOLO model."""
global model_yolo, _model_ready
try:
# Use pretrained YOLO26n model from Ultralytics base image
model_yolo = YOLO("yolo26n.pt")
_model_ready = True
except Exception as e:
print(f"Error initializing YOLO model: {e}")
_model_ready = False
model_yolo = None
# Initialize model on module import
_initialize_model()
def is_model_ready() -> bool:
"""Check if the model is ready for inference."""
return _model_ready and model_yolo is not None
```
This will load the model once when the container starts, and the model will be shared across all requests. If your model will be handling heavy inference load, it is recommended to select a machine type with more memory when importing a model in Vertex AI at a later step.
Next, create two utility functions for input and output image processing with `pillow`. YOLO26 supports PIL images natively.
```python
def get_image_from_bytes(binary_image: bytes) -> Image.Image:
"""Convert image from bytes to PIL RGB format."""
input_image = Image.open(io.BytesIO(binary_image)).convert("RGB")
return input_image
```
```python
def get_bytes_from_image(image: Image.Image) -> bytes:
"""Convert PIL image to bytes."""
return_image = io.BytesIO()
image.save(return_image, format="JPEG", quality=85)
return_image.seek(0)
return return_image.getvalue()
```
Finally, implement the `run_inference` function that will handle the object detection. In this example, we will extract bounding boxes, class names, and confidence scores from the model predictions. The function will return a dictionary with detections and raw results for further processing or annotation.
```python
def run_inference(input_image: Image.Image, confidence_threshold: float = 0.5) -> Dict[str, Any]:
"""Run inference on an image using YOLO26n model."""
global model_yolo
# Check if model is ready
if not is_model_ready():
print("Model not ready for inference")
return {"detections": [], "results": None}
try:
# Make predictions and get raw results
results = model_yolo.predict(
imgsz=640, source=input_image, conf=confidence_threshold, save=False, augment=False, verbose=False
)
# Extract detections (bounding boxes, class names, and confidences)
detections = []
if results and len(results) > 0:
result = results[0]
if result.boxes is not None and len(result.boxes.xyxy) > 0:
boxes = result.boxes
# Convert tensors to numpy for processing
xyxy = boxes.xyxy.cpu().numpy()
conf = boxes.conf.cpu().numpy()
cls = boxes.cls.cpu().numpy().astype(int)
# Create detection dictionaries
for i in range(len(xyxy)):
detection = {
"xmin": float(xyxy[i][0]),
"ymin": float(xyxy[i][1]),
"xmax": float(xyxy[i][2]),
"ymax": float(xyxy[i][3]),
"confidence": float(conf[i]),
"class": int(cls[i]),
"name": model_yolo.names.get(int(cls[i]), f"class_{int(cls[i])}"),
}
detections.append(detection)
return {
"detections": detections,
"results": results, # Keep raw results for annotation
}
except Exception as e:
# If there's an error, return empty structure
print(f"Error in YOLO detection: {e}")
return {"detections": [], "results": None}
```
Optionally, you can add a function to annotate the image with bounding boxes and labels using the Ultralytics built-in plotting method. This will be useful if you want to return annotated images in the prediction response.
```python
def get_annotated_image(results: list) -> Image.Image:
"""Get annotated image using Ultralytics built-in plot method."""
if not results or len(results) == 0:
raise ValueError("No results provided for annotation")
result = results[0]
# Use Ultralytics built-in plot method with PIL output
return result.plot(pil=True)
```
### Create HTTP inference server with FastAPI
Now that you have the core YOLO26 inference logic, you can create a FastAPI application to serve it. This will include the health check and prediction endpoints required by Vertex AI.
First, add the imports and configure logging for Vertex AI. Because Vertex AI treats stderr as error output, it makes sense to pipe the logs to stdout.
```python
import sys
from loguru import logger
# Configure logger
logger.remove()
logger.add(
sys.stdout,
colorize=True,
format="<green>{time:HH:mm:ss}</green> | <level>{message}</level>",
level=10,
)
logger.add("log.log", rotation="1 MB", level="DEBUG", compression="zip")
```
For a complete Vertex AI compliance, define the required endpoints in environment variables and set the size limit for requests. It is recommended to use [private Vertex AI endpoints](https://docs.cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type) for production deployments. This way you will have a higher request payload limit (10 MB instead of 1.5 MB for public endpoints), together with robust security and access control.
```python
# Vertex AI environment variables
AIP_HTTP_PORT = int(os.getenv("AIP_HTTP_PORT", "8080"))
AIP_HEALTH_ROUTE = os.getenv("AIP_HEALTH_ROUTE", "/health")
AIP_PREDICT_ROUTE = os.getenv("AIP_PREDICT_ROUTE", "/predict")
# Request size limit (10 MB for private endpoints, 1.5 MB for public)
MAX_REQUEST_SIZE = 10 * 1024 * 1024 # 10 MB in bytes
```
Add two Pydantic models for validating your requests and responses:
```python
# Pydantic models for request/response
class PredictionRequest(BaseModel):
instances: list
parameters: Optional[Dict[str, Any]] = None
class PredictionResponse(BaseModel):
predictions: list
```
Add the health check endpoint to verify your model readiness. **This is important for Vertex AI**, as without a dedicated health check its orchestrator will be pinging random sockets and will not be able to determine if the model is ready for inference. Your check must return `200 OK` for success and `503 Service Unavailable` for failure:
```python
# Health check endpoint
@app.get(AIP_HEALTH_ROUTE, status_code=status.HTTP_200_OK)
def health_check():
"""Health check endpoint for Vertex AI."""
if not is_model_ready():
raise HTTPException(status_code=503, detail="Model not ready")
return {"status": "healthy"}
```
You now have everything to implement the prediction endpoint that will handle the inference requests. It will accept an image file, run the inference, and return the results. Note that the image must be base64-encoded, which additionally increases the size of the payload by up to 33%.
```python
@app.post(AIP_PREDICT_ROUTE, response_model=PredictionResponse)
async def predict(request: PredictionRequest):
"""Prediction endpoint for Vertex AI."""
try:
predictions = []
for instance in request.instances:
if isinstance(instance, dict):
if "image" in instance:
image_data = base64.b64decode(instance["image"])
input_image = get_image_from_bytes(image_data)
else:
raise HTTPException(status_code=400, detail="Instance must contain 'image' field")
else:
raise HTTPException(status_code=400, detail="Invalid instance format")
# Extract YOLO26 parameters if provided
parameters = request.parameters or {}
confidence_threshold = parameters.get("confidence", 0.5)
return_annotated_image = parameters.get("return_annotated_image", False)
# Run inference with YOLO26n model
result = run_inference(input_image, confidence_threshold=confidence_threshold)
detections_list = result["detections"]
# Format predictions for Vertex AI
detections = []
for detection in detections_list:
formatted_detection = {
"class": detection["name"],
"confidence": detection["confidence"],
"bbox": {
"xmin": detection["xmin"],
"ymin": detection["ymin"],
"xmax": detection["xmax"],
"ymax": detection["ymax"],
},
}
detections.append(formatted_detection)
# Build prediction response
prediction = {"detections": detections, "detection_count": len(detections)}
# Add annotated image if requested and detections exist
if (
return_annotated_image
and result["results"]
and result["results"][0].boxes is not None
and len(result["results"][0].boxes) > 0
):
import base64
annotated_image = get_annotated_image(result["results"])
img_bytes = get_bytes_from_image(annotated_image)
prediction["annotated_image"] = base64.b64encode(img_bytes).decode("utf-8")
predictions.append(prediction)
logger.info(
f"Processed {len(request.instances)} instances, found {sum(len(p['detections']) for p in predictions)} total detections"
)
return PredictionResponse(predictions=predictions)
except HTTPException:
# Re-raise HTTPException as-is (don't catch and convert to 500)
raise
except Exception as e:
logger.error(f"Prediction error: {e}")
raise HTTPException(status_code=500, detail=f"Prediction failed: {e}")
```
Finally, add the application entry point to run the FastAPI server.
```python
if __name__ == "__main__":
import uvicorn
logger.info(f"Starting server on port {AIP_HTTP_PORT}")
logger.info(f"Health check route: {AIP_HEALTH_ROUTE}")
logger.info(f"Predict route: {AIP_PREDICT_ROUTE}")
uvicorn.run(app, host="0.0.0.0", port=AIP_HTTP_PORT)
```
You now have a complete FastAPI application that can serve YOLO26 inference requests. You can test it locally by installing the dependencies and running the server, for example, with uv.
```bash
# Install dependencies
uv pip install -e .
# Run the FastAPI server directly
uv run src/main.py
```
To test the server, you can query both the `/health` and `/predict` endpoints using cURL. Put a test image in the `tests` folder. Then, in your Terminal, run the following commands:
```bash
# Test health endpoint
curl http://localhost:8080/health
# Test predict endpoint with base64 encoded image
curl -X POST -H "Content-Type: application/json" -d "{\"instances\": [{\"image\": \"$(base64 -i tests/test_image.jpg)\"}]}" http://localhost:8080/predict
```
You should receive a JSON response with the detected objects. On your first request, expect a short delay, as Ultralytics needs to pull and load the YOLO26 model.
## 2. Extend the Ultralytics Docker image with your application
Ultralytics provides several Docker images that you can use as a base for your application image. Docker will install Ultralytics and the necessary GPU drivers.
To use the full capabilities of Ultralytics YOLO models, you should select the CUDA-optimized image for GPU inference. However, if CPU inference is enough for your task, you can save computing resources by selecting the CPU-only image as well:
- [Dockerfile](https://github.com/ultralytics/ultralytics/blob/main/docker/Dockerfile): CUDA-optimized image for YOLO26 single/multi-GPU training and inference.
- [Dockerfile-cpu](https://github.com/ultralytics/ultralytics/blob/main/docker/Dockerfile-cpu): CPU-only image for YOLO26 inference.
### Create a Docker image for your application
Create a `Dockerfile` in the root of your project with the following content:
```dockerfile
# Extends official Ultralytics Docker image for YOLO26
FROM ultralytics/ultralytics:latest
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
# Install FastAPI and dependencies
RUN uv pip install fastapi[all] uvicorn[standard] loguru
WORKDIR /app
COPY src/ ./src/
COPY pyproject.toml ./
# Install the application package
RUN uv pip install -e .
RUN mkdir -p /app/logs
ENV PYTHONPATH=/app/src
# Port for Vertex AI
EXPOSE 8080
# Start the inference server
ENTRYPOINT ["python", "src/main.py"]
```
In the example, the official Ultralytics Docker image `ultralytics:latest` is used as a base. It already contains the YOLO26 model and all necessary dependencies. The server's entrypoint is the same as we used to test the FastAPI application locally.
### Build and test the Docker image
Now you can build the Docker image with the following command:
```bash
docker build --platform linux/amd64 -t IMAGE_NAME:IMAGE_VERSION .
```
Replace `IMAGE_NAME` and `IMAGE_VERSION` with your desired values, for example, `yolo26-fastapi:0.1`. Note that you must build the image for the `linux/amd64` architecture if you are deploying on Vertex AI. The `--platform` parameter needs to be explicitly set if you are building the image on an Apple Silicon Mac or any other non-x86 architecture.
Once the image build is completed, you can test the Docker image locally:
```bash
docker run --platform linux/amd64 -p 8080:8080 IMAGE_NAME:IMAGE_VERSION
```
Your Docker container is now running a FastAPI server on port `8080`, ready to accept inference requests. You can test both the `/health` and the `/predict` endpoint with the same cURL commands as before:
```bash
# Test health endpoint
curl http://localhost:8080/health
# Test predict endpoint with base64 encoded image
curl -X POST -H "Content-Type: application/json" -d "{\"instances\": [{\"image\": \"$(base64 -i tests/test_image.jpg)\"}]}" http://localhost:8080/predict
```
## 3. Upload the Docker image to GCP Artifact Registry
To import your containerized model in Vertex AI, you need to upload the Docker image to Google Cloud Artifact Registry. If you don't have an Artifact Registry repository yet, you will need to create one first.
### Create a repository in Google Cloud Artifact Registry
Open the [Artifact Registry page](https://console.cloud.google.com/artifacts) in the Google Cloud Console. If you are using the Artifact Registry for the first time, you may be prompted to enable the Artifact Registry API first.
<p align="center">
<img width="70%" src="https://github.com/lussebullar/temp-image-storage/releases/download/docs/create-artifact-registry-repo.png" alt="Google Cloud Artifact Registry repository creation">
</p>
1. Select Create Repository.
2. Enter the name of your repository. Select the desired region and use default settings for other options, unless you need to change them specifically.
!!! note
Region selection may affect the availability of machines and certain compute limitations for non-Enterprise users. You can find more information in the Vertex AI official documentation: [Vertex AI quotas and limits](https://docs.cloud.google.com/vertex-ai/docs/quotas)
1. Once the repository is created, save your PROJECT_ID, Location (Region), and Repository Name to your secrets vault or `.env` file. You will need them later to tag and push your Docker image to the Artifact Registry.
### Authenticate Docker to Artifact Registry
Authenticate your Docker client to the Artifact Registry repository you just created. Run the following command in your terminal:
```sh
gcloud auth configure-docker YOUR_REGION-docker.pkg.dev
```
### Tag and push your image to Artifact Registry
Tag and push the Docker image to Google Artifact Registry.
!!! note "Use unique tags for your images"
It is recommended to use unique tags every time you will be updating your image. Most GCP services, including Vertex AI, rely on the image tags for automated versioning and scaling, so it is a good practice to use semantic versioning or date-based tags.
Tag your image with the Artifact Registry repository URL. Replace the placeholders with the values you saved earlier.
```sh
docker tag IMAGE_NAME:IMAGE_VERSION YOUR_REGION-docker.pkg.dev/YOUR_PROJECT_ID/YOUR_REPOSITORY_NAME/IMAGE_NAME:IMAGE_VERSION
```
Push the tagged image to the Artifact Registry repository.
```sh
docker push YOUR_REGION-docker.pkg.dev/YOUR_PROJECT_ID/YOUR_REPOSITORY_NAME/IMAGE_NAME:IMAGE_VERSION
```
Wait for the process to complete. You should now see the image in your Artifact Registry repository.
For more specific instructions on how to work with images in Artifact Registry, see the Artifact Registry documentation: [Push and pull images](https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling).
## 4. Import your model in Vertex AI
Using the Docker image you've just pushed, you can now import the model in Vertex AI.
1. In Google Cloud navigation menu, go to Vertex AI > Model Registry. Alternatively, search for "Vertex AI" in the search bar at the top of the Google Cloud Console.
<p align="center">
<img width="80%" src="https://github.com/lussebullar/temp-image-storage/releases/download/docs/vertex-ai-import.png" alt="Vertex AI Model Registry import interface">
</p>
1. Click Import.
1. Select Import as a new model.
1. Select the region. You can choose the same region as your Artifact Registry repository, but your selection should be dictated by the availability of machine types and quotas in your region.
1. Select Import an existing model container.
<p align="center">
<img width="80%" src="https://github.com/lussebullar/temp-image-storage/releases/download/docs/import-model.png" alt="Vertex AI import model dialog">
</p>
1. In the Container image field, browse the Artifact Registry repository you created earlier and select the image you just pushed.
1. Scroll down to the Environment variables section and enter the predict and health endpoints, and the port that you defined in your FastAPI application.
<p align="center">
<img width="60%" src="https://github.com/lussebullar/temp-image-storage/releases/download/docs/predict-health-port.png" alt="Vertex AI environment variables configuration">
</p>
1. Click Import. Vertex AI will take several minutes to register the model and prepare it for deployment. You will receive an email notification once the import is complete.
## 5. Create a Vertex AI Endpoint and deploy your model
!!! note "Endpoints vs Models in Vertex AI"
In Vertex AI terminology, **endpoints** refer to the **deployed** models, since they represent the HTTP endpoints where you send inference requests, whereas **models** are the trained ML artifacts stored in the Model Registry.
To deploy a model, you need to create an Endpoint in Vertex AI.
1. In your Vertex AI navigation menu, go to Endpoints. Select your region you used when importing your model. Click Create.
<p align="center">
<img width="60%" src="https://github.com/lussebullar/temp-image-storage/releases/download/docs/endpoint-name.png" alt="Vertex AI create endpoint interface">
</p>
1. Enter the Endpoint name.
1. For Access, Vertex AI recommends using private Vertex AI endpoints. Apart from security benefits, you get a higher payload limit if you select a private endpoint, however you will need to configure your VPC network and firewall rules to allow access to the endpoint. Refer to the Vertex AI documentation for more instructions on [private endpoints](https://docs.cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type).
1. Click Continue.
1. On the Model settings dialog, select the model you imported earlier. Now you can configure the machine type, memory, and GPU settings for your model. Allow for ample memory if you are expecting high inference loads to ensure there are no I/O bottlenecks for the proper YOLO26 performance.
1. In Accelerator type, select the GPU type you want to use for inference. If you are not sure which GPU to select, you can start with NVIDIA T4, which is CUDA-supported.
!!! note "Region and machine type quotas"
Remember that certain regions have very limited compute quotas, so you may not be able to select certain machine types or GPUs in your region. If this is critical, change the region of your deployment to one with a bigger quota. Find more information in the Vertex AI official documentation: [Vertex AI quotas and limits](https://docs.cloud.google.com/vertex-ai/docs/quotas).
1. Once the machine type is selected, you can click Continue. At this point, you can choose to enable model monitoring in Vertex AI—an extra service that will track your model's performance and provide insights into its behavior. This is optional and incurs additional costs, so select according to your needs. Click Create.
Vertex AI will take several minutes (up to 30 min in some regions) to deploy the model. You will receive an email notification once the deployment is complete.
## 6. Test your deployed model
Once the deployment is complete, Vertex AI will provide you with a sample API interface to test your model.
To test remote inference, you can use the provided cURL command or create another Python client library that will send requests to the deployed model. Remember that you need to encode your image to base64 before sending it to the `/predict` endpoint.
<p align="center">
<img width="50%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/vertex-ai-endpoint-test-curl-yolo11.avif" alt="Vertex AI endpoint testing with cURL">
</p>
!!! note "Expect a short delay on the first request"
Similarly to the local testing, expect a short delay on the first request, as Ultralytics will need to pull and load the YOLO26 model in the running container.
You have successfully deployed a pretrained YOLO26 model with Ultralytics on Google Cloud Vertex AI.
## FAQ
### Can I use Ultralytics YOLO models on Vertex AI without Docker?
Yes; however, you will first need to export the model to a format compatible with Vertex AI, such as TensorFlow, Scikit-learn, or XGBoost. Google Cloud provides a guide on running `.pt` models on Vertex with a complete overview of the conversion process: [Run PyTorch models on Vertex AI](https://cloud.google.com/blog/topics/developers-practitioners/pytorch-google-cloud-how-deploy-pytorch-models-vertex-ai).
Please note that the resulting setup will rely only on the Vertex AI standard serving layer and will not support the advanced Ultralytics framework features. Since Vertex AI fully supports containerized models and can scale them automatically according to your deployment configuration, it allows you to leverage the full capabilities of Ultralytics YOLO models without needing to convert them to a different format.
### Why is FastAPI a good choice for serving YOLO26 inference?
FastAPI provides high throughput for inference workloads. Async support allows handling multiple concurrent requests without blocking the main thread, which is important when serving computer vision models.
Automatic request/response validation with FastAPI reduces runtime errors in production inference services. This is particularly valuable for object detection APIs where input format consistency is critical.
FastAPI adds minimal computational overhead to your inference pipeline, leaving more resources available for model execution and image processing tasks.
FastAPI also supports SSE (Server-Sent Events), which is useful for streaming inference scenarios.
### Why do I have to select a region so many times?
This is actually a versatility feature of Google Cloud Platform, where you need to select a region for every service you use. For the task of deploying a containerized model on Vertex AI, your most important region selection is the one for the Model Registry. It will determine the availability of machine types and quotas for your model deployment.
Additionally, if you will be extending the setup and storing prediction data or results in Cloud Storage or BigQuery, you will need to use the same region as for Model Registry to minimize latency and ensure high throughput for data access.

243
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/view-results-in-terminal.md Normal file
View File

@@ -0,0 +1,243 @@
---
comments: true
description: Learn how to visualize YOLO inference results directly in a VSCode terminal using sixel on Linux and MacOS.
keywords: YOLO, inference results, VSCode terminal, sixel, display images, Linux, MacOS
---
# Viewing Inference Results in a Terminal
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/sixel-example-terminal.avif" alt="Sixel example of image in Terminal">
</p>
Image from the [libsixel](https://saitoha.github.io/libsixel/) website.
## Motivation
When connecting to a remote machine, normally visualizing image results is not possible or requires moving data to a local device with a GUI. The VSCode integrated terminal allows for directly rendering images. This is a short demonstration on how to use this in conjunction with `ultralytics` with [prediction results](../modes/predict.md).
!!! warning
Only compatible with Linux and MacOS. Check the [VSCode repository](https://github.com/microsoft/vscode), check [Issue status](https://github.com/microsoft/vscode/issues/198622), or [documentation](https://code.visualstudio.com/docs) for updates about Windows support to view images in terminal with `sixel`.
The VSCode compatible protocols for viewing images using the integrated terminal are [`sixel`](https://en.wikipedia.org/wiki/Sixel) and [`iTerm`](https://iterm2.com/documentation-images.html). This guide will demonstrate use of the `sixel` protocol.
## Process
1. First, you must enable settings `terminal.integrated.enableImages` and `terminal.integrated.gpuAcceleration` in VSCode.
```yaml
"terminal.integrated.gpuAcceleration": "auto" # "auto" is default, can also use "on"
"terminal.integrated.enableImages": true
```
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/vscode-enable-terminal-images-setting.avif" alt="VSCode enable terminal images setting">
</p>
2. Install the `python-sixel` library in your virtual environment. This is a [fork](https://github.com/lubosz/python-sixel?tab=readme-ov-file) of the `PySixel` library, which is no longer maintained.
```bash
pip install sixel
```
3. Load a model and execute inference, then plot the results and store in a variable. See more about inference arguments and working with results on the [predict mode](../modes/predict.md) page.
```{ .py .annotate }
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Run inference on an image
results = model.predict(source="ultralytics/assets/bus.jpg")
# Plot inference results
plot = results[0].plot() # (1)!
```
1. See [plot method parameters](../modes/predict.md#plot-method-parameters) to see possible arguments to use.
4. Now, use [OpenCV](https://www.ultralytics.com/glossary/opencv) to convert the `np.ndarray` to `bytes` data. Then use `io.BytesIO` to make a "file-like" object.
```{ .py .annotate }
import io
import cv2
# Results image as bytes
im_bytes = cv2.imencode(
".png", # (1)!
plot,
)[1].tobytes() # (2)!
# Image bytes as a file-like object
mem_file = io.BytesIO(im_bytes)
```
1. It's possible to use other image extensions as well.
2. Only the object at index `1` that is returned is needed.
5. Create a `SixelWriter` instance, and then use the `.draw()` method to draw the image in the terminal.
```python
from sixel import SixelWriter
# Create sixel writer object
w = SixelWriter()
# Draw the sixel image in the terminal
w.draw(mem_file)
```
## Example Inference Results
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/view-image-in-terminal.avif" alt="YOLO inference results displayed in terminal">
</p>
!!! danger
Using this example with videos or animated GIF frames has **not** been tested. Attempt at your own risk.
## Full Code Example
```{ .py .annotate }
import io
import cv2
from sixel import SixelWriter
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Run inference on an image
results = model.predict(source="ultralytics/assets/bus.jpg")
# Plot inference results
plot = results[0].plot() # (3)!
# Results image as bytes
im_bytes = cv2.imencode(
".png", # (1)!
plot,
)[1].tobytes() # (2)!
mem_file = io.BytesIO(im_bytes)
w = SixelWriter()
w.draw(mem_file)
```
1. It's possible to use other image extensions as well.
2. Only the object at index `1` that is returned is needed.
3. See [plot method parameters](../modes/predict.md#plot-method-parameters) to see possible arguments to use.
---
!!! tip
You may need to use `clear` to "erase" the view of the image in the terminal.
## FAQ
### How can I view YOLO inference results in a VSCode terminal on macOS or Linux?
To view YOLO inference results in a VSCode terminal on macOS or Linux, follow these steps:
1. Enable the necessary VSCode settings:
```yaml
"terminal.integrated.enableImages": true
"terminal.integrated.gpuAcceleration": "auto"
```
2. Install the sixel library:
```bash
pip install sixel
```
3. Load your YOLO model and run inference:
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
results = model.predict(source="path_to_image")
plot = results[0].plot()
```
4. Convert the inference result image to bytes and display it in the terminal:
```python
import io
import cv2
from sixel import SixelWriter
im_bytes = cv2.imencode(".png", plot)[1].tobytes()
mem_file = io.BytesIO(im_bytes)
SixelWriter().draw(mem_file)
```
For further details, visit the [predict mode](../modes/predict.md) page.
### Why does the sixel protocol only work on Linux and macOS?
The sixel protocol is currently only supported on Linux and macOS because these platforms have native terminal capabilities compatible with sixel graphics. Windows support for terminal graphics using sixel is still under development. For updates on Windows compatibility, check the [VSCode Issue status](https://github.com/microsoft/vscode/issues/198622) and [documentation](https://code.visualstudio.com/docs).
### What if I encounter issues with displaying images in the VSCode terminal?
If you encounter issues displaying images in the VSCode terminal using sixel:
1. Ensure the necessary settings in VSCode are enabled:
```yaml
"terminal.integrated.enableImages": true
"terminal.integrated.gpuAcceleration": "auto"
```
2. Verify the sixel library installation:
```bash
pip install sixel
```
3. Check your image data conversion and plotting code for errors. For example:
```python
import io
import cv2
from sixel import SixelWriter
im_bytes = cv2.imencode(".png", plot)[1].tobytes()
mem_file = io.BytesIO(im_bytes)
SixelWriter().draw(mem_file)
```
If problems persist, consult the [VSCode repository](https://github.com/microsoft/vscode), and visit the [plot method parameters](../modes/predict.md#plot-method-parameters) section for additional guidance.
### Can YOLO display video inference results in the terminal using sixel?
Displaying video inference results or animated GIF frames using sixel in the terminal is currently untested and may not be supported. We recommend starting with static images and verifying compatibility. Attempt video results at your own risk, keeping in mind performance constraints. For more information on plotting inference results, visit the [predict mode](../modes/predict.md) page.
### How can I troubleshoot issues with the `python-sixel` library?
To troubleshoot issues with the `python-sixel` library:
1. Ensure the library is correctly installed in your virtual environment:
```bash
pip install sixel
```
2. Verify that you have the necessary Python and system dependencies.
3. Refer to the [python-sixel GitHub repository](https://github.com/lubosz/python-sixel) for additional documentation and community support.
4. Double-check your code for potential errors, specifically the usage of `SixelWriter` and image data conversion steps.
For further assistance on working with YOLO models and sixel integration, see the [export](../modes/export.md) and [predict mode](../modes/predict.md) documentation pages.

180
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/vision-eye.md Normal file
View File

@@ -0,0 +1,180 @@
---
comments: true
description: Discover VisionEye's object mapping and tracking powered by Ultralytics YOLO26. Simulate human eye precision, track objects, and calculate distances effortlessly.
keywords: VisionEye, YOLO26, Ultralytics, object mapping, object tracking, distance calculation, computer vision, AI, machine learning, Python, tutorial
---
# VisionEye View Object Mapping using Ultralytics YOLO26 🚀
## What is VisionEye Object Mapping?
[Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) VisionEye offers the capability for computers to identify and pinpoint objects, simulating the observational [precision](https://www.ultralytics.com/glossary/precision) of the human eye. This functionality enables computers to discern and focus on specific objects, much like the way the human eye observes details from a particular viewpoint.
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/visioneye-object-mapping-with-tracking.avif" alt="VisionEye object mapping with YOLO tracking">
</p>
!!! example "VisionEye Mapping using Ultralytics YOLO"
=== "CLI"
```bash
# Monitor objects position with visioneye
yolo solutions visioneye show=True
# Pass a source video
yolo solutions visioneye source="path/to/video.mp4"
# Monitor the specific classes
yolo solutions visioneye classes="[0, 5]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("visioneye_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Initialize vision eye object
visioneye = solutions.VisionEye(
show=True, # display the output
model="yolo26n.pt", # use any model that Ultralytics supports, e.g., YOLOv10
classes=[0, 2], # generate visioneye view for specific classes
vision_point=(50, 50), # the point where VisionEye will view objects and draw tracks
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or video processing has been successfully completed.")
break
results = visioneye(im0)
print(results) # access the output
video_writer.write(results.plot_im) # write the video file
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
The `vision_point` tuple represents the observer's position in pixel coordinates. Adjust it to match the camera perspective so the rendered rays correctly illustrate how objects relate to the chosen viewpoint.
### `VisionEye` Arguments
Here's a table with the `VisionEye` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "vision_point"]) }}
You can also utilize various `track` arguments within the `VisionEye` solution:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Furthermore, some visualization arguments are supported, as listed below:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## How VisionEye Works
VisionEye works by establishing a fixed vision point in the frame and drawing lines from this point to detected objects. This simulates how human vision focuses on multiple objects from a single viewpoint. The solution uses [object tracking](https://docs.ultralytics.com/modes/track/) to maintain consistent identification of objects across frames, creating a visual representation of the spatial relationship between the observer (vision point) and the objects in the scene.
The `process` method in the VisionEye class performs several key operations:
1. Extracts tracks (bounding boxes, classes, and masks) from the input image
2. Creates an annotator to draw bounding boxes and labels
3. For each detected object, draws a box label and creates a vision line from the vision point
4. Returns the annotated image with tracking statistics
This approach is particularly useful for applications requiring spatial awareness and object relationship visualization, such as surveillance systems, autonomous navigation, and interactive installations.
## Applications of VisionEye
VisionEye object mapping has numerous practical applications across various industries:
- **Security and Surveillance**: Monitor multiple objects of interest from a fixed camera position
- **Retail Analytics**: Track customer movement patterns in relation to store displays
- **Sports Analysis**: Analyze player positioning and movement from a coach's perspective
- **Autonomous Vehicles**: Visualize how a vehicle "sees" and prioritizes objects in its environment
- **Human-Computer Interaction**: Create more intuitive interfaces that respond to spatial relationships
By combining VisionEye with other Ultralytics solutions like [distance calculation](https://docs.ultralytics.com/guides/distance-calculation/) or [speed estimation](https://docs.ultralytics.com/guides/speed-estimation/), you can build comprehensive systems that not only track objects but also understand their spatial relationships and behaviors.
## Note
For any inquiries, feel free to post your questions in the [Ultralytics Issue Section](https://github.com/ultralytics/ultralytics/issues/new/choose) or the discussion section mentioned below.
## FAQ
### How do I start using VisionEye Object Mapping with Ultralytics YOLO26?
To start using VisionEye Object Mapping with Ultralytics YOLO26, first, you'll need to install the Ultralytics YOLO package via pip. Then, you can use the sample code provided in the documentation to set up [object detection](https://www.ultralytics.com/glossary/object-detection) with VisionEye. Here's a simple example to get you started:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("vision-eye-mapping.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init vision eye object
visioneye = solutions.VisionEye(
show=True, # display the output
model="yolo26n.pt", # use any model that Ultralytics supports, e.g., YOLOv10
classes=[0, 2], # generate visioneye view for specific classes
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or video processing has been successfully completed.")
break
results = visioneye(im0)
print(results) # access the output
video_writer.write(results.plot_im) # write the video file
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### Why should I use Ultralytics YOLO26 for object mapping and tracking?
Ultralytics YOLO26 is renowned for its speed, [accuracy](https://www.ultralytics.com/glossary/accuracy), and ease of integration, making it a top choice for object mapping and tracking. Key advantages include:
1. **State-of-the-art Performance**: Delivers high accuracy in real-time object detection.
2. **Flexibility**: Supports various tasks such as detection, tracking, and distance calculation.
3. **Community and Support**: Extensive documentation and active GitHub community for troubleshooting and enhancements.
4. **Ease of Use**: Intuitive API simplifies complex tasks, allowing for rapid deployment and iteration.
For more information on applications and benefits, check out the [Ultralytics YOLO26 documentation](https://docs.ultralytics.com/models/yolov8/).
### How can I integrate VisionEye with other [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) tools like Comet or ClearML?
Ultralytics YOLO26 can integrate seamlessly with various machine learning tools like Comet and ClearML, enhancing experiment tracking, collaboration, and reproducibility. Follow the detailed guides on [how to use YOLOv5 with Comet](https://www.ultralytics.com/blog/how-to-use-yolov5-with-comet) and [integrate YOLO26 with ClearML](https://docs.ultralytics.com/integrations/clearml/) to get started.
For further exploration and integration examples, check our [Ultralytics Integrations Guide](https://docs.ultralytics.com/integrations/).

215
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/workouts-monitoring.md Normal file
View File

@@ -0,0 +1,215 @@
---
comments: true
description: Optimize your fitness routine with real-time workouts monitoring using Ultralytics YOLO26. Track and improve your exercise form and performance.
keywords: workouts monitoring, Ultralytics YOLO26, pose estimation, fitness tracking, exercise assessment, real-time feedback, exercise form, performance metrics
---
# Workouts Monitoring using Ultralytics YOLO26
<a href="https://colab.research.google.com/github/ultralytics/notebooks/blob/main/notebooks/how-to-monitor-workouts-using-ultralytics-yolo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Workouts Monitoring In Colab"></a>
Monitoring workouts through pose estimation with [Ultralytics YOLO26](https://github.com/ultralytics/ultralytics/) enhances exercise assessment by accurately tracking key body landmarks and joints in real-time. This technology provides instant feedback on exercise form, tracks workout routines, and measures performance metrics, optimizing training sessions for users and trainers alike.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/Ck7DW96dNok"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Monitor Workout Exercises with Ultralytics YOLO | Squats, Leg Extension, Pushups and More
</p>
## Advantages of Workouts Monitoring
- **Optimized Performance:** Tailoring workouts based on monitoring data for better results.
- **Goal Achievement:** Track and adjust fitness goals for measurable progress.
- **Personalization:** Customized workout plans based on individual data for effectiveness.
- **Health Awareness:** Early detection of patterns indicating health issues or over-training.
- **Informed Decisions:** Data-driven decisions for adjusting routines and setting realistic goals.
## Real World Applications
| Workouts Monitoring | Workouts Monitoring |
| :----------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------: |
| ![YOLO push-ups counting with pose estimation](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/pushups-counting.avif) | ![YOLO pull-ups counting with pose estimation](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/pullups-counting.avif) |
| PushUps Counting | PullUps Counting |
!!! example "Workouts Monitoring using Ultralytics YOLO"
=== "CLI"
```bash
# Run a workout example
yolo solutions workout show=True
# Pass a source video
yolo solutions workout source="path/to/video.mp4"
# Use keypoints for pushups
yolo solutions workout kpts="[6, 8, 10]"
```
=== "Python"
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
# Video writer
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("workouts_output.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
# Init AIGym
gym = solutions.AIGym(
show=True, # display the frame
kpts=[6, 8, 10], # keypoints for monitoring specific exercise, by default it's for pushup
model="yolo26n-pose.pt", # path to the YOLO26 pose estimation model file
# line_width=2, # adjust the line width for bounding boxes and text display
)
# Process video
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = gym(im0)
# print(results) # access the output
video_writer.write(results.plot_im) # write the processed frame.
cap.release()
video_writer.release()
cv2.destroyAllWindows() # destroy all opened windows
```
### KeyPoints Map
![YOLO pose estimation keypoints order diagram](https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/keypoints-order-ultralytics-yolov8-pose.avif)
### `AIGym` Arguments
Here's a table with the `AIGym` arguments:
{% from "macros/solutions-args.md" import param_table %}
{{ param_table(["model", "up_angle", "down_angle", "kpts"]) }}
The `AIGym` solution also supports a range of object tracking parameters:
{% from "macros/track-args.md" import param_table %}
{{ param_table(["tracker", "conf", "iou", "classes", "verbose", "device"]) }}
Additionally, the following visualization settings can be applied:
{% from "macros/visualization-args.md" import param_table %}
{{ param_table(["show", "line_width", "show_conf", "show_labels"]) }}
## FAQ
### How do I monitor my workouts using Ultralytics YOLO26?
To monitor your workouts using Ultralytics YOLO26, you can utilize the [pose estimation capabilities](https://docs.ultralytics.com/tasks/pose/) to track and analyze key body landmarks and joints in real-time. This allows you to receive instant feedback on your exercise form, count repetitions, and measure performance metrics. You can start by using the provided example code for push-ups, pull-ups, or ab workouts as shown:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
gym = solutions.AIGym(
line_width=2,
show=True,
kpts=[6, 8, 10],
)
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = gym(im0)
cv2.destroyAllWindows()
```
For further customization and settings, you can refer to the [AIGym](#aigym-arguments) section in the documentation.
### What are the benefits of using Ultralytics YOLO26 for workout monitoring?
Using Ultralytics YOLO26 for workout monitoring provides several key benefits:
- **Optimized Performance:** By tailoring workouts based on monitoring data, you can achieve better results.
- **Goal Achievement:** Easily track and adjust fitness goals for measurable progress.
- **Personalization:** Get customized workout plans based on your individual data for optimal effectiveness.
- **Health Awareness:** Early detection of patterns that indicate potential health issues or over-training.
- **Informed Decisions:** Make data-driven decisions to adjust routines and set realistic goals.
You can watch a [YouTube video demonstration](https://www.youtube.com/watch?v=LGGxqLZtvuw) to see these benefits in action.
### How accurate is Ultralytics YOLO26 in detecting and tracking exercises?
Ultralytics YOLO26 is highly accurate in detecting and tracking exercises due to its state-of-the-art [pose estimation](https://www.ultralytics.com/blog/how-to-use-ultralytics-yolo11-for-pose-estimation) capabilities. It can accurately track key body landmarks and joints, providing real-time feedback on exercise form and performance metrics. The model's pretrained weights and robust architecture ensure high [precision](https://www.ultralytics.com/glossary/precision) and reliability. For real-world examples, check out the [real-world applications](#real-world-applications) section in the documentation, which showcases push-ups and pull-ups counting.
### Can I use Ultralytics YOLO26 for custom workout routines?
Yes, Ultralytics YOLO26 can be adapted for custom workout routines. The `AIGym` class supports different pose types such as `pushup`, `pullup`, and `abworkout`. You can specify keypoints and angles to detect specific exercises. Here is an example setup:
```python
from ultralytics import solutions
gym = solutions.AIGym(
line_width=2,
show=True,
kpts=[6, 8, 10], # For pushups - can be customized for other exercises
)
```
For more details on setting arguments, refer to the [Arguments `AIGym`](#aigym-arguments) section. This flexibility allows you to monitor various exercises and customize routines based on your [fitness goals](https://www.ultralytics.com/blog/ai-in-our-day-to-day-health-and-fitness).
### How can I save the workout monitoring output using Ultralytics YOLO26?
To save the workout monitoring output, you can modify the code to include a video writer that saves the processed frames. Here's an example:
```python
import cv2
from ultralytics import solutions
cap = cv2.VideoCapture("path/to/video.mp4")
assert cap.isOpened(), "Error reading video file"
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
video_writer = cv2.VideoWriter("workouts.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
gym = solutions.AIGym(
line_width=2,
show=True,
kpts=[6, 8, 10],
)
while cap.isOpened():
success, im0 = cap.read()
if not success:
print("Video frame is empty or processing is complete.")
break
results = gym(im0)
video_writer.write(results.plot_im)
cap.release()
video_writer.release()
cv2.destroyAllWindows()
```
This setup writes the monitored video to an output file, allowing you to review your workout performance later or share it with trainers for additional feedback.

301
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/yolo-common-issues.md Normal file
View File

@@ -0,0 +1,301 @@
---
comments: true
description: Comprehensive guide to troubleshoot common YOLO26 issues, from installation errors to model training challenges. Enhance your Ultralytics projects with our expert tips.
keywords: YOLO, YOLO26, troubleshooting, installation errors, model training, GPU issues, Ultralytics, AI, computer vision, deep learning, Python, CUDA, PyTorch, debugging
---
# Troubleshooting Common YOLO Issues
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/yolo-common-issues.avif" alt="YOLO common training and deployment issues">
</p>
## Introduction
This guide serves as a comprehensive aid for troubleshooting common issues encountered while working with YOLO26 on your Ultralytics projects. Navigating through these issues can be a breeze with the right guidance, ensuring your projects remain on track without unnecessary delays.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/TG9exsBlkDE"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Ultralytics YOLO26 Common Issues | Installation Errors, Model Training Issues
</p>
## Common Issues
### Installation Errors
Installation errors can arise due to various reasons, such as incompatible versions, missing dependencies, or incorrect environment setups. First, check to make sure you are doing the following:
- You're using Python 3.8 or later as recommended.
- Ensure that you have the correct version of [PyTorch](https://www.ultralytics.com/glossary/pytorch) (1.8 or later) installed.
- Consider using virtual environments to avoid conflicts.
- Follow the [official installation guide](../quickstart.md) step by step.
Additionally, here are some common installation issues users have encountered, along with their respective solutions:
- Import Errors or Dependency Issues - If you're getting errors during the import of YOLO26, or you're having issues related to dependencies, consider the following troubleshooting steps:
- **Fresh Installation**: Sometimes, starting with a fresh installation can resolve unexpected issues. Especially with libraries like Ultralytics, where updates might introduce changes to the file tree structure or functionalities.
- **Update Regularly**: Ensure you're using the latest version of the library. Older versions might not be compatible with recent updates, leading to potential conflicts or issues.
- **Check Dependencies**: Verify that all required dependencies are correctly installed and are of the compatible versions.
- **Review Changes**: If you initially cloned or installed an older version, be aware that significant updates might affect the library's structure or functionalities. Always refer to the official documentation or changelogs to understand any major changes.
- Remember, keeping your libraries and dependencies up-to-date is crucial for a smooth and error-free experience.
- Running YOLO26 on GPU - If you're having trouble running YOLO26 on GPU, consider the following troubleshooting steps:
- **Verify CUDA Compatibility and Installation**: Ensure your GPU is CUDA compatible and that CUDA is correctly installed. Use the `nvidia-smi` command to check the status of your NVIDIA GPU and CUDA version.
- **Check PyTorch and CUDA Integration**: Ensure PyTorch can utilize CUDA by running `import torch; print(torch.cuda.is_available())` in a Python terminal. If it returns 'True', PyTorch is set up to use CUDA.
- **Check GPU compatibility**: Support for GPU architectures earlier than Turing and compute capability (SM) < 7.5 was abandoned since cuDNN 9.11.0. So if you have an older GPU - like 1080Ti - you may have to use a version of PyTorch built against an older version of CUDA/cuDNN. You can check this by running `import torch; cap = torch.cuda.get_device_capability(0) if torch.cuda.is_available() else (0, 0); cudnn = torch.backends.cudnn.version() or 0; ok = "not compatible" if cudnn >= 91100 and (cap[0] < 7 or (cap[0] == 7 and cap[1] < 5)) else "should be ok"; print(f"Compute capability: SM {cap[0]}.{cap[1]}, cuDNN: {cudnn} => {ok}")`
- **Environment Activation**: Ensure you're in the correct environment where all necessary packages are installed.
- **Update Your Packages**: Outdated packages might not be compatible with your GPU. Keep them updated.
- **Program Configuration**: Check if the program or code specifies GPU usage. In YOLO26, this might be in the settings or configuration.
### Model Training Issues
This section will address common issues faced while training and their respective explanations and solutions.
#### Verification of Configuration Settings
**Issue**: You are unsure whether the configuration settings in the `.yaml` file are being applied correctly during model training.
**Solution**: The configuration settings in the `.yaml` file should be applied when using the `model.train()` function. To ensure that these settings are correctly applied, follow these steps:
- Confirm that the path to your `.yaml` configuration file is correct.
- Make sure you pass the path to your `.yaml` file as the `data` argument when calling `model.train()`, as shown below:
```python
model.train(data="/path/to/your/data.yaml", batch=4)
```
#### Accelerating Training with Multiple GPUs
**Issue**: Training is slow on a single GPU, and you want to speed up the process using multiple GPUs.
**Solution**: Increasing the [batch size](https://www.ultralytics.com/glossary/batch-size) can accelerate training, but it's essential to consider GPU memory capacity. To speed up training with multiple GPUs, follow these steps:
- Ensure that you have multiple GPUs available.
- Modify your `.yaml` configuration file to specify the number of GPUs to use, e.g., `gpus: 4`.
- Increase the batch size accordingly to fully utilize the multiple GPUs without exceeding memory limits.
- Modify your training command to utilize multiple GPUs:
```python
# Adjust the batch size and other settings as needed to optimize training speed
model.train(data="/path/to/your/data.yaml", batch=32)
```
#### Continuous Monitoring Parameters
**Issue**: You want to know which parameters should be continuously monitored during training, apart from loss.
**Solution**: While loss is a crucial metric to monitor, it's also essential to track other metrics for model performance optimization. Some key metrics to monitor during training include:
- Precision
- Recall
- [Mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP)
You can access these metrics from the training logs or by using tools like TensorBoard or wandb for visualization. Implementing early stopping based on these metrics can help you achieve better results.
#### Tools for Tracking Training Progress
**Issue**: You are looking for recommendations on tools to track training progress.
**Solution**: To track and visualize training progress, you can consider using the following tools:
- [TensorBoard](https://www.tensorflow.org/tensorboard): TensorBoard is a popular choice for visualizing training metrics, including loss, [accuracy](https://www.ultralytics.com/glossary/accuracy), and more. You can integrate it with your YOLO26 training process.
- [Comet](https://bit.ly/yolov8-readme-comet): Comet provides an extensive toolkit for experiment tracking and comparison. It allows you to track metrics, hyperparameters, and even model weights. Integration with YOLO models is also straightforward, providing you with a complete overview of your experiment cycle.
- [Ultralytics Platform](https://platform.ultralytics.com/): Ultralytics Platform offers a specialized environment for tracking YOLO models, giving you a one-stop platform to manage metrics, datasets, and even collaborate with your team. Given its tailored focus on YOLO, it offers more customized tracking options.
Each of these tools offers its own set of advantages, so you may want to consider the specific needs of your project when making a choice.
#### How to Check if Training is Happening on the GPU
**Issue**: The 'device' value in the training logs is 'null,' and you're unsure if training is happening on the GPU.
**Solution**: The 'device' value being 'null' typically means that the training process is set to automatically use an available GPU, which is the default behavior. To ensure training occurs on a specific GPU, you can manually set the 'device' value to the GPU index (e.g., '0' for the first GPU) in your .yaml configuration file:
```yaml
device: 0
```
This will explicitly assign the training process to the specified GPU. If you wish to train on the CPU, set 'device' to 'cpu'.
Keep an eye on the 'runs' folder for logs and metrics to monitor training progress effectively.
#### Key Considerations for Effective Model Training
Here are some things to keep in mind, if you are facing issues related to model training.
**Dataset Format and Labels**
- Importance: The foundation of any [machine learning](https://www.ultralytics.com/glossary/machine-learning-ml) model lies in the quality and format of the data it is trained on.
- Recommendation: Ensure that your custom dataset and its associated labels adhere to the expected format. It's crucial to verify that annotations are accurate and of high quality. Incorrect or subpar annotations can derail the model's learning process, leading to unpredictable outcomes.
**Model Convergence**
- Importance: Achieving model convergence ensures that the model has sufficiently learned from the [training data](https://www.ultralytics.com/glossary/training-data).
- Recommendation: When training a model 'from scratch', it's vital to ensure that the model reaches a satisfactory level of convergence. This might necessitate a longer training duration, with more [epochs](https://www.ultralytics.com/glossary/epoch), compared to when you're fine-tuning an existing model.
**[Learning Rate](https://www.ultralytics.com/glossary/learning-rate) and Batch Size**
- Importance: These hyperparameters play a pivotal role in determining how the model updates its weights during training.
- Recommendation: Regularly evaluate if the chosen learning rate and batch size are optimal for your specific dataset. Parameters that are not in harmony with the dataset's characteristics can hinder the model's performance.
**Class Distribution**
- Importance: The distribution of classes in your dataset can influence the model's prediction tendencies.
- Recommendation: Regularly assess the distribution of classes within your dataset. If there's a class imbalance, there's a risk that the model will develop a bias towards the more prevalent class. This bias can be evident in the confusion matrix, where the model might predominantly predict the majority class.
**Cross-Check with Pretrained Weights**
- Importance: Leveraging pretrained weights can provide a solid starting point for model training, especially when data is limited.
- Recommendation: As a diagnostic step, consider training your model using the same data but initializing it with pretrained weights. If this approach yields a well-formed confusion matrix, it could suggest that the 'from scratch' model might require further training or adjustments.
### Issues Related to Model Predictions
This section will address common issues faced during model prediction.
#### Getting Bounding Box Predictions With Your YOLO26 Custom Model
**Issue**: When running predictions with a custom YOLO26 model, there are challenges with the format and visualization of the bounding box coordinates.
**Solution**:
- Coordinate Format: YOLO26 provides bounding box coordinates in absolute pixel values. To convert these to relative coordinates (ranging from 0 to 1), you need to divide by the image dimensions. For example, let's say your image size is 640x640. Then you would do the following:
```python
# Convert absolute coordinates to relative coordinates
x1 = x1 / 640 # Divide x-coordinates by image width
x2 = x2 / 640
y1 = y1 / 640 # Divide y-coordinates by image height
y2 = y2 / 640
```
- File Name: To obtain the file name of the image you're predicting on, access the image file path directly from the result object within your prediction loop.
#### Filtering Objects in YOLO26 Predictions
**Issue**: Facing issues with how to filter and display only specific objects in the prediction results when running YOLO26 using the Ultralytics library.
**Solution**: To detect specific classes use the classes argument to specify the classes you want to include in the output. For instance, to detect only cars (assuming 'cars' have class index 2):
```bash
yolo task=detect mode=segment model=yolo26n-seg.pt source='path/to/car.mp4' show=True classes=2
```
#### Understanding Precision Metrics in YOLO26
**Issue**: Confusion regarding the difference between box precision, mask precision, and [confusion matrix](https://www.ultralytics.com/glossary/confusion-matrix) precision in YOLO26.
**Solution**: Box precision measures the accuracy of predicted bounding boxes compared to the actual ground truth boxes using IoU (Intersection over Union) as the metric. Mask precision assesses the agreement between predicted segmentation masks and ground truth masks in pixel-wise object classification. Confusion matrix precision, on the other hand, focuses on overall classification accuracy across all classes and does not consider the geometric accuracy of predictions. It's important to note that a [bounding box](https://www.ultralytics.com/glossary/bounding-box) can be geometrically accurate (true positive) even if the class prediction is wrong, leading to differences between box precision and confusion matrix precision. These metrics evaluate distinct aspects of a model's performance, reflecting the need for different evaluation metrics in various tasks.
#### Extracting Object Dimensions in YOLO26
**Issue**: Difficulty in retrieving the length and height of detected objects in YOLO26, especially when multiple objects are detected in an image.
**Solution**: To retrieve the bounding box dimensions, first use the Ultralytics YOLO26 model to predict objects in an image. Then, extract the width and height information of bounding boxes from the prediction results.
```python
from ultralytics import YOLO
# Load a pretrained YOLO26 model
model = YOLO("yolo26n.pt")
# Specify the source image
source = "https://ultralytics.com/images/bus.jpg"
# Make predictions
results = model.predict(source, save=True, imgsz=320, conf=0.25)
# Extract bounding box dimensions
boxes = results[0].boxes.xywh.cpu()
for box in boxes:
x, y, w, h = box
print(f"Width of Box: {w}, Height of Box: {h}")
```
### Deployment Challenges
#### GPU Deployment Issues
**Issue:** Deploying models in a multi-GPU environment can sometimes lead to unexpected behaviors like unexpected memory usage, inconsistent results across GPUs, etc.
**Solution:** Check for default GPU initialization. Some frameworks, like PyTorch, might initialize CUDA operations on a default GPU before transitioning to the designated GPUs. To bypass unexpected default initializations, specify the GPU directly during deployment and prediction. Then, use tools to monitor GPU utilization and memory usage to identify any anomalies in real-time. Also, ensure you're using the latest version of the framework or library.
#### Model Conversion/Exporting Issues
**Issue:** During the process of converting or exporting machine learning models to different formats or platforms, users might encounter errors or unexpected behaviors.
**Solution:**
- Compatibility Check: Ensure that you are using versions of libraries and frameworks that are compatible with each other. Mismatched versions can lead to unexpected errors during conversion.
- Environment Reset: If you're using an interactive environment like Jupyter or Colab, consider restarting your environment after making significant changes or installations. A fresh start can sometimes resolve underlying issues.
- Official Documentation: Always refer to the official documentation of the tool or library you are using for conversion. It often contains specific guidelines and best practices for model exporting.
- Community Support: Check the library or framework's official repository for similar issues reported by other users. The maintainers or community might have provided solutions or workarounds in discussion threads.
- Update Regularly: Ensure that you are using the latest version of the tool or library. Developers frequently release updates that fix known bugs or improve functionality.
- Test Incrementally: Before performing a full conversion, test the process with a smaller model or dataset to identify potential issues early on.
## Community and Support
Engaging with a community of like-minded individuals can significantly enhance your experience and success in working with YOLO26. Below are some channels and resources you may find helpful.
### Forums and Channels for Getting Help
**GitHub Issues:** The YOLO26 repository on GitHub has an [Issues tab](https://github.com/ultralytics/ultralytics/issues) where you can ask questions, report bugs, and suggest new features. The community and maintainers are active here, and it's a great place to get help with specific problems.
**Ultralytics Discord Server:** Ultralytics has a [Discord server](https://discord.com/invite/ultralytics) where you can interact with other users and the developers.
### Official Documentation and Resources
**Ultralytics YOLO26 Docs**: The [official documentation](../index.md) provides a comprehensive overview of YOLO26, along with guides on installation, usage, and troubleshooting.
These resources should provide a solid foundation for troubleshooting and improving your YOLO26 projects, as well as connecting with others in the YOLO26 community.
## Conclusion
Troubleshooting is an integral part of any development process, and being equipped with the right knowledge can significantly reduce the time and effort spent in resolving issues. This guide aimed to address the most common challenges faced by users of the YOLO26 model within the Ultralytics ecosystem. By understanding and addressing these common issues, you can ensure smoother project progress and achieve better results with your [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks.
Remember, the Ultralytics community is a valuable resource. Engaging with fellow developers and experts can provide additional insights and solutions that might not be covered in standard documentation. Always keep learning, experimenting, and sharing your experiences to contribute to the collective knowledge of the community.
## FAQ
### How do I resolve installation errors with YOLO26?
Installation errors can often be due to compatibility issues or missing dependencies. Ensure you use Python 3.8 or later and have PyTorch 1.8 or later installed. It's beneficial to use virtual environments to avoid conflicts. For a step-by-step installation guide, follow our [official installation guide](../quickstart.md). If you encounter import errors, try a fresh installation or update the library to the latest version.
### Why is my YOLO26 model training slow on a single GPU?
Training on a single GPU might be slow due to large batch sizes or insufficient memory. To speed up training, use multiple GPUs. Ensure your system has multiple GPUs available and adjust your `.yaml` configuration file to specify the number of GPUs, e.g., `gpus: 4`. Increase the batch size accordingly to fully utilize the GPUs without exceeding memory limits. Example command:
```python
model.train(data="/path/to/your/data.yaml", batch=32)
```
### How can I ensure my YOLO26 model is training on the GPU?
If the 'device' value shows 'null' in the training logs, it generally means the training process is set to automatically use an available GPU. To explicitly assign a specific GPU, set the 'device' value in your `.yaml` configuration file. For instance:
```yaml
device: 0
```
This sets the training process to the first GPU. Consult the `nvidia-smi` command to confirm your CUDA setup.
### How can I monitor and track my YOLO26 model training progress?
Tracking and visualizing training progress can be efficiently managed through tools like [TensorBoard](https://www.tensorflow.org/tensorboard), [Comet](https://bit.ly/yolov8-readme-comet), and [Ultralytics Platform](https://platform.ultralytics.com/). These tools allow you to log and visualize metrics such as loss, [precision](https://www.ultralytics.com/glossary/precision), [recall](https://www.ultralytics.com/glossary/recall), and mAP. Implementing [early stopping](#continuous-monitoring-parameters) based on these metrics can also help achieve better training outcomes.
### What should I do if YOLO26 is not recognizing my dataset format?
Ensure your dataset and labels conform to the expected format. Verify that annotations are accurate and of high quality. If you face any issues, refer to the [Data Collection and Annotation](https://docs.ultralytics.com/guides/data-collection-and-annotation/) guide for best practices. For more dataset-specific guidance, check the [Datasets](https://docs.ultralytics.com/datasets/) section in the documentation.

517
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/yolo-data-augmentation.md Normal file
View File

@@ -0,0 +1,517 @@
---
comments: true
description: Learn about essential data augmentation techniques in Ultralytics YOLO. Explore various transformations, their impacts, and how to implement them effectively for improved model performance.
keywords: YOLO data augmentation, computer vision, deep learning, image transformations, model training, Ultralytics YOLO, HSV adjustments, geometric transformations, mosaic augmentation
---
# Data Augmentation using Ultralytics YOLO
<p align="center">
<img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/albumentations-augmentation.avif" alt="YOLO data augmentation examples showing original and augmented images for training">
</p>
## Introduction
[Data augmentation](https://www.ultralytics.com/glossary/data-augmentation) is a crucial technique in computer vision that artificially expands your training dataset by applying various transformations to existing images. When training [deep learning](https://www.ultralytics.com/glossary/deep-learning-dl) models like Ultralytics YOLO, data augmentation helps improve model robustness, reduces overfitting, and enhances generalization to real-world scenarios.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/e-TwqFtay90"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to use Mosaic, MixUp & more Data Augmentations to help Ultralytics YOLO Models generalize better 🚀
</p>
### Why Data Augmentation Matters
Data augmentation serves multiple critical purposes in training computer vision models:
- **Expanded Dataset**: By creating variations of existing images, you can effectively increase your training dataset size without collecting new data.
- **Improved Generalization**: Models learn to recognize objects under various conditions, making them more robust in real-world applications.
- **Reduced Overfitting**: By introducing variability in the training data, models are less likely to memorize specific image characteristics.
- **Enhanced Performance**: Models trained with proper augmentation typically achieve better [accuracy](https://www.ultralytics.com/glossary/accuracy) on validation and test sets.
Ultralytics YOLO's implementation provides a comprehensive suite of augmentation techniques, each serving specific purposes and contributing to model performance in different ways. This guide will explore each augmentation parameter in detail, helping you understand when and how to use them effectively in your projects.
### Example Configurations
You can customize each parameter using the Python API, the command line interface (CLI), or a configuration file. Below are examples of how to set up data augmentation in each method.
!!! example "Configuration Examples"
=== "Python"
```python
import albumentations as A
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Training with custom augmentation parameters
model.train(data="coco.yaml", epochs=100, hsv_h=0.03, hsv_s=0.6, hsv_v=0.5)
# Training without any augmentations (disabled values omitted for clarity)
model.train(
data="coco.yaml",
epochs=100,
hsv_h=0.0,
hsv_s=0.0,
hsv_v=0.0,
translate=0.0,
scale=0.0,
fliplr=0.0,
mosaic=0.0,
erasing=0.0,
auto_augment=None,
)
# Training with custom Albumentations transforms (Python API only)
custom_transforms = [
A.Blur(blur_limit=7, p=0.5),
A.CLAHE(clip_limit=4.0, p=0.5),
]
model.train(data="coco.yaml", epochs=100, augmentations=custom_transforms)
```
=== "CLI"
```bash
# Training with custom augmentation parameters
yolo detect train data=coco8.yaml model=yolo26n.pt epochs=100 hsv_h=0.03 hsv_s=0.6 hsv_v=0.5
```
#### Using a configuration file
You can define all training parameters, including augmentations, in a YAML configuration file (e.g., `train_custom.yaml`). The `mode` parameter is only required when using the CLI. This new YAML file will then override [the default one](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/default.yaml) located in the `ultralytics` package.
```yaml
# train_custom.yaml
# 'mode' is required only for CLI usage
mode: train
data: coco8.yaml
model: yolo26n.pt
epochs: 100
hsv_h: 0.03
hsv_s: 0.6
hsv_v: 0.5
```
Then launch the training with the Python API:
!!! example "Train Example"
=== "Python"
```python
from ultralytics import YOLO
# Load a COCO-pretrained YOLO26n model
model = YOLO("yolo26n.pt")
# Train the model with custom configuration
model.train(cfg="train_custom.yaml")
```
=== "CLI"
```bash
# Train the model with custom configuration
yolo detect train model="yolo26n.pt" cfg=train_custom.yaml
```
## Color Space Augmentations
### Hue Adjustment (`hsv_h`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ hsv_h }}`
- **Usage**: Shifts image colors while preserving their relationships. The `hsv_h` hyperparameter defines the shift magnitude, with the final adjustment randomly chosen between `-hsv_h` and `hsv_h`. For example, with `hsv_h=0.3`, the shift is randomly selected within `-0.3` to `0.3`. For values above `0.5`, the hue shift wraps around the color wheel, that's why the augmentations look the same between `0.5` and `-0.5`.
- **Purpose**: Particularly useful for outdoor scenarios where lighting conditions can dramatically affect object appearance. For example, a banana might look more yellow under bright sunlight but more greenish indoors.
- **Ultralytics' implementation**: [RandomHSV](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomHSV)
| **`-0.5`** | **`-0.25`** | **`0.0`** | **`0.25`** | **`0.5`** |
| :----------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_h_-0.5.avif" alt="Hue shift -0.5 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_h_-0.25.avif" alt="Hue shift -0.25 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_h_0.25.avif" alt="Hue shift 0.25 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_h_0.5.avif" alt="Hue shift -0.5 augmentation"/> |
### Saturation Adjustment (`hsv_s`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ hsv_s }}`
- **Usage**: Modifies the intensity of colors in the image. The `hsv_s` hyperparameter defines the shift magnitude, with the final adjustment randomly chosen between `-hsv_s` and `hsv_s`. For example, with `hsv_s=0.7`, the intensity is randomly selected within `-0.7` to `0.7`.
- **Purpose**: Helps models handle varying weather conditions and camera settings. For example, a red traffic sign might appear highly vivid on a sunny day but look dull and faded in foggy conditions.
- **Ultralytics' implementation**: [RandomHSV](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomHSV)
| **`-1.0`** | **`-0.5`** | **`0.0`** | **`0.5`** | **`1.0`** |
| :-------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_s_-1.avif" alt="Saturation -1.0 grayscale augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_s_-0.5.avif" alt="Saturation -0.5 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_s_0.5.avif" alt="Saturation 0.5 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_s_1.avif" alt="Saturation 1.0 vivid augmentation"/> |
### Brightness Adjustment (`hsv_v`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ hsv_v }}`
- **Usage**: Changes the brightness of the image. The `hsv_v` hyperparameter defines the shift magnitude, with the final adjustment randomly chosen between `-hsv_v` and `hsv_v`. For example, with `hsv_v=0.4`, the intensity is randomly selected within `-0.4` to `0.4`.
- **Purpose**: Essential for training models that need to perform in different lighting conditions. For example, a red apple might look bright in sunlight but much darker in the shade.
- **Ultralytics' implementation**: [RandomHSV](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomHSV)
| **`-1.0`** | **`-0.5`** | **`0.0`** | **`0.5`** | **`1.0`** |
| :--------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_v_-1.avif" alt="Brightness -1.0 dark augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_v_-0.5.avif" alt="Brightness -0.5 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_v_0.5.avif" alt="Brightness 0.5 augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_hsv_v_1.avif" alt="Brightness 1.0 bright augmentation"/> |
## Geometric Transformations
### Rotation (`degrees`)
- **Range**: `0.0` to `180`
- **Default**: `{{ degrees }}`
- **Usage**: Rotates images randomly within the specified range. The `degrees` hyperparameter defines the rotation angle, with the final adjustment randomly chosen between `-degrees` and `degrees`. For example, with `degrees=10.0`, the rotation is randomly selected within `-10.0` to `10.0`.
- **Purpose**: Crucial for applications where objects can appear at different orientations. For example, in aerial drone imagery, vehicles can be oriented in any direction, requiring models to recognize objects regardless of their rotation.
- **Ultralytics' implementation**: [RandomPerspective](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomPerspective)
| **`-180`** | **`-90`** | **`0.0`** | **`90`** | **`180`** |
| :-----------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_degrees_-180.avif" alt="Rotation -180 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_degrees_-90.avif" alt="Rotation -90 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_degrees_90.avif" alt="Rotation 90 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_degrees_180.avif" alt="Rotation 180 degrees augmentation"/> |
### Translation (`translate`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ translate }}`
- **Usage**: Shifts images horizontally and vertically by a random fraction of the image size. The `translate` hyperparameter defines the shift magnitude, with the final adjustment randomly chosen twice (once for each axis) within the range `-translate` and `translate`. For example, with `translate=0.5`, the translation is randomly selected within `-0.5` to `0.5` on the x-axis, and another independent random value is selected within the same range on the y-axis.
- **Purpose**: Helps models learn to detect partially visible objects and improves robustness to object position. For example, in vehicle damage assessment applications, car parts may appear fully or partially in frame depending on the photographer's position and distance, the translation augmentation will teach the model to recognize these features regardless of their completeness or position.
- **Ultralytics' implementation**: [RandomPerspective](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomPerspective)
- **Note**: For simplicity, the translations applied below are the same each time for both `x` and `y` axes. Values `-1.0` and `1.0`are not shown as they would translate the image completely out of the frame.
| `-0.5` | **`-0.25`** | **`0.0`** | **`0.25`** | **`0.5`** |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_translate_-0.5.avif" alt="Translation -0.5 shift augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_translate_-0.25.avif" alt="Translation -0.25 shift augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_translate_0.25.avif" alt="Translation 0.25 shift augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_translate_0.5.avif" alt="Translation 0.5 shift augmentation"/> |
### Scale (`scale`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ scale }}`
- **Usage**: Resizes images by a random factor within the specified range. The `scale` hyperparameter defines the scaling factor, with the final adjustment randomly chosen between `1-scale` and `1+scale`. For example, with `scale=0.5`, the scaling is randomly selected within `0.5` to `1.5`.
- **Purpose**: Enables models to handle objects at different distances and sizes. For example, in autonomous driving applications, vehicles can appear at various distances from the camera, requiring the model to recognize them regardless of their size.
- **Ultralytics' implementation**: [RandomPerspective](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomPerspective)
- **Note**:
- The value `-1.0` is not shown as it would make the image disappear, while `1.0` simply results in a 2x zoom.
- The values displayed in the table below are the ones applied through the hyperparameter `scale`, not the final scale factor.
- If `scale` is greater than `1.0`, the image can be either very small or flipped, as the scaling factor is randomly chosen between `1-scale` and `1+scale`. For example, with `scale=3.0`, the scaling is randomly selected within `-2.0` to `4.0`. If a negative value is chosen, the image is flipped.
| **`-0.5`** | **`-0.25`** | **`0.0`** | **`0.25`** | **`0.5`** |
| :-------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_scale_-0.5.avif" alt="Scale 0.5x zoom out augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_scale_-0.25.avif" alt="Scale 0.75x zoom out augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_scale_0.25.avif" alt="Scale 1.25x zoom in augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_scale_0.5.avif" alt="Scale 1.5x zoom in augmentation"/> |
### Shear (`shear`)
- **Range**: `-180` to `+180`
- **Default**: `{{ shear }}`
- **Usage**: Introduces a geometric transformation that skews the image along both x-axis and y-axis, effectively shifting parts of the image in one direction while maintaining parallel lines. The `shear` hyperparameter defines the shear angle, with the final adjustment randomly chosen between `-shear` and `shear`. For example, with `shear=10.0`, the shear is randomly selected within `-10` to `10` on the x-axis, and another independent random value is selected within the same range on the y-axis.
- **Purpose**: Helps models generalize to variations in viewing angles caused by slight tilts or oblique viewpoints. For instance, in traffic monitoring, objects like cars and road signs may appear slanted due to non-perpendicular camera placements. Applying shear augmentation ensures the model learns to recognize objects despite such skewed distortions.
- **Ultralytics' implementation**: [RandomPerspective](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomPerspective)
- **Note**:
- `shear` values can rapidly distort the image, so it's recommended to start with small values and gradually increase them.
- Unlike perspective transformations, shear does not introduce depth or vanishing points but instead distorts the shape of objects by changing their angles while keeping opposite sides parallel.
| **`-10`** | **`-5`** | **`0.0`** | **`5`** | **`10`** |
| :----------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_shear_-10.avif" alt="Shear -10 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_shear_-5.avif" alt="Shear -5 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_shear_5.avif" alt="Shear 5 degrees augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_shear_10.avif" alt="Shear 10 degrees augmentation"/> |
### Perspective (`perspective`)
- **Range**: `0.0` - `0.001`
- **Default**: `{{ perspective }}`
- **Usage**: Applies a full perspective transformation along both x-axis and y-axis, simulating how objects appear when viewed from different depths or angles. The `perspective` hyperparameter defines the perspective magnitude, with the final adjustment randomly chosen between `-perspective` and `perspective`. For example, with `perspective=0.001`, the perspective is randomly selected within `-0.001` to `0.001` on the x-axis, and another independent random value is selected within the same range on the y-axis.
- **Purpose**: Perspective augmentation is crucial for handling extreme viewpoint changes, especially in scenarios where objects appear foreshortened or distorted due to perspective shifts. For example, in drone-based object detection, buildings, roads, and vehicles can appear stretched or compressed depending on the drone's tilt and altitude. By applying perspective transformations, models learn to recognize objects despite these perspective-induced distortions, improving their robustness in real-world deployments.
- **Ultralytics' implementation**: [RandomPerspective](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomPerspective)
| **`-0.001`** | **`-0.0005`** | **`0.0`** | **`0.0005`** | **`0.001`** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_perspective_-0.001.avif" alt="Perspective -0.001 transformation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_perspective_-0.0005.avif" alt="Perspective -0.0005 transformation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_perspective_0.0005.avif" alt="Perspective 0.0005 transformation"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_geometric_perspective_0.001.avif" alt="Perspective 0.001 transformation"/> |
### Flip Up-Down (`flipud`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ flipud }}`
- **Usage**: Performs a vertical flip by inverting the image along the y-axis. This transformation mirrors the entire image upside-down but preserves all spatial relationships between objects. The flipud hyperparameter defines the probability of applying the transformation, with a value of `flipud=1.0` ensuring that all images are flipped and a value of `flipud=0.0` disabling the transformation entirely. For example, with `flipud=0.5`, each image has a 50% chance of being flipped upside-down.
- **Purpose**: Useful for scenarios where objects can appear upside down. For example, in robotic vision systems, objects on conveyor belts or robotic arms may be picked up and placed in various orientations. Vertical flipping helps the model recognize objects regardless of their top-down positioning.
- **Ultralytics' implementation**: [RandomFlip](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomFlip)
| **`flipud` off** | **`flipud` on** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation" width="38%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_flip_vertical_1.avif" alt="Vertical flip augmentation enabled" width="38%"/> |
### Flip Left-Right (`fliplr`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ fliplr }}`
- **Usage**: Performs a horizontal flip by mirroring the image along the x-axis. This transformation swaps the left and right sides while maintaining spatial consistency, which helps the model generalize to objects appearing in mirrored orientations. The `fliplr` hyperparameter defines the probability of applying the transformation, with a value of `fliplr=1.0` ensuring that all images are flipped and a value of `fliplr=0.0` disabling the transformation entirely. For example, with `fliplr=0.5`, each image has a 50% chance of being flipped left to right.
- **Purpose**: Horizontal flipping is widely used in object detection, pose estimation, and facial recognition to improve robustness against left-right variations. For example, in autonomous driving, vehicles and pedestrians can appear on either side of the road, and horizontal flipping helps the model recognize them equally well in both orientations.
- **Ultralytics' implementation**: [RandomFlip](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.RandomFlip)
| **`fliplr` off** | **`fliplr` on** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation" width="38%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_flip_horizontal_1.avif" alt="Horizontal flip augmentation enabled" width="38%"/> |
### BGR Channel Swap (`bgr`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ bgr }}`
- **Usage**: Swaps the color channels of an image from RGB to BGR, altering the order in which colors are represented. The `bgr` hyperparameter defines the probability of applying the transformation, with `bgr=1.0` ensuring all images undergo the channel swap and `bgr=0.0` disabling it. For example, with `bgr=0.5`, each image has a 50% chance of being converted from RGB to BGR.
- **Purpose**: Increases robustness to different color channel orderings. For example, when training models that must work across various camera systems and imaging libraries where RGB and BGR formats may be inconsistently used, or when deploying models to environments where the input color format might differ from the training data.
- **Ultralytics' implementation**: [Format](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.Format)
| **`bgr` off** | **`bgr` on** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation" width="38%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_bgr_channel_swap_1.avif" alt="BGR channel swap augmentation" width="38%"/> |
### Mosaic (`mosaic`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ mosaic }}`
- **Usage**: Combines four training images into one. The `mosaic` hyperparameter defines the probability of applying the transformation, with `mosaic=1.0` ensuring that all images are combined and `mosaic=0.0` disabling the transformation. For example, with `mosaic=0.5`, each image has a 50% chance of being combined with three other images.
- **Purpose**: Highly effective for improving small object detection and context understanding. For example, in wildlife conservation projects where animals may appear at various distances and scales, mosaic augmentation helps the model learn to recognize the same species across different sizes, partial occlusions, and environmental contexts by artificially creating diverse training samples from limited data.
- **Ultralytics' implementation**: [Mosaic](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.Mosaic)
- **Note**:
- Even if the `mosaic` augmentation makes the model more robust, it can also make the training process more challenging.
- The `mosaic` augmentation can be disabled near the end of training by setting `close_mosaic` to the number of epochs before completion when it should be turned off. For example, if `epochs` is set to `200` and `close_mosaic` is set to `20`, the `mosaic` augmentation will be disabled after `180` epochs. If `close_mosaic` is set to `0`, the `mosaic` augmentation will be enabled for the entire training process.
- The center of the generated mosaic is determined using random values, and can either be inside the image or outside of it.
- The current implementation of the `mosaic` augmentation combines 4 images picked randomly from the dataset. If the dataset is small, the same image may be used multiple times in the same mosaic.
| **`mosaic` off** | **`mosaic` on** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation" width="38%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_mosaic_on.avif" alt="Mosaic 4-image augmentation enabled" width="55%"/> |
### Mixup (`mixup`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ mixup }}`
- **Usage**: Blends two images and their labels with given probability. The `mixup` hyperparameter defines the probability of applying the transformation, with `mixup=1.0` ensuring that all images are mixed and `mixup=0.0` disabling the transformation. For example, with `mixup=0.5`, each image has a 50% chance of being mixed with another image.
- **Purpose**: Improves model robustness and reduces overfitting. For example, in retail product recognition systems, mixup helps the model learn more robust features by blending images of different products, teaching it to identify items even when they're partially visible or obscured by other products on crowded store shelves.
- **Ultralytics' implementation**: [Mixup](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.MixUp)
- **Note**:
- The `mixup` ratio is a random value picked from a `np.random.beta(32.0, 32.0)` beta distribution, meaning each image contributes approximately 50%, with slight variations.
| **First image, `mixup` off** | **Second image, `mixup` off** | **`mixup` on** |
| :-----------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="First image for MixUp blending" width="60%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_mixup_identity_2.avif" alt="Second image for MixUp blending" width="60%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_mixup_on.avif" alt="MixUp blending augmentation enabled" width="85%"/> |
### CutMix (`cutmix`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ cutmix }}`
- **Usage**: Cuts a rectangular region from one image and pastes it onto another image with given probability. The `cutmix` hyperparameter defines the probability of applying the transformation, with `cutmix=1.0` ensuring that all images undergo this transformation and `cutmix=0.0` disabling it completely. For example, with `cutmix=0.5`, each image has a 50% chance of having a region replaced with a patch from another image.
- **Purpose**: Enhances model performance by creating realistic occlusion scenarios while maintaining local feature integrity. For example, in autonomous driving systems, cutmix helps the model learn to recognize vehicles or pedestrians even when they're partially occluded by other objects, improving detection accuracy in complex real-world environments with overlapping objects.
- **Ultralytics' implementation**: [CutMix](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.CutMix)
- **Note**:
- The size and position of the cut region is determined randomly for each application.
- Unlike mixup which blends pixel values globally, `cutmix` maintains the original pixel intensities within the cut regions, preserving local features.
- A region is pasted into the target image only if it does not overlap with any existing bounding box. Additionally, only the bounding boxes that retain at least `0.1` (10%) of their original area within the pasted region are preserved.
- This minimum bounding box area threshold cannot be changed with the current implementation and is set to `0.1` by default.
| **First image, `cutmix` off** | **Second image, `cutmix` off** | **`cutmix` on** |
| :------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_cutmix_identity_1.avif" alt="First image for CutMix" width="85%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_cutmix_identity_2.avif" alt="Second image for CutMix" width="85%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_cutmix_on.avif" alt="CutMix augmentation enabled" width="85%"/> |
## Segmentation-Specific Augmentations
### Copy-Paste (`copy_paste`)
- **Range**: `0.0` - `1.0`
- **Default**: `{{ copy_paste }}`
- **Usage**: Only works for segmentation tasks, this augmentation copies objects within or between images based on a specified probability, controlled by the [`copy_paste_mode`](#copy-paste-mode-copy_paste_mode). The `copy_paste` hyperparameter defines the probability of applying the transformation, with `copy_paste=1.0` ensuring that all images are copied and `copy_paste=0.0` disabling the transformation. For example, with `copy_paste=0.5`, each image has a 50% chance of having objects copied from another image.
- **Purpose**: Particularly useful for instance segmentation tasks and rare object classes. For example, in industrial defect detection where certain types of defects appear infrequently, copy-paste augmentation can artificially increase the occurrence of these rare defects by copying them from one image to another, helping the model better learn these underrepresented cases without requiring additional defective samples.
- **Ultralytics' implementation**: [CopyPaste](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.CopyPaste)
- **Note**:
- As pictured in the gif below, the `copy_paste` augmentation can be used to copy objects from one image to another.
- Once an object is copied, regardless of the `copy_paste_mode`, its Intersection over Area (IoA) is computed with all the object of the source image. If all the IoA are below `0.3` (30%), the object is pasted in the target image. If only one the IoA is above `0.3`, the object is not pasted in the target image.
- The IoA threshold cannot be changed with the current implementation and is set to `0.3` by default.
| **`copy_paste` off** | **`copy_paste` on with `copy_paste_mode=flip`** | Visualize the `copy_paste` process |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_copy_paste_off.avif" alt="Original image without augmentation" width="80%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_copy_paste_on.avif" alt="Copy-paste augmentation enabled" width="80%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_copy_paste_demo.avif" alt="Copy-paste augmentation animated demo" width="97%"/> |
### Copy-Paste Mode (`copy_paste_mode`)
- **Options**: `'flip'`, `'mixup'`
- **Default**: `'{{ copy_paste_mode }}'`
- **Usage**: Determines the method used for [copy-paste](#copy-paste-copy_paste) augmentation. If set to `'flip'`, the objects come from the same image, while `'mixup'` allows objects to be copied from different images.
- **Purpose**: Allows flexibility in how copied objects are integrated into target images.
- **Ultralytics' implementation**: [CopyPaste](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.CopyPaste)
- **Note**:
- The IoA principle is the same for both `copy_paste_mode`, but the way the objects are copied is different.
- Depending on the image size, objects may sometimes be copied partially or entirely outside the frame.
- Depending on the quality of polygon annotations, copied objects may have slight shape variations compared to the originals.
| **Reference image** | **Chosen image for `copy_paste`** | **`copy_paste` on with `copy_paste_mode=mixup`** |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_mixup_identity_2.avif" alt="Second image for MixUp blending" width="77%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_copy_paste_off.avif" alt="Original image without augmentation" width="80%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_copy_paste_mixup.avif" alt="Copy-paste with MixUp mode" width="77%"/> |
## Classification-Specific Augmentations
### Auto Augment (`auto_augment`)
- **Options**: `'randaugment'`, `'autoaugment'`, `'augmix'`, `None`
- **Default**: `'{{ auto_augment }}'`
- **Usage**: Applies automated augmentation policies for classification. The `'randaugment'` option uses RandAugment, `'autoaugment'` uses AutoAugment, and `'augmix'` uses AugMix. Setting to `None` disables automated augmentation.
- **Purpose**: Optimizes augmentation strategies automatically for classification tasks. The differences are the following:
- **AutoAugment**: This mode applies predefined augmentation policies learned from datasets like ImageNet, CIFAR10, and SVHN. Users can select these existing policies but cannot train new ones within Torchvision. To discover optimal augmentation strategies for specific datasets, external libraries or custom implementations would be necessary. Reference to the [AutoAugment paper](https://arxiv.org/abs/1805.09501).
- **RandAugment**: Applies a random selection of transformations with uniform magnitude. This approach reduces the need for an extensive search phase, making it more computationally efficient while still enhancing model robustness. Reference to the [RandAugment paper](https://arxiv.org/abs/1909.13719).
- **AugMix**: AugMix is a data augmentation method that enhances model robustness by creating diverse image variations through random combinations of simple transformations. Reference to the [AugMix paper](https://arxiv.org/abs/1912.02781).
- **Ultralytics' implementation**: [classify_augmentations()](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.classify_augmentations)
- **Note**:
- Essentially, the main difference between the three methods is the way the augmentation policies are defined and applied.
- You can refer to [this article](https://sebastianraschka.com/blog/2023/data-augmentation-pytorch.html) that compares the three methods in detail.
### Random Erasing (`erasing`)
- **Range**: `0.0` - `0.9`
- **Default**: `{{ erasing }}`
- **Usage**: Randomly erases portions of the image during classification training. The `erasing` hyperparameter defines the probability of applying the transformation, with `erasing=0.9` ensuring that almost all images are erased and `erasing=0.0` disabling the transformation. For example, with `erasing=0.5`, each image has a 50% chance of having a portion erased.
- **Purpose**: Helps models learn robust features and prevents over-reliance on specific image regions. For example, in facial recognition systems, random erasing helps models become more robust to partial occlusions like sunglasses, face masks, or other objects that might partially cover facial features. This improves real-world performance by forcing the model to identify individuals using multiple facial characteristics rather than depending solely on distinctive features that might be obscured.
- **Ultralytics' implementation**: [classify_augmentations()](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.classify_augmentations)
- **Note**:
- The `erasing` augmentation comes with a `scale`, `ratio`, and `value` hyperparameters that cannot be changed with the [current implementation](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/data/augment.py#L2502). Their default values are `(0.02, 0.33)`, `(0.3, 3.3)`, and `0`, respectively, as stated in the PyTorch [documentation](https://docs.pytorch.org/vision/main/generated/torchvision.transforms.RandomErasing.html).
- The upper limit of the `erasing` hyperparameter is set to `0.9` to avoid applying the transformation to all images.
| **`erasing` off** | **`erasing` on (example 1)** | **`erasing` on (example 2)** | **`erasing` on (example 3)** |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------: |
| <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_identity.avif" alt="Original image without augmentation" width="85%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_erasing_ex1.avif" alt="Random erasing example 1" width="85%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_erasing_ex2.avif" alt="Random erasing example 2" width="85%"/> | <img src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/augmentation_erasing_ex3.avif" alt="Random erasing example 3" width="85%"/> |
## Advanced Augmentation Features
### Custom Albumentations Transforms (`augmentations`)
- **Type**: `list` of Albumentations transforms
- **Default**: `None`
- **Usage**: Allows you to provide custom [Albumentations](https://albumentations.ai/) transforms for data augmentation using the Python API. This parameter accepts a list of Albumentations transform objects that will be applied during training instead of the default Albumentations transforms.
- **Purpose**: Provides fine-grained control over data augmentation strategies by leveraging the extensive library of Albumentations transforms. This is particularly useful when you need specialized augmentations beyond the built-in YOLO options, such as advanced color adjustments, noise injection, or domain-specific transformations.
- **Ultralytics' implementation**: [Albumentations](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.Albumentations)
!!! example "Custom Albumentations Example"
=== "Python API"
```python
import albumentations as A
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Define custom Albumentations transforms
custom_transforms = [
A.Blur(blur_limit=7, p=0.5),
A.GaussNoise(var_limit=(10.0, 50.0), p=0.3),
A.CLAHE(clip_limit=4.0, p=0.5),
A.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.5),
A.HueSaturationValue(hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=20, p=0.5),
]
# Train with custom Albumentations transforms
model.train(
data="coco8.yaml",
epochs=100,
augmentations=custom_transforms, # Pass custom transforms
imgsz=640,
)
```
=== "More Advanced Example"
```python
import albumentations as A
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n.pt")
# Define advanced custom Albumentations transforms with specific parameters
advanced_transforms = [
A.OneOf(
[
A.MotionBlur(blur_limit=7, p=1.0),
A.MedianBlur(blur_limit=7, p=1.0),
A.GaussianBlur(blur_limit=7, p=1.0),
],
p=0.3,
),
A.OneOf(
[
A.GaussNoise(var_limit=(10.0, 50.0), p=1.0),
A.ISONoise(color_shift=(0.01, 0.05), intensity=(0.1, 0.5), p=1.0),
],
p=0.2,
),
A.CLAHE(clip_limit=4.0, tile_grid_size=(8, 8), p=0.5),
A.RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3, brightness_by_max=True, p=0.5),
A.HueSaturationValue(hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=20, p=0.5),
A.CoarseDropout(
max_holes=8, max_height=32, max_width=32, min_holes=1, min_height=8, min_width=8, fill_value=0, p=0.2
),
]
# Train with advanced custom transforms
model.train(
data="coco8.yaml",
epochs=100,
augmentations=advanced_transforms,
imgsz=640,
)
```
**Key Points:**
- **Python API Only**: Custom Albumentations transforms are currently supported only through the Python API. They cannot be specified via CLI or YAML configuration files.
- **Replaces Default Transforms**: When you provide custom transforms via the `augmentations` parameter, they completely replace the default Albumentations transforms. **The default YOLO augmentations (like `mosaic`, `hsv_h`, `hsv_s`, `degrees`, etc.) remain active and are applied independently**.
- **Bounding Box Compatibility**: Be cautious when using spatial transforms (transforms that change the geometry of the image). Ultralytics handles bounding box adjustments automatically, but some complex transforms may require additional configuration.
- **Extensive Library**: Albumentations offers over 70+ different transforms. Explore the [Albumentations documentation](https://albumentations.ai/docs/) to discover all available options.
- **Performance Consideration**: Adding too many augmentations or using computationally expensive transforms can slow down training. Start with a small set and monitor training speed.
**Common Use Cases:**
- **Medical Imaging**: Apply specialized transforms like elastic deformations or grid distortions for X-ray or MRI image augmentation
- **Aerial/Satellite Imagery**: Use transforms optimized for overhead perspectives
- **Low-Light Conditions**: Apply noise and brightness adjustments to simulate challenging lighting
- **Industrial Inspection**: Add defect-like patterns or texture variations for quality control applications
**Compatibility Notes:**
- Requires Albumentations version 1.0.3 or higher
- Compatible with all YOLO detection and segmentation tasks
- Not applicable for classification tasks (classification uses a different augmentation pipeline)
For more information about Albumentations and available transforms, visit the [official Albumentations documentation](https://albumentations.ai/docs/).
## FAQ
### There are too many augmentations to choose from. How do I know which ones to use?
Choosing the right augmentations depends on your specific use case and dataset. Here are a few general guidelines to help you decide:
- In most cases, slight variations in color and brightness are beneficial. The default values for `hsv_h`, `hsv_s`, and `hsv_v` are a solid starting point.
- If the camera's point of view is consistent and won't change once the model is deployed, you can likely skip geometric transformations such as `rotation`, `translation`, `scale`, `shear`, or `perspective`. However, if the camera angle may vary, and you need the model to be more robust, it's better to keep these augmentations.
- Use the `mosaic` augmentation only if having partially occluded objects or multiple objects per image is acceptable and does not change the label value. Alternatively, you can keep `mosaic` active but increase the `close_mosaic` value to disable it earlier in the training process.
In short: keep it simple. Start with a small set of augmentations and gradually add more as needed. The goal is to improve the model's generalization and robustness, not to overcomplicate the training process. Also, make sure the augmentations you apply reflect the same data distribution your model will encounter in production.
### When starting a training, a see a `albumentations: Blur[...]` reference. Does that mean Ultralytics YOLO runs additional augmentation like blurring?
If the `albumentations` package is installed, Ultralytics automatically applies a set of extra image augmentations using it. These augmentations are handled internally and require no additional configuration.
You can find the full list of applied transformations in our [technical documentation](https://docs.ultralytics.com/reference/data/augment/#ultralytics.data.augment.Albumentations), as well as in our [Albumentations integration guide](https://docs.ultralytics.com/integrations/albumentations/). Note that only the augmentations with a probability `p` greater than `0` are active. These are purposefully applied at low frequencies to mimic real-world visual artifacts, such as blur or grayscale effects.
You can also provide your own custom Albumentations transforms using the Python API. See the [Advanced Augmentation Features](#advanced-augmentation-features) section for more details.
### When starting a training, I don't see any reference to albumentations. Why?
Check if the `albumentations` package is installed. If not, you can install it by running `pip install albumentations`. Once installed, the package should be automatically detected and used by Ultralytics.
### How do I customize my augmentations?
You can customize augmentations by creating a custom dataset class and trainer. For example, you can replace the default Ultralytics classification augmentations with PyTorch's [torchvision.transforms.Resize](https://docs.pytorch.org/vision/stable/generated/torchvision.transforms.Resize.html) or other transforms. See the [custom training example](../tasks/classify.md#train) in the classification documentation for implementation details.

209
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/yolo-performance-metrics.md Normal file
View File

@@ -0,0 +1,209 @@
---
comments: true
description: Explore essential YOLO26 performance metrics like mAP, IoU, F1 Score, Precision, and Recall. Learn how to calculate and interpret them for model evaluation.
keywords: YOLO26 performance metrics, mAP, IoU, F1 Score, Precision, Recall, object detection, Ultralytics
---
# Performance Metrics Deep Dive
## Introduction
Performance metrics are key tools to evaluate the [accuracy](https://www.ultralytics.com/glossary/accuracy) and efficiency of [object detection](https://www.ultralytics.com/glossary/object-detection) models. They shed light on how effectively a model can identify and localize objects within images. Additionally, they help in understanding the model's handling of false positives and false negatives. These insights are crucial for evaluating and enhancing the model's performance. In this guide, we will explore various performance metrics associated with YOLO26, their significance, and how to interpret them.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/q7LwPoM7tSQ"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> Ultralytics YOLO26 Performance Metrics | MAP, F1 Score, <a href="https://www.ultralytics.com/glossary/precision">Precision</a>, IoU & Accuracy
</p>
## Object Detection Metrics
Let's start by discussing some metrics that are not only important to YOLO26 but are broadly applicable across different object detection models.
- **[Intersection over Union](https://www.ultralytics.com/glossary/intersection-over-union-iou) (IoU):** IoU is a measure that quantifies the overlap between a predicted [bounding box](https://www.ultralytics.com/glossary/bounding-box) and a ground truth bounding box. It plays a fundamental role in evaluating the accuracy of object localization.
- **Average Precision (AP):** AP computes the area under the precision-recall curve, providing a single value that encapsulates the model's precision and recall performance.
- **[Mean Average Precision](https://www.ultralytics.com/glossary/mean-average-precision-map) (mAP):** mAP extends the concept of AP by calculating the average AP values across multiple object classes. This is useful in multi-class object detection scenarios to provide a comprehensive evaluation of the model's performance.
- **Precision and Recall:** Precision quantifies the proportion of true positives among all positive predictions, assessing the model's capability to avoid false positives. On the other hand, Recall calculates the proportion of true positives among all actual positives, measuring the model's ability to detect all instances of a class.
- **F1 Score:** The F1 Score is the harmonic mean of precision and recall, providing a balanced assessment of a model's performance while considering both false positives and false negatives.
## How to Calculate Metrics for YOLO26 Model
Now, we can explore [YOLO26's Validation mode](../modes/val.md) that can be used to compute the above discussed evaluation metrics.
Using the validation mode is simple. Once you have a trained model, you can invoke the model.val() function. This function will then process the validation dataset and return a variety of performance metrics. But what do these metrics mean? And how should you interpret them?
### Interpreting the Output
Let's break down the output of the model.val() function and understand each segment of the output.
#### Class-wise Metrics
One of the sections of the output is the class-wise breakdown of performance metrics. This granular information is useful when you are trying to understand how well the model is doing for each specific class, especially in datasets with a diverse range of object categories. For each class in the dataset the following is provided:
- **Class**: This denotes the name of the object class, such as "person", "car", or "dog".
- **Images**: This metric tells you the number of images in the validation set that contain the object class.
- **Instances**: This provides the count of how many times the class appears across all images in the validation set.
- **Box(P, R, mAP50, mAP50-95)**: This metric provides insights into the model's performance in detecting objects:
- **P (Precision)**: The accuracy of the detected objects, indicating how many detections were correct.
- **R (Recall)**: The ability of the model to identify all instances of objects in the images.
- **mAP50**: Mean average precision calculated at an intersection over union (IoU) threshold of 0.50. It's a measure of the model's accuracy considering only the "easy" detections.
- **mAP50-95**: The average of the mean average precision calculated at varying IoU thresholds, ranging from 0.50 to 0.95. It gives a comprehensive view of the model's performance across different levels of detection difficulty.
#### Speed Metrics
The speed of inference can be as critical as accuracy, especially in real-time object detection scenarios. This section breaks down the time taken for various stages of the validation process, from preprocessing to post-processing.
#### COCO Metrics Evaluation
For users validating on the COCO dataset, additional metrics are calculated using the COCO evaluation script. These metrics give insights into precision and recall at different IoU thresholds and for objects of different sizes.
#### Visual Outputs
The model.val() function, apart from producing numeric metrics, also yields visual outputs that can provide a more intuitive understanding of the model's performance. Here's a breakdown of the visual outputs you can expect:
- **F1 Score Curve (`F1_curve.png`)**: This curve represents the [F1 score](https://www.ultralytics.com/glossary/f1-score) across various thresholds. Interpreting this curve can offer insights into the model's balance between false positives and false negatives over different thresholds.
- **Precision-Recall Curve (`PR_curve.png`)**: An integral visualization for any classification problem, this curve showcases the trade-offs between precision and [recall](https://www.ultralytics.com/glossary/recall) at varied thresholds. It becomes especially significant when dealing with imbalanced classes.
- **Precision Curve (`P_curve.png`)**: A graphical representation of precision values at different thresholds. This curve helps in understanding how precision varies as the threshold changes.
- **Recall Curve (`R_curve.png`)**: Correspondingly, this graph illustrates how the recall values change across different thresholds.
- **[Confusion Matrix](https://www.ultralytics.com/glossary/confusion-matrix) (`confusion_matrix.png`)**: The confusion matrix provides a detailed view of the outcomes, showcasing the counts of true positives, true negatives, false positives, and false negatives for each class.
- **Normalized Confusion Matrix (`confusion_matrix_normalized.png`)**: This visualization is a normalized version of the confusion matrix. It represents the data in proportions rather than raw counts. This format makes it simpler to compare the performance across classes.
- **Validation Batch Labels (`val_batchX_labels.jpg`)**: These images depict the ground truth labels for distinct batches from the validation dataset. They provide a clear picture of what the objects are and their respective locations as per the dataset.
- **Validation Batch Predictions (`val_batchX_pred.jpg`)**: Contrasting the label images, these visuals display the predictions made by the YOLO26 model for the respective batches. By comparing these to the label images, you can easily assess how well the model detects and classifies objects visually.
#### Results Storage
For future reference, the results are saved to a directory, typically named runs/detect/val.
## Choosing the Right Metrics
Choosing the right metrics to evaluate often depends on the specific application.
- **mAP:** Suitable for a broad assessment of model performance.
- **IoU:** Essential when precise object location is crucial.
- **Precision:** Important when minimizing false detections is a priority.
- **Recall:** Vital when it's important to detect every instance of an object.
- **F1 Score:** Useful when a balance between precision and recall is needed.
For real-time applications, speed metrics like FPS (Frames Per Second) and latency are crucial to ensure timely results.
## Interpretation of Results
It's important to understand the metrics. Here's what some of the commonly observed lower scores might suggest:
- **Low mAP:** Indicates the model may need general refinements.
- **Low IoU:** The model might be struggling to pinpoint objects accurately. Different bounding box methods could help.
- **Low Precision:** The model may be detecting too many non-existent objects. Adjusting confidence thresholds might reduce this.
- **Low Recall:** The model could be missing real objects. Improving [feature extraction](https://www.ultralytics.com/glossary/feature-extraction) or using more data might help.
- **Imbalanced F1 Score:** There's a disparity between precision and recall.
- **Class-specific AP:** Low scores here can highlight classes the model struggles with.
## Case Studies
Real-world examples can help clarify how these metrics work in practice.
### Case 1
- **Situation:** mAP and F1 Score are suboptimal, but while Recall is good, Precision isn't.
- **Interpretation & Action:** There might be too many incorrect detections. Tightening confidence thresholds could reduce these, though it might also slightly decrease recall.
### Case 2
- **Situation:** mAP and Recall are acceptable, but IoU is lacking.
- **Interpretation & Action:** The model detects objects well but might not be localizing them precisely. Refining bounding box predictions might help.
### Case 3
- **Situation:** Some classes have a much lower AP than others, even with a decent overall mAP.
- **Interpretation & Action:** These classes might be more challenging for the model. Using more data for these classes or adjusting class weights during training could be beneficial.
## Connect and Collaborate
Tapping into a community of enthusiasts and experts can amplify your journey with YOLO26. Here are some avenues that can facilitate learning, troubleshooting, and networking.
### Engage with the Broader Community
- **GitHub Issues:** The YOLO26 repository on GitHub has an [Issues tab](https://github.com/ultralytics/ultralytics/issues) where you can ask questions, report bugs, and suggest new features. The community and maintainers are active here, and it's a great place to get help with specific problems.
- **Ultralytics Discord Server:** Ultralytics has a [Discord server](https://discord.com/invite/ultralytics) where you can interact with other users and the developers.
### Official Documentation and Resources:
- **Ultralytics YOLO26 Docs:** The [official documentation](../index.md) provides a comprehensive overview of YOLO26, along with guides on installation, usage, and troubleshooting.
Using these resources will not only guide you through any challenges but also keep you updated with the latest trends and best practices in the YOLO26 community.
## Conclusion
In this guide, we've taken a close look at the essential performance metrics for YOLO26. These metrics are key to understanding how well a model is performing and are vital for anyone aiming to fine-tune their models. They offer the necessary insights for improvements and to make sure the model works effectively in real-life situations.
Remember, the YOLO26 and Ultralytics community is an invaluable asset. Engaging with fellow developers and experts can open doors to insights and solutions not found in standard documentation. As you journey through object detection, keep the spirit of learning alive, experiment with new strategies, and share your findings. By doing so, you contribute to the community's collective wisdom and ensure its growth.
## FAQ
### What is the significance of Mean Average Precision (mAP) in evaluating YOLO26 model performance?
Mean Average Precision (mAP) is crucial for evaluating YOLO26 models as it provides a single metric encapsulating precision and recall across multiple classes. mAP@0.50 measures precision at an IoU threshold of 0.50, focusing on the model's ability to detect objects correctly. mAP@0.50:0.95 averages precision across a range of IoU thresholds, offering a comprehensive assessment of detection performance. High mAP scores indicate that the model effectively balances precision and recall, essential for applications like [autonomous driving](https://www.ultralytics.com/solutions/ai-in-automotive) and surveillance systems where both accurate detection and minimal false alarms are critical.
### How do I interpret the Intersection over Union (IoU) value for YOLO26 object detection?
Intersection over Union (IoU) measures the overlap between the predicted and ground truth bounding boxes. IoU values range from 0 to 1, where higher values indicate better localization accuracy. An IoU of 1.0 means perfect alignment. Typically, an IoU threshold of 0.50 is used to define true positives in metrics like mAP. Lower IoU values suggest that the model struggles with precise object localization, which can be improved by refining bounding box regression or increasing annotation accuracy in your [training dataset](https://docs.ultralytics.com/datasets/).
### Why is the F1 Score important for evaluating YOLO26 models in object detection?
The F1 Score is important for evaluating YOLO26 models because it provides a harmonic mean of precision and recall, balancing both false positives and false negatives. It is particularly valuable when dealing with imbalanced datasets or applications where either precision or recall alone is insufficient. A high F1 Score indicates that the model effectively detects objects while minimizing both missed detections and false alarms, making it suitable for critical applications like [security systems](https://docs.ultralytics.com/guides/security-alarm-system/) and [medical imaging](https://www.ultralytics.com/solutions/ai-in-healthcare).
### What are the key advantages of using Ultralytics YOLO26 for real-time object detection?
Ultralytics YOLO26 offers multiple advantages for real-time object detection:
- **Speed and Efficiency**: Optimized for high-speed inference, suitable for applications requiring low latency.
- **High Accuracy**: Advanced algorithm ensures high mAP and IoU scores, balancing precision and recall.
- **Flexibility**: Supports various tasks including object detection, segmentation, and classification.
- **Ease of Use**: User-friendly interfaces, extensive documentation, and seamless integration with tools like Ultralytics Platform ([Platform Quickstart](../platform/quickstart.md)).
This makes YOLO26 ideal for diverse applications from autonomous vehicles to [smart city solutions](https://docs.ultralytics.com/guides/queue-management/).
### How can validation metrics from YOLO26 help improve model performance?
Validation metrics from YOLO26 like precision, recall, mAP, and IoU help diagnose and improve model performance by providing insights into different aspects of detection:
- **Precision**: Helps identify and minimize false positives.
- **Recall**: Ensures all relevant objects are detected.
- **mAP**: Offers an overall performance snapshot, guiding general improvements.
- **IoU**: Helps fine-tune object localization accuracy.
By analyzing these metrics, specific weaknesses can be targeted, such as adjusting confidence thresholds to improve precision or gathering more diverse data to enhance recall. For detailed explanations of these metrics and how to interpret them, check [Object Detection Metrics](#object-detection-metrics) and consider implementing [hyperparameter tuning](https://docs.ultralytics.com/guides/hyperparameter-tuning/) to optimize your model.

225
algorithms/dms_yolo/code.embedded.bak/docs/en/guides/yolo-thread-safe-inference.md Normal file
View File

@@ -0,0 +1,225 @@
---
comments: true
description: Learn how to ensure thread-safe YOLO model inference in Python. Avoid race conditions and run your multi-threaded tasks reliably with best practices.
keywords: YOLO models, thread-safe, Python threading, model inference, concurrency, race conditions, multi-threaded, parallelism, Python GIL
---
# Thread-Safe Inference with YOLO Models
Running YOLO models in a multi-threaded environment requires careful consideration to ensure thread safety. Python's `threading` module allows you to run several threads concurrently, but when it comes to using YOLO models across these threads, there are important safety issues to be aware of. This page will guide you through creating thread-safe YOLO model inference.
<p align="center">
<br>
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/jMbvN6uCIos"
title="YouTube video player" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen>
</iframe>
<br>
<strong>Watch:</strong> How to Perform Thread Safe Inference with Ultralytics YOLO Models in Python | Multi-Threading 🚀
</p>
## Understanding Python Threading
Python threads are a form of parallelism that allow your program to run multiple operations at once. However, Python's Global Interpreter Lock (GIL) means that only one thread can execute Python bytecode at a time.
<p align="center">
<img width="800" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/single-vs-multi-thread-examples.avif" alt="Single-thread vs multi-thread inference">
</p>
While this sounds like a limitation, threads can still provide concurrency, especially for I/O-bound operations or when using operations that release the GIL, like those performed by YOLO's underlying C libraries.
## The Danger of Shared Model Instances
Instantiating a YOLO model outside your threads and sharing this instance across multiple threads can lead to [race conditions](https://www.ultralytics.com/glossary/algorithmic-bias), where the internal state of the model is inconsistently modified due to concurrent accesses. This is particularly problematic when the model or its components hold state that is not designed to be thread-safe.
### Non-Thread-Safe Example: Single Model Instance
When using threads in Python, it's important to recognize patterns that can lead to concurrency issues. Here is what you should avoid: sharing a single YOLO model instance across multiple threads.
```python
# Unsafe: Sharing a single model instance across threads
from threading import Thread
from ultralytics import YOLO
# Instantiate the model outside the thread
shared_model = YOLO("yolo26n.pt")
def predict(image_path):
"""Predicts objects in an image using a preloaded YOLO model, take path string to image as argument."""
results = shared_model.predict(image_path)
# Process results
# Starting threads that share the same model instance
Thread(target=predict, args=("image1.jpg",)).start()
Thread(target=predict, args=("image2.jpg",)).start()
```
In the example above, the `shared_model` is used by multiple threads, which can lead to unpredictable results because `predict` could be executed simultaneously by multiple threads.
### Non-Thread-Safe Example: Multiple Model Instances
Similarly, here is an unsafe pattern with multiple YOLO model instances:
```python
# Unsafe: Sharing multiple model instances across threads can still lead to issues
from threading import Thread
from ultralytics import YOLO
# Instantiate multiple models outside the thread
shared_model_1 = YOLO("yolo26n_1.pt")
shared_model_2 = YOLO("yolo26n_2.pt")
def predict(model, image_path):
"""Runs prediction on an image using a specified YOLO model, returning the results."""
results = model.predict(image_path)
# Process results
# Starting threads with individual model instances
Thread(target=predict, args=(shared_model_1, "image1.jpg")).start()
Thread(target=predict, args=(shared_model_2, "image2.jpg")).start()
```
Even though there are two separate model instances, the risk of concurrency issues still exists. If the internal implementation of `YOLO` is not thread-safe, using separate instances might not prevent race conditions, especially if these instances share any underlying resources or states that are not thread-local.
## Thread-Safe Inference
To perform thread-safe inference, you should instantiate a separate YOLO model within each thread. This ensures that each thread has its own isolated model instance, eliminating the risk of race conditions.
### Thread-Safe Example
Here's how to instantiate a YOLO model inside each thread for safe parallel inference:
```python
# Safe: Instantiating a single model inside each thread
from threading import Thread
from ultralytics import YOLO
def thread_safe_predict(image_path):
"""Predict on an image using a new YOLO model instance in a thread-safe manner; takes image path as input."""
local_model = YOLO("yolo26n.pt")
results = local_model.predict(image_path)
# Process results
# Starting threads that each have their own model instance
Thread(target=thread_safe_predict, args=("image1.jpg",)).start()
Thread(target=thread_safe_predict, args=("image2.jpg",)).start()
```
In this example, each thread creates its own `YOLO` instance. This prevents any thread from interfering with the model state of another, thus ensuring that each thread performs inference safely and without unexpected interactions with the other threads.
## Using ThreadingLocked Decorator
Ultralytics provides a `ThreadingLocked` decorator that can be used to ensure thread-safe execution of functions. This decorator uses a lock to ensure that only one thread at a time can execute the decorated function.
```python
from ultralytics import YOLO
from ultralytics.utils import ThreadingLocked
# Create a model instance
model = YOLO("yolo26n.pt")
# Decorate the predict method to make it thread-safe
@ThreadingLocked()
def thread_safe_predict(image_path):
"""Thread-safe prediction using a shared model instance."""
results = model.predict(image_path)
return results
# Now you can safely call this function from multiple threads
```
The `ThreadingLocked` decorator is particularly useful when you need to share a model instance across threads but want to ensure that only one thread can access it at a time. This approach can save memory compared to creating a new model instance for each thread, but it may reduce concurrency as threads will need to wait for the lock to be released.
## Conclusion
When using YOLO models with Python's `threading`, always instantiate your models within the thread that will use them to ensure thread safety. This practice avoids race conditions and makes sure that your inference tasks run reliably.
For more advanced scenarios and to further optimize your multi-threaded inference performance, consider using process-based parallelism with [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) or leveraging a task queue with dedicated worker processes.
## FAQ
### How can I avoid race conditions when using YOLO models in a multi-threaded Python environment?
To prevent race conditions when using Ultralytics YOLO models in a multi-threaded Python environment, instantiate a separate YOLO model within each thread. This ensures that each thread has its own isolated model instance, avoiding concurrent modification of the model state.
Example:
```python
from threading import Thread
from ultralytics import YOLO
def thread_safe_predict(image_path):
"""Predict on an image in a thread-safe manner."""
local_model = YOLO("yolo26n.pt")
results = local_model.predict(image_path)
# Process results
Thread(target=thread_safe_predict, args=("image1.jpg",)).start()
Thread(target=thread_safe_predict, args=("image2.jpg",)).start()
```
For more information on ensuring thread safety, visit the [Thread-Safe Inference with YOLO Models](#thread-safe-inference).
### What are the best practices for running multi-threaded YOLO model inference in Python?
To run multi-threaded YOLO model inference safely in Python, follow these best practices:
1. Instantiate YOLO models within each thread rather than sharing a single model instance across threads.
2. Use Python's `multiprocessing` module for parallel processing to avoid issues related to Global Interpreter Lock (GIL).
3. Release the GIL by using operations performed by YOLO's underlying C libraries.
4. Consider using the `ThreadingLocked` decorator for shared model instances when memory is a concern.
Example for thread-safe model instantiation:
```python
from threading import Thread
from ultralytics import YOLO
def thread_safe_predict(image_path):
"""Runs inference in a thread-safe manner with a new YOLO model instance."""
model = YOLO("yolo26n.pt")
results = model.predict(image_path)
# Process results
# Initiate multiple threads
Thread(target=thread_safe_predict, args=("image1.jpg",)).start()
Thread(target=thread_safe_predict, args=("image2.jpg",)).start()
```
For additional context, refer to the section on [Thread-Safe Inference](#thread-safe-inference).
### Why should each thread have its own YOLO model instance?
Each thread should have its own YOLO model instance to prevent race conditions. When a single model instance is shared among multiple threads, concurrent accesses can lead to unpredictable behavior and modifications of the model's internal state. By using separate instances, you ensure thread isolation, making your multi-threaded tasks reliable and safe.
For detailed guidance, check the [Non-Thread-Safe Example: Single Model Instance](#non-thread-safe-example-single-model-instance) and [Thread-Safe Example](#thread-safe-example) sections.
### How does Python's Global Interpreter Lock (GIL) affect YOLO model inference?
Python's Global Interpreter Lock (GIL) allows only one thread to execute Python bytecode at a time, which can limit the performance of CPU-bound multi-threading tasks. However, for I/O-bound operations or processes that use libraries releasing the GIL, like YOLO's underlying C libraries, you can still achieve concurrency. For enhanced performance, consider using process-based parallelism with Python's `multiprocessing` module.
For more about threading in Python, see the [Understanding Python Threading](#understanding-python-threading) section.
### Is it safer to use process-based parallelism instead of threading for YOLO model inference?
Yes, using Python's `multiprocessing` module is safer and often more efficient for running YOLO model inference in parallel. Process-based parallelism creates separate memory spaces, avoiding the Global Interpreter Lock (GIL) and reducing the risk of concurrency issues. Each process will operate independently with its own YOLO model instance.
For further details on process-based parallelism with YOLO models, refer to the page on [Thread-Safe Inference](#thread-safe-inference).