视觉特征-自监督-同源不同crop视图一致性【2025-08】：DINOv3【DINOv2+Gram损失=DINO损失+（iBOT损失+Gram损失）+KoLeo正则化损失】

Abstract 摘要

Self-supervised learning holds the promise of eliminating the need for manual data annotation, enabling models to scale effortlessly to massive datasets and larger architectures. By not being tailored to specific tasks or domains, this training paradigm has the potential to learn visual representations from diverse sources, ranging from natural to aerial images- using a single algorithm. This technical report introduces DINOv3, a major milestone toward realizing this vision by leveraging simple yet effective strategies. First, we leverage the benefit of scaling both dataset and model size by careful data preparation, design, and optimization. Second, we introduce a new method called Gram anchoring, which effectively addresses the known yet unsolved issue of dense feature maps degrading during long training schedules. Finally, we apply post-hoc strategies that further enhance our models’ flexibility with respect to resolution, model size, and alignment with text. As a result, we present a versatile vision foundation model that outperforms the specialized state of the art across a broad range of settings, without fine-tuning. DINOv3 produces high-quality dense features that achieve outstanding performance on various vision tasks, significantly surpassing previous self- and weakly-supervised foundation models. We also share the DINOv3 suite of vision models, designed to advance the state of the art on a wide spectrum of tasks and data by providing scalable solutions for diverse resource constraints and deployment scenarios.

自监督学习（Self-supervised Learning）有望消除对人工数据标注的需求，使模型能够轻松扩展到海量数据集和更大规模的架构。由于该训练范式并不针对特定任务或领域进行定制，因此它有潜力通过单一算法从多样化的数据源中学习视觉表征（Visual Representations），涵盖从自然图像到航空图像等不同类型的数据。

本技术报告介绍了 DINOv3，这是朝着实现这一愿景迈出的重要里程碑，其通过采用简单但高效的策略实现了显著进展。

• 首先，我们通过精心的数据准备（Data Preparation）、模型设计（Model Design）和优化（Optimization），充分利用数据集规模和模型规模扩展所带来的优势。
• 其次，我们提出了一种称为 Gram Anchoring（Gram 锚定） 的新方法，该方法有效解决了一个已知但尚未解决的问题：在长时间训练过程中，密集特征图（Dense Feature Maps）会出现性能退化。
• 最后，我们应用了一系列后处理策略（Post-hoc Strategies），进一步增强模型在分辨率（Resolution）、模型规模（Model Size）以及与文本对齐（Text Alignment）方面的灵活性。

因此，我们提出了一种通用视觉基础模型（Vision Foundation Model），无需进行微调（Fine-tuning），便能够在广泛场景下超越针对特定任务优化的先进模型（State of the Art）。

DINOv3 能够生成高质量的密集特征（Dense Features），并在多种视觉任务（Vision Tasks）中实现卓越性能，显著超过此前的自监督（Self-supervised）和弱监督（Weakly-supervised）基础模型（Foundation Models）。

我们还发布了 DINOv3 视觉模型套件（DINOv3 Suite of Vision Models），旨在通过提供可扩展的解决方案，推动广泛任务和数据场景下的技术发展。

这些模型能够适应不同的资源限制（Resource Constraints）和部署场景（Deployment Scenarios），进一步提升视觉领域的先进水平。

1 Introduction 1 引言

Foundation models have become a central building block in modern computer vision, enabling broad generalization across tasks and domains through a single, reusable model. Self-supervised learning (SSL) is a powerful approach for training such models, by learning directly from raw pixel data and leveraging the natural co-occurrences of patterns in images. Unlike weakly and fully supervised pretraining methods (Radford et al., 2021; Dehghani et al., 2023; Bolya et al., 2025) which require images paired with high-quality metadata, SSL unlocks training on massive, raw image collections. This is particularly effective for training large-scale visual encoders thanks to the availability of virtually unlimited training data. DINOv2 (Oquab et al., 2024) exemplifies these strengths, achieving impressive results in image understanding tasks (Wang et al., 2025) and enabling pre-training for complex domains such as histopathology (Chen et al., 2024). Models trained with SSL exhibit additional desirable properties: they are robust to input distribution shifts, provide strong global and local features, and generate rich embeddings that facilitate physical scene understanding. Since SSL models are not trained for any specific downstream task, they produce versatile and robust generalist features. For instance, DINOv2 models deliver strong performance across diverse tasks and domains without requiring task-specific finetuning, allowing a single frozen backbone to serve multiple purposes. Importantly, self-supervised learning is especially suitable to train on the vast amount of available observational data in domains like histopathology (Vorontsov et al., 2024), biology (Kim et al., 2025), medical imaging (Pérez-García et al., 2025), remote sensing (Cong et al., 2022; Tolan et al., 2024), astronomy (Parker et al., 2024), or high-energy particle physics (Dillon et al., 2022). These domain often lack metadata and have already been shown to benefit from foundation models like DINOv2. Finally, SSL, requiring no human intervention, is well-suited for lifelong learning amid the growing volume of web data.

基础模型（ Foundation Models ）已经成为现代计算机视觉（ Computer Vision ）中的核心基础组件。通过一个可重复使用的统一模型，基础模型能够在不同任务和领域之间实现广泛的泛化能力。

自监督学习（ Self-supervised Learning, SSL ）是训练此类模型的一种强大方法。它直接从原始像素数据中学习，并利用图像中各种模式自然共同出现的规律来获取有效的视觉表征。与弱监督预训练（ Weakly-supervised Pretraining ）和全监督预训练（ Fully-supervised Pretraining ）方法（Radford et al., 2021；Dehghani et al., 2023；Bolya et al., 2025）不同，后两者需要图像与高质量元数据（ Metadata ）配对，而 SSL 则能够直接利用海量的原始图像集合进行训练。

由于实际上可以获得近乎无限的训练数据，这种方式尤其适合训练大规模视觉编码器（ Large-scale Visual Encoders ）。

DINOv2（Oquab et al., 2024）充分体现了这些优势：它在图像理解任务（ Image Understanding Tasks ）中取得了令人瞩目的结果（Wang et al., 2025），同时还能支持组织病理学（ Histopathology ）等复杂领域的预训练（Chen et al., 2024）。

采用 SSL 训练的模型还具有一些额外的理想特性：它们能够有效应对输入分布偏移（ Input Distribution Shifts ），能够提供强大的全局特征（ Global Features ）和局部特征（ Local Features ），并生成信息丰富的嵌入表示（ Embeddings ），从而促进对物理场景的理解。

由于 SSL 模型并不是针对任何特定的下游任务（ Downstream Task ）进行训练，因此它们能够生成通用且鲁棒的泛化特征（ Generalist Features ）。例如，DINOv2 模型无需针对具体任务进行微调（ Task-specific Finetuning ），便能在多种不同任务和领域中取得出色表现，因此可以使用同一个冻结的主干网络（ Frozen Backbone ）来服务于多种用途。

尤为重要的是，自监督学习非常适合利用多个领域中海量的观测数据（ Observational Data ）进行训练，例如

• 组织病理学（ Histopathology ）（Vorontsov et al., 2024）、
• 生物学（ Biology ）（Kim et al., 2025）、
• 医学影像（ Medical Imaging ）（Pérez-García et al., 2025）、
• 遥感（ Remote Sensing ）（Cong et al., 2022；Tolan et al., 2024）、
• 天文学（ Astronomy ）（Parker et al., 2024），
• 以及高能粒子物理学（ High-energy Particle Physics ）（Dillon et al., 2022）。

这些领域通常缺乏元数据，而已有研究表明，它们能够从 DINOv2 等基础模型中显著受益。

最后，由于 SSL 不需要人工干预，因此非常适合在网络数据规模持续增长的背景下进行终身学习（ Lifelong Learning ）。

Figure 1: Figure 1: (a) Evolution of linear probing results on ImageNet1k (IN1k) over the years, comparing fully-(SL), weakly- (WSL) and self-supervised learning (SSL) methods. Despite coming into the picture later, SSL has quickly progressed and now reached the Imagenet accuracy plateau of recent years. On the other hand, we demonstrate that SSL offers the unique promise of high-quality dense features. With DINOv3, we markedly improve over weakly-supervised models on dense tasks, as shown by the relative performance of the best-in-class WSL models to DINOv3 (b). We also produce PCA maps of features obtained from high resolution images with DINOv3 trained on natural © and aerial images (d).

图 1：(a) 多年来在 ImageNet1k 数据集上，基于全监督学习、弱监督学习以及自监督学习方法所得到的线性探测结果的演变情况。尽管自监督学习出现的时间较晚，但其发展速度极快，现已达到近年来在 ImageNet 数据集上的最佳性能水平。另一方面，研究表明自监督学习能够生成质量极高的密集特征。如(b)所示，通过 DINOv3，其在处理密集型任务时的性能明显优于那些基于弱监督学习的模型。此外，我们还利用 DINOv3 处理了来自自然场景和航空影像的高分辨率图像，并生成了相应的 PCA 特征图，见©和(d)。

In practice, the promise of SSL, namely producing arbitrarily large and powerful models by leveraging large amounts of unconstrained data, remains challenging at scale. While model instabilities and collapse are mitigated by the heuristics proposed by Oquab et al. (2024), more problems emerge from scaling further. First, it is unclear how to collect useful data from unlabeled collections. Second, in usual training practice, employing cosine schedules implies knowing the optimization horizon a priori, which is difficult when training on large image corpora. Third, the performance of the features gradually decreases after early training, confirmed by visual inspection of the patch similarity maps. This phenomenon appears in longer training runs with models above ViT-Large size (300M parameters), reducing the usefulness of scaling DINOv2.

在实际应用中，自监督学习（ Self-supervised Learning, SSL ）所承诺的愿景——即利用海量、无约束的数据来训练任意规模且能力强大的模型——在大规模场景下仍然面临诸多挑战。

尽管 Oquab 等人（2024）提出的启发式方法（ Heuristics ）能够缓解模型不稳定（ Model Instabilities ）和模型坍塌（ Model Collapse ）问题，但随着规模进一步扩大，新的问题也随之出现。

• 首先，如何从无标签数据集合（ Unlabeled Collections ）中筛选和收集有用的数据，目前仍不明确。

• 其次，在常规训练实践中，采用余弦调度（ Cosine Schedules ）意味着必须预先知道优化周期（ Optimization Horizon ），而在大规模图像语料库（ Large Image Corpora ）上进行训练时，这一点往往很难做到。

• 第三，在训练早期阶段之后，模型特征（ Features ）的性能会逐渐下降，这一点也通过对图像块相似度图（ Patch Similarity Maps ）的可视化检查得到了证实。对于规模超过 ViT-Large、参数量达到 3 亿（300M）以上的模型，这种现象会在更长时间的训练过程中出现，从而降低继续扩展 DINOv2 规模所能带来的实际收益。

Addressing the problems above leads to this work, DINOv3, which advances SSL training at scale. We demonstrate that a single frozen SSL backbone can serve as a universal visual encoder that achieves stateof-the-art performance on challenging downstream tasks, outperforming supervised and metadata-reliant pre-training strategies. Our research is guided by the following objectives: (1) training a foundational model versatile across tasks and domains, (2) improving the shortcomings of existing SSL models on dense features, (3) disseminating a family of models that can be used off-the-shelf. We discuss the three aims in the following.

为了解决上述问题，我们开展了本项工作，即 DINOv3，旨在推动大规模自监督学习训练（ SSL Training at Scale ）的发展。

我们证明，单个冻结的 SSL 主干网络（ Frozen SSL Backbone ）可以作为通用视觉编码器（ Universal Visual Encoder ），并在具有挑战性的下游任务（ Downstream Tasks ）上达到当前最先进的性能（ State-of-the-art Performance ），同时超越监督式预训练（ Supervised Pre-training ）以及依赖元数据的预训练策略（ Metadata-reliant Pre-training Strategies ）。

我们的研究主要围绕以下三个目标展开：

1. 训练一个能够适用于不同任务和领域的基础模型（ Foundational Model ）；
2. 改进现有 SSL 模型在密集特征（ Dense Features ）方面存在的不足；
3. 发布一系列可以直接开箱即用（ Off-the-shelf ）的模型。

下文将分别讨论这三个目标。

Figure 2: Figure 2: Performance of the DINOv3 family of models, compared to other families of self- or weaklysupervised models, on different benchmarks. DINOv3 significantly surpasses others on dense benchmarks, including models that leverage mask annotation priors such as AM-RADIO (Heinrich et al., 2025).

图 2：在不同评估标准下，DINOv3 系列模型与其他自监督或弱监督模型系列的性能对比。在那些需要利用掩码标注信息的任务中，DINOv3 的表现明显优于其他模型，包括那些依赖 AM-RADIO（Heinrich 等人，2025 年）这类方法的模型。

Strong & Versatile Foundational Models DINOv3 aims to offer a high level of versatility along two axes, which is enabled by the scaling of the model size and training data. First, a key desirable property for SSL models is to achieve excellent performance while being kept frozen, ideally reaching similar stateof-the-art results as specialized models. In that case, a single forward pass can deliver cutting-edge results across multiple tasks, leading to substantial computational savings-an essential advantage for practical applications, particularly on edge devices. We show the wide breadth of tasks that DINOv3 can successfully be applied to in Sec. 6. Second, a scalable SSL training pipeline that does not depend on metadata unlocks numerous scientific applications. By pre-training on a diverse set of images, whether web images or observational data, SSL models generalize across a large set of domains and tasks. As illustrated in Fig. 1(d), the PCA of DINOv3 features extracted from a high-resolution aerial image clearly allows to separates roads, houses, and greenery, highlighting the model’s feature quality.

1.1 强大且通用的基础模型（ Strong & Versatile Foundational Models ）

DINOv3 旨在沿两个维度实现高度的通用性（ Versatility ），而这种能力主要得益于模型规模和训练数据规模的扩展。

• 首先，对于自监督学习模型（ SSL Models ）而言，一个非常重要且理想的特性是：在模型参数保持冻结（ Frozen ）的情况下，依然能够获得优异的性能，并且最好能够达到与专门针对特定任务设计的模型相近的当前最先进水平（ State-of-the-art, SOTA ）。在这种情况下，仅通过一次前向传播（ Forward Pass ），同一个模型就能够在多种任务上取得领先性能，从而显著节省计算资源。这对于实际应用而言是一个至关重要的优势，尤其是在边缘设备（ Edge Devices ）上。第 6 节展示了 DINOv3 能够成功应用于多种不同任务，体现出其广泛的适用范围。
• 其次，一个无需依赖元数据（ Metadata ）且具有可扩展性的 SSL 训练流水线（ Scalable SSL Training Pipeline ），能够为大量科学应用提供新的可能性。通过在多样化的图像集合上进行预训练——无论这些图像来自互联网，还是来自观测数据（ Observational Data ）——SSL 模型都能够在大量不同领域和任务之间实现良好的泛化。

如图 1(d) 所示，对从一幅高分辨率航空图像（ High-resolution Aerial Image ）中提取的 DINOv3 特征进行主成分分析（ Principal Component Analysis, PCA ）后，可以清晰地区分道路、房屋和植被区域，这充分体现了模型所学习特征的高质量。

Superior Feature Maps Through Gram Anchoring Another key feature of DINOv3 is a significant improvement of its dense feature maps. The DINOv3 SSL training strategy aims at producing models excelling at high-level semantic tasks while producing excellent feature maps amenable to solving geometric tasks such as depth estimation, or 3D matching. In particular, the models should produce dense features that can be used off-the-shelf or with little post-processing. The compromise between dense and global representation is especially difficult to optimize when training with vast amounts of images, since the objective of high-level understanding can conflict with the quality of the dense feature maps. These contradictory objectives lead to a collapse of dense features with large models and long training schedules. Our new Gram anchoring strategy effectively mitigates this collapse (see Sec. 4). As a result, DINOv3 obtains significantly better dense feature maps than DINOv2, staying clean even at high resolutions (see Fig. 3).

1.2 通过 Gram 锚定获得更优的特征图（ Superior Feature Maps Through Gram Anchoring ）

DINOv3 的另一个关键特性，是其密集特征图（ Dense Feature Maps ）质量得到了显著提升。

DINOv3 的 SSL 训练策略旨在使模型既能够在高层语义任务（ High-level Semantic Tasks ）中表现出色，又能够生成高质量的特征图，从而适用于深度估计（ Depth Estimation ）、三维匹配（ 3D Matching ）等几何任务（ Geometric Tasks ）。

特别是，这些模型应当能够生成可直接使用（ Off-the-shelf ）的密集特征，或者只需要进行少量后处理（ Post-processing ）即可使用。

然而，在使用海量图像进行训练时，对密集表征（ Dense Representation ）与全局表征（ Global Representation ）之间的权衡进行优化尤其困难。

这是因为，高层语义理解（ High-level Understanding ）的优化目标可能会与保持高质量密集特征图的目标产生冲突。

这两类相互矛盾的目标会导致大型模型在采用较长训练周期（ Long Training Schedules ）时出现密集特征坍塌（ Dense Feature Collapse ）现象。

我们提出的新型 Gram 锚定策略（ Gram Anchoring ）能够有效缓解这种坍塌问题，具体内容见第 4 节。

因此，与 DINOv2 相比，DINOv3 能够生成质量显著更高的密集特征图，并且即使在高分辨率输入下，特征图依然能够保持清晰、稳定，如图 3 所示。

The DINOv3 Family of Models Solving the degradation of dense feature map with Gram anchoring unlocks the power of scaling. As a consequence, training a much larger model with SSL leads to significant performance improvements. In this work, we successfully train a DINO model with 7B parameters. Since such a large model requires significant resources to run, we apply distillation to compress its knowledge into smaller variants. As a result, we present the DINOv3 family of vision models, a comprehensive suite designed to address a wide spectrum of computer vision challenges. This model family aims to advance the state of the art by offering scalable solutions adaptable to diverse resource constraints and deployment scenarios. The distillation process produces model variants at multiple scales, including Vision Transformer (ViT) Small, Base, and Large, as well as ConvNeXt-based architectures. Notably, the efficient and widely adopted ViT-L model achieves performance close to that of the original 7B teacher across a variety of tasks. Overall, the DINOv3 family demonstrates strong performance on a broad range of benchmarks, matching or exceeding the accuracy of competing

1.3 DINOv3 模型家族（ The DINOv3 Family of Models ）

通过 Gram 锚定解决密集特征图退化（ Dense Feature Map Degradation ）问题之后，进一步扩大模型规模所带来的潜力得以充分释放。

因此，使用自监督学习训练规模大得多的模型，可以带来显著的性能提升。在本项工作中，我们成功训练了一个拥有 70 亿参数（7B Parameters） 的 DINO 模型。

由于如此庞大的模型在实际运行时需要消耗大量计算资源，我们进一步采用知识蒸馏（ Knowledge Distillation ）技术，将大型模型中的知识压缩到规模更小的模型变体中。

最终，我们提出了 DINOv3 视觉模型家族（DINOv3 Family of Vision Models）。这是一套全面的视觉模型系列，旨在解决广泛的计算机视觉挑战。

该模型家族通过提供可扩展的解决方案，力求进一步推动当前最先进水平，同时能够适应不同的资源约束（ Resource Constraints ）和部署场景（ Deployment Scenarios ）。

蒸馏过程（ Distillation Process ）产生了多个不同规模的模型变体，其中包括视觉 Transformer（ Vision Transformer, ViT ）架构的 Small、Base 和 Large 版本，以及基于 ConvNeXt 的架构。

值得注意的是，高效且得到广泛应用的 ViT-L 模型，在多种任务上的性能已经接近原始的 70 亿参数教师模型（ 7B Teacher Model ）。

总体而言，DINOv3 模型家族在广泛的基准测试（ Benchmarks ）上均展现出强劲性能。在全局任务（ Global Tasks ）上，其准确率能够达到或超过其他竞争模型；而在密集预测任务（ Dense Prediction Tasks ）上，其性能则显著领先于这些模型，如图 2 所示。

Figure 3: Figure 3: High-resolution dense features. We visualize the cosine similarity maps obtained with DI-NOv3 output features between the patches marked with a red cross and all other patches. Input image at 4096×4096. Please zoom in, do you agree with DINOv3?

图 3：高分辨率的密集特征。我们展示了使用 DI-NOv3 生成的特征所得到的余弦相似度图，这些特征是针对那些标有红色叉号的区域与其他所有区域之间的相似度。输入图像的尺寸为 4096×4096。请放大查看，您认为 DINOv3 的结果可靠吗？

Overview of Contributions In this work, we introduce multiple contributions to address the challenge of scaling SSL towards a large frontier model. We build upon recent advances in automatic data curation (Vo et al., 2024) to obtain a large “background” training dataset that we carefully mix with a bit of specialized data (ImageNet-1k). This allows leveraging large amounts of unconstrained data to improve the model performance. This contribution (i) around data scaling will be

1.4 贡献概述（ Overview of Contributions ）

在本项工作中，我们提出了多项贡献，以解决将自监督学习（ Self-supervised Learning, SSL ）扩展到大规模前沿模型（ Large Frontier Model ）时所面临的挑战。

我们基于近期在自动数据筛选与整理（ Automatic Data Curation ）方面的进展（Vo et al., 2024），构建了一个大规模的“背景”训练数据集（ Background Training Dataset ），并经过精心设计，将其与少量专门数据（即 ImageNet-1k）进行混合。这使我们能够充分利用海量、无约束的数据来提升模型性能。围绕数据规模扩展（ Data Scaling ）的这一贡献（i）将在第 3.1 节中详细介绍。

We increase our main model size to 7B parameters by defining a custom variant of the ViT architecture. We include modern position embeddings (axial RoPE) and develop a regularization technique to avoid positional artifacts. Departing from the multiple cosine schedules in DINOv2, we train with constant hyperparameter schedules for 1M iterations. This allows producing models with stronger performance. This contribution (ii) on model architecture and training will be described in Sec. 3.2.

我们通过定义一种定制的 ViT 架构变体，将主要模型规模扩展至 70 亿参数（7B Parameters）。

我们引入了现代化的位置嵌入（ Position Embeddings ），即轴向旋转位置编码（ Axial Rotary Position Embedding, Axial RoPE ），同时开发了一种正则化技术（ Regularization Technique ），用于避免位置伪影（ Positional Artifacts ）。

与 DINOv2 中采用多个余弦调度（ Cosine Schedules ）不同，我们在 100 万次迭代（ 1M Iterations ）的整个训练过程中采用恒定的超参数调度（ Constant Hyperparameter Schedules ）。这种方法使我们能够获得性能更强的模型。围绕模型架构与训练（ Model Architecture and Training ）的这一贡献（ii）将在第 3.2 节中介绍。

With the above techniques, we are able to train a model following the DINOv2 algorithm at scale. However, as mentioned previously, scale leads to a degradation of dense features. To address this, we propose a core improvement of the pipeline with a Gram anchoring training phase. This cleans the noise in the feature maps, leading to impressive similarity maps, and drastically improving the performance on both parametric and non-parametric dense tasks. This contribution (iii) on Gram training will be described in Sec. 4.

借助上述技术，我们能够按照 DINOv2 算法进行大规模模型训练。

然而，正如前文所述，模型规模的扩大也会导致密集特征（ Dense Features ）质量下降。

为了解决这一问题，我们对训练流水线进行了核心改进，引入了 Gram 锚定训练阶段（ Gram Anchoring Training Phase ）。

该方法能够清除特征图（ Feature Maps ）中的噪声，从而得到效果非常出色的相似度图（ Similarity Maps ），并显著提升模型在参数化密集任务（ Parametric Dense Tasks ）和非参数化密集任务（ Non-parametric Dense Tasks ）上的性能。围绕 Gram 训练（ Gram Training ）的这一贡献（iii）将在第 4 节中详细介绍。

Following previous practice, the last steps of our pipeline consist of a high-resolution post-training phase and distillation into a series of high-performance models of various sizes. For the latter, we develop a novel and efficient single-teacher multiple-students distillation procedure. This contribution (iv) transfers the power of our 7B frontier model to a family of smaller practical models for common usage, that we describe in Sec. 5.2.

遵循此前的实践，我们训练流水线的最后几个步骤包括一个高分辨率后训练阶段（ High-resolution Post-training Phase ），以及将大模型蒸馏（ Distillation ）为一系列不同规模的高性能模型。

针对后者，我们开发了一种新颖且高效的单教师多学生蒸馏方法（ Single-teacher Multiple-students Distillation Procedure ）。

这一贡献（iv）将我们 70 亿参数前沿模型的能力迁移到一系列规模更小、更加实用、适合日常使用的模型中，具体内容将在第 5.2 节中介绍。

As measured in our thorough benchmarking, results in Sec. 6 show that our approach defines a new standard in dense tasks and performs comparably to CLIP derivatives on global tasks. In particular, with a frozen vision backbone, we achieve stateof-the-art performance on longstanding computer vision problems such as object detection (COCO detection, mAP 66.1) and image segmentation (ADE20k, mIoU 63.0), outperforming specialized fine-tuned pipelines. Moreover, we provide evidence of the generality of our approach across domains by applying the DINOv3 algorithm to satellite imagery, in Sec. 8, surpassing all prior approaches.

通过我们全面而系统的基准测试（ Benchmarking ），第 6 节的实验结果表明，我们的方法

• 在密集任务（ Dense Tasks ）上确立了新的性能标准，
• 同时在全局任务（ Global Tasks ）上的表现能够与 CLIP 衍生模型（ CLIP Derivatives ）相媲美。

特别是在保持视觉主干网络冻结（ Frozen Vision Backbone ）的情况下，我们在多个长期存在的计算机视觉问题上取得了当前最先进的性能（ State-of-the-art Performance ）。例如，

• 在目标检测（ Object Detection ）任务中，COCO 检测基准上的平均精度均值（ Mean Average Precision, mAP ）达到 66.1；
• 在图像分割（ Image Segmentation ）任务中，ADE20K 数据集上的平均交并比（ Mean Intersection over Union, mIoU ）达到 63.0，其性能超过了针对具体任务专门微调的流水线（ Specialized Fine-tuned Pipelines ）。

此外，为了证明我们方法在不同领域中的通用性（ Generality Across Domains ），我们在第 8 节中将 DINOv3 算法应用于卫星影像（ Satellite Imagery ），其性能超过了此前所有相关方法。

2 Related Work

2 相关工作

Self-Supervised Learning Learning without annotations requires an artificial learning task that provides supervision in lieu for training. The art and challenge of SSL lies in carefully designing these so-called pre-text tasks in order to learn powerful representations for downstream tasks. The language domain, by its discrete nature, offers straightforward ways to set up such tasks, which led to many successful unsupervised pre-training approaches for text data. Examples include word embeddings (Mikolov et al., 2013; Bojanowski et al., 2017), sentence representations (Devlin et al., 2018; Liu et al., 2019), and plain language models (Mikolov et al., 2010; Zaremba et al., 2014). In contrast, computer vision presents greater challenges due to the continuous nature of the signal. Early attempts mimicking language approaches extracted supervisory signals from parts of an image to predict other parts, e.g. by predicting relative patch position (Doersch et al., 2015), patch re-ordering (Noroozi and Favaro, 2016; Misra and Maaten, 2020), or inpainting (Pathak et al., 2016). Other tasks involve re-colorizing images (Zhang et al., 2016) or predicting image transformations (Gidaris et al., 2018).

2.1 自监督学习（ Self-Supervised Learning, SSL ）

在无需标注数据的情况下进行学习，需要设计一种人工构造的学习任务，为训练过程提供替代性的监督信号。自监督学习（Self-Supervised Learning, SSL） 的核心艺术与挑战在于，如何精心设计这些所谓的 前置任务（Pre-text Tasks），从而学习能够有效支持下游任务（ Downstream Tasks ）的强大表示。

由于语言领域具有离散性特点，因此可以较为直接地构造这类任务，这推动了许多针对文本数据的成功无监督预训练方法（ Unsupervised Pre-training Approaches ）。

相关方法包括词嵌入（ Word Embeddings ）（Mikolov et al., 2013; Bojanowski et al., 2017）、句子表示（ Sentence Representations ）（Devlin et al., 2018; Liu et al., 2019）以及纯语言模型（ Plain Language Models ）（Mikolov et al., 2010; Zaremba et al., 2014）。

相比之下，由于计算机视觉信号具有连续性（ Continuous Nature of the Signal ），视觉领域面临更大的挑战。

早期模仿语言领域方法的尝试，通常从图像的局部区域中提取监督信号，并利用这些信号预测图像的其他部分。例如，

• 通过预测图像块之间的相对位置（ Relative Patch Position ）（Doersch et al., 2015）、
• 图像块重新排序（ Patch Re-ordering ）（Noroozi and Favaro, 2016; Misra and Maaten, 2020），
• 或者图像修复（ Inpainting ）（Pathak et al., 2016）来构造训练任务。

其他任务则包括图像重新着色（ Image Re-colorization ）（Zhang et al., 2016）以及预测图像变换（ Image Transformation Prediction ）（Gidaris et al., 2018）。

Among these tasks, inpainting-based approaches have gathered significant interest thanks to the flexibility of the patchbased ViT architecture (He et al., 2021; Bao et al., 2021; El-Nouby et al., 2021). The objective is to reconstruct corrupted regions of an image, which can be viewed as a form of denoising auto-encoding and is conceptually related to the masked token prediction task in BERT pretraining (Devlin et al., 2018). Notably, He et al. (2021) demonstrated that pixel-based masked auto-encoders (MAE) can be used as strong initializations for finetuning on downstream tasks. In the following, Baevski et al. (2022; 2023); Assran et al. (2023) showed that predicting a learned latent space instead of the pixel space leads to more powerful, higher-level features-a learning paradigm called JEPA: “Joint-Embedding Predictive Architecture” (LeCun, 2022). Recently, JEPAs have also been extended to video training (Bardes et al., 2024; Assran et al., 2025).

在这些任务中，基于图像修复的方法（ Inpainting-based Approaches ）受到了广泛关注，这主要得益于基于图像块的视觉 Transformer 架构（ Patch-based ViT Architecture ）所提供的灵活性（He et al., 2021; Bao et al., 2021; El-Nouby et al., 2021）。

其目标是重建图像中被破坏的区域，这可以被视为一种去噪自编码（ Denoising Auto-encoding ）形式，并且在概念上类似于 BERT 预训练（Devlin et al., 2018）中的掩码词元预测任务（ Masked Token Prediction Task ）。

值得注意的是，He et al.（2021）证明，基于像素的掩码自编码器（ Pixel-based Masked Auto-Encoder, MAE ）可以作为下游任务微调（ Fine-tuning ）时的强大初始化模型。

随后，Baevski et al.（2022; 2023）以及 Assran et al.（2023）进一步表明，与预测像素空间（ Pixel Space ）相比，预测学习得到的潜在空间（ Learned Latent Space ）能够产生更强、更高层次的特征。这种学习范式被称为 JEPA（Joint-Embedding Predictive Architecture，联合嵌入预测架构）（LeCun, 2022）。

近年来，JEPA 方法也被扩展到视频训练领域（ Video Training ）（Bardes et al., 2024; Assran et al., 2025）。

A second line of work, closer to ours, leverages discriminative signals between images to learn visual representations. This family of methods traces its origins to early deep learning research (Hadsell et al., 2006), but gained popularity with the introduction of instance classification techniques (Dosovitskiy et al., 2016; Bojanowski and Joulin, 2017; Wu et al., 2018). Subsequent advancements introduced contrastive objectives and information-theoretic criteria (Hénaff et al., 2019; He et al., 2020; Chen and He, 2020; Chen et al., 2020a; Grill et al., 2020; Bardes et al., 2021), as well as self clustering-based strategies (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; 2021). More recent approaches, such as iBOT (Zhou et al., 2021), combine these discriminative losses with masked reconstruction objectives. All of these methods show the ability to learn strong features and achieve high performance on standard benchmarks like ImageNet (Russakovsky et al., 2015). However, most face challenges scaling to larger model sizes (Chen et al., 2021).

另一条与本文更加相关的研究路线，是利用图像之间的判别信号（ Discriminative Signals Between Images ）来学习视觉表示（ Visual Representations ）。

这一类方法可以追溯到早期深度学习研究（Hadsell et al., 2006），但随着实例分类技术（ Instance Classification Techniques ）的提出而逐渐流行起来（Dosovitskiy et al., 2016; Bojanowski and Joulin, 2017; Wu et al., 2018）。

后续研究进一步引入了对比学习目标（ Contrastive Objectives ）和信息论准则（ Information-theoretic Criteria ）（Hénaff et al., 2019; He et al., 2020; Chen and He, 2020; Chen et al., 2020a; Grill et al., 2020; Bardes et al., 2021），以及基于自聚类的策略（ Self-clustering-based Strategies ）（Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; 2021）。

近期的一些方法，例如 iBOT（Zhou et al., 2021），进一步将这些判别式损失（ Discriminative Losses ）与掩码重建目标（ Masked Reconstruction Objectives ）结合起来。

所有这些方法都展示了学习强大特征表示（ Strong Feature Representations ）的能力，并且能够在 ImageNet（Russakovsky et al., 2015）等标准基准数据集上取得较高性能。

然而，大多数方法在扩展到更大模型规模（ Larger Model Sizes ）时仍然面临挑战（Chen et al., 2021）。

Vision Foundation Models The deep learning revolution began with the AlexNet breakthrough (Krizhevsky et al., 2012), a deep convolutional neural network that outperformed all previous methods on the ImageNet challenge (Deng et al., 2009; Russakovsky et al., 2015). Already early on, features learned end-to-end on the large manually-labeled ImageNet dataset were found to be highly effective for a wide range of transfer learning tasks (Oquab et al., 2014). Early work on vision foundation models then focused on architecture development, including VGG (Simonyan and Zisserman, 2015), GoogleNet (Szegedy et al., 2015), and ResNets (He et al., 2016).

2.2 视觉基础模型（ Vision Foundation Models ）

深度学习革命始于 AlexNet 的突破（Krizhevsky et al., 2012）。AlexNet 是一种深度卷积神经网络（ Deep Convolutional Neural Network, CNN ），在 ImageNet 挑战赛（Deng et al., 2009; Russakovsky et al., 2015）中的表现超过了此前所有方法。

早在这一时期，人们就发现，在大规模人工标注的 ImageNet 数据集上通过端到端学习（ End-to-end Learning ）获得的特征，对于广泛的迁移学习任务（ Transfer Learning Tasks ）都非常有效（Oquab et al., 2014）。

随后，早期关于视觉基础模型（ Vision Foundation Models ）的研究主要聚焦于模型架构的发展，包括 VGG（Simonyan and Zisserman, 2015）、GoogleNet（Szegedy et al., 2015）以及 ResNet（He et al., 2016）。

Given the effectiveness of scaling, subsequent works explored training larger models on big datasets. Sun et al. (2017) expanded supervised training data with the proprietary JFT dataset containing 300 million labeled images, showing impressive results. JFT also enabled significant performance gains for Kolesnikov et al. (2020). In parallel, scaling was explored using a combination of supervised and unsupervised data. For instance, an ImageNetsupervised model can be used to produce pseudo-labels for unsupervised data, which then serve to train larger networks (Yalniz et al., 2019). Subsequently, the availability of large supervised datasets such as JFT also facilitated the adaptation of the transformer architecture to computer vision (Dosovitskiy et al., 2020). In particular, achieving performance comparable to that of the original vision transformer (ViT) without access to JFT requires substantial effort (Touvron et al., 2020; 2022). Due to the learning capacity of ViTs, scaling efforts were further extended by Zhai et al. (2022a), culminating in the very large ViT-22B encoder (Dehghani et al., 2023).

鉴于扩大规模（ Scaling ）所展现出的有效性，后续研究开始探索利用大型数据集训练更大规模的模型。Sun et al.（2017）利用包含 3 亿张有标签图像的专有 JFT 数据集扩展了监督训练数据（ Supervised Training Data ），并取得了令人瞩目的结果。JFT 数据集同样帮助 Kolesnikov et al.（2020）实现了显著的性能提升。

与此同时，研究人员还探索了结合监督数据（ Supervised Data ）和无监督数据（ Unsupervised Data ）进行规模扩展的方法。例如，可以利用一个在 ImageNet 上进行监督训练的模型，为无监督数据生成伪标签（ Pseudo-labels ），随后使用这些伪标签来训练规模更大的网络（Yalniz et al., 2019）。

之后，JFT 等大型监督数据集的出现，也推动了 Transformer 架构向计算机视觉领域的迁移（Dosovitskiy et al., 2020）。特别是，如果无法使用 JFT 数据集，那么想要达到原始视觉 Transformer（ Vision Transformer, ViT ）相当的性能，需要付出大量额外努力（Touvron et al., 2020; 2022）。

由于 ViT 具有强大的学习能力，Zhai et al.（2022a）进一步推动了模型规模扩展，并最终发展出了规模极大的 ViT-22B 编码器（ViT-22B Encoder）（Dehghani et al., 2023）。

Given the complexity of manually labeling large datasets, weakly-supervised training-where annotations are derived from metadata associated with images-provides an effective alternative to supervised training. Early on, Joulin et al. (2016) demonstrated that a network can be pre-trained by simply predicting all words in the image caption as targets. This initial approach was further refined by leveraging sentence structures (Li et al., 2017), incorporating other types of metadata and involve curation (Mahajan et al., 2018), and scaling (Singh et al., 2022). However, weakly-supervised algorithms only reached their full potential with the introduction of contrastive losses and the joint-training of caption representations, as exemplified by Align (Jia et al., 2021) and CLIP (Radford et al., 2021).

由于人工标注大规模数据集十分复杂，弱监督训练（ Weakly-supervised Training ）成为监督训练的一种有效替代方案。在这种训练方式中，标注信息是从与图像相关联的元数据（ Metadata ）中提取的。

早期，Joulin et al.（2016）证明，只需让网络预测图像标题（ Image Caption ）中的所有词语作为训练目标，就能够对网络进行预训练。此后，这一初始方法得到进一步改进，包括利用句子结构（ Sentence Structures ）（Li et al., 2017）、引入其他类型的元数据并进行数据筛选整理（ Data Curation ）（Mahajan et al., 2018），以及进一步扩大训练规模（Singh et al., 2022）。

然而，直到引入对比损失（ Contrastive Losses ）以及对图像标题表示（ Caption Representations ）进行联合训练之后，弱监督算法才真正发挥出其全部潜力。代表性方法包括 ALIGN（Jia et al., 2021）和 CLIP（Radford et al., 2021）。

This highly successful approach inspired numerous open-source reproductions and scaling efforts. Open-CLIP (Cherti et al., 2023) was the first open-source effort to replicate CLIP by training on the LAION dataset (Schuhmann et al., 2021); following works leverage pre-trained backbones by fine-tuning them in a CLIP-style manner (Sun et al., 2023; 2024). Recognizing that data collection is a critical factor in the success of CLIP training, MetaCLIP (Xu et al., 2024) precisely follows the original CLIP procedure to reproduce its results, whereas Fang et al. (2024a) use supervised datasets to curate pretraining data. Other works focus on improving the training loss, e.g. using a sigmoid loss in SigLIP (Zhai et al., 2023), or leveraging a pre-trained image encoder (Zhai et al., 2022b). Ultimately though, the most critical components for obtaining cutting-edge foundation models are abundant high-quality data and substantial compute resources. In this vein, SigLIP 2 (Tschannen et al., 2025) and Perception Encoder (PE) (Bolya et al., 2025) achieve impressive results after training on more than 40B image-text pairs. The largest PE model is trained on 86 B billion samples with a global batch size of 131 K . Finally, a range of more complex and natively multimodal approaches have been proposed; these include contrastive captioning (Yu et al., 2022), masked modeling in the latent space (Bao et al., 2021; Wang et al., 2022b; Fang et al., 2023;

这种取得巨大成功的方法进一步推动了大量开源复现和规模扩展工作。OpenCLIP（Cherti et al., 2023）是首个通过在 LAION 数据集（Schuhmann et al., 2021）上进行训练来复现 CLIP 的开源项目；后续研究则利用预训练主干网络（ Pre-trained Backbones ），并采用类似 CLIP 的方式对其进行微调（Sun et al., 2023; 2024）。

考虑到数据收集（ Data Collection ）是 CLIP 训练成功的关键因素之一，MetaCLIP（Xu et al., 2024）严格遵循原始 CLIP 的训练流程，以复现其结果；而 Fang et al.（2024a）则利用监督数据集对预训练数据（ Pretraining Data ）进行筛选和整理。

其他研究则重点改进训练损失函数（ Training Loss ）。例如，SigLIP（Zhai et al., 2023）采用 Sigmoid 损失（ Sigmoid Loss ），还有一些工作利用预训练图像编码器（ Pre-trained Image Encoder ）（Zhai et al., 2022b）。

归根结底，要获得最先进的基础模型（ Cutting-edge Foundation Models ），最关键的因素仍然是海量的高质量数据（ Abundant High-quality Data ）以及充足的计算资源（ Substantial Compute Resources ）。

沿着这一方向，SigLIP 2（Tschannen et al., 2025）和感知编码器（ Perception Encoder, PE ）（Bolya et al., 2025）在超过 400 亿对图像-文本数据（40B Image-text Pairs） 上进行训练后取得了令人瞩目的结果。其中，最大的 PE 模型使用 860 亿个样本（86B Samples） 进行训练，全局批大小（ Global Batch Size ）达到 13.1 万（131K）。

此外，还出现了一系列更加复杂、原生支持多模态（ Natively Multimodal ）的方法，包括

• 对比式图像描述（ Contrastive Captioning ）（Yu et al., 2022）、
• 潜在空间中的掩码建模（ Masked Modeling in the Latent Space ）（Bao et al., 2021; Wang et al., 2022b; Fang et al., 2023; Wang et al., 2023a），
• 自回归训练（ Auto-regressive Training ）（Fini et al., 2024）。

In contrast, relatively little work has focused on scaling unsupervised image pretraining. Early efforts include Caron et al. (2019) and Goyal et al. (2019) utilizing the YFCC dataset (Thomee et al., 2016). Further progress has been achieved by focusing on larger datasets and models (Goyal et al., 2021; 2022a), as well as initial attempts at data curation for SSL (Tian et al., 2021). Careful tuning of the training algorithms, larger architectures, and more extensive training data lead to the impressive results of DINOv2 (Oquab et al., 2024); for the first time, an SSL model matched or surpassed open-source CLIP variants on a range of tasks. This direction has recently been further pushed by Fan et al. (2025) by scaling to large models without data curation, or by Venkataramanan et al. (2025) using open datasets and improved training recipes.

相比之下，针对无监督图像预训练（ Unsupervised Image Pretraining ）规模扩展的研究相对较少。早期工作包括 Caron et al.（2019）和 Goyal et al.（2019），它们使用了 YFCC 数据集（Thomee et al., 2016）。

此后，通过使用更大的数据集和模型（Goyal et al., 2021; 2022a），以及尝试针对自监督学习进行数据筛选整理（ Data Curation for SSL ）（Tian et al., 2021），这一方向取得了进一步进展。

通过对训练算法进行细致调优、采用更大规模的模型架构以及使用更加丰富的训练数据，DINOv2（Oquab et al., 2024）取得了令人瞩目的结果。它首次证明，自监督学习模型（ SSL Model ）能够在一系列任务上达到甚至超过开源 CLIP 变体（ Open-source CLIP Variants ）的性能。

最近，这一研究方向又得到了进一步推进。例如，Fan et al.（2025）在不进行数据筛选整理的情况下，将模型扩展到了更大的规模；Venkataramanan et al.（2025）则通过使用开放数据集（ Open Datasets ）和改进后的训练方案（ Improved Training Recipes ）进一步提升了性能。

Dense Transformer Features A broad range of modern vision applications consume dense features of pre-trained transformers, including multi-modal models (Liu et al., 2023; Beyer et al., 2024), generative models (Yu et al., 2025; Yao et al., 2025), 3D understanding (Wang et al., 2025), video understanding (Lin et al., 2023a; Wang et al., 2024b), and robotics (Driess et al., 2023; Kim et al., 2024). On top of that, traditional vision tasks such as detection, segmentation, or depth estimation require accurate local descriptors. To enhance the quality of SSL-trained local descriptors, a substantial body of work focuses on developing local SSL losses. Examples include leveraging spatio-temporal consistency in videos, e.g. using point track loops as training signal (Jabri et al., 2020), exploiting the spatial alignment between different crops of the same image (Pinheiro et al., 2020; Bardes et al., 2022), or enforcing consistency between neighboring patches (Yun et al., 2022). Darcet et al. (2025) show that predicting clustered local patches leads to improved dense representations. DetCon (Hénaff et al., 2021) and ORL (Xie et al., 2021) perform contrastive learning on region proposals but assume that such proposals exist a priori; this assumption is relaxed by approaches such as ODIN (Hénaff et al., 2022) and SlotCon (Wen et al., 2022). Without changing the training objective, Darcet et al. (2024) show that adding register tokens to the input sequence greatly improves dense feature maps, and recent works find this can be done without model training (Jiang et al., 2025; Chen et al., 2025).

2.3 密集 Transformer 特征（ Dense Transformer Features ）

现代视觉应用中，有相当广泛的一类任务会直接使用预训练 Transformer 的密集特征（ Dense Features ），包括

• 多模态模型（ Multi-modal Models ）（Liu et al., 2023; Beyer et al., 2024）、
• 生成模型（ Generative Models ）（Yu et al., 2025; Yao et al., 2025）、
• 三维理解（ 3D Understanding ）（Wang et al., 2025）、
• 视频理解（ Video Understanding ）（Lin et al., 2023a; Wang et al., 2024b）
• 机器人学（ Robotics ）（Driess et al., 2023; Kim et al., 2024）。

除此之外，目标检测、图像分割以及深度估计等传统视觉任务，也需要准确的局部描述符（ Local Descriptors ）。

为了提升通过自监督学习训练得到的局部描述符质量，大量研究致力于设计局部自监督学习损失（ Local SSL Losses ）。例如，

• 可以利用视频中的时空一致性（ Spatio-temporal Consistency ），将点轨迹循环（ Point Track Loops ）作为训练信号（Jabri et al., 2020）；
• 也可以利用同一图像不同裁剪区域之间的空间对齐关系（ Spatial Alignment ）（Pinheiro et al., 2020; Bardes et al., 2022）；
• 还可以强制相邻图像块之间保持一致性（ Consistency Between Neighboring Patches ）（Yun et al., 2022）。

Darcet et al.（2025）表明，对经过聚类的局部图像块（ Clustered Local Patches ）进行预测，可以改善密集表示（ Dense Representations ）。

DetCon（Hénaff et al., 2021）和 ORL（Xie et al., 2021）则在区域提议（ Region Proposals ）上执行对比学习（ Contrastive Learning ），但这类方法假设区域提议事先已经存在。ODIN（Hénaff et al., 2022）和 SlotCon（Wen et al., 2022）等方法则放宽了这一假设。

在不改变训练目标（ Training Objective ）的情况下，Darcet et al.（2024）表明，在输入序列中加入寄存器词元（ Register Tokens ）能够显著改善密集特征图（ Dense Feature Maps ）。

近期研究进一步发现，即使无需重新训练模型，也可以实现类似的改进（Jiang et al., 2025; Chen et al., 2025）。

Figure 4: Figure 4: DINOv3 at very high resolution. We visualize dense features of DINOv3 by mapping the first three components of a PCA computed over the feature space to RGB. To focus the PCA on the subject, we mask the feature maps via background subtraction. With increasing resolution, DINOv3 produces crisp features that stay semantically meaningful. We visualize more PCAs in Sec. 6.1.1.

图 4：高分辨率下的 DINOv3。我们通过将特征空间上进行的 PCA 分析所得到的前三个分量映射到 RGB 颜色空间，从而展示 DINOv3 中的密集特征。为了让 PCA 分析更聚焦于目标物体，我们通过背景减除技术来屏蔽掉无关的特征图。随着分辨率的提高，DINOv3 能够生成清晰且具有语义意义的特征。更多关于 PCA 的分析内容将在 6.1.1 节中介绍。

A recent trend are distillation-based, “agglomerative” methods that combine information from multiple image encoders with varying in global and local feature quality, trained using different levels of supervision (Ranzinger et al., 2024; Bolya et al., 2025): AM-RADIO (Ranzinger et al., 2024) combines the strengths of the fully-supervised SAM (Kirillov et al., 2023), the weakly-supervised CLIP, and the self-supervised DI-NOv2 into a unified backbone. The Perception Encoder (Bolya et al., 2025) similarly distills SAM(v2) into a specialized dense variant called PEspatial. They use an objective enforcing cosine similarity between student and teacher patches to be high, where their teacher is trained with mask annotations. Similar losses were shown to be effective in the context of style transfer, by reducing the inconsistency between the Gram matrices of feature dimensions (Gatys et al., 2016; Johnson et al., 2016; Yoo et al., 2024). In this work, we adopt a Gram objective to regularize cosine similarity between student and teacher patches, favoring them being close. In our case, we use earlier iterations of the SSL model itself as the teacher, demonstrating that early-stage SSL models effectively guides SSL training for both global and dense tasks.

近年来还出现了一类基于蒸馏的“聚合式”方法（ Distillation-based Agglomerative Methods ）。这类方法将多个图像编码器（ Image Encoders ）中的信息进行融合，而这些编码器在全局特征和局部特征质量方面各有差异，并且可能使用不同程度的监督信号进行训练（Ranzinger et al., 2024; Bolya et al., 2025）。

例如，

• AM-RADIO（Ranzinger et al., 2024）将全监督的 SAM（Kirillov et al., 2023）、弱监督的 CLIP，以及自监督的 DINOv2 各自的优势整合到一个统一的主干网络（ Unified Backbone ）中。
• 感知编码器（ Perception Encoder, PE ）（Bolya et al., 2025）同样将 SAM(v2) 蒸馏到一个专门面向密集任务的变体 PEspatial 中。该方法采用一种训练目标，使学生模型和教师模型对应图像块之间的余弦相似度（ Cosine Similarity ）尽可能高，其中教师模型使用掩码标注（ Mask Annotations ）进行训练。

类似的损失函数在风格迁移（ Style Transfer ）领域中也已经被证明十分有效，其思路是降低不同特征维度对应 Gram 矩阵（ Gram Matrices ）之间的不一致性（Gatys et al., 2016; Johnson et al., 2016; Yoo et al., 2024）。

在本项工作中，我们采用一个 Gram 目标函数（ Gram Objective ）来正则化学生模型与教师模型图像块之间的余弦相似度，使两者的特征更加接近。

与上述方法不同的是，我们将 SSL 模型自身在较早训练迭代阶段的版本作为教师模型。

实验表明，处于训练早期阶段的 SSL 模型能够有效指导后续的 SSL 训练，并同时改善全局任务（ Global Tasks ）和密集任务（ Dense Tasks ）的性能。

Other works focus on post-hoc improvements to the local features of SSL-trained models. For example, Ziegler and Asano (2022) fine-tune a pre-trained model with a dense clustering objective; similarly, Salehi et al. (2023) fine-tune by aligning patch features temporally, in both cases enhance the quality of local features. Closer to us, Pariza et al. (2025) propose a patch-sorting based objective to encourage the student and teacher to produce features with consistent neighbor ordering. Without finetuning, STEGO (Hamilton et al., 2022) learns a non-linear projection on top of frozen SSL features to form compact clusters and amplify correlation patterns. Alternatively, Simoncini et al. (2024) augment self-supervised features by concatenating gradients from different self-supervised objectives to frozen SSL features. Recently, Wysoczańska et al. (2024) show that noisy feature maps are significantly improved through a weighted average of patches.

另一些研究则着重于对 SSL 模型的局部特征进行训练后的改进（ Post-hoc Improvements ）。

例如，

• Ziegler and Asano（2022）利用密集聚类目标（ Dense Clustering Objective ）对预训练模型进行微调；
• 类似地，Salehi et al.（2023）通过在时间维度上对齐图像块特征（ Temporal Alignment of Patch Features ）进行微调。两类方法都能够提升局部特征的质量。
• 与本文更接近的是，Pariza et al.（2025）提出了一种基于图像块排序（ Patch-sorting ）的目标函数，用于鼓励学生模型与教师模型生成具有一致邻域排序（ Consistent Neighbor Ordering ）的特征。
• 在无需微调的情况下，STEGO（Hamilton et al., 2022）会在冻结的 SSL 特征之上学习一个非线性投影（ Non-linear Projection ），从而形成更加紧凑的聚类，并增强特征之间的相关性模式（ Correlation Patterns ）。
• 另一种方法由 Simoncini et al.（2024）提出，其通过将不同自监督学习目标产生的梯度与冻结的 SSL 特征进行拼接，从而增强自监督特征。
• 近期，Wysoczańska et al.（2024）表明，通过对不同图像块进行加权平均（ Weighted Average of Patches ），可以显著改善存在噪声的特征图。

Related, but not specific to SSL, some recent works generate high-resolution feature maps from ViT feature maps (Fu et al., 2024), which are often low-resolution due to patchification of images. In contrast with this body of work, our models natively deliver high-quality dense feature maps that remain stable and consistent across resolutions, as shown in Fig. 4.

此外，还有一些相关研究虽然并非专门针对 SSL，但同样致力于从 ViT 特征图中生成高分辨率特征图（ High-resolution Feature Maps ）（Fu et al., 2024）。由于图像需要被划分为图像块（ Patchification ），ViT 产生的特征图通常具有较低的空间分辨率。

与上述工作不同，我们的模型能够原生生成高质量的密集特征图（ High-quality Dense Feature Maps ），并且这些特征图在不同分辨率下均能够保持稳定且一致，如图 4 所示。

3 Training at Scale Without Supervision

3 无监督下的大规模训练

DINOv3 is a next-generation model designed to produce the most robust and flexible visual representations to date by pushing the boundaries of self-supervised learning. We draw inspiration from the success of large language models (LLMs), for which scaling-up the model capacity leads to outstanding emerging properties. By leveraging models and training datasets that are an order of magnitude larger, we seek to unlock the full potential of SSL and drive a similar paradigm shift for computer vision, unencumbered by the limitations inherent to traditional supervised or task-specific approaches. In particular, SSL produces rich, high-quality visual features that are not biased toward any specific supervision or task, thereby providing a versatile foundation for a wide range of downstream applications. While previous attempts at scaling SSL models have been hindered by issues of instability, this section describes how we harness the benefits of scaling with careful data preparation, design, and optimization. We first describe the dataset creation procedure (Sec. 3.1), then present the self-supervised SSL recipe used for this first training phase of DINOv3 (Sec. 3.2). This includes the choice of architecture, loss functions, and optimization techniques. The second training phase, focusing on dense features, will be described in Sec. 4.

DINOv3 是一种新一代模型，旨在通过推动 自监督学习（Self-Supervised Learning, SSL） 的边界，生成迄今为止最具鲁棒性（ Robust ）和灵活性的视觉表示（ Visual Representations ）。

我们的设计理念受到大语言模型（ Large Language Models, LLMs ）成功经验的启发：对于大语言模型而言，扩大模型容量（ Model Capacity Scaling ）能够带来显著的涌现能力（ Emergent Properties ）。

通过使用规模提升一个数量级的模型以及训练数据集（ Training Datasets ），我们希望释放 自监督学习（SSL） 的全部潜力，并推动计算机视觉领域类似的范式转变（ Paradigm Shift ），摆脱传统监督学习（ Supervised Learning ）或特定任务方法（ Task-specific Approaches ）固有的限制。

具体而言，自监督学习（Self-Supervised Learning, SSL） 能够产生丰富、高质量的视觉特征（ Visual Features ），这些特征不会受到特定监督信号或任务目标的偏置影响，因此可以作为广泛下游应用（ Downstream Applications ）的通用基础。

尽管此前对 SSL 模型进行规模扩展（ Scaling SSL Models ）的尝试受到训练不稳定性（ Instability Issues ）的限制，本节将介绍我们如何通过精心的数据准备（ Data Preparation ）、模型设计（ Model Design ）以及优化策略（ Optimization ）来充分利用规模扩展带来的优势。

我们

• 首先介绍数据集构建流程（ Dataset Creation Procedure ）（第 3.1 节），
• 随后介绍用于 DINOv3 第一阶段训练的自监督训练方案（ Self-supervised SSL Recipe ）（第 3.2 节）。该部分包括模型架构（ Architecture ）选择、损失函数（ Loss Functions ）设计以及优化技术（ Optimization Techniques ）等内容。
• 第二阶段训练主要关注密集特征（ Dense Features ）的学习，将在第 4 节中进行介绍。

表 1：训练数据对特征质量的影响，通过其在下游任务中的表现来体现。我们将经过聚类处理的数据集（Vo 等人，2024 年）与经过检索处理的数据集（Oquab 等人，2024 年）与原始数据以及我们的混合数据集进行了比较。这项消融实验的迭代次数为 20 万次。

3.1 Data Preparation

3.1 数据准备

Data scaling is one of the driving factors behind the success of large foundation models (Touvron et al., 2023; Radford et al., 2021; Xu et al., 2024; Oquab et al., 2024). However, increasing naively the size of the training data does not necessaril translate into higher model quality and better performance on downstream benchmarks (Goyal et al., 2021; Oquab et al., 2024; Vo et al., 2024): Successful data scaling efforts typically involve careful data curation pipelines. These algorithms may have different objectives: either focusing on improving data diversity and balance, or data usefulness-its relevance to common practical applications. For the development of DINOv3, we combine two complementary approaches to improve both the generalizability and performance of the model, striking a balance between the two objectives.

数据规模扩展（ Data Scaling ）是推动大型基础模型（ Large Foundation Models ）成功的重要因素之一（Touvron et al., 2023; Radford et al., 2021; Xu et al., 2024; Oquab et al., 2024）。

然而，简单地增加训练数据规模并不一定能够带来更高的模型质量以及更好的下游基准测试（ Downstream Benchmarks ）性能（Goyal et al., 2021; Oquab et al., 2024; Vo et al., 2024）：成功的数据规模扩展通常依赖于精心设计的数据筛选流程（ Data Curation Pipelines ）。

这些算法可能具有不同的目标：有些关注提升数据多样性和均衡性（ Data Diversity and Balance ），而另一些则关注数据有效性（ Data Usefulness ）——即数据与实际常见应用场景之间的相关程度。

在 DINOv3 的开发过程中，我们结合了两种互补的方法，以同时提升模型的泛化能力（ Generalizability ）和性能（ Performance ），在这两个目标之间取得平衡。

Data Collection and Curation We build our large-scale pre-training dataset by leveraging a large data pool of web images collected from public posts on Instagram. These images already went through platformlevel content moderation to help prevent harmful contents and we obtain an initial data pool of approximately 17 billions of images. Using this raw data pool, we create three dataset parts. We construct the first part by applying the automatic curation method based on hierarchical -means from Vo et al. (2024). We employ DINOv2 as image embeddings, and use 5 levels of clustering with the number of clusters from the lowest to highest levels being 200M, 8M, 800k, 100k, and 25k respectively. After building the hierarchy of clusters, we apply the balanced sampling algorithm proposed in Vo et al. (2024). This results in a curated subset of 1,689 million images (named LVD-1689M) that guarantees a balanced coverage of all visual concepts appearing on the web. For the second part, we adopt a retrieval-based curation system similar to the procedure proposed by Oquab et al. (2024). We retrieve images from the data pool that are similar to those from selected seed datasets, creating a dataset that covers visual concepts relevant for downstream tasks. For the third part, we use raw publicly available computer vision datasets including ImageNet1k (Deng et al., 2009), ImageNet22k (Russakovsky et al., 2015), and Mapillary Street-level Sequences (Warburg et al., 2020). This final part allows us to optimize our model’s performance, following Oquab et al. (2024).

3.1.1 数据收集与筛选（ Data Collection and Curation ）

我们利用从 Instagram 公开帖子中收集的大规模网络图像数据池（ Large-scale Web Image Data Pool ）构建预训练数据集（ Pre-training Dataset ）。

这些图像已经经过平台级内容审核（ Platform-level Content Moderation ），有助于避免有害内容的出现。我们获得了一个初始数据池，其中包含约 170 亿张图像。

基于该原始数据池，我们创建了三个数据集部分。

• 第一部分通过应用 Vo et al.（2024）提出的基于层次化 k-means 的自动筛选方法（ Automatic Curation Method based on Hierarchical k-means ）构建。我们使用 DINOv2 作为图像嵌入（ Image Embeddings ），并采用 5 层聚类结构（ Hierarchical Clustering ）。从最低层到最高层，聚类数量分别为 200M、8M、800k、100k 和 25k。在构建完聚类层级结构（ Hierarchy of Clusters ）之后，我们应用 Vo et al.（2024）提出的均衡采样算法（ Balanced Sampling Algorithm ）。最终得到一个包含 16.89 亿张图像的精选子集（ Curated Subset ），命名为 LVD-1689M。该数据集能够保证对网络中出现的所有视觉概念（ Visual Concepts ）进行均衡覆盖。
• 对于第二部分，我们采用了一种基于检索的筛选系统（ Retrieval-based Curation System ），其流程类似于 Oquab et al.（2024）提出的方法。我们从数据池中检索与选定种子数据集（ Seed Datasets ）中的图像相似的图像，从而构建一个能够覆盖与下游任务相关视觉概念的数据集。
• 对于第三部分，我们使用公开可用的原始计算机视觉数据集（ Raw Publicly Available Computer Vision Datasets ），包括：
  • ImageNet1k（Deng et al., 2009）
  • ImageNet22k（Russakovsky et al., 2015）
  • Mapillary 街景序列（ Mapillary Street-level Sequences ）（Warburg et al., 2020）
  • 这一最终部分使我们能够进一步优化模型性能，遵循 Oquab et al.（2024）的方法。

Data Sampling During pre-training, we use a sampler to mix different data parts together. There are several different options for mixing the above data components. One is to train with homogeneous batches of data that come from a single, randomly selected component in each iteration. Alternatively, we can optimize the model on heterogeneous batches that are assembled by data from all components, selected using certain ratios. Inspired by Charton and Kempe (2024), who observed that it is beneficial to have homogeneous batches consisting of very high quality data from a small dataset, we randomly sample in each iteration either a homogeneous batch from ImageNet1k alone or a heterogeneous batch mixing data from all other components. In our training, homogeneous batches from ImageNet1k account for 10% of training.

3.1.2 数据采样（ Data Sampling ）

在预训练过程中，我们使用一个采样器（ Sampler ）将不同数据部分混合在一起。

对于上述数据组件的混合方式，存在多种选择。

一种方式是在每次训练迭代（ Iteration ）中，从随机选择的单一数据组件中构造同质批次（ Homogeneous Batches ）进行训练。

另一种方式是优化异质批次（ Heterogeneous Batches ），即按照一定比例从所有数据组件中选择数据并组合成批次。

受到 Charton 和 Kempe（2024）的启发，他们发现由小规模数据集中的高质量数据组成的同质批次具有积极作用，因此我们在每次迭代中随机选择以下两种方式之一：

1. 仅从 ImageNet1k 中采样一个同质批次；
2. 混合其他所有数据组件的数据，形成异质批次。

在我们的训练过程中，来自 ImageNet1k 的同质批次占总训练过程的 10%。

表 2：DINOv2 与 DINOv3 模型中所使用的教师网络架构对比。这两种模型的结构都包含 40 层，且嵌入维数均为4096。需要注意的是，我们采用了 16 像素的块大小，这会改变在特定分辨率下序列的有效长度。

要么使用仅来自 ImageNet1k 的同类数据构成的训练集，要么使用来自所有其他数据源的混合数据构成的训练集。在我们的训练中，来自 ImageNet1k 的同类数据占训练数据的 10%。

Data Ablation To assess the impact of our data curation technique, we perform an ablation study to compare our data mix against datasets curated with clustering or retrieval-based methods alone, and the raw data pool. To this end, we train a model on each dataset and compare their performance on standard downstream tasks. For efficiency, we use a shorter schedule of 200k iterations instead of 1M iterations. In Tab. 1, it can be seen that no single curation technique works best across all benchmarks, and that our full pipeline allows us to obtain the best of both worlds.

3.1.3 数据消融实验（ Data Ablation ）

为了评估数据筛选技术（ Data Curation Technique ）的影响，我们进行了消融研究（ Ablation Study ），比较以下数据配置：

• 我们提出的数据混合方案；
• 仅使用聚类方法（ Clustering-based Method ）筛选的数据集；
• 仅使用检索方法（ Retrieval-based Method ）筛选的数据集；
• 原始数据池（ Raw Data Pool ）。

为此，我们分别基于每个数据集训练模型，并比较它们在标准下游任务（ Standard Downstream Tasks ）上的表现。

为了提高实验效率，我们采用较短的训练计划（ Training Schedule ），训练 200k 次迭代，而不是完整的 1M 次迭代。

如表 1 所示，没有任何单一的数据筛选技术能够在所有基准测试中取得最佳效果，而我们完整的数据处理流程（ Full Pipeline ）能够结合两种方法的优势，实现两者之间的最佳平衡。

3.2 基于自监督的大规模训练

While models trained with SSL have demonstrated interesting properties (Chen et al., 2020b; Caron et al., 2021), most SSL algorithms have not been scaled-up to larger models sizes. This is either due to issues with training stability (Darcet et al., 2025), or overly simplistic solutions that fail to capture the full complexity of the visual world. When trained at scale (Goyal et al., 2022a), models trained with SSL do not necessarily show impressive performance. One notable exception is DINOv2, a model with 1.1 billion parameters trained on curated data, matching the performance of weakly-supervised models like CLIP (Radford et al., 2021). A recent effort to scale DINOv2 to 7 billion parameters (Fan et al., 2025) demonstrates promising results on global tasks, but with disappointing results on dense prediction. Here, we aim to scale up the model and data, and obtain even more powerful visual representations with both improved global and local properties.

虽然使用自监督学习（ Self-Supervised Learning, SSL ）训练的模型已经展现出一些有趣的特性（Chen et al., 2020b; Caron et al., 2021），但大多数 SSL 算法尚未扩展到更大规模的模型尺寸。

这主要是由于以下两个原因之一：

• 训练稳定性问题（ Training Stability Issues ）（Darcet et al., 2025）；
• 过于简单的解决方案无法捕获视觉世界的完整复杂性（ Full Complexity of the Visual World ）。

当 SSL 模型进行大规模训练（ Scale Training ）时（Goyal et al., 2022a），其性能并不一定表现出令人印象深刻的提升。

一个值得注意的例外是 DINOv2，它是一个具有 11 亿参数（ 1.1 Billion Parameters ）的模型，在经过筛选的数据（ Curated Data ）上训练，并达到了与弱监督模型（ Weakly-supervised Models ）（例如 CLIP（Radford et al., 2021））相当的性能。

最近，一项将 DINOv2 扩展到 70 亿参数（ 7 Billion Parameters ）的工作（Fan et al., 2025）

• 在全局任务（ Global Tasks ）上展示出了良好的结果，
• 但在密集预测（ Dense Prediction ）任务上的表现令人失望。

在本文中，我们旨在同时扩大模型规模和数据规模（ Model and Data Scaling ），并获得更强大的视觉表示（ Visual Representations ），使其同时具备更优的

• 全局特性（ Global Properties ）
• 局部特性（ Local Properties ）。

图 5：(a)显示了 ViT-g 与 ViT-7B 在 ImageNet1k 数据集上的余弦相似度变化情况；(b)和(c)则分别展示了这两种模型在 VOC 数据集上的图像分割任务中的准确率变化情况。我们发现，当图像块向量与类别向量之间的余弦相似度较低时，图像分割性能最佳。随着训练的进行，这种相似度会逐渐上升，从而导致模型在密集型任务上的性能下降。

Learning Objective We train the model with a discriminative self-supervised strategy which is a mix of several selfsupervised objectives with both global and local loss terms. Following DINOv2 (Oquab et al., 2024), we use an image-level objective (Caron et al., 2021) , and balance it with a patch-level latent reconstruction objective (Zhou et al., 2021) . We also replace the centering from DINO with the Sinkhorn-Knopp from SwAV (Caron et al., 2020) in both objectives. Each objective is computed using the output of a dedicated head on top of the backbone network, allowing for some specialization of features before the computation of the losses. Additionally, we use a dedicated layer normalization applied to the backbone outputs of the local and global crops. Empirically, we found this change to stabilize ImageNet kNNclassification late in training (+0.2 accuracy) and improve dense performance (e.g. +1 mIoU on ADE20k segmentation, -0.02 RMSE on NYUv2 depth estimation). In addition, a Koleo regularizer is added to encourage the features within a batch to spread uniformly in the space (Sablayrolles et al., 2018). We use a distributed implementation of Koleo in which the loss is applied in small batches of 16 samples-possibly across GPUs. Our initial training phase is carried by optimizing the following loss:

3.2.1 学习目标（ Learning Objective ）

我们采用一种判别式自监督策略（ Discriminative Self-Supervised Strategy ）训练模型，该策略结合了多种自监督目标（ Self-Supervised Objectives ），并同时包含

• 全局损失项（ Global Loss Terms ）
• 局部损失项（ Local Loss Terms ）。

遵循 DINOv2（Oquab et al., 2024），我们使用：

1. 图像级目标（ Image-level Objective ）：

$$L_{DINO}$$

该目标来自 Caron et al.（2021）。

2. 图像块级潜变量重建目标（ Patch-level Latent Reconstruction Objective ）：

L i B O T L_{iBOT}