are there any model which is entirely based on vision transformer (like plain vit backbone + detr head) for object detection?
Probably, yes:
If by “full Vision Transformer” you mean an object detector whose main visual backbone is a ViT rather than a CNN, there are several examples now.
For the specific shape you mentioned — something like a plain ViT backbone + a DETR-style head/decoder — I would start with LW-DETR. Hugging Face describes its architecture as a plain ViT encoder + projector + shallow DETR decoder, so it is probably the closest off-the-shelf match to your example.
If you want a newer, practical detector built around a pretrained ViT backbone, RF-DETR is also worth looking at. It uses a DINOv2 ViT backbone and is now integrated directly into Transformers; the current Hugging Face object-detection guide even uses RF-DETR for the fine-tuning example.
And if what you mean is closer to “vanilla ViT itself turned into an object detector”, then YOLOS is probably the cleanest reference point. It modifies ViT only minimally and adds detection tokens/heads rather than attaching a conventional DETR encoder-decoder stack.
So, roughly:
| What you are looking for | Candidate | Transformers usability |
|---|---|---|
| plain ViT + DETR-style decoder | LW-DETR | native pipeline / AutoModelForObjectDetection |
| modern ViT detector, practical/fine-tunable | RF-DETR | native Transformers + current HF training guide |
| vanilla-ViT-like detector | YOLOS | native pipeline / AutoModelForObjectDetection |
| plain ViT backbone + stronger research DETR | DINO + ViTDet in detrex | research stack rather than a single HF detector class |
| explicitly remove the CNN backbone | WB-DETR | mainly a research reference |
The only thing I would be careful about is the word “entirely”, because there are a few different meanings hiding inside it.
What does “full / pure ViT” mean here?
I think it helps to separate at least four slightly different requirements:
A. no ResNet / CNN backbone
B. a plain/non-hierarchical ViT as the backbone
C. ViT backbone + DETR-family decoder/head
D. literally no convolution operator anywhere in the model
A–C are fairly well represented in existing detectors.
D is much stricter.
For example, LW-DETR definitely has a plain ViT encoder, but it also has a C2f projector derived from the YOLOv8 design. The HF documentation describes the flow roughly as:
image
↓
plain ViT encoder
↓
multi-level encoder features
↓
C2f projector
↓
shallow deformable DETR decoder
↓
boxes/classes
So I would call LW-DETR a ViT-based / ViT-backbone detector, but not a literally zero-convolution network.
YOLOS is closer to the conceptual “just use ViT for detection” end of the spectrum. Its high-level structure is more like:
image patches + detection tokens
↓
ViT
↓
class / box prediction heads
There is no separate DETR decoder.
There is another subtlety here: many ViT implementations use Conv2d to implement the initial non-overlapping patch projection. A layer like
Conv2d(
in_channels=3,
out_channels=hidden_dim,
kernel_size=16,
stride=16,
)
is effectively applying the same learned linear projection independently to each 16×16 patch. So:
contains an nn.Conv2d module
does not necessarily mean
uses a conventional CNN feature extractor
Those are different architectural questions.
That distinction becomes useful if “entirely Transformer” is meant at the operator level rather than at the architectural/backbone level.
A few useful reference points
1. YOLOS: probably the cleanest “can vanilla ViT detect objects?” example
The original idea behind YOLOS is deliberately simple: take a Vision Transformer and see how far it gets on object detection with minimal modifications.
In Transformers it is particularly easy to try:
from transformers import pipeline
detector = pipeline(
"object-detection",
model="hustvl/yolos-small",
)
or:
from transformers import AutoImageProcessor, AutoModelForObjectDetection
processor = AutoImageProcessor.from_pretrained("hustvl/yolos-small")
model = AutoModelForObjectDetection.from_pretrained("hustvl/yolos-small")
The linked hustvl/yolos-small checkpoint is Apache-2.0.
Conceptually this is probably the best baseline if the question is:
“Can a nearly vanilla ViT itself serve as the detector?”
It is less appropriate if the exact requirement is:
“I specifically want ViT backbone → DETR decoder.”
because YOLOS does not have that conventional DETR decoder boundary.
2. LW-DETR: closest to “plain ViT + DETR”
LW-DETR is much closer to the architecture in the question.
The HF docs explicitly describe:
- a plain ViT encoder;
- a projector;
- a shallow DETR decoder.
It also mixes window and global attention in the ViT and gathers intermediate encoder features before the projector, so “plain ViT” here should not be read as “absolutely no detection-oriented adaptation”.
But if I wanted to try the closest existing answer with the least setup, this is probably where I would start:
from transformers import pipeline
detector = pipeline(
"object-detection",
model="AnnaZhang/lwdetr_small_60e_coco",
)
The linked lwdetr_small_60e_coco checkpoint and the official LW-DETR implementation are Apache-2.0.
3. RF-DETR: newer practical ViT + DETR-family route
RF-DETR is interesting if the motivation is practical rather than architectural purity.
It uses a DINOv2 Vision Transformer backbone, and the implementation descends from ideas in LW-DETR / Deformable DETR. It is integrated into current Transformers, including the normal object-detection model interface.
HF’s current object detection tutorial uses RF-DETR for fine-tuning, which makes it one of the easier current options if the eventual goal is custom-data training rather than just studying the architecture.
One license detail is worth checking before choosing a size: the RF-DETR project separates licensing by component/model tier. The core package and Nano-through-Large models are Apache-2.0, while the Plus components / XL / 2XL detection models use PML 1.0.
So I would check the exact checkpoint rather than assuming every RF-DETR weight has identical terms.
4. ViTDet + DINO: useful if you want to assemble the pieces yourself
ViTDet is a particularly useful reference for the backbone side of this question.
Its important result was that you do not necessarily need to redesign ViT into a hierarchical vision architecture first: a plain, non-hierarchical ViT can be adapted to detection.
However, in Transformers, ViTDet is exposed as a backbone, not as a complete ViTDetForObjectDetection equivalent.
For an actual “ViT backbone + modern DETR” research configuration, detrex is a useful concrete example. Its model zoo contains pretrained:
- DINO-ViTDet-Base
- DINO-ViTDet-Large
configurations, and the project documents how it builds a Simple Feature Pyramid around ViTDet.
So this combination is not just a hypothetical wiring diagram; trained ViT + DINO/DETR-family configurations exist.
The trade-off is that this route is more of a Detectron2/detrex research stack than the one-line Transformers experience of LW-DETR, RF-DETR, or YOLOS.
5. WB-DETR: useful if “no CNN backbone” is the important part
If your definition of “full Transformer” is specifically:
“I do not want a CNN feature-extraction backbone at all”
then WB-DETR is a useful historical reference.
Its stated goal was explicitly to test whether a DETR-style detector really needed a CNN feature extractor. It serializes the image into tokens and uses an encoder/decoder without the usual CNN backbone.
I would treat this mainly as a useful architecture/paper reference today rather than the easiest model to deploy from Hugging Face.
Recent direction: foundation-model ViTs are increasingly being used as detector backbones
There is also a newer pattern that may be relevant depending on why you want a “full ViT” detector:
large/self-supervised ViT
↓
small detection adaptation/projector
↓
DETR-family detector
RF-DETR is one example using DINOv2.
Meta’s current DINOv3 repository also ships detector code around its ViT backbones. The detector implementation and detection configuration contain Deformable-DETR/DINO-style machinery rather than a plain minimal DETR head.
That makes DINOv3 another strong piece of evidence that:
ViT backbone + DETR-family detection head is a very active design pattern,
but I would not call it a minimal “plain ViT + plain DETR” implementation.
Also, DINOv3 uses the project’s own DINOv3 License Agreement, so its licensing/deployment story is different from the Apache-2.0 examples above.
Small implementation sanity check
I also did a small smoke check with a current Transformers installation, mainly because the term “pure Transformer” can be misleading when looking only at architecture diagrams.
I loaded:
hustvl/yolos-small
AnnaZhang/lwdetr_small_60e_coco
and ran a forward pass plus a simple nn.Conv2d module inventory.
The useful observation was:
YOLOS-small:
Conv2d modules = 1
└─ the ViT patch projection (16×16 kernel, stride 16)
LW-DETR-small:
Conv2d modules = 9
├─ 1 ViT patch projection
└─ 8 additional convolutions in the C2f projector
This was only an implementation smoke test, not an accuracy/speed benchmark, but it illustrates the terminology issue nicely:
YOLOS
≈ architecturally very close to a vanilla ViT detector
but not literally "contains zero Conv2d objects"
LW-DETR
= plain ViT backbone + DETR-family decoder
but also contains a convolutional projector
So if your requirement is architectural simplicity, YOLOS/LW-DETR/RF-DETR are all useful references.
If your requirement is literally zero convolution operators, I would inspect the exact implementation rather than rely on labels such as “pure Transformer”, “full Transformer”, or even “plain ViT”.
My short version
If I were choosing where to start:
Want the closest thing to "plain ViT backbone + DETR head"?
→ LW-DETR
Want a newer practical ViT-based detector with good current HF support?
→ RF-DETR
Want the cleanest vanilla-ViT-as-detector reference?
→ YOLOS
Want to build a ViT backbone + strong DETR-family detector yourself?
→ ViTDet + DINO/detrex
Want specifically "no CNN backbone" as a research constraint?
→ WB-DETR
Want literally zero Conv2d / convolution operations?
→ treat that as a separate requirement and audit the exact model implementation
So the answer is definitely not “there is no full-ViT object detector”; rather, there are several families already. The main choice is what you mean by “full” and whether your priority is architectural purity, a DETR-like component boundary, current Transformers usability, or practical detection performance.
Yes. DETR is probably the closest match to what you’re describing: a transformer-based object detector where the image is represented as tokens and processed by a transformer encoder-decoder, although the original DETR uses a CNN backbone rather than a plain ViT. For a more fully ViT-based approach, you may want to look at models such as ViTDet, which uses a Vision Transformer backbone for detection, or DINO/Deformable DETR variants with transformer-based detection pipelines. The main distinction is whether you mean a strictly plain ViT with no CNN components at all, or simply a detector built around a ViT backbone.
Thank you for the reply. I would like to know if it is practical to build a object detection model with Vanilla Vit backbone and the head part (decoder) from detr.