gaze-lle-p150 on Tenstorrent p150
Derived from Gaze-LLE (fkryan/gazelle) by Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman and James M. Rehg: the
gazelle_dinov2_vitb14_inoutcheckpoint on the DINOv2 ViT-B/14 backbone (facebookresearch/dinov2), with unmodified weights mirrored at changh95/gaze-lle-weights. Paper: arXiv:2412.09586. Port: changh95/tt-Gaze-LLE. See the upstream repository for training and evaluation details.
Gaze-LLE ViT-B/14 with the in/out head (gazelle_dinov2_vitb14_inout): a frozen DINOv2 ViT-B/14 encoder (86M parameters) and a 2.8M-parameter gaze decoder (a head-position prompt added to the scene features, then 3 transformer layers), trained on GazeFollow and fine-tuned on VideoAttentionTarget. One RGB image goes in and is squashed to 448x448, together with 1 to 10 head boxes, one per person. Out come, for each person, a 64x64 heatmap of where the person is looking over the whole image, the gaze target point (the heatmap maximum, in pixels of the original image) and the probability that the gaze target is inside the frame. The encoder runs once per image and all heads share it.
Runs on Tenstorrent p150 (mesh P150).
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
At a glance
| Hardware | p150 |
| License | MIT and Apache-2.0 |
Intended use
Gaze-target estimation (where each person in an image is looking) for research, prototyping and offline analysis of images and video frames, given one head box per person from a separate head or face detector.
Prerequisites
- The Tenstorrent CLI,
tt:uv tool install tenstorrent(or use tt-model alone, see the Quickstart) - Docker
- Tenstorrent hardware: p150
Quickstart
tt serve changh95/gaze-lle-p150
tt serve downloads the Docker image and the changh95/gaze-lle-weights at f83e53f0f17dd2175f31ceaf9fa650044147ada1 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.
Without tt-cli: tt-model serve changh95/gaze-lle-p150.
Send the sample image of this repo with its three head boxes to /predict, on the port tt serve reported:
curl -sLO https://huggingface.co/changh95/gaze-lle-p150/resolve/main/media/source_1.png
printf '{"image":"%s","bboxes":[[116,28,176,104],[269,37,301,81],[87,57,107,76]]}' "$(base64 -w0 source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
Capabilities
This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β its request and response shapes are the model's own. See the author's notes for the payload it expects.
POST /predict takes one image and the head boxes of the people in it, and returns one gaze prediction per box.
GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.
request field (POST /predict) |
meaning |
|---|---|
image |
base64 PNG or JPEG of one image, any size; a data:image/...;base64, prefix is accepted |
bboxes |
1 to 10 head boxes [x1, y1, x2, y2], one per person; clamped to the image, and each must span more than 1/32 of the image in both axes |
bbox_format (optional) |
pixels (default: pixels of the original image) or normalized (fractions of width and height) |
heatmap_format (optional) |
png (default: one base64 8-bit grayscale 64x64 PNG per head, uint8(round(p * 255))), npz (one base64 .npz with heatmap float32 (N, 64, 64) and inout float32 (N,)), json (nested lists, 5 decimals) or none |
Response for the sample image (media/source_1.png, 500x334) and the three boxes of the quickstart (heatmaps elided):
{"model": "gaze-lle-p150", "image_size": [500, 334], "input_size": [448, 448], "num_heads": 3,
"heatmap_size": [64, 64], "heatmap_format": "png", "heatmap_note": "...",
"predictions": [
{"bbox": [116.0, 28.0, 176.0, 104.0], "bbox_normalized": [0.232, 0.083832, 0.352, 0.311377], "inout": 0.037841796875,
"gaze_target": {"x": 289.0625, "y": 172.21875, "x_norm": 0.578125, "y_norm": 0.515625, "heatmap_row": 33, "heatmap_col": 37, "score": 0.349609375},
"heatmap_png": "..."},
{"bbox": [269.0, 37.0, 301.0, 81.0], "bbox_normalized": [0.538, 0.110778, 0.602, 0.242515], "inout": 0.75390625,
"gaze_target": {"x": 296.875, "y": 99.15625, "x_norm": 0.59375, "y_norm": 0.296875, "heatmap_row": 19, "heatmap_col": 38, "score": 0.515625},
"heatmap_png": "..."},
{"bbox": [87.0, 57.0, 107.0, 76.0], "bbox_normalized": [0.174, 0.170659, 0.214, 0.227545], "inout": 0.67578125,
"gaze_target": {"x": 93.75, "y": 62.625, "x_norm": 0.1875, "y_norm": 0.1875, "heatmap_row": 12, "heatmap_col": 12, "score": 0.212890625},
"heatmap_png": "..."}],
"timing_ms": {"preprocess": 9.75, "inference": 3.8, "encode": 1.04, "total": 14.59}}
heatmapvalues are probabilities in [0, 1] on a 64x64 grid that covers the whole original image (row = y, column = x).gaze_targetis its maximum:x = column / 64 * width,y = row / 64 * heightin pixels of the original image,x_norm/y_normthe same as fractions,scorethe maximum value.inoutis the probability that the person's gaze target is inside the frame. Below 0.5 the person is predicted to look outside the image, andgaze_targetis then not meaningful.bboxechoes the box as sent;bbox_normalizedis the box as used (fractions, clamped to the image).timing_msgives the server-side stages of the request:preprocess(base64 and image decode, resize to 448x448 on the host CPU),inference(upload, device run, readback),encode(heatmap encoding).- Errors: 400 for an image that cannot be decoded, more than 10 boxes, an empty or too small box, or invalid request fields; 503 while the server is starting.
- A Python API with the same outputs (
gaze_lle.GazeLLE) is described incode/PYTHON.md, with an example incode/examples/quickstart.py. It needs a host ttnn build of the tt-metal commit in Provenance withpatches/tt-metal-8b98410.local.patchapplied.
Example output
Outputs of this package drawn with gaze_lle.visualize: every head box with its in/out probability, and for
each head with inout at least 0.5 its heatmap and a line from the box centre to the predicted gaze target. The
first image is the response above (the sample image with the three quickstart boxes); the other two are
GazeFollow test images 1000 and 2600 with the head box of their annotated person. Inputs:
media/source_1.png, media/source_2.png and media/source_4.png (see Licensing).
Expected performance
Gaze-target estimation on the two benchmarks the Gaze-LLE paper reports, and agreement with the same model run in fp32 on the CPU:
| eval | this package | reference | images |
|---|---|---|---|
| VideoAttentionTarget test: AUC / L2 / AP in-out | 0.9316 / 0.1084 / 0.8986 | 0.933 / 0.107 / 0.897 (published, this checkpoint, all 13,127 test frames) | 1,384 frames (3,355 heads) |
| VideoAttentionTarget test, same frames: AUC / L2 / AP in-out | 0.9316 / 0.1084 / 0.8986 | 0.9330 / 0.1072 / 0.9014 (fp32 CPU reference) | 1,384 frames |
| GazeFollow test: AUC / Avg L2 / Min L2 | 0.9542 / 0.1129 / 0.0508 | 0.956 / 0.104 / 0.045 (published for the GazeFollow-only checkpoint gazelle_dinov2_vitb14, not this one) |
4,782 |
| GazeFollow test, first 1,000 images: AUC / Avg L2 / Min L2 | 0.9530 / 0.1166 / 0.0530 | 0.9529 / 0.1115 / 0.0498 (fp32 CPU reference) | 1,000 |
| Heatmap PCC against the fp32 CPU reference, mean / median | 0.987 / 0.993 (GazeFollow), 0.985 / 0.993 (VideoAttentionTarget) | β | 1,000 / 1,384 |
In/out decision (inout β₯ 0.5) equal to the fp32 CPU reference |
98.1 % (GazeFollow), 97.4 % (VideoAttentionTarget) | β | 1,000 / 1,384 |
The published figures are the "Gaze-LLE (ViT-B)" row of Table 3 of the paper. Its VideoAttentionTarget columns
are for the checkpoint served here (gazelle_dinov2_vitb14_inout); its GazeFollow columns are for the
GazeFollow-only checkpoint. VideoAttentionTarget (ejcgt/attention-target-detection):
every 10th frame of each of the 131 test clips. GazeFollow: the whole test set (the
vikhyatk/gazefollow copy), one annotated person per image.
Metrics and data preparation follow fkryan/gazelle. The fp32 CPU reference is the PyTorch Gaze-LLE model of code/gaze_lle/reference/
(adapted from fkryan/gazelle) with the same weights and the upstream preprocessing.
| speed (batch 1, warm) | input | median | p99 | throughput |
|---|---|---|---|---|
| Device time per image, back-to-back trace replays | 448x448, 1 head | 2.79 ms | 2.79 ms | 359 images/s |
| Device time per image, back-to-back trace replays | 448x448, 3 heads | 3.07 ms | 3.07 ms | 326 images/s |
| Device time per image, back-to-back trace replays | 448x448, 10 heads | 4.10 ms | 4.13 ms | 244 images/s |
Python model() call (upload, device run, readback, output decoding) |
uint8 RGB array 448x448, 1 head | 3.25 ms | 3.70 ms | 307 images/s |
Python model() call |
uint8 RGB array 448x448, 3 heads | 3.60 ms | 4.58 ms | 278 images/s |
Python model() call |
uint8 RGB array 448x448, 10 heads | 4.61 ms | 12.9 ms | 217 images/s |
Python model() call, also decoding and resizing the file |
sample PNG 500x334, 3 heads | 14.2 ms | 17.9 ms | 70 images/s |
Served POST /predict, server timing_ms.total |
sample PNG 500x334, 1 head | 12.0 ms | 16.1 ms | β |
Served POST /predict, client round trip on loopback |
sample PNG 500x334, 1 head | 14.9 ms | 19.5 ms | β |
Served POST /predict, server timing_ms.total |
sample PNG 500x334, 3 heads | 12.6 ms | 16.6 ms | β |
Served POST /predict, client round trip on loopback |
sample PNG 500x334, 3 heads | 15.5 ms | 19.9 ms | β |
Measured with this package's image on one p150 in an 8-vCPU AMD EPYC-Rome host (host load average about 10 to
13 during the runs), with code/scripts/bench_p150.py (500 calls per Python row; each device row is 20 rounds
of 500 replays) and code/scripts/bench_http.py (1,000 requests per served row, one at a time on one keep-alive
connection, PNG heatmaps); of the served 3-head request, base64 and PNG decode and the resize on the host take
7.9 ms (median).
Limitations
- Not for decisions about people: not for surveillance, for monitoring the attention of employees, students, drivers or other individuals, for profiling, or for any legal, medical or safety decision. Not for driving or other vehicle or robot control. It estimates a gaze target from the appearance of the head and the scene; it is not an eye tracker, is not calibrated to any person or camera, and does not identify people.
- Not for training or fine-tuning, and not for other Gaze-LLE checkpoints: inference only, with the ViT-B/14
_inoutweights. The ViT-L and GazeFollow-only checkpoints are not packaged. - Needs one head box per person from a separate head or face detector; the package detects no heads. 1 to 10 boxes per image; more are refused.
- One image per request, batch 1, requests served one at a time; each video frame is processed on its own, with no temporal smoothing. Every image is squashed to 448x448 without letterboxing, so the aspect ratio is not kept; EXIF orientation is not applied.
- Tested on GazeFollow and VideoAttentionTarget test images only. Other domains (top-down or fisheye cameras, very small or occluded heads, crowds) were not tested.
- One p150 only (one chip): multi-chip meshes are not supported, and other boards were not tested.
- The device computes in bfloat16, with block-float8 weights in most of the encoder's matrix multiplications. Against the fp32 CPU reference on the first 1,000 GazeFollow images, Avg L2 is 0.0051 higher (0.1166 against 0.1115) while AUC is the same; for 8 % of the heads the heatmap maximum lies more than 0.05 of the image size away from the reference's. The in/out decision differs from the reference for 1.9 % (GazeFollow) and 2.6 % (VideoAttentionTarget) of the heads.
- Image decode and resize run on the host CPU, so the latency of
model()and/predictdepends on the host's CPU and rises with its load; the device time does not.
Licensing
- Weights: changh95/gaze-lle-weights at commit
f83e53f0f17dd2175f31ceaf9fa650044147ada1, downloaded bytt serveat that commit, not redistributed here. It holds the Gaze-LLE decoder checkpointgazelle_dinov2_vitb14_inout.pt(the fkryan/gazelle v1.0.0 release file, MIT) and the DINOv2 ViT-B/14 backbone checkpointdinov2_vitb14_pretrain.pth(Apache-2.0). - Port and serving code (
code/): Apache-2.0. The CPU reference model incode/gaze_lle/reference/is adapted from fkryan/gazelle (MIT; its notice is incode/gaze_lle/THIRD_PARTY_NOTICES.md) and follows facebookresearch/dinov2 (Apache-2.0);code/models/common/lightweightmodule.pyis from tt-metal (Apache-2.0). patches/tt-metal-8b98410.local.patch: the uncommitted changes the image's tt-metal carries on top of the public commit in Provenance; tt-metal is Apache-2.0.media/source_1.png,media/source_2.pngandmedia/source_4.pngare GazeFollow test images 800, 1000 and 2600 (GazeFollow, Recasens et al., NeurIPS 2015, whose images come from SUN, MS COCO, Actions 40, PASCAL, ImageNet and Places), andmedia/target_1.png,media/target_2.pngandmedia/target_4.pngare this package's outputs drawn on them. The photographs belong to their owners and are not covered by this package's licences; they are shown for illustration only, under the terms of GazeFollow and of its source collections.
Feedback
Questions or problems with this package: open a discussion at https://huggingface.co/changh95/gaze-lle-p150/discussions β that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
Provenance
The exact sources the image was built from β code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 8b98410e730bb504fea43a88609756e34821d91d (dirty tree β the image includes uncommitted changes) |
code/ digest |
0d89972ed53d2bce (sha256, first 16 hex digits) |
| built | 2026-11-12T09:08:42+00:00 by tt-model 0.1.0 |
Model tree for changh95/gaze-lle-p150
Base model
changh95/gaze-lle-weights

