gaze-lle-p150 on Tenstorrent p150

Derived from Gaze-LLE (fkryan/gazelle) by Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman and James M. Rehg: the gazelle_dinov2_vitb14_inout checkpoint on the DINOv2 ViT-B/14 backbone (facebookresearch/dinov2), with unmodified weights mirrored at changh95/gaze-lle-weights. Paper: arXiv:2412.09586. Port: changh95/tt-Gaze-LLE. See the upstream repository for training and evaluation details.

Gaze-LLE ViT-B/14 with the in/out head (gazelle_dinov2_vitb14_inout): a frozen DINOv2 ViT-B/14 encoder (86M parameters) and a 2.8M-parameter gaze decoder (a head-position prompt added to the scene features, then 3 transformer layers), trained on GazeFollow and fine-tuned on VideoAttentionTarget. One RGB image goes in and is squashed to 448x448, together with 1 to 10 head boxes, one per person. Out come, for each person, a 64x64 heatmap of where the person is looking over the whole image, the gaze target point (the heatmap maximum, in pixels of the original image) and the probability that the gaze target is inside the frame. The encoder runs once per image and all heads share it.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware p150
License MIT and Apache-2.0

Intended use

Gaze-target estimation (where each person in an image is looking) for research, prototyping and offline analysis of images and video frames, given one head box per person from a separate head or face detector.

Prerequisites

  • The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
  • Docker
  • Tenstorrent hardware: p150

Quickstart

tt serve changh95/gaze-lle-p150

tt serve downloads the Docker image and the changh95/gaze-lle-weights at f83e53f0f17dd2175f31ceaf9fa650044147ada1 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/gaze-lle-p150.

Send the sample image of this repo with its three head boxes to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/gaze-lle-p150/resolve/main/media/source_1.png
printf '{"image":"%s","bboxes":[[116,28,176,104],[269,37,301,81],[87,57,107,76]]}' "$(base64 -w0 source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β€” its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one image and the head boxes of the people in it, and returns one gaze prediction per box. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

request field (POST /predict) meaning
image base64 PNG or JPEG of one image, any size; a data:image/...;base64, prefix is accepted
bboxes 1 to 10 head boxes [x1, y1, x2, y2], one per person; clamped to the image, and each must span more than 1/32 of the image in both axes
bbox_format (optional) pixels (default: pixels of the original image) or normalized (fractions of width and height)
heatmap_format (optional) png (default: one base64 8-bit grayscale 64x64 PNG per head, uint8(round(p * 255))), npz (one base64 .npz with heatmap float32 (N, 64, 64) and inout float32 (N,)), json (nested lists, 5 decimals) or none

Response for the sample image (media/source_1.png, 500x334) and the three boxes of the quickstart (heatmaps elided):

{"model": "gaze-lle-p150", "image_size": [500, 334], "input_size": [448, 448], "num_heads": 3,
 "heatmap_size": [64, 64], "heatmap_format": "png", "heatmap_note": "...",
 "predictions": [
   {"bbox": [116.0, 28.0, 176.0, 104.0], "bbox_normalized": [0.232, 0.083832, 0.352, 0.311377], "inout": 0.037841796875,
    "gaze_target": {"x": 289.0625, "y": 172.21875, "x_norm": 0.578125, "y_norm": 0.515625, "heatmap_row": 33, "heatmap_col": 37, "score": 0.349609375},
    "heatmap_png": "..."},
   {"bbox": [269.0, 37.0, 301.0, 81.0], "bbox_normalized": [0.538, 0.110778, 0.602, 0.242515], "inout": 0.75390625,
    "gaze_target": {"x": 296.875, "y": 99.15625, "x_norm": 0.59375, "y_norm": 0.296875, "heatmap_row": 19, "heatmap_col": 38, "score": 0.515625},
    "heatmap_png": "..."},
   {"bbox": [87.0, 57.0, 107.0, 76.0], "bbox_normalized": [0.174, 0.170659, 0.214, 0.227545], "inout": 0.67578125,
    "gaze_target": {"x": 93.75, "y": 62.625, "x_norm": 0.1875, "y_norm": 0.1875, "heatmap_row": 12, "heatmap_col": 12, "score": 0.212890625},
    "heatmap_png": "..."}],
 "timing_ms": {"preprocess": 9.75, "inference": 3.8, "encode": 1.04, "total": 14.59}}
  • heatmap values are probabilities in [0, 1] on a 64x64 grid that covers the whole original image (row = y, column = x). gaze_target is its maximum: x = column / 64 * width, y = row / 64 * height in pixels of the original image, x_norm / y_norm the same as fractions, score the maximum value.
  • inout is the probability that the person's gaze target is inside the frame. Below 0.5 the person is predicted to look outside the image, and gaze_target is then not meaningful.
  • bbox echoes the box as sent; bbox_normalized is the box as used (fractions, clamped to the image).
  • timing_ms gives the server-side stages of the request: preprocess (base64 and image decode, resize to 448x448 on the host CPU), inference (upload, device run, readback), encode (heatmap encoding).
  • Errors: 400 for an image that cannot be decoded, more than 10 boxes, an empty or too small box, or invalid request fields; 503 while the server is starting.
  • A Python API with the same outputs (gaze_lle.GazeLLE) is described in code/PYTHON.md, with an example in code/examples/quickstart.py. It needs a host ttnn build of the tt-metal commit in Provenance with patches/tt-metal-8b98410.local.patch applied.

Example output

Gaze-LLE on a Tenstorrent p150: the three predictions of the response above drawn on the sample image Gaze-LLE on a Tenstorrent p150: prediction for the annotated person of GazeFollow test image 1000 Gaze-LLE on a Tenstorrent p150: prediction for the annotated person of GazeFollow test image 2600

Outputs of this package drawn with gaze_lle.visualize: every head box with its in/out probability, and for each head with inout at least 0.5 its heatmap and a line from the box centre to the predicted gaze target. The first image is the response above (the sample image with the three quickstart boxes); the other two are GazeFollow test images 1000 and 2600 with the head box of their annotated person. Inputs: media/source_1.png, media/source_2.png and media/source_4.png (see Licensing).

Expected performance

Gaze-target estimation on the two benchmarks the Gaze-LLE paper reports, and agreement with the same model run in fp32 on the CPU:

eval this package reference images
VideoAttentionTarget test: AUC / L2 / AP in-out 0.9316 / 0.1084 / 0.8986 0.933 / 0.107 / 0.897 (published, this checkpoint, all 13,127 test frames) 1,384 frames (3,355 heads)
VideoAttentionTarget test, same frames: AUC / L2 / AP in-out 0.9316 / 0.1084 / 0.8986 0.9330 / 0.1072 / 0.9014 (fp32 CPU reference) 1,384 frames
GazeFollow test: AUC / Avg L2 / Min L2 0.9542 / 0.1129 / 0.0508 0.956 / 0.104 / 0.045 (published for the GazeFollow-only checkpoint gazelle_dinov2_vitb14, not this one) 4,782
GazeFollow test, first 1,000 images: AUC / Avg L2 / Min L2 0.9530 / 0.1166 / 0.0530 0.9529 / 0.1115 / 0.0498 (fp32 CPU reference) 1,000
Heatmap PCC against the fp32 CPU reference, mean / median 0.987 / 0.993 (GazeFollow), 0.985 / 0.993 (VideoAttentionTarget) – 1,000 / 1,384
In/out decision (inout β‰₯ 0.5) equal to the fp32 CPU reference 98.1 % (GazeFollow), 97.4 % (VideoAttentionTarget) – 1,000 / 1,384

The published figures are the "Gaze-LLE (ViT-B)" row of Table 3 of the paper. Its VideoAttentionTarget columns are for the checkpoint served here (gazelle_dinov2_vitb14_inout); its GazeFollow columns are for the GazeFollow-only checkpoint. VideoAttentionTarget (ejcgt/attention-target-detection): every 10th frame of each of the 131 test clips. GazeFollow: the whole test set (the vikhyatk/gazefollow copy), one annotated person per image. Metrics and data preparation follow fkryan/gazelle. The fp32 CPU reference is the PyTorch Gaze-LLE model of code/gaze_lle/reference/ (adapted from fkryan/gazelle) with the same weights and the upstream preprocessing.

speed (batch 1, warm) input median p99 throughput
Device time per image, back-to-back trace replays 448x448, 1 head 2.79 ms 2.79 ms 359 images/s
Device time per image, back-to-back trace replays 448x448, 3 heads 3.07 ms 3.07 ms 326 images/s
Device time per image, back-to-back trace replays 448x448, 10 heads 4.10 ms 4.13 ms 244 images/s
Python model() call (upload, device run, readback, output decoding) uint8 RGB array 448x448, 1 head 3.25 ms 3.70 ms 307 images/s
Python model() call uint8 RGB array 448x448, 3 heads 3.60 ms 4.58 ms 278 images/s
Python model() call uint8 RGB array 448x448, 10 heads 4.61 ms 12.9 ms 217 images/s
Python model() call, also decoding and resizing the file sample PNG 500x334, 3 heads 14.2 ms 17.9 ms 70 images/s
Served POST /predict, server timing_ms.total sample PNG 500x334, 1 head 12.0 ms 16.1 ms –
Served POST /predict, client round trip on loopback sample PNG 500x334, 1 head 14.9 ms 19.5 ms –
Served POST /predict, server timing_ms.total sample PNG 500x334, 3 heads 12.6 ms 16.6 ms –
Served POST /predict, client round trip on loopback sample PNG 500x334, 3 heads 15.5 ms 19.9 ms –

Measured with this package's image on one p150 in an 8-vCPU AMD EPYC-Rome host (host load average about 10 to 13 during the runs), with code/scripts/bench_p150.py (500 calls per Python row; each device row is 20 rounds of 500 replays) and code/scripts/bench_http.py (1,000 requests per served row, one at a time on one keep-alive connection, PNG heatmaps); of the served 3-head request, base64 and PNG decode and the resize on the host take 7.9 ms (median).

Limitations

  • Not for decisions about people: not for surveillance, for monitoring the attention of employees, students, drivers or other individuals, for profiling, or for any legal, medical or safety decision. Not for driving or other vehicle or robot control. It estimates a gaze target from the appearance of the head and the scene; it is not an eye tracker, is not calibrated to any person or camera, and does not identify people.
  • Not for training or fine-tuning, and not for other Gaze-LLE checkpoints: inference only, with the ViT-B/14 _inout weights. The ViT-L and GazeFollow-only checkpoints are not packaged.
  • Needs one head box per person from a separate head or face detector; the package detects no heads. 1 to 10 boxes per image; more are refused.
  • One image per request, batch 1, requests served one at a time; each video frame is processed on its own, with no temporal smoothing. Every image is squashed to 448x448 without letterboxing, so the aspect ratio is not kept; EXIF orientation is not applied.
  • Tested on GazeFollow and VideoAttentionTarget test images only. Other domains (top-down or fisheye cameras, very small or occluded heads, crowds) were not tested.
  • One p150 only (one chip): multi-chip meshes are not supported, and other boards were not tested.
  • The device computes in bfloat16, with block-float8 weights in most of the encoder's matrix multiplications. Against the fp32 CPU reference on the first 1,000 GazeFollow images, Avg L2 is 0.0051 higher (0.1166 against 0.1115) while AUC is the same; for 8 % of the heads the heatmap maximum lies more than 0.05 of the image size away from the reference's. The in/out decision differs from the reference for 1.9 % (GazeFollow) and 2.6 % (VideoAttentionTarget) of the heads.
  • Image decode and resize run on the host CPU, so the latency of model() and /predict depends on the host's CPU and rises with its load; the device time does not.

Licensing

  • Weights: changh95/gaze-lle-weights at commit f83e53f0f17dd2175f31ceaf9fa650044147ada1, downloaded by tt serve at that commit, not redistributed here. It holds the Gaze-LLE decoder checkpoint gazelle_dinov2_vitb14_inout.pt (the fkryan/gazelle v1.0.0 release file, MIT) and the DINOv2 ViT-B/14 backbone checkpoint dinov2_vitb14_pretrain.pth (Apache-2.0).
  • Port and serving code (code/): Apache-2.0. The CPU reference model in code/gaze_lle/reference/ is adapted from fkryan/gazelle (MIT; its notice is in code/gaze_lle/THIRD_PARTY_NOTICES.md) and follows facebookresearch/dinov2 (Apache-2.0); code/models/common/lightweightmodule.py is from tt-metal (Apache-2.0).
  • patches/tt-metal-8b98410.local.patch: the uncommitted changes the image's tt-metal carries on top of the public commit in Provenance; tt-metal is Apache-2.0.
  • media/source_1.png, media/source_2.png and media/source_4.png are GazeFollow test images 800, 1000 and 2600 (GazeFollow, Recasens et al., NeurIPS 2015, whose images come from SUN, MS COCO, Actions 40, PASCAL, ImageNet and Places), and media/target_1.png, media/target_2.png and media/target_4.png are this package's outputs drawn on them. The photographs belong to their owners and are not covered by this package's licences; they are shown for illustration only, under the terms of GazeFollow and of its source collections.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/gaze-lle-p150/discussions β€” that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from β€” code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal 8b98410e730bb504fea43a88609756e34821d91d (dirty tree β€” the image includes uncommitted changes)
code/ digest 0d89972ed53d2bce (sha256, first 16 hex digits)
built 2026-11-12T09:08:42+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for changh95/gaze-lle-p150

Finetuned
(1)
this model

Collection including changh95/gaze-lle-p150

Paper for changh95/gaze-lle-p150