adriandj3/TernaryPilot-v1-Computer-Browser-Use-LoRA

TernaryPilot v1 (deprecated) · computer & browser use for Ternary-Bonsai-2-9B

The first version: click, act and read on Windows and the web, trained on a single GTX 1080 with 8 GB.

By Adrian Ciprian Iancu · GitHub · Hugging Face · LinkedIn

Deprecated: TernaryPilot v2 is out. Use adriandj3/TernaryPilot-v2-Computer-Browser-Use-LoRA: same prompts and API (drop-in), trained from scratch on 60,516 examples (GUI-360, AgentNet Windows and macOS, Mind2Web). This page stays for reproducibility and for the GTX 1080 training story.

Results against the base model

Full tables in Original results below.

v1 and v2 on public benchmarks

Same server, same cases, greedy, images at 1024 tokens.

benchmark	cases	base model	v1	v2
ScreenSpot-v2	1,272	46.5%	80.5%	60.7%
ScreenSpot-v2, desktop + web only	771	37.0%	78.6%	71.5%
ScreenSpot-Pro	423	1.7%	6.4%	11.1%
GUI-360 grounding	2,000	16.9%	42.2%	66.4%
GUI-360 actions	1,192	0.5%	17.3%	42.2%

v1 is still ahead on ScreenSpot-v2, whose targets are short element names: its training had dedicated grounding sets of that kind (ShowUI, Wave-UI, GroundCUA). v2 is ahead on professional apps, Office grounding and multi-step actions.

The machine had a fixed stop time, so some runs were cut before the end. ScreenSpot-Pro ran the first 423 of 1,581 cases in the benchmark's file order, which goes application by application, so later applications are missing (not a random sample); GUI-360 actions ran 1,192 of its 2,000 sampled cases, asked in random order, so they are a random subset.

Pick a file

file	for	size	what it is
TernaryPilot-v1-Ternary-Bonsai-2-9B-LoRA-F16.gguf	llama.cpp (PrismML fork) --lora	80 MB	rank 16, alpha 32, F16

What it can do

Skill	You give it	It answers
Click grounding	a screenshot and a target ("save file", "the Donate link")	{"x": 512, "y": 233} on a 0-1000 grid
Next action	a task, the actions done so far, the current screen	a short thought and one action: click(x=, y=), write("..."), press("enter"), hotkey("ctrl+s"), scroll(-3), done() ...
Multi step tasks	the same, step after step	the next step, keeping track of the history
Screen reading	a screenshot	which app or site is open, its state and the important text

Original results (GTX 1080)

Measured on a GTX 1080 with the PrismML llama.cpp fork, greedy decoding, images at 1024 tokens. None of the evaluation screenshots were used in training (also filtered out of the training pool by perceptual hash).

Click grounding	Base model	My earlier grounding LoRA	v1
ScreenSpot-v2, 200 case stratified subset	80 / 200	153 / 200	156 / 200
Real Windows apps in Italian, 37 targets with UI Automation boxes	2 / 37	13 / 37	15 / 37
Real web pages, 210 targets with DOM boxes	28 / 210	99 / 210	101 / 210

Agent actions on 145 held-out steps (85 Windows steps from AgentNet tasks never seen in training, 60 web steps from Mind2Web). Right = same action type and, for actions with coordinates, within 40 units of the true point on the 0-1000 grid.

	Base model	Earlier grounding LoRA	v1
Right actions	7 / 145	28 / 145	67 / 145
Windows	7 / 85	15 / 85	41 / 85
Web	0 / 60	13 / 60	26 / 60
Answers in the expected format	131 / 145	93 / 145	145 / 145

With the adapter at scale 0 outside GUI requests, 12 out of 12 chat prompts give the exact same greedy output as the base model.

How to use it

Same as v2 (see its card): the PrismML fork of llama.cpp, the base model and vision projector from prism-ml/Ternary-Bonsai-2-9B-gguf.

llama-server -m Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf \
  --mmproj Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf \
  --lora-scaled TernaryPilot-v1-Ternary-Bonsai-2-9B-LoRA-F16.gguf:0 \
  --image-max-tokens 1024 -ngl 999 -fa on --jinja -c 16384 --parallel 2

Turn it on per request with "lora": [{"id": 0, "scale": 1}], "temperature": 0, "presence_penalty": 0, and give GUI requests their own slot ("id_slot": 1). The click, action and screen-reading prompts are on the v2 card and did not change.

How I trained it

• Hardware: one NVIDIA GTX 1080 (Pascal, 8 GB, fp32 only, no bf16, no flash attention, no Triton).
• Method: Ternary-Bonsai-2-9B rebuilt exactly in PyTorch from the PQ2_0 GGUF, checked against llama.cpp, LoRA on top of the frozen ternary weights, exported in the GGUF LoRA format.
• Speed work: my CUDA kernel that unpacks the ternary weights in one pass, the 4096 Hadamard rotation as two 64x64 factors, one shared image prefix for several targets of the same screenshot: from 20.2 to about 6 to 9 s per example.
• Run: rank 16, alpha 32, AdamW lr 1e-4, 5.5 hours, about 2,100 examples seen, starting from my earlier grounding LoRA.
• Data: 4,251 examples on 2,079 screenshots: AgentNet Windows actions, clicks and screen reading (MIT), Multimodal-Mind2Web actions and clicks (OpenRAIL), ShowUI-desktop, Wave-UI and GroundCUA clicks.

Full kit and write-up: bonsai2-ternary-pascal/lora-training. Why a LoRA is needed at all (the ternary weights cost most of the pixel-click skill: Qwen3.5-9B 36/38, Ternary Bonsai 19/38 on the same cases): docs/ternary-gui-grounding.md.

Limitations

• Superseded by v2. It saw about half of its small dataset once.
• No safety layer: keep a human in the loop for anything that deletes, buys or sends.
• Responsible use: part of the data comes from Multimodal-Mind2Web under an OpenRAIL license.

Citation

@misc{iancu2026ternarypilot,
  author       = {Iancu, Adrian Ciprian},
  title        = {TernaryPilot v1: a computer use and browser use LoRA for Ternary-Bonsai-2-9B},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/adriandj3/TernaryPilot-v1-Computer-Browser-Use-LoRA}}
}

Thanks to PrismML for Ternary Bonsai and the llama.cpp fork, the OpenCUA team for AgentNet, OSU NLP for Mind2Web, Show Lab for ShowUI, AgentSea for Wave-UI and ServiceNow for GroundCUA.

Keywords: computer use, browser use, GUI agent, web agent, GUI grounding, LoRA, GGUF, llama.cpp, ternary, 1.58-bit, Bonsai, Qwen3.5, ScreenSpot, AgentNet, Mind2Web, Windows automation, local AI, low VRAM, GTX 1080