How to run in ComfyUI + samples
I was able to use loras after patching with Claude using this script
"""Convert LoRAs to ComfyUI text-encoder LoRA key naming.
Source keys : layers.{N}.{path}.lora_a / .lora_b (rank-major / out-major)
Target keys : text_encoders.qwen3vl_8b.transformer.model.layers.{N}.{path}.lora_A.weight.lora_B.weight
Rationale: comfy/lora.py:model_lora_keys_clip builds the key map as "text_encoders." + <te state_dict key without .weight>, and the Ideogram4 TE exposes qwen3vl_8b.transformer.model.layers.N... . comfy/weight_adapter/lora.py only recognises lora_A/lora_B (capitals), lora_up/lora_down, etc.
"""
import sys
from safetensors.torch import load_file, save_file
PREFIX = "text_encoders.qwen3vl_8b.transformer.model."
SUFFIX = {".lora_a": ".lora_A.weight", ".lora_b": ".lora_B.weight"}
def convert(src, dst):
sd = load_file(src)
out = {}
for k, v in sd.items():
for old, new in SUFFIX.items():
if k.endswith(old):
out[PREFIX + k[: -len(old)] + new] = v
break
else:
raise SystemExit(f"unexpected key: {k}")
save_file(out, dst, metadata={"format": "pt"})
print(f"{len(out)} keys -> {dst}")
print("sample:", next(iter(out)))
if __name__ == "__main__":
convert(sys.argv[1], sys.argv[2])
Ive run the model with turbo version on several prompts, here is examples if you're curious:
https://github.com/novmikvis/ideogram-4-prompt-adherence-test
Here is what I've noticed :
- it definitely reduces gray box failure mode (triggers less with small amount of text)
- Lightly reduced prompt adherence and overall coherence (see image with set of icons: plasters became pill blister packs)
- Produces more of a wide-angle shot in realistic scenes
Also here are some additional notes from Claude:
Merged checkpoints may be losing more of the delta than the LoRA path
While comparing the merged encoder against base + LoRA, they read:manifests/merge_step_00001000.json. scale_policy ispreserve_stock_per_tensor_scale, and the per-projection metrics in that same file
show the delta taking a reduction. Across all 252 projections:
| min | p25 | median | max | |
|---|---|---|---|---|
delta_retention_norm_ratio |
0.724 | 0.872 | 0.915 | 1.017 |
delta_cosine |
0.553 | 0.688 | 0.731 | 0.920 |
157 of 252 projections land below 0.75 cosine; the weakest islayers.0.self_attn.k_proj at 0.553 cosine / 0.724 retention.
That looks like a consequence of keeping the base model's per-tensor FP8 scales: the
adapted weights no longer fit the range those scales were fitted for. ComfyUI's runtime
LoRA path does preserves them — comfy/ops.py convert_weight dequantises, the delta is
added in a higher-precision compute dtype, and set_weight requantises withscale="recalculate" plus stochastic rounding. So applying the adapter at runtime may
well preserve more of what you trained than the pre-merged file does.
Might be worth re-merging with recalculated scales and comparing — if the metrics in
the manifest are computed the way I'm reading them, there could be some free quality
sitting there.
I will make a note to take a look at that and desperately hope I remember to check my notes
Following up for later
Investigation handoff: ComfyUI LoRA path vs. merged FP8 encoder
I investigated the converted LoRA, the published merge manifests, current ComfyUI code, and the prompt-adherence comparison images.
Findings so far
The LoRA conversion appears correct for this adapter:
- lora_a is rank-major and correctly maps to lora_A.weight.
- lora_b is output-major and correctly maps to lora_B.weight.
- The text_encoders.qwen3vl_8b.transformer.model. prefix matches ComfyUI’s generic text-encoder LoRA mapping.
- This adapter is rank 64 / alpha 64, so omitting explicit .alpha keys still gives the intended effective multiplier of 1. This conversion would need adjustment for adapters where alpha differs from rank.
- A useful runtime validation is that ComfyUI reports all 504 LoRA tensors/252 projections loaded with no “key not loaded” warnings.
Relevant ComfyUI code:
- Text-encoder key mapping
- lora_A/lora_B handling
Claude’s description of the current ComfyUI runtime path is substantially correct. For a quantized weight, ComfyUI:
- Dequantizes the base weight.
- Applies the LoRA in its LoRA compute dtype—normally FP16 on modern GPUs, otherwise FP32.
- Requantizes with scale="recalculate" and seeded stochastic rounding.
Relevant code:
- LoRA patching path
- Dequantize/requantize implementation
- LoRA compute dtype
The quoted step-1000 manifest statistics were independently verified:
- 252 adapted projections
- Median delta-norm retention: approximately 0.915
- Median delta cosine: approximately 0.731
- 157/252 projections below 0.75 cosine
- Worst projection: layers.0.self_attn.k_proj, approximately 0.724 retention and 0.553 cosine
The same pattern is present throughout the released lineage. From steps 510–5000, median delta cosine remains around 0.72–0.73, with approximately 154–174 projections below 0.75. This is systematic rather than unique to step 1000.
However, there is an important caveat to the “recalculate scales for free quality” theory: the adapter was explicitly trained quantization-aware. According to the model card, its training forward formed the dequantized stock weight plus the LoRA delta and then requantized using the stock per-tensor FP8 scale. Therefore, the published fixed-scale merge intentionally matches the train-time forward path.
A recalculated-scale merge may represent the floating-point base + LoRA weights more closely, but it is not automatically more faithful to the behavior optimized during training. Recalculating also changes quantization error across the entire base tensor. It can remove clipping while making the FP8 grid coarser for most weights.
To establish that the original scales no longer fit the adapted weights, the manifests would ideally include:
- Saturated/clipped value counts
- Adapted maximum divided by the stock representable maximum
- Error with fixed versus recalculated scales
- Text-encoder activation or Ideogram velocity error—not only weight-delta cosine
Image-test interpretation
The comparison set is valuable paired evidence: it keeps the seed, sampler and prompt fixed while changing the adapter checkpoint.
Observed examples support several qualitative claims:
- The realistic portrait becomes noticeably more loosely framed with the LoRA.
- At step 1000, the requested plasters become a pill blister pack.
- Steps 3010 and 5000 return to something bandage-like, so that substitution is not monotonic across checkpoints.
- The framing tendency may be a learned teacher bias: the adapter distills LLM-generated structured captions, which often encode explicit lens, shot-size and compositional priors.
The gray-box reduction is expected. Official Ideogram documentation says raw plain text often produces the safety-warning image because the model was trained exclusively on structured JSON captions. Moving plain-text conditioning toward the JSON-teacher manifold should reduce those false positives.
There is also a safety caveat: reducing the learned gray-box response could weaken true positives as well. The adapter corpus deliberately includes R/X/XXX source-rating groups, so application-level safety screening remains important.
The current comparison should still be treated as exploratory:
- It has 16 prompts but only one seed per prompt.
- Changing conditioning can substantially redirect a diffusion trajectory even with the same initial seed.
- The prompts are approximately 270–600 words, so this does not yet evaluate genuinely short natural-language prompts.
- Training used cached 512×512 V4_DEFAULT_20 trajectories, while the comparison uses multiple larger resolutions with the Turbo/Default-20 configuration.
Recommended next experiment
Compare the same checkpoint through five paths:
- Stock FP8 encoder
- Stock FP8 plus runtime LoRA
- Published fixed-stock-scale FP8 merge
- Locally produced recalculated-scale FP8 merge
- BF16 or FP16 merge as a reference, if memory permits
Use identical workflows and at least 8–16 seeds per prompt. Include separate suites for:
- Truly short prompts
- Long descriptive prompts
- In-image text/OCR
- Exact object counts
- Spatial relations
- Photographic framing
- Benign prompts prone to gray-box failures
- A controlled safety set
Record:
- Gray-box rate
- OCR character/word accuracy
- Object/count and spatial compliance
- Blind pairwise coherence/quality preferences
- Text-encoder tap differences
- Ideally, conditional Ideogram velocity MSE against the structured teacher
The main unresolved question is therefore not whether recalculated scales retain the raw floating-point delta better—they plausibly do—but whether that produces conditioning closer to the teacher and better images than the quantization behavior used during training.