Instructions to use pragmaticcs/PentaCoder-9B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pragmaticcs/PentaCoder-9B-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="pragmaticcs/PentaCoder-9B-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("pragmaticcs/PentaCoder-9B-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use pragmaticcs/PentaCoder-9B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use pragmaticcs/PentaCoder-9B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "pragmaticcs/PentaCoder-9B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pragmaticcs/PentaCoder-9B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
- SGLang
How to use pragmaticcs/PentaCoder-9B-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "pragmaticcs/PentaCoder-9B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pragmaticcs/PentaCoder-9B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "pragmaticcs/PentaCoder-9B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pragmaticcs/PentaCoder-9B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use pragmaticcs/PentaCoder-9B-GGUF with Ollama:
ollama run hf.co/pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use pragmaticcs/PentaCoder-9B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use pragmaticcs/PentaCoder-9B-GGUF with Docker Model Runner:
docker model run hf.co/pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
- Lemonade
How to use pragmaticcs/PentaCoder-9B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.PentaCoder-9B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use pragmaticcs/PentaCoder-9B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use pragmaticcs/PentaCoder-9B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "pragmaticcs/PentaCoder-9B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Architectural Specifications
- Composition & Triad Specialization
- Merge Methodology & Mathematical Formulation (HSCM-DP)
- 1. Anchor-Relative Task Formulation
- 2. Continuous Depth Schedule & Partition of Unity
- 3. Gram-Schmidt Orthogonal Task Disentanglement
- 4. MoE Router Gate Frobenius Isometry (Entropy Calibration)
- 5. Outlier Feature & Channel-Kurtosis Shielding
- 6. Row-Wise Continuous Directional Forget-Gating
- 7. Truncated SVD Denoising of Dense Projections
- 8. Expert-Salience Dynamic Routing & Pareto Pruning
- 9. Contractive Spectral-Radius Clamping for DeltaNet Recurrence
- 10. Geodesic Norm Conservation
- 1. Anchor-Relative Task Formulation
- Layer-Stratified Component Policies
- Agentic Chat Template & Multi-Tier CoT
- Recommended Generation Parameters
- Citation & References
Patreon Exclusive: This model available exclusively to my free tier Patreon members. To get access, become a member for free and checkout the post with a special secret link which will automatically grant you access.
This repository provides quantized GGUF builds of pragmaticcs/PentaCoder-9B
Most coding fine-tunes generate clean snippets in isolation but choke on real developer workflows: running bash commands, parsing runtime errors, and navigating large repositories. Generalist reasoning models have the opposite problem: solid high-level logic, but brittle git/diff formatting and inconsistent tool calls.
Qwen3.8-KAT-AgentWorld-35B-A3B-Coder merges three specialized models on the Qwen 3.5 35B A3B architecture into a fast, practical coding workhorse:
- Kwaipilot/KAT-Coder-V2.5-Dev (Anchor): SOTA open coding core (69.4% SWE-bench Verified), providing raw patch generation, refactoring, and code synthesis.
- Qwen/Qwen-AgentWorld-35B-A3B (Donor 1): Trained on 10M+ trajectories across terminal, SWE, MCP, and OS tasks to keep tool use disciplined and prevent repetition loops when commands fail.
- empero-ai/Qwen3.8-35B-A3B-Distill (Donor 2): Frontier reasoning and algorithmic problem-solving to handle complex logic without prompt brittleness.
Contents
- Architectural Specifications
- Composition & Triad Specialization
- Merge Methodology & Mathematical Formulation (HSCM-DP)
- Layer-Stratified Component Policies
- Agentic Chat Template & Multi-Tier CoT
- Recommended Generation Parameters
- Citation & References
Architectural Specifications
| Parameter | Specification |
|---|---|
| Total Parameters | 34.7B (Text Backbone; Vision & MTP excised) |
| Active Parameters per Token | ~3.3B (8 Routed Experts + 1 Shared Expert) |
| Architecture Class | Hybrid Recurrent-Attention MoE Causal LM (Qwen3_5MoeForCausalLM) |
| Hidden Dimension (dmodel) | 2048 |
| Intermediate Dimension (dintermediate) | 512 per routed expert (131,072 collective) / 2048 shared expert |
| Decoder Layers | 40 |
| Routed Experts | 256 Total (Top-8 routing via softmax gate) |
| Shared Experts | 1 Unconditional Shared Highway Expert |
| Attention Topology | 10 Blocks × (3 Gated DeltaNet Recurrent Layers : 1 Gated Softmax Layer) |
| Full Attention Layers | Layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39 |
| Linear Attention Configuration | 16 Key Heads / 32 Value Heads (dk = dv = 128) |
| Full Attention Configuration | 16 Query Heads / 2 Key-Value Heads (GQA, dh = 256) |
| Position Embedding | YaRN RoPE (θ = 10⁷, Factor = 4.0, Original = 262,144) |
| Context Window Length | 262,144 tokens (256k) |
| Native Precision | bfloat16 |
| Vocabulary Size | 248,320 (Synchronized for tool-calling and CoT tokens) |
Composition & Triad Specialization
The architecture treats KAT-Coder-V2.5-Dev as the topological parameter anchor (W₀). Two complementary donor models inject non-overlapping cognitive subspaces governed by a continuous depth schedule over normalized layer index u = l / 39, where l ∈ [0, 39]:
| Model | Structural Role | Primary Capabilities | Depth Priority |
|---|---|---|---|
| Kwaipilot/KAT-Coder-V2.5-Dev | *Anchor Baseline (W₀)* | SOTA 69.4% SWE-bench verified patch generation, repo AST parsing, code diff synthesis, refactoring purity. | Global Foundation (50%–75%) |
| Qwen/Qwen-AgentWorld-35B-A3B | Environment Donor (Δ₁) | Language world modeling, terminal execution anticipation, multi-turn tool schema integrity, error rollback heuristics. | Deep Output Layers (Ascending to 35%) |
| empero-ai/Qwen3.8-35B-A3B-Distill | Reasoning Donor (Δ₂) | Frontier CoT distillation, competitive algorithmic decomposition, math logic, multi-language semantic grounding. | Central Decoder Core (Peaking at 30%) |
Merge Methodology & Mathematical Formulation (HSCM-DP)
Standard merging strategies (such as naive SLERP or uniform DARE-TIES) degrade complex MoE hybrid networks. In a 35B A3B backbone, uniform coordinate dropping corrupts the recurrent eigenvalues of linear attention kernels and induces routing collapse across the 256 expert gates.
The HSCM-DP (Heterogeneous Subsystem Consensus Merging with DeltaNet Preservation) engine solves these failure modes by dividing the parameter space into distinct mathematical manifolds.
1. Anchor-Relative Task Formulation
Task delta vectors are computed strictly relative to the verified coding anchor W₀:
2. Continuous Depth Schedule & Partition of Unity
Donor mixing weights are continuously modulated across layer depth l ∈ [0, 39]:
The scalar coefficients are normalized such that:
This schedule ensures the coding specialist (W₀) retains majority control throughout the stack, while Qwen3.8 provides deep reasoning in the middle layers, and AgentWorld directs terminal command emission at the final projection layers.
3. Gram-Schmidt Orthogonal Task Disentanglement
Fine-tuned models frequently share conversational alignment drift, creating collinear task vectors that cause destructive cancelation or over-amplification. The shared collinear component is projected out of Donor 2:
By injecting only the strictly orthogonal delta Δ₂⊥, the model absorbs 100% of Qwen3.8's unique algorithmic reasoning without double-counting shared fine-tuning updates.
4. MoE Router Gate Frobenius Isometry (Entropy Calibration)
When high-dimensional routing gate matrices (Wgate ∈ ℝ2048 × 256) are linearly blended, Euclidean norm shrinkage contracts the logit scale:
In a softmax routing gate, smaller logit magnitudes act as an artificial temperature spike:
This causes uniform expert firing and expert starvation. HSCM-DP enforces Frobenius Norm Isometry anchored on KAT-Coder:
This mathematically guarantees that the routing entropy, sharpness, and expert dispatch calibration of the base coding specialist are preserved.
5. Outlier Feature & Channel-Kurtosis Shielding
Large language models exhibit emergent outlier feature channels (Z > 3.0) along the input dimension that maintain code indentation, markdown block delimiters, and AST tree boundaries.
For projection matrices W₀ ∈ ℝdout × din, column-wise L₂ norms are evaluated:
Donor updates along channels with Zj > 3.0 are attenuated by a protection factor:
This prevents donor perturbation from degrading whitespace tracking and syntax formatting.
6. Row-Wise Continuous Directional Forget-Gating
When fine-tuning on conversational datasets, certain neuron rows rotate counter to core coding representations. Rather than applying a binary sign mask, HSCM-DP evaluates row-wise directional cosine alignment:
Neurons exhibiting counter-directional drift (cos θi < 0) are exponentially dampened:
This allows donors to enrich feature representations while preventing them from inverting established coding weights.
7. Truncated SVD Denoising of Dense Projections
In dense attention and shared expert projections, task vectors are projected into their principal low-rank singular subspace to remove high-frequency optimizer noise:
The spectrum is truncated to retain 85% of the total Frobenius energy (S²):
8. Expert-Salience Dynamic Routing & Pareto Pruning
For the 256 routed experts (dintermediate = 512), standard SVD would reduce representational capacity. Instead, experts are merged using Relative Drift Salience:
Dynamic blending coefficients for expert e are calculated as:
Surviving weights are filtered via deterministic Pareto energy thresholding (Eratio = 0.90) and smoothed using continuous magnitude-weighted consensus damping:
If AgentWorld extensively modified an expert (e.g., shell parsing), that expert receives a higher donor weight. If an expert was untouched, KAT-Coder maintains full control.
9. Contractive Spectral-Radius Clamping for DeltaNet Recurrence
In Gated DeltaNet layers (30 of the 40 layers), linear attention relies on the state-space recurrent update:
To prevent numerical divergence over long contexts, the recurrent transition operators (linear_attn kernel and decay projections) must remain contractive (spectral radius σmax ≤ 1.0).
Using power iteration, the dominant singular values of the anchor and merged matrices are computed:
If the merged matrix exceeds the anchor's spectral bound, it is scaled down:
This ensures numerical stability across the entire 262,144 token context window.
10. Geodesic Norm Conservation
Linear combinations in high dimensions contract the matrix norm. For all projections, the output matrix is rescaled to match the target energy norm:
This prevents hidden-state variance decay across the 40 decoder layers.
Layer-Stratified Component Policies
| Parameter Subsystem | Target Identifiers | Applied Policy | Density / Rank | Mathematical Invariants |
|---|---|---|---|---|
| Embeddings & LM Head | embed_tokens, lm_head |
Geodesic Norm Blend | 1.0 | Convex projection; vocabulary dimension aligned to 248,320. |
| MoE Router Gates | mlp.gate.weight |
Frobenius Norm Isometry | 1.0 (No DARE) | Logit scale matched to |W₀|F; preserves softmax routing entropy. |
| Linear Attention (DeltaNet) | linear_attn.* |
DARE-Free Smooth Addition | 1.0 | Power-iteration spectral clamping guarantees σmax ≤ σ₀. |
| Full Attention Projections | self_attn.q/k/v/o_proj |
GS-Orthogonal + SVD Denoise | Rank ≤ 128 | Kurtosis protection (Z > 3.0); 85% Frobenius energy retention. |
| MoE Shared Expert | mlp.shared_expert.* |
GS-Orthogonal + SVD Denoise | Rank ≤ 128 | Provides common syntactic grounding across all tokens. |
| 256 Routed Experts | mlp.experts.* |
Salience Routing + Soft TIES | 90% Energy | Pareto pruning; consensus agreement damping (A1.5). |
| Normalization Layers | *layernorm*, norm |
Convex Weighted Average | 1.0 | Preserves variance calibration across residual layers. |
Agentic Chat Template & Multi-Tier CoT
This model uses the Improved Chat Template for Qwen 3.x to support multi-tier Chain-of-Thought reasoning, dual-format agentic tool execution (JSON and XML), automatic error recovery, and strict token-waste elimination.
Template Features
- Multi-Tier
<thought>Reasoning: Supports explicit internal reasoning traces before tool calls and final code synthesis. - Dual Tool Calling Schemas: Compatible with both JSON-RPC function calls and XML tool tags for terminal, editor, and search operations.
- Circuit-Breaker Recovery: Suppresses repetition loops when shell commands or linters return non-zero exit codes.
Recommended Generation Parameters
For repository-level software engineering, bug localization, and automated tool use:
| Parameter | Recommended Value | Description |
|---|---|---|
| Temperature | 0.6 |
Balances strict syntactic validity with algorithmic path exploration. |
| Top-P | 0.95 |
Eliminates low-probability token tails while preserving alternative logic paths. |
| Top-K | 20 |
Restricts token candidate pools to prevent architectural syntax drift. |
| Min-P | 0.0 (Off) |
Disabled in favor of explicit Top-K / Top-P governance. |
| Repetition Penalty | 1.0 (Off) |
Disabled to prevent syntax degradation in repetitive code patterns (indentation, braces). |
| Presence Penalty | 0.0 |
Prevents naming mutations across long-context symbol resolution. |
Citation & References
- Kwaipilot/KAT-Coder-V2.5-Dev
- Qwen/Qwen-AgentWorld-35B-A3B
- empero-ai/Qwen3.8-35B-A3B-Distill
- Improved Chat Template for Qwen 3.x
@inproceedings{yadav2023ties,
title={Resolving Interference When Merging Models},
author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
volume={36},
pages={7093--7115},
year={2023}
}
@article{yu2024dare,
title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
author={Yu, Le and Yu, Bowman and Yu, Haiyang and Huang, Fei and Li, Yongbin},
journal={arXiv preprint arXiv:2311.03099},
year={2023}
}
@article{yang2024deltanet,
title={Gated Delta Networks: Improving Mamba2 with Delta Rule for Linear Recurrent Attention},
author={Yang, Songlin and Wang, Bailin and Shen, Yikang and Panda, Rameswar and Kim, Yoon},
journal={arXiv preprint arXiv:2412.06464},
year={2024}
}
- Downloads last month
- 627
4-bit
5-bit
8-bit
