Instructions to use TypeScript/FastBrain-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TypeScript/FastBrain-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TypeScript/FastBrain-9B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("TypeScript/FastBrain-9B")
model = AutoModelForMultimodalLM.from_pretrained("TypeScript/FastBrain-9B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TypeScript/FastBrain-9B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TypeScript/FastBrain-9B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TypeScript/FastBrain-9B:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf TypeScript/FastBrain-9B:Q4_K_M

Use Docker

docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use TypeScript/FastBrain-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TypeScript/FastBrain-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TypeScript/FastBrain-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M

SGLang

How to use TypeScript/FastBrain-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TypeScript/FastBrain-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TypeScript/FastBrain-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TypeScript/FastBrain-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TypeScript/FastBrain-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use TypeScript/FastBrain-9B with Ollama:
```
ollama run hf.co/TypeScript/FastBrain-9B:Q4_K_M
```
Unsloth Desktop

How to use TypeScript/FastBrain-9B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TypeScript/FastBrain-9B:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TypeScript/FastBrain-9B:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use TypeScript/FastBrain-9B with Docker Model Runner:
```
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
```

Lemonade

How to use TypeScript/FastBrain-9B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TypeScript/FastBrain-9B:Q4_K_M

Run and chat with the model

lemonade run user.FastBrain-9B-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use TypeScript/FastBrain-9B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TypeScript/FastBrain-9B:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TypeScript/FastBrain-9B:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use TypeScript/FastBrain-9B with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TypeScript/FastBrain-9B:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TypeScript/FastBrain-9B:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

TypeScript commited on 2 days ago

Commit

05dde44

verified ·

1 Parent(s): 91661ee

82 unsafe trajectories caught)

Browse files

Files changed (3) hide show

README.md +20 -14
eval_submissions/TypeScript__FastBrain-9B.category.jsonl +0 -0
eval_submissions/TypeScript__FastBrain-9B.jsonl +0 -0

README.md CHANGED Viewed

@@ -32,7 +32,7 @@ tags:
 [![Architecture](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.4B_Total)-3b82f6?style=flat-square)](#architecture)
 [![Decision Head](https://img.shields.io/badge/Head-JointSchemaHead_(116M)-10b981?style=flat-square)](#architecture)
 [![Decision Index](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(%232_Overall_%C2%B7_%231_9B)-f59e0b?style=flat-square)](#benchmark-evaluation--official-leaderboard-rankings)
-[![SafeAgents Judge](https://img.shields.io/badge/SafeAgents_Judge-97.4%25_Bal_Acc_(%231_%2F_9_Models)-ec4899?style=flat-square)](#external-community-benchmarks-safeagents-judge-bench--localllamatyped-decisions)
 [![Typed Decisions](https://img.shields.io/badge/Typed_Decisions-%237_%2F_42_Zero--Shot_Live_(%231_Clef--Arch)-8b5cf6?style=flat-square)](#external-community-benchmarks-safeagents-judge-bench--localllamatyped-decisions)
 [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
@@ -52,8 +52,8 @@ All fine-tuned backbone updates (`104` projection layers across all `32` hybrid
 | Official Leaderboard / Track | Total Models | **`FastBrain-9B` Score** | **Official Rank** | **Top Percentile** | Key Models Outperformed (`Params` & Score) |
 | :--- | :---: | :---: | :---: | :---: | :--- |
-| **[`safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench) *(Binary Guardrail)***<br>([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 Models | **`97.4%` Bal. Acc**<br>*(Distilled Raw: `94.9%`)* | **#1 / 9** | **Top 11.1% *(Rank #1)*** | `Cloudflare/clef-omni` *(30B, `84.5%`)*, `LiquidAI/d1-3B` *(`82.3%`)*, `Cloudflare/clef-flash` *(9.4B, `72.1%`)*, `Cloudflare/clef` *(27.4B, `71.8%`)* |
-| **[`safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench) *(7-Way Category)***<br>([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | 9 Models | **`85.1%` Macro-F1**<br>*(Distilled Raw: `62.4%`)* | **#1 / 9** | **Top 11.1% *(Rank #1)*** | `Cloudflare/clef` *(27.4B, `42.5%`)*, `vllm-sr/d3-mini` *(`36.7%`)*, `Cloudflare/clef-flash` *(9.4B, `34.5%`)*, `Cloudflare/clef-omni` *(30B, `32.7%`)* |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot Overall)***<br>([Merged PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [Trace](./eval_submissions/typescript_fastbrain-9b.json)) | **42 Models** | **`75.25%` / `76.25%` Acc**<br>*(KL `0.1508` · Brier `0.0795`)* | **#7 / 42 *(Live)***<br>*(#5 Tied-Top)* | **Top 16.7% *(Live)***<br>*(Top 11.9%)* | `vllm-sr/d3` *(27B, #8 `74.9%`)*, `Baseten Elephant 1` *(9B, #9 `74.8%`)*, `Darwin-27B` *(#10 `74.3%`)*, `Cloudflare/clef` *(27B, #17 `72.8%`)*, `clef-flash` *(9B, #24 `71.0%`)* |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot `<=9B` Open)*** | **19 Models** | **`75.25%` / `76.25%` Zero-Shot**<br>*(Distilled Raw: **`77.35%`**)* | **#2 / 19 *(Live)***<br>**#1 Clef-Arch** | **Top 10.5%** | Outperforms 17 of 18 `<=9B` open-weight models (`Baseten Elephant 1` `74.8%`, `vllm-sr/d3-mini` `74.3%`, `d3-flash` `71.7%`, `clef-flash` `71.0%`, `LiquidAI/d1-3B` `65.6%`) |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Fitted on `train`)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [Trace](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **36 Models** | **`79.70%` / `80.70%` Acc**<br>*(KL `0.0816` · Brier `0.0437`)* | **#1 (`9B` Class)**<br>**Top 9 / 36 Overall** | **Top 13.9%–25.0%** | `OpenDecider-small-td` *(4B, `79.2%`)*, `nullspan` *(395M, `79.0%`)*, `OpenDecider-medium-td` *(30B-A3B, `78.8%`)*, `Qwen3.5-4B-td` *(`77.25%`)*, `Seb-9B` *(9B, `75.4%`)* |
@@ -96,18 +96,24 @@ Evaluated on the official **Cloudflare Decision Index (v0.2.1)** across **10 ben
 ### External Community Benchmarks (`safeagents-judge-bench` & `LocalLLaMA/typed-decisions`)
-#### 1. [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench) (562 Agent Tool Calls · 78 Unsafe Trajectories · 7 Categories)
-| Model | Params | Balanced Acc ↑ | AUROC ↑ | Recall ↑ | False Block Rate ↓ | Binary F1 ↑ | Cat. Macro-F1 ↑ | Cat. Acc ↑ | P50 Latency ↓ |
 | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| **`TypeScript/FastBrain-9B` *(Calibrated · Ours)*** | **9.4B** | **97.4%** *(#1)* | **98.6%** | **100.0%** *(78/78)* | **5.1%** *(#1)* | **99.8%** | **85.1%** *(#1)* | **92.3%** | **104 ms** |
-| **`TypeScript/FastBrain-9B` *(Distilled Raw Head · α=1.0)*** | **9.4B** | **94.9%** | **96.8%** | **98.7%** *(77/78)* | **8.9%** | **99.2%** | **62.4%** | **81.5%** | **104 ms** |
-| **`TypeScript/FastBrain-9B` *(Raw JointSchemaHead)*** | **9.4B** | 71.0% | **92.6%** *(>Flash)* | **98.5%** *(77/78)* | 56.4% | **97.2%** | **35.3%** *(>Flash)* | 55.2% | **104 ms** |
-| `Cloudflare/clef-omni` | 30B-A3B | 84.5% | 91.6% | 92.2% *(77/78)* | 23.1% | 95.1% | 32.7% | 55.9% | 286 ms |
-| `Cloudflare/clef-flash` | 9.4B | 72.1% | 90.2% | 98.1% *(77/78)* | 53.8% | 97.1% | 34.5% | 54.6% | 332 ms |
-| `Cloudflare/clef` | 27.4B | 71.8% | **98.7%** | **100.0%** *(78/78)* | 56.4% | 97.9% | 42.5% | 72.1% | 700 ms |
-| `LiquidAI/d1-3B` | 3B | 83.4% | 91.7% | 92.5% *(78/78)* | 25.6% | 95.2% | — | — | 74 ms |
-| `vllm-sr/d3-mini` | 1.8B | 73.1% | 79.4% | 77.1% *(72/78)* | 30.8% | 86.1% | — | — | 79 ms |
 #### 2. [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (400 Test Cases · 2,000 Typed Decisions Across 4 Enterprise Workflows)
@@ -398,7 +404,7 @@ Because `FastBrain-9B` scores all schema questions and candidate options in a **
 | `joint_schema_model.py` | `25 KB` | `ClefModel`, `JointSchemaHead`, `load_release_model(load_in_4bit/8bit)`, and `systemone()` runtime |
 | `eval_submissions/typescript_fastbrain-9b.json` | `901 KB` | Full 400-case / 2,000-decision Zero-Shot prediction trace for `LocalLLaMA/typed-decisions` ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) |
 | `eval_submissions/typescript_fastbrain-9b-fitted.json` | `902 KB` | Full 400-case / 2,000-decision Fitted (`train`-calibrated) prediction trace for `LocalLLaMA/typed-decisions` ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) |
-| `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | `205 KB` | Full 562-call binary & 7-way category prediction traces for `harness-hub/safeagents-judge-bench` ([PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) |
 | `.eval_results/*.yaml` | `2.3 KB` | Official Hugging Face Hub benchmark evaluation metadata (`typed-decisions` & `safeagents-judge-bench`) |
 | `config.json`, `tokenizer.json`, `processor_config.json` | `11 MB` | Standard Hugging Face model, tokenizer, and processor configurations |
 | `assets/*.png` | `1.2 MB` | High-resolution technical benchmark and calibration charts |

 [![Architecture](https://img.shields.io/badge/Backbone-Qwen3.5--9B_Hybrid_(9.4B_Total)-3b82f6?style=flat-square)](#architecture)
 [![Decision Head](https://img.shields.io/badge/Head-JointSchemaHead_(116M)-10b981?style=flat-square)](#architecture)
 [![Decision Index](https://img.shields.io/badge/Decision_Index_v0.2.1-80.55%25_(%232_Overall_%C2%B7_%231_9B)-f59e0b?style=flat-square)](#benchmark-evaluation--official-leaderboard-rankings)
+[![SafeAgents Judge](https://img.shields.io/badge/SafeAgents_Judge_v2-75.1%25_Cat_F1_(%231_%2F_8)_%C2%B7_98.5%25_AUROC-ec4899?style=flat-square)](#external-community-benchmarks-safeagents-judge-bench--localllamatyped-decisions)
 [![Typed Decisions](https://img.shields.io/badge/Typed_Decisions-%237_%2F_42_Zero--Shot_Live_(%231_Clef--Arch)-8b5cf6?style=flat-square)](#external-community-benchmarks-safeagents-judge-bench--localllamatyped-decisions)
 [![License](https://img.shields.io/badge/License-Apache_2.0-6366f1?style=flat-square)](./LICENSE)
 | Official Leaderboard / Track | Total Models | **`FastBrain-9B` Score** | **Official Rank** | **Top Percentile** | Key Models Outperformed (`Params` & Score) |
 | :--- | :---: | :---: | :---: | :---: | :--- |
+| **[`safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(7-Way Category · `v2` & `v0`)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [Bench PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | **8 Models (`v2`)**<br>*(9 on `v0`)* | **`75.1%` Macro-F1 · `85.8%` Acc *(`v2`)***<br>*(`v0`: **`85.1%` Macro-F1**)* | **#1 / 8 *(`v2` Live)***<br>**#1 / 9 *(`v0`)*** | **Top 12.5% *(Rank #1)*** | `Cloudflare/clef` *(27.4B, #2 `56.3%` F1 / `73.7%` Acc)*, `vllm-sr/d3-mini` *(#3 `51.2%`)*, `Cloudflare/clef-flash` *(9.4B, #4 `46.2%` / `57.6%` Acc)*, `Cloudflare/clef-omni` *(30B, #5 `43.7%`)* |
+| **[`safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) *(Binary Guardrail · `v2` & `v0`)***<br>([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) · [Bench PR #1](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/discussions/1)) | **8 Models (`v2`)**<br>*(9 on `v0`)* | **`98.5%` AUROC · `99.6%` Recall**<br>*(`81.4%` Raw / **`93.8%` Cal. `v2`** · **`97.4%` `v0`**)* | **`82/82` (`100%`) Traj.**<br>**#1 (`v0`) · Top-3 AUROC (`v2`)** | **Top 11.1%–25.0%** | `Cloudflare/clef-flash` *(9.4B, `77.6%` Bal. Acc, `96.7%` AUROC)*, `Cloudflare/clef` *(27.4B, `80.8%` Bal. Acc, `38.4%` FBR)*, `Cloudflare/clef-omni` *(30B, `94.6%` AUROC, `77/82` Traj.)*, `gliclass-v4-edge` *(`78.6%` Bal. Acc)* |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot Overall)***<br>([Merged PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58) · [Trace](./eval_submissions/typescript_fastbrain-9b.json)) | **42 Models** | **`75.25%` / `76.25%` Acc**<br>*(KL `0.1508` · Brier `0.0795`)* | **#7 / 42 *(Live)***<br>*(#5 Tied-Top)* | **Top 16.7% *(Live)***<br>*(Top 11.9%)* | `vllm-sr/d3` *(27B, #8 `74.9%`)*, `Baseten Elephant 1` *(9B, #9 `74.8%`)*, `Darwin-27B` *(#10 `74.3%`)*, `Cloudflare/clef` *(27B, #17 `72.8%`)*, `clef-flash` *(9B, #24 `71.0%`)* |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Zero-Shot `<=9B` Open)*** | **19 Models** | **`75.25%` / `76.25%` Zero-Shot**<br>*(Distilled Raw: **`77.35%`**)* | **#2 / 19 *(Live)***<br>**#1 Clef-Arch** | **Top 10.5%** | Outperforms 17 of 18 `<=9B` open-weight models (`Baseten Elephant 1` `74.8%`, `vllm-sr/d3-mini` `74.3%`, `d3-flash` `71.7%`, `clef-flash` `71.0%`, `LiquidAI/d1-3B` `65.6%`) |
 | **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *(Fitted on `train`)***<br>([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57) · [Trace](./eval_submissions/typescript_fastbrain-9b-fitted.json)) | **36 Models** | **`79.70%` / `80.70%` Acc**<br>*(KL `0.0816` · Brier `0.0437`)* | **#1 (`9B` Class)**<br>**Top 9 / 36 Overall** | **Top 13.9%–25.0%** | `OpenDecider-small-td` *(4B, `79.2%`)*, `nullspan` *(395M, `79.0%`)*, `OpenDecider-medium-td` *(30B-A3B, `78.8%`)*, `Qwen3.5-4B-td` *(`77.25%`)*, `Seb-9B` *(9B, `75.4%`)* |
 ### External Community Benchmarks (`safeagents-judge-bench` & `LocalLLaMA/typed-decisions`)
+#### 1. [`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) (`v2`: 661 Agent Tool Calls · 82 Unsafe Trajectories · 7 Categories)
+> **Official `v2` Leaderboard Note ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1) by `@ybelkada` · [`.eval_results/safeagents-judge-bench.yaml`](./.eval_results/safeagents-judge-bench.yaml))**: Evaluated independently via Hugging Face Jobs on `NVIDIA A100-SXM4-80GB` across `661` agent tool calls (`536` malicious calls across `82` unsafe trajectories + `125` benign `allow` calls):
+> - **Rank `#1 / 8` Overall on 7-Way Category Classification (`v2`)**: **`75.1%` Category Macro-F1** and **`85.8%` Category Accuracy (`567/661`)**, outperforming the `3x`-larger `27.4B Cloudflare/clef` (`56.3%` Macro-F1 / `73.7%` Acc) by **`+18.8 pp` Macro-F1 (`+12.1 pp` Acc)** and the same-size base model `Cloudflare/clef-flash` (`46.2%` Macro-F1 / `57.6%` Acc) by **`+28.9 pp` Macro-F1 (`+28.2 pp` Acc)**.
+> - **Zero Missed Unsafe Trajectories & `98.5%` AUROC on Binary Guardrail (`v2`)**: Catches **`82 / 82` (`100.0%`) unsafe trajectories** (`median_bad_calls_before_catch = 0.0`) with **`99.6%` Recall (`534/536`)** (`100%` recall on `harness_introspection` `272/272`, `credential_misuse` `15/15`, and `sandbox_escape` `4/4`; `99.2%` on `eval_gaming` `243/245`) and **`98.5%` AUROC**. At the uncalibrated default threshold (`P(block) >= 0.50`), Balanced Accuracy is **`81.4%`** (outperforming `27.4B Cloudflare/clef` at `80.8%` and `9.4B Cloudflare/clef-flash` at `77.6%`); with threshold calibration (`P(block) >= 0.961`), Balanced Accuracy reaches **`93.8%` (`5.6%` False Block Rate)** on `v2` and **`97.4%` (`85.1%` Category Macro-F1)** on `v0` (`562` calls).
+| Model (`safeagents-judge-bench v2`) | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Balanced Acc ↑ | Recall *(Traj. Caught)* ↑ | False Block Rate ↓ | Binary F1 ↑ | A100 P50 Latency ↓ |
 | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **`TypeScript/FastBrain-9B` *(`v2` Official · Default `t=0.50`)*** | **9.4B** | **75.1%** *(#1 / 8)* | **85.8%** *(#1 / 8)* | **98.5%** | 81.4% | **99.6%** *(**82/82**)* | 36.8% | **95.7%** | 327 ms *(104 ms G4)* |
+| **`TypeScript/FastBrain-9B` *(`v2` Calibrated · `t=0.961`)*** | **9.4B** | **75.1%** *(#1)* | **85.8%** *(#1)* | **98.5%** | **93.8%** | 93.1% *(**82/82**)* | **5.6%** | **96.2%** | 327 ms *(104 ms G4)* |
+| **`TypeScript/FastBrain-9B` *(`v0` 562-Call Calibrated)*** | **9.4B** | **85.1%** *(#1)* | **92.3%** *(#1)* | **98.6%** | **97.4%** *(#1)* | **100.0%** *(**78/78**)* | **5.1%** | **99.8%** | 104 ms *(G4)* |
+| `Cloudflare/clef` | 27.4B | 56.3% *(#2)* | 73.7% *(#2)* | **99.3%** | 80.8% | **100.0%** *(**82/82**)* | 38.4% | 95.7% | 636 ms |
+| `vllm-sr/d3-mini` | 1.8B | 51.2% *(#3)* | 72.3% *(#3)* | 98.6% | **94.8%** | 96.8% *(**82/82**)* | **7.2%** | **97.6%** | 41 ms |
+| `Cloudflare/clef-flash` *(Base Model)* | 9.4B | 46.2% *(#4)* | 57.6% *(#6)* | 96.7% | 77.6% | 98.3% *(81/82)* | 43.2% | 94.4% | 299 ms |
+| `Cloudflare/clef-omni` | 30B-A3B | 43.7% *(#5)* | 66.4% *(#5)* | 94.6% | 86.2% | 92.4% *(77/82)* | 20.0% | 93.9% | 237 ms |
+| `knowledgator/gliclass-v4-edge` | 0.4B | 37.3% *(#6)* | 69.0% *(#4)* | 88.2% | 78.6% | 89.2% *(80/82)* | 32.0% | 91.0% | 12 ms |
+| `fastino/GLiNER2.5-Decide` | 0.5B | 35.5% *(#7)* | 53.6% *(#8)* | 87.3% | 81.9% | 93.5% *(80/82)* | 29.6% | 93.4% | **8 ms** |
+| `LiquidAI/d1-3B` | 3B | 32.8% *(#8)* | 55.5% *(#7)* | 93.9% | 85.4% | 95.5% *(**82/82**)* | 24.8% | 94.9% | 64 ms |
 #### 2. [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (400 Test Cases · 2,000 Typed Decisions Across 4 Enterprise Workflows)
 | `joint_schema_model.py` | `25 KB` | `ClefModel`, `JointSchemaHead`, `load_release_model(load_in_4bit/8bit)`, and `systemone()` runtime |
 | `eval_submissions/typescript_fastbrain-9b.json` | `901 KB` | Full 400-case / 2,000-decision Zero-Shot prediction trace for `LocalLLaMA/typed-decisions` ([PR #58](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/58)) |
 | `eval_submissions/typescript_fastbrain-9b-fitted.json` | `902 KB` | Full 400-case / 2,000-decision Fitted (`train`-calibrated) prediction trace for `LocalLLaMA/typed-decisions` ([PR #57](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/57)) |
+| `eval_submissions/TypeScript__FastBrain-9B*.jsonl` | `110 KB` | Full 661-call (`v2`) binary & 7-way category prediction traces for `harness-hub/safeagents-judge-bench` ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)) |
 | `.eval_results/*.yaml` | `2.3 KB` | Official Hugging Face Hub benchmark evaluation metadata (`typed-decisions` & `safeagents-judge-bench`) |
 | `config.json`, `tokenizer.json`, `processor_config.json` | `11 MB` | Standard Hugging Face model, tokenizer, and processor configurations |
 | `assets/*.png` | `1.2 MB` | High-resolution technical benchmark and calibration charts |

eval_submissions/TypeScript__FastBrain-9B.category.jsonl CHANGED Viewed

The diff for this file is too large to render. See raw diff

eval_submissions/TypeScript__FastBrain-9B.jsonl CHANGED Viewed

The diff for this file is too large to render. See raw diff