---
title: Qwen3.8 27B Responses API
emoji: 🚀
colorFrom: purple
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
short_description: Authenticated OpenAI Responses API for Qwen3.8-27B GGUF
---

# Qwen3.8-27B OpenAI-compatible inference server

This Space serves `unsloth/Qwen3.8-27B-GGUF` through the current CUDA build of
`llama-server`. It exposes an authenticated OpenAI-compatible API, including
`POST /v1/responses`.

The default quantization is `Q4_K_M` (about 17.1 GB). The deployment is intentionally
text-only (`--no-mmproj`) so the model, KV cache, and runtime have enough VRAM on a
24 GB GPU. The native model context is larger, but the operational default is bounded
to 16,384 tokens for predictable memory use.

## Required Space configuration

1. Select paid GPU hardware with at least 24 GB of VRAM. An NVIDIA L4 or A10G is the
   minimum target for the default quantization and context.
2. Add a Space secret named `INFERENCE_API_KEY` containing at least 32 random
   characters.
3. Optional persistent storage mounted at `/data` avoids downloading the model again
   after a cold rebuild or restart.

The server fails closed when `INFERENCE_API_KEY` is missing or too short. Do not expose
this key in a browser, mobile application, Godot project, or public repository. Calls
from an end-user application should go through a trusted backend.

## OpenAI Python SDK

```python
import os

from openai import OpenAI


client = OpenAI(
    base_url="https://pablo-flores-mollinedo-inferecence.hf.space/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
)

response = client.responses.create(
    model="qwen3.8-27b",
    instructions="Answer clearly and accurately.",
    input="Explain continuous batching in two sentences.",
    max_output_tokens=512,
)

print(response.output_text)
```

Streaming uses the same SDK surface:

```python
with client.responses.stream(
    model="qwen3.8-27b",
    input="Write a short story about a golden house.",
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
```

## HTTP endpoints

| Endpoint | Purpose | Authentication |
| --- | --- | --- |
| `GET /health` or `/v1/health` | Readiness | Public |
| `GET /v1/models` | Served model metadata | Public |
| `POST /v1/responses` | OpenAI Responses-compatible inference | Bearer key |
| `POST /v1/responses/input_tokens` | Responses input token count | Bearer key |
| `POST /v1/chat/completions` | OpenAI Chat Completions compatibility | Bearer key |
| `GET /metrics` | Prometheus metrics | Bearer key |

The Responses implementation is provided by `llama-server` and converts Responses
requests into the model's chat template. It supports the text-generation subset needed
by the smoke test. OpenAI-hosted tools, durable response storage, and OpenAI service-side
state are not provided by this self-hosted deployment.

## Runtime controls

| Variable | Default | Allowed range or values |
| --- | --- | --- |
| `MAX_CONTEXT` | `16384` | `1024..32768` |
| `MAX_OUTPUT_TOKENS` | `4096` | `1..8192`, not greater than context |
| `PARALLEL_REQUESTS` | `1` | `1..2` |
| `MODEL_QUANT` | `Q4_K_M` | Letters, numbers, and underscores from this model repository |
| `INFERENCE_API_KEY` | none | Required, at least 32 characters |

Increasing context, output length, concurrency, or quantization size increases VRAM
pressure. Change one dimension at a time and run the smoke test after each deployment.

## Validation

Run configuration and contract tests:

```bash
python3 -m unittest discover -s tests -v
```

After deployment, run the public-path smoke test without printing the key:

```bash
INFERENCE_BASE_URL="https://pablo-flores-mollinedo-inferecence.hf.space/v1" \
INFERENCE_API_KEY="..." \
python3 smoke_test.py
```

## Operations and recovery

- Model weights are authoritative in the configured Hugging Face repository and are not
  written by this service.
- Response state and KV cache are ephemeral. A restart may discard conversation state;
  callers should resend required context.
- Prompt and response bodies are not logged by the startup configuration.
- `/metrics`, Space logs, and `/health` provide the initial operational signals.
- Roll back by reverting this Space repository to the previous known-good commit. Never
  roll back to a mutable or unreviewed container image.
- If VRAM exhaustion occurs, first reduce `MAX_CONTEXT`, then reduce
  `MAX_OUTPUT_TOKENS`, or select a smaller quantization from the same model repository.

See [`docs/architecture.md`](docs/architecture.md) for boundaries, invariants, validation,
and recovery details.