Router

Concurrency

Not thread-safe. Create one Router per thread, or use Client which manages Router instances internally.

Example

>>> # Usually created via Client
>>> client = Client("Qwen/Qwen3-0.6B")
>>> # client._router is the Router instance

Attributes

model

Model identifier (e.g., "Qwen/Qwen3-0.6B", "openai::gpt-4o").

endpoint

Optional custom endpoint URL (overrides provider defaults).

spec

The ModelSpec used to create this target.

Attributes

text

Decoded token text.

item_type

Item type (e.g. ItemType.MESSAGE, ItemType.REASONING).

content_type

Content type (e.g. ContentType.OUTPUT_TEXT, ContentType.REASONING_TEXT).

Mutability

GenerationConfig is mutable, allowing in-place modification of session defaults:

>>> chat = Chat("model", config=GenerationConfig(temperature=0.7))
>>> chat.config.temperature = 1.2  # Modify for subsequent calls
>>> chat.config.max_tokens = 500

For creating variations without modifying the original, use .override():

>>> base = GenerationConfig(temperature=0.7)
>>> creative = base.override(temperature=1.2)  # New config, base unchanged

Design Note

GenerationConfig serves as "Request Configuration" - not just sampling parameters, but all per-request settings including prompting concerns (chat_template, extra_context, schema_strategy, stop_sequences).

This design enables

1. Session-level defaults via Chat(config=GenerationConfig(...)) 2. In-place modification via chat.config.temperature = 0.5 3. Per-request overrides via chat.send(..., schema_strategy="typescript") 4. A/B testing by passing different configs to the same Chat

Parameters like schema_strategy live here (not on Chat) because:

• "auto" handles 99% of cases without user intervention
• Session defaults work via Chat(config=...)
• Per-request flexibility is preserved for experimentation
• Chat.__init__ stays clean (identity only: model, system, storage)

Attributes

max_tokens

Maximum number of tokens to generate. Default is 256. One token is roughly 4 characters or 0.75 words in English.

temperature

• Controls randomness in generation. Default is 0.7.
• 0.0: Deterministic (greedy decoding)
• 0.1-0.5: Focused and consistent
• 0.7-1.0: Balanced creativity
• 1.0-2.0: More creative and varied

top_k

Limits token selection to the k most likely. Default is 50. Set to 0 to disable top-k filtering.

top_p

Nucleus sampling threshold. Default is 0.9. Selects from smallest set of tokens whose cumulative probability exceeds this threshold.

min_p

Minimum probability threshold. Default is 0.0 (disabled). Tokens with probability below min_p * max_prob are excluded. Modern alternative to top_p for some use cases.

repetition_penalty

Penalty for repeating tokens. Default is 1.0. Values > 1.0 discourage repetition, < 1.0 encourage it.

stop_sequences

List of strings that stop generation. Default is None. When any of these strings is generated, generation stops immediately. The stop sequence itself is NOT included in the output.

Multi-token sequences are fully supported. Each stop sequence is
tokenized and the full token sequence is matched during generation.
For example, "User:" will only trigger when the complete sequence
is generated, not when just "User" appears.

stop_token_ids

List of token IDs that stop generation. Default is None. When any of these token IDs is generated, generation stops immediately.

This overrides the model's default EOS tokens for this request only.
If you want to ADD to the default EOS tokens (not replace them),
include the model's eos_token_ids in your list.

Use cases

• Override model's EOS tokens for specific prompts
• Add additional stop tokens (like newline) for line-by-line generation
• Implement custom stop logic without string matching overhead

Example: stop_token_ids=[151645, 198]  # EOS + newline for Qwen

seed: Random seed for reproducibility. Default is None (random). Set to a fixed value for deterministic outputs.

response_format: Structured output format specification. Default is None. Use ResponseFormat to constrain output to JSON or regex patterns. Note: Not yet implemented in the runtime.

logprobs: Whether to return token log probabilities. Default is False. When True, GenerationResult.logprobs will contain log probabilities for each generated token.

top_logprobs: Number of top token alternatives to return. Default is None. When set (1-20), returns log probabilities for the top N most likely tokens at each position. Requires logprobs=True.

logit_bias: Dictionary mapping token IDs to bias values. Default is None. Positive values increase the likelihood of a token being sampled, negative values decrease it. Use -100 or lower to effectively ban a token from appearing.

Example: {1234: -100}  # Ban token 1234
Example: {5678: 5.0}   # Strongly prefer token 5678

chat_template: Custom chat template for this request. Default is None. When set, uses this template instead of the session's chat_template or the model's built-in template. Accepts either a template string or a PromptTemplate object.

This follows standard configuration precedence

• GenerationConfig.chat_template (per-request) takes priority over
• Chat.chat_template (session-level) which takes priority over
• Model's default template (from tokenizer_config.json)

The template uses Jinja2 syntax with standard variables:
- messages: List of message dicts with 'role' and 'content'
- bos_token, eos_token: Special tokens from model config
- add_generation_prompt: Whether to append assistant prompt

Example: "{% for m in messages %}{{ m.role }}: {{ m.content }}\n{% endfor %}"

extra_context: Additional template variables. Default is None. Dictionary of extra variables to inject into the template context. These become available alongside standard variables (messages, etc.).

Use cases

• Tool definitions: {"tools": [{"name": "search", ...}]}
• System metadata: {"date": "2024-01-15", "user_name": "Alice"}
• Custom flags: {"enable_thinking": True}

Example

>>> config = GenerationConfig(
        ...     extra_context={"tools": [{"name": "calculator"}]}
        ... )

tools_json: Tool definitions as a JSON array string. Default is None. This is set internally by Chat when tool calling is enabled.

tool_choice: Tool choice directive ("auto", "required", "none", or function name). Default is None and set internally by Chat when tool calling is enabled.

• schema_strategy: Strategy for injecting schema into prompts. Default is "auto". Controls how JSON schemas are formatted when using response_format.
• "auto": Automatically select based on model architecture (recommended)
• "typescript": TypeScript interface syntax (best for code-trained models)
• "json_schema": Raw JSON Schema (for older/simpler models)
• "xml_schema": XML-wrapped schema (for Anthropic-style models)

Most users should leave this as "auto". Override only when experimenting
with new models or when you know a specific format works better.

inject_schema_prompt: Whether to inject schema into the prompt. Default is True. When True and response_format is provided, the schema is automatically injected into the system prompt or user message.

Set to False if you've manually included the schema in your prompt
and want grammar enforcement without auto-injection.

allow_thinking: Enable chain-of-thought reasoning mode. Default is False. When True, allows the model to output a <think>...</think> block before the structured response. Useful for complex reasoning tasks.

max_thinking_tokens: Maximum tokens for thinking block. Default is 512. Only applies when allow_thinking=True. Limits the reasoning output to prevent excessive token usage.

validation_retries: Number of automatic retries on SchemaValidationError. Default is 0. When using response_format with Pydantic validators, the grammar ensures syntactically valid JSON but cannot enforce semantic constraints (e.g., field_validator that requires age < 120).

When > 0, if parsing raises SchemaValidationError, the Chat automatically:
1. Appends the error message to conversation history
2. Regenerates with the same grammar constraint
3. Repeats until validation passes or retries exhausted

This closes the loop between grammar (syntactic) and Pydantic (semantic).
Set to 1-3 for most use cases; higher values rarely help.

extra_body: Extra parameters for remote API requests. Default is None. Dictionary of provider-specific parameters that are merged into the request body for OpenAI-compatible APIs. This is the "escape hatch" for using new or provider-specific features not yet in GenerationConfig.

For local inference, this parameter is ignored.

Example: extra_body={"repetition_penalty": 1.1, "top_a": 0.5}

Example

>>> # Creative writing config
>>> creative = GenerationConfig(
...     temperature=1.2,
...     top_p=0.95,
...     max_tokens=500
... )

>>> # Precise/deterministic config
>>> precise = GenerationConfig(
...     temperature=0.0,
...     max_tokens=100
... )

>>> # JSON extraction config
>>> json_config = GenerationConfig(
...     temperature=0.0,
...     stop_sequences=["}"],
...     max_tokens=200
... )

>>> # With logprobs
>>> config = GenerationConfig(
...     logprobs=True,
...     top_logprobs=5,
...     max_tokens=50
... )

>>> # With logit bias (ban specific tokens)
>>> config = GenerationConfig(
...     logit_bias={1234: -100, 5678: -100},  # Ban tokens 1234 and 5678
...     max_tokens=100
... )

>>> # Use with Chat
>>> chat = Chat("model", config=precise)
>>> chat.send("What is 2+2?")  # Uses precise config
>>> chat.send("Write a poem", config=creative)  # Override for this call

Parameters

response_format

A dataclass type or JSON schema dict defining the output structure.

Raises

StructuredOutputError

If the schema is invalid or compilation fails.

Example - Basic usage

>>> from dataclasses import dataclass
>>> from talu.router import Grammar, Chat
>>>
>>> @dataclass
... class Answer:
...     value: int
...     reasoning: str
...
>>> # Compile grammar once
>>> grammar = Grammar(Answer)
>>>
>>> # Reuse for multiple generations
>>> chat = Chat("Qwen/Qwen3-0.6B")
>>> r1 = chat.send("What is 2+2?", response_format=grammar)
>>> r2 = chat.send("What is 3+3?", response_format=grammar)

Example - Server context (compile at startup)

>>> # At startup
>>> ANSWER_GRAMMAR = Grammar(Answer)
>>>
>>> # In request handler
>>> async def handle_question(question: str):
...     chat = AsyncChat("model")
...     response = await chat.send(question, response_format=ANSWER_GRAMMAR)
...     return response.parsed

Attributes

token_ids

Pre-tokenized input. Bypasses tokenizer. Use case: External tokenization, token-level testing, sending tokens directly from another process. Does NOT make sense with role-based chat formatting (user/assistant markers).

continue_from_token_id

Continue generation from a specific token ID. Use case: Autocomplete systems, prefix completion, debugging token alignment. Does NOT make sense with chat-formatted prompts that expect conversation turns.

echo_prompt

Return input prompt plus generated completion combined. Use case: Auto-regressive training data generation, completion-style debugging where you need to verify input+output pairing. Chat applications already have full conversation history in messages list, so explicit echo doesn't make sense.

Example

>>> from talu.router import CompletionOptions

>>> # Pre-tokenized input
>>> opts = CompletionOptions(
...     token_ids=[1234, 5678, 9012],
...     continue_from_token_id=151645
... )
>>> response = talu.raw_complete(
...     "Qwen/Qwen3-0.6B",
...     "Continue: ",
...     completion_opts=opts
... )

>>> # Echo mode
>>> opts = CompletionOptions(echo_prompt=True)
>>> response = talu.raw_complete(
...     "Qwen/Qwen3-0.6B",
...     "Hello",
...     completion_opts=opts
... )

Usage

stop_flag = StopFlag()

# In generation thread or async task
for token in router.stream(chat, msg, stop_flag=stop_flag):
    yield token

# In cancellation handler (e.g., asyncio.CancelledError)
stop_flag.signal()

The stop flag can be reused after calling reset().

Attributes

base_url

Base URL for the API (e.g., "http://localhost:8080/v1").

api_key

API key for authentication.

org_id

Organization ID for multi-tenant APIs.

timeout_ms

Request timeout in milliseconds (0 = default).

max_retries

Maximum retry attempts for failed requests (0 = default).

extra_params

Additional parameters to include in API requests. These are merged into the request body for OpenAI-compatible APIs. Useful for provider-specific parameters not covered by GenerationConfig.

Example::

backend = OpenAICompatibleBackend(
    base_url="https://api.together.xyz/v1",
    api_key="...",
    extra_params={"repetition_penalty": 1.1}
)

headers

Custom HTTP headers to include in every request to this backend. Useful for enterprise networking requirements like authentication proxies, custom routing, or tracing headers.

Example::

backend = OpenAICompatibleBackend(
    base_url="https://internal-proxy.corp.example.com/llm/v1",
    api_key="...",
    headers={
        "X-Request-ID": "abc123",
        "X-Team-ID": "ml-team",
        "X-Proxy-Auth": "secret-token",
    }
)

Attributes

id

Model identifier (e.g., "Qwen/Qwen3-4B-Instruct-2507").

object

Object type (usually "model").

created

Unix timestamp when the model was created (optional).

owned_by

Owner/organization of the model.

Parameters

base_url

Base URL of the server (e.g., "http://localhost:8000").

api_key

Optional API key for authentication.

timeout

Request timeout in seconds.

Returns

List of RemoteModelInfo objects describing available models.

Raises

IOError

If the server is not reachable.

ValidationError

If the response is invalid.

Example

>>> from talu.router import list_endpoint_models
>>> models = list_endpoint_models("http://localhost:8000")
>>> for m in models:
...     print(m.id)
Qwen/Qwen3-4B-Instruct-2507

Parameters

base_url

Base URL of the server (e.g., "http://localhost:8000").

api_key

Optional API key for authentication.

timeout

Request timeout in seconds.

Returns

True if the endpoint is available, False otherwise.

Example

>>> from talu.router import check_endpoint
>>> if check_endpoint("http://localhost:8000"):
...     print("vLLM is running")

Parameters

base_url

Base URL of the server (e.g., "http://localhost:8000").

api_key

Optional API key for authentication.

timeout

Request timeout in seconds.

Returns

List of model ID strings.

Example

>>> from talu.router import get_model_ids
>>> ids = get_model_ids("http://localhost:8000")
>>> "Qwen/Qwen3-4B-Instruct-2507" in ids
True