Guide · GLRMask

Install

The Python package and Rust crate expose the same core idea from different sides of the stack.

Pythonpip

python -m pip install glrmask

RustCargo

cargo add glrmask

Quickstart

This is the basic Python shape used by the library: create a vocabulary adapter, compile a JSON Schema constraint, then start an independent runtime state.

import glrmask

vocab = glrmask.Vocab.from_llama_cpp(llm)
end_token_ids = vocab.llama_cpp_end_token_ids

schema = '{"type":"string","enum":["positive","negative","neutral"]}'
constraint = glrmask.Constraint.from_json_schema(
    schema,
    vocab,
    end_token_ids=end_token_ids,
)

state = constraint.start()

Two lifetimes

Constraint is compiled, immutable, and reusable. The object returned by start() is the small mutable state for one generation.

Runtime loop

At each step, ask the constraint state for a vocabulary mask while the model computes logits. Apply that mask, sample, then commit the token that actually won.

state = constraint.start()

for _ in range(max_tokens):
    logits = model_step()
    mask = state.mask(vocab_size)
    logits[~mask] = -float("inf")

    token = sample(logits)
    state.commit_token(token)
    model_commit(token)

    if token in end_token_ids:
        break

mask() depends only on the current constraint state. That is the useful scheduling property: mask generation can run alongside the model forward pass instead of waiting until logits already exist.

Grammars

JSON Schema is the main practical entry point. GLRMask also accepts its own EBNF-like GLRM format, along with Lark and EBNF grammars.

GLRM can refer to exact model-token IDs as terminals. That matters for tokenizer-level structure—end tokens are the obvious example—that does not have a byte representation inside the ordinary grammar.

General context-free grammars can introduce lexer ambiguity, reductions, and multiple parser stacks. GLRMask keeps those alternatives in a graph-structured stack rather than forcing an early choice.

Reuse and caching

The compiled constraint is the expensive object. Treat it like a build artifact: serialize it, cache it, and load it where inference happens. Each request then creates only its own runtime state.

Deployment shape

For large or reused grammars, compilation can happen away from the inference server. The finished artifact is what the hot path needs.

Dynamic constraints

DynamicConstraint uses the same runtime interface when a cached compilation is not available. It starts much faster, but leaves more work on mask generation. That makes it a useful fallback while a fully compiled constraint is being produced for later requests.

For the current API surface, examples, and build instructions, use the repository. The rest of this site focuses on the architecture and performance trade-offs rather than duplicating generated API documentation.

Compile a constraint. Reuse it. Keep generation simple.