laya.cpp: standalone C++ inference with CUDA and Vulkan backends
ggml-based C++ runtime for all three checkpoints. Native tokenization, decision heads and JSON output; optimized FP16/BF16 CUDA on RTX with FP32 accumulation, plus Vulkan.
Throughput measured on an RTX PRO 6000 Blackwell at batches 1–8; precision checked against the Python baseline.
# laya.cpp Standalone C++ inference for Laya typed decisions using ggml, with optimized CUDA execution on NVIDIA RTX GPUs and a Vulkan FP32 backend. Model loading, Unicode/BPE tokenization, transformer inference, decision heads, and JSON output all run natively. All three checkpoints are supported: | Variant | Context | Encoder width / layers | Tokenizer | |---|---:|---:|---| | `english` | 512 | 1024 / 28 | NFC byte-level BPE | | `multilingual` | 1024 | 768 / 22 | Metaspace BPE with byte fallback | | `typed-decisions` | 1024 | 1024 / 28 | NFC byte-level BPE | Strict FP32 is the default. The optimized CUDA path uses exact checkpoint FP16 weights, paired activation components, FP32 accumulation, and fused packing kernels. Enable it with `--tensor-core-fp32 --flash-fp32`. Native mixed BF16 is available with `--bf16` using the validated CUDA 13.0 / cuBLAS 13.1.0 build profile. Correctness is checked against the baseline at matching precision; see [precision comparisons](docs/precision.md). ## Performance Measured on an **NVIDIA RTX PRO 6000 Blackwell (96 GB), capped at 450 W**, using all 250 fixed acceptance questions at batches 1, 2, 4 and 8. Throughput is **questions/second**; higher is better. Python denotes the original implementation. The 16-bit mode is **BF16**. | Model | Batch | Python BF16 | laya.cpp BF16 | Python FP32 | laya.cpp FP32 | |---|---:|---:|---:|---:|---:| | english | 1 | 149.2 | 365.8 | 147.9 | 341.7 | | english | 2 | 267.5 | 585.6 | 201.6 | 420.6 | | english | 4 | 459.8 | 761.1 | 232.9 | 437.2 | | english | 8 | 662.5 | 809.8 | 231.7 | 385.7 | | multilingual | 1 | 178.9 | 485.4 | 191.7 | 475.3 | | multilingual | 2 | 320.7 | 816.0 | 299.8 | 610.0 | | multilingual | 4 | 550.2 | 1,164.0 | 388.5 | 673.4 | | multilingual | 8 | 828.1 | 1,246.4 | 408.0 |