// benchmarking
Stress-test.
At any scale.
Most teams run a model once and proceed. liwaiwai runs it a thousand times recording every deviation, every latency spike, every inconsistency before and after deployment.
Start benchmarking$ liwaiwai run --models 3 --iters 100 --prompt "Classify this contract clause..."
› initialising... 100 iterations across 3 models
✓ gpt-4o avg 812ms 98.2% consistent $0.0041/run
✓ claude-3-5-sonnet avg 934ms 97.8% consistent $0.0038/run
⚠ gemini-2.5-flash avg 441ms 91.3% consistent $0.0009/run
› 3 deviations flagged see guardrail report
▸ trust matrix: PENDING (712/1000 runs)
Concurrent multi-model runs
Test up to 6 models in parallel with a single prompt. Latency, token cost, and output quality side by side.
Iteration at scale
Run the same prompt 10, 100, or 1,000 times. Statistical drift, outliers, and edge-case failures surface over time not just on first contact.
Latency and throughput tracking
Every run captures response time in milliseconds, token counts, and cost per call stored historically for longitudinal analysis.
Consistency scoring
The Trust Matrix assigns each model a consistency score across iterations. Low variance is a signal. High variance is a finding.
// who benchmarks with liwaiwai
Government agencies
Independent benchmark data to support AI procurement decisions and policy evaluation.
Research institutions
Publication-grade LLM evaluation with reproducible methodology and full audit trails.
Enterprise AI teams
Regression detection across model versions before changes reach production.