Let's help help help devs.
Target: <200ms. Several important LLM CLI tools take multiple seconds. PRs welcome ❤️
| library | cold | warm (10 runs) | version | measured on |
|---|---|---|---|---|
vllm --help | 15610ms | 7473ms | 0.14.1+cpu | 2026-01-28T21:50Z |
sglang --help | 15662ms | 6825ms | v0.5.9 | 2026-02-25T11:16Z |
VLMEvalKit --help | 13936ms | 5076ms | v0.2 | 2026-02-25T11:21Z |
tensorrt-llm --help | 7467ms | 2600ms | 1.0.0 | 2026-02-25T11:14Z |
datasets --help | 3074ms | 907ms | 4.5.0 | 2026-02-25T11:02Z |
llm --help | 1205ms | 569ms | 0.28 | 2026-02-25T11:03Z |
openai --help | 1053ms | 504ms | 2.24.0 | 2026-02-25T11:03Z |
hf --help | 990ms | 331ms | 1.4.1 | 2026-02-25T11:02Z |
langchain-cli --help | 845ms | 259ms | 0.0.37 | 2026-02-25T11:03Z |
lm-eval --help | 800ms | 228ms | 0.4.11 | 2026-02-25T11:04Z |
ollama --help | 15ms | 15ms | 0.17.0 | 2026-02-25T11:02Z |
llama.cpp --help | 19ms | 13ms | b8149 | 2026-02-25T11:04Z |
transformers --help | 0ms | 0ms | 5.2.0 | 2026-02-25T11:04Z |