Benchmarking System One small LLM decision APIs vs. in-process tree ensembles and exact TreeSHAP on 64-feature tabular payloads

There has been a lot of discussion lately around “System One” decision models: fast, small LLM endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls. If the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense. Where I still see teams get burned is routing structured numeric/categorical feature vectors (say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop) through an LLM endpoint so they can get a score plus a natural-language “reason code” in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and prompt-generated reason codes are not true feature attributions. While running an 18-model verification matrix for SMILE 7.2 (smile-serve running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS, alongside ONNX Runtime), I measured where the latency actually goes when you pair classical tree ensembles with exact per-row TreeSHAP in-process. A few numbers stood out: Scoring itself is almost free. Across 50 to 500 trees and 20 to 200 features (500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P), single-row GradientTreeBoost and RandomForest scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP (and 0.21 ms p50 / 0.43 ms p99 on exported XGBoost .onnx). Exact TreeSHAP is where naive Random Forests blow up. Scoring a tree scales linearly with depth O(T * D), but exact TreeSHAP scales with depth squared, O(T * L * D^2). On an unconstrained 300-tree, 64-feature RandomForest (max_depth=20), single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding ?explain=true pushed latency to 192.5 ms p50 / 274.4 ms p99 (worse than a cloud LLM call). Here is the depth/leaf ablation on the 300-tree, 64-feature Random Forest (JVM, 100k measured requests after 5k warm-up): RF Hyperparameters (300 trees, 64 features) .sml Size Cold Start Scoring p50 / p99 Exact TreeSHAP p50 / p99 500 rps p99 Stream (rows/s) max_depth=20, max_nodes=0 (unbounded) 16.8 MB 12.50 s 0.41 ms / 0.92 ms 192.5 ms / 274.4 ms 12.9 ms 5,201 /s max_depth=10, max_nodes=0 3.40 MB 3.91 s 0.31 ms / 0.98 ms 22.5 ms / 28.9 ms 2.43 ms 9,195 /s max_depth=8, max_nodes=256 1.56 MB 2.49 s 0.28 ms / 0.56 ms 7.81 ms / 9.09 ms 0.90 ms 11,039 /s max_depth=6, max_nodes=128 637 KB 2.32 s 0.26 ms / 0.54 ms 2.14 ms / 2.88 ms 0.78 ms 14,662 /s GradientTreeBoost (300 trees, 64 features) 292 KB 1.55 s 0.24 ms / 0.47 ms 0.69 ms / 0.91 ms 0.64 ms 19,241 /s One interesting failure mode we hit on GraalVM Native Image: while GradientTreeBoost passed <5 ms p99 TreeSHAP across all 9 matrix cells (1.26 ms to 3.97 ms p99 even at 500 trees x 200 features), the 300-tree RandomForest on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms (p50 = 4.48 ms, p95 = 4.54 ms), zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside TreeSHAP triggering GraalVM Native Image’s default Serial GC (~29 ms stop-the-world pause roughly once every 85 requests), whereas HotSpot’s G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms. The takeaway for hybrid agent + ML pipelines is pretty simple: use small LLMs when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms. Code and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: GitHub - haifengl/smile: Statistical Machine Intelligence & Learning Engine · GitHub For folks running real-time SHAP or feature attributions in production latency budgets (<10 ms), are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path? Open article →