Your code got faster. Did it finish the job?

“My code is faster than yours.”

Engineers know the feeling.

Ten percent faster feels good.
Fifty percent is worth sharing.
Three times faster?

That can do wonders for an engineer’s ego.

Before asking how, I would ask:

“Did it do all the work?”

A crash is usually easy to notice. A program that runs quickly and produces plausible results is harder to question—even when it has skipped something important.

Doing less computation can be a perfectly good optimization. The question is whether the work we removed changes the result we need.

The experiment

I recently built tiny-infer-lab: a small C inference engine and a performance-validation toolkit.

The aim wasn't to prove that C beats Python.

Pick a workload that disadvantages Python, give C its most favorable conditions, and “C wins” is easy to arrange.

That's choosing the winner before organizing the race.

Instead, I compared six implementations using the same model weights and 450 held-out test examples, at batch sizes 1, 8, 32, 128 and 450:

Handwritten C, system BLAS, emlearn, ONNX Runtime, NumPy and scikit-learn.

Results and timing

All six agreed on every predicted class at every tested batch size: 438/450 correct, or 97.33%. Full probability outputs also passed checks against an independent float64 reference within predefined tolerances.

emlearn was fastest at batch size 1. ONNX Runtime was fastest at every larger tested batch.

At batch size 450, warm batch time amortized per prediction was about:
• ONNX Runtime: 0.059 µs
• C BLAS: 0.142 µs

Each figure is the median of six independent process medians, not individual-request latency. Timing covers Python-facing APIs, input checks, probability output and result collection; loading, warmup and correctness checks are outside it. Batch implementations and internal checks differ, so this isn't a bare-kernel ranking.

My C engine didn’t win every comparison. That was useful: the results showed the trade-offs.

Challenge the checks

I also deliberately broke C implementations and corrupted reports to check whether validation would reject those specific faults. It did.

I wanted tests that would notice when the code was wrong.

Memory and limits

Median warm process RSS at batch size 450 was about 60.4 MiB for ONNX Runtime and 38.9 MiB for C BLAS. Those totals include Python, NumPy and the measurement environment. They are not model-only RAM.

One 64→32→10 model, one machine with an Apple M5 Pro, BLAS and ONNX Runtime compute threads set to one. These results don't establish a universal winner.

Before a performance number guides a decision, I want to know whether the measurement system itself has been challenged.

Slow code can be optimized. Wrong code can be fixed.

The uncomfortable case is wrong code with a benchmark giving it a standing ovation.