CoreML was enabled. CPU inference was faster.

GPU. NPU. Neural Engine.
The names sound fast before you run anything.

So I had a question: would CoreML actually make this model faster?

I built inference-doctor to check ONNX deployments: output correctness, timing, worker memory, and the execution evidence behind a provider's name.

Correctness and timing results

On an Apple M5 Pro, I compared ONNX Runtime's CPU and CoreML Execution Providers using the same small Digits MLP and 450 held-out examples, with maximum batch sizes of 1, 16, 64, and 128.

Both matched the independent reference classes, and both scored 436/450 correct (96.89%). Logits and probabilities also passed their reference tolerance checks.

Then came the timing results.
CPU was faster at all four settings. At a maximum batch size of 128, warm inference time per example was:

CPU: ~0.076 µs
CoreML: ~0.694 µs

It felt like bringing a sports car to a race and losing to a family sedan—then discovering the finish line was the corner shop.

Choose the car after you know the route.

What provider evidence can establish

That analogy doesn't explain the result. Model size, workload, and deployment overhead need further investigation. Nor does a CoreML EP event prove GPU or Neural Engine use: profiling ran in a separate session, so it cannot establish identical placement in every timing trial.

This compares two deployment paths, not the theoretical power of two chips.

Amortized timing and request latency

The timing needs context too.

For maximum batch 128, each fresh process ran five complete timed passes over 450 examples after warmup, including the final 66-row batch. Total synchronous inference-call time was divided by 450×5; the reported value is the median of four trial averages.

It is not a 0.076 µs response time for an individual request.

Cook 100 bowls of noodles in ten minutes and you can advertise six seconds per bowl. Order one and wait ten minutes, though, and “on average” becomes a remarkably poor apology.

A separate wall-clock measurement of the timed passes includes slicing, output checks, class scoring, and coverage checks. Neither measure includes data loading, session creation, or warmup.

The scope of worker memory

Worker RSS includes the interpreter, libraries, full test/reference data, validation allocations, and runtime. It excludes the controller, profiling workers, CoreML helper services, and device memory. It cannot rank total deployment memory costs.

Measurement goals and engineering judgment

For me, the useful question is what each number can support.

Before investing in acceleration, are we targeting request latency, batch throughput, or total deployment cost? Each needs its own evidence.

When results disagree with intuition, change the judgment—not the benchmark.

Otherwise, the slide is ready before the speedup is:

“AI Acceleration Successfully Enabled.”

Enabled: yes.
Faster: not yet established.
The presentation, however, was finished in record time.