Your tensor arena got 30% smaller. Why didn’t inference get faster?

Show an engineer a block of memory, and they’ll ask:
“Does it need to stay there?”

Think of an office with ten desks and six people.
You see space. An engineer sees a scheduling problem.

“If they use the desks at different times, could we get by with seven?”

The goal isn’t faster work. It’s fitting the work into the space we have.

Reuse from tensor lifetimes

That led to tiny-arena-lab.

I reused the model and 450 held-out examples already evaluated in tiny-infer-lab, comparing separate tensor storage with reuse after a tensor is no longer needed.

“No longer needed” matters.

An empty desk doesn’t mean someone left the company. They might be in the bathroom.
Give their computer to someone else too soon, and both people may still submit reports.
The reports just get a little more interesting.

My small C11 inference engine plans reuse from tensor lifetimes:
“Are you finished with that? Can I borrow the space?”

Every later read must be accounted for. Otherwise, memory optimization becomes memory corruption—and you save space by losing the right answer.

Capacity results and timing

At batch size 16, the digit classifier’s tensor arena needed:

Separate: 8,832 bytes.
Reuse: 6,144 bytes.

About 30.4% less arena space.
The separate layout exceeded the experiment’s 7 KiB arena budget; reuse completed inference within it.

Both layouts scored 438/450 (97.33%), with every predicted class matching the scikit-learn reference.

Speed?

Separate: about 0.685 µs per valid example.
Reuse: about 0.686 µs.

The observed process ranges overlap. That doesn’t establish a slowdown or a reliable speedup.

These are warm batch times amortized over valid examples, not individual-request latency: the median of six independent process medians. Timing includes final-batch padding, work counters and output-fingerprint calculation; counter/fingerprint comparisons happen after timing stops.

Did the optimization work?
For faster inference, these results don’t demonstrate success.
For fitting within the arena budget, they do.

One result. Two questions.

Check execution and validation

I also checked intermediate values and actual address offsets, used independent numerical references, and added compiled fault-injection tests and layout diagrams.

A neat diagram doesn’t prove the executor follows the plan. I deliberately made code skip work in the timed path to check whether validation would notice.

Experiment limits and engineering judgment

6,144 bytes is arena space, not total RAM. Weights, metadata, inputs and other runtime costs still count. The 7 KiB budget is an experiment setting; this Apple M5 Pro host experiment doesn’t prove MCU feasibility.

For engineering leaders, name the constraint before judging an investment. Capacity, latency, throughput and deployment cost aren’t interchangeable.

Otherwise:
“I finally got everything into my carry-on.”
“Will the plane fly faster now?”

No. But at least the bag fits.