What It Actually Takes to Trust a Local Model

I'm building a personal AI assistant that runs a small fleet of local models instead of one general-purpose model: a fast, cheap worker model for grunt-work tasks like pulling structured data out of messy text, and a slower, larger model as a fallback for anything that needs actual reasoning. That design only works if the worker calls are cheap and fast enough to run many of them in parallel without hitting a commercial API every time. Cheap and fast, on a Mac, meant asking a question most MLX users never have to: is Apple's own native inference stack, not MLX, actually viable for this?

That stack has two names that will be unfamiliar if you've only used MLX. CoreML is Apple's existing framework for running models directly on the Neural Engine, GPU, or CPU. Core AI is the newer, still-beta successor Apple is building specifically for this kind of on-device LLM work. Neither is what most local-inference guides point you toward, and there's a reason for that: they're harder to use, more narrowly documented, and it wasn't obvious going in whether either could actually hold up model quality once real requests started hitting them.

Six weeks and three phases of work later, here's what actually happened, and why the process that got me there turned out to matter more than the answer itself.

Getting weights into a shape macOS can run fast

CoreML is the obvious starting point on macOS. Apple's framework compiles model weights into .mlmodelc bundles that load directly onto the Neural Engine, GPU, or CPU, whichever the hardware and firmware will allow. LFM2.5-350M from LiquidAI was the candidate for the worker-tier role: small enough that several variants fit on disk alongside a larger fallback, fast enough to handle many concurrent calls.

Converting LFM2.5 from its HuggingFace checkpoint to CoreML meant running it through coreml-llm, which produces an Apple .mlpackage that gets compiled on-device into .mlmodelc. That compilation step is the whole point. It's what makes CoreML fast at inference time: ahead-of-time compilation to hardware-specific IR, not just weight packaging.

The conversion wasn't one artifact. Two axes, quantization (fp16, int8) and context window (2K through 32K), produced a matrix of candidates, because context length drives KV-cache size per concurrent request and quantization trades quality against throughput. Three configs survived to quality testing:

• lfm2.5-350m-int8-2k: the speed option. ~284 tokens/sec, minimal context, lowest memory.
• lfm2.5-350m-fp16-16k: longer context, ~47 tokens/sec, room for bigger documents.
• lfm2.5-1.2b-fp16-32k: the fallback. Nearly 4x the parameters, ~3.2 tokens/sec, the candidate for tasks the 350M models couldn't handle.

One thing surfaced during conversion that shaped everything after it: LFM2.5's architecture carries a rolling convolutional state, conv_state_in / conv_state_out, tracked separately from the standard KV cache. A normal transformer only needs a KV state between decode steps. LFM2 needs both. Get the state management wrong and the output doesn't obviously break. It just quietly degrades. That's a worse failure mode than a crash, because nothing tells you it happened.

Answering the architecture question before writing a line of bridge code

Before any serving code got written, there was a prior question worth settling properly: how should concurrent requests share a loaded model?

The naive approach, one MLModel per request, is easy to reason about and expensive in practice. Loading an MLModel from disk costs 250MB to 2,300MB depending on the config, and that cost hits on every load. Ten concurrent requests means ten loads, which on a 32GB unified-memory MacBook is a real constraint, not a theoretical one.

The correct pattern, confirmed across 16 benchmark configurations in a prior test harness, is to load one MLModel once and allocate a fresh MLState per request. The state holds the KV cache and conv state for a single sequence: cheap (24–385MB) and fully isolated. The weights never duplicate. This is Apple's intended architecture, but it's not something you'd derive from the API docs alone; the benchmark is what turned "probably" into "confirmed." Concurrency gains were real but modest, with the 350M int8-2K config hitting roughly 1.16x aggregate throughput at 4 concurrent requests, which mattered for capacity planning more than it mattered for the yes/no on the architecture itself.

The same benchmark also caught an environmental bug that would keep resurfacing: macOS 26 Tahoe's ANE firmware fails silently on .all compute units and falls through to GPU or CPU without saying so. The fix isn't in the code. It's a fallback chain: request .all, catch the failure, try .cpuAndGPU, then .cpuOnly. When Apple ships the firmware fix, nothing needs to change; .all just starts working.

Building something that boots clean

The bridge (the service that takes a request and routes it to the right model) is a Swift binary on Vapor, structured around two actors. ModelRegistry holds one ModelWorker per model ID and pre-loads everything listed in config.json at boot. If any model fails to load, the process exits immediately, because a loud failure at startup beats a mysterious one three requests into production. ModelWorker holds the loaded model and, for each request, allocates a fresh state, runs prefill, decodes, then discards the state. No cross-request leakage, which matters specifically because LFM2's rolling conv state would otherwise carry one caller's context into another caller's response.

Getting it to boot cleanly took one real debugging round. The 1.2B FP16 model triggered an OOM crash during GPU compilation at startup. The driver tried to build an execution plan for a 2.4GB model in GPU memory, hit a limit, and macOS issued SIGKILL. The log just stopped:

No outputs specified. @ Impl

The fix was pinning the 1.2B model to .cpuOnly in config. Slower, but it loads. Two smaller issues got fixed in the same pass: force-unwraps on CoreML output values (safe until a model returns an unexpected shape, then a crash) became guarded unwraps with real errors, and the bridge started rejecting over-length prompts outright instead of silently passing invalid KV-cache positions into the model. After that, all three models loaded and warmed on the first try:

[lfm2.5-350m-int8-2k] Loaded successfully and warmed up with computeUnits: MLComputeUnits(rawValue: 1)
[lfm2.5-350m-fp16-16k] Loaded successfully and warmed up with computeUnits: MLComputeUnits(rawValue: 0)
[lfm2.5-1.2b-fp16-32k] Loaded successfully and warmed up with computeUnits: MLComputeUnits(rawValue: 0)
[Boot] CoreML Bridge HTTP server starting on http://127.0.0.1:8090

The question that actually mattered: can it do the work

A model that boots and serves requests hasn't proven anything yet. The real test was a 13-task suite across five categories, built to map directly onto what a worker model like this actually needs to do: extraction, classification, retrieval Q&A, verification, and code review. Every task is retrieval-grounded, meaning the answer lives in the provided context and nowhere else. That's not a design preference. A worker that answers from parametric memory when the passage doesn't contain the answer is a hallucination risk in production, not a quality nitpick to shrug off.

All three models ran the full suite at temperature 0.0, 39 calls total:

Both 350M variants landed on exactly 5/13, failing the identical tasks for identical reasons, which is itself the finding. Quantization and context window made no difference to the failure pattern, ruling out int8 precision loss as the culprit. D1 and D2 require holding a claim and a source in attention at once and reasoning about the relationship between them, which is inference rather than retrieval, and at 350M parameters that gap held across both variants. C1 was the more interesting failure: a direct question with the answer plainly stated in the passage, and the model failed it anyway. That pointed to a prompt-attractor effect, where the model's prior for how to answer a question competed with the instruction to pull the answer from the given text. Rephrasing the prompt to anchor harder on the passage improved C1 sometimes, but not consistently enough to change the verdict. E2 showed a different pattern entirely: the model flagged the correct function (one with a legitimate empty-list guard) as suspicious, exactly the false-positive tendency that task was built to catch.

Ruling out the bridge before blaming the model

Before concluding anything about the 350M model's ceiling, there was a more careful question to ask: is any of this the bridge's fault? HTTP routing, chat template rendering, input validation, token decoding through CoreML's Swift API. Any layer in that stack could in principle degrade quality relative to what the raw weights can do. Ruling out the infrastructure before blaming the model is the only methodologically honest order to do this in.

This is where MLX earned its keep, as the ground truth to test against rather than the thing being tested. The isolation test loaded the identical weights through mlx_lm, pulling LiquidAI/LFM2.5-350M-MLX-8bit straight from HuggingFace and running it natively through MLX. No bridge, no HTTP layer, no Swift CoreML API in between. Same 13 tasks, same prompts, same grading, temperature 0.0.

Result: 5/13. Same tasks failed. Same tasks passed.

That settles it cleanly. The 350M LFM2.5 model has a genuine capability ceiling at this scale, and it shows up identically whether you run it through a Swift HTTP bridge or MLX's Python runtime with unmodified HuggingFace weights. The bridge isn't degrading anything. The model is the model. And the 1.2B model's 10/13 proves the suite itself is solvable; nothing in the test design demands capability that doesn't exist anywhere in the LFM2.5 family, it just doesn't exist yet at 350M parameters for reasoning and multi-step synthesis. Extraction and classification, the 350M models handle well. Hallucination refusal, the one property this whole design actually depends on for safety, held across every config.

350M is unreliable for reasoning. 1.2B is viable. The runtime choice doesn't change either verdict.

A five-step protocol, not a one-time answer

None of this was really about LFM2.5. It's a reusable protocol for evaluating any local model under any runtime, and it's the reason this exercise was worth doing carefully instead of quickly.

Pin the weight provenance first: exact checkpoint, exact quantization, exact conversion pipeline. "350M int8" isn't specific enough on its own; context window and conversion tool both change behavior. Establish a runtime isolation baseline second, running raw weights through a direct runtime like MLX before testing anything that touches a serving layer. Without that baseline, a failure through the bridge is unattributable. Run the pass/fail quality suite third. The 13-task structure is directly reusable against any candidate, and the hallucination-refusal test stays non-negotiable regardless of how well a model scores elsewhere. Compare the bridge's results against the isolation baseline fourth. A match means the serving layer is clean; a divergence narrows the search to exactly what the bridge adds. And for anything landing just below threshold, run a prompt-sensitivity ablation fifth. The C1 result proved that wording shifts individual task outcomes even when the underlying capability is fixed, which doesn't change a model's verdict but matters enormously for what ships in production.

The whole suite runs in minutes. The isolation test adds an hour, most of it spent waiting on a model download. That's a small price for a clean, well-characterized capability verdict, and a much smaller price than shipping a worker that fails quietly at reasoning tasks once it's live.

Then Apple shipped a beta, and the framework got to prove itself

The original plan filed Core AI, Apple's .aimodel format and the intended successor to CoreML's .mlpackage, under "test this later." Later arrived the same day macOS 27 Golden Gate beta 4 shipped, with an explicit changelog entry for bug 176210080: a fix for ANE failures on certain weight configurations, precisely the failure class the 16-pass benchmark had been documenting for weeks. The upgrade happened immediately.

The ANEF -14 error the bridge had been working around didn't go away, because the fix targeted .aimodel models specifically, and the bridge's models are .mlpackage, a different format running a different runtime path entirely. "Core AI fixed the ANE bug" turned out to be true and also not applicable here. One real improvement did show up: .cpuAndGPU now loads and warms cleanly for the small models, where it previously triggered a Metal OOM on macOS 26. GPU execution became available for .mlpackage models that didn't have it before, but it's still not ANE, and still not the format the fix actually targeted.

Testing Core AI properly meant testing it on its own terms. apple/coreai-models, Apple's own export recipes, Python primitives, Swift runtime, and CLI tools, built cleanly against the beta 4 SDK. Exporting Qwen3-0.6B to .aimodel format took one command:

uv run coreai.llm.export Qwen/Qwen3-0.6B --output-dir /tmp/qwen3-0.6b-int4

The benchmark result: 1,426 tokens/sec prompt processing, 159.5 tokens/sec generation. That's the first end-to-end Core AI inference on this machine, and 159 tok/s for a 0.6B model is fast, roughly 19x faster than the CPU/BNNS path the CoreML bridge runs. powermetrics showed it landing on GPU, not ANE: 6,261 mW peak, zero ANE watts, and the export directory literally named gpu-pipelined. The compute placement decision gets made at export time by Apple's own pipeline, and for macOS today, that pipeline picks GPU for LLM generation. The ANE fix in beta 4 removed a blocker for ANE-targeted models. It didn't change where Apple's own scheduler decides to run an LLM on an M2.

The harder test was LFM2.5-8B-A1B, already sitting on disk in Core AI format via a community conversion. Apple's own llm-runner rejected it outright:

Error: Invalid output type for 'Expected 2 states (KV cache), got 3:
["keyCache", "valueCache", "convState"]'

Apple's toolkit assumes two KV states. LFM2.5 has three: the same rolling conv state that mattered back at the CoreML conversion stage is still there, and it's still not something a standard transformer runner knows to allocate for. Not a bug. A gap in what Apple's catalog currently supports. A community coreai-kit has a custom LFM2 engine that does handle the three-state architecture, but getting it to build against beta 4 meant patching three separate API breaks: a get-only property that used to be mutable, a type that became noncopyable and needed explicit ownership semantics, and finally a runtime crash resolving tensor layout for the MoE routing tensors:

CoreAIRuntime/NDArray+Layout.swift:126: Fatal error:
Cannot make a Tensor.Layout from an unresolved TensorRequirements

That last one lives inside Apple's own CoreAIRuntime, not in the community kit's code. The beta's API surface moved faster than the kit could track it. Two independent paths, two independent verdicts, both landing in the same place: LFM2.5 isn't runnable on Core AI yet on beta 4, whether the wall is architectural (Apple's toolkit) or a moving beta target (the community kit). MLX, with first-party LiquidAI weights and a mature generation loop and no three-state gap to work around, stays the correct runtime for LFM2.5 for now.

What Core AI actually is, now that it's been used instead of read about

The export pipeline works cleanly. The benchmark tooling is solid. Generation throughput on GPU beats the CoreML/BNNS path by a wide margin for models the toolkit already supports. Qwen3, Gemma3, Mistral, and Mixtral all convert and run without incident, and the repository is actively maintained under a real open-source license. That's the genuine upside, and it's not a small one.

The nuance is that Core AI is GPU-first for LLMs today, not ANE-first, regardless of what the beta 4 changelog might suggest to someone skimming it. The power efficiency story that made ANE the interesting target in the first place doesn't show up yet for generative models the way it does for smaller, non-generative ones, and whether that changes in a future beta is Apple's call, not something a weight-format fix decides. The limitation that actually matters here: any architecture that isn't a plain two-state KV transformer needs explicit engine support before it runs at all, and LFM2.5's hybrid conv-attention design with its third state tensor is outside that support today. The community model zoo is building toward it. It's chasing a beta that keeps moving.

Where this leaves things, and what's next

The bridge works. It serves three models from one process over a clean OpenAI-compatible API, the error handling is explicit instead of mysterious, and the quality it delivers is exactly what the raw weights are capable of, no more, no less. For 350M-scale models specifically: trustworthy for extraction and classification, not yet trustworthy for reasoning or multi-step synthesis, and the routing logic needs to reflect that ceiling rather than hope around it. For Core AI: real, fast, and ready today for standard transformer architectures, with a clear migration path once it's needed, but not yet an option for LFM2.5-family workers until either Apple adds engine support or the community kit stabilizes against a beta that's stopped moving.

The part worth keeping regardless of which model wins is that the testing protocol doesn't care what it's evaluating. Weight provenance, runtime isolation, the pass/fail suite, bridge comparison, prompt-sensitivity ablation: the same five steps, in the same order, produce a clean verdict whether the subject is a 350M CoreML worker or whatever ships in Core AI's LFM2.5 support six months from now (including the rest of the local model fleet I'm now running it against — planner, coder, and reviewer candidates, a story for another post). That's the property worth building for in infrastructure testing generally: not a framework that hopes for a particular answer, but one that tells you what's actually true and moves on.