laya-coreml

laya-coreml moves Laya decision models onto Apple's Neural Engine, the low-power AI chip in every Apple Silicon Mac. Laya answers multiple choice, rating, and yes or no questions with probabilities instead of generated text, the open counterpart to TypeSafe's Jev. This is the same author's follow-up to mizorewww/laya-mlx, and it is Apache-2.0.

Install is pip install laya-coreml on macOS 15 or later. Weights download from Hugging Face on first load, then everything runs offline, with no PyTorch needed. The author measured about 5 ms per short question on an M3 Max, roughly 1.4x faster than the MLX port and about 2.8x less energy per decision.

Free at every size. Use it when a Mac app needs a fast yes or no or a routing decision thousands of times a session and battery matters. For longer prompts, laya-mlx is simpler.

The catch is the fine print. The Neural Engine build caps each request at 96 tokens total, question and options included. An experimental 1,024-token Neural Engine build took about 92 ms per request, and the author says the short-prompt speedup does not carry over to long prompts. The weights sit on a pseudonymous Hugging Face account, and this is a port of a port, a few days old.