LiteRT

LiteRT is Google's runtime for running machine learning models directly on phones, laptops, browsers, and small boards instead of in the cloud. It is the renamed successor to TensorFlow Lite, which already runs inside a lot of Android apps. Version 2 adds a simpler compiled-model API that picks the GPU or NPU (the dedicated AI chip in newer phones) for you. Apache-2.0 and free.

Getting started is mostly an app-integration job, not a server to run. You convert a PyTorch, TensorFlow, or JAX model to the .tflite format and load it from Kotlin, C++, or JavaScript. NPU support covers Qualcomm, MediaTek, Google Tensor, Intel, and Broadcom. For large language models, Google points you to google-ai-edge/LiteRT-LM, which sits on top of this.

Use it when you ship an Android app or cross-platform product with on-device AI. On an Apple-only stack, Core ML is still the native choice. ExecuTorch from the PyTorch team and ONNX Runtime cover similar ground if your models already live in those ecosystems.

The catch is Apple and the upgrade path. Apple's Neural Engine is still marked coming soon, so on iPhone and Mac you run on the CPU or Metal GPU. And moving from the V1 API to V2 is a real upgrade, with its own migration guide.