A software runtime that executes trained machine learning models to generate predictions or text from new input data. Inference engines optimize for latency, throughput, and hardware utilization — converting model weights into responses as efficiently as possible. Modern LLM inference engines (vLLM, TensorRT-LLM, llama.cpp) use techniques like continuous batching, KV-cache management, and quantization to maximize tokens-per-second on GPU hardware.
A DigitalOcean GenAI Platform deployment uses an inference engine under the hood to serve Llama 3 70B. When a user sends a chat message, the engine loads the model from GPU memory, processes the prompt tokens through attention layers, and streams back the response — achieving 2× higher throughput than a naive model server through continuous batching.