Question 1

What's the difference between Model Serving and Model Training?

Accepted Answer

Model training is the process of learning model parameters from data (usually offline and compute-heavy). Model serving is putting the trained model behind an API so applications can send inputs and get predictions quickly and reliably in production.

Question 2

When should I use Model Serving?

Accepted Answer

Use model serving when an application needs predictions on demand (milliseconds to seconds), such as fraud checks during checkout, real-time recommendations, chat or translation APIs, or dynamic pricing. If you only need periodic predictions (e.g., nightly scoring of a customer list), batch prediction may be simpler and cheaper than always-on serving.

Question 3

How much does Model Serving cost?

Accepted Answer

Cost depends mainly on (1) compute type (CPU vs GPU/accelerators), (2) how many replicas you run and whether they are always on, (3) autoscaling behavior and traffic volume, (4) memory/model size and cold-start overhead, (5) networking and load balancing, and (6) observability features (logs/metrics/traces). Managed endpoints typically charge per instance-hour (or per request for serverless options) plus any data transfer and storage. Foundation-model APIs usually charge per token or per request.

Model Serving

Definition

Real-World Example

Related Terms

Cloud Provider Equivalencies

Explore More Cloud Computing Terms