Experiment Tracking
Definition
Recording and comparing different machine learning experiments, including their configurations, metrics, and results for informed decision-making.
Use Cases
- Databricks: Tracking and comparing model training runs across teams to improve reproducibility and speed up iteration — Databricks created and maintains MLflow, which logs parameters, metrics, and artifacts for each training run. Teams use MLflow Tracking to compare runs, and MLflow Model Registry to promote selected models to staging/production. (Standardized experiment logging improves reproducibility and collaboration, reducing time spent re-running experiments and helping teams identify best-performing configurations faster.)
- Microsoft: Managing and comparing machine learning experiments in enterprise workflows — Azure Machine Learning integrates MLflow Tracking so data scientists can log runs from notebooks and pipelines, store artifacts centrally, and compare metrics across experiments in the Azure ML studio UI. (Centralized tracking and governance make it easier for teams to audit results, reproduce training, and operationalize models with consistent metadata.)
Provider Equivalents
- AWS: Amazon SageMaker Experiments
- Azure: Azure Machine Learning (MLflow Tracking)
- GCP: Vertex AI Experiments
- OCI: OCI Data Science (MLflow Tracking)
Frequently Asked Questions
- What's the difference between experiment tracking and model registry?
- Experiment tracking records what happened during training (parameters, code version, metrics, artifacts) so you can compare runs and reproduce results. A model registry is for managing the lifecycle of selected models after training—versioning them, adding approvals, and promoting them to staging or production.
- When should I use experiment tracking?
- Use it as soon as you run more than a few training attempts or need reproducibility. It’s especially useful when you tune hyperparameters, compare different feature sets, run A/B model candidates, collaborate across a team, or must audit how a model was produced for compliance.
- How much does experiment tracking cost?
- Costs depend on (1) the tracking service pricing model and (2) the storage/compute you use. Many platforms include basic experiment tracking features as part of their managed ML service, but you still pay for training compute, artifact storage (e.g., object storage for models), and sometimes metadata storage or managed workspace fees. Self-hosting MLflow can reduce direct service fees but shifts cost to your infrastructure and operations.
Category: ai-ml
Difficulty: intermediate
Related Terms
See Also