FSx for Lustre

AWS's fully managed high-performance parallel file system, built on the open-source Lustre file system and designed for workloads that require sub-millisecond latency and hundreds of gigabytes per second of aggregate throughput. Where EFS and FSx for Windows File Server are general-purpose shared file systems, and FSx for NetApp ONTAP targets enterprise multi-protocol environments, FSx for Lustre is purpose-built for compute-intensive workloads — HPC simulations, ML training jobs, genomics pipelines, financial risk modelling, and video rendering — where thousands of compute nodes must read and write large datasets simultaneously without a storage bottleneck. FSx for Lustre offers two deployment types: scratch file systems, which are optimised for short-lived, cost-sensitive workloads and provide no data replication or persistence after the file system is deleted; and persistent file systems, which replicate data within an Availability Zone and are suited for long-running jobs where data durability matters. Both types can be natively linked to an S3 bucket — lazy-loading objects from S3 on first access and optionally writing results back to S3 when processing is complete, so you only pay for active compute storage rather than cold data at rest. Integration with SageMaker and AWS Batch is first-class: training jobs and batch tasks can mount an FSx for Lustre file system directly, eliminating the overhead of copying a large training dataset from S3 to local instance storage before each run. Storage is POSIX-compliant, so existing Linux applications and MPI-based HPC codes work without modification. The closest equivalent on other clouds is Azure Managed Lustre (preview) and, for lower-performance HPC storage, GCP Filestore Enterprise. Use FSx for Lustre when job throughput is storage-bound and your workload is compute-intensive and burst-oriented; use EFS when you need a simple, automatically scaling NFS share across multiple Linux instances; use FSx for NetApp ONTAP when you need multi-protocol access or ONTAP data management.

A genomics company runs an AWS Batch pipeline that processes thousands of whole-genome sequencing files per day. They create an FSx for Lustre file system linked to the S3 bucket holding the raw FASTQ files. Each Batch job mounts the Lustre file system at startup and reads only the files it needs — which are transparently loaded from S3 on first access — achieving 200 GB/s aggregate read throughput across the compute fleet. Finished VCF output files are exported back to S3 automatically, and the scratch file system is deleted at the end of the batch run to avoid ongoing storage costs.