Category: Amazon FSx for Lustre

At Snap Inc., the company behind Snapchat, cost is an important factor in every infrastructure decision. With research scientists and machine learning (ML) engineers working across a wide gamut of projects, from large language models to computer vision to video generation, the AI/ML platform team faces a constant balancing act: deliver the high-performance storage that […]

When running large language model (LLM) inference at scale on AWS, the GPU might not be the only thing that limits you. The GPU generates tokens fast, but what then contributes to performance is everything around it: memory, storage, and the network path that connects them. That’s the difference between a demo and production. Training […]

For nearly two decades, organizations across healthcare, drug discovery, and financial services have relied on SAS Grid to power their most critical analytics workloads. SAS Grid (formally known as SAS Grid Manager) is an enterprise software platform developed by SAS Institute that distributes and manages large-scale analytical workloads across a cluster of servers. It acts […]

The exponential growth in size and complexity of foundation models (FMs) has created unprecedented infrastructure demands across compute, networking, and storage resources. Storage systems, in particular, face intense requirements for throughput, latency, and capacity. In machine learning (ML) model training, these storage demands are particularly evident in checkpointing—a critical reliability mechanism that periodically saves and […]

Managing high-performance file storage can be a significant operational and cost challenge for many organizations, especially those running compute-intensive workloads such as high-performance computing (HPC) or data analytics. This is particularly true for organizations with existing data lakes on Amazon S3 who need POSIX-compliant, high-performance file system access. Amazon FSx for Lustre provides a scalable, […]

Imagine a company like a movie studio, one that works with enormous volumes of video files, scripts, and animation assets. They store these files on a high-performance file system such as Amazon FSx for Lustre, a fully managed shared storage built on the world’s most popular high-performance file system. Each file has metadata, such as […]

One of the primary benefits of using memory mapping (MMAP) in applications is saving memory on the client, sharing data amongst multiple threads and processes, and reducing file system impact to the application. Users are constantly looking for ways to improve application performance, and often this means diving deep into their workloads’ storage profiles to […]

Update 4/9/2025: The cross-account bucket policy in the blog has been updated. It was missing a required principal: “arn:aws:iam::accountID:role/AWS-Signed-In-Console-Role.” This omission causes an access denied error. As enterprises evolve their cloud governance practices, multiple teams working in separate accounts may need to share data. One team may oversee an enterprise data lake in one account, […]

AWS re:Invent 2023 is happening November 27th through December 1st in Las Vegas Nevada. AWS re:Invent offers cloud enthusiasts, experts, and practitioners an experience like no other. You get five days of opportunities to find productive solutions that will transform your company and your pick of 2,000+ learning sessions to quickly master new skills. And, […]

Building and testing software is a resource-intensive operation that usually involves a fleet of very powerful servers waiting in the wings for build jobs. With the rise of cloud native continuous integration/continuous development (CI/CD) systems on Kubernetes (i.e., Tekton, Jenkins X), we’re seeing a shift from the large (and often over-provisioned) static fleet of build […]