Source time Edition date
IndustryAWS Machine Learning Blog
Distributed training with Ray Train DLCs and EFA on EKS
在 EKS 上使用 Ray Train DLC 和 EFA 进行分布式训练
Teams must decide whether Ray Train's prebuilt stack is worth trading image control for, since fine-tuning Qwen2.5-1.5B on eight GPUs works only within the container's fixed CUDA-to-EFA configuration.
Judgment
Platform teams deciding whether to standardize on AWS-built Ray Train DLCs must weigh faster cluster bring-up against ceding control of the CUDA, NCCL, EFA, and Ray versions baked into the image.
Basis
Sunita Koppar, Kevin Wang, and Sukhmeet Singh describe the container as the scaling bottleneck: every node needs matching CUDA, PyTorch, NCCL, EFA userspace libraries, the AWS OFI NCCL plugin, and an agreed Ray version. AWS builds, tests, patches, and publishes the Ray Train DLC in Amazon ECR Public; KubeRay builds the cluster from one RayCluster manifest, and Ray Train places eight workers, one per GPU, across EFA-enabled g6.12xlarge nodes. The mechanism trades image-level control for a pre-integrated stack.
Unclear The source does not quantify the operational cost of rebuilding custom images or compare performance against a self-managed Ray stack.
Next Check whether the published DLC's CUDA, NCCL, EFA, and Ray versions match the team's required PyTorch build before committing to the migration.