MMBatch Overview
What is Memory Machine™ Batch?
Memory Machine Batch (MMBatch) captures the entire running state of a Batch Job into a consistent image and restores the Job on a new Compute Instance without losing any work progress. It ensures a high quality of service at the Batch level using low-cost, but unreliable Spot-based Compute Instances. For more details, visit the MMBatch website.
The MMBatch's key features include:
- Full integration into the customer Batch environment
- Automated checkpoint and restore
- No change to the customer workflow
- No change to the Job applications and Workflow Manager scripts
- Scalable across thousands of Batch Jobs and Compute Instances
- Secure data processing within the customer VPC
Supported Clouds and Batch Environment
AWS Batch ECS
Supported Machine Image
Amazon Linux 2023 with ECS support (kernel 6.1)
MMBatch Architecture
The MMBatch integrates with AWS Batch ECS. The architecture is shown below:
The AWS Batch and MMBatch components include:
- In the AWS Batch Queue, under Compute Environment, each Compute Instance needs to have the MMBatch installed. The MMBatch is responsible for the checkpoint and restore of the Batch job containers, as well as checkpoint image lifecycle management. The MMBatch is shown as a solid blue block and delivered as an installer package.
- Each AWS Batch Queue requires dedicated storage for the MMBatch checkpoint images. The storage options include:
a. S3 bucket and the open-source JuiceFS to map file access to S3’s object storage, shown as a shaded blue block.
b. Alternatively, MMBatch checkpoint images can be stored directly in EFS, FSx Lustre, and other NFS systems.
List of Cloud Service Provider Service Used
| Service | Required/Optional | Usage |
|---|---|---|
| AWS CloudWatch | Optional | MMBatch logging |
| AWS Cognito | Optional | MMBatch Management Plane login |