Configuring Data Volumes

A job that generates file system I/O requires a definition for one or more file systems.

Introduction

If you submit a job that generates file system I/O, that is, the job reads from or writes to files, you must specify the data volumes (the data storage spaces) to support the file systems created by the worker node. The OpCenter supports a range of data volume types. The type that is appropriate for your job depends on your requirements including the following.

Persistence after the job ends
I/O performance
Storage capacity
Cost

Note

For compliance (or other) reasons, some organizations require that the working directories generated by Nextflow or Cromwell workflows are archived securely. In such cases, a non-persistent data volume is not appropriate.

Architecture

After the user submits a job to the OpCenter (using the CLI or the web interface), the OpCenter selects and then instantiates a virtual machine (host). After the host is up and running, the host loads (and then starts) the container specified by the user. This is the worker node that executes the job.

There are existing volumes that are automatically mounted by the worker node and there are new volumes that are automatically created (and then mounted) by the worker node. The latter volumes are deleted once the job completes. The data volumes that are specified by the user when the user submits the job are mounted by the container running inside the worker node. The relationship of the different data volumes to the OpCenter and to the worker node is shown in the figure.

Volumes mounted by Worker Node

The Worker Node automatically mounts the following volumes.

Directory exported by the OpCenter to share binaries with the Worker Node and to provide an area for the Worker Node to store log files.
Root volume automatically created by the CSP when the host instance starts. The MMCloud user can specify the root volume capacity with the float submit --rootVolSize option or accept the default size of 40 GB.
Container image volume (/mnt/float-image) that acts as the root volume for the container. The MMCloud user can specify the image volume capacity with the float submit --imageVolSize option or accept the size automatically calculated by the OpCenter.
AppCapsule volume (/mnt/float-data) to store data and metadata needed to resume executing a job on a different host. The OpCenter automatically sizes the AppCapsule volume based on the memory capacity of the Worker Node.

AppCapsule defaults to using a local volume (for example, EBS in AWS). You can also use a JuiceFS file system on an object storage service, such as S3 in AWS.
- For an individual job, use float submit --snapLocation local|[s3]://BUCKETNAME to make your choice between local or JuiceFS (replace s3 with gs if you are running MMCloud in Google Cloud).
- To override the default behavior for all jobs, change the cloud.snapLocation system setting from local to [s3]://BUCKETNAME (replace s3 with gs if you are running MMCloud in Google Cloud).

The root, container image, and AppCapsule volumes are deleted after the job completes and the host is deleted.

The worker node also mounts data volumes if they are specified when the job is submitted. Data volumes are specified by using float submit with the --dataVolume option. Specify multiple data volumes by using --dataVolume multiple times in a single float submit command.

The OpCenter supports the following data volume types.

AWS Elastic Block Store (EBS) volume created by OpCenter for the duration of the job.
Existing EBS volume (for example, previously created by the AWS account holder) that persists after the job completes.
Directories that are exported by an NFS server and NFS-mounted by the worker node.
AWS Simple Storage Service (S3) bucket mounted as a file system by the worker node using a FUSE (Filesystem in USErspace) client.
Distributed file system that provides a POSIX-compliant API to access files where the underlying data is split into chunks and stored in one or more object storage systems (for example, S3). Files are only accessible using the client. The current release supports the following distributed file systems.
- JuiceFS
  
  JuiceFS is a high-performance, open-source distributed file system that can use a range of cloud storage services as the underlying object store. For example, JuiceFS can use Amazon's S3.
  
  JuiceFS separates raw data (for example, data chunks stored in S3) from the metadata used to locate and access the files. The metadata is stored in a database. The OpCenter instantiates a temporary Redis database. When the Redis database is deleted, files stored in JuiceFS are not accessible, so JuiceFS is considered a non-persistent file system in MMCloud
- Fusion
  
  Fusion is a distributed file system developed by Seqera for Nextflow pipelines to use cloud storage services provided by AWS S3 or Google Cloud Storage. Fusion is implemented as a FUSE client with an NVMe-based cache fronting a cloud storage service. Fusion requires the use of Wave, a container provisioning service integrated with Nextflow. SpotSurfer and WaveRider are not supported when Fusion is used with MMCloud.
- Lustre
  
  Lustre (the name is a combination of Linux and cluster) is an open-source distributed file system originally developed for the world's largest supercomputers and now widely used in high-performance computing clusters. Similar to JuiceFS, Lustre separates metadata from physical storage, which allows the Lustre file system to support access to the physical storage in parallel operations. A Lustre deployment requires multiple services that are deployed on separate servers, for example, management servers and metadata servers. Amazon provides a fully-managed Lustre service called Amazon FSx for Lustre. MMCloud users access Lustre file system resources by mounting directories provided by Amazon FSx for Lustre.

The following table compares data volume types.

Data Volume Type	Persistence	Performance	Capacity	Cost	Ease of Use
New EBS volume	No	High	Low	High	High
Existing EBS volume	Yes	High	Low	High	High
NFS mount	Yes	High	Medium	Medium	Low if need to create NFS server. Else, high.
S3 bucket	Yes	Low	High	Low	Medium
JuiceFS	No	High	High	Low	Medium
Fusion	Yes if cache is flushed	High	High	Low (open-source). High (commercial version).	Low
Lustre	Yes	High	High	High	Low

New EBS Volume

Use the float submit CLI command with the -D or --dataVolume option as follows.

--dataVolume [size=SIZE,throughput=RATE]:/DATADIR

Replace:

SIZE: capacity of EBS volume in GB
RATE: I/O throughput in Mbps (can omit the throughput parameter)
DATADIR: point where worker node mounts the EBS volume

Example:

--dataVolume [size=10]:/data

Equivalently, open the web interface and go to Submit Job>Start from Scratch>Basic.

Scroll down to Storage Volumes
Click Add and select Volume-New from the drop-down menu
Fill in the Size and Mount Point fields

Existing EBS Volume

Log in to your AWS Management console
Go to the EC2 Dashboard
If needed, create an EBS volume by completing the following steps.
- From the left-hand panel, select Elastic Block Store > Volumes
- On top, right-hand corner click Create volume
- Fill in the form and then click Create volume at the bottom, right-hand side of the page. If the volume is created successfully, you are returned to the Volumes console.
  
  Note
  
  AWS offers EBS volumes with different performance characteristics. Select gp3 for a general purpose volume. Select io2 for a high-performance volume.
From the Volumes console, select the volume to use
Copy the Volume ID. It is a string that looks like vol-08eafe32dac03f9a0.
Use the float submit CLI command with the -D or --dataVolume option as follows.

--dataVolume VOLUME_ID:/DATADIR

Replace:
- VOLUME_ID: volume ID identifying the existing EBS volume
- DATADIR: point where worker node mounts the EBS volume

Example:

--dataVolume vol-08eafe32dac03f9a0:/data

Equivalently, open the web interface and go to Submit Job>Start from Scratch>Basic.

Scroll down to Storage Volumes
Click Add and select Volume-Existing from the drop-down menu
Fill in the Volume ID and Mount Point fields

NFS Mount

If needed, create an NFS server in the same VPC as the OpCenter. To use the NFS server successfully, check the following items.
- There is network connectivity between the private IP addresses of the NFS server and the OpCenter.
- The NFS server allows inbound access to port 2049 (apply an appropriate security group if needed).
- The subnet mask in /etc/exports allows the worker node to mount file systems from the NFS server.
Use the float submit command with the -D or --dataVolume option as follows.

--dataVolume nfs://NFS_PRIVATE_IP/EXPORTED_DIR:/MOUNTED_DIR
Replace:
- NFS_PRIVATE_IP: private IP address of the NFS server
- EXPORTED_DIR: directory (or path to directory) exported by the NFS server
- MOUNTED_DIR: mount point (or path to mount point) where worker node mounts the exported directory

Example:

--dataVolume nfs://172.31.53.99/mnt/memverge/shared:/data

Equivalently, open the web interface and go to Submit Job>Start from Scratch>Basic.

Scroll down to Storage Volumes
Click Add and select NFS from the drop-down menu
Fill in the NFS Path (use format: nfs://NFS_SERVER_IP/EXPORT_DIR) and Mount Point fields

S3 Bucket

If needed, create an S3 bucket by completing the following steps.

Log in to your AWS Management Console
Open the Amazon S3 console
From the left-hand panel, select Buckets
On the right-hand side, click Create bucket and follow the instructions

Note

You must choose a bucket name that is unique across all regions except China and the US government. Buckets are accessible across regions.

Access Keys

Depending on how your AWS account is set up, you may have to provide security credentials to enable access to the S3 bucket you created. To generate security credentials for your AWS account, complete the following steps.

Log in to your AWS Management Console
On the navigation bar, all the way to the right, click your username and go to Security credentials
Scroll down the page to the section called Access keys and click Create access key
Download the csv file. The csv file has two entries, one called Access key ID and one called Secret access key.

Note

If the S3 bucket belongs to another AWS user, obtain the security credentials from the S3 bucket owner.
Store the security credentials (in encrypted form) in the OpCenter secret manager by entering the following two commands.
```
float secret set KEY_NAME ACCESS_KEY_ID
float secret set SECRET_NAME SECRET_ACCESS_KEY
```
Replace:
- KEY_NAME: name to associate with the access key ID
- ACCESS_KEY_ID: access key ID
- SECRET_NAME: name to associate with secret access key
- SECRET_ACCESS_KEY: secret access key
Example:
```
$ float secret set bucketaccesskeyid A***C
Set bucketaccesskeyid successfully
$ float secret set bucketaccesssecret X***S
Set bucketaccesssecret successfully
```

Retrieve the security credentials by entering the following two commands.

$ float secret get KEY_NAME
 A***C
$ float secret get SECRET_NAME
 X***S

S3 Bucket as Data Volume

Use the float submit command with the -D or --dataVolume option as follows.

--dataVolume [accesskey=A***C,secret=X***S,endpoint=s3.REGION.amazonaws.com,mode=rw]s3://BUCKETNAME:/MOUNTED_DIR

Replace:

A***C: access key ID (omit if not needed)
X***S: secret access key (omit if not needed)
REGION: region in which S3 bucket is located
BUCKETNAME: S3 bucket name (or bucket-name/subfolder)
MOUNTED_DIR: mount point (or path to mount point) where worker node mounts the S3 bucket

Equivalently, open the web interface and go to Submit Job>Start from Scratch>Basic.

Scroll down to Storage Volumes
Click Add and select Cloud Storage from the drop-down menu
Fill in the S3 Path and Mount Point fields. For S3 Path, use format: [accesskey=xxx,secret=yyy,endpoint=s3.region.amazonaws.com,mode=rw]s3://bucketname/path

JuiceFS

JuiceFS is used with applications (such as Nextflow) that require high-performance access to large data sets and the cost of a block storage volume is prohibitive.

When JuiceFS is used as a data volume for a job that runs on a single worker node, the JuiceFS metadata server runs as a process on the worker node. When JuiceFS is used as a shared volume for a pipeline (such as Nextflow), the JuiceFS metadata server runs on one host (for example, the Nextflow host) that is accessed by all worker nodes. In this case, a firewall rule to open port 6868 is applied to the host and all worker nodes.

To use JuiceFS as a data volume for an application running on a single worker node in AWS, complete the following steps.

Log in to your AWS Management console
Create an S3 bucket to use as the underlying data store
Use the float submit command with the -D or --dataVolume option set as follows:

--dataVolume [accesskey=A***C,secret=X***S]jfss3://BUCKETNAME:/MOUNTED_DIR

Replace:
- A***C: access key ID (if user needs credentials to access S3 bucket)
- X***S: secret access key (if user needs credentials to access S3 bucket)
- BUCKETNAME: S3 bucket name
- MOUNTED_DIR: mount point (or path to mount point) where worker node mounts JuiceFS
Note

Do not include any folders in the S3 bucket name.

Example:

float submit -i blast --tag latest -j jfsblast.sh -t c6i.8xlarge --dataVolume [accesskey=***,secret=***]jfss3://welcometojuicefs:/data

Equivalently, open the web interface and go to Submit Job>Start from Scratch>Basic.

Scroll down to Storage Volumes
Click Add and select Cloud Storage from the drop-down menu
Fill in the jfss3 Path (use format: [accesskey=xxx,secret=yyy,mode=rw]jfss3://bucketname/path ) and Mount Point fields

Fusion

Fusion is only used with Nextflow pipelines. Using Nextflow with MMCloud requires nf-float, the Nextflow plugin for MMCloud. A description of nf-float, its use, and the configuration changes required to use Fusion, are available here.

When Fusion is used with MMCloud, SpotSurfer and WaveRider are not supported. To turn off WaveRider and to specify On-demand Instances when using Fusion, insert these lines into the Nextflow configuration file (don't duplicate lines if they are already in the configuration file).

fusion {
  enabled                  = true
  exportStorageCredentials = true
  exportAwsAccessKeys      = true
}

float {
    address = 'OPCENTER_PRIVATE_IP'
    username = 'USERNAME'
    password = 'PASSWORD'
    vmPolicy = [
        onDemand: true,
        retryLimit: 3,
        retryInterval: '10m'
    ]
    migratePolicy = [
        disable: true
    ]
}

Lustre

Before using Lustre with MMCloud, you must configure AWS FSx for Lustre.

Sign in to the AWS Management Console
Go to the FSx for Lustre console
Click Create file system
Follow the three-step procedure for creating an Amazon FSx for Lustre file system
When the process completes, copy the DNS name (it is a string that looks like fs-00cbcc67abcb6872d.fsx.us-east-1.amazonaws.com) and the mount name (it is a string that looks like yqgfrb4v).

Use the float submit command with the -D or --dataVolume option set as follows.

--dataVolume [opts=xxx,xx]lustre://DNSNAME/MOUNTNAME/PATH:/MOUNTDIR

Replace:

DNSNAME: DNS name associated with Amazon FSx for Lustre file system
MOUNTNAME: mount name associated with Amazon FSx for Lustre file system
PATH (optional): path to directory in Lustre file system
MOUNTDIR: directory where Lustre file system is mounted by the container running the job

If providing additional mount options enclosed in [], string must begin with opts= and must not contain spaces. Separate options with a comma.