Job Schedulers and Workflow Management Systems

MMCloud includes a job scheduler for running batch jobs in the cloud. MMCloud integrates with workflow managers to enable pipeline execution.

Introduction

Workload managers, also known as job schedulers, are used to schedule, execute, and monitor batch jobs on clusters of compute nodes. Contention for resources is resolved by maintaining a queue of pending work. Workflow management systems are used to manage data-analytic pipelines, i.e., sequences of computational tasks such as found in bioinformatics.

Examples of job schedulers are:

• Slurm
• IBM Spectrum LSF ("load sharing facility")
• AWS Batch

Examples of workflow management systems are:

• Nextflow
• Cromwell
• Flyte

Job Scripts

A job script is a text file that has job setup information for the workload manager followed by commands to execute. A job script must start with a shebang (#!/bin/bash is commonly used but you can use others). A job script may be as simple as a few lines of shell script, but is often more complicated.

Most jobs submitted to OpCenter include a job script, but it is possible to submit a job without a job script as long as the container image has an "entrypoint" (in Docker terminology).