Job Schedulers and Workflow Management Systems
Memory Machine CE includes a job scheduler for running batch jobs in the cloud.
Introduction
Workload managers, also known as job schedulers, are used to schedule, execute, and monitor batch jobs on clusters of compute nodes. Contention for resources is resolved by maintaining a queue of pending work. Workflow management systems are used to manage data-analytic pipelines, i.e., sequences of computational tasks such as found in genomic analyses.
- Slurm
- IBM Spectrum LSF ("load sharing facility")
- AWS Batch
- Cromwell
- Nextflow
Job Scripts
A job script is a text file that has job setup information for the workload manager followed by commands to execute. A job script must start with a shebang (#!/bin/bash is commonly used but you can use others). A job script may be as simple as a few lines of shell script but is often more complicated.
Most jobs submitted to OpCenter include a job script, but it is possible to submit a job without a job script.