Using Cromwell with MMCloud

To use Cromwell with MMCloud, you need the following:

• MMCloud Carmel 2.0 release or later
• OpCenter instance with valid license
• Cromwell Host (can be your local computer or a Linux virtual machine running in same VPC as the OpCenter)
• Cromwell Host with the following:
  • Java 11
  • Cromwell jar file
  • MemVerge's Cromwell configuration file
  • Job file in wdl format
  • Input file(s) in json format
  • Options file in json format (optional)
  • MMCloud CLI binary. You can download it from the OpCenter.

• Check the version of java installed on the Cromwell Host by entering:

  java -version
  openjopenjdk version "11.0.18" 2023-01-17 LTS
  OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-3.el9) (build 11.0.18+10-LTS)
  OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-3.el9) (build 11.0.18+10-LTS, mixed mode, sharing)

• If needed, install Java 11. Commercial users of Oracle Java need a subscription. Alternatively, you can install OpenJDK under an open-source licence by entering (on a Red Hat-based Linux system):

  sudo dnf install java-11-openjdk

• Create a directory called cromwell and cd to it
• Download the Cromwell jar file (85 is a recent version) from here and place it in the cromwell directory
• Check the Cromwell jar file by entering:

  java -jar cromwell-85.jar --help
  cromwell 85
  Usage: java -jar /path/to/cromwell.jar [server|run|submit] [options] args>...
  
    --help                   Cromwell - Workflow Execution Engine
    --version
  Command: server
  Starts a web server on port 8000.  See the web server documentation for more details about the API endpoints.
  Command: run [options] workflow-source
  Run the workflow and print out the outputs in JSON format.
    workflow-source          Workflow source file or workflow url.
    --workflow-root <value>  Workflow root.
    -i, --inputs <value>     Workflow inputs file.
    -o, --options <value>    Workflow options file.
    -t, --type <value>       Workflow type.
    -v, --type-version <value>
                             Workflow type version.
    -l, --labels <value>     Workflow labels file.
    -p, --imports <value>    A zip file to search for workflow imports.
    -m, --metadata-output <value>
                             An optional JSON file path to output metadata.
  Command: submit [options] workflow-source
  Submit the workflow to a Cromwell server.
    workflow-source          Workflow source file or workflow url.
    --workflow-root <value>  Workflow root.
    -i, --inputs <value>     Workflow inputs file.
    -o, --options <value>    Workflow options file.
    -t, --type <value>       Workflow type.
    -v, --type-version <value>
                             Workflow type version.
    -l, --labels <value>     Workflow labels file.
    -p, --imports <value>    A zip file to search for workflow imports.
    -h, --host <value>       Cromwell server URL.

• Download the OpCenter CLI binary for Linux hosts from the following URL:

  https://<op_center_ip_address>/float

  where <op_center_ip_address> is the public (if you are outside the VPC) or private (if you are inside the VPC) IP address for the OpCenter. If you download the CLI binary (called float) to your local machine, move the file to the Cromwell Host.
• Make the CLI binary file executable and add the path to the CLI binary file to your PATH variable.
• Open a file called cromwell-float.conf and insert the following contents.

  include required(classpath("application"))
  # This is an example of how you can use Cromwell to interact with float.
  
  backend {
    default = float
  
    providers {
      float {
        actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
        config {
          runtime-attributes="""
                  String user = "USERNAME"
                  String pass = "PASSWORD"
                  String addr = "OPCENTER_IP_ADDRESS"
                  String f_cpu = "MINVCPUS"
                  String f_memory = "MINMEMORY"
                  String f_docker = ""
                  String f_extra = ""
          """
  
          # If an 'exit-code-timeout-seconds' value is specified:
          # - check-alive will be run at this interval for every job
          # - if a job is found to be not alive, and no RC file appears after this interval
          # - Then it will be marked as Failed.
          # Warning: If set, Cromwell will run 'check-alive' for every job at this interval
          exit-code-timeout-seconds = 30
  
          submit = """
              mkdir -p ${cwd}/execution
              tail -n +22 ${script} > ${cwd}/execution/no-header.sh
              head -n $(($(wc -l  < ${cwd}/execution/no-header.sh) - 14)) ${cwd}/execution/no-header.sh > ${cwd}/execution/float-script.sh
              float sbatch -a ${addr} -u ${user} -p ${pass} -i ${f_docker} -j ${cwd}/execution/float-script.sh --cpu ${f_cpu} --mem ${f_memory} ${f_extra} >  ${cwd}/execution/sbatch.out 2>&1
              cat ${cwd}/execution/sbatch.out | sed -n 's/id: \(.*\)/\1/p' > ${cwd}/execution/job_id.txt
              echo "receive float job id: "
              cat ${cwd}/execution/job_id.txt
  
              mkdir -p float-instance-$(cat ${cwd}/execution/job_id.txt)
              cd float-instance-$(cat ${cwd}/execution/job_id.txt)
  
              echo "cd ${cwd}/execution" > float-check-alive.sh
              echo "float -a ${addr} -u ${user} -p ${pass} log cat -j \$1 stdout.autosave > stdout" >> float-check-alive.sh
              echo "float show -a ${addr} -u ${user} -p ${pass} -j \$1 --runningOnly" >> float-check-alive.sh
  
              echo "cd ${cwd}/execution" > float-kill.sh
              echo "float scancel -a ${addr} -u ${user} -p ${pass} -j \$1" >> float-kill.sh
              cat ${cwd}/execution/sbatch.out
          """
  
          kill = """
              source float-instance-${job_id}/float-kill.sh ${job_id}
          """
  
          check-alive = """
              source float-instance-${job_id}/float-check-alive.sh ${job_id}
          """
  
          job-id-regex = "id: (\\w+)\\n"
        }
      }
    }
  }

• Replace the following (keep the quotation marks).
  • USERNAME: username to log in to the OpCenter.
  • PASSWORD: password to log in to the OpCenter.
  • OPCENTER_IP_ADDRESS: IP address of the OpCenter (public or private if the Cromwell Host is outside or inside the VPC, respectively).
  • MINVCPUS: minimum number of vCPUs to use as the default for a Worker Node
  • MINMEMORY: minimum memory capacity (in GB) to use as the default for a Worker Node.

• The string following f_extra is combined with the string following f_extra in the wdl file and appended to the float submit command sent to the OpCenter. The f_extra string in the wdl file overrides the f_extra string in the configuration file if they are in conflict.

• Create a file called "helloworld.wdl" and insert:

  # Example workflow
  workflow myWorkflow {
      call myTask
  }
  
  task myTask {
      command {
          echo "hello world"
      }
      output {
          String out = read_string(stdout())
      }
  }

• Run the "hello world" job by entering:

  java -jar cromwell-85.jar run helloworld.wdl

  The output from Cromwell is verbose. To determine that the job runs successfully and to view the output, look for the following section:

  [2023-05-08 21:47:01,69] [info] BackgroundConfigAsyncJobExecutionActor [cb7c6fd7myWorkflow.myTask:NA:1]: Status change from - to Done
  [2023-05-08 21:47:03,27] [info] WorkflowExecutionActor-cb7c6fd7-9020-41cf-841f-c95f43ce86da [cb7c6fd7]: Workflow myWorkflow complete. Final Outputs:
  {
    "myWorkflow.myTask.out": "hello world"
  }
  [2023-05-08 21:47:06,67] [info] WorkflowManagerActor: Workflow actor for cb7c6fd7-9020-41cf-841f-c95f43ce86da completed with status 'Succeeded'. The workflow will be removed from the workflow store.
  [2023-05-08 21:47:09,04] [info] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.
  {
    "outputs": {
      "myWorkflow.myTask.out": "hello world"
    },
    "id": "cb7c6fd7-9020-41cf-841f-c95f43ce86da"
  }

A Cromwell workflow file can call many tasks. If the tasks are independent, Cromwell can scatter the tasks into parallel "shards." Once the shards are complete, the results can be gathered into a single output. The following example demonstrates different aspects of Cromwell.

• Task executed using Local as backend
• Task executed by OpCenter as backend
• Parallel sharding
• JSON files to supply input data and parameters

To run this example, complete the following steps:

• Create a file called 3step.wdl and insert the following content.

  workflow scatterGather {
      Array[String] names
  
      call intro
      scatter (name in names) {
          call addList { input: name=name }
      }
      call compileList { input: items=addList.out }
  }
  
  task intro {
      command {
          echo "Starting to compile the grocery list"
      }
      output {
          String out = read_string(stdout())
      }
      runtime {
          backend: "Local"
      }
  }
   
  
  task addList {
      String name
  
      command {
          printf "[cromwell-addList] Add ${name} to list\n"
          sleep 30
      }
      output {
          String out = read_string(stdout())
      }
      runtime {
          f_docker: "cactus"
          f_cpu: "2"
          f_memory: "4"
          f_extra: "--name addList"
      }
  }
  
  task compileList {
      Array[String] items
  
      command {
          printf "[cromwell-compileList] These items are on the grocery list:\n" > my_file.txt
          sleep 1
          echo ${sep=". " items} >> my_file.txt
          cat my_file.txt
          sleep 30
      }
      output {
          String out = read_string(stdout())
      }
      runtime {
          f_docker: "cactus"
          f_extra: "--name compileList"
      }
  }

• Create an input file called 3step.json and insert the following content.

  {
     "scatterGather.names": ["Apples", "Bananas", "Milk", "Bread"]
  }

• Create an options file called options.json and insert the following content.

  {
      "write_to_cache": false,
      "read_from_cache": false
  }

• Run the worklow by entering the following command.

  java -Dconfig.file=cromwell-float.conf -jar cromwell-85.jar run 3step.wdl --inputs 3step.json --options options.json
              

OpCenter shows the five tasks (four "scatter" tasks and one "gather" task).

float squeue -A -f name=List -d

+-----------------------+-------------+-------+-----------+----------------------+----------+------------+
|          ID           |    NAME     | USER  |  STATUS   |     SUBMIT TIME      | DURATION |    COST    |
+-----------------------+-------------+-------+-----------+----------------------+----------+------------+
| fihe9H73Mw7stcdCN1JPD | compileList | admin | Completed | 2023-05-09T20:51:17Z | 2m37s    | 0.0089 USD |
| JO2HXLD0WJLYVPW2npISI | addList     | admin | Completed | 2023-05-09T20:45:29Z | 4m3s     | 0.0025 USD |
| N8JDsSywHuKa3c7s8FZrl | addList     | admin | Completed | 2023-05-09T20:45:29Z | 3m59s    | 0.0024 USD |
| 4t9qV52z8B7423YtfLbve | addList     | admin | Completed | 2023-05-09T20:45:29Z | 3m59s    | 0.0024 USD |
| HsBFzGvPvhfekjFPSJ97A | addList     | admin | Completed | 2023-05-09T20:45:29Z | 4m5s     | 0.0025 USD |
(edited)

To start a Cromwell server on a local Cromwell Host, perform the following steps.

• On the Cromwell Host, enter the following command.

  java -Dconfig.file=cromwell-float.conf -jar cromwell-85.jar server

• Check that the Cromwell Host's inbound filtering rules allow access to port 8000
• Open a browser and go to http://<cromwell_host_ip>:8000 where <cromwell_host_ip>is the public (private) IP address of the Cromwell Host if you are outside (inside) the VPC.

To submit a job using the Cromwell web interface, complete the following steps.

• Expand the Workflows section and then click Submit a workflow for execution

  

• Click Try it out (on the right-hand side)

  

• Browse your local computer for a workflowSource wdl file
• Browse your local computer for a workflowInputs json file
• Browse your local computer for a workflowOptions json file
• Click Execute (at the bottom of the section). If the job is accepted, the server returns a code 201.
• Copy the workflow ID
• Click Get the outputs of a workflow
• Click Try it out and then paste the workflow ID into the id box
• Click Execute