Skip to content

Filters

Use filters to limit a command to a subset of all jobs, hosts, or gateways.

Feature Description

One OpCenter may schedule hundreds of workflows and thousands of jobs. Each workflow may comprise hundreds of jobs. Each job runs on its own host. Without any filters, an action applied to jobs (such as listing or canceling), both current and historical, may select thousands of entries, making the display difficult to navigate. With job filters, you can limit the action to a subset of all jobs. For example, you can select only the jobs submitted by a certain user or jobs submitted in a certain time range or jobs in a certain operational state. You can also combine filters using boolean operators.

Similarly, host filters select a subset of all hosts and gateway filters select a subset of all gateways.

Operation

Every job scheduled by the OpCenter has associated metadata, including the user who submitted the job, the time the job was submitted, the job name, the container image requested, the host(s) on which the job executes, the gateway the job connects to, and so on. You can use many of these metadata components (or keys) to filter jobs, hosts or gateways.

Filters have the following structure.

[key][operator][value]

where key is one of metadata parameters, value matches the data type of key (that is, string, integer, float, datetime or boolean), and operator is one of the following.

  • = (use with any data type)
  • >= (use with integer, float or datetime only)
  • <= (use with integer, float or datetime only)
  • > (use with integer, float or datetime only)
  • < (use with integer, float or datetime only)

Wildcards are allowed with string variables. Use "*" to match any number of characters and "?" to match a single character.

Build complex filters by combining single filters with "and" or "or" (use parentheses to specify the order in which filters are evaluated).

Examples:

  • user=admin?
  • name=*
  • cpu>16
  • running=true
  • submitTime>=2024-02-01T21:08:43-05:00 and submitTime<=2024-02-20T21:08:51-05:00
  • (owner=bean or owner=pea) and duration>4h
  • (status=Cancelled or status=Completed) and imageID=*docker.io/memverge/sentieon:latest*

Filters are used with float commands that select a group of jobs, hosts, or gateways. Such commands include the following.

  • cancel
  • gateway list
  • hosts
  • list (squeue)
  • report get

Note

When using filters with the CLI running in a linux shell, the shell may interpret filters as shell commands. To avoid this, enclose the filter in quotes, for example,
--filter "duration>1h".

Job Filters

The table shows the keys that you can use in job filters.

Key Type Value Example
id string id=w700gdr6owzjjgdim5m0p
name string name=admin
user string user=admin
duration duration (integer) duration>1h
status string status=Executing
running boolean running=true
submitTime datetime submitTime>2024-02-16
lastUpdate datetime lastUpdate>=2024-02-07T00:28:00-05:00
update duration (integer) update<=3d
endTime datetime endTime>2024-02-16
tags string tags=nextflow-io-process-name:MULTIQC,nextflow-io-run-name:furious
envs string envs=RSTUDIO_USER=rstudio,RSTUDIO_PASS=memverge
exitCode string exitCode=1
output string output=*Loaded image: docker.io/memverge/blast:latest*
cpu integer cpu<=16
memory integer memory>=128
dataVolumes string dataVolumes=*size=10*
publishes string publishes=8787:8787
imageID string imageID=*rstudio*
resumable boolean resumable=true
suspendable boolean suspendable=true
cancellable boolean cancellable=true
cost float cost>1.25 (Do not use "=". No exact matches found.)

Note

The output filter looks for a matching string in the output messages generated by each job. Display all output messages by entering the following command:
float show -v -F yaml -j JOB_ID

or by opening the Job Details - JOB_NAME screen on the web interface.

List the filtered jobs output in ascending order by using the --order KEY flag. List in descending order by prepending KEY with "-". Use any of the following keys order output:

  • id
  • name
  • user
  • status
  • duration
  • start (default)
  • cost
  • cpu
  • memory

Examples:

$ float squeue --filter "cost>1.00 and cost<2.00 and status=Completed"
+-----------------------+-----------------------+...+-----------+----------+------------+---------+-----------+
|          ID           |         NAME          |   |  STATUS   | DURATION |     SUBMIT TIME      |    COST   |
+-----------------------+-----------------------+...+-----------+----------+----------------------+-----------+
| tg19ijew3lzhdlfceuwhg | sentieon-c5a.12xlarge |...| Completed | 2h7m20s  | 2024-02-16T16:21:46Z | 1.3464 USD|
| u6zg20ug0ppa7r5itn7hr | sentieon-c5a.12xlarge |...| Completed | 2h3m8s   | 2024-02-19T20:19:35Z | 1.3287 USD|
+-----------------------+-----------------------+...+-----------+----------+----------------------+-----------+
$ float squeue --filter  "(cost>2.00 or duration>2h) and status=Completed"
+-----------------------+-----------------------+...+-----------+----------+----------------------+------------+
|          ID           |         NAME          |...|  STATUS   | DURATION |     SUBMIT TIME      |    COST    |
+-----------------------+-----------------------+...+-----------+----------+----------------------+------------+
| m8tfroy50mwt20kohhgkp | blast-c5.9xlarge      |...| Completed | 8h44m18s | 2024-02-14T04:07:38Z | 5.4851 USD |
| w700gdr6owzjjgdim5m0p | blast-c5.9xlarge      |...| Completed | 9h3m16s  | 2024-02-14T04:22:45Z | 5.3502 USD |
| tg19ijew3lzhdlfceuwhg | sentieon-c5a.12xlarge |...| Completed | 2h7m20s  | 2024-02-16T16:21:46Z | 1.3464 USD |
| u6zg20ug0ppa7r5itn7hr | sentieon-c5a.12xlarge |...| Completed | 2h3m8s   | 2024-02-19T20:19:35Z | 1.3287 USD |
+-----------------------+-----------------------+...+-----------+----------+----------------------+------------+
$ float squeue --filter  "status=Completed" -o -duration
+-----------------------+-----------------------+...+-----------+----------+----------------------+------------+
|          ID           |          NAME         |...|  STATUS   | DURATION |     SUBMIT TIME      |    COST    |
+-----------------------+-----------------------+...+-----------+----------+----------------------+------------+
| w700gdr6owzjjgdim5m0p | blast-c5.9xlarge      |...| Completed | 9h3m16s  | 2024-02-14T04:22:45Z | 5.3502 USD |
| m8tfroy50mwt20kohhgkp | blast-c5.9xlarge      |...| Completed | 8h44m18s | 2024-02-14T04:07:38Z | 5.4851 USD |
| tg19ijew3lzhdlfceuwhg | sentieon-c5a.12xlarge |...| Completed | 2h7m20s  | 2024-02-16T16:21:46Z | 1.3464 USD |
| u6zg20ug0ppa7r5itn7hr | sentieon-c5a.12xlarge |...| Completed | 2h3m8s   | 2024-02-19T20:19:35Z | 1.3287 USD |
| bxulaob02n1d91cmymvxk | Rtest                 |...| Completed | 1h53m1s  | 2024-02-14T16:51:35Z | 0.0371 USD |
...[edited]

Host Filters

The table shows the keys that are available to use with the float hosts (sinfo) command.

Key Type Value Example
instanceID string instanceID=i-0f86428a68f9ca248
status string status=normal
instType string instType=c5a*
cores integer cores>=16
memoryInGB integer memoryInGB>4
payType string payType=OnDemand
publicIP string publicIP=3.90.*.*
start datetime start>2024-02-16T23:20:14Z
end datetime end<2024-02-21T23:20:14Z
zone string zone=us-east-1f
duration duration (integer) duration>1m
cost float cost>0.5 (Do not use "=". No exact matches found.)

List the filtered host output in ascending order by using the --order KEY flag. List in descending order by prepending KEY with "-". Use any of the following keys to order output:

  • instanceID
  • status
  • instType
  • cores
  • memoryinGB
  • payType
  • publicIP
  • start (default)
  • end

Examples:

$ float hosts -o cores --filter "payType=Spot and status=normal"
...+--------+----------------------------+---------+---------------+------------+----------------------+...
...| STATUS |        INSTANCETYPE        | PAYTYPE |   PUBLICIP    |    ZONE    |      STARTTIME       |...
...+--------+----------------------------+---------+---------------+------------+----------------------+...
...| normal | t3.medium(2 vCPU, 4 GB)    | Spot    | 44.201.45.252 | us-east-1d | 2024-02-22T22:07:07Z |...
...| normal | c5a.large(2 vCPU, 4 GB)    | Spot    | 44.223.40.14  | us-east-1f | 2024-02-22T22:06:47Z |...
...| normal | c5.9xlarge(36 vCPU, 72 GB) | Spot    | 3.236.182.133 | us-east-1d | 2024-02-22T22:06:55Z |...
...+--------+----------------------------+---------+---------------+------------+----------------------+...
$ float hosts -o -cost --filter "start>2024-02-20 and status=reclaimed and cost>1.00"
...+-----------+----------------------------+----------+----------------+...+----------------------+...+------------+
...|  STATUS   |        INSTANCETYPE        | PAYTYPE  |    PUBLICIP    |...|      STARTTIME       |...|    COST    |
...+-----------+----------------------------+----------+----------------+...+----------------------+...+------------+
...| reclaimed | c5.9xlarge(36 vCPU, 72 GB) | Spot     | 54.166.157.114 |...| 2024-02-21T20:24:18Z |...| 4.2346 USD |
...| reclaimed | c5.9xlarge(36 vCPU, 72 GB) | OnDemand | 54.81.111.170  |...| 2024-02-22T03:24:31Z |...| 3.4920 USD |
...+-----------+----------------------------+----------+----------------+...+----------------------+...+------------+
[edited]

Gateway Filters

The table shows the keys that are available to use with the float gateway list command.

Key Type Value Example
id string id=g-*
name string name=gateway1
status string status=stopped
cpu integer cpu>=2
memory integer memory<4
ip string publicIP=3.90.*.*
portStart integer portStart>1000
portEnd integer portEnd<=1500
start datetime start>2024-02-21T23:20:14Z
usedPorts integre usedPorts=1
cost float cost>0.5 (Do not use "=". No exact matches found.)

List the filtered gateway output in ascending order by using the --order KEY flag. List in descending order by prepending KEY with "-". Use any of the following keys to order output:

  • id
  • name
  • status
  • cpu
  • memory
  • ip
  • portStart
  • portEnd
  • start
  • usedPorts
  • cost
  • lastUpdate (default)

Examples:

$ float gateway list --filter "portStart>=2000"
+-------------------------+----------+--------+---------------+---------------+-----------+----------------------+...
|           ID            |   NAME   | STATUS | CONFIGURATION |   IPADDRESS   | PORTRANGE |      STARTTIME       |...
+-------------------------+----------+--------+---------------+---------------+-----------+----------------------+...
| g-2q7z42l4wpz9cyeepdskw | gateway3 | Ready  | 2Cores4GB     | 3.232.133.116 | 4000-4500 | 2024-02-23T19:39:47Z |...
| g-jvala2gu26me69ewvqtrc | gateway2 | Ready  | 2Cores4GB     | 44.195.82.162 | 2000-3000 | 2024-02-23T19:39:02Z |...
+-------------------------+----------+--------+---------------+---------------+-----------+----------------------+...
float gateway list -f "status=ready and usedPorts>0" -o -usedPorts
...+----------+--------+---------------+----------------+-----------+----------------------+-----------+------------+
...|   NAME   | STATUS | CONFIGURATION |   IPADDRESS    | PORTRANGE |      STARTTIME       | USEDPORTS |    COST    |
...+----------+--------+---------------+----------------+-----------+----------------------+-----------+------------+
...| gateway1 | Ready  | 2Cores4GB     | 44.200.255.208 | 1000-1050 | 2024-02-23T19:38:23Z | 2         | 0.9054 USD |
...| gateway2 | Ready  | 2Cores4GB     | 44.195.82.162  | 2000-3000 | 2024-02-23T19:39:02Z | 1         | 0.9050 USD |
...+----------+--------+---------------+----------------+-----------+----------------------+-----------+------------+
$ float gateway list --filter "status=stopped"
+-------------------------+----------+---------+---------------+---------------+-----------+...
|           ID            |   NAME   | STATUS  | CONFIGURATION |   IPADDRESS   | PORTRANGE |...
+-------------------------+----------+---------+---------------+---------------+-----------+...
| g-2q7z42l4wpz9cyeepdskw | gateway3 | Stopped | 2Cores4GB     | 3.232.133.116 | 4000-4500 |...
+-------------------------+----------+---------+---------------+---------------+-----------+...

[edited]