Skip to content

Understanding Workload Priority and Preemption

Preemption ensures that higher-priority workloads can acquire resources even if lower-priority workloads are already using them. In this simple demonstration, you will create two Projects with different priority levels and observe how a running medium-priority workload is paused when a higher-priority workload starts—and how it automatically resumes once the higher-priority job finishes.

PreRequsites

  • A Node Group with a single GPU that will be used by workloads in the two Projects, one at a time.
  • At least one non-admin user exists. Create a new local user if required.

1. Create Two Projects with Different Priorities

  1. Open the Projects Dashboard
  2. Login to the MemVerge.ai UI as a Platform Administrator.
  3. Select Projects from the left navigation bar.
  4. Click + Create Project to add your first Project
  5. Complete the required fields. See Managing Projects for more information.
  6. Assign Priority
  7. In the Priority field, choose Medium for this first Project.
  8. Create the Second Project
  9. Repeat the process for a second Project (e.g., high-priority-project).
  10. Set the Priority to High or Highest.

2. Assign the Same User to Both Projects

There are two methods to assign users to a project:

Method 1:

  1. Go to Users & Authentication → Users.
  2. Click on the username that will run both workloads. The user details page will open.
  3. Click + Add to Project
  4. Select each newly created Project (medium priority, high priority) from the dropdown.
  5. Click Confirm
  6. Repeat Step 3 so the user has permissions to create Workspaces in both Projects.

Method 2:

  1. Navigate to Projects
  2. Click the Name of the project to see the project details
  3. Navigate to the Members tab
  4. Click + Add Members
  5. Search or select the user
  6. Click Confirm

3. Login as the User

  • Logout of the AI Platform as the Platform Admin
  • Login as the user

4. Prepare a Storage Volume

  1. Create or Use an Existing Volume
  2. From Storage → Volumes, choose + New Volume.
  3. Enter a name and allocate size for your AI workloads (e.g., ml-training-vol).
  4. This volume can be used by both Projects if attached to each Workspace configuration.

5. Create and Start a Workspace in the Medium-Priority Project

  1. Select the Medium Project
  2. In the Projects dashboard, find medium-priority-project.
  3. Create a Workspace
  4. Click + Create Workspace. Assign the new workspace to the medium-priority-project.
  5. Attach the ml-training-vol if needed for your workload’s data.
  6. Launch the Workspace
  7. Wait for it to reach a Running or Ready state.

5. Create and Start a Workspace in the High-Priority Project

  1. Switch Projects
  2. In Projects, find high-priority-project.
  3. Create a Workspace
  4. Similar steps as above: + Create Workspace, attach any required volume (can be the same or a new one).
  5. Observe Preemption
  6. Upon launching this high-priority workspace, the platform will pause (preempt) the medium-priority workspace’s job if the cluster resources are insufficient to run both simultaneously.
  7. The medium-priority workspace status transitions to Preempted or NotReady, indicating it has been paused.

6. Stopping the High-Priority Workload

  1. Stop the High-Priority Workspace
  2. In the Workspaces dashboard, locate the workspace running under high-priority-project.
  3. Click Stop, then confirm in the popup dialog.
  4. Automatic Resume
  5. Once the high-priority workload fully stops, the platform detects freed resources.
  6. The medium-priority workspace automatically resumes from its preemption state.

7. Confirm the Medium-Priority Workspace Has Resumed

  1. Check Status
  2. Refresh the Workspaces dashboard. The medium-priority workspace should revert from Preempted or NotReady back to Running or Ready.
  3. Verify Workload Progress
  4. Connect to the resumed workspace (if interactive) or check logs to ensure the job continues from where it left off.

Congratulations! You have successfully demonstrated a simple preemption scenario using two workloads in different priority Projects. This setup ensures mission-critical or time-sensitive jobs always have the necessary resources, while lower-priority jobs automatically pause and resume as availability changes.