Configuring PMEM

About this task

This procedure describes how to create a pod to contain the DPME. This must be done on each container node.

Label all nodes that have PMEM

$ kubectl label node <node_name> storage=pmem

Procedure

  1. Label all nodes that have PMEM:
    $ kubectl label node node_name storage=pmem
  2. On each container node, use the supplied script to create a pmem-fsdax-config run-once daemonSet:
    $ scripts/config-pmem.sh
    daemonset.apps/pmem-fsdax-config created
    waiting for pmem-fsdax-config pods to run....
    waiting for pmem-fsdax-config daemonset to complete
    Successfully configured PMEM namespace to fsdax
    daemonset.apps "pmem-fsdax-config" deleted
    $
  3. Optional: Change the PMEM file size for the containerized DPME.
    1. To change the PMEM file size for mvmallocd, modify the sudo fallocate line in mount.sh:
    sudo fallocate -l PMEM_size /mnt/pmem${numa}/mm
    Where PMEM_size is the required size suffixed with G for gigabytes. For example: 16G.
    Important:

    The PMEM file size must be less than the VM PMEM capacity.

    For example, if you assigned less than 16G PMEM to a VM, change the fallocate size to be less than this capacity:

    sudo fallocate -l 12G /mnt/pmem${numa}/mm
  4. Associate the VM with specific NUMA node as recommended by VMware. See "Associate Virtual Machines with Specified NUMA Nodes" on the VMware documentation website.
  5. Associate PMEM allocation for that VM to the same NUMA node. See VMware knowledge base article 78094.
  6. Run mount.sh. This script takes two optional arguments: --ssh-key-path and --nvdimm:
  7. Run the mount script:
    $ scripts/mount.sh --ssh-key-path ssh_key_path \
    --nvdimm nvdimms
    $
    Where
    --ssh-key-path
    is the path to SSH private key file.
    --nvdimm
    is the number of NVDIMMs on each container node. Default value is one.
    Both parameters are optional.
  8. Verify that the PMEM device is mounted on each container node:
    $ ssh -i ssh_key_file core@node_ip
    [core@node_ip ~]$ df -h
    Filesystem                            Size  Used Avail Use% Mounted on
    ...
    /dev/pmem0                            32G   42M   32G   1% /var/mnt/pmem0
    ...
    $