Troubleshooting Hazelcast Cluster Snapshot

Hazelcast cluster snapshot activities are logged in mvcsnapd_log and mvm_cluster_snapshot_hazelcast.log. Check the logs for troubleshooting or technical support.

Dealing with Error Messages

Cause: Failed to Register Cluster

$ mvmcli cluster register -n cluster-name -P hazelcast -c contact_member_address:port -C coordinator_address1:port,coordinator_address2:port
Failed to register cluster: Failed connection validation: Cannot contact coordinator coordinator_address1:port; Cannot contact coordinator coordinator_address2:port; …
$ mvmcli cluster snapcreate -n cluster-name -l snapshot-name
Failed to create cluster snapshot: failed to connect to the primary coordinator coordinator_address:port

Remedy: Check the mvcsnapd Service

Procedure

  1. Check whether the mvcsnapd service is running on each host corresponding to every coordinator_address.
  2. Check whether mvcsnapd is started on the corresponding node.

Cause: Failed to Register Cluster

$ mvmcli cluster register -n cluster-name -P hazelcast -c contact_member_address:port -C coordinator_address1:5679,coordinator_address2:5679 
Failed to register cluster: Failed connection validation: failed to query contact member contact_member_address:port

Remedy: Check for a Port Conflict

Procedure

Check if one of the Hazelcast members is listening to the address and port. If not, assign one.

Cause: Contact Member Not a Member of the Cluster

$ mvmcli cluster snapcreate -n cluster-name -l snapshot-name
The contact member contact_member_address:port is not a member of the cluster

This error indicates that the list of members returned by the Hazelcast cluster does not contain the contact member that you registered with the cluster register command. This usually means that the Hazelcast member is listening to multiple network interfaces and that Hazelcast is using a different network interface from contact_member_address for the cluster.

Remedy: Re-Register the Cluster

Procedure

  1. Check which address the contact member is using. 
    curl http://contact_member_address:port/hazelcast/rest/cluster
  2. Register the cluster with the corresponding address instead.
  3. Optional: Alternatively, use Hazelcast’s <interface> configuration directive to restrict which network interfaces Hazelcast is allowed to use for the cluster, so that it matches the one that you registered.

Cause: Port Not Prepared

$ sudo mvmcli cluster snapcreate -n cluster-name -l snapshot-name
Member contact_member_address:port is not successfully prepared by any coordinator

Remedy: Check Configuration

Procedure

  1. Check the script file mvm_cluster_snapshot_hazelcast.sh to verify that the Hazelcast cluster name and password are configured correctly.
  2. Check for mvcsnapd_log and mvm_cluster_snapshot_hazolcast.log for details.

Cause: Restore Fails

$ mvmcli cluster snaprestore -n cluster-name -l snapshot-name
Coordinator 10.0.1.76:5679 failed: PreRestoreSnapshot failed.

Remedy: Check for Existing Nodes

Procedure

  1. Check if any of the nodes in the snapshot already exist.
  2. Check the logs mvcsnapd_log and mvm_cluster_snapshot_hazelcast.log for more information.