Troubleshooting Hazelcast Cluster Snapshot
Hazelcast cluster snapshot activities are logged in mvcsnapd_log and mvm_cluster_snapshot_hazelcast.log. Check the logs for troubleshooting or technical support.
Dealing with Error Messages
Cause: Failed to Register Cluster
$ sudo mvmcli cluster register -n cluster-name -P hazelcast -c contact_member_address:port -C coordinator_address1:port,coordinator_address2:port Failed to register cluster: Failed connection validation: Cannot contact coordinator coordinator_address1:port; Cannot contact coordinator coordinator_address2:port; …
$ sudo mvmcli cluster snapcreate -n cluster-name -l snapshot-name Failed to create cluster snapshot: failed to connect to the primary coordinator coordinator_address:port
Remedy: Check the mvcsnapd Service
Procedure
- Check whether the mvcsnapd service is running on each host corresponding to every coordinator_address.
- Check whether mvcsnapd is started on the cooresponding node.
Cause: Failed to Register Cluster
$ sudo mvmcli cluster register -n cluster-name -P hazelcast -c contact_member_address:port -C coordinator_address1:5679,coordinator_address2:5679 Failed to register cluster: Failed connection validation: failed to query contact member contact_member_address:port
Remedy: Check for a Port Conflict
Procedure
Check if one of the Hazelcast members is listening to the address and
port. If not, assign one.
Cause: Contact Member Not a Member of the Cluster
$ sudo mvmcli cluster snapcreate -n cluster-name -l snapshot-name The contact member contact_member_address:port is not a member of the cluster
This error indicates that the list of members returned by the Hazelcast cluster does not contain the contact member that you registered with the cluster register command. This usually means that the Hazelcast member is listening to multiple network interfaces and that Hazelcast is using a different network interface from contact_member_address for the cluster.
Remedy: Re-Register the Cluster
Procedure
-
Check which address the contact member is using.
curl http://contact_member_address:port/hazelcast/rest/cluster
- Register the cluster with the corresponding address instead.
- Optional: Alternatively, use Hazelcast’s <interface> configuration directive to restrict which network interfaces Hazelcast is allowed to use for the cluster, so that it matches the one that you registered.
Cause: Port Not Prepared
$ sudo mvmcli cluster snapcreate -n cluster-name -l snapshot-name Member contact_member_address:port is not successfully prepared by any coordinator
Remedy: Check Configuration
Procedure
- Check the script file mvm_cluster_snapshot_hazelcast.sh to verify that the Hazelcast cluster name and password are configured correctly.
- Check for mvcsnapd_log and mvm_cluster_snapshot_hazolcast.log for details.
Cause: Restore Fails
$ sudo mvmcli cluster snaprestore -n cluster-name -l snapshot-name Coordinator 10.0.1.76:5679 failed: PreRestoreSnapshot failed.
Remedy: Check for Existing Nodes
Procedure
- Check if any of the nodes in the snapshot already exist.
- Check the logs mvcsnapd_log and mvm_cluster_snapshot_hazelcast.log for more information.