Wednesday 1 October 2014

Stop and start a Nutanix cluster

I suppose that under normal circumstances one would not need to shut down a Nutanix cluster very often but I had to do it twice recently because of a network upgrade and a complete shutdown of a datacenter.

The nodes in a Nutanix cluster are not only ESXi hosts; they provide storage services as well. If you need to shut down a node or a cluster for any reason, the Nutanix software needs to be shut down properly to avoid storage-related issues.

If you are CVM making use of a vDS ensure that your system is configured with a 64 GB satadom.
If your local CVM datastore is showing up as vmfs you will be fine and there will be no impact. If you have a 16 GB satadom contact Nutanix support

This issue is only applied to certain configuration / NOS version.
When the system is configured with 16Gb satadom, local datastore is mounted as a pynfs.
You can check this by looking at the local datastore on vsphere client - if local datastore is vmfs, this means satadom is 64Gb in size and directly mounted as local datastore and there will be no impact on shutting down / starting cluster.

If local datastore is nfs and displayed size is 1Tb, then the system is using pynfs, which by default is none-persistent and will not be mounted unless vCenter is available (dvSwitch information is stored in vCenter database).

Stopping a cluster
  • Confirm autostart is enabled for all CVM
  • Shut down all VM on the Nutanix cluster except the CVM
  • SSH into any CVM and stop Nutanix cluster with ‘cluster stop’ command
  • Ensure all services are stopped on all CVM (Zeus and Scavenger will be UP and  that is fine)

 Shutting down nodes in a cluster
  • Shut down guest VM or move them to other node. You can only shut down one node in cluster If you need to shut down more than one node you will need to shut down entire cluster.
  • Log into ESX host or vCenter
  • Shut down CVM by using  Power > Shut Down Guest
  • Enter maintenance mode
  • Uncheck Move Powered Off and suspended virtual machines to other hosts in cluster.
  •  Shut down ESX node

Starting nodes in a cluster
  • Power on node(s)
  • Exit maintenance mode
  • Power on the CVMs and wait approx. 5 minutes for all services to start
  • Start Nutanix cluster with ‘Cluster start’ command
  • Confirm cluster services are running with ‘ncli Cluster status | grep –A 15 cvm_ip_address’ command
  • Ensure ESXI can see datastore
  • Verify all services are up on all controller VM with ‘cluster status’ command
  • Once all nodes are up you will get to log into each esxi host and re-enable automatic startup for the controller VM. Go to Configuration > Virtual Machine Startup/Shutdown

4 comments:

  1. Thanks for these instructions! Curious - Is the CVM auto-start a necessary step?

    ReplyDelete
  2. It is recommended somewhere in the setup documentation. Auto start is not supported on VM that are in an HA cluster but the CVM should have HA disabled. Hence the reason that you enable this via ESX host.
    My instructions say to power on CVM but that should happen automatically according to settings. Will test at earliest opportunity and correct

    ReplyDelete
  3. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete