Cluster in inconsistent state when resizing to 0 worker nodes #665

Activity

1. riccardomurri commented on Nov 6, 2019

   

   riccardomurri

   on Nov 6, 2019

   Contributor

   More actions

   When a resize of a running cluster removes all worker nodes the yaml file containing the cluster state reads:

   worker: []
   

   however when trying to add back worker node or starting the cluster from the config file an error pops up:

      ERROR Could not start cluster test: 'worker'
   

   I can not reproduce this error with the "master" branch of ElastiCluster; I am removing nodes using elasticluster resize --no-setup -r ... and then trying to (re)configure using elasticluster setup. Combining the two steps into one elasticluster resize -r ... also works.

   $ elasticluster resize ge -r 1:worker --no-setup
   [...]
   Removing 1 worker node(s) from the cluster.
   The following nodes will be removed from the cluster.
       worker001
   Do you really want to remove them? [y/N]: y
   2019-11-06 21:53:35 monia elasticluster[29370] INFO Shutting down node `worker001` (VM instance `6b667441-a231-41f2-9499-aea28e6e328d`) ...
   [...]
   NOT configuring the cluster as requested.
   [...]
   
   $ fgrep 'worker:' ~/.elasticluster/storage/ge.yaml
   worker: []
   
   $ elasticluster setup ge
   Updating cluster `ge`...
   [...]
   Configuring cluster `ge`...
   [...]
   No config file found; using defaults
   
   PLAY [Prepare VM for running Ansible] ************************************************************************************************************
   
   TASK [Ensure Python is installed] ****************************************************************************************************************
   [...]
   

   (Ansible starts, but fails later on since playbooks assume that groups.gridengine_worker exists and is not empty.)
2. blmt commented on Nov 14, 2019

   

   blmt

   on Nov 14, 2019

   Author

   More actions

   The removal part was not the issue, though. The problem arises when you try to add back any number of worker nodes: did you test that as well?