Skip to content

Managing Clusters

There are two ways to manage your clusters: (1) through the Banyan dashboard and (2) with the client library.

Cluster Status

A cluster is always in one of the following states:

  • Creating - Cluster creation is in progress.
  • Ready/Running - Cluster is created and ready to run computation on.
  • Destroying - Cluster is currently destroying/deleting.
  • Terminated - Cluster is terminated, and can be restarted.

Clusters that are deleted will not be displayed.

Managing Clusters from Banyan Dashboard

Sign into the dashboard to view and manage your clusters. To view your clusters, first navigate to the Clusters tab in the Banyan dashboard. Click on the reload icon in the top left. If you currently have clusters, they will be displayed in the table, along with information about the cluster, including the current status, the number of sessions currently running, the estimated total cost of the cluster so far (Banyan cost + AWS cost), the time of the last session started on the cluster, and the time of cluster creation.

To destroy a cluster, click the destroy icon to the right of the cluster listing in the table. You will see the cluster in the Destroying status. When the cluster is completely destroyed, it's status will be Terminated. You will not be charged for clusters that are not running.

You can (re)create this cluster by clicking the create icon in the row corresponding to the cluster you would like to create. The cluster will get created using the same configuration specified.

You can also permanently delete a cluster configuration. This will remove it from the table and delete all AWS resources associated with the cluster (including the S3 bucket).

Managing Clusters from the Client Library

The client library provides several functions for managing clusters. See the full documentation for more details.

All clusters that have not been permanently deleted can be retrieved by the following:

get_clusters()

get_clusters()

You can use get_running_clusters() to get all clusters that are currently running.

Individual clusters can be destroyed or deleted. Both destroy and delete will terminate all running nodes. However, destroy will save the cluster configuration and will not delete the S3 bucket (including data and session logs). delete will delete all resources associated with the cluster. You are not charged for destroyed or deleted clusters. Use the following:

destroy_cluster("weekly-data-playground")
delete_cluster("weekly-data-playground")

destroy_cluster("weekly-data-playground")
delete_cluster("weekly-data-playground")

Connecting to a cluster with SSH

Banyan uses an extension of the open-source CfnCluster for low-level cluster management. To connect to the head node of a cluster with SSH:

  1. Ensure that you created an SSH/EC2 key pair. When SSHing into a cluster that you have created, ensure to use the EC2 key pair that you specified when creating the cluster. You can find this information on the Clusters page of the dashboard. when you created the cluster and downloaded the key pair file to a secure location
  2. Ensure you have pip installed, as well as AWS ParallelCluster version 3. Follow the instructions here
  3. To SSH into your cluster, run the following in a terminal or command prompt session: 
    pcluster ssh -n your-cluster-name -i PATH/TO/your_ec2_key_pair

When running pcluster ssh, provide the name of the cluster and the path (location) that you downloaded and and store your key pair file that you created when you created the cluster.

If you are having troubling connecting to your cluster, please ensure that you have corectly authenticated your AWS credentials.

What's next?

If you have clusters that are in the running state, you can run some computation by starting a session and then run automatically parallelized code with BanyanArrays.jl or BanyanDataFrames.jl or custom scripts.