Skip to content

Creating Clusters

Once you have completed the setup steps in Getting Started, you are ready to create a cluster. A cluster is virtual supercomputer running in an AWS Virtual Private Cloud in your AWS account. A single cluster can be shared for Julia compute and Python compute. You can create a cluster on the Banyan dashboard or from the client library.

Creating a Cluster from the Banyan Dashboard

To create a cluster, first navigate to the Clusters tab in the Banyan dashboard. At the bottom of the page, fill out the required details.

  1. Enter a Cluster name.
  2. Select the AWS Region you would like your cluster/resources provisioned in.
  3. Enter in the name of an SSH/EC2 key pair that you would like to use to later SSH into the cluster. If you do not have an EC2 Key Pair, please create one here and download the key pair file to a secure location. Please provide just the name and not your local path to the file.
  4. Select the Worker type that you would like your cluster to use. See here for more details on AWS EC2 instance types.
  5. Drag the slider to select the number of nodes in the cluster. The total number of workers available for compute in the cluster will be num_nodes x workers_per_node, where workers_per_node varies based on the worker type
  6. Optionally, you can provide an IAM Policy ARN to give additional AWS permissions to the cluster. The name of the IAM policy must begin with "Banyan". See here for additional details on how to create IAM policies.
  7. Optionally, you can provide an S3 Bucket ARN of a bucket that you have created for the cluster to read and write to. Ensure that this bucket is created in the same region as the cluster.
  8. Click Create Cluster
    Form for creating a cluster

Once you have done this, you will see your cluster in the Creating status in the clusters table. When creation is complete, your cluster will be in the Ready state, and you can now run computation on the cluster.

Clusters table on Banyan dashboard

Creating a Cluster from the Client Library

You can alternatively create a cluster from the client library.

create_cluster(
    name="weekly-data-playground",
)

create_cluster(
    name="weekly-data-playground",
)

What's next?

Now, that you have created a cluster, you are ready to run some computation on the cluster by starting a session and using functions in BanyanArrays.jl or BanyanDataFrames.jl or custom scripting for processing big datasets. Also, see here for how to destroy or delete your cluster and how to connect to it through SSH if you wish to.