Skip to content

create_cluster()

Create a cluster.

create_cluster(name=None, instance_type="m4.4xlarge", max_num_workers=2048, min_num_workers=0, iam_policy_arn=None, s3_bucket_arn=None, s3_bucket_name=None, scaledown_time=25, vpc_id=None, subnet_id=None, nowait=False ...)

Description

This function creates a new cluster with the given configuration or re-creates a previously destroyed cluster. If no vpc_id and subnet_id are provided, the cluster is by default created in the default public VPC and subnet in your AWS account.

When creating a cluster, you must provide the name of the AWS EC2 Key Pair that you would like to use to SSH into the cluster. Create and EC2 Key Pair here and securely save the file. Banyan does not store your EC2 Key Pair. You can use your EC2 Key pair to SSH into your cluster once it is created, for troubleshooting. You do not need to SSH into your cluster, as Banyan fully manages compute and can update your cluster for you.

Optional Parameters

name (str)

Name of cluster. We recommend naming your cluster for its intended use. For example, you may name a cluster ml-experimentation if it is used for experimenting with machine learning.

instance_type (str)

Type of EC2 instance for the compute nodes of the cluster. All compute nodes are the same instance types. EC2 instance types differ in the number of workers (vCPUs) for compute, memory resources, and network capacity. Currently, Banyan supports the following instance types: "t3.large", "t3.xlarge", "t3.2xlarge", "m4.4xlarge", "m4.10xlarge", "c5.2xlarge", and "c5.4xlarge". Defaults to "m4.4xlarge".

max_num_workers (int)

Maximum number of compute workers the cluster can scale up to. You should increase this if you plan to run sessions that require high computational resources or if you plan to run many sessions on the same cluster. Defaults to 2048.

Note that you get charged by the number of running workers in the cluster at any given point in time, not by the maximum number of workers.

disk_capacity (Union{String,Real})

Either "auto" or the number of bytes to allocate for the shared Lustre file system available to all workers used by any session running on the cluster.

If "auto" is specified, an Amazon EFS file system is shared across all workers used by any session running on the cluster. Otherwise, a number of bytes might be specified with a string or float for the capacity of a Lustre file system that is shared by all workers.

The default is "1200 GB" for a 1.2-TB-capacity Lustre parallel file system for optimal performance. There is a default quota of 100 Lustre file systems so after that quota is exhausted, you may have to switch to "auto" for EFS file systems.

min_num_workers (int)

Minimum number of compute workers to always keep running on the cluster. Cluster creation does not complete until this number of workers have been spun up. These workers are not subject to scaledown_time. A nonzero value allows for sessions to get started faster, since there will be less wait time for workers to get spun up. Since these workers are always running, you will be charged for them for the entire duration of the cluster.

iam_policy_arn (str)

AWS ARN of optional IAM policy that contains additional permissions that the cluster should have. For example, if you need to read/write to an Amazon DynamoDB table, you should create a policy with these permissions on your AWS Management Console and provide the ARN for that policy here. The name of this policy must begin with "Banyan"

region (str)

AWS region to create the cluster in. The following regions are supported: us-east-1 (N. Virginia), us-east-2 (Ohio), us-west-1 (N. California), us-west-2 (Oregon), eu-central-1 (Frankfurt), eu-west-2 (London), and sa-east-1 (São Paulo).

s3_bucket_arn (str)

AWS ARN of the Amazon S3 bucket used to store cluster data and logs. This bucket is located in your AWS account. If not provided, an S3 bucket for the cluster is created in your account. You can retrieve the ARN of this bucket using get_clusters()

s3_bucket_name (str)

Name of the Amazon S3 bucket used to store cluster data and logs. This bucket is located in your AWS account. If not provided, an S3 bucket for the cluster is created in your account. You can retrieve the ARN of this bucket using get_clusters()

scaledown_time (int)

Number of consecutive minutes of inactivity after which workers should get spun down on the cluster. Workers are inactive when there are no sessions running on them. When workers get spun down, the next session that is started may take longer to get fully set up. Defaults to 25.

vpc_id (str)

ID of the AWS VPC in which to create cluster. VPC must be public. If not provided, the default public VPC in your AWS account is used. For security purposes, we recommend using this VPC solely for Banyan or creating a new AWS account to use with Banyan.

subnet_id (str)

Subnet in which to create cluster. Subnet must be in the given VPC. If not provided a subnet in the default public VPC in your AWS account is used.

nowait (bool)

Indicates not to wait for cluster creation to complete before returning. Defaults to False and will block until cluster creation is complete.

Returns (Cluster)

Banyan struct Cluster dictionary, containing the cluster configuration.

Cluster(
    "mycluster",  # name
    "creatin",  # status
    0,  # num_sessions_running
    "arn:aws:s3:::banyan-cluster-data-mycluster-1234abcd"  # s3_bucket_arn
)

Example Usage

# To create cluster with default configuration
create_cluster()

# To re-create a cluster that has been previously destroyed
create_cluster(name="myfirstcluster")

create_cluster(
    name="weekly-data-playground",
    instance_type="c5.2xlarge",
)