create_cluster()
Create a cluster.
create_cluster(name=None, instance_type="m4.4xlarge", max_num_workers=2048, min_num_workers=0, iam_policy_arn=None, s3_bucket_arn=None, s3_bucket_name=None, scaledown_time=25, vpc_id=None, subnet_id=None, nowait=False ...)
Description
This function creates a new cluster with the given configuration or re-creates
a previously destroyed cluster. If no vpc_id
and subnet_id
are provided,
the cluster is by default created in the default public VPC and subnet in your
AWS account.
When creating a cluster, you must provide the name of the AWS EC2 Key Pair that you would like to use to SSH into the cluster. Create and EC2 Key Pair here and securely save the file. Banyan does not store your EC2 Key Pair. You can use your EC2 Key pair to SSH into your cluster once it is created, for troubleshooting. You do not need to SSH into your cluster, as Banyan fully manages compute and can update your cluster for you.
Optional Parameters
name
(str
)
Name of cluster. We recommend naming your cluster for its
intended use. For example, you may name a cluster ml-experimentation
if it is
used for experimenting with machine learning.
instance_type
(str
)
Type of EC2 instance for the compute nodes of the cluster. All compute nodes are the same instance types. EC2 instance types differ in the number of workers (vCPUs) for compute, memory resources, and network capacity. Currently, Banyan supports the following instance types: "t3.large", "t3.xlarge", "t3.2xlarge", "m4.4xlarge", "m4.10xlarge", "c5.2xlarge", and "c5.4xlarge". Defaults to "m4.4xlarge".
max_num_workers
(int
)
Maximum number of compute workers the cluster can scale up to. You should increase this if you plan to run sessions that require high computational resources or if you plan to run many sessions on the same cluster. Defaults to 2048.
Note that you get charged by the number of running workers in the cluster at any given point in time, not by the maximum number of workers.
disk_capacity
(Union{String,Real}
)
Either "auto"
or the number of bytes to allocate for the shared Lustre file system available to all workers
used by any session running on the cluster.
If "auto"
is specified, an Amazon EFS file system is shared across all workers used by any session running on the cluster.
Otherwise, a number of bytes might be specified with a string or float for the capacity of a Lustre file system
that is shared by all workers.
The default is "1200 GB"
for a 1.2-TB-capacity Lustre parallel file system for
optimal performance. There is a
default quota of 100 Lustre file systems so after that quota is exhausted, you may have to switch to "auto"
for
EFS file systems.
min_num_workers
(int
)
Minimum number of compute workers to always keep running on the cluster. Cluster
creation does not complete until this number of workers have been spun up.
These workers are not subject to scaledown_time
.
A nonzero value allows for sessions to get started faster, since there will be less
wait time for workers to get spun up. Since
these workers are always running, you will be charged for them for the entire
duration of the cluster.
iam_policy_arn
(str
)
AWS ARN of optional IAM policy that contains additional permissions that the cluster should have. For example, if you need to read/write to an Amazon DynamoDB table, you should create a policy with these permissions on your AWS Management Console and provide the ARN for that policy here. The name of this policy must begin with "Banyan"
region
(str
)
AWS region to create the cluster in. The following regions are supported: us-east-1 (N. Virginia), us-east-2 (Ohio), us-west-1 (N. California), us-west-2 (Oregon), eu-central-1 (Frankfurt), eu-west-2 (London), and sa-east-1 (São Paulo).
s3_bucket_arn
(str
)
AWS ARN of the Amazon S3 bucket used to store
cluster data and logs. This bucket is located in your AWS account. If not provided,
an S3 bucket for the cluster is created in your account. You can retrieve the
ARN of this bucket using get_clusters()
s3_bucket_name
(str
)
Name of the Amazon S3 bucket used to store
cluster data and logs. This bucket is located in your AWS account. If not provided,
an S3 bucket for the cluster is created in your account. You can retrieve the
ARN of this bucket using get_clusters()
scaledown_time
(int
)
Number of consecutive minutes of inactivity after which workers should get spun down on the cluster. Workers are inactive when there are no sessions running on them. When workers get spun down, the next session that is started may take longer to get fully set up. Defaults to 25.
vpc_id
(str
)
ID of the AWS VPC in which to create cluster. VPC must be public. If not provided, the default public VPC in your AWS account is used. For security purposes, we recommend using this VPC solely for Banyan or creating a new AWS account to use with Banyan.
subnet_id
(str
)
Subnet in which to create cluster. Subnet must be in the given VPC. If not provided a subnet in the default public VPC in your AWS account is used.
nowait
(bool
)
Indicates not to wait for cluster creation to complete before returning. Defaults to False and will block until cluster creation is complete.
Returns (Cluster
)
Banyan struct Cluster
dictionary, containing the cluster configuration.
Cluster(
"mycluster", # name
"creatin", # status
0, # num_sessions_running
"arn:aws:s3:::banyan-cluster-data-mycluster-1234abcd" # s3_bucket_arn
)
Example Usage
# To create cluster with default configuration
create_cluster()
# To re-create a cluster that has been previously destroyed
create_cluster(name="myfirstcluster")
create_cluster(
name="weekly-data-playground",
instance_type="c5.2xlarge",
)