Viewing Logs
Each data processing session that you run on a cluster has logs. These logs include errora details in the case of a session failing.
Viewing Session Logs
The standard output and standard error from the session running on a cluster is
redirected to a log file stored on disk on the
cluster. If the print_logs
parameter is set to true
when the session was
started, the logs will get streamed and printed out on the client side as
code gets executed on the cluster. When the session is ended, if the store_logs_in_s3
parameter was set to true
when the session was started, this log file will be persisted in cloud storage
in your AWS account. You can either view this log file from the Banyan dashboard
or download it to your local machine for viewing using the client library.
To view the log file from the Banyan dashboard, navigate to the Sessions tab, search for the session based on its session ID or session name, and click on the button to view logs.
To download the log file to your local machine, use download_session_logs()
.
download_session_logs(
session_id="2021-06-10-2240237175e6921d0c69f8a93aaa49ffd30287",
cluster_name="weekly-data-playground"
)
download_session_logs(
session_id="2021-06-10-2240237175e6921d0c69f8a93aaa49ffd30287",
cluster_name="weekly-data-playground"
)
Viewing Cluster Logs
You should usually not need to view any logs for your cluster. However, if cluster
creation fails and the information provided by get_cluster_status()
is not
sufficient, you may want to view more detailed logs. You can find the logs at the
following URL: https://<AWS_REGION>.console.aws.amazon.com/cloudwatch/home?region=<AWS_REGION>#logsV2:log-groups/log-group/$252Faws$252Fparallelcluster$252F<CLUSTER_NAME>