我正在尝试使用AWS ParallelCluster (AWS Research Workshop)遵循教程，一旦我得到CloudFormation启动并运行，我无法通过Rest API连接到集群。我使用的是Slurm 21.08.8和ParallelCluster 3.1.4。我能够SSH到头节点并启动作业，但是当我试图通过SageMaker笔记本访问node时，我得到了一个连接拒绝。下面是我为ParallelCluster配置的YAML。请注意，这是使用gravon2 ARM处理器:
Region: ${REGION}
Image:
Os: alinux2
SharedStorage:
- Name: myebs
StorageType: Ebs
MountDir: /shared
EbsSettings:
VolumeType: gp2
Size: 200
HeadNode:
InstanceType: c6g.medium
Networking:
SubnetId: ${SUBNET_ID}
ElasticIp: true
Ssh:
KeyName: ${KEY_NAME}
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: q1
CapacityType: ONDEMAND
ComputeResources:
- Name: cr1
InstanceType: c6g.2xlarge
MinCount: 0
MaxCount: 20
Efa:
Enabled: false
CustomActions:
OnNodeConfigured:
Script: ${POST_INSTALL_SCRIPT_LOCATION}
Args:
- ${POST_INSTALL_SCRIPT_ARGS_1}
- ${POST_INSTALL_SCRIPT_ARGS_2}
- ${POST_INSTALL_SCRIPT_ARGS_3}
- ${POST_INSTALL_SCRIPT_ARGS_4}
- ${POST_INSTALL_SCRIPT_ARGS_5}
- ${POST_INSTALL_SCRIPT_ARGS_6}
- ${POST_INSTALL_SCRIPT_ARGS_7}
- ${POST_INSTALL_SCRIPT_ARGS_8}
- ${POST_INSTALL_SCRIPT_ARGS_9}
Iam:
AdditionalIamPolicies:
- Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite
S3Access:
- EnableWriteAccess: true
BucketName: '*'
Networking:
SubnetIds:
- ${SUBNET_ID}
AssignPublicIp: true
PlacementGroup:
Enabled: true
这是我的slurm.conf:
#
# Example slurm.conf file. Please run configurator.html
# (in doc/html) to build a configuration file customized
# for your environment.
#
#
# slurm.conf file generated by configurator.html.
#
# See the slurm.conf man page for more information.
#
# CLUSTER SETTINGS
ClusterName=mypc6g2
SlurmUser=slurm
SlurmctldPort=6820-6829
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm.state
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ReconfigFlags=KeepPartState
#
# CLOUD CONFIGS OPTIONS
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,cloud_dns
CommunicationParameters=NoAddrCache
SuspendProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
ResumeProgram=/opt/parallelcluster/scripts/slurm/slurm_resume
ResumeFailProgram=/opt/parallelcluster/scripts/slurm/slurm_suspend
SuspendTimeout=120
ResumeTimeout=1800
PrivateData=cloud
ResumeRate=0
SuspendRate=0
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=180
UnkillableStepTimeout=180
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
MessageTimeout=60
#
# SCHEDULING, JOB, AND NODE SETTINGS
EnforcePartLimits=ALL
SchedulerType=sched/backfill
ProctrackType=proctrack/cgroup
MpiDefault=none
ReturnToService=1
TaskPlugin=task/affinity,task/cgroup
#
# TRES AND GPU CONFIG OPTIONS
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
# LOGGING
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
JobCompType=jobcomp/none
#
# WARNING!!! The slurm_parallelcluster.conf file included
# get updated by pcluster process, be careful
# when manually editing!
include slurm_parallelcluster.conf
# Enable jwt auth for Slurmrestd
AuthAltTypes=auth/jwt
#
## /opt/slurm/etc/slurm.conf
#
# ACCOUNTING
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30
#
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=<IP ADDRESS> # cluster headnode's DNS
AccountingStorageUser=db_user
AccountingStoragePort=6839
sudo journalctl -u slurmrestd(隐藏IP)的输出:
-- Logs begin at Thu 2022-05-12 10:46:45 UTC, end at Thu 2022-05-26 03:00:38 UTC. --
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
sudo systemctl status slurmrestd(隐藏IP)的输出:
● slurmrestd.service - Slurm restd daemon
Loaded: loaded (/etc/systemd/system/slurmrestd.service; disabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2022-05-25 22:30:25 UTC; 4h 31min ago
Main PID: 12872 (code=killed, signal=SEGV)
May 25 22:30:25 systemd[1]: Started Slurm restd daemon.
May 25 22:30:25 slurmrestd[12872]: debug: _establish_config_source: using config_file=/opt/slurm/etc/slurmrestd.conf (environment)
May 25 22:30:25 slurmrestd[12872]: debug: slurm_conf_init: using config_file=/opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: Reading slurm.conf file: /opt/slurm/etc/slurmrestd.conf
May 25 22:30:25 slurmrestd[12872]: debug: NodeNames=q1-dy-cr1-[1-20] setting Sockets=8 based on CPUs(8)/(CoresPerSocket(1)/ThreadsPerCore(1))
May 25 22:30:25 systemd[1]: slurmrestd.service: main process exited, code=killed, status=11/SEGV
May 25 22:30:25 systemd[1]: Unit slurmrestd.service entered failed state.
May 25 22:30:25 systemd[1]: slurmrestd.service failed.
systemctl status slurmctld的输出:
● slurmctld.service - Slurm controller daemon
Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-05-25 22:30:25 UTC; 4h 33min ago
Main PID: 12899 (slurmctld)
CGroup: /system.slice/slurmctld.service
├─12899 /opt/slurm/sbin/slurmctld -D
└─12903 slurmctld: slurmscriptd
May 26 00:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 00:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 01:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:20:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 02:40:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
May 26 03:00:01 slurmctld[12899]: slurmctld: auth/jwt: auth_p_token_generate: created token for root for 1800 seconds
我不能得到slurmrestd服务的启动和运行，我不知道在其他地方看，因为这不是非常好的文档在Slurm网站和Rest API不是AWS ParallelCluster支持的功能(根据AWS支持团队)。
我正在运行教程中的以下代码块:
import requests
import json
slurm_openapi_ep = 'http://'+slurm_host+':8082/openapi/v3'
print(slurm_openapi_ep)
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.35'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.36'
# slurm_rest_base='http://'+slurm_host+':8082/slurm/v0.0.37'
_, get_headers = pcluster_helper.update_header_token()
print(get_headers)
resp_api = requests.get(slurm_openapi_ep, headers=get_headers)
print(resp_api)
if resp_api.status_code != 200:
# This means something went wrong.
print("Error" , resp_api.status_code)
with open('build/slurm_api.json', 'w') as outfile:
json.dump(resp_api.json(), outfile)
print(json.dumps(resp_api.json(), indent=2))
print输出
http://<HEAD NODE IP ADDRESS HERE>:8082/openapi/v3
{'X-SLURM-USER-NAME': 'ec2-user', 'X-SLURM-USER-TOKEN': '<MY TOKEN HERE>', 'Content-type': 'application/x-www-form-urlencoded', 'Accept': 'application/json'}
和错误:
ConnectionError: HTTPConnectionPool(host='<HEAD NODE IP ADDRESS HERE>', port=8082): Max retries exceeded with url: /openapi/v3 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6151b9940>: Failed to establish a new connection: [Errno 111] Connection refused',))
提前感谢任何建议或想法来解决。
对于任何试图找到答案的人来说，aws更新了他们的文档和教程来支持Graviton处理器。现在很有魅力了