我写了一个使用spark上下文的example.jar程序。 我如何在使用Slurm的集群上运行这个？ 这与https://stackoverflow.com/questions/29308202/running-spark-on-top-of-slurm有关，但答案不是很详细，不在serverfault。
为了运行一个使用spark上下文的应用程序，首先需要运行一个启动master和一些worker的Slurm作业。 在使用Slurm时，您需要注意以下几点：
我正在将Linux二进制文件安装到$HOME/spark-1.5.2-bin-hadoop2.6/ 。 请记住用脚本中的一些有效值replace<username>和<shared folder> 。
#!/bin/bash #start_spark_slurm.sh #SBATCH --nodes=3 # ntasks per node MUST be one, because multiple slaves per work doesn't # work well with slurm + spark in this script (they would need increasing # ports among other things) #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=4 #SBATCH --mem-per-cpu=500 # Beware! $HOME will not be expanded and invalid paths will result Slurm jobs # hanging indefinitely with status CG (completing) when calling scancel! #SBATCH --output="/home/<username>/spark/logs/%j.out" #SBATCH --error="/home/<username>/spark/logs/%j.err" #SBATCH --time=01:00:00 # This section will be run when started by sbatch if [ "$1" != 'srunning' ]; then this=$0 # I experienced problems with some nodes not finding the script: # slurmstepd: execve(): /var/spool/slurm/job123/slurm_script: # No such file or directory # that's why this script is being copied to a shared location to which # all nodes have access to: script=/<shared folder>/${SLURM_JOBID}_$(basename "$0") cp "$this" "$script" # This might not be necessary on all clusters module load scala/2.10.4 java/jdk1.7.0_25 cuda/7.0.28 export sparkLogs=$HOME/spark/logs export sparkTmp=$HOME/spark/tmp mkdir -p "$sparkLogs" "$sparkTmp" export SPARK_ROOT=$HOME/spark-1.5.2-bin-hadoop2.6/ export SPARK_WORKER_DIR=$sparkLogs export SPARK_LOCAL_DIRS=$sparkLogs export SPARK_MASTER_PORT=7077 export SPARK_MASTER_WEBUI_PORT=8080 export SPARK_WORKER_CORES=$SLURM_CPUS_PER_TASK export SPARK_DAEMON_MEMORY=$(( $SLURM_MEM_PER_CPU * $SLURM_CPUS_PER_TASK / 2 ))m export SPARK_MEM=$SPARK_DAEMON_MEMORY srun "$script" 'srunning' # If run by srun, then decide by $SLURM_PROCID whether we are master or worker else source "$SPARK_ROOT/sbin/spark-config.sh" source "$SPARK_PREFIX/bin/load-spark-env.sh" if [ $SLURM_PROCID -eq 0 ]; then export SPARK_MASTER_IP=$(hostname) MASTER_NODE=$(scontrol show hostname $SLURM_NODELIST | head -n 1) # The saved IP address + port is necessary alter for submitting jobs echo "spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT" > "$sparkLogs/${SLURM_JOBID}_spark_master" "$SPARK_ROOT/bin/spark-class" org.apache.spark.deploy.master.Master \ --ip $SPARK_MASTER_IP \ --port $SPARK_MASTER_PORT \ --webui-port $SPARK_MASTER_WEBUI_PORT else # $(scontrol show hostname) is used to convert eg host20[39-40] # to host2039 this step assumes that SLURM_PROCID=0 corresponds to # the first node in SLURM_NODELIST ! MASTER_NODE=spark://$(scontrol show hostname $SLURM_NODELIST | head -n 1):7077 "$SPARK_ROOT/bin/spark-class" org.apache.spark.deploy.worker.Worker $MASTER_NODE fi fi
现在开始sbatch作业，并在example.jar ：
mkdir -p $HOME/spark/logs jobid=$(sbatch ./start_spark_slurm.sh) jobid=${jobid##Submitted batch job } MASTER_WEB_UI='' while [ -z "$MASTER_WEB_UI" ]; do sleep 1s if [ -f $HOME/spark/logs/$jobid.err ]; then MASTER_WEB_UI=$(sed -nE 's|.*Started MasterWebUI at (http://[0-9.:]*)|\1|p' $HOME/spark/logs/$jobid.err) fi done MASTER_ADDRESS=$(cat $HOME/spark/logs/${jobid}_spark_master) $HOME/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --master $MASTER_ADDRESS example.jar firefox $MASTER_WEB_UI
正如maxmlnkn的回答所述，您需要一种机制来在Spark jar提交之前执行Spark jar，然后在Slurm分配中设置/启动相应的Spark守护进程。
已经开发了几个脚本/系统来为你做这个设置。 上面链接的答案提到了Magpie @ https://github.com/LLNL/magpie （完全公开：我是这些脚本的开发者/维护者）。 Magpie提供了一个作业提交文件（submission-scripts / script-sbatch-srun / magpie.sbatch-srun-spark），供您编辑并执行您的集群特定信息和作业脚本。 configuration完成后，您可以通过“sbatch -k ./magpie.sbatch-srun-spark”提交。 有关更多详细信息，请参阅doc / README.spark。
我会提到有其他的脚本/系统为你做这个。 我缺乏与他们的经验，所以不能评论，除了下面的链接。