Spark Standalone in AWS

Spark is also good tool for processing small size job.

Prepare docker image

from alpine(need to install more packages) or ubuntu iamge.

FROM openjdk:8-alpine
MAINTAINER xxx@mail
RUN apk add --update python2 python3 bash coreutils procps gcompat
# Install your python library
RUN python3 -m pip install bs4
WORKDIR /app/service
RUN wget http://apache.mirror.cdnetworks.com/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz && tar xfz spark-2.4.0-bin-hadoop2.7.tgz && ln -s spark-2.4.0-bin-hadoop2.7 spark
RUN rm -f *.tar.gz *.tgz
ENV SPARK_HOME=/app/service/spark
ENV SPARK_LOG_DIR=/app/logs/spark
ENV SPARK_LOCAL_DIRS=/app/tmp
ENV PATH=$SPARK_HOME/sbin:$PATH
ENV SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dcom.amazonaws.services.s3.enforceV4=true"

Build Dockerfile

docker build -t your_docker_hub_address/spark:2.4.0 .
docker push your_docker_hub_address/spark:2.4.0

Start standalone-master node in 1 node

docker run -d --net=host -v /app/logs:/app/logs -v /app/tmp:/app/tmp your_docker_hub_address/spark:2.4.0  /bin/bash -c "SPARK_NO_DAEMONIZE= start-master.sh"

Start standalone-slave nodes using ASG

In my case, I put this to AWS-AutoScalingGroup(user-data section). So, I can increase ASG size anytime.

CPU=$(cat /proc/cpuinfo | grep processor | wc -l)
docker run -d --net=host -v /app/logs:/app/logs -v /app/tmp:/app/tmp your_docker_hub_address/spark:2.4.0\
/bin/bash -c "SPARK_NO_DAEMONIZE= start-slave.sh spark://master_ip_address:7077 -c $(($CPU -1))"

Set-up standalone-driver node

In my case, I set-up spark driver in Jenkins Slave node.

Download below 2 files to folder (ex: /opt/spark/jars)

To access S3, version is important.

wget http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.3/hadoop-aws-2.7.3.jar
wget http://central.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar

Edit conf/spark-defaults.conf

• com.amazonaws.services.s3.enforceV4 : because aws library version is too old. Korean users have to add this property.
• For Hadoop, uploading files are write to tmp directory and directory names are replaced. But S3, it based on object.(prefixes cannot be changed.)

spark.master                     spark://MASTER_URL:7077
spark.driver.core                1
spark.driver.memory              2g
spark.driver.extraJavaOptions    -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Dcom.amazonaws.services.s3.enforceV4=true
spark.executor.extraJavaOptions  -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Dcom.amazonaws.services.s3.enforceV4=true
spark.executor.memory 50g
spark.jars      /opt/spark/jars/aws-java-sdk-1.7.4.jar,/opt/spark/jars/hadoop-aws-2.7.3.jar
spark.jars.packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.3,org.mongodb:mongo-java-driver:3.8.0,org.elasticsearch:elasticsearch-spark-20_2.11:6.2.4
spark.ui.showConsoleProgress    true
spark.speculation   false
spark.serializer    org.apache.spark.serializer.KryoSerializer
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version    2
spark.hadoop.mapreduce.fileoutputcommitter.cleanup.skipped  true
spark.hadoop.parquet.enable.summary-metadata    false
spark.hadoop.fs.s3a.endpoint    s3.ap-northeast-2.amazonaws.com
spark.hadoop.fs.s3a.impl    org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3n.impl    org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3.impl     org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.experimental.input.fadvise  random
spark.hadoop.fs.s3a.fast.upload     true
spark.hadoop.fs.s3a.fast.upload.active.blocks   8