Spark runs fine as a single-box cluster: our 2-core Ubuntu VM counted 110 million words in about 13 seconds. What usually goes wrong is everything around the job. start-master.sh advertises whatever the hostname resolves to, nothing comes back after a reboot, and the master opens an unauthenticated submission port that most guides never mention. You’ll fix all three here.
This guide shows how to install Spark on Ubuntu as a standalone cluster from a verified tarball, with the master and worker under systemd and the firewall closed to everything but your LAN. You then check the web UIs on ports 8080, 8081 and 4040, test spark-shell and PySpark, and time SparkPi and word count jobs. Everything ran on one host, so a multi-node layout was not tested. The commands were run on Ubuntu 26.04 LTS with Spark 4.2.0 and OpenJDK 21 in October 2026.
Prerequisites
You need very little to follow along, because Spark’s standalone mode ships its own cluster manager and doesn’t need Hadoop, YARN or HDFS:
- Ubuntu 26.04 LTS with sudo access
- 2 vCPUs, 4 GB RAM and 10 GB free disk
- Outbound HTTPS to downloads.apache.org and dlcdn.apache.org
- A LAN subnet you trust for the web UIs
That spec is a floor for following along, not a production recommendation. Spark sizing comes from executor memory per core and from how much data a stage shuffles. A real worker usually lands at 4 to 8 GB of RAM per core it offers, plus fast local disk for shuffle spill, so size yours from the biggest stage you expect to run. If you are still deciding whether Spark is the right engine for the job, one of the Kafka and Spark books we reviewed will save you more time than any tuning flag. Debian 13 readers have a separate Spark guide for Debian.
Set the shell variables
Two values repeat through this guide: the IP the master binds to and the Spark release you download. Pin the master to the host’s real LAN IP rather than its hostname. Left alone, start-master.sh advertises the output of hostname -f, which on a cloud image is usually a bare name like spark01 that other machines can’t resolve. Find the address first:
hostname -I | awk '{print $1}'Put that address in HOST_IP. The second line asks the Apache download server for the newest release directory, so you always get the current stable build without editing the command.
export HOST_IP="10.0.1.50"
export SPARK_VERSION=$(curl -s https://downloads.apache.org/spark/ | grep -oP "spark-\K[0-9]+\.[0-9]+\.[0-9]+(?=/)" | sort -V | tail -1)
echo "Master IP: ${HOST_IP} Spark: ${SPARK_VERSION}"The echo must print an IP and a clean three-part version string that matches the newest release on the downloads page. An empty version means curl couldn’t reach the mirror, so fix outbound HTTPS before going further. Both variables only live in the current shell, so run the export block again if you reconnect.
Install Java 21
Spark’s current documentation lists Java 17, 21 and 25 as supported runtimes, with Scala 2.13 bundled and Python 3.10 or newer for PySpark. On Ubuntu 26.04 the default-jre-headless package pulls Java 25, which works, but we install openjdk-21-jre-headless by name. Java 21 is a long-term support release on that list, and the same package is the default runtime on Ubuntu 24.04, so the JAVA_HOME path in this guide stays the same if you move between the two LTS releases:
sudo apt update
sudo apt install -y openjdk-21-jre-headless curl gnupgCheck which runtime is on the PATH, because a leftover Java 8 or 11 from an older install would stop Spark from starting at all.
java -versionYou should see a 21.x build from the Ubuntu archive. If you see an older version, run sudo update-alternatives --config java and pick the 21 entry.
openjdk version "21.0.12.1" 2026-08-18
OpenJDK Runtime Environment (build 21.0.12.1+1-1-26.04.4-Ubuntu)
OpenJDK 64-Bit Server VM (build 21.0.12.1+1-1-26.04.4-Ubuntu, mixed mode, sharing)A headless JRE is enough, since Spark ships its own Scala compiler for spark-shell. If you also compile Java code on this box or want to switch runtimes with update-alternatives, the OpenJDK install guide for Ubuntu 26.04 covers the full JDK.
Download and verify Spark
Spark has no apt repository, so the tarball from the Apache Spark downloads page is the install. Pull the binary from the CDN, but take the checksum, signature and signing keys from downloads.apache.org, because Apache only publishes those on its own server and a mirror cannot vouch for itself.
cd /tmp
curl -sSfLO "https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz"
curl -sSfLO "https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz.sha512"
curl -sSfLO "https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz.asc"
curl -sSfLO https://downloads.apache.org/spark/KEYSThe tarball is about 555 MB and took 40 seconds to arrive on our lab line, about 155 MB more than the 3.5 build this guide used to install, and 456 MB of the unpacked tree is jars. Check the SHA-512 first:
sha512sum -c "spark-${SPARK_VERSION}-bin-hadoop3.tgz.sha512"Anything other than OK means a truncated or tampered download, so delete the file and fetch it again before you go any further.
spark-4.2.0-bin-hadoop3.tgz: OKA matching checksum only proves the file wasn’t damaged in transit. The GPG signature proves a Spark release manager built it, so import the project’s KEYS file and verify it.
gpg --import KEYS
gpg --verify "spark-${SPARK_VERSION}-bin-hadoop3.tgz.asc" "spark-${SPARK_VERSION}-bin-hadoop3.tgz" 2>&1 | grep -E "Good signature|BAD signature" | cut -d"<" -f1The line you want starts with Good signature. A BAD signature line means the tarball doesn’t match what the release manager signed, so stop there.
gpg: Good signature from "Huaxin Gao (CODE SIGNING KEY) Without the grep, gpg also warns that the key is not certified with a trusted signature. That only means you haven’t personally signed the release manager’s key, and it’s expected on a first import, so the checks pass and the tarball is safe to unpack.
Install Spark on Ubuntu under /opt
Spark should run as its own unprivileged user, not as root or as your login, so a bad job cannot touch anything outside the Spark tree. Create a system account, unpack the release into a versioned directory and point a stable /opt/spark symlink at it. The symlink is what makes upgrades painless later: unpack the next release beside this one, copy conf/spark-env.sh across (it lives inside the versioned directory), move the link and restart the services.
sudo useradd --system --home-dir /opt/spark --shell /usr/sbin/nologin spark
sudo tar -xzf "spark-${SPARK_VERSION}-bin-hadoop3.tgz" -C /opt
sudo ln -sfn "/opt/spark-${SPARK_VERSION}-bin-hadoop3" /opt/spark
sudo mkdir -p /opt/spark/logs /opt/spark/work
sudo chown -R spark:spark "/opt/spark-${SPARK_VERSION}-bin-hadoop3"Next, put the Spark binaries on every user’s PATH. A file in /etc/profile.d/ applies to all login shells, which beats editing each user’s ~/.bashrc and keeps the setting in one place you can remove later.
echo 'export SPARK_HOME=/opt/spark' | sudo tee /etc/profile.d/spark.sh
echo 'export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin' | sudo tee -a /etc/profile.d/spark.sh
source /etc/profile.d/spark.shRun which spark-shell and it should print /opt/spark/bin/spark-shell. New SSH sessions pick the file up on their own, so only the current shell needs the source. At this point the install has written these paths:
/opt/spark-<version>-bin-hadoop3, the unpacked release/opt/spark, the symlink every config points at/opt/spark/work, application files and executor stdout/stderr/var/lib/spark/local, shuffle and spill scratch, created in the next section/etc/profile.d/spark.sh, the PATH snippet
Removing Spark later means deleting those five, the two systemd units you add next and the spark user. The scratch directory sits outside the release tree on purpose, so an upgrade never wipes it or leaves it behind in an old release.
Configure the master and worker
Every Spark launcher sources conf/spark-env.sh, so the bind address, ports and worker resources live in one file. Without it, our master came up as spark://spark01:7077, because start-master.sh falls back to hostname -f and advertises that bare name, which a worker on another machine can’t resolve without DNS or a hosts entry. It also opened port 6066, the REST submission server, which is on by default and accepts jobs without any authentication. Anyone who can reach 6066 can run code as the spark user, so we switch it off unless you have a reason to use it.
sudo vim /opt/spark/conf/spark-env.shPaste the settings below. HOST_IP_HERE is a placeholder that the next command swaps for your HOST_IP, since shell variables do not expand inside a file you paste into vim.
SPARK_MASTER_HOST=HOST_IP_HERE
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_WEBUI_PORT=8081
SPARK_LOCAL_DIRS=/var/lib/spark/local
SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=false"
JAVA_HOME=/usr/lib/jvm/java-21-openjdk-$(dpkg --print-architecture)Replace the placeholder and hand the file to the spark user, because the daemons read it as that account.
sudo sed -i "s/HOST_IP_HERE/${HOST_IP}/g" /opt/spark/conf/spark-env.sh
sudo chown spark:spark /opt/spark/conf/spark-env.shSPARK_WORKER_CORES and SPARK_WORKER_MEMORY set what the worker offers the cluster, not what it uses at idle. Leave a core and at least 1 GB for the OS and the two daemons on a shared box. Our master and worker sat at 216 MiB and 198 MiB of resident memory after a reboot, so the daemons themselves are cheap; the executors are where the RAM goes.
SPARK_LOCAL_DIRS moves shuffle files and spill off /tmp, which is Spark’s default scratch location. On Ubuntu 26.04 /tmp is a tmpfs, so a big shuffle would land in RAM and compete with the executors. The JAVA_HOME line asks dpkg for the architecture, so the same file works on arm64 servers, where the JDK lives in java-21-openjdk-arm64.
The scratch directory needs one more decision, because spark-submit reads the same spark-env.sh, so your own driver process writes scratch there as well as the spark user. Give it to the spark group with the setgid bit and add the people allowed to submit jobs to that group. Anyone outside the group cannot read driver scratch or fill the disk through it:
sudo mkdir -p /var/lib/spark/local
sudo chown spark:spark /var/lib/spark/local
sudo chmod 2770 /var/lib/spark/local
sudo usermod -aG spark "$USER"Log out and back in so the new group membership applies, then check that id lists spark. During a word count run we saw the driver’s blockmgr-* directory owned by josphat:spark next to the executor’s directory owned by spark:spark, and an account outside the group got Permission denied listing the directory.
Run the master and worker with systemd
The start-master.sh and start-worker.sh scripts fork into the background by default, which is fine at a terminal but means nothing restarts them after a crash or reboot. Setting SPARK_NO_DAEMONIZE=true keeps the Java process in the foreground, so systemd can supervise it directly and the logs land in the journal instead of /opt/spark/logs. Create the master unit.
sudo vim /etc/systemd/system/spark-master.serviceAdd this unit definition. Restart=on-failure brings the master back if the JVM dies, and a clean systemctl stop leaves it down.
[Unit]
Description=Apache Spark standalone master
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=spark
Group=spark
Environment=SPARK_NO_DAEMONIZE=true
ExecStart=/opt/spark/sbin/start-master.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetThe worker unit is almost the same, except it takes the master URL as an argument and orders itself after the master on this host.
sudo vim /etc/systemd/system/spark-worker.servicePaste the worker definition, again with the placeholder for the master IP. On a second machine you would drop spark-master.service from After=, since no master runs there.
[Unit]
Description=Apache Spark standalone worker
After=network-online.target spark-master.service
Wants=network-online.target
[Service]
Type=simple
User=spark
Group=spark
Environment=SPARK_NO_DAEMONIZE=true
ExecStart=/opt/spark/sbin/start-worker.sh spark://HOST_IP_HERE:7077
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetFill in the master address, reload systemd and start both services. The worker retries its connection every few seconds, so it does not matter if it comes up a moment before the master is ready.
sudo sed -i "s/HOST_IP_HERE/${HOST_IP}/g" /etc/systemd/system/spark-worker.service
sudo systemctl daemon-reload
sudo systemctl enable --now spark-master spark-workerCheck the master’s journal for the lines that matter: the URL it advertises, the leader election and the worker registration.
journalctl -u spark-master --no-pager | grep -E "Starting Spark master|ALIVE|Registering worker"A healthy master reports the IP URL, goes ALIVE and registers a worker with the cores and memory you set in spark-env.sh.
Oct 17 21:57:17 spark01 start-master.sh[31296]: 26/10/17 21:57:17 INFO Master: Starting Spark master at spark://10.0.1.50:7077
Oct 17 21:57:18 spark01 start-master.sh[31296]: 26/10/17 21:57:18 INFO Master: I have been elected leader! New state: ALIVE
Oct 17 21:57:29 spark01 start-master.sh[31296]: 26/10/17 21:57:29 INFO Master: Registering worker 10.0.1.50:37551 with 2 cores, 2.0 GiB RAMIf the master line shows a hostname instead of your IP, spark-env.sh wasn’t read, which usually means the placeholder is still in it. No “Registering worker” line means the worker is pointed at the wrong URL; the troubleshooting section shows that error. We rebooted the VM to confirm the units come back on their own, and the worker registered one second after the master went ALIVE.
The terminal capture below shows the release, both services active and the three ports listening, with the master’s RPC port bound only to the LAN address:
The web UIs on 8080 and 8081 listen on all interfaces regardless of SPARK_MASTER_HOST, so the firewall is what keeps them private. Until you add the rules in the next step, anyone who can reach the VM can browse both pages.
Open the firewall
Spark’s web UIs have no login by default, and the standalone master accepts any application that can reach port 7077. Treat all of them as LAN-only, and open only these ports on a single-host cluster:
- 7077/tcp: master RPC, where workers and drivers connect
- 8080/tcp: master web UI
- 8081/tcp: worker web UI
- 4040/tcp: application UI, open only while a job runs
UFW ships inactive on Ubuntu, so allow SSH before you enable it or you will cut off your own session. Swap 10.0.1.0/24 for the subnet your browsers and clients sit on.
sudo ufw allow OpenSSH
sudo ufw allow from 10.0.1.0/24 to any port 7077,8080,8081,4040 proto tcp
sudo ufw --force enable
sudo ufw status numberedThe status should list the SSH rule and one Spark rule scoped to your subnet. If the Spark rule says Anywhere, you dropped the from clause, so delete it by number and add it again.
Status: active
To Action From
-- ------ ----
[ 1] OpenSSH ALLOW IN Anywhere
[ 2] 4040,7077,8080,8081/tcp ALLOW IN 10.0.1.0/24
[ 3] OpenSSH (v6) ALLOW IN Anywhere (v6) When you add workers on other machines, these four ports are no longer enough. Each worker listens on a random RPC port, and executors open random ports back to the driver, so either allow all TCP between cluster nodes or pin SPARK_WORKER_PORT and spark.driver.port to fixed values. The same goes for a driver on your laptop in client mode: executors must reach its spark.driver.port and spark.blockManager.port, and a second application on one host takes UI port 4041, then 4042. We only submitted jobs on the Spark host itself, so those remote paths were not tested here. The UFW firewall guide covers subnet rules and logging in more depth.
Check the Spark web UIs
Open http://10.0.1.50:8080 in a browser on your LAN, using your own HOST_IP. The master page is the quickest health check you have: one worker in the ALIVE state, the cores and memory you configured, and every application that has run against the cluster with its duration:
Notice that every application took 1024 MiB per executor even though the worker offers 2 GiB. That is the default spark.executor.memory of 1 GB, and it’s covered in the sizing section at the end. The worker UI on port 8081 lists the executors it has launched:
Finished executors show the state KILLED, which looks alarming but is how standalone mode reports an executor it tore down after the application exited. A real failure shows FAILED or EXITED with a non-zero code, and the stderr link next to it has the stack trace.
Test spark-shell and PySpark
Both shells default to local[*] mode, which runs inside the shell’s own JVM and never touches your cluster. Pass --master so the test goes through the master and a real executor on the worker.
spark-shell --master "spark://${HOST_IP}:7077"Sum the numbers from 1 to a million at the scala> prompt. The answer must be 500000500000, and the context line should show the spark:// master rather than local[*]:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0
/_/
Using Scala version 2.13.18 (OpenJDK 64-Bit Server VM, Java 21.0.12.1)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context Web UI available at http://spark01:4040
Spark context available as 'sc' (master = spark://10.0.1.50:7077, app id = app-20261017215836-0000).
Spark session available as 'spark'.
scala> spark.range(1, 1000001).selectExpr("sum(id)").show()
+------------+
| sum(id)|
+------------+
|500000500000|
+------------+
scala> :quitPySpark uses the system python3, which is Python 3.14 on Ubuntu 26.04, and it worked without extra packages because the PySpark library ships inside the tarball. Start it the same way.
pyspark --master "spark://${HOST_IP}:7077"Build a two-row DataFrame and show it. A table back from the executor proves the Python worker, the JVM and the cluster all talk to each other, which is what you need before you submit real jobs.
Using Python version 3.14.4 (main, Aug 20 2026 10:41:58)
Spark context Web UI available at http://spark01:4040
Spark context available as 'sc' (master = spark://10.0.1.50:7077, app id = app-20261017215913-0001).
SparkSession available as 'spark'.
>>> df = spark.createDataFrame([("ubuntu", 26), ("debian", 13)], ["distro", "release"])
>>> df.show()
+------+-------+
|distro|release|
+------+-------+
|ubuntu| 26|
|debian| 13|
+------+-------+
>>> exit()For notebook work, point a JupyterLab install at the same master by building a SparkSession with .master("spark://10.0.1.50:7077"). If you need a different Python than the system one, the Python on Ubuntu 26.04 guide covers venv and uv, and you then set PYSPARK_PYTHON to that interpreter.
Run a measured Spark job
An interactive shell proves the wiring, but spark-submit is how real jobs run. Start with the SparkPi example that ships in the tarball, which estimates pi across 100 partitions and needs no input data.
time spark-submit --master "spark://${HOST_IP}:7077" "${SPARK_HOME}/examples/src/main/python/pi.py" 100The estimate changes slightly on every run because the samples are random. What you’re checking is the exit and the time, not the digits.
Pi is roughly 3.142720
real 0m17.492s
user 0m12.896s
sys 0m0.726sFor something closer to a data job, generate a text file with a known answer, so you can check the result as well as the timing. Ten million copies of the same eleven-word line give exactly 10 million of each word and 20 million of “the”. Keep it out of /tmp, because Ubuntu 26.04 mounts /tmp as tmpfs: the file would sit in RAM and disappear on reboot.
sudo mkdir -p /srv/spark-data
sudo chown "$USER": /srv/spark-data
yes "the quick brown fox jumps over the lazy dog on spark" | head -n 10000000 > /srv/spark-data/words.txtWrite the word count job to a file in your home directory. In client mode, which spark-submit uses by default, the script runs on your machine and only the input file has to be readable by the executors.
vim ~/wordcount.pyThe job reads the file, splits lines into words, counts them and prints the top five with the elapsed time:
import sys
import time
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, explode, split
spark = SparkSession.builder.appName("wordcount").getOrCreate()
start = time.time()
words = spark.read.text(sys.argv[1]).select(explode(split(col("value"), " ")).alias("word"))
counts = words.groupBy("word").count().orderBy(col("count").desc())
counts.show(5)
print(f"Job finished in {time.time() - start:.1f}s")
spark.stop()Submit it to the cluster. You get a table and one timing line back once the job finishes.
spark-submit --master "spark://${HOST_IP}:7077" ~/wordcount.py /srv/spark-data/words.txtEvery count must be an exact multiple of 10 million. Anything else means lines were dropped or split differently:
+-----+--------+
| word| count|
+-----+--------+
| the|20000000|
| lazy|10000000|
|jumps|10000000|
| dog|10000000|
| on|10000000|
+-----+--------+
only showing top 5 rows
Job finished in 13.6sWe ran each job at least three times on the 2 vCPU, 4 GB VM with the worker offering 2 cores. “Wall” is the full spark-submit run measured with /usr/bin/time, including JVM and Python startup:
| Job | Input | Spark time | Wall time |
|---|---|---|---|
SparkPi, Python (pi.py 100) | none | 12 to 13 s app duration | 16.8 to 17.9 s |
SparkPi, Scala (run-example SparkPi 100) | none | 7 s app duration | 10.8 to 11.1 s |
| Word count, PySpark DataFrame | 506 MB, 110M words | 12.3 to 14.0 s | 21.7 to 23.0 s |
The Python SparkPi takes about 6 seconds longer than the Scala one for identical work. pi.py runs its sampling function in Python worker processes and ships every result through serialization, while the Scala version stays in the JVM. The word count shows the other side. It uses the DataFrame API, so the split, explode and group run as JVM code and Python only drives the plan. That is why you can write DataFrame jobs in PySpark without a penalty, and why row-by-row Python functions are slow. Roughly 9 of the 22 wall seconds are fixed startup cost, so tiny jobs are dominated by overhead and benchmarks on small inputs tell you little.
While an application runs, its own UI is on port 4040. The Jobs tab for the word count shows the scan job reading the file in 4 tasks:
The Stages tab explains where those 4 tasks came from. Spark split the 505.6 MiB input into 128 MB partitions, and with only 2 cores it ran them two at a time, so adding cores would cut this stage’s time until the worker reaches 4 cores, one per partition. Past that, extra cores sit idle on an input this size:
The shuffle between the two stages is only 2.6 KiB because there are just ten distinct words. On real text with millions of distinct keys the shuffle grows into gigabytes, and that’s the number to watch when you size disks. The 4040 UI disappears as soon as the application stops, so set up the Spark history server if you need it afterwards. For a streaming source to feed jobs like this one, the Kafka install guide gets a broker running on the same LAN.
Troubleshoot Spark errors
Each error below came up on the lab VM while writing this guide. The heading is the exact text you would search for, and the paragraph under it gives the cause and the fix we applied.
[PATH_NOT_FOUND] Path does not exist: file:/tmp/words.txt
Our first word count input lived in /tmp, and after a reboot the next run failed with this AnalysisException because /tmp on Ubuntu 26.04 is a tmpfs that starts empty on every boot. Check with findmnt /tmp; if it says tmpfs, move your data to a disk path such as /srv/spark-data. On a multi-node cluster the same error also appears when a file exists on the driver but not on the worker, because each executor reads file: paths from its own disk. Use shared storage or copy the input to every node at the same path.
ERROR DiskBlockManager: Failed to create any local dir
We hit this the first time SPARK_LOCAL_DIRS pointed at a directory owned only by the spark user. The job died on submit with java.nio.file.AccessDeniedException: /opt/spark/local/blockmgr-..., because the driver runs as whoever calls spark-submit and it reads the same spark-env.sh. Make the directory group-owned by spark with mode 2770, add the submitting user to the spark group with sudo usermod -aG spark "$USER", log in again and resubmit. No service restart is needed. A world-writable 1777 directory also works, but only on a single-admin lab box, since every local user could then read driver scratch.
WARN Worker: Failed to connect to master localhost:7077
This repeats every few seconds when the worker’s master URL doesn’t match the address the master bound to. We reproduced it by starting a worker against spark://localhost:7077 while the master listened on the LAN IP.
26/10/17 22:07:29 INFO Worker: Connecting to master localhost:7077...
26/10/17 22:07:29 WARN Worker: Failed to connect to master localhost:7077
Caused by: java.io.IOException: Failed to connect to localhost/127.0.0.1:7077The master URL in spark-worker.service must match the “Starting Spark master at” line in the master’s journal character for character. Fix the unit, then run sudo systemctl daemon-reload and restart the worker.
NameError: name ‘spark’ is not defined
PySpark only creates the spark and sc objects through Python’s startup hook, and Python skips that hook when stdin isn’t a terminal. Piping code into the shell, as in echo "spark.range(5).show()" | pyspark, fails with this error. Put the code in a script and run it with spark-submit, building the session yourself with SparkSession.builder as wordcount.py does.
Size executors to fit the worker
Each application gets one executor per worker by default, with 1 GB of heap and every core the worker offers. Raising --executor-memory to 1800m made no difference to the word count (13.7 seconds against 13.3 to 14.0 at the default), because the job streams its input and keeps almost nothing in memory. Extra heap only pays off when a job caches data or shuffles wide aggregations. Asking for more than the worker has is worse than useless, as this 3g request against our 2g worker showed.
spark-submit --master "spark://${HOST_IP}:7077" --executor-memory 3g ~/wordcount.py /srv/spark-data/words.txtSpark does not fail the job. It waits for resources that will never appear and logs this every 15 seconds until you kill it with Ctrl+C.
26/10/17 22:11:54 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
26/10/17 22:12:09 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resourcesThe same warning appears when no worker is registered at all, so check the master UI first. If the worker is there, the request is bigger than SPARK_WORKER_MEMORY or SPARK_WORKER_CORES. Keep executor memory at or under what one worker offers, raise the worker’s offer in spark-env.sh when the box has spare RAM, and restart the worker. We’d add cores before memory on a box like this one, because the word count stage ran its 4 tasks two at a time and more cores cut that wait directly.
The link to download spark 2.4.5 is wrong.
Please replace with https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz
Thanks for capturing the issue. We’ve updated the article for the latest release.
Really very usefull document for me… its very helpful to avoid so many mistakes while installing Spark
Thanks, was the only setup that works to install spark, I hope to build nice project after your post.
Just live the right url to download: https://downloads.apache.org/spark/spark-3.2.1/
Awesome we’re happy to hear this.
Same here. Thanks to Josphat! Your post saved me a lot of time.
Welcome Fedrick.