Home > Articles
Here is an easy Step by Step guide to installing PySpark and Apache Spark on MacOS. Step 1: Get Homebrew Homebrew makes installing applications and languages on a Mac OS a lot easier. Homebrew makes installing applications and languages on a Mac OS a lot easier. To execute this recipe, you will need a bash Terminal and an internet connection. Also, before we start any work, you should clone the GitHub repository for this book.The repository contains all the codes (in the form of notebooks) and all the data you will need to follow the examples in this book.
␡- Installing Spark in Standalone Mode
< BackPage 3 of 9Next >
This chapter is from the book Apache Spark in 24 Hours, Sams Teach Yourself
This chapter is from the book
This chapter is from the book
Installing Spark in Standalone Mode
In this section I will cover deploying Spark in Standalone mode on a single machine using various platforms. Feel free to choose the platform that is most relevant to you to install Spark on.
Getting Spark
In the installation steps for Linux and Mac OS X, I will use pre-built releases of Spark. You could also download the source code for Spark and build it yourself for your target platform using the build instructions provided on the official Spark website. I will use the latest Spark binary release in my examples. In either case, your first step, regardless of the intended installation platform, is to download either the release or source from: http://spark.apache.org/downloads.html
This page will allow you to download the latest release of Spark. In this example, the latest release is 1.5.2, your release will likely be greater than this (e.g. 1.6.x or 2.x.x).
FIGURE 3.1 The Apache Spark downloads page.
Installing a Multi-node Spark Standalone Cluster
Using the steps outlined in this section for your preferred target platform, you will have installed a single node Spark Standalone cluster. I will discuss Spark’s cluster architecture in more detail in Hour 4, “Understanding the Spark Runtime Architecture.” However, to create a multi-node cluster from a single node system, you would need to do the following:
- Ensure all cluster nodes can resolve hostnames of other cluster members and are routable to one another (typically, nodes are on the same private subnet).
- Enable passwordless SSH (Secure Shell) for the Spark master to the Spark slaves (this step is only required to enable remote login for the slave daemon startup and shutdown actions).
- Configure the spark-defaults.conf file on all nodes with the URL of the Spark master node.
- Configure the spark-env.sh file on all nodes with the hostname or IP address of the Spark master node.
- Run the start-master.sh script from the sbin directory on the Spark master node.
- Run the start-slave.sh script from the sbin directory on all of the Spark slave nodes.
- Check the Spark master UI. You should see each slave node in the Workers section.
- Run a test Spark job.
Related Resources
- Book $39.99
- eBook (Watermarked) $31.99
- Book $27.99
I decided to teach myself how to work with big data and came across Apache Spark. While I had heard of Apache Hadoop, to use Hadoop for working with big data, I had to write code in Java which I was not really looking forward to as I love to write code in Python. Spark supports a Python programming API called PySpark that is actively maintained and was enough to convince me to start learning PySpark for working with big data.
In this post, I describe how I got started with PySpark on Windows. My laptop is running Windows 10. So the screenshots are specific to Windows 10. I am also assuming that you are comfortable working with the Command Prompt on Windows. You do not have to be an expert, but you need to know how to start a Command Prompt and run commands such as those that help you move around your computer’s file system. In case you need a refresher, a quick introduction might be handy.
Often times, many open source projects do not have good Windows support. So I had to first figure out if Spark and PySpark would work well on Windows. The official Spark documentation does mention about supporting Windows.
Installing Prerequisites
PySpark requires Java version 7 or later and Python version 2.6 or later. Let’s first check if they are already installed or install them and make sure that PySpark can work with these two components.
Java
Java is used by many other software. So it is quite possible that a required version (in our case version 7 or later) is already available on your computer. To check if Java is available and find it’s version, open a Command Prompt and type the following command.
If Java is installed and configured to work from a Command Prompt, running the above command should print the information about the Java version to the console. For example, I got the following output on my laptop.
Instead if you get a message like
It means you need to install Java. To do so,
- Go to the Java download page. In case the download link has changed, search for
Java SE Runtime Environment
on the internet and you should be able to find the download page. - Click the Download button beneath JRE
- Accept the license agreement and download the latest version of
Java SE Runtime Environment
installer. I suggest getting the exe for Windows x64 (such asjre-8u92-windows-x64.exe
) unless you are using a 32 bit version of Windows in which case you need to get the Windows x86 Offline version. - Run the installer.
After the installation is complete, close the Command Prompt if it was already open, open it and check if you can successfully run
java -version
command.Python
Python is used by many other software. So it is quite possible that a required version (in our case version 2.6 or later) is already available on your computer. To check if Python is available and find it’s version, open a Command Prompt and type the following command.
If Python is installed and configured to work from a Command Prompt, running the above command should print the information about the Python version to the console. For example, I got the following output on my laptop.
Instead if you get a message like
It means you need to install Python. To do so,
- Go to the Python download page.
- Click the Latest Python 2 Release link.
- Download the
Windows x86-64 MSI installer
file. If you are using a 32 bit version of Windows download theWindows x86 MSI installer
file. - When you run the installer, on the Customize Python section, make sure that the option Add python.exe to Path is selected. If this option is not selected, some of the PySpark utilities such as
pyspark
andspark-submit
might not work.
After the installation is complete, close the Command Prompt if it was already open, open it and check if you can successfully run
python --version
command.Installing Apache Spark
- Go to the Spark download page.
- For Choose a Spark release, select the latest stable release of Spark.
- For Choose a package type, select a version that is pre-built for the latest version of Hadoop such as Pre-built for Hadoop 2.6.
- For Choose a download type, select Direct Download.
- Click the link next to Download Spark to download a zipped tarball file ending in .tgz extension such as
spark-1.6.2-bin-hadoop2.6.tgz
. - In order to install Apache Spark, there is no need to run any installer. You can extract the files from the downloaded tarball in any folder of your choice using the 7Zip tool.Make sure that the folder path and the folder name containing Spark files do not contain any spaces.
In my case, I created a folder called
spark
on my C drive and extracted the zipped tarball in a folder called spark-1.6.2-bin-hadoop2.6
. So all Spark files are in a folder called C:sparkspark-1.6.2-bin-hadoop2.6
. From now on, I will refer to this folder as SPARK_HOME
in this post.To test if your installation was successful, open a Command Prompt, change to SPARK_HOME directory and type
binpyspark
. This should start the PySpark shell which can be used to interactively work with Spark. I got the following messages in the console after running binpyspark
command.The last message provides a hint on how to work with Spark in the PySpark shell using the
sc
or sqlContext
names. For example, typing sc.version
in the shell should print the version of Spark. You can exit from the PySpark shell in the same way you exit from any Python shell by typing exit()
. The PySpark shell outputs a few messages on exit. So you need to hit enter to get back to the Command Prompt.
Configuring the Spark Installation
Starting the PySpark shell produces a lot of messages of type INFO, ERROR and WARN. In this section we will see how to remove these messages.
By default, the Spark installation on Windows does not include the
winutils.exe
utility that is used by Spark. If you do not tell your Spark installation where to look for winutils.exe
, you will see error messages when running the PySpark shell such asThis error message does not prevent the PySpark shell from starting. However if you try to run a standalone Python script using the
binspark-submit
utility, you will get an error. For example, try running the wordcount.py
script from the examples
folder in the Command Prompt when you are in the SPARK_HOME directory.which produces the following error that also points to missing
winutils.exe
Installing winutils
Let’s download the
winutils.exe
and configure our Spark installation to find winutils.exe
.- Create a
hadoopbin
folder inside the SPARK_HOME folder. - Download the winutils.exe for the version of hadoop against which your Spark installation was built for. In my case the hadoop version was 2.6.0. So I downloaded the winutils.exe for hadoop 2.6.0 and copied it to the
hadoopbin
folder in the SPARK_HOME folder. - Create a system environment variable in Windows called
SPARK_HOME
that points to the SPARK_HOME folder path. Search the internet in case you need a refresher on how to create environment variables in your version of Windows such as articles like these. - Create another system environment variable in Windows called
HADOOP_HOME
that points to the hadoop folder inside the SPARK_HOME folder.
Since the
hadoop
folder is inside the SPARK_HOME folder, it is better to create HADOOP_HOME
environment variable using a value of %SPARK_HOME%hadoop
. That way you don’t have to change HADOOP_HOME
if SPARK_HOME
is updated.If you now run the
binpyspark
script from a Windows Command Prompt, the error messages related to winutils.exe
should be gone. For example, I got the following messages after running the binpyspark
utility after configuring winutils
The
binspark-submit
utility can also be successfully used to run wordcount.py
script.Configuring the log level for Spark
There are still a lot of extra INFO messages in the console everytime you start or exit from a PySpark shell or run the
spark-submit
utility. So let’s make one more change to our Spark installation so that only warning and error messages are written to the console. In order to do this- Copy the
log4j.properties.template
file in theSPARK_HOMEconf
folder aslog4j.properties
file in theSPARK_HOMEconf
folder. - Set the
log4j.rootCategory
property value toWARN, console
- Save the
log4j.properties
file.
Now any informative messages will not be logged to the console. For example, I got the following messages after running the
binpyspark
utility once I configured the log level to WARN.Summary
In order to work with PySpark, start a Windows Command Prompt and change into your SPARK_HOME directory.
Pyspark Download Mac Free
- To start a PySpark shell, run the
binpyspark
utility. Once your are in the PySpark shell use thesc
andsqlContext
names and typeexit()
to return back to the Command Prompt. - To run a standalone Python script, run the
binspark-submit
utility and specify the path of your Python script as well as any arguments your Python script needs in the Command Prompt. For example, to run thewordcount.py
script fromexamples
directory in your SPARK_HOME folder, you can run the following commandbinspark-submit examplessrcmainpythonwordcount.py README.md
References
I used the following references to gather information about this post.
- Downloading Spark and Getting Started (chapter 2) from O’Reilly’s Learning Spark book.
Share on: Twitter ❄ Facebook ❄ Google+ ❄ Email
Any suggestions or feedback? Leave your comments below.
Pyspark Download Mac Iso
Please enable JavaScript to view the comments powered by Disqus.comments powered by Disqus