Aren’t you thinking why there is one more post on the installation of on Mac OS X ?? Apache Spark If you are think so, then you wouldn’t have had success in following the previous tutorial or if this is the first time you are seeing such tutorial like this then, congrats you have come to the right place. :P The purpose of me writing is just that I wanna write a post in Medium, because it has been quite a time since I published one. Ok let gets back to the core topic. So you would have done this step of for installation steps. Googling You did this right ? The first link is not really an installation guide, the second one is an installation guide, but its uses a very old version of Spark, the others are basically steps to follow and are not pictorial. We love screen shots aren’t we ? because we know the saying “A picture says a thousand words.” So first steps first. Downloads & Installations a package manager for OS X. Brew , Execute the below command to install the tool. ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" Java Mac already has Java. So skipping this part. Scala using brew we can install Scala. brew install scala At the time of writing this tutorial, the Scala version that shall be installed is 2.11.7 Apache Spark I am a fan of this IDE for writing Python applications. PyCharm , (I will write one more tutorial on how to directly run the Python scripts from IDE on the Spark environment.) , If you want to use the Scala/Java API of Spark. IntelliJ IDEA The community editions of the IDEs are free. If you are student, JetBrains unlock many features. You just have to request using your school email id. Setting up Spark I would always like to keep things organized, so lets move the downloaded Spark folder to /usr/local/spark tar -xvzf spark-1.6.0.tgz # to extract the contents of the archive mv spark-1.6.0 /usr/local/spark # moves the folder from Downloads to local cd /usr/local/spark Now we have to build the spark that was downloaded, or else we cannot run the programs or examples. This might take a little time (Actually more than you can think of). build/sbt clean assembly # builds the spark environment Testing Spark installation If you already have a hands on experience with Spark write your own Word count program in the shell else there are few examples that Spark will provide us to get used to the environment. Shells : ./bin/spark-shell # Scala shell OR ./bin/pyspark # Python shell Pi example: ./bin/run-example org.apache.spark.examples.SparkPi All the best for building some world class Big data apps.
