Setup Apache Spark on Windows.

It’s great to try Spark on Windows. Here is how you can setup Spark Standalone mode on Windows. This article shows you how to setup without mistakes. 

To install Spark on a windows based environment the following prerequisites should be fulfilled first.

Requirement:

1. Java 6+
2. Scala 2.10.x
3. sbt (Simple build Tools)
4. GIT
5. Spark 1.*.*
6. Intelij IDEA or Scala IDE for programming

Installations:

Step1: Download and install Java: 
Now Java 8 works on Windows when you build Spark it takes a mistake. It's notice below.


Step2: Download and install Scala

Scala 2.10.x works better. Also note: please make sure there is no space in your installation directory(e.g. you may install it in C:\scala, but you cannot install it in C:\Program Files\scala)


Step3: Download and install Git

You have to make sure git is installed and “git” can be executed in commend line(it is an option when you install git)

Step4: Download and install SBT
The latest version of sbt is compatible with JAVA 8 and scala 2.10.x

Step5: Download and extract Spark 
Extract file Spark in anywhere, but it is better in: c:\spark-1.6.0 

Step 6: Build Spark 

•  press WIN + R and run “cmd”
•  go to your spark boot directory: cd c:\spark-1.6.0
•  input “sbt assembly” and run

Videos tutorial:

Note:

The installation will take about 10-20 minutes

Then, you may run spark-shell, and spark has been already built if you see the following picture:

Troubleshoot:

1. java/javac/git/sbt is not recognized as an internal or external command

You didn't put all the software path into environment variables path(they are not executive in commend line )

2. [error]: not a valid commend: package/assemble

please make sure you installed the software version as I indicated before

3. [error]: java.lang.OutOfMemoryError: Java heap space

A: It is because sbt does not get enough memory allocated. Please go to your sbt/conf directory, and find the file “sbtconfig”.

Revise -Xmx512M to -Xmx2048M; -XX:MaxPermSize=256m to -XX:MaxPermSize=1024m; ReservedCodeCacheSize=512m