Installation Spark Cluster on Windows

How to install Spark Cluster on Windows?

Installation spark cluster on windows isn't the same on Unix. When you using Unix terminal you start a standalone master

./sbin/start-master.sh

But on Windows it's more complex. How to do it right, without mistake and don't spend too much time.
Do it step by step and carefully.

Spark MLlib: From quick start to Scikit-Learn

Presentation by Joseph Bradley from Databricks.
He was speaking about Spark’s distributed Machine Learning Library - MLlib.
We will start off with a quick primer on machine learning, Spark MLlib, and a quick overview of some Spark machine learning use cases. We will continue with multiple Spark MLlib quick start demos. Afterwards, the talk will transition toward the integration of common data science tools like Python pandas, scikit-learn, and R with MLlib

Watch the movie about Webinar: Spark MLlib: From quick start to Scikit-Learn

Attachments:

• Slide
• Pop vs. Price SQL
• demo - 1 - sklearn
• demo - 2 - MLlib
• Pop. vs. Price LR

Interactive Analysis with the Spark Shell

Basics

Spark’s shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. It is available in either Scala (which runs on the Java VM and is thus a good way to use existing Java libraries) or Python. Start it by running the following in the Spark directory:

Scala

./bin/spark-shell

Spark’s primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from Hadoop InputFormats (such as HDFS files) or by transforming other RDDs. Let’s make a new RDD from the text of the README file in the Spark source directory:

scala> val textFile = sc.textFile("README.md")
textFile: spark.RDD[String] = spark.MappedRDD@2ee9b6e3

RDDs have actions, which return values, and transformations, which return pointers to new RDDs. Let’s start with a few actions:

scala> textFile.count() // Number of items in this RDD
res0: Long = 126

scala> textFile.first() // First item in this RDD
res1: String = # Apache Spark

Now let’s use a transformation. We will use the filter transformation to return a new RDD with a subset of the items in the file.

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
linesWithSpark: spark.RDD[String] = spark.FilteredRDD@7dd4af09

We can chain together transformations and actions:

scala> textFile.filter(line => line.contains("Spark")).count() // How many lines contain "Spark"?
res3: Long = 15

Video Tutorial

Hadoop - Eclipse configuration for Hadoop/MapR on Ubuntu

You installed Hadoop on computer and want to write code with Eclipse. But you don't know how to configure Eclipse for Hadoop? and How to import jar file to Eclipse?
You could read very much articles from internet but you spent too much time these problems. And when you make a mistake, you continue search and search too much. Of course it resolved or didn't.

Now i write this short article for you. Read and you'll find necessary information.

I don't write about " How install Hadoop on ***?" "How to setup Eclipse on ***?" and so on. From internet you can search or you known about it.

Now, configuration Eclipse for Hadoop

Requirements:  

1.  Installed Java 
2.  Installed Hadoop
3.  Have Eclipse IDE
4.  Download Hadoop plugin for Eclipse: here 

Configuration:

1. Combine the source code and build jar file with below command:

$ ant jar -Dversion=2.6.0 -Declipse.home=/opt/eclipse/ -Dhadoop.home=/usr/lib/hadoop/hadoop-2.6.0/

( my version 2.6.0, if your ver. is 2.4.0 change it by your version -2.4.0). 

2. Copy plugin jar file from hadoop-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.6.0.jar to /eclipse/plugin(directory).

3. After restart Eclipse, the MapReduce perspective will be available.

Video tutorial

Comment & share it if you find it's helpful.
Thank you for reading!

Setup Apache Spark on Windows.

It’s great to try Spark on Windows. Here is how you can setup Spark Standalone mode on Windows. This article shows you how to setup without mistakes. 

To install Spark on a windows based environment the following prerequisites should be fulfilled first.

Requirement:

1. Java 6+
2. Scala 2.10.x
3. sbt (Simple build Tools)
4. GIT
5. Spark 1.*.*
6. Intelij IDEA or Scala IDE for programming

Installations:

Step1: Download and install Java: 
Now Java 8 works on Windows when you build Spark it takes a mistake. It's notice below.


Step2: Download and install Scala

Scala 2.10.x works better. Also note: please make sure there is no space in your installation directory(e.g. you may install it in C:\scala, but you cannot install it in C:\Program Files\scala)


Step3: Download and install Git

You have to make sure git is installed and “git” can be executed in commend line(it is an option when you install git)

Step4: Download and install SBT
The latest version of sbt is compatible with JAVA 8 and scala 2.10.x

Step5: Download and extract Spark 
Extract file Spark in anywhere, but it is better in: c:\spark-1.6.0 

Step 6: Build Spark 

•  press WIN + R and run “cmd”
•  go to your spark boot directory: cd c:\spark-1.6.0
•  input “sbt assembly” and run

Videos tutorial:

Note:

The installation will take about 10-20 minutes

Then, you may run spark-shell, and spark has been already built if you see the following picture:

Troubleshoot:

1. java/javac/git/sbt is not recognized as an internal or external command

You didn't put all the software path into environment variables path(they are not executive in commend line )

2. [error]: not a valid commend: package/assemble

please make sure you installed the software version as I indicated before

3. [error]: java.lang.OutOfMemoryError: Java heap space

A: It is because sbt does not get enough memory allocated. Please go to your sbt/conf directory, and find the file “sbtconfig”.

Revise -Xmx512M to -Xmx2048M; -XX:MaxPermSize=256m to -XX:MaxPermSize=1024m; ReservedCodeCacheSize=512m