install Spark on Windows 10
Go to http://spark.apache.org/docs/latest/ and findout in the downloading section, which java version you need to install on your PC.
Go to java.com, download and install that version of java.
go to http://spark.apache.org/downloads.html
select pre-built for hadoop 2.6
click spark-1.6.1-bin-hadoop2.6.tgz will take you to another page
There, click http://ftp.wayne.edu/apache/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz
unzip it in a folder and name it spark (optional).
Go to dos prompt and navigate to bin directory inside that folder.
type spark-shell and ENTER, you should get scala prompt. if so, its ok.
Go to https://www.continuum.io/downloads and download appropriate version of anaconda and install it.
Go to DOS prompt and run:
conda update conda
conda update python
From https://ysinjab.com/2015/03/28/hello-spark/
Go to http://spark.apache.org/docs/latest/ and findout in the downloading section, which java version you need to install on your PC.
Go to java.com, download and install that version of java.
go to http://spark.apache.org/downloads.html
select pre-built for hadoop 2.6
click spark-1.6.1-bin-hadoop2.6.tgz will take you to another page
There, click http://ftp.wayne.edu/apache/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz
unzip it in a folder and name it spark (optional).
Go to dos prompt and navigate to bin directory inside that folder.
type spark-shell and ENTER, you should get scala prompt. if so, its ok.
Go to https://www.continuum.io/downloads and download appropriate version of anaconda and install it.
Go to DOS prompt and run:
conda update conda
conda update python
From https://ysinjab.com/2015/03/28/hello-spark/
How to use IPython & IPython notebook with Apache Spark:
- IPython:
- Spark version <= 1.1.0 : set environment variable IPYTHON = 1
- Spark version > 1.1.0 : set environment variable PYSPARK_DRIVER_PYTHON = ipython
- IPython notebook:
- Spark version <= 1.1.0 : set environment variables IPYTHON = 1 and IPYTHON_OPTS = notebook
- Spark version > 1.1.0 : set environment variables PYSPARK_DRIVER_PYTHON = ipython and PYSPARK_DRIVER_PYTHON_OPTS = notebook
Go to spark/bin folder from command prompt and run pyspark. ipython notebook should fire up. issue the command print sc. you should get a valid object.
