SparkR環境搭建

原創

legotime

2020-02-22 12:27

------------------------------------------目錄----------------------------------------------------------

R的安裝

Rstudio的安裝

SparkR啓動

Rstudio操作SparkR

------------------------------------------------------------------------------------------------------

----------------------------------------step1:安裝R------------------------------------------------------

sudo apt-get update
sudo apt-get install r-base

啓動R，界面輸入R

在界面輸入

----------------------------------------------step2:Rstudio安裝------------------------------------------------------

1、官網下載系統對應的版本：https://www.rstudio.com/products/rstudio/download/

雙擊之後-->選擇安裝

---------------------------------啓動SparkR-----------------------------------------------------------

cd 到sparkR下的bin目錄

./sparkR

--------------------------------------end---------------------------------------------------------------

-----------------------------Rstudio下操作sparkR-------------------------------------------------------------------------------

前提：啓動spark集羣

if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
  Sys.setenv(SPARK_HOME = "/usr/local/spark/spark-1.6.0-bin-hadoop2.6")
}
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sc <- sparkR.init(master = "local[*]", sparkEnvir = list(spark.driver.memory="2g"))

sqlContext <- sparkRSQL.init(sc)
df <- createDataFrame(sqlContext,faithful)

registerTempTable(df,"faith")
res <- sql(sqlContext,"SELECT eruptions,waiting From faith WHERE waiting <= 55")
head(res)

如果出現以下，you get it!!