阿龍學堂-SparkGraphx基本操作篇-第二篇

0、需求

加載外部文件進行圖計算，數據存在於該外部文件當中。

1、加載依賴包

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cn.along.com</groupId>
    <artifactId>along-root</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spark.version>2.0.2</spark.version>
        <hbase.version>1.3.1</hbase.version>
        <hadoop.version>2.7.5</hadoop.version>
        <scala.version>2.11.8</scala.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <finalName>networdcount</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>cn.itcast.streaming.WorldCount</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

2、準備數據

2.1、文件follers.txt，內容如下所示：

2 1
4 1
1 2
6 3
7 3
7 6
6 7
3 7

2.2、文件users.txt，內容如下所示：

1,BarackObama,Barack Obama
2,ladygaga,Goddes of Love
3,jereslg,John Resig
4,justinbleber,Justin Bieber
6,matel_Zahair,Matel Zaharia
7,ordersky,Matrin,Odersky
8,anonsys

3、實現代碼

def main(args: Array[String]): Unit = {
    //準備環境
    val conf: SparkConf = new SparkConf().setAppName("test2").setMaster("local[4]")
    val sc = new SparkContext(conf)
    sc.setLogLevel("WARN")
    //讀取數據
    val path = "follers.txt"
    val graph: Graph[Int, Int] = GraphLoader.edgeListFile(sc, path).partitionBy(PartitionStrategy.RandomVertexCut)
    //打印圖的數據
    graph.edges.foreach(println(_))
    graph.vertices.foreach(println(_))
    //構建圖的三角關係的網絡
    val tricount = graph.triangleCount().vertices
    val userRDD: RDD[String] = sc.textFile("users.txt")
    userRDD.foreach(println(_))
    val users = userRDD.map {
      lines =>
        val fileds = lines.split(",")
        (fileds(0).toInt, fileds(1))
    }
    users.foreach(println(_))
  }

4、其實核心代碼就兩個：

1、如何加載圖關係文件 

2、如何和現有的數據集進行生成關係

 

有問題請聯繫QQ:765120845