neo4j 连接 spark

最新推荐文章于 2025-01-21 09:43:33 发布

原创最新推荐文章于 2025-01-21 09:43:33 发布 · 2.5k 阅读

4 ·

本内容遵循CC 4.0 BY-SA版权协议

neo4j 同时被 2 个专栏收录

4 篇文章

订阅专栏

图数据库

3 篇文章

订阅专栏

本文详细介绍了如何使用Neo4j-spark-connector在Neo4j 3.4和Spark 1.6.0之间建立连接。通过配置SparkConf并设置必要的连接参数，如URL、用户名和密码，演示了如何从Neo4j中读取数据并将其转换为Spark DataFrame，以及如何使用Cypher查询来创建Neo4j Row RDD。

今天使用neo4j连接spark
neo4j版本3.4
spark版本1.6.0
(1) 首先，需要添加jar包 neo4j-spark-connector_2.10-1.0.0-RC1.jar或者添加maven依赖

org.neo4j.spark
neo4j-spark-connector_2.10
1.0.0-RC1

(2) 设置spark连接信息
val conf : SparkConf = new SparkConf().setAppName(“InitSpark”).setMaster(“local[*]”) conf.set(“spark.neo4j.bolt.url”,”bolt://localhost:7687”)
conf.set(“spark.neo4j.bolt.user”,”neo4j”)
conf.set(“spark.neo4j.bolt.password”,”123456”)

简单例子

object Neo4jDataFrameTest { 
 def main(args: Array[String]): Unit = {      
val sQLContext = InitSpark.getSqlContext   
val query =" MATCH p=(m:ITEM)<-[r:rel*]-(n:ITEM) where m.item = '84010420' and  ALL(c in r where c.exdt>'2018-12-15')" +  " with  m.item as mark,length(r) as len, last(r).number as num,reduce(s=1.0 ,x in rels(p)| s*tofloat(x.number)) as nums, last(r).mitem as mitem , last(r) as bom  " +  " return  mark,bom.erpid as erpid, bom.virtual as virtual,mitem,bom.item as item,bom.pono as pono," +  " bom.comment as comment, bom.warehouse as warehouse, bom.exdt as exdt, bom.indt as indt,tofloat(num) as num,nums,len "   
val df = Neo4jDataFrame.withDataType(sQLContext,query,Seq.empty,"mark" ->    StringType,"erpid" ->StringType,"virtual"->LongType,  "mitem"->StringType,"item"->StringType,"pono"->LongType,"comment"->StringType,"warehouse"->StringType,"exdt"->StringType,  "indt" ->StringType,"num"->DoubleType,"nums"-> DoubleType ,"len"->LongType)    
df.show(1000)  }}

object Neo4jConnectSparkGraph {
  def main(args: Array[String]): Unit = {
     val sc = InitSpark.getSC
     val sQLContext = InitSpark.getSqlContext
     val cypher = "match (n:ITEM) return n.item limit 10"
     val neo = Neo4jRowRDD(sc,cypher)
   }
 }