Convert Pyspark dataframe to dictionary

                    Convert Pyspark dataframe to dictionary

input:

Col0, Col1
-----------
A153534,BDBM40705
R440060,BDBM31728
P440245,BDBM50445050

output:

{'A153534': 'BDBM40705'}, {'R440060': 'BDBM31728'}, {'P440245': 'BDBM50445050'}

Method1:

df.rdd.map(lambda x: {x.Col0: x.Col1}).collect()

[{'A153534': 'BDBM40705'}, {'R440060': 'BDBM31728'}, {'P440245': 'BDBM50445050'}]

Method2:

df = spark.read.csv('/FileStore/tables/Create_dict.txt',header=True)

df = df.withColumn('dict',to_json(create_map(df.Col0,df.Col1)))

df_list = [row['dict'] for row in df.select('dict').collect()]

print df_list

['{"A153534":"BDBM40705"}',
 '{"R440060":"BDBM31728"}',
 '{"P440245":"BDBM50445050"}']