Skip to content
Advertisement

Replace Spark array values with values from python dictionary

I have a Spark dataframe column having array values:

| data | arraydata |
| ---- | ---------
| text | [0,1,2,3] |
| page | [0,1,4,3] |

I want to replace [0,1,2,3,4] with [negative,positive,name,sequel,odd]

Advertisement

Answer

mapping = {0: "negative", 1: "positive", 2: "name", 3: "sequel", 4: "odd"}
mapping_column = map_from_entries(array(*[struct(lit(k), lit(v)) for k, v in mapping.items()]))

df = df.withColumn("mapping", mapping_column) 
       .withColumn("arraydatav2", expr(""" transform(arraydata, x -> element_at(mapping, x))""")) 
       .drop("mapping")
Advertisement