I have a Spark dataframe column having array values:
| data | arraydata | | ---- | --------- | text | [0,1,2,3] | | page | [0,1,4,3] |
I want to replace [0,1,2,3,4] with [negative,positive,name,sequel,odd]
Advertisement
Answer
mapping = {0: "negative", 1: "positive", 2: "name", 3: "sequel", 4: "odd"} mapping_column = map_from_entries(array(*[struct(lit(k), lit(v)) for k, v in mapping.items()])) df = df.withColumn("mapping", mapping_column) .withColumn("arraydatav2", expr(""" transform(arraydata, x -> element_at(mapping, x))""")) .drop("mapping")