I am working on a data frame that looks like this : I’m trying to make a Haverisne distance matrix. Basically for each zone, I would like to calculate the distance between it and all the others in the dataframe. So there should be only 0s on the diagonal. Here is the Haversine function that I use but I can’t
Tag: haversine
Calculate the minimum distance to destinations for each origin in pyspark
I have a list of origins and destinations along with their geo coordinates. I need to calculate the minimum distance for each origin to the destinations. Below is my code: I got error like below: my question is: it seems that there is something wrong with withColumn(‘Distance’, haversine_vector(F.col(‘Origin_Geo’), F.col(‘Destination_Geo’))). I do not know why. (I’m new to pyspark..) I have