Skip to content

Tag: pyspark

Get the most consecutive day from Date column with PySpark

Original dataframe: member_id AccessDate 111111 2020-02-03 111111 2022-03-05 222222 2015-03-04 333333 2021-11-23 333333 2021-11-24 333333 2021-11-25 333333 2022-10-11 333333 2022-10-12 333333 2022-10-13 333333 2022-07-07 444444 2019-01-21 444444 2019-04-21 444444 2019-04-22 444444 2019-04-23 444444 2019-04-24 444444 2019-05-05 444444 2019-05-06 444444 2019-05-07 Result dataframe: member_id Most_Consecutive_AccessDate total 111111 2022-03-05 1 222222 2015-03-04 1 333333 2022-10-11, 2022-10-12, 2022-10-13 3 444444 2019-04-21, 2019-04-22, 2019-04-23,

Extract first fields from struct columns into a dictionary

I need to create a dictionary from Spark dataframe’s schema of type pyspark.sql.types.StructType. The code needs to go through entire StructType, find only those StructField elements which are of type StructType and, when extracting into dictionary, use the name of parent StructField as key while value would be name of only the first nested/child StructField. Example schema (StructType): Desired result:

PySpark: Performing One-Hot-Encoding

I need to perform classification task on a dataset which consists categorical variables. I performed the one-hot encoding on that data. But I am confused that whether I am doing it right way or not. Step 1: Lets say, for example, this is a dataset: Step 2: After performing one-hot encoding it gives this data: Step 3: Here the fourth

Replicate a function from pandas into pyspark

I am trying to execute the same function on a spark dataframe rather than pandas. Answer A direct translation would require you to do multiple collect for each column calculation. I suggest you do all calculations for columns in the dataframe as a single row and then collect that row. Here’s an example. Calculate percentage of whitespace values and number