Currently I have a spark job that reads the file, creates a dataframe, does some transformations and then move those records in “year/month/date” format. I am achieving this by:
df.write.option("delimiter", "t").option("header", False).mode( "append" ).partitionBy("year", "month", "day").option("compression", "gzip").csv( config["destination"] )
I want to achieve the same by pythonic way. So, in the end it should look like:
data/2022/04/14 data/2022/04/15
Advertisement
Answer
Based on your question , instead of using partitionBy
you can also modify your config['destination']
, as s3 will take care of the necessary folder creations underneath the s3 path
s3_dump_path = config["destination"] ### 's3:/test-path/' >>> curr_date = datetime.now().date() >>> year,month,day = curr_date.strftime('%Y'),curr_date.strftime('%m'),curr_date.strftime('%d') >>> s3_new_path = '/'.join([s3_dump_path,year,month,day]) >>> s3_new_path 's3:/test-path//2022/04/14' >>> config["destination"] = s3_new_path df.write.option("delimiter", "t").option("header", False).mode( "append" ).option("compression", "gzip").csv( config["destination"] )