python code on Heroku errors where it works locally

I am getting too many errors on Heroku, so any print statements are not showing up in logs. This code works fine locally (python on windows 10) but wont work on Heroku, where it tells me there is nothing to concat…

path = r'.storetx' # use path for csv files
all_files = glob.glob(os.path.join(path , "tastyworks_tx*.csv")) # 

li = []
print(all_files)    

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    print('filename: ', filename)
    input('pause here..')
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)   #append all the files to a single dataframe called "frame"

I have checked the storetx folder on Heroku and the tastyworks_tx*.csv files are present and I have other functions that can access files inside of the store folder but they use forward slanting (e.g. hist_data = pd.read_csv('./store/hist5m.csv', index_col=0, header=[0, 1]) works fine ).

On Heroku I get the following error.

2022-08-18T05:35:28.470644+00:00 app[web.1]: File “/app/./main.py”, line 291, in tastytx 2022-08-18T05:35:28.470644+00:00 app[web.1]: frame = pd.concat(li, axis=0, ignore_index=True) #append all the files to a single dataframe called “frame”

…

raise ValueError(“No objects to concatenate”) 2022-08-18T05:35:28.470645+00:00 app[web.1]: ValueError: No objects to concatenate

my python dependencies used are same version as on the local setup, though the local setup has more installed. Here is my “requirements.txt” file for Heroku that installs fine:

pandas==1.4.2, pandas-datareader==0.10.0, DateTime==4.4, yfinance==0.1.72, fastapi==0.79.0, uvicorn==0.18.2, beautifulsoup4==4.11.1

I dont want to give Heroku my credit card info so cannot install Papertrail or better logging addons and cant figure out how to find out more of what is going wrong here due to limits on the logs.

Answer

This is likely because you are using instead of / in the path.

On Linux, which is probably the OS for your Heroku machine, directories are separated with ‘/’. Your code will be looking for a file called precisely .\store\tx\tastyworks_tx_2020.csv rather than a file called tastyworks_tx_2020.csv in .store/tx/.

Another (IMO more rubust) solution to replacing with / would be to use pathlib, which should work on all operating systems.

from pathlib import Path

the_path = Path("store") / Path("tx")
all_files = the_path.glob("tastyworks_tx*.csv") 
...

Advertisement

Answer