os.getcwd() raise Exception with django dev server

I have a django project running inside docker, and the service is up with the command of python manage.py runserver, with file autoreload open, and using threadings.

My code invokes shutil.make_archive() which will then invoke os.getcwd(), and from time to time, os.getcwd() will raise FileNotFoundError, by searching more information, now I realized that the error might be caused by the path is no more there because the path is deleted by somewhere else after I enter the path.

The error only raised sometimes and I couldn’t find a stable way to reproduce it, once it happens, I can fix the problem by making any code changes to trigger file autoreload or restart the docker service, looks like make sure the process restarts will do the help, otherwise the error keeps raising.

When things going properly, os.getcwd() will return my django project root dir path. And I’m 100% sure my code does nothing related with dir manipulation like os.chdir().

So in a nutshell, the django server works fine, suddenly os.getcwd() starts to raise error until I restart the process.

My question is, what could be the root cause? is it possible that python manage.py runserver might cause any issue?

Python: v3.8.6
Django: v3.2
Docker: v20.10.12

Answer

Turns out that it has nothing to do with dev server or docker. It’s because shutil.make_archive is not thread safe.

What shutil.make_archive does is:

1. call os.getcwd() to save current dir path
2. os.chdir to whatever dir it needs
3. do the archiving process
4. os.chdir back to the dir path saved in first step (similar with doing pushd .; popd;)

I’m calling shutil.make_archive from multiple threads simultaneously, and also rmdir those temp dirs, and since os.chdir takes effect process-wise, so things happen in the order of:

1. os.chdir(dir_path)
2. os.rmdir(dir_path)
3. os.getcwd()

and in the final step FileNotFoundError raised thanks to the race condition.

For those who want to dig more about shutil.make_archive‘s thread-safety, there is a thread discussed about it.