Skip to content
Advertisement

Python3 cgi.FieldStorage parses file name but not contents between boundary tags

I inherited a python3 project where we are trying to parse a 70 MB file with python 3.5.6 . I am using cgi.FieldStorage

File (named: paketti.ipk) I’m trying to send:

JavaScript

Headers:

JavaScript

Temporary file /tmp/nginx/0000000001:

JavaScript

Code:

JavaScript

Now the strangest things. Logs show:

JavaScript

Look at the /tmp/nginx directory:

JavaScript

So, it is like partially working because the name is got. But why it does not parse the data contents? What am I missing?

Is this even doable on python or should I just write a C utility? The file is 70 MB and if I read it in memory, OOM-killer kills the python3 process (and rightfully so, I’d say). But yeah, where does the data contents go?

Advertisement

Answer

There were more issues at play than I first thought.

First, /tmp was coming from tmpfs having maximum size of 120MB.

Secondly, my nginx.conf was problematic. I needed to comment out stuff like this to clean it up:

JavaScript

Then I needed to add these

JavaScript

After this the code

JavaScript

started to “work”. I’m monitoring /tmp usage and it uses first 70MB and then full 120 MB. The uploaded file is truncated to 50 MB.

So, when I am reading and writing parsed cgi.FieldStorage even in a loop of 4096 characters, the system reads it automatically FULLY to somewhere in /tmp once and then tries to write the final file and encounters “No space left on device” error.

To fix this I keep the nginx.conf additions and just read the self.rfile manually myself in a loop, totally reading [‘Content-Length’] (anything other makes it go bonkers). This is able to save it cleanly with one pass; there is no more than single time 70MB usage of /tmp .

Advertisement