Python socket receive – incoming packets always have a different size

I’m using the SocketServer module for a TCP server. I’m experiencing some issue here with the recv() function, because the incoming packets always have a different size, so if I specify recv(1024) (I tried with a bigger value, and smaller), it gets stuck after 2 or 3 requests because the packet length will be smaller (I think), and then the server gets stuck until a timeout.

class Test(SocketServer.BaseRequestHandler):

def handle(self):

   print "From:", self.client_address

   while True:    

     data = self.request.recv(1024)
     if not data: break

     if data[4] == "x20":              
       self.request.sendall("hello")
     if data[4] == "x21":
       self.request.sendall("bye")
     else:
       print "unknow packet"
   self.request.close()
   print "Disconnected", self.client_address

launch = SocketServer.ThreadingTCPServer(('', int(sys.argv[1])),Test)

launch.allow_reuse_address= True;

launch.serve_forever()

If the client sends multiples requests over the same source port, but the server gets stuck, any help would be very appreciated, thank !

Answer

Note: As people have pointed out in the comments, calling recv() with no parameters is not allowed in Python, and so this answer should be disregarded.

Original answer:

---

The network is always unpredictable. TCP makes a lot of this random behavior go away for you. One wonderful thing TCP does: it guarantees that the bytes will arrive in the same order. But! It does not guarantee that they will arrive chopped up in the same way. You simply cannot assume that every send() from one end of the connection will result in exactly one recv() on the far end with exactly the same number of bytes.

When you say socket.recv(x), you’re saying ‘don’t return until you’ve read x bytes from the socket’. This is called “blocking I/O”: you will block (wait) until your request has been filled. If every message in your protocol was exactly 1024 bytes, calling socket.recv(1024) would work great. But it sounds like that’s not true. If your messages are a fixed number of bytes, just pass that number in to socket.recv() and you’re done.

But what if your messages can be of different lengths? The first thing you need to do: stop calling socket.recv() with an explicit number. Changing this:

data = self.request.recv(1024)

to this:

data = self.request.recv()

means recv() will always return whenever it gets new data.

But now you have a new problem: how do you know when the sender has sent you a complete message? The answer is: you don’t. You’re going to have to make the length of the message an explicit part of your protocol. Here’s the best way: prefix every message with a length, either as a fixed-size integer (converted to network byte order using socket.ntohs() or socket.ntohl() please!) or as a string followed by some delimiter (like ‘123:’). This second approach often less efficient, but it’s easier in Python.

Once you’ve added that to your protocol, you need to change your code to handle recv() returning arbitrary amounts of data at any time. Here’s an example of how to do this. I tried writing it as pseudo-code, or with comments to tell you what to do, but it wasn’t very clear. So I’ve written it explicitly using the length prefix as a string of digits terminated by a colon. Here you go:

length = None
buffer = ""
while True:
  data += self.request.recv()
  if not data:
    break
  buffer += data
  while True:
    if length is None:
      if ':' not in buffer:
        break
      # remove the length bytes from the front of buffer
      # leave any remaining bytes in the buffer!
      length_str, ignored, buffer = buffer.partition(':')
      length = int(length_str)

    if len(buffer) < length:
      break
    # split off the full message from the remaining bytes
    # leave any remaining bytes in the buffer!
    message = buffer[:length]
    buffer = buffer[length:]
    length = None
    # PROCESS MESSAGE HERE