Issue returning data from Python to Unity

This question is an evolution of this: Issue returning header byte from Python to Unity

Now the issue about header byte is solved, but I have a problem during the loop.

The data incoming good but after “n” times the loop break because receive a json string with an entire structure of data with open/close bracket PLUS the next structure of data with open bracket, some data and in the “image:” some data and many of the “” and unless the close bracket.. And the json parser obviously breaks. Why and what can I do? Thanks Below the code revised:

IEnumerator Client()
{
   while (!IsConnected(client))
   {
      try
      {
         client = new TcpClient(host, port);                 
         s = client.GetStream();                
              
         byte[] byteBuffer = Encoding.UTF8.GetBytes("Connected to client");               
         s.Write(byteBuffer, 0, byteBuffer.Length);

         while (true)
         {
            if (s.DataAvailable)
            {            

               while (s.DataAvailable)
               { 
                  var header = new byte[4];
                  s.Read(header, 0, header.Length);
    
                  var fileSize = BitConverter.ToUInt32(header,0);

                  StringBuilder myCompleteMessage = new StringBuilder();
                  MemoryStream ms = new MemoryStream();
                       
                  int increment = 0;

                  while (ms.Length < fileSize)
                  {
                     byte[] dataReceived = new byte[fileSize];
                     increment = s.Read(dataReceived, 0, dataReceived.Length);
    
                     ms.Write(dataReceived.Take(increment).ToArray(), 0, increment);
                  }
                       
                  myCompleteMessage.AppendFormat("{0}",   Encoding.UTF8.GetString(ms.GetBuffer()));

                  JSONNode data = JSONNode.Parse(myCompleteMessage.ToString());
               
                  ....
                  // Do work with myCompleteMessage

                  yield return null;
               }
            }
         }
      }
   }
}

Answer

I don’t have your Unity environment here, so I can’t have tested this.

Something like a loop (likely run in a coroutine or something that doesn’t block Unity’s default handling) like this should be enough to handle a stream of messages of the shape we established before (a 4-byte header signifying the message length, followed by that many bytes of JSON).

while (!IsConnected(client))
{
    try
    {
        client = new TcpClient(host, port);
        s = client.GetStream();
        while (true)
        {
            var data = ReadPacket(s);
            // Do work with data
        }
    }
}

ReadPacket should be something like

JSONNode ReadPacket(Stream s)
{
    var buffer = new byte[131072];

    // Read 4 bytes (we will assume we can always read that much)
    var n = s.Read(buffer, 0, 4);
    if(n < 4) throw new Exception("short read");
    var fileSize = BitConverter.ToUInt32(buffer, 0);

    Debug.Print("Reading a blob of length", fileSize);

    // Memory stream to accumulate the reads into.    
    var ms = new MemoryStream();
    while (ms.Length < fileSize)
    {
        // Figure out how much we can read -- if we're near the end,
        // don't overread.
        var maxRead = Math.Min(buffer.Length, fileSize - ms.Length);
        var increment = s.Read(buffer, 0, maxRead);
        // ... and write them into the stream.
        ms.Write(buffer, 0, increment);
        Debug.Print("Read", ms.Length, "of", fileSize);
    }
    // Decode the bytes to UTF-8 and parse.
    return JSONNode.Parse(Encoding.UTF8.GetString(ms.GetBuffer()));
}