Situation : I have a basler camera connected to a raspberry pi, and I am trying to livestream it’s feed with FFmpg to a tcp port in my windows PC in order to monitor whats happening in front of the camera.
Things that work : I manage to set up a python script on the raspberry pi which is responsible for recording the frames, feed them to a pipe and streaming them to a tcp port. From that port, I am able to display the stream using FFplay.
My problem : FFplay is great for testing out quickly and easily if the direction you are heading is correct, but I want to “read” every frame from the stream, do some processing and then displaying the stream with opencv. That, I am not able to do yet.
Minimaly reprsented, that’s the code I use on the raspberry pi side of things :
command = ['ffmpeg', '-y', '-i', '-', '-an', '-c:v', 'mpeg4', '-r', '50', '-f', 'rtsp', '-rtsp_transport', 'tcp','rtsp://192.168.1.xxxx:5555/live.sdp'] p = subprocess.Popen(command, stdin=subprocess.PIPE) while camera.IsGrabbing(): # send images as stream until Ctrl-C grabResult = camera.RetrieveResult(100, pylon.TimeoutHandling_ThrowException) if grabResult.GrabSucceeded(): image = grabResult.Array image = resize_compress(image) p.stdin.write(image) grabResult.Release()
On my PC if I use the following FFplay command on a terminal, it works and it displays the stream in real time :
ffplay -rtsp_flags listen rtsp://192.168.1.xxxx:5555/live.sdp?tcp
On my PC if I use the following python script, the stream begins, but it fails in the cv2.imshow
function because I am not sure how to decode it:
import subprocess import cv2 command = ['C:/ffmpeg/bin/ffmpeg.exe', '-rtsp_flags', 'listen', '-i', 'rtsp://192.168.1.xxxx:5555/live.sdp?tcp?', '-'] p1 = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE) while True: frame = p1.stdout.read() cv2.imshow('image', frame) cv2.waitKey(1)
Does anyone knows what I need to change in either of those scripts in order to get i to work?
Thank you in advance for any tips.
Advertisement
Answer
We may read the decoded frames from p1.stdout
, convert it to NumPy array, and reshape it.
Change
command
to get decoded frames inrawvideo
format and BGR pixel format:command = ['C:/ffmpeg/bin/ffmpeg.exe', '-rtsp_flags', 'listen', '-i', 'rtsp://192.168.1.xxxx:5555/live.sdp?tcp?', '-f', 'rawvideo', # Get rawvideo output format. '-pix_fmt', 'bgr24', # Set BGR pixel format 'pipe:'] # Use stdout as output
Read the raw video frame from
p1.stdout
:raw_frame = p1.stdout.read(width*height*3)
Convert the bytes read into a NumPy array, and reshape it to video frame dimensions:
frame = np.frombuffer(raw_frame, np.uint8) frame = frame.reshape((height, width, 3))
Now we can show the frame by calling cv2.imshow('image', frame)
.
The solution assumes, we know the video frame size (width
and height
) from advance.
The code sample below, includes a part that reads width
and height
using cv2.VideoCapture
, but I am not sure if it’s going to work in your case (due to '-rtsp_flags', 'listen'
. (If it does work, you can try capturing using OpenCV instead of FFmpeg).
The following code is a complete “working sample” that uses public RTSP Stream for testing:
import cv2 import numpy as np import subprocess # Use public RTSP Stream for testing in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4' if False: # Read video width, height and framerate using OpenCV (use it if you don't know the size of the video frames). # Use public RTSP Streaming for testing: cap = cv2.VideoCapture(in_stream) framerate = cap.get(5) #frame rate # Get resolution of input video width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) # Release VideoCapture - it was used just for getting video resolution cap.release() else: # Set the size here, if video frame size is known width = 240 height = 160 command = ['C:/ffmpeg/bin/ffmpeg.exe', # Using absolute path for example (in Linux replacing 'C:/ffmpeg/bin/ffmpeg.exe' with 'ffmpeg' supposes to work). #'-rtsp_flags', 'listen', # The "listening" feature is not working (probably because the stream is from the web) '-rtsp_transport', 'tcp', # Force TCP (for testing) '-max_delay', '30000000', # 30 seconds (sometimes needed because the stream is from the web). '-i', in_stream, '-f', 'rawvideo', # Video format is raw video '-pix_fmt', 'bgr24', # bgr24 pixel format matches OpenCV default pixels format. '-an', 'pipe:'] # Open sub-process that gets in_stream as input and uses stdout as an output PIPE. ffmpeg_process = subprocess.Popen(command, stdout=subprocess.PIPE) while True: # Read width*height*3 bytes from stdout (1 frame) raw_frame = ffmpeg_process.stdout.read(width*height*3) if len(raw_frame) != (width*height*3): print('Error reading frame!!!') # Break the loop in case of an error (too few bytes were read). break # Convert the bytes read into a NumPy array, and reshape it to video frame dimensions frame = np.frombuffer(raw_frame, np.uint8).reshape((height, width, 3)) # Show the video frame cv2.imshow('image', frame) if cv2.waitKey(1) & 0xFF == ord('q'): break ffmpeg_process.stdout.close() # Closing stdout terminates FFmpeg sub-process. ffmpeg_process.wait() # Wait for FFmpeg sub-process to finish cv2.destroyAllWindows()
Update:
Reading width and height using FFprobe:
When we don’t know the video resolution from advance, we may use FFprobe
for getting the information.
Here is a code sample for reading width
and height
using FFprobe
:
import subprocess import json # Use public RTSP Stream for testing in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4' probe_command = ['C:/ffmpeg/bin/ffprobe.exe', '-loglevel', 'error', '-rtsp_transport', 'tcp', # Force TCP (for testing)] '-select_streams', 'v:0', # Select only video stream 0. '-show_entries', 'stream=width,height', # Select only width and height entries '-of', 'json', # Get output in JSON format in_stream] # Read video width, height using FFprobe: p0 = subprocess.Popen(probe_command, stdout=subprocess.PIPE) probe_str = p0.communicate()[0] # Reading content of p0.stdout (output of FFprobe) as string p0.wait() probe_dct = json.loads(probe_str) # Convert string from JSON format to dictonary. # Get width and height from the dictonary width = probe_dct['streams'][0]['width'] height = probe_dct['streams'][0]['height']