OpenCV Background Subtraction and Music in Background of Video
A lot of us like to travel, however due to the lockdown, we were not able to travel to places we wished, in this post, i will explain a method in which you can travel to places virtually 😉
In this post one can understand how to perform background subtraction on a video. Apart from this we will go a step ahead and add this video without background into another video of your choice. We also describe a method to add music to your file.
The inputs needed are as follows:
1- A video file (*.mp4) [This is the video to which the original webcam video stream without background is added]
2- An audio file (*.mp3) [This is the music you wish to add to your new video]
Background Subtraction: it is a common and widely used technique for generating a foreground mask (namely, a binary image containing the pixels belonging to moving objects in the scene) by using static cameras. [1]
Background Subtraction calculates the foreground mask performing a subtraction between the current frame and a background model, containing the static part of the scene or, more in general, everything that can be considered as background given the characteristics of the observed scene. [1]
Having understood the concept of background subtraction, I will now explain how the code works.
I have posted the snippets of the code, because I believe that one can concentrate better if the mind, hands, eyes are working towards the same goal.
At Lines 2–3: we import the libraries required:
pygame: It is a library that has modules designed for writing video games, but in our case we will use it to play music in the background. [2]
cv2: It is the OpenCV library that has modules to perform image processing, in our case it is used to perform background subtraction. [3]
The function above is created to match the Frame size of the Video Stream from the laptop webcam, with the Frame size of the Video Provided.
In the code above:
line 4: Opens the video stream.
line 5: Opens the Background Video File, (In my case, I chose a beach video from youtube: https://www.youtube.com/watch?v=NABKLowMMcY
line 6: Captures the first Frame from the webcam, (This is my reference Frame, please ensure that the first Frame is empty, and does not have any object[in our case it is the person whose background must be cropped])
In code above we initialize, load and start playing the music, (In my case I chose the sound of the Sea, as my background) https://www.freesoundslibrary.com/sea-sound-effect/
In the code above:
line 24: Captures the current Frame.
lines 26–30: Performs Background Subtraction.
line 32: Captures the Background video Frame.
line 34: Matches the Frame size of Video Stream from webcam, with the background video.
line 36: Evaluates the region where an object is present in video stream, and based on these values, it crops the object.
lines 38–44: Crops the object and replaces the pixels in the background video, with the pixel values from the video stream where the object is present.
lines 46–47: Displays the 2 Video files, one with frames from video stream, another with frames of object, with background from selected video.
lines 49–52: When ‘q’ key is pressed from keyboard, the frames stop being displayed, the audio is stopped.
line 54: Destroys all opened windows (video stream, and background removal windows).
line 55: Stops the video capture (Closes webcam).
The Output.
Conclusion:
1 — Currently the video is slow, this is because of the ‘For’ loop used to change the pixel values of Background video with the object in the webcam video stream, if we can eliminate ‘For’ loop, our video can play faster.
2 — One may observe that not all the pixel values of the object are added to the background video. Currently we are using “createBackgroundSubtractorMOG2” for detecting the object, if we can find a more efficient algorithm or the right value for “varThreshold”, the efficiency of the algorithm can be improved.
References:[1]https://docs.opencv.org/3.4/d1/dc5/tutorial_background_subtraction.html
[2] https://www.pygame.org/wiki/about
[3] https://pypi.org/project/opencv-python/
