We present a basic technique to fill-in missing parts of a video sequence taken from a static camera. Two important cases are considered. The first case is concerned with the removal of non-stationary objects that occlude stationary background. We use a priority based spatio-temporal synthesis scheme for inpainting the stationary background. The second and more difficult case involves filling-in moving objects when they are partially occluded. For this, we propose a priority scheme to first inpaint the occluded moving objects and then fill-in the remaining area with stationary background using the method proposed for the first case. We use as input an optical-flow based mask, which tells if an undamaged pixel is moving or is stationary. The moving object is inpainted by copying patches from undamaged frames, and this copying is independent of the background of the moving object in either frame. This work has applications in a variety of different areas, including video special effects and restoration and enhancement of damaged videos. The examples shown in the paper illustrate these ideas.
Paper [PDF] ( To appear in the proceedings of IEEE-ICIP 2005 )
Latest - "Video Inpainting Under Camera Motion" (under review)
The result below illustrates the first case as mentioned above. The person with the orange jacket, the phone-box in the center and the lamp-post towards the left of center, are filled in with the stationary background. The inpainted sequence is illustrated on the right. Notice that there are no inconsistencies from one frame to another in the areas that are filled-in.
The more important and difficult to handle problem is, filling-in an occluded moving person. To address this, we assume that we are given a "motion confidence image" Mc, which indicates if a pixel belongs to moving foreground or stationary background. The following synthetic example shows that given a perfect Mc, we can perfectly inpaint the moving object.
The implementation of our algorithm on a real-life video sequence (50 frames) is shown below. We use a crude optical-flow based "motion confidence image" Mc. The sequence on the right shows completed foreground, and the one below shows the completely inpainted sequence. Observe that the motion of the completed person and the background filled-in, is globally consistent.
The sequence below illustrates the application of our technique to removing large moving objects from videos. Notice how well the moving person is synthesized in the region of object removal. The orange basket-ball visible in the completely inpainted sequence is seen only for a few frames in the original video (60 frames). Our background completion ensures that all available temporal information is preserved. ( Frame size has been reduced from 640x480 to 320x240, for faster viewing )
Note: During background completion in a damaged frame at a location (say) p, we find a matching patch of background and not only copy it to Ψp (refer to paper for explanation) but also to all damaged frames that have a data loss at that particular location, thus achieving consistency. When we are faced with the problem of inpainting a moving person (as above), it is therefore important that we FIRST complete the moving person and THEN complete the static background, so that we do not synthesize background at a location where otherwise we would want part of the completed moving object to be.
Last updated on : Sunday, July 31, 2005