This all started with the question of, “can I extract just the keyframes from a piece of video?”
It’s relatively easy with standard mpeg based digital video but, becomes problematic with digital surveillance video with a non standard GOP Structure.
So, after finding a possible solution I thought I had better run a test.
I’m starting off in Windows 7. Using my Virtualdub pack I have created a test video. This is really easy to do by selecting Tools > Test Video and I have used RGB Cube.
For the purposes of this test I have then changed the frame rate to 30 FPS exactly. I have then trimmed the video to 30 seconds.
Having now got a 900 frame video I have placed a timecode overlay using a subtitle creator.
The next job is to output an mpeg video. Using the ffdshow codec plugin included in the Vdub pack, I have selected Compression > ffdshow, and then configured the codec to use mpeg2 at high quality. Its important to remember that when you then select File > Save as avi, you need to select the dropdown for file types and select ‘all files’. You will then be able to place the desired file extension after your filename. In this case .mpg.
After creating my test file I have opened it again in Vdub using the mpeg2 file handler.
As can be seen from the following two images, the frame types are being correctly identified within the Group Of Pictures (GOP).
Its now over to the Linux virtual desktop. For virtualisation, there are a number of free options. Virtualbox is open source but does not support drag and drop between your host PC and the virtual. You can, however, set up a shared folder between the two. Virtualbox is also good at creating snapshots of a virtual system. This is handy if you are configuring a linux system for a specific purpose and happen to corrupt the system files. The other one for Windows hosts is VMware player. This is the free version of VMware’s various packages and does support drag and drop between host and virtual.
For todays task I have used VMware player and a Virtual PC running Linux Mint 12. I have moved to Mint from Ubuntu as I prefer the build. Its now up to version 13, with a number of interface options. There are so many Linux distributions (distros), and each works slightly differently. It helps if your chosen distro has a good user community as it’s here that you will find the help needed to complete certain tasks.
We now need to work with our video – my test video is only small so its been dragged and dropped into my Mint desktop.
The video tool we are going to use is FFmpeg. This little utility is immensely powerful and used as the backend to many players and encoding programs in Windows, OSX and Linux. These programs, with Graphical User Interfaces, are great for the quick and easy tasks but you can get stuck if you want to do something a little more taxing! ffmpeg has a very good userbase from all aspects of video and broadcast and its worth doing a little research to learn a bit more and see what can be done.
As can be seen from my package manager, ffmpeg is not installed.
If you install from the recommended repositories, then you will only get a standard ffmpeg build. This is fine for the majority of standard tasks but there is a lot more it can do if you build it yourself. This can be a bit daunting but there are very good step-by-step instructions online. I used the one here…
You can then check its all ok but opening a terminal window and just typing ffmpeg
OK, what do we need to do? The task was to identify and extract all the I frames from a video. I know that it can be done in Virtualdub using the decimate option but this is only suitable when the GOP structure is routine.
The good thing with scripting in ffmpeg is that its relatively easy to understand once you know the basics. There are loads of tutorials online but in very simplistic terms you tell the computer to use ffmpeg. You tell ffmpeg what file to look at. You tell ffmpeg what to do with the file and you finally tell it what to output.
ffmpeg /home/spreadys/Desktop/video.264 -f avi -sameq -vcodec copy /home/spreadys/Desktop/videoout.avi
The above line, when entered in terminal would tell ffmpeg to use the file on my desktop named as video.264. Note that I have used the full file path.
-f avi means force avi as the output file type.
-sameq means retain the same quality as original.
-vcodec copy means use the same codec as original – no transcoding.
Lastly I have placed the file path and name for my output file.
This little script would read the raw 264 stream and wrap it inside an avi container. Very handy!
Anyway, back to the task at hand!
Remember what I was saying about the ffmpeg userbase being so large and across a huge variety of video professionals. I found this script whilst searching the web and have then tweaked it very slightly for our purpose.
ffmpeg -vf select=”eq(pict_type\,PICT_TYPE_I)” -i /home/david/Desktop/Origmpg2.mpg -vsync 2 -qscale 1 -f image2 /home/david/Desktop/Iframes/Iframes-%02d.tif -loglevel debug 2>&1 | grep “pict_type:I -> select:1” > /home/david/Desktop/keyframe-timecodes.txt
Seconds later, after running this in terminal, my Iframes folder fills with Images. Using the math, we can calculate how many we should have.
900 Frames. A 12 Image GOP. 900 divided by 12 = 75.
There are 75 Images in my Iframes folder!
I also now have a txt file on my desktop. This gives me the timecodes of all I frames and their corresponding frame number.
Part of the output shown below…
n:0 pts:33333 t:0.033333 pos:8212 interlace_type:P key:0 pict_type:I
n:12 pts:433333 t:0.433333 pos:106592 interlace_type:P key:0 pict_type:I
n:24 pts:833333 t:0.833333 pos:214442 interlace_type:P key:0 pict_type:I
n:36 pts:1233333 t:1.233333 pos:329014 interlace_type:P key:0 pict_type:I
n:48 pts:1633333 t:1.633333 pos:428926 interlace_type:P key:0 pict_type:I
n:60 pts:2033333 t:2.033333 pos:520700 interlace_type:P key:0 pict_type:I
n:72 pts:2433333 t:2.433333 pos:612634 interlace_type:P key:0 pict_type:I
n:84 pts:2833333 t:2.833333 pos:701742 interlace_type:P key:0 pict_type:I
This method for extracting I frames rom a piece of video is now a work in progress! I need to do a bit more work on the script and the output txt file. I also need to figure out why the tifs are fine within Linux but have ‘issues’ in Windows and OSX.
If anybody wants to pick this up and run with it….and perhaps write a Windows GUI, then feel free!
Feb 2012 Update:
I’m still working when I can on this one! Struggling in windows with the latests builds of ffmpeg and also avconv. If anybody knows of how to do it, them please let me know!
May 2013 Update:
Cracked it – it was all down to telling ffmpeg to output the tiffs with the correct pixel format. There is a full write up including the new way of extracting the I frames only in this entry: https://spreadys.wordpress.com/2013/04/28/update-on-i-frame-only-extraction/