There is a huge array of digital video formats used by the surveillance industry and when taken together, in real life situations, it can all get a little confusing. By having an awareness of these common issues, manufacturers can increase the usefulness of their video.
To break things down I have started to categorise the formats rather than try to deal with each one individually.
A Standard Format – an example being an Mpeg4 with all ISO Parts retained.
A Modified Format – Such as a motion jpeg based video that does not conform to documented standards.
Just to mix things up a bit, we then have containers. These too can be either standard (.AVI) or modified (.DVR).
The one thing that I am not going to do here is any format bashing. There is enough of that already on many security forums and manufacturer’s sites. My EEPIP blogs are to assist those in the industry in Evidential Export, Playback, Investigation and Presentation. If you have read any of my previous entries on how to deal with certain formats, it should be clear that if a security manufacturer doesn’t take these processes into account, their format of choice can actually hinder an investigation. I am also not suggesting that all modified formats are bad.
What’s the difference between a standard format and a modified one?
Standard formats retain the structure and build as documented within video industry standardisation protocols, such as ISO or ITU. The benefit here is that they are transparent. The video can be analysed and understood at great detail. The results of any analysis can be verified against other software and explanations to anomalies can be fully investigated. Standardisation also allows for much more flexibility in the use of that piece of video.
Modified formats allow a manufacturer to step outside the standard and change the video encoding for their own use. The main problem when a format is modified is that documentation is usually very hard to acquire or, more often, secretive. This then makes the footage hard to examine and verify its validity.
Let’s look further at this using a standard Motion JPEG stream.
The file is made up with a series of jpeg images, one after the other. With each image is metadata relating to the date and system time it was taken. The CCTV player for this format would be able read the metadata and then display it within the interface. If I were to use a standard file recovery tool to identify and extract the images, each jpeg would be able to be identified, viewed and investigated. The images are independently verifiable.
Now compare this to two Modified Motion JPEG streams.
The first one utilises the player to hold all the information on the jpeg format, including resolution and decoding instructions. The benefits of this are that the storage overhead on the DVR is reduced as it only needs to retain the basic information. The difficulty during investigation is that although the player presents an image, how can we verify that it was the one recorded? The most common abuse of this method is rescaling. Many record at field level and store an image of 704 x 288, (we are in PAL land here!) The player then presents and exports this at 704 x 576. It is usually enough to just be aware of this up-scaling, but in other cases a more detailed understanding is required on the method used. Duplication, interpolation….? It’s important to know. I have also seen recording at 352 x 288 and then having an image playback and export from the player at 704 x 576!
The next type of system utilises a form of motion compensated compression by only retaining the parts of an image that have changed from a reference image. The encoder captures an image and this is retained. The encoder captures the next image and it is analysed to detect what parts are the same and what parts are different. If it detects changes, only those parts are retained. This goes on until a set number of images have been reached and then a new reference image is recorded. The pattern is then repeated. The player designed for that format presents the whole image on the screen. Parts of that image come from the reference image and the the moving / changing parts then come from that new image. To an owner, this has huge storage and bandwidth benefits. To an investigator, how can we verify the accuracy of the system to determine change or location of an object?
In both of these ‘modified’ examples, it’s also commonly difficult to deal with the video for its final purpose. In the first one, I would not be able to extract all the images because the data to decode them is not there – it’s in the manufacturer’s player. In the second, I may be able to extract the images but only one out of x (the number of images before the next reference), would be a full image.
And it doesn’t stay with JPEG. This modification has carried on through to Mpeg4 and H264. A few more examples:
A manufacturer retains standards and records in Mpeg4. Those camera streams are all recorded onto the same drive. When evidentially exported, that stream is copied as it was recorded, bit for bit. The player for that format reads this and places each camera stream into the appropriate window within its interface. By using standards, it is very easy for a trained investigator to take this file, analyse it and extract all the individual camera streams in their original form.
Compare this to a modified file. It may be understandable as an mpeg type of video but it cannot be interpreted or analysed out of its player, and the camera streams cannot be extracted. If the player designed by the manufacturer does not allow for these issues then how can the footage be correctly assessed and validated?
In the h264 world there are just too many examples, but a new twist is a container called v264. This is basically a modified h264 that appears to be causing one or two problems. Tenvis has compiled a v264 to h264 converter but in the conversion process everything changes – including the resolution. Disaster!
Standards also cover metadata. Date and time information should be transparent and verifiable. With this standardisation comes the flexibility of using that time information in any way necessary during the investigation.
What I used to call a proprietary format now just sits under the modified umbrella! These are formats specifically designed but still have links back towards a standard. Examples of this are Wavelet and, the much newer, MxPeg.
MxPeg is Jpeg based but, although modified, it has been included in many open standard based players such as VLC. This does therefore, make it easy to manage.
Although the video is a sales pitch from MxPeg, I am sure some of the issues raised regarding H264 will ring true with anyone working with evidential video over the past few years. The size of GOP in some some situations never cease to amaze me. A recording of 5 Images per second with a 200 Image GOP. That’s a new clean image every 40 seconds! When you add poor spatial compression, you can imagine the video quality.
Finally then, what about standard containers? A good example of this is Quadrox. The company detail at length their use of the standard .wmv container but then utilize their own modified jpeg encoding with the JR24 Codec. As such, if a video has been encoded with this, it must be installed in order to play the video. It’s a standard container with a modified format.
It is becoming more popular to not even bother with a container file. The raw stream(s) just sits in a file with a random file extension. If it follows a standard, then once identified, it’s pretty easy to deal with. However, if it’s modified, and the manufacturer has failed to provide a substantive piece of software, then it becomes increasingly frustrating.
I estimate that 50% of current evidential exports are of a standard format, with a further 25% being a modified format with manufacturer support to conduct certain tasks (not all!). That leaves 25% that are difficult, with little support to conduct analysis or further processing.
It is likely that modified formats with limited support will soon start to become a rarity. I believe that manufacturers are slowly beginning to understand that a transparent recording format allows for openness in their infrastructure and gives them an ability to communicate and link with other devices. The hidden benefit of following a standard is that it can be fully investigated and understood.
But why is it a hidden benefit? Manufacturers and installers should promote this fact. It should be included in their documentation that they follow a standard that can be understood and dealt with for any reason. On the other hand, if they continue to utilise some form of modification, they can document how this benefits the user. They can then promote the fact that they work together with end users of their format to ensure interoperability throughout.
Sorry, what was that you said……..You are a manufacturer but don’t work with end users of your format?
Don’t worry – you are not alone…… yet!