The Reliability of Image Representation

It’s been a long time since I posted here.
I am the first one to admit that I can get a little passionate about Forensic Video Analysis and CCTV Investigation, and my frustrations with various aspects of this niche field of forensics can take their toll on my wellbeing.
Sometimes you just have to take a breath, rather than bang your head harder on the brick wall.

There are though a handful of other analysts who have continued to shout about certain issues for some considerable time. Unfortunately, we often seem to get drowned out and the problems get lost and ignored within the legal system. It was a refreshing change therefore a few weeks ago, to see and hear from one of those respected colleagues on the UK’s Channel 4 news.

It is clear that the news item has probably been heavily edited, cut down and slanted to concentrate on facial analysis BUT there are some important takeaways from this and things that we, as analysts, must consider.

Firstly, let me make this clear. CCTV is not putting innocent people behind bars.
Misunderstanding, incorrect handling, poor analysis and biased presentation will put innocent people behind bars. It’s not CCTV’s fault!

Second.. the images and comparative analysis displayed in this video leave a lot to be desired and cause damage to the role of the Forensic Video Analyst…but worryingly, I have seen worse.

Professor Gillian Tully, right at the outset states, “When it comes to the interpretation of CCTV Images, its difficult for us at the moment to have the assurance that it’s done properly across the board…” She then goes onto the fact that people and organisations do not meet the required standards set by the Forensic Science Regulator (FSR).

I am not going to get into the huge mess surrounding the initial setting of the FSR’s guidance in the UK.

I am not going to get into the conflicts between the FSR’s guidance, the training delivered to most police organisations in Forensic Video Analysis, and then the guidance documents published by the National Police Chiefs Council.

I am also not going to get into the ISO accreditation of video units and private service providers and the difficulties this has caused…

We have to go all the way back to the start, and clearly define what a person should do and what is their role to the court.

I have been lucky, or unlucky in many cases, to review lots of Forensic Video and Image reports and there are some constant problems. The most common is an unclear definition of what that person’s role is within the investigation.

In any case where video evidence is to be used, whether that from CCTV, Cellphones or Body Worn Video, an analyst must ensure and report on the integrity of that video, the authenticity of that video and the reliability of the image representation.

Why is an expert in video required to do this? You all watch the TV, you all watch YouTube. The latest viral video of a dog skiing is completely different, however, to a piece of footage recorded on a CCTV system that could result in an innocent person being convicted of a crime they did not commit.

Regardless of the task or question being asked, the analyst has a duty to the court to assess and understand the evidence.

The analyst’s first responsibility is to identify any changes in the image since the time it was first created. This ensures the integrity of the exhibit. Any changes must adhere to a scientific workflow and therefore be repeatable, reproducible and explainable. Any item produced as a result of a change will differ from the original and, as such, will be a new exhibit. Any undocumented changes must be disclosed as these will affect the integrity of the produced item.

The second responsibility is to ensure authenticity. Any item must be a true and accurate representation of that which it purports to be. The video may not be authentic in its original state. The capture and encoding of the video may have changed the shape, dimensions, speed or colour, and consequently, anything in the scene would suffer from this incorrect view. Authenticity therefore not only relates to the identification of fabricated evidence, either through malcous manipulation or incompetence, but more commonly to the method of aquisition. In these cases, authenticity can often be restored through restoration and enhancement.

This is where the third responsibility comes in. Ensuring the reliability of the image representation. This is often termed the scientific stage, as processing, reconstruction, testing, restoring, enhancing, collating, annotating and presenting video evidence must adhere to a scientific workflow. If those are followed, and any required peer review completed, the produced item or presented fact should be reliable.

It is not the analyst’s role to form an opinion as to an item, or a mark. It is an analysts role to inform the judge or jury that the item or mark constructed within the image is reliable and can be trusted.

It is not the analysts role to make up terms and use incorrect, unscientific procedures.

It is not the analyst’s role to fit the narrative of those who are employing them.

Comparative Video and Image Analysis can be extremely time-consuming work…. but also very frustrating. Most CCTV used is not installed for the purpose it is being used for and, as such, identifying and documenting the limitations is key for transparency and trust.

When the analysis starts with the identification of the facts, it is easy to perform unbiased analysis and, if necessary, report that an object or views representation is unreliable.

And this workflow is not just for comparative analysis. Timelines, storyboards and the tracking of persons through scenes can all be presented incorrectly with unreliable visual information if the original data is not understood and analyzed correctly first. The staged disclosure of video exhibits is also of key importance to reduce unconcious bias.

An analyst is there to identify the facts in the video or image. There should be very little subjective interpretation. The facts are presented after objective analysis.
If an analyst analyzes a video and is able to confirm though reconstruction and restoration that an object or person has a number of atributes then those atributes are facts.

However, an analysts most important skill is the ability to identify features that are not reliable or that contrast, rather than compare with a known object or person.

I am sure that Reuben could have said a lot more (I imagine there was a lot edited), but the report has highlighted the scary fact that there are people in the UK and around the world presenting this sort of stuff and they are full of unreliable images caused by a lack of understanding and questionable terms to baffle a jury.

The more times that things like this get shouted about, the better it will be for Forensic Video Analysis and the legal system.

Advertisement

App Reviews – Clock Check and iCrimeFighter

These two iOS apps fit nicely together so I thought that joining the reviews would help more. (If you are an Android user and know of similar apps then please let me know or, if you have done a review, send me the link and I will include it here).

The first one, Clock Check, is “an application designed to aid CCTV engineers and Law Enforcement personnel when calculating the time discrepancies on CCTV systems.” Firstly, it does what it says it does, and with no fuss. It also has a few little extras that make it a little more than a basic time calculator.

It has the ability to sync with your devices time. This is great if you have your iDevice set to sync time automatically. The problem though is that the time shown within the app doesn’t move. Its static. I think a great update would enable you to see the realtime moving along and then after setting the system time hit the save button. The times would then be saved.

As it is now, having the ability to sync to realtime is good but you still need to change this slightly.

Time checks can be saved with a user generated title and the app also saves a number of recent checks.

The other good function is the ability to enter in the time of an event and then for the app to show you what time that would be on a DVR.

The app definitely helps the mathematically challenged like me! Its a good start but needs a few refinements to be a great tool in the box. The first one, as stated earlier would be the ability have your system clock shown within the app so you would not have to change the real time at all.

The second would be the ability to export the saved checks, perhaps to a dropbox or email address as a pdf. These would then be added into your case records.

So, it it worth £1.49? Well that’s up to you. On its own you may not think so but if you included it with another app then your opinion may change.

This next app, iCrimeFighter, is an “investigative/evidence gathering application! Use your iOS device to collect all the information you need at the crime scene or event. With iCrime Fighter you can capture photographs, record videos, record interviews, and write field notes.”

This is a free app so offsets the money spent on Clock Check. You can add job details and then see these on a map.

Starting Interface

Once the ‘job’ is entered into the database, you have the ability to add items to it.

This is where you could add screenshots from Clock Check, photographs of the Camera locations and camera views, notes on the scene etc. You can then email them all to yourself as one job.

Both apps together make a good package for Digital Multimedia Evidence Retrieval.

Lastly, and just to close this up, you may need to log other reference number’s or even design your own purpose built database. Take a look at Bento for Ipad. Although this is currently at £6.99, if you need something to keep track and link various reference numbers specific to your role then this is a very good tool to use.

DM NetVu Observer

I have always found Dedicated Micros DVR’s (DM’s) pretty nice to deal with. When you are offered a multitude of ways to get the native video, there is rarely an issue if one of those options is broken (defective Optical drive etc). It’s after the Evidential Export that some problems can occur.

Luckily, DM also give us some help on what player is being used and this is especially helpful in this case as NetVu Observer utilizes Java. You may find that some of the older versions of the player have problems with the new Java update.

So, all the .Par files that need dealing with can play as expected within this player. The problem now is that there is a lot of unnecessary footage and whilst dealing with that we need to ensure that nothing is being altered.

By selecting File > Export  – the box above appears. Initially this looks promising. Evidence disk creates a raw copy of what we have already and Video DVD is a built in Video DVD converter. These are both very handy in the right circumstances but I want to do a bit more to my DVD Copy. Due to this, AVI is my export of choice and after that I can assess what type of AVI its given me. The first issue with the player version detailed above is that the maximum avi time length that it could export was 15 mins.   There is obviously a problem with the AVI Export option as no error appears and the dialogue box states completed but the file is not what was requested!

Under the Video display window, there is a Download Video Button (highlighted above). This is usually used when utilizing the software to recover video over a network. However, it also works for exporting video already recovered.

This brings up the box above which is slightly different from the initial export box. In order to create the .avi files, the software creates temporary .par files consisting of just the footage you request in the time period input boxes. It is handy to change this to somewhere more accessible as you may want to save this. Then, after selecting the cameras of choice, the .avi export works!

We now have a number of avi wrapped video files, one for each camera selected. I requested exactly 1 hour…

Duration : 1h 0mn
Duration : 1h 0mn 16s 280ms
Duration : 1h 0mn
Duration : 01:00:16.280
Frame rate : 25.000
Frame rate : 25.000 fps
Frame count : 90407
Resolution : 8
Resolution : 8 bits
Color space : YUV
Chroma subsampling : 4:2:0

As you can see from the mediainfo report, I have a video that is reporting an extra 16secs and 280ms…… this is something to be aware of!

Although this is a format known to me I quickly had a look at the original .par files within Hxd and confirmed that I was dealing with MotionJpeg streams. Various different formats could be wrapped up inside the PAR files so its always worth checking. Most now utilise H264.

The nice thing with these is that each image is referenced with their associated timecode and this is detailed after the Frame number and jpeg header. By checking these on the temporary produced Par files I could see that the last image recorded was 16s 280ms over the 1hr mark. Keeping this file in Hxd and then dropping in one of the avi files I could compare and verify these were just avi wrapped motion jpeg streams and each image was the same bit for bit.

Looking within mediainfo again, I could establish certain other details:

Width : 704
Width : 704 pixels
Height : 256
Height : 256 pixels
Pixel aspect ratio : 0.500
Display aspect ratio : 1.375
Display aspect ratio : 1.375
Frame rate : 25.000
Frame rate : 25.000 fps
Frame count : 90407

The interesting thing here is that there are 90407 frames and at 25fps……. however when I look at the .avi in Virtualdub, there are a large amount of duplicate frames.

Dealing with these can be done in a number of ways.

As they are MotionJpeg avi’s, you have the option of using JpegSnoop. This will scan the file and extract each jpeg from within the file. It also keeps each jpegs metadata (the specific date / time etc).

Another way is FFmpeg. The command:

ffmpeg (input.avi) -vcodec copy -vsync drop (output.avi)

…. results in a motionjpeg file of all unique frames and no duplicates. Its also very fast. There are a number of benefits in doing this but the main one is that when reviewing frame by frame, you don’t have to deal with the duplicates that are created in the original avi export. Each jpeg remains intact with the important metadata.

Using both methods resulted in 2631 frames.  As we have a start time (from the first frame) and the end time (from the last frame) it’s now possible to establish the frame rate and, if required,  re-do the FFmpeg convert to add in the -r command to enter a specific frame rate. (0.72 FPS).

For grabbing frame images or review, Virtualdub or Forevid work great.

For creating a presentation multi-camera Video DVD, I found that by dropping the image only avi’s into Edius and then adjusting their duration / speed, resulted in easier scrubbing whilst editing. They also had to be re sized within the NLE to deal with the 704 x 256 field based recording.

At least this way, I can monitor any changes to the images throughout the workflow and utilize the produced files for numerous uses.

If you want to dig a little deeper and analyse the specific time-code for each image, take a look at this entry, in particular the details at the bottom.

.264 files and FFmpeg

Back to FFmpeg again!

If you have arrived here regarding V264 – Click HERE

I have recently begun to notice a few mp4 / h264 / AVC stream files that don’t want to play nicely within FFmpeg or the other small GUI’s designed to rewrap the raw streams into a container file. The need to do this is also becoming more regular due to the increasing amount of streams arriving with no other player or information file packaged with them.

So, a folder on a disk named RecordFile. Within this are a number of .264 files with a long naming convention consisting of channel number, date and time. Although there are numorous h264 players out there, my usual route is to stick a file first into MediaInfo and see what it reports.

If it is, as suspected an AVC/h264 file, then its dropped into FFplay to assess how FFmpeg handles it. If all goes smoothly then it’s either FFmpeg to rewrap or AVC2AVI.

The four files here though crashed AVC2AVI and would not initially play in FFplay.

Taking a closer look at the files in HXD, reveals a useful piece of text within the initial data. Its always worth keeping these so if you obtain data or even drives where this appears, you have a head start with your research.

Start of Data in HXD

The video data itself started at offset 10000 and there was nothing else within the data stream that looked out of place.

Data Start

It is not usually necessary to tell FFmpeg what file it is reading but with these files, as it turns out, that’s exactly what the problem is. Once I forced FFmpeg to treat the input file as a h264 video, the rewrap started.

I have also begun to document the entire FFmpeg procedure for inclusion in reports.

The workflow and FFprobe / FFmpeg commands are below:

ffprobe -show_streams -count_frames -pretty originalfile.264 > ffprobeREPORT.txt

ffprobe -show_frames -pretty > allframesREPORT.txt

I then use Windows Powershell to strip this down to just the Coded Picture Numbers and Frame Type

Now that I know the details on amount of frames and the FPS I can use FFmpeg to rewrap my stream, create an index within it to enable scrubbing back and forth and also compile a report on the process.

ffmpeg -f h264 -i originalfile.264 -f avi -r 25 -vcodec copy -vsync drop -debug_ts -loglevel verbose -report originalfile-rewrap.avi

We now have a fully scrubbable and editable file that plays great inside Virtualdub along with the ffmpeg input library.

Virtualdub Info

So, in conclusion… if files don’t want to do something within FFmpeg and you think that they should it may be worthwhile ‘forcing’ the issue and actually telling FFmpeg what the input file is by placing the -f flag before the -i input file.

On a connected note – pay particular attention to the frame numbering and counting between Mediainfo, Virtualdub and FFprobe etc. Be prepared to justify why the numbers may be different. I am finding that some rewrapped avi’s do have frames missing. When the original .264 file and the report compiled during the process are analyzed, it reveals that they are missing, damaged or out of the correct decoding order. This seems more common on files from the lower end of the market where dropping a frame every now and then is not immediatly noticeable.

Just keep your fingers crossed its not right in the middle of something important!

SSF Files with BKPlayer.exe

The BKplayer.exe comes in a variety of flavours and is linked with various different file extensions. The specific one I am dealing with here has come on an optical disk which presents the contents as seen in the image below.

If things are looking different for you, you may find that your specific version is mentioned in another page:

Part 2

Part 3

Part 4

BKPlayer file and folder structure

Note the icon for the player. There are different versions with different icons. The specific version can be found be reviewing its properties.

BKPlayer File Properties

Wouldn’t it be great if companies placed their details in here along with some contact details? There are no company details anywhere on the player.

If you have a player that is single screen only but you appear to have files relating to different cameras then check this out! SSF Files with BKPlayer – Part 2

Within the DATA folder, there is a folder with a date/numbered naming convention, and then inside that there are two large files with the .ssf file extension. This must be our video!

First, lets see if the video will play with the player.

BKPlayer Interface

The player suffers with the usual lack of information with no ‘About’ or ‘Help’ area and no details on the video type or make-up.

In the top right corner of the interface, there is a folder icon that when selected gives the option to search for video. If you have copied the contents of the disk to your HDD in order to investigate the contents you can’t just navigate to the folder and select the file. You have to open the folder with the BKPlayer in and then the program itself will identify that there is a DATA folder. It will then present the SSF files to you for selection.

Finding the SSF Files!

It would appear that although the player supports multicamera playback, the person conducting the export in this case has created a file for each camera. Although this can cause a problem if viewing a number of cameras, its actually better when we come to start analyzing the video.

Effective playback using the BKplayer seems to be rather hit and miss. I noted that when paused and using the frame fw function, I could only move approx. 10 frames before it froze. Hitting play again would unfreeze it. This caused problems when trying to establish frame rate. I couldn’t move forward and count the image changes. One of the cameras also appeared to skip frames. This was an obvious concern and required further inspection.

The third icon down gives the ability to export an ‘avi’. By selecting this, the interface displays the time duration of the selected SSF file.

By reviewing the SSF Files I recorded that:

Ch03 had a time duration of 3 minutes 21 seconds,

Ch05 had a time duration of 3 minutes 19 seconds

Upon the creation of the new avi’s, these were examined in Gspot and MediaInfo. Both avi’s were Divx 5.0 format with the DX50 4CC code. Both had a GOP structure of IPPPP.

Ch03 – 4513 Frames, 22FPS, 3 minutes 25 seconds

Ch05 – 2509 Frames, 12FPS, 3 minutes 29 seconds

I finally compared the raw data from the avi’s to that in the original SSF’s.

Comparing Hex

We now have a problem. The footage has been transcoded, the duration’s have changed and when reviewing Ch05, it plays very slowly. For some reason the transcoding process has slowed it down – Its got the frame rate wrong!

The conclusion here is that those AVI’s are not worth much and are pretty unreliable.

Back to the SSF’s….

MediaInfo gives us some useful information:

Video
Count : 248
Count of stream of this kind : 1
Kind of stream : Video
Kind of stream : Video
Stream identifier : 0
Inform : 352*288 (1.222), AVC (PAL) (Baseline@L3.0) (1 Ref Frames)
Format : AVC
Format/Info : Advanced Video Codec
Format/Url : http://developers.videolan.org/x264.html
Commercial name : AVC
Format profile : Baseline@L3.0
Format settings : 1 Ref Frames
Format settings, CABAC : No
Format settings, CABAC : No
Format settings, ReFrames : 1
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=24
Internet media type : video/H264
Codec : AVC

As both SSF’s had been identified as H264 streams I used FFMPEG to wrap up the video into a standard avi container.

FFMPEG -i filename.ssf -an -vsync drop -vcodec copy -f avi out.avi

I told FFmpeg to wrap the video using avi (-vcodec copy -f avi) but to ignore the audio (-an) and most importantly to rewrite the Presentation Time Stamp (-vsync drop).

The resulting files played and scrubbed perfectly within Virtualdub when using the ffmpeg input driver. The I frames were also identified in the GOP structure. I also compared and verified the raw data was present in both the SSF and the new avi.

Scrubbing through the Video in Virtualdub (Imaged blurred on purpose)

Due to the timecodes being removed from the new avi, in order for it to play correctly, a frame rate needs to be established.

For Ch03, this was the total frames (4518) divided by the duration of the video in seconds (201). This equates to 22 FPS.

The result to all this was that I now had a correct piece of video that I could analyze effectively. There were two questions though…

Was it possible to analyze the original SSF before wrapping it into an avi and how to deal with the other camera that appeared to have an incorrect frame rate?

I have been using FFmpeg for a while but this seemed to be a job for FFprobe. This program, that comes along with FFmpeg, attempts to open an input file and then probe it for information. Its similar to a lot of video information tools and is actually used as a backend for a few of them. However, if you use it from the command line, you can see whats going on and get the information you require.

Although its possible to do the first bit here in Windows, I chose to do both in Linux. The reason is that in the second part, I am going to only output specific information into a text file. To do this I will use the GREP utility that is not as easy to get done in Windows.

ffprobe -show_streams -count_frames -pretty /home/david/Desktop/14263902.SSF > /home/david/Desktop/Stream-FrameCount.txt

The above  command results in a txt file being created with the following information… (remember when saving these from Linux to use in Windows – you have to change the encoding from Unix to Windows. That way your text will appear in the same format).

The text file:

STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Baseline
codec_type=video
codec_time_base=1/50
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=352
height=288
has_b_frames=0
sample_aspect_ratio=0:1
display_aspect_ratio=0:1
pix_fmt=yuv420p
level=30
timecode=N/A
is_avc=0
nal_length_size=0
id=N/A
r_frame_rate=50/2
avg_frame_rate=25/1
time_base=1/1200000
start_time=N/A
duration=N/A
bit_rate=N/A
nb_frames=N/A
nb_read_frames=4518
nb_read_packets=N/A
[/STREAM]

Obviously, we knew a lot of this from parsing the SSF file through MediaInfo but we didnt know the frame count until we created the .avi. By using this we can get the frame count directly from the original file.

Then it’s time to identify the GOP structure and individual frame types.

ffprobe -show_frames -pretty /home/david/Desktop/14263902.SSF | grep ‘pict_type\|coded_picture_number’ > /home/david/Desktop/FrameID.txt

This will output a text file showing each frame number and its associated frame type, part of it is seen below…

coded_picture_number=21
pict_type=P
coded_picture_number=22
pict_type=P
coded_picture_number=23
pict_type=I
coded_picture_number=24
pict_type=P
coded_picture_number=25
pict_type=P
coded_picture_number=26
pict_type=P
coded_picture_number=27
pict_type=P
coded_picture_number=28
pict_type=P

From this I can now verify my frame readings in Virtualdub. Comparing these resulted in a positive match.

If I had not used Grep to narrow down the output, my result for each frame would have been the text below:

[FRAME]
media_type=video
key_frame=1
pkt_pts=N/A
pkt_pts_time=N/A
pkt_dts=N/A
pkt_dts_time=N/A
pkt_duration=48000
pkt_duration_time=0:00:00.040000
pkt_pos=0
width=352
height=288
pix_fmt=yuv420p
sample_aspect_ratio=N/A
pict_type=I
coded_picture_number=0
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
reference=3
[/FRAME]

See the following posts for updates on dealing with FFprobe outputs

FFprobe and Powershell

Update on I frame Only Extraction

So, the first question was answered…It is possible to analyze the SSF file and obtain the information needed so you can create the wrapped avi along with a correct frame rate (by adding -r 22 to the ffmpeg command) and then verify the GOP structure between the two.

Lastly, the other camera that suffered with the missing frames.

As I stated earlier, the player was difficult to use in order to establish exactly what was going on and why it was skipping during playback. I followed the same methodology for this camera so I could now get the file out and check a few things. It would appear that the missing frames was as a result of a poorly instigated Video Motion Detection system. There are small gaps in the video when there is low movement. This is not clear in the player and does not come across when using the players inbuilt export to avi function. The export function doesn’t know there are gaps so it stretches the video to fit the time scale – thats why it appeared in slow motion. This is very important to note as it could cause a lot of confusion and misunderstanding if these exports are used.

There is a lot of information on FFprobe online and all the various options that it can output. Hope it helps!

Part 2 – The Single Player