Stereo Camera Calibration and Tools

Working with Stereo Footage

This guide will take you through the steps necessary to take captured data from a Head Mounted Stereo Camera Couple, and prepare it for processing with the MetaHuman for Unreal Engine plugin. The data is expected in specific file formats and directory structure. If the data is formatted correctly, it is possible to compute and export the calibration parameters needed for reconstruction. This page provides Python scripts and a calibration application to support these tasks.

In this guide you will go through the installation and usage of these tools to prepare example data. For guidance on how to capture the best data, please see the Capture Data page.

Tools and Dependencies

Tool Locations

Download the MetaHuman Stereo Tools plugin from the Marketplace and go to the following directory:

C:\Program Files\Epic Games\UE_5.4\Engine\Plugins\Marketplace\StereoCaptureTools\Content\StereoCaptureTools.zip

Unzip the file and open the folder.

Install the Calibration App

Inside the StereoCaptureTools > Calibration App folder, you will find a CalibrationApp.exe file for the calibration app:

C:\Program Files\Epic Games\UE_5.4\Engine\Plugins\Marketplace\StereoCaptureTools\Content\StereoCaptureTools\Calibration App\CalibrationApp.exe

In order to run this application without needing to remember its location, add this folder to the path environment variable. You can do this via Windows Settings > Environment Variables, and then adding it as a new entry to your path.

You can do this via Windows Settings > Environment Variables, and then adding it as a new entry to your path.

Install FFmpeg

The bundled Python scripts require FFmpeg to be installed and available on the system path. You can download FFmpeg from ffmpeg.org.

Once installed make sure to add the folder where the ffmpeg.exe binary resides to the system path. You can do this via Windows Settings > Environment Variables, and then adding it as a new entry to your path.

Install Python

To run the bundled Python scripts, install Python 3.9.7 or above. Once installed, add Python to your user’s path via the checkbox in the installer. You can download Python from python.org.

Install Python Dependencies

You will find the Python script(s) used for the conversion process in this directory:

C:\Program Files\Epic Games\UE_5.4\Engine\Plugins\Marketplace\StereoCaptureTools\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools

These scripts have dependencies which you will need to install before they can function. It is recommended that you install these dependencies into a virtual environment to avoid installation in the main Python environment.

To create a Python virtual environment, enter the installation path, including the folder name. In this example, you will create a virtual environment inside a folder called conversion_venv. To do this, open a cmd or powershell window and run the following command:

python -m venv conversion_venv

Before you can use the virtual environment, you must activate it for your current shell. The activation method is the same, but the extension on the script varies.

For cmd:

.\conversion_venv\Scripts\activate.bat

For powershell:

.\conversion_venv\Scripts\activate.ps1

When you have successfully activated the virtual environment, you will see an indicator besides the shell prompt containing the name of the virtual environment, in this case “conversion_venv”.

Now that you are inside an activated virtual environment, you are ready to install the dependencies. The dependencies are specified inside the file called requirements.txt within the Stereo Capture Tools folder.

To install them run the following command:

pip install -r requirements.txt

Once that command has finished, the dependencies will be installed inside the virtual environment.

Data Preparation

Preparing the data consists of two steps: converting the data and creating the calibration, with the goal of creating the layout described in Ingest Structure. This data structure is required in order to ingest captured data into Unreal Engine.

Convert the Data

Overview

The script used to convert stereo data into a suitable format for MetaHuman ingest is called mh_ingest_convert.

The script does the following:

• Takes two videos from the stereo cameras, along with any audio, and extracts the video frames into a pair of image sequences.

• Generates the necessary metadata.

• Copies the audio to the output folder.

The basic command structure is as follows:

python mh_ingest_convert.py bot <bot_video> top <top_video> <image_extraction_method> <output_folder> --options

To see all of the available options, run the following command:

python mh_ingest_convert.py --help

Image Extraction Method

The image extraction method can have a significant impact on the disk space needed for the image sequences, so it is important to choose the most appropriate method for the videos you have.

The available options are as follows:

If you have a mjpeg-encoded video, then the script extracts the (already jpeg encoded) frames into jpeg files, with minimal overhead in data size compared to the original video. This method will not work for any other video encoding. You can use jpg_copy with both grayscale and color videos, as long as your video is mjpeg-encoded.

If your videos are not mjpeg-encoded and have grayscale data, use the png_gray or png_rgb24 method.

The jpg_lossy method reduces the image data size even further, at the expense of reconstruction/animation quality. By using this method you are reducing the quality of your footage and this may impact the depth maps generated during ingest. For this reason, we only provide the minimum compression level to the jpeg encoder, to reduce the impact on the depth maps.

Example Conversion

To demonstrate how to prepare sample footage, you will run through a quick example. Here we assume that you have some calibration footage and two actor performances. One actor performance will be used for identity creation and the other as an actual performance.

In each of these cases, there are two video files (bot.mov and top.mov) with an additional audio file (audio.wav) for the performances. Or, more visually:

+---example_data
|   +---identity
|   |       audio.wav
|   |       bot.mov
|   |       top.mov
|   |
|   \---performance
|           audio.wav
|           bot.mov
|           top.mov

For the following example, we assume an Unreal Engine version of 5.3. We also assume mjpeg-encoded video, so we may use the jpg_copy image extraction method.

Activate the Virtual Environment

Activate your virtual environment, if you have not already done so. See the previous section for instructions on how to do this. Once you are inside an “activated” shell prompt you can proceed to convert the data.

Create the Output Folder

mkdir C:\MyData\example_data_prepared

Calibration

To convert the calibration footage, pass the following information:

• The top and bottom videos.

• The image extraction method (jpg_copy in this case).

• An output folder.

python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHuman\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py"
bot C:\MyData\example_data\calibration\bot.mov  
top C:\MyData\example_data\calibration\top.mov 
jpg_copy 
C:\MyData\example_data_prepared\calibration

Identity

The conversion of the identity performance works the same way as calibration, just with an additional parameter to specify the location of the audio file (if desired):

python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHuman\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py" 
--audio-path C:\MyData\example_data\performance\audio.wav 
bot C:\MyData\example_data\performance\bot.mov
top C:\MyData\example_data\performance\top.mov 
jpg_copy
C:\MyData\example_data_prepared\identity

Performance

As the performance data has the same layout as the identity footage, you can convert it in the same way by changing the input and output paths:

python "C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace\MetaHuman\Content\StereoCaptureTools\stereo_capture_tools\stereo_capture_tools\mh_ingest_convert.py" 
--audio-path C:\MyData\example_data\performance\audio.wav
bot C:\MyData\example_data\performance\bot.mov
top C:\MyData\example_data\performance\top.mov 
jpg_copy
C:\MyData\example_data_prepared\performance

Repeat this performance conversion step for as many performances as you wish to convert.

Things to keep in mind

• If a calibration file is provided on the command line, it will not be automatically copied into the output folder, nor does it need to exist at the time the script is run. As far as the script is concerned, the calibration path is a string to be written into the take.json file.

• The slate name will, by default, be determined from the root folder of the first video file path. However, this can be overridden on the command line.

Create the Calibration

Overview

To create a stereo camera calibration file, use the HMC calibration app you installed earlier.

To see all of the available options, run the following command:

CalibrationApp.exe --help

Defining the Calibration Board

The parameters of the calibration board used to capture the calibration footage need to be defined and specified as input to the calibration app.

To do this, make a copy of the default configuration from the following location:

C:\Program Files\Epic Games\Calibration App\default_hmc_config.json

Open the file and update the following parameters to match the characteristics of your calibration board:

patternHeight

• The number of internal corners on the height of the board.

• This is the number of squares minus one.

patternWidth

• The number of internal corners on the width of the board.

• This is the number of squares minus one.

squareSize

• The edge size of each square in cm.

If you are using the example calibration board, found here

StereoCaptureTools.zip\StereoCaptureTools\stereo-calib-grid-16v11h-7p5mm.png

Then the configuration file should look something like this:

{
    "cameras": [
        [
            "top",
            "bot"
        ]
    ],
    "defaultBlurThreshold": 7.5,
    "patternHeight": 15,
    "patternWidth": 10,

To use your new configuration, pass it to the calibration app using the -c parameter as such:

CalibrationApp
-f C:\MyData\example_data_prepared\calib
-e C:\MyData\example_data_prepared\calibration\calib.json
-n 30
-c C:\MyData\example_data_prepared\custom-config.json

Alternatively, these parameters can be overridden on the command line using the --sq (--squareSizeOverride), --ph (--patternHeightOverride) or --pw (--patternWidthOverride). For example:

CalibrationApp 
-f C:\MyData\example_data_prepared\calib 
-e C:\MyData\example_data_prepared\calibration\calib.json 
-n 30 
--sq 0.75 --ph 15 --pw 10

Example Calibration

Continue here with the same example from the Converting the Data section.

To create a calibration file, run the HMC calibration app with the following information:

• Where to find the calibration footage.

• Where to output the calibration file.

• The number of frames to use.

CalibrationApp 
-f C:\MyData\example_data_prepared\calibration 
-e C:\MyData\example_data_prepared\calibration\calib.json 
-n 30

The number of frame samples specified with the -n argument are spread throughout the full length of the footage, so the longer the calibration takes, the more spread out those samples will be.

When you run the above command, you should see an output similar to the following:

Detection Started!
Detection Finished!
Intrinsic Calibration Started.
bot: 30/30 selected images for intrinsic calibration. (0 blurry images).
top: 30/30 selected images for intrinsic calibration. (0 blurry images).
Intrinsic Calibration Finished.
Extrinsic Calibration Started.
60 selected images for extrinsic calibration.
2.12488e+08 3084.77
Calibration has been saved to: C:\MyData\example_data_prepared\calibration\calib.json

Once your calibration file has been created, copy it into each of your performance folders. For the example data you created earlier (see Ingest Structure for more details), do the following:

copy 
C:\MyData\example_data_prepared\calibration\calib.json 
C:\MyData\example_data_prepared\identity\calib.json

copy 
C:\MyData\example_data_prepared\calibration\calib.json 
C:\MyData\example_data_prepared\performance\calib.json

Ingest Structure

The output of the stereo tools ingest conversion script is a directory structure containing a mixture of mandatory and context dependent files. The directory structure is as follows:

+---Output Folder
|   +---bot
|   |       001.png
|   |       002.png
|   |       ...
|   +---top
|   |       001.png
|   |       002.png
|   |       ...
|   take.json

The bot and top directories contain image sequences created from the input video files. These will be in .png or .jpeg format depending on the command line arguments supplied to the mh_ingest_convert.py script.

The thumbnail.jpg will be displayed in Unreal Engine to help identify the take. The mh_ingest_convert.py script extracts the first frame from one of the input video sequences in order to generate the thumbnail.

The take.json is a metadata file which contains all the necessary information about a take in order for it to be ingested into Unreal Engine. This includes details about the Slate and Take number, timecode information, and frame ranges alongside paths to the associated take files. These files include image sequences, thumbnail, and calibration.

Depending on the arguments supplied to the mh_ingest_convert.py script, there may be additional files copied into the output directory structure. For example, using the --audio-path argument will additionally copy the supplied audio file into the output folder.

Things to Keep in Mind

The take.json must reference a calibration (calib.json) file. By default, the take.json references a calib.json in the chosen output folder. The calib.json must be generated and copied to this location manually. If the take.json does not reference a valid calibration file, or that calibration is not present at the specified location, this will result in an error when trying to ingest into Unreal Engine.