The Audio Driven Animation feature in MetaHuman Animator gives you the ability to process audio into realistic facial animation.
MetaHuman Animator has the ability to take depth footage (either from an iPhone with a depth sensor or from a stereo camera setup) and process it into facial animation. You can learn more about capturing your facial performance by reading the MetaHuman Facial Performance Capture Guidelines.
The Audio Driven Animation feature adds another input option to process SoundWave (audio) assets into facial animations in an analogous way.
The facial performance capture method requires a more complex and expensive capture setup when compared to using audio to drive facial animations.
This feature is also available in UEFN, with the exception of exporting a level sequence from a performance, which is currently not supported.
Audio Capture Recommendations
The quality of the generated facial animations will depend on the clarity of the recorded audio.
Follow these recommendations to improve the animation results:
Recommendation | Configuration |
---|---|
A minimum 16kHz sampling rate | Lower sampling rates will remove important speech related frequencies from the signal. |
Minimize background noise | Audio with a higher signal-to-noise ratio will result in cleaner animations. |
Avoid reverb and echoes | The animation quality will be reduced if these effects are present in the recording. |
One speaker per audio file | The feature will animate to all voices present in the recording, so having multiple speakers / background voices could affect the resulting animation. |
MetaHuman Animator can use any audio format supported by Unreal Engine. Read the Importing Audio Files documentation to learn more.
Processing in the Performance Asset Editor
Create a new MetaHuman Performance asset by right-clicking in the Content Browser, then select MetaHuman Animator > MetaHuman Performance.
Double-click the MetaHuman Performance file to configure the Details panel.
Select Audio as the Input Type (1).
In the Audio sample, select the SoundWave asset that will be used to generate the animation (2).
In the Visualization Mesh, select your MetaHuman face mesh (3).
Click Process to generate the facial animations.
This will process the audio and create the Facial Rig animation tracks in Sequencer.
Once the process is complete, you can click Export Animation or Export Level Sequence to export the facial animation. If you choose Level Sequence, remember to assign your MetaHuman character blueprint.
Option | Description |
---|---|
Downmix Channels | If enabled, this will downmix multi channel audio (most commonly stereo) down to a mono channel when processing. If disabled, the channel used for processing is specified by the “Audio Channel Index”. |
Generate Blinks | This option will enable or disable the generation of blinks with the audio solve. |
Batch Processing from the Content Browser
With batch processing, you can process several audio assets simultaneously. To do this, follow these steps.
In the Content Browser, select all the SoundWave assets you want to process.
Right-click and select the MetaHuman Performance menu. This menu has 3 options:
Create Performance and Process
Process and Export to Anim Sequences
Process and Export to Level Sequences
Create Performance and Process
This creates a MetaHuman Performance asset for each SoundWave and processes them all individually.
Upon selecting this option, a dialog appears.
Select the output path.
Add a prefix and/or suffix to the output files.
Click Export.
In the next dialog window, select the MetaHuman face mesh to use for the animation.
Once you click Create, processing will commence and add each performance asset to the specified output folder.
Process and Export to Anim Sequences
This creates a MetaHuman Performance and Animation Sequence for each SoundWave and processes them all individually.
Select the output path.
Add a prefix and/or suffix to the output files.
Click Export.
In the next dialog window, select the Target Skeleton or Skeletal Mesh of the MetaHuman to use for the animation.
Once you click Create, processing will commence and add each performance asset in the specified output folder.
Process and Export to Level Sequences
This creates a MetaHuman Performance and Level Sequence for each SoundWave and processes them all individually.
Select the output path.
Add a prefix and/or suffix to the output files.
Click Export.
In the next dialog window, select the MetaHuman blueprint for the Target MetaHuman Class.
Once you click Create, processing will commence and add each performance asset in the specified output folder.
Processing with Python
In the plugin’s /Content/Python folder, there is an example script
process_audio_performance.py
which can be used to create and process a MetaHuman performance from a SoundWave asset.The example script
export_performance.py
can be used to export anim sequences and level sequences from performances.The
/Content/Python/ReadMe.txt
gives some additional information on how to call these scripts.