ML Deformer Framework

When selected this button will begin training the model using the properties in the ML Deformer Editor’s Details panel, to generate Morph Targets of your character’s mesh deformations.

Here you can select the ML Deformer model you would like to use. You can select which model you would like to train the ML Deformer with from the available options in the drop down-menu. Changing the selected model will wipe your current session, so it is recommended to only change this right after creating a new ML Deformer asset.

Here you can toggle the ML Deformer to either the Training mode or the Testing mode.

• The Training mode allows you to inspect your training data. You will find all training related settings in the Details panel.
• The Testing mode is used to view the resulting deformation of the trained model. You can access testing and debug settings in the Visualization panel during testing.

Here you can access the ML Deformer Editor’s Tools. Using the Drop Down Menu, select the tool to open each tool’s options where you can adjust its properties and run the operation. You can access the following tools using the Tools menu button:

• Key Pose Extraction Tool
• Get Neighbor Stats

The tools contained in this menu are currently mostly applicable when working with the Nearest Neighbor ML Deformer model.

For more information about the individual tools and a list of their properties and a description of their functionality, see their respective sections below.

Here you can select what Actor you would like to debug. By selecting an actor using the drop-down menu, you can view the deformations on an actor in an active Play In Editor (PIE ) session. After selecting an actor and world, the Actor in the ML Deformer Editor’s viewport will show the same pose as in your PIE session. Only actors using the currently opened ML Deformer asset can be picked to debug.

Projects often blend several animations together and do procedural modifications such as IK, debugging can be helpful, as you can test only with a single animation sequence in the ML Deformer editor when not debugging.

You can refresh the list of available actors for debug using the Refresh button in the toolbar.

Set the data to extract key poses from the following options:

• Poses: Set the poses to be clustered.

• Cache: Set the geometry cache for each pose. This is only required when Extract Geometry Cache is checked.

  The number of frames in Cache must match the number of keys in Poses

• Must Include Frames: Set frames that must be included in the final output.

Here you can assign the Animation Sequence asset to extract the pose clusters too. While you can overwrite an existing asset in your project, you can also use the New Poses button to create a new asset in order to save your extracted pose clusters.

When the Extract Geometry Cache property is enabled, here you can assign the asset to save your extracted geometry cache data to. While you can overwrite an existing asset in your project, you can also use the New Geometry Cache button to create a new asset in order to save your extracted geometry chase data.

Set the Animation Sequence asset that will be played on the Skeletal Mesh in the Viewport. Use this property to set an Animation Sequence other than your training Animation Sequence to see how the ML Deformer performs during playback.

Set the Deformer Graph asset to use on the ML Deformer Editor’s deformed test actor. Deformer Graph assets define deformers in a graph based system and can do things such as performing skinning, morphing and recomputing normals at runtime. You can set the Deformer Graph asset you intend to use with your character at runtime here to observe the results. If you want to use a deformer graph in combination with this ML Deformer asset at runtime in your project, you will need to specify the deformer graph asset in the "Mesh Deformer" property of your Skeletal Mesh Component as well. Changing the deformer asset in the ML Deformer asset visualization settings does not automatically add it to your skeletal mesh components.

Set the scale factor of the mesh deformation deltas applied on the ML Deformed mesh in the Viewport. A value of 0.0 will render the same mesh deformations as the Linear Skinned mesh, while a value of 1.0 will render the full ML Deformer mesh deformation correction.

Here you can manually set the frame number in the playback of the Training Anim Sequence. This value will automatically update when playing or scrubbing the animation in the sequence playback controls.

When enabled, a debug view of the vertex delta values between the Linear Skinned Mesh and the Ground Truth mesh will be rendered in the viewport.

When the Draw Deltas property is enabled, you can toggle whether the vertex delta debug drawing should also display the deltas that are normally occluded by the mesh or not.

When enabled, a heatmap showing the delta of change between the Linear Skinned Mesh and the ML Deformer corrected results will be rendered on the ML Deformed mesh in the Viewport. Cooler colors indicate small values, while warmer colors depict larger values. Grey indicates no values are present.

Here you can set the mode of the heat map, to observe different data. You can select the mode using the drop down menu, and can select the following modes:

• Activations visualizes areas where the ML Deformer is applying corrections, cooler colors represent little corrections, while warmer colors represent larger changes to the mesh. Grey indicates that no changes are being made.
• Ground Truth visualizes the error between the Ground Truth mesh and the ML Deformed mesh. This mode requires that a Geometry Cache asset is set in the Ground Truth property. Cooler colors indicate little error, while warmer colors indicate large errors. Grey indicates there is no error.

After training your ML Deformer model, you can enable this property to see a debug drawing of the Skeletal Mesh’s Morph Targets in the Viewport. This property can be helpful to observe how sparse the generated Morph Targets are.

Here you can set the index value of the generated Morph Target that will be rendered in the Viewport.

When enabled, all ML Compare meshes will be visible in the Viewport.

Here you can set the visualization mode for your meshes masks. Each bone, curve, bone group, and curve group has a specific mask area on the mesh. This mask defines areas where generated Morph Targets can be active. They can be used to filter out deformations in undesired areas. For example if you rotate the left arm, you don't want the right arm to deform. The mask for the left arm can be set up in a way that it only includes vertices around the area of the left arm to enforce this.

You can select the Mask Viz Mode using the property’s drop-down menu. You can select from the following options:

• Off: Disables the Mask visualization entirely.
• When in Focus: Only shows the mask inside the viewport when the input widget on the right side of the UI is in focus. Show the masks for the selected item (bone, curve, bone group, curve group).
• Always: Always show the selected mask. The mask selected is defined by what is selected in the input widget on the right side of the UI.

Here you can reference the Main System memory and GPU specific memory usage of your model in megabytes (mb).

Here you can reference the estimated cooked and uncooked sizes of your model in megabytes (mb).

Set the linear skinned Skeletal Mesh asset for your character you are training.

Set a list of Training Inputs that will be used to train the selected ML Deformer Model. Each list should contain a pair of Animation Sequences containing mesh poses, and Geometry Cache assets that contain mesh deformation data. Both assets per pair should run at the same frame rate, and contain the same number of frames, with each frame of each asset containing the same pose. Each asset’s mesh should also contain the same number of vertices.

To create a new input index use the (+) Add. After creating an Input index, you can expand the settings for each index to access the following properties:

• Anim Sequence: Set the Animation Sequence asset that will be used to sample mesh poses to train the selected ML Deformer model. This sequence can contain random poses, or a specific training sequence. If the Animation Sequence contains any Animation Curves, they will be extracted during training if they are provided as Network Inputs.
• Geometry Cache: Set the Geometry Cache asset that contains the target high quality mesh deformations, which will be used to to train the selected ML Deformer model. The Geometry Cache asset should contain the same framerate, number of frames, and poses on each frame as the defined Animation Sequence in this Training Input index. The Geometry Cache asset can contain fewer meshes than the Skeletal Mesh, for example, if you are only training the body deformations of a character and not the head.
• Use Custom Range: When enabled you can set a custom range of frames from the assigned Animation Sequence and Geometry Cache assets.
• Start Frame: When the Use Custom Range property is enabled, set a start frame for the desired custom range.
• End Frame: When the Use Custom Range property is enabled, set an end frame for the desired custom range.
• Enabled: When enabled, this property will include this index in the training process. Using this property you can add multiple Training Input Anims indexes and toggle which ones to use for specific use cases or to test results with different data.

At least one index of Training Inputs is required to train the ML Deformer model. You can add additional training inputs to further improve your mesh deformation results when, for example, you notice a particular pose is not deforming properly. Additional data, containing targeted data around these poses can improve your results.

Here you can set a transform value to adjust the positioning of the Geometry Cache with the Skeletal Mesh. Use this property if you notice your Geometry Cache asset is not aligned with your Animation Sequence poses correctly. It is recommended to use this property to apply scale and rotation adjustments if your Geometry Cache mesh’s vertices are not overlapped with your Animation Sequence mesh’s vertices. The goal of the training process is to calculate the distance between each vertice on the meshes when deformations occur, so correctly aligned vertices are critical to good results. Keep in mind that the Mesh Spacing setting in the Visualization details panel still separates the Linear Skinned mesh and your Geometry Cache. Make sure the meshes overlap and have the same scale and orientation when Mesh Spacing is set to 0.

Here you can set the bones and curves that will be considered when training the ML Deformer Model. To add a bone or curve from your character’s Skeletal Mesh asset, use (+) Add and select either Add Bone or Add Curve to add a single bone or curve, or Add All Animated Bones or Add All Animated Curves to import all of your characters available animated bones or curves. It is good practice to first add all animated bones or curves, and then trim them down by for example removing helper bones.

Added bones will be used to define the pose of the character. The model will relate the orientation of the bones during each pose to the corrective deformations the model will output. It is recommended to set as few bones as possible, while still capturing all of the necessary movement of your character to achieve the best results, for example you should exclude all helper bones. When using the Add All Animated Bones function, the Editor will automatically add all bones to the list that contain animation data. This process can add twists to bones that you will most likely want to remove afterwards.

If you use the Neural Morph Model and have set the Mode to Local, then after adding bones to the list, you can right-click a selected bone and select the Edit Bone Mask option to open a window where you can edit if that bone should be included in the bone mask. By default, every vertex that is skinned to the bones inside the bone mask are included in the bone mask. After editing the bones in the bone mask you can right click within the list of bones and select the Reset All Bone Masks option, to set all bones masks to their default settings.

Added curves can be used to influence the model using alpha values that can represent many things, such as weights, alphas, facial controls, or muscle activations. Using the Add All Animated Curves option will add all curves contained within the Animation Sequence asset that have value animated value changes.

You can remove individual bones or curves from the list by selecting the item, right-clicking and selecting Delete Selected. You can also clear either the Bone List or the Curve List entirely by right-clicking the list and selecting the Clear List option.

Set the maximum number of frames from the assigned assets in the Input property to use when training the model. By reducing this number you can train the model using a smaller section of your training data. Samples taken will alternate between the input animations. It will take a sample of the first enabled one, then the next one, and so on, until we reach the maximum number of training frames or there are no frames left to sample.

Set the maximum length of a delta value that will be considered during training. All delta values longer than the specified amount will be ignored and set to zero length. This property can be helpful to reduce issues caused by very long delta values. You can view a debug render of delta lengths in the Viewport by enabling the Visualization panel’s Draw Deltas property, when the ML Deformer Editor’s mode is set to Training.

Set the number of frames per batch when training the ML Deformer model. Higher values will use more GPU memory during training but can reduce how long the training process takes. Batch size can also influence the quality of your results. The iteration count in combination with batch size impacts quality. If you halve your batch size, you need double the number of iterations to get to the same quality.

Set the initial step size in the optimization process during the ML Deformer model training. Larger values can deliver faster results during training, but may overlook critical data, delivering less optimal results. Lower values can deliver higher quality results, but could take an infinite amount of time to train. The goal is to find an optimal value that captures enough data during the training process without requiring incredibly long training times. It is recommended to start with larger values, and work down to achieve a loss level that is acceptable for your project's needs. Each ML Deformer Model will deliver different results, even with the same value, so it is recommended to approach each model differently.

Set the maximum number of Skeletal Mesh LOD levels should be used to generate ML Deformer LODs. A value of 1 will only allow the ML Deformer on the highest quality (LOD0) of the Skeletal Mesh.

Set the dimensions of hidden layers in the network. To set a hidden layer, use (+) Add, then set the value of the index to the layer’s value you want to hide.

Here you can set a value that will determine the cutoff for the Morph Target delta values. Delta values smaller than or equal to this threshold will be ignored and set to 0.0. This property is useful to remove small deltas from Morph Targets, which will lower the memory usage at runtime. However when this value is set too high, it can also introduce visual artifacts to your results. A value of 0.0 will result in the highest quality Morph Targets to be generated, at the cost of higher runtime memory usage.

Set the generated Morph Target’s compression level. Higher values will apply more compression, but could result in visual artifacts on the mesh. Lower values will apply less compression, resulting in high quality results, but will require more memory usage. It is recommended to use values between 20.0 and 200.0, for optimal results.

Set the channel data that represents the delta mask multipliers. You can use this property to break down and distribute influences on the ML Deformer in specific areas, such as targeting high motion joint seams, such as a character's neck or shoulders. The painted vertex color values will be used like weight multipliers on the ML Deformer deltas applied to that vertex. You can select a Mask Channel using the property’s drop-down menu. You can select from the following channels:

• Vertex Color Red: Uses the Red Vertex Color Mask channel.
• Vertex Color Blue: Uses the Blue Vertex Color Mask channel.
• Vertex Color Green: Uses the Green Vertex Color Mask channel.
• Vertex Color Alpha: Uses the Alpha Vertex Color Mask channel.

This property can be useful to target the training process on specific areas of your mesh you want to improve mesh deformations with unique settings, rather than training the entire mesh using the same settings.

Here you can add sections to break down the Nearest Neighbor Model’s training process for individual mesh components of your character. Each section contains a set of vertices in the original Skeletal Mesh. The nearest neighbor search is then performed on each section separately. For example, if a character asset has both a shirt and pants mesh, you can define each mesh as a separate section in order to perform the nearest neighbor training on each mesh individually. You can add a section index using (+) Add, then expand the section properties to access the following settings:

• Num PCACoeffs: Set the number of PCA (Principal Component Analysis) coefficients for this section.
• Weight Map Creation Method: Set a method to create weight maps for this section. You can select From Text to include all vertices from text with a weight of 1 using the Vertex Indices property, Selected Bones to use skinning weights from the selected bones using the Selected Bones property, or Vertex Attributes to use weights from a vertex attribute set using the Attribute Name** property.
• Vertex Indices: When the Weight Map Creation Method is set to From Text, you can select the string value that represents the vertex index you want to use to create a weight map, using the drop-down menu.
• Selected Bones: When the Weight Map Creation Method is set to Selected Bones, you can use the Select Bones button to select a bone from your character’s skeleton, to use to create a weight map. You can also use the Create Attributes button to open a dialogue box, where you can name and create a new Vertex Attribute that will be used with the selected bone when creating a new weight map.
• Neighbor Poses: Set an Animation Sequence asset that contains the poses of the nearest neighbor ROMusing the asset selection drop-down menu..
• Neighbor Meshes: Set a Geometry Cache asset that contains the nearest neighbor ROM using the asset selection drop-down menu.
• Excluded Frames: Set an array of frames to exclude from the nearest neighbor ROM using (+) Add.

When enabled, the ML Deformer will cache intermediate results on your computer’s hard drive.

Set the mode in which the ML Deformer’s neural network will operate in. You can select from the following options:

• Local: Operates with a local neural network per bone. This mode will operate with higher performance but you may experience lower quality deformations.

This mode is optimized to learn deformations around individual joints, and requires the least amount of training data. This mode also produces more sparse Morph Targets which reduces the memory usage, and can be processed faster on the GPU. You will also see faster CPU performance using the Local mode, as evaluations of bones are broken up into smaller parts rather than evaluating the entire skeleton at once.

This option should be used by default unless you encounter difficulty reconstructing the training data. Each bone or curve will create a set of morph targets in this model. The total number of morph targets generated will be:

(num_bones + num_bone_groups) * num_morphs_per_bone_or_curve + (num_curves + num_curve_groups) * num_morphs_per_bone_or_curve + 1

The + 1 at the end is just one Morph Target that is always created and is used for normalization of the data. This is always the first morph target in the list of generated morph targets. The Local mode can handle structured data, such as structured ROM’s better than the Global mode.

• Global: Operates using a global network that considers all bones together. This mode will operate with a lower performance, but can deliver higher quality mesh deformations.

This mode learns deformation from coordinated movement of multiple joints, but requires more training data. This mode acts similar to the Local mode, but acts like all bones are in one big bone group. This works best with randomized poses as training data.

This value represents how many Morph Targets will be generated for your mesh during the training data. Higher values will produce higher quality results but will require more memory and performance. It is recommended that you aim to set this value to the lowest possible value while still retaining the quality of your deformations.

When the Mode is set to Local, this value represents the number of Morph Targets that will be generated for each bone, curve, bone group, and curve group. When using the local mode, it is recommended to input the lowest values possible while still retaining your expected quality. It is recommended that this value be somewhere between 4 and 12 but may differ depending on your project's needs.

When the Mode is set to Global, this value represents the total number of Morph Targets that will be generated for your Mesh. It is recommended that this value be set between 64 and 256 but may differ depending on your project's needs.

Set the number of iterations to train the model for. For testing purposes and in-development projects smaller values, such as 1000 - 3000 will deliver usable results. For final asset generation high values, such as 100000 - 1000000 will produce higher quality results.

Set the number of hidden layers that the neural network model will have. Higher values will reduce your project's performance, but can work with more complex mesh deformations. When using the Local Mode, a value of 1 is usually adequate. When using Global Mode a value between 1 and 3 typically provides the best balance of performance and quality results.

Set the number of units per neuron per hidden layer. Higher numbers will reduce performance but will allow for more complex mesh deformations.

Set the Regularization Factor of the training data which helps with generalizing the model. Higher values can also lead to sparser morph targets, which can reduce GPU memory usage and improve GPU performance. The default value of this property is 1.0, if this does not provide good results, try disabling this property with a value of 0.0, and working up, until you find the smallest value that gives good results.

Set the beta parameter in the smooth L1 loss function. This property smoothes the results of the generated deformations and is dependent on the scale of the mesh. The lower the value the, the less smoothing is applied. For a normal sized character mesh, a value of 1.0 is typical, on smaller meshes, lower values will provide better results. A value of 0.0 will disable the smoothing loss completely.

When the Mode property is set to Local, enabling this property will allow for the use of per bone and bone group masks by generating an influence mask per bone based on skinning information. This will enforce deformations localized to the area around the joint. Enabling this property increases localized joint deformation quality and can reduce the GPU memory usage, and improvesGPU performance.

Here you can set a value that will determine the cutoff for the Morph Target delta values. Delta values smaller than or equal to this threshold will be ignored and set to 0.0. This property is useful to remove small deltas from Morph Targets, which will lower the memory usage at runtime. However when this value is set too high, it can also introduce visual artifacts to your results. A value of 0.0 will result in the highest quality Morph Targets to be generated, at the cost of higher runtime memory usage.

Set the generated Morph Target’s compression level. Higher values will apply more compression, but could result in visual artifacts on the mesh. Lower values will apply less compression, resulting in high quality results, but will require more memory usage. It is recommended to use values between 20.0 and 200.0, for optimal results.

Set the channel data that represents the delta mask multipliers. You can use this property to break down and distribute influences on the ML Deformer in specific areas, such as targeting high motion joint seams, such as a character's neck or shoulders. The painted vertex color values will be used like weight multipliers on the ML Deformer deltas applied to that vertex. You can select a Mask Channel using the property’s drop-down menu. You can select from the following channels:

• Vertex Color Red: Uses the Red Vertex Color Mask channel.
• Vertex Color Blue: Uses the Blue Vertex Color Mask channel.
• Vertex Color Green: Uses the Green Vertex Color Mask channel.
• Vertex Color Alpha: Uses the Alpha Vertex Color Mask channel.

This property can be useful to target the training process on specific areas of your mesh you want to improve mesh deformations with unique settings, rather than training the entire mesh using the same settings.

• Consider increasing the number of frames of training data. It is recommended to use, at minimum, 3000 frames of training data for average characters, but more frames will result in better deformations. It is recommended for good results to have at least 7500 frames of training data.
• Your training data poses may not cover enough range of motion for your project. It is recommended to extend the character motion in your training data to cover all motions you need deformation coverage for.
• Ensure your training data Animation Sequence and Geometry Cache assets have matching frame numbers, frame rate playback, vertex counts for the meshes they have in common, and matching poses per frame.
• Ensure your Animation Sequence and Geometry Cache asset meshes are properly aligned to accurately capture vertex deltas. Verify your vertex deltas in Training Mode over several frames. Enable Draw Deltas.
• Set the Weight property to 1.0, and the Ground Truth Lerp to 0.0.
• Ensure your Morph Target’s normals are being computed at runtime. To fix this. either use a Deformer Graph, or enable the Skeletal Mesh asset’s Recompute Tangents property.
• Ensure the Max Num Training Frames property is not limiting your available training data.
• Set the Num Iterations property to a high value such as 5000. This value can be reduced later if need be.
• Your ML Deformer Inputs may contain too many bone references. Reduce this value to only the maximum number of animated bones necessary to capture your character's motion during training.
• When using the Local mode, you may be generating too many Morph Targets, or hidden layers, try reducing these values.
• Set the Batch Size property to 64 or 128.
• Disable the Smooth Loss Beta, or Regularization Factor property by setting a value of 0.0. If this solves the issue, try using a smaller value than you had previously. It is recommended to find the smallest value possible, while still retaining the level of quality your project needs.
• Set the Learning Rate to a value of 0.001, 0.0001 or 0.01. If you are working with a smaller mesh than a normal character mesh, such as a face, try adjusting the Smooth Loss Beta and Regularization properties in combination with these values.

If your GPU performance is low, try reducing the number of Morph Targets. It is recommended to have a total number of generated Morph Targets, for the whole character, between 64 and 256.

• Decrease the value for the Delta Zero Threshold to rescue the number of small delta values from your generated Morph Targets.
• Increase the Num Iterations property, to create more sparse Morph Targets.
• Increase the Regularization Factor property.
• When using the Global mode, try reducing the Num of Hidden Layers property.