Introducing the Machine Learning Cloth Simulation System
The Machine-Learning (ML) Cloth Simulation system provides high-fidelity, highly performant real-time cloth simulation in Unreal Engine.
The system provides higher fidelity than the traditional physics-based model by using a trained dataset that can be used in real-time to produce results that were previously only achievable with offline simulation.
Benefits of using Machine Learning
In traditional game development, you simulate character clothing by using a physics solver. However, this process can be computationally intensive due to the time-consuming updates for elasticity, and the complicated interactions between cloth and body.
A machine learning system can assist users in training a model using high-quality simulation data generated in advance. This system is capable of producing clothing meshes that are of similar quality to pre-simulated data, while being fast and efficient in terms of memory usage.
Technical Implementation
For each frame, the ML Cloth Simulation system takes the character's skeletal pose as input and predicts a final cloth pose by deforming a base cloth mesh using linear blend skinning. The deformation has two major components.
The first component is a low-frequency deformation similar to the neural morph model, but instead of outputting vertex deltas, the network outputs a set of principal component analysis (PCA) coefficients.
Using this technique, the PCA delta is calculated as follows:
pca_delta = mean_delta + pca_coeff * pca_basis
This component is modeled with a multi-layer perceptron (MLP) network, which predicts PCA coefficients using the Skeletal Mesh pose as input. To train this component, you should prepare a dataset of random poses and simulate clothing to settle on each pose. For this component, we recommend 5000 poses for a humanoid character.
The second component is a high-frequency deformation computed using a Nearest Neighbor model, which is a variation of the ML Deformer model.
This component takes the PCA coefficients from the first component and applies a nearest neighbor search in a Nearest Neighbor dataset. For this component, one should prepare a small, but diverse set of poses and simulate clothing to settle on each of the poses. Ideally the poses should be taken from some key frames of the animations in-game. For optimal results, between 50 and 100 poses are recommended for a humanoid character.
Using this technique, the vertex delta is calculated as follows:
vertex_delta = pca_delta + nearest_neighbor_delta
For the trained dataset, you can generate the final cloth position for each pose with any cloth simulation solver (such as Houdini Vellum). For each pose, simulate the cloth until settlement and save the settled cloth in the Geometry Cache. You can import the cache into Unreal Engine as an Alembic file.
For best results, you should manually choose the most relevant poses for the dataset. However, it is possible to do automatic pose selection with the KMeans Pose Generator. The generator takes an animation sequence and runs the KMeans algorithm to generate a list of entries for the Nearest Neighbor dataset.
Current Limitations
The current system implementation has one main limitation - The clothing is assumed to be quasi-static. This means the system can predict shape details like wrinkles but cannot predict dynamic motion like swinging or draping.
This system is currently best suited for tight-fitting clothing like pants and T-shirts, but not loose garments like dresses or capes.
You can learn how to use the ML Cloth system by following the ML Deformer - Nearest Neighbor Model tutorial on the Epic Developer Community.