Unreal Engine support for the Magic Leap One, Creator Edition is available now! The custom editor with integration is available both on the Epic Games Launcher and GitHub for you to download today.
Projects built with 4.19 will break when they are upgraded to 4.20 or higher.
Setup
Lumin SDK
Unreal Engine 4.23 supports the Lumin SDK version 0.19.0. Projects built with later versions of the Lumin SDK will fail. Unreal Engine 4.24 supports the Lumin SDK version 0.22.0 and Unreal Engine 4.26 supports the Lumin SDK version 0.24. See Lumin SDK Development Requirements for more.
First, we will need to install the Lumin SDK.
-
Download, install, and then run Magic Leap Package Manager from https://creator.magicleap.com.
-
Sign in to Magic Leap Package Manager with your registered email address and 6-digit code.
-
The default destination for Magic Leap packages is C:\Users\you\MagicLeap. To change this, click Settings and then specify a different path in the Download/Install Directory.
-
Select Lumin SDK under Common Packages,and then click Download & Install.
-
In the confirmation message at the bottom of Package Manager, click Apply changes.
Unreal Editor for Magic Leap
-
Grab the Magic Leap editor from the Epic Games Launcher by heading over to the Library tab.
-
Select Option from the Launch dropdown.
-
Select Android as a Target Plaform.
-
Select Lumin as a Target Platform, and click Apply.
-
Once installed, you will want to point to where you have installed the Lumin SDK. In the Project Settings menu, head over to Platforms > Magic Leap SDK and set the path to your Lumin SDK folder. (Example: C:\Users\you\MagicLeap\mlsdk\v0.x)
System Configuration
In order for UE4 to run commands from the SDK, you will need to set a few system variables (not user variables) in your computer’s Environment Variables. You can find this window in Control Panel > System > Advanced System Settings > Environment Variables...
First, you wil need to add an entry called “MLSDK”, which contains the path to the folder containing /mlsdk you installed above.
Second, you will need to add “%MLSDK%\internal\tools\mldb” to your PATH environment variable.
Finally, reboot your machine.
Sample Project
As part of our early support for the Magic Leap headset, we have put together a small sample project with examples of how many of the platform’s unique features are used in Unreal Engine.
Download the LuminSample project for use with the custom editor to reference when reading through the following feature docs.
Plug in the device and power it up. Note where the device initially boots up, as the nose position will be what the device chooses as your world origin in Unreal Engine, with the X-axis increasing as it moves away from the wearer. The LuminSample project assumes that the headset is started facing the user as all of the content is rotated 180 degrees.
Navigating the packaged LuminSample project requires the Controller. On the Main Menu, you will use the touchpad to highlight a specific example that you would like to check out. Once highlighted, press down on the touchpad to load the level. You can hide and show the Main Menu at any time by holding down the trigger and pressing the shoulder button.
Each sublevel is fully independent and can be loaded individually if desired.
Features
Player Location
Examples can be found in the PlayerLocationAndGaze map.
Getting the player’s head location is often used when building mixed reality experiences. We can access this by calling GetWorldTransform on the CameraComponent in our LuminSamplePawn - not unlike any other HMD in UE4.
If you are building your own pawn from scratch, you will need to add a Camera Component.
In LuminSample, the Bot from Robo Recall will look, aim, and shoot at you as you move around the scene. There is logic that will tell them to stop if you move behind them. The Drone will follow you as you move around the space.
Click for full image.
Have a look at the Lumin_Biped, Lumin_Drone and LuminSamplePawn Blueprints to see how these behaviors are hooked up. In short, we are driving Actor rotation or Anim Blueprints based on the result of FindLookRotation of the Actor that needs to face the player and the player position.
Click for full image.
If you are not using eye-tracking, you can use the CameraComponent’s forward vector to get an estimate of the player’s gaze and trace from that direction. Using this technique, the Drone will emote when you focus your gaze on it.
Click for full image.
Displaying World Origin
Example Found in LuminSamplePawn.
This one is fairly simple, but useful for understanding where your content “is.”
Knowing where the world origin is can help debug anything that you need to update in world space. As mentioned above, the world origin is set when the device first boots up and is right around the location of the nose. The default forward vector of the world origin is facing away from the headset in the direction of what would be the user’s gaze.
In LuminSample, we have added a boolean variable, bShowWorldOrigin, to LuminSamplePawn for you to toggle some simple debug information about where the world origin is. Set the default value of this variable in the pawn or toggle it at runtime to show the world origin.
Click for full image.
Gesture and Hand Tracking
Examples Found in LuminSamplePawn and GesturesAndTotem Map.
The Magic Leap One supports numerous gestures and tracks the position of certain points on hands when they are in view. By default, you must enable the gestures that you want to look for. You can set this at any time using the SetConfiguration node. Selecting fewer gestures helps the device focus on accurately recognizing the gestures you care about. We are doing this in LuminSamplePawn.
You can get gestures from either or both hands by calling GetCurrentGesture and checking the enum return. This can also be found in LuminSamplePawn.
You can also get location information of the center of the hand as well as many "Keypoints" that are found representing different parts of the hand such as a fingertip or knuckle. Using Keypoints (GetGestureKeypointTransform) and hand center (GetHandCenter), you have a lot of information about where parts of the user’s hands are and what they are doing. In the GesturesAndTotem map, we are toggling bShowGestureDebug on LuminSamplePawn to draw debug spheres on hand centers and keypoints every frame.
Taking the above information, it is possible to “touch” things in world space. In the GesturesAndTotem map, there is an InteractionSphere that is looking for overlaps, and we are toggling bTryTouchOverlap on LuminPlayerPawn to overlap on all Keypoints on both hands. There are probably more performant ways to go about this, but we wanted to show all available options when touching virtual objects.
Click for full image.
Looking for gestures comes with a performance cost, so keep that in mind as you plan out your content.
Controller Input
Examples found in LuminSamplePawn and GesturesAndTotem map.
For the most part, the Controller works the same way as other VR motion controllers. The Touchpad can be polled by checking GetMotionController (R) Thumbstick X, for left and right, and GetMotionController (R) Thumbstick Y for up and down. Also, the touchpad has a Z-axis value for pressure sensitivity. This scales from 0-1 based on how hard the user is pushing down on the touchpad.
The trigger operates the same way as well. You can poll the axis or grab the event. Look for GetMotionController (R) TriggerAxis and MotionController (R) TriggerAxis.
Like the others, the shoulder button uses the MotionController (R) Shoulder to get it’s pressed and released events.
All of this can be seen hooked up in LuminSamplePawn. In the case of LuminSample, we call Event Dispatchers that other Blueprints can subscribe to to take action when input is received. This is how we are controlling MainMenuActor and the TotemInputActor inside of our GesturesAndTotemContainer in the GesturesAndTotem map.
Room Scanning and Meshing
Examples found in WorldMeshing map and LuminMeshActor.
The Magic Leap One has the ability to scan the room and provide geometric data about the environment. We have worked this into our existing Mixed Reality Mesh (MRMesh) system for the platform, so developers can query the area and dynamically add content in the space.
What you will need is a Blueprint Actor with two components - MRMeshComponent and MeshTrackerComponent.
When you start scanning the world to get mesh data, you will want to associate these two components with each other using the ConnectMRMesh function where the MeshTracker is the Target and MRMesh is passed as the In MRMesh Ptr.
In the MeshTrackerComponent on LuminMeshActor, we have toggled a few settings to ensure that we are looking for the right things in the tracked space. You will see that we have edited the following in the Meshing > MagicLeap section in the Details panel:
-
Scan World to true
-
Mesh Type to Blocks
-
Meshing Poll Time to 0.5
-
Bounding Volume as Box Collision
-
Ignore Bounding Volume to true (to scan the whole room)
-
Planarize to true
The mesh reacts to physics the same way as any other collider. In the WorldMeshing map, a press of the shoulder button on the Controller will trace from the HMD’s forward vector into the world and draw debug information on the impact location and normal. We toggle TestWorldHit on our LuminSamplePawn to handle this.
Click for full image.
Microphone Input
Examples found in LuminSamplePawn and AudioExamples map.
The Magic Leap One has a built-in microphone that allows developers to capture the input for a variety of uses. In the AudioExamples map we are using an AudioCaptureComponent added into the LuminSamplePawn to monitor the microphone envelope values for a gameplay feature.
When activated, the component sends an event with the microphone envelope value. You can add an event from the component by selecting it from the components list and clicking the Green Plus box at the bottom of the Details pane.
In the example, players can blow to change the direction of the wind, the Kite and the Boy will then adapt to the new wind direction. We bind the Custom Event MicInputAudioValue to OnAudioEnvelopeValue to gather information from the device microphone only when we want to listen for it.
Click for full image.
Click for full image.
Spatialized Audio
Examples found in AudioActor_SoundSpawn and AudioExamples map.
When developing for the ML1 you can spatialize audio using the same techniques you would when authoring for other platforms. In addition to this workflow, we are using an actor specifically built for creating spatialized random sounds. This actor can be found in the sample content here:
.../LuminSampleContent/AudioExample/AudioBlueprints/AudioActor_SoundSpawn
The intent of this actor is to allow Sound Designers to shape a common ambient audio playback behavior through easy to understand parameters. The AudioActor_SoundSpawn actor can be configured to generate random sounds within a defined radius, or set to play a single sound in a very specific location. In the AudioExamples sample we are using this actor for 2 unique purposes. In the scene, two actors are set to create random call and response bird chirps. A third actor is being attached to the kite to create looping rustling sounds.
The birds are set to Auto Start and immediately begin playing. They are also set to Repeat, meaning they will continue to play the same sound using the set configuration after they have completed playing. None of the actors are set to Use Overlaps. When this bool is checked the actor will turn on and off as the player enters and exits the defined radius.
There are a number of settings to play with inside the actor. The settings allow you to control the time it takes between spawns of your sounds, the random number of sounds to spawn, volume, pitch, and most importantly the radius in which the sound will spawn. When this value is set the system will randomly select a location in X and Y coordinate space inside the radius. If you want to include random Z offsets, check B Include Height.
Later in the scene, we deactivate the bird audio simply by setting the Repeat bool to false in the Level Blueprint.
For the Kite idle audio we use a separate actor, AudioActor_KiteIdle, that is being attached at Begin Play.
AudioActor_KiteIdle is configured differently from the other audio actors. Here Auto Start is unchecked in the settings and we manually activate the audio using Execute Timer in the Level Blueprint.
This actor also has the Min and Max Timer values set very low to create a quick rustling loop. The Min and Max Spawns are set to 1 so we do not have overlapping sounds.
The Spawn Radius is also set to 1, to make sure the sound is localized to the Kite’s location.
Again, later in the scene, the sound is turned off by setting Repeat to false in the Level Blueprint.
Engine Settings
Vulkan and OpenGL
The Magic Leap One supports both OpenGL and Vulkan for rendering. You can toggle between these two options in Project Settings > Plugins > Magic Leap Plugin.
Renderer - Desktop or Mobile
You can also choose between the desktop and mobile rendering paths by choosing “Use Mobile Rendering” in Project Settings > Platforms > Magic Leap.
Desktop rendering and Vulkan are currently only supported on source builds and not in the binary (launcher) build. We expect both to be fully supported in the 4.20 binary.
Package, Deploy, Simulate
Launch
Like other platforms, you can launch directly to the device from the editor by clicking the drop-down next to Launch, then selecting your Magic Leap device.
Package
You can also package your .mpk via the toolbar under File > Package Project.
Launch will run the application directly on the device as expected. To install your packaged project, open the command line and run mldb install /yourproject.mpk.
You can then launch your application from the menu in the headset or run mldb launch from the command line. Of course, you will need to replace com.yourcompany.yourproject with the identifier that you have set up in Project Settings.
You must sign your applications and packages with a developer certificate generated from the Creator Portal to install or run them on Magic Leap One. You can find more information about Magic Leap Developer Certificates on their web site under Get a Developer Certificate. You can add a Magic Leap certificate in Project Settings under Platforms > Magic Leap > Distribution Signing > Certificate File Path.
Simulate
Alternatively, you can use Magic Leap Remote to iterate in the simulator in the event that you do not have a device. Head to Project Settings > Plugins > Magic Leap Plugin and check the Enable Zero Iteration checkbox. After an editor restart, you can play in VR Preview by selecting Play > VR Preview.
Terminal Commands
Here are some other useful MLDK terminal commands that we have used regularly that may be helpful:
-
mldb devices // shows list of recognized devices connected to your computer
-
mldb packages // shows packages installed on the device
-
mldb uninstall // uninstalls package from the device
-
mldb launch // launches the package if it is installed on the device
-
mldb terminate // kills the application if it is running
-
mldb reboot // reboots the devices attached
-
mldb shutdown // shuts down the device