LLM Caption UI

This section looks specifically at capturing an LLM’s dialog and binding it to a caption UI in a user widget. 

To create captions in the Persona template, the following files are used:

• WB_Score.uasset

• ui_manager.verse

• persona_behavior.verse

Setting Up the User Widget

The WB_Score user widget is where the captioned dialog is generated. The widget contains all of the UI gameplay elements of the Player HUD, including the captions. The closed caption design and text are created inside an Overlay widget. The Overlay named CaptionOverlay controls the visibility of the caption UI:

• An Image widget named CaptionBackground creates the background for the captions.

• A Text Block widget named Captions displays the caption text.

The user widget has two important variables that provide a way to modify the caption elements at runtime. The variables are CloseCaption (a message variable) and CCVisible (a logic variable).

CloseCaption is bound to the Captions widget Text in the Details panel.

The CCVisible variable is bound to the CaptionBackground widget and Captions widget Visibility behavior in the Details panel.

These bindings and their specific setups provide a way to control when the Captions become visible in the UI and what the UI displays. This functionality comes from the ui_manager Verse script.

Verse Setup

The InitializeUI function adds the user widget to the player’s HUD.

var HUD:UIAssets.WB_Score = UIAssets.WB_Score{}


    # Function that will add the HUD to the Player's screen, and handle all required initialization for the UI
    InitializeUI():void =
        if:
            Player := GetPlayspace().GetPlayers()[0]
            TargetUI := GetPlayerUI[Player]
        then:
            . . .

The ShowCaptions and HideCaptions functions are created to modify the bound variables created in the user widget.

    # Function that allows other Verse classes to show and modify the Captions
    ShowCaptions<public>(IncomingCaption:message):void=
        # Set 'CloseCaption' to 'IncomingCaption'. This variable is the direct message we use in the Caption's Widget element
        set HUD.CloseCaption = IncomingCaption
        # Set 'CCVisible' to true. This sets the Overlay element that holds the caption to be shown
        set HUD.CCVisible = true


    # Function that allows other Verse classes to hide and clear the Captions
    HideCaptions<public>():void=

These functions are used and bound to events in the PersonaComponent. The PersonaComponent has events that help to determine the exact moment the LLM has successfully passed a message — when the LLM begins speaking, and when it stops speaking. 

These events are used to modify the Captions text and control the visibility of the closed captions.

This code runs in the persona_behavior.verse device during the OnBegin function, which provides a way to bind the CommitSayEvent and StopSayEvent to helper functions that communicate the text changes to the ui_manager.verse device.

            # Subscribe 'SetCloseCaptions' to the event 'CommitSayEvent'. This allows the NPC's last spoken message to be displayed on screen
            PersonaComponent.CommitSayEvent.Subscribe(SetCloseCaptions)
            # Subscribe 'ClearCloseCaption' to the event 'StopSayEvent'. This allows us to clean all of our Close Caption variables for the next message
            PersonaComponent.StopSayEvent.Subscribe(ClearCloseCaptions)

These functions are required as intermediaries between the events of the PersonaComponent and the UIManager.

    # This function calls ShowCaptions() to make the Captions element visible and set the message inside it to the latest talk point from the NPC
    SetCloseCaptions(NPC:?agent, Message:message):void=
        UIManager.ShowCaptions(Message)
   
    # This function clears the contents of the Captions element and hides it when the NPC is done talking
    ClearCloseCaptions(Interrupted:logic):void=
        UIManager.HideCaptions()

Everything the LLM says is caught in a closed caption box and displays in the HUD.

Complete Script

Below is the complete script for the ui_manger device.

using { /Fortnite.com/Devices }
using { /Verse.org/Simulation }
using { /UnrealEngine.com/Temporary/Diagnostics }
using { /UnrealEngine.com/Temporary/UI }
using { /Verse.org/Input }
using { /Verse.org/Input/UI }
using { /Verse.org/Simulation/Tags }

# See https://dev.epicgames.com/documentation/en-us/uefn/create-your-own-device-in-verse for how to create a verse device.

  Each page below is independent of the other pages. Click the topic you're interested in knowing more about.  

• 

  LLM Characters as Gameplay Drivers

  Drive gameplay using LLM characters.

• 

  LLM Game Mechanics with Verse

  Use Verse and Structured Output to refine LLM output.