Drive Gameplay with Structured Output

Structured Output gives a deeper dimension to your gameplay with NPCs. Structured Output is a mechanism through which the LLM backing your NPC returns user-defined, LLM instantiated structs to drive gameplay logic. 

This means your LLM character can have deeper interactions, such as return answers to multiple choice questions, and decide if the price of an item is worth the cost, and more.

Overview

This section customizes your persona_npc_behavior with a single Structured Output for your persona. The interactions_response struct:

• Records the number of times this NPC has interacted with the player.

• The current feeling the persona has about this interaction with the player.

• An array recording the history of the persona's feelings about all interactions with the player.

Structured Output requires that you:

• Create a struct defining the Structured Output.

• Provide instructions for how the persona should fill the Structured Output struct.

• Define and subscribe to an event signaled upon receipt of Structured Output.

You can configure multiple Structured Outputs on a single persona at the same time and specify different subscriptions for each type of Structured Output for your persona.

Supported Types

Structured Output supports the following Verse types as fields inside structs:

• int

• float

• bool

• enum

• message

Displaying Messages

Message types are redacted when printed to the log or in-game, but you can display message types using devices that are designed to display them, such as:

• billboard_device

• hud_message_device

• popup_dialog_device

Prerequisites

• Add an NPC to a Conversation

Add Structured Output to NPC Behavior Script

The code below uses Structured Output to extend the capabilities of multiple LLMs by using a prompt_binding_definition type and RegisterAction() function to introduce new response types based on player prompts.  

Prompt Binding is a struct given to the AI to register an action defined in the RegisterAction() function. Together these tell the LLM when to do a certain action or why a certain action should be used based on a player’s prompt.

Below is a Verse script containing an expanded persona_npc_behavior in the section persona_npc_behavior.verse. This expands upon the code in the Add an NPC to a Conversation documentation including Structured Output. 

To amend the code in your existing persona_npc_behavior, follow these steps:

1. Ensure that you have your Verse Project open in VS Code:

   1. In the Menu Bar, select Verse > Open Project in VS Code…

   2. Once VS Code opens, find your persona_npc_behavior.verse file in the VS Code Explorer.

2. Copy and paste the content of persona_npc_behavior.verse below into your Verse file in your project.

3. Compile your Verse code either in the UEFN Toolbar or within VS Code.

For a step-by-step walkthrough of the Verse script, follow the steps under Explanation of Script Changes. This section picks up immediately after where the script explanation in Add Persona to NPC Character left off.

persona_npc_behavior.verse

using { /Fortnite.com/AI }
using { /Fortnite.com/Devices }
using { /Fortnite.com/Playspaces }

using { /UnrealEngine.com/Conversations }
using { /UnrealEngine.com/Temporary/Diagnostics }

using { /Verse.org/Chat }
using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }

Explanation of Script Changes

1. A struct named interactions_response holds variables that the functions pass to the LLM. The LLM uses the variables to evaluate the prompt and how to respond.

   Verse

   interactions_response := struct:
       @ai_description("Use the Number field to output the number of times you have spoken to this player.")
       Number:int
    
    
       @ai_description("Use the Feeling field to output how you are feeling about your current interaction with the player.")
       Feeling:message
   

   Copy full snippet(7 lines long)

2. Two different @ai_descriptions tell the LLM what the variables are and the reason the LLM should use them when creating a response.

   Verse

       @ai_description("Use the Number field to output the number of times you have spoken to this player.")
       Number:int
    
    
       @ai_description("Use the Feeling field to output how you are feeling about your current interaction with the player.")
       Feeling:message
   

   Copy full snippet(6 lines long)

3. OnInteractionsResponse is the function that drives gameplay logic using Structured Output. The callback that fires whenever the AI session produces structured output matching the interactions_response struct. It's the handler registered by RegisterAction in AddStructuredOutput(), and its Response parameter carries the data the model filled in.    

   Verse

   OnInteractionsResponse(Response:interactions_response):void =
           Logger.Print("Structured response received")
           Logger.Print("Number of Interactions: {Response.Number}")
           Logger.Print("Feeling: {Localize(Response.Feeling)}")
           # Drive gameplay logic here
   

   Copy full snippet(5 lines long)

4. The AddStructuredOutput function registers the NPC's structured-output action with its AI session. This provides a way for the model to report interaction data back into gameplay. This code does two things. First it builds a prompt_binding_definition, a small descriptor giving the action a Name and a Description (both wrapped through the MakeMessage helper to turn the string literals into message values). The description text is what tells the model when to use this action: here, whenever the player has spoken to the NPC.

   Then it calls Session.RegisterAction, passing the binding, true, the interactions_response struct type, and the OnInteractionsResponse handler. This tells the session that when it emits structured output shaped like interactions_response, it should route that data to OnInteractionsResponse. RegisterAction returns a cancelable, which is wrapped in option{} and stored in InteractionsResponseSubscription so the subscription can be torn down in the OnEnd method.

   Verse

       AddStructuredOutput():void =
           if:
               NPCEntity := GetEntity[]
               PersonaComponent := NPCEntity.GetComponent[persona_component]
                Session := PersonaComponent.GetAISession()
           then:
               # Create a prompt_binding so that we can use it on RegisterAction
               InteractionBinding:prompt_binding_definition = prompt_binding_definition:
                   Name := MakeMessage("InteractionBinding")
                   Description := MakeMessage("Signals whenever the player has spoken to you.")
   
   Expand code Copy full snippet(14 lines long)

5. OnBegin adds the structured output to the session and sets the player as the conversation target. In a game where players can join late, you want to ensure you add them on the PlayerAddedEvent or when a player walks up to an NPC via a trigger.

   Verse

       OnBegin<override>()<suspends>:void=
           AddStructuredOutput()
           AddPlayersAsConversationTarget()
   

   Copy full snippet(3 lines long)

6. In OnEnd the InteractionsResponseSubscription is cleaned and removed from all players. This is good practice.

   Verse

       OnEnd<override>():void=
           if (Subscription := InteractionsResponseSubscription?):
               Subscription.Cancel()
           for (Player : GetEntity[].GetPlayspaceForEntity[].GetPlayers()):
               RemovePlayerAsConversationTarget(Player)
   

   Copy full snippet(5 lines long)

Test Structured Output

The NPC is now configured with Structured Output. Every time the persona speaks, the interactions_response is signaled.

Test your LLM character's behavior for more engaging responses and reactions to your prompts in-game.