Perception with Scene Graph

Core Concepts

The Perception API captures everything that could be considered detection or a “sense” in–game to help drive engaging, interactive experiences. Every NPC that chases a player, every security camera that detects intruders, every creature that reacts to sound — each one needs a way to perceive this information and decide what to do about it.

The Perception API functions as a suite of sensors that serve as an NPC’s senses in-game. These senses are each represented by a pair of components that all report detections in one common shape:

• An EMITTER component on a thing that can be sensed (a sneaking player, a ringing phone, a pressure plate).

• A DETECTOR component on a thing that senses (a guard's eyes, ears, or skin).

• A SUBSCRIPTION from any receiver entity, with a filter, that says which detections it wants.

• A common event — every sense raises a sense_detection_event subclass that carries DetectedEntity and a DetectionState.

A sight event, a hearing event and a touch event all derive from sense_detection_event, and they all use the same three-state lifecycle (Active, Inactive, Instant). Code written to consume one sense is a template for the others.

Terminology

Following are some useful terms that you should become familiar with to better understand how the API works:

• Perception — an observation of the world around an entity.

• Sense — a specific type of perception (sight, hearing, touch, smell, etc).

• Emitter — a component that marks an entity as something that can be perceived through a given sense.

• Detector — a component that marks an entity as something that perceives through a given sense.

• Receiver — an entity registered with a detector to be notified of perception events.

• Represented entity — The identity of the detected object. Usually the emitter's own entity, but an emitter can stand for something else, e.g. a held gun counts as the character holding it.

• Sense data — a sense-specific description of what was perceived (for example, how clear, how loud, or the collision that happened.

• Filter — a function or expression provided by a receiver that the framework reevaluates against every perception event to specify which events are sent to the receiver. For example,only passing on noises above a certain loudness threshold.

Emitters and Detectors

The AI Perception API allows you to define how entities sense the world and report their perceptions. It provides implementations for the most common senses (sight, hearing, touch) and a framework for you to build your own senses using the same primitives.

It can be simply summarized as consumers subscribe to a detector with a filter and react to perception events.

To make an event such as a footstep perceivable, the event must emit data for a specific sense that can be detected. The typical flow is:

• An entity with an attached emitter component can be perceived by the sense it emits. For example, attaching a sight_emitter_component onto an entity allows it to be perceived by sight.

• Another entity with an attached detector component for that specific sense can perceive the sense data emitted.

• Once the emitted sense is perceived, the perceiving entity will respond to the sensory data. For example, this could be a guard spotting a player, moving toward a sound source, or a pressure trap being triggered.

You can use the API in three ways:

• EPIC built-in senses — sight, hearing, and touch exist as ready-made emitter/detector pairs. Wire them up and go for immediate use or use them as inspiration to make your own senses!

• Custom spatial senses —  Make custom senses by subclassing component, implement a spatial_sense_emitter / spatial_sense_detector pair, and call RegisterEmitter() / RegisterDetector(). The framework's spatial pipeline handles overlap dispatch.

• Custom senses from scratch — Write your own components and event, and send the event to subscribers yourself .

Senses

Senses in the AI Perception API are broken down into several key aspects:

• Sense data, such as  intensity of sense, position, and so on.

• Emitter and detector components, based on information like how the data is transmitted and detected and how the received data is filtered.

• Detection events and logic related to how a sense can be perceived.

Epic provides three senses by default: sight, hearing, and touch.

Sight

Sight is handled by the paired sight_emitter_component and sight_detector_component. The sight_emitter_component designates the entity it is attached to as something that can be seen. The sight_detector_component is a sensor that detects stimuli in a cone in a given range and angle, these are defined in Verse. The detector reports Active when an emitter enters the cone and has line of sight, and Inactive when it leaves or is obscured from line of sight. 

The general setup follows the following format:

1. On the observer, add a sight_detector_component and set HalfAngle and Range.

   1. Optionally, you can set  HysteresisAngle - the additional radians beyond HalfAngle before Active transitions to Inactive, the HysteresisRange - the additional centimeters beyond Range before Active transitions to Inactive and NearClipRadius, the radius in centimeters where no emissions are detected. HysteresisAngle, HysteresisRange and NearClipRadius default to 0.

   2. Angles are defined in radians (half-angle clamps to 0 - pi/2), distances are defined in centimetres. Negative hysteresis and near-clip values clamp to zero.

2. On the thing to be seen, add a sight_emitter_component. Passive emission is on by default.

3. Subscribe a receiver to the detector.

WARNING:

The following example is a sight detector that logs what a guard sees when patrolling.

using { /Verse.org/Native }
using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }
using { /Verse.org/SpatialMath }
using { /Verse.org/Perception }

AcceptAllSight(E:sight_detection_event)<decides><reads>:void = true?

sight_logger_component<public> := class<final_super>(component):
    var Subscription<private>:?cancelable = false

You can test the sight detector by creating an intruder entity for the detector to detect:

sight_demo_setup_component<public> := class<final_super>(component):
    OnSimulate<override>()<suspends>:void =
        if (Sim := Entity.GetSimulationEntity[]):
            Origin := Entity.GetGlobalTransform().Translation

            GuardEntity:entity = entity{}
            GuardEntity.AddComponents(array{
                sight_detector_component{Entity := GuardEntity, HalfAngle := 0.8, Range := 2000.0},
                sight_logger_component{Entity := GuardEntity}})
            Sim.AddEntities(array{GuardEntity})

Hearing

A hearing_emitter_component marks something that makes sound. A hearing_detector_component marks something that listens. 

There are two ways a sound reaches a detector:

• PASSIVE — An emitter with PassiveEmissionEnabled set to true is continuously audible to detectors it overlaps. The detector sees Active when they begin to overlap and Inactive when they stop.

• PULSE — Calling Emit(hearing_data) on an emitter sends a one-shot Instant event to every detector it overlaps at that moment.

The general setup follows the following format:

• Put the hearing_detector_component on an entity that has a mesh_component. The mesh's shape is the listening zone (its bounding box if the mesh is not queryable). The mesh does not need to block anything; an overlap-only mesh such as one using the Trigger collision preset works.

• Put the hearing_emitter_component on the sound source. PassiveEmissionLoudness and PassiveEmissionSoundTags are @editable.

• Subscribe a receiver to the detector. The first subscriber enrolls the detector.

IMPORTANT:

The following example is a hearing detector that logs when it hears a pulse.

using { /Verse.org/Native }
using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }
using { /Verse.org/SpatialMath }
using { /Verse.org/Perception }

AcceptAllHearing(E:hearing_detection_event)<decides><reads>:void = true?

hearing_logger_component<public> := class<final_super>(component):
    var Subscription<private>:?cancelable = false

This example creates a sound source that pulses once, and one that is passively audible:

clacker_component<public> := class<final_super>(component):
    OnSimulate<override>()<suspends>:void =
        Sleep(3.0)
        if (Emitter := Entity.GetComponent[hearing_emitter_component]):
            Emitter.Emit(hearing_data{Loudness := 80.0, Position := Entity.GetGlobalTransform().Translation})

hum_setup_component<public> := class<final_super>(component):
    OnSimulate<override>()<suspends>:void =
        if (Sim := Entity.GetSimulationEntity[]):
            HumEntity:entity = entity{}

Touch

A touch_detector_component listens to collisions on its own meshes. When one of those meshes touches something carrying a touch_emitter_component (or a child of an entity that carries one), the detector reports Active. When the last touching pair separates, it reports Inactive. Touch is continuous: it never raises Instant.

The general setup follows the following format:

• The detector entity needs a collidable mesh. Either leave Meshes empty and use the mesh_component on the detector's entity, or fill Meshes with several meshes (when detecting a complex object such as a vehicle).

• The thing to be touched needs a collidable mesh and a touch_emitter_component on that entity or an ancestor.

• Subscribe. The first subscriber makes the detector wire its collision listeners; the last cancel tears them down.

• Something must make the two meshes actually collide. Touch is fed only by Scene Graph collision events between meshes.

The following is a simple example that logs detected touches.

using { /Verse.org/Native }
using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }
using { /Verse.org/SpatialMath }
using { /Verse.org/Perception }

AcceptAllTouch(E:touch_detection_event)<decides><reads>:void = true?

touch_logger_component<public> := class<final_super>(component):
    var Subscription<private>:?cancelable = false

A further example shows how you can create a scoring pattern on touch. Coins carry a touch_emitter_component. The collector carries a touch_detector_component. Touching a coin scores and removes it. Removing the coin also closes its contact, so an Inactive follows. The default clause ignores it.

using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }
using { /Verse.org/SpatialMath }
using { /Verse.org/Perception }

AcceptAllTouch(E:touch_detection_event)<decides><reads>:void = true?

coin_collector_component<public> := class<final_super>(component):
    var Score<public>:int = 0
    var Subscription<private>:?cancelable = false

Custom Spatial Senses

You can create custom spatial senses with the Perception API to extend beyond the standard, Epic-provided senses. These senses are overlap-driven, and function similarly to hearing. The system reports when an emitter and detector start or stop overlapping; your sense decides what data to send and who receives it once it has passed through a filter. 

Spatial senses follow the same format as preexisting senses:

• A spatial_sense_emitter - A component you write that implements spatial_sense_emitter on a thing that can be sensed

• A spatial_sense_detector - A component you write that implements spatial_sense_detector on a thing that senses

• A SUBSCRIPTION from the receiver entity, with a filter, that says which detections it wants.

• A sense_detection_event where the Spatial sense extends the sense_detection_event subclass that carries a DetectedEntity and a DetectionState.

The perception system tracks where each enrolled emitter and detector is, using the same Scene Graph mesh_component volumes and points as hearing, and calls the detector's OnSpatialChange when an emitter starts or stops overlapping it. The sense author supplies everything else: the event, its data, subscriptions, filtering and delivery.

For a custom sense, decide deliberately which side carries the volume: 

• Emitters that have an area of effect (a scent cloud) carry the mesh, and detectors can be points.

• Detectors that have a sensing zone carry the mesh, and emitters can be points.

A detector declares an EmitterType. It perceives every enrolled emitter whose class is that type or a subtype of it within the same simulation. 

A custom emitter that derives from a built-in emitter is therefore perceived by that derived sense's built-in detectors.

IMPORTANT:

The Rules of a Custom Sense

1. The emitter calls RegisterEmitter[] once it is in the scene within OnBeginSimulation and cancels the returned handle in OnEndSimulation.

2. The detector calls RegisterDetector[] when it gets its first subscriber and again on scene re-entry if it still has subscribers and cancels when the last subscriber leaves.

   1. Both calls FAIL, with no handle, if the component is already enrolled in that role, if the implementer is not a component, or if the game isn’t running (registration only works during a session, not in the editor viewport). Keep the handle only on success so a refused attempt can be retried.

3. OnSpatialChange runs inside the transaction. It can record the event but it cannot call SendDown. Record to a pending list and deliver from the detector's own OnSimulate.

   1. OnSpatialChange runs inside a transaction, so it can't call SendDown. Record the event there, and deliver it from a loop in the detector's OnSimulate that checks once per tick.

4. sense_detection_state is an open enum, so every case needs a default clause.

5. Always cancel registration in OnEndSimulation. A component destroyed without cancelling leaves its detections open forever because the owed Inactive has no component left to name.

The following is an example of a custom spatial sense, scent.

scent_data<public> := class<castable>:
    Strength<public>:float = 1.0

scent_detection_event<public> := class(sense_detection_event):
    Emitter<public>:scent_emitter_component
    Detector<public>:scent_detector_component
    DetectedEntity<override>:entity
    DetectionState<override>:sense_detection_state
    ScentData<public>:scent_data

This example extends the scent sense idea to a scent hound. A prey animal leaves a scent cloud: its entity has a mesh_component for the cloud and a scent_emitter_component. The hound has a scent_detector_component and no mesh, so it is a point. It smells the scent when its point is inside the cloud's volume.

using { /Verse.org/SceneGraph }
using { /Verse.org/Simulation }
using { /Verse.org/SpatialMath }
using { /Verse.org/Perception }

AcceptAnyScent(E:scent_detection_event)<decides><reads>:void = true?

hound_component<public> := class<final_super>(component):
    var ScentsInRange<public>:int = 0
    var Subscription<private>:?cancelable = false