Leveraging Brain Activity for Combat Systems
How can the player’s cognitive states contribute to game feel? Cinnamon & Antonyms is a top-down, melee-driven RPG that features layered combat systems. The game interfaces with an electroencephalogram (EEG) to modulate the avatar’s abilities based on the player’s level of concentration.
I began this project to memorialize my first foray into combat systems. Cinnamon & Antonyms was inspired by two talks at the 2022 Game Developers Conference: Ian Cofino’s “Dreamscaper: Killer Combat on an Indie Budget” and John Muñoz’s “Future Realities Summit: How NASA Has Translated Aerospace Research into Biofeedback Game Experiences.”
June 2022 — [Current]
Unreal Engine 4
WHAT KIND OF INFORMATION CAN WE GET OUT OF BRAIN ACTIVITY?
Understanding What You Can Do Versus What You Should Do
In my eleven years (!!) as an aspiring neuroscientist, there have only been a handful of moments when I'm reminded why I care so much about the brain—or rather, why my brain cares so much about herself. I was lucky to capture one of those moments during the early stages of this project, which I’ve clipped below.
A Side Note
It goes without saying that I tried a couple (read: a ton) of signal processing schemes before landing on this one. EEG headsets typically have built-in dimensionality reduction, which means that we’re aggregating signals from billions of neurons into a modest five channels. The main architecture of the processor involves a 1–30 Hz frequency band filter and fast Fourier transform (FFT) to isolate and categorize the spike waveforms. The resultant signal is a 5 x t matrix, where t represents the neural manifold over which the Kalman filter estimates the current state. As a general rule, smaller t values allow for higher fidelity but are more computationally intensive and susceptible to noise. Given that the Muse 2 headset samples at a rate of 256 Hz, it’s safe to choose a t value well into the double digits. We can make our math a little easier by using our FFT values to multiply by a 5 x t matrix of decoder weights, so we’re left with a single time-varying float value. And that’s brain control, folks!
Brain control is a longstanding fantasy within our cultural imagination, capturing the attention of tech bros with the budding field of ubiquitous computing and Trekkies with the near-future promise of psychokinesis. It speaks to a level of authority over ourselves and our environment that we will likely never achieve within our lifetimes.
So, why didn’t I make a fully brain-controlled game? Whenever I consider a new feature, I ask myself two questions. Firstly, what portion of the product’s resources (time, money, computational power, or otherwise) does this feature consume? Secondly, how much of my vision for this product can I achieve with this feature?
Cinnamon & Antonyms aims to deliver a visceral power fantasy that rewards strategy and improvisation. Based on that goal, I'd place brain control in the low-impact range with an astronomically high cost. Even when approaching new games, hardcore gamers have already accumulated a mental model for how they should interface with combat systems. We can capitalize on those heuristics by using genre conventions—such as standard movement controls—instead of incurring the cost of additional training. Peripheral devices also tend to have high latencies and rarely offer the fidelity for fast-paced gameplay. I'm not arguing that brain control in games is always bad—think about how it could be used as an awesome accessibility feature—but it’s not right for this product.
Capturing Brain Activity
To establish a wireless connection to the Muse 2 headset, I used three acquisition servers that leverage different forms of serial communication to successively process the incoming data. BlueMuse streams real-time EEG data through the Lab Streaming Layer (LSL) protocol. The OpenVibe acquisition client filters and writes the input stream to a Transmission Control Protocol (TCP) socket, which feeds into the UE4-BCI plugin.
Inside the Decoder
Now that we’ve gotten our signal into the engine, we have to decide how we want to interpret it. As shown on the right, the headset’s conductive sensor collects electrical signals from outside of our frontal lobe. Because we’re working with a non-invasive device, the fidelity of our signal is limited to large-scale patterns of activity rather than transient changes.
One such EEG pattern is the index of engagement, which describes a user’s level of focus. Researchers at NASA quantified the index of engagement by the formula β / (α + θ), for which beta, alpha, and theta describe discrete wave frequency bands.
HOW CAN WE LEVERAGE CERTAIN COGNITIVE STATES DURING COMBAT?
Theory of Flow
To leverage the player’s brain activity as a mechanic, the primary objective of Cinnamon and Antonyms’ gameplay should revolve around inducing changes in the player’s index of engagement. Psychologist Mihaly Csikszentmihalyi characterized the phenomenon of flow as a sense of immersion and energized focus that arises from certain forms of gameplay. He theorized that flow occurs when a game offers focus, freedom, feedback, and challenge. Using the index of engagement as a proxy for flow, I derived six design pillars that capitalize on the player’s cognitive resources to encourage flow states.
Players should be able to
understand the situation, tactically
assess options, and make
strategic decisions on the fly
Combat should feel fluid and
allow for multiple approaches
that can all work intuitively
Tough, But Fair
The players' fate is in their hands,
not the hands of the RNG system.
Player actions allow for correction and enemies feel balanced
Combat should drive variety and replayability through overlapping
Players should understand and
enjoy how their actions impact the
Players should feel like the
emotional and cognitive state they
bring into the game is reflected by
Genre conventions around the camera position inform players’ expectations of the speed and snappiness of combat. As a rule of thumb, the fidelity of gameplay typically decreases with camera distance; a farther camera reduces the weight of combat and often feels less cinematic, but the macroview provides the player with more information on their positioning. To balance the immediacy of melee combat and macroview required for ranged combat, the camera’s field of view decreases when enemies cluster around Cinnamon and increases when she draws her ranged weapon.
Dynamic camera field of view for melee (left) and ranged (right) combat
Due to the distance of the camera, Cinnamon & Antonyms selectively leverages color and values to draw the player’s eye towards certain targets. The values of the background, middle ground, and avatars are clamped to increasingly higher levels of contrast to establish a read hierarchy. Enemies, abilities, and the player character are also categorized into different regions of the color wheel to make them feel visually distinct.
Value (left) and color (right) draw the player's attention towards enemies and the player character
Feel and Animation
Six techniques emphasize the weight and readability of attacks.
Favoring Player Input - The player may cancel all attack animations to defensively break out of inputs
Secondary Motion - The human eye is attuned to natural movement. The cloth simulation on Cinnamon’s scarf alleviates the robotic movement of her base animations
Hitlag - Freezing both characters at the moment of contact sells the illusion of force and allows the player's eyes to register the collision
Zero Tween Time - The distance of the camera creates a need to convey large differences between attacks. Instead of blending between animations, hit reactions jump to the middle of the montage, which clearly telegraphs movement
Hit Flashes - The colors of hit flashes provide feedback on collisions and damage
Translate Forward - Forward momentum connects chains of attacks and feels more committal
Polishing the Camera Movement
The previous camera setup felt static and did not reflect the dynamism of Cinnamon's movements. To provide more passive motion, the new camera system connects to Cinnamon's hip socket instead of her capsule collider, which creates an organic sway in the camera's location based on her movement pattern. A slight camera lag dampens the sway and helps mitigate nausea.
To emphasize the difference in weight between Cinnamon's light and heavy attacks, a camera coom, subtle motion blur, and slow motion effect accompany heavy attacks.
Frame Data Breakdown
Physical interactions in Cinnamon & Antonyms are mediated by three types of colliders. Pushboxes (blue) convey the physical space occupied by the character and prevent overlap between game objects. The size of the pushbox changes based on the movement of the character—for instance, rolling shrinks Cinnamon's pushbox to half of its original height—but is otherwise static. Hurtboxes (yellow) describe when and where the character can be damaged; as shown in the images below, enemy hurtboxes are larger than their pushboxes to provide some leeway in the player's aim. Certain types of defensive abilities, such as rolling and dashing, may disable hurtboxes to create untargetable states. Collisions between the attacker's hitbox (red) and victim's hurtbox apply damage.
Pushbox (blue), hurtbox, (yellow), and hitbox (red) in the startup, active, and recovery frames of a melee attack
Notably, Cinnamon's hitbox is only enabled during the middle segments of the attack animation, followed by a recovery phase in which the player cannot apply damage. The duration of an ability's recovery is directly proportional to its damage output, introducing a tradeoff between the player's rate of queuing attacks and each attack's potential damage. This technique also prevents spam by injecting a delay between successive attacks, which encourages more purposeful combat.
Enemy states provide utility during combat by communicating how enemies respond to specific inputs. This form of diegetic tutorialization allows players to build mental models for enemy attack patterns and apply them to novel situations.
With the exception of her ultimate ability, all of Cinnamon’s attacks include soft targeting, in which the avatar automatically orients towards the nearest enemy during the first frame of an attack. Typical third-person combat systems require the player to control their character's yaw axis and location. Soft targeting alleviates the technical burden on the player to consider the avatar’s rotation and reduces the likelihood of missing attacks.
A Side Note
Why does hitlag work? Hitlags originated from fighting games as a means to emphasize the impact of certain attacks. But, if we think about our experience of collisions, hitlags don’t seem to have a real-world analog.
Imagine scattering a group of billiard balls during a game of pool. We perceive collisions through tons of different stimuli: we can hear the balls clacking together and feel vibrations from the cue stick, but the most telling indicator is an abrupt change in the objects’ momentum. Humans are great at estimating a moving object’s trajectory—after all, we have a lifetime of experience with physics—and we can use that estimation to gauge an object’s time-to-collision (TTC). However, physics systems in game engines often don’t align with our expectations for how objects should move in the real world. By maintaining the change in momentum over a longer period of time, hitlags create a margin of error that ensures the player’s TTC estimation falls within the duration of the lag. We register this effect as a collision even though it doesn't follow any of our other expectations of physical interactions.
Cinnamon’s basic attacks set the enemy into a stun state, which facilitates target prioritization and helps sell the weight of different strikes. The heavy hit state influences combat pacing by putting space between the player and enemy, hence providing an opportunity for the player to reposition. The popup state from Cinnamon’s ultimate attack sets the enemy into a juggle state, which encourages the player to press their advantage when they know the enemy cannot react.
Incorporating Brain Activity
Given that Cinnamon’s kit focuses on melee combat while the enemies’ kit focuses on ranged combat, the player requires gap-closing tools to engage enemies at a distance. Cinnamon’s ultimate ability is a near-instantaneous dash that provides temporary immunity to damage. If enemies are within the vicinity of Cinnamon at the end of her dash, they take area-of-effect damage and transition into a popup state. This combination offers a safe method of either engaging or disengaging from combat scenarios by allowing the player to reposition themselves without taking damage.
Cinnamon's ultimate ability is enabled by the player's index of engagement
Cinnamon’s ultimate is enabled by the player’s index of engagement crossing a predetermined threshold, represented by her scarf glowing. This threshold creates a positive feedback loop by offering deeper mechanics for increased levels of focus, which subsequently elicits more frequent flow states. Notably, the average signal produced by the player’s brain changes on an individual and daily basis. To normalize the ultimate across different players, the first ten seconds of gameplay incorporate a real-time calculation of the player’s flow threshold.
HOW CAN WE MAKE COMBAT SYSTEMS MORE RESPONSIVE TO NEURAL INPUT?
// Hey there, stranger. This section will be updated in September, 2022, so everything below this line is temporary. Thanks for taking the time to look through my work!
Resource management is one of the more subtle themes in Bloodline's gameplay; it emerges only when Bloodrush is on cooldown, forcing the player to rely on other systems before the ability becomes available. However, the player's unlimited supply of bullets leaves little reliance on Bloodrush.
To enhance the theme of resource management and thus add a new source of tension, future iterations of Bloodline will leverage heartbeat counts as currency. At its current state, Bloodrush is triggered regardless of the player's desires; players may feel that they wasted the ability during periods out of combat or wish to save the ability for another time. With a currency system, the player may spend a single beat on a bullet or ten beats to use Bloodrush. Not only will this condition provide players with more opportunities for skill expression, but it will also necessitate more frequent usage of the parkour movement system, which emphases satisfying traversal and drives heart rate changes.
Opacity vs. Depth
During one particular session, I explained the metrics behind Bloodrush and its threshold to one of my playtesters. The next time he played, he began holding his breath to increase his heart rate above the threshold. Though I initially conceptualized Bloodrush as an organic relief valve to high-tension scenarios, he demonstrated that some players want to control the game's pacing rather than allow it to be dictated by external systems.
Opacity surrounding Bloodrush engendered a unique sense of discovery; the player can feel when it should be triggered, but they are not told precisely when or what it does. However, that sensation wears off after the first-time experience. Systems that enable strategizing and skill expression—such as resource management—permit greater depth and longevity than short-term sensational experiences.
Bloodline now utilizes a currency system, in which the player may spend one heartbeat on a bullet or ten heartbeats on a Bloodrush effect. This modification forces the player to find new avenues for skill expression through the parkour system. It additionally includes updated UI and alerts to notify the player when Bloodrush becomes available, which enhances learnability without compromising the player's sense of discovery.
Bloodline's protagonist offers a high-agility, high-fragility power fantasy that facilitates rapid decision-making and frequent reruns. However, early feedback from playtesters suggested that the damage system was too unforgiving. I implemented a 5-cost shield that blocks all incoming damage for ten seconds to allow the player to experience longer sessions of fast-paced gameplay.
The shield, mapped to the E key, is indicated by a honeycomb overlay
Progression systems create a positive feedback loop between the player's and avatar's mastery. Player progression—the player improving their knowledge of the game—often facilitates abstracted progression—in-game statistics that evolve a character's abilities. Think about snowballing in League of Legends; the autotelic reward of landing Yasuo's wombo-combo provides more gold to launch even deadlier attacks. Once you hit your 0/7/0 power spike, much of the apprehension of the early game subsides, producing a faster pacing and more frequent opportunities for emotional payoff.
Bloodline takes a different approach to progression systems by eliminating abstracted progression. The heartbeat system demands a consistent difficulty and pacing within levels; if the player finds a level too easy, their sense of urgency dissipates, their heart rate slows, and they run out of bullets. If the player finds a level too hard, their sense of urgency skyrockets, their heart produces an infinite supply of bullets, and we're back to square one. Therefore, Bloodline must deliver the end goal—the feeling of complete mastery over the character—without altering the game's pace.
A Balancing Act
The ten-second shield over-compensated for the unforgiving damage system. The following changes have been implemented to maintain a consistent difficulty level and fast pace.
- Enemy firing period decreased from 2 seconds to 1.25 seconds
- Damage increased from 33 to 40
- Shield cost increased from 5 to 10, and Bloodrush cost increased from 10 to 20
- Shield blocks all damage for 5 seconds, then blocks half of all damage for 5 seconds
- Bloodrush decreases the camera's field of view to improve the player's aim
As an added bonus for the most resourceful Bloodline players, headshots kill enemies instantly.
Bloodline is my first foray into the more analytical approach to design that underlies most of the games I love. Though I have only been making games for one year, the kind of designer I am and the kind of player I am have emerged as very different people. Bloodline represents my attempt to bring those two individuals closer together. I would love to expand the core mechanics of Bloodline into a multiplayer setting, and I invite you to check back on this page in a couple months as I sort out my motivations as a designer.