Leveraging Brain Activity for Combat Systems
How can the player’s cognitive states contribute to game feel?
Cinnamon & Antonyms is a top-down, melee-driven roguelike that features layered combat systems. The game interfaces with an electroencephalogram (EEG) to modulate the avatar’s abilities based on the player’s level of concentration.
I began this project to apply my background in neuroscience to game design and memorialize my first foray into combat systems. Cinnamon & Antonyms was inspired by two talks at the 2022 Game Developers Conference: Ian Cofino’s “Dreamscaper: Killer Combat on an Indie Budget” and John Muñoz’s “Future Realities Summit: How NASA Has Translated Aerospace Research into Biofeedback Game Experiences.”
May 2022 — [Current]
Unreal Engine 4
WHAT KIND OF INFORMATION CAN WE GET OUT OF BRAIN ACTIVITY?
Understanding What You Can Do Versus What You Should Do
In my eleven years (!!) as an aspiring neuroscientist, there have only been a handful of moments when I'm reminded why I care so much about the brain—or rather, why my brain cares so much about herself. I was lucky to capture one of those moments during the early stages of this project, which I’ve clipped below.
A Side Note
It goes without saying that I tried a couple (read: a ton) of signal processing schemes before landing on this one. EEG headsets typically have built-in dimensionality reduction, which means that we’re aggregating signals from billions of neurons into a modest five channels. The main architecture of the processor involves a high-frequency band filter and fast Fourier transform to isolate spike waveforms. The resultant signal is a 5 x t matrix, where t represents the neural manifold over which the Kalman filter estimates the current state. As a general rule, smaller t values allow for higher fidelity but are more computationally intensive and susceptible to noise. Given that the Muse 2 headset samples at a rate of 256 Hz, it’s safe to choose a t value well into the double digits. We can make our math a little easier by multiplying by an empirically-determined 5 x t matrix of decoder weights, so we’re left with a single time-varying float value. And that’s brain control, folks!
Brain control is a longstanding fantasy within our cultural imagination, capturing the attention of tech bros with the budding field of ubiquitous computing and Trekkies with the near-future promise of psychokinesis. It speaks to a level of authority over ourselves and our environment that will likely remain unresolved within our lifetimes.
So, why didn’t I make a fully brain-controlled game? Whenever I consider a new feature, I ask myself two questions. Firstly, what portion of the product’s resources (time, money, computational power, or otherwise) does this feature consume? Secondly, how much of my vision for this product can I achieve with this feature?
Cinnamon & Antonyms aims to deliver a visceral power fantasy that rewards strategy and improvision. Based on that goal, I'd place brain control in the low-impact range with an astronomically high cost. Even when approaching new games, hardcore gamers have already accumulated a mental model for how they should interface with combat systems. We can capitalize on that model by using genre conventions—such as standard movement controls—instead of incurring the cost of a tutorial level. Peripheral devices also tend to have high latencies and rarely offer the fidelity for fast-paced gameplay. I'm not arguing that brain control in games is always bad—think about how it could be used as an awesome accessibility feature—but it’s not right for this product.
Capturing Brain Activity
To establish a wireless connection to the Muse 2 headset, I used three acquisition servers that leverage different forms of serial communication to successively process the incoming data. BlueMuse streams real-time EEG data through the Lab Streaming Layer (LSL) protocol. The OpenVibe acquisition client filters and writes the input stream to a Transmission Control Protocol (TCP) socket, which feeds into the UE4-BCI plugin.
Inside the Decoder
Now that we’ve gotten our signal into the engine, we have to decide how we want to interpret it. As shown on the right, the headset’s conductive sensor collects electrical signals from outside of our frontal lobe. Because we’re working with a non-invasive device, the fidelity of our signal is limited to stable, large-scale patterns of activity.
One such EEG pattern is the index of engagement, which describes a user’s level of focus. Researchers at NASA quantified the index of engagement by the formula β / (α + θ), for which beta, alpha, and theta describe discrete wave frequency bands.
HOW CAN WE INDUCE CERTAIN COGNITIVE STATES THROUGH COMBAT?
Theory of Flow
To leverage the player’s brain activity as a mechanic, the primary objective of Cinnamon and Antonyms’ gameplay should revolve around inducing changes in the player’s index of engagement. Psychologist Mihaly Csikszentmihalyi characterized the phenomenon of flow as a sense of immersion and energized focus that arises from certain forms of gameplay. He theorized that flow occurs when a game offers focus, freedom, feedback, and challenge. Using the index of engagement as a proxy for flow, I derived six design pillars that capitalize on the player’s cognitive resources to encourage flow states.
Players should be able to
understand the situation, tactically
assess options, and make
strategic decisions on the fly
Combat should feel fluid and
allow for multiple approaches
that can all work intuitively
Tough, But Fair
The players' fate is in their hands,
not the hands of the RNG system.
Player actions allow for correction and enemies feel balanced
Combat should drive variety and replayability through overlapping
Players should understand and
enjoy how their actions impact the
Players should feel like the
emotional and cognitive state they
bring into the game is reflected by
Genre conventions around the camera position inform players’ expectations of the speed and snappiness of combat. As a rule of thumb, the fidelity of gameplay typically decreases with camera distance—a farther camera reduces the weight of combat and often feels less cinematic, but the macroview provides the player with more information on their positioning. To balance the immediacy of melee combat and macroview required for ranged combat, the camera’s field of view decreases when enemies cluster around Cinnamon and increases when she draws her ranged weapon.
Game Feel and Animation
Four techniques emphasize the weight and readability of attacks.
Favoring Player Input - The player may cancel all attack animations to defensively break out of inputs
Secondary Motion - The human eye is attuned to unnatural movement. The cloth simulation on Cinnamon’s scarf alleviates the robotic movement of her base animations
Translate Forward - Adding forward momentum connects chains of attacks and gives the impression of powerful impulses
Instant Reactions - The distance of the camera creates a need to convey large differences between hits. Instead of blending between animations, hit reactions jump to the middle of the montage, which allows for clear telegraphing of movement. Hit flashes also provide feedback on collisions
Genre Selection and Movement System
Borrowing from the above three games, I chose to create an FPS due to the high standardization of the controls. Drawing from the themes in Ghostrunner, the player—controlling a criminal protagonist—must escape a prison by killing all of the guards in a given time frame. I found that the parkour movement in Ghostrunner and Mirror's Edge facilitated more long-term changes in my heart rate than VALORANT, and the organic nature of the movement system paired well with the biological theme.
I built an analogous parkour system to Mirror's Edge by adding crouch, slide, sprint, wall run, mantle, and ledge grab to the default FPS controls. I noted that the input design of both Mirror's and Ghostrunner felt uncomfortable due to reliance on both the left Shift and left Ctrl keys. Therefore, I removed the use of the Shift key by making sprint, wall run, mantle, and ledge grab trigger automatically based on proximity to walls.
Mirror's Edge's (left) and Bloodline's (right) gray box parkour system
Emotional payoff is difficult to harness in parkour systems because interruptions—such as missing a wall run in Mirror's Edge—are disruptive to the game's pacing. In our heartbeat analogy, think about having an extremely high heart rate before flatlining; it's an unpleasant shift that can break the player's immersion.
Ghostrunnner's pace feels more like a sine wave than a binary function, which I attribute to the "sensory boost" mechanic. While the player is suspended in air, they can trigger an effect that slows down time and highlights enemies. For a brief period, the player can gauge their positioning and queue their next moves. This ability provides a relief valve during high-friction situations and enhances the player's sense of mastery over their environment.
Sensory boost in Ghostrunner
I mimicked Ghostrunner's sensory boost with the "Bloodrush" ability by triggering a time dilation when the player's heart rate reaches a certain threshold. Moreover, the enemies and prediction lines from their guns become highlighted to provide the player with more information to evaluate the situation. To limit the usage of this ability, the player can only trigger Bloodrush once every ten seconds.
The lower HUD elements show the player's health (left) and a custom heart rate metric (right). Most measures of BPM utilize an aggregate average over ten-second intervals, causing slower rate fluctuations. My metric shows real-time beat counts and resets every ten seconds, evaluating to faster rate changes. Bloodrush is triggered when the player's heart beats over ten times in ten seconds.
HOW CAN WE FURTHER EMBED HEARTBEAT IN THE CORE GAME LOOP?
Resource management is one of the more subtle themes in Bloodline's gameplay; it emerges only when Bloodrush is on cooldown, forcing the player to rely on other systems before the ability becomes available. However, the player's unlimited supply of bullets leaves little reliance on Bloodrush.
To enhance the theme of resource management and thus add a new source of tension, future iterations of Bloodline will leverage heartbeat counts as currency. At its current state, Bloodrush is triggered regardless of the player's desires; players may feel that they wasted the ability during periods out of combat or wish to save the ability for another time. With a currency system, the player may spend a single beat on a bullet or ten beats to use Bloodrush. Not only will this condition provide players with more opportunities for skill expression, but it will also necessitate more frequent usage of the parkour movement system, which emphases satisfying traversal and drives heart rate changes.
Opacity vs. Depth
During one particular session, I explained the metrics behind Bloodrush and its threshold to one of my playtesters. The next time he played, he began holding his breath to increase his heart rate above the threshold. Though I initially conceptualized Bloodrush as an organic relief valve to high-tension scenarios, he demonstrated that some players want to control the game's pacing rather than allow it to be dictated by external systems.
Opacity surrounding Bloodrush engendered a unique sense of discovery; the player can feel when it should be triggered, but they are not told precisely when or what it does. However, that sensation wears off after the first-time experience. Systems that enable strategizing and skill expression—such as resource management—permit greater depth and longevity than short-term sensational experiences.
Bloodline now utilizes a currency system, in which the player may spend one heartbeat on a bullet or ten heartbeats on a Bloodrush effect. This modification forces the player to find new avenues for skill expression through the parkour system. It additionally includes updated UI and alerts to notify the player when Bloodrush becomes available, which enhances learnability without compromising the player's sense of discovery.
Bloodline's protagonist offers a high-agility, high-fragility power fantasy that facilitates rapid decision-making and frequent reruns. However, early feedback from playtesters suggested that the damage system was too unforgiving. I implemented a 5-cost shield that blocks all incoming damage for ten seconds to allow the player to experience longer sessions of fast-paced gameplay.
The shield, mapped to the E key, is indicated by a honeycomb overlay
Progression systems create a positive feedback loop between the player's and avatar's mastery. Player progression—the player improving their knowledge of the game—often facilitates abstracted progression—in-game statistics that evolve a character's abilities. Think about snowballing in League of Legends; the autotelic reward of landing Yasuo's wombo-combo provides more gold to launch even deadlier attacks. Once you hit your 0/7/0 power spike, much of the apprehension of the early game subsides, producing a faster pacing and more frequent opportunities for emotional payoff.
Bloodline takes a different approach to progression systems by eliminating abstracted progression. The heartbeat system demands a consistent difficulty and pacing within levels; if the player finds a level too easy, their sense of urgency dissipates, their heart rate slows, and they run out of bullets. If the player finds a level too hard, their sense of urgency skyrockets, their heart produces an infinite supply of bullets, and we're back to square one. Therefore, Bloodline must deliver the end goal—the feeling of complete mastery over the character—without altering the game's pace.
A Balancing Act
The ten-second shield over-compensated for the unforgiving damage system. The following changes have been implemented to maintain a consistent difficulty level and fast pace.
- Enemy firing period decreased from 2 seconds to 1.25 seconds
- Damage increased from 33 to 40
- Shield cost increased from 5 to 10, and Bloodrush cost increased from 10 to 20
- Shield blocks all damage for 5 seconds, then blocks half of all damage for 5 seconds
- Bloodrush decreases the camera's field of view to improve the player's aim
As an added bonus for the most resourceful Bloodline players, headshots kill enemies instantly.
Bloodline is my first foray into the more analytical approach to design that underlies most of the games I love. Though I have only been making games for one year, the kind of designer I am and the kind of player I am have emerged as very different people. Bloodline represents my attempt to bring those two individuals closer together. I would love to expand the core mechanics of Bloodline into a multiplayer setting, and I invite you to check back on this page in a couple months as I sort out my motivations as a designer.