CINNAMON &
ANTONYMS.
Leveraging Brain Activity for Combat Systems
How can the player’s cognitive states contribute to game feel? Cinnamon & Antonyms is a top-down, melee-driven RPG that features layered combat systems. The game interfaces with an electroencephalogram (EEG) to modulate the avatar’s abilities based on the player’s level of concentration.
Cinnamon & Antonyms was inspired by two talks at the 2022 Game Developers Conference: Ian Cofino’s “Dreamscaper: Killer Combat on an Indie Budget” and John Muñoz’s “Future Realities Summit: How NASA Has Translated Aerospace Research into Biofeedback Game Experiences.”
Winner of the Best Overall Award at the 2023 University of Chicago Winter Tech Showcase
Timeline
June 2022 — November 2022
5 months
My Contribution
Solo development
Tools
Unreal Engine 4
Muse 2
OpenVibe
BlueMuse
Mind Monitor
Audio & UI
Dreamscaper (2020)
Transistor (2014)
League of Legends (2009)
Undertale (2015)
OVERVIEW.
WHAT KIND OF INFORMATION CAN WE GET OUT OF BRAIN ACTIVITY?
Understanding What You Can Do Versus What You Should Do
In my eleven years (!!) as an aspiring neuroscientist, there have only been a handful of moments when I'm reminded why I care so much about the brain—or rather, why my brain cares so much about herself. I was lucky to capture one of those moments during the early stages of this project, which I’ve clipped below.
A Side Note
for Nerds
It goes without saying that I tried a couple (read: a ton) of signal processing schemes before landing on this one. EEG headsets typically have built-in dimensionality reduction, which means that we’re aggregating signals from billions of neurons into a modest five channels. The main architecture of the processor involves a 1–30 Hz frequency band filter and fast Fourier transform (FFT) to isolate and categorize the spike waveforms. The resultant signal is a 5 x t matrix, where t represents the neural manifold over which the Kalman filter estimates the current state. As a general rule, smaller t values allow for higher fidelity but are more computationally intensive and susceptible to noise. Given that the Muse 2 headset samples at a rate of 256 Hz, it’s safe to choose a t value well into the double digits. We can make our math a little easier by using our FFT values to multiply by a 5 x t matrix of decoder weights, so we’re left with a single time-varying float value. And that’s brain control, folks!
Brain control is a longstanding fantasy within our cultural imagination, capturing the attention of tech bros with the budding field of ubiquitous computing and Trekkies with the near-future promise of psychokinesis. It speaks to a level of authority over ourselves and our environment that we will likely never achieve within our lifetimes.
So, why didn’t I make a fully brain-controlled game? Whenever I consider a new feature, I ask myself two questions. Firstly, what portion of the product’s resources (time, money, computational power, or otherwise) does this feature consume? Secondly, how much of my vision for this product can I achieve with this feature?
Cinnamon & Antonyms aims to deliver a visceral power fantasy that rewards strategy and improvisation. Based on that goal, I'd place brain control in the low-impact range with an astronomically high cost. Even when approaching new games, hardcore gamers have already accumulated a mental model for how they should interface with combat systems. We can capitalize on those heuristics by using genre conventions—such as standard movement controls—instead of incurring the cost of additional training. Peripheral devices also tend to have high latencies and rarely offer the fidelity for fast-paced gameplay. I'm not arguing that brain control in games is always bad—think about how it could be used as an awesome accessibility feature—but it’s not right for this product.
Capturing Brain Activity
To establish a wireless connection to the Muse 2 headset, I used three acquisition servers that leverage different forms of serial communication to successively process the incoming data. As shown below, BlueMuse streams real-time EEG data through the Lab Streaming Layer (LSL) protocol. The OpenVibe acquisition client filters and writes the input stream to a Transmission Control Protocol (TCP) socket, which feeds into the UE4-BCI plugin.
Inside the Decoder
Now that we’ve gotten our signal into the engine, we have to decide how we want to interpret it. As shown on the right, the headset’s conductive sensor collects electrical signals from outside of our frontal lobe. Because we’re working with a non-invasive device, the fidelity of our signal is limited to large-scale patterns of activity rather than transient changes.
One such EEG pattern is the index of engagement, which describes a user’s level of focus. Researchers at NASA quantified the index of engagement by the formula β / (α + θ), for which beta, alpha, and theta describe discrete wave frequency bands.
.png)
HOW CAN WE LEVERAGE CERTAIN COGNITIVE STATES DURING COMBAT?
Theory of Flow
To leverage the player’s brain activity as a mechanic, the primary objective of Cinnamon and Antonyms’ gameplay should revolve around inducing changes in the player’s index of engagement. Psychologist Mihaly Csikszentmihalyi characterized the phenomenon of flow as a sense of immersion and energized focus that arises from certain forms of gameplay. He theorized that flow occurs when a game offers focus, freedom, feedback, and challenge. Using the index of engagement as a proxy for flow, I derived six design pillars that capitalize on the player’s cognitive resources to encourage flow states.
Purposeful
Players should be able to
understand the situation, tactically
assess options, and make
strategic decisions on the fly
Improvisational
Combat should feel fluid and
allow for multiple approaches
that can all work intuitively
Tough, But Fair
The players' fate is in their hands,
not the hands of the RNG system.
Player actions allow for correction and enemies feel balanced
Dynamic
Combat should drive variety and replayability through overlapping
systems
Feedback-Driven
Players should understand and
enjoy how their actions impact the
world
Personalized
Players should feel like the
emotional and cognitive state they
bring into the game is reflected by
their avatar
Framing Combat
Genre conventions around the camera position inform players’ expectations of the speed and snappiness of combat. As a rule of thumb, the fidelity of gameplay typically decreases with camera distance; a farther camera reduces the weight of combat and often feels less cinematic, but the macroview provides the player with more information on their positioning. To balance the immediacy of melee combat and macroview required for ranged combat, the camera’s field of view decreases when enemies cluster around Cinnamon and increases when she draws her ranged weapon.
Dynamic camera field of view for melee (left) and ranged (right) combat
Due to the distance of the camera, Cinnamon & Antonyms selectively leverages color and values to draw the player’s eye towards certain targets. The values of the background, middle ground, and avatars are clamped to increasingly higher levels of contrast to establish a read hierarchy. Enemies, abilities, and the player character are also categorized into different regions of the color wheel to make them feel visually distinct.
Value (left) and color (right) draw the player's attention towards enemies and the player character
Feel and Animation
Six techniques emphasize the weight and readability of attacks.
Favoring Player Input - The player may cancel all attack animations to defensively break out of inputs
Secondary Motion - The human eye is attuned to natural movement. The cloth simulation on Cinnamon’s scarf alleviates the robotic movement of her base animations
Hitlag - Freezing both characters at the moment of contact sells the illusion of force and allows the player's eyes to register the collision
Zero Tween Time - The distance of the camera creates a need to convey large differences between attacks. Instead of blending between animations, hit reactions jump to the middle of the montage, which clearly telegraphs movement
Hit Flashes - The colors of hit flashes provide feedback on collisions and damage
Translate Forward - Forward momentum connects chains of attacks and feels more committal
Polishing the Camera Movement
The previous camera setup felt static and did not reflect the dynamism of Cinnamon's movements. To provide more passive motion, the new camera system connects to Cinnamon's hip socket instead of her capsule collider, which creates an organic sway in the camera's location based on her movement pattern. A slight camera lag dampens the sway and helps mitigate nausea.
To emphasize the difference in weight between Cinnamon's light and heavy attacks, a camera coom, subtle motion blur, and slow motion effect accompany heavy attacks.
Frame Data Breakdown
Physical interactions in Cinnamon & Antonyms are mediated by three types of colliders. Pushboxes (blue) convey the physical space occupied by the character and prevent overlap between game objects. The size of the pushbox changes based on the movement of the character—for instance, rolling shrinks Cinnamon's pushbox to half of its original height—but is otherwise static. Hurtboxes (yellow) describe when and where the character can be damaged; as shown in the images below, enemy hurtboxes are larger than their pushboxes to provide some leeway in the player's aim. Certain types of defensive abilities, such as rolling and dashing, may disable hurtboxes to create untargetable states. Collisions between the attacker's hitbox (red) and victim's hurtbox apply damage.
Pushbox (blue), hurtbox, (yellow), and hitbox (red) in the startup, active, and recovery frames of a melee attack
Notably, Cinnamon's hitbox is only enabled during the middle segments of the attack animation, followed by a recovery phase in which the player cannot apply damage. The duration of an ability's recovery is directly proportional to its damage output, introducing a tradeoff between the player's rate of queuing attacks and each attack's potential damage. This technique also prevents spam by injecting a delay between successive attacks, which encourages more purposeful combat.
Enemy States
Enemy states provide utility during combat by communicating how enemies respond to specific inputs. This form of diegetic tutorialization allows players to build mental models for enemy attack patterns and apply them to novel situations.
With the exception of her ultimate ability, all of Cinnamon’s attacks include soft targeting, in which the avatar automatically orients towards the nearest enemy during the first frame of an attack. Typical third-person combat systems require the player to control their character's yaw axis and location. Soft targeting alleviates the technical burden on the player to consider the avatar’s rotation and reduces the likelihood of missing attacks.
A Side Note
for Nerds
Why does hitlag work? Hitlags originated from fighting games as a means to emphasize the impact of certain attacks. But, if we think about our experience of collisions, hitlags don’t seem to have a real-world analog.
Imagine scattering a group of billiard balls during a game of pool. We perceive collisions through tons of different stimuli: we can hear the balls clacking together and feel vibrations from the cue stick, but the most telling indicator is an abrupt change in the objects’ momentum. Humans are great at estimating a moving object’s trajectory—after all, we have a lifetime of experience with physics—and we can use that estimation to gauge an object’s time-to-collision (TTC). However, physics systems in game engines often don’t align with our expectations for how objects should move in the real world. By maintaining the change in momentum over a longer period of time, hitlags create a margin of error that ensures the player’s TTC estimation falls within the duration of the lag.
.png)
Cinnamon’s basic attacks set the enemy into a stun state, which facilitates target prioritization and helps sell the weight of different strikes. The heavy hit state influences combat pacing by putting space between the player and enemy, hence providing an opportunity for the player to reposition. The popup state from Cinnamon’s ultimate attack sets the enemy into a juggle state, which encourages the player to press their advantage when they know the enemy cannot react.
Incorporating Brain Activity
Given that Cinnamon’s kit focuses on melee combat while the enemies’ kit focuses on ranged combat, the player requires gap-closing tools to engage enemies at a distance. Cinnamon’s ultimate ability is a near-instantaneous dash that provides temporary immunity to damage. If enemies are within the vicinity of Cinnamon at the end of her dash, they take area-of-effect damage and transition into a popup state. This combination offers a safe method of either engaging or disengaging from combat scenarios by allowing the player to reposition themselves without taking damage.
Cinnamon's ultimate ability is enabled by the player's index of engagement
Cinnamon’s ultimate is enabled by the player’s index of engagement crossing a predetermined threshold, represented by her scarf glowing. This threshold creates a positive feedback loop by offering deeper mechanics for increased levels of focus, which subsequently elicits more frequent flow states. Notably, the average signal produced by the player’s brain changes on an individual and daily basis. To normalize the ultimate across different players, the first ten seconds of gameplay incorporate a real-time calculation of the player’s flow threshold.
HOW CAN WE MAKE COMBAT SYSTEMS MORE RESPONSIVE TO NEURAL INPUT?
Pain Points
An early version of the combat system was tested with a handful of players, and I’ve summarized their feedback below:
-
The acquisition servers produced high latencies. The game was averaging at 45 FPS with hitches below 10 FPS, which meant that players’ indices of engagement were not synced with the values in the engine.
-
The ultimate ability has limited utility in combat. Though this ability doubles as a disengage and gap-closing tool, players wanted more optionality outside of positioning.
-
Players didn't understand why they unlocked their ultimate ability. The game indicates when the player’s index of engagement is above a certain threshold, it doesn’t tell them when they are approaching that threshold.
Simplified Tech Stack
The latency highlighted by the playtesters was primarily caused by the number of acquisition servers concurrently running on my computer. I offloaded part of the signal acquisition to a mobile application called Mind Monitor and decoded the signal in-engine. Rather than updating every frame, the new scheme leverages OSC messaging to only update the engine when the wave frequency distribution changes, which brought the framerate back up to 60 FPS.
Diversifying the
Player's Toolkit
Instead of a binary system that unlocks an ultimate ability when the player crosses their focus threshold, the new kit adopts a flexible mana system. Players gain one mana each time they cross their threshold, which they may spend on three abilities.
Outside of the original dash ability, Cinnamon’s updated kit incorporates melee, AOE, and long-ranged elements to validate a variety of playstyles. The mana cost for each ability is proportional to its damage output.
_edited.png)
LMB — Basic Melee
"Let's go!"
Cost: 0 mana
Damage: 10–15
Using the left mouse button, Cinnamon lunges forward with her sword, swiping up to three times. Using the right mouse button, Cinnamon spins her sword, damaging all enemies around her.
Indicating Incremental Success
The most common feedback I received on the first iteration of Cinnamon & Antonyms was that players didn't understand whether they were making progress toward unlocking their ultimate ability. A combination of HUD and in-world elements visually indicate real-time increases in their index of engagement.
Health and mana bars display passive regeneration and real-time increases in the player's index of engagement.
Ability indicators are highlighted when the player has enough mana to use them.
Ability descriptions appear
on hover.
Cinnamon's scarf glows when she has full mana to encourage the player to use their abilities.
Hey stranger,
I conceptualized Cinnamon & Antonyms near the end of my junior year of college, just before starting my first roles in the AAA space at Electronic Arts and later Wizards of the Coast. Cinnamon and I have grown a lot over the past few months, but this project will probably be my last substantial personal endeavor before starting my post-graduate career.
There’s something disorienting about fulfilling a dream—it comes with an annoyingly momentary sense of relief followed by existential self-reflection (what now? why me? was it worth it?). The weird thing about dreams is that they don’t always repeat themselves in the exact same way. I remember writing in my reflection on Zero that I desperately wanted to become a concept artist at Riot Games, and even though I’ve strayed a lot from that path, I’ll be joining Riot in six months. I’ve taken some time this year to consider what my next dream would look like. I can’t say I have any more clarity than you do, stranger, but here’s what I know: I experienced a sense of joy creating Cinnamon & Antonyms that I haven’t felt with projects I've worked on before. Hopefully, this feeling is the beginning of something special.
A special thanks to everyone who shared honest feedback, playtested, judged, and listened to me talk about Cinnamon & Antonyms ad nauseam over the past five months. I'm honored to have witnessed all of the light and joy you brought into Cinnamon's life.