Anyone who's played enough Mafia has some hunches about how town and scum use their votes. Some of those might be right and others might be wrong! Here let's subject all these ideas to some scrutiny.
After a bit of work, I have my hands on a dataset tracking every vote anyone's ever made across hundreds of completed non-theme games. I've got each vote's post number, target, and maker, whatever. I even know which day phase they happened. And for each game in the data set, I also have all the stuff you usually see in a mod's OP - stuff like who won, each slot's role/alliance/fate, replacements, etcetera. I also of course have every one of each game's posts. And more or less, so do you -- though I could still do much more work to make it all much more accessible.
In my posts below I'll provide further details about my work on and plans for the project and so on but above all I'm interested in you and your ideas and connecting them to data. Pass on your ideas and there's a solid chance me or someone else can do some work on them. Here are starter questions I'd welcome anyone to provide input on:
What are some memorable ways people's votes have lately pinged you as suspicious (or the opposite!)?
What are your most confident beliefs about how town and mafia tend use their votes?
What are some things you always wondered about how players vote but could never confirm?
Do you have an approach to vote count analysis that you apply in your games, even a vague one?
How do matters like WIFOM or the context of votes color your interpretation of them?
If you're particularly code or stats-savvy or just feeling creative, I'd also appreciate any advice (or of course contributions) in that space for making the most of all this. And from me, you can expect a steady flow of new plots and analyses, and eventually models and tools, focused on growing and turning this dataset into usable knowledge about how people play this game.
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
i think i remember something similar to this thread (or maybe this thread's prelude) existing somewhere else and some people asking some pretty good questions that maybe could be answered. here are a few of my prior beliefs, though:
On a day 1 elimination wagon on scum, other scum are more likely to be at the end of the wagon (and possibly slightly less likely but still >rand to be at the very start of the wagon)
In 9 player games, scum are almost never on day 1 scum elimination wagons
This one might be hard to test—In games where multiple scum members are being voted early on (essentially they have been "caught" early in the game), they will be more likely to vote each other (i.e crossvote).
i'll probably try to come up with more but those ones come to mind easiest for what i'd be interested to see tested on the data.
Ego! Hoping this thread will confirm my longstanding belief that, broadly speaking, it is not what is voted but how it is voted that actually matters, although the points in the post above seem a little more plausible to me than most vote oriented arguments I have seen so certainly possible there may be some findings.
In post 5, northsidegal wrote:On a day 1 elimination wagon on scum, other scum are more likely to be at the end of the wagon (and possibly slightly less likely but still >rand to be at the very start of the wagon)
Based on some of the early data that still needs to be verified, this basically held true for all scum eliminations. There was a drop off on early votes for wagons with fewer players alive.
Hopefully psyche doesn't get mad at me for posting this (it's his).
In post 6, Cephrir wrote:Ego! Hoping this thread will confirm my longstanding belief that, broadly speaking, it is not what is voted but how it is voted that actually matters, although the points in the post above seem a little more plausible to me than most vote oriented arguments I have seen so certainly possible there may be some findings.
I wonder if this project will warp the meta.
Ooh the vca vs anti vca wars finally will have an answer... right?
i'm just as interested in finding ways to get at the how as at the what though it will take some work
atm doing some finishing touches to make sure the apparent contrasts in these plots between "people DO differ by faction" and "people DON'T differ by faction" are real (or not real)
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
In post 0, Psyche wrote:What are your most confident beliefs about how town and mafia tend use their votes?
In the broadest of broad definitions, I would say:
Town use their votes to try and gain information and/or place pressure on the scum with the end goal of eliminating scum (this definition I feel covers basically every playstyle and philosophy surrounding votes but I admit it could be imperfect);
Scum use their votes to try and manipulate the town's perception of the gamestate and the information within to make things more favorable for them. (This definition I feel covers every scum use of their vote but I admit is very very broad. Bussing is manipulating the town's perception of the gamestate to make the town be less inclined to think two individuals are partnered together; self-voting is often manipulating the town's perception of the gamestate to make the town be less inclined to think the individual is scum; hammering oneself as scum to deny the town information is a manipulation of the gamestate and information within by denying the town that info; voting in general for scum is a tool they use to attempt to make the town believe the scum doing the voting isn't scum except in situations where it's meant to make the town believe the voted isn't scum per the above on bussing; voting town in general for scum is a tool they use to attempt to manipulate the town's perception of the gamestate and the information within to make a town elimination more likely.)
But while these broad definitions may be close to universal, obviously the specifics differ and there's far more nuance when you get down to the details which involve a lot more, so to speak, guesswork.
Something I've been pondering as a modeling approach is perhaps to focus on differences in information and in incentives between
factions
while focusing on differences in sensitivity to these incentives between players.
Take the dilemma posed by the possibility of bussing, for instance. Mafia face a choice between the apparent towncred that comes with limming scum and their reduced mathematical odds of victory when scum is eliminated. Any measured differences between how factions vote when a mafia player is some distance from being eliminated presumably stem primarily from how these incentives are weighed.
The challenge in modeling voting behavior this way though is that it sort of also requires a model of the expected value (EV) of a vote at any point in the game. But paradoxically, a good model of vote EVs depends on a a good model of how people actually vote! Furthermore, if people's perception of vote EV differs from true EV, then finding EVs with math won't really cut it. So it's possible this whole framework is too tautological to be of use.
But that's how I see this going.
1. Start with a baseline, probably wrong, model of town voting behavior that includes some sensitivity to voting patterns. EV calculations often assume something close to random voting but we'd want to bake in something like a bias toward voting people who have not voted known mafia.
2. Using this baseline model plus a traditional EV analysis of how different phase outcomes (e.g. a teammate getting eliminated) affect mafia win probability, build a vote EV model that tracks the expected value for mafia of making any given vote given a current vote distribution (and maybe also voting history). The model should predict the highest EVs when 1) a voting history is indistinguishable from town voting behavior but 2) makes miselims and similar favorable outcomes more likely.
3. Assume mafia votes with these EVs in mind, but potentially with different degrees of sensitivity to different sources of risk.
4. Improve on or replace these models through analysis of their internal logic and comparison of their simulated predictions with analysis outcomes from the actual data.
Issues with this approach include the potential circularity if approached wrong (we're basically simulating WIFOM!), little initial sensitivity to how actual discussion drives voting behavior (we'd only get more sophisticated later!), and the trouble of factoring stuff like slots having named roles and abilities into any EV calculations. But one has to start somewhere I guess.
For now though we need a good overview of what kinds of phenomena the model has to account for of course!
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
I didn't share my code but I'm happy to walk you through how I did it. I feel like I remember people attempting to build similar things but I can't remember where to find them now.
If yall decide to mess with the dataset, it's in `data/votes.json` in the repository linked above. I just updated it with useful metadata (e.g. voter and voted factions, whether a vote is "terminal" or not, stuff like that) sufficient for most initial analyses one might have in mind. Hope to have some initial plots for yall soon - they mostly exist but I wouldn't feel responsible if I distributed them w/o confirming a few more things.
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
Our first batch of analyses focus specifically on characterizing how different factions position themselves on the
hammered wagon
of a phase - the wagon that eliminates a player and ends the Day.
To help shed some light on questions like these, I identified the votes in my dataset that were part of a phase's hammered wagon and tracked across all these votes whether they were made by TOWN or MAFIA and then visualized the proportion of votes from each faction between various features. To help interpret these results, I also plot (in gray) the proportions you'd expect to see if voter faction had nothing to do with how people vote. To put it simply:
Colored lines and objects in the plots below come from my actual data
Gray lines indicate what you'd see if faction had nothing to do with how people vote
Error bars roughly indicate how far away colored lines should probably be from the gray lines before you even think about getting excited
Spoiler: Details
When aggregating across many games with different setups, it can be hard to interpret when a proportion is genuinely interesting and not just the base rate you'd expect if TOWN and MAFIA voting behavior were indistinguishable. If this
null hypothesis
were true, then we would be able to label slots in our dataset as MAFIA or TOWN at random and still get the same analysis results. To help put our results in context, I did exactly that: I repeatedly shuffled slot identities in the dataset around and performed my analyses on each shuffling. When analysis outcomes on my real dataset differ enough from those using my null hypothesis dataset, we can feel more sure that we're registering a real effect of slot faction on voting behavior. This technique is called monte carlo testing, and it's very cool. But honestly it's been tough to implement, so let me know if you see any signs of flaws with my approach.
Do MAFIA disproportionately show up on hammered wagons?
We can track across phases the rate at which players on a hammered wagon are MAFIA, and contrast that with what we'd expect if being TOWN or MAFIA makes no difference in how you vote. We can also
only consider phases that don't end the game
- i.e. the phases that someone in-game a TOWN player might look back to to find MAFIA. Doing that, unless my code is wrong, we find that the ratio of MAFIA on hammered wagons may depend considerably on both game phase and on the faction of the slot voted on.
First, we find that on balance, phases we have enough data for end with MAFIA on the hammered wagon at a rate meaningfully lower than what you'd expect if factions vote the same. It's a small difference though, so not much to write home about.
When we focus exclusively either on wagons where the eliminated slot is TOWN or where the eliminated slot is MAFIA though, we make a striking observation: As a game goes on (e.g. in Days 3, 4, and 5), MAFIA disproportionately end Days away from hammered wagons on TOWN while disproportionately ending Days on hammered wagons that vote out other MAFIA.
Since the MAFIA faction's win condition to eventually outnumber TOWN, this result may seem pretty unintuitive. Importantly (at least as a sanity check), this pattern completely reverses if you do also include the last phase of each game in the analysis. This makes sense, because in these final phases, whether the limmed player is TOWN or MAFIA is decisive about who wins the game.
Spoiler: Same Analysis But Just the Final Day of Each Game
So MAFIA do readily switch up this pattern when it would be throwing to do otherwise. But what's going on in those other phases?
"Towncred" is the only reason MAFIA ever really have to help vote out other MAFIA or avoid helping vote out TOWN. My hunch is that on Days 3-5, lims are especially likely to be foregone conclusions among the TOWN thanks to power role night actions and/or the emergence of consensus reads. In these cases, MAFIA have no room to drive eliminations on their own and instead focus on collecting credit (for Mafia lims) or avoiding blame (for mislims). Even though TOWN are aware that MAFIA might apply this strategy, the MAFIA still sees value in it despite the downsides. Alternatively, perhaps they suffer from a Curse of Knowledge and more readily acknowledge the merits of players' accurate reads and demerits of wrong reads (this wouldn't explain why Day moderates the effect though!). Whether one or another account explains what we're seeing, the above plots are the result.
Do mafia disproportionately commit to hammered wagons at particular points?
If it were true that (on balance) MAFIA worry more about towncred than optimizing their numbers advantange when they choose where to land their vote before the Day ends, we should expect an additional effect of faction on a slot's
position
on hammered wagons. For example, we might expect MAFIA to pay extra attention to who ends up casting the hammer vote ending the Day and finally confirming a slot's elimination.
There are a few ways to track a player's position on a wagon, and which is best to use depends on the analysis question we're interested in. For example, we can track either the
entry position
of a vote -- how many voters were on the wagon when you first joined it -- or the
terminal position
-- the number of votes made before yours in the wagon that never changed. For example, of you vote someone and I vote someone but then you unvote and the Day ends, my vote's entry position is 2 but its terminal position is 1.
When comparing wagons across phases, it's also pertinent whether we track the absolute position of a vote or specifically how close a vote gets a wagon to a hammer. Being the third vote on a wagon has drastically different weight in a game where it takes 7 votes to eliminate someone compared to when it just takes 4. Similarly, the attention an E-2 vote gets might depend a lot on how many votes it took to get there. Basically, I'm saying these analyses get pretty weird if you think about it enough! I'm here going to cherry-pick the analysis variations I find interesting.
First we consider whether different distances of a vote position from hammer on a hammered wagon feature more or less MAFIA than you'd expect if faction didn't affect voting behavior. To help shed light on the results above, we'll look at the overall stats as well as the stats for Days 1-2 and Days 3-5 - again always excluding the final phases of each game where applicable.
Spoiler: Distance from Elimination Subsets
We do find across phases that MAFIA hammer other MAFIA especially (but still only a bit) more often than you'd expect if faction didn't affect how people vote. But while we continue to see the same avoidance of participation in mislims among MAFIA, it's hard to identify consistent patterns in the final position of MAFIA on hammered TOWN wagons from these plots that we might develop an interpretable theory from.
We also see some evidence that on D1/D2 MAFIA are loathe to show up as early as E-6 on a teammate's wagon and stay there until the wagon is hammered. However, since our sample size of games/phases where E-6 votes are possible is necessarily smaller than those of games/phases where E-3 or other E-<6 votes are possible, we might want to try further analysis.
Instead of looking at how far away a wagon is from being hammered at each vote, let's instead align our counts at the absolute size of the wagon at each vote - its
entry position
.
Spoiler: Entry Position Subsets
Considering vote position this way highlights some patterns not evident in the previous analysis. Sharpening our previous observation, we find that in the earlygame (D1 and D2) regardless of exactly how many votes it takes in a given phase to eliminate someone, MAFIA are biased
against
ending a phase as the first vote on a hammered MAFIA wagon in much the same way that they are biased
toward
being the hammer.
Furthermore, we observe that much of MAFIA's avoidance of participation in TOWN mislims is concentrated around the wagon's early positions; when the wagon gets more support, it seems MAFIA vote more like random - though the story gets complicated again near the very end of the wagon.
We'll explore one final approach to analyzing wagon position even as stuff might vary between phases like the number of players around. We'll track for each vote the "percentage completion" of the corresponding wagon. For example, a wagon with two votes on it when it takes 5 to eliminate is 40% complete. For plotting, we'll group these percentages into 10% bins. The idea is that this might be a better way to factor how player count and wagon size interact to define the weight of a given vote than absolute position or distance from hammer. One great feature of this approach is that the final bin collects all hammer votes, and the initial bin collects all initial votes, so the plots are more interpretable at least in those areas.
Spoiler: Binning by Percentage Completion Subsets
This approach seems to do the trick. It helps localize MAFIA hestitancy to participate in miselims to the early positions in a TOWN miselim, and the similar but weaker early positional avoidance pattern for MAFIA lims, too. And of course it's got me confident that MAFIA really are a bit biased toward hammering their buds when presented the opportunity -- even in Days 1 and 2.
Conclusions
These comments probably make too much of a big deal about a lot of weak patterns that don't really decisively speak to one single account of How Mafia Vote. The deviations from our null hypothesis identified here aren't very big even if we do suppose they're real and might be observed in games yet-to-be-played. When you factor in the consequences of performing MANY statistical tests and comparisons as I do here, the weight of any "significant" result lessens a bit.
But this is a good start I think. So far we've only considered a sliver of the voting data I've collected -- just the 7 or less votes per phase that actually get someone voted out -- but are nonetheless identifying what I'd be willing to bet really are small but systematic patterns in how players with mafia role pms pursue their win conditions. These patterns suggest that while MAFIA do an impressive job of mimicking TOWN behavior, on balance their knowledge of the setup and strategies for countering common scumhunting approaches actually distinguish their behavior from other players.
But I guess it'll take more work to find out for sure.
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
will probably edit and pare the above down when i'm up for that.
but lately i'm leaning toward a slight "shy scum" prior for interpreting votecounts. Even though it's in their numerical interest, they tend to avoid helping lim town and join in on limming other mafia. When you look at the positions of players' votes on these hammered wagons, mafia are particularly unlikely to end a phase at the earliest positions on town limmed wagons but especially likely to cast the hammer vote on other mafia. There are some hitches to the overall narrative and the differences underlying it are pretty small, but the evidence seems consistent with the idea that on balance scum don't exactly play for keeps, instead preferring a longer game focused on protecting towncred.
Will need more evidence from behavior outside these hammered wagons but that's the hypothesis I'm going to run with and examine as I work on more analyses.
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
A weekly thing is probably not in the cards given my several other projects, but you can still expect somewhat regular updates from me on this. I think in the short term I'm going to focus on cleaning up what I have so far (including my analyses above) and turning my codebase into a coherent and well-documented library so novice and experienced coders alike can ask questions of the data if they want to and get answers with less work.
as gregor samsa awoke one morning from uneasy dreams he found himself transformed in his bed
ego
