[Requesting Help] MTG Swiss Tournament Simulations (Can you program?)

[4eak]

I've re-written an old Python program of mine from a while ago. I wanted to do some testing on the nature of magic's swiss tournament structure through mass simulations. My previous work forced me to call into question the validity or degree of relevance of tournament results. I worry we put too much stock in tournament results, that we trust them too much, etc. I wonder how much justice is to be found in the DCI's swiss tournament structure. Do the best decks/players generally top 8, and in what size tournaments, and by what margins? How much improvement in discovering the best decks would we see by increasing the number of rounds played? My preliminary analysis (which I generated about a year and half ago) suggested that this game may be less about skill, and frankly, more like gambling, to some degree, than I initially thought.

Anyways, I want clearer and more developed answers to some set of questions regarding the swiss tournament structure. Simulations (according to two math departments I've talked to) are really the best way to get the answers. I also have a few curiosities I might satiate with this tool, such as developing a deeper understanding of the nature of glass-cannons and other highly metagamed decks (e.g. the RG anti-artifact decks that existed when Affinity was legal in T2).

Unfortunately, I'm a noob at statistics. In addition, I'm a terrible C++ coder, and I am only marginally better at Python. My code works, but it ain't pretty, and problematically, my code is crazy slow (even after the meager optimizations I can make). For example, for 1000 simulations of a 1000 person tournament, single-threaded on my i7-2600k@4.4Ghz takes 7 minutes (with Pypy). From what I've seen, I really need far more simulations (lots of variance in the answers I get with so few simulations). With what I've got, it just isn't feasible to sketch out satisfactory answers to these kinds of questions.

I need help.


I'm looking for people to help me figure out how to ask and answer the right questions. There are probably many salient statistical features of this topic of which I am not aware.
I'm looking for people to either help optimize my code (not preferred) or write a simulation program from scratch (preferably in a decent language, built by someone who actually knows what they are doing).




peace,
4eak

[Dissolution]

I can code (did the Charbelcher goldfish program and I write MTGO bots).
I write in C#, so it's easy to understand. I also have a lot of experience making simulation code (dicerolling, mostly) that's quite fast.

I'd love to help your project out!
-T

[stage]

I'm looking for people to help me figure out how to ask and answer the right questions. There are probably many salient statistical features of this topic of which I am not aware.
I'm looking for people to either help optimize my code (not preferred) or write a simulation program from scratch (preferably in a decent language, built by someone who actually knows what they are doing).


This is super interesting! I'm not working the week of 12/20 so I'll have a fair amount of time. It would also be a great way for me to learn more about the tournament structure in case I play in large tournaments in the future (one can always hope, lol). I use MATLAB extensively in my work; my background is in economics (game theory and finance) and statistics.

e: saw Dissolution is interested so I'll sit this one out. I'm looking forward to seeing what they come up with!

[jandax]

Frank Karsten did this type of thing at CFB for an article or two. In his program, millions of games were simulated to produce a sort of an aggregate metagame. It's a long shot but maybe you can try to reach out to Frank (via FB/Twitter I assume) and go from there.

Best of luck, hope this project takes off! It'd be a helluva case study

[apple713]

Wouldnt you have to determine the matchup win % for each deck and its opponents in any given meta for this?

If you did have to gather that information i would be very interested in it and it would be an amazing resource.

[stage]

Wouldnt you have to determine the matchup win % for each deck and its opponents in any given meta for this?

Yeah having winrates would probably be needed. It might be possible to determine winrates empirically using available data. Not sure how much accessible tournament data is out there.

[4eak]

PM'd Dissolution. We'll hash out the details. For now, I'm thinking:


User input, things we control, preferably as arguments/flags from a command line

Number of Simulations
Number of Decks
The matchup %'s of decks. Most of the time it will be one deck with specific settings (this one is easy) and the rest will be zombies which go 50/50 against each other. I'd like to have control over all of them though. Not sure how about to go about doing that.
Number of Rounds, if we want to go non-standard
Top X Cut off, if we want to go non-standard
Whether or not we play out Top. If Top X is played, maybe I want to be able to choose between Swiss (as some German tournaments do, if I recall correctly), but definitely we need standard Single-Elimination


Output:

Doesn't really matter. Might as well be somewhat verbose one-liners we can parse. Should it just output to the command line or to a file? Will there be an IO bottleneck to an SSD? Could we handle it?
I'd like to have at least this for output:

The Top X, likely ranked from highest to lowest seed.
The results of Top X single-elim tournament



Problems:

Tie-breakers. When you have so many decks that are 50/50 against each other, tie-breakers that we currently have often don't work. After tiebreakers, we may only know 7 of the decks which are in the Top 8. We should record this, somehow. I don't know quite what to do with it.


@ stage
This is super interesting! I'm not working the week of 12/20 so I'll have a fair amount of time. It would also be a great way for me to learn more about the tournament structure in case I play in large tournaments in the future (one can always hope, lol). I use MATLAB extensively in my work; my background is in economics (game theory and finance) and statistics.Do you have any ideas? It is best to know our requirements before the program is written. If there are things you want tested, let us know now. =)

Also, you may find these useful:
https://www.wizards.com/dci/downloads/Swiss_Pairings.pdf
https://www.wizards.com/dci/downloads/tiebreakers.pdf


Yeah having winrates would probably be needed. It might be possible to determine winrates empirically using available data. Not sure how much accessible tournament data is out there.Yeah, not only is it a records problem, but I believe there have never been enough games played in the lifetime of MTG to make determinations about the vast majority of decks regarding the vast majority of metagames and players. We won't have to worry about this though. See below.

@ jandax
Frank Karsten did this type of thing at CFB for an article or two. In his program, millions of games were simulated to produce a sort of an aggregate metagame. It's a long shot but maybe you can try to reach out to Frank (via FB/Twitter I assume) and go from there.I just looked at it. I wrote the same program (one for burn, and then later another for sligh) about 4 years ago (http://www.mtgthesource.com/forums/showthread.php?15820-61-or-more-cards-in-toolbox-decks). My favorite part was getting the mulligans to work right. Unfortunately, this project, which is a worthy one that we can all learn from, has nothing directly to do with the Swiss tournament structure.

Ha! Browsing through that thread reminded me: I need to bring back Rainbow Troll.

What a terrible script.

@ apple713

Wouldnt you have to determine the matchup win % for each deck and its opponents in any given meta for this?

If you did have to gather that information i would be very interested in it and it would be an amazing resource.I'll let you know right now, you can't feasibly determine the matchup win % for each deck, for many reasons. Too many combinations of decks and metagames (you might be able to make some huge reductions to archetypal decklists and metagames), you have to take into account varying degrees of player skill (no idea how to do that), and you can't write an AI (let alone an AI with varying degrees of skill) for the vast majority of Legacy decks (genetic code is a start?). Each of these issues are there own problem to tackle. That is a whole different can of worms.

The fact is that we will just be hardcoding matchup %'s. For example, we will just assume that a player/deck pair has some matchup % against another player/deck pair. We can just ignore decks and players, in a sense. This is perfectly reasonable though, since we are testing the swiss tournament structure itself. Note, this means we're analyzing something which isn't just specific to magic in large part (although, the tiebreakers are specific).



peace,
4eak

[Dissolution]

Those pdfs helped a bit. Working on implementing the system that pairs up players. Does anyone know if there is any bias in the system to keep people from being paired up more than once in the swiss?
Also, with Byes, it says they don't count for opponent percentages. Does this mean they count as 0 or do they not even factor into the calculations?
I will be doing pre and post-board matchup percentages, as well as support for draws (if we can figure out a way for them to be useful).
Should have some concrete data in a day or two.
-T

[4eak]

Generally, players are not paired with the same opponent more than once in the rounds. Not everyone uses the same pairings generator though. Given the bias, when there are only two players at a particular match point value who have already played each other, this causes some weird things to happen. Admittedly, regarding 50/50 zombies, this probably isn't an issue, but if we wish for more complicated matchup grids (and that would be nice), then it will matter. Note, however, that there are some tournaments I've been to which used different pairing procedures (programs) that did allow for opponents to play each other more than once in the swiss rounds. This is an interesting issue in the Swiss tournament structure itself. Consider having a flag option for this.

As to the byes, I can only go by what the PDF says:


When a player is assigned a bye for a round, he or she is considered to have won the match 2–0. Thus, that player earns 3 match points and 6 game points. A player’s byes are ignored when computing his or her opponents’ match-win percentage and opponents’ game-win percentageIn the examples given, a player's bye is used for his MW% and GW% calculation, but ignored for his OMW% and OGW% calculation. So, if there were 8 rounds played, and a player got a bye, then to figure out his OMW%, we'd only add up his 7 opponents' match-win percentages and divide by 7 (rather than 8, because of the bye). For the same player's OGW%, we also ignore that player's 6 games points earned via the bye. So, when adding up his opponents' game-win percentages, we ignore the bye round, adding up only the opponents' 7 GW%'s and dividing by 7*3 (rather than 8*3).

This is more a theoretical note, but I wonder why we even allow byes to be calculated at all in match-win and game-win percentages. That seems to be a mistake, at first glance. It isn't like these are earned wins - shouldn't we only look at those wins which have been earned? Can anyone give me a justification for the DCI's choice on this one? Further, can anyone justify why .33, in particular, is chosen as the floor for the MWP and GWP used for calculating OMW% and OGW%? These might be interesting issues to explore, and perhaps we should have flag options for these as well.

I hadn't even thought about draws in this program, but I can see it. I have long worried that even if slower control decks which were prone to draws did have significant advantages over a metagame without time limits, these decks may still be largely unplayable in tournament settings (in part due to how exhausting they can be to play). We'd need a way to set it choose the percentage of times a deck will draw with another (maybe a draw matchup grid, or we can may our current matchup grid more complex). This might not even be worth the effort though.



peace,
4eak

[JDK]

This reminds me that I wanted to program an Android application designed for MTG's swiss structure...

[Dissolution]

When a tournament cuts to top X players, how does it do the pairings from there?
I think it's 1st & Nth, 2nd & N-1th. Just want to be sure I'm taking that into account.

[4eak]

I've seen first seed vs. lowest seed, second vs. second lowest, and so on. I've also seen random pairings in the top. Anyone seen anything else we should take into account?

Also, I have a question, and I have no idea where to find the answer. I'm hoping some more knowledgeable folks can either explain the answer or point me in the right direction. Currently we have this:

17–32 5 rounds of Swiss
33–646 6 rounds of Swiss
65–128 7 rounds of Swiss
129–226 8 rounds of Swiss
227–409 9 rounds of Swiss
410+ 10 rounds of Swiss


If I recall, we should add 9-16 is 4 rounds of Swiss, and anything below 9 is 3 rounds of Swiss.

The problem, however, is that in the largest tournaments in the world, the rounds and cutoffs for days don't follow this list (and may even vary).

What rules should we employ for figuring out number of rounds and cutoffs for larger tournaments?


peace,
4eak

[thecrav]

The only info I was able to find regarding number of rounds and how pairings are done were both for MODO:

Players Rounds
2 1
3-4 2
5-8 3
9-16 4
17-32 5
33-64 6
65-128 7
129-212 8
213-384 9
385-672 10
673-1248 11
1249-2272 12
2273+ 13

Re: Seeding:
"If a top 8 playoff is used, pairing for the single-elimination rounds generally use a seeding system (1st seed plays 8th seed, 2nd plays 7th, and so on)."

Yes, that's right, it says generally. Mind blown by the specificity.

edit: The MTR confirms that it's a seeded bracket.

double edit: 2^r is the max number of players for r rounds... until you get to 8 rounds... that's weird.

[Dan Turner]

How do you account for the "human" factor. Numbers are all good but you also have piloting skill. My wife can pilot Elves! to a win 80% of the time but if I handed her death and taxes or solidarity she would wash out. You have to have a way to account for the randomness of jank.i.was.not.expecting.that.shit.deck as well. It is all good to plan for all the dtb but that one rogue deck can ruin all your planning.

[Dissolution]

That's why I'm only working with one 'special' deck at a time, in a field of 'zombies' (50/50). Support for multiple decks is in the works, as well as pre and post board matchup percentages.
The most difficult part is the ranking and pairing algorithms, which are working, if slow.
Once I add TopX cutoffs and verify all the math, I'll get the code up on github and post some results.
-T

[4eak]

Any status update? =) I can't wait to try it.


peace,
4eak

[Dissolution]

Sorry, insanely busy with the holidays and work. I have the program mostly completed, will post it to github soon.
If anyone posts requested stats (% for the main deck, # of rounds, cut to top X?), I'll run the sim and post the results. Otherwise, a few days.
-T