Comprehensive Legacy Metagame Analysis and a Request for Data

It would be useful to compile systematic data on Legacy format trends to help answer questions we've all discussed on the Source, such as the following:

• What are the most prevalent decks in the format?
• What are the strongest decks in the format?
• Are there decks that can prey on the current metagame?
• Do the decks that appear most frequently in Top 8's do so primarily because so many players are playing them, because the deck is inherently powerful, or because the deck has the tools to help stronger pilots outplay their opponents?

Unfortunately, none of these answers are really meaningful without large data sets, and ever since SCG stopped publishing the Too Much Information series, the Legacy community hasn't had access to these data. Furthermore, no one has the time to compile all these datasets, and a group effort would cause in inconsistencies in formatting that would make analysis difficult.

To address this problem, I've written computer scripts to help streamline the number crunching, and I need your help as tournament organizers to provide me with tournament results and decks associated with each participant. This would represent an attempt to capture every single match in these tournaments.

All I need from you for each tournaments are two things:

• Match results: I will take either WER (Wizard Event Reporter) files, or a spreadsheet with match results. If you provide me with a spreadsheet, please observe the formatting rules discussed in detail in the following section.
• Decks associated with each player: Again, please observe the formatting rules discussed below.

Data Submission

If you're interested in submitting data files to me, please send me a PM, and I will provide you with my email address. WER files are preferred for providing match results, because they are consistently formatted, but otherwise the following also works:

http://i.imgur.com/LDJb1Na.png

Note that there are generally four fields for each match result. The first column represents the round number, the second column the name of the first player, the third column the match result (in W-L-D format to represent games won, lost, and drawn by the first player), and the fourth column the opponent for the player in column 2. The screenshot above shows match results from a fictious "Duel of the Planeswalkers" (this actually is derived from real tournament data, but with participant names changed and decks arbitrarily assigned to each player). In Round 6, Oona defeated Mangara 2-0, whereas Elspeth Tirel defeated Venser in two games.

Note that there are only two fields when a player has a bye. In this case, Suleiman has a Round 6 bye.

In addition, there is a fifth field with "ID" to denote when a drawn match is done so intentionally. Note that I automatically assume any matches that are reported as 0-0-0 or 0-0-3 are intentional draws.

http://i.imgur.com/oDzsAxJ.png

Please use Q, S, and F to denote the quarterfinal (top 8), semifinal (top 4), and final (top 2) rounds, as illustrated above.

Note that participant names can be given in any format, as long as they are consistent from round to round. That is, don't write "Elspeth Tirel" in one round then change it to "Tirel, Elspeth" in the next.

http://i.imgur.com/0qmlvRs.png

Please submit decks using an Excel spreadsheet or a CSV file. The first column denotes the player name and should be consistent with the names used in the match results. The second field represents the deck archetype associated with the player in column 1. If you want to include subarchetype information, please do so in the second field in parenthesis as shown above. In this case, Augustin IV played the Esper subarchetype of the Stoneblade deck.

There is a lot of debate on how to name decks, and there is just as much controversy on whether a deck qualifies as an archetype or a subarchetype. I am likely to rename a lot of archetypes from the submissions I receive for the sake of consistency.

Data Output

For every tournament data submission I receive, I will generate a spreadsheet (download) and a series of graphs to illustrate what happened at the tournament. To convince you tournament organizers to provide me with data, I'll let you in on how sweet this is going to be.

http://i.imgur.com/QCh0zTq.png

The first sheet of the spreadsheet will list every competitor and their associated deck by final Swiss standings. All fields are filterable and sortable, so you can easily look up how a player or an archetype performed. Tiebreakers are automatically calculated from match results.

http://i.imgur.com/Z8XNyWp.png

The second sheet of the spreadsheet will list every match result (win, loss, draw, bye, ID) for each round, along with game results. Again, all fields are filterable and sortable.

http://i.imgur.com/sVHGAZO.png

The third sheet provides a breakdown of how each archetype performed, along with its prevalance in the field.

http://i.imgur.com/SNclTzX.png

The fourth sheet provides a breakdown of how the most popular archetypes performed against each other.

http://i.imgur.com/LNAv71L.png

A series of pie graphs illustrates the prevalance of each deck within the top 8, top 16, top 32, top 64, and in the general field.

http://i.imgur.com/SRDsBcU.png

A stacked bar plot will similarly convey the composition of archetypes in the top 8 or in other standing groups.

http://i.imgur.com/mRrfkPR.png

And a collection of histograms helps determine whether a particular deck over or underperformed. In this example, Jund was quite underwhelming, as all of its players finished outside of the top 64. (Note: decks were arbitrarily assigned to each player, don't read too much into this fake data!)

A final note: I assume any data that is shared with me is fair game for me to make public on the Source. If you have reservations about this arrangement, please let me know or I will assume otherwise.

I have some tournament data, but it's not available as a WER event file or in a spreadsheet as formatted. Can you still process it?

As long as you have results for all matches played, decklists associated with all players, and all the names are internally consistent, I am willing to take a look. I am also currently able to parse HTML files as outputted by SCG.

How exactly are tiebreakers calculated?

Tiebreaker rules are described in the official DCI guide. Two other considerations which I have discovered that are not mentioned are as follows: (1) the minimum match win contribution from each opponent is 1/3, not 0.33; and (2) tiebreakers are locked after a player drops, so subsequent rounds will not affect the OMW% or OGW% of a dropped player. For instance, in the examples above, if Freyalise dropped after round 6, her OMW% and OGW% would not be affected by the outcome of any of Worzel's matches beginning with round 7 (nor those of any of her other opponents).

Match results from elimination rounds are NOT used in the determination of tiebreakers.

Why are the results of some of your elimination rounds denoted as "1-0" or "0-1"?

When I have information about which player advanced in each elimination round, but not the number of games the match took, I indicate the winner as having won 1-0.

In the event of a top X split in which the matches are not played out, but one participant concedes to another (e.g. for byes, points, etc.) and for which no officially reported DCI result is available to me, I will also indicate the outcome as 1-0.

Why do your standings differ from the final Swiss standings available elsewhere?

There are several possible sources of discrepancy:

• Since 8/29/14, I have used results from elimination rounds (when available) to modify the standings of top 8 participants.
• In rare instances when two participants have the OMW%, GW%, and OGW% tiebreakers, the relative ranking of those participants is determined arbitrarily in my calculations.
• I have noticed occasional floating point errors when comparing two players. This describes a situation when Player A has 15 points, 66.6667% OMW, and a 80% GW, while player B has 15 points, an identical 66.6667% OMW, and a 75% GW. The computer may decide that the representation for player B's 66.6667% OMW is greater than the representation for player A's 66.6667% OMW. To avoid this problem, I compare percentages to only 6 decimal points, but please let me know if you suspect a recurrence of this bug.
• There may be inconsistent data entry regarding matches played, particularly if match results for the round are published and then subsequently readjusted when players later realize a match slip was filled out incorrectly. I have methods to check for this problem, but this may certainly slip through the crack.
• I have noticed some inconsistency in publically match results and standings regarding how disqualified players affect OMW% and OGW%; fortunately, these are rare occurrences.
• And of course, there can certainly be problems with my implementation of tiebreaker calculations, so please let me know if you observe any problems!

Why do you keep track of Byes and IDs?

I score Byes as wins in the calculation of final standings, as outlined by the official DCI rules above. I do not include them in tracking performance of archetypes.

I also score IDs as draws in the calculation of final standings, as outlined by the official DCI rules above. I do not include them in tracking performance of archetypes. Note that as there is no way to officially report an intentional draw, the decision to flag a draw as an ID is a judgment call either made by me or otherwise annotated by the tournament organizer providing the information.

Your section on Archetype performance is inconsistent with the Match Results.

Please note that I do not include results from Byes, IDs, and mirror matches in assessment of archetype performance. I also exclude pseudo-mirror matches involving two different subarchetypes of the same deck (for instance, Jund vs. Punishing Jund).

What other features can we expect in the future?

If I get more data, I'd like to keep track of other general pieces of information such as:

• How much tiebreakers change in the final round of an X round tournament?
• How often does a player in top 8 contention intentionally draw himself into 9th or 10th?
• How much more often does the higher seed win in elimination rounds? Does this correlate with performance in the Swiss rounds? How much more likely is the higher seed to win by virtue of being on the play? (Note: a 1st seed who went 9-1 has won 90% of his played matches, while a 2nd seed who went 8-0-2 with two IDs has won 100% of his played matches, so one might indicate that the 2nd seed as having performed better in Swiss despite not having the draw/play advantage.)

Events with Complete Information

• Dark Tower Games Open, August 2014 (spreadsheet) (graphs)
• AZMagicPlayers.com Legacy Championship, September 2014 (spreadsheet) (graphs)

Events with Incomplete Information

Match results are not available for these events, but the final Swiss standings are.

• Knightware, June 2013 (graphs)
• Knightware, September 2013 (graphs)
• Knightware, October 2013 (graphs)
• Knightware, December 2013 (graphs)
• Knightware, January 2014 (graphs)
• Knightware, February 2014 (graphs)
• Knightware, April 2014 (graphs)
• Knightware, June 2014 (graphs)
• Knightware, August 2014 (graphs)
• Knightware, all (graphs)

Wow. This is fantastic. A huge undertaking. Thank you for attempting something like this.

Why did SCG stop with the TMI series? Was there any official statement?

I know Worth tried to black out the Hive Mind on MtGO by reneging on their daily events publishing. There wasn't enough of an outcry to reverse it. Seems as if Wizards loves to try and keep its players in the dark about the game they play. I suspect it has to do with their increasingly abhorrent design paradigm. And serves as a way to thwart players from exposing the flaws in such designs and mismanagement of printings and formats. That is to say WotC cannot handle players solving formats because of their printing philosophy and consequently punish the player base in response.

Sorry for the tin foil hat rant and hats off to you for this.

Nice initiative. Don't mean to be negative nancy but the problem I've found with these long term metagame analysis is the difficulty in naming and categorizing decks. There is always a statistically meaningful percentage of the metagame which place with non-standard 'in-between' decks. You look at Esperblade and realize that they are running Deathrite Shaman as their only black source or see some Deathblade lists with stuff like Lingering Souls, so you start losing consistency in your naming conventions. You take a look at Bant lists, you see some Maverick decks with a minor splash of blue and some Stoneblade decks with a minor splash of green being lumped together as the same archetype. You start lumping various nuances of BUG control and BUG midrange together due to minor variences but then end up losing the clarity on the main archetype for each. UW Thopters lists start popping as Miracles list with SFM and vice versa.

Another problem is deck evolution. A deck starts as a minor variant of a current archetype so you ignore the differences and still lump them together, 6 months down the road it starts occupying its own space in the meta and you fail to capture the rise of the deck. I would be amazed if there could be a consistent way of managing all this variance to do a proper metagame analysis given the data.

I'm wet with excitement. I'll help you out.

-Matt

Let it be known that I appreciate the Vorthosness you displayed when pairing Legacy decks up with particular MTG personalities. That work has not gone unnoticed.

Quote:

---

Originally Posted by menace13

Wow. This is fantastic. A huge undertaking. Thank you for attempting something like this.

Why did SCG stop with the TMI series? Was there any official statement?

I know Worth tried to black out the Hive Mind on MtGO by reneging on their daily events publishing. There wasn't enough of an outcry to reverse it. Seems as if Wizards loves to try and keep its players in the dark about the game they play. I suspect it has to do with their increasingly abhorrent design paradigm. And serves as a way to thwart players from exposing the flaws in such designs and mismanagement of printings and formats. That is to say WotC cannot handle players solving formats because of their printing philosophy and consequently punish the player base in response.

Sorry for the tin foil hat rant and hats off to you for this.

---

SCG events used to be 300-400 players for Standard and 150-250 for Legacy - so the amount of time it takes to generate them now is probably a lot since the events have doubled in size. The ROI probably isn't in there for them.

I'd love to see the results. Some hard proof that Brainstorm decks are outperforming non-BS decks would be nice instead of the common "Hurr, maybe there are just 90% Brainstorm decks" retort.

Since there are match-ups included, it would also allow to understand why particular decks performed well/not-so-well, which is great.

In any case, I applaud to the effort.

Note that if you cannot send a WER file for whatever reason, I can also parse anything generated from the Wizards Event Reporter, such as results pages from SCG.

I have done a similar project in the past and its a really exhausting thing to do. Back then I used data from the "Too Much Information" series too. Also I was collecting data by hand asking people what they played, against which deck they played and how it went. You had a really good idea in asking the public for help with the data. My project "failed" because I couldn't get a representative number of match-ups from a lot of decks. I'm curious about the progress of your project.

Quote:

---

Originally Posted by GoboLord

I have done a similar project in the past and its a really exhausting thing to do. Back then I used data from the "Too Much Information" series too. Also I was collecting data by hand asking people what they played, against which deck they played and how it went. You had a really good idea in asking the public for help with the data. My project "failed" because I couldn't get a representative number of match-ups from a lot of decks. I'm curious about the progress of your project.

---

Several Sourcers and other local tournament organizers have told me that they would send me data. I currently have numbers compiled for one local 52-man tournament last week, but I'm waiting for confirmation that I'm allowed to make things public. We'll see if I do actually get enough data to draw meaningful conclusions.

All of my work is frontloaded in the scripts, so once I get match results and decks for each player, I can generate spreadsheets and graphs similar to TMI within minutes, so at least the data processing end isn't going to be bad.

As a side project, I've gone ahead and downloaded results for over 150 SCG Legacy Opens, although some of them have incomplete links. I won't have decklists associated with them, but I hope to generate meaningful data about the play/draw advantage in elimination rounds, distribution of OMW% shift after a last round ID, etc.

I received permission to post compiled information from a recent event in the Pacific Northwest (Dark Tower Games Open), and I received partial information from Koby regarding eight Knightware events from April 2013 to August 2014, from which I was able to put together graphs (but no spreadsheet).