Wizards: "Please don't solve our metagames with data-driven methods. Thanks."

[Jamaican Zombie Legend]

There's been a bit of a dustup on Reddit about the reasons behind a feature on MTGGoldfish being pruned. Apparently, Wizards prefers that sites that produce Magic content avoid publishing data that leads to metagames being solved too quickly.

A post from the owner of the Goldfish clarifies:

I want to provide some clarification on the situation:

R&D contacted us and asked us to remove the matchup win percentages and analysis. This was the feature that said Jeskai was 55% favored against Abzan Aggro. This feature was removed back in September. The winning decklists, the metagame breakdown, the strategy articles, and basically everything you see on the site today is unaffected and will continue to be there.

The motivation for removing the data from Magic R&D at Wizards of the Coast:

We asked MTGGoldfish to suspend the Constructed Metagame series on MTGGoldfish that compiles large volumes of MTGO tournament results to paint a picture of a current Constructed format. While these articles are informative and interesting, we feel that this level of data-driven metagame analysis ultimately damages the health of those formats.

Magic thrives on being a recurring puzzle for players to solve, and intensive data mining leads to a more rapid understanding of a new metagame. Solving the puzzle then becomes less interesting, and the format grows stale ahead of its time. The number of tournament-viable deck options for players is reduced, and player interest in the format shrinks along with it. This hurts everyone creating content for Magic players, which is ultimately why we stopped publishing such a high volume of winning MTGO decklists on our own site, and why other Magic content sites have also subscribed to this philosophy after becoming aware of this impact.

We appreciate MTGGoldfish's cooperation in this matter, and while we expect some readers will be disappointed by this news, we hope players will understand that the decision comes from our desire to keep the game as healthy as possible.

The matchup statistics were gathered via Magic Online replays. Wizards can easily block access to this data by either blocking bot access to replays, or removing replays altogether. I would rather remove the data (which we did) rather than having the Magic Online team divert resources to block access to the data (instead of developing and improving Magic Online), or worse yet, everyone losing the ability to watch replays altogether. Either way, Wizards can block this data since Magic Online is the source.

Now if you disagree or want to discuss the merits of having access to such data, that is fine. But please keep it civil, and provide feedback, examples and reasons that are constructive and forward the discussion.

It's an interesting precedent and, judging from the Reddit thread, a rather unpopular one as well.

To be fair to Wizards, throttling data and analysis isn't entirely unwarranted. Magic is a game created by a small handful of people and is, mostly, a physical game. It's hard for a few designers at WotC to be able to match the collective metagaming power of the millions-strong Magic-playing community, making the balancing act they have to perform quite a tough one; the end-users are going to have a lot more play-hours to break the game than the designers had to craft and balance it. And, unlike digital games, there's no luxury of balance tweaks via patching. The widespread use of power-level errata or haphazard bannings would be deleterious to the health of the game; even with the proliferation of smartphones, checking Gatherer before every interaction for the updated Oracle text would be miserable and jam up the flow of gameplay.

All this adds up to a situation in which formats, especially Standard, will be solved soon after the release of new cards, with little recourse available to Wizards besides printing more cards (or speeding up the release/rotation schedule), "gittinggudder" at design (kind of a trivial response*), or engaging in unorthodox tactics like this one to keep things (more) fresh and interesting.

However, this tactic of suppressing critical information might not have any real benefits.

For one thing, the information is still out there, just not as readily available to the average player. Hypothetically, any determined individual or group could replicate the MTGGoldfish data-mining methods and have access to the metagame/matchup stats. The demand for such data is unlikely to falter; competitive players are going to want every morsel of info that could help to bump up their win percentage. And it would seem Wizards hasn't denied access to the replay-watching methods from which the raw data is scraped from. So the (likely) end result is containing the spread of Magic analytics to the more spikey set, giving them an additional leg up over the more "casual" competitors (if that term makes sense).

Secondly, analytics aren't always necessary to understand a format. Many times the best cards/decks are strikingly obvious to anyone with sufficient knowledge of the way Magic works. Most players likely didn't need to see hard numbers to *know* Affinity was busted; anecdotal evidence from local events and games played was mostly enough. Turns out things like "Draw 2 cards" for U and "free" 4/4s on turn two are kinda busted. Same thing with Shards->Zen Jund, Caw-Blade, U/W DelverGeist, and other dominant decks; players could just *tell*. Sure, big data offers empirical backing that's a bit more robust, but scarcely do most players demand such a burden of proof when talking about meta (for better or for worse).

And lastly, a bit on the peripheries, what exactly does data-driven analysis really crowd out? People still enjoy jamming pet-decks, brews, and other choices borne out as less competitive by MagiMetrics™. And there's always budget players who will be precluded from playing the best decks due to monetary constraints. The point is, not everyone will want to act upon the information or even be able to. I guess a lot of the "Gotta make deadline, gotta move stock!" articles penned by pros lose their luster in the face of hardcore analytics. No, that brew designed to sell Narset Transcendent posts a ~40% winrate against the field and is not going to break the meta wide open.

A bit of a rambling post, but I figure this is an interesting enough topic and might spark some discussion.


* In practice, it's true, but not particularly interesting to talk about. Bashing Wizards is fun and all (donning my amatuer-designer hat and getting salty during spoiler season is probably the most amusing thing in Magic!), but it never really goes anywhere. That being said, there are definitely certain trends, likely motivated by sales/financial concerns, that are making contemporary sets/metas a more simplistic and easier to solve.

[phonics]

Statistics require sufficient sample size to have any sort of validity, fresh metas dont have that because they are new, so how would statistics help solve a format, when the meta is already established before any valuable statistics are available? The most undercosted cards drive the meta way more than any kind of statistic ever could, its pretty hard to criticize this analysis when you have stuff like the seige rhino meta of standard.

[Julian23]

Judging from the quote, it sounds like they would be fine with the bots analyzing e.g. a month of Legacy on MODO. Would love to see those results.

[iatee]

Why not just take down the replays? Having them all there is not key to anyone's MTGO experience. I find the data interesting to look at, but I would barely notice if they took it away.

I don't really play Standard, but the Modern matchup data I looked it always seemed kinda off. I think SB cards are so powerful in older formats that the matchup %s are really affected by how people build their boards. In Standard you can improve your matchup vs a deck a bit but there aren't the same type of trump cards. I doubt we would find anything too interesting if they put the data up on the relatively small set of legacy games. Player skill matters a lot more too, especially with something like Miracles.

[menace13]

"Data mining is 2 stronk. Please help us make ourselves look better" -WotC

[Quasim0ff]

I actually think this is entirely reasonable.

I can only imagine this would lead to a terrible metagame otherwise.

Also: Lol'd at people comparing sports to mtg.

[Whitefaces]

I actually think this is entirely reasonable.

I can only imagine this would lead to a terrible metagame otherwise.

Also: Lol'd at people comparing sports to mtg.

Agreed. My initial reaction was 'what the fuck?', but it actually makes sense. Solved metagames are no fun and I welcome anything to try and hinder this.

[Julian23]

So where do you draw the line when it comes to figuring out Standard? Assume someone finds a way to break it within 2 weeks of release; does that mean you would be ok with WotC telling said person not to publish his findings for the sake of the excitement?

It feels so strange to me that people are be ok with a company restricting interpretation of legally collected data about their product.

[Quasim0ff]

So where do you draw the line when it comes to figuring out Standard? Assume someone finds a way to break it within 2 weeks of release; does that mean you would be ok with WotC telling said person not to publish his findings for the sake of the excitement?

It feels so strange to me that people are be ok with a company restricting interpretation of legally collected data about their product.

The problem, from my point of view, is that there will be a clear best deck, ie Caw-Blade, which everyone will then just play. it leads to an inbred environment where people will eventually just grind mirror/specific matches all day, which will in turn most likely push some people to quick magic.

It also leads to a stigma regarding people not playing said deck, as well as questioning their intentions with regards to not playing "the clear best deck".

[HSCK]

But that's untrue, we saw a standard season that wasn't solved at all last year. Ditto with the dice rolling Modern format which is also unsolved despite the enormous amounts of data collected.

[Quasim0ff]

But that's untrue, we saw a standard season that wasn't solved at all last year. Ditto with the dice rolling Modern format which is also unsolved despite the enormous amounts of data collected.

Did the same amount of data exist last year?

[Dice_Box]

Yes.

[jrsthethird]

The problem, from my point of view, is that there will be a clear best deck, ie Caw-Blade, which everyone will then just play. it leads to an inbred environment where people will eventually just grind mirror/specific matches all day, which will in turn most likely push some people to quick magic.

It also leads to a stigma regarding people not playing said deck, as well as questioning their intentions with regards to not playing "the clear best deck".

If Affinity is the best deck and everyone plays it, then the guy who shows up with 4 Shatterstorm and 4 Ancient Grudge main wins the tournament. More people pick up the anti-Affinity deck, and then people create tech for the anti-Affinity mirror. Perhaps a new anti-anti-Affinity deck comes out. The metagame is no longer "solved".

If anything, the data mining just makes life too easy. Maybe it's too fast for a 2 year rotation with only 2 central block themes (multicolor, graveyard, lands, etc.), but moving to an 18-month rotation with 3 different block themes creates more (hypothetically) archetypes and more diversity, which slows down the solving of the format.

Hell, we're seeing all-Abzan top 4's right now. A couple weeks ago MTGGoldfish featured a Standard LD deck that fared very well against the metagame. When every deck is 4 colors with a slow-ish curve and several ETB tapped lands (not to mention only a few sources of the splash color), intelligent land destruction keeps the 4-color goodstuff decks under control. But hey, Wizards hates LD. Just play all the good cards in one deck and spend 700 dollars doing it.

[MGB]

This reminds me of "security through obscurity" in the information security field.

Companies that try to maintain integrity of their services or products by obscuring internal data only succeed in concentrating exploit information in the hands of the hackers. Hackers will find security holes in your products even if your source code and internal documentation is obscured. The people hurt by this are the consumers and the "good guys" who don't have the financial incentives or resources to pore over disassembled binaries.

Similarly, in this case, the people hurt most by this are the average player and the non-pro. People who have the time and resources to generate metagame data will still find a way to do so, but obscuring the data from the average player will only concentrate the advantage in the hands of a few.

[iatee]

They should just hide the replays. They are not key to anyone's enjoyment of MTGO.

[iatee]

'enjoyment'

[menace13]

This reminds me of "security through obscurity" in the information security field.

Companies that try to maintain integrity of their services or products by obscuring internal data only succeed in concentrating exploit information in the hands of the hackers. Hackers will find security holes in your products even if your source code and internal documentation is obscured. The people hurt by this are the consumers and the "good guys" who don't have the financial incentives or resources to pore over disassembled binaries.

Similarly, in this case, the people hurt most by this are the average player and the non-pro. People who have the time and resources to generate metagame data will still find a way to do so, but obscuring the data from the average player will only concentrate the advantage in the hands of a few.

This is what I tweeted out to Worth when he disabled replays in events on MtGO. His reply was "I respectfully disagree". Which is cool and all, but he's still wrong, so yeah... About that.

[rufus]

I actually think this is entirely reasonable.

I can only imagine this would lead to a terrible metagame otherwise.

Also: Lol'd at people comparing sports to mtg.

A degenerate metagame is a consequence of poor set design. Data driven methods can only hasten the arrival of that degenerate endstate. ("Terrible" is subjective - there might be people out there who love seeing Seige Rhino every other game and relish optimizing the Mantis Rider / Crackling Doom split for their Dark Jeskai deck.)

In this particular instance, the huge drop in power level from Khanz to BfZ is also a factor.

[CutthroatCasual]

Players: "Please don't release a super weak set, forcing everyone to play the same cards as before because there's not enough power in the new ones. Thanks."