The Art of Testing - Magic: the Gathering article

Written by Weedmonkey on April 03, 2011

We've all done it. PV does it, and LSV does it. We may stay up all night doing it, or we might just sleep the night away and do it at a Grand Prix site an hour before the tournament begins. Often, how much we do it directly relates to how much success we have - after all, the more we do it the more experience we have, and the more experience we have the more chance we have of coming out on top.

That's right boys and girls, I'm talking about testing.

I've written a number of articles now for Magic-League in regards to deckbuilding. Mostly these have been detailing the processes I take and sharing ideas as to how to effectively build a deck. My last 'strategy' article on UB Proliferate Control didn't go enough into the why for certain card choices I felt, and I realised that I was missing so much behind deckbuilding and playtesting in the article.

What I'm going to be exploring in this article is exactly that - strategies for deckbuilding and playtesting. This is one area that I feel is often not explored by writers, especially considering it is such a crucial tool for tournament players.

Obligatory Disclaimer: This article's author is by no means a professional player, or someone whose words should be considered gospel. However, this article's author has been called competent before, and has also won a tournament using nothing but three Etherwrought Page as win conditions. Take this article as an opportunity to look at new ideas, and you may find ways to improve your own abilities :).

In The Beginning, there was Brian Weissman's...

It seems that for every Magic player, there is a unique twist on testing that they have. Some players take all decks in a given metagame and throw X matches against each other deck to find the one they want to pilot, while others will build a deck from scratch and test it to high heaven. While there is no single 'right' or 'wrong' way to test, you can improve the quality of your own testing by understanding what works and what doesn't in the context of what you are trying to achieve.

Contrary to popular belief, I don't 'rogue deck' for the sake of it. The reason why I often don't pilot common archetypal decks is because I don't feel comfortable playing them, even after playing enough matches with them to get used to the deck. Back in Lorwyn/Shards Standard, I played Mannequin. Before that, it was Teachings in both Ravnica/Time Spiral Standard and Time Spiral Block Constructed. In both cases, I ended up modifying both decks to suit how I play.

The process I'll be using as an example today is the process I actually use to build and test decks:

Define the purpose for selecting a deck
Building a deck
Initial testing and adjusting the deck
Comprehensive testing
Evaluation

The First Step

In everything you do, you should have a reason for doing it - this goes not only in Magic, but in life also. When you're looking for a deck you want to pilot, you should know what you want to achieve with that deck. For most tournament players, this may only extend to 'I want to win a PTQ' or 'I want to day 2 this GP' - and that's okay. You don't need to have a ten page philosophy on why you want to pilot a deck, only an understanding of exactly what you want to achieve with it.

For players looking to develop a deck, this statement should be slightly more complex. While you may want to have good success at a tournament like a tournament player would, it is also crucial that you understand what you want to achieve with a deck within the scope of the format and the metagame. For example, when i first built UB Proliferate Control I wanted to pilot a deck that had a good matchup against both UW and UB control - the two top decks of the format that I saw at the time. By understanding that, I saw the opportunity for a Tezzeret-based control deck in the format, as it could take advantage of lack of pressure from those two decks allowed it to achieve a board position that often the top-tier control decks couldn't regain control from. When the results from Pro Tour - Paris came in, I discarded the deck soon after. The format seemed to shift towards a creature-heavy one, and the advantages that UB Prolfierate Control had - as well as the purpose of the deck in the first place - were shifting away from the makeup of the format.

After discarding UB Proliferate Control and taking the time to look at what decks seemed to be making the biggest waves in Standard, I decided that what I wanted out of a deck was:

I wanted a deck that had a good Caw! Blade matchup. Half of the Pro Tour - Paris top 8 consisted of the UW machine, and there were 24 out of a possible 32 Squadron Hawks in the whole top 8. It seemed that equipment was going to be one of the cornerstones of the format, and it seemed that Caw! Blade would be public enemy number one.
I wanted a deck with at minimum a control streak in it. One of the most important things a player needs to know is what sort of decks they're most comfortable with piloting. While top-level professional players may be able to jump between different styles of decks with minimal difficulty, Peter PTQer more than likely won't have the skill to be able to do that. I know that I'm the kind of player who is most comfortable with being able to have some level over control over how the opponent's board develops. To this end, I know that I can pilot aggro-control decks well, as well as permission-based control decks.

Although it seems trivial, having a purpose is important because it helps to give focus to everything else involved in finding a deck to play. Without that focus, often I've found myself flip-flopping between half a dozen decks without dedicating myself to one, and using cards in decks that aren't at all achieving what I set out to achieve. Giving yourself a purpose helps to avoid that, and ensures that you're giving yourself the best possible opportunity to succeed.

Job Available: Standard Deck

Once you have a purpose, you can start looking at either selecting or developing a deck that you believe will help you best fulfill that purpose. The number of methods a player may employ to find their deck of choice can be great in number. For someone whose only purpose is to perform well in a tournament, they may look at previous tournament results and collate the information before looking at specific decklists, or they may look at decklists that they like the look of for a specific archetype or overarching strategy. For a deck developer, they may look at existing deck ideas for ideas they can modify and tune. They may also decide to collate information on different archetypes in the format and find similar weaknesses in archetypal decks to exploit.

From looking at the PT Paris decklists and results, I knew that I had to keep in mind that Stoneforge Mystic packages were going to be popular for any decks playing white. Initially, I randomly brewed a few decks that I was more or less unimpressed with - some weren't playing out as well as I wanted them to when I got around to the first part of testing with them, while others were just flat-out horrible on paper.

One important thing a deckbuilder should understand is why their ideas aren't working, and why they are working. This becomes especially true when sticking to strategies that existed in previous formats - even though the strategies were successful then, they may or may not also be successful now for a number of reasons (the metagame is different, cards that fulfilled important roles may be missing from the format, and so on). Understanding why a deck doesn't work beyond "it's bad" will help you go a long way to understanding why a deck does work, and through that being able to develop better decks.

I ruled out brewing a deck after a while because I felt I didn't have enough information on Standard to make it work. That left me with finding a pre-existing idea to tune and develop. Looking at the PT Paris decklists, I found this little number:

Ken Yukihiro's Death Snow
Main Deck		Sideboard
4 Birds of Paradise 4 Lotus Cobra 4 Stoneforge Mystic 2 Llanowar Elves 2 Thrun, the Last Troll 4 Jace, the Mind Sculptor 4 Mana Leak 4 Negate 3 Sword of Body and Mind 2 Sword of Feast and Famine	2 Gideon Jura 4 Forest 4 Island 4 Misty Rainforest 4 Seachrome Coast 3 Razorverge Thicket 3 Stirring Wildwood 2 Verdant Catacombs 1 Celestial Colonnade	4 Baneslayer Angel 3 Nature's Claim 3 Ratchet Bomb 2 Linvala, Keeper of Silence 1 Basilisk Collar 1 Mortarpod 1 Thrun, the Last Troll

What initially drew me to this deck was control streak that ran alongside the mana acceleration. It can look a lot like a controllish deck on paper, but playing it you're often pressing attacks with an (often multiple) sworded-up creature of choice. However, in early testing of this particular list, I found that the list wasn't performing as well as it could be. As the aggressive player, often the mana accelerators prove to be non-threats mid-to-late game, even with swords beefing them up. As the defensive player, the control elements of the deck simply aren't enough to be able to swing board position back in your favour. Understanding the strengths and weakness of the deck, I set out to modify the deck to be more thraet-heavy while maintaining the permissionesque control streak that it had in it.

When working with a pre-existing deck, it's important to understand what the most important roles of the deck are, and why those roles exist. From there, you can make informed choices about what cards to use to fulfill what roles. For Death Snow, I identified the roles as:

Mana Accelerators. Part of what made Death Snow work was that it was able to keep up with faster decks through mana acceleration in creating threats (even if it was arming up a mana accelerator with a Sword). I chose to cut the Lotus Cobras, as I find them to be lightning rods against aggro decks (which this deck already has trouble with handling). In their place, I added Viridian Emissaries. The Emissary is a card I find to be excellent in a deck suited to it - it's a great lemming for aggro decks and also helps to fix your mana (something that 3-color decks can have trouble with in this format). The two power is also nothing to scoff at if you need to play beatdown. Finally, I cut the pair of Llanowar Elves as I found ten mana accelerators to be overkill.
Threats. When building a deck that aims to be beating down with creatures, you're usually having to sacrifice ways to control the board in your deck to make room for them. If you're not a weenie deck, your creatures are almost always going to be deployed slower. Because of this, it's important that your creatures are either resilient or going to have a significant effect on the board position. Thrun in Yukihiro's deck is an excellent example of this - many decks are going to have pains trying to dispatch it. Also, Caw! Blade and its ilk really, really have trouble dealing with a resolved Thrun from my experience. I also brought the Baneslayer Angels in from the board. Baneslayer is something I think is underplayed right now - with so many creatures in the format, a 5/5 first striker with flying is something that can get out of hand - quickly.
Equipment Package. The equipment package is a cornerstone of the Death Snow deck (and well, every single other white deck in the format). Yukihiro's split of 3 Sword of Body and Mind/2 Sword of Feast and Famine I felt was completely incorrect. First, Sword of Feast and Famine is relevant against far more archetypes than Sword of Body and Mind is. Secondly, having more than two of any one sword is overkill. To that end, I shaved the excess Swords of Body and Mind and threw in a miser's Sword of Vengance. Although the equip cost is steep, in a deck like this the ability go grant first strike can become surprisingly relevant - especially against other sword-wielding nasties.
Permission Spells. Yukihiro opted for four Mana Leak and four Negate in his deck. In Death Snow, the counterspells serve as a way of pushing through your sworded-up creatures and making sure nothing untoward happens to them. With the way I built the deck, I replaced the Negates with Spell Pierces, as without as much mana acceleration it was often easier to deploy a threat and leave Pierce mana up than it was to leave Negate mana up.

Once I had the main deck sorted out, I ran the deck through some initial testing to see what I needed in a sideboard. Hammer of Ruin was one card in particular i was impressed with - although it seems like a card that doesn't seem to do much unless you're already ahead, the ability to equip and swing with it out of nowhere is not to be underestimated.

Once you've built your deck, leave it alone for a while. Go and do something else - watch TV, take your significant other out to dinner (because he/she will prefer wanting to go out with you to watching you tweak a deck endlessly), or something else. Your brain will do a lot of work behind-the-scenes, and you may pick up things at a second glance that you didn't initially because you were on only one train of thought.

Once you've reached a point where you're happy with your deck, you can begin with getting a feel for it from testing.

Initial T

I know you've heard the term a few times above, now allow me to explain it to you before I get pelted with rotten fruit and vegetables. The initial testing phase is where you start playing with the deck to get a feel for it. You might goldfish with it for a few games first to get a feel of different play sequences before trying to find a few games with opponents.

Your aim here should be to see whether the deck has potential first and foremost. If you find the deck's strategy is different to what you planned, or if you aren't achieving the purposes you had in mind when first looking at a deck, you may find yourself making significant changes to the deck or discarding it completely - and that's okay. The idea of having an initial testing phase is to feel out the deck.

For deck developers, often you'll find yourself jumping back and forth between deckbuilding and testing - and you should be. Your aim at this point should be trying to modify the deck to find a balanced list that does have potential to achieve the purposes you've given it. The key is to test to understand the deck above all other aims.

While it's not exactly clear-cut where you would move on from initial testing, I draw the line where I don't feel I need to look at card choices for the main deck anymore. When you come out of initial testing, you're looking at a more or less balanced list that isn't going to require significant modifications to it. Once you're at that stage, you can look at testing in-depth and fine tuning your deck.

Comprehensive Testing

The aim of comprehensive testing is to thoroughly evaluate the deck's potential within the scope of the format and/or metagame. You should be coming into this stage of development and testing with a good idea of your deck and how it plays. You should only need to make minor adjustments - significant changes can negatively impact on the quality of your testing.

I came into comprehensive testing with this list:

Roo's Death Snow
Main Deck		Sideboard
4 Birds of Paradise 4 Viridian Emissary 3 Stoneforge Mystic 3 Thrun, the Last Troll 3 Baneslayer Angel 3 Mirran Crusader 4 Mana Leak 3 Spell Pierce 3 Jace, the Mind Sculptor 2 Gideon Jura 2 Sword of Feast and Famine	1 Sword of Body and Mind 1 Sword of Vengeance 3 Misty Rainforest 3 Tectonic Edge 2 Razorverge Thicket 2 Seachrome Coast 2 Glacial Fortress 2 Sunpetal Grove 3 Island 3 Forest 3 Plains	3 Divine Offering 3 Ratchet Bomb 3 Condemn 3 Obstinate Baloth 1 Hammer of Ruin 1 Mirran Crusader 1 Thrun, the Last Troll

Now, how do you go about good testing?

The honest answer is I don't know. For every two people you talk to, there seems to be two different ways of going about testing. While over time you find similarities between people, there are so many small variations. What I can say is that what professional players have said on it makes the process seem a lot less involve than it actually is - the difference is that a lot of what they explain is internalised. Additionally, there isn't a great deal of information out there on the testing process itself - most of it is testing within the context of an idea or an recent/upcoming tournament.

The key to testing comprehensively is to be able to get as much information as you can. You want to be able to pilot the deck effectively and adapt to a range of situations and game states you may come across, you want to know what cards are and aren't working effectively, and you want to know what the optimal sideboards are for both archetypes and oponnents. Ultimately, you want to be able draw a definitive conclusion regarding the viability of the deck, and the appropriateness of piloting it.

Some of the more common debates on testing:

Pre- vs post-sideboarded games. There are more than a few players of a school of thought were most of your testing should come from pre-sideboarded matches. I have absolutely no idea where this has come from, but logically this is flawed. More than half of all matches you play will be after you sideboard - to ignore more than half your matches in testing is really punishing yourself.
Playtesting groups - who/when/how?. What makes a good playtest group can mean different things to different people. A popular idea is that your playtest group should only consist of the highest quality players that you can find. This i agree with - to a point. Having quality players on your team will ensure that your testing will have as few play errors as possible, and will help to ensure that against skilled pilots any deck you test you will be well-prepared to pilot. However, there is a chance that your playtest group will suffer from an issue known as 'groupthink' - where the members of a group conform to the same pattern of thinking . This can compromise your playtesting where the group misses issues as a whole - which is why I like to have less experienced players in any playtesting group I'm a part of. Not only do they get experience and the opportunity to improve, but they can sometimes catch things that others in the group may miss.
Quantity of testing. There are some people who believe that cramming in X amount of matches against every deck is effective, especially as you can get all the results and then tune your deck against the weak matchups. The problem with this is that when you tune the deck, your previous testing results are effectively invalidated. You can still draw what you learned about playing the deck and general ideas about the matchup from those results, but if you really want to fine-tune your deck and balance it, then you need to retest. Furthermore, if you cram a lot of testing into a short period of time, you can negatively impact on the quality of your testing by following the same thought patterns every single time. Magic isn't chess - there is a lot of variance from game to game. I would rather play a few matches against each archetype daily with a few rest days than cram a lot of testing in and get stuck on the same ideas.
Testing against archetypal decks only. This is something I've seen some PTQ players instantly snub in testing. Often, it's because they're 'garbage decks' and 'I won't ever actually see one in a tournament'. While you may never see one in a tournament, don't be so quick to dismiss rogue decks as garbage. You will come across someone piloting something that hasn't been discussed to death on a forum sooner or later. You may not need a significant amount of testing against these decks, but it doesn't hurt to see whether your deck is going to have trouble against decks that aren't pre-existing archetypes.

The key in testing is finding out what works for you.

For me, one strategy I use when keeping track of comprehensive testing is using a spreadsheet to track match results, and a notepad file to keep track of notes during those matches. This allows me to visually get an idea of what the deck's statistics look like, as well as track individual areas of the game that I like to focus on (such as number of lands during a game). The spreadsheet for Death Snow you can find here. Go on, open it.

Here's the different parts of the spreadsheet I have:

Archetypes. This is the general summary of records against archetypal decks that I would reasonably expect to surface in a tournament. I keep track of the wins and losses (and the record of the match for those wins/losses) as general ideas of how difficult I expect winning the match against a given archetype is. The win/loss record (as a percentage) I record as a quantitative summary of each matchup.
Match Summary. This is a summary of each individual match. Being able to record information about each individual match can provide far more insight than simply recording the outcome of each match, especially when combined with recording notes about each individual match. One important thing to understand is quantitative analysis can only provide so much information about a match - they need to be combined with qualitative, holistic match records in order to provide the best possible understanding.
Play/Draw. Whether I started a game on the play or on the draw. 0 is Draw, 1 Play. This helps to provide more information about matches when used in conjunction with:
Won/Lost. 0 represents a loss, and 1 a win. Combined with Play/Draw, it helps show me whether I win more on the play or the draw. Additionally, it can help provide more information in regards to whether I'm winning more game pre- or post-sideboard against different archetypes.
Number of Lands. For those unfortunately individuals who I've playtested with extensively, they may have realised I have a notoriously bad habit of trying to cut as many lands from a deck as I can. Having grown up playing in an era where 20 or less lands for an aggro deck was fine, I still find it a little awkward when running anything less than 22 lands in a Standard format is asking for trouble. This statistic allows me to track the amount of lands I'm typically finding myself having at the end of a game, and also helps to see how relevant/useful any nonland mana accelerators are in contributing to developing my mana base.

For the purposes of this article, I've tested twenty matches comprehensively with Death Snow. I generally draw a line at twenty matches as a point where you can generally conclude whether the deck has enough potential to continue testing and refining. Most of us work/study and only play Magic as a hobby - we don't have the time to test enough matches to draw precise data on a given deck. If you were to however, I'd recommend no less than 25-30 matches against each archetype to get an accurate idea - more if the matchups aren't clear-cut in terms of the superior deck.

Once you feel you have enough data to satisfy yourself, you can look at evaluating the data you have.

Judgement Day

When you have enough data from testing comprehensively, you want to look at drawing definitive conclusions about your chosen deck. Throughout your testing, you will have already begun to draw some conclusions about the deck; the main aim about evaluating your deck is trying to provide a definitive answer in your own mind with some sort of evidence (be it quantitative or qualitiative) to support your decision. If you cannot draw a definitive conclusion for every aspect of the deck, then you need to go back to testing until you can. The primary questions you should ask are:

Am I acheiving what I want to achieve with this deck? Go back to your purpose for starting testing back in the beginning of the process. Look at your purpose(s) as a question. If you can't answer that positively, you need to look at your testing and determine whether it's worthwhile going back and developing/testing more to achieve what you're aiming to achieve.
What conclusions can I draw from the matches I have tested? Even if you're evaluating your testing from as few as twenty matches (like I am here), that should at minimum be at least enough data to draw some conclusions from your testing, even if they aren't overly precise.
Is it worth pursuing this particular deck? Why?. This is by far the most important one for both tournament players and deck developers. Throughout your testing, you may or may not have begun to feel whether the deck is worth pursuing. For some, it will come down to whether or not they feel comfortable piloting the deck. For others, there may be strengths or weaknesses that weigh heavily on them when they test the deck. No matter what conclusion you draw, you absolutely must be able to justify it. Firstly, this shows that you have been able to fully understand the deck through development and testing. Secondly, it ensures that you have tested sufficiently to be able to draw a conclusion that it is not worthwhile continuing with the deck.

Let's look at these questions with Death Snow.

Am I achieving what I want to achieve with Death Snow? There were two purposes I wanted to achieve with Death Snow:

I wanted a deck that had a good Caw! Blade matchup. Even though I've only had one test match with Death Snow in-depth with Caw! Blade, I tested extensively with it initially to ensure that the matchup was a positive as I wanted it to be. I can definitely say that for Death Snow the Caw! Blade matchup is excellent. A Thrun that survives to untap is a nightmare for them to deal with, especially with a sword in hand. Post-board it only gets better, as you have a number of options to deal with opposing Swords.
I wanted a deck that had at a minimum a control streak in it. The lesson here is that counterspells do not a control deck make - it's how they fit in with the overall strategy that matters. This deck plays more like an aggro deck that can disrupt key spells if needed than an aggro deck with a control streak. This isn't the kind of deck I'm most suited to piloting.

So, in the end it's a no to this question. While the first purpose is more important than the second, I couldn't say yes to both.

What conclusions can I draw from the matches I have tested? From the matches I've played, there are a number of conclusions I can draw from my experiences. These include:

The main deck is inconsistent. This I draw from losing over half my pre-board games. With only 20 creatures pre-board, there isn't a lot of threat density. Against aggro decks I was often boarding out all of my counters and going up to 24 or 25 creatures. Those games were often erratic because I would be drawing counters when I didn't need them (either because I wasn't wanting to keep counter mana up or it was too far along in the game for them to be relevant). Post-board I was able to provide more accurate answers to individual archetypes and thus my results improved.
The deck can easily seal the deal against aggro decks with a good draw. Let's face it - Baneslayer Angel was a $50+ rare at one point because you had to deal with it immediately or lose the game. This is still the case - against aggro decks I found that untapping with Baneslayer often meant that I won the game whether it was in one turn or ten. Death Snow on a good draw can win against an aggro deck quite easily - the issue is that the deck can't do it consistently.
The matchups which theoretically should be good aren't that good. This in particular is aimed at Red Deck Wins and Vampires - two archetypes which on paper should favor Death Snow quite well. Against RDW, even the life gain that Obstinate Baloth and Baneslayer provides often isn't enough to handle the large amount of reach that the deck has. Against Vampires, while the record on paper is positive the games were often close - a better draw as opposed to a superior deck.
The deck was hurting for removal. This is a major point. While the deck did pilot well overall against aggro decks, they could have been significantly improved with access to more removal than the deck currently has available to it. Surprisingly, this was most relevant against UB Control - if a Grave Titan or other Big Bad Creature^TM resolved, it was often impossible to deal with it permanently.
The deck couldn't operate on any less than five lands. While I did win some games on less than five lands, overall I needed five lands in order to do what I needed to do with the deck. Even with eight mana accelerators, 24 lands seems to be too low for the deck.
The deck could afford more card draw. Mid- to late-game after you have played out your hand you often find yourself knocking on the top of your deck for more threats - an issue exacerbated by the low number of creatures in the deck.

As you can see, I was left wanting by and large by the deck. It always felt promising, but it always seemed to be missing something to it. I kept the main deck intact for the purposes of this article - ideally if I was to continue with this deck I would pull out the counterspells, but that wouldn't match the general idea of Death Snow that Ken Yukihiro piloted. From these conclusions, it's clear that I would need to go back to developing the deck and going through the testing process again - and that's okay. Just becuse you identify a number of flaws in the deck doesn't mean it isn't viable - you may only need to change some cards to improve the deck's weaknesses.

Is it worth pursuing this particular deck? Why? The all-important question. When looking at this question I personally am looking for a combination of positive numerical results and positive holistic records from my notes to determine whether the deck is worth pursuing. Even though the number of in-depth matches I've played with Death Snow is few, I'd still be looking numerically for a 60% win record at least in order to satisfy my own standard for pursuing a deck. From the spreadsheet you can see that over 20 matches the record was 50% - far below that. Even taking into account the last three matches were all losses, the deck was hovering between 55 and 60% throughout the majority of testing.

Taking into account the testing, the answers to my questions above and my own feelings about the deck, I'd discard it. If I wanted to play this style of Stoneforge Mystic-packaged aggro deck with the extra removal it needs, I would rather play something like Naya (which based on my testing with Death Snow seems to be a worthwhile avenue of exploration). As for Death Snow as an archetype, I think it has some significant flaws that prevent it from becoming a worthwhile option.

Famous Last Words

There's a lot of information here, and it is by no means gospel. My testing methods aren't perfect, and a lot of what is contained in here could make for good topics of discussion. What I want people to take away from reading this article is ideas to potentially improve their quality of testing, and to understand that if you want to succeed in Magic you need to be willing to put in the effort to get there. If people liked the spreadsheet I used to document my own testing, I'll also give you this template for your own personal use.

I'd also like to thank tchiseen, Bigshow, EatsMortals and Coboney for their putting up with me and helping me collate what I needed for this article. You're all awesome guys :D.

- Roo

Back to Magic: the Gathering Articles