The Art of Testing
Written by Weedmonkey on April 03, 2011
We've all done it. PV does it, and LSV does it. We may stay up all night doing it, or we might just sleep the night away and do it at a Grand Prix site an hour before the tournament begins. Often, how much we do it directly relates to how much success we have - after all, the more we do it the more experience we have, and the more experience we have the more chance we have of coming out on top.
That's right boys and girls, I'm talking about testing.
I've written a number of articles now for Magic-League in regards to deckbuilding. Mostly these have been detailing the processes I take and sharing ideas as to how to effectively build a deck. My last 'strategy' article on UB Proliferate Control didn't go enough into the why for certain card choices I felt, and I realised that I was missing so much behind deckbuilding and playtesting in the article.
What I'm going to be exploring in this article is exactly that - strategies for deckbuilding and playtesting. This is one area that I feel is often not explored by writers, especially considering it is such a crucial tool for tournament players.
Obligatory Disclaimer: This article's author is by no means a professional player, or someone whose words should be considered gospel. However, this article's author has been called competent before, and has also won a tournament using nothing but three Etherwrought Page as win conditions. Take this article as an opportunity to look at new ideas, and you may find ways to improve your own abilities :).
In The Beginning, there was Brian Weissman's...
It seems that for every Magic player, there is a unique twist on testing that they have. Some players take all decks in a given metagame and throw X matches against each other deck to find the one they want to pilot, while others will build a deck from scratch and test it to high heaven. While there is no single 'right' or 'wrong' way to test, you can improve the quality of your own testing by understanding what works and what doesn't in the context of what you are trying to achieve.
Contrary to popular belief, I don't 'rogue deck' for the sake of it. The reason why I often don't pilot common archetypal decks is because I don't feel comfortable playing them, even after playing enough matches with them to get used to the deck. Back in Lorwyn/Shards Standard, I played Mannequin. Before that, it was Teachings in both Ravnica/Time Spiral Standard and Time Spiral Block Constructed. In both cases, I ended up modifying both decks to suit how I play.
The process I'll be using as an example today is the process I actually use to build and test decks:
The First Step
In everything you do, you should have a reason for doing it - this goes not only in Magic, but in life also. When you're looking for a deck you want to pilot, you should know what you want to achieve with that deck. For most tournament players, this may only extend to 'I want to win a PTQ' or 'I want to day 2 this GP' - and that's okay. You don't need to have a ten page philosophy on why you want to pilot a deck, only an understanding of exactly what you want to achieve with it.
For players looking to develop a deck, this statement should be slightly more complex. While you may want to have good success at a tournament like a tournament player would, it is also crucial that you understand what you want to achieve with a deck within the scope of the format and the metagame. For example, when i first built UB Proliferate Control I wanted to pilot a deck that had a good matchup against both UW and UB control - the two top decks of the format that I saw at the time. By understanding that, I saw the opportunity for a Tezzeret-based control deck in the format, as it could take advantage of lack of pressure from those two decks allowed it to achieve a board position that often the top-tier control decks couldn't regain control from. When the results from Pro Tour - Paris came in, I discarded the deck soon after. The format seemed to shift towards a creature-heavy one, and the advantages that UB Prolfierate Control had - as well as the purpose of the deck in the first place - were shifting away from the makeup of the format.
After discarding UB Proliferate Control and taking the time to look at what decks seemed to be making the biggest waves in Standard, I decided that what I wanted out of a deck was:
Although it seems trivial, having a purpose is important because it helps to give focus to everything else involved in finding a deck to play. Without that focus, often I've found myself flip-flopping between half a dozen decks without dedicating myself to one, and using cards in decks that aren't at all achieving what I set out to achieve. Giving yourself a purpose helps to avoid that, and ensures that you're giving yourself the best possible opportunity to succeed.
Job Available: Standard Deck
Once you have a purpose, you can start looking at either selecting or developing a deck that you believe will help you best fulfill that purpose. The number of methods a player may employ to find their deck of choice can be great in number. For someone whose only purpose is to perform well in a tournament, they may look at previous tournament results and collate the information before looking at specific decklists, or they may look at decklists that they like the look of for a specific archetype or overarching strategy. For a deck developer, they may look at existing deck ideas for ideas they can modify and tune. They may also decide to collate information on different archetypes in the format and find similar weaknesses in archetypal decks to exploit.
From looking at the PT Paris decklists and results, I knew that I had to keep in mind that Stoneforge Mystic packages were going to be popular for any decks playing white. Initially, I randomly brewed a few decks that I was more or less unimpressed with - some weren't playing out as well as I wanted them to when I got around to the first part of testing with them, while others were just flat-out horrible on paper.
One important thing a deckbuilder should understand is why their ideas aren't working, and why they are working. This becomes especially true when sticking to strategies that existed in previous formats - even though the strategies were successful then, they may or may not also be successful now for a number of reasons (the metagame is different, cards that fulfilled important roles may be missing from the format, and so on). Understanding why a deck doesn't work beyond "it's bad" will help you go a long way to understanding why a deck does work, and through that being able to develop better decks.
I ruled out brewing a deck after a while because I felt I didn't have enough information on Standard to make it work. That left me with finding a pre-existing idea to tune and develop. Looking at the PT Paris decklists, I found this little number:
What initially drew me to this deck was control streak that ran alongside the mana acceleration. It can look a lot like a controllish deck on paper, but playing it you're often pressing attacks with an (often multiple) sworded-up creature of choice. However, in early testing of this particular list, I found that the list wasn't performing as well as it could be. As the aggressive player, often the mana accelerators prove to be non-threats mid-to-late game, even with swords beefing them up. As the defensive player, the control elements of the deck simply aren't enough to be able to swing board position back in your favour. Understanding the strengths and weakness of the deck, I set out to modify the deck to be more thraet-heavy while maintaining the permissionesque control streak that it had in it.
When working with a pre-existing deck, it's important to understand what the most important roles of the deck are, and why those roles exist. From there, you can make informed choices about what cards to use to fulfill what roles. For Death Snow, I identified the roles as:
Once I had the main deck sorted out, I ran the deck through some initial testing to see what I needed in a sideboard. Hammer of Ruin was one card in particular i was impressed with - although it seems like a card that doesn't seem to do much unless you're already ahead, the ability to equip and swing with it out of nowhere is not to be underestimated.
Once you've built your deck, leave it alone for a while. Go and do something else - watch TV, take your significant other out to dinner (because he/she will prefer wanting to go out with you to watching you tweak a deck endlessly), or something else. Your brain will do a lot of work behind-the-scenes, and you may pick up things at a second glance that you didn't initially because you were on only one train of thought.
Once you've reached a point where you're happy with your deck, you can begin with getting a feel for it from testing.
I know you've heard the term a few times above, now allow me to explain it to you before I get pelted with rotten fruit and vegetables. The initial testing phase is where you start playing with the deck to get a feel for it. You might goldfish with it for a few games first to get a feel of different play sequences before trying to find a few games with opponents.
Your aim here should be to see whether the deck has potential first and foremost. If you find the deck's strategy is different to what you planned, or if you aren't achieving the purposes you had in mind when first looking at a deck, you may find yourself making significant changes to the deck or discarding it completely - and that's okay. The idea of having an initial testing phase is to feel out the deck.
For deck developers, often you'll find yourself jumping back and forth between deckbuilding and testing - and you should be. Your aim at this point should be trying to modify the deck to find a balanced list that does have potential to achieve the purposes you've given it. The key is to test to understand the deck above all other aims.
While it's not exactly clear-cut where you would move on from initial testing, I draw the line where I don't feel I need to look at card choices for the main deck anymore. When you come out of initial testing, you're looking at a more or less balanced list that isn't going to require significant modifications to it. Once you're at that stage, you can look at testing in-depth and fine tuning your deck.
The aim of comprehensive testing is to thoroughly evaluate the deck's potential within the scope of the format and/or metagame. You should be coming into this stage of development and testing with a good idea of your deck and how it plays. You should only need to make minor adjustments - significant changes can negatively impact on the quality of your testing.
I came into comprehensive testing with this list:
Now, how do you go about good testing?
The honest answer is I don't know. For every two people you talk to, there seems to be two different ways of going about testing. While over time you find similarities between people, there are so many small variations. What I can say is that what professional players have said on it makes the process seem a lot less involve than it actually is - the difference is that a lot of what they explain is internalised. Additionally, there isn't a great deal of information out there on the testing process itself - most of it is testing within the context of an idea or an recent/upcoming tournament.
The key to testing comprehensively is to be able to get as much information as you can. You want to be able to pilot the deck effectively and adapt to a range of situations and game states you may come across, you want to know what cards are and aren't working effectively, and you want to know what the optimal sideboards are for both archetypes and oponnents. Ultimately, you want to be able draw a definitive conclusion regarding the viability of the deck, and the appropriateness of piloting it.
Some of the more common debates on testing:
The key in testing is finding out what works for you.
For me, one strategy I use when keeping track of comprehensive testing is using a spreadsheet to track match results, and a notepad file to keep track of notes during those matches. This allows me to visually get an idea of what the deck's statistics look like, as well as track individual areas of the game that I like to focus on (such as number of lands during a game). The spreadsheet for Death Snow you can find here. Go on, open it.
Here's the different parts of the spreadsheet I have:
For the purposes of this article, I've tested twenty matches comprehensively with Death Snow. I generally draw a line at twenty matches as a point where you can generally conclude whether the deck has enough potential to continue testing and refining. Most of us work/study and only play Magic as a hobby - we don't have the time to test enough matches to draw precise data on a given deck. If you were to however, I'd recommend no less than 25-30 matches against each archetype to get an accurate idea - more if the matchups aren't clear-cut in terms of the superior deck.
Once you feel you have enough data to satisfy yourself, you can look at evaluating the data you have.
When you have enough data from testing comprehensively, you want to look at drawing definitive conclusions about your chosen deck. Throughout your testing, you will have already begun to draw some conclusions about the deck; the main aim about evaluating your deck is trying to provide a definitive answer in your own mind with some sort of evidence (be it quantitative or qualitiative) to support your decision. If you cannot draw a definitive conclusion for every aspect of the deck, then you need to go back to testing until you can. The primary questions you should ask are:
Let's look at these questions with Death Snow.
Am I achieving what I want to achieve with Death Snow? There were two purposes I wanted to achieve with Death Snow:
So, in the end it's a no to this question. While the first purpose is more important than the second, I couldn't say yes to both.
What conclusions can I draw from the matches I have tested? From the matches I've played, there are a number of conclusions I can draw from my experiences. These include:
As you can see, I was left wanting by and large by the deck. It always felt promising, but it always seemed to be missing something to it. I kept the main deck intact for the purposes of this article - ideally if I was to continue with this deck I would pull out the counterspells, but that wouldn't match the general idea of Death Snow that Ken Yukihiro piloted. From these conclusions, it's clear that I would need to go back to developing the deck and going through the testing process again - and that's okay. Just becuse you identify a number of flaws in the deck doesn't mean it isn't viable - you may only need to change some cards to improve the deck's weaknesses.
Is it worth pursuing this particular deck? Why? The all-important question. When looking at this question I personally am looking for a combination of positive numerical results and positive holistic records from my notes to determine whether the deck is worth pursuing. Even though the number of in-depth matches I've played with Death Snow is few, I'd still be looking numerically for a 60% win record at least in order to satisfy my own standard for pursuing a deck. From the spreadsheet you can see that over 20 matches the record was 50% - far below that. Even taking into account the last three matches were all losses, the deck was hovering between 55 and 60% throughout the majority of testing.
Taking into account the testing, the answers to my questions above and my own feelings about the deck, I'd discard it. If I wanted to play this style of Stoneforge Mystic-packaged aggro deck with the extra removal it needs, I would rather play something like Naya (which based on my testing with Death Snow seems to be a worthwhile avenue of exploration). As for Death Snow as an archetype, I think it has some significant flaws that prevent it from becoming a worthwhile option.
Famous Last Words
There's a lot of information here, and it is by no means gospel. My testing methods aren't perfect, and a lot of what is contained in here could make for good topics of discussion. What I want people to take away from reading this article is ideas to potentially improve their quality of testing, and to understand that if you want to succeed in Magic you need to be willing to put in the effort to get there. If people liked the spreadsheet I used to document my own testing, I'll also give you this template for your own personal use.
I'd also like to thank tchiseen, Bigshow, EatsMortals and Coboney for their putting up with me and helping me collate what I needed for this article. You're all awesome guys :D.
by WeedSeed on 2011-04-03 13:42 CET
really nice article and mostly usefull
by EatsMortals on 2011-04-03 14:14 CET
great article, gogo infect :)
by GreenBear on 2011-04-03 16:33 CET
You missed out the sort of try out approach to testing. You should always have in mind i am happy with this card, I am in love with this card but this card I am not happy with and would like to try something else in its place. You should constantly be evaluating your cards as you are going along with testing, looking whats winning games, whats to slow, whats underpowered, why cards are disappointing you as this may give you more idea what you need in there place.
by alexandria1 on 2011-04-03 22:55 CET
Great article! Nice job Roo.
by Bigshow on 2011-04-04 03:01 CET
I'll help you collate any time
by tchiseen on 2011-04-04 05:44 CET
tldr; i was expecting more of a shout out. BOO!
by turdferguson on 2011-04-05 09:09 CET
i might be wrong, but 59 cards ftw :P :)
by mangoface on 2011-04-06 06:24 CET
my favorite part is when you touch the fact that you should brainstorm why you want to build a good deck; like what is the purpose of this deck; my purpose of my decks is to have the highest winning record whats your?
by warpg8 on 2011-04-07 05:55 CET
this article is good in and of the testing method, however, this particular testing method in standard is basically straight garbage because the card pool is so small and the power is so centralized in a select few cards. if you look at most standard decks, they're not running "new tech", especially maindeck.
Download & Guides