So you have an idea you throw it onto some cardboard and you test it yourself to make sure it makes sense. Great! Now you take your idea and walk out into the real world. After each play you are bombarded left and right with new idea's and people trying to change the game. For the most part it may seem like noise. What do you do? Ignore them and push on? Try to implement their ideas?
I have done several big conventions and many many many playtest sessions on Robit Riddle at this point. I might not know all the answers yet, but this is how I approach playtest feedback. I haven't done much blind playtesting yet, so this is mainly for feedback when I am there and easily accessible. I will review my plan for blind playtesting at the end of this article.
Bias
Feedback is tricky because there is almost always bias on the givers part. This might be based on their interests or other players influence or who knows what. Getting past that prejudice and finding the gems that will help you might take some effort, but in my experience is completely worth it.
There is also two forms of constructive feedback. The first is, I didn't like thing x. The second is, you should do x or y. The first form is the easiest to deal with, because it gets right to the point. They are telling you what they perceived to be an issue. The later however is harder, because you have to dig out of them what they thought the issue was to begin with.
Before the Playtest Starts
Getting feedback starts way before the end of the game, it starts when you first meet everyone. I try figure out each persons background to get to know what they like and don't like. This will give me a basis for where their bias may come from.
For instance if I have a hard-core RPGer sitting down to play my extremely light RPG like game. They could get the idea of a light RPG or they could try to change it into a more heavy system.
During the Game
I think this is the most important part. Try not to playtest with the group. Instead observe from afar. I get way more information by watching the playtesters and their reactions to things than from their feedback. Reading body language is much more important than listening to what is said.
Take notes! Even better record video for later.
When someone makes a confused face take note of the current state of the game and why the confusion occurred. Was it poorly explained? or was it overly complex? I try to interpret what happened to the best of my ability, while trying to keep out my biased point of view.
After the Game
My own bias can get in the way. What I think this players motivation may sway my opinion of their feedback. To avoid this, I treat all feedback as important and playtesters with respect. I try to get to the root of what their issue is, and why they think it is an issue. I take notes and show that I am listening and care about what they have to say.
What I don't do is defend my game. My game is something that is completely in flux and can be changed at any time to make it an even better experience. If I am constantly defending it, then something seems very wrong to me. Being defensive will immediately turn off the feedback loop, and I could miss out on some valuable data.
With these steps this is also how I convert some of the playtesters into fans and evangelists for me and my game. This is very helpful for creating a follow! But I don't try to focus on this, I just let it happen naturally.
Playtest Review
After the playtest or day of playtesting I review my notes. My notes are always about what the players thought were the issues, not the solutions. Well I sometimes write down the solutions if I think they are really great.
I try to pick issues that were the most consistent across all playtesters and resolve those items first. Then I iterate on that and any other issues that seem important. I don't act on all of the feedback, but I do keep a running list of possible solutions to issues. I make small changes and playtest myself.
Then repeat this whole process again and again until I have a great game.