- Fun is not measurable, but reflections of it are.
- Determining Proxies in terms of accuracy, precision, and granularity of measurement is important.
- Definitions and importance of Hypotheses and Theories
- Different game analysis methods
Update from Future!
I created this post from Roman Szomolai‘s blog post; then shared it in one of my other blogs. Then I thought that this blog’s purpose is more suitable for this post. Therefore, I publish this post here with humble modifications and addons! If you want to read more of my posts about technical stuff related to games, don’t forget to check the Technical Spot of Games Category.
Measuring fun is generally an overlooked topic of game design. Do you know if your game is fun or not? In this topic, we are criticizing one of the most important topics in game design: measuring fun.
Have you ever thought of measuring your emotions?
Let us talk about some important as well as hard topic to discuss: what is fun? As game designers we forget one important point which is if we do not have fun while playing a game; this does not mean that game is not fun. Have you ever thought of measuring your feelings? I mean, how much are you happy while reading this, for example? Fun is also unmeasurable; but players’ behaviors are observable.
What is Proxy?
Mr. Szomolai points one important term which is called as proxy. In general usage, proxy simply means medium between you and server. It serves as a waiter that carries you a meal (website). In the game industry, this term has similar analogy. Everything which is measurable like player actions or expression are called as proxies. You do not need to know what fun really is, still, you can observe proxies which are reflections of fun. Let us discuss how to use these proxies.
Hypothesis & Theories
First, we need to understand what hypothesis is. We all create theories in our daily life; for example, we can think that if we lie to our friend and s/he understands it; then s/he will be mad at us. This is one example of hypothesis: if ‘a’ happens, then ‘b’ will always follow it.
Theory is group of hypotheses and these hypotheses are interconnected and consistent with each other. Interconnection between them means that if you know the relationship in one hypothesis, then you can make inferences about another hypothesis.
In practice, this relationship between ‘a’ and ‘b’ can not be fully understood. You try to get as closer as possible to reality. Then the ladder continues as:
- Better relationship leads you to better hypothesis.
- Better hypotheses lead you to better theory.
- At the end, better theories lead you to happier players.
Mr. Szomolai explains their point as if the hypothesis is not improving their theory, then it is not good enough. If it is not good enough, they start with a new one. If it is good enough, then they update their theory. Once they updated their theory, then they start with a new hypothesis based on their theory; and the loop continues.
Proxies: Valuable Indicators
When you have hypothesis, it is time to measure it with valuable indicators: proxies. If you talk about scale, then you deal with numbers. But do you really want to measure numbers? Mr. Szomolai explains measurement as an approximation of one phenomenon to another; till you get to an approximation that you really understand intuitively, or which is already part of your inner theory. Proxy is every single example of these phenomena that different game designers may understand differently. Mr. Szomolai indicates a very valuable point: Junior game designers argue about whose theory is true; however, seniors will experiment with theories and measure outcomes. I want to quote his words as exactly, here:
A great theory with bad measurements will lead to worse results than a weak theory with great measurements. Bad measurements will improve theory only a bit (if at all) while good measurement can move theory by miles (this is iteration once again, you probably heard of that before).Mr. Roman Szomolai
Accuracy, Precision, and Granularity
We talked about the importance of proxies and measurement; let us discuss parameters that we should watch.
Accuracy: Degree of truthfulness. Increases with repetition.
Precision: Difference in small parts. Requires focus on details.
The granularity of measurement: Do you need to measure it in kilograms or grams?
Here is a good example from Mr. Szomolai, to fully understand the difference between these terms:
Imprecise but accurate weight will show +/- 25% of the real value each time you step on it. Scale with rough granularity will show only kilograms and precise but inaccurate weight will show exact numbers each time but it will be exactly 8 kilograms off. In each case you will not know exactly what you wanted to know but you will be closer to the truth than before.– Mr. Roman Szomolai
Then why don’t we always have the best precision, accuracy, and granularity of measurement? It is about time, money, and know-how. We should make our decisions between them as accurately as possible according to the situation that we are in.
Designing is not simply “I know how to do it, and I want it in this way!”. To create a theory, we need to know how every part of our game works; and we need to have a theory about how they affect player experience. Good game designers should be doing progressive design: they do not say “we know everything”, but continuously update their theory.
We -as game designers- should always be checking for noise while considering data that we collected. Looking for counterexamples will make us aware of if the data is as accurate as we want or not.
Different analysis methods used while measuring proxies
We have different analysis methods while measuring proxies that are used for measuring fun. Let us discuss them:
This is based on your instincts as “player”. You are, probably, already playing games and observing what else do you want to happen, or what things did you dislike in the game. You can use this way with your friend: you make him/her play the game and ask his/her opinions.
Advanced way of improving your heuristic skills is continuously playing different games and creating patterns from them.
This is the cornerstone of game development. No game can be thought without playtesting. This is also can be done by yourself or your friends. You should do this as much as possible.
Mr. Szomolai points that there are several advanced ways of doing this like Kleenex playtesting, company-wide testing, focus group testing, etc. These methods should be analyzed in-depth to choose accurately.
This type of analysis is based on players’ feedback and game dynamics. You may know that dynamics are created from mechanics by players. Their verbal feedbacks are so important for us to criticize our decisions.
Advanced ways of doing this analysis are using machine learning to analyze as much feedback as possible, and directly communicating with players -by community manager for example- about their opinions and transferring them to developers.
Thought Experiment / Model
In this model, we do not experiment with all the components of the game, but some parts of it: for example, multiplayer matchmaking mechanics of the lowest rank. Then we can make deduction about the rest of it. You can create sub-parts of your whole game system then ask yourself what-if questions. The advanced version of this approach is making this process with machine learning prediction.
This analysis type is based on your competitors in your genre. What commonalities or differentiations do their games have? You can simply do this playing and noting these points manually. The more advanced way of doing this includes listing all games and relationships in your genre, watching interviews with creators, doing in-depth analysis with specific frames.
This is the area where you play with numbers: KPIs. You can simply do this by feedbacks from players in specific part of game. Advanced way of doing this is working with data companies.
Interview / Discussion
This analysis is done directly between players and developers. You ask players about their opinion, then just listen to them without defending your game; then observe their comments. Advanced version of this analysis is done with wisely chosen questions and qualified interviewers.