Alexandria Grimaldi and I both work in an innovation lab—so we often find ourselves in the company of amazing people from very diverse backgrounds. But this also means that many of these wonderful people have never run a survey. While guiding our lab members to put together the survey they needed, we realized that (1) it can be very stressful and frustrating for them, (2) it’s really hard to find in-depth yet accessible help online and (3) our tips could be very helpful for a larger community. So in this series, we wanted to talk about how to design your very first survey from scratch.
The curse of knowledge is that you underestimate how powerful “obvious” ideas can be to anyone for whom it is new.
Surveys are an important research tool to leverage. They can help to quickly derive quantitative insights to inform major decisions. Some successful examples are:
- How Superhuman leverage surveys to find product-market fit
- How Buffer used Sean Ellis’ framework and tools
- How Quartz surveyed 1,500 executives to improve their product for the changing habits, behaviors and preferences, of business leaders
But, if you don’t have an experienced researcher or the budget to outsource, you end up having to run surveys all by your lonesome. Uh oh.
😣 How do I know what type of survey to run?
😫 What questions should I be asking?
😣 Where do I even start?
😫 What choices do I have?
There are survey platforms we love, like Typeform, Survey Monkey, and Qualtrics, that are awesome at answering these questions. However, there is also an underlying assumption that if you are using their services, you likely have fundamental knowledge of survey design. This is great for people like Alexandria Grimaldi and I, who have years of research experience and may need quick refreshers on learned methodologies. However, if you’re completely new to this field, getting started probably feels like a shot in the dark. 😶 🔫
🎬 So, let’s talk about designing a survey!
🤓 First, Table Stakes.
- There is such a thing as bad questions.
- Put demographic questions last—unless they are being used to screen participants (More about that here).
- Make sure the entire survey (e.g. question order) flows naturally for the participant.
- Grammar check, spell check, etc. after you’re finished writing it.
- Always QA your surveys once they’re program —especially if programming is outsourced.
We’ve left the table stakes deliberately brief to concentrate more on the stickier points, and also to talk about points that readers’ have brought up around why no one trusts surveys. If you are looking to dig into some of the above points, here is a great article.
Next, Writing Your Survey…
1. Design your survey around well-defined goals
The very first step when writing a survey is to ask yourself the following questions:
- What exactly are we trying to achieve?
- What are the most important things we need to learn right now?
- How am I going to use the information once I have it?
To keep things simple, the answers to these questions will inform your goals. (These “goals” are also sometimes known as assumptions, hypotheses, objectives, etc.) Throughout the survey creation process, your goals are your lighthouse; they will keep you focused and guide you back on course when you go astray. 🐈
The easiest way to write questions is to work backwards from your set of goals. It’s important to remember that you have to make a very conscious effort not to bias the survey while doing this. It’s a tough balancing act, which is why writing surveys is much more of an art than a science.
For example, if you want to find out how many people love the color red, the appropriate question to ask should be “What is your favorite color?” Then, list a few colors for the participant to select from. It becomes a much more compelling story when red is selected against all other options. Directly asking people whether or not they like the color red creates a false dilemma. These are the sorts of logical fallacies that hurt the credibility of survey results.
Your survey will undergo a lot of revisions; either from yourself, other team members, or stakeholders. Sometimes in the torrent of edits, you’ll lose sight of the original goals. Loose questions will stick out like sore thumbs. Be firm and don’t be afraid to remove or re-write the problem questions.
This is especially true if you are working with external stakeholders or teams that have other priorities besides your own. If there isn’t a clear alignment, you’ll have multiple stakeholders using the survey to achieve different outcomes, which makes things extremely messy and counter-productive. The revision process becomes exponentially easier when everyone is aligned from the start on the goals of the survey.
How to deal with conflicting and multiple goals
If your survey has more than one goal, these goals need to work together in a way that makes sense. If they don’t, you either need to re-define your goals or run separate surveys instead. Cramming everything into one survey may seem frugal and efficient, but it will inevitably create a lot of unnecessary noise. You’ll find yourself endlessly splicing and digging into data that may not be relevant. Because you have all of this data, all of it might (momentarily) seem important. But here’s a protip: just because you have 95% stat on something, doesn’t mean it’s necessarily useful or meaningful.
2. Keep the survey as short as possible
The reality is: no one wants to take a survey.
Do you remember the last time you took a survey? I’m sure you were extremely engaged and considerate of your responses in the beginning, but the more time you spent on it, the less you read the questions or thought through your answer. The longer it is, the higher the drop-out rate will be, and the lower quality of responses will be. (More about survey bloat here.)
To keep your survey short but comprehensive: only include responses that help with your goal (remember Step 1). You can (and should) include questions with data exploration in mind. For example, when validating a persona, do include demographic and behavioral questions. And, if you are managing disparate stakeholder goals, remember that running a separate survey is always an option.
If the question doesn’t make or break things right now: take them out. Less is more when it comes to survey size.
“To ensure high response rates and avoid misleading survey results, keep your surveys short and ensure that your questions are well written and easy to answer.” —Jakob Nielsen on NNgroup
3. Turn open-ended questions into close-ended questions
This is one of the trickier bits in research: balancing of qualitative vs quantitative. To have a rich understanding of our users, you often need to collect varying proportions of both.
Most of the time when we use open-ended responses it’s because we want to hear the answer described in the participants’ own words and / or don’t want to eliminate any possible options for them. We can get a lot of inspiration from people that think outside the box!
But, let’s not make the participants hate you.
Open-ended questions are very taxing on the participants. The more someone has to think and explain themselves, the longer the survey will feel. You’ll begin to see a lot of nonsensical responses like “n/a,” “ok,” “idk,” “asdfg” and “I'm just going to copy the question straight into the response section.”
Prior proper research is very important, and it shows.
You’ve probably already done a ton of research around your survey topic, so you should have a good idea of what the options should be. Use them! If you have no idea what some of the potential responses could be, think about doing more research or delay asking that question.
Save yourself a lot of time and effort
When you turn an open-ended response into a close-ended one, you’ll also save yourself time when you’re analyzing the results. Including qualitative responses means that you have to read through and quantify qualitative responses, which can be a time-consuming endeavor. Why not make it much easier for yourself to splice and analyze the results?
We want to continue discussing how to design a survey, but Tzeying is going to kill Alexandria Grimaldi for how long this is already. The next article will focus on the topics of:
- How to avoid writing biased questions
- Deciding which questions should be primed vs when to get a gut reaction
- Targeting the right audience
- Contextualising the survey for the correct audience
If there’s anything else you’d like us to explore or elaborate more on, let me or Alexandria know!