cool stuff.

Surveys

A time and place for surveys

Often misused for research, this tool can still play an instrumental role when it is deployed in the right context.

Updated May 25, 2019

Loved by your marketing team and reviled by virtually everyone else, the survey has a bad reputation as a research tool.

It is relatively easy to write a lousy survey, but the real problem is that we rely on them far too often as the ONLY source of data. This misuse says more about designers as the collective involved than saying much at all about the tool itself. A survey can still be a valuable tool if constructed and deployed effectively.

Constructing better surveys

When I was transitioning from a print designer to UX design, I was a big fan of surveys. This was particularly true when working with small startups operating on tiny budgets. Surveys were easy to create and deploy, but the real key was that they generated the beloved data that we all desire so much.

In a world where you are pursuing answers, it’s easy to turn a blind eye to the quality of the data that you’re collecting.

What I would come to realize after working in the field for a while is that the data doesn’t matter if the survey is terrible. Worse yet, the data a bad survey produces doesn’t seem bad.

Here are a few steps you can take to create useful surveys.

Ask if you need a survey

A survey that is explicitly deployed in the service of gathering data is inviting disaster. This process generally begins with someone in your office, suggesting that your organization “put a survey on the site to see what people like.”

First, this is why we need you in those meetings. This is a design decision, and you can’t help steer the conversation from your desk with your headphones on.

Second, there are better ways to tell what people like. For instance, if you’re in the meeting, you could suggest pulling the analytics for your site to get a sense of what people “like.” This goes right back to what was discussed in the chapter on observational research, “actions speak louder than words.”

Finally, if the pressure is still on from your team to create a survey, use that opportunity to make the survey itself serve as an entryway to another form of research, the user interview.

A very, very brief introduction

Don’t start by saying too much upfront. It’s easy to taint a participant by leading the survey with a poorly constructed introduction. Start by mentioning what the survey is about ” seeking your insight regarding industry/product X” and get out of the way. There is no need to mention your product or desired outcome. The survey will do the rest of the work for you.

If there is an incentive, mention it

Typically when we discuss incentives in the research phase of a project, it is tied explicitly to spending time talking with a real person. If you are offering an incentive, mentioning it in the introduction of your survey can be helpful to spur people along. _ “Receive a $10 Amazon gift card for completing a 10-minute call with our team, “_ informs everyone that there is a possible payoff for the action but only after completing the call.

Shorter surveys are completed surveys

We’ve all been on the receiving end of long surveys seeking our feedback. Regardless of how we feel about the entity asking for our feedback, studies show that completion rates fall sharply for any survey that takes longer than five minutes to complete.

According to SurveyMonkey, the magic number is 10 questions. Anything longer than that and you’re going to see people bail in more significant amounts.

Draft your survey

Regardless of the platform, you opt to create your survey in, it’s pretty easy to lose your focus when you can’t see the length of the survey. Drafting out the questions you want to ask on a piece of paper or in Google Docs will give you a chance to consider all possible options before you begin the editing phase.

Order matters

An important consideration when constructing a survey is the order of the questions that you’re going to be asking.

There are two primary groups, questions relating to the topic, and questions relating to the person. If your survey is intended to find you quality candidates for a specific topic, you should lead with those topic-related questions first.

Demographics vs. psychographics

Thinking ahead to the next stage of this process, you’ll eventually need to sort out your possible participants. Do you want a specific group of people, or are you seeking people who behave a particular way?

Ultimately, these considerations will come into play as you write your survey questions.

Demographics will focus your attention on who the user is, where psychographics focuses on what the user does.

A demographic-focused question could be age or gender-related. A psychographic-focused question would be more behavior-related. For instance, ” how many cups of coffee have you consumed in the past five days “ is a question that illuminates a particular behavior that’s important if your survey is related to coffee consumption.

Some demographic questions are obvious to ask, but determining which behavioral questions matter could play a significant role when you later try to sort your field of interview candidates.

Avoid unanswerable questions

Nothing will stop a survey in its tracks like a question that the participant can not answer.

The most obvious form is the follow-up to a question that the user has already told you that they could not answer. If a user ever responds “No” to a question like “Do you use product X,” you can’t then ask them about their experience with the product. That’s literally the reason every survey platform can use conditional logic to reroute the participant around questions that do not apply to them.

Author and noted design researcher Erika Hall wrote a great breakdown on surveys and the questions you shouldn’t ask. In On Surveys, Hall clearly outlines the types of questions that you should NOT ask:

  • Never ask what people like.
  • Never ask for someone to predict the future.
  • Never ask someone to rate anything on a 10-point scale.

Recruiting survey participants

Ultimately, the path forward for you will be determined by two factors. The first is focused on the maturity of your product and whether it has a user base. The second centers around whether you have a budget or not.

If you are working on an existing product or service, one of the easiest ways to recruit participants is from an existing user base. Using a service like Ethnio or Positly allows you to invite current users to join a research panel, providing a chance to use your survey to build up a database of participants.

Another way to build from your existing user base if to recruit from users who have contacted your customer support team. These are users generally have opinions about your product that are valuable to hear when considering adjustments to make.

You won’t always have the luxury of an existing user base to recruit from, especially when working on a new company or service. In these instances, it comes down to a discussion of cost vs. time.

When you create a survey, you’ll generate a link that can be passed around. Ask friends and family to help get the word out that will typically take the form of a quick blast via social media. Within 24 hours, this approach will have generated as many responses as you’ll receive.

You can also recruit in public forums like Reddit, Facebook, and Craigslist but your response rate is likely to be very low unless you considering increasing your incentive. Proceed with caution here as the incentive could lead to the recruitment of less qualified participants.

End by requesting permission to follow-up

Nobody likes ending up on a mailing list for a service, which is why most participants will likely pause for a moment before handing over this crucial bit of information. However, if you explicitly ask for their permission to follow-up with them regarding this survey, they will be far more likely to approve your request and thus, finish the survey.

Exercise

As illustrated in this chapter, surveys are useful in the right situation. With that in mind, we’ve limited the role of the survey to that of a screening tool — specifically in the service of gathering potential interview subjects.

This exercise will build upon your previous competitive analysis work while building a small database for your upcoming testing, and user interviews exercises focused on meal delivery services.

Using the information you have gathered, construct a survey that:

  • Has a clear introduction stating the purpose
  • Is shorter than ten questions in length
  • Presents no questions to the participant that cannot be answered
  • Asks permission for follow-up contact

Once complete, update your Program Journal with links to any assets produced in this exercise. Post your Journal in the #Feedback-Loop channel for review.

Once your work has been reviewed, you’ll release your survey to the public via friends, family, and social media. Continue pushing your survey until you have accumulated between 20-to-30 responses. You’ll use this base of potential participants in the next chapter.

Up next Competitor Testing

New Pragmatic’s Intro to Product Design and UX Design courses are now open for enrollment — but don't wait! Space is limited.

Sign Up Now