The Missing Step in Your Fancy Survey Design

Whether it’s for marketing, community engagement or an academic project, many of us will have written a survey of some kind. Some folks keep it old-school with a list of scrappy questions emailed to people they like, and some go all 2020 using awesome online tools to design global, digital-marketing extravaganzas. 

I’ve been writing, using, commissioning and formally reviewing surveys for over 20 years, and have seen exactly what the skills-gaps are among technical and expert teams in terms of survey design, and how much demand there is for external support. 

There’s now a growing supply of survey design services and software which, to a degree, satisfies that demand. 

But there’s one step in the survey design process that too many people still miss out entirely or execute poorly, which costs them time, money and sometimes their hard-won reputation.

How familiar is the following scenario? 

You’ve put together a beautiful survey and sent it around to your intended audience of respondents. It looks amazing and it took you hours to prepare. You even paid a social-media wunderkind to do the fancy, online, digital bits for you. Responses are finally coming in and you’re excited to see the numbers. But, you soon find that there are problems with the data:

• There’s not much of it;

• It’s incomplete;

• It doesn’t address the points you were getting at;

• It makes no sense;

• You can’t use it as you intended to;

• Respondents are giving negative feedback;

• You’ve blown your budget;

• You’re frustrated;

• The clock is ticking…

At best this has earned you a stern word and a hard stare from a director, board member or co-worker. At worst you’ve dropped the ball on an unmissable opportunity or damaged a relationship with a stakeholder group. 

Poor survey design really has cost people their jobs and closed organisations. 

Whether you’re new at designing surveys or you’ve been writing them for years, you need to understand that all good surveys are piloted (or tested) by someone other than their creator and revised based on feedback before they begin.

Why? There are two broad reasons and a hundred smaller ones. The broad ones are:

1. People generally don’t like completing surveys and don’t need much of an excuse to dismiss or abandon yours. 

2. Questions must generate a meaningful amount of useful, relevant and new data, that you know how to work with.

Piloting involves asking others to review and give detailed feedback on your proposed survey tool and survey method, to check that questions make sense, that the gathered data actually helps you achieve your objectives, that the approach is equitable, and will attract relevant respondents who then participate.

You could pilot the survey by distributing to a select few of your intended audience in a kind of pretend version of the survey. 

However, if the survey is revised after testing, you may need those individuals to engage again as their initial responses won’t necessarily align with the new format. But, pilot respondents are notorious for not engaging with revised versions. If you have a very specific or hard-to-reach audience, this can really affect your overall numbers.

The other piloting option (which doesn’t eat into your response rate) is to get an experienced, independent person to review your draft survey and your proposed approach. 

Sure, an un-piloted survey might work just fine, and you might get heaps of really valuable insight into whatever it was about. But you know that’s just a lucky break, right? 

And if your survey relates to an auditable or regulated area of work, you’ll generally have some kind of good-governance or fiduciary obligation to do the work as efficiently and diligently as possible; luck is not a strategy.

Getting solid advice on design is a great idea, however, I’d strongly encourage everyone to also pilot their surveys. 

Academic researchers, students, non-profits and charities, public service providers, startups, small businesses and big-hitters can all benefit from the exercise. 

My advice as always is to make sure your precious resources deliver both impact and value, and complement rather than complicate your activities.

***

This is something I’ve offered advice on in a consulting capacity on many occasions. If you’d like to discuss further or if you need some specific help, hit the Contact button (at the top of this page) and drop me a line.