Ultimate Guide to Effective Citizen Surveys

As a city leader you’re facing complex problems. There are a lot of voices and many competing demands, and it’s critical that you prioritize wisely to have the biggest impact and make the best decisions for your community.

Citizen feedback is a crucial piece of the puzzle. It ensures citizens have a voice, but when it’s done poorly the process can require an inordinate amount of time and resources for results that never translate into actionable understandings. But it doesn’t have to be this way! Surveys can effectively give you insight into what citizens think and provide the direction you need to confidently make better decisions. It all starts with a well-designed survey.

At OnPointe Insights, we help many city leaders with effective citizen surveys, and we’ve had years of experience helping leaders at major global enterprises. We’ve boiled it all down to 5 guiding principles you should rely on to make sure your next citizen survey delivers.

The first secret of citizen surveys is that short (but effective) is the way to go. We aim to keep our surveys to about 9 minutes max. There are definite benefits to constraining yourself to a short survey.

Keeping your citizen survey short…

1. Forces you to focus on the important issues
2. Minimizes curiosity questions that usually won’t matter
3. Helps you create better and more meaningful questions
4. Creates goodwill with those who invest time to respond and help
5. Builds trust with citizens who appreciate your focus

We know it’s tempting to put everything requested into a survey. Don’t let internal politics and your desire to please get in the way of doing what is right for effective survey feedback.

Today’s reality is that people want things to be intuitive, simple, and short—and they largely scan rather than read content. A long survey causes people to quit early or to zone out and give halfhearted answers. So, keep your survey short. Ask each question once in the best way you can.

We encourage you to welcome a short survey as a friend that helps you be thoughtful to your citizens and focused on what really matters.

Remember: You can always do another survey.

It goes without saying that most people are never without their phones. What you may not realize is just how many people will use their phones to take your survey. Consider this: In a recent Florida study (think older, retired people), 50% completed the survey on their phones. In a recent Texas study (where we can assume a more mixed-age demographic), 62% used a mobile device. Just imagine the results if you have a younger population in your community! Clearly, it is vital to design for mobile. But, just because the survey platform you use can work on mobile phones does not mean that your survey is automatically optimized for the best experience. The type of questions you ask and the way you structure them are key for a good mobile experience. Here are a few key Dos and Don’ts as you’re optimizing your survey for mobile. (And, of course, there are more. But if you do these 4, your surveys will be much improved.)

• DO make the questions you ask short. This goes back to our previous guideline, but it’s even more important on mobile. Make sure you’re not taking paragraphs to explain or set up your question. Crystal clear and long is worse than understandable and brief. If you can’t ask the question quickly, re-think, re-write, and tighten up your questions. The same thing goes for question choices. Always be as short as possible.
• DON’T use long batteries of questions asking if they “Strongly Agree, Somewhat Agree, Neither Agree nor Disagree, Somewhat Disagree or Strongly Disagree.” Such questions were common in the past, and even though they work on phones, they quickly become long and burdensome. You can do some battery-type questions, but minimize them.
• DON’T use 10-point scales (except the Net Promoter Score). There are 10 point scales, 7-point scales, 5-point scales, and on and on—so you have lots of options. Longer scales only give you slightly more value, and they do more harm than good on mobile phones as they simply take up space.
• DO use the microphone function. We always encourage using the microphone for open-ended questions on the phone. Allowing participants to provide open-ended answers this way makes for comments that are richer and far more complete as well as being MUCH easier for your participants. Speaking is much faster than typing—so this one is a win-win for everyone.

Your citizen surveys should be about your city. Ensure you’re focusing on things you could change (or even would change). Here are a few examples of areas where surveys can be incredibly powerful:

• Do you need to prioritize constrained resources against the many demands you face? Design to get the focus you need and prioritize the city budget. Even if you need to prioritize 50 or more areas, the right survey design can address these issues clearly and give you unifying feedback.
• Do you need to determine which events to support more and which ones can go? Design to provide that focus. The right feedback can help you spend time on events citizens care about without wasting valuable time and resources on ones that don’t matter much.
• Do city leaders have differing visions about where to invest the budget on city improvements? Your design can intelligently and briefly cover various options for feedback.

The point is, if your questions are designed wisely, you can address many city issues and still keep the survey under 9 minutes long. Someone who understands survey design can effectively get you there.

We recommend no more than 2 open-ended questions that ask for free-form feedback or comments. When asked, open-ended questions should typically be broad to seek thoughts overall.

We’ve found that the heavy lifting is done with well-designed questions that are not free-form. You should use open-ended questions as the icing on the cake. They add flavor and context.

Why should you limit open-ended questions?

• They are the enemy of a short and sweet survey. By adding an open-ended question you’re tacking on at least a minute to your survey completion time.
• They are difficult to answer. With open-ended questions, people have to think rather than respond. They have to put thoughts into words rather than evaluate options. Good
  survey design makes it as simple as possible for the participant.
• They are difficult to analyze. We have tools that make it easier, but it takes a skilled analyst to create meaning from open-ended questions beyond a few interesting quotes. This is because free-form responses can be about anything, from park improvements, streets, taxes, barking dogs, sidewalks, how things used to be, and so forth.
• Little is learned after the second open-ended question. Having analyzed survey responses for years, we find people will share a great deal in the first two open-ended questions. Thereafter, they usually repeat themselves.
• More open-ended questions add work for both the participant and those doing the analysis. Remember, <=>

So what should you do when including these questions in your survey? Ask for open-ended questions to be coded and filtered.

With today’s technology, it’s possible to group or code comments into meaningful categories. Then, it’s possible to create easy filters that allow your team to review the comments on specific topics.

For example, if many people comment on park improvements, the right coding and filtering will let you apply the filter and read them together. Even with hundreds (or thousands!) of comments, coding and filtering can make it possible to see themes and inspire improved responses and plans from citizen comments.

Be Careful: Not all research providers make it easy to handle open-ended comments. Be sure to ask about their plans to code and filter responses for your benefit.

Finally, spend time thinking about the quality of responses. The old saying is very true in survey research: “Garbage in, garbage out.”

Here are 3 ways to ensure quality in your survey:

1.  Ensuring quality if distributing via your site/QR code anyone can access.
   It’s good to distribute surveys on the city website, but you must be sure to clean the data to assure high quality responses because distributing the survey this way also allows people from outside of your city, state, and country to take your survey. And they can take it dozens of times, effectively stuffing the ballot box. How do you address the issue?
   • Remove duplicate IP addresses.
     Check IP addresses and filter out those who complete it multiple times from the same one. We like to allow two from the same household, but we remove 3 and above.
   • Remove those outside of your city.
     We add a zip code field to our surveys as well as ask them to choose their city from a list. We then remove those that are invalid.
2. Ensuring quality with a large and representative sample.
   It’s possible to get 10% or more of citizens to respond. To do so, we like to use several ways of distributing. We love emails and SMS invitations (usually from utility billings), and we also use the city website and other things the city often does. By using various ways of reaching people, we get a strong response. With lots of replies, there are a few key ways to ensure quality:
   • Track the source.
     Your survey can include special codes to tell you which channel a response came from. This allows you to see which distribution channels worked best for the next time, and you can double-check the ones that allow several people to take them.
   • Show a map.
     We like to have citizens indicate where they live on a city map, broken down into 4 or 5 regions. This makes it easier to check to see if the sample is representative.
3. Ensure quality by removing speeders. Another important measure of quality control is checking how long it took a person to complete the survey. It’s a good idea to make sure you can see how long each respondent spent on the survey—and then remove the speeders. Is this a 9-minute survey? Remove those finishing in under 4 minutes. Their responses tend to be superficial, and they are typically disengaged and/or cheating. Removing them will increase quality.

There is more we can help you do to ensure quality, but if you do these 3 things, your data will be reliable and consistent year-after-year. Give us a call if you need some help designing a survey or would rather have someone do it for you. We’ll gladly do what we can.