A closer look at the City of Edmonton’s Downtown Arena Questionnaire

The City’s online questionnaire for the proposed downtown arena has received quite a lot of criticism, especially online. Twitter users have not been shy about calling it “leading” and questioning its intent. And that criticism has come up in face-to-face conversations I’ve had with people at arena consultation sessions and other events as well.

At the consultation sessions, officials have made it very clear that the questionnaire is meant simply to gather feedback, and is not intended to be a statistically valid survey. It feels like a survey though, so it’s no surprise that people treat it that way.

The introduction of the questionnaire reads:

This questionnaire is one of several methods being used to gather thoughts and concerns about a proposed downtown sports and entertainment facility. The feedback received will be provided to City Council to assist in their decision-making.

And it explicitly asks that you only fill it out once, though technically there’s nothing stopping you from filling it out dozens of times. There’s also no way to ensure that only people who live in Edmonton fill out the questionnaire.

Proposed Downtown Arena Consultation

I emailed the City with some questions on the survey, and they wrote back with detailed responses.

Was the questionnaire written by City staff? Calder Bateman staff?

A team worked on the online questionnaire. It was one of the early elements developed to support public input and was meant to be a quick and convenient way people could share their thoughts. It was never intended and not designed as a statistically valid, formal survey – but a technology-based tool for use by those who might not make the open houses in person. It means we will have a richer range of input to report on.

While other elements of the open house and discussion forums have evolved based on the input and suggestions we’ve received, we didn’t feel it appropriate to adjust the online questionnaire – so it has remained consistent with what was originally posted.

How many entries have you received to-date?

As of 4:00 p.m. Nov 9 we had 17,030 directly through the online questionnaire plus another 60 from those who called 311. We’re pleased with these results. It’s likely that people are seeing the online questionnaire as an option to attendance at the open house.

Any indication of how many received entries are duplicates?

No. At this point, we haven’t done an analysis on the questionnaire submissions. We really need to wait until the entire process concludes to get the full picture. There are a number of ways people are providing input including email, sticky notes, completion of the discussion feedback forms and the questionnaires. The inputs from all areas will be consolidated into the final report.

Are there any plans for a follow-up questionnaire later in the process?

There will be other opportunities for public involvement and input throughout the process, and another questionnaire might well be an option.

Are there any plans for a statistically valid survey?

We realize the online questionnaire only reflects the opinion of the people who complete it – it is not a representative analysis of what the population of Edmonton might think. The City will be very clear when presenting the results to ensure they are not considered a statistically valid sample of opinion.

A statistically valid survey continues to be a possibility. Our focus right now is on the current suite of public involvement opportunities.

I also asked my colleague Greg Pope, Analytics & Psychometrics Manager for Questionmark, for his thoughts and feedback on the questionnaire. He suggested this best practice guide (PDF) for good survey design as a starting point. Here are some of the comments & suggestions he had:

  • The current 1-5 “definitely” likert scale is unclear (no label for 2,3,4). Greg suggested an even number of choices (1-4) so that there is no “maybe” or “unsure” category.
  • “A rule of thumb is to try and keep all the questions on the same scale with the same values, so converting all questions to statements with a 4 point agreement scale would work well.”
  • “Another rule of thumb is to phrase the questions all the same way. It is not leading or misleading to ask “I want a new downtown arena built in Edmonton.” Because you are asking whether they agree or not with the statement. “
  • If the responses were going to be analyzed in any way, Greg said he would have asked some demographic questions. As a simple questionnaire, that’s probably not as important.
  • Another suggestion Greg had was to combine the open-ended fields into one at the end, to make it quicker to complete the questionnaire. He said there’s often sorting that has to happen anyway.

Good feedback to keep in mind if a survey is created in the future. Thanks to both Greg and the City of Edmonton for helping me out with this.

I think there are two key takeaways here. First, the online questionnaire is not a survey. Second, the City of Edmonton is consistent in treating the questionnaire as just another way for people to provide input. If you think the questions are leading, don’t fill it out. There’s nothing stopping you from emailing the City with your thoughts or calling 311. And if you can, attend one of the public consultation sessions.

3 thoughts on “A closer look at the City of Edmonton’s Downtown Arena Questionnaire

  1. Your first hint that the questionnaire isn’t a survey should be the fact that the questionnaire is not random response. Unless you’re getting responses from the universe of people affected (in this case, each and every Edmontonian), nothing that asks you to participate can be seen as repesentative of the population as a whole. Only random sampling can do that.

  2. Well in the end they used it to support their position and did calculate the number of people who agreed verus disagree. After the Patrick LaForge’s phone call to 350,000 people they did not choose to remove the calls made on that date. So I am sorry but they say one thing and use it to achieve their own means – positive support for their pet project.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s