The City’s online questionnaire for the proposed downtown arena has received quite a lot of criticism, especially online. Twitter users have not been shy about calling it “leading” and questioning its intent. And that criticism has come up in face-to-face conversations I’ve had with people at arena consultation sessions and other events as well.
At the consultation sessions, officials have made it very clear that the questionnaire is meant simply to gather feedback, and is not intended to be a statistically valid survey. It feels like a survey though, so it’s no surprise that people treat it that way.
The introduction of the questionnaire reads:
This questionnaire is one of several methods being used to gather thoughts and concerns about a proposed downtown sports and entertainment facility. The feedback received will be provided to City Council to assist in their decision-making.
And it explicitly asks that you only fill it out once, though technically there’s nothing stopping you from filling it out dozens of times. There’s also no way to ensure that only people who live in Edmonton fill out the questionnaire.
I emailed the City with some questions on the survey, and they wrote back with detailed responses.
Was the questionnaire written by City staff? Calder Bateman staff?
A team worked on the online questionnaire. It was one of the early elements developed to support public input and was meant to be a quick and convenient way people could share their thoughts. It was never intended and not designed as a statistically valid, formal survey – but a technology-based tool for use by those who might not make the open houses in person. It means we will have a richer range of input to report on.
While other elements of the open house and discussion forums have evolved based on the input and suggestions we’ve received, we didn’t feel it appropriate to adjust the online questionnaire – so it has remained consistent with what was originally posted.
How many entries have you received to-date?
As of 4:00 p.m. Nov 9 we had 17,030 directly through the online questionnaire plus another 60 from those who called 311. We’re pleased with these results. It’s likely that people are seeing the online questionnaire as an option to attendance at the open house.
Any indication of how many received entries are duplicates?
No. At this point, we haven’t done an analysis on the questionnaire submissions. We really need to wait until the entire process concludes to get the full picture. There are a number of ways people are providing input including email, sticky notes, completion of the discussion feedback forms and the questionnaires. The inputs from all areas will be consolidated into the final report.
Are there any plans for a follow-up questionnaire later in the process?
There will be other opportunities for public involvement and input throughout the process, and another questionnaire might well be an option.
Are there any plans for a statistically valid survey?
We realize the online questionnaire only reflects the opinion of the people who complete it – it is not a representative analysis of what the population of Edmonton might think. The City will be very clear when presenting the results to ensure they are not considered a statistically valid sample of opinion.
A statistically valid survey continues to be a possibility. Our focus right now is on the current suite of public involvement opportunities.
I also asked my colleague Greg Pope, Analytics & Psychometrics Manager for Questionmark, for his thoughts and feedback on the questionnaire. He suggested this best practice guide (PDF) for good survey design as a starting point. Here are some of the comments & suggestions he had:
- The current 1-5 “definitely” likert scale is unclear (no label for 2,3,4). Greg suggested an even number of choices (1-4) so that there is no “maybe” or “unsure” category.
- “A rule of thumb is to try and keep all the questions on the same scale with the same values, so converting all questions to statements with a 4 point agreement scale would work well.”
- “Another rule of thumb is to phrase the questions all the same way. It is not leading or misleading to ask “I want a new downtown arena built in Edmonton.” Because you are asking whether they agree or not with the statement. “
- If the responses were going to be analyzed in any way, Greg said he would have asked some demographic questions. As a simple questionnaire, that’s probably not as important.
- Another suggestion Greg had was to combine the open-ended fields into one at the end, to make it quicker to complete the questionnaire. He said there’s often sorting that has to happen anyway.
Good feedback to keep in mind if a survey is created in the future. Thanks to both Greg and the City of Edmonton for helping me out with this.
I think there are two key takeaways here. First, the online questionnaire is not a survey. Second, the City of Edmonton is consistent in treating the questionnaire as just another way for people to provide input. If you think the questions are leading, don’t fill it out. There’s nothing stopping you from emailing the City with your thoughts or calling 311. And if you can, attend one of the public consultation sessions.