Guest blog: Bill Dashfield looks at the 2018 New Zealand Census, teases out some reasons for the low response rate and suggests fixes. Ironically, the 2018 Census process prevented him from filling in his own census form online… This article expresses his personal views.
Summary
The 2018 ‘Digital First’ census has seen
- 82% of respondents (74% of New Zealanders) taking part online
- “quality of data” very high
BUT
- an 82% increase in non-respondents (from 5.5% to around 10%)
- more use of ‘imputation’ to increase response rate, resulting in a 5 month delay in the first results
- an apology from the government statistician
- anxiety for people wanting to take part but unfamiliar with the internet
- accessibility issues, especially for the elderly and disabled
- anger about questions ‘forcing people to lie’
- widespread concern about some groups being greatly under-counted
- widespread concern about the incompleteness of census information, used to prioritise government services
Some real advantages for online
There are advantages to an online form over and above the lower cost; a well designed form that works with reader software can be easier for people with visual or some physical disabilities; it can take people through a process step by step, warning them of missed questions for example; it allows more predefined answer options than can be can accommodated on a paper form; it can be faster and more convenient.
But… 82% increase in non-respondents
Despite a long gestation period, considerable testing and considerable effort and investment in outreach, the digital 2018 Census did not run smoothly. Leaving aside organisation problems like letters not being sent early enough, the switch from census takers calling at every dwelling to being ‘digital first’ caused multiple issues. The most important result is contributing to the response rate dropping from 94.5% in 2013 to ‘at least’ 90% in 20181. Yes, response rates have been dropping for twenty years, by about 0.9% per census; the ‘online by default census’ did not reverse the trend but increased it five-fold.
| We’re working alongside our community outreach centres in Rotorua to target our whānau who are not digitally connected. The reality for us is that only 24% of households competed Census in 2013 which equals how we’ve been funded by Govt since then. And it’s assumed that more households will complete it online – we still have a district where we believe at least 40% are still not connected. |
This is drop is serious and important2: government funding for social and other services is being increasing targeted to the groups in most need, and planners need good data on them, despite many of these groups are the hardest to reach by online methods3. Stats will use statistically valid imputation methods to improve the coverage / response rate but this is not ideal.
There are multiple reasons for the drop in response, and Department of Statistics will be doing a full independent review to identify them; this will doubtless greatly improve the next census as the Canadians did to go from a disastrous 68% in 2011 to a world-leading 98.5% in 20164.
However as someone experienced in testing, online surveys and accessibility, I found some of the problems obvious and foreseeable5. Here is my take, as an outsider, on some things I think went wrong, and how to avoid them in future:
- The Census was not properly accessible (paper and online).
- The online process were not flexible enough, and simple improvements were missed.
- The helpdesk staff were inexperienced and not properly briefed. The FAQ was fairly superficial.
- The push for ‘Digital First’ was too strong; it was not easy enough to get paper forms.
- The different user experience of paper versus online was not fully appreciated.
- Some groups were alienated by the pre-set answers not including them.
- There were justified concerns about privacy.
- The testing was fragmented, not ‘whole process’ and didn’t include all the right groups.
The Census was not properly accessible (paper and online).
The first accessibility test I do is for adequate contrast between text and background. This is vital for people with poor eyesight; the international standard is for a contrast ratio of 4.5:1.
The Dwelling Forms had small text in orange) which failed this: orange/white was 3.2:1. The purple/white of the Individual Form passed online, but the downloadable sample print form failed. As the Census website is no longer available, I can’t do a full accessibility check.
Stats tests of the forms6 seems to have focussed on usability rather than accessibility for the disabled, and on the questions rather than the process.
Jonathan Mosen, a blind man with a Master’s degree who runs an international consultancy on technology for the blind and an internet radio station, hit another problem He was unable to access the online survey – despite all his knowledge, skills and equipment. The access code letter could not be read by his scanner7.
Accessibility testing of the whole process should have picked up these and any other issues.
The online process were not flexible enough, and simple improvements were missed.
In my case, I filled in the Dwelling Form in advance online, as I was going to be away on the Whanganui River on a 3-day camping trip, followed by 2 weeks touring. So I listed myself as ‘away’ on Census Day. The online Census application then refused to let me fill in my Individual Form in advance – it was expecting me to do the form in another ‘Dwelling’. But my tent had not been sent a code….
And although I could request paper forms via the internet, there was no option to download them to print them off. Instead, you have to wait for them to arrive in the post, which is slow, not delivered every day and would arrive after Census Day.
And although the Individual Form allowed you to say where you usually lived, the Dwelling Form did not. Thus a group of families on holiday in their internet-connected motorhomes will each appear to be living in a single room mobile dwelling at a road end rather than their three- or four-bedroom houses.
The helpdesk staff were inexperienced and not properly briefed. The FAQ was fairly superficial.
As the FAQ did not cover this (though more FAQs were added after Census Day), I rang up the Census Helpline, who offered to send someone round to the Whanganui River campsite with a form, as there was no internet access there. When I explained this involved 3 days in a kayak or one day in a jetboat, she agreed this was not a sensible option. She then suggested that I (and my wife and overseas guests) could just not take part. I objected to not being counted. Eventually, after another call, the obvious suggestion of posting the forms to us to fill and return after our trip was made, which we did.
Apparently the 5 motorhome families had contacted the helpdesk too, and got 5 different answers.
All this could have been avoided by 1) thinking through and role-playing the possibilities 2) covering them in the FAQ 3) better helpdesk staff with better training and 4) having downloadable forms or emailable forms and/or access codes instead of a fallback dependent solely on snail-mail and field officers.
| I am doing census help sessions in Dunedin/Mosgiel. We have been inundated with not only young but older people with no internet access or confidence. |
The push for ‘Digital First’ was too strong; it was not easy enough to get paper forms or assistance.
The access code letter sent to households made it very clear that you were expected to fill it in online. To get paper forms you had to find the small, un-obvious number on the letter, call it, and wait, and wait…
For something as important – and a legal requirement – as the Census, this is making it too hard. Make the online way the easy way, but always allow another way for those not online. Paper forms should have been included in the letter, which could also have included a local place to get help with online or paper forms – for example trusted local groups, libraries, CABs, community centres. Stats made a good start on this by engaging with over 1,000 organisations and 400 events8 to ‘promote the benefits and value of completing the census’, and at least some of them offered assistance too. This needs to become the norm for the next Census.
We must avoid ‘online only’ situations, or we discriminate. This has happened in the UK , where Age UK has found that 41% of councils insist on online claims for two key financial benefits for older people9; a third of them (14% of the total) did not even offer assistance to do this. This led to calls for a legal right to use off-line methods.
The different user experience of paper versus online was not fully appreciated.
Forms on paper have a number of advantages that online forms lack:
- The unfilled forms are a visible, physical reminder.
- They can be filled in in private – privacy is difficult if there is one household computer, or if you have to go to a library for access
- They can be carried around and filled in bit by bit when convenient.
- They are easier to read.
- It’s easier to show them to trusted adviser.
Plus of course the paper forms used to be delivered by someone who could explain them and help you to complete them, if you wanted. (This was only initially offered in a small number of New Zealand locations in 2018.)
Some groups were alienated by the pre-set answers not including them.
There has been a lot of comment about the only gender options being Male or Female, Pakeha not being an ethnic group etc. You can scrawl your own answer on a paper form; if an online form doesn’t allow ‘Other’, your only options are to ‘lie’ or not take part.10
This is quite unnecessary: the paper form could include an ‘Other’ box, the content of which is then coded to a list of subgroups – a standard research procedure; the online form could include a supplementary question including the list of subgroups and an Other box for the rare cases of the list not being exhaustive.
‘Data quality’ does not consist of forcing all answers to match a small set of arbitrary values; it consists of answers that reflect the true situation of the individual, and a Census where the greatest number possible of New Zealanders have responded.
There were (partly justified) concerns about privacy.
As well as possible lack of privacy from using public computers, there are concerns that that assurances about sensitive data handling may not be totally correct11 and some access codes were incorrectly delivered giving people access to other households’ data12.
However, a key issue with many of the hardest to reach groups is that they are distrustful of authority and not confident with technology. Often the best way to reach them is to work through local organisations and people they trust (Wellington Downtown Community Ministry’s work with the homeless is a great example 13 ), who can meet them face to face and guide them on their way, whether it is to fill in a Census form that will help get resources targeted where most needed, or to equip them with computer skills, confidence and home internet access (as 20/20 programmes, when resourced, do). Only once they’ve got those skills, confidence and access can they interact online with the Census and other services.
The testing was fragmented, not ‘whole process’ and didn’t include all the right groups.
A lot of the problems mentioned above can be detected and pre-empted by
- testing the whole process – not just testing question wording and online forms in isolation,
- including representatives of all population groups in testing – for instance test with a group of people with disabilities as well as online accessibility experts (as a company I work with, AccEase, does). And pay them for their time.
- don’t ignore or gloss over any issues raised.
- don’t coerce people online, entice them. Make is easy for them to go online but provide a genuine off-line alternative.
The 2018 Census problems have been years in the making, and I suspect many of them come from an underlying push to go online to increase efficiency and reduce government expenditure rather than to improve the way the Census serves New Zealand citizens.
Footnotes and References
- https://www.stats.govt.nz/news/2018-census-update
- “Ninety percent is really not good”, Professor of statistics Thomas Lumley, University of Auckland, reported in The Spinoff
- See also https://2020.org.nz/blog/2018/06/22/digital-divide-worse-than-seems-in-latest-report/ where the World Internet Project NZ 2017 survey undercounted and misreported New Zealanders offline.
- The Press “Census: a case for and against” 5 March 2018
- Hindsight is a wonderful thing, but I did tell Stats about some issues beforehand…
- “We held a small number of usability tests of the online forms to test for a streamlined logon
process and the usability of the redesigned Internet Access Code. Usability testing focused on respondents accessibility of the forms, navigation, and selection of responses…” https://www.stats.govt.nz/methods/2018-census-design-of-forms - Independent Herald, Jun 6, 2018, pg 4 “Local man leads way with e-petition”
- Stats said “We worked with all universities, 320 libraries, 21 district health boards, and 73 regional and local councils to promote the benefits and value of completing the census. Our engagement team worked with more than 600 community organisations, 330 champions, and 8,000 volunteers to support more than 400 community-led promotions and events.” However, I do not recall there being any local information being included in my letter. The Census site has been removed, and the archive of it does not include a sample letter.
- https://www.charitydigitalnews.co.uk/2018/05/29/age-uk-research-shows-digital-excluding-elderly-from-benefits/
- https://thespinoff.co.nz/society/23-01-2018/damned-lies-and-statistics-why-stats-nz-should-listen-to-queer-communities/
- https://pursuit.unimelb.edu.au/articles/the-key-to-unlocking-your-privacy
- https://www.newsroom.co.nz/2018/07/10/148425/census-red-flags-raised-in-march
- https://thespinoff.co.nz/society/21-03-2018/in-the-census-do-the-homeless-count/
Backgrounder
Bill’s knowledge comes from being an Internet professional with over 20 years experience in online surveys, online usability and accessibility; and as a worker and activist reaching offline New Zealanders with digital inclusion and digital literacy programmes.
This article expresses his personal views. Follow Bill on @BillDashNZ.
Bill has been working in IT all his life and for community ICT and digital inclusion since the mid 1990s. He has carried out surveys and research into ICT for 20/20, government, business and for a Master in Information Management (VUW*) and has been (at different times) sponsor, volunteer, Trustee and contractor for the 20/20 Trust; NZ web master strategist, marketeer, training manager and customer support for a top 3 global IT company; web developer, quality control, accessibility and usability assessor.
Whilst managing CommunityNet Aotearoa in the mid 2000s for the Department of Internal Affairs he was part of the working group on the whole of government New Zealand Digital Strategy, advised on community IT project proposals and received an online accessibility award.
He is passionate about web accessibility, working with AccEase to supplement their team of disabled testers with technical accessibility / usability assessment and expert analysis. He also organises NetSquared Wellington, a monthly meetup group for community organisations and change-makers using technology for social good.