Department of Education, Skills and Employment
Ensuring meaningful survey results with cognitive interviewing.
Job Seeker Survey Cognitive Interviewing
Cognitive interviewing
Cognitive interviewing is an invaluable tool to explore respondents’ comprehension and interpretation of survey questions and provide assurance that a survey will deliver meaningful results.
During cognitive interviewing, Wallis’ experienced consultants utilise cognitive interviewing frameworks to observe and understand behaviours and thought processes of a participant when interpreting and responding to a question.
Where cognitive interviewing identifies a problem, the survey content can be revised to ensure survey questions are interpreted as intended.
Process
Initial pre-test refinement of the survey instrument
An initial refinement of the questionnaire was conducted based on Wallis’ experience conducting research with job seekers, together with data the Department had from their own quantitative testing of the initial questionnaire.
Recruitment
Recruitment was managed by Wallis’ qualitative fieldwork coordinator and utilised a rigorous recruitment and reminder process. A combination of phone calls, email and SMS ensured that once participants were booked into the research, they had all the information required to participate.
Phase one of cognitive testing
The first phase of cognitive testing consisted of ten cognitive interviews with a diverse range of job seekers. Testing was carried out across both CATI and online modes to test for the different ways questions were interpreted whether read or spoken.
Following the completion of round one, a report was provided with recommended changes and refinements to the questionnaire.
Phase two of cognitive testing
Following consultation with the Department regarding changes to the questionnaire, an updated questionnaire was prepared, and a second phase of five more cognitive tests was carried out. These interviews allowed for further testing of existing concepts, but with particular attention paid to the testing of changes made to the survey instrument after phase one.
Final report
Following the second phase of cognitive testing, a final report covering the entirety of the exercise was prepared, with a focus on action-centred recommendations for the final survey instrument.
“This project has been really informative, it’s helped improve the survey a lot and has built on our survey design skills more generally too. [Wallis] have been really great to work with – everything ran very smoothly from our perspective which we really appreciated. So a big thank you for that!”