Briefing Notes

B: Further Analysis of Research Methods

B2: Ask if you don't know ... the research questionnaire

Are questionnaires making a comeback? My mail so far this year has regularly been spiked with consumer data collection surveys (I replied to the one from the British Library), questionnaires from government departments, publishers, pressure groups and professional bodies, even one from a General Election candidate and, of course, the decennial Census form. How useful are they? If we are silly enough to voluntarily surrender consumer profile information to interested businesses this is likely to be very useful (to them), and the politician collected my vote by taking the trouble to ask some questions. As for the UK Census, the more recent parts of this unique time series (now covering 160 years) are invaluable to planners and, with the 1901 records due for public release next year, what would the family history buffs do without the earlier returns?

More depressing are the questionnaires arriving by post and e-mail rehearsing variations on "My organisation is agonising over its position in the world/ going through the motions of Best Value Review. As a really valued stakeholder, please take the time to ..." The main problem here is that survey managers will keep trying to get complex information and highly subjective responses from people using an instrument that is really designed to systematically gather small amounts of comparable information from a targeted group of respondents. Once you move beyond the intended territory, you are never likely to know whether you have projected your preconceptions as questions so that replies are likely to come from like-minded individuals, people who guess the meaning, or people who interpreted your questions in their own way and answered accordingly. Their replies to open questions are often messy and ambiguous and you have no way of finding out what people really meant. All this is before the results are interpreted and conclusions drawn. Last year I watched one public library senior management team wrestling with the dire implications of key messages from non-users of their service, until I took a hand and began grilling the Corporate Best Value Guru. He eventually admitted that the response rate from non-users to this particular questionnaire was lower than 2%. Why was he so reluctant to tell us this?

Towards better questionnaires?

Sheer despondency, however, is induced by the poorly designed questionnaires that roll in from LIS students, service providers and once in a while from other researchers. Some of these can be instantly recognised through their hotchpotch presentation – if in doubt cram in another question and the smaller the amount of space left for replies the better (fewer words to analyse?). Others are better presented; I wonder how many questionnaire replies are abandoned half way through when the ambiguous questions or lack of any sense of direction dawn? Here are some ground rules gleaned from too many hours looking at poorly designed questionnaires:

• Give attention to the structure of the questionnaire
   
  • start with a straightforward question to get the recipient to start replying
     
  • move from the general to the specific in each section (usually)
     
  • try to ensure some variety in the types of question asked (see below)
     
  • draw skidpaths if there are alternative routes (of the type 'if 'Yes' go on to question x') to make sure that every eventuality is covered. Then choose whether to leave these in to help people move through the questionnaire
     
  • group related topics into modules
     
  • make the structure clear to the respondent by using headings
     
• Choose the question types to fit the purpose. The main options are:
   
  • closed questions – pre-assigned response categories or 'yes' and 'no' boxes (make sure that each category is distinct [especially for age ranges] and that all eventualities are covered – if your response category is not provided what does this say about the competence of the designers?)
     
  • open questions – at simplest this may be a 'Why is this?' after a closed question; the intention is that the respondents should reply in their own words (don't forget to analyse these and that categorisation/synthesis takes time)
     
• Use response scales where appropriate. These are a form of closed question. Most common are:
   
  • Likert scales: a set of choices to record agreement/disagreement
     
  • Guttman scales: statements arranged according to the strength of attitude
     
  • Thurlstone scales: forced choice to agree/disagree
     
  • Semantic differential: seeking quantitative measures by offering scales between extremes.
     
• Offer clear and consistent instructions for completing the questionnaire (easily forgotten if questions are considered individually)
   
• Pay attention to question wording – since this is a topic in its own right we'll come back to this theme in a later column.
   

To summarise, the overall aim of the research questionnaire is to obtain research-relevant information efficiently. You need active co-operation from your target group when doing this. Anything that will help people to co-operate with you by carefully completing and returning your questionnaire should be actively encouraged. This takes time, planning and attention to detail – even before you reach the data analysis stage.

Contents © Information Management Associates 2000 - 2003

Design © ConnectWorks Ltd. 2000 - 2003