Briefing Notes

B: Further Analysis of Research Methods

B1: Getting under the skin: the critical incident approach

Much of my recent and current research is focused on assessing the impact of library services. Whether we are working in education, health, social care or public libraries we keep hitting the same set of linked issues:

• Library service managers rarely stand back from concern with service provision for long enough to clarify the aims driving their services.
   
• As a result, they give little attention to how they can judge whether they are succeeding in meeting their aims and they have corresponding few impact assessment indicators in place (as distinct from service performance indicators that, broadly, tell whether the service is running efficiently or not).
   
• Unsurprisingly, when they do identify areas in which they want their success in meeting aims to be assessed, they seldom have enough baseline information about these areas on which to judge any progress.
   
• Finally, they are sometimes at a loss about how appropriate evidence of impact can be gathered.
   

The first three of these are development planning issues; indeed, success in addressing them depends partly on being able to distinguish between development planning and infrastructure maintenance planning. The fourth issue is closer to home, because it usually translates into the question, what constitutes acceptable evidence? There is an underlying political question here. Organisations usually seek to assess the impact of their services for more than one reason: to account to an external agency; to reassure or impress the Board or equivalent body; to provide ammunition to promote the service; or to give to service managers information on which to take resource allocation decisions. Traditional performance indicators combined with judicious benchmarking against similar services tend to be demanded by the first two groups (even if they don't do anything with them); good qualitative information about impact may help to promote the service and will certainly be of use to the service managers. But how can good impact information be obtained?

When we talk about the impact of services we are necessarily concerned with the impact on people, usually service users or potential users. As we have already noted in talking about qualitative research methods in an earlier column, there are only a limited number of ways of gathering this information and they boil down to variations on the questionnaire, the self-completed log, the interview and (other forms of) observation.

Which approach will work best for libraries? Obviously this depends upon what you are trying to find out, but one potentially useful approach is probably under-used. All libraries provide answers to people's questions and most spend a significant amount of time on enquiry work. In some libraries, completed enquiries are categorised to provide limited performance information but it is possible to go much further.

Critical incident interviews

I have regularly been employed by national education and health information agencies as well as library services to conduct critical incident interviews based on enquiries that they receive. The usual approach in this type of interview is to select up to fifty substantial enquiries (not request for contact addresses etc.) which form the critical incidents. Then:

1. Telephone interviews are arranged with the enquirers.
    
2. At the beginning of the interview the enquirer is reminded about the particular contact with the agency (using the enquiry record completed at the time). Two difficulties to watch for here are:
    
   • when the enquirer is in such frequent contact that recall of the particular incident is hazy (options: proceed and hope that remembrance returns or negotiate discussion of a more recent and memorable enquiry)
      
   • when what the agency treated as an enquiry is regarded as a conversational gambit or part of an information exchange by the 'enquirer' (response: proceed but record the different interpretation of what was going on).
      
3. Contextual information is gleaned. The enquirer is asked why they were seeking that information at that time (what triggered them to make the contact; what was happening in their organisation or in their own life that brought this to the fore). Questions are also asked about any other information sources tried and why.
    
4. Evaluation: what useful information, if any, was obtained from any of the sources tried and why was it useful (and if relevant, how did the agency response compare with other people's)?
    
5. Quality assurance: were the staff of the agency outstandingly rude, slow to respond, slapdash – or polite versions of these questions (people do tend to remember outstandingly good treatment as well as the opposite).
    
6. Impact: was any use made of the information provided by the agency and if so, did anything happen as a result?
    
7. General comments on the enquirer's experience of the services provided by the agency.
    

Following this approach to enquiries tends to provide a better picture of how the service fits into the evolving world of a range of users, as well as of the main factors that affect people's readiness to make enquiries. This approach can also provide a basis for enquirers to make direct comparisons of services (including such elements as format and presentation) and can produce occasional but powerful information about the real impact of information on people's life and work, because it offers a very specific and direct focus for the user to make a judgement.

More strategically, sets of interviews can readily be translated into case studies for use in promoting services and discussing options with senior managers. Perhaps most importantly, service managers report that they find this type of information useful in reviewing and fine tuning their services in ways that traditional library statistics don't touch.

Contents © Information Management Associates 2000 - 2003

Design © ConnectWorks Ltd. 2000 - 2003