Briefing Notes

B: Further Analysis of Research Methods

B5: Rounding up the literature

• token references to the recognised authorities on the topic in question. This approach is fairly common in journal articles; my favourite example appeared in a piece on the early work of Dostoevsky, where the author intended to write that "The views of Professor Brown on this topic are worth noting". Unfortunately, the last word acquired an additional 'h' in the course of publication!
   
• more or less systematic summaries of relevant earlier research on each aspect of the topic in question, complete with citations, leading up to a presentation of the new project's approach and findings. This is the accepted approach for most doctoral theses and formal research reports in journals;
   
• a variant in which the authors cite all the sources that they have drawn upon directly in their work and then add an 'additional reading' section. This section usually serves to defend the authors against accusations of not being aware of other significant research. Unfortunately, such lists too often serve merely to enable the next researchers on the topic to protect their own backs by recycling the same references. Some years ago I tried to obtain the originals of all the publications cited by the main researchers on my research theme (information use in social services) and found that virtually all the citations were incomplete, most were inaccurate, many were garbled, some were downright misleading and a few apparently did not exist. One frequently-cited 'seminal' work turned out to be an amalgam of three works by different authors;
   
• fully-fledged annotated literature reviews covering all relevant research published within specified parameters (usually of language and theme).
   

Systematic literature reviews

The move towards evidence-based health care is gathering momentum in the UK, following on from the examples set in Canada and the USA. Putting the focus on decision-making based on the best available research evidence has led to renewed emphasis on being able to obtain relevant research reports, resulting in various initiatives such as the creation of the National electronic Library for Health. The evidence-based approach has also led to greater attention being placed upon the quality of research, how the research is reported and to its interpretation by practitioners. Elements of the evidence-based approach are now being picked up in other areas such as health care and education. As a result, there is growing government interest in commissioning systematic literature reviews of research and in developing appropriate methods for carrying them out. For example, the recently created EPPI-Centre and the National Foundation for Educational Research in England and Wales have evolved an approach to systematic literature reviews that goes well beyond assembling citations and adding annotations. What is offered below is an amalgam of such approaches, aimed at ensuring a rigorous, transparent and systematic critical review of the literature.

The main steps in this process are to establish:

1. the parameters of the research and specify the inclusion criteria;
    
2. the key questions to be pursued;
    
3. the keywords to be used (including any modifications during the project);
    
4. which sources are to be searched and for what time periods;
    
5. what kinds of material are to be included (e.g. only research reports; other strategic reports; other specified types of publications expressing opinions etc.; grey literature);
    
6. in relation to each item retrieved: whether it fits the stated criteria, how well the research has been conducted (including the type of research and an assessment of its likely reliability), and how well this evidence answers one or more of the research questions;
    
7. what has been looked at. The record for each item covered should include:
    
   • a description of the research design, methods and purpose (including any missing evidence)
      
   • a summary of the research findings (the crux of this approach)
      
   • whether this research has been included in the final review;
      
8. what has been found. The relevant research should be grouped by question and by type of research (with, for example, survey-based projects grouped and considered together) and the answers to the questions should then be offered;
    
9. what gaps remain. Which questions have not been answered or have only been partially answered?
    

This approach to the literature review is clearly more time-consuming and expensive than most traditional approaches, even before various quality assurance checks, such as the use of panels of assessors or double-reviewing of research reports are introduced. Even so, the trend appears to be towards the creation of more transparent and reliable guides of this kind, which gives rise to some interesting questions: How long will it be before the ritual research review at the beginning of each new project will be replaced by a once-only reference to the relevant critical literature review and its conclusions? Or will academic examiners and journal referees continue to insist upon customised literature reviews being produced to show the awareness of the researchers? If so, how much of the rigour associated with systematic reviews will be expected from doctoral students and potential authors?

Contents © Information Management Associates 2000 - 2003

Design © ConnectWorks Ltd. 2000 - 2003