Briefing Notes

B: Further Analysis of Research Methods

B6: Fitness for the (research) purpose: some traps and how to avoid them

A few years ago I was asked to review an educational research project. This piece of work had been executed in a painstaking and methodical way, using well-tried and reliable research methods and applying them well – and therein lay the trap! Unfortunately, one dimension was missing from what would otherwise have been a good project. Although great attention was paid to getting people to participate in the research and provide information, most of the people targeted in the research were not in a position to provide the information being sought!

This research was focussed on barriers to lifelong learning presented by further and higher educational institutions and on perceptions of the education institutions by people who were socially excluded. Who would you target if you were interested in these questions? The project focussed on the people who were already sufficiently socially included to be attending college or university. This is problematic: most further and higher education students are by definition involved in lifelong learning and few would fall within the usual definition of the socially excluded. It could be argued that New Deal students in further education are drawn from the socially excluded, but they only formed an (unspecified) minority of the people interviewed in the project. The vast majority of the people interviewed in this study were higher education students.

A sensible approach was taken to the linked survey of students, by targeting particular classes (i.e. lessons) and persuading the students to fill questionnaires in on the spot (thus ensuring a high response rate). However, if you do this it is vital to analyse which respondents are in each class to ensure that the views of New Deal students or other categories of 'potentially socially excluded' students are properly considered, bearing in mind that FE courses range between franchised HE courses and HE access courses, through to basic literacy and numeracy. This differentiation didn't happen (or at least was not reported). Nevertheless, the project came forward with a series of findings about barriers to further and higher education access. Is it unfair to suggest that the research team was so busy getting on with the research job and making sure that it was all done well that they forgot to ask themselves whether they were talking to the right people to get the information they wanted?

The pre-packaged trap'

This is not the first time that I have encountered problems around fitness for purpose in relation to research projects. The usual way this issue manifests itself is statistical, or rather it involves statistical misinterpretation. At various levels from Masters' Dissertation upwards I have encountered examples of the wrong tests being applied in asserting the significance of statistical findings. This might be termed the 'pre-packaged trap' since what happens is that the chosen statistical package obligingly offers a standard test and the student (or occasionally the professional researcher) equally obligingly reports the result and interprets the data accordingly, regardless of whether the test is appropriate in that case. The message here is, if in doubt consult a statistician.

'Painting yourself into a corner'

Occasionally, the urge to be seen as 'scientific' in carrying out a research study gets in the way of any effective pursuit of the research question. An interesting example occurred last year, when I was asked to referee an article for an international information journal. The authors had collaborated on a project in which they tried to assess whether mediated e-learning packages worked better or worse than traditional teaching, or e-learning only, for various groups of business students. The research design appeared impressive: students were randomly chosen from across a range of courses, parallel cohorts were subjected to different combinations of teaching delivery methods for particular parts of their course, their views on the different learning experiences were collected and their success or otherwise in completing the course was logged.

A closer look at the research question led to further questions – what about the prior IT and e-learning experience of the students and what was being compared? Were the teachers in face-to-face sessions merely reading out what was stored in the electronic versions and, if not, how could the experiences of the students be compared? In this case, the authors appear to have exerted their energies in the wrong direction – rather than concentrating on random samples they would have been better occupied in carefully selecting comparable cohorts of students as the basis for their study, before subjecting them to consistently presented and mediated learning opportunities (not easy!).

How can some of these pitfalls be avoided?

The answer boils down to a two step process:

1. The researchers asking themselves a set of questions. Obvious candidates are:
    
   • What information do we really need to gather?
      
   • Whom should we be talking to? If we can't get at them are there any acceptable surrogates?
      
   • How can we get at enough of them and how should they be chosen?
      
   • How can we engage with them to ensure that we get the information that we need and that we understand what they are telling us?
      
   • How can we get their interpretations of what we are finding out?
      
   • And (later), is the information that we have gathered relevant to our research question?
      
2. Then, to the interesting bit - asking the same questions of colleague researchers and potential research 'subjects' – the people in the research settings.

Contents © Information Management Associates 2000 - 2003

Design © ConnectWorks Ltd. 2000 - 2003