How do policy makers find the right evidence?

Decision makers want to use evidence because it can lead to highly effective campaigns, programs and policies – and better outcomes for Australians. But how do they pinpoint the best evidence to use when there’s so much to sift through?

This is the topic of the Sax Institute’s upcoming training course, Critical Appraisal of Research to Support Policy and Program Professionals, which is being held online on 8 May. We caught up with course co-presenter Professor Mary Haines for her top tips when appraising research.

Q: Is the importance of using research in policy now generally accepted, or are there still barriers?  

Professor Haines: I think most people consider using research in policy as important – that argument has been won! If you look at most policy documents nationally and internationally, you’ll see the notion of evidence-based policy. A great example of it from decades ago is Australia’s ‘Slip! Slop! Slap!’ campaign, our first sun-awareness campaign that responded to evidence about the ozone layer, UV rays and sunscreen. And the best recent example of evidence use on a global scale was the COVID-19 pandemic response. So yes, for at least 10 to 15 years there’s been a widespread will to use research in policy and a general understanding of it.

The key challenge now is that we’ve almost got too much research. We’re in an information-rich environment, where there’s been a tidal wave of published research. So the big barrier to using research in policy now is: how do you determine which research is going to help you?   

Q: What are the mistakes people make when assessing research?

Professor Haines: There are three big mistakes. The first one is not clearly framing your question. It’s not enough to say, “I’m working in policy area X and I have problem Y”. What specific piece of evidence do you need to do your job? Do you need evidence that tells you the scale and extent of the problem? Or do you need evidence to tell you what the solution would be? Do you need evidence about what intervention will be acceptable to the community?

The second mistake is to think that all research is equal, therefore a quick search will give you what you need. And the third mistake happens after the research has been selected: poor presentation of your findings.

Q: How can critical appraisal help policy makers assess and use research?   

Professor Haines: Critical appraisal gives you a mindset for approaching research as well as a framework and tools for assessing and presenting research, and these are all taught in our course.

The mindset is what my course co-presenter, Professor Carmen Huckel-Schneider, calls a “neutral position”. It’s a dispassionate point of view that allows you to ask, “What is the research really telling me?”

The framework of critical appraisal first helps you determine the higher-level question you’re asking.  Are you after evidence on the problem itself, the possible solutions to the problem, or the best solution for your community. Knowing this allows you to be able to search for the exact evidence you need among that tidal wave of evidence.

The screening tools of critical appraisal are used to find the best evidence. When you read papers using critical appraisal, you’ll be looking at things like sample size and the strength of outcomes to assess the quality of the research. You’ll also be questioning the research as to how well it answers your question – is the factor it’s telling me about small or significant in causing the problem? Is the solution it’s telling me about feasible in Australia or my community?

The report is an important part of critical appraisal that’s often overlooked. How you package or present your appraisal can determine whether that research influences policy. A great package will clearly explain the question, summarise the chosen research, list its strengths and weaknesses, and then clearly state where the science lands. No-one needs to sift through a 100-page report!

Q: What hands-on training do participants get from your course?

Professor Haines: Carmen and I have made this a very practical course – we want to make life easy for our students, who are busy policy makers and program professionals. We aim to de-mystify research for our students and get them using the framework and tools of critical appraisal.

One of the ways we do this is by running a scenario where the boss needs a brief on what the science says about a particular issue because they’re going to present on it at an important meeting. We guide students through each part of the process, including the packaging of the actual brief. The feedback on this style of training is really positive.

Professor Mary Haines has more than 20 years of senior experience across government, corporate, academic and not-for-profit sectors, with expertise in health research, evaluation and translational initiatives. Mary has held senior executive service and Board positions in the NSW Government and serves as board member of Sydney Local Health District. Mary is an Adjunct Professor at the University of Sydney, School of Public Health, Menzies Centre for Health Policy and a Senior Adviser at the Sax Institute.