Evaluating Effectiveness ASCSA event Adelaide July 19 2010

19 July 2010
6:30 pmto8:30 pm

On Monday 19 July we held an event at the Science Exchange on evaluating science communication activities.

A key challenge of science engagement activity is showing your effectiveness and measuring success.  How do we know that our programs and activities are making a difference?  Do we have the impact we desire?  Are we engaging with the right audiences? How do we evaluate what we’re doing?  At the July ASCSA meeting we’re inviting several people from community engagement/education programs, including science shows, museums and science events, to share their experiences of how they go about evaluating their programs.  This event will be relevant to anyone involved in community engagement/education programs, and we’d especially love to see you if you’re involved in planning a National Science Week event for August this year.

A podcast of the evening is now available at http://www.riaus.org.au/science/whats_on/past_events/audio_recordings.jsp

Some references on evaluation:

RCUK:  http://www.rcuk.ac.uk/per/bestprac.htm
Beacons for public engagement: http://www.publicengagement.ac.uk/sites/default/files/EvaluatingPublicEngagement.pdf
National STEM Centre: http://www.nationalstemcentre.org.uk/res/documents/page/STEM_Does_it_work_revised_Oct_09.pdf

The importance of evidence

There are many ways of communicating science. But what evidence do we have that any of it is effective in improving the understanding of science? The answer is very little. When it comes to engaging students or the public in science, we may be doing no more than moving deck chairs on the Titantic.

Sless and Shrensky characterise the situation perfectly “…the evidence for the effectiveness of (science) communication is about as strong as the evidence linking rainmaking ceremonies to the occurrence of rain,” Sless, D. and Shrensky, R. (2001) Science Communication in Theory and Practice, eds Stocklmayer, S., Gore, M.M. and Bryant, C.R., Kluwer Academic Publishers.

Some years ago a leading American science communicator, Dr Rick Borchelt, led a team of ‘blue ribbon’ science journalists and communicators to produce a best practices roadmap for NASA’s Marshall Spaceflight Center. A key finding of the 2001 report was that it was surprising that science, as a data-driven enterprise, did not demand evidence of the effectiveness of science communication.

There are several reasons why evidence does not exist beyond simple evaluation.  One is lack of funding to gather the evidence. While the resources required may be less than that needed for science research, it is still significant enough to need substantial funding.

Another aspect is the value of science communication, especially in a university environment.  No ‘research quantum’ (monetary value to a university) is assigned to science communication successes because Government funding does not include it. For example, I led a team in collaboration with NASA to produce a science-related project, which in 18 months attracted five million hits on the associated wiki.  Why is that not worth quantum for the research centre it was done in?

Science communication is still – more often than not – included almost as an afterthought in relation to science research. I’ve been involved in a major report on a specific area of research where for more than two years I fought for the education and outreach to be recognised as more than the required add-on with little or no funding.  We are reaching out to tomorrow’s scientists – and with three decades of decline in interest in science in our high schools, goodness knows we need it.

Scientists are frequently called on to do more in communicating their science to the public. But what evidence is there that it does anything to improve the public understanding of science?

Borchelt and his team found, after two years of research, that the principle reason scientists communicate with the public is not the one they intend. The actual aim is to please their bosses in getting column or airspace to promote their institution’s science. The Borchelt team recognised there is nothing wrong with this kind of promotion of the awareness of science. Nevertheless, it is not the same as improving the public understanding of science. And in either case there is no evidence of success (beyond pleasing an institution’s boss).

One of the key tenants of good communication (science or otherwise) is ‘know your audience’. On this rests the most serious issue of all for Australia. Unlike the US, Europe, and elsewhere, we do not test public audiences for their understanding of science. We also do not test our students leaving their high school science education for that understanding of science. We do not test for the effectiveness of science curricula. And the proposed Australian science curriculum is based on…you’ve got it…nothing – at least in the box that says ‘how effective are current science curricula?’

How does this impact on public audience consumers of science in the mass media? Journalists have little but intuition to make assumptions about reader/listener/viewer understanding of science, let alone know how information via the mass media shapes personal world views about science. But as a past science journalist, I know the response: we do not aim to educate: we inform – based on the evidence that science provides.

Perhaps the Australian word for a type of camping gear – a swag – is actually an acronym for the effectiveness of science communication: “Stupid Wild Assumptions and Guesses”!

More seriously, how can we understand what we are doing (or achieving) in science education, outreach, and the public understanding of science, unless we have the evidence of effectiveness in any of those areas?

We need significant change to happen if we are going to do anymore than what seems to be intuitively good to do – from high school science education to science communication in the public domain. We have a lot of work to do.

•    Dr Carol Oliver is a science communication specialist with the Australian Centre for Astrobiology at the University of NSW.