Golden Rules | Evaluating Evaluation

Golden Rules | Evaluating Evaluation

I’m quite new to the world of evaluation. When I went part-time at the Museums Association a few years ago, as well as becoming a partner in the Museum Consultancy, one of the people I hooked up with was Christian Heath at King’s College London. We were both surprised that in spite of the millions spent on summative evaluation, it seems to have had a rather limited impact on practice. We wanted to find out why – and we wanted to find out what evaluations could tell museums generally about making engaging visitor experiences.

We devised a project called Evaluating Evaluation and raised some money to do it from two of the main funders of museum evaluation: the Wellcome Trust and the Heritage Lottery Fund. You can read a little more here.

We’ve spent much of 2012 immersed in evaluations, mainly of permanent displays in museums and galleries, with a few from science centres and historic buildings. I soon came across Eric Jensen from Warwick University, a man on a mission running a series of seminars to encourage evaluators to use more rigorous methodologies.

And, at first, I thought that the reasons why evaluations weren’t better shared and compared was because of methodological problems. But, as the Evaluating Evaluation investigation progressed it became clear that even if all evaluations used perfect social science methods they still wouldn’t have a huge impact.

As Nicky Boyd says, there are institutional and organisational issues that prevent evaluation being shared and used to its full extent. Evaluators often come from outside the museum, or have a relatively marginal role inside it, and so do not have a big influence; project teams disperse after completing their work, their knowledge often disperses, too; few museums have proper systems for recording the findings of evaluations and feeding their lessons into future work [although some, such as the British Museum, do]; in the case of summative evaluations of permanent displays, which we focused on, it’s often simply impractical to make remedial changes to a complex gallery.

As our project got started, some evaluators got in touch and told us how dissatisfied they are with the way some museums approach evaluation. They feel museums often prepare lousy briefs, expect far too much for the available fee, have little idea what they are going to do with the evaluation once complete and confuse evaluation with advocacy.

This latter point is crucial. I think summative evaluation can be expected to play four conflicting roles. First is the role we’d all say is the most important: to help the project team learn from their work and improve their future practice.

Second, the similar but subtly different role of supporting learning beyond the project team, or in the wider sector. Here, the fear mentioned by Nicky Boyd comes into play. Project teams can be reluctant to expose their failures – sometimes even within the institution they work for.

Third, most summative evaluation comes at the end of an externally funded project and is a funder’s requirement. There’s a real risk that this confuses evaluation with the funder’s systems for monitoring and accountability. Unless the funder and the museum have a very trusting relationship, it’s likely that the evaluation will be under pressure to ‘prove’ to the funder that the museum met all its objectives.

Finally, in similar vein, some museums don’t want evaluation at all – they just want good news that they can use for advocacy.

I fear that because of the organisational issues, it will be hard for the sector to move to a position where it can realise all the potential benefits from summative evaluation. [I should stress that our focus has been on summative evaluation; the problems seem fewer, and the benefits easier to realise, in the case of front-end and formative evaluation.]

After reading dozens of summative evaluation reports I have to say that my main feeling is one of disappointment that they actually tell us rather little about what visitors actually experience. There’s lots and lots of information in summative evaluations, much of it very interesting, but little of it seems actually that useful beyond the museum that commissioned it – and in some cases it might not in fact be that useful even there.

Sorry to say, if we’re looking for ‘golden rules’ about how to create great museum displays, I don’t think we’ll find them in summative evaluations. [It may be, of course, that the golden rules in fact don’t exist.]

Some evaluators have suggested that museums might move slightly away from summative evaluation towards purer audience research. Perhaps, for example, a group of museums could jointly align all the evaluations they commission to address a consistent set of research questions. Then, over time, comparable data would build up and we might gain some solid empirical evidence about how to better engage our audiences.

Author | Maurice Davies is a visiting senior research fellow in the Department of Management at King’s College London, a partner in the Museum Consultancy and Head of Policy and Communication at the Museums Association  | maurice.davies@ntlworld.com

A Hard Habit to Break

A Hard Habit to Break

Sustainability | How we agonise about the rhetoric matching the action

The view from Tony Butler’s desk at the Museum of East Anglian Life where he sat and wrote this blog

The Happy Museum project proposes that museums are well placed to re-imagine a world where economic growth is not the principle measure of success of a society. Museums can show that materialism need not characterise our relationships to the world around us. A more powerful retelling of history by way of cooperation and collaboration and a more humble approach to the natural environment might help civic society address pressing issues such as climate change and resource depletion. The project suggests that museums take a more activist approach to working with communities and use their status as places for encounters to foster local campaigns.  

How far museums are able to help lead this change remains to be seen.

For some time museums have been pioneering thinking around environmental sustainability (The Garden Museum) and doing great work within their communities (most local museums!) However very few combine the two to foster the sort of citizenship that Matthew Taylor of the RSA says ‘helps people to be the people they need to be, to do what they think they want to do’. At The Happy Museum Symposium last January, in typically provocative style, the Museums Association’s Maurice Davies suggested that museums were not able to link well-being with sustainability.  Our project is seeking to create a community of practice of UK museums committed to supporting transition to a high well-being, sustainable society. Two of the six museums we have commissioned to make these links explicit are:

• Godalming Museum is working with allotment holders, cycling clubs and transition groups to develop displays and resources about a Victorian hydro electricity project in a nearby tanning works.
• The Story Museum in Oxford has employed well-being experts and psychologists to work with architects and surveyors to influence the sustainable design of their new museum.

Faith in exponential economic growth is a hard habit to break.

Mainstream culture seems to be reluctant to question the primacy of economic growth or be brave enough (an agony for liberal cultural professionals) to suggest to the public that lower consumption might contribute to a kinder safer society. This mirrors a similar attitude in mainstream politics. The Labour politician Caroline Flint MP and the Shadow Climate and Energy Secretaries wrote a blog for The Guardian (02 April 2012) reporting that the public were more likely to accept climate change if policies were concerned with people’s everyday concerns - bills not bears!

As a Clore Fellow at the Southbank Centre in 2008 I was asked to design a sustainability programme.  On the one hand the redeveloped Royal Festival Hall had a sustainable ground source heat pump system and the Participation Team was developing programming strands around climate change through an artists in residence programme. The programme (run with Cape Farewell) pioneered a cultural response to climate change by taking artists to witness the impact of melting ice in the Arctic.  On the other hand, Southbank continued to fly in artists from half way around the world for one-off performances and relied on sponsorship from a big oil company to support the Classical Music Programme.  Here, one of the UK’s best-known cultural centres had an opportunity to have a dialogue with itself and its audience about the ethical dilemmas in bridging the gap between rhetoric, artistic excellence and the leadership responsibilities of a cultural institution.

Four years on, public funding priorities seem to be driving cultural organisations to take an ever more short-term approach. Creative Scotland recently announced that it would like to move to one-year or project-by-project funding agreements. The Department for Culture Media and Sport is driving organisations to make the most of corporate and private philanthropy. These further complicate the judgements made by organisations like museums in their programming choices and their own beliefs and values. So, big institutions that receive sponsorship from oil companies or weapons manufacturers (of late Tate - BP and the Science Museum - BAE Systems) should be prepared to substantiate why they enter into these relationships.  Bridget McKenzie, writes in a recent blog that organisations should not believe 'that when  money directly gained from ecocide or violence is given to a museum it is washed clean.' 

Tony and the Museum’s objects and land.  What sustainable challenges does he face?

So now, as the Director of the Museum of East Anglian Life (MEAL), I face my own dilemma of balancing the Happy Museum rhetoric with the desire to develop new work and the need to pay the increasing costs of rates and energy bills.  In a blog 18 months ago I suggested that museums might look to the Transition Movement as a means to co-operate, share resources and innovate but this seems a long way off – even for me!  And in a recent conversation about the future of funding to museums with my local council the chief executive talked about how this would be dependent on local economic growth, in particular new housing development. With central government grants cut and council tax rises politically unacceptable, the only source of new income to fund some services would be through payment schemes such as the New Homes Bonus incentive.  I find myself arguing that one of the strongest justifications for the continued funding of MEAL is the economic contribution to the locality - which is over £1.2m a year.

Can you spot the molehills?  One of costs the Museum encounters through nature.

So what’s the likelihood of a museum developing a financial model that combines both a cash and exchange economy? 

Time banks or skill sharing schemes are great, but they won’t pay fuel, insurance and maintenance bills for the 20 historic buildings at MEAL. Through a Social Return on Investment study we carried out in 2010 we know that our museum creates £4.50 of social value for every £1 of cash invested. We are able to use these statistics to increase funding streams.  But to be fully sustainable shouldn’t we be working to convert the social value into something practical to exchange?  

A great place to start is Mission Models Money re-think programme which looks at how culture can contribute to the Transition and to a high well-being low carbon economy. In a society re-framed by an economy more aware of its environmental limits we can expect that people will consume differently, spend their time differently, and have different expectations of their lives.  This is all food for thought for our Museum Activist who will be starting in August and will be seeking a more cooperative and collaborative approach to our engagement with local people. 

What challenges will the new Museum Activist face to bring well-being and sustainability together?

Author | Tony Butler, Director of Museum of East Anglian Life and supporter of LMG.

Photographs | Julie Reynolds, LMG Blogger.