Golden Rules | Evaluating Evaluation

Golden Rules | Evaluating Evaluation

I’m quite new to the world of evaluation. When I went part-time at the Museums Association a few years ago, as well as becoming a partner in the Museum Consultancy, one of the people I hooked up with was Christian Heath at King’s College London. We were both surprised that in spite of the millions spent on summative evaluation, it seems to have had a rather limited impact on practice. We wanted to find out why – and we wanted to find out what evaluations could tell museums generally about making engaging visitor experiences.

We devised a project called Evaluating Evaluation and raised some money to do it from two of the main funders of museum evaluation: the Wellcome Trust and the Heritage Lottery Fund. You can read a little more here.

We’ve spent much of 2012 immersed in evaluations, mainly of permanent displays in museums and galleries, with a few from science centres and historic buildings. I soon came across Eric Jensen from Warwick University, a man on a mission running a series of seminars to encourage evaluators to use more rigorous methodologies.

And, at first, I thought that the reasons why evaluations weren’t better shared and compared was because of methodological problems. But, as the Evaluating Evaluation investigation progressed it became clear that even if all evaluations used perfect social science methods they still wouldn’t have a huge impact.

As Nicky Boyd says, there are institutional and organisational issues that prevent evaluation being shared and used to its full extent. Evaluators often come from outside the museum, or have a relatively marginal role inside it, and so do not have a big influence; project teams disperse after completing their work, their knowledge often disperses, too; few museums have proper systems for recording the findings of evaluations and feeding their lessons into future work [although some, such as the British Museum, do]; in the case of summative evaluations of permanent displays, which we focused on, it’s often simply impractical to make remedial changes to a complex gallery.

As our project got started, some evaluators got in touch and told us how dissatisfied they are with the way some museums approach evaluation. They feel museums often prepare lousy briefs, expect far too much for the available fee, have little idea what they are going to do with the evaluation once complete and confuse evaluation with advocacy.

This latter point is crucial. I think summative evaluation can be expected to play four conflicting roles. First is the role we’d all say is the most important: to help the project team learn from their work and improve their future practice.

Second, the similar but subtly different role of supporting learning beyond the project team, or in the wider sector. Here, the fear mentioned by Nicky Boyd comes into play. Project teams can be reluctant to expose their failures – sometimes even within the institution they work for.

Third, most summative evaluation comes at the end of an externally funded project and is a funder’s requirement. There’s a real risk that this confuses evaluation with the funder’s systems for monitoring and accountability. Unless the funder and the museum have a very trusting relationship, it’s likely that the evaluation will be under pressure to ‘prove’ to the funder that the museum met all its objectives.

Finally, in similar vein, some museums don’t want evaluation at all – they just want good news that they can use for advocacy.

I fear that because of the organisational issues, it will be hard for the sector to move to a position where it can realise all the potential benefits from summative evaluation. [I should stress that our focus has been on summative evaluation; the problems seem fewer, and the benefits easier to realise, in the case of front-end and formative evaluation.]

After reading dozens of summative evaluation reports I have to say that my main feeling is one of disappointment that they actually tell us rather little about what visitors actually experience. There’s lots and lots of information in summative evaluations, much of it very interesting, but little of it seems actually that useful beyond the museum that commissioned it – and in some cases it might not in fact be that useful even there.

Sorry to say, if we’re looking for ‘golden rules’ about how to create great museum displays, I don’t think we’ll find them in summative evaluations. [It may be, of course, that the golden rules in fact don’t exist.]

Some evaluators have suggested that museums might move slightly away from summative evaluation towards purer audience research. Perhaps, for example, a group of museums could jointly align all the evaluations they commission to address a consistent set of research questions. Then, over time, comparable data would build up and we might gain some solid empirical evidence about how to better engage our audiences.

Author | Maurice Davies is a visiting senior research fellow in the Department of Management at King’s College London, a partner in the Museum Consultancy and Head of Policy and Communication at the Museums Association  | maurice.davies@ntlworld.com

Why are we not sharing

Why are we not sharing |

A blog by Anna Salaman, Head of Formal Learning, Royal Museums Greenwich (incorporating the National Maritime Museum, Royal Observatory, Greenwich, Queen’s House and Cutty Sark)

Evaluation is good… isn’t it?

Before exploring some of the reasons why we might not be sharing, or joining up our thinking around evaluation across the sector, it’s useful to think about why we are doing it, and whether or not it’s a good thing.

In order to give evaluation value, we must, of course, value audiences. In the last 20 years the museums sector has come a long way in its relationship with its audiences. Learning is now at the heart of everything we do; consultation and participation underpins our interpretation strategies, the object is no longer ‘king’ but a focus for conversation between the museum and its visitor, museums are no longer places to passively receive packages of knowledge and perceived wisdom but rich arenas for skills-building, nurturing understanding and inspiring creativity. They are welcoming, social spaces where everyone can happily enjoy a cup of coffee and a chat without the pressure of reading every label, aren’t they?

Sometimes they are – and there are many wonderful examples of this, but often they are not and, it is often coherently argued, neither should they be. As a fully paid up member of the fraternity[1] my vote is always audience-centred. But I cannot, hand-on-heart, say with full confidence that this is the view of all my colleagues across the many departments here, or across the sector as a whole. And, if we are still debating the place for learning, the audience and participation, then perhaps we are a long way from being able to adopt a united approach to evaluation.

Are we still learning how to evaluate?

Let’s assume we do, as a sector, value evaluation. There is, after all, much evidence to support this, with strong investment in evaluation strategies and consultative planning and development from many leading institutions. But it appears that expertise and confidence in these disciplines is still growing and has yet to be fully embedded as standard practice across the sector. Many of us are, all too often, going back to first principles and replicating practice every time we begin a new piece of work when the answers are often already there if we know where to look for them. I wonder…

How many institutions have a permanent evaluation team?

How many institutions prioritise audience and evaluation research over content and collections research?

Why national projects such as the Arts Council England funded Schools and Museums initiative require partners to undertake individual evaluation rather than evaluating the project as a whole?

Do we value practice-based expertise as highly as research-based expertise?

This is not an exercise in finger-pointing, simply a reflection that perhaps we are still learning how to evaluate.

 Sharing is good…isn’t it?

Why aren’t we sharing..?

So, if we are still learning how to evaluate as a sector, can we be expected to know how to share as a sector? Or to have the confidence to share something we don’t fully understand or embrace? Is it that we won’t share, or that we simply can’t? Or – most likely – a bit of both?

At this point, it is important to point out that there is actually some sharing going on. Strands of recent MA and GEM conferences, the Visitor Studies Group training and academic studies from leading centres in evaluation and Museology [such as Leicester and Warwick Universities], as well as individual organisations have all had a good go. In my own field of museum learning I have found colleagues from other institutions only too willing to share both their practice and their findings – a quick email or phone call has often yielded fantastic results and, in many cases, built new working relationships. However, a quick Google search has been less successful. Perhaps all this points not to a lack of willingness to share but, perhaps, to a lack of opportunity or structure for doing so in a coherent and sustainable way.

Here at the Royal Museums Greenwich [RMG is made up of: National Maritime Museum, The Queen’s House, Royal Observatory, and the Cutty Sark and has a 2 million-strong collection] we have gone on a bit of an evaluation journey [or ‘voyage’ – we are maritime museums after all] over the last few years. In fact, we are still on it, still learning, and long may the voyage last.

Even within my own organisation we have struggled to share: Did your evaluation ask the right questions for my project? Do I need to fully understand the project before I can fully understand the evaluation? But that’s a digital project and this is a whole gallery! Have I got time to look into this in depth with everything else going on?

The answer is, of course, a Royal Museums, Greenwich-wide evaluation strategy which all of these findings feed into and out of. To make sense of all our visitor research and map it in such a way that evaluation for the Maritime Art Collection and the Queen’s House yields useful information for the Royal Observatory and Astronomy; that evaluation outcomes for digital interactives in the National Maritime Museum are useful for Cutty Sark.  

And if that is indeed the answer, then can the wider museum sector effectively share evaluation without a nationwide strategy?

An interesting ‘sharing’ analogy came recently during the research and front-end evaluation [oh yes!!] phase of a forthcoming Children’s Gallery project here. In a workshop with Jo Graham [Learning Unlimited] we learned about the characteristics of different types of play and the evolution of ‘sharing’ in children. Very young children don’t share – they play in parallel, often the very same game with the very same toys, each brumming the same red car along parallel pieces of carpet. As pre-schoolers they become more social and friendships begin to form. By Key Stage 1 they are developing a strong sense of friendship and are often well-versed in the joys and advantages of cooperation and sharing [well, some of them…]

Perhaps, as a sector, we are in our infancy when it comes to evaluation. Perhaps as we grow into confident, well-adjusted, joined-up evaluators we will cease working in parallel – following the same methods and asking the same questions as each other – and begin to cooperate and collaborate.  

Or is it all far less interesting than all this – is it simply a question of resource? Which brings us back to ‘value’ again. Do we value evaluation highly enough to invest the necessary time, money and people in strategic, effective, long-term, joined-up evaluation? In this age of austerity for the cultural sector it is still, sadly, commonplace for learning to bear the brunt of the cuts. If this is an indication of where our priorities lie then perhaps that says something significant about how much, as a sector, we value evaluation and the voice and opinion of our audience.

On the bright side, however, key funding organisations value evaluation extremely highly, demanding ‘evidence’ of need for any funding application and ‘a fully comprehensive evaluative report’ at the end of a project – often withholding actual cash until they see it in black and white!

And there other questions about sharing evaluation:

Is it because evaluation is not always valued as ‘proper research’?

Is it because we still value the object over the audience, or our own expertise and knowledge over that of the audience?

Are we afraid that we will expose our failings and be found woefully lacking?

Should we be sharing beyond the sector and, if so, how far? How about our recent secondary evaluation? Should we share it with the Times Education Supplement? Teachers’ and Headteachers’ conferences? Ofsted?

Are we seen as valuable contributors to the wider [formal] learning debate?

I feel I could go on and on and on….

So how should we share?

I have recently been involved in two significant evaluation processes for the National Maritime Museum with Nicky Boyd. One to support the development of the Museum’s forthcoming ‘Nelson, Navy Nation’ gallery, and one to, once again, try to solve the sticky problem of why we don’t get more visits from secondary schools. Both evaluations yielded fascinating results, which I am keen to share. But shouldn’t sharing go beyond simply posting the reports on GEM or sticking them on our website? And are the results really that fascinating – or just fascinating to us and the particular issues we face? If evaluation has taught me anything, it’s to ‘ask the audience’ so, answers on a [digital] postcard please: would you be interested in the findings of these evaluations and how would you like them disseminated? And have you got any interesting stuff I could have a look at?!

Drop me a line at: asalaman@rmg.co.uk

Author | Anna Salaman, Head of Formal Learning, Royal Museums, Greenwich

---

[1] This is not an official organisation – simply a way of expressing that I am professionally dedicated to museums learning

The Late Shows

 Evaluation case study | The Late Shows

To share or not to share? When it comes to sharing evaluation for The Late Shows, Newcastle and Gateshead’s annual Museums at Night event, the short answer is yes, we do need to share evaluation. But how?

The team here at Tyne & Wear Archives & Museums has been running The Late Shows for six years now. In 2007 it started as a one-night event with 14 cultural venues taking part including Discovery Museum, Baltic Centre for Contemporary Art and the Laing Art Gallery. All the venues stayed open late and put on free events and exhibitions, which aimed to entice a new, younger audience through their doors [see image].

 The Late Shows 2012 took place over two nights on 18 and 19 May and we had a record 56 venues in the line-up and saw thousands of visitors brandishing their Late Shows glowsticks as they swarmed through the streets of Newcastle and Gateshead, visiting as many venues as they could [see the image of the Late Show CitySightSeeing Bus]. Venues which took part in the 2012 event included the Castle Keep, Mushroom Works artists’ studios and Newcastle’s Literary and Philosophical Society. We achieved 30,000 visits across The Late Shows venues in 2012!

With so many venues and artists taking part, and The Late Shows being a free, non-ticketed event, it has the potential to be a difficult or expensive project to evaluate but we have tried to keep it as simple as possible.

All the marketing and evaluation for The Late Shows is carried out in-house by the Tyne & Wear Archives & Museums communications team. Through our evaluation we aim to not only evaluate our marketing and track visitor satisfaction [see image of happy visitors] but also to see whether we have achieved various aims which include encouraging people to visit venues they have never been to before and inspiring them to return again throughout the year.

Every year we work with a fantastic team of Late Shows volunteers [see image] who act as ambassadors for the event, welcoming visitors to each venue, keeping track of visitor numbers and, of course, handing out those sought-after glowsticks. Another of their key tasks is to encourage people to fill in our incentivised evaluation survey which they can do online [using Survey Monkey] or by returning a freepost form after the event.

The findings from our evaluation are essential when it comes to shaping subsequent years’ events. It’s vital that the results are shared with all 50-plus participating venues so they can take the feedback into account when they programme their event for next year. We have seen everything from ghost tours and snake handling to mass drawing and performance art in The Late Shows programme over the years and each have played a role in making The Late Shows what it is – a showcase for the outstanding creativity and culture here in the North East. However, no venue wants to take the risk of programming something blind, with no knowledge of the audience and what they value.

The important thing about The Late Shows is that it’s a collaborative effort, with more than 50 different venues coming together to make one of the biggest events in our region’s cultural calendar. That can only work if we share what we know.

All venues are sent the headline survey results so they can see who came, which venues they went to and what they thought of them. Comments on specific venues or artists are shared individually, although many of the venues then choose to share these with each other. There is close, two-way communication between The Late Shows team and staff from all the venues so that experiences can be passed back and forth.

We also share specific feedback with some of our suppliers and partners – for example, feedback on the free Late Shows bus services, which transport visitors around the cities on either an open-topped bus or a mini-bus, is shared with the bus drivers so we can work together to tweak the route or timetable the next year.

We are also keen to share knowledge more widely. Every May, thousands of venues around the country – and the world – stay open late for Museums at Night. We are in conversation with Culture24, who promote the Museum at Night festival nationally, to share our experiences with other areas whom may wish to put on a partnership cluster event.

Although it’s so important to know the answers to those quantitative questions – did we meet our visitor target? Did we reach our target audience? - every year, I find that some of the most enlightening feedback comes in the form of the visitor comments we receive via social media and through Survey Monkey, which can really capture the spirit and atmosphere of The Late Shows.

We always include an open ended question so that people can give us any suggestions they might have for improving the event or let us know if there is something they especially enjoyed or didn’t like. One of my favourites from 2012 was: ‘It was an amazing event. I took my two children (both under six) and they had an amazing time learning to make a clay pot on a wheel, making masks, decorating tiles, learning circus skills and seeing magic. My son said "it was the best night of my life". Thank you!'

The Late Shows is a collaborative effort and with feedback like that, why would we keep it to ourselves?

The Late Shows 2013 takes place on 17 and 18 May 2013. www.thelateshows.org.uk

Author | Bill Griffiths, The Late Shows Project Manager bill.griffiths@twmuseums.org.uk

Tyne & Wear Archives and Museums

All images courtesy of Tyne & Wear Archives and Museums