Sharing evaluation | don't reinvent the wheel...

Sharing evaluation | don't reinvent the wheel...

Barriers

For over 12 years now I’ve worked within the museum and heritage sector in the field of evaluation and audience research and I’m reasonably confident the sector can now see the benefits of evaluation. However, I often feel frustrated at the barriers that we, as museum professionals, collectively seem to have raised, which often prevent us from sharing what we find out from the evaluation process.  We are still reticent about telling others what hasn’t worked, we still ask the same old questions, and we don’t look internally or externally as much as we could for useful data.  We just don’t seem to learn as well as we potentially could from the experiences of our colleagues. Recently a colleague suggested that perhaps it’s just part of the human condition that we need to keep asking the same questions of museum audiences.  Does this suggest we need reassurance or is it, fundamentally, about the fact that we believe our organisations are so unique we don’t feel we can apply what others have found out about their audiences to our own situation?  Does it feel easier to keep asking the same questions rather than spending time applying someone else’s research to our own organisation?   I’ve come to the conclusion that breaking down the barriers to sharing is much more than the question of whether we publish our findings. Many museums already do this really well, e.g. the Victoria and Albert, who commission a lot of research and put reports on line and the Natural History Museum, who share evaluations of exhibitions and learning programmes.  I think we need to do more. 

Understanding

We know that carrying out evaluation provides evidence about how projects and programmes are meeting aims and objectives and encourages on-going project improvements. We know it can act as an excellent tool for telling others about the great work being done in the sector, enabling us to share lessons learned with colleagues.  Evaluation can help the project team and participants feel like they ‘own’ the project, it can develop relationships with visitors and play a really important role in developing ideas and planning future projects. [see image]  We also know that if we don’t carry out any evaluation we could waste time and money, produce something that we can’t change, lose interest from our target audience, and waste an opportunity to learn something useful and lose funding.  There are a number of excellent online toolkits readily available which steer willing participants through the whole evaluation process e.g. the East of England Museum Hub’s Evaluation Toolkit for Museum Practitioners and the Smithsonian Centre for Education and Museum Studies has just set up an online community of practice around evaluation, which looks really exciting.

Facilitating a focus group at the Grant Museum of Zoology as part of the London Museums Hub Project ‘Say it again say it differently’ 2004-2006.

Fear

So we understand the benefits of evaluation and how to do it, but why are we not sharing what we find out through the evaluation process more effectively? [see image] One of the important factors may be that many of us are still reticent about telling others about what hasn’t worked so well.  I really understand the fear behind this.  For many reasons we want to portray projects, programmes and events in the best light, to secure future funding or to retain professional standing, but in its purest sense evaluation should be seen as an opportunity for us to learn and improve our professional practice. It should not be done to point fingers or assign blame. Rather it should help create a culture where we are able to take risks and share what hasn’t gone so well. Only then can we learn and move forward as a sector.

Re-packaging

I think a possible way forward is to re-package the data already held within organisations and think more strategically about how we plan evaluation. The organisations I regularly work with are often surprised with how much data they actually have once they start looking! Recently, working with the Education and Interpretation teams in a heritage organisation a colleague and I tried this as a practical exercise.  After some initial hesitation, followed by discussion and deliberation, the team realised they could pull out useful audience information from their existing evaluation data which they could put to efficient use in planning future projects and programmes.  Maybe the next step would be to work with two or three similar organisations to share audience consultation and evaluation data around specific audiences. Maybe a group of museums could agree some overarching research questions so all the evaluation and audience consultation work they do individually could collectively feed into answering these questions.

Facilitating a focus group at the Brunel Museum as part of the London Museums Hub Project ‘Say it again say it differently’ 2004-2006.

Let’s be confident about what we know

As a profession we need to be confident about what we already know.  I am often asked to find out the barriers secondary school teachers have in taking their students out to museums. I can safely say that I’ve asked a lot of teachers this question and I know what the answers usually are, so I encourage the organisations I work with to ask something which might be fundamentally more useful [see image]. A recent project with The National Maritime Museum [NMM] in Greenwich took this a step further when they asked secondary teachers what the NMM could do to help teachers advocate internally for a museum trip with their students. Should the NMM guarantee that the school could bring a whole year group at once or guarantee specialist input from a member of museum staff for the visit? Should they provide evidence of impact on attainment targets or information that highlights the benefits of learning outside the classroom? The Museum plans to share the results of their research with the sector.  Having the confidence that they know about the barriers teachers have to visit the museum with students has enabled the NMM to find out new information which will ultimately improve their school offer.

Evaluating with children at Bromley Museum as part of the London Museums Hub Project ‘Say it again say it differently’ 2004-2006.

Sharing documents is not enough

As a committee member for the London Museums Group there are certainly more conversations we can have as to how Share London, the London Museums Group online forum for sharing practice, can help us to share evaluation and audience consultation findings more widely.  I think we need to remember that posting our analysed and interpreted evaluation and audience research data might not be enough and more work and conversations need to be had about how we stop reinventing the wheel. Ultimately, time is always precious, and if we can stop asking the same questions, use the data we already have and share our findings more efficiently we [and our audiences] can all only benefit in the longer term.  

Author | Nicky Boyd, Museum Consultant [Audience Research & Evaluation] and LMG | nickyboyd@btinternet.com

Images | all photographs were taken by J. Neligan during focus groups as part of the London Museums Hub project Saying it differently 2004-06.  

For more information on the project please go to: http://www.museumoflondon.org.uk/Corporate/About-us/Regional-Programmes/Publications+and+Resources.htm

Nicky Boyd's Blog has been reprinted on the Guardian's Culture Professionals Network.  Sign up free to become a member of the Culture Professionals Network.

Why are we not sharing

Why are we not sharing |

A blog by Anna Salaman, Head of Formal Learning, Royal Museums Greenwich (incorporating the National Maritime Museum, Royal Observatory, Greenwich, Queen’s House and Cutty Sark)

Evaluation is good… isn’t it?

Before exploring some of the reasons why we might not be sharing, or joining up our thinking around evaluation across the sector, it’s useful to think about why we are doing it, and whether or not it’s a good thing.

In order to give evaluation value, we must, of course, value audiences. In the last 20 years the museums sector has come a long way in its relationship with its audiences. Learning is now at the heart of everything we do; consultation and participation underpins our interpretation strategies, the object is no longer ‘king’ but a focus for conversation between the museum and its visitor, museums are no longer places to passively receive packages of knowledge and perceived wisdom but rich arenas for skills-building, nurturing understanding and inspiring creativity. They are welcoming, social spaces where everyone can happily enjoy a cup of coffee and a chat without the pressure of reading every label, aren’t they?

Sometimes they are – and there are many wonderful examples of this, but often they are not and, it is often coherently argued, neither should they be. As a fully paid up member of the fraternity[1] my vote is always audience-centred. But I cannot, hand-on-heart, say with full confidence that this is the view of all my colleagues across the many departments here, or across the sector as a whole. And, if we are still debating the place for learning, the audience and participation, then perhaps we are a long way from being able to adopt a united approach to evaluation.

Are we still learning how to evaluate?

Let’s assume we do, as a sector, value evaluation. There is, after all, much evidence to support this, with strong investment in evaluation strategies and consultative planning and development from many leading institutions. But it appears that expertise and confidence in these disciplines is still growing and has yet to be fully embedded as standard practice across the sector. Many of us are, all too often, going back to first principles and replicating practice every time we begin a new piece of work when the answers are often already there if we know where to look for them. I wonder…

How many institutions have a permanent evaluation team?

How many institutions prioritise audience and evaluation research over content and collections research?

Why national projects such as the Arts Council England funded Schools and Museums initiative require partners to undertake individual evaluation rather than evaluating the project as a whole?

Do we value practice-based expertise as highly as research-based expertise?

This is not an exercise in finger-pointing, simply a reflection that perhaps we are still learning how to evaluate.

 Sharing is good…isn’t it?

Why aren’t we sharing..?

So, if we are still learning how to evaluate as a sector, can we be expected to know how to share as a sector? Or to have the confidence to share something we don’t fully understand or embrace? Is it that we won’t share, or that we simply can’t? Or – most likely – a bit of both?

At this point, it is important to point out that there is actually some sharing going on. Strands of recent MA and GEM conferences, the Visitor Studies Group training and academic studies from leading centres in evaluation and Museology [such as Leicester and Warwick Universities], as well as individual organisations have all had a good go. In my own field of museum learning I have found colleagues from other institutions only too willing to share both their practice and their findings – a quick email or phone call has often yielded fantastic results and, in many cases, built new working relationships. However, a quick Google search has been less successful. Perhaps all this points not to a lack of willingness to share but, perhaps, to a lack of opportunity or structure for doing so in a coherent and sustainable way.

Here at the Royal Museums Greenwich [RMG is made up of: National Maritime Museum, The Queen’s House, Royal Observatory, and the Cutty Sark and has a 2 million-strong collection] we have gone on a bit of an evaluation journey [or ‘voyage’ – we are maritime museums after all] over the last few years. In fact, we are still on it, still learning, and long may the voyage last.

Even within my own organisation we have struggled to share: Did your evaluation ask the right questions for my project? Do I need to fully understand the project before I can fully understand the evaluation? But that’s a digital project and this is a whole gallery! Have I got time to look into this in depth with everything else going on?

The answer is, of course, a Royal Museums, Greenwich-wide evaluation strategy which all of these findings feed into and out of. To make sense of all our visitor research and map it in such a way that evaluation for the Maritime Art Collection and the Queen’s House yields useful information for the Royal Observatory and Astronomy; that evaluation outcomes for digital interactives in the National Maritime Museum are useful for Cutty Sark.  

And if that is indeed the answer, then can the wider museum sector effectively share evaluation without a nationwide strategy?

An interesting ‘sharing’ analogy came recently during the research and front-end evaluation [oh yes!!] phase of a forthcoming Children’s Gallery project here. In a workshop with Jo Graham [Learning Unlimited] we learned about the characteristics of different types of play and the evolution of ‘sharing’ in children. Very young children don’t share – they play in parallel, often the very same game with the very same toys, each brumming the same red car along parallel pieces of carpet. As pre-schoolers they become more social and friendships begin to form. By Key Stage 1 they are developing a strong sense of friendship and are often well-versed in the joys and advantages of cooperation and sharing [well, some of them…]

Perhaps, as a sector, we are in our infancy when it comes to evaluation. Perhaps as we grow into confident, well-adjusted, joined-up evaluators we will cease working in parallel – following the same methods and asking the same questions as each other – and begin to cooperate and collaborate.  

Or is it all far less interesting than all this – is it simply a question of resource? Which brings us back to ‘value’ again. Do we value evaluation highly enough to invest the necessary time, money and people in strategic, effective, long-term, joined-up evaluation? In this age of austerity for the cultural sector it is still, sadly, commonplace for learning to bear the brunt of the cuts. If this is an indication of where our priorities lie then perhaps that says something significant about how much, as a sector, we value evaluation and the voice and opinion of our audience.

On the bright side, however, key funding organisations value evaluation extremely highly, demanding ‘evidence’ of need for any funding application and ‘a fully comprehensive evaluative report’ at the end of a project – often withholding actual cash until they see it in black and white!

And there other questions about sharing evaluation:

Is it because evaluation is not always valued as ‘proper research’?

Is it because we still value the object over the audience, or our own expertise and knowledge over that of the audience?

Are we afraid that we will expose our failings and be found woefully lacking?

Should we be sharing beyond the sector and, if so, how far? How about our recent secondary evaluation? Should we share it with the Times Education Supplement? Teachers’ and Headteachers’ conferences? Ofsted?

Are we seen as valuable contributors to the wider [formal] learning debate?

I feel I could go on and on and on….

So how should we share?

I have recently been involved in two significant evaluation processes for the National Maritime Museum with Nicky Boyd. One to support the development of the Museum’s forthcoming ‘Nelson, Navy Nation’ gallery, and one to, once again, try to solve the sticky problem of why we don’t get more visits from secondary schools. Both evaluations yielded fascinating results, which I am keen to share. But shouldn’t sharing go beyond simply posting the reports on GEM or sticking them on our website? And are the results really that fascinating – or just fascinating to us and the particular issues we face? If evaluation has taught me anything, it’s to ‘ask the audience’ so, answers on a [digital] postcard please: would you be interested in the findings of these evaluations and how would you like them disseminated? And have you got any interesting stuff I could have a look at?!

Drop me a line at: asalaman@rmg.co.uk

Author | Anna Salaman, Head of Formal Learning, Royal Museums, Greenwich

---

[1] This is not an official organisation – simply a way of expressing that I am professionally dedicated to museums learning