The Late Shows

 Evaluation case study | The Late Shows

To share or not to share? When it comes to sharing evaluation for The Late Shows, Newcastle and Gateshead’s annual Museums at Night event, the short answer is yes, we do need to share evaluation. But how?

The team here at Tyne & Wear Archives & Museums has been running The Late Shows for six years now. In 2007 it started as a one-night event with 14 cultural venues taking part including Discovery Museum, Baltic Centre for Contemporary Art and the Laing Art Gallery. All the venues stayed open late and put on free events and exhibitions, which aimed to entice a new, younger audience through their doors [see image].

 The Late Shows 2012 took place over two nights on 18 and 19 May and we had a record 56 venues in the line-up and saw thousands of visitors brandishing their Late Shows glowsticks as they swarmed through the streets of Newcastle and Gateshead, visiting as many venues as they could [see the image of the Late Show CitySightSeeing Bus]. Venues which took part in the 2012 event included the Castle Keep, Mushroom Works artists’ studios and Newcastle’s Literary and Philosophical Society. We achieved 30,000 visits across The Late Shows venues in 2012!

With so many venues and artists taking part, and The Late Shows being a free, non-ticketed event, it has the potential to be a difficult or expensive project to evaluate but we have tried to keep it as simple as possible.

All the marketing and evaluation for The Late Shows is carried out in-house by the Tyne & Wear Archives & Museums communications team. Through our evaluation we aim to not only evaluate our marketing and track visitor satisfaction [see image of happy visitors] but also to see whether we have achieved various aims which include encouraging people to visit venues they have never been to before and inspiring them to return again throughout the year.

Every year we work with a fantastic team of Late Shows volunteers [see image] who act as ambassadors for the event, welcoming visitors to each venue, keeping track of visitor numbers and, of course, handing out those sought-after glowsticks. Another of their key tasks is to encourage people to fill in our incentivised evaluation survey which they can do online [using Survey Monkey] or by returning a freepost form after the event.

The findings from our evaluation are essential when it comes to shaping subsequent years’ events. It’s vital that the results are shared with all 50-plus participating venues so they can take the feedback into account when they programme their event for next year. We have seen everything from ghost tours and snake handling to mass drawing and performance art in The Late Shows programme over the years and each have played a role in making The Late Shows what it is – a showcase for the outstanding creativity and culture here in the North East. However, no venue wants to take the risk of programming something blind, with no knowledge of the audience and what they value.

The important thing about The Late Shows is that it’s a collaborative effort, with more than 50 different venues coming together to make one of the biggest events in our region’s cultural calendar. That can only work if we share what we know.

All venues are sent the headline survey results so they can see who came, which venues they went to and what they thought of them. Comments on specific venues or artists are shared individually, although many of the venues then choose to share these with each other. There is close, two-way communication between The Late Shows team and staff from all the venues so that experiences can be passed back and forth.

We also share specific feedback with some of our suppliers and partners – for example, feedback on the free Late Shows bus services, which transport visitors around the cities on either an open-topped bus or a mini-bus, is shared with the bus drivers so we can work together to tweak the route or timetable the next year.

We are also keen to share knowledge more widely. Every May, thousands of venues around the country – and the world – stay open late for Museums at Night. We are in conversation with Culture24, who promote the Museum at Night festival nationally, to share our experiences with other areas whom may wish to put on a partnership cluster event.

Although it’s so important to know the answers to those quantitative questions – did we meet our visitor target? Did we reach our target audience? - every year, I find that some of the most enlightening feedback comes in the form of the visitor comments we receive via social media and through Survey Monkey, which can really capture the spirit and atmosphere of The Late Shows.

We always include an open ended question so that people can give us any suggestions they might have for improving the event or let us know if there is something they especially enjoyed or didn’t like. One of my favourites from 2012 was: ‘It was an amazing event. I took my two children (both under six) and they had an amazing time learning to make a clay pot on a wheel, making masks, decorating tiles, learning circus skills and seeing magic. My son said "it was the best night of my life". Thank you!'

The Late Shows is a collaborative effort and with feedback like that, why would we keep it to ourselves?

The Late Shows 2013 takes place on 17 and 18 May 2013. www.thelateshows.org.uk

Author | Bill Griffiths, The Late Shows Project Manager bill.griffiths@twmuseums.org.uk

Tyne & Wear Archives and Museums

All images courtesy of Tyne & Wear Archives and Museums

Leaping over crocodiles

Leaping over crocodiles

Evaluation, can it work? … I think it can.  I have worked for several years as a museum evaluator for Renaissance East of England, Norfolk Museums and Archaeology Service and as a freelancer. I started with a distinctly academic approach, having studied an MSc in Research Methods and was often surprised at what passed as hard evidence in the museum sector– even in studies commissioned from professional researchers. Research or evaluation reports can make poor use of existing theory and literature, may contain only minimal explanations about approaches to sampling, data collection and data analysis and, when making claims from numerical data, do not always use of appropriate statistical methods.

There is, therefore, a case for improved skills around evaluation in museums, especially in critically evaluating and reviewing research. This way we can be more demanding when commissioning research and more proactive in using what evidence is already out there.  A further issue with evaluation in the sector is that it often leans towards advocacy and has an emphasis on demonstrating success rather than identifying what can be built on and improved. The reasons for this are understandable as museums are not a statutory service and are often focussed on making a case for their own worth.  The issue, however, is not just the quality of the evaluation, but about how it is able to influence practice.

Does evaluation make a difference? This is a thorny issue and, in the past, I have felt that the gap between a formal research report and its implementation might as well be a leap over a crevasse filled with hungry crocodiles.

Nile Crocodile (Crocodylus niloticus), Norwich Castle Natural History Collection

I have begun to realise that the word ‘evaluation’ has different meanings and that there are different levels of evaluation, from formal research studies to what might better be termed ‘reflective practice’.  When looking for evidence that evaluation makes a difference, the focus is often on the big decisions and on matters of policy – but what about the smaller things that affect our everyday service delivery?

How evaluation is implemented will depend on its ‘level’. When an individual is involved in evaluating their own work, or has a distinct connection to it, they often have the power to make the changes they deem necessary first-hand. On the other hand, where the evaluation concerns structures, policy or a number of stakeholders, implementing change is more complicated and the level of ‘proof’ needed to persuade an organisation to change is higher. Therefore, when planning evaluation, we should start with the organisation’s structure and ask ourselves in each case ‘How will the decisions be made here?’ and ‘What is the level of proof needed to instigate change?’

When it comes to getting research findings implemented, I have had a lot to learn in my own approach. By and large, staff are too overloaded to be interested in the nuances, caveats and methods of an evaluation - preferring concrete recommendations or clearly defined concepts. Working in this way can take some nerve, because it involves sticking your neck out and making judgements, and as an evaluator you are all too aware of the limitations of your data. It is also important to find a number of ways of getting the evaluation findings across, and nothing’s better than getting out there and talking to people, whether informally or formally. . Doing evaluation is only one piece of the puzzle when it comes to bringing about change. Evaluation needs ‘embedding’ within systems – this might mean involving the evaluator in a project team, ensuring there are built-in cycles of service review, or having a clearly laid-out process for implementing evaluation findings.

So where is my evidence that evaluation can work?  When writing this blog I asked around our organisation for examples of how evaluation had been used – here is my evidence.

School children enjoying a trip to Time and Tide Museum in Great Yarmouth

‘I always try to speak to the lead teacher during an event so I can gauge how an event is working and will actively ask for an honest (not just pleasing) evaluation. Often, I find that with written evaluations people are reluctant to criticise or reflect on what could be improved. My fear is that if the event didn't work, the school might not tell us and simply not book again. We also always hand out evaluation forms, most of which are returned completed. In either case, when we receive comments which indicate something is amiss, we will contact the teacher direct and either explain why x,y or z happened, or ask for ideas to improve whatever didn't work. This follow -up is essential because the teachers feel that their comments are valued. ,They become part of the planning process and are more likely to book with us again.   Over the years, our evaluation and assessment procedures have enabled me to build up good working relationships with local teachers and our events have benefited from this process.’ PD: Time and Tide Museum

‘When I first started running events in the Norwich Castle Study Centre, I had a very small budget for marketing.  I imagine this is how a lot of smaller museums find themselves.  I did a basic feedback form for the first ten or so events, asking people how they found out about the event as well as how much they enjoyed it etc.  The information about how much they enjoyed it was pretty much useless in a way, because I just had 60 people telling me they enjoyed it!  Which is, of course, great, but it didn't help me change anything.  The evaluation was far more useful for targeted marketing as we were quickly able to see what worked  (whether people found out about the event from the local paper or e-postcards).  In the end I saved a lot of money on flyers and it meant that when we were arguing for support from marketing, we were able to advocate those services.  So I suppose the moral of that story is to only ask questions where you care what the answer is, and that you can use the results from….I think the most powerful and immediate evaluation is probably verbal.  It is so informal, that we often overlook it.  I have realised this from working with the public on a daily basis.  Every time someone asks a question about what we do, or why something is the way it is, or enquire about information, you sub-consciously make a note to ensure that information is available in future - whether you include it in a tour, change a bit of text, or raise it yourself as a question.  So I suppose that comment is about having an attitude which seeks feedback and investigates it.’

RB: Norwich Castle Study Centre

‘A great example of how audience data has changed a museum in Great Yarmouth is when we did some mapping of how visitors moved through galleries before we did any redisplay work. I compiled a floor plan and gave this to the Front of House Team. I then asked them to map ten visitor movements per day for three weeks. Using the results of this data (as well as written feedback) it seemed that a high percentage of visitors used part of one gallery as a corridor, walking straight through rather than looking in some half-bays to one side.  When we were designing the new gallery plans, this data gave me the ability to persuade staff  that we should break up the long corridor space by adding an extra wall and enhance the interpretation – this made the new Romans gallery more of an individual statement space rather than just part of a bigger room.’ JO: Time and Tide Museum

‘The repeated requests for a guide book in the visitor book enabled us to make a good case for this - and we now have a lovely inexpensive souvenir guide. I regularly look at the 'What could we improve section' of the visitor book to check we're not missing anything or to pick out and rectify anything that has obviously gone 'pear-shaped.’ CT: Strangers Hall

‘Evaluation is very important to the way I work as an individual, and I encourage my team to do likewise.  I rely on feedback forms primarily, but also other sources like ‘Trip advisor’.  I use evaluation to find out if there is a problem, and if so I'll sort it out right away. As a manager I always read all feedback as it helps me keep up to date with what's going on.’

CM: Norwich Castle

I have come to the conclusion that, for evaluation to work, you need both the will and the way:

• A genuine desire to use evidence and a culture that is open to change
• Systems for embedding the use of evidence within planning and decision-making cycles. 

Evaluation is deeply linked to organisational change and should be an ingredient in the cake, rather than the icing.. So, happy cake making, and good luck leaping over those crocodiles!'

Author | Amanda Burke, Evaluation Officer | Amanda.burke@norfolk.gov.uk / 01603 493657

Norfolk Museums and Archaeology Service. 

Image courtesy of Norfolk Museums and Archaeology Service

Participation begins within the Museum

Participation begins within the Museum

The Museum redevelopment plans being discussed.  Image courtesy of National Museum of Wales 

At St Fagans: National History Museum we have recently been given the good news that we have funding for a major redevelopment of the Museum. The overall goal is to transform St Fagans into a world-class national museum of history that is driven by the needs of its users [see image above] . With funding from the Heritage Lottery Grant and Welsh Government, the work of constructing a new gallery and refurbishing the existing galleries will start next year. Both funders emphasise the importance of participation and listening to our users. The question is: How do we reconcile this with the usual pattern of front-end, formative and summative evaluation?  If we are to place users’ needs at the heart of the museum’s work, surely we must develop more adaptive and participative ways of carrying out evaluation.

Sharing with staff

Most museums are practised at producing visitor studies, reports and gathering data about visitors and non-visitors. Very rarely, however, are their findings shared and discussed across the whole range of staff, let alone made relevant to the work of individuals. We believe that participation begins ‘within’ the museum and at St Fagans, we have tried to do this by involving over 70 members of staff in the evaluation and consultation processes of preparing our plans for redevelopment. Staff trained as scribes and facilitators for focus group discussions, and conducted surveys to track and map current visitor behaviour.  This may not be as scientifically rigorous as work undertaken by professional evaluators, but it has exposed a wide range of staff to first-hand interaction with visitors and audiences we find hard to reach. This in turn has led to informed internal discussion about how we need to move forward. The findings were perceived to be less unfairly critical than reports produced by outsiders, yet harder to dismiss.

Sharing with other museums

Research reports published online by museums such as Glasgow Museum and the Victoria and Albert Museum are a useful source of information for practice that has worked well elsewhere. The majority, however, are summative evaluations published long after the project is live, and driven by the needs of funding bodies. Their purpose is mainly to persuade funders of the success of projects and not necessarily to impact on present and future practice. How is it that most museums, despite many years of evaluating their work, are still producing fundamentally similar gallery displays and attract the same types of visitors? Are we evaluating the wrong things?

One of the issues is that there is little consistency of approach across the sector. The methodologies used and the knowledge gained often resides with external audience research companies and consultants. Comparing data can therefore be challenging. It would be a real step forward if museums could find a way of working together to find a common approach, share methods and standardise questions so that research findings can be shared and inform the whole sector. Could a central place be provided for all museums to store and post recent research that would avoid trawling through numerous websites, blogs and publications?

Sharing with community partners

Our ambition at St Fagans is to be a participatory museum on a national scale, providing opportunities for all the diverse peoples of Wales to participate in creating and sharing content. Our inspiration is the work of Nina Simon (www.participatorymuseum.org) who defines such museums as a ‘multidirectional content experience, a platform that connects different users who act as content creators, distributors, consumers, critics and collaborators.’ There is very little evaluation of participatory projects in the museum world. If museum objectives are changed by the participation of users, then the process of evaluation needs to be ongoing and adaptive.

There are potential partners in other organisations who already have expertise in this area. Through our involvement in the Paul Hamlyn initiative, Our Museum: Communities and Museums as Active Partners, we are already learning how to involve others in our decision making – to share authority and ways of working with voluntary and other third sector organisations who have much deeper experience of working with those we traditionally find hard to reach. The lessons learned from Our Museum will inform work with our other participatory forums, representing all our target audiences.

These will help us to create a programme of evaluation, considering how we evaluate the success and impact of the process as well as the product. We need to create measuring techniques that are useful to both museum and community partners, which can be implemented by in-house staff, volunteers and community partners. We are in this together, hopefully for the long term.

This is not a strategy for the risk averse.  We learn as much from failure as success. Both should be shared honestly, as they happen, with our colleagues in other museums if we are to create truly culturally democratic museums and reduce inequalities in participation in the cultural activities we provide.  We are at the start of the journey and you can follow our progress at http://www.museumwales.ac.uk.

Author |  Nia Williams, Learning Manager | Nia.Williams@museumwales.ac.uk

St Fagans: National History Museum

Amgueddfa Cymru-Museum Wales