Evaluating Culture Change

There are seven different evaluators working across the project – one for each beacon and one for the NCCPE. These evaluators spent the morning reflecting on the lessons they were learning about the issues, challenges and opportunities being faced in each setting. These were fed back immediately after lunch.

Summary of feedback

• Different approaches are being followed across the project but there are common elements and shared successes

• The project is really starting to happen this year – real momentum which is reflected across all beacons. It takes time for this activity to get going

• Need understanding of how to meet funder expectations when thinking about future programmes – this isn’t always the same as what projects understand as their objectives; need to clarify early on

• Be clear about what the objectives are - as it guides the projects and helps tracking if it has been effective

• Behavioural change is messy and not easy to measure. Often people have light bulb moments and do things differently afterwards. Capturing that is difficult. What process have they gone through to make those changes? Beacons are often a trigger – catalyst

• Partnerships are important. Beginning to show that trust is building and relationships are deeper, but this takes time – evaluating this process is important but not necessarily built into any evaluation framework

• Complications – lots of stuff going on and different activities impacting on this agenda. Impossible to say ‘this is’ directly the result of beacon intervention – lots of change is part of the wider context. Need to be careful in over-surveying; people can get surveyed out

• Balance of quantitative and qualitative – do we need to count stuff? The qualitative analysis – the learning, the journeys, the anecdotes are important. Capturing process of change is also important. Evaluating process, and outcomes, is not the norm in academia. What effect/impact does something have, not just what you’ve delivered (outputs)? Working with people to think about wider impact.

• Embedding evaluation and monitoring – we need to do it with you – not to you. Evaluators are your critical friend. Think of it at the beginning and at the end. Role of evaluators is to help capture the wider impact.

• Audiences – Beacon evaluators feel their focus is different to Oakleigh’s (the latter more related to funder expectations?).

Podcast of this feedback on Slide Share (external site).