Evaluation Discussion Summary

The evaluators then joined tables, and each table was assigned a question that had been posed by one of the delegates. Responses to these questions were fed back at the end of the day:

Summary of Key Insights and Recommendations from each of these Discussions

Question: What new concepts, methods and philosophy of evaluation are emerging through the Beacons?

Insight: We have to do something different in evaluating this project.

Recommendation: We are measuring the value of partnership working and culture change and how trust is built up. It’s the approach to this evaluation that is important not the “landscape”.

Question: Next time someone confidently state that we can’t reward or incentivise engagement because “there’s no way of assessing engagement success”, what should I tell them?

Insight: Different forms of engagement need to be considered on their own. No one size fits all. Need to understand what success looks like.

Recommendation: Need to get away from audit culture. Need to engage with evaluators. A lot of evidence comes from self-reflection and conversation, rather than hard facts. Balance quantitative and qualitative.

Question: What balance should there be between quantitative and qualitative methods to evaluate the whole Beacon impact?

Insight: Who is the audience? What is the aim of the research? Necessary for transparency ie. size of dataset, etc.

Recommendation: Recognise that the two are intrinsically linked. The weighting between the 2 variables depends on the aims/audiences but need for transparency however you approach it.

Question: How can we capture change as it happens? “Live” so to speak as well as through reflection?

Insight: Evaluation is part of change process

Recommendation: Tailor anecdotes to stakeholders.

Question: How do we capture the effects and benefits of partnership building on culture change?

Insight: This is the same issue for evaluating culture change. Evaluate what’s happening in the partnership.

Recommendation: Identify the partnership and how it has helped deliver specific objectives/aims of the programme.

Question: What’s the purpose of evaluation? Will it change policy and make a difference in the sector and externally?

Insight: We need to add another level to capture collaborative learning and to plan. Make it a dynamic process to involve policymakers so they learn with us. Make the learning useful to others.

Recommendation: We need to take action. Danger Oakleigh evaluation doesn’t pick up on learning. In autumn 2011 recruit peer review evaluation to pull out learning.