Guide: Introduction to evaluation
- Section:
- Project management
Introduction
Most funders will ask you to tell them how you have measured the impact of your public engagement work – and it can sometimes feel like an unwelcome add-on. Evaluation is a valuable tool which enables you to learn from your experiences. Think of it as the research element of your activity. It is a process of collecting evidence and reflection that will help you with your next public engagement event.
Benefits can include:
- assistance with planning. Evaluation helps you focus on what you want to achieve, how you will achieve it and how you will know if you have been successful
- provides evidence. It proves the value and benefits of your activity and a record of your achievements (for you and your line manager/supervisor and funder)
- demonstrates value for money. Which is important when reporting to funders
- learning that can be shared with others and inform future activities
- an opportunity to reflect on your approach and improve it
Guidelines and approach
Planning your impact evaluation
It is likely that you will be conducting the evaluation yourself rather than paying for an external evaluator – although really big projects that have a budget would do well to bring in an expert.
Ideally plan your evaluation strategy at the beginning. Plan it as you would a research project. Evaluation can take place before, during and after an activity and can be used to consider the delivery process and the quality of the content. The aim is to assess whether you have met your objectives, what changed as a result of the activity, how it could have been more effective and if there were any unintended outcomes. It is important to understand not only what happened but also why something worked or didn't work.
There are various stages to an impact evaluation strategy:
1. Overarching Aims. What do you hope to achieve? eg Raise awareness of health research, recruit new scientists, obtain public experiences of an issue
2. Objectives. What do you need to do to achieve your aims? Make your objectives SMART: S(pecific) M(easurable) A(chievable) R(elevant) and T(ime limited) and think about what you can realistically achieve
3. Evaluation Questions. These are the questions you want to answer. They are similar to research questions and they should relate to evidence that can be collected
4. Methodology. You may want to assess your audience to create a baseline before your activity and then see if there has been any change after your event. You can undertake on-going evaluation (formative) to assess how successful your event is in engaging with your audience. This allows you to modify what you are doing. Finally, you can conduct summative evaluation at the end of the event to assess the success in achieving your outcomes and the overall impact
5. Data collection. There are two main types of data collection methods – quantitative and qualitative. Quantitative methods use facts and figures and multiple choice responses whereas qualitative methods ask what people thought. A mixed approach is recommended. There are various data collection techniques that you can use, for example observation, face to-face interviews, questionnaires (paper or on-line using programmes such as Survey Monkey), focus groups, and creative techniques such as graffiti walls, video and drama
6. Data analysis. You need to consider how much raw data you plan to gather and how you will analyse the results. It is advisable to use a spreadsheet and if possible number all your responses. Open qualitative responses can be coded into common themes to make analysis easier
7. Reporting. When writing your report think about the evidence you have collected, what it tells you and who is going to read your findings. Reflect on what have you learned from the experience. Write a clear and accessible report including lessons learned. Make judgements on what was successful and what was not and how you might do things differently next time
What is the difference between an output, an outcome and impact?
Outputs are the results of your activity (eg. event, website, science show, wall painting)
Outcomes are the overall benefits you aim to achieve (eg. deeper understanding of climate change, the skills to create a computer animation, shared values).
Impact is the overall effect or influence of the activity. It is the sum of the outputs and outcomes.
Other resources
RCUK have published a guide to evaluating projects. Published in 2005, Evaluation: Practical Guidelines has some really useful content for people wanting to evaluate their engagement activities. We are working with RCUK to update this guide for publication in Spring 2011.
We have also pulled together some pages of links to other relevant resources.
Examples of project evaluations
Evaluating science, arts and community projects
Evaluating Learning and Higher Education
In addition you can find out more about evaluation approaches in our How to support it section or read our Literature Review: Auditing, Evaluating and Benchmarking Public Engagement.
Authors
This guide has been adapted from the Engaging Researcher Booklet - written by Suzanne Spicer, from the Manchester Beacon and Sophie Duncan, from the NCCPE.
