Why evaluate?
Evaluation is a process of collecting evidence and reflection that will help you understand the dynamics and effect of your work and help inform future projects or approaches. Used correctly, evaluation is a valuable tool that enables you to learn from your experiences and to assess the impact of your work.
Benefits can include:
- Assistance with planning. Evaluation helps you focus on what you want to achieve, how you will achieve it and how you will know if you have been successful.
- Evidence of impact. It can help measure the value and benefits of your activity and provides a record of your achievements (for you and your line manager/supervisor, funders and potentially for the REF).
- Critical reflection. Evaluation provides a tool for critical reflection for you and the people you collaborate with, and helps improve a project.
- Accountability. Evaluating can help to demonstrate whether your project is value for money which is important when reporting to funders.
- Learning. Evaluation generates learning that can be shared with others and inform future activities.
Planning your evaluation
It's best to develop your evaluation plan at the beginning of a project, as this will help you think through what you are trying to achieve and ensure you collect relevant data from your project as you go. Evaluation can be used to inform the development of your project and to assess whether you have met your objectives, what changed as a result of the activity, how it could have been more effective and if there were any unintended outcomes (both positive and negative). It is important to understand not only what happened but also why something worked or didn't work.
You may choose to evaluate in-house or to buy-in some external expertise. This decision will be informed by project size and budget, the complexity of the project and relative ease with which data can be collected and analysed. In addition you need to consider whether you have the skills in-house to evaluate your work. There are lots of people who can help you develop effective evaluation plans, so do get them involved. An external perspective can be really valuable.
Developing an evaluation plan
Here is a list of things you need to think about when creating an evaluation plan.
Overarching aim of activity
What do you hope to achieve? E.g. Raise awareness of health research; inspire school students to become researchers, gather public experiences of an issue of interest.
Objectives
What do you plan to do to achieve your aims? It helps if you make your collective objectives SMART: S(pecific) M(easurable) A(chievable) R(elevant) and T(ime limited) e.g. run 3 workshops for school groups on healthy eating by May 2019, involving 90 students in total.
Evaluation questions
What do you want to know? These questions are similar to research questions, and should relate to the activity, e.g., have the students learnt how to make healthy food choices.
Methodology
How will you answer these questions? It is important to remember that you don’t have to wait until the delivery of the project to evaluate. Formative evaluation can be used to develop your approach, or to continually assess and reflect on how a project is going. Summative evaluation enables you to consider the outcomes and impact.
Broadly speaking there are two main types of evaluation data you might collect:
- Quantitative: providing facts and figures such as the number of participants who said they learnt something new.
- Qualitative: providing narrative and descriptive data, which can be useful for capturing perspectives and providing texture to your numbers.
Methodology considerations include: whether you are able to create a baseline, to measure change against; to what extent you will gather qualitative or qualitative data; time and resources you have for data analysis; who can participate in your evaluation (for example you may want to gather feedback from participants, your team, and independent observers to triangulate your findings).
Data collection
There are a number of different ways in which you can collect data for your evaluation. When planning data collection you need to consider a range of factors:
- Participants in your project
- Location
- Ethics
- Time available to gather data
There are different data collection techniques including: focus groups, questionnaires, case studies, interviews, post cards, video diaries, etc. Each have strengths and weaknesses.
Data analysis
You need to consider how much raw data you plan to gather and how you will analyse the results. Again there are a range of options to do this, including software packages that can help you analyse your data. Open qualitative responses can be coded into common themes to make analysis easier. To do this you will need to make use of a coding framework. One useful example is the Generic Learning Outcomes framework that was developed by the Museums and Libraries Association.
Reporting
Who is interested in your evaluation? You need to consider this when developing your evaluation plan. First and foremost, it is important that you and your team learn from the evaluation and reflect this in future projects.
When writing your report think about the evidence you have collected, what it tells you and what's relevant to your potential readers.
Useful resources
We have created a guide to evaluating your public engagement activities, which includes a worked example of an evaluation plan
- STFC’s framework for evaluating public engagement
How the NCCPE can help
The NCCPE runs various evaluation courses, and we can offer bespoke consultancy and training. Do get in touch if you could like advice or guidance about what we can offer.