Most funders will ask you to tell them how you have evaluated your public engagement work – and it can sometimes feel like an unwelcome add-on. However, used correctly, evaluation is a valuable tool that enables you to learn from your experiences and to assess the impact of your project. Evaluation is a process of collecting evidence and reflection that will help you understand the dynamics and effect of your work, and help you with your next public engagement project.
Benefits can include:
- Assistance with planning. Evaluation helps you focus on what you want to achieve, how you will achieve it and how you will know if you have been successful
- Evidence of impact. It can help measure the value and benefits of your activity and provides a record of your achievements (for you and your line manager/supervisor, funders and potentially for the REF)
- Critical reflection. Provides a tool for critical reflection for you and the people you collaborate with, in order to help improve a project.
- Demonstrates whether your project is value for money which is important when reporting to funders
- Generates learning that can be shared with others and inform future activities
Planning your evaluation
It's best to develop your evaluation plan at the beginning of a project, as this will help you think through what you are trying to achieve and ensure you collect relevant data from your project as you go. Evaluation can be used to inform the development of your project and to assess whether you have met your objectives, what changed as a result of the activity, how it could have been more effective and if there were any unintended outcomes (both positive and negative). It is important to understand not only what happened but also why something worked or didn't work.
You may choose to evaluate in-house, or to buy-in some external expertise. This decision will be informed by project size and budget, the complexity of the project and relative ease with which data can be collected and analysed. In addition you need to consider whether you have the skills in house to evaluate your work. There are lots of people who can help you develop effective evaluation plans, so do get them invovled!
Developing an evaluation plan
Here is a list of things you need to think about when creating an evaluation plan.
- Overarching aim of activity
- What do you hope to achieve? e.g. Raise awareness of health research; recruit new scientists, gather public experiences of an issue of interest.
- What do you plan to do to achieve your aims? It helps if you make your collective objectives SMART: S(pecific) M(easurable) A(chievable) R(elevant) and T(ime limited) e.g. run 3 workshops for school groups on healthy eating by May 2015, involving 90 students in total.
- Evaluation Questions
- What do you want to know? These questions are similar to research questions, and should relate to the activity!
- How will you answer these questions? You may choose between on-going evaluation (formative) to continually assess and reflect on how a project is going. You may also want to consider outcomes and impact, in which case summative evaluation would be most appropriate.
- Broadly speaking there are two main types of evaluation data:
- Quantitative: providing facts and figures such as the number of participants who said they learnt something new.
- Qualitative: providing narrative and descriptive data, which can be useful for capturing perspectives and providing texture to your numbers.
Methodology considerations include: whether you are able to create a baseline, to measure change against; to what extent you will gather qualitative (numbers) data or qualitative data; time and resources you have for data analysis; who can participate in your evaluation (for example you may want to gather feedback from participants, your team, and observers to triangulate your findings.)
- Data collection
- There are a number of different ways in which you can collect data for your evaluation. When planning data collection you need to consider a range of factors:
- - Participants in your project
- - Location
- - Ethics
- - Time available to gather data
- There are different data collection techniques including: focus groups, questionnaires, case studies, interviews, post cards, video diaries etc., and each have strengths and weaknesses.
- Data analysis
- You need to consider how much raw data you plan to gather and how you will analyse the results. Again there are a range of options to do this, including software packages that can help you analyse your data. Open qualitative responses can be coded into common themes to make analysis easier. To do this you will need to make use of a coding framework. One great example is the Generic Learning Outcomes that was developed by the Museums and Libraries Association.
- Who is interested in your evaluation? You need to consider this when developing your evaluation plan. First and foremostly it is important that you and your team learn from the evaluation and reflect this in future projects.
- When writing your report think about the evidence you have collected, what it tells you and what's relevant to your potential readers.