The Beacons for Public Engagement (BPE) were established as pilots to trial and test new approaches to public engagement within higher education institutions (HEI) and research centres. Evaluation and learning plays a role in the BPE programme. This Mini Story of Change outlines the lines of inquiry to evaluate UCL’s programme as part of the Beacons for Public Engagement (BPE) initiative.
The approach to evaluation
In July 2009 an Evaluation Officer was appointed to the UCL-led BPE programme, working within the PEU. The Evaluation Officer is responsible for assessing the success of UCL’s programme as part of the BPE scheme, specifically examining the PEU’s overall operation and its performance in achieving the aims and objectives of the BPE programme. In practice this has involved:
- Devising an effective way to evaluate UCL’s public engagement work (i.e. an Evaluation Framework)
- Evaluating past, present, and ongoing UCL public engagement activities (at programme and project level)
- Learning from and sharing knowledge on evaluation practices and experiences with practitioners, academics, researchers, fellow Beacon evaluators and partners
- Creating and applying monitoring, evaluation and reporting systems. Providing feedback on activities to ensure continuous improvement of practice
- Reviewing the performance of the PEU in disseminating the cumulative and integrative value of public engagement within UCL
- Evaluating the validity and potential utility of guidance, documentation and tools developed, and training and mentoring programmes undertaken
- Developing and trialling evaluation tools and techniques
The evaluation framework
Due to the nature of the BPE programme, a responsive, qualitative evaluation approach has been adopted. An evaluation framework has been developed for the evaluation of UCL’s programme as part of the BPE initiative. The framework has been developed from a process of consultation, collaboration and contemplation drawing upon a range of literature, research studies and the experiences of the UCL PEU, other Beacons, public engagement academics and practitioners. Information from a number of public engagement academics and practitioners from a range of UCL departments (e.g. Geography, Science and Technology Studies, Bartlett School of Graduate Studies, Archaeology, Engineering) and organisations (e.g. Oxford Brookes University, London 21, NCCPE, CUE East) has fed feed into the development the evaluation framework.
The UCL-led BPE Evaluation Framework is downloadable here.
- Through the adoption of an internal evaluator, working within the UCL PEU, there is a close relationship between the programme and the evaluation undertaken. Monitoring and evaluation is embedded in the delivery of the UCL-led BPE programme. The collection of information at different stages and levels is not only useful to document the achievements of the UCL-led BPE but to reflect upon experiences and developments. Within the UCL evaluation and monitoring approach there is an emphasis on understanding and learning, which is consistent with the ethos of the UCL-led BPE.
- The evaluation of the UCL-led BPE is based on the strategic aims of the BPE programme - the five Beacon aims provide clear conceptual boundaries for the evaluation.
- The UCL-led BPE evaluation uses a quantitative and qualitative approach that allows for plurality and diversity. The mixed methodology adopted enabled the collection of material from a range of sources to offer an interpretation of the different strategies evident in the UCL-led BPE from a range of perspectives. These are used to build up a picture of the UCL-led BPE.
- Understanding impacts of public engagement is complex as they are often multi-layered and hard to assess. The framework developed potential indicators to provide proxies for impact; these indicators are evolving.
- It is recognised that evaluation within the HE sector is a complex issue; with various approaches to evaluation public engagement being tried and tested by each Beacon. This poses an interesting challenge for the BPE programme.
- A number of evaluation tools and techniques have been developed drawing upon a range of sources, including the experience of UCL staff and students, the UCL Public Engagement Unit and its programme partners, and literature on public. However, in reality people often need more than tools, they need ongoing support and guidance to undertake evaluation.
Gemma Moore, Public Engagement Evaluation Officer
0207 679 4112