Supporting the development of an integrated, regional approach to programme evaluation in East Asia
At a glance
Operating in 11 countries in East Asia, the British Council’s Higher Education Partnerships (HEP) programme facilitates a range of collaborative activity between the UK and these 11 countries to support internationalisation of their higher education systems. Monitoring and evaluation (M&E) of British Council activities has often by hindered by the extreme differences in internationalisation and local systems across the region, as well as limited internal capacity to support robust M&E activity. IFF Research has worked collaboratively with the East Asia regional team to develop and implement a regional M&E programme that enables robust, consistent data collection across the region, with the ability to adapt to the unique needs of each country and its programmes. This programme has been paired with capacity building components to improve M&E understanding and skills within the organisation and support internal M&E ownership longer-term.
About the client
The British Council is the United Kingdom’s international organisation for cultural relations and educational opportunities. The organisation offers English language training and funded programmes in arts, culture and education to build connections, understanding and trust between people in the UK and other countries. Their programmes work directly with individuals and institutions, and with governments and policymakers to deliver locally relevant programmes and change. In 2019, the British Council worked in more than 100 countries globally, connecting 80 million people directly and millions more through online, broadcasts and publications.
Challenges and objectives
Internationalisation of higher education (HE) systems is at the heart of the British Council’s work globally. British Council East Asia operates regional HEP programmes across 11 countries to support regional and national internationalisation of HE systems. A diverse region, these 11 countries range from those early in the internationalisation journey (such as Myanmar or Vietnam) through to those very internationalised (such as Singapore and China), each requiring a different set of policy and/or collaborative research or teaching activities to support the needs of their country. Across the region, however, all countries are broadly working towards the same three objectives:
- To help the HE sector in each country achieve a greater degree of internationalisation, and to drive up standards across the HE sector
- To draw on and promote British HE sector expertise to support the internationalising agendas of countries in East Asia, and to increase opportunities for UK HE institutions in the region across a range of areas, as these agendas develop.
- To forge links between specific institutions in the UK and East Asia for the purposes of developing shared research programs and greater mobility between institutions.
These objectives are delivered through the Higher Education Partnerships (HEPs) with UK HE institutions, with British Council helping institutions build connections with each other and supporting partnerships through funding and expertise.
However, limited M&E activity and infrastructure paired with such incredible diversity of programmes and associated evaluation needs has made measuring the impact of programmes in a consistent fashion challenging. As a result, this has limited the organisation’s ability to demonstrate its programmes’ value and effect at both the national and regional levels.
“Since 2019, IFF Research have been the British Council’s evaluation partner on two projects in our Education portfolio in East Asia: Generation UK (an international mobility programme for UK graduates), and Higher Education Institutional Partnerships. IFF’s work with us over the past year has been transformational in terms of delivering high quality evidence of programme outcomes.”
Gary,
Regional M&E Advisor, East Asia, British Council
Solution
Since June 2019, IFF Research and British Council colleagues in East Asia have been working collaboratively to develop and implement a new process and tool for HEP outcome monitoring in the region. This included:
- Re-design of the regional Theory of Change (TOC) and development of an associated evaluation framework;
- Regional workshop to co-develop and agree the revised TOC and evaluation framework with the regional teams, provide introductory training to support capacity building, and introduce the programme’s aims and approach;
- Establishment of a standardised questionnaire and survey order form comprised of core and project-specific metrics to be used for programme monitoring, adaptable to each project’s specific objectives;
- Measurement of baseline, interim and (upcoming) final colleague capacity metrics to assess the skills, confidence and ongoing training needs of British Council colleagues over time;
- Identification and implementation of survey design, hosting and monitoring software set up by IFF then handed to the British Council team (with training) for ongoing internal monitoring;
- Qualitative interviews with British Council colleagues and programme participants to assess performance and get first-hand accounts of project impact; and
- Delivery of interim and final evaluation report, providing the first national and regional evaluation of the HEP’s performance.
Outcomes are being monitored for ongoing internal use, with a full, formal regional evaluation due in Summer 2021. Over time, ownership of monitoring activities are being transitioned to the British Council with appropriate training, to enable ongoing, robust M&E following the project’s completion.
Impact
In its first 15 months, the regional programme has been designed and implemented, with seven of the 11 countries having already adopted the revised approach. Following an initial trial phase, the final metrics, survey ordering process and survey software was finalised and implemented across the region, resulting in 172 surveys completed across 11 programmes in the seven countries between February and July 2020. Results of these first surveys were combined and analysed in an interim report, providing a first view of national and regional performance across the key metrics and outcomes and representing the first aligned, consolidated view of regional performance – from an overall, UK and local perspective.
Interim data has shown an overwhelmingly positive picture at the regional level:
- Nine in 10 agree their partnership was mutually beneficial.
- Nearly eight in 10 felt the partnership helped them realise their institutional strategy.
- Seven in ten felt their project’s intended outcomes had been met and feel increased confidence in working with their respective UK/East Asian counterparts.
- More than half believe the partnership would not have happened without the British Councils’ support.
However, the interim results have also identified clear distinctions in experience and perceptions between UK and local (East Asia) partners, as well as considerable differences in performance between the seven countries, raising interesting questions for further analysis and qualitative interviews in subsequent stages.
Access to this ongoing monitoring data and regional results has had a positive impact on the British Council by improving internal interest in and capacity for M&E, as well as provide robust data supporting the programmes’ successes at the local and regional levels. Such data creates a powerful justification for future activity and has established an internal appetite for good evidence and its role in programme delivery.
Related case studies
Evidence that subject-level TEF can enhance teaching quality, but at what cost to academic staff?
Graduate Outcomes: the UK’s biggest annual social survey
Reimagining the non-academic student experience in a Covid world
