Evaluating ‘Beyond the Program’ to Drive Institutional Change

Introduction

In the complex landscape of higher education, equity programs stand as beacons of hope, aiming to dismantle barriers and foster inclusivity. Program evaluation can provide valuable insights into the effectiveness and efficiency of equity programs and their contributions to addressing societal issues.

However, evaluations often concentrate on assessing the programs’ activities rather than considering the broader context in which the programs are developed and executed. To fully understand their impact, we must widen our evaluation lens and examine how these initiatives interconnect with the broader environment of institutional policies, practices and cultural norms. By analysing equity program data in relation to structural and cultural elements, we can better understand their true impact and drive meaningful institutional change.

The Importance of Context in Evaluating Equity Programs

Let me say right from the outset, that I’m not an evaluations expert. However, I deeply value the insights that evaluation offers us, whether we are equity practitioners, researchers, leaders, policymakers or other stakeholders.

I think most of us recognise that equity in higher education isn’t just about individual initiatives. It’s about transforming the institution as a whole. Equity programs are more than standalone efforts; they hold the potential to drive meaningful institutional change. Take, for example, an outreach program in a rural area that successfully increases the interest of students in higher education. At first glance, this program will be evaluated as successful, and rightly so.

Yet, if these students then face a wall of systemic barriers preventing access and participation, we must question the completeness of our evaluation.

Recognising that evaluations are tools for improving rather than proving, we should ask ourselves: Could our understanding of equity programs and progress become even more impactful by extending our program evaluations to link to the broader structural and cultural contexts these students navigate?

Linking Equity Programs to their Institutional Environment

Let’s consider the potential to drive institutional change through our program evaluations.
While equity programs and their thorough evaluations are essential in addressing inequities in higher education, true transformation also demands shifts in policies, culture, and practices that go beyond individual programs.

Despite over $1.5 billion investment in equity initiatives by the government since 2010, the representation of students from equity backgrounds in universities has shown minimal change. This period saw substantial efforts in developing and implementing well-designed equity programs that often achieved their outcomes. Yet, the real impact of these programs is beyond observable outcomes and the boundaries of the program activities.Universities are complex, with a maze of policies, governance structures and cultural underpinnings.

Understanding how equity programs interact with these broader elements is essential for understanding their effectiveness and real (or potential) impact. The effectiveness of these programs can be significantly influenced by the institutional environment—a program successful in one setting may face challenges in another due to external factors such as policy constraints or structural barriers.

Simply adjusting or even developing a new program might not lead to the desired results. Addressing broader systemic issues could unlock the program’s potential impact without changing its fundamental design. As such, evaluating equity programs within the context of the entire institution is key to gaining a comprehensive view of their actual and potential effectiveness and impact.

Let’s consider the example of equity program designed to increase the number of students in a rural community who are considering higher education, with the overarching aim of increasing the number of students enrolling in the university. The program is evaluated to show that it successfully increased interest, but actual enrolment numbers do not improve due to the admission policies.

Following a shift towards more equitable admission practices, the same program, without any change in its design or delivery, suddenly achieves its enrolment aims, highlighting how external factors can significantly affect the evaluation of a program’s efficacy and outcomes.

This leads us to consider: How are our institutional structures and cultures facilitating or hindering the potential impact of equity programs and real change?
How can we then use these understandings from our evaluations to inform, influence and transform these very structures and cultures towards a more equitable system?

Broadening the Evaluative Lens

By integrating the evaluation of equity program data with the broader institutional context, we can also contribute to driving institutional change. Through this perspective, we not only measure the outcomes of individual programs, but also gain an understanding of how and why these outcomes are either achieved or not achieved. We can draw on identified constraints or enablers at the institutional level as potential catalysts to inform and drive institutions toward adopting more inclusive and equitable structures and cultures.

Consider our earlier example of the equity program aimed at increasing student enrolment for students in a rural community. Suppose the program’s broader evaluation reveals a need for more inclusive admission policies for the program to achieve its potential impact. These insights could prompt the institution to reevaluate and possibly revise its admissions policies.

This then challenges us to consider expanding our equity program evaluation frameworks, to encompass how these programs interact within the wider institutional ecosystem.

Towards Evaluating Institutional Change

As we incorporate evaluation into our equity programs, we can see that evaluating outcomes and their contribution to broader equity goals need to go beyond the program activities. A broader evaluation of these initiatives can also help drive systemic change within institutions.

This perspective urges us to expand our program evaluation frameworks to encompass the broader objective of institutional change for equity goals, marking it a critical step in our evaluation practices.