Measuring our Impact: How Developmental Evaluation Informs our Practice Part 2
Steve Williams is a developmental evaluator at LEDlab and has extensive professional experience in evaluation, impact measurement, and data visualization for sustainability and social change projects. Steve is currently a PhD candidate in Resources, Environment and Sustainability at the University of British Columbia where he is a Liu Institute for Global Issues Scholar and a UBC Public Scholar. He is researching the societal impacts of participatory processes focusing on the Alberta Energy Futures Lab.
In our previous post on evaluation, we talked about Developmental Evaluation and how that helps us tune and adjust our program. But how do we evaluate systems change and the impact the LEDlab is having?
We use the multi-level perspective of transition theory (F.Ww Geels, “Technological Transitions as Evolutionary Reconfiguration Processes: a Multi-Level Perspective and a Case-Study,” Research Policy 31, no. 8 (2002): 1257–74.) to guide us. This framework highlights the importance of interactions among niche innovations, policy regimes and the socio-cultural landscape in which complex change processes are embedded. The theory suggests that changes in socio-technical systems arise mainly from the intersection of two processes, one in which collections of niche innovations align to put pressure on the policy regime, and the second wherein landscape level changes (i.e. elections, economic crises) create windows of opportunity within the regime for niche innovations to take hold. It is important to note that the time scales of regime and landscape change can be over years, decades or even generations. Over the life of the LEDlab, we may see system shifts or might be able to report on early indicators of systems change.
Based on our experiences in the field, we’ve observed the following changes at the niche/regime/landscape levels related to local economic development in the DTES.
It is important to note that while these changes are happening, not all are caused by the LEDlab. However, it is still important to track these changes. For example, the Fentanyl crisis has shifted resources, impacted community organizations across the DTES and affected individual projects within the LEDlab. The multi-level perspective gives us a framework to observe these changes and respond to them in real time.
To track the impact of the LEDlab, we are using data collected through the DE process. Here are some of the questions we are asking and indicators we are tracking:
- Niche – What is the impact of project level innovations?
- Hours of capacity building workshops delivered to community organizations and graduate students
- Instances of organizational learning (E.g. idea exchanges, new knowledge or vocabulary, new technology skills, new communication skills, consensus building, decision-making processes)
- Regime – What impact are we having on policy, regulation and institutional change?
- Number of networks strengthened and extended (E.g. collaborative grant application, strategic partnerships, MoUs, space sharing)
- Instances of structural socio-economic changes (E.g Policies/laws passed, new public discourse, infrastructure changes)
- Instances of structural organizational changes (E.g. shifts in roles/ responsibilities, new jobs, new business models, shifts in collective purpose and vision)
- Landscape – How are we influencing broader social and cultural change? What is the impact of political and economic shifts on the LEDlab and how is the lab responding?
- Shifts in public narrative
- LEDlab adaptations to changes in landscape
In addition to early indicators of systems change, we are also observing impacts at the level of products (e.g. web sites, documents), enhanced learning/capacity, and new or strengthened networks. We also see impacts at the level of organizational (new business models, changes in roles or structure) and policy (new policy, changes in policy discourse). The table below summarizes these effects. Note that this is not a comprehensive list but illustrates the types of impacts we are seeing in different categories.
Taken together, these indicators show the rich impact that LEDlab is having in the community. This impact goes beyond individual projects to include policy, supporting community organizations and strengthening networks in the community.
Be sure to sign-up to LEDlab’s Newsletter to stay updated on all our activities!