Measuring our Impact: How Developmental Evaluation informs our practice

Steve Williams is a developmental evaluator at LEDlab and has extensive professional experience in evaluation, impact measurement, and data visualization for sustainability and social change projects. Steve is currently a PhD candidate  in Resources, Environment and Sustainability at the University of British Columbia where he is a Liu Institute for Global Issues Scholar and a UBC Public Scholar. He is researching the societal impacts of participatory processes focusing on the Alberta Energy Futures Lab.

Like most projects, we at LEDlab want to make sure we are delivering what we set out to do and are meeting our program goals. But when working with complex emergent projects with a goal of social innovation, it can be challenging to use traditional evaluation methods that expect a defined strategy and outcomes. A challenge we face is that the system we are working within is changing rapidly, we don’t know all the answers going in (that’s the whole point of the Lab approach!) and we are co-creating the project with our partners.

Developmental Evaluation (DE) is an evaluation approach well suited to an initiative such as the LEDlab. DE focuses on project developments and key pivot points within the project to surface learning for the project team. The idea is to conduct this evaluation and learning in real time so that the team can respond and adapt to changing conditions rather than waiting until review at the end of the project to see what went well and what didn’t.

We manage this process through our bi-weekly DE Reports, which are reflections, that are later used as a management and strategic learning tool throughout the organization. These reports focus on three key questions:

  • What? What is happening now? What is being produced? What new learning is happening? What networks are being developed or strengthened?
  • So What? What are the implications of this? What insights have you had? What was surprising and unexpected?
  • Now What? What are the opportunities? What about roadblocks? What are the next steps?

Graduate student project coordinators complete DE Reports every two weeks and send to lab staff and to their community partner; similarly lab staff complete DE reports every two weeks and send to parent organizations (Ecotrust and RADIUS). DE Reports are used to support quarterly partner meetings between parent organizations and lab staff re: fiduciary oversight and high level strategic direction and are also used to support monthly meetings between lab staff, community partners and project coordinators re: project strategy. Coaching that happens in response to DE Reports also supports the personal development and growth for graduate students and lab staff.

Screen Shot 2017-05-14 at 10.26.31 PM

This framework and process help us to capture and activate three key types of impacts:

  • Impact on students by providing a tool for self reflection, having a place to document emotions/concerns/challenges, and surfacing roadblocks.
  • Impact on lab strategy by providing a mechanism for co-design and co-creation with feedback from the system and community partners
  • Impact on community by sharing back insights and developmental outcomes with the system to help it better understand itself through community partner workshops and other venues.

So, Developmental Evaluation helps us tune and adjust our program, but how do we evaluate systems change and the impact the LEDlab is having?

We use the multi-level perspective of transition theory (F.Ww Geels, “Technological Transitions as Evolutionary Reconfiguration Processes: a Multi-Level Perspective and a Case-Study,” Research Policy 31, no. 8 (2002): 1257–74.) to guide us. This framework highlights the importance of interactions among niche innovations, policy regimes and the socio-cultural landscape in which complex change processes are embedded. The theory suggests that changes in socio-technical systems arise mainly from the intersection of two processes, one in which collections of niche innovations align to put pressure on the policy regime, and the second wherein landscape level changes (i.e. elections, economic crises) create windows of opportunity within the regime for niche innovations to take hold. It is important to note that the time scales of regime and landscape change can be over years, decades or even generations. Over the life of the LEDlab, we may see system shifts or might be able to report on early indicators of systems change.

Evaluation Blog Part 2 of 2 – Image 1

Based on our experiences in the field, we’ve observed the following changes at the niche/regime/landscape levels related to local economic development in the DTES.

Evaluation Blog Part 2 of 2 – Image 2

It is important to note that while these changes are happening, not all are caused by the LEDlab. However, it is still important to track these changes. For example, the Fentanyl crisis has shifted resources, impacted community organizations across the DTES and affected individual projects within the LEDlab. The multi-level perspective gives us a framework to observe these changes and respond to them in real time.

To track the impact of the LEDlab, we are using data collected through the DE process. Here are some of the questions we are asking and indicators we are tracking:

  • Niche – What is the impact of project level innovations?
    • Hours of capacity building workshops delivered to community organizations and graduate students
    • Instances of organizational learning (E.g. idea exchanges, new knowledge or vocabulary, new technology skills, new communication skills, consensus building, decision-making processes)
  • Regime – What impact are we having on policy, regulation and institutional change?
    • Number of networks strengthened and extended (E.g. collaborative grant application, strategic partnerships, MoUs, space sharing)
    • Instances of structural socio-economic changes (E.g Policies/laws passed, new public discourse, infrastructure changes)
    • Instances of structural organizational changes (E.g. shifts in roles/ responsibilities, new jobs, new business models, shifts in collective purpose and vision)
  • Landscape – How are we influencing broader social and cultural change? What is the impact of political and economic shifts on the LEDlab and how is the lab responding?
    • Shifts in public narrative
    • LEDlab adaptations to changes in landscape

In addition to early indicators of systems change, we are also observing impacts at the level of products (e.g. web sites, documents), enhanced learning/capacity, and new or strengthened networks. We also see impacts at the level of organizational (new business models, changes in roles or structure) and policy (new policy, changes in policy discourse). The table below summarizes these effects. Note that this is not a comprehensive list but illustrates the types of impacts we are seeing in different categories.

Evaluation Blog Part 2 of 2 – Image 3
Evaluation Blog Part 2 of 2 – Image 4

Taken together, these indicators show the rich impact that LEDlab is having in the community. This impact goes beyond individual projects to include policy, supporting community organizations and strengthening networks in the community.