Measuring our Impact: How Developmental Evaluation informs our practice
Steve Williams is a developmental evaluator at LEDlab and has extensive professional experience in evaluation, impact measurement, and data visualization for sustainability and social change projects. Steve is currently a PhD candidate in Resources, Environment and Sustainability at the University of British Columbia where he is a Liu Institute for Global Issues Scholar and a UBC Public Scholar. He is researching the societal impacts of participatory processes focusing on the Alberta Energy Futures Lab.
Like most projects, we at LEDlab want to make sure we are delivering what we set out to do and are meeting our program goals. But when working with complex emergent projects with a goal of social innovation, it can be challenging to use traditional evaluation methods that expect a defined strategy and outcomes. A challenge we face is that the system we are working within is changing rapidly, we don’t know all the answers going in (that’s the whole point of the Lab approach!) and we are co-creating the project with our partners. Developmental Evaluation (DE) is an evaluation approach well suited to an initiative such as the LEDlab. DE focuses on project developments and key pivot points within the project to surface learning for the project team. The idea is to conduct this evaluation and learning in real time so that the team can respond and adapt to changing conditions rather than waiting until review at the end of the project to see what went well and what didn’t.
We manage this process through our bi-weekly DE Reports, which are reflections, that are later used as a management and strategic learning tool throughout the organization. These reports focus on three key questions:
- What? What is happening now? What is being produced? What new learning is happening? What networks are being developed or strengthened?
- So What? What are the implications of this? What insights have you had? What was surprising and unexpected?
- Now What? What are the opportunities? What about roadblocks? What are the next steps?
Graduate student project coordinators complete DE Reports every two weeks and send to lab staff and to their community partner; similarly lab staff complete DE reports every two weeks and send to parent organizations (Ecotrust and RADIUS). DE Reports are used to support quarterly partner meetings between parent organizations and lab staff re: fiduciary oversight and high level strategic direction and are also used to support monthly meetings between lab staff, community partners and project coordinators re: project strategy.
The data we collect through the DE process also allows us to track how we are impacting systems. Coaching that happens in response to DE Reports also supports the personal development and growth for graduate students and lab staff.
So, how does this evaluation system help us to evaluate in complex and consistently changing circumstances? We are able to use this framework to capture and activate three key types of impacts:
- Impact on students by providing a tool for self reflection, having a place to document emotions/concerns/challenges, and surfacing roadblocks.
- Impact on lab strategy by providing a mechanism for co-design and co-creation with feedback from the system and community partners
- Impact on community by sharing back insights and developmental outcomes with the system to help it better understand itself through community partner workshops and other venues.
The data we collect also helps us track impact we are having at a systems level. More on that on our June evaluation blog!
Be sure to sign-up to LEDlab’s Newsletter to stay updated on all our activities!