By Laura Clark
Social innovation without impact is null. It is safe to say that all social innovations strive to create as much impact as possible. However, is it possible to get lost in the art of measuring impact?
The measurement of impact is particularly interesting as it pertains to a project that I am leading. EmpowerU works with women exiting the sex trade, teaching them financial literacy and entrepreneurial skills to help them gain independence. Upon graduation from the financial literacy and entrepreneurship workshops, women are able to apply for post-secondary scholarships to help them advance their career. They are also given the option to participate in EmpowerU’s Micro Loan Program to further support them in receiving the skills, mentorship, and finances they need to pursue their dreams of running their own social enterprises.
The question remains about how to implement impact metrics for this project. How does one measure the impact that is clearly taking place and avoid getting lost in the process? Although it can be a cumbersome task trying to find an appropriate measure, it is of utmost importance in order to raise awareness and support for the project and its cause.
More Than Economics and Financials
Antadze and Westley (2012) discuss that in order to adequately measure impact, the metrics for social innovation must go beyond economic and financial metrics (p. 134). They analyze this topic through combining research from many sources. A struggle in measuring the impact of social innovations that is mentioned is measuring them in a linear way, which does not suit the complex system that these innovations act in (Antadze & Westley, 2012, p. 134). This was especially a struggle within the EmpowerU project. The main component of the project that would reveal impact is the human component with the women the project works with, which is complex and very non-linear in nature. For example, one woman receives a scholarship through the EmpowerU program. This allows her to return to school to pursue classes that would help her advance her entrepreneurial career. She gains skills and confidence and is interested in opening her own social enterprise. She becomes a part of the EmpowerU Micro Loan Program, allowing her to pursue her business and eventually hire other disadvantaged women within her social enterprise. Simply using a statistic based strictly on number of women impacted, one woman was impacted. This would fail to take into account the major impact of the journey involved and the ongoing support throughout the process, therefore missing a major component of impact. In this respect, the relation of social innovation as a process and “journey” is relevant (Antadze & Westley, 2012, p. 134). Just as each woman’s life journey will differ and require adaptation, so will the social innovation that addresses it.
Finding an Appropriate Evaluation Method
Formative and summative evaluation focus on testing a model (Antadze & Westley, 2012, p. 146). Formative evaluation works to improve the model, while summative evaluation determines its success and effectiveness upon its completion and determines whether it should be continued, improved upon, or eliminated (Antadze & Westley, 2012, p. 146). The suggestion to use developmental evaluation with summative and formative evaluation is appealing as it does not rely solely on summative or formative evaluation and avoids the shortcomings associated with these methods (Antadze & Westley, 2012, p. 146). Developmental evaluation focuses on ongoing adaptation and development that is needed for a complex, rapidly changing system. Summative and formative methods of evaluation do not suffice on their own.
Refocus
Gugerty and Karlan (2018) vouch for evaluations of impact implemented in the right circumstances, adding that organizations should be building a positive internal culture in which data is collected on a regular basis (p. 41). This rings true for EmpowerU as there was a time when precious project resources were being dedicated to impact metrics that were not necessarily measuring impact effectively. A “mosaic” of different pieces was considered as the project progressed and this gave a clearer picture of what impact looks like and of how EmpowerU would measure impact as a result (Gugerty & Karlan, 2018, p. 41).
Surveys and conversations are used to better understand the impacts of EmpowerU on each woman’s personal life, relationships, and sense of fulfillment and purpose.
The discussion of asking the right questions but taking a wrong approach resonates (Gugerty & Karlan, 2018, p. 43). Although the EmpowerU team was asking valuable questions that were important to consider, the mark was missed in the actual impact the answers to these questions would reveal. Taking a step back and refocusing efforts allowed irrelevant metrics to be put aside. Indirect effects discussed by Gugerty and Karlan (2018) are important to keep in mind when measuring impactful change (p. 44). The EmpowerU team was able to better see the indirect impact of project efforts once metrics were re-evaluated. This sparked the realization that even more impact was being created than originally thought. Instead of exploring new measurement methods and investing resources into that process, we were able to direct our efforts into how we can better measure impact using the methods we already have. For example, by changing questions within EmpowerU’s pre-workshop and post-workshop surveys, we were better able to target and measure impact. We did not need a whole new measurement system, just an update of one that was already working effectively and with some improvements could work even better. This brings up the point of going with what is already known when measuring impact (Gugerty & Karlan, 2018, p. 45). Sometimes an answer is revealed through research that has been done, and more research and impact evaluations are not necessary (Gugerty & Karlan, 2018, p. 45). The information that was already held through previous EmpowerU research and impact metrics gave us the answer to our impact metric issue. By researching metrics that were already recorded, redundant work was being done. A re-organization of information was needed to put EmpowerU back on track and effectively measure our impact.
Summary
Re-organization allowed EmpowerU to consider our values, focus on what really mattered, and base our measurement choices on this. With effective measurements in place, EmpowerU is able to measure impact and communicate our impact effectively, while allowing room for improvement if needed. Effective measurements have made all the difference. We are able to communicate to team members and partners what our goals are and what we have accomplished. This has had far-reaching effects in raising awareness and within the project overall.
How have you approached impact measurement for social innovations?
References
Antadze, N., & Westley, F. R. (2012). Impact Metrics for Social Innovation: Barriers or Bridges to Radical Change?. Journal of Social Entrepreneurship, 3(2), 133-150, https://doi.org/10.1080/19420676.2012.726005
Gugerty, M. K., & Karlan, D. (2018). Ten Reasons Not to Measure Impact – and What to Do Instead. Stanford Social Innovation Review. Retrieved from https://ssir.org/articles/entry/ten_reasons_not_to_measure_impact_and_what_to_do_instead
Laura Clark is a student at Mount Royal University in Calgary, and project lead for EmpowerU. This blog post was adapted from a short précis she wrote for a course: Historical Case Studies for Social Innovation.
ABSI Connect works to connect, align, celebrate, strengthen, and learn from Albertan change-makers who are finding innovative ways to address the complex problems their communities face. Do you have a story, idea, or insight you’d like to share with the ABSI Connect community? Let us know.