Background

Anne Goodman’s question to the Leap Ambassadors Community in February 2015, about how to structure a new Director of Outcomes & Evaluations position at the Saint Luke’s Foundation sparked insightful and substantive responses from five ambassadors based on experiences within their own organizations.

Guidance Highlights

(FULL RESPONSES ORGANIZED BY THEME, BELOW)

Think of this as the first hire in building a team; they won’t be able to do it all on their own.

• Consider EQ (emotional intelligence) as much as IQ; evaluation and quality improvement lead to change management–look for people who are good with people.
• Increase the influence of the position by making it a direct report to the CEO/President/Exec. Director.
• Start internally. Working in partnership with key leadership, the person in this position should start by:
  • Setting the cultural foundation within your organization
  • Sharpening your organizational goals/outcomes and evaluation
  • Solidifying your organization’s theory of change.
• The first hire needs to know they are driving culture change; discussions and planning with key leadership should focus on change management, establishing small wins from the beginning.
• Then, the person in this position can help grantees understand what role they play in the larger scope of your work and identify which of your indicators they can help support.
  • Practical cautionary note: Grantees have only so much time and resources to put into data collection and monitoring. Decide whether to emphasize the needs of grantees or the needs of the foundation.
• Keep in mind that the internal work will evolve, but never be “done.”

Recommended Reports

Benchmarking Evaluation in Foundations: Do We Know What We Are Doing?, The Foundation Review, January 2013 (recommended by Patti Patrizi who for 15 years ran the Evaluation Roundtable, which convenes evaluation directors from philanthropy and also benchmarks practice and is now running the Center for Evaluation Innovation in DC).

Necessary and Not Sufficient: the State of Evaluation Use in Foundations, which “highlights an important and interesting variation among those foundations where evaluation units report directly to the CEO,” and other relevant case studies, articles, and papers from the Evaluation Roundtable here (per Patti Patrizi).

Building Capacity to Measure and Manage Performance, Bridgespan (recommended by Isaac Castillo).

People Recommended to Consult

• Isaac Castillo, Director of Outcomes, Assessment, and Learning at Venture Philanthropy Partners  and a Leap Ambassador, icastillo@vppartners.org
• Dominique Bernardo, CEO at Variety – The Children’s Charity (recommended by Cynthia Figueroa who noted that Dominique served as the VP of Quality Assurance at Congreso) and a Leap Ambassador, dominique.bernardo@varietyphila.org
• Juan de Angulo, Vice President of Quality Assurance at Congreso (also recommended by Cynthia Figueroa who noted that Juan “can provide great perspective regarding the role he plays in the agency”), deanguloj@congreso.net
• Gabriel Rhoads, former Edna McConnell Clark Foundation’s Director of Evaluation and Learning, (recommended by Lissette Rodriguez who noted that he has a role similar to the one Anne laid out and could also tell her about the two other people on his team), grhoads@projectevident.org

Sample Job Descriptions (See Appendix)

• Director of Data and Evaluation at DC Promise Neighborhoods Initiative (Isaac Castillo noted that unlike Saint Luke’s Foundation, DCPNI is a direct service organization, but since they do grantmaking, ™[they] have had to create an internal culture of outcome measurement and evaluation∫ and ™provide technical assistance [in some cases, very intensive technical assistance] to our grantees/partners∫).
• Director of Evaluation and Learning at Edna McConnell Clark Foundation (shared by Gabriel Rhoads).
• Director of Evaluation and Quality Improvement at a nonprofit (Anonymous upon request).
• Director of Program Evaluation and Planning at an agency (Anonymous upon request).
• Director of Data and Evaluation at Congreso (shared by Juan de Angulo).

Important Note: As Patti Patrizi explained, “There is a huge range of emphasis in the job descriptions. The direct service providers (rightfully) need to emphasize data collection, data quality, and improvement of service. This is not necessarily the same job as that of an evaluation person in a foundation (of course depending on what kind of foundation it is, for instance an operating foundation would have some of the same needs as that of a service provider). And the person who is good at working internally within a foundation–that is doing theory of change and helping to identify strategy indicators etc.–is not likely to be the same person who can work in a hands-on way with grantees to help them improve their own data systems and use.”

Sample organizational structure: Also see Appendix for a detailed table describing how the Data & Evaluation department at Congreso is structured.

Responses

(ORGANIZED BY THEME)

THINK OF THIS AS THE FIRST HIRE

Isaac Castillo: “First, I will say that it will likely be difficult for a person to do both the internal work and the external/grantee work at the same time without any help. You would need a true superstar that has done both types of work already, and even then they would probably need help. And even when things are up and running, it will likely require multiple people to maintain the work (depending on the size of your org, and the number of grantees you would be working with). First piece of advice is to think of this as the first hire in building a team, and that this person would eventually hire one or more people to help with their work.

“When I was working in the Director of Data and Evaluation position, my time was split into thirds. One third of my time was spent working on internal cultural issues –working with staff to get them more oriented to outcomes measurement and performance management. Another third was working directly with grantees–helping them also understand outcome measurement and performance management, but also focusing on simple things like proper use of measurement tools and even just the benefits of simple technology (like Excel) to assist in the process. The last third of the work was focused on collecting community wide data and staying up on current program evaluation and program design research to help inform our work.

“As my position evolved, I hired someone to take over a large chunk of the external work–it was taking up a lot of time to do technical assistance and support for our 8 grantees, especially once the grantees left the paper/pencil stage and needed more support with their data systems. So that made us a staff of two. But really, we need a third person–someone to focus heavily on the internal work as we slowly expand the amount of programming we delivery ourselves.”

CONSIDER EQ (EMOTIONAL INTELLIGENCE) AS MUCH AS IQ—LOOK FOR PEOPLE WHO ARE GOOD WITH PEOPLE

Gabriel Rhoads (EMCF’s Director of Evaluation and Learning, recommended by Lissette Rodriguez): “When you’re hiring for an Evaluation Director, I’ve often been told you want to consider EQ as much as IQ. Evaluation and quality improvement can lead to change management, and when you’re hiring, be sure to keep a lookout for people that are good with people. It’s possible to bring on a team that’s fantastic with data, but the leader should be able to interpret, explain, and make all the information understandable.”

POSITION THE ROLE FOR MAXIMUM INFLUENCE

Alice Shobe: “Very quickly after hiring Mei Ling [our first measurement and learning position], I moved her to become a direct report to me so that I could increase her influence agency-wide. It was clear that I was prioritizing this measurement and learning culture shift and I helped positioned Mei Ling with the right technical knowledge to have the most influence/authority possible.”

START INTERNALLY WITH YOUR THEORY OF CHANGE, CULTURE, GOALS

Isaac Castillo: “I think that it is important that this person start by making sure there is a good theory of change and set of outcomes for your organization that everyone is in agreement upon.”

Alice Shobe: “My advice from our journey is to really focus internally first and make sure you’ve set the cultural and expertise foundation. We really spent a good couple of years doing that by focusing on sharpening our own organizational goals/outcomes and evaluation and personal goals/evaluation. From there we began to move from exclusively hiring external evaluators to more direct outcomes work on individual grants and work with our grantees. As we’ve moved into this current chapter, we are also working closely with our public sector partners to make sure we were focused on the learning/insights needed to advance our homeless work in our counties.”

FIRST HIRE IS DRIVING CULTURE CHANGE, SHOULD BE FOCUS OF KEY LEADERSHIP

Cynthia Figueroa: “The first person hired needs to know they are driving culture change in the organization. Discussions and planning with key leadership should be equally focused on change management. Need to establish what will be some small wins from the beginning.”

THEN HELP GRANTEES IDENTIFY WHICH INDICATORS THEY CAN SUPPORT IN BROADER SCOPE OF YOUR WORK

Isaac Castillo: “Once [your theory of change] is in place, then it should be presented to the grantees to help them understand what role they would play in the larger scope of Saint Luke’s work. At DCPNI, when we first start working with grantees (and at the beginning of each new grant cycle) we have them look at our theory of change and indicators and have them talk about which indicator they think they can help support. This way, right from the beginning, we get grantees to focus on a few indicators (we really don’t let them choose more than three) and we know pretty quickly where we have gaps in our service strategy. Ultimately, this helps get everyone oriented to a performance and outcome driven culture–internal and external people are all aligned to achieving the set of outcomes you have laid out. And it speaks to the importance of the data collection and outcome measurement work–if we can’t measure what we are doing, then how do we know if we are actually making a difference towards what we committed to?”

PRACTICAL, CAUTIONARY NOTE: GRANTEES ONLY HAVE SO MUCH TIME FOR DATA COLLECTION, DECIDE WHETHER TO EMPHASIZE NEEDS OF FOUNDATION OR GRANTEES

Patti Patrizi: “Grantees have only so much time and resources to put into data collection and monitoring. Foundations need to decide what they hope to emphasize in terms of data collection–the needs of grantees or the needs of the foundation. It is also important to be very clear about the different types of responsibility and accountability for a foundation or for its grantees. It is not the same.”

KNOW THE INTERNAL WORK WILL NEVER BE “DONE”

Isaac Castillo: I do agree with your strong internal push first–I think you need to do that to make sure your organization is in a good place. However, I do want to caution that the internal work will never be `done’–it will just evolve. Staff turnover will mean that there will always be basic culture work and training that will need to be done. And as existing staff become more bought into the ideas, they will have more sophisticated questions that will require more time from your evaluation staff to answer. At some point you will need to decide when it is time to do the external work as well–and that time may be signaled by hiring of a second person.”

Acknowledgments

Our thanks to Anne Goodman, Isaac Castillo, Cynthia Figueroa, Dominique Bernardo, Alice Shobe, Patti Patrizi, Lissette Rodriguez, and Gabriel Rhoads for sharing their insights.