Appreciating evaluation

If you want someone’s eyes to glaze over and suddenly remember an urgent meeting, just mention the phrase, “project evaluation.” It is astounding to me that people will dump millions of dollars into a project and have no idea whether or not it accomplished anything but spending money.

Since project evaluation is a passion of mine (but most of the time I don’t admit it), I was delighted to find a panel discussion at SOCAP 2012 on this topic. The speakers, Margo Brandenburg of the Rockefeller Foundation, Jenny Everett from the Aspen Institute, Rick Jacobus of Cornerstone Partnership, and Emily Tucker and Steve Wright from the Grameen Foundation, are all using different models but they are all looking at the same thing: how do we measure success? How do we know our project achieved the desired outcome?

For a research fanatic, like me, I was also pleased to see that there is terrific creativity in how they approach measuring success. For most large government projects, there is no evaluation of any sort. The last time I checked the figures, 5 percent were evaluated. Several academic Ph.Ds would be brought in at the conclusion of the program and spend two or three years trying to figure out what happened. 

I believe in rigorous methodology, but I also believe there is also a place for clever, non-academic ways to get information specific to a project. 

The SOCAP panel looks to put metrics in place before the project starts and to make measurement part of the process. If you wait until the end, then bring in people who were not involved in the project, there’s no way to recover the important events that happened along the way. Project staff gets reassigned. Some things were never written down. New people coming in after the fact will only get a skewed perspective. 

Each panel member had a different process, but they needed different information for their unique projects. They came up with processes that worked for their project. Regardless of the measurements used, if they result in evaluation metrics that provide useful numbers that can guide future projects and provide a baseline to start from, what does it matter if the Ph.Ds turn up their noses at the methodology?

But in addition to project measurement, there’s another aspect of project planning that is obviously overlooked, disregarded, or never thought of in the first place. There are so many example of this you wonder what hole the project team’s head is hiding in.

In a conversation between SOCAP sessions, I listened to someone talk about helping the poor people of Nigeria. This was a project to teach Nigerians how to make crafts to help them eke out an unsustainable living.  While I listened, I wondered why no one mentions Shell and other foreign oil companies that are making out like bandits by extracting Nigerian oil. Unimaginable and unknown sums are paid to the Nigerian government. The Nigerian government could probably pay its citizen a living wage from this money, but it’s going into government officials’ coffers, most likely located in Switzerland.

The oil companies do not hire locals, therefore they strike, kidnap western oil workers and otherwise act out their anger. This proves they are only worthy of incarceration. If I were a desperately poor Nigerian watching all those rich western oil workers take my country’s natural wealth, for which I get nothing, I’d be a bit upset too. And I’d be insulted by being offered a crafts class instead of letting me participate in my country’s natural wealth.