MSD and job creation: can we measure what counts?

The need to ‘prove results’ in job creation easily creates incentives for unsustainable quick fixes that do not shift the conditions holding problems in place.

In December last year a major British pub chain made national headlines by announcing plans to invest £200 m to create 10,000 jobs in provincial UK pubs and hotels.

More retail jobs generally relies on more spending by consumers.  But sector analysis is forecasting weaker trading conditions and a ‘nightmare scenario’ for British hoteliers. I wondered if the company’s strategy is to enlarge its market share, while smaller businesses struggle. If so, then with insecure contracts and low wages, the real impact might not be more jobs but worse jobs.

These days the UK economy manages to combine record high employment with record rates of in-work poverty. And job-rich poverty is not just a first world problem. The quality of growth matters for developing countries too. Economists agree that shifts towards higher-productivity, higher value-added activities and innovations are needed to drive job creation and lift living standards.

A key ambition for donors therefore is to influence the conditions that guide economies onto paths of inclusive and sustainable job-rich growth: so improving the position of disadvantaged groups such as women.

It follows that donors have a strong interest, not just in levels of investment, but in system change. And for now, I will define that simply as “shifting the conditions that are holding a problem in place”.

Kania, Kramer & Senge, who I borrowed that phrase from, identify six general types of conditions. Three types they label as explicit:

• Policies, Practices, Resource Flows

But, they argue these are underpinned by three types of intangible conditions:

• Relationships and connections, Power dynamics, Mental models

And it is these intangible conditions we need to shift in order to achieve lasting systemic change.

So it is not surprising that as MSD practitioners, we often end up talking about and trying to address intangibles such as ‘Innovation, Adaptation, Institutions and norms, Supporting functions’

Now here comes the problem - these intangibles are ‘intangible’ - they’re hard to measure

People are trying, and here is an example. I recently reviewed a study (not yet published) which examined the impact of two labour market programmes (EYE Kosovo, Risi Albania) on skills development, intermediation, investment and regulation in the ICT sector.

The researchers looked at some very intangible issues:

1. Had the programmes influenced thinking, attitudes, behaviours and the way things are assessed, planned and done in the sector?
2. Had the programmes influenced the way people look at gender relations, and at the participation of people from minority ethnic groups in the workforce?

The study did find significant change in actors' behaviors, relationships, activities, policies and practices. These changes were affecting types of business models, advocacy and policy change, collaboration and institutional change. However, at best, the study told a narrative story and certainly did not provide rigorously objective or numerical evidence.

Now, some people get very impatient with all this. They acknowledge the complexity of markets, but believe from a practical point of view, systems are no more than the sum of their parts. Intangible concepts, such as innovation, resilience, adaptive capacity, relationships, norms and so on are unnecessary, they argue, or must be defined in terms of observable actions and their resource impacts so that they be readily understood and measured.

I think we have a clash of ontologies here. That is theories about the kinds of things that have existence:

• a holistic world view assumes that complex phenomena (such as ecosystems, physiology, consciousness or market economies) are best understood in terms of interacting systems and networks - where the whole is manifestly more than the sum of its parts
• a reductionist world view believes complex phenomena can only be properly explained and understood by unpacking the fundamental elements and laws that underpin them (like physics and chemistry)

So can we look at measuring jobs in a rigorously quantitative (i.e. reductionist) way? By the way this study is one of the very few pieces of evidence we have about formal job creation in the BEAM Evidence Map.

PEPE is a £43 m DFID programme in Ethiopia aiming to create 45,000 formal manufacturing jobs mainly in garments, leather and textiles sectors, partly using an MSD approach. The PEPE mid-term evaluation actually used mixed methods, but I want to talk about the quantitative element, which was very elaborate.

The independent evaluation team used a computable general equilibrium (CGE) approach, based on an existing high-quality model of the Ethiopian economy. They combined this model with firm-level survey regression analyses to generate a range of possible employment effects that combined labour created in Hawassa Industrial Park with estimates of the effect of increased firm profits.

In the garment sector they went further by conducting a detailed case study using a process tracing methodology. Then used this to create a second modelling scenario to estimate plausible employment effects.

In the end, the evaluation found evidence supported a ‘contribution claim’ of between 4,645 and 12,436 jobs, attributable to PEPE support.

Considering the high level of resources and technical evaluation skills needed, this seems to me like a surprisingly imprecise result. And, by the way, the methodology did not allow disaggregation by gender, social inclusion or other criteria such as ‘decent work’.

Fortunately, the evaluation report is far more useful where it reports on three in-depth case studies that qualitatively assessed PEPE’s contribution to solving critical firm-level constraints separately in each sector. These findings tell DFID and PEPE managers a great deal about which interventions are working, which are not and why. The surveys were not as ‘scientifically rigorous’ but their explanatory power provides a convincing steer for where PEPE goes next.

So where does this all lead us?

It seems to me that whether it is 10,000 jobs for £200 m in the UK, or 45,000 jobs for £43 m in Ethiopia, in the messy, complex, dynamic world of real economic systems we are vulnerable to simplistic equations and expectations about job creation.

The need to ‘prove results’ in job creation easily creates incentives for unsustainable quick fixes that do not shift the conditions holding problems in place.

In most cases I cannot see how reductionist thinking is going to allow us to understand or evaluate systemic change. Moreover, rigorous quantitative methods - using only objective data about tangible effects - are actually surprisingly difficult and costly (as we saw with PEPE Evaluation). And still produce relatively imprecise answers.

On the other hand, a drive to ‘improve results’ creates incentives to examine the intangible conditions for decent job creation. Most of the time we will have to do this using relatively subjective qualitative methods and accept the lack of ‘rigour’ in exchange for useful insights.

This will probably never produce simple neat answers. But when it comes to job creation, I think it is better to try to measure what counts, than only to count what you can measure.