What works centre or what should work centre?
The What Works Network is a UK central government initiative to gather evidence, analyse it, and present it to policy makers so that they can make better decisions. I absolutely love the concept.
There are problems — six of the seven centres are in London for example — but overall it’s a great idea. Government is listening and the centres have produced some really interesting and influential outputs.
My worry is that a few of those outputs are getting too influential. I’m going to focus on the What Works Centre for Local Economic Growth (WWCLEG) because it’s the only centre whose output I follow. I’ll share my understanding of the methods they’ve used and suggest areas where I think that their work could be interpreted better.
Before I start I want to be really clear. I’m not having a go. WWCLEG is really important and has done a good job. This is about improving that work, improving how the existing work is used, and maybe opening this area of evidence-led decision making up to a much wider and possibly more agile set of organisations and methods.
The What Works Centre for Local Economic Growth (WWCLEG)
WWCLEG is based at the London School of Economics and is run largely by academics from that institution. They work in collaboration with Arup and London-based think-tank Centre for Cities. It aims to answer questions like “does investing in transport infrastructure generate economic growth?” and “does hosting large cultural and sporting events generate economic growth?”.
So far, so good. These are extremely important questions where lots of people make huge claims without any basis in evidence. Specifically on those questions, WWCLEG have done reports on Transport and Sports & Culture. Read them, they’re accessible, well-presented, and well-evidenced.
The problems start when we look at how they went about answering those questions and when we look at exactly what question they answered. I have three concerns here,
- WWCLEG apply a really high bar for papers to be considered good enough to consider. Of 2300 studies they considered for their transport study, just 29 made the grade. Of 550 studies considered for their sports & culture study, just 36 made the grade. There’s nothing wrong with that, but we need to keep it in mind. Now that we’ve seen the results we may need to rethink this approach.
- WWCLEG is a UK body and yet it considers studies from all over the OECD even where our peculiar politics makes them less relevant.
- WWCLEG largely report a lack of evidence for the impact of intervention. This is unsurprising given their rigorous approach. But they then refrain from making difficult political statements on the back of that finding. WWCLEG looks at what should work in the UK more than they report what does work in the UK.
Let’s look at an example and I’ll show you what I mean.
Manchester City stadium
Manchester applied to host the Olympic Games in 1996 and 2000 and hosted the Commonwealth Games in 2002.
Helped by its Olympic bids Manchester got the UK’s largest indoor arena (built for £52m with £38m coming from government and EU grants) and a velodrome (built with £9.5m of government grants). For the commonwealth games it got lots of stuff including a new stadium and a new aquatics centre.
I say got deliberately because of the £110m that the Commonwealth Games cost to host, £73m came from national funds with Manchester paying for just £37m.
So in total, thanks to big sports & cultural events Manchester got £171.5m worth of stuff and only had to spend £37m of its own money. At the same time it was building its tram system, now the largest of any city in the UK, attracting a significant chunk of the BBC, and cementing its position as the UK’s second city. It got most of this too and good on it.
Given this you’d expect Manchester to be an example city in WWCLEG’s reports on both transport and on sports & culture. Right?
I have some bad news. Manchester is mentioned just once in WWCLEG’s transport assessment. It is not mentioned at all in its Sports & Culture assessment. For comparison, London is mentioned 7 and 21 times respectively.
Despite having only just occurred, the 2012 Olympics is mentioned 10 times in the Sports & Culture assessment alone. Manchester’s Commonwealth Games is mentioned nowhere.
Estimating the return on investments in Manchester
It’s pretty easy to start estimating the economic impact of the investment we’ve seen in Manchester. Thanks in large part to their new stadium, now connected to Metrolink, Manchester City were bought by wealthy investors. Their wage bill is now about £200m per year.
To make this a study we need to compare that to another club which we can use as a control. We need one of a similar size and in a similar situation before the intervention. Leeds United, whose wage bill is about £20m, fit the bill perfectly.
So assuming a 50% total income from tax on wages, Manchester City are generating an extra £90m of government revenue per year. Using the standard return period of 30 years you get an extremely rough return on investment ratio of about 16 to 1.
These are extremely rough calculations. If I had a month I could do much better calculations, open my methods up to public debate and wider datasets, and greatly improve the accuracy of my impact estimate. But this would still fall well short of the standard needed to be considered in a WWCLEG report. To do that I guess I’d need a full economics department at a University and about two years. We’re looking at a total cost, per study, of £200,000 at the absolute minimum. One million pounds is a more reasonable estimate and is well above the disposable funds that local governments have to consider impact.
WWCLEG’s reports completely missed Manchester, the best opportunity to study the impact of transport, sports & culture in the UK in the last 30 years. We need to understand why and ask whether a methodology that could not even consider the UK’s second city is useful in helping us make decisions. I’m not sure it is.
What’s the alternative?
I started with three concerns and so I’ll end with the three reasons why I think the problems I've found occured.
- WWCLEG set the barrier of entry too high. This mean they excluded all but the largest and most formal studies. They then failed to notice or warn sufficiently of the huge and obvious gaps this leaves. WWCLEG has been quick to suggest that “there is no evidence that investment generates growth”. Good. But too few people citing its reports seem aware that their methods made finding such a link very difficult. I'm not blaming WWCLEG for that, but it's a problem.
- WWCLEG looked at papers from around the world. It therefore looked at cities which largely pay for transport, sports & culture investment themselves. In the UK this does not happen. Instead the central government is responsible for most such funding. Manchester funded around 20% of the total investment it received. The return on investment from Manchester or another city bidding for transport and cultural investment is therefore five times higher than the return to the UK.
- And so in the end, WWCLEG answers the wrong question. It asks what should work instead of what does work. If UK cities raised their own money and selected their own priorities for investment that would be less of a problem. In the UK they can’t.
I am almost certain that if the WWCLEG were not so heavily concentrated in London they would have seen and highlighted the problems I have with asessing Manchester. I do not believe that this bias is deliberate. Instead it is a beautiful illustration of how investment decisions in the UK are inherently political and accidentally biased.
A city looking for guidance on what works for local economic growth should be aware of this. It should use economics to challenge politics rather than rely on economic arguments alone. Cities should make decisions within the existing political framework rather than within an ideal one. There should be more reports that are designed to assist them do that.
Lastly, cities should be developing their own tools. It makes no sense to do this individually and they cannot afford to do so within the old model of academia, peer-review, and large projects funded from London. That is why I am helping cities build open models of impact. Instead of peer-review they rely on open criticism and input. Instead of presenting overviews to politicians, they involve politicians in the process. They will not meet the gold-standard that academics want but I am hopeful that they will deliver the economic growth this country needs.