The What Works Network is a UK central government initiative to gather evidence, analyse it, and present it to policy makers so that they can make better decisions. I absolutely love the concept.
There are problems — six of the seven centres are in London for example — but overall it’s a great idea. Government is listening and the centres have produced some really interesting and influential outputs.
My worry is that a few of those outputs are getting too influential. I’m going to focus on the What Works Centre for Local Economic Growth (WWCLEG) because it’s the only centre whose output I follow. I’ll share my understanding of the methods they’ve used and suggest areas where I think that their work could be interpreted better.
Before I start I want to be really clear. I’m not having a go. WWCLEG is really important and has done a good job. This is about improving that work, improving how the existing work is used, and maybe opening this area of evidence-led decision making up to a much wider and possibly more agile set of organisations and methods.
WWCLEG is based at the London School of Economics and is run largely by academics from that institution. They work in collaboration with Arup and London-based think-tank Centre for Cities. It aims to answer questions like “does investing in transport infrastructure generate economic growth?” and “does hosting large cultural and sporting events generate economic growth?”.
So far, so good. These are extremely important questions where lots of people make huge claims without any basis in evidence. Specifically on those questions, WWCLEG have done reports on Transport and Sports & Culture. Read them, they’re accessible, well-presented, and well-evidenced.
The problems start when we look at how they went about answering those questions and when we look at exactly what question they answered. I have three concerns here,
Let’s look at an example and I’ll show you what I mean.
Manchester applied to host the Olympic Games in 1996 and 2000 and hosted the Commonwealth Games in 2002.
Helped by its Olympic bids Manchester got the UK’s largest indoor arena (built for £52m with £38m coming from government and EU grants) and a velodrome (built with £9.5m of government grants). For the commonwealth games it got lots of stuff including a new stadium and a new aquatics centre.
I say got deliberately because of the £110m that the Commonwealth Games cost to host, £73m came from national funds with Manchester paying for just £37m.
So in total, thanks to big sports & cultural events Manchester got £171.5m worth of stuff and only had to spend £37m of its own money. At the same time it was building its tram system, now the largest of any city in the UK, attracting a significant chunk of the BBC, and cementing its position as the UK’s second city. It got most of this too and good on it.
Given this you’d expect Manchester to be an example city in WWCLEG’s reports on both transport and on sports & culture. Right?
I have some bad news. Manchester is mentioned just once in WWCLEG’s transport assessment. It is not mentioned at all in its Sports & Culture assessment. For comparison, London is mentioned 7 and 21 times respectively.
Despite having only just occurred, the 2012 Olympics is mentioned 10 times in the Sports & Culture assessment alone. Manchester’s Commonwealth Games is mentioned nowhere.
It’s pretty easy to start estimating the economic impact of the investment we’ve seen in Manchester. Thanks in large part to their new stadium, now connected to Metrolink, Manchester City were bought by wealthy investors. Their wage bill is now about £200m per year.
To make this a study we need to compare that to another club which we can use as a control. We need one of a similar size and in a similar situation before the intervention. Leeds United, whose wage bill is about £20m, fit the bill perfectly.
So assuming a 50% total income from tax on wages, Manchester City are generating an extra £90m of government revenue per year. Using the standard return period of 30 years you get an extremely rough return on investment ratio of about 16 to 1.
These are extremely rough calculations. If I had a month I could do much better calculations, open my methods up to public debate and wider datasets, and greatly improve the accuracy of my impact estimate. But this would still fall well short of the standard needed to be considered in a WWCLEG report. To do that I guess I’d need a full economics department at a University and about two years. We’re looking at a total cost, per study, of £200,000 at the absolute minimum. One million pounds is a more reasonable estimate and is well above the disposable funds that local governments have to consider impact.
WWCLEG’s reports completely missed Manchester, the best opportunity to study the impact of transport, sports & culture in the UK in the last 30 years. We need to understand why and ask whether a methodology that could not even consider the UK’s second city is useful in helping us make decisions. I’m not sure it is.
I started with three concerns and so I’ll end with the three reasons why I think the problems I've found occured.
I am almost certain that if the WWCLEG were not so heavily concentrated in London they would have seen and highlighted the problems I have with asessing Manchester. I do not believe that this bias is deliberate. Instead it is a beautiful illustration of how investment decisions in the UK are inherently political and accidentally biased.
A city looking for guidance on what works for local economic growth should be aware of this. It should use economics to challenge politics rather than rely on economic arguments alone. Cities should make decisions within the existing political framework rather than within an ideal one. There should be more reports that are designed to assist them do that.
Lastly, cities should be developing their own tools. It makes no sense to do this individually and they cannot afford to do so within the old model of academia, peer-review, and large projects funded from London. That is why I am helping cities build open models of impact. Instead of peer-review they rely on open criticism and input. Instead of presenting overviews to politicians, they involve politicians in the process. They will not meet the gold-standard that academics want but I am hopeful that they will deliver the economic growth this country needs.