Written by Alan Hudson and Parker Essick
We are not epidemiologists, or behavioral scientists. Neither can we claim any expertise in modeling the potential consequences of policy actions to inform policy design and implementation. But we do have lots of experience of exploring whether and how data and learning can support the design and implementation of policies to address complex challenges relating to corruption and the use of public resources (see our strategy and our recent submission to the World Bank on “data and development”). Data, theory and learning are part of our DNA. Maybe models should be too.
Complexity, adaptation and models
As we’ve adapted our work to the context of COVID-19 and sought to keep up to speed on the many things that have been written about COVID-19 and governance, we’ve been interested to see discussions about how to respond to the new challenges bring to the fore a number of issues that have been the focus of our work over the last few years. These include issues relating to the complex nature of the challenges, the need for adaptive responses, and the value and limits of models in informing the design of those responses.
First, much discussion has focused on the complex, systemic and globally interdependent nature of the challenges posed by COVID-19, and the value of a systems perspective in addressing those challenges. Pieces that have caught our attention include: COVID-19 means systems thinking is no longer optional, by Seth Reynolds; A systems perspective on the coronavirus: If the Wire was about COVID-19, what would the seasons be?, by Monalisa Salib; and, Why a systems response to COVID-19 is critical, by Olivia Leland.
Second, there have also been a number of interesting contributions that have focused on the need for adaptive and learning-centered responses. These include an excellent paper on adaptive leadership by Ben Ramalingam, Leni Wild and Matt Ferrari; Smart Containment with Active Learning: A Proposal for a Data-Responsive and Graded Approach to COVID-19, by Tahir Andrabi and colleagues; Seeing pandemics as complex adaptive problems, by Peter Harrington; and, A call for a new generation of COVID-19 models by Alex Engler.
Third, there have been many pieces that have explored the role that science, data and modeling can play in deciding amongst policy options, while also highlighting the complicated path which leads from science and data (whether that’s data about the past, or data derived from modeling the future), through values and politics, to policy, practice and results. Pieces that we have found particularly interesting include: Science isn’t a clear-cut pandemic guide, by Therese Raphael; There’s no such thing as just “following the science”, by Jana Bacevic; The truth about scientific models, by Sabine Hossenfelder; The mudfight over “wild-ass” COVID numbers is pathological, by Roger Pielke Jr.; Role of modelling in COVID-19 policy development, by Emma McBryde and colleagues; The problem of modelling: Public policy and the coronavirus, by Paul Collier; 10 Tips for Making Sense of COVID-19 Models for Decision-Making, by Elizabeth Stuart and colleagues; Sharing models for COVID-19: Guidance and tools, by Fionntánn O’Donnell; and, a super-interesting piece on Modeling the pandemic: Attuning models to their contexts, by Tim Rhodes and colleagues.
For those with more time, and enthusiasm to learn about modeling, we recommend Computational modelling: Technological futures, by the UK Government’s Office for Science, Council for Science and Technology. And for some additional insights into how models are, and might, be used in different parts of the world, see: South Africa’s use of COVID-19 modelling has been deeply flawed. Here’s why, by Seán Mfundza Muller; How to forecast outbreaks and pandemics: America needs the contagion equivalent of the national weather service, by Caitlin Rivers and Dylan George; Newsom Announces New COVID-19 Modeling Website, Open-Source Tools For ‘Citizen Scientists’; and, if you can get beyond the paywall, Our modelling must be the best as Britain comes out of lockdown, by Nigel Shadbolt.
Exploring the value and limits of modeling
Inspired by the attention given to models and modeling in relation to COVID-19, and struck by the absence of modeling approaches in the governance and development space with which we are most familiar, we thought it would be worth exploring the role that models can play in the design and implementation of public policies relating to complex, systemic and social challenges.
To this end, we plan to convene a small group of people who work on and around issues including data and evidence, adaptive development, modeling, and public policy, to explore some key questions:
- Potential for modeling impact of governance-related policies: How useful and feasible would it be to deploy models that enable the exploration of policy impacts, to inform the design and implementation of policies – or policy commitments in the Open Government Partnership process, for instance – to address governance-related challenges, such as those relating to corruption and the use of public resources?
- Lessons from other spheres and systems: What lessons and insights does the use of models to explore the possible impacts of different policies in different spheres and systems – for instance, in relation to economic systems, ecological dynamics, traffic flows, and epidemiological crises – hold for efforts to make use of models in relation to governance-related challenges?
- Models and their use: What sorts of models might best support progress along the pathway from data, through politics and policy, to impact, as regards challenges relating to governance, corruption and the use of public resources? What implications does the fact that the effectiveness of policy in these areas depends upon relationships, trust, legitimacy and compliance have for how such models would need to be developed and used?
Our discussions may conclude that modeling the potential impacts of different policies is not a useful or feasible way to go in the governance and development space. Or, that there is a wealth of experience about the value and challenges of modeling in relation to public policy that we need to get up to speed on. Or, that exploring the role that models might play in informing adaptive responses to complex social challenges – and governance-related challenges in particular – is a rich seam to explore.
We have an open mind.
If this piques your interest, please drop us a line. We’d love to learn along together!
¹ Models take many many forms. In this context, we are referring to computer-based dynamic representations of reality that combine theory and data to explore possible future scenarios by changing the parameters and assumptions which drive the model’s dynamics. We are particularly interested in models that include actors and behavior as key steps on the pathway from policy to impact, rather than ones that focus on relationships between more abstract variables.
I appreciate these reflections, Alan, and the useful collection of links. Is there an illustrative case study that shows how modeling may be useful to governance work? I could imagine it going in so many directions that it would be helpful to narrow down the parameters. For example, are you thinking about modeling cost savings in procurement processes by looking at data from previous years to model what could be gained from preventing faulty contracts in the future? Or maybe you’re thinking about modeling the potential increase in learning outcomes if the Tanzanian government were to adopt the pay incentives for teachers advocated by Twaweza?
It either case, my first impression is that there is a tension between Global Integrity’s focus on politics as an under-appreciated dimension of development and modeling as a typically apolitical approach to identify the most salient variables to influence outcomes. For example, modeling can offer estimates about the spread of the disease with or without face masks. But it doesn’t have much to offer us in understanding the political dynamics as to why wearing a face mask would become a partisan issue, or how political leaders address the trade-offs between economic and health damage.
I’m looking forward to seeing what comes out of your group discussions and thanks as always for sharing so openly.
Thanks for the feedback, David.
In terms of your main point, yes, there can certainly be a tension between models (follow the science!) and politics, as we’ve seen so clearly in the US and the UK. But there is scope for models to be used in ways that inform policy decisions that will inevitable be both about what works (the technical) and what is valued (the political). Of course, in particularly challenging contexts, getting to that sort of constructive combination of models and politics is far from easy!
A number of the pieces referenced get into these issues. And your questions helpfully add to the agenda for discussion.
This, from Open Ownership, provides a good idea of the sort of thing I’m thinking of.
Subject to capacity, I plan to collate more, and have interesting discussions scheduled with a number of colleagues, in, and out, of the governance and development space (including Johnny West, below).
Further reflection … your concern re the tension between a model-focused approach and a politics-focused approach seems to be a variant of a tension between approaches that focus on data, and approaches that focus on politics.
I imagine you would agree that we need policy processes that combine data (inc. data from models) and values? Data/models might help to tell you how to get to a particular outcome. But the data/models can’t tell you what value to place on that outcome. [they also don’t get into the sphere of emotions, which is another important area, and at least as much a driver of politics as are facts].
I’m reminded here of Ruth Levine’s p 2017 piece on the moral case for evidence, where she wrote: ‘Empirical analysis is not a substitute for the value judgments that inform a theory of justice in any society. But empirical analysis is an essential complement to those value judgments, helping to turn the “what we believe” into the “what we do.”’
As such, the many many years of discussion and action re evidence-based, or evidence-informed, policy – lots of which Hewlett has supported – seem very pertinent in thinking about the value and limits of models.
Thanks, Alan. Again, very interested if there is a case study from the “governance for development” field that illustrates your thinking.
I’ve not come across one, which is interesting in itself. The closest I’ve found is the open ownership idea, referenced above. I will share what I find, of course. I think the search might be more fruitful in relation to particular sectors and service delivery areas, where there’s sometimes greater clarity about outcomes, mechanisms, behavior and actors (that is, theories of change that are more clearly specified).
Hi, this sounds really interesting and is something that UNDP here in Afghanistan is exploring albeit in a tentative manner from an economic perspective rather than the broader more qualitative perspective of an integrated approach required for more balanced socio-ecological development. An energy programme, for example, is modelled in terms of its impact on the economy not in terms of the capabilities that it fosters once people are connected to an electricity supply. I am not sure if I am the right person to involve if you want a small gp of specialists, but I am doing a Masters in Systems thinking and have 25 years experience in international development so let me know if my contribution may be of interest
Thanks Keith. I’ll add you to the list!
I’ve been working on exactly this topic since March… starting from my own little corner of practise – project modelling in natural resources – but since Covid looking across all areas of public policy where modeling is used (and as the UK report you link to shows, modelling is taking hold in nearly every aspect of public policy fast). It’s a hugely rich area! One observation: I believe the confusion about what models can and can’t do arises – understandably – from a lag in awareness of how the use of models has broadened so much in recent decades. Modelling was used for centuries – using the same techniques as computational modelling with orders of magnitude less calculations and data – for ballistics, irrigation, astronomy, land surveying. This continued into the first generational of computational modelling – think Los Alamos, NASA, and traffic network analysis. These are high validity environments, largely because calculations, even if complex, are less likely to be self-referring… The results were what we could call “logical positivist miracles” – a model can put Armstrong on the Moon! In more recent decades, modelling extended into systems with massive feedback loops – economics, epidemiology, long-term climate change impact – and that’s when they morphed considerably in terms of how they are set up, and what are good outputs and outcomes. But the popular understanding – including that of all the current generation of decision makers in the policy space – has stuck essentially with the earlier conception. Hence the self-serving formula of politicians who claim to just be “following the science”. Boffins make models, we just follow ’em. Which is a total misread (since Covid is squarely in the second low validity class of models). Anyway… would love to chat!
Very exciting to hear that you have been exploring similar issues, Johnny. Looking forward to comparing notes as part of a wider discussion.
This is great to see Alan. I’d suggest that a couple of interesting topics are the use of actor-based change models within systems paradigms (see article in the American Journal of Evaluation “The Actor-Based Change Framework: A Pragmatic Approach to Developing
Program Theory for Interventions in Complex Systems”, which includes a governance example from Nepal) and efforts at modeling around the spread of social norms or the diffusion of innovation might be interesting sources to review. My suggestion with respect to the question of data and politics would be to try to convert politics into data, which investigations around social norms and opinion surveys can begin to do, rather than keep them as distinct lines of inquiry.
Governance definitely suffers from low validity as a scientific space compared with engineering, but I see the value of working on systems-based models less as the computational power and more around the improved accuracy that modeling requires of us. Very cool initiative!
Thanks David – your comment gave me plenty of food for thought.
I agree that the value add would not be computational power; this is not about models spitting out solutions. Instead, it would be the increased clarity that even trying to model policy implications would require and encourage. I also think models can provide a great focus for collaborative efforts to explore how change happens and what might be done, by whom, to accelerate particular sorts of changes.
And yes, Actor-Based Change frameworks do seem to be a step towards the sort of clarity that modeling requires, and that would – modeling or not – contribute to clearer theories of change, more insightful evidence, enhanced learning, and greater effectiveness and impact. This is something we are exploring as part of a project that involves designing and piloting a participatory systems mapping approach to addressing governance challenges relating to health.
I also appreciate your comment re not treating data and politics as distinct lines of enquiry. Indeed, perhaps the key feature of our revised strategy is its sharper focus on exploring whether and how data and learning can lead to behavior changes, when those behaviors are shaped by power, incentives and norms. It’s also something we addressed from a slightly different angle in our recent submission to the World Bank’s World Development Report for 2021 on “Data for Better Lives”. /2020/07/14/wdr2021/