This month Steve Wilson, founder and Director of Standpoint Decision Support, lends his experience with data analytics and legal regulation to unpack how regulators can better harness data for more effective regulation.
By now everyone is familiar with the wave of “big data” that is invading nearly every aspect of private and public life. There are “recommendation engines” implemented by companies trying to sell to us, “nudge” units in governments prompting us to pay taxes, and exercise and sleep trackers combating our poor habits.
Legal regulators have always collected a large volume of data from their members, and about their members’ behaviour, because collection and review of this information is central to a regulator’s mission of ensuring ethical conduct and promoting best practices. But like other organisations, law regulators are beginning to ask the question: is there more that we can be doing to leverage our data and better fulfill our mission? The answer is probably ‘yes,’ but there is a right way and a wrong way of pursuing this goal.
First, the wrong way, which can be summarized as working from data to theory. This usually manifests itself in projects asking the following kinds of questions:
-
- What meaning can we extract from the data we have?
- What other data can we collect?
- How can we use the data we have to prove we’re implementing the best strategy?
There are plenty of consultants who will happily charge handsome fees to help clients pursue these kinds of open-ended questions, but organisations that have successfully implemented evidence-based management practices work from theory to data. That is, by starting with the identification of key issues/questions/problems, and then implementing the data collection, analysis and reporting procedures necessary to support improved decision-making. There are many facets of law regulation to which the theory-to-data approach can be applied. Here we focus on one that we think could generate the most significant benefits for regulators.
An old adage in marketing is that “half of all advertising is wasted; we just don’t know which half!” This concept can be extended to many other domains, where multiple actions are taken by organisations to improve outcomes, but rarely are the actions analysed or monitored to assess their individual effectiveness. This is certainly true in law regulation, where regulators enforce rules, provide services and encourage practices, all to promote better outcomes for their members and for the public. But how often are these various initiatives evaluated to determine whether they are having their intended effects, either individually or in aggregate? From our experience, quite rarely.
Legal regulators are not alone in this; the failure to properly monitor outcomes is a long-standing problem, primarily because it isn’t easy to assess the effectiveness of interventions in complex systems. It takes foresight and planning, and success is not guaranteed. And although the tools of “big data” can help, the experience and expertise of an organisation’s leadership and staff are crucial.
We recommend that organisations use adaptive management to address their key issues and questions. Adaptive management is an iterative process that follows scientific methods to generate reliable inferences about the effectiveness of management actions. For example, imagine a regulator is planning to implement a new program aimed at improving lawyer performance. The following steps comprise the adaptive management process:
- Asking the right question, framed as an hypothesis: by implementing program X, we expect to see changes in Y, which is an indicator of our desired outcome Z. This is not a trivial step, because there could be reasons we might expect to see changes in Y that are unrelated to the implementation of X. There are several approaches to mitigating these “confounding” factors and doing so is critical.
- Acquiring the necessary data, often relying on both retrospective and newly collected information. Data collection can be compromised by various biases and needs to be planned carefully.
- Analysing the collected data. The sophistication required at this step will vary, depending on the complexity of the question and on the underlying data-generated system.
- Acting on the results of the analysis, by both reporting plain-language outcomes to stakeholders and by implementing learnings through improved policies and procedures.
- Monitoring performance on an ongoing basis to ensure that expected outcomes are being achieved. Monitoring often leads to additional questions that begin the adaptive management process anew, creating a feedback loop of continuous improvement.
Note that in this workflow, most analytics consultants will generally assist only with acquiring and analysing data. The other three steps are equally critical to success and depend heavily on organizational leadership and internal knowledge and capacity.
The emerging world of big data analytics is providing opportunities to improve law regulation, but it is important to understand the limits of the new tools and the role of proper planning and institutional knowledge.