Mis-Leadership and Metrics

The American military command thought this was like any other war: you searched out the enemy, fixed him, killed him, and went home. The only measure of the war the Americans were interested in was quantitative. . .That the body count might be a misleading indicator did not penetrate the command; large stacks of dead Vietcong were taken as signs of success.

— David Halberstam, The Best and the Brightest

Economists preach that incentives matter, and this is true. But when we preach that incentive systems should be based on quantitative measures, we encourage business leaders and government policy makers to go badly wrong. Misleading statistics cause mis-leadership.

The debate over these practices too often gets channeled into a search for better measures of performance. But this side-steps the deeper problem, which is the illusion of control. Executives in business, non-profits, and government sometimes have too much faith in their ability to direct processes that operate deep inside the organization or the larger society. The statisticians and efficiency experts fail to appreciate the autonomy of human beings.

During the Vietnam War, Secretary of Defense Robert McNamara relied on quantitative measures, including the infamous body count, that showed progress. It turned out that these were the wrong indicators, or perhaps it was the wrong war to be fighting in the first place. The war was a tragedy of mis-leadership

Closer to home, years ago when we moved to a new neighborhood a friend excitedly told us that average test scores at the local elementary school were the highest in the region. It turned out that this was achieved by the school administrators telling weaker students to stay home on test day. The state Board of Education, believing that publicizing test scores would lead schools to undertake real improvements, was engaging in mis-leadership.

For more examples of mis-leadership involving quantitative measures, I recommend a recent book, The Tyranny of Metrics, by Jerry Z. Muller. Also, a recent article by Megan McArdle, Metrics and Their Unintended Consequences, gives still more examples.

Goal Diversion and Corruption

The problems with quantitative measures are well known. One is what Muller calls goal diversion. In a complex situation, such as the Vietnam War, there are many intermediate objectives to be met. Measuring one objective, attrition of the enemy, diverted attention from other objectives, including winning the hearts and minds of the local population.

Another problem is what Muller calls corruption, in which the numerical goal is met by means that subvert its intent. Corruption is illustrated by my example of the school that boosted its scores by telling weaker students to stay home on test days. Another example of corruption is the Wells Fargo fake accounts scandal.

Both goal diversion and corruption played big roles in the financial crisis of 2008. Goal diversion resulted from regulatory “affordable housing goals,” which were intended to pressure Freddie Mac and Fannie Mae to expand mortgage credit to low-income borrowers. But this diverted attention from the goal of safety and soundness at the two mortgage giants. Their regulator should have required the two firms to augment their loss reserves and capital buffers to align with the additional risks that these loans represented.

Corruption was involved in the way that banks used regulatory arbitrage to adopt flimsy capital structures while apparently meeting or exceeding the risk-based capital requirements imposed by regulators. The exotic financial instruments of the pre-crisis period, including “collateralized debt obligations” (CDOs) and “structured investment vehicles” (SIVs) were created to meet the letter of the regulations while totally violating their spirit.

Judgment vs. Formulas

Some of the worst problems of goal diversion and corruption can be reduced by following Amar Bhide’s advice to incorporate judgment rather than relying solely on rigid formulas. For example, bank examiners can assess the purpose of a bank’s use of exotic financial instruments. They can assess the ability of bank executives to understand and control the risks that are embedded in their operations.

In fact, most businesses do not rely on formulas alone to set compensation. Instead, they give supervisors latitude to use judgment in determining bonuses and salary adjustments for the employees who work for them. This is because formulas cannot capture all of the performance goals that are embedded in today’s complex business environment. Supervisors who are closer to the employee are in a better position to weigh various factors, including contributions that are difficult to quantify.

Even when businesses tie compensation partly to formulas, they tinker with the formulas from year to year. The longer they stick with any one formula, the more ways that employees will come up to “game the system.” Every formula is subject to becoming corrupted after it has been in place for a while.

The Illusion of Control

The worst cases of mis-leadership come about when metrics are used to feed the illusion of control. Sometimes, achieving a goal is not within the organization’s power. In that case, the intermediate measures produce arbitrary rewards and punishments, and they lead to behavior that makes things worse rather than better.

For example, raising student test scores nationwide, using policies devised and implemented in Washington, D.C., is not an achievable goal. The overwhelming majority of rigorously-studied educational interventions fail to achieve a meaningful, lasting effect. Yet under the No Child Left Behind Act and the efforts that followed, schools are assumed to have solutions at hand for educating every pupil. They are measured and graded according to quantitative measures of performance. In short, Washington acts as if schools have exquisite control over student learning as reflected in (un-corrupted) test scores, even though research suggests otherwise. Politicians operate with the illusion of control.

Instead of holding teachers accountable to a centralized statistical office, I believe that teacher evaluation should be undertaken by peers, principals, and parents. Test scores can be unreliable indicators for many reasons. Parents can readily assess whether a teacher is working conscientiously and effectively.

I have only started reading Nassim Taleb’s latest book, Skin in the Game. But I gather that he would make the point that when it comes to children’s education, parents have real skin in the game. If I allow my child to try to learn from a bad teacher, I suffer from that. If a bureaucratic system fails to remove a bad teacher, the designer of the system does not suffer consequences.

What is true in education is also true in other important realms. Statistical measures can help in certain well-defined activities, like running a baseball team. But in complex realms, including education, health care, and financial regulation, the best judges of performance are constituents who are close to the local situation (with skin in the game), not remote statisticians.

Seeing Like a State

Data collection and statistics have always been part of the process that James Scott calls Seeing Like a State. But no matter how intently the officials stare, their vision is blurry and out of focus.

Regardless of what may take place in the minds of leaders, human beings are autonomous and difficult to control. Successful business executives understand this and learn to live with limitations on their ability to fine-tune organizational behavior. They know that they cannot measure performance with precision or design a perfect incentive system. Rather than stick rigidly to formulaic systems, successful executives encourage subordinates to exercise autonomy and judgment.

In government, trying to rely on judgment raises the issue of favoritism. There is a bias toward using formulas to determine compensation, and there is a bias against flexibility in the design and especially in the implementation of those systems.

In the public sector, there is an expectation of consistency and uniformity. This tends to run counter to giving autonomy to low-level managers.

Government officials often do not work with realistic goals. Politicians, unfortunately, are rewarded for sounding confident about their ability to control the uncontrollable. The economists that appeal most to politicians are those who claim to offer solutions, and economists’ solutions typically involve metrics and incentives. But the statistical measures do not move organizations toward the desired objectives. Instead, they produce harmful unintended consequences.

Because problems like education and financial regulation are complex, it probably would be best to approach them with greater flexibility than is typical in private business. But the nature of government works in the other direction. In this context, rewarding and punishing people based on metrics is mis-leadership.