The KPI Black Market: The Informal Metrics Organizations Actually Trust
The Official Market
Modern organizations are obsessed with measurement. Pop into virtually any executive meeting, and you’re likely to find glowing dashboards, reports, infographics, charts, and scorecards packed with performance metrics.
Revenue growth, customer satisfaction, engagement, productivity, utilization, retention, cycle time: if it can be quantified, it’s probably being tracked.
This is the Official Market, or what organizations buy and sell in an attempt to take the inherent complexity of business and turn it into something understandable: reports, scorecards, dashboards, intelligence platforms, and Balanced Scorecards.
Business needs complexity to transform into something manageable. Leadership must see into what happens everywhere in the organization, and metrics give teams a common language to express how they perform. Formal metrics are the right tool for holding employees accountable and for allowing leaders to assess the performance of individuals or teams against others over time.
Yet the potential harms of overreliance on formal measurement systems aren’t merely abstract or as far-fetched as many make them out to be.
Natlia Cuguer-Escofet, a researcher at the University of Pompeu Fabra, and Josep M. Rosanas at Universitat de Barcelona analyzed a set of cases in which performance management systems, when implemented rigidly, led to unintended outcomes, including cases from Spanish banks in the period leading up to the 2008 crisis.
In one such instance, a manager who was resistant to escalating loan-making practices was moved out of a position where loan decisions could be made, despite being given a promotion (in the form of an increased salary and improved office space).
In another instance, a board member who was hesitant about an asset’s value was reluctant to express his reservations because the organization’s incentive systems largely tied everyone’s performance to profit. The performance systems were working as they were designed to, but their output was discouraging decision-makers from making professional decisions when they were most needed.
The takeaway from these scenarios isn’t that measurement is itself flawed; it’s that every organization’s performance system will eventually hit a ceiling where it can’t foresee every contingency. When they become overly committed to using objective indicators, organizations risk inhibiting the human intelligence that might warn them of a problem before it shows up in the results.
The crux of the problem isn’t so much a reliance on measuring performance; it’s the assumption that measurable equals significant.
Organizations are often obsessed with measuring metrics, even though those same metrics sometimes do not truly matter to an organization’s success. These metrics aren’t designed to provide a clear picture of why something is or isn’t working; rather, they simply demonstrate what is going on.
They aren’t about showing how frustrated customers are; instead, they show that the satisfaction level has fallen. They aren’t about the employee turnover going up but about what has led to employee discontentment over time. An organization’s ability to identify the causes of declining numbers is critically important, yet metrics cannot illustrate the complexities driving performance from the bottom up.
It can show what must be seen, not what must be fathomed. It can show performance that is visible.
This weakness is exacerbated, in many cases, by the fact that metrics are, by their very nature, selective. Each KPI necessarily prioritizes some aspects of performance while overlooking others. Organizations use metrics to measure performance based on what they perceive as critical, yet the business environment and consumer expectations require new perspectives. The KPIs organizations rely on may therefore cease to align with the reality on the ground.
It is simply a matter of fact that metrics are better indicators than drivers. When the system measures certain behaviours and outcomes, employees quickly adapt by doing what the system wants them to do. When employees are measured on customer service call time, for example, they learn to hang up with customers as quickly as possible rather than solve their problem. Metrics lead us to manipulate an organization’s output through what we measure, even if what we measure isn’t indicative of success.
“The problem isn’t a measurement one; it’s a knowledge one. You can know the velocity; you just don’t know where you’re headed, and therefore you just don’t know what to do, which makes managing impossible.” – Jeff Bezos
This system of performance relies upon measurement for decision and action-taking but neglects the human aspect; instead, it relies on information that is already visible or reportable. The challenge is that all of this is usually evident on a dashboard if it is tracked or measured.
With that said, not all aspects of the performance in the workplace are quantifiable:
- Trust is hard to measure
- Honesty cannot be quantified
- Creativity or foresight doesn’t have to be demonstrated on a chart
These are not all reflected in The Official Market, as every metric, KPI, report or business scorecard makes choices about what’s relevant and what isn’t.
Sadly, by focusing solely on what we can readily identify as critical and important, many organizations inadvertently start to devalue or even ignore areas they cannot easily quantify. That, it has been said, means the information in their reporting systems may be missing valuable pieces or even be flat-out misleading.
One example is a business intelligence system that tells people how busy employees were in the office (measured by time spent at the desk, use of specific tools, etc.) but does not measure the outcomes of that work. This system has become completely removed from the actual outcomes that would determine whether employees were actually working effectively or not.
In such instances, organizations become overly dependent on such formally measured criteria and risk suppressing human judgment or observation that would otherwise point them toward a problem early on.
The Black Market
There’s one in every organization.
It might not show up in your year-end results. It might not be mentioned in a quarterly review. It definitely will not be in your executive dashboard, but nearly everyone in the company knows it.
This is the KPI Black Market; this is where you would go when your formal measures do not tell the same story. The name is controversial, but it should not be when people search for additional data to navigate a complex business. If an organization goes to great lengths, many beneficial ideas may fall outside measurement standards.
The Conversations That Never Make the Dashboard
Companies have invested significant resources over the past few years in business intelligence tools that afford a live view of operations. However, much of an organization’s most useful intelligence still travels via conversation.
- A sales leader hears multiple account managers mention the same customer pain point.
- A product leader notices an increase in “what is that for?” type questions about a new feature.
- A team lead finds conversation in their team’s hushed post-all-hands meeting.
- A customer success manager starts hearing unusually similar wording in otherwise unrelated client calls, hinting at a shared frustration that hasn’t been logged anywhere yet.
- A regional manager notices that high performers are suddenly asking more “confirmation” questions instead of making autonomous decisions.
- A project lead observes that status updates remain technically positive, but the tone of delivery shifts: shorter messages, fewer details, less narrative confidence.
- An HR partner hears recurring “soft exits” in development conversations – people talking more about uncertainty, optionality, or “keeping an eye on things” rather than commitment.
Those aren’t standard metrics, but they often show trouble before it hits the Profit & Loss (P&L).
That’s partly why leaders place so much importance on informal conversation – it’s where emerging signals like doubt, disappointment, enthusiasm, and apprehension get aired while they’re still in their most formative (and useful) stage.
Once a signal is a metric, it’s already past the critical inflection point. That is due to the fact that dashboards chronicle what happened, while conversations signal what’s about to happen.
We write down and archive at an unforeseen speed, yet much of our knowledge is often contained…elsewhere. That knowledge often moves through the Black Market, with almost blinding celerity.
The Mental Dashboard
Try asking an experienced sales leader what will make the quarter miss your target. They often start with “I have a feeling.” It’s the kind of thing a data scientist will probably break out maniacally in a feverish rash at the sound of it.
How can they predict they might miss when the company invests millions in data and analytics to give you objectivity?!
However, the data science in judgment and forecasting actually supports this kind of intuitive forecasting: experts use intuition often not at random but rather to detect patterns that may not show up explicitly and may even be unable to be easily and systematically articulated, due to experience (e.g., having interacted with customers, products, markets, negotiations, and company stakeholders), which can be more sensitive to some cues than others.
You might experience it as a feeling or a sense:
- A salesperson feeling the heat because customer engagement seems “off” but has not yet been captured by metrics.
- A regional manager in your organization who believes they sense unusual nervousness in the sales reps during customer interactions.
Customer success may note that the typical post-demo and pilot behaviour among clients has changed slightly, but it is not yet affecting metrics such as engagement and churn. In these kinds of instances, they are not officially being recognized by your data platform.
Yet these sorts of signals often influence forecast judgments, however indirectly. Leaders, in essence, operate with two dashboards: one that they see on their screen and another that resides in their head.
One is evidence of what is happening. The other is the interpretation of what’s happening. Neither works well without the other.
The Spreadsheet Nobody Talks About
Perhaps one of the most unaddressed elements of organizational life is the presence of shadow forecasting mechanisms.
Officially, there’s an organization’s forecast. Unofficially, there often exists a second forecast, which may exist in the form of an individual’s private spreadsheet, in an individual’s notebook, or through individual or team discussion.
It is likely, in some form, that this meeting has been heard in every organization where one exists.
- The company forecast is presented.
- The numbers look perfectly healthy.
- Then inevitably someone pipes up, “OK, but what do we actually think?”
- The numbers look perfectly healthy.
The line dividing the Official and Black markets is drawn with that phrase. The Official forecast might be the organization’s most formal assessment, but the Black Market forecast often represents a compilation of individual experience, customer issues, the news and anything else that doesn’t easily lend itself to tabulation.
It’s curious that shadow forecasts don’t necessarily always compete directly against official outputs. Indeed, they can arise as employees try to circumvent gaps they see in the official mechanisms. The fact of there being a spreadsheet doesn’t necessarily matter, since it is the quest for a depiction of reality that people believe in.
Tribal Knowledge and Unofficial Indicators
Arguably the hottest currency in the KPI Black Market is tribe experience. Most organizations have individuals who seem to be aware of certain things well before the rest of the population becomes aware of them. Those individuals understand which projects are real and which generate polished-looking status charts. They can usually predict the top truly unserved and unhappy customer base even before official complaints surface.
Such employees know which operational hazards warrant attention, even when they do not appear in risk analyses. What’s truly fascinating is that these fellows often don’t even have access to data; however, they have contextual gut feelings. By virtue of experience or informed hunches, they understand and see patterns that systems simply can’t capture.
These people remember what happened last time. They recall the anger, the shouts, the boasts, the merriment, or the frustration. They are living archives, in a sense. As such, organizations often defer significantly to individuals who cannot effectively translate the value of their insights into metrics, yet that value is very much there.
This then creates an interesting paradox.
On the one hand, companies may champion objectivity; on the other hand, in uncertain environments, they often turn to sources of experience who, by their nature, are not subject to objective measurement systems. Similar principles are evidenced in how folks make decisions in the informal universe on a day-to-day basis.
- Managers observe how quickly answers are transmitted for questions and inquiries.
- Account teams pick up on the customer’s emotional tone rather than on official customer satisfaction reports.
- Product team leaders keep their eyes on the number of surprises.
- Executive team members will note when the “unhappy camper” stops raising their objections.
These signals often aren’t included on charts but have a tremendous impact on decision-making, at times having a significantly greater impact than the official scores themselves.
Why the Black Market Exists
It may be tempting to see these informal arrangements as proof of the ultimate failure of formal measurement. This is a faulty assumption that relies on a complete misunderstanding of the very premise. The existence of the KPI Black Market signals that organizations are, ultimately, human systems operating in contexts far more complex than can ever be fully captured by numbers.
Dashboards cannot account for every variable. KPIs cannot enumerate every risk. Reports cannot portray trust, morale, judgment, intuition, confidence, or culture. When people and groups try to find order in increasingly chaotic surroundings, it is natural that they create complementary information systems – the KPI Black Market – that support and backstop formal systems. The KPI Black Market is thus not a conspiracy against data, but a very reasonable response to its ultimate shortcomings. It is a natural evolution of a most logical process.
Perhaps the most important irony is that most organizations already rely upon the inputs of this unrecorded channel: they simply do so informally and under the table.
The highest-trusted and most timely signals usually originate elsewhere – between peers, during hallway discussions, through personal observation, or based on embodied tacit knowledge. The KPI Black Market is more prevalent in complex environments where reality is perennially richer than our metrics, and, more generally, in organizations that have simply done too poor a job of creating formal indicators.
What Should Leaders Do About the KPI Black Market?
The existence of the KPI Black Market does not, of course, suggest that companies should discard their dashboards, scorecards, or formal reports. Au contraire!
Formal measurement is crucial if organizational performance is to be consistent and comparable, and if accountability is to be meaningful rather than arbitrary, and so much so that the Official Market is an essential part of organizational life.
The problem isn’t that organizations formally measure performance; it’s that they treat formal measurements as complete representations of reality rather than partial ones.
Great leaders recognize that the best dashboard or scorecard cannot do their thinking for them, but can help them think, and that, in addition to the question “What does this metric say?”, a second question needs to be asked.
“What’s missing from this metric?”
A necessary shift in focus leads to the treatment of signals and the observations of employees as information & value, not noise. The aim here isn’t the wholesale abandonment of measurement, but the supplementation of metrics by insight.
A similar approach can be found in the management literature, and the argument has long been made that formal management controls necessarily contain gaps that must be filled by managerial judgment, a concept of “informal justice”.
In essence, such judgments may allow us to question the validity of a metric because it has not kept pace with changing circumstances. Perhaps the easiest way in which to undertake a measure of diagnosis is for a team of leaders to take the time to ask management to list all of the things that management considers important, and then see what doesn’t appear on the board.
Final Thoughts
It is increasingly common to portray organizations as rational, analytical, almost-organic beings in which decisions are data-driven, and metrics are king.
To a certain extent, this is true. Most of them are, indeed, social entities – powered by the experience, intuition, confidence, and understanding coming from their members. Such a sentiment would be historically true as well, as it was the case well before dashboards existed – managers trusted their intuition and vision. Long before business intelligence platforms were born, people discussed and understood their environment to proceed forward, even when uncertainty loomed like an overcast sky.
Although our reporting tools now offer unprecedented visibility, this doesn’t deny the need for those implicit ways of leading teams to progress. Frankly speaking, they simply shouldn’t impede this.
It would be a naive mistake to assume all critical variables can be measured, as the key predictors of an organization’s success are sometimes elusive to quantification. These signals originate from talks, gut feelings, interactions, observations, and events, which never exactly translate onto a metric dashboard.
However, leading companies leverage both, sometimes in equal measure, and often to great success. A dashboard illustrates the past, while those close to the business can provide current-state insights and often foresight.
This is the key takeaway one should derive from the KPI Black Market. The real value is found where the most trustworthy predictors remain off the official dashboard.
**********
Looking to strengthen your expertise in KPI design and performance measurement? Gain practical knowledge, proven methodologies, and globally recognized certification through The KPI Institute’s Online Certified KPI Professional.
