Measurement with AI Needs Methodology

I heard from one of our students about how counterproductive AI had been for their measurement development efforts. They were working on a thought leadership-related objective, and their AI prompt was something as simple as “KPIs for Thought Leadership”. AI suggested measures such as sentiment analysis, backlinks, podcast performance, and advisory board requests. None of these elements had anything to do with their strategy but they sounded impressive and sophisticated. Before long, the organization was trying to measure and track sentiment and backlinks without really understanding what to do with the scores. Plus, they had teams launching a podcast, which turned out to be a major initiative, that had little connection to the organization’s actual strategy. One poorly written AI prompt, without the proper strategic context, had created weeks of counterproductive work.

This demonstrates an important strategy/measurement lesson in the AI era: in the hands of someone know knows what they are doing, AI can be magic. When the user doesn’t know what they are doing, it can produce mountains of waste.

Method Matters More Than the Tool

When you follow the basic principles of a disciplined measurement methodology such as the MPRA methodology, with or without using AI, the results can be terrific. The effort (and the AI results) won’t be very useful if you don’t start with the most important context: what results are we trying to achieve and why? Human dialog is needed to form consensus around strategic intent and to manage the related change. Once the team agrees on the intended result of an objective, AI can help analyze the objective based on key principles. Did we describe the desired result in clear, concrete language? Did we focus too much on actions instead of outcomes? Did we merge multiple strategies? Can we objectively assess the result?

Then we can generate dozens of potential measures to consider using the tools we teach in our programs. If we can’t directly measure the success, maybe the logic model, Ishikawa diagram, or a process flow analysis can help identify potential contributing factors to our success. In the past these exercises took hours, but with an intelligently drafted prompt we can now generate dozens of ideas instantly. Then real live humans will need to discuss and evaluate the usefulness of the ideas.

Selecting the right measures will be done by the human team, but the analysis can be assisted by AI prompts that estimate the value of a measure versus the availability of the data. It can help assess how well the measure captures the intended results of the objective, avoids subjectivity or inconsistent interpretation, and is likely to drive the desired behaviors or performance improvements. Again, a human team can determine if the assessment fits the context on the ground.

Conclusion

AI is extraordinarily powerful. It can analyze data, generate ideas, summarize information, identify patterns, draft plans, and produce in seconds what once took hours. But that does not mean it always produces good work. Dumb measurement ideas used to be apparent and now they come shrouded by the polish of well-written consultant speak.

The quality of AI output depends heavily on the quality of the thinking behind the prompt, the judgment applied to the response, and the expertise of the person deciding what to accept, reject, or refine. In the hands of someone who knows what they are doing, AI can be a tremendous force multiplier. In the hands of someone who does not, it can simply help them make mistakes faster.

Check out our upcoming Introduction to KPIs course to learn a practical, step-by-step approach to developing meaningful measures that align with your organization’s strategy.