As some companies become skeptical about the returns on Generative AI initiatives, many are faced with the crucial task of evaluating return on investment (ROI). This article explores the multifaceted nature of AI ROI, offering a comprehensive framework for assessment.

Understanding the Dual Nature of AI Returns

Generative AI's impact on an organization can be categorized into two main types of returns:

Direct Returns

• Time Savings: Automation of repetitive tasks, such as AI-powered document processing, knowledge augmentation, and discovery, can significantly reduce manual labor hours.
• Productivity Increase: Many workflows shift from pure manual execution to data-backed decision making. Productivity increases as employees do not need to spend time aggregating data, but rather are able to make informed decisions (human in the loop).
• Cost Savings: Lower-skilled labor can be retrained to provide value in other functions rather than focusing on mundane workflows that can be outsourced to an AI workflow.
• Revenue Increase: New personalized services or products can open up additional revenue streams.

Indirect Returns

• Better Experience: Improved customer and employee satisfaction often results from personalized interactions and processes.
• Skills Retention: The presence of cutting-edge AI technology can attract and retain talent that values innovation. Inaction in adopting new technology can lead to frustration and loss of talent.
• Agility: Building AI capabilities prepares the organization for future opportunities and challenges. As with every emerging technology, GenAI needs time to manifest its economic potential. Starting now with small initiatives prepares the workforce to adapt to future changes.

Common Pitfalls in ROI Evaluation

To accurately assess AI ROI, it's crucial to avoid these common mistakes:

Overlooking Error Rates

AI tools, while powerful, are not infallible. Failing to account for error rates can lead to overestimation of ROI. For example, in defect detection, while AI can significantly reduce defects in Manufacturing-QA, error rates emerge due to the non-deterministic nature of AI outputs. This has to be accounted for, especially at scale.

Single-point ROI Analysis

Viewing AI systems as "set-and-forget" solutions ignores the ongoing adjustments and improvements necessary for optimal performance. Updates and maintenance are crucial for long-term success and should be factored into ROI calculations.

Isolated Project Evaluation

Assessing AI projects in isolation overlooks potential synergies within the broader project portfolio and opportunities to leverage previous investments.

Best Practices for Comprehensive ROI Assessment

To gain a more accurate picture of generative AI's value, consider the following approaches:

Holistic Evaluation

Incorporate both direct and indirect returns in your ROI calculations. While direct returns often provide immediate, quantifiable benefits, indirect returns can offer substantial long-term value.

Long-term Perspective

Account for the evolving nature of AI technology. Initial ROI may be modest, but as systems learn and improve, returns often increase over time. As an example, in Generative AI technologies, the cost per token has drastically reduced over the last 12 months. Similar developments can be expected in other areas of AI.

Error Rate Consideration

Factor in the potential impact of AI errors on overall ROI. This includes not just the direct costs of errors, but also potential reputational risks and mitigation expenses.

Portfolio Approach

Consider how new AI initiatives complement existing projects and technologies. For example, if you decide to implement a ChatGPT clone using your own or cloud-provided AI infrastructure, you may re-use this for future projects. Look for opportunities to re-use or build upon previous AI investments.

Conclusion

Quantifying the ROI of generative AI requires a nuanced approach that goes beyond simple cost-benefit analysis. By considering both direct and indirect returns, avoiding common evaluation pitfalls, and adopting a comprehensive assessment strategy, CIOs can gain a clearer picture of AI's true value to their organization.

As you evaluate current or future AI initiatives, remember that the most successful implementations often deliver a combination of immediate efficiency gains and long-term strategic advantages. By taking a balanced and thorough approach to ROI assessment, you can make more informed decisions about AI investments and maximize their value for your organization.

‍