In April 2024, Arun Chandrasekaran, Distinguished Vice President Analyst at Gartner, whose research focuses on artificial intelligence, wrote in a Gartner blog about a prediction: By 2027, more than 50% of the GenAI models used by enterprises will be specific to either an industry or a business function—up from approximately 1% in 2023.
Additionally, in 2023, businesses began spending money much more actively, as confirmed by a report from Statista. Based on these facts, he calls for planning to deploy and manage multiple domain-specific GenAI models.
However, before doing so, he suggests looking for off-the-shelf, domain-specific models that can be trained or tuned to meet enterprise needs.
This sounds like a plan, but I think it's very important to have one's own data. Reports about the ROI of AI that has been implemented, or predictions of future plans for implementing AI, are crucial before starting to invest in popular solutions or trying new optimization methods with AI.
So, in this article, I’ve compiled some real-world results on the ROI of implemented AI, along with a short guide on how to measure the ROI of AI based on best practices.
Introduction to ROI in AI
At its core, ROI is a financial ratio that compares an investment's gain or loss to its cost. In the context of AI, a successful investment should yield benefits that outweigh the associated costs.
In other side, calculating ROI for AI projects presents unique challenges:
Time discrepancy: Costs are typically incurred in the present or near future, while benefits may materialize at an uncertain point in the future.
Uncertainty: The timing of benefit accrual is often less predictable than the timing of expenditures.
Comprehensive valuation: ROI calculations for AI should account for both the time value of money invested and the uncertainty of future benefits.
In sum, it is important to distinguish between two types of ROI when evaluating AI investments.
Hard ROI: This is the standard financial measure, focusing on quantifiable monetary gains or losses relative to costs.
Soft ROI: This broader concept encompasses less tangible benefits such as improved employee satisfaction and retention, enhanced skill acquisition, brand strengthening, and potential increases in company valuation.
By considering both hard and soft ROI factors, organizations can gain a more comprehensive understanding of the value their AI investments bring to the business.
Mistakes When Calculating ROI in AI
By the way, When calculating the Return on Investment (ROI) for AI initiatives, companies often fall into three major pitfalls. Understanding and avoiding these can be crucial for accurate ROI assessment. Let’s look on the core three.
Uncertainty of Benefits
Many organizations perform simplistic ROI calculations that fail to account for the uncertainty involved in realizing AI benefits or for the quality of existing data sets. This oversight can lead to overly optimistic projections and unrealistic expectations.
Key considerations:
- AI models may have errors and rarely achieve 100% accuracy
- Real-world performance often differs from training environments
- Estimating error rates requires comparing AI to human baseline performance
- Calculating the cost of errors can be challenging and far-reaching
Example: An AI system categorizing customer complaints may miscategorize a high-severity issue as low-priority, potentially leading to customer dissatisfaction or even customer loss.
Best practice: Review both hard and soft investments/benefits before initiating an AI project, review the quality of data sets on which you will train the model, and create a theory (how you expect the AI model to influence your business function).
Computing ROI Based on a Single Point in Time
Many companies calculate ROI shortly after AI deployment, typically a few months post-implementation. This approach fails to account for potential performance deterioration over time.
Key points:
- Machine learning models may degrade in performance as time passes
- Continuous performance measurement is essential
- Budget for ongoing maintenance to preserve long-term AI value
Best practice: Implement continuous ROI assessment or real-time measurement of results to track AI performance and value over time.
Treating Each AI Project Individually
Companies often evaluate AI projects in isolation, neglecting the broader impact of their entire AI implementation initiative.
Key points:
- Missed opportunities for synergies between projects
- Incomplete understanding of overall AI impact on the organization
Best practice: Adopt a portfolio approach when evaluating ROI, considering the collective impact and interrelationships of all AI initiatives.
By avoiding these common mistakes and adopting a more comprehensive approach to ROI calculation, organizations can make more informed decisions about their AI investments and maximize the value derived from their AI initiatives.
So let's draw a line and create a step-by-step guide on measuring ROI based on knowledge about KPIs and challenges.
Key Metrics for Measuring ROI in AI
According to a study by Deloitte, key areas yielding significant returns include customer service and experience (74%), IT operations and infrastructure (69%), and planning and decision-making (66%). However, it's also noted that not all companies experience a tangible ROI.
But, before starting, business need to be ensured what all decision makers and shareholders in understanding the key components of ROI in AI:
This diagram illustrates the complex interplay between the benefits derived from AI models and the investments required to create and maintain them. With this framework in mind, let's explore the six key steps to track and optimize ROI for AI investments:
Step 1: Define your goals and KPIs
Ask yourself: What strategic objectives does this AI project support? Are you aiming for faster innovation, improved customer interaction, or reduced operational costs?
Choose relevant KPIs that directly track progress towards your stated goal. E.g.:
- Sales conversion rates
- Customer churn reduction
- Decision-making time
- Error reduction
- Customer satisfaction scores (e.g., NPS)
Step 2: Establish your baseline
Gather all relevant data on your current performance. This includes sales cycles, customer complaints, and anything else that matters to your business.
- Compare your metrics with industry standards
- Identify areas where you're lagging (these are prime candidates for AI improvement)
- Look for hidden insights in your data. For example - Are there unusual cost spikes or recurring customer issues?
Remember, perfection isn't necessary. The goal is to have a point of comparison for when your AI is up and running.
Step 3: Estimate revenue gains
Consider potential gains from:
- Production line optimization
- Quality control automation
- Image recognition systems
- Increased productivity through automation
Factor in:
- New production capacities
- Additional products sold
- Newly acquired customers
- The impact of false positives and negatives in classification or recommendation tools
Step 4: Identify and Assess Costs
Calculate costs for:
- Tasks and resources (including your tech stack)
- Team composition and rates
- Project duration and number of iterations (training)
- Implementation and maintenance
- Data collection
Be aware of potential “bill shock” from unexpected charges, especially with cloud-based AI solutions.
Step 5: Determine intangible benefits
Assess “soft” ROI factors specific to your AI project:
- Impact on brand recognition
- Employee loyalty improvements
- Reduced physical and mental health issues (e.g., burnout) due to task automation
Use a qualitative approach to evaluate these benefits, as they're often harder to quantify but still crucial.
Step 6: Set a realistic timeframe
- Recognize that AI projections are ongoing processes, not one-time events.
- Account for model maintenance periods and associated costs.
- Consider both short-term goals and long-term vision when setting your timeframe.
Step 7: Develop a status quo scenario
- Calculate the Risk Of Non-Investment (RONI).
- Estimate the financial impact of not investing in an AI solution.
- This helps you understand the opportunity cost of inaction.
Step 8: Crunch the Numbers
Use the NPV (Net Present Value) approach to determine if your AI project will deliver on its long-term promises:
1. Calculate the difference between your AI scenario and the status quo.
2. Apply the NPV formula to account for the time value of money.
3. Conduct a sensitivity analysis by adjusting discount rates or cost/benefit forecasts to assess project risk.
Step 9: Account for ongoing challenge
- Data quality: Ensure your data is clean, accessible, and compliant with regulations like GDPR or HIPAA.
- Rapid technological change: Plan for regular model updates or replacements as technology evolves.
- Ethical considerations: Factor in potential costs related to bias, privacy concerns, or reputational risks.
Step 10: Review and adjust
- Regularly reassess your ROI calculations as the project progresses.
- Be transparent with stakeholders about short-term goals vs. long-term vision.
- Account for the AI learning curve—models often improve over time as they're exposed to more data.
Remember, calculating AI ROI is an ongoing process. Stay flexible, keep learning from your data, and be prepared to adapt your approach as your AI project evolves.
Real-world Case Studies of AI ROI
Case 1: Quantifying the return on investment of hospital artificial intelligence
In March 2024, an article was published in the American College of Radiology journal presenting the results of an exploration of the capabilities of AI-powered tools in healthcare, including the ROI, which highlighted a set of deficiencies.
Problem:
Hospitals like those in the United States face challenges in achieving an accurate ROI from AI-powered radiology platforms.
The complexity of integrating AI into existing workflows, coupled with the high initial costs, often makes it difficult for hospital administrators to justify these investments.
Idea:
The development of an AI-powered radiology diagnostic platform aims to streamline workflows, reduce labor times, and enhance diagnostic accuracy, addressing both clinical and financial challenges.
Solution:
To quantify the financial and non-financial benefits of implementing the AI platform across various hospital departments, such as radiology, IT, and administration, a comprehensive ROI calculator was created. The calculator includes scenarios that factor in different hospital settings, allowing for a tailored analysis of potential returns.
Results:
The AI platform demonstrated a 451% ROI over five years, which increased to 791% when radiologist time savings were included. However, ROI was found to be highly sensitive to factors like hospital type and time horizon, with significant variations in ROI depending on the specific hospital setting.
For instance, diagnostic centers without stroke management accreditation showed a much lower ROI, highlighting the need for careful consideration when deploying AI solutions in different healthcare environments.
Despite these challenges, the platform's integration resulted in time savings, additional diagnoses, and increased revenue from follow-up procedures, reinforcing the value of AI in hospital operations.
Case 2: PayPal's AI-Driven Cybersecurity Strategy
In November 2023, VentureBeat interviewed Assaf Keren, CISO and VP of enterprise cybersecurity at PayPal, revealing insights into the company's use of AI in cybersecurity and fraud prevention.
Problem:
As a global leader in digital payments, PayPal faced significant challenges in maintaining robust cybersecurity and fraud prevention measures while handling massive transaction volumes and sensitive financial data.
The company needed to find ways to bolster security without negatively impacting the customer experience, especially in the face of evolving cyber threats and fraud patterns.
Idea:
PayPal recognized the potential of AI, particularly generative AI, to enhance its cybersecurity capabilities, improve fraud detection, and streamline risk management processes. The company aimed to leverage AI to adapt quickly to changing fraud patterns and protect customers more effectively.
Solution:
PayPal implemented an AI-driven cybersecurity strategy focusing on:
1. Employing transformer-based deep learning for fraud reduction and customer protection.
2. Developing AI algorithms for risk management and global commerce empowerment.
3. Creating rapidly adaptable deep learning models that can be trained and deployed within 2–3 weeks.
4. Utilizing their vast data resources (over 200 petabytes of payment data) to power AI models.
5. Exploring generative AI-driven defense mechanisms and next-generation automated threat detection systems.
Results:
1. Financial Impact:
- For Q2 2023, PayPal reported $7.3 billion in revenue, up 7% year-over-year.
- The company achieved an 11% reduction in losses through improved risk management.
2. Fraud Prevention:
- From 2019 to 2022, as annual payment volumes nearly doubled from $712 billion to $1.36 trillion, PayPal cut its loss rate by nearly half, largely due to advances in AI algorithms and technology.
3. Operational Efficiency:
- AI-powered deep learning models can now be trained and pushed to production within 2 to 3 weeks, allowing for rapid adaptation to new fraud patterns.
4. Customer Protection:
- The AI systems have enhanced PayPal's ability to protect customers by quickly incorporating new fraud patterns and integrating feedback from internal agents and customers.
5. Future Potential:
- PayPal is actively exploring generative AI to further enhance efficiency, increase security, improve customer experience, and grow the business.
Despite these successes, PayPal acknowledges the need for careful evaluation and responsible deployment of AI technologies, particularly in handling sensitive financial data. The company emphasizes the importance of considering factors such as data quality, intellectual property, security, privacy, and compliance when implementing AI solutions.
Case 3: The Closure of Swell Investing–WiseBanyan
Swell Investing launched in 2015 as part of a wave of robo-advisors targeting millennials. In 2019 the company announced its closure on its website, ceasing to accept new clients and deposits and cease all operations.
Problem:
Swell Investing, a digital advisory firm specializing in socially responsible portfolios, failed to achieve the necessary scale to sustain operations in a crowded market of robo-advisors targeting millennials.
The company struggled with high customer acquisition costs and competition from lower-fee alternatives, ultimately failing to achieve a positive ROI on its AI and technology investments.
Idea: Swell Investing aimed to capitalize on the growing interest in ethical investment options among millennials while leveraging the robo-advisor model and AI-driven portfolio management.
The company bet on the combination of automated guidance, socially responsible investing options, and targeting tech-savvy millennials to achieve the scale necessary for profitability.
Solution:
The platform offered socially responsible portfolios with a low minimum investment of $50 and used AI and machine learning technologies to automate portfolio management and provide personalized investment advice. To differentiate itself, Swell focused on offering socially responsible options that aligned with millennial interest in environmental and social issues.
Results:
Despite its targeted approach and technological investments, Swell Investing struggled to achieve profitability and scale:
1. By the time of its closure, Swell had accumulated only $35 million in assets under management across 15,000 clients, far below the scale needed for sustainability.
2. Customer acquisition costs were unsustainably high, starting at $350 per person in the first year and dropping to about $150 in the final year.
3. The company's annual fee of 0.75% was considered high in a market where some competitors offered zero-fee options, making it less attractive to cost-sensitive millennials.
4. Swell faced intense competition from a growing number of SRI-focused robo-advisors targeting the same millennial demographic.
This case illustrates several key challenges in the robo-advisor and AI-driven financial services sector:
1. The high costs of customer acquisition and the need to balance these against potential lifetime value of clients.
2. The challenges of differentiating in a crowded market, even with a focus on trending areas like socially responsible investing.
3. The struggle to achieve positive ROI on AI and technology investments, particularly for smaller, specialized firms that may not have the resources to sustain longer periods of negative returns while building scale.
While Swell Investing's closure doesn't necessarily signal the end of SRI-focused robo-advisors, it does highlight the need for such services to carefully balance their value proposition, fee structure, and customer acquisition strategies to achieve the scale necessary for long-term viability in this highly competitive market.
Conclusion: ‘Tracking the Value Every Step’
So, I’ve attempted to explain that the widespread adoption of AI presents organizations with challenges in demonstrating and calculating ROI. These challenges arise from:
1. Premature adoption: Many companies implement AI without defined use cases, driven by competitive pressure.
2. Lack of expertise: Insufficient internal skills hinder effective AI utilization.
3. Measurement challenges: Quantifying AI's value, particularly in time and productivity, is complex.
4. Scale and cost considerations: Expanding AI usage often leads to unexpected costs.
5. Industry-specific needs: Effective AI solutions must be tailored to specific industries.
Steps to improve ROI:
1. Define clear use cases
2. Establish measurable metrics
3. Pilot before full-scale deployment
4. Invest in training
5. Partner with industry-specific AI experts
6. Systematically track AI impact
7. Set realistic expectations
By strategically adopting AI, companies can better realize tangible benefits and demonstrate a positive ROI. Careful planning, partner selection, and ongoing evaluation are key to success.
With the right strategy, AI offers significant growth opportunities. Explore our AI services at Techstack, where our experts can guide you through the entire process, ensuring your AI implementation meets your industry-specific needs and drives real value.