Focus instead on where best to run your workloads and start using cost-conscious coding 

The big cloud service providers are moving into AI – and this has some folks sounding the alarm.

The narrative is that when businesses embrace the AI capabilities of AWS, Google Cloud, and Microsoft Azure, they are handing over even more power to these already powerful companies.

But AI is just another service that the cloud vendors are going to provide. It can’t be stopped.

Microsoft 365 is a phenomenal example. Excel will have Copilot, so will PowerPoint and your email. Companies that are already on Microsoft Azure will embrace these capabilities. They have to because AI is getting integrated into an ecosystem of which they are already a part, and it’s happening at an incremental cost. Those that don’t use these capabilities to write content, create PowerPoints, and otherwise do things better, could miss out on valuable opportunities.

Now, for custom AI solutions, you may have documents and volumes of data on premises to which you want to apply AI technology. So, do you want to use Azure AI or do you use Amazon Bedrock? Well, if you already put your data lake on AWS, you can now point all those documents to Bedrock as opposed to moving big chunks of your data to enable your organization to employ Azure AI.

Understand that expensive data movement and cloud costs are the real threat

My point is that it’s not just AI that is driving business decisions about which vendors and technologies to use. It’s the associated data, the associated infrastructure, and the associated compute cost that organizations have to pay for a new cloud if they have to move their data.

Also, not everything related to AI involves chatbots. Different companies have different AI use cases, and AI involves huge volumes of data. If a company needs to move its data across clouds to use one cloud service provider over another, that creates big challenges. It’s a struggle.

The cost of the cloud is still a puzzle that many companies are putting together. And AI has made this a lot more complex with added cost that is even harder to compute or predict accurately.

Ask yourself: Would you be better off keeping that workload on premises?

That is prompting many companies to consider whether they can leverage their on-premises infrastructure so that they don’t have to move their data into the cloud. The thinking is that they already have the hardware, and the on-premises model will give them more influence over their business and costs.

Given the options with large language models (LLMs) across local LLMs and cloud-based LLMs, and the added confusion around compliance and data security, more thought is being given to whether staying on-premises for certain workloads would make sense. Things you will want to consider in determining whether a local LLM and an on-premises footprint may be more beneficial than leveraging public cloud include, but are not limited to, the training frequency and training data.

Workloads that constantly generate more revenue, have a need to handle burst traffic, and need continuous feature uplift are ideal for the cloud while a more standard workload that’s lights on and not requiring continuous uplift may be left on-prem if the strategy is still to have a data center. Typically, in any organization, we estimate about 20-30% of enterprise workloads that run in the cloud actually generate revenues. This is true for any workload, not just AI-based workloads.

Considering all the factors above, conscious decisions have to be made on whether we continue paying for APIs and hosting or train, host, and use an AI model on premises.

Do cloud optimization and get ahead of excessive costs with cost-conscious coding

Cloud sticker shock has driven excitement about and investment in financial and operational IT management and optimization (FinOps). For example, IBM in June revealed plans to buy FinOps software company Apptio for $4.6 billion, and TechCrunch notes “the ongoing rise of FinOps.”

But the FinOps framework and many related tools are reactive in nature. You deploy your application to the cloud, and then try to use FinOps tools to control your costs. By the time controls are put in place, the money is already spent.

Cost-conscious coding is a far more effective approach to cloud optimization. It enables you to design for cost, reliability, and security in any cloud workload that your company is deploying. With AI, this becomes all the more important as algorithms that are not tuned or optimized will consume significantly larger compute and storage than the ones that are consciously developed.

While DevOps tries to bring engineering closer to operations, it has not solved for the above problem. Although development methodology changed with DevOps, the philosophy of coding has not. Most developers today still write code for business requirements and functionality only and not for cost.

Cost-conscious coding changes that, which is extremely valuable to the bottom line because designing for cost is critical. But to benefit from cost-conscious coding you will need to build internal expertise or work with an experienced partner to control your cloud costs in this way.

Organizations are now trying to get their arms around what AI means for their businesses. As you do this, analyze what your infrastructure and compute costs will look like now and in the future if you run them on premises vs. in the cloud, and whether or not you do cost-conscious coding; define AI use cases that will be most beneficial for your business; decide how much you are willing to spend on those use cases; consider compliance, control, reliability, security, and training data and frequency requirements; and understand the revenue potential and opportunities for optimization involved with your AI use cases and all of your workloads.

By Premkumar Balasubramanian