CTO Insights

The high percentage of AI initiatives in the Beta and Production phases show how eager companies are to integrate AI into their existing workflows. Over the next year, we anticipate a higher number of projects to transition into the production phase.

Moving to production would require organizations to integrate a variety of tools and processes to maintain quality of service, security and compliance. Organizations would have to create novel processes, from legal and compliance to product and engineering, which may result in delays in transitioning into production.

CTO Insights

Seems like everyone working on AI initiatives want to either have it live in production, or has actively been planning to do so. The increased efficiency and additional benefits of real world AI applications make it a top priority for many organizations.

CTO Insights

Organizations worldwide recognize the pivotal role of LLMs and AI in driving business efficiency. The budgets for generative AI reflect its importance and anticipated ROI. Companies are investing significantly in these technologies, expecting them to drive innovation, competitive advantage, and future cost savings.

These high budgets, however, make it challenging to achieve sustainable unit economics in AI applications. As large models require large amounts of compute and expensive GPU machines, there’s a race to reduce inference and training costs. The high costs associated with AI are some of the main blockers of its usage in production.

As we expect AI budgets to remain high, we believe that inference costs and token generation costs to reduce significantly over the coming years.

CTO Insights

AI initiatives are multi-faceted by nature, requiring a variety of tools and disciplines, from prompt engineering to model deployment and software engineering. Creating a production ready AI product requires this variety of skills to tackle the non-deterministic nature of LLMs. It’s only natural that we’ll see a wide range of roles involved in these projects, from product manager , prompt engineers and data scientists.

As AI and LLM initiatives grow, we’ll see prompt engineers, software developers, PMs and analysts taking an active role in shaping and working on them. Part of the reason is the fact that LLMs provide a higher level of abstraction, allowing non-technical people to take an active part in shaping AI-based products.

CTO Insights

OpenAI continues to dominate the LLM market due to their highly-performant models and Azure private deployments. Anthropic comes right after, with their capable Claude models served through AWS Bedrock and their native API. Claude models are affordable and performant, providing an increasingly popular competition to the OpenAI GPT-4 series.

The high rate of adoption of models on AWS Bedrock and Google Vertex shows that companies favor deployments on their own cloud environments, for additional privacy, security and stability.

The usage of open-source models is expected to grow as well, as capable models such as Llama-3 and Mistral (and their future generations) are released. These models enable companies to save costs, improve performance, and fine-tune models for their own needs.

CTO Insights

Integrating AI and LLM in real-life workflows is challenging. Enterprises are still using a variety of platforms and systems tailored for their own use-cases. Combine this with the unique challenges integrating AI, and you’re up for a real challenge. The non-deterministic nature of LLMs demand safeguards, checks and processes, to maintain the highest level of quality. Achieving this, while connecting to the various platforms in every company, and staying compliant, is challenging.

A top concern for companies is security and privacy. First, LLMs require a lot of data to provide valuable responses. These models are hosted on different providers, outside the cloud perimeter of organizations. Creating secure and compliant AI applications will remain top priority for organizations, and it follows that we’ll see more bespoke security solutions tailored for large enterprises.

CTO Insights

Prompts are critical assets for AI applications. Crafting the right prompt is a mix of experimentation, trial and error and best practices. As crafting prompts doesn’t require technical expertise, we see the rise of Prompt Engineers and PMs in handling prompts. Ensuring collaboration among these roles can enhance the effectiveness and efficiency of AI initiatives. The introduction of LLMOps practices can streamline prompt management and improve outcomes. We see the importance of prompts in AI applications continue as AI initiatives move into their production phases.

CTO Insights

Debugging and evaluation remain top priorities for taking AI applications to production. New tools for debugging and evaluating LLM applications are increasingly being improved. We believe that more tools will eventually be created around LLMOps and tailored specifically to handle the unique challenges in LLM applications.

CTO Insights

Security and talent shortages are critical challenges in deploying LLM applications. AI models bring security concerns including data privacy, integrity, and compliance with regulations like GDPR. Additionally, the non-deterministic nature of LLMs introduces new security vulnerabilities and attack vectors. On the talent side, deploying LLMs requires specialized skills in a rapidly changing world, making it difficult for organizations to effectively train and hire relevant teams to handle these complex applications.

CTO Insights

Companies leverage AI in text-intensive processes. These naturally include code-generation, customer support and knowledge management. So far, these areas have also proven to be the most valuable in integrating AI for the current generation of LLMs. As models become more powerful, and unique integrations are created, we’ll see AI and LLMs integrated in new business processes in new domains as well.

CTO Insights

As companies use LLMs and models in a variety of ways, a need for a higher budget is generated. Even though inference cost is slowly dropping, use-cases are becoming more common, which in turn increases the budgets for AI. We believe that companies that want to leverage AI will still have to spend higher budgets in the coming years.