Since launch From ChatGPT, a stampede of tech company leaders has been chasing the rumor: everywhere you look, another company is touting its pioneering ai feature. But true business value comes from delivering product capabilities that matter to users, not just from using cutting-edge technology.
We achieve a 10x return on engineering effort with ai by starting with basic principles of what users need from your product, developing an ai capability that supports that vision, and then measuring adoption to make sure it hits the mark.
The first feature of our ai product was not aligned with this idea and it took us a month to reach a disappointing 0.5% adoption among returning users. After refocusing on our core principles of what our users need from our product, we developed an “ai as an agent” approach and launched a new ai capability that reached 5% adoption in the first week. This formula for success in ai can be applied to almost any software product.
The waste of excessive haste
Many startups, like ours, are often tempted by the lure of integrating the latest technology without a clear strategy. So, after the groundbreaking release of the various incarnations of OpenAI’s pre-trained generative transformer (GPT) models, we began looking for a way to use large language model (LLM) ai technology in our product. Very soon, we secure our place on the hype train with a new ai-powered item in production.
This first ai capability was a small summary function that uses GPT to write a short paragraph describing each file our user uploads to our product. It gave us something to talk about and we did marketing content, but it didn’t have a significant impact on our user experience.
Many startups are often tempted by the lure of integrating the latest technology without a clear strategy.
We knew this because none of our key metrics showed an appreciable change. Only 0.5% of returning users interacted with the description during the first month. Additionally, there were no improvements in user activation or changes in the pace of user registration.
When we thought about it from a broader perspective, it became clear that this feature would never change those metrics. The core value proposition of our product is about big data analytics and using data to understand the world.
Generating a few words about the uploaded file won’t result in any meaningful analytical insights, which means it won’t help our users much. In our rush to deliver something ai-related, we missed an opportunity to deliver real value.
Success with ai as an agent: 10 times greater profitability
The ai approach that made us successful is an “ai as Agent” principle that allows our users to interact with our product data through natural language. This recipe can be applied to almost any software product built on API calls.
After our initial ai feature, we checked the box, but we weren’t satisfied because we knew we could do better for our users. So we did what software engineers have been doing since the invention of programming languages: get together for a hackathon. Starting from this hackathon, we implemented an ai agent that acts on behalf of the user.
The agent uses our own product by making API calls to the same API endpoints that our web interface calls. It builds API calls based on a natural language conversation with the user, attempting to accomplish what the user asks it to do. The agent’s actions are manifested in our web UI as a result of API calls, as if the user had performed the actions themselves.
