1
0
The release of ChatGPT in late 2022 didn’t just start an AI boom; it redefined the horizon for the entire IT industry. As the world realized the immense potential of Large Language Models (LLMs), my team and I began exploring how these capabilities could be harnessed. By early 2023, we hit a critical realization: we didn’t just need AI applications, we needed an AI application platform.
The Challenge: Security and Sovereignty
I was tasked with building an AI platform tailored for our tech family. The initial goal was modest: support effective prompt engineering. I began by scouting the open-source landscape, but in those early days, the available solutions fell short of enterprise-grade requirements. Specifically, we needed robust security and internal compliance that off the shelf tools couldn’t yet guarantee. So, I started building from scratch.
From Prompt Engineering to Versioned Apps
Prompt engineering is the heart of LLM interaction, so that’s where I focused first. I built a system where users could manage, test, and refine their system prompts, eventually “publishing” them as standalone Apps accessible via HTTP APIs.
To ensure reliability, a non-negotiable for enterprise use, I implemented:
- Version Control: Every change to a prompt is tracked.
- Staged Rollouts: Users can deploy new versions safely without disrupting existing workflows.
The impact was immediate. By abstracting the complexity, we empowered non-technical colleagues to build functional AI apps simply by defining a prompt. Seeing “non-tech” folks transition into creators was a turning point; ease of use quickly became my top priority as interest poured in from across the organization.
Overcoming the Context Barrier with RAG
While system prompts are powerful, they are constrained by their training data and “cutoff dates.” Internally, our most popular request was for an FAQ agent based on our proprietary, private knowledge.
However, we faced a major hurdle: our internal documentation far exceeded the 32,000 token context window available at the time. To solve this, I implemented Retrieval-Augmented Generation (RAG).
RAG transformed the platform’s search capability. Instead of simple keyword matching, the AI began understanding intent. If a user searched for “fruit,” the system was now smart enough to retrieve data regarding apples, oranges, and pears even if the word “fruit” never appeared in the source text.
The Future: Codifying Human Knowledge
The true value of this platform isn’t just the tech, it’s the ability to codify human expertise into highly available, efficient digital tools. Today, these apps live where our work happens: on Slack. They answer menial questions, automate repetitive tasks, and allow our teams to focus on high-level problem-solving.
What started as a search for an open-source tool has grown into the backbone of our AI strategy in Southeast Asia.
