The Art of Not Knowing: How We Are Building an AI Coach from Scratch
Learnings and advice from our experiments with building an AI web app and two mobile apps with zero prior experience.
We didn’t plan for this. Not really.
Our cursor is hovering over the “Submit for Review” Button for our AI Agile Coach apps we’ve been building for the last 4 weeks. We didn’t really know how to make apps 4 weeks ago. We did it anyway.
“What’s the worst that can happen?”
“They could reject it.”
“Hit the button.”
Spoiler Alert: Both Apps were accepted within a couple of days.
How did we get here?
It’s 2023, and AI is everywhere. ChatGPT is the shiny new toy. “Look, it can make a poem about Scrum Masters!” someone posted in our Slack. “Just had it write my email; it’s like magic," another wrote. We start using AI for little things—generating ideas, handling mundane tasks—and just having fun.
But it was more than a gimmick, and we all knew it. We were stunned, and maybe feeling behind the curve. This was new, big, and moving fast, and here we were, trying to catch up.
That’s when it hit us: this wasn’t just something to watch from the sidelines. We could build something with this.
Here’s what we did, what we learned, plus some peeks behind-the-scenes.
1. Diving In: No Perfect Idea, Just Start
We didn’t have a grand plan or a clear idea of what to create. But we knew we’re all about learning and sharing knowledge in a way that sticks.
And now, AI was this huge, shiny tool we could use in a new way. But what exactly were we going to build? And how would we do it?
Instead of waiting for the perfect idea, we just started tinkering. No fancy roadmaps, no long debates—just us, a handful of ideas, and the urge to see what we could make.
Learning:
Don’t wait for the perfect idea. Start small, experiment with the tech, and let the direction emerge from what you learn. With fast-moving tech, waiting too long could mean missing the moment.
2. Establishing a Vision: Beyond Just Another Chatbot
From the start, we didn’t want to build just another chatbot. ChatGPT was already there and we knew we couldn’t (and didn’t want to) compete with its scale. But what ChatGPT offered in scale, it lacked in quality for niche topics, where details matter.
What if our AI Agile Coach could leverage our expertise to provide trustworthy, thought-provoking guidance beyond simple, generic responses?
Quality, trust, and a human touch became our vision’s pillars. We wanted to counter the common issues of AI—being generic, impersonal, and unreliable—by making these our strengths.
Learning:
Start with a vision and work backwards from there. You don’t need all the details upfront; new ideas will naturally emerge as you progress.
3. Forming a Lean Team and Process: Keeping It Simple
Once we decided to build the AI coach, a small team formed—one designer, two developers, and the rest of the team in the background for feedback and providing ideas.
Without much planning, we adopted a lean process: quick daily syncs, informal retro-like conversations to tweak tools, and weekly demos to share progress. It wasn’t about sticking to a strict agile framework; it was about keeping things simple and responsive.
Learning:
Let your process evolve naturally. Daily check-ins, regular feedback, and simple demos can keep you aligned. Don’t overthink it—sometimes, the best approach is the one that fits your team’s rhythm.
4. The Art of Prompting: Trial and Error
We knew that innovative prompts (the instructions you give to the language model) were going to be elemental to our app. Prompting sounds easy until you try to do it well. We experimented with different prompts—tweaking words, rewriting sentences, testing until it felt right.
Our current prompt is over 5,000 characters long, packed with details that make our AI sound like us. It’s like writing a secret recipe, and it took a lot of tasting to get there and we continue tweaking it regularly.
Learning:
Experiment with prompts. Test different variations to see what works, and adjust them frequently. Good prompts are crucial for the AI’s performance.
5. Releasing Early and Often: Letting Go of Perfection
We didn’t wait for perfection. As soon as our first version was good enough, we released it. Was it flawless? Did it include the countless ideas and features we thought of? Definitely not. But it didn’t make sense to continue without feedback from real people.
With AI, you can’t predict every outcome, and every user interaction is different. The only way to know what works is to get it out there, see how people use it, and adapt based on what you learn.
Learning:
Don’t aim for perfection before launch. Get a minimum viable product out quickly, and learn from user feedback for rapid improvements. This isn’t a new idea, but in the rapidly evolving AI landscape it matters more than ever.
6. Standing on the Shoulders of Giants
Turns out, you don’t have to build everything yourself. We relied on frameworks and boilerplates like Gravity, which handles all the tedious but necessary stuff like user management, authentication, and email , so we could focus on what mattered. We leaned heavily on tools like LangChain and of course the OpenAI API, which were doing a lot of the heavy lifting behind the scenes.
Learning:
Leverage existing tools to speed up development. Don’t build from scratch when someone else has solved the problem. The right tools can save a lot of time.
7. Learning from Experts (Like the real Henrik)
We owe a lot to Henrik Kniberg, who guided us through some of the most critical parts of the development process. He kicked things off with a workshop that set the tone and direction for our work. As a mentor to the team and the namesake for our product, Henrik pushed us into new directions and helped us shape the AI Agile Coach into what it is today. His influence has been invaluable, and we’re grateful for his support along the way.
Learning:
Reach out and learn from experts. A fresh perspective from someone experienced can open up new paths you hadn’t considered.
8. The Content Challenge: What Do We Feed the AI?
We’ve spent years creating and curating content we believe in, and now we had to figure out how to make that the foundation for our AI coach. It wasn’t just about dumping our materials into the system. We wanted our AI to be reliable, engaging, and genuinely helpful. So we spent hours picking and refining the right bits of knowledge, and ensuring the AI could understand and use them effectively.
Learning:
Invest time in curating your content. Ensure it’s accurate and reflects the tone and voice for your AI. Better content leads to a more trustworthy AI.
9. Iterating the LLM Implementation: Adapt, Adapt, Adapt
Improving the AI coach took multiple iterations, each teaching us what worked and what didn’t:
1. Using Custom GPT Inside ChatGPT:
We started with custom GPTs inside ChatGPT. It was easy—no coding required. However, we quickly hit limits with control and output quality. We also knew that we wanted to release our own product outside of the OpenAI ecosystem with its own branding and a more direct connection to our other products.
2. Custom UI with the Assistant API:
Initially we built a custom UI using the Assistant API. This gave us more control over which documents the AI used. But we soon hit limitations on scalability and it was hard to control which parts of the custom knowledge were used to generate answers.
3. Custom UI with LangChain and Weviate Vector DB:
Next, we implemented a custom Assistant using LangChain and knowledge from a Weviate vector database. This setup allowed us to pull in only relevant knowledge, improving quality. The results were great in most cases, but sometimes the Assistant got lost in endless loops or provided weird, unnatural answers.
4. LangChain Agent:
To counter these issues, we implemented a LangChain agent. This fairly new approach delivered the best results yet. The agent can decide which tools and information to use based on the user’s query and is more efficient with his memory of past interactions. For example, if a user says “Hi”, the agent decides that it doesn’t need to search his specialized knowledge to simply greet back.
The trade-off is that you give up some control because the agent has more autonomy. However, the benefits of improved quality and efficiency far outweigh this. The Langchain agent library offers many options and configurations to fine-tune the model's performance.
5. Some new, awesome solution.
What comes after agents, we don’t know yet. Let’s see...
Learning:
If a setup isn’t working, don’t stick to it. Keep trying new configurations and learn from each. New solutions are emerging daily in the AI landscape, so stay open and ready to adapt to the latest tools and approaches to improve your product.
10. Building the Mobile App: A Leap of Faith
We always knew we’d need a mobile app, but building one seemed daunting. When Sohrab asked what it would take to create an app, we were hesitant. Mobile apps are complex, and we’d never done it before. Despite our web app still being in the early stages with plenty left to do, it made sense to start the mobile app early. By getting the app working with a limited feature set, we avoided the risk of having to catch up later when the product was more complex. Plus, it gave us an early opportunity to learn and adapt.
Using Expo and React Native, we built iOS and Android versions in four weeks. Hitting the “Submit for Review” button was pretty exciting. It reminded us to take the leap and see where we land.
Learning:
If users express a need, take the leap. The learning curve might be steep, but practical experience beats theoretical planning. Frameworks like Expo and React Native work well for building multi-platform apps. In the past, non-native apps could feel clunky or out of place, but these frameworks have evolved significantly. You only sacrifice a bit of the native feel, and in return, you gain rapid development and easier maintenance across platforms.
11. Using AI to Build AI: Enabling Everyone
One of the coolest things was using AI to help build our AI. We set up different GPTs for specific tasks: one for coding, another for writing, and so on. Even I, a designer with limited coding abilities, found myself coding with the help of Cursor, a code editor with a built-in assistant. AI became an enabler, allowing everyone to contribute in new ways.
Learning:
Use AI to enhance your development process and build specialized tools. As we learned from Henrik Kniberg: With every task on your list, ask yourself: How can AI help me with that?
12. Keeping the Human Touch
In an AI-driven project, it’s easy to lose sight of the human element. So, we made a conscious effort to prevent that:
- The logo was first sketched on a post-it.
- Code was manually checked, selected, and written.
- Marketing copy was first written on paper—with a pencil.
- Emails to users were written and sent manually with a bootstrapped approach, with no scaling in mind yet. (Hint: Don’t optimize for scale prematurely)
Learning:
AI is a powerful tool, but it’s the human elements that make a product feel genuine and relatable. Keep the personal touch alive in your project.
13. We’re Just Getting Started
We built a lot quickly, but we’re still figuring it out. Every day brings new challenges, lessons, and ideas to test. We’re still learning, iterating, and trying to make something that works.
And that’s okay because it’s how we’ve always operated. We live by the principles we teach: learn fast, adapt quickly, and keep moving forward.
Learning:
Stay curious and flexible. Keep testing assumptions, embrace what you learn, and be ready to change direction.
And don’t forget to have fun along the way.
---
If you want to check out what we’ve built, join the waitlist for our AI coach, Henrik, here. If you have questions, reach out. We’re always up for a chat about our learnings.
