How not to add AI to your product

Our second attempt at adding AI to Fillout, an online form builder.

Do not index
Do not index
Hide CTA
Hide CTA
Hide cover
Hide cover
In late 2022, we excitedly added AI to our product. It flopped.
Fast-forward 7 months and we just released our second attempt. In this post, I share what went wrong with our first launch, what we learned, and what we’re doing differently this time.
I focus on using AI for a form-building tool. But the learnings apply to a wide range of products and to anyone using AI to solve the “blank canvas” onboarding problem in SaaS.

Our (failed) first attempt

First, some context: our product is Fillout, a powerful form builder (think Google Forms, but with more styling options, better integrations, and advanced features). We first introduced AI into our onboarding flow to help solve the “blank canvas” problem.
The UX was simple: describe the form you want to make, with as much or as little detail as you want. Then, “AI” designs a form that you can add to and modify. We thought it would be the equivalent of a “form template”, but personalized to your exact use case.
A screenshot of our first attempt to add AI to Fillout. Read about how we built it here.
A screenshot of our first attempt to add AI to Fillout. Read about how we built it here.
Back in October 2022, using generative AI in-product was a fairly new idea. We launched on Product Hunt, pitching ourselves as the “AI-powered form builder” and got several-hundred signups.
Unfortunately, almost all the people who tried the AI onboarding immediately churned. The few who did stick around usually removed the AI-generated questions from their form pretty fast. As the final blow, a few of our existing users asked us to re-enable our old onboarding flow. That’s all we had see to be sure - we removed the AI feature completely, just one week after launch.

What we learned

After the launch, we took a step back and watched a few of our friends use Fillout for the first time. Here’s what we learned:
  1. The experience was overwhelming. If you start from a blank form, you can gradually learn how to use the form editor as you make your first form. But with the AI onboarding, new users had to get acquainted with the Fillout editor and parse all the questions that were added to their form, at the same time.
  1. There was no ability to undo and iterate. Once you landed in the pre-built form, that was it. You couldn’t re-prompt Fillout to modify the form or undo some of the changes if they weren’t what you wanted. We understood this better a few weeks later when OpenAI launched ChatGPT. One of the things that makes ChatGPT great is the ability to iterate on a prompt if you don’t get the outcome you need - something Fillout lacked.
  1. The AI was a one trick pony. Even if the AI produced a helpful initial form, the user was left hanging, expecting the rest of the form building experience to be super-charged in some way. This was doubly true since we marketed Fillout to be an “AI-powered form builder” - lesson learned.
  1. We didn’t teach users how to prompt effectively. Many people gave generic prompts like Order form , which, unsurprisingly, yielded generic results. The few who did succeed provided details like the tone of the form, specifics on the questions they wanted to ask, and how long the form should be. Although we had a basic placeholder and tooltip, that clearly wasn’t enough to guide users to success.

What we’re doing differently

The last few months has been a whirlwind of startups incorporating AI into their products. Though a lot of these new AI features have similar utility to our first Fillout attempt, we’ve seen several approaches that have been well-received and provide clear value (Notion and Retool come to mind).
Though we haven’t sat completely on the sidelines (we launched an experimental tool called and an AI quiz maker), we’ve mostly held off adding AI to the core form building experience — until now.
How the new AI-feature works and why it’s different
  1. First, the experience is entirely in-editor. To trigger the feature, you click on the sparkle icon ✨ in the top left corner of the screen. You can use it whenever you need it, or not at all. Because we don’t generate an initial form with AI, you get the benefit of gradually discovering the form builder without being overwhelmed. Once you start using the AI, you’re better acclimated to understand what is changing.
    1. The user experience for version 2.0 of AI in Fillout
      The user experience for version 2.0 of AI in Fillout
  1. We show tips on how to prompt effectively and let you choose whether you want to add one question or multiple questions at once. This gives you more control over what the AI actually does.
  1. The AI is context-aware. Similar to how Github Copilot takes into account your most recently viewed code files, Fillout takes clues from your current form to ensure that the new AI content fits in as you’d expect it to.
  1. We stream partial completions from OpenAI to reassure you that something is happening and that you’re not just stuck on a loading screen.
  1. Most importantly, we clearly show you what questions were AI generated and give you the option to undo with one click.
Accept or reject proposed changes to your form.
Accept or reject proposed changes to your form.
The verdict’s not out yet, but so far in our own personal testing this new approach has proven significantly more useful.
Creating mini AI tools
Lastly, we started creating mini-tools that use AI, like our AI survey maker, AI form builder and AI quiz maker. These tools don’t require signing up, so they’re an easy no-commitment way to try Fillout and built a form, survey or quiz very quickly.
If you want to try the in-app AI for yourself, you can sign up for free here. Let me know what you think:
Thanks for reading!
Dominic Whyte

Written by

Dominic Whyte

Dominic is a co-founder at Fillout. He previously worked in engineering & product at Retool. Prior to Retool, he started Cheer (backed by Sequoia and acquired by Retool in 2020).

Related posts

How we used AI to improve new user onboardingHow we used AI to improve new user onboarding
How we made our own help center with NotionHow we made our own help center with Notion
The Developer’s Case for Building Native IntegrationsThe Developer’s Case for Building Native Integrations
Database Wars 2.0: Why Everyone’s Building a DatabaseDatabase Wars 2.0: Why Everyone’s Building a Database
AI has killed software switching costsAI has killed software switching costs