Launching An AI Product Part 1 - User Problems, Research and Alignment
A 4 part case study on launching an AI Product. Part 1 is all about identifying user problems, prioritization, achieving alignment and user research techniques.
Welcome! If you’re new to PM-ing, consider this series your playbook for launching your first AI product. If you’re a seasoned PM, get ready to relive and get thanked for all the invisible work that goes into product launches. This isn’t your typical 5-minute read—it’s a thought-provoking exploration of product concepts. There’s no other material like this online, so I suggest grabbing some water and snacks and settling in!
For more of an explanation to this case study series, consider revisiting my introductory post.
By the end of the first article, you should know everything about completing phase 1 :
What’s the problem and why should we solve it now?
How do I identify the problem?
What are the implications of solving this problem now vs later?
How do I drive alignment with engineering, business development, marketing and customer success to support me in solving this problem?
I need to understand what users think. What is the best approach to user research in my scenario?
I’m going to need you to imagine that you're a product manager at a sports management company, which has become tech forward with the launch of a mobile app with an AI-powered chat.
This chat isn’t just any chat—it’s a sports oracle. It spills the beans on games, sports legends, and epic matches.
But wait, there’s more - We’re also making money by selling sports gear/merchandise and game tickets right within the chat.
Your mission? Boost the chat’s mojo—get those Daily Active Users (DAU) soaring! Why? Because that’s the launchpad for merchandise sales and ticket bonanzas.
Per your regular PM duties, you are working hard to think of ways to get that DAU to increase. You have a dashboard of metrics to look at on a daily basis - one day, you notice something odd.
While the number of users visiting the app is steadily increasing, they're not exactly translating to the same extent when it comes to engagement. People are opening the app, but they’re not using the chatbot feature much. Wait, is this a problem you need to look into solving?!
Phase 1A : Define the problem at sight in an objective manner.
In order to get a clearer picture, you as a Product Manager will typically dive in and figure out quantitatively, what the problem exactly is. Here’s the result of the first pass.
Even though the net new users visiting the mobile app is growing at about 5% month over month, not enough of the new users are actually interested in engaging with the chatbot. The new users’ engagement on the merch buying and ticket booking features is also pretty low, compared to the old users.
Why would the new users behave so differently from the existing users?
Upon slicing the data further and also talking to the marketing team, we realize that the new users coming in are due to a growth initiative - a collaboration with a streaming service. This seems to be a company wide growth strategy and will continue to get popular as we make more and more business partnerships.
Now you set out on an investigation to understand and create a rough persona of these new users because they’re the ones you’re also going to be catering to. Collaboration is key when it comes to such investigative journeys. Working together with an analyst, data scientist, user researcher or some sort of a domain expert to gain insights about the new users - login details, entered interests, chats and data that the streaming service would be providing to us are good places to start.
You finally come up with the following insight:
80% of the new users coming into our app land there accidentally via a link on the streaming service that they use.
The new users coming in are barely interested in using the app’s chat feature to learn about sports. They don't care about buying merchandise unless it's related to a movie they’ve watched and are clearly not interested in attending games.
While it's always interesting to keep gaining a ton of insights about these users, you want to also stop the investigation at some point and move on to the next phase - quantifying any potential damage caused by this growth initiative on your KPIs (Key performance indicators) and whether this presents an opportunity to meet your goals sooner.
Key Takeaways:
Partner with SMEs (analysts, data engineers etc.) to collect historical data.
Discuss with cross functional partners (marketing, customer success etc) to identify any gaps in understanding.
Quantify the problem in terms of change in key user behavior metrics.
Phase 1B : Is this a problem we need to solve now?
Surfacing and quantifying problems that could affect the business is the single biggest value-add by a product manager prior to ideating a solution.
If we allow things to progress as is, what’s the damage expected at the end of the year?
So you’ve gone ahead and pulled more data - given the current month over month increase in the number of new users on the app, taking their limited engagement into account, the overall DAU on the app drops far behind our yearly goal for the year.
It naturally also impacts the overall sales of merchandise and tickets booked on the app to an extent which you’d quantify (helps to make some assumptions for quick calculations)
While addressing challenges, also explore potential opportunities. As a product manager, aim not only to highlight concerns but also to chart a course toward solutions.
To get good at doing this, collaborate closely with other product managers responsible for various other functions of the chat. Understand the impact of this problem on not just your KPI, but also theirs. This “Inform as you go” partnership across different work streams will help align efforts in the long run.
Pro Tip: Maintain a user funnel and conversion dashboard. Even a basic one can provide valuable insights about users who reach the app but don't convert into paying customers eventually. Personally, I regularly review real-time metrics with PMs from various work streams to enhance user experience, growth, and monetization.
You can now translate the problem into several opportunities. The main one can be something like the one below.
Opportunity
With a net-new user app install rate of 5% MoM for the next 12 months, growth in DAU by X% within the net-new users can improve DAU overall by Y% and bring in $Z M ARR due to sales of merchandise and tickets.
* ARR = Annual recurring revenue is the yearly value of revenue generated from a feature or product.
If we work on increasing engagement for these new users on the app, it could eventually translate to some monetization since they can purchase tickets and merchandise based on chat prompts.
As a product manager, you’re likely working with a well-defined product roadmap established at the beginning of the fiscal year. However, unexpected situations like this one can arise, prompting the need for adjustments. Upon encountering such problems (or opportunities), you’ll need to revise the roadmap and incorporate new initiatives.
Taking a peek at the list of work items from the roadmap for this quarter, the product backlog could look something like this -
Upon reviewing that list, it appears there are no other initiatives that could generate such a substantial increase in ARR this year. It’s worth considering strategies to enhance engagement for these new users and developing a well-defined solution with product-market fit. When the timing aligns, you can mobilize the engineering team into execution mode if you have a product specification document prepared.
Key Takeaways
Partner with other PMs to understand all other KPIs for the product.
Quantify the impact to all other KPIs due to previously quantified change in user behavior.
Peruse your backlog to assess the urgency of solving this problem.
Cool, now it’s time to discuss this opportunity with business, product and engineering teams to make sure your whole team is aligned in your thought process.
If you made it this far in the article, amazing, grab a snack. Just 2 more phases to go in this article!
Phase 1C : Get aligned with the broader team on the new development
As a product manager (PM), you’re the architect of future product developments. This involves collaborating with various teams. For instance:
You strategize with business development to assess timely product innovations.
You work with marketing to create press announcements.
You keep customer success informed for seamless support.
You inform design and engineering about upcoming features.
Given the significant opportunity you’ve uncovered for the chat product, maintain stakeholder involvement throughout—from inception to launch. A pro tip is also to chat with these partners regularly, giving them updates about the product's progress towards meeting yearly goals. This ensures retained context instead of having to catch them up on everything from months ago.
To articulate yourself in the best way to the stakeholders, I’d suggest creating a simple presentation which answers 3 crucial questions - What, Why and How.
Here’s how I’d do it for our scenario :
What is the Problem?
Issue: Despite a 5% month-over-month growth in new users visiting our mobile app, they show limited interest in viewing merchandise or purchasing tickets.
Impact: As a result, our Chatbot Daily Active Users (DAU) has dropped to 6.2 million, falling short of our yearly goal of 8.4 million.
Why does it matter?
Context: With the ongoing initiative, the net new users contribute to the overall engagement.
Consequence: Failing to get the new users to engage with the chat regularly brings down our overall DAU to 6.2M, which is way lower than our goal of hitting 8.4M DAU by EoY.
How do we fix this?
The solution is to investigate ways to get these new users to engage on the chat far more than what is happening currently.
The opportunity it brings is this: Growth in DAU by X% within the net-new users can improve DAU overall by Y% and bring in $Z M ARR due to sales of merchandise and tickets.
Note that in answering “How do we fix this?”, you’re not yet proposing what the solution will look like. You are merely proposing the direction to take to meet certain goals.
If you want to do an even better job, I’d look into metrics that each of these teams care about as part of their yearly goals. For instance, business development cares about competition & customer acquisition costs (CAC). You could add in these metrics as well:
The Chatbot engagement on our competitors’ apps (X, Y, Z) are growing steadily while ours is steadily declining.
CAC increases by A% if we assume B% of new users switch to X, Y or Z apps.
I would suggest doing a dry run of this with a peer before presenting to leads because you don't want to be “the PM who cried idea”. Once your thought process has been vetted and your stakeholders agree with the fact that this is a problem worth solving, you not only have their support but are also ready to make the app better for these new users!
The final step is to restate and solidify the goal that everyone has mutually agreed upon chasing.
Goal : Grow the daily active net-new users for the chat by X% by EoY.
Key takeaways:
Prepare a simple presentation covering key aspects of the problem and opportunity - The What, The Why and the How.
Incorporate stakeholder friendly metrics into your presentation for more impact.
Address stakeholder feedback and incorporate changes.
Solidify and restate the new goal.
Phase 1D : Let’s understand the new users better.
What better way to understand more about users than actually talking to them?
First, I’d get a high level understanding of user groups and slice and dice them based on several characteristics. This is called, creating user personas. Personas keep the team focused on user needs rather than assumptions. They serve as a constant reminder that the product exists to solve real-world problems for specific individuals.
Here’s an example of a sample group that after much data pulling, represents 70% of the new users coming into the app, which we will call out top user persona.
User persona : People living in US within the age range 25-35 who don’t follow any sports teams/events, landing on our app’s chat because they clicked a promotional link upon opening the streaming service. In short, these users are sports newbies!
The next step involves collaborating with the user research team to create a survey.
This survey will be administered to several users who fit the majority description, specifically for our focused user study. Once we have gathered their insights, you’ll continue working with the user research team to formulate a set of questions. These questions should encourage users to discuss their experiences, pain points, and challenges related to your product.
Keep in mind that these users are unfamiliar with sports, so your product might be a bit of a mystery to them. In this context, it’s essential to encourage users to share their general difficulties when it comes to incorporating sports into their lives—after all, that aligns with the company’s mission!
What I’m describing is really a super dumbed down version of how to partner with a UX research team to conduct an efficient study of users to understand their pain points with the app. When you’ve got the time, give this, this and this a read. Or, wait for my article on this very topic!
Assuming you’ve combined the results from focused user studies and survey responses, ideally using a mix of both to mitigate biases from in-person interviews, here are some aggregated stats based on feedback from a subset of people within your focused user persona:
10% of new users seek information about upcoming local matches or sports events in different cities they might be traveling to.
30% of new users struggle with choosing birthday gifts for sports-loving friends or family members.
40% of new users want to learn about sports teams followed by their friends or family but lack a starting point.
Based on these problems, several opportunities immediately present themselves to you to better the app. These ideas could come to you via brainstorming, while doing market research, or even the shower. Note down every idea!
Filtering through these ideas will become easier once we reconsider our goal - increase Chatbot DAU for the new app users, ie selected user persona. Let’s go through the options and pick one that could increase the chatbot DAU in a couple of months and add the rest to the ideas backlog!
Clearly, acting on problems #1 & #2 will further app monetization immediately. However working on fixing problem #3 will systematically improve the engagement on the chat, which is more inclined with our goal.
So 1 & 2 are either:
Picked up because they have huge monetization potential after a round of evaluation.
Picked up because they are small enough to tackle alongside other work that’s going on.
Dropped because they don't have a high enough monetization potential at the moment.
Referencing our previously mentioned user funnel, we see that solving for problem #3 will impact a larger chunk of users thereby increasing the daily prompts on the chat and will help us meet our goal of net-new DAU by X%
So we’re going with solving Problem #3, which can be stated as follows:
Problem: Many new users aspire to actively participate in sports activities with their family and friends. However, for beginners, there is a noticeable lack of user-friendly resources in the market to learn about sports topics comprehensively.
Opportunity: Our app’s chat functionality currently focuses on answering specific questions related to sports, primarily catering to individuals who already possess substantial knowledge about teams, events, and sports personalities. However, there exists a fantastic opportunity to create a welcoming starting point specifically for sports newcomers. By doing so, we can ignite their interest in sports and provide them with accessible pathways to learn and engage with this exciting world!
At this point, you start listing down a whole bunch of solutions!
Daily quiz which opens to the app and provides info about all the options.
Movie or TV show suggestions around a topic of choice
And many more ideas!
Key takeaways:
User Personas: Create and prioritize 1-2 dominant user personas.
Research Collaboration: Craft unbiased questions for interviews/surveys with the User Research team.
Identify Pain Points: Summarize user pain points related to the product and its topics.
Goal-Oriented Solutions: Solve pain points aligned with the stated goal.
Document for Roadmaps: Record all pain points and ideas for future improvements.
Phase 1E : A solution presents itself!
The next article will focus on:
Picking a solution to build.
Understanding product market fit for AI products.
Brainstorming with Design and Engineering.
Requirements formulation for AI products.
Much more!
If you made it this far in the article, thank you. I enjoyed reliving Phase 1 of having launched a product I dearly cared about. :)
If you think someone else might benefit from this, give this post a share and subscribe to my newsletter for more such content. Cheers!
Great post, Nimisha! Thanks for sharing. How are we defining a fall in yearly DAU to $6.2M?
Also curious to know how much of the documentation or user research work you have to do yourself? Given the XFN collaboration wouldn't want any team's waiting of details from PM.