Pro Tips For Building Your AI Product Management Skillset

The Importance of Seamless AI Integration [06:46]

Lorilyn emphasizes integrating AI deeply into product design, avoiding an “add-on” feel.
Ideal AI integration makes features feel natural and unnoticed as “AI” by users.
Superhuman’s “Instant Event” feature illustrates this approach, allowing users to create calendar events from emails with a single click.
Initially, the feature was in a less accessible location for testing purposes, using Superhuman’s command menu, “Command K.”
After testing and learning from feedback, the team relocated it to more intuitive spots, like a button in emails and a calendar shortcut.
This seamless integration ensures AI features feel useful and accessible when needed.
The “Instant Event” feature now allows users to create events by simply clicking on dates in emails, with the cursor changing to a hand icon to signal this action.
Initially tested in the command menu, the feature is now fully integrated for ease of use.
Users naturally discover and use the feature without explicit instructions, which aligns with Lorilyn’s goal for seamless AI integration.

Optimizing AI for Speed and Accuracy [10:54]

Superhuman prioritizes proactive features like auto-summary and instant replies, displaying them instantly for user convenience.
The auto-summary feature provides a one-line summary of every email, enabling faster information access without requiring user action.
For instant replies, the team faced a trade-off between a “turtle bot” (smart but slow) and a “fast, awkward uncle” model (quick but less accurate).
Despite the turtle bot’s higher quality, the team chose the faster model, prioritizing speed and user experience.
They fine-tuned the faster model to improve accuracy, ensuring responses are quick and reasonably accurate.
Balancing quality with speed is essential in deciding whether features should be proactive or user-initiated.

Technical Decision-Making in AI [14:24]

Lorilyn uses a tool to compare multiple prompts and models side by side, assessing performance with real data.
She creates a dataset from internal user feedback (thumbs up/down) to reflect real-world scenarios.
Evaluations focus on latency, aiming for responses within 1–2 seconds to meet user needs.
Through repeated testing, she identifies issues (e.g., misinterpreting context in replies) and refines prompts to improve accuracy.
Iterative testing helps establish success criteria, balancing speed with accuracy and learning the specific types of accuracy needed.

Mastering Prompt Engineering [16:38]

Lorilyn developed prompt engineering skills through on-the-job learning and experimentation.
Occasionally, she researched online for specific solutions.
She also received tips from OpenAI, like using techniques such as all-caps commands, repeating instructions, and including one-shot examples.
Fine-tuning involved adding examples to address challenging cases and improve accuracy.
Lorilyn used a technique where she had the LLM explain its reasoning to refine prompts.
For example, she would ask the LLM who it was replying to and why.
This process revealed misunderstandings in the LLM’s responses, allowing her to make specific adjustments to the prompts.
The “explain yourself” method helped clarify the LLM’s thought process, aiding in more effective prompt tuning.

Business Challenges in AI Integration at Superhuman [18:54]

Integrating AI into Superhuman involved balancing complexities and challenges.
Lorilyn compared AI models: GPT-4 (slow but high quality) vs. GPT-3.5 (faster but less smart).
Initially, GPT-4’s slowness posed significant challenges in scaling email summarization.
Superhuman prioritized quality and user needs, considering speed and correctness.
They chose GPT-3.5 for better latency while employing prompt engineering to meet quality standards.
The business focus was on ensuring user needs were met, regardless of costs.

Hands-On Learning and Practical AI Skills [20:54]

Interview Highlights