The two metrics I’m constantly told to optimise as a startup founder came dead last in a panel on responsible AI companions.
Only 24% of the audience picked engagement and retention as important measures of a 'good' companion.
Ironically, those are among the first numbers I’m expected to prove.
Last week at Lorong AI, I joined Kenny Choo, Gerard Yeo and Hakim Norhashim to discuss relational AI across human behaviour, computational psychology, AI governance and the messier reality of actually building one.
Kenny excellent at catching the tensions between our answers instead of letting us agree politely for 2 hours.
One of our best debates started with a simple poll.
What does 'emotional safety' actually mean? Is wellbeing how someone feels after a conversation, or what she DOES afterwards? Does user satisfaction mean the same thing for Maibel, where the relationship is meant to support wellness behaviour, as it does for a role-playing companion?
Most of the room prioritised emotional safety, wellbeing and healthy offline behaviour over engagement and retention.
Companies have clean success metrics: did she return, how often did she talk, did she convert?
But the audience wanted us to measure what happened to the person because she returned.
Gerard's work on emotion and trust exposed the flaw in judging AI responses in isolation. The same reply can reassure one person, alienate another, or land differently after months of accumulated trust.
Our audience gave AI an average trust score of 2.5/5 for something personally important.
I don’t think the goal should be getting that anywhere near 5.
AI can sound caring and confident long before its judgment deserves that confidence.
Hakim's work in AI governance exposed the next gap: regulation needs standards, but relational AI resists clean labels.
If the same pattern can look like healthy engagement or unhealthy dependence, accountability gets difficult fast. Both can look remarkably similar on a dashboard.
The audience leaned towards shared responsibility for keeping AI companions safe, yet few placed regulators first.
I’m less convinced regulators can stay that far away.
Startups are rewarded for growth. Retention is traction.
If responsible design sometimes means choosing the less engaging behaviour, relying entirely on companies to absorb that commercial trade-off voluntarily is a weak system.
This tension is already encoded into how we build Maibel.
Before our companion gets to coach, encourage or simply continue the conversation, our behavioural cascade checks safety, emotional state, receptivity and whether she has already told us no.
Sometimes the response most likely to increase engagement is exactly the response we should NOT send.
I still need women to come back.
But if retention rises while her agency falls, we’ve built for the wrong kind of retention.
Hi, I’m Mabel.
I’m the founder of Maibel. I write about emotional AI, women’s wellness and learning to build the things I wish existed.
More about me
