“Evaluate trustworthiness and reliability while gauging emotional impact of AI interactions”
Challenges
To strengthen their respected brand, Mercer Master Trust decided to develop an AI pension agent capable of providing tailored information and guidance. The business was already prioritising usability and satisfaction with a traditional UX interface, but this expansion sought to integrate a human-AI experience and create a more interactive, intuitive platform.
Mercer knew they had to ensure the advisor addressed real user needs for the project to succeed. Understanding that emotional design plays a crucial role in the human-AI experience, they set out to evaluate how users would perceive the tool’s trustworthiness and reliability while also gauging the emotional impact of interactions with the AI.
“Reach beyond traditional UX research methods and utilise sophisticated biometric responses”
The REO approach
To de-risk this human-AI experience, REO evaluated user trust and value in Mercer’s AI pension agent by reaching beyond traditional UX research methods.
These methods can fall short in the world of human-AI interactions as these systems learn and adapt over time, changing behaviour at a rapid pace. They involve a complex emotional choreography in user interactions.
REO utilised more sophisticated and contemporary biometric responses such as heart rate, skin conductance, eye-tracking, and facial expression analysis. These tools offer real-time objective data on users’ emotional and cognitive states during interactions with the AI agent.
REO recruited 12 participants for 90-minute in-person testing sessions, ensuring that a variety of people were involved. Testers ranged from Mercer Master Trust members and new employees only just starting to think about their pensions, to people approaching retirement who are increasingly thinking about their pensions.
Tracking Biometrics
REO’s Dr Thomas Hein interviewed participants and observed their interactions with the AI agent in our London Biometrics Lab, a controlled environment where a suite of biometric sensors are used to collect real-time data, including:
Skin Conductance – To measure stress responses.
Eye-Tracking – To analyse attention, engagement, and cognitive load using pupillometry.
Participants performed a series of tasks using the AI agent designed to mimic realistic pension guidance scenarios.
This approach afforded REO a way to navigate the maze of self-report biases and probe each user’s behavioural interactions and emotional choreography throughout their experience.
Heart Rate Monitoring – To assess anxiety and physiological arousal.
Facial Expression Analysis – To interpret emotional states and reactions.
“Deep, actionable insights for enhancement to ensure an elevated user experience”
The outcome
The advanced emotional biometric research conducted offered the Mercer Master Trust valuable, in-depth insights into how their members interacted with the AI pension agent.
Personalisation Enhancement
Customers expressed a desire for more tailored responses, but the prototype AI pension agent does not yet have access to users’ personal data. Given that the AI will interact with logged in users who have established pension accounts, this confirms that there is a clear opportunity to leverage more personalised guidance, rooted in data, for members.
These insights were highly actionable and revealed specific patterns in member behaviour, which could drive improvements. For example:
Boosting Trust around Personal Data
Some users were explicitly wiling to allow the AI agent to have access to their personal financial data. Others were vocally willing, but physiologically anxious, as shown in the biometric of reduced heart-rate variability. Another was both explicitly and electrophysiologically anxious. Informing users that the AI was internally trained and fact-checked increased all users’ perceived confidence, presenting a key opportunity to build trust and credibility.
Improved Conversational Accuracy
Biometric feedback, including facial recognition data, highlighted moments of frustration when users needed to rephrase questions to get the guidance they expected. This insight represents an opportunity to boost user trust and satisfaction by enhancing the agent’s ability to interpret and respond to variable input.
Enhanced Guidance Mechanisms
While the AI agent provided helpful directions, users suggested it would be helpful if it would prompt them to get started with conversations. A stress response was captured when some users felt at a loss as to how to start or continue a conversation without suggestions or recommendations. Mercer will strengthen the AI agent’s ability to guide users as to what the next best questions might be, improving trust and engagement.
Increasing Engagement with Multimedia Features
Participants expressed a preference for the AI agent to proactively offer helpful resources and guidance, expecting interactive elements to make the experience more engaging. These insights furnished Mercer with actionable recommendations on how to enrich user interactions.
Elevated User Experience
These insights have equipped Mercer Master Trust with a clear roadmap for refining their AI-driven pension agent, ensuring it meets the evolving needs and expectations of their users while enhancing trust and engagement.
