Creating a culture of experimentation
Industry
Fintech Payments
The team
Lead Product Designer (me), Lead Product Manager, Engineering Manager & Data Analyst
My role
I led design across discovery, hypothesis planning, stakeholder management and execution:
- Facilitated workshops to generate experiment ideas
- Created hypotheses for experiments
- Led managing internal and external stakeholders to get experiment buy in
User Impact
- Improved user experience - Being able to experiment quickly on and optimise the experience for the end user has provided a positive impact.
- Faster access to new features - Experimentation has allowed us to be confident prioritising user centric features like Gift card swaps and introducing Payment products.
- Less risk of negative impact - On the flip side to the above, experimentation has stopped us from perusing and shipping changes that would negatively impact the user experience.
Business Impact
- Increased revenue - In the first 6 months of testing, we prioritised and launched one feature which resulted in an estimated £1m additional revenue for Runa (Total yearly revenue ~ £9m)
- Increased confidence in shipping features - Testing appetite for features has led to an easier and better informed prioritisation process for the product teams.
- Improved partner relationships - This one definitely took work, as you will see below, but experimentation has cemented Runa as a growth partner for our customers and merchants.
The Challenge
Runa is a B2B2C fintech looking to build the next-gen global network, allowing businesses to pay, reward and send digital value anywhere in the world as easily as sending a message. Runa started off mainly in gift cards and it’s network of gift card merchants worldwide is it’s biggest USP.
As the lead designer, I set out to:
- Establish a culture of experimentation inside Runa
- Prove the value of testing through an early, high-impact experiment
- Build processes that customers and internal teams could trust
First Steps: Workshop & Hypothesis
I facilitated a multi-disciplinary workshop (Account Managers, Partnerships, CX, Engineering) to surface pain points in the recipient journey.
We mapped these into an Opportunity Solution Tree and prioritised our first experiment.
Experiments workshop through Miro
Hypothesis
One of the key insights that we uncovered during the experiments workshop was that there was currently an issue for both our customers and their recipients when they were redeeming their gift cards.
Customers - When their CX teams are trying to cancel orders, they were accidentally clicking on the links provided by Runa which then opened the gift card and charged the customer for the card and made it so it could not be cancelled.
Recipients - If there is an issue with their gift card, they are currently presented with a generic error page that does not give them enough information around who to contact or how to fix the issue.
By adding a confirmation step before recipients accessed gift card codes, we could:
- Prevent accidental redemptions by external CX teams
- Reduce user confusion when errors occurred
Experiment 1: Gift Card Confirmation Step
Test: Add a confirmation step before code reveal
Duration: 24 hours across 100% of traffic (~1 million visits per month)
Measure:Drop-off, error rates, accidental redemptions
Gift card redemption flow before
Gift card redemption flow after
Experiment 1: Results
Key results are were follows:
- 97.9% completion (minimal drop-off)
- 14.5% reduction in support tickets for recipient errors
- 100% elimination of accidental CX redemptions
- £1m/year in savings from reduced refunds and support overhead
This experiment proved the value of testing and won early buy-in across Runa.
Scaling Experiments and Hitting a Roadblock
After the big success of the first experiment, there was an eagerness from the business to really push forward with experimentation. For our next rounds of experiments, we focused on the hypothesis that certain recipients wanted the option to swap their gift card for a different one, or for a different form of payment.
We were running 2-3 experiments per week, testing different swap offerings as well as different pricing structures related to the swaps.
Different Swap Option Experimentations
As we tested more, what we were testing became more visible to our customers, who were receiving feedback from their recipients about the different options we were offering in our experimentations. It wasn’t long before customers started asking could they be excluded from our experiment traffic.
This was a big challenge:
- Customers began receiving feedback from recipients about tests
- Many asked to be excluded, and soon 90% of traffic was off-limits
- Time to statistical significance grew from ~24h → 4–5 days
Experimentation stalled.
Notion page with exclusions list
Reframing Experimentation with Customers
To try and regain trust, I worked with Product marketing to create some transparent comms decks explaining:
- Why we experiment
- How tests are run
- The benefits for customers, recipients and merchants
As part of this, I also redesigned our experiment governance, moving away from an exclusion list:
- From “Exclusion list” → “Themed opt-ins”
- Customers could opt into experiments relevant to their use case (e.g. swaps, pricing, redemption flows)
- This reframed testing as a choice and partnership, not a risk
Marketing deck example
Experiment themes
Building a Culture of Experimentation
Today, experimentation is embedded in Runa’s product process:
- I co-chair weekly experiment stand ups and ideation sessions across multiple teams
- We now have a shared Experiments Roadmap in Notion, with templates for hypothesis, setup, and outcomes
- Our customers and merchants are now engaged as partners, and we proactively share insights from tests they are involved in to help build trust
- Experimentation has had a direct influence on roadmap prioritisation: led to launching Reloadable Prepaid Cards and Pay to Bank as well as building out the functionality to swap gift cards for payments and vice versa
Notion experiment template
Reflection
A big part of what I have learned in this process, is that in the B2B world experimentation is as much about trust and communication with your key stakeholders as it is about the actual testing processes.
As a designer, I was able to provide impact:
- Facilitating alignment through workshops with key stakeholders to identify areas we could experiment
- Visualising, prototyping and pitching experiment ideas to stakeholders to get buy in
- Working with Product Marketing to create template decks that could be easily filled out and shared with our customers after the conclusion of an experiment
- Evangelising experimentation with colleagues and C-level stakeholders to create a culture where Runa is constantly learning and has become a growth partner for our merchants and customers
Experimentation at Runa is now a driver of revenue, innovation, and customer trust.
Other projects
Building out Payment products at Runa
Evolving Runa from a gift card network to a global payouts infrasctucture with the addition of flexible FIAT payment products.
See more
Designing accessible car insurance forms
How does the structure of a car insurance form impact the actual time taken, perceived time taken and ease of use for all user types.
See more