Innovating philanthropy: leveraging insights to transform donor education through AI

Thank you so much, Amy. And, yes, it's true. I can still make the cinnamon rolls from scratch, and that's a that's a job that you never quit, let me tell you. They still keep making them to this day. But I wanna thank you all for joining me today. This has been a really amazing conference with a lot of really exciting speakers. And so I am just so humbled and grateful that you're joining me today.

I'm really excited to talk about how Giving Compass is using user testing insights to leverage, insights into generosity through donor education using artificial intelligence.

And so just a little bit about me. My name is Madeline Allegria. I'm an impact analyst at Giving Compass where I've been for over four years now. I also hold a bachelor's degree from New York University and a master's in public policy from the University of Washington in Seattle, which is where I'm coming from today. My role at Giving Compass is to lead internal research and to collaborate with our product and engineering teams to help, better our technology and our products using the research innovations and insights that we've gleaned through some of the research that we've done in our collaborations with user testing.

So before we get started, I just wanna let you know what you can expect, throughout my presentation. I wanna talk to you a little bit about some of the problems that we're seeing in the philanthropic sector, a little bit about Giving Compass and our methodology and how we think we can address those problems with AI. And then I wanna talk you through a very specific use case of how we've used user testing to help cultivate digital trust amongst our donor users, and the nonprofits in their communities.

And then I'll finish up by telling you a little bit about how we're leveraging these insights for future innovations.

So let's get started about donor decision paralysis.

So what's the problem? There are over one point eight million registered charities in the United States, and that is a lot. So when you are sitting there trying to make a decision about where to put your money to make the greatest impact, especially if you're new to philanthropy or new to giving, you might go to a Google or a chat GPT and try to get some out of the box solutions or recommendations.

And what you'll find is that these, recommendations are not tailored to your individual goals or values, and that they're typically churning out the top ten list that they're getting to every single person who asks that question. And they're not really gonna give you that tailored recommended advice that you would get from, just being in your community and knowing about the issues that you care about the most.

So what's missing is a trusted guide that simplifies the process by giving you information and resources that are backed by experts in the sector and align with what matters most to you.

And so just to give you a visual, we're seeing a donor journey that looks something like this. A donor will come in and they're very aware of the issues that matter the most to them. And they're very aware of how they're defining, that impact and what that impact can look like in their communities.

But they don't have access to the most relevant tailored information that's really guided by trusted resources in the sector.

And because of that, they're unable to discover and trust and connect with organizations that are specific to their cause area or their community, and so they're unable to track or plan the impact of their donations and their giving.

And so what we're hoping that this next generation of artificial intelligence can do is help streamline that information process. So that way, donors are able to have a seamless experience from understanding and being inspired by an issue or that they care about. And then immediately are able to get access to the information that is critical to them understanding how they can make a difference in their community or cause area.

And then from there, we wanna connect donors with organizations that they can trust, that have been vetted, by experts in the sector. So that way, they can start giving with trust and impact and ultimately be able to define and achieve, the results that they wanna see in that issue area or in their community.

Ultimately, we think that this this being able to, define the impact will help them to track and plan for future giving. And, ultimately, when they have a giving strategy that works for them, they'll share and advocate with their friends and family and get new donors back into that cycle where people will have a seamless journey to finding organizations and advocacy for the organizations and causes that they care about.

So what about give Giving Compass? Really, at Giving Compass, we know that giving is deeply personal. And so because giving is so personal to your individual values and your goals, we know that an out of the box solution or recommendation or plan for giving is not going to work for everybody.

We know that we're gonna need tailored advice and tailored information from trusted sources, from sources that we can track and that we understand where that information and data is coming from. So that way, we can encourage donors to so we can tailor our recommendations to donors that are very specific to their values and goals for their giving journeys.

So for some context, I wanna give you a little bit of a background now about Giving Compass, and our methodology as well as some of our partners.

So Giving Compass was founded in two thousand and seventeen by Jeff and Trisha Raikes.

Jeff and Trisha Raikes were early, employees at at Microsoft and then went on to lead the Gates Foundation for a number of years before starting their own foundation at the Raikes Foundation in Seattle, Washington.

Early in their giving journey, they found challenges finding targeted resources that were guiding this values aligned community inclusive and ultimately effective giving strategy. And so thus, Giving Compass was born. So today, with over a hundred thousand monthly users, Giving Compass is a donor education and artificial intelligence technology platform, and we have started to take a behavioral approach to helping donors be more thoughtful and impactful givers. So by aggregating data across the nonprofit sector, including vetted nonprofits, research best practices advice, and donor education content, as well as video and audio issue funds and volunteer opportunities, we're really striving to guide donors to give more, give more frequently, and to consider giving to smaller local and grassroots nonprofits, a little bit like what you see right outside of that table with Kuritos.

She's another great example of the types of organizations that we really want donors to consider giving to, and we wanna help connect.

And so like I talked about before in the donor journey, and you'll hear me say journey a lot because we're all on one. And it's the same here at Giving Compass, but the join the streamlined donor journey looks a little something like this. So a donor comes in. They are inspired by an issue area or an organization that's near and dear to their heart. They are immediately able to access all the information from trusted resources where they can track the sources verified by experts in the sector, and then immediately are connected with personalized recommendations for nonprofits in that issue area that have been vetted again by experts in the sector.

And once they're able to start tracking their gifts, they're able to, define what impact looks like for them and start working towards achieving that impact in their community, plan giving for the next year, and then ultimately share with their friends and family to get more people back into that cycle. So Giving Compass thinks that by dividing or by bridging that information gap, that we can help streamline this process and start to aggregate some of the data that's been so scattered across many different sources, especially in the nonprofit sector.

So how do we get there?

First and foremost, we believe that donor education is a behavior change challenge.

So like Anoom for weight loss or a Thrive Global for mental health, we know that if we want donors to be more thoughtful and intentional givers, that we have to get them to change their behavior. That means getting them to sit down and be intentional about the ways that they're giving and creating a giving strategy that works for them. So to do that, we know that we need to take a science backed approach, and that we needed to keep the human element at the forefront of our of our strategy.

Because giving is so personal, we know that really just a standardized or generalized advice would not work at keeping people engaged in the long term.

We also know that the way that digital search and exploration with the onslaught of AI is rapidly changing what is possible for the nonprofit sector. So just to give you an example, all the information that's aggregated on nonprofits is all comes from a single source, the IRS nine ninety database. And that database is a series of downloadable PDFs, which if you've ever tried to click on a PDF, it takes you a very long time to get to, the information that's needed. So in order to we're we found that by using AI, we can quickly scan and read and aggregate that data much more quickly. So that way, it's easily searchable, by donors who are looking to give for very specific causes in very specific areas.

And so we're also questioning how we can leverage this type of data ethically. And so that way, we know that we are guiding donors towards the most impactful organizations and the organizations that we really think are making the biggest difference. We also know that the way people want to learn is changing.

Giving Compass is first and foremost an education platform. We do not accept, donations on behalf of organizations. We really just wanna connect donors through that pipeline. And so we know that content is gonna be at the forefront of that. Gone are the days of extensive long reports, that people don't really read fully. We know that donors want concise and engaging content, and we're moving more towards a video based learning model. And, ultimately, we understand the power of a platform.

We know that a comprehensive platform is essential to getting donors to, from point a to point b in the giving journey and using nudges and other behavioral science backed approaches that we can in help influence donors from step to step.

And so we are not alone in this ecosystem because our AI powered tool is, is is, uses con content that is syndicated from partners across the sector. We know that the people that we work with are the we have to trust the most in order for our data to be reliable and consistent. So we work and cater towards high net worth individuals as well as donors who are interested in getting serious about their giving strategy. We work with other giving platforms. We, of course, work with an broad range of nonprofits across the country, as well as financial advisors in firms and community foundations. So some of our partners include Schwab and Fidelity Charitable as well as other, community foundations across the country.

Okay. So I've told you a lot about the problem, and I told you a little bit about giving Compass. And we know now that there's a clear problem. Hopefully, I've convinced you of that, but there's an information asymmetry, and we wanna provide a solution.

Ultimately, we wanna provide a better and more comprehensive, smarter, AI powered search, but we're still exploring how we can bridge that digital trust divide. We know that giving is deeply personal and that the tools that digital learners are using are paramount to getting them to give more effectively. But we also know that there's a lot of mistrust in the nonprofit sector and there's a lot of mistrust in the AI sector. And so bridging that digital trust is gonna be crucial if we wanna help change donor perceptions of this big scary new technology and the nonprofits that are doing the amazing work in their communities.

And so what we did is we decided to run, a very interesting experiment using a very niche portion of our audience.

So in February of this year, we surveyed eighty eight active DAF donors. And for those who are unfamiliar, as I'm sure most people are, very niche part of the philanthropic sector, a donor advised fund or a DAF is a tax advantage philanthropic vehicle that allows donors to make an irrevocable contribution and receive an immediate tax deduction.

These accounts are allocated for philanthropic giving. So once the money goes into these accounts, they have to be spent on philanthropic grants.

Why DAFs? Well, we partnered with a lot of DAF institutions, including a number of banks, including Schwab and Fidelity, And we really wanted to see if this technology could be implemented with some of their users and some of our users. And we also know that DAF's or DAF account holders are very sophisticated donor audience.

These are really amongst the highest net worth of individual donors with an average estimated income of over one point three million, and DAFs are extremely popular amongst this demographic. So our thought is that if we could help convince this type of donor, perhaps that we could then trickle down that effect towards donors who are newer to the philanthropic ecosystem.

So in February this year, we launched a study. We recruited eighty eight DAF donors, which is absolutely incredible because this is a very niche sector of the population. We found all of these users through our user testing platform. Forty nine of them were men, thirty nine were women. They ranged anywhere from age twenty six to seventy two with the average age being forty four.

And so in that session, we surveyed eighty eight of them. We asked them a variety of questions about their past giving habits, their giving behaviors, how they were thinking about philanthropic giving, when the last time they made a donation was, all this type of information.

And we got a lot of great quantitative information about who this niche audience of users were that were using our site.

We found things that they cared about the most in the philanthropic sector, including that their donation dollars were used effectively and that really we found that they wanted to see that their donations were improving people's lives.

But it really validated some of our early theories that people know what they're interested in, and they know what impact looks like.

So we really thought, that this is a great set starting point for developing this type of audience.

But what we also found is that there was a mismatch between what the donors were telling us their most impactful organization or donation was and what their most recent donation was. So in addition to all this great, quantitative information about our users and what they were doing and what their behavior was previously, we also got a lot of really rich qualitative information about our donors.

We this donor in particular really stuck out to me.

As you read her story, she talked about the most impactful donation was giving to a small, local arts community or nonprofit in her community. She talked about giving that gift in memoriam of her mother who had recently passed away because her mother was previously a retired artist and who gave that up to raise children and how she was really looking for ways to honor her mother's wishes. And she's not the only one. We got a number of stories from these donors who were telling us all of these amazing things about what giving meant to them, about what an impactful gift meant to them, and really about the different types of ways that they were engaging with giving.

However, when we asked these donors where their most recent donation was, the majority of these donors told us that they were giving to major national or international chapters of large organizations. So these are organizations with net assets over a hundred million, over five hundred million, really big organizations. And these organizations are great. Do not get me wrong.

We really do think that they are part of a healthy philanthropic budget. But we also know that those dollars can go a lot further at a smaller organization that's working on issues in your community, helping your neighbors, your friends, your kids' neighbors, your kids' friends, etcetera, etcetera. And we also know that the donors were telling us that those were the most, impactful gifts that they were making in a calendar year. And so it really validated to us that there is this information asymmetry, That when they when these donors go and think about the causes that they care about, they know what they care about, and they know what they wanna give to.

But when it comes down to make a donation at the end of the year, maybe for tax purposes, maybe for end of year giving, that people default to what they know because the search tools are search constantly surfacing the same results over and over and over again, and they're not tailored to what these donors want to see or what they care about the most.

And so we wanted to change this we wanted to change donor behavior, and we were experimenting with different ways that maybe content could do that. Maybe the different types of content that we were syndicating in our AI search could help change the ways that donors were thinking about organizations and maybe influence giving them giving to different types of organizations. So the second part of our study involved having, the donors go and explore our site. We asked them to choose any nonprofit that they could think of, and read just simple biographic text for that organization. So you might get, like, mission information, some basic financials, basically, just a great summary of what the organization was doing.

But then we had them watch a forty second video on this nonprofit.

And the the forty second video was for a nonprofit called Amarias Art Academy. And Amarias Art Academy was a grassroots nonprofit in the Atlanta, Georgia metro area. And it's an organization that is helping teach students about, comic book that comic book industry, comic book illustration, as well as teaching them skills that they would need to go and pursue that career. So it just showed some some, program staff.

It showed some of the program recipients. It talked about, the the founders of the organization, etcetera, etcetera. Oops. Sorry.

Can you go back one?

And after both, conditions, we ask the donors to rate their agreement with the following statement. I have enough information to donate.

And now what we found is a fifteen percent increase in willingness to donate using, after watching the video. This type of, improvement in donation readiness is extremely crucial in the nonprofit industry.

It's hard to overstate how, in the private sector, we might understand technology and we might be quicker to adapt to technology. But in the nonprofit sector, due to a number of resource constraints, technology adaptation is really slow. And so understanding that there's this much of an increase in a willingness to donate after watching a video really changes the game for a small nonprofit and when they're spending their resources on allocating and reaching a different type of audience.

And so not only did we see this increase in the willingness to donate, we also saw a whole bunch of metrics and an increase in trust amongst the donors. So this helps extremely bridge the the trust divide that that you might see between a donor and not only our technology, but the donor and a nonprofit in their community. So we see over forty two percent of donors who then said, I can see the organization's impact.

I can see the program staff, and I can see that the founder are credible. And I can I want to learn more about this organization?

This is an incredible development for connecting donors, with nonprofits in their communities, as well as connecting nonprofits to donors that might be interested in their cause, all using technology and artificial intelligence to do so.

And so, again, not only do we see that in the quantitative data and the quantitative analysis, we also see it again and again in the user testimonials.

So as we're going back and watching the the each user testing user, all of our donors, and seeing what they're doing, they're telling us, wow. I would have never considered this organization. This is not something that I really would have known was out there. But after watching the video, I know what they do.

There's a face to this organization. It feels personal. They feel connected, and it really sinks in for them how unique the programming is. Again, when you're a small nonprofit and you're operating on a, operating budget of maybe less than a hundred thousand dollars, a forty second video to increase donation readiness by fifteen percent in in a donor population amongst people that you have basically no.

It's basically virtually free.

That is a huge, huge, huge development for nonprofits. And if we can aggregate and syndicate that type of data, we're doing a huge service to the sector in bridging that technology divide.

And so how do we leverage these insights for the future?

So we're not done. We are still at the beginning of our donor journey ourselves as an organization.

We are really definitely experimenting with the different types of ways that, we can use user testing to really understand our donor audience.

But really what we wanna know is that now that we know for sure, we can scientifically prove using user testing that this content is changing the way that donors interact with nonprofits, we want to know if it'll change the way that they give. So we really wanna know, can we get donors to now give to these nonprofits? Can we get them to give more than they would otherwise? And can we give them we can we get them to give to more impactful nonprofits?

So what's next? As we speak, I'm running several user testing surveys, and we're running a randomized control trial on giving donors up to fifty dollars to donate to a charity of them of their choice or to keep to themselves.

So we've recruited two hundred American donors, the user testing. I actually think that number is two hundred and eighty now. And yeah. And we've assigned half of them to our site to research a nonprofit and half of them to Giving Compass, or to the Internet as a control group.

And so what we've done is that we've we're asking them, to give us a moderated session where they're looking at they're researching on our site or they're researching on the Internet, and we're seeing what they do. We wanna see how many of them are going to a chat GPT, what they're searching for, what the results that they're getting back, are they satisfied with those results, and we wanna see the same thing on our side. Then the next week, we're we followed up with a survey to all of these donors to see, hey. Are you still researching?

What are you doing? How's that changed? Has your behavior changed? Has the information that you've gotten changed?

Are you satisfied with that information? Etcetera.

And then this final week, we've asked the donors to go ahead and give a fifty dollar donation to any nonprofit that they've chosen. And right now, we're currently analyzing whether or not, donors who are using Giving Compass are giving more and where they're giving. So that's a little stay tuned because it's still going on right now, but, just another idea of how we're using we're scaling this type of impact.

And so some other key takeaways, you know, that was a sneak peek of what we're we're looking for for the future.

But, throughout our testing with user testing, we have definitely learned a lot.

The first is improved UI and UX.

If there's anything that our experiments have shown us, it is that the visuals matter a lot. They matter a lot, not only in cultivating that trust between the donor, but in making the applications more user friendly and more visually appealing so we can keep time on the page at a higher. So what we've done is improve our loading messages to make them a little bit friendlier, to really adapt to the circumstances that they're these donors are looking in, as well as improve the visibility of our nudges.

So that way we're helping to guide donors from step to step saying, like, if you're interested in this cause, you might be interested in this organization. If you're interested in this organization, maybe you're interested in this one that's a little bit closer to you.

Things like that, as well as updating some of the information that we're displaying. We're also really trying to understand how we can diversify our data sources. So like I've said, and, like, everybody who's works who's worked or used any sort of AI will know that it's only as powerful as the data that you're giving it. And so we're trying to understand how we can make our data not only as diverse as possible, but as trusted as possible. We wanna make sure that when you're using our solution, that you are getting information that has been verified by experts in the sector and that we are guiding you towards nonprofits that have been vetted by a human in the sector that you're looking for.

And we're changing again the presentation of this data. So we're changing it to include a lot of video and visual content because we understand now from our research just how important and critical it is. Going forward, because we're not done yet, we are doing a competitive analysis using user testing again with some of our competitors at other charity meet charity rating sites as well as other artificial intelligence searches, as well as testing the effectiveness of our nudges and doing a stakeholder study and map to really understand the different types of donors, the different types of donor personas, as well as the different types of organizations that might be interested in using a tool like this.

But that is all I have for you today. I think I wanna talk a little bit fast, but that's I wanna thank you all for joining me. It's been, truly such an honor, and and thank you to Amy and the user testing team. It's been really great, but happy to answer any questions if you have them.

Well, before we do that, let's give Maddie a big round.

Thank you. Thank you.

That was great, Maddie.

Thank you so much.

Thank you.

So do we have any questions?

Hi.

I think I might have missed this, but it's it's a big chunk that I might have missed it. But could you walk us through, like, what the AI search tool that you you talked about, that, like, experience look like? Like, where does this live? How donors access this tool, and also how partners and nonprofits also input their data?

Yeah. Sure. So, you can go to give encompass dot org and use the tool right now. It's definitely we're, again, in the early stages.

The way that it works is that it's a closed source, art and you'll you'll know by me talking about this that I'm on the research side and not the engineering side, so you can tell you that right now.

But it's a closed source artificial intelligence chatbot.

And so what it it can do is you can go to this, or to this our site and type in, I'm in Austin, Texas, and I'm looking for an organization that's serving, homeless veterans. And what we'll do is we'll scan our database. We have access to all one point eight million charities that are, that are on the IRS database. But what we've also done is vet, human vet within a team a a team in our in our organization as is human vetting a whole batch of organizations to provide you and boost the organizations that we vetted across a number of criteria, including financial criteria, impact reports, metrics that are important to people in the sector.

So I think that's one part of your question. And the other part is, our tools are white labeled to our partners. So it for example, at a banking institution where you have a donor portfolio, a bit really big use case for us is financial advisers. So you might go to a financial adviser at the end of the year and say, okay.

Here's my financial strategy, and I'm I'm ready to make a tax deduction, and where where can I donate to now? So we found a really strong use case with financial advisors who are interested in using our tool to help, create a more personalized, approach to, philanthropic giving the same way that they would for a financial plan for a family.

And so our tools have been white labeled under their sites as well as, they've helped us a lot in the development of this.

Did I get all the parts of your question? Okay. Great. Just making sure. Thank you.

Hello. Oh, thank you. Great talk.

I guess my question is, are you using AI with as you have your donors trying to find those nonprofits, do they get an influx of donors? But how do you leverage that tool to make sure there's actually sustained donors for those local communities?

Yeah. That is a really great question, and that's a problem that we are actively working on right now and something that I talk a lot or I've mentioned a lot about the nudges, but I'm happy to share more.

Yeah. That's we want donors to not only connect, but also keep giving to that organization over and over again. And so we really encourage donors to plan their giving because we know when donors sit down at the beginning of the year or the end of the year, whenever you do your financial planning and you're making a plan the same way that you would for a financial plan, that you're keeping that organization in the top of your mind, and you might have a recurring donation for perpetuity.

And so that's part of us, trying to keep donors engaged on the site, keep donors engaged in the cycle because we know that once they get to the tracking and planning stage, if they've already made a gift and they've made it this far, if we can get them to keep that organization in their plan, we can keep those donations coming through. Hopefully, that answers. But, yes, a definite challenge, not only for us, but across the sector.

Great. We're gonna go with one last question here.

Hi. Great presentation. I'm really interested in how you do are doing the a hundred, control group and the, is that all survey? Are there usability sessions? What is the method?

Yeah. Thank you for asking. I am definitely, heads down in that one. That is a huge study that we're doing.

So, basically, we've done, an unmoderated session for the first the first wave. So it's a three week longitudinal study. So what that means is that, we have a baseline, at the for week one. So for week one, it's gonna look like an unmoderated session using user testing. We have them in different blocks based on their condition.

And, they've just gone through our site. They've, they've researched. They've done different things that we've asked them to do, not only on our site, but also on the, Internet. So we've used the Internet as a control, to see, like, what happens when you just, like, Google a nonprofit.

So we do we've done that. At the end of week one, there's also a survey where we ask them a whole series of questions about what they thought, just trying to get some product feedback, as well as trying to understand if they use the control, where did it go? Did you go to ChatGPT? Did you go to Google? Did you go to a charity rating site?

Etcetera, etcetera. Week two, we followed up with a survey. So we've I've added all those users into my little favorites and sent them a survey to follow-up.

And that survey just basically is a tracking in. Like, hey. What did you do over the past week? We told all of our our donors at the beginning that we'd like you to continue researching your donation decision for all three weeks. And so it was just a follow-up survey. And then week three, similar follow-up survey. So week one was an unmoderated session and a survey.

Week two, just a survey. And week three, just a survey.

Hopefully, that answers.

Yeah. That's really great.

So thank you. One more round of applause for Maddie. Thank you so much. Thank you.

Thank you so much.

So I guess most of you have not heard of universal robots before, or maybe you have. But just to point out, it's a Danish company. So I flew all the way from Denmark to here just to give the presentation for you today. So I've really been looking forward to this.

As Cecilia just said, my name is Enja Sove. I'm head of UX and product design in Universal Robots.

And I think before we get started, I just wanted to point out what is great UX and Design actually for me. So just to tell you a little story here, in 1960s, even before the title UXs was invented, there were some really, really, really cool people developing how the push button phone set should be done.

And I don't have this example. I didn't, I wasn't born at that time, but I actually got the inspiration here from Nelson Newman Group. So the engineers here, they looked at how they should do the setup. And as you can see over here, they found out that most people prefer the middle solution down the button because that's very easy to understand. It's how we normally see the digits.

But then there is one thing that you can always be sure of as a UXer, and that is that your users, they don't know what they want. So you have to be the one right asking the right questions and you have to be the one able to do the right conclusions.

And what they did, those UXers at that time, was that they actually tested this. And they found out that the fastest solution to put in an eight digit or ten digit number for people is actually the solution up here, number one in line four, which we actually know today on all our push button phones. Right? And you also know it on your mobile phone.

So, what that is is that if we assume that the push button telephone set has been used forty trillion times, by doing a good UX and a good design, instead of just taking the one that everybody preferred, the world has saved forty billion work years, which is actually quite a lot. So, that is what good UX and design is to me. Imagine what a difference we can do out there if we actually do our job so good. So, this is what the presentation is about here.

So, I just wanted to introduce to you my family here.

This is the robots that we actually design.

And, we have, as you can see, some robots' arms over here, and then we have a control box, and we have something called a teeth pendant, which is similar to an iPad.

We design the physical product, we design the software on the Teeth Pendant that you actually use to control the robot with, and then we design all the digital touchpoints that our users have with Universal Robots. So I would like to ask you, just raise your hand if you both have a physical product and a digital product in your organisation.

Wow. That's good. That's a lot of people here. Great. Okay. So how do we make that come together? This is very much what the presentation here is about.

So, in Universal Robots, we are guided as designers strategic insight. And you can see the difference here in our product where you have the old products to this side and you have the new products over here. They are a little bit larger, but also the design is different. And the reason for that design being different is that's because when we started to work with insights from real users, we learned, uh-oh, we have an issue in our product, the way that it looks and the way that it's designed, and we need to change that going forward. But you can also see that we have kept some of the branding from our old product range to our new product range. So that's the journey that we have been on for the last four years.

And just to let you know a little bit about it, we are supporting three fifty engineers in our organizations. We work worldwide.

When I started in twenty twenty, we were four engineers in a corner of a software organization, only looking at the software for our TP.

Then we have grown quite a lot. In twenty twenty two, we worked, twenty UX designers and researchers working across what we at that time called product creation and marketing.

And then we have developed even further. And the newest thing is is that in autumn this year, we have actually been part of product management. So I'm now referring to our VP of product management in the organization, which allows us to really provide good insights to our product managers and allows us to be much more user driven than we were before.

If you look at the curve here, the stages of UX majority which maturity, which I hope you also know from Nelson Norman Group, I would pretend that we are between five and six up here.

So when we started, there was, like, the colors were not defined in Universal Robot. There was not a lot of best practices written down and all of that stuff. And that's some of the things that we have been doing working through all the phases here. Now we are there. And now when we are building the future strategy of the company, now when we are looking into what is the next new product that needs to come out, everybody says, hey, remember to ask UX and product design. They might have some information that can help you. So that is a great achievement for me and the team here.

We are a multidimensional role in Universal Robots. It means that we cover those four areas here and then also, of course, the micro and macro trends underneath. We work with design research, We work with interaction design. We work with visual design and behavior analytics.

And in my belief, when those four areas come together, you can actually create good UX. So one can now stand next to another. And that's actually why I'm here today to present some of the ways that we have been working so I can pass on that knowledge to you guys as well.

All the areas are important when we need to design a new product or we need to design a new experience.

And we have a strong foundation to work as a customer driven organization if we remember to include all the areas here.

Of course, all the areas have different expertise and the objective is to collect cultivate a holistic understanding of our user experience across the products and all touchpoints within the organization.

In our situation, it's very, very, very difficult because the ones that actually buys our products is probably a CTO or somebody else in the company, and the one who actually uses our product is the person working on the production floor with the cohort every day. And now you maybe think, oh, she called it cohort instead of robot? Is her English really that bad?

No. It's not, but I excuse for my for my English. But, actually, the reason why is because Cobot is taking collaborative and robots and putting that together because that is what is unique about our products here, that you can actually come very, very, very close to the robot. And if it should hit a person, it will automatically stop. So it's very, very, very safety proof to be close to a robot. It's not all robots that are like that. Many robots are behind big fences and working in there, but our product is very, very, very close to the user, and that means that you can actually stand next to it and maybe do similar work or prepare the work that the cobot actually has to do.

So I thought a lot about what should I bring to you here today? And I decided that I'm going to present ten must haves for building a user driven organization today. This is very much learning from our own journey. This is how we have done it. And you're more than welcome to ask questions in the end as Cecilia says because it can seem like, okay, you've been super, super, super busy, which we have, but we also try to work with this in a structured way.

And the reason for working with this in a structured way is because I don't know if I'm putting people too much in boxes here, but we might have some engineers over here, and they think with one part of the brain. And then we might have some designers over here, and they think about with the other part of the brain. And when we need to collaborate across, it's very, very important that we meet both the engineers' needs, but also the designers' needs. So we need to have the creativity, but we also need to have the structure that makes it okay for the engineers to actually go in and work with the experience. So I think a lot of my role has been to also communicate that inside the company.

So first of all, what you have to do is to build a great relationship in the organisation.

We call it that we are divine in universal robots, and it means that we have combined the developers and the designers.

We are not going to use the engineers as designers, and we are not going to use the designers as engineers, But we collaborate very, very close together. The reason for doing that is also the high difference in numbers of three hundred and fifty versus fifteen people over here. But it actually makes us really, really, really good when working together because it makes it possible that we can see the engineers in the eyes and say, hey, hey, this is how we should do it. And the engineers feel empowered. They feel like, okay, I was part of the process as well, which is really, really, really cool in product creation.

So the first thing, be divine.

The next thing is to create a vision. In Universal Robots, we have a vision that we want to create a world where people work with robots, not like robots. I'll just spend two seconds on explaining that because what our product does is that it works within production, it works within manufacturing.

And normally when we have it there, you can see that people are doing the same job over and over again. So if they're doing like that with their arm for fifty years, try to feel if they have a little back pain here maybe. So some of them have the risk of actually being worn out, and we don't want that. I'm a mom of three boys.

I don't want them to grow up and have a job where they will actually be worn out. I want them to be able to use their brain and have an intelligent job in the future as well. So that's what we are creating with this vision. We want to take away the jobs that actually is not nice for people to do.

Then you can say, okay, that's the overall vision. And then in UX and product design, we have created our own vision as well. We want to make everybody fall in love with our brand, our company, our product, and solution.

And we want to do that by creating the amazing design like Apple. Okay. So you might be thinking, hey, Anja, that's easy. I could say that as well.

Yes, it is. But everybody knows Apple and everybody knows that they are super, super, super gut. So why should we not put the the the the goal up here and say, hey, this is what we want to go do? So this is the vision for all the designers and they will be challenged by that when we need to go along.

And then give the unique user experience like Tesla. And that is because three years ago, I actually did a big study of Tesla and how they actually work and what is actually the thing. Of course, they hit a wave for electronic cars. But there's one thing there is for sure.

If you ever sat close to a person in a with a nice dinner or something like that who just bought a Tesla, that person is so annoying because he's trying to sell you the Tesla. Okay? He's so super, super, super happy with his product. So how do we get our users into that mindset as well?

That's my goal, and that's why it's here. And then you can see the last line, be universal like universal robots. Okay. Our product is just an arm.

It's cut off here. It has no hand.

But the unique thing about it is that it can work within nearly every environment that you can imagine. We can do nearly everything about it that you can imagine, and we need to keep that because that's a huge strength for our product in the future.

Yeah. So that's a little bit about our vision. And then you can say, okay, design is not just how it looks and feels. It's also about how it works. Of course, it is. And that's why we have to be divine. That's why we have to work closely with the developers as well.

Working closely with the developers and engineers.

So what do you imagine, Anja? Can you add some colors here or what? No. We are not the ones adding in some colors. So when I started at that, I needed to make sure, okay, I'll build a fundament here that everybody can hook into. I'll build some guidelines that they actually understand. And that's why number three is create a guiding design and experience philosophy.

These are two very, very, very short pages, as you can see here. But this describes exactly what it is that we have to go do when we talk about design. It describes about how we want the future experience to be for our users going forward as well. And it's written in very, very few words as you can see here.

I can't publish them to you guys. I'm sorry about that. But at least you can see how long they are up here. It's nothing more than that.

And this has been super, super, super effective when communicating to the engineers because this is something that they understand. And now they can make a cross point. Oh, I did this. I understood this.

And then we can build that into the product as well. So guiding them this way is a really, really cool thing as well.

And then number four. Oh, can we go one back, please?

Thank you. Number four, map the customer journey.

So automation is a journey. It's not a destination.

When we first tried to map the customer journey, we did, like, those long, long, long posters that could not even hang on this wall here because there were so many touch points and way that you could get through the customer journey and so on. And every time we tried to present that, some people would be like, okay, falling asleep or something because we're so happy. But what we wanted to do with this was actually trying to create a tool that everybody in the organization understands, that everybody in the organization can use. And this led us to this customer journey that you see up here. The customer journey goes around in an infinity loop like you see here and we have put it into different phases. The thing about putting it into phases is really, really good for us because then we can say who's responsible for each phases of the customer journey.

I'll just take you through the customer journey here.

So in phase number one, we have the automation need. That is exactly the situation that I described before. Okay. So Linda is coming to work every day.

She's done that for the last fifty years, and she's done this for the last fifty years, eight hours a day, just like that. It's really, really, really cool. And one day, she finds out, okay, my my bag is hurting here. I need to go and see the doctor.

She goes to the doctor. The doctor says, you cannot do that anymore. She goes, she tells her boss, I need to quit my job.

And then her boss, he's really, really, really clever. So he thinks, oh, I'm going to put in a job ad. Maybe also advertise it on social media and so on. Woo hoo. That's really something. But unfortunately for him, nobody wants the job.

So he's an automation need now because he needs the job done. But what is he going to do about it? And then he starts thinking about, is there a different way that I could actually react to this? Could I do something differently here?

And that sends him into trigger because now he starts to understand, hey, other people are working smarter than me. What do I need to do to get to that level as well? And into transform, he understands, hey, a cobot, that's a collaborative robot. It could be my, solution as well.

And then he jumps into onboard. We give him a lot of training. We give him a lot of knowledge about what is actually universal robots, how can he actually use the robot. And not until the end of this phase, we actually try to sell him the cobot.

So he's very well dressed now and he's very well on board of what it is that he's going to buy. One of the best features is that it works twenty four hours a day and it doesn't need lunch in between. So it's a really good feature for him as well. And then you can say, okay, he purchases the robot in end of onboard, and then he jumps into deploy.

That means that we send him the robot and he will start activating the robot in his factory floor.

We need to make sure that he had a good setup there. And sometimes we are so successful that he would go directly from that phase and into automation it again. Because whoops, over here is standing Wendy, and she has been doing this for fifty years. And she also has feeling bad in her arm. So maybe we could also replace her job with some automation. And that makes him go into automation need and go the round again.

If there is not an urgent need for more automation, he just goes into play. And in play, the robot is just running. It's super good. But we know that at some point, he will run into trouble.

That's guidance. That's our full service organization. What do we need to support him the best? If that robot stops working and he needed to do this twenty four hours a day, he's in trouble, and we gotta be there.

We gotta serve him very, very, very fast, and we gotta help him very, very fast. So the guidance space is super important for us here as well. And if we succeed with all of this, he will jump right into advocacy. He will be the one going out telling his friends, hey, I found something.

This is amazing. I've been helped by this. And then you can say he will run into automation need again or send one of his friends there. Or if he's just happy, he will just continue going around in the loop over here.

So this is how we describe the customer journey. This is actually what we want for our users in Universal Robots. And now we can start building all the touchpoints along the customer journey. We can start saying who's responsible for it and how should we actually design it.

And then the next thing, measure the customer journey.

So, we have the customer journey here, and I'm actually able to measure the customer journey. What we have done so far is that we have done a study, a customer journey survey that we send out. And we have, as I said before, customers all around the world. We have twelve hundred people, participating in the survey. And in the survey, there were one hundred and twenty two questions.

That's an effort for people to actually wanting to spend that amount of time and giving data to us. And this data will be our baseline. This data allows us to actually measure all the phases that we have here, but it also allows us to say, Okay, with this little product improvement, have we actually moved something here? And this is where it becomes really, really interesting.

So now we can benchmark against ourselves. We also, of course, use the NPS. We use the QX score. We use benchmarks from user Zoom as well, so we can see how we are doing compared to all of you.

Great guys sitting here. And then, okay, we have set KPIs for all our stakeholders. So, we've told all the stakeholders and all the people who are responsible for different areas of customer journey, This is your problem. We would really like to help you with more insights and so on, but you need to reach this target as well.

And we are being supported by our senior management team here, so this is really, really cool. And it's a cool tool that everybody can understand in the organization.

Yes. Okay. In order to be able to do this, we have created some systems underneath, and this is really something that I would like to share with you as well. We have our Insight system, Polaris.

Polaris is known as the Northern Star. It's used for navigation. So when you are out in a boat in the middle of nowhere, you can actually look at the stars and you can find the North Star and then you can navigate from that. So, of course, that's the right name for our Insight system. Everybody should navigate from insights from our users, right, in order for us to be a user driven organization, a customer driven organization.

So number six is create a research and insight system.

We have created this system four years ago without really knowing that that was the thing that we're doing at that time. We needed a program to fill in all our data, and we choose EnjoyFQ, which we are really, really happy with. We structured all the data in there, and now we have more than thirty thousand data points in there from eleven thousand users, giving us a huge advantage when we need to build strategy, when we need to build new products, etcetera, because we can search in there and we can find all the information in there. And then, of course, as I said before, we have our customer experience survey with all the feedback there that is also structured into a GeoHQ.

So it also gives us a lot of knowledge. And then we are using user zoom and user testing now to also add in a lot of knowledge there to make sure that we are on top of everything. And every time we have a prototype, we can go out and test. And if a user says, hey, this is the thing that I don't like or I like the old software actually better on the teeth bending or something like that, we can add that into the system and become shareable for everyone in the organization.

So how did we actually build the Insight system? Well, we needed the structure, as I said before. We were probably not one hundred percent aware of what it is that we were doing. I will totally admit that.

But, actually, we took all the data we had, and we added it into EnjoyHQ. And that took about three months, but it was under COVID. So everybody was working from home. So it was a good little task to do at the end of the day or something like that.

But we got there, and it was a huge achievement for us. In in in in GeoHQs, we used the data.

We structured the data using tags, and, also, we democratized the system by giving everyone in the organization the unique possibility to go in and read our insights. But what we learned from that is that to make good decisions and to make good insights, you need also good people. So I think it's it's really a thing that you really need to educate your users if you democratize research in the way that we did. So please think about that. It's not the only read only access. It's also about understanding what it is that they actually read.

And then we were so lucky, and it helped us a lot that user zoom user testing in JSQ is now one company. So that was a thing that we can control, but it was really really useful for us.

Okay. Then you might be thinking, what is the outcome actually of Polaris? And that's number seven. Get familiar with your users.

So there's been a lot of people here talking about the archetypes. What we have in our organization is that we have something that we call the behavioral archetypes.

We did a research with eighty companies, eighty customers around the world, and we Cobot from us? When they receive the product? What is their behavior when they interact with us? What is their behavior in digital touchpoint?

What is their behavior when using the actual product out there? And then we tried to make those five descriptions up here. And I'll just give you an example with one of them. Here's the yellow one.

He's called the scrappy hacker. The scrappy hacker is the first user of our product because he is on forehand of technology and he can fix almost everything himself. Okay? So when he receives the Cobot, he just rips the box open and he takes the Cobot out and he installs it.

He doesn't read the manual. He doesn't care about security, blah, blah, blah, all of those things. He will just start using the Cobot. And that's so super, super, super cool.

That was the first many users. Now we're seeing a shift in our users as well, more and more getting into the Meticulous Masters, the purple yeah, the purple one up here. Meticulous masters is a group of people being put together with different expertise.

And the people in that group will use their expertise to know a little bit about the product, each one of them. So that changes the scenario that we have to do. Now we need to present the manual even before. We need to make sure that when the box arrives, the box looks very, very nice. It has description on how to do it, etcetera, etcetera. So a lot of changes is going on also just by following our users this way.

But the cool thing about this is that we have created a corporate language with this. When we are talking about our users, I will just say that we are one thousand plus employees in our company. When we talk about the users across the world, we now have a corporate language that we can speak to. We can say, he's probably a scrappy hacker. And then everybody knows what it's like. And the reason for doing that is because we have meeting rooms decorated as the different archetypes.

So, we took five meeting rooms and we said, Okay, we're going to name them as our archetypes and we're going to design them inside so they look like our archetypes' mind.

So, adding in a lot of visual examples in there and a lot of information about our archetypes have been very, very important to the engineers as well, understanding what it is that they are working with and who it is they are building the solution for.

And then we have a ten minute recorded presentation on the intranet that everybody can go and watch if they have issues with finding out who our users are and so on. But it's not personas. It's behavioral archetypes that we have created here.

Yes.

Then we have our design system. And I guess most of you guys are researchers in here, but I wanted to make sure also to mention our design system because the design system is super, super, super important to be able to send the right signal to our users and all the touchpangs out there.

We want to build a holistic design and ensure a good user experience. And we can only do that if we meet the user the same way in doubt when you are in contact with Apple. When you see an Apple Store, you know exactly, okay, this is the Apple Store. Of course, When you see an Apple Store, you know exactly, okay, this is the Apple Store. Of course, it looks like that.

When you are in contact with Tesla, you know exactly what it's like. You know exactly what you should expect from that. And that's the same thing that we want to do with our design language here. So that is why it's important for us to describe When we started doing this, there was no rules in Universal Robots. Blue was a color that we used, but we had no code for the color or anything like that. So it's just a similar color that everybody could point at.

What we have done here is that we have defined a new visual language. And I'm from Denmark, as I said before. We are famous for the Nordic design tradition. We are famous for the Scandinavian design. You know that from furnitures as well. And we've tried to take some of that historical heritage and add it into our products.

That's why we are blue.

Blue is the same colour as the sky. Blue is not a signal colour. A signal colour is red, yellow, and green, and it will tell you to stop, pause, or continue.

Blue is the color of the sky, and blue is a relaxing color. It also means that when you see our product here, it's very, very approachable. It's safe just because it's using the color blue.

And then you have the friendly landscape of the Danish and the northern nature as well. So we are protected very much in Denmark from dangerous things.

We have the sand curves here close to the sea, protecting the land from the sea. And the most dangerous animal in Denmark is a wasp.

So if you go out, you sleep outside or anything like that, you don't need to be afraid of snakes or something like that because you just need to be afraid of the wasp. That's it. And I think everybody can manage that. We don't have sharp cliffs in our in our nature, so there's nothing that you can cut yourself on.

Most of the beaches are sandy beaches, so it's very, very nice to walk there and so on. And this is some of the things that we have taken into our design as well. And then the curves. You don't see any sharp edges or anything here, but we try to take that into our product.

And that's when you look at our new product, you will see that we have used the shapes. And, actually, the the blue on our new product is shaped like a water drop. And the reason for that is because we wanted people to think about this. We wanted people to have a more understanding about this.

And isn't that insane when I'm talking about an industrial, cobalt that we actually have thoughts like that? But this is something I think means a lot of the for the people actually working together with the robots. And these are the people who are most interesting for us because they are the cobot champions. And I'll come back to that in a little while.

I already said a lot about the colors here, but you can see that we have the blue color here. And actually, the choice of the blue color was a lucky choice from our founder when he had to do it.

So what we did with all this knowledge is that we built a design system. Does any of you have kids at home?

Yeah. Quite a lot of have that. Okay. So you probably played with LEGO. And you know that it's super, super, super nice playing with LEGO because you can build almost everything in the world. But, goddamn, it's hot on a Saturday morning when somebody wants to build a big van with four wheels, and you can only find three of the wheels. Right?

And you need to go through the whole box and you cannot find it. So that's the same thing. That's the same way that I'm looking at a design system. If you have everything in a mess, like the first picture up here, it's super hard to build from a design system. But if you make sure to structure your components in the design system, it becomes so much easy and you can build everything in the world from a design system. That's my thought about a design system and that's what I want in Universal Robots because I want us to be effective and I want us to be able to build products even faster. And then we need the structure.

Our design system looks like this. It's called Sirius. So we are still in the stars as well. We have Sirius Digital first that actually helps the digital experience. It means it has a lot of knowledge about how you should build a website, how you should look at getting people to interact with you online. And then we have the serious hardware describing how our products should be built and what it is that we set up of rules for building new products. And then we have serious software, which is describing how our software should work.

All of those three things were separate before. Now they are one thing and it relates to each other. This makes us capable of saying, okay, we give a holistic user experience both when people interact with us digital, when they see our product, and also when they use our software. And then the benefit of this is, of course, also your external use.

So we can share the design system to other companies and we can say, hey, can you help us build a great marketing campaign? Yes, I can because they can access all our assets here and they can help do that. So we can use it for that purpose as well. That's a huge benefit, by the way.

So it creates a lot of value for us. It creates consistency and cohesion in here. And, yeah, it makes the user recognize and trust us in all the different touch points, which is super, super important.

It creates efficiency and speed so designers and developers can actually quickly find the new the assets that they need to build new products, which is a huge advantage for us as well. And then it empowers collaboration between the designers and the developers.

It creates easier scaling for us because we are only fifteen and they are three fifty. So when we want to do something, it's so much easier for us to actually send them to a system instead of saying, hey, can you wait? And then becoming a bottleneck. Right?

So really, really easier scaling as well. That's also a huge benefit of our design system. And then also the quality and the accessibility in there. The focus is accessibility, improves the quality of the products, and it meets our legal and ethical, restrictions as well and standards.

And that is something that was not possible before. So this is also something that you can build into a design system and make it really, really cool.

And then in the end, cost savings. Hey, take that to your leaders all over there. It's really, really a cool thing. We can save a lot of money.

We can save a lot of energy by being structured like this. And this is new for designers. And a lot of designers have, a real cool feeling when they hear that I want them to work with system. But I tell them to leave their ego outside the door if they need to work with us.

Not that they should not have an ego, but they should be able to work within the system because it benefits us all instead of them wanting to have rounded corners or something over here when we decided that we are doing it like this. So this is what they need to put into. And then you can say the last thing, prove the value. Is there a connection between great user experience and company results?

In my opinion, there is. Because, as I said before, those people that I send into advocacy, they will be our COBRA champions. They will be the ones being so proud working with our product out there. And I love to go around and see them. And that's a huge value for us when we have people around the world being proud to work with our cohort. Right?

So create, repeat customers. A cohort champion will advocate for universal robots and our products. They will advocate for more automation, and they will recommend us to friends, family, and peers. And they will post about us on social media, which is also very, very, very cool.

And then a fun fact for you guys in here. When I go around visiting our customers and I really find a true Copa champion that really loves his new arm standing next to him over here, he has named the arms.

Sometimes he's also dressed the arm. But the robot, the most common robot name is Robot of Roberta. Maybe it's not a big thing, but it's a big thing for me. So if I hear a person naming their cobot, then I will say, okay, you are a cobot champion. It's really, really, really cool.

So how did we do it? We built the great relationship in the organization.

We created the vision. The vision is the same as four years ago, so nothing has changed there. We created a guiding design and experience philosophy to make it easier to work with the the developers and the engineers.

And we mapped the customer journey.

We measured the customer journey. And then we created a research and insight system, which we have been truly, truly, truly happy for.

Structuring that information and not having it laying around in thousands of PowerPoints is just the best. So think about that. And then also get familiar with your users. Use a language that everybody in the organization can understand and can know.

And then build the design system. Know what is your color, what is your components, what is everything that you need to build the right touchpoints with your users. And then prove the value. Go to your leadership and say, hey, this is the value that we provide to the organization.

We should be famous for that. Right? I want to be famous for that. Like, Apple is famous for their design and Tesla is famous for their unique user experience. Hey, we want to be that as well. And then create repeat customers. If you can have people coming back in the infinity loop, it's really, really, really cool.

So work hard, party hard. That's also a thing. And then be patient. That is my best recommendations from you out here. So, yeah.

I feel like this weird owl thing needs a little explanation.

So what I am really obsessed with, oh, is, distracting my two I have a tween and a teen, eleven and thirteen year old girls who are on the hormonal ride. And you seriously cannot rage while you're listening to weird alcophamania people. Okay? Look it up. It is hysterical.

Yes. So as Amy said, I am Amy Troupe, and I am the, vice president of experience in digital strategy for Centene.

We are absolutely thrilled to be here today in Austin among so many fellow UX and CX practitioners.

Today, Jen Jones, who is our, who leads our CX insights practice, and Melissa Spikemayer, who who leads our UX research practice, we're gonna share with you some of the approaches we've taken to really begin to alter the DNA of our company and and begin to transition it into a customer obsessed culture.

I do wanna admit and note that this is a journey, and we really are at the very beginning stages.

So before we dive in, by a show of hands, who here had actually heard of Centene before today's session?

Oh, wow.

Okay. Yeah. I think that was about a third, and that's not surprising to me.

So Centene is one of the biggest companies that you have never heard of. We were founded as a single Medicaid health plan, back in in nineteen eighty four in Wisconsin.

And now here we are forty years later, and we sit at number forty two on the Fortune five hundred list. We serve roughly ninety million members across all fifty states, and with about one hundred and sixty billion in revenue.

So there's a couple of reasons you've never heard of us. First is our brand strategy.

Stentine, it's a corporate brand. We don't sell any health plans or operate any health plans under that brand name. They all bear their own distinct brand name.

And then the second reason is that we specialize in government sponsored as opposed to employer sponsored health care.

So we are the number one provider of, Medicaid health plans, of marketplace health plans, and then what we refer to as dually eligible Medicaid Medicare health plans.

So Centene was born out of this b to g, business model that really did not necessitate a focus on experience.

The company grew historically, both organically and through acquisition, but really focused on creating hard value through things like administrative cost savings and efficiency, not soft value drivers like experience and satisfaction.

And so when we embarked on this endeavor a couple of years ago, we said to ourselves, hey. In order to change the culture of Centene, let's start by embracing it.

So Centene, you know, we recognized the fact that the company is comprised of actuaries and epidemiologists who really trust it in math and models. And so we leaned into this, and we leveraged it to empirically tie experience to hard business value.

This this morning, we're gonna share with you how we've really quantified and operationalized experience at Centene.

First, we look to answer these questions. Does experience matter in terms of business value, and which factors matter the most?

How should we measure those factors such that that measure is actually predictive of that value?

And how do we translate their impact into dollars and cents?

To answer these questions, it definitely requires the effort of some smart statisticians, but it's pretty straightforward.

Then comes the trickier part. How do you begin to actually operationalize this all the way down to your UX practices?

Jen and Melissa are gonna share the steps there. I'll talk through the modeling.

So to answer these questions, we first have to talk we have to start with the customer that we're talking about and the business value we're looking to realize.

We started with Ambetter, which is our marketplace line of business and focused on the business value of member retention.

Let me let me talk you through our methodology.

So first, we fielded a quantitative study, large scale, of those members who renewed and those who did not, asking them to evaluate their experience, on on on the factors of ease, trust, satisfaction, and likelihood to recommend.

We then ran a regression analysis to determine the relationship between their evaluation of their experience and whether or not they stayed or left.

We, isolated specific moments of the experience to identify the relative impact each of those individual moments had.

And then we confirmed which measure was actually predictive of that renewal behavior.

And then lastly, we did the math to translate the experience measure into both top line revenue and bottom line income.

So to answer our first most basic question, does experience matter?

Yes. Experience matters. And I'm I'm certain this is welcome news to everybody in this room.

So members who scored their experience at a six or higher were, were more likely to have renewed.

Okay. Great. But we know that not all experiences are created equal, not all moments.

Which aspects of the experience have the greatest influence on that score?

So the re the regression analysis also described how much effect each of the experience factors had on the score and, in turn, that likelihood to renew.

We found that there were four factors that have the strongest influence on that renewal behavior.

Getting help from customer service, getting care approved and paid for, getting an appointment with an in network provider, and understanding plan coverage.

These became our moments that matter.

We use moments that matter as a very specific term now, and we could we we say we will only assign this term to, you know, specific experiences that we have proven have a material influence on business value.

These are the areas that we must maniacally monitor. These are the areas where we should make our investments.

In a, you know, in a in an environment where you're experiencing daily fires, I don't know if any of you have this scenario in your company, The moments that matter, they really give us a way to focus our efforts and to prioritize our investments.

Next, we looked at how effective each of the measures of ease, trust, satisfaction, and NPS were at predicting a member's likelihood to renew.

We also evaluated a composite score that combined ease, trust, and satisfaction.

So the headline here is that, independently, trust and NPS were the best indicators of member retention at this relational level. Remember, this was a large scale quant study, really, trying to understand the brand relationship.

However, the so the CX composite score, that combination of ease, trust, and satisfaction, also performed very, very well, and it had this added benefit of being able to tie what's happening at the transactional level, everyday interactions up to the relational level. And we really wanted to establish a framework where we could tell, you know, what's happening day in and day out. How do we expect that to ladder up to the overall relationship in our brand?

So, and and this is, I I think, common sense, but if you think about it, how easy or difficult is it for you to accomplish what you want to on a website, or when how helpful is it when you when you talk to customer service? That directly ties to your satisfaction with your experience, and then in turn, your trust with a brand.

And so, for this reason, we have chosen to implement the CX composite score as our enterprise standard member experience metric. And Jen is gonna talk a little bit more about how we've, done that.

Okay. So finally, we did the math to translate the impact of a change in experience to revenue and income.

For I I took out some of the most proprietary data on this slide, but for for Ambetter, the model showed that a one unit increase or decrease in that composite score corresponded to a two point eight percent increase or decrease in the likelihood to renew.

So then from there, all you really need to know and understand is sort of the size of your customer base, in our case, our membership base. What is the lifetime value of that member? So what's the, you know, average, revenue per member, average income per member? And then you can do the math, and you can equate improvements in CX to top line and bottom line.

Now we've just looked at the model. This is our model for the marketplace line of business. We have also developed a model for Medicare.

And I think it's important to note that the influence, the relative influence that experience had on retention does vary by line of business, and the moments that matter are distinct and are unique to each line of business. So it's really, really essential that you do this analysis for each of your customer types.

So with these models, we are now able to demonstrate and articulate, you know, the return on investing in experience to our super hard numbers minded organization.

We can use the models to focus attention, of the organization on the experience factors that matter the most to each of our customer types.

We can use them to weigh investment options, for improving different aspects of the member experience, and we can use them to build out substantiated business, cases for making those investments.

We are currently working with our partners across lines of business to standardize the way in which we we measure CX and to use the new CX composite score to set their experience goals for twenty twenty five.

It is no easy feat, as Jen will attest to next. So when I hired Jen roughly two years ago, one of the very first, efforts she, embarked upon was to go around the company and to inventory all the different measurement methodologies that were in place. And as you can imagine, it was all over the map. So some of our lines of business were using CSAT, others were using NPS.

And then even within NPS, she found use of a seven point scale, a ten point scale, and the eleven point scale.

So, Jen, now that we've, you know, seen how can we identify and how should we measure moments that matter, can you talk us through the trickier part of this exercise, which is, how to actually operationalize it?

Thanks, Amy. I sure can. And thank you for that memory because I have that. It's a little PTSD, but it's the wild west of what we inherited.

And so as we applied this new metric, as we applied this new model, Amy, Melissa, and other leaders, we sat down and we said, what is our vision? What is the framework we wanna create for how to truly infuse this into our business and make it work? So we then documented our listening posts. And thankfully, over the last two years, we've stood up listening posts everywhere we possibly can in the customer journey.

When Amy said you haven't heard of Centene, that's because we have over three sixty websites across all of our brands. And yes, we have intercept surveys on every single one. I'm also happy to tell you that there's a post call survey after a phone call to customer experience across all of our brands. But making sure that we tied that to our UX study results, making sure we tied it to our relational instruments.

My team is largely responsible for the CX, where my sister here, Melissa, is responsible for the UX. And we are partners in crime because while my team's out doing more of that generative work, her team is constantly looking at the evaluative side of it. So our annual survey, our monthly pulse survey, we wanted to make sure that this all tied together. And you can imagine after we got all those results back from the actuaries and from the statisticians, we're like, I sure hope this works.

So we sat down and looked at the data, and we went retroactive and said, well, what happened this past year? So month over month, when we looked at our transactional survey scores in aggregate and our relational survey scores, thankfully whew. I remember the moment this happened. I was, like, breathing a very, very sigh of relief because those lines on that graph lined up.

And that's the alignment we were hoping to have. But our groups pride ourselves on a tagline, which is delivering actionable insights. And this aggregate score isn't enough to do that. As Amy talked about, we now have the moments that matter.

So looking at those, we wanted to make sure that those categories of tasks were lining up in their scores at this composite, both relational and transactional. You can see for the top four that we listed on that bar chart, they in fact did.

That allows us the confidence going forward that when we see a score dip in the month over month, tracking of it, we can absolutely get to the root cause of what happened and quickly get something into testing to see how we can make an improvement happen.

So here's a little legend that we're providing our executives on how they can remember that the CX composite score matters. It matters to the business bottom line at Centene, and it matters to them because they're going to be setting a retention target. In our Ambetter line of business, which Amy spoke of, they're looking at somewhere around seventy percent. We haven't finalized it yet. We're still in the fourth quarter of doing that work. But if they are setting that retention target at seventy percent, we are telling them that our dashboard should show a target line of four point o for our straight average of trust, ease, and satisfaction.

What we did was then provide the numbers from this year's annual monthly and ongoing surveys so they could see, hey, we're not so far off. Four point o is achievable next year. We need to work hard though to analyze where our root issues are occurring so we can have a very quick impact on those month over month as they come in.

So that's the what. That's the what of operationalizing this new framework we've developed.

I think that's easier than the how. The how is how we impact our culture. How do we get in and convince that people who are married to NPS or who truly love the old way we did top two box CSAT scores, that that's the way they want to work? We did it by looking at these three parts of our culture.

We looked at process, the process for how people make decisions at Centene.

We looked at relationships.

And Johan actually referenced this this morning when he talked about relationships that need to exist across the customer journey to make this come to life. And then finally, insights.

Making those actionable then allows us to pull that voice of customer through the model and make sure that everywhere we go, we are truly gaining alignment in what should be the priorities for the improvements we make.

So I hope you'll play along with me. I have a little illustration on the first one. So when it comes to process, do you know how your leaders make decisions?

Do you know how the prioritization happens when an enhancement goes in and a change gets made? So to illustrate this, we'll play a little trivia game. I have a question for you. Here's your challenge.

When I throw the question on the screen, do not touch your phone. Do not Google this answer. Trust your gut. Tell me if you know which band is the most popular of all time.

Just think about it. Don't use a device.

I'll give you a spoiler alert that when I was rehearsing this in front of my college age daughter, she's like, definitely Taylor.

Taylor is an artist, not a band. So let me give you a list because Taylor's not on the list, Danny. And, among these seven now, maybe your brain's kind of switching over a little bit. Like, wait. I thought I for sure knew it, but my band is not on the list.

So let's ask this question a different way because if we were truly in a UX or a CX study, we would be debating as a team of researchers, was this specific enough? What do we mean by popular?

Let's add a caveat.

What if I asked you for two different metrics? What if I asked you about record sales or concert sales?

Now we have something much more objective, and the answer is Beatles and YouTube.

That is the objective truth. You can look up that statistic. You can find where the data lives, and you can point to it and trust it. Your opinion, or my daughter's opinion of Taylor, is the subjective truth. That is truly your opinion.

And in my world, our team likes to have a little joke. We call it the n equals one syndrome. And I bet this has happened to you. It's Monday morning.

You're just trying to get that second cup of coffee going, and suddenly your phone rings, and it's an executive. Or in my world, the chat rings because we're all virtual. So the chat's popping up. Hey, Jen.

Do you have a minute?

It's like, okay. What is it? Well, this weekend, I was at a party, and I talked to one of our members.

I know exactly what's wrong with our website.

Oh, good. We have n equals one. I was waiting for one customer to tell us how to solve all the problems. So my team laughs about that, but it's truly something to think about.

Are your insights objective or subjective? Is that kind of groupthink occurring by one really persuasive executive who has their pet project or their friend who's a customer? That inside out thinking can be problematic if that alone is how your decisions are made. It's a little bit more siloed.

It relies on popular opinion instead of the facts that we're all busy gathering in our research roles.

That combined with outside in thinking is really what we're striving for. That's where we're getting those data and facts. We're looking at concert sales. We're looking at record Not that these should be done alone, but together. You don't wanna stifle the brainstorming and innovation that comes from the inside out thinking.

Lastly, I'll talk about relationships. And again, I have to thank Johan for setting us up so properly because this slide was ready to go.

We had to stop and think about the customer journey and make sure that we realize that our customers don't care that there isn't a relationship from our website development team and our contact center.

So when a customer goes to pay a bill on the website and struggles, they are expecting, when the customer service agent picks up the phone, for them to pick up where they left off, help them with more advanced knowledge, help them figure out their task, not have a repeat experience of that customer service agent sitting in front of the same screen that they were on five minutes ago.

In order to increase that empathy and make that awareness known, we had to foster relationships between those groups. We had to get people together in meetings and say, hey, here's a situation where we're watching people struggle on the web and then several minutes later make a phone call about the same task. So thankfully, we have the metrics to show that. And Melissa's gonna come up next and tell you about a case study where it came to life.

Alright. Thank you.

Alright. So thanks, Jen. We are so grateful to have such a tight partnership with Jen's team, who she already mentioned we affectionately call our sister team.

So as we saw earlier, Jen talked you through the CX composite framework, score that's built with the transactional and relational data.

So these data are all collected in the wild, essentially, what has already gone out to production.

My team conducts our research mostly prior to that phase. We're we're evaluating designs prior to launch. So this gives us an opportunity to look at satisfaction, ease, and trust during the design evaluation of these tasks, which ladder up to those moments that matter. So this can give us an early read on how these features are performing with respect to the CX metrics, and we can make changes hopefully prior to before going to production.

So we, as UX professionals, all know the value that we bring to the table, but it can sometimes be difficult to show that value to our stakeholders, to leadership. So being part of this framework, really shows how tightly coupled UX and CX are and how UX research can contribute to those higher level CX metrics and business goals.

So we're going to dig a little bit into how the UX research team operates and how we tie CX in throughout our process.

So earlier this year, the UX team actually started working in more of a matrix model. So it used to be more of a service model. Now our designers and our researchers are embedded into the development squad. So designers, one to one. Researchers are responsible for two squads just due to the size of our team.

UX research leadership. So we are responsible. We are the people leaders still of the UX researchers, and so we stay really tightly close, close with them in the work they're that they're doing. So, you know, continuing to mentor them, you coach them, whatever they need throughout the process.

We're also making sure that their capacity is good, that they don't they're not, overloaded or underloaded. We're also overseeing that, you know, UX research process and strategy.

And then when it comes to our process, I'm sure what we have laid out here looks a lot like what you all are doing in your organizations as well. So we start with the initialize phase, which is really just understanding the problem.

You know, better understanding any research goals, any hypotheses that are out there, talking through success metrics. And this, a lot of times, is a conversation. We work through this together.

Next, we go into the gather phase. So that's really just pulling together all of the information that we have, all the data that we know so that we can lay it all out there and understand if there are any gaps in our understanding.

If there aren't any gaps, we don't necessarily need to conduct research. We can share out what we know and inform the work going forward. But if there are gaps, that helps inform our research plan.

And then we get into creating our research materials, our study. We rely heavily on user Zoom for a lot of our research, and then we launch to participants.

And next, we get into the synthesis, of course. So, you know, extracting those themes, those insights, and, packaging it up into, you know, a report where we really try to prioritize prioritize actionable insights for our stakeholders.

And then finally, this last part is really key and really a big benefit to us being embedded in the teams because instead of moving on to the next project, we're saying, hey. Let's let's watch this. Let's see how we're doing in production so that we can really, understand and assess the effectiveness of the change that we've made.

So this is definitely not something done in a silo. You know, as I mentioned, we're working through this with, you know, our UX partners, with the squads that we're part of, you know, many other folks, including our sister team. So, we're always in close collaboration with others. You know, again, those those relationships are really key.

Okay. And, you know, as I mentioned, there's really few places along the way where we're not in close partnership or having those touch points with CX. So, again, during that gather phase, we're pulling all the research we can from her team, you know, looking at what what, generative work has been done. What are the surveys saying, all those hundreds of intercept surveys that we have. You know, what are people calling about in the call center? So really pulling all of that together to help inform our research plans.

When we're conducting the research, we are implementing those those validation or those metrics as part of our validation. So the satisfaction, the ease, and the trust is part of that measurement protocol.

And then with synthesis, this is key too is really making sure that we're triangulating all of these insights together. So looking at what the surveys told us, looking what we learned, and tying that all together really helps us build a stronger story of what's going on, where did we come from, what did we learn, and where should we go from here.

And then finally, again, at that last sort of it's not really the last step. It's ongoing. Right? But we really look to our CX partners to say, hey.

Like, let's measure the impact of the change. So what where were we before? Now that we've released this, what are we what are we seeing now? And that helps us understand the the effect of the change.

So as Jen mentioned, we're gonna go through a real world example.

So This is the Appelo is a case where we set our sights on facilitating collaboration, again, to bring those actionable omnichannel insights to Centene. So this was a larger effort where we really pulled together a lot of different groups. Right? So our call center analysts, our digital product owners, UX, CX, amongst others.

And so, kinda setting the landscape here, we're looking at data for the task of resetting a password on, one of our secure sites.

And so the graph shows the CX composite metric scores in the blue dotted line and our calls to the call center in the solid purple line.

And, again, these are showing, the the task of resetting a password.

They are the week over week scores of call volumes and those CX composite metric scores.

So, again, this project is showcasing where we uncovered an issue that had occurred with an update to our website and people were starting to start to call the call center, with confusion over the website. So we saw calls related to this task almost double in volume, so that's definitely not not probably a good sign.

So we took a look at we started to dig into the intercept survey data so that we could kinda understand more context around the problem and then, really start to understand, like, is this something we need to take a look at, prioritize a fix?

And so it's this really encompasses some of the feedback we were getting. Setting up password with entry key, extremely confusing.

Worst password experience ever. So not a good one. We see this as the very sad face on our journey map.

Definitely something that we, are not happy with. So this allowed us to prioritize a change. Right? So that's where UX came in.

You know, we looked we took all of the data and created an updated design, and then the UX research team validated the change.

The fix was then implemented, rolled out to production, and almost immediately, we saw those calls drop. So if you see that nice little, line down, that's very good news for us. So that was definitely showing that, we were having an impact. Right? And then, of course, we need to go back, either to the the other data and kinda looking at our other metrics to to really make sure that the change is, in fact, having that positive impact. So we were able to validate that with the survey data and, as you can see, the CX composite metric came back to its original levels.

Very good news.

So this is an example of measuring the CX composite metric at that transactional, level and how UX and CX work together to validate the impact of the change.

Alright. So in conclusion, Amy, Jen, and I have shown you how we have built a customer obsessed culture at Centene.

The experience factors that matter the most will be a function of the customer that you are considering and the business value that you are looking to drive. But as mentioned earlier, not all are created equal.

So then we select a metric based on its ability to predict that business value.

You saw our formula. You'll need to build one that translates your metric to the hard value statement that is most impactful to your company and then measure it regularly.

Get obsessed.

By understanding the process of how decisions are made, fostering relationships between groups, and delivering actionable omnichannel insights, we have shown you how we've operationalized our model.

And finally, we've shown you how a strong connection between UX and CX allows us to tie our data together to build stronger insights that then fuel amazing experiences.

Thank you.

Great. We have time for a few questions.

So Hello.

Hi. So I'm not sure if I remember your names correctly. Jen and Melissa, I believe. I represent the UXR and I talk to customer support now and then to kind of have touch points, but that relationship is not strong enough to be able to pass on, like, hey, we have this problem. Let's do something more tactical about it. So I'm curious what those touch points might look like if it's a weekly meeting or how you guys choose to prioritize CS.

I wanna make sure to clarify. You're talking about the customer support team, like the call center team?

Yeah. What whatever it is that you guys, hear customer feedback from. So if it is those surveys, if it is the call center, however you're taking that boots on the ground information and then being able to pass it down to the UX team, what that looks like.

So one thing we did, I don't know if you have this at your company, but we do have a metric that tells us how much it costs per phone call. So while we've stood up here today and talked to you about trying to preserve revenue, there's also an expense side of this equation that you could be monitoring. So we worked in partnership with the web analytics group to make sure, like, when Melissa was showing you, that was a secure logged in site that we were studying.

When that particular customer departs from that site and makes a phone call because we're doing call analysis as well, we can see this is where channel switching occurs and is really expensive to the customer. Because these are self-service tasks. They should be doing on the website or the app by themselves. So when they have to pause now and go repeat the task by calling customer service, that's an expense we could avoid. So we tried to quantify that by showing that cost per phone call as something that could be avoided. And then they came to the table and they were like, oh, yeah. We'd like to reduce calls.

When they came to the table?

Yeah. Yeah.

She wants to know about the UX. Yeah.

So however, one of the I shared it.

Like once you guys are hearing, how do you pass it on to the UX team so you can approach it. What are those conversations look like?

I mean, sometimes they're just very organic in nature, right, where Jen's like, hey, these guys called me and we definitely need to pull you in. And which is awesome because we have that really close tie together and we so we will do that quite often.

And my team is also digging into data as we're doing that gathering phase two. We we have dashboards and things that we can look at or we also partner with Jen's team as well and be like, hey. Like, can you help me interpret this?

And so there's just a lot of organic back and forth conversation.

There one one additional, I guess, forum or capability, I'll call it, that we're in the process of standing up. We're probably about nine months into forming this group is, what we call an experience management team. They are sort of consumers. They're ex they're experts in the end to end journey of each of our customer types, and they breathe day in, day out the data and information that Jen's team is producing.

And their job is to know the moments that matter and to monitor those. And if there is a blip or in the case of, like, the password reset, that's like an enabling capability to many of the other moments. Right? If they see something like that that's impacted, it is their job to work in the, existing processes depending upon the group they need to reach out to for the fix. But in the case of the web team, it would be reaching out to the product manager who would then likely get in touch with UX design and work it into, is it something that we need to fit into this sprint now? Is it something that we can leave till the next PI planning? Or, you know so I think that's, I don't know if that helps.

Yeah. That's exactly what I was looking at. Thank you. I appreciate it.

I think we have time for one more question because We can stay after.

Yeah. We could be a little late to lunch. I know. I know.

I wanna make everyone late lunch, but you're going to be here the whole conference.

So, yes.

One more question for now, but I'll tell you where you can find them during lunchtime.

Thank you. Hi, I'm Julie. I work in the climate tech sector and I'm a B2C marketing team, but we can only market through our eighty utility clients. So we don't have direct, end customer interaction unless we go through them. One thing that we find is that there's often a lot of pushback with, like, nagging the customer too much. I saw that you have a lot of different touch points where you're asking for feedback.

How do you balance getting all of these different types of touch points with potentially creating a more frustrating experience by asking too many times for people to pig in?

So like trying to avoid survey fatigue in that sense? Exactly.

And like making sure that you're prioritizing the most important parts for sourcing that feedback.

Yeah. So the first thing we did early on, which was kind of at the one hundred and one level, was to make sure that we put cookies on all those surveys and some of them are pretty lengthy. One of our lines of business said, I only want that pop up to show once every ninety days after a customer visits our site. So we use the power of cookies to prevent that.

What we're working on now that's a work in progress is trying to get it all the way down to the customer level as a data source to say, these are all the interactions we have had with them in total. Whether it's a marketing interaction, a research request, maybe something transactional happened that they got an email from us confirming a payment. We wanna know, like, we're not bothering them too much. So having that data in more of a journey view at the exact individual customer level times millions and millions of members is what we're trying to do for next year.

Thank you. A girl can dream. Right?

Yeah. We'll let you know.

Awesome. I want to, have us give a big, thank you to Team Centene for their wonderful presentation.

I'm here.

We spend a lot of time thinking about consumption and news habits, how we fit into everyone's news diet, and then how we can uncover unmet needs and deliver the best experiences for our target user base.

Thank you, Nicole. So why don't we move on to you, Tara?

I'm Tara Smith. I'm the user experience design manager at Best Buy.

If you haven't heard of Best Buy, it's a Fortune one hundred company, with over a thousand stores in the US. And I'm and I'm I've worked on the Best Buy app, and currently, I'm supporting the Geek Squad and in stores experiences and support over twenty thousand employees.

And last but not least.

Hi, everyone. I'm Veronica.

I'm the senior manager UX design and innovation at United Airlines.

I lead the, design teams for shopping, so anything related to booking, repurchasing your tickets, as well as your day of travel. So when you check-in, you take your flight off board, on board, all of those stuff. And as well as our power packed qualitative research team, some of them are here with me today.

I've been here at United for eight years. I've been fortunate enough to work in both enterprise and consumer product at United Airlines and create some kickass tools for our customers as well as our consumers, as well as our employees.

And, here in my current role, along with the design and research team, I also spearhead the digital product innovation, which means creating more cool stuff for our travelers.

That's great. Thank you. I'm so glad, Grai, that you were all here.

Now let's move on to our first question. We know that in today's rapidly evolving consumer landscape, consumers have more choice than ever. And so one of the ways that brands can stand out from the competition is to have these breakthrough experiences.

How is it that you are ensuring that your brand stands out to deliver best in class experiences against the competition.

Why don't we start with you, Nicole?

Okay. So I think about this in two different ways. One, I'm gonna say voice of the customer.

So I will be celebrating my two year anniversary at Advanced Local this December.

When I joined, I quickly realized that we only had regular feedback loops with our digital subscribers and only via email surveys. So as many of you know, that's not your full user base, and that's assuming people are in email, open it, all that good stuff. So to be best in class, we have to know what all of our user types are thinking. So right now, I'm spearheading an initiative, working in a cross functional form with the team to, stand up channels that if you think about the marketing funnel from the anonymous user to getting them to return, then maybe they register with us and then they become a paying customer. We want all those user types when they're using our product in context to be able to give us feedback. So I think it's really important that we hear from everybody and we use that feedback to help us prioritize what should go on our road map and what we should focus on.

The second, I would simply say is differentiation and personalization.

When I did assess the two years worth of subscriber data that we had, it was very clear signals that, we are not treating our paying customers any meaningfully differently than we are our casual transactional users. So we have to do better because I mean, they may they may not be writing us and saying I want a fully personalized experience, but they're telling us things like something else should happen. I should have more access of something. I should have less of certain things, like maybe less ad units served up to them.

So we are on a journey to think differently and differentiate our user experience based on who you are, what we know about you, where did you come from, and, just really optimizing that funnel along the way. And what we really want our paying customers to know is that we know you, we know what you registered for, we know if you've downloaded the app, and we're paying attention. And I think that's the least we can do, and to strengthen our brand, promise to our paying customers.

That is fantastic.

Now, Venika, I understand that you're working on redesigning some experiences to uplevel them for your customers. Can you tell us a bit more about that?

Of course.

So United Airlines is the world's largest airline, which comes with a lot of weight because we have a hundred and eight million unique customers. Each customer has a very unique need. So talking on the same veins as Nicole mentioned, personalization is a huge, huge thing for us as well because we want to cater to our customers to meet their needs and their expectation in their journey. So when you think about experiences of, let's say, someone who is in frequent travel, maybe takes a flight once in two years, we help them tailor their experience more in terms of we'll help you provide terminal guide, which is a tool that helps you walk through the airport, understand how to get to a different gate when you're connecting, what are the needs that you have, Do you go through customs?

Do you not? Do you take your bags? Do they automatically check them in themselves? What happens to your strollers?

What happens to your wheelchairs if you're traveling with them? So all of these questions will help you answer those questions. But when you think about those very frequent travelers, some of us might be here. They're like business travelers.

We call them as a persona type. They travel too often. They're like, cut the noise. I know where I'm going.

I know how to get to from terminal a to b, and I know what Austin Air looks like or where Chicago airport looks like. Right? So we help cater those experiences in personalization, and we are redesigning certain experiences to really up our game in personalization to really help cater to those needs of our customers. For example, very recently, on fourth of July, which is a huge, huge, huge summer travel day for us, we had flight disruptions, which meant servers broke down and bad weather, flights were grounded, no one was flying.

So imagine a the crowded airport, and it was a chaos at that time. And what we did was we created a new tool that can help customers rebook their flight. So we design new experiences. We redesign those experiences thinking about our customers in mind.

So someone who might need help in rebooking, how do we help them in those guided steps? Versus someone who's a pro at rebooking, make it easy for them. Give them their options then and there so that they don't have to go and talk to the agents to really help self-service them. So it's really thinking about the self-service lens because catering to hundred and eight million user base is really a big deal.

And we are trying to champion those experiences as much as we can with personalization as well as using innovative technologies.

So I don't know about you all, but I'm hearing a very strong theme about personalization being used to uplevel experiences. Tara, is there anything that you would add to that?

Yeah. So Best Buy is undergoing a pretty big transformation, from a brand standpoint right now. So if you've opened up the app recently, it might look a lot different from the last time you opened it. One of the big things that we're using is machine learning on our home page to, you know, personalize the experience even more just like, you know, Vanica is doing and what Nicole is doing.

But I work on the employee side, and I'm supporting the Geek Squad. And when people come into Best Buy, they need help with things, and they need things fixed. We you know, your phone breaks, your laptop breaks, you need help. And so what we're doing is we're getting rid of some of our legacy systems that we've had for fifteen years, and we're thinking about how can we give the employee a consumer grade tool to help them work better with our customers so that they kind of have a three sixty view about everything that's going on.

So that when they come to the store, we know everything about them. We know some things, you know, that they didn't think that we knew, and, we are able to provide them a much better experience than what they had before. And a lot of that has to do with personalizing the experience even for the employee and giving them some tools to help them with their job. And if every if you don't know other people at work, you guys could have the same job and be doing things completely differently.

And so, the tools that we use are really important to make sure that at least we're best buy that we're able to work really efficiently.

Okay.

Well, we'll just take that as a cue to move on to the next question.

We'll do that.

Alright.

So as you are, your teams are working with different stakeholders and various stakeholder groups, How is it that you're thinking about this balance between, and we got to we talked about this a little bit earlier today about how do you tie to ROI, but think of it more as how you're balancing the business value you're providing with the value that you're designing around these customer experiences.

Why don't we start with you, Veronica, on this one?

United Airlines, as I mentioned, is, is a beast, right? It's a huge organization. I have I work with stakeholders who have been in their roles for more than twenty years. I've also met people who have been at the company for more than sixty years. So they spend their lifetime knowing these tools, understanding the customers.

And when you are in a certain role for a very long time, you feel like you know all the perspective of bringing in a new design perspective, a new design decision?

What we did, on my team is I created and, brought along the value of a design thinking and b, a qualitative research team, which did not exist prior to more than two and a half years ago.

And this awesome rock star team, cheers to my team members here in the crowd today.

This team has really, kind of did a one eighty in terms of how our product owners work with us. They come up with such qualitative strong insights. They bring our stakeholders to the table. When we are creating these, kind of research proposals, they bring the stakeholders to the table.

Okay. Hey, guys. What do you want to ask? They get a skin in the game and you go and ask them, hey.

What are you looking for? What type of answers are you seeking from our customers? And they feel more invested in those experiences.

This team has also included them when we go do, research studies at the airport. So we are out in the airport talking to the customer, and they are part of that group. They are with us trying to do the same task. They understand it into the much better perspective of what does it mean to create new experiences. When we do this, when we make them part of the process, they become a shareholder or stakeholder in those conversations, and they are more than willing to then drive the change that we are seeking. And I think that's, like, the shared goal and shared purpose that really drives innovation and creating better seamless experiences for our customers.

I love what you're saying there about creating a shared goal, a shared vision, but it doesn't always start out that way at first. And Tara, you have some experiences, you know, working with different teams and bringing them along the journey.

Yeah. So just like Vanneco was saying, like, really early on building relationships with your partners are really important. So when I first set started at Best Buy, the team that I'm working with, they hadn't worked with UX design before, and they really didn't know what you user research was. And so I'm introducing all of this to them and having them be able to lead we have the luxury of being able to go into stores and, you know, actually talk to people and see what's going on.

Because those conversations can be in a bubble when you're, you know, in a corporate office or something like that. Your partners may think that they completely understand the person because they talked to somebody one time five years ago, and that's just not the truth. So I think it's really important to really understand and really build empathy with your partners and build trust with them. And how to build trust with them is to, you know, do these things together, to do workshops together, to have conversations together, and build up that that time together.

I even do funny things like, you know, doing team building activities and things like that just to help us feel more comfortable so that they can trust me more when I'm come up with a really big idea. And like I said, I'm trying to turn a employee facing tool into a consumer grade tool. And, you know, coming up with the with the UX basics, you know, like, this should be more minimal. We tend to overcomplicate things that are employee facing.

So that's really important to me, and, there definitely has to be that balance because at the same time, we're trying to make money and that really matters.

So, I think that there's always a a bridge that you can gap with your partners, and it's really important to meet with them early on in all of them, not just your product partners, your engineering partners, your even your analytics partners, like, whoever it is that you need to meet with. I think it's really important to have constant conversations with them and really make them a part of your team and part of your process.

I completely agree that building empathy and trust is so critical with any stakeholder team that you're working with. What I'm really curious, to ask Nicole about now is just how are you managing that balance between the customer's needs and and the business needs? Sometimes it feels like it may be a conflict, but maybe it doesn't need to be.

It definitely does feel like it's a conflict. I would say it's a constant balancing act. It's what I spend a good percentage of my weeks, focused on.

So I think about it this way. Make sure you and your team really know what the business goals are and not just, like, look at a brief and say, do we have KPIs on there? Like, really understand what your stakeholders are trying to achieve. Where are they in that journey with that? Maybe what things they have tried in the past and what's failed because oftentimes, that tells you where some scars are and where they might be resistant if you're coming out and and proposing a solution or even proposing to test something in a certain way.

Know what the trade offs are and the nuances of that. And the more you've conversant you as a leader can be and your team can be, the better.

And then take the time to share with your business stakeholders why you have confidence in your UX insights. And so, I've spent a good amount of time the last couple of years trying to prove, you know, the value of qualitative research. You know, I still get the question yet, but that's just five or eight people. Why are we gonna listen to that? I'm still having that conversation, but we're gonna, you know, push through to that. So it's always connecting what you're learning and what you set out to do, to how it can improve sort of the bottom line and sort of map it to those goals.

But to get over some of the qualitative research resistance, I'll call it, we do a couple of same things.

One, if we feel like we have learned something in an individual study, we we have some signals or some trends, but we've heard this over time in other reports and studies. We just say that upfront. I mean, I've we've worked we have a couple of canned slides of, like, like, we're here, but, like, we've heard this before. This is how this all fits in. So while we might be coming at you talking, we've talked to ten target users, really, we're looking at hundreds at this point so we can sort of have confidence.

We really also work very closely with our, quantitative research, market research group, and we're always discussing the push pull, the need, the balance of quant and qual because that's the whole story.

We, bring our data scientists in really early. We ask a lot of data questions. I've surprised a lot of people when I joined. I wanted to know what's converting, what's our turn rate, and this given UX flow, where's the abandonment?

And they're like, well, that's cool that you're asking that. But we're getting all of those points of view together is really, key to when you have the recommendations, you have the actual insights that you you can say, I have a lot of confidence in this because of x y z. And then on the flip side, really being honest about, like, hey, this might be a new signal, or this might be something that we really should validate before we make this change. Maybe we wanna have a quant follow-up study before we, I don't know, flip check out on its head.

But be honest and open about that and have discussions about that. I feel like that goes a long way of sort of being honest about, like, where are where are you again on that confidence scale of your insights and what else is needed?

I think I could we could talk forever about this.

Yes. But that's those are stuff.

I was just gonna add in real quick, like, like, when on on my side, you know, employee facing tools, some if we make a big mistake, that can really make a huge dent inefficiency for some for for a place like Best Buy. And if we're not finding a cheap way to test ideas and to fail, that could be really expensive for us. And, you know, like like you said, like, having those kinds of insights, but also really understanding the business of where we could, you know, make better efficiencies, where we can help our partners sometimes with the knowledge that we have based on what we know from our, you know, from all of our testing and all of our knowledge that our partners may not realize because they're so focused on making money. You know?

I think I'd like to build on on that point of data. Data is super powerful.

When we show to our people people in product roles and technical roles, they're very analytical. Like, they think from numbers perspective. They are not always very, convinced on the qualitative side, the empathetic side of design. Right?

So when we come with data and numbers and we show them those numbers, it really helps them. For us, for example, numbers would be, like, people visiting our site, people failing in certain tasks, people achieving a certain task. Right? But then we get further nuance in terms of who these customers are.

If we can tie a story with those numbers and build that story upon with qualitative research, that's incredible to get the buying that you guys are looking for.

Yeah. It makes a huge difference. Fantastic.

Okay. So for this next question, I'd like for you to tap into your crystal ball. Did you bring them today with you?

You did.

Okay. You're good. Okay. But let's talk about how you think about the future of design and how it's evolving.

There have been a lot of conversations about accessibility and more inclusive design, but it's evolving and it's changing. Where do you see this evolving over the next five years or so?

Nicole.

So I think about this question, a couple of different ways. So one from news and media products.

There's just sort of a big gap in understanding across the age demographics of things I think that the industry has taken for granted before. And so, an example of that is when we do testing, whether we're trying to change a staff bio or we're changing a layout for an article design, we're moving elements around.

Our younger demographic are expressing that they don't quite understand the difference between straight up reporting and opinion piece or something from an editorial.

So, to being inclusive of like the context that our different user base might have and that right now it's really important to, be credible and authoritative and be a trusted source of information.

I see us having to go extra lengths and really think differently about how we share the information, show what the source is, show what the process it was for the journalist to get that information out there.

As, you know, generative AI is playing to a big role in our industry right now, like being transparent if say, audio was generated by GenAI, or, you know, if if an image was, like, how transparent we need to be in our product experience is something that we are just beginning to grapple with. We don't have all the answers.

But on the flip side, we've long known that people consume information differently. People have different needs as it relates to really understanding information.

But it's really hard to design, like, take a piece of content and design it five different ways for five different user needs. We are starting to experiment with generative AI in in in ways of, like, taking a straight up full long form article, scraping that data, and then reproducing it, whether it could be bulleted summaries for a user type that needs sort of a quick scan, or flipping that into audio for people who need to hear it rather than read it, or maybe they can't read it, Or flip it into video and people, some people process information better from quick little bite sized information that they can sort of swipe through. I'm excited by that as a future because, like, again, we've all we've long known that consumption, modes, and needs differ. We just really haven't had the resources at scale to bring that to our user base, and we're starting to maybe see some light there that, that could become a reality for us.

That's great. Yeah. And, Monica, how are you looking at inclusive design at United Airlines?

For us, accessibility and inclusive are twofold. A, the customers were interactive with the digital channels. Do require government mandate. We have to be accessible.

Their website works and all of those things. But the biggest challenge that we work with are, folks who travel with us with disabilities. Right? With someone traveling with a wheelchair, someone has, like, temporary disabilities or superman and disabilities.

How we tailor their experiences is something we are really emphasized at United right now. For example, we just launched a new tool.

When you are booking a flight, you can actually provide the details of the wheelchair if you're traveling with it, and then we'll cater the search results to tell you which flight would fit that wheelchair.

It's huge in terms of the it cut down the cost for, like, call centers where people are asking, hey. I need to travel with a wheelchair, and you are being on hold with the amazing music that always plays when you are on full hold.

But it also helps and empowers our customer. It they feel more trust when they're traveling with us because they know we care. They know that we understand them. They know that you we are caring for them, and we are being proactive about their needs.

So then that instills a trust at the beginning of their journey. I think the other things that we are looking for are more like cool on the similar lanes. It's like AI innovations in terms of what can we do being mindful about our customers. Because in that huge customer base, we have baby boomers, and we have generation alpha traveling with us.

These are completely different needs of customers. Someone that texting age versus someone that calling age, how do we cater to both of those people? Right? So thinking about conversational search, using voice as a search, these are the things that we are tinkering on at United right now to really include them in our experiences so that we can help you and cater you in terms of whatever is your comfort level.

So we reach out to you rather than you having to adapt to us, from your digital experiences.

And, Tara, you have anything similar that you're doing as and I have very similar like, even though she's an airline, I'm a retailer.

Like, our customers are actually very similar because Best Buy is everybody. And then from our the people that we hire to be Geek Squad agents, sometimes they're right out of high school, and some of them have been working with us for twenty years and they're older. So they're interacting with software differently. So accessibility inclusivity has been top of mind, and my team hasn't actually really considered that much.

It's been an afterthought up until I started on this team. And so something when we started doing our vision work and started thinking about, how we wanted the future of the software to be, accessibility needed to be one of the first things that we do, not one of the last things that we do as a checkbox. And it goes beyond checking, you know, color contrast and things like that. So recently, I went into the store I went into the stores and I talked to over thirty agents over a week, and I discovered that, you know, probably, like, twenty percent of them were, colorblind and had dyslexia.

And that was something that nobody had ever talked about. We'd never tried to design for that, and there are so many pieces of our software that are color coded red as, like, an action item. Like, like, this does not work for those people because we found out that those handful of people that were color blind, they actually see it as green, which is the opposite of, like, the action that's red. So this is something we need to use different colors.

We need to be careful about the icons that we're using, but it goes beyond that. You need to think about the environment that somebody is using your product. Like, think about are they walking around in a noisy environment? Are they gonna be riding their bike when they're using their product?

Is somebody gonna be pregnant whenever they're using, you know, on your journey? Like, you know, think about all those other things that you need to do to be inclusive as well as the language that you're using and the images that you're showing. Those are also really important because I think most of us know here that representation matters. And when you see someone that looks like you and the things that you're using, that's really impactful.

Mhmm.

I'd like to build on that. I think language, is something that I'm very passionate about. I'm an immigrant. I'm not from here. I do not understand a lot of cultural references as well.

Sometimes I'm sitting in a room and people are talking about some random jargon industry American language, and I would not get it at all. So I'm always very mindful when we are building these experiences for our international travelers as well because we have a huge customer base. Even people living in America like me for a decade, I might not understand few words because that's not the vocabulary I'm used to. I'm someone who's always confused between the British spelling of colors and American spelling of colors because that's just my mind how I've been conditioned.

Right? So it's always very powerful to understand the type of customers, and we work with a great editorial team, to really help us build language that a five year old not five year. I think it's eight years old can understand from different experiences. So that's like it's not always thought of.

Like, language isn't always, like, a big criteria for inclusive design, but it is a very powerful one.

Yes. Yeah.

And one last thought.

Again, I think we could go on forever about this topic.

Being inclusive in your recruiting for your test participants, making sure that you're testing with, a broad spectrum of users as well as just not just thinking about the design for them.

So I I know we're running close to time, but I've just got one more question for you all. I am sitting here very inspired by you all, just the journey that you've been on as I'm sure many people in the audience are. And what I'm I'm just kind of dying to know from you is if you're gonna do this all over again or you had some advice just to impart on us today, if someone who's maybe in the middle of the journey, what would you tell us just to keep going?

I'd say get comfortable with being uncomfortable and pushing yourself in arenas that you probably didn't think you would be if you're in design and want to be a design leader. So again, it's the business acumen. I mean, what are we doing oftentimes? We're just bringing about change. So learn about change management, and become conversing in that because that's gonna come up. Yeah.

Negotiating and stakeholder management, all of these really sexy sounding things, but that becomes increasingly part of what your day to day life is.

Mhmm. Alright. I'll same. So like I've mentioned before, relationship building is a is a big thing that I wish that I had more courage to do because, you know, as somebody who works in an organization where UX design is pretty new, I really have to kind of shove my way as a seat at a table sometimes.

And, you know, somebody said, no. No. You don't need to be there. I would kind of before I would kinda be like, yeah.

Okay. I I guess I don't need to be there. But now I'm definitely a lot more I need to be there because I'm representing, you know, so many people that I'm designing for, and my voice needs to be there so that I can represent all these people. So that's something that I've been learning that I wish that I had done a lot earlier.

And, you know, not just speaking up for myself and speaking up for a UX designer, UX research, but also sticking up for the person that I'm designing for and what they need and what they what their pain points are because those things can get lost very easily when you're having those business conversations.

And I think it's really important to to make sure that you're building good relationships so that you have a safe environment to step up.

Yeah. I think, similar for me is the a lesson that I learned, over time is don't be afraid. Don't be afraid to speak your mind. Don't be afraid to speak your opinion because the best idea should not always trickle down.

Like, some people have been in their roles for a very long time. They're stuck on certain ideas. When you speak up, take the courage to speak up, and you never know what conversations you would spark. And I think the other thing that I am very passionate about and I hold I try to keep very sure to that is just be nice.

Just be nice to everyone around you. It makes your day easier because work shouldn't be stressed. It should be fun. And when you're nice to people, they tend to agree with you more.

I agree. You can get what you've done, what you want a lot more if you're just kind.

Be kind.

That's a great note for us to end on. Be kind to everyone.