Hello, my name's Ed Fullman. Our topic today is harnessing large language models for customer experience. We're going to look back at the past a little bit and then forward into AI and new technologies and their impact on customer experience management. A little bit about me. I've been working in technology for over 40 years. I'm an engineer. He started as a programmer. I. Many years ago, my focus now is mainly on innovation and new product introduction and specifically product management and lead leading, global engineering teams. In the past, I've been more directly involved with call centers and CX operations. the largest was A BPO that I was a leader of. For a large telecom, and their broadband support where there were 4,000 agents in seven countries from multiple vendors, that I pulled together and, That was a experience that obviously was very informative about some of the opportunities and some of the challenges that call centers have to deal with. But I've also worked on the technology side of call centers and, CX in general, multimodal. a lot of my focus, in the two thousands and 2010s was about, the technology to deliver the calls. voiceover IP and other things like that. And then in, in more recent years, in the last 15 or so, machine learning applied to, targeting for loyalty and sales and also for trying to personalize calls and personalized offers. So a lot of the stuff we're gonna talk about today, I've done with one technology or another, or set of technologies. And I've done it in multiple industries. Today at Veritas, I'm Chief Product Officer and our focus is on, cloud Native Edge and distributed architectures running various kinds of technologies to deliver high value automation for customers. Veritas means truth and automation. the kind of problems that we're after have a need to, they range from solving problems that involve hardware to fully automated solutions. what's probably different at some level with what we do is that, we're not just building these things. We're operating also these kind of platforms and, those platforms have generated. Billions of dollars for big brands that we work with through our, partners. And, we've generated billions of transactions each year, that are processed by our systems. So there's a lot of, customer involvement, consumer involvement as well. And, we're, we have a lot of familiarity with that and a lot of focus on that. So some of the technologies and areas of development that we focus on are, everything from staff supplementation and building applications to more process and project and product oriented solutions that we deliver. the, we're involved a lot in development of AI and analytics. we specialize in, cloud native solutions involving Kubernetes. from the cloud to the edge, we have expertise in bare metal. we've had specific expertise using blockchain and digital chain of custody applications, not so much, payment or crypto, but using blockchains to do other, kinds of supply chain oriented things. And then. we've implemented these kinds of solutions altogether to create product frameworks that we use with clients. So this chart gets us to, the blend of my career and the world at large around, customer success and, and how you. Drive different kinds of customer experiences. So I started out in live call centers in the nineties and, specifically for telecoms focused on sales and service. in the two thousands I worked on, as I said before, one of the largest BPOs of. Of customer experience management. And this involved, broadband technical service for, technical support for, one of the largest telecoms. And in that environment, we were, I was working in seven countries. And we had people also in the US and we had people, spread out around the world and we were trying to deal with, keeping the unit cost down through, broader scheduling across multiple vendors in multiple countries. And, a lot of tactics that, have been upgraded over the years with the advent of machine learning and ai, different kinds of AI solutions. So the first two, boxes on this chart, really the focus was on, call routing. some work on, improving the personalization of the experience and routing around that. a lot about scheduling and training, because the difference between the BPOs and the live call centers and the web self-help was a huge difference. the work I was doing at the time was creating time, reducing costs and creating time for web self-help to be built and rolled out by, by teams that I was working with. web self-help was all about, getting in the getting calls captured into a process without a call center where, the users could find answers to problems that they, wanted to find and move on, or use some kind of system that could actually create some kind of settings change or something like that would, Enable the service to work better for the client in one way or another. So that kind of avoidance was an enormous cost, savings for call centers. It diverted, calls away in a very large way, and it still does today. So web self help, which has become AI enhanced is now. Essentially eliminating calls and turning calls into other kinds of things. And the AI and LLM addition to that is just really taking it to another level rather than have it be structured and following a flow chart. Now it's, it's derived by a chatbot or some other kind of technology around, that ai, which gives us many more points of data to do things like scheduling workloads or, understanding sentiment or, some kind of call avoidance in, in a multimodal way that, really is designed to, to get the customer. To a point where, they're happy with the experience. And, and this has been the last 40 years, the last 30 for sure, is on this chart. So the strengths and challenges of call centers are pretty well understood, but for the broader audience, The value proposition of a call center is really the personal touch, real time support from the perspective of you're actually talking to somebody who's working with a system that is going to change something for you, change settings, make a payment happen, reverse a payment, whatever. So the The value is real because someone's there doing something for you. that's your sense when you're talking to someone on the phone. The challenge is it's a very high cost solution. If it's, if your call center is handling all kinds of calls, including calls that could be avoided, the attrition is very high and it's, very hard to deal with. certain types of call centers like collection call centers are extremely high attrition. and the issue here is that, as a product manager, it's very hard to use your call centers or, it's the only way to deal with. A customer encountering new features that have been released into software or SaaS or some kind of solution like that. without some way of diverting calls that relate to customers, especially those dealing with a new service or a change in service through a new feature being released, the cost is extremely high. The value is not as high either because you've got. Low value calls mixed with high value calls, and that's what actually causes the attrition in a call center. Nobody wants to do that work. The era of AI has different opportunities and challenges. AI takes away some of these issues we just discussed around call centers, mainly by diverting it away from the call, but. Also assisting or helping the agent be better informed, doing the right thing, and using AI to get that to happen. So it creates scalability, interactions become more cost effective. The flip side is the technology implementation costs. the training of models to be effective and not misleading or provide misinformation and the management of trust of that situation. So, customers are not, I'm sure some are unaware that they're working with ai, but customers are not blind to the what's happening, so they need to feel like these are the right answers that they're getting. the idea that they could be ideas that adapt based on who the person is exciting at some level, but it also means it's not deterministic. In other words, the answer is not the same for every customer. So that leads to trust, issues that have to be like focused on, and mainly, what that means is. In the end, even though the message might be different, this, the problem needs to be resolved. The issue needs to be resolved. And if it is, then the trust goes up. So what's the role of AI and the transformation of CX to be this way? so here's a few of the things that we're gonna go through in more detail. personalized customer interactions, proactive resolution. These two are really the, the holy grail. This is what we're all after, right? If we can create an interaction that is specific to the, the consumer's problem, if we can get in front of it and use some kind of mode other than a call center to actually deliver. an issue resolution or inform someone of a problem that they may have and how to resolve it. These are really, this is the, all the value we want from the customer experience. enhancing self-help. Is a huge opportunity, and making it adapted to the customer based on their skill level with technology and so forth. These are all things that are possible, and, being done at some level by different companies. efficient resource allocation is an area as well, and sentiment analysis has been upgraded through the latest AI technology. So let's take it apart. personalized interactions. mainly what we're talking about here is chat bots, that people get a response from based on their question, which makes it a personalized interaction. it. It's able to be used as well by support, by the actual call center agents, so it gets to faster resolution for their clients, right? If they can see what they've done and. They understand what they've been shown and the problem was figured out properly, then you can accelerate the whole process here. so the chat bots, as we discussed before, they avoid call volume, right? So there, it's an avoidance strategy, which is, has been a good one in general for, many decades. So the opportunity here is to really. Supercharge again, the whole experience around virtual assistants and chat bots, and it's got a lower cost, and it delivers a personalized experience. So all around it's, this is why all this excitement exists in this area. From a perspective of proactive issue, re resolution, what we're talking about is detecting patterns to prevent customer complaints, to actually get in the way of an escalation and stop it before it starts by being proactive in an outreach. An outreach that doesn't have to be a phone call, but it could be. in the right situation, that might be the choice, but generally the, this is multimodal. So it could be text or some email or some other method that gets, issue escalation stop before it starts. So this is an area of, Prediction, right? So predict prediction of potential service failures based on reviews of calls coming in and, finding like backtracking to some kind of bug that's been released or, and in other cases just avoiding a certain kind of configuration which causes problems, competing configurations in software. So predicting these service failures based on, customer experience interactions and what's going on with your software solution, for example, is a way that, that this technology is applied here. alerting businesses, before customer escalate is a goal here. and reducing negative experiences. So the other area here is, enhanced self-help. So what we're talking about is going from, the web health that started to emerge in the 2010s to where we are today. And the last, decade or so is the application of, machine learning to find patterns and then. More recently chatbot to create, dynamic FAQs, dynamic responses to, that are designed to be aligned with, the skill level of the, consumer involved, continuous updates based on other queries that customers are making, which indicate some kind of pattern, like an outage or, Some other kind of problem that's emerged as part of a new feature or something. So AI powered Certs is designed to improve the efficiency. it's going to look across a larger set of data points and basically pull together. based on all of that, including the most recent information, what's going on, and try to deliver a more efficient solution. And it reduces the need for human agents at some level. human agents are gonna be informed by this technology as well. although this topic isn't specifically about the agents, and there's another topic about that, this is how. They'll become more efficient if they get on a call. efficient resource allocation is something that's enhanced by the, increased number of data points that, the latest AI can look at. There's been a lot of work for decades on scheduling efficiency and, that kind of baseline backend process is pretty stable right now. What's the input to that is really what's changing. chat bots and other kinds of technology that are more AI and machine learning oriented are able to figure out new patterns and lay those on top of existing ways of, Displaying scheduling information and figuring out, peak support times and figuring out routes and inquiries, and generally leading to better balance of workload in the call center. And I think, some of this, again, from product management's perspective is including in your data that you're looking at, New features that have been deployed, new features that have been pulled back in a continuous delivery environment where you can see maybe there was an in, there were calls coming in on something and then it was pulled back, right? Something was like being tested and some type of AB testing and, maybe there were a group of failures. So. the new AI technologies are able to consider those and then, wait, the response. and from a product management perspective, you're going to probably want to have, the customer experience that's handled by the AI to be influenced by what you're doing as far as releasing new features. So workforce optimization, as I had mentioned before, it's an area that's received tons of investment and there are numbers of companies that are very good at this kind of work. And now AI is being blended in at a significant pace. They're basically see themselves as AI companies. Now from an CX perspective, But generally the goal here is to reduce operational costs, to flex calls and have workforces optimized to not only take calls from customers, but also, in some environments play a role as, reinforcement learner, people that are focused on grading and reinforcing learning from the ai. these are some of the biggest experts. That exist in these companies as far as how things should operate. So looking at, proposed AI responses and grading them is, a realistic role for the call center to play, in my opinion. And this is the way that you get them involved in reducing calls that they don't need to take, where the answers are somewhat deterministic and you can get to something that will solve people's problems. Sentiment analysis is another area that's had tons of investment already. and AI is just a new enhancement as it gets to a place where, more can be done. So in the NLP area where you're actually listening to calls with ai, which has been going on for some time, the improvements are substantial. And the opportunity to figure out tone and sentiment, not just in calls with NFP, but also analyzing text, and text responses and chat bots is something that is key to avoiding frustration. here's a couple of case studies. This one is actually one in which I worked, that I mentioned before. And, this is a very large, technical support, set of call centers and, actually web self-help and some chat, that was, deployed to basically deal with problems with broadband. what was interesting for me about this particular experience is when it started, this was, an engagement I was involved with for about five, six years. when it started, the early, we were still into the early adopters, and those were mainly technical pe people that understood the technology at some level, not people that, were afraid of the technology or. Didn't even buy it. Someone else bought it for 'em. So as time went on though, that's who we started to serve. So a, a very common thing that you see in call centers is that as you bring up a new service or a new technology, or as you introduce new features, you as a product manager, you end up with situations where call center satisfaction goes down because. The problems aren't well understood that they're gonna have to be solved. And then there's a little bit of adjustment that has to happen. AI dramatically speeds that up enormously. So the call avoidance is very high in those spaces, like telecom because a lot of it's technical. There's only a few ways to do something, so it's relatively deterministic. And the opportunity here is substantial, as with this particular company. this is talking really about their last five years or so, five to 10. And, they've had major reductions of basic call volume, really beginning in the, two thousands and early two 2010s. And now they're just getting farther and farther into eliminating call volume. So where they were thousands of agents in let's say tier two and tier one. now that's been reduced substantially. And my ex, my expectation it would be that the cost of individual agents has gone up, but the specialization that they do and the value that they create has gone up as well. And they, they can target very carefully who needs to talk to an agent and at the same time have good exit paths from their, process so that if someone needs to escalate out 'cause they don't understand what's going on or. The chatbot doesn't, has never seen their problem before 'cause it's new. those kind of exit paths are easy to get, put in place and just generally faster issue resolution. in, in this environment, in the telecom environment, especially broadband where you're talking about configuring a router to, to some extent, in the past it was much more detailed than it is now. getting off the phone. was very difficult, but there was just a lot to tie down. So the opportunities with AI are really resolved as offline, not in a call, in an actual call or something, and drive a big, a faster resolution, which is what people want. another that is, I've experienced more as a consumer than an operator is, proactive call rebooking. When a flight gets canceled or it's late or whatever, this is truly where everyone wants to get to as far as the ability to find ways to avoid a call and resolve a situation. through technology, there's nothing better. This is the kind of the highest value. So while reducing call volume is a great thing. essentially avoiding it through ai. What would be even better is to resolve the problem before, the customer has to deal with it, because that frustration is gonna be what causes churn. Or in this case, somebody choosing one airline or another in the future, banks have also. Had a kind of experience that is similar in some respects to the previous two, but different. this first line AI assistant handled 1 billion plus interactions. It's extremely unlikely that those 1 billion interactions would've occurred. Without, online web self-help or chat bots or something else, right? No one would. It's unlikely that people would check their balances often. It's unlikely that, people would have interactions with their individual transactions as much as they do now without AI and chat bots. So this is actually. An example of increasing, satisfaction of customers and along with that, increasing the volume of interactions, but dealing with them in a low cost multimodal way. 30% call re reduction is on what they had coming inbound on calls. So they still get that big lift that you saw in telecom. and then they have this additional, Volume, which is all customer, success that is focused on how the people use the service. So improved high value service interactions, really the result. So here are a few of the metrics that you want to, be tracking. if you're not measuring it, you're not gonna be able to do anything about it. So. what's new here is really AI deflection rate, first call resolution and CSAT have been in place for decades. agent retention has been also measured o obviously, for especially, large ca call center ops or operations with outsourcing. The cost of retraining and bringing agents online is super high. So whether you were a call center outsourcer, or. The owner of the call center, both time, both ways. This was a big focus. Satisfaction of agents is also something to be monitored here, in my opinion, because I think that you're gonna end up with learnings from that. That are gonna show you where you're, where you need to tighten up on communication and so forth. Like what you're communicating, coming out of, software development, for example, and into production. How you're communicating that and what the training is associated with that. A lot of that is, is hard to understand unless you're looking at satisfaction of agents. As we look forward to AI being included into our operations and figuring out ways to do some of the big value prop, prop, big value propositions, how we're going to, proactively get involved in resolving problems and other things like that. You wanna start with something small as. is always advised pretty much in all technology. So start with a small project that's gonna have measurable value so that you can actually see some kind of improvement. ensure that you have human escalation paths well figured out, especially if you're going to do something that's got, potentially lots of steps involved. anything like that ensure that you've got a way to jump out of the process, because what you'll be able to see is where the process is weak, depending on where people jump out and, maybe where it has to be enhanced. So if you provide good escalation paths out, then you won't have people hang up. For example, mid. Mid interaction or kill the text in the chat bot, what you'll see is where the problems got too complicated and the user was not able to get the answer that they wanted. So escalation paths out of the process that you're trying to implement is pretty important. And then just focus on feedback, which is also hard to get, which is why you want to focus on these human escalation paths. What comes out of those human conversations with people about like why they couldn't get it to work is gonna be much more easy to pull out then, even though it's anecdotal, then real world feedback where people are doing, survey responses. So pretty typical red map here. The focus here is just that this is a technology deployment and even if you buy a third party service, you need to follow, a reasonable path here. So that you don't create, paths within your customer experience management that are half finished and unable to really deal with the problems that are going on. So a lot of monitoring of chatbots, for example, a lot of, reinforcement learning going on with your agents is only gonna help move this along because it. Where AI differs a little bit is that you are, you're managing the growth of a model and a knowledge base of a model and the true ups of a model. it's a different kind of iterative, almost research oriented work. So the more data that they can get during the early phases, is pretty critical. And then as it gets into production. You need to be focused on getting feedback both from escalations as well as after action feedback from a survey. So what's the future of where all this is going? AI plus some kind of hybrid human service where a person can jump into a call based on. Priority or, re relative to, product management that's going on, like new features getting released, tying those more tightly together. That's pretty much the future. doing things in multiple modes has been happening for decades as well, and that's only gonna continue. And then really the focus, the opportunity is predictive and proactive customer support. So as some takeaways, I would say generally that AI is not in a place where it's designed to replace human interactions. AI is here to help enhance them. so you know, where we can replace human interactions is where we've been replacing human interactions. So basic call volume. questions that have very deterministic outcomes, right? So they're always the same answer every time. Tho those are gonna be the places where we're gonna get call avoidance and they always have been. So the opportunity here is to really focus on that with the new technologies. But it's not gonna eliminate call centers 100% unless your business is that deterministic. Right where there's complexity, you're gonna need people. And without those people you risk losing high value, customer interactions or even losing the customer. AI is going to help improve efficiency and satisfaction to customers. So that's what you should be looking for and measuring. And product managers need to more tightly integrate what's going on in product management to this CX environment. There has to be a very tight integration so that if people are reporting problems, that it becomes very clear to the product managers as quickly as possible what those issues are that they're encountering. And almost in real time you should be looking at it. so even if your product is, has a high touch deployment with the clients, and you get feedback from those kind of deployments and installations, at least the early ones, you, as you get into more of a steady state and you're changing features out. Releasing new features, you really need to be tied together with your CX and you should expect that you can get these kind of responses from it, which are gonna be super valuable for product management and overall customer success. thank you very much for the time. if there's questions, please send 'em our way and I'll be glad to get back to you. Thank you again for your attention and feel free to reach out to us or me. You can use this QR code to reach. Thank you.