Hello everyone. my name is Hamad VAAs. I am a lead technical product manager at Quali. Qua is basically a game development and publishing company. we work both in mobile console, PC and all the other platforms including Nintendo, and, any other platforms that support games, whether it's web games, as well. I have, almost more than seven years of experience now in product management. I am a software engineer basically, and, I have devoted my entire professional experience to product management. and, this topic, leveraging LLM, For your data-driven product management is something that I am, myself learning and trying to implement that in my own company, and I'm seeing, terrific results already. So I'm very excited for this session. so let's quickly jump onto the session itself. So I'm gonna move to the session. so the topic itself, leveraging LLMs for data-driven. Decision making, in product management. this is something that, as I said, I'm learning myself, but I've seen, some amazing results to, use a large language models and to understand how natural it is. It's something that even I was surprised as well. So let's go through. So let's start with what is, large language models, as a brief, and the most simplest definition is that LLMs are advanced AI models, that are trained on diverse data sets to understand and generate the next sequence of humanlike text. So for example, if I'm saying, if my answer to how are you is I am fine. So the model would understand that a normal human talks like that and then they would try to, imitate. and try to generate text as humanly as possible. That's the most simplest of definitions of LLMs, however. It has so many vast usages, and you can now see, like chat, GPT, deep seek, Claude, all these, ai tools and AI search engines are now trying to imitate how a normal human would respond to a text. So it doesn't sound like a machine originated text, and that's how. LLM used the powerful, natural language processing to make sure that it's very human. you can use that in your AI agents, you can use that in your normal, conversations as your first point of conversations, as well as your entire, your entire customer management can just be used, can just be managed using the powerful LLMs. That's the brief definition of it, but we will, we'll dive deep into that later on. Now, the other part of this, session is data-driven decision making right now. Product management is taking a huge turn towards using large sets of data, plotted in very user-friendly graphs, and funnels. this is very crucial when you are planning your roadmap, when you're making sure that your product has the feature that the people will actually use. So it's very critical. That not just, in the roadmap, you use this data driven decision making, but once you have deployed this feature, you make sure that your customer touchpoints are properly implemented and that you make sure that once you have deployed a feature. You know that what KPIs I'm going for. Those might be retention, those might be conversion rate, those might be engagement rate. Whatever your KPI is, someone has revenue, number of orders, number of clicks, number of conversions, whatever you're going for, you have to make sure those touch points are the ones that is. making an impact in your decision making. That is your data-driven decision making. This approach heavily relies on metrics. Now, imagine data-driven decision making combined together with LLMs, LLM, bringing huge sets of data that basically define that how a human would emulate your features before even you deploy that in the market. That's how powerful this can be. So that's the. The use and the importance of data-driven decision making in these times, basically. So now. When you have understood LMS and you have understood the need for decision making, with proper data, let's combine together and understand how LMS can enhance your product management and your decision making. The first thing and the very, very important is market research and trend analysis. Now, when I'm doing it alone with just my own research, I can probably find 50 competitors, right? Or even a hundred competitors if I have enough time, but. LMS can now scan large data sources because it is, it has a vast data sources. it can help you using your competitor updates, sorry, updates, industry reports. Your news, identify key trends. you might even get real time insights. So for example, if I am. If I'm building an e-commerce website, for example, with, a specific niche of products, LLM would tell me that what is the segment of users you should target? How they would behave in the market with this spec specific audience that you're going for, and whether you have to make early decisions before going into the market and understanding. The next is customer feedback analysis. So once your market is done, market research is done, then you have to understand how you are collecting your customer feedback. Now LLMs can again give you, some proper categorization, whether it's sentiment analysis, whether it's, highlighting your recurring issues with the customer. for example, you have 10 million users and your LLM can eventually tell you that what is the issue that is constantly occurring for a specific audience, and then you can actually make decisions without scanning all those customer feedbacks. Your feature prioritization. This is very crucial as a product manager and I hope the product managers would agree with me that when you are prioritizing features, you have so many different influences. Your CEOs, your COOs, your lead product, your business analysts, they all will try to influence features. As they want. But when it comes to decision driven, data-driven decision making and LLMs, you actually have the data in front of you where you can prioritize. You can actually emulate how your features were due using the LLM dataset, and then you can eventually give them the data even before going into the market. It can give you historical feature management, like how historically this feature. This kind of feature has behaved in the market. and also what's the customer demand? LMS can even predict the potential impact of new features on your product before you release the product. And you can actually show your C-E-O-C-O CFOs that this is how the data. is talking, so you wanna listen or not AB testing, again, no product manager can survive in this industry without properly ab testing. The, your features before you, like when you launch them. LMS can actually automate the interpretation of test results, identifying which variations perform best and viable. Not just the, with the current data, but actually the historical, the predictive data and the current data that you have. So if I'm a product manager, I only make decision with the current data, but LLN can give me predictive analysis, historical analysis, and current analysis. Automating the reporting part, again, is very crucial that LMS can generate ex executive summaries, dashboards, key takeaways is pretty self-explanatory. Speaker notes, again, by leveraging lms, product managers can actually automate market research, analyze customer feedback at scale, and generate meaningful insights with minimal manual effort. that's very crucial that you're actually talking at scale with LMS and not just your current data set. Moving along, what kind of tools or technologies LMS are being used currently in the market? First thing is you have OpenAI your chat GPTs, your, all the buzz in the market with the chat g bt that they all are using, for, they're used for text generations. They're used for summarizations, LLMs. This literally are the data sets that actually give you the human output. You have your Google Bird. again, optimized for natural living, lang natural language understanding using the LLMs. you can use Claude Ro, you can use, again, hugging face transformers. that's basically up to you what kind of combination of tools you're going for. You can even go for your custom LLM integration that's. That's a bit, resource heavy, but if you want to make your own use case, you can actually use open source APIs and then you can make it in and in the way that you want. I would suggest to start very simple to use like, let's try chat GPT. Try and understand how the data set's working. Just make sure that your features are prioritized properly with chat g bt, for example, or Google Bird. And then slowly and gradually. Start about, a humiliating your ab test, your insights, trends, your predictive analysis. Just make sure to start small, but make sure that you are using historical and predictive data for your future planning. What challenges and considerations you need to make whilst you are, transforming to normal product management, to data driven LRM based product management. the quality, the data quality, and the bias. One thing that I always say is you cannot a hundred percent rely on large set of data. You have to make sure that. the quality of the data and the source of the data is validated. If it's not validated, you are basically planning to fail. Just make sure that the sources of your data is properly validated. You know that the, where the data is coming from, and once your entire segmentation of data is ready, only then you make decisions. Otherwise, you will go towards very biased features, biased results, and inaccurate predictions, which is something that. The, some product managers like are actually making this mistake in the market right now. Interpretability is very crucial that you have to understand the data. If you cannot understand the data, make sure that you have a data engineer, you have a data scientist, you have a, an analyst who can tell you, what this data means. So just make sure that you understand the data, scalability, and cost. As everyone knows, it's not cheap to, to, to use these models. Just make sure that you start small. You have to make sure that it is actually useful for you. Don't just jump into the bandwagon. Make sure that this is actually useful for you. If it's not, if you have the data and you only need to use the current data and real data, you don't need to, go towards this direction. Ethical and privacy concerns. Some data sets are not properly, sourced. just make sure if your organization is going towards this route, your legal, actually check the data sources as well. some best practices. Again, this is. Purely subjective, purely coming from my side. It might be different for you. It might be different for anyone else. some guidelines for, from my side, for success in this, in this domain is clearly define the problem. Just don't give the problem. Like customers are not logging in. That's not a problem. That is an outcome That is. An outcome of a problem, make sure your problem is very well defined. Only then start talking ai. AI is not step one. It's not step two. It's not step three. It's probably step five. Step six, make sure all your steps before AI are properly sorted. Use high quality, unbiased data for model training. Make sure that your data is properly vetted, properly validated. Treat AI as an assistant, not a replacement. that's something that I can just make a headline. AI is not here to replace anyone. People who use AI will replace the people who don't use ai, but AI will not replace anything. Make sure that AI is your assistant. It's not making decisions for you. Continuously test, monitor and refine models to improve accuracy. Again, let's face self-explanatory. Communicate AI insights transparently to stakeholders, but make sure your analyst actually agrees with the data as well. Sometimes when you're talking with large data sets, some of the data will not make sense. So your analyst might cut down garbage data and give you a proper graph, and then transparently show it to stakeholders. What future I see of AI in product management. I think it's quite clear. Your bis, your proper business inside should properly be using ai. It's very crucial that you have predictive models. You have current and anana model and you have historical models. They all should work seamlessly merging with business intelligence systems, real time AI, decision making. That is something that is still quite far away, but I have seen intelligent CEOs referred to as I CEOs who are actually CEOs, but an AI and who are making business decisions or actually just suggest suggesting business decisions. So it can be happening in real time, but still a bit far away. But I think. AI should still be vetted, should still be properly managed, but there will be a point where AI will constantly be making decisions without actually involving anyone else, but that's still quite far away right now. As I said, AI cannot replace anyone at this point. Improved explainability, AI models will become more transparent in how they generate insights. Fully AI driven product management, still quite far away, but that is the future. it might not be, an automated solution, but product management, product development, product management is different. Product development is different, but product development will heavily be influenced with AI code sourcing. So AI might be writing code for basic models, but still you need software engineers to properly validate that code and actually improve that code as well. So that is everything from my side. As I said, this topic is still something that I'm learning as well, but it's a need of the time. You have to make sure that your, your product management is properly, improving with these emerging technologies. You like lms. and make sure to be, on track for yourself to be improved. And, for improving not just yourself, but your product, your company, and make sure that your customers are getting the best out of your product using the LLMs and using all the technologies that the current industry has to offer. Thank you so much. I'm very thankful to, the Con 42 conference, for inviting me to speak and, please feel free to connect with me on LinkedIn and, I'll make sure to respond anything that you guys have to ask. Thank you so much. Have a wonderful day.