Hello everyone. My name is, I am a solutions engineer at Symbiotic LLC. I have just over six years of experience in warehouse automation engineering, where I design, implement large scale automated distribution centers. My work spans fortune hundred retailers across grocery, healthcare app, RL, and paid food sectors. I help them transform supply chain operations with robotics, ai. And advance ML operation practices. Today we'll be discussing why AI system that look flawless in the lab often fail in real world warehouses or experiences, and most importantly, how we can prevent those failures. So let me begin with Paradox. The autonomous mobile market, mobile robot market is surging worldwide. Warehouses are inventing investing millions into AI systems that are in the lab and deliver success rates of above 90%. But there is reality when these systems enter real warehouses, performances often collapses. We have seen projects delayed for months, millions wasted, and operations disrupted. Why does this happen? That's the story I will share with you today. So here is what I'll cover in next 20 minutes for devastating failure patterns in ml in real world that are not considered, that issues were not considered in labs. The blind spot in the ML operations that caused them performance breakdowns, game changing strategies for success. My goal is my goal is to leave you a practical playbook that you can take back to your teams. So first failure pattern is the physical reality gap in the lab. Robotic picking bot looks flawless as you can see in the picture. But what happens when the system meets the real world? Scenarios like transparent shrink wrap. When the pallet is built completely by robots and they wrapped with plastic. Then second one is irregular shape products, and third one is dust interfering with vision sensors which are built, or which are, which have placed on the robots, the accuracy plunges, sometimes from 95% down to early 60%, and. What are the blind spots do you think of? I think blind spots are clear here. First of all, if I was mentioning about lab, that means training data. That lab is considering lacked messy real world cases or real world experiences. I would say second one is validation happen only in labs, not on life floors. And when errors happen, no feedback loop pushed back into retraining. So teams optimized for benchmarks instead of reality. They set their benchmark and they aim high in lab, but in reality it was different. So the results was brittle systems I, so second failure pattern I would say is forecasting models predicted demand perfectly until market volatility hit volatility, hit. Suddenly we had overstocking of wrong SKUs, empty shelves during peak demand and error rates, spiraling, unchecked. And what other root cause do you think of? So root cause first one is no robust drift detection, outdated ground truth, and no human in loop oversight. That means human in the loop verification points led to cascading errors. When error cost, and fourth one is no fallback plan when things went wrong, there should be some plan B. When we think of going system in wrong direction, it's like driving blind. The model keeps staring even as it heads off the clip. Third pattern is similar, I would say to everyone. Have you ever had a deployment where warehouses ran seven to eight middleware systems, and each one seemed fine in isolation, but the moment traffic spiked, the latency cascade across them like falling dominoes? The single schema mismatch become single point of failure, the entire deploy deployment. Collapsed like house of cards and the root cause issues. The root cause issues are monitoring was not shared across teams. When we are monitoring any issues or we are finding any, we are, if we are, we have new findings, we should always share across the teams and get their feedback. Second one is testing only covered components, not the full system, and no graceful degradation plans. There should be some plan B always in term when we know there might be some chances of failures. And the last failure I would think of is human factor. You can design most advanced systems, but if workers don't trust, they won't use it. Think of this what happens when early failures cause operators to bypass the automation I have dealt with. Operators with 20 years of experience, they are doing things manually. And when it comes, when the point comes of automation implementation and using those, the operators are hesitant to use the new advanced systems due to fear of losing job or they are not ready to use advancement. They think they are more productive the way they're doing the things. That's exactly what happened at one autonomous. Parts distributor, they invested approximate 7 million system and, but the no work, but the workforce rejected it. The entire system was abandoned and they wasted $7 million. So what are the game changing strategies you think of here? So I think first one is phase deployment. Start with small control pilots running shadow mode alongside existing operations Scale progressively when metrics are validated. So I have been working closely with one of the customer a retailer biggest customer in USA. They have implementing their warehouse automation into. Different phases. For one of the FA Warehouse, they have three to four phases and they will work on only one phase at a time. And alongside they will have their existing operations running. And this help that organization drastically, which reduce integration risk by nearly 40% compared to Big Bang launches. So obviously phase deployment always helps. This is the first strategy. I would think of and I would suggest definitely. The second one is comprehensive testing protocols. We should have a comprehensive testing protocols use advise testing, stress test edge cases. Always use the worst case scenarios for your testing. Use chaos engineering deliberately break things and see how your system approaches or deals with these things. Augment training with synthetic edge cases. And test, test the whole integrated system under peak load. And one other thing is always this system we should test in peak modes, which is like when you have festivals like Christmas, black Friday, sales. So we should consider those peak times and test your systems so that you get a hundred percent results and you understand where the failure might come from. Those who adopted these methods cut failures drastic dramatically. Third strategy, I would think of human AI approaches. Hybrid human AI approach. Define clear human in loop. In the loop protocols, there should be always some protocols to inter intervene humans in the automation and they should work side by side. Feed human corrections back into the training. If there are any correction errors made by human, we should always feed them into training so that those errors are not, have, will not happen in future. And design AI to augment humans, not to replace them. The best outcomes I have, the best outcomes I have seen were not just accuracy. Some warehouses achieved 99 plus percent of inventory accuracy and improved workforce morality. So as I promised, there will be a playbook. So here's the playbook I would think of. First, I would say it's validation. Validate the in the new invention in real world condition, not just in labs. Always use worst case scenarios and test your. Operations, monitor everything across all systems. Integrate humans in the loop and take their feedback and use that data to retrain them. Deploy incrementally, not all at once. Plan for resilience, graceful degradation, and fallback modes. There should be always some plan beef when it comes or when you know that the system might fail, For closing. And the key takeaways I would think of is the most successful warehouse automation projects don't always have most advanced algorithms. They succeed because they follow machine learning operation practices. So remember, the Lab of Success is not operational success. Monitor everything and plan for it. Plan for the drift test systems holistically, not just in parts that. Test them in parts as well as a hundred percent with a hundred percent implementation. Never underestimate the human factor. Always consider humans operators when implementing a new warehouse automation, because they're the one will be working. Ask them questions. Ask or take their feedback in the loop and apply when you are designing or when you're implementing things. Also make sure or feel them, let them or try to make them comfortable with the new machineries or new system so that they don't feel that they're losing their jobs and phase deployments to manage risk. Always deploy these new automation in different phases, not whole warehouse system in one go. So that's all. Thank you everyone. Thank you for your.