Conf42 DevOps 2024 - Online

Looking Back at a Lifetime of Poor Tech Choices

Video size:


Watching your technology choice go down the drain—because of broken promises, vendor implosion, or obsolescence—can feel like a career-ending experience. But in my experience, there’s usually a lot of good that comes out of those seemingly bad tech choices


  • Leonidato: In tech, we get to choose a lot. There is no such thing as a poor technical choice. Here are some common decisions which are clearly bad. Leonidato hopes to give you an idea that you can layer over with your own experiences.
  • In 1984, case Western Reserve University, IBM, and at and T wanted to build the world's first free online community. Freenet was basically a glorified message board. Without those hours I spent on freenet, I wouldn't have gotten the chance to move up.
  • Borland got his start in tech as a classroom instructor teaching adults how to use things like Wordstar Lotus. When he was out of work, he learned HTML and started building websites for people on the side. His leap into the world of monitoring started with a tool called Tivoli.
  • The sheer number of choices and possible negative outcomes can be paralyzing. Do not try to hide your mistakes. It's always worth working with what you have now compared with waiting for the next thing.


This transcript was autogenerated. To make changes, submit a PR.
I'd like to start out my talk by dropping a little bit of a truth bomb on you, and that's that. If there's one thing I've learned in my career in tech, it's that we get to choose a lot. Now, we may not like what we're choosing things horror shows from my past that I've had to choose between, and we may not like why we have to choose any of these things at all. And we definitely don't like how often we get to choose. Don't bother reading the details of this slide. It's an eye exam that basically is saying, shit changes all the time. In fact, one of the things that I laughed about the most at the start of the pandemic was all the media outlets saying, oh, it's the new normal. And I'm thinking, oh, you mean when everything we know gets thrown out the window and we have to find a completely different way to do every aspect of our jobs? Yeah, in tech, we have a word for that. It's called, oh, my gosh, look, it's Tuesday. The sheer volume of opportunities to choose creates a number of challenges, but maybe none as much as the paralyzing fear that we might choose wrong, which is the point of my talk today. Thank you for joining me. As I look back across a lifetime of poor technical choices or another title for this is when you stare into the face of failure and success, smiles back. I don't know. It just happens. Thank you for joining me. My name is Leonidato. I'm a middle aged white dude giving a conference talk. I selected this picture to give you confidence that I know what the hell I'm talking about. Nothing else on that slide really matters, except that you can find me, as at Liana Dotto, on almost all social media. More importantly, I want you to understand that this is what I call an oyster talk. Now, I got that idea from the parenting educator, Barbara Colaroso. And the point is that I have about half an hour to talk to you here, and I cannot teach you anything in that amount of time. In fact, in that amount of time, about the only thing I can do is irritate you. Now, when an oyster becomes irritated, it does one of two things. It either creates a pearl or it dies. Now, it is my sincere hope that I do not irritate you to death today. However, I did think it was fair to let you know that those are some risks. Instead, what I hope is I give you just the grain of an idea that you will then think about and layer over with your own experiences until what you have is something that is polished and unique and valuable to you. Now, after 35 years in tech, I've learned something, and I want to share it with you all right up front. And that's that. There is no such thing as a poor technical choice. I know because I've made some doozies. And today I'm just going to share a few of the residents living in my technical house of horrors. Okay, I take that back. There are some common decisions which are clearly bad. For example, not having backups, that's bad. Or thinking that replicas or mirrors or raid are backups, that's also a bad technical choice. Moving fast and breaking things in test, sure. In prod, not so good. You do have a test environment, right? Like we all should have a test environment. We should all have an environment that we test things in that is not actually production. Okay, you get my point. Bad technical choice. Trying to do everything all by yourself. Bad technical choice. Not a personal choice. A bad technical choice. Using oracle for literally anything at any time can be considered a bad technical choice. Starting a land war in Asia. Okay, fine, that was a princess bride quote, but I still had to include it there. All right, so let's start. In 1984, case Western Reserve University, IBM, and at and T wanted to build the world's first free online community. Now, just for context, case Western is a university not known for its technical program. IBM is the stodgiest computer company in existence. And at T is the zombified remains of a phone provider that got more or less legislated out of existence. What could possibly go wrong with this plan? And unbelievably, they succeeded. They created a free public community computer system which bore a strong resemblance to the BBS software I downloaded onto my dad's compuware pc and ran when he wasn't home, until he got the bill after the first month, the telephone bill after the first month, and shut that all down. Now, I spent a lot of time on freenet for a few reasons. First of all, it was free. Second of all, you could reliably download uu encoded ASCII porn on it. And third, there was a one limit, time limit on that BBS system unless you became a moderator in one of the SiGs or special interest groups. So I became an admin on the dungeons and Dragons Sig because, of course I did. And I also became an admin on the word processing Sig and a few others along the way. Now, you'd have to be a child of the understand exactly how much time you could sync into a dial up service. But you would argue it's a complete waste of time, right? So freenet was basically a glorified message board. And what you did on it was that you posted messages on different areas and you sent comes to the maybe 20 other people in the world who had an email address in 1986. And to do that, you needed to select an editor to actually write that stuff. Now, you had a choice. You could use a homegrown abomination called Fred or Emacs or Vi entirely at random. I chose Vi as my default editor. I was 16 and I didn't know any better. And I learned Vi like it was the most natural thing in the world. So, fast forward to one of my first jobs. I was working desktop support in an engineering company, and the Unix admin came down. He was looking for one of us to train into supporting one of the new systems. And so he sat us down. He was one of those typical, you know, grumpy big beard and everything. And he talked to us and he said, okay, well, I'm going to have to pick one of you boneheads to do this. And even before I can teach you the system, I have to teach you this weird thing that nobody ever gets, and it takes forever for you people to learn. It's called Vi. You know that scene in Jurassic park? Yeah. The point is that without those hours I spent on freenet, I wouldn't have gotten the chance to move up. By the way, the system that I learned was bind version eight, the first production version of what would eventually be called DNS. There was no part of this talk where I say that learning DNS was or is a good choice, just to be clear. So next story, I got my start in tech as a classroom instructor teaching adults how to use things like Wordstar Lotus one, two, three. Word perfect. And my boss thought it was important that I have some certification since I was coming from outside the tech sphere. And so I spent 80 hours taking my word perfect certified resource test, not studying for the test. It took me 80 hours to take the test. Back in my day, we had to certify 10 miles uphill both ways in the snow, and we liked it anyway. Where's word perfect now? I mean, it's kind of part of comes, but not really, but bought by Novell anyway, it was a complete waste of time. Right? By the way, you can see the certification right there. Yeah, complete waste of time, right? Well, Wordperfect had gone the revolutionary route of hiding formatting codes before that, if you wanted to format something, you actually put the formatting codes on the screen with everything else, but you would see instead a representation of bold or underline or whatever. Now, sometimes you had to fix overlapping codes, and for that there was a feature called reveal codes. It looked like that. So you can see down below in the second bottom half of my screen here where there's underline and where those bold starts and stops and all that stuff. Which is why a few years later, it took me about 15 minutes to wrap my head around how HTML worked. And that helped me in a lot of ways that I'm going to talk about in a little bit. But let me keep going. So, like you, my browser has about 8000 open tabs. But before the Internet, the way that you would keep up to date on stuff is voraciously reading every tech magazine. I would tear out articles that I wanted to keep. I'd tape the pages into scrapbooks and create my own IT technology murder board and everything. And maybe worst of all, I really liked what I was learning and so I wanted to share it with everybody I knew. And so I would try to engage my coworkers in conversations. Did you see how Borland is taking on the limb standard? I was really a lot of fun at parties, let me tell you. I was so desperate to share the information that the Unix admin, the one who I knew Vi, and so he took a liking to me. Well, he knew I knew Vi and he knew I learned HTML pretty quickly. So he set up a web server for me and so I could type the snippets of the magazines into web pages and share them. Now, literally nobody ever read those pages, but I learned how to create web pages really quickly. And when I was out of work a few years later, I had everything I need to start building websites for people on the side and do it faster than most people could. Now, I'll explain more about that in a minute also. But knowing HTML literally helped me put food on the table. But there's more. Typing HTML week after week got tedious, so I asked the Unix admin how to automate it. He taught me literally enough perl to be dangerous. I wrote ridiculously complicated scripts that would take the raw text input out of a text file and then smash it together into web pages and figure out where the line breaks were and add the paragraph marks and everything and figure out what the next page name was supposed to be and then update the index HTML file. I was very, very proud of my achievement. But like I said, nobody ever read those pages. So what value did any of that have? So my leap into the world of monitoring started with a tool called Tivoli. Now later on, I would include tools, everything from openview to Nagios, to SolarWinds, to new relic to Shinkan to Zabix and Grafana. But I have to admit, Tivoli was my first tool for its time. Tivoli was a comprehensive multi million dollar solution. It had modules for system inventory and software distribution and metrics collection and event correlation. Tivoli was also basically a gang of perl scripts and a trench coat and that lobbed monitoring data into a Corba backend database. Nobody with an ounce of common sense wanted anything to do with this freak show of a tool. But of course, I heard the word perl and I was all in. And just like when I was 16, learning Vi, I was too new to everything to understand how truly awful it was. It all seemed perfectly normal. But everything I learned set me up for the next 25 years of my career. Now, Tivoli was good at a lot of things. Windows was not one of them. So we had to find this weird third party tool called SMS, which you see on the screen. But it's not actually the Microsoft SMS it was. Before that, it was just SMS systems, server monitoring system. And integrating the data wasn't just hard tools at that time not only didn't easily integrate or combine themselves with other tools, they actively resisted the effort of integration. It took months to get this all to fit together. And in all honesty, it was a bad tool and we probably should have built it ourselves at the time. But we learned a lot about integration and data types and transforms. And believe it or not, I use everything that I learned back at that time, even today as I'm working with newer and more robust tools. And yes, that SMS was bought by Microsoft and turned to Microsoft SMS, which became SCCM, which then became SCOM, which then became ECM, which then became whatever the heck Microsoft is calling it this week. And again, knowing the underlying foundations of how tools are built definitely helps. Tivoli had a GUI for doing things like grouping systems together for software distribution. And naturally you could create groups, and you could create groups of groups, and then you could create a group of groups of. Actually you couldn't do that because every time I did that, the entire database would crash and corrupt and I would have to either rebuild it from scratch or restore it from a backup. And I really wanted to know why. And so I did some digging. Remember that Corba backend that I talked about. It was one of the first object oriented databases in history, but it could only handle three levels of containership, of object containership. After that, it would just die. Now, Corba wasn't my choice. It was Tivoli's choice when they. But the tool. But over the three separate weekends that I spent rebuilding the Tivoli database, I learned a lot about the impact of downstream consequences and also the importance of understanding the architecture that was being implemented and knowing the ramifications of that. Because sometimes a software tool's pedigree really does matter. Okay. The work that I did on Tivoli got me noticed at the company I was at, which was Nestle, and they offered me the chance of a lifetime. They were going to move me and my family to the world headquarters and work on a really big project, global monitoring. 250,000 systems in 5000 locations. So they packed up my family and we landed in Switzerland on September 1, 2000, and 110 days before 911. It was also six months, almost to the day before bubble burst, at which point everyone in the project was sent home in the middle of a recession. I was out of any kind of steady work for 18 months. I spent a lot of time feeling that that choice that I'd made to accept the opportunity to move to a new country and everything, I really spent time feeling that that choice had ruined everything. And over the next few months, I learned a little bit about humility. I learned a lot about myself and mental health and my value as a person and not just the source of a paycheck. I learned who my friends were and who I could rely on. And I learned that times can get very, very tight and still contain moments of joy and how important it was to recognize those moments and relish them and just be in those moments, regardless of what's happening around you. Okay. The next story actually wasn't me, but I observed it very close up and I wanted to share it with you. It's kind of poor technical choice. Voyeurism or rubber necking, as the slide says. All right. I saw two different sev one events. Events that brought the entire system crashing down. Two events, one week apart. In both cases, it was because of a configuration change that happened in production. One person was immediately walked out the door. The other person was given a stern talking to. So what was the difference? The first person made the change, saw that everything had crashed, immediately flagged it, let people know, owned up to it. They couldn't fix it. They didn't have the permissions or the skill, but they offered to be there through the entire process. They offered to bring coffee, to bring pizza, to watch, to learn, to make sure that they understood the impact of what they had done so that they would never do it again and that they could actually help when they saw other problems that were similar happen like it. The second person saw what happened and tried to bury it. There's always a log file. It's always in the logs. Now, those lesson here should be pretty obvious. Do not try to hide your mistakes. There isn't or shouldn't be the expectation of perfection, but there is, or at least there should be an expectation of decency. A few years ago I was tweaking an alert and I accidentally caught 772 tickets twice in 15 minutes. And the worst part was, I didn't even know that I had done it until the entire help desk came up to my cubicle and invited me down to their area to help them manually close the 1544 tickets one by one. Yes, there was a mass close function. They wouldn't let me use it. Now, with those benefit of time and experience, maybe some alcohol, I can reflect on this formative event and wonder whose fault was it? Was it the tool that allowed such a thing to happen? Or was it the operator? There's probably enough blame to go around, but in a larger sense it was an immature philosophy about what an alert actually is. Now, all of that is the topic of a completely different talk. But for today, the lesson is on the screen right there. That screen was created by the vendor after the event I just described, because I told them about it and said, why did you let this happen? I wasn't the first person to experience it, but when I became a Devrel advocate for that vendor a couple of years later, it was something that I could articulate very passionately and convincingly and get them to make a meaningful change until they resolved it and gave a screen that said, oh my gosh, you're about to do something bad. Maybe think about it. So that's the point, is that the experience I had fed into my ability to make meaningful change. As I said at the beginning, the sheer number of choices and possible negative outcomes can be paralyzing. It is a Whitman sampler of possible crap. In fact, you get this sort of choose your own adventure kind of list of things. For example, do you do kubernetes or do you choose not to do kubernetes? Do you implement an integration to chat jeopardy. That is how you say it. Or do you not do it? Do you use blockchain or not use blockchain or this framework or that language or this platform? Or do you put your system on AWS or Azure or GCP or all three of them? Do you use oracle for literally anything ever? No, you don't. Do you use on prem versus colo versus cloud versus a hybrid approach? Do you use monitoring with this solution or that solution, or all the solutions combined, or do you build it yourself? All of these choices hold potential success or disaster, and there's almost no way of knowing Seth Godin, one of my personal heroes, addressed those inherent fear many of us have. He called it technical fomo, the fear of missing out, the fear of not getting into or onto the next thing. And it causes us to hesitate to get involved in something now. But what he said is so very relevant. This thing, those thing we have now, is worth working with because it offers so many opportunities compared with merely waiting for the next thing. It's always a good choice to work with what you have now and not worry that you'll be too engaged to pick up the next thing, which may be better, but I know I said that my point was there were no poor technical choices. But can that be it? I mean, that seems really trite, really banal, really predictable. Maybe the lesson is that the real treasure was the cables that we collected along the way. Or maybe it's something else. One last set of lessons. I learned the word loquacious very early. Loquacious means tending to talk a great deal. I learned it in fourth grade when it appeared on my report card. Because my teacher used it to describe me, I'd end up seeing that word a lot. Also overly communicative and oversharing and diarrhea of the mouth. I also wrote a lot all through school. Now, my notes were comprehensive, but not exactly on topic. If you wanted to know whose chair made that weird farting noise at 1115 in social studies class, I was your guy. My notes may not have had the facts or details, but the 1812 war, but in their own way they were comprehensive. Also, from the time I was ten years old until I graduated from college, 100% of my time was spent pursuing a career in theater. In fact, my senior year I had two academic classes and seven classes that were choir, drama, band, and so on. All that time spent, and I'm not doing any of that today. So you could argue that all three of these things that you see were poor uses of my time. But on the other hand, I'm a developer relations advocate or evangelist or whatever you want to call it. And how do you do that job? Well, you do a lot of talking to customers. You participate by typing in online forums. You create videos like this one you present at conference talks, you are with people on podcasts and so on, along with everything you see here. Remember that my entry into technology was to provide training to learn how to relate to a group and present detailed information in an interesting and even entertaining way, and sometimes to take a technical pie in the face just to keep everyone enjoying themselves. All that writing early on served me pretty well in the blogging circuit and theater definitely helps me be comfortable in situations like this where I'm standing up in front of an audience of people and sharing ideas with you. Today there really aren't any poor technical choices, and that's because it's not about the tech. In fact, it was never about the tech. It will always be about you. I've left this slide blank on purpose to leave room for what you bring to the story. You're probably already thinking about your experiences and how they relate to the ones I've been sharing. You, or are similar or different in many different ways. We pick up technical puzzle pieces throughout our journey in our career, and often it seems like the things that we pick up have nowhere to go, but the truth is that they will all fit somewhere eventually. In fact, I felt so strongly that everyone's journey is valuable and worth talking about that I made this talk open source. If you go onto my GitHub repo, which you'll see there, you can download it and submit it as a CFP to your own conferences, share your own amazing details, and run with it. There is never going to be a part of your skills, your experience, your life that won't apply at comes point. The trick the real lesson of this talk is that we have to commit ourselves to showing up every day ready to apply both who we are and what we know. And if we do that, there really will be no such thing as a poor technical choice. As I mentioned at the start of this talk, I can't teach you much, but I might irritate you. So if you're feeling irritated, it's time to start mulling over that grain of an idea and creating a pearl of your very own. Meanwhile, online line in the chat of those conference I am ready for your questions. Thank you very much.

Leon Adato

Principal Technical Evangelist @ Kentik

Leon Adato's LinkedIn account Leon Adato's twitter account

Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways