Conf42 Cloud Native 2024 - Online

Just don’t do it – Databases in Kubernetes

Video size:

Abstract

You are not really crazy, are you? Running your beloved database in Kubernetes? This infrastructure is built to break, and my database should not break! What actually is this Cloud Native nonsense? The latest and greatest marketing gig, let’s get out our buzzword-bingo cards!

Summary

  • We'll just stick with postgres in Kubernetes in the next 30 minutes or so. I want to take you through some of the considerations that we have when we look at postgres. New ways of developing applications and incorporating data processing.
  • Jan Karremans: Let's look at postgres and cloud. He says postgres is the most transformative technology since Linux. Running data management solutions like postgres in Kubernetes is really the way that in the next decade or so, he says.
  • How does agile, how does microservices help you deliver more value for your company? And when it comes to cloud, cloud native and that mindset part of it. These are really four elements that have matured over time and are now part of the way we look at operations.
  • Dora is for me at least a report, research and capability that Google Drives that really focuses on the KPI part of things. How do you start measuring your success as a cloud native inspired organization?
  • What are cloud made of? Well, mostly Linux servers these days. Looking at deconstructing the application piece by piece, defining very nice specific parts of the application specific function. These kind of developments is what cloud enables us to do.
  • The cloud native PG project was created, now part of CNCF. It's run postgres, the Kubernetes way. How can we start bringing data management in this cloud native mindset? Become part of the project again, a community governed open source project.
  • Kubernetes and postgres together gives you two very strong tools that allow you to take your journey exactly there where you need it to go. We would love to hear, come back and get in touch with you and hear your thoughts.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello, everybody, and welcome to the session. Just don't do it. Postgres in Kubernetes, or I should perhaps should say postgresql in Kubernetes, which is a very hard thing to pronounce. So we'll just stick with postgres in Kubernetes in the next 30 minutes or so. I want to take you through some of the considerations that we have when we look at postgres, if we look at new, new ways of developing applications and actually incorporating data processing in the world of postgres and into the world of data processing. So without much further ado, let's just jump into our first slide. So basically what we're going to do is look at postgres and cloud. And there is a lot to say about this. There is very little time because we only have like a very short slot here. But before we dive in, just let me quickly introduce myself. Who is this guy sitting here in this presentation thinking that he could tell you something about this? Well, my name is Jan Karremans. I'm a techie in sales, which is a little bit strange, but I started out in database technology many years ago, first in the world of proprietary database technology, but at some point decided there should be more in the world. So I migrated myself from Oracle to Postgres, currently a member of the international database community. And just over three years ago, I guess I got really involved in Kubernetes and really trying to understand how this technology helps transform the way we build applications. And that kind of put me on a mission. I worked very closely together with the team that did a lot of work on how do you put postgres in a container? How do you put the elephant in a box, if you like? So my background again, I said I'm proprietary databases Oracle in my case, I am KCA trained and I have a little bit of a postgres background. Meanwhile, currently at this company called Cybertech, a pure play postgres company that helps guide organizations in their journey in adopting postgres. And really looking at what are some of the challenges that you face if you move from a traditional world to this new world and how can you get the most benefit out of it. I believe that running data management solutions like postgres in Kubernetes is really the way that in the next decade or so, looking at some of the things that we're going to talk about from an application development perspective, from an application architecture perspective, really are things that are going to contribute. Good friend of mine, Mark Lindster, currently postgres fellow described this as postgres is the most transformative technology since Linux. I think he's right. If you look at the first wave of open source as I've once defined it, bringing open source solutions to the world really started with a little bit and I don't mean this in a demeaning way, but with a little bit simpler to replace technology. So you've got tooling, you've got perhaps even something which is a little bit more complex like an operating system, but they're less intrinsically interwoven into an organization as perhaps database technology. So this first wave of open source really focused on the simpler quote unquote technologies replacing those from proprietary solutions to open source solutions. I think operating systems are a very good example of that. In the early days we had proprietary operating systems like VMS, like Solaris, like Unix, et cetera. Those have all been replaced with this open source equivalent called Linux. Most every system is running Linux. This change, this transformation is now also coming to the data processing world where really truly community governed open source solutions like postgres is built. Also like Kubernetes is are taking over this high end of the IT tool spectrum. So postgres in that sense the most transformative technology since Linux. And this is one of my personal most favorite slides. It's super hectic, it's super busy, but it discusses the origins of relational database theory. And even though relational database theory is the foundation of postgres with all of the additions, with all of the extras that has been built in postgres over the years, Postgres really is a multilingual data processing platform. It speaks relational technology, it speaks document database or JSON based approaches. So that is really where this is going today. And if you look at the chart, it started with System R, the great debate 1974. For those of you who are interested in history, had a bit of a discussion around which kind of database would win relational database theory at that point. One. So that technology split it up in oracle on the one side and ingress and later post ingress or postgres project on the other hand. And that evolved through the years to where we are today, where you can clearly see postgres from a data processing platform is winning. And this is why I think also in this setting, cloud native setting, having a good understanding of how postgres plays here is of a vital importance from that perspective. If you then start looking, okay, so we had postgres or we have postgres. I should say this has been around for over 30 years. It's part of the traditional way in which we do monolithic application development. Then since 1015 years, the common arising of cloud, the maturing of cloud. And then based on top of that, you have where cloud native comes in, cloud native really as a mindset for how do I build applications that in this world of cloud actually use the power that this agile and quick evolving technology brings? It requires a different way of building applications. Even though postgres is 30 years old, it feels like it was created to go through these motions of cloud. And cloud native, I think that's a good thing to have in the back of your head as you start looking at big monolithic applications, which is nothing bad to the way that these applications are built. But if you look at today's requirements in organizations where new features, we need to have the application do new things, it needs to do something different, really is a big driver for changing applications. If you do this with your traditional big monolithic application, it's really hard to make that swift shift. I mean, traditionally you would say, I'm going to do requirement engineering, then I'm going to do a technical design, I'm going to do an architectural design, I'm going to build my pre code, I'm going to actually code the application. And by the way, by the time you had your first viable thing that you could test, you would be two years ahead. In those two years, so many of these requirements would have changed that it's virtually impossible to keep up with the evolving of your business these days. So from that traditional model we're shifting to, and I like to use this equivalent with this cloud of cards in the sky where every bird reacts to its neighbor, right? If that swift shifts a little bit, then all of the birds around it will shift also a little bit and you get these gracious patterns in the sky. I think that comparison, for me at least, looking at how requirements influence this, I think this is also how applications, and if you start looking at microservices applications, where the individual parts of the application change and evolve, much like the position of the bird in the cloud of cards, is basically a model that gives you the ability to swiftly interact on changing requirements in your business as postgres, as a data processing engine fits in this model, or if you would be able to do that, that would add tremendous value. We looked at cloud native in one of the previous slides. This basically boils down to how do you adopt the cloud native mindset, which basically discusses how developers and code reviewers, et cetera, work on a repository, really make that into something that evolves around a development branch, automated testing. So there is a lot of these things defined on how you build an application. What are some of the KPIs around application building an application evolvement and development, et cetera. And how do you deliver this application towards end users? Are your continuous integration? How do you do continuous delivery? When I started to look into this, one of the most profound things that I found is that continuous integration, I thought was the hardest part, right, where everything you do always have to be, sorry. Continuous delivery was the hardest part because everything you needed to do, you need to be able to automatically make that available to your end users, and that needs to go fluently. But as you start diving deeper in, the continuous integration part actually is the harder part because that defines that your application always has to be in a state in which you can actually deploy it. So that puts extreme requirements on how do I implement changes, how do I test changes, how do I continuously ensure that the application that I have sitting in my repository is in a state that I can put it out there in production. So this model then starts defining and focusing on the performance indicators actually required to start doing this, this journey for me, because originally I'm a database person looking at this and bridging the gap from my world where I think about data persistence and how do I record data and et cetera, et cetera, and shifting that mind towards, okay, so I am now also part of this development chain. I need to start thinking about how do I actually work through the changes? How do I look at continuous testing, how do I look at test data management, et cetera. So that was really a learning experience. And that kind of brings us back to how do you go from a monolithic application to agile delivery? Cloud in that sense, changes much more than just looking at. We all know, and again, no disrespect, but we all know the manager that comes in, right? And I think, Dilbert, for those of you who remember that comics, trip also had like a phase in that manager comes in tomorrow, we will be cloud native or we will be agile and just like flipping a switch. Unfortunately, it doesn't really work like that in reality. So going from a traditional monolithic application, from this big steam train that is moving forward to this flock of birds, requires many changes in the way you look at applications, how you approach application development, et cetera. So this is not something you do just on the side overnight, right? So getting into this, thinking through this, evolving, this is really a next challenge. Application and we saw that in the previous slide already. The buzzwordbingo, I think in it we're really good in buzzwordbingo stuff. Everybody today is talking about AI, AI, AI. The big question is AI matures, which will happen in, I presume, just a couple of years. What will it actually change in everyday life? What will it actually change in your day to day business? The reason for this is because we've seen this a couple of times, right? Big data, cloud would change everything. Yes, cloud has changed everything, but only now, and this is 1015 years after cloud started. We're now really starting to understand how cloud influence day to day it activities. So also from that perspective, keep in mind that a lot of this stuff is buzz, it's marketing buzz, it's sales buzz, where people try to get you to focus on their solutions, on the things that they're bringing. But one of the things is please step back, look at your own requirements. And how do these buzzword things, how does agile, how does microservices help you deliver more value for your company? And when it comes to cloud, cloud native and that mindset part of it, I think we're getting to a point where this has matured so far that we can say, yes, we can start out of the buzzwordbingo part and actually look at what do these things define. And I just put four on the slide here. So agile development, it focuses on deploying, redeploying of applications, keeping your mind in the back of your mind, the KPIs that you want to fulfill in how fast can I deploy an application? When we were looking at a tanker or at a train, a change could easily take six months to a year. Today we're looking at bringing out an application every other hour instead of every other year. Right? So that's one. The microservices part really has to do with going from this big block to the individual nuclei in that cloud. How can we break down an application? How can we find the essential granules? If you like to start working with this flexible mechanism, this grid, if you like that, your application is. So that's another element which sounds very much like marketing, but actually really has a good meaning. The DevOps part as well. I think DevOps over the past couple of years has really become really settled and really well defined and ingrained in every other organization. You don't just install a single server, but you write an ansible playbook or you write some script that actually does this work for you in an omnipotent, repeatable way. Every time you run the script, the outcome is exactly the same. And if you want to change something, you don't fix something on the server, but you just change the script and let the script do it for you. As we're going to this new world, we're just going to throw away the container and spin up a new one. And then what I talked about just now is CI CD. How do you continuously guarantee that your application is in a state that you can deploy it, but also that you can ploy it by the push of a button? So these are really four elements that have matured over time and really are now in a position where they're part of the way we look at it operations. So that's kind of interesting. With that, I want to briefly introduce Dora. Dora is for me at least a report, research and capability that Google Drives that really focuses on the KPI part of things. How do you start measuring your success as a cloud native inspired organization? How do you perform? How quickly do you roll out applications? What's the mean time between changes? How fast can you restore a specific service back to operation? If you start working through this, what's your deployment frequency? What I said earlier, going from an application which has a new version every year or every six months, to an application that has a new version every 2 hours and successfully deploy that and really have reliable numbers around, how do I do this? So sre site reliability engineer really came out of this motion where understanding how we build and how we deploy applications really is part of how we work in it these days. Again, I think all very interesting background, both from a cloud native perspective as well as from a data management perspective where we were looking to bring this together. And that kind of brings us back to this a little bit. Daddy, what are cloud made of? Well, mostly Linux servers these days having this understanding and really taking a step back from, again, the buzzwordbingo and all of the spin that's been giving, and I can't say this often enough, just sit down and think about what does my organization require that I can start building that actually add value? And I've been working with a specific organization that has a big monolithic, old fashioned application and what they're actually doing is looking at deconstructing the application piece by piece, defining very nice specific parts of the application specific function that that application is doing. You might even think, okay, so this is the part of the application where a truck driver comes to a factory and they need to go from the entrance to a specific loading bay. How do you make this go fluent. How do you give the truck driver an app in which he just could say, I need to go to loading bay three, dont 15. That is, I can actually have a nice chart on my mobile that says go left, go right, and you'll end up actually at that specific loading base, specifically at the time where you need to be. So these kind of developments is what cloud actually enables us to do because of the interaction and because of the open access that these technologies and this development brings. And that brings us a bit back to the flock of cards, right? It's almost like a neural network where all of these microservices, and specifically when then you look at postgres and the things that postgres as a platform can bring, and the way that data can be replicated between some of these services makes for something where not just the individual granules, but also the link between the granules and where every element knows of another element, just exactly what it needs to know can be built in such a way that it starts working like a neural network. I think this is a very nice future perspective for what things that we could start doing when we bring postgres and cloud native and the cloud native mindset even further together. Super busy slide. And please pause the video if you want to read this, but this really describes a little bit what the cloud native mindset is about. So it's really a definition slide that I just wanted to include for you guys and girls to look over as we start wrapping this up. You're probably familiar with Kubernetes, Kubernetes as a community governed open source project. We're going to bring this together with the Postgres project, one of the most powerful, one of the most widespread community governed open source projects resulting from the work of Michael J. Michael Stonebreaker at the University of Berkeley, and after him, Mr. Bruce Mumgen, the founding father of the Postgres global Development group. So we have two very powerful communities, and we're actually bringing them together in this new environment where we bring these two elements to bear for what we've just talked about. How can we start bringing data management in this cloud native mindset? So to that extent, the cloud native PG project was created, now part of CNCF. And thank you to Gabriele Bartolini for building and driving this project. And basically what it is, it's run postgres, the Kubernetes way. So to Kubernetes, Postgres is an application that has its specific requirements, that has some specific elements to it, running persistent data loads in Kubernetes has often been viewed as that's challenging, that's perhaps very awkward to do. But at this point in time, we're at a point where you can run postgres the Kubernetes way. So rather than bringing Kubernetes to postgres, what we did is take postgres and actually bring it to Kubernetes and make it build and run the way that it should run in both the cloud native mindset as defined by the CNCF, what we saw in the previous slide, but also take the goodness and benefits of postgres and actually exploit them as best as we can. So if you want to be part, please scan the QR code or just look up cloudnative PG, IO and look at this project. Become part of the project again, a community governed open source project. We recently hit 3000 stars on GitHub. So from a postgres operator perspective, this is really the way forward. It's one of the most powerful operators at this point. And what it basically does is it captures a database administrator, puts it in a box, right? So day one, day two operations. How do you bring all of the requirements for running a data processing system to Kubernetes? That means high availability, switch over, load balancing. All of those elements are covered by cloud native PG, in this case by the operator. The operator basically teaches Kubernetes postgres and for you as an end user, it's basically just deploy and use. So from that perspective, it's a very interesting and complete product and process that brings me a little bit to the last slide and I think this picture is a very nice illustration of that thought. We're here at the date this presentation is given. We're at this specific point in our journey. This is going to go somewhere from this place. And the beauty of this is as these are community governed open source projects, I said it before, you and I together define what the rest of our journey will be like. Where will we take this technology from here? What will be the existing requirements? What will be some new requirements? Or what will be things that will be asked of us tomorrow, next week, next month, next year? Trust in that being and adopting Kubernetes and postgres together gives you two very strong tools that allow you to take your journey exactly there where you need it to go. And I think that's fairly unique and unparalleled in this environment. So that's basically the message that I wanted to share with you. What we do is we build our applications that we build, and we try to help you, basically, for the things that you need. So we would love to hear, come back and get in touch with you and hear your thoughts to what we've talked about here today, but also what some of the other elements are. So for that, thank you very much for being part of this presentation and until soon.
...

Jan Karremans

Sales Leader @ CYBERTEC

Jan Karremans's LinkedIn account Jan Karremans's twitter account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways