What if you could work with some of the world's most innovative companies, all from the comfort of a remote workplace? Andela has matched thousands of technologists across the globe to their next career adventure. We're empowering new talent worldwide, from Sao Paulo to Egypt and Lagos to Warsaw. Now the future of work is yours to create. Anytime, anywhere. The world is at your fingertips. This is Andela. Hello, everyone. Thank you for joining my talk. So I'm Jerve. I'm going to talk today about successful. Go for successful, successful microservices architecture. Few years ago, when I joined my current company, the team was working on we had almost monolithic architecture and they were using a programming language called Scala. So then we decided to make the transition using go and building a proper microservices architecture software. So the aim of my talk is not so much about praising go or the microservices architecture, but it's more about talking about the journey and what we learned through that journey. So very briefly, about myself. So I'm Jerve. I'm a senior software engineer. I'm based in London. The company I work for is called smart numbers and I joiners them in 2019. So we are a software company specializing telecommunication. So we deal with things like fraud prevention, call management, telecoms migration or mobile solution and so on. So if you want more detail, you have the website there, the link. Myself, I'm part of the fraud prevention team. So I'm working on fraud prevention products. So as you can imagine, we have quite a weird and strong constraint because we're dealing with telephony, especially latency. So we'll come back to that a few times through the presentation. And if you have any question or any comments or feedback for me, you have on the right for you, I think. Yeah, you have on these right my twitter, and you also have the link to my YouTube channel where I'm trying to teach go the basics, giving you tricks, tips, teaching you design patterns and these kind of things. So that's for me. So now let's dive into the subject. So, microservices, obviously, I said a few minutes ago that I don't want to spend too much time in praising microservices, but I'd like you to understand what's a microservices architecture and also what are the caveats of using one of those. So to set the scene properly, I'm going to have to compare it with a monolith architecture. So no disrespect for a monolith architecture. So it worked very well for a few years it still work in certain eases, but now the trend is more into microservices for some reason, for agile and so on. So the cliche of a monolith, so you have a monolith on the left. So usually a monolith is like a gigantic mono service, years and years of code accumulated. It's using a single database with a horrific schema, difficult to migrate. And this kind of stuff, it's usually in a single version control repo into a single GitHub. So it means that you have to have a strong discipline in using GitHub. Like you have to use branches and you have to do rebase, you have to do commit, pull requests and all this drama. So usually it's quite difficult to extend and maintain, and it's not such a nice dev experience. So as opposed two that we have on the right, successful, successful microservices architecture, the name suggests, instead of having this mono blockers, we have multiple services communicating with each other. So usually a microservice is like an autonomous, meaningful service representing business unit. They can have these own database and they are in their own GitHub. So the advantages of that are that they are easily testable on their own. You can upgrade these on their own. So if you want to upgrade a component, a library, on one service, you can do that and it won't impact the other services. Your schema is simple because you have tinier databases and if you have let's say four colleagues in your team, and it means that they can work independently on four different services in different repo without stepping on each other toes. So you have quite of autonomous. So it usually translates into a faster release deployment or a better dev experience overall. So that's on paper, that's on theory. But obviously I'm here to talk to you about the caveats of this pattern. So I took a screenshot for you about a screenshot of our microservices. So we are using AWS, Amazon, Amazon Services. So this is x ray, which is a monitoring tracing tool. So the dots represent services and databases or any entry points. So you can see visually straight away it's complexity. So here we are introducing a lot of complexity between the services in term of communication. And we are also introducing what we call cognitive load, meaning that it's a lot to take on for one single person or for a team, as in if you have 100 services, it's not humanly possible to remember what each single service is doing or responsible for. Then secondly, you are increasing the overall latency of your those architecture. If you're not careful, it's simply because you can have a service calling another service calling over multiple services. And as I told you, I'm working for a telephony company. So if a person is trying to contact emergency services, you can't wait forever to process the call. And then lastly I mentioned that it was easily extendable, maintainable, yes and no. Because if you have a service sending a request, two another service and you want to change, you want to remove or you want to add another property on those request response. It's quite a bit of gymnastic because you don't want to introduce break and change and you might not aware if you change the response of one service you can think about these service consuming it, but then you might forget others. So it becomes really complex. And it's even more complex if you have multiple environments. Like you have a dev environment, you have a UAT environment, you have a pro environment, you want to propagate this chain smoothly without breaking anything. So it's quite a lot of scheduling. Also maintainable? Yes and no, because I said earlier, they are upgradable on their own. That's true, but let's say you are using a common library for each single of them, and that library requires a security patch. If you have 100 services, you're going to have to propagate those patch security 100 times, unfortunately. So obviously any programming language could help solving mitigating those problems or would have those problems. But the goal of this talk is how do we make the most of go to tackle them? Right, let's see how we can use go the most efficiently possible. So our first challenge was three years ago when we had call start problem with our serverless functions with Scala. So a serverless function is a programmatic function where your cloud provider takes care of hosting, managing and provisioning your infrastructure, and then you just execute your code. The problem with that is when you want to invoke your function, your cloud provider has to download your code and then start your new execution environment as shown on the diagram. So this existing time is your call start and, and then your code executes. And the duration, the length of a call start depends on three factors. So first one is the language. Unfortunately some languages take longer than others, these the package size. So it depends if your language is natively. Once compile is natively more voluminous than others, and also if you're using bigger dependencies or more dependencies. So obviously if you're using more packages or dependency, the longer is going to be your cost because you have to download those dependencies and also doesn't matter here in our case, but it also depends if it's in a private network or not. I'll put a link for you here. It's only about AWS, the Amazon services about call starts. It's a really good article. So the problem we face and what we saw, so we had awful call start with scanner. So Scala is part of the JVM language family, so it compiles to Java byte machine, Java code byte machine, sorry. And so we had call start of one to 2 seconds and then our code executing. So that was not acceptable for our constraint, our product, because you can wait forever for a call to be processed. And you can imagine in that scenario that we had a scala function calling another scalar function. So it means that you could hit a chain of call starts, which is a really awful experience for the customer. So then we makes a trial, we tried a Golemba and it was much much better. So we had a call start of millisecond and then your function execution. So that makes sense because go binaries are really lightweight, so therefore the provisioning, when your cloud provider is downloading your code it's much quicker. So then that's why the consists are much much tinier with go. So that was a first success for our microservices, especially for the latency. So we decided to make the full transition to Golang basically. Second point is Golang is quite a well supported and popular language. So it seems a bit simple and trivial to say that, but it does have an impact. So let me explain because it's quite a well known language now and it was already three years ago. So you often have sdks available. So we use aws, but it's available for Google and for Asia and other probably. And as we had to make the transition we had to write services from scratch. So for instance we had to write HTTP server, graphql server, we had to write test to generate mock mocks, we had to encrypt, to hash, we had to talk to a database, we had to services as request response. So none of us had experience in go and we had plenty of tutorials available, a ton of tutorials available, which made our life really easy. So then yeah, these ecosystem was and still is extremely alive. And we have a supported community and it's a living language, it's really well used. So it means that the services we wrote three years ago still beneficiate from updates, from improvements. It's a bit of a teaser, but we had a few months, a couple of years ago go modules for these dependency management we had generic like a few days ago, so it really does help in building high quality services. So my next points is quite a bit of a continuity of my previous point, but more in detail. So go is quite a well designed language. So here you have all the standard libraries built in when you download go, I think it's worth mentioning because all of that, as I said, is inbuilt. Like when you install go, you have all of that ready to be used and you don't have to download any external dependency or this kind of stuff. So worth mentioning, you have the crypto for hashing or to salt or the secret and this kind of stuff, you have these encoding library for JsON marshalling and marshalling. You have the amazing errors package, very simple but really effective. Have the formatting library, you have the OS library for manipulating files, you have string manipulation, you have the sync library for the existing group, the Mutex and these kind of things. And you also have the really good time library, really well built. So a bit of comparison here you have for instance JavaScript. When you want to use TAM, you have to download external library like moment or momentum or date or date fn or something like that. Also the testing framework. So the testing framework is actually a better example. So when you want to test your code with JavaScript, you have a ton of libraries, can't really remember, but you have mocha, you have Chinese Sinan, you have cypress perpetual js. It's not uniformized, it's not like inbuilt. Whereas in Go, if you want to write go and you want to test your code straight away, you have a ready two be used library dependencies management. So we started with an external tool called depth. But fortunately enough in Go 1.13 the dev introduced Go modules, or Go mod, called Go mod. So it's just fantastic tool which just work. So you can add a dependency, you can download them, you can initiate a file, it's just proper dependency management available to you when you download go. So that's fantastic. All right, some libraries worth mentioning, obviously you could use only standard libraries to build your pieces of software, like you could write from scratch a library on your own to generate graphql server or this kind of stuff. But turns out that you really have good library out there, really well maintained, really well tested, and it really helped us in our journey to build high quality services. So the first two are testing libraries. So Gomega and Ginkgo. So these are matcha assertion library and BDD test framework. And these, the third one is really important because when you're writing microservices you have a lot of common parts used by your different services. So then the solution for that is to write common library that you can share between your services. So those kind of libraries are codec serialization framework or these kind of things. Or they can be models to share between service to ensure that you are using the correct request, the correct contracts between them. So you want to do that smoothly. So you have to have a tool which can properly distribute tag version your library for that. So Gorilla releaser is a really nice binaries builder then. Lastly, I thought it was worth mentioning this GraphQl server generator. So it just works out of the box and from GraphQl schema it's generating a full server ready to deploy. So it really smooth the experience, the dev experience and really help us to deliver value quicker. A very quick special mention to Golan. So Golan is an did. It's part of the Jetbrain family. So you probably know Intellij. I know that ids are quite controversial, but if you're familiar with the Dreadbrain ids, you can go pretty quickly with Golan, with all the shortcuts and stuff. So the learning here about the tooling is because you have so go is well built and you have really good libraries available. So the tooling is really excellent. So that makes the dev experience really solid. And for our microservices it means you build faster and you build with high quality. And overall it's just really enjoyable to write go at the end of the day. All right, so my last point is quite an interesting one. It's about the new joiner experience. So for a person new to go, or maybe his or her first language or second, Golang is quite easy to learn. So you have very few keywords and the syntax is quite explicit. So just in a few weeks you can already understand and you can already contribute massively. Two, the team. So that's really good. And also even after a few years, if you still have appetite for challenges, you can always find subjects, topics to deal with, like garbage collection, or understand how the concurrency model works, all these kind of things. So in our team, just a few months after embracing the Go universe, one junior member of our team did a presentation. Two, the whole engineering department about go. That was a massive success. So at the bottom I put two links, two resources that I like, I'm time to comes go back there to just check some pieces of knowledge. So the first one is a tour of go and the second one is go by example. So Golang is really beginner friendly. So it's immediately rewarding for them. But also from the mentor side, as you can build a solid base and you can consolidate very crucial and important concepts like solid principle, clean code, and you can also build on that eases even further concepts. So for our microservices, it means that for a junior point of view, you don't spend much time on struggling, on learning a new language, but you can also focus on the overall architecture and try to understand the place of a service in the whole fleet of services. All right, so now I'd like to just contrast all those advantages with some limitations. So the very first limitation with go that we face was how to organize our folders and packages. So on the left you have these traditional organization, the old fashioned way, MVC model, view controller. So we started with that. We had a package called controller with all the controllers. Then we had a package called services. Then we had all our services and was gigantic package, not so much meaningful, and so on and so on. You had views, models, and we just agreed on a better organization, which per context. So context here does not mean go context. So context is more like a logical unit, like you have on the right, the order context, where you have everything about order and then everything about customer, and so on and so on. And we also found out that having too much level of nesting didn't work either. So what we tend to do usually is to have only one level of death, like you can see on these, right? Or at most two level of def, of nesting. Sorry. So it really help us by organizing those packaging in meaningful context to identify common packages and extract them as common library, or to do not misidentify them. Do not extract something where it's not necessary. And then the tooling did the rest, as I told you, with that go release library, the second limitation we face. So here, it's not necessarily impacting our microservices, but I thought it was worth mentioning. So the first caveat was the pointer problem. So I'm going two, take a really controversial shortcut here. But usually you use pointer if you want to mutate something or if your struct is too heavy to be passed around. So in that case you use pointer. So then we had a few problems with that. So first is like, it's a bit daunting for someone who has never seen pointer in his or her life. So we just took the decision to avoid punters like the plague, basically. So we try to do it more in a functional programming way. So let's try to avoid mutate things, and let's try to do not use pointer in our struct if it's not necessary, because we had a bit of a drama where we had a pointer, a misuse, and we broke prod once, so that was terrible, so we didn't do that again. And then about interfaces. So sometimes the concept of implicit interfaces are also called the typing. It's quite difficult for a beginner to understand. So about the future, what else could we do with go in order to have even better successful Microservices architecture? Wrote a tiny user story there. So as a Go software engineer, I want to write infrastructure in go, deployment pipeline in go, my task runner in go, and even writing go to generate my documentation. So then I don't write awful yaml. So here it's a bit of a joke, yes and no, because go is such an efficient language which brings so much high quality to the services that you'd like to have that resilience, that kind of quality level for everything around. So that would be the dream. Second point is about protobuff. So protocol buffer. So we are a bit late in the game, so we don't have that implemented at the minute in our architecture. So protobuff is about serialization of structured data. As you can see, you can do it in go. So here it would help, we believe it would help to propagate any changes between our services, because it can do validate schema validation and these kind of fancy things. And it's also very lightweight and very efficient in term of latency. So that's something on our to do list. Another things on our to do list is the generics. So just a few days ago go 1.18 did release these generics, finally. So it's ready to be used. So who knows what we're going to found. So in conclusion, we saw that successful, successful microservices architecture, really good pattern, but unfortunately you have to be careful about two points. So the first one was these complexity between your services, and the second one was the latency that you could introduce in your services. So then we also saw that go was really helping to ease those pain points. So first one was excellent performances, the library binaries, the language speed, and then you have the tooling available because it's quite a popular language, you have a lot of good tooling available, a lot of tutorials and so on. So overall providing a really good developer experience. And then who knows, maybe there will be even more go features in these future to even make the experience better. So thank you for watching. Hi,