Conf42 Golang 2021 - Online

Go learning by project: Souin (rex)

Video size:

Abstract

I’m Træfik user since v1.4 but there was no caching system. I scrolled over the internet to know if any solution exists but nothing appear then I decided to write my own Træfik cache system. This talk it will be a feedback on Golang learning, issues, tips and tricks to start on this awesome language

I discovered Go language but didn’t have any time to follow multiples tutorials to learn it. But one day I discovered Træfik reverse-proxy project when I wanted to switch my infrastructure into fully dockerized one. I’m Træfik user since v1.4 but after many months using it I encountered an issue : there were no caching system in this reverse-proxy. I scrolled over the internet to know if any solution exists but nothing appears.

Then I decided to write my own Træfik cache system, but the main question was “Which language?”
- PHP ? Nah.
- Nodejs ? What a joke !
- C++ ? I didn’t learn this language at school and it’s really insane to learn.

Then I was on Træfik github repository when I decided to write it in Go. Another good point: that’s compatible with docker integration.

So I started the project and called it Souin
This talk will be a feedback on Go learning, issues, tips and tricks to start on this awesome language.

Summary

  • Steve Combrak is creator of a cache system called Swan Reader in Go. He is also traffic ambassador and active open source contributor. Today we'll talk about corning and specify in my case the project called Swap.
  • Go started on 2003 at Google but open sourced on 2009 by Francis McCabe and case Clark. The advantages of Go is you have one binary file as output. It's really easy to learn. But Go is not really for Ops. And you can do some web app using Go.
  • Golang is normalized through some variable environment variables like the Gopath. You have bin folder, PKG folder and SRC one. You can combine it with the goarch to tell Go to build compile for a specific arch with a specific OS.
  • Learn to develop in Go with our live demo. Let's see together how to do our first Edward. Then we will deep dive into the language, complexity and concepts in go.
  • A channel is like a pipe between your main function and your goutins. You can push and pull data inside it. To block the main thread we can do that.
  • The next step would be the testing. While discovering Golang, I discovered the unit testing and go is really well, it embed the test inside the language itself. Just run using an ide, because I'm a nerd, not really a nerd.

Transcript

This transcript was autogenerated. To make changes, submit a PR.
Hello everybody. Today we'll talk about corning and specify in my case the corarning by project, the project called Swap. You have QR code to be able to retrieve my slides on slides.com. Before dive into the subject, let me introduce myself and make my self promotion to you. My name is Steve Combrak. I'm from really in France. Sorry for my get accent but I try my best to speak as well as possible. You can find me on GitHub searching Darkwick and Darkwick underscore dev or Twitter. I'm creator of a cache system called Swan Reader in Go, Im Trfik Helper, traffic ambassador and active open source contributor and also and freelance too. So after that we can dive into the beautiful world of Go and see together. What are the issues in Go? And I try to teach you what are the main conflicts with Go. We will talk about Golang story because Go is very young language. It started on 2003 at Google but open sourced on 2009 by Francis McCabe and case Clark. Google tried his own language to answer some performances issues. It's written in c, but you don't have to know c language to code in Golang. The advantages of Go is you have one binary file as output. You can build cross platform for macOS, Windows and Linux for example. It's really easy to learn. You can do multi threading and concurrency really easily and it's really easy to learn it because it's not a tricky language. Here are some software using Go. The first one is Docker, it's a famous one. It's the soft to manage containers for those who doesn't know. And the second one is Kubernetes for the open source name and Borg for Google instance. It's an orchestrator to manage containers. And you have other software like traffic or Mercury. But Go is not really for Ops. And you can do some web app using Go. For example Hugo is static website generator from Markdown and Yaml configuration file. You can do fail to ban like using bad actor and you have a reverse proxy called Gadi to serve PhP app or any web app instead of using Nginx, Apache or any other reverse proxies. So what is my history with coding? Let me replace the context. I'm traffic contributor and traffic user since one four. I'm also helper and traffic ambassador. So any other names to describe this? And for those who doesn't know traffic solution yet, it's a reverse proxy written in go to help you to manage container architectures. And it's a container first approach. It's designed to manage containers, it supports the odd detection deployment, the scaling, it load balances your request automatically. Only a specific port for all protocols, then traffic manage for you the service resolution depending Austin entry, then it's a pivot point of your infrastructure. And after months to use this solution, there was an issue I was facing too. There wasn't any cache system internally to traffic. Then all requests would have to resolve the associated service. Then it took many milliseconds to be done and I searched after cache system solution but nothing appeared to be as efficient as I wanted or it was only poc. So after many nights spent to look after that I decided to create my first program in go. Because traffic is regional in go two and it souin good. So this project called Swan is actually in version one five two and for the record the release names are free beer names because I love to drink beers. So we witched over the 100 stars and don't forget to leave one because I've got an oversized ego and I like to view the Stargazers count growing. So the first thing in go I have to teach you is that Golang is normalized through some variable environment variables like the Gopath. The Gopath will tell to go, where to find binaries, where to find your sources, your dependencies and the normalized architectures. So under your Gopath it's normalized architectures. You have bin folder, PKG folder and SRC one. And I made a table to resume what is the goal of which folder? So sources contain the readable development sources with your dependencies. The PKG contains copackage objects compiled from SRC directory and your go source code packages, and your bin folder contains fully compiled go software based on PKG objects. Then after the go path you have your go route which is the Golang SDK path. It tells to go which version it has to use and you have one of the most useful environment variable, the goos. It tells to go which kind of OS it has to build for. By default it will use the OST OS. If you are on macOS it will use Darwin, on Linux it will use Linux and Windows it will use Windows. So go OS isn't really important as standalone environment variable, but you can combine it with the goarch to tell Go to build compile for a specific arch with a specific OS. With that you can do cross platform build and build for Windows even if you are on macOS and Im 64 with an rim laptop. So you can do some matrix to build for any os with any possible architectures, so it will generate over hundreds binaries, but you'll cover every system with that. So now we can go to the next section which is the live demo. You have QR code and the plane URL to clone the repository and practice with us during this session and learn to develop in Go. The first thing to do with Go is the editor. So let's see together how to do our first Edward. Then we will deep dive into the language, complexity and concepts in go. So let's begin together. I just cloned the repository. We have main go file to tell Go. It is our entry point. We can run only main packages with a main function with Go. So if I try to run my main package, I can just type go run main go and it will run my go file. So if I do that, I'm absolutely doing nothing in my script. So let's type simple hello world in the console. So we have the FMT package which expose some methods to be able to debug our code or print something in the console. So we will print with a new line our hello world. It's a little bit laggy because I'm recording, so sorry for the inconvenience. Then if I save and I run main go, it prints hello world in the console. Now we have our first hello world working. We have hello world in the console, but the softwares we build with go are not easy like a simple hello world, and because we want to make some calculations or any other operations. So let's see together an advanced mode of go. It's go routines. The Go routines are basically a part of a logical thread. Then go sits on it to be as efficient as it is, because the wall language is built around the go routines. So let me explain more precisely, what is it? It's asynchronous, detached from the main thread. Then if you kill your main thread, the go routine won't be able to talk with your terminal or your instance, the instance it was called by. So we have to take care about keeping the main thread running to be able to get back the data and be sure the souin is finished once the main thread will be closed. So let's see together how it works. Okay, so now I commented my hello world print and we can go under grutins folder and create main go file. After thats we create a function called go routines. It will return absolutely nothing. And we can do a simple loop for an iterator initialize. Thats zero. We can tell if it under ten with an incrementation by one just print something and it will be sprintf. Then from the main thread, the main method iteration like that and just pass our iteration and we can tell that we have another function. This function will be called for example my detached function like thats and just copy this. Or we can tell we will give it an iterator like thats and that's it. So to call it we can tell my detached function, pass an iterator and we can share our iterator here between the detached function and the main function. But if I do that, it will be a synchronous was. So the first will be the detached function. Then after he finished it will run this part of code, but we want to run them was a detached mode and concurrency. So go run main go we have the main obviously it, then we have our detached iterations, then after that the main iteration. So to run it as concurrent c just add the go keyword before our function. Then it will create a new logical thread and just run it inside it. Just run like that. So the main method iterate ten times, but the detached one at the time to iterates only four times because the go instantiation will take some microseconds. But during this time as it in synchronous, our for loop will be running before the detached will start. So to be able to wait this, we could add a simple time sleep, then tell okay, you will sleep for 1 second and I have my iterations, then it sleep for 1 second and close the main thread. So that's why we have our detach method here. So that's not a really clean way to do that. If you want to wait for go routines, we have something from the sync package called weight groups. And what is a weight group? It's a pool of go routine and it will tell. Okay, while all go routines didn't finish, I will just wait for it. To achieve that we have to define a variable called wg, for example for weight group from the sync package and weight group after thats I can just tell, okay, my weight group will add a new asynchronous function and we can just pass our weight group to our detached function. I will group my declarations and here we want to pass our weight group. So it will be pointer because we don't have to copy our weight group pool. Then we could do just wg done. But it's cleaner to tell before reach the end of the function. So you can use the keyword defer to delay the execution of what we placed after at the really end of the function. So we will delay the done, and now we can pass our reference like that. And now let's printf print ln. And that's it. But if I do that, it will run and okay, it won't wait for our detached function end. And to do that we can do just wg wait and just run like that. So we have the first iteration, the second, and even if I add some sleeping here, for example, millisecond, and I will do the same thing here. And here I will sleep for half second. And now we have our detached between main method and everything is running. And while we don't reach the end of the function, it won't close the go routine, so it will wait for it. So now we've seen the easy was for managing go routines. But we have another concept in go called channels. So what is a channel? A channel is like a pipe between your main function and your goutins. You can push and pull data inside it. You're able to tell okay, I'm at this step, and if I send you that, you can do this, or okay, if I do that, you can close this or like that. So let's see together how it works and what are the main issues. So, now we can delete our go routines. I will copy everything except the package name, thats. And I can remove this. I will go inside channel folder, main go and just pass this. We can remove references to weight group, like that. Like that. And now I would have to use the channel. So let's instantiate a channel, just temporarily channel with make and it's a channel. And my channel will contain only Internet. We can pass absolutely anything, the interface or your type or anything else thats just for the behavior. And tell to go, okay, I will send you that under it, like that. So in my champ channel I can push data. So if I try to tell I want to pursue an integer, then if I try to print this, I can do it like that. Then it will wait for an incoming value. So we send it here and it will print this, and it will print if the channel is closed or not. And to block the main thread we can do that. So while I don't receive value anymore after this, and after this reading, I will block the main thread. So we have to edit our main function. So that's not growoutine, but channel like that. So if I try to run go, run main go, it will create a deadlock, since all go routines are not up anymore. But to patch this, I have to use my channel here it will be an integer channel and I can defer thats one will be considered was true. So I will create a simple function and I send data in it and I have to pass my temporarized channel. I do that I would be able to read data in it but it doesn't. But if I try to read data like that then I get my one. But here I don't get any data and all go functions are stopped. So we could just move this and replace the plane temporal channel. And to read data in it can get the value and the open or close and just debug these values. And I got true because it's open and one because I send the value one in it. But if I tell okay I will close my channel here. I don't want to get any data anymore to be pushed in it. Im able to do that like this and I consider thats this will be longer than the temporized channel. When doing that I got zero because I've got nothing and the default value is zero for an integer and false because it's not open anymore. I think we cover the main part of the channels now I think we can go further and go the next step and the next tips would be the testing because I came from an environment that we didn't test really, we didn't test a lot. So while I'm discovering Golang, I discovered the unit testing and go is really well, it embed the test inside the language itself. So we could just comment this. And I don't have folder containing test, just create it testing or just test main go. So we will have some functions like two is two, so it will return a boolean and here we take an integer and tell see if n is equal to two. And obviously I have to return this for example Frank, which will return an integer and return two plus two. And if I want to go further I can tell fizz buzz like so it will take an Internet and return a string and if n equal to zero I will return this n module. Here I return unbuzz and the default will return buzz for example. So to test this I have to export my function. We can create a go file, your go file underscore test. And if we want to create a function to be tested, generally it's an exported function and it will ensure that for example test frank, two plus two it will use testing package and in that we can have some asserts. But that's really low level and you don't have any helpers. So my function two plus two will return four. So if Frank two plus two is different than four, we will throw an error, should return four. Then if I do that, it will pass. So just run using an ide, because I'm a nerd, not really a nerd. So it will compile and I'm a noob. That's the good word. Then it's okay, my tests are green. And if I enable this and I say okay, two plus two for me, should return three, then it will compile, then the test fail and it should return four. That's it. But if you're absolutely sure, two plus two should return three, so you can tell to your code, okay, I just return three and it's okay. So now we've done this test, what are the other functions? Two is two like that. So it will implement the testing package and just tell if hang to these two, three. So if it returned true here, it shouldn't, then just show three is not two. And I will disable this. And f for the two, it will return false, two should be. So that's really useless here, but that's for testing. So it will test any cases like it's unit testing. And now we will test the fizz buzz. And now we want to run this, what I've done in Fisburg. So if I try to return something like if Fizzburz from zero, different fizz, I think if it's different than fizz, then I can display a narrow message. For zero, it should return fizz. And if fizz buzz, one return buzz, no buzz. Yep. And for two, it should return buzz. And for this, it should return buzz. And that's it. So if I run it, we can see everything, every test path, and we can run tests like go test on each packages and it works. But I prefer on my ide because it's visual. And to explain, it's simpler. So if you want to go further, you have a more explicit example in the solution. You will run, what is the word, a web server, you will send JSon response and any other things using what we've seen together, so little useful tools. For example, for testing, you have gotest got tool for tooling and for linting and best practices you have golint. It will tell you you're so bad, that's not the right way to do that. And the same thing for govet, but govet will under any bad implementations or not just the syntax, for example. So here are some examples using that, some useful commands, go import, go FMT, et cetera. But I think you know already these commands and some useful links. And thanks to Avelino for making this repository exist. It will aggregate the data related to code language and if you want to read something you'll find obviously on this repository. So you have code wars and coding challenge to change you on catas. So that's little exercise. It takes around five or ten minutes to be done and you can see the other responses when you find something working and you can see better comprehensions or faster script. And I'm working on other side projects so im working on Smurf. That's an open source pen test oriented platform written in angular for the front end, PHP for the back. That's a good project to help you to manage your pen test missions or just pentest. I created Swan WordPress repository to be able to manage Swan through WordPress admin and there is the same thing for Prestashop created by Luke. So take a look at this. That's good project especially for smash and I think I'm done here. Nah thank you everybody for your attention and im available to discuss about some things. So like Gordon Ramsay is saying thank you so much and thank you to comfort to organizers team.
...

Sylvain Combraque

Software Developer @ Les-Tilleuls.coop

Sylvain Combraque's LinkedIn account Sylvain Combraque's twitter account



Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways