Conf42 Golang 2023 - Online

goyek - using Go for automation

Video size:


Are you tired of using Bash and Makefiles for automation? How about using Go instead? With Go it should be easier to create a cross-platform, portable, maintainable, and reusable automation. Use goyek library for creating build automation in an Go idiomatic way.


  • During the presentation, I will talk about using Go for automation. Goyek is used to create build automation in Go as opposed to many other tools. During development, you format code, run unit tests, generate code, deploy, et cetera. My presentation will be interesting even for those not interested in yet another automation tool.
  • Make file is the facto standard for automating builds in go. I wanted to explore the possibility of creating build pipelines in Go. Mage is a makerake like tool using go. You write plain old goat functions and Mage automatically uses them as make file write runnable targets.
  • Goyek is a tool that lets you build applications using Go. It can be used to create reusable, customizable, build pipelines. It logging the execution and logging and using error to report a problem. Easy to debug and develop like regular boring go code.


This transcript was autogenerated. To make changes, submit a PR.
During the presentation, I will talk about using Go for automation and ad advertise Goyek. Goyek is used to create build automation in Go as opposed to many other tools. Goyek is just a go library with API inspired by testing Cobra flag HTTP packages. However, I really think that my presentation will be interesting even for those not interested in yet another automation tool. Here is the agenda. First, I will tell a few words about automation. They will demonstrate free tools that can be used for automating. In the end, I'm going to share with you my subjective conclusions. During development, you format code, run unit tests, generate code, deploy, et cetera. You want to have these things automated for sake of continuous integration, continuous deployment, basically automating your daily routine stuff for the sake of this presentation so that I can demonstrate the automation tools. By example, we will have a simple build workflow for a Go repository. We'll have an all target which will basically contain the whole build workflow as it will depend on FMT and test targets. Then the FMT target will simply run go FMT for all the packages and this target will be imported as it can be used by multiple repositories. Test target will run unit tests and generate code coverage even if any test fails. All of the code is hosted on a GitHub repository, so you can look at it at any time you want. We will start our journey by looking at make file is the facto standard for automating builds in go. Let's look at the simple build pipeline created in make. Let's start by running make which executes the default make target the default make target prints the usage. Let's run the old target. As we can see, it executes the dependencies. It run FMT and then test which had failed. We see that the go test had failed, but still the HTML code coverage report has been created here. Let's look at the make file which contains the definition of the make targets. The first line is a declaration that the common make file should be included. We'll look into that a little bit later. We have the all target which is the build pipeline which depends on the FMT and test targets. Here's the definition of the test target. It's a script, a bash script, which first declares a variable exit which is equal to zero in the beginning. Then it runs go test with code coverage, and if it fails, it assigns to the exit its exit code so that the HTML coverage report can be generate later and finally exits with the code assigned here or with zero if the go test had passed. Now let's look at the command at the go mk file. We are setting that the default shell is bash, so it is more probable that it will work on other creating systems. For example, macOS zsh is often the default shell. We are setting the default goal which is the default target as help. And help is a magic target which basically prints the usage based on the comments which are added to the targets. The comments needs to be defined as the double bash characters. We have a reusable print target function which says which function is executed and then we are having some reusable targets. And here we have the definition of the FMT target which executes go FMT. What are the challenges when using make? Writing complex logic is hard for anything more complex. I always need to look at docs, I need to look up at some recipes, look at some make targets which I already was using, or I need to use stack overflow or basically Google some stuff. Also, I do not like the bugging bash nor make which is often like print into the console. At last. It's not easy to create automation in bash at make that works on Linux, macOS and Windows. When I was starting to learn Go, I heard that some sres prefer writing go apps instead of bash scripts for compatibility, maintainability and reusability reasons. Most Go programs using the standard library work perfectly on all supported operating systems. Moreover, I prefer debugging go programs rather than bash scripts or make targets in other language. It is quite popular to use its own ecosystem for automation. As far as I know, everything started with rabbi's rake, but now Java has gradle, net has nuke, etc. Therefore, I wanted to explore the possibility of creating build pipelines in Go. The first thing which I found was mage. Mage is a makerake like tool using go. You write plain old goat functions and mage is automatically using them as make file write runnable targets. It's time to look at mage. Let's run it by executing go run mage go as previously for make we see the usage. Let's run the old target. As previously, we can see that the test has been executed, have been executed, and we have the HTML report generated. However, we have no information what test had failed and we have no information what target had failed. Therefore we need to run mage inverbose mode. Now the output is similar to make file to make let's look at mage go which we are running using go run. It's just a wrapper script on mage main which executes mage. There are two ways of using or executing mage. One is using this wrapper mage go file. The benefit of it is that thanks to the Go modules, we'll make sure that all developers are using the same version of mage. The other way is to install the mage CLA app. Mage works magically by finding the targets defined in mage files which are required to use to have the mage go build tags. So these two lines are required. Then we have regular go code. We have no regular imports, and here we have the magical mage import which basically includes all the mage targets defined in demo common package which we will look at later. Here is the definition of the all target. The targets are defined as an export as exported functions. In Mage we have an information that we are depending on the FMT target from the command package and on test target from this package. This is the definition of the test target. It returns an error because it can fail. We are using the sh package provided by mage here and it executes go test and go tool cover to run tests and execute test coverage. We are using errors joined from the Gostanta library to make sure to return an error if any of those two execution failed. Here is the definition of the common build targets. This is a simple definition of FMT target. Let me share with you the issues that I have with mage. For me, mage is too magical. The target discovery requires the magical build tags and because of that you often lose the intellisense in your id like vs code or goend. It's also recording the magical import commands which you need to add before importing the packages which contains the reusable targets. It also has some gotchas. Debugging is hard. Even the author admitted that he is debugging by printing stuff to the output, like in make concurrency. Logging is tricky when you're running concurrently many targets, the logs are not synchronized, and also without the verbose mode you do not get a lot of information in the output and when the things are failing in no verbose mode you get almost no information. So as we had seen in the example, we have, for example, no information which target had failed. At last, the API surface of the sh package is huge and I always had trouble which function I should run. There are functions like run run, v run with v run with et cetera, et cetera. And the API is so big that it makes hard to know which ones should be used. And even if you know what you should use, then the readability is a problem. When you read a code that was written some time ago, I felt that something was wrong and I could do something easier, simpler, more idiomatic. It took me one year to create initial version of Goyek, so after many long nights I thought that it should be a library. As libraries tend to be more usable and flexible. The tasks and targets should look like unit tests. Registering the targets can be done similar to registering Cobra commands. Parsing arguments should be leveraging the flag packages package and extensibility could be done similar to HTTP middleware's pattern. Now let me introduce to Goyek, let's run it by executing Go run build. As previously, you see the usage of the goyek. We have the information which tasks are available, but also we have the information which flags you can use. As usual, let's run the old target. When something is failing and you're not running in various mode, you have the same experience like when using go tests. So you just got the information that the test task had failed and you had the output from this one task. If you want similar experience to the make file, you just run it in verbose mode like for go test and you have the information of the whole execution. Let's look at how everything is defined. The build pipeline is defined in build folder and the main function contains an execution of a convenient boot main function. The boot package contains, I hope it's an extension of GOIC main which defines some reusable flags and configures the flow basically in a convenient way. So basically it is a set of commonly used middlewares and flags which are used when you're using Goyek for a build pipeline. If we look at all, it contains the definition of the all tasks and the definition looks similar to the definition of a cobra command. So we have a name all, we have the usage and we have also listed the dependencies of this task which is FMT and test. If we look at the test definition, we can see an action which is similar concept to a unit test. When you have the testing t so you here we have the definition of the task action which executes go test and go to cover to generate HTML report. The CMD exec is a convenient function for running commands in a shell like way. When you use CMD exec and test and the execution will fail, it will mark the task as failed, but it will continue the execution. So under the hood, under the hood it's just logging the execution and logging and using error to report a problem. Reusing reusing task is simple. We simply have a package demo task which contains the definition of an FMT task which is defined as FMT usage. Go FMT and is just executing go FMT. One of the big benefits of using Goyek is that intellisense is working because it doesn't use any build tags. Also because it's a regular go application, you can easily debug anything. So I can put a breakpoint here, I can press f five another breakpoint which I put and we can see that here. Now it has not failed because it was running fine, but if we step it's running executing go test, we need to wait a moment and now we see that the task is failing. We see that the value of the failed has changed and probably if we looked at the output, I just need to find it was debug console. Yeah, we received the output here so we can continue the execution. So as we can see, we can debug. We have problems and we can use intellisense during development, which is very convenient. It was just a teaser. I hope you enjoy it and I encourage you to check out more because Goyek is more powerful than just that. You can add middlewares, create reusable, customizable, build pipelines, customize creating, integrate with viper and any package, and I encourage you to check out the repository for documentation and see the examples. I especially encourage you to look at how Goyek is dog fooding its own functionalities for its own build pipeline. Here there's another quick example of usage. You can see that in the output you see in the locks. You can see the file name and line number and it's very helpful during debugging. And GOIC is very useful when you have a little more, little even a little more complex logics, like if statements for loops, you can use a helper to mark some functions as helpers. Then this function will be not printed when logging and it's really easy to debug and develop like regular boring go code. Let me summarize the presentation. I still use make not only for Gil repositories, but for non go repositories as well. For stuff like automating, setup of kubernetes, local clusters, protobuff validation. Usually for automating simple things I do not state make is bad, it's just complex, maybe even some aspects more complex than go. Moreover, I find doing complex things in make not straightforward. Regarding mage, the good thing it requires go. It has community, a lot of stars, and I know a lot of repositories are using it. But it has gotchas that are annoying me and they're annoying me so much that personally I was never using it in production. GoiC also requires go. It's a library instead of a framework with AK inspired by popular go packages, and I hope it's more idiomatic. It's simply yet extensible, and it works very well with your Ed and I use it in production and the behavior is similar to go test, but and if you like it, make a shout out. Feel free to try it, and any feedback is more than welcome. Here are some hyperlinks. Feel me to ask questions. I'm open to any feedback and thank you very much for your time and your attention. I hope you enjoyed the talk. Thank you. See you later.

Robert Pajak

Senior Software Engineer @ Splunk

Robert Pajak's LinkedIn account

Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways