Transcript
This transcript was autogenerated. To make changes, submit a PR.
So let's get started with building Go CLI application I am
Abhisek Pattnaik, full stack software engineer,
currently working as senior software engineer at Impulses.
To know more about me, please go to the link below. Before we
start with building CLI apps, let's get some prerequisites.
You should have some knowledge about basics syntax of
Go. You may use vs code or
intellij Goland IDE for writing the code.
You should also have some basics of understanding
about JSON and XML and
to give you some general information about the talk. All the relevant
links in the slides are clickable. All the code
images are clickable and point to the repository with
the appropriate commit. Slide and app project
are published in speaker Deck and GitHub respectively.
Relevant code commits are tagged as per the slide number.
So this is the flow in which we will walk through
this talk. I will provide the introduction to CLI
app development and we will get started with
flag model. In Go. We will learn how to parse
various types of data from the CLI arguments and then
finally use all the knowledge we gained to build a CLI
application to generate random data.
Introduction to CLI app why
do we choose go for building CLI apps?
Go provides reliable, efficient and
scalable CLI application development.
It makes it easy to build and it is fast.
Go has a rich set of libraries for various tasks such
as manipulating text files and data.
It also has built in and third party module
support for parsing commandline arguments and build CLI
applications. CLI stands
for command line interface where you write commands
and its related information in textual form.
You can launch programs and automate various tasks.
That means you have native integration or you can run the
program using any other programming languages.
It is easy to write and understand, unlike complex graphical
applications. Now let's see
what Go's built in flag module offers. We can
implement parsing the command line flags with various data
types, specify defaults,
and also provide the usage description about each
flag. It also allows to parse the data
into custom data structures. Getting started
with the flag module let's look
at the syntax for passing the flags and data to
our CLI app. We can use single or
double dashes before the name of the flag and optionally
a value either with a space or equal symbol between
the flag and value. Here is an example how
we can pass the commandline flags. Let us
look at the code so we check out the
slide number eleven and
to go mode vendor.
And as you can see here,
there is a name variable that is declared and we
are passing the address of the name variable to the string
VAR method with a
name as the name of the flag itself and
the default value as world, and the usage
is as the name to say hello to.
Then we parse the values, the argument
command line arguments, and then we print the hello
and the name that is provided. If no name is provided,
then it will take the world as the value. So let's
build it and test it out.
So as you can see, we can provide double
dash name and provide the actual name and
it will print the greeting.
And if we do not provide any value
or any name, then it provides
the default value.
Parsing data with flagged module the
flag module supports various ways to parse the command
line argument into, such as to string integer,
boolean float duration of time, which is
a non primitive data structure you can parse
to any other custom data structures using
flag VAR and flag funk methods
we saw how we parse flags where we need to provide the
name of the flag, and often it happens that the name
is too long to type. For this we can use flag aliases which
are shorter way to pass data to the flag variables.
Let's see the code.
We will switch to the slide 14.
Here you can see we have declared a variable name
and we are using the same variable name
for having a longer version of
the flag and a shorter version of the flag and
we parse the data and as before we are
printing the data. So let us build that and
let's use with
full version and with a
shorter version. As you can see, we were able to
provide the name to the
longer and the shorter version alias.
If we check the help and
we can see that we have two different
flags defined with a shorter and the longer version
in the usage.
So conventionally we write
a shorter version as the longer
version first character of the longer version.
It is often required that we have multiple flags
and there are longer flag names that we need to type to handle
it gracefully. With better maintainability and with ease,
we can make use of getopt third party
library. It provides a clean way to define the aliases
and also show the usage in a better way.
Let's look at the code.
We'll switch to slide 15
and make a go mod vendor.
So here we use the get opt third
party library to declare the longer
version of the we declared the longer version of the flag using
the flag model and using getopt. We define the
alias with the formal name,
same as the flag that is defined and a
shorter name. And then we parse using getopt
and we provide give the default print
the defaults. So let's build and look at how
it appears. As we can see,
the shorter and the longer version of the flag
appear in a much better way than the previous
way of showing the usage.
A user of the CLI app might not know what
flags are available for CLI app and how to
use them. To find out, we conventionally use
hyphen h or double dash help
to show the usage of the CLI application.
Let's see how to implement that.
Let's see the code we check
out to slide 16.
In this we have a
name and repeat
two different flags. We get the
total arguments that are provided in the commandline.
Using the OS args, we discard the first argument,
which usually is the name
of the commandline program that is used to run.
Then we define a help flag
and defaulting to false or
true based on the total number of arguments.
So if the total number of arguments is zero,
the help will run and we
will get the defaults the usage of the flags.
So let's try that. Let's build it
and without giving any
arguments. As we can see, we got the help.
Even if we provide hyphen h or double
help, we get the usage
of the command line.
You built a nice CLI application, but a creation
without a name is like a ship without a cell. Let's give
a name to our application. For example greater CLI.
We use the new flagset constructor function
to create a new flagset with a name.
Let's look at the code. We switch to the
slide number 17 here.
We create a new flag set with new
flagset. Here we are using the gitopt
library and we can provide
the new name that we have reacher Cli
as the first parameter to the new flagset
constructor function and we use the
new flag set to define
the flags and also provide aliases
so this can be used with the flag
module as well. And then we parse
the arguments. If help is true, we provide
the usage. So let's see.
Let's compile this and provide
the help. And as you can see when we see
the help, the usage of greeter
CLI is as follows. So you can see
the name of the CLI application that we have
given. As noted before,
we can parse a CLI flag value to a
custom data type. For that you need to implement the
flag dot value interface for the given custom data
type. Let's see how
to implement.
We go to the slide number 19.
It here we have address
structure, which is a custom data type.
Then we use the address structure
as a pointer in the address value
and we implement the get
and the value interface. In the value interface,
you implement the string method
and the set method. In the string
method, we simply return the string value of the address
and in the set method we split
the address that is provided in the given format
and then assign it to the
address structure.
So let's build this and
see how we can use it. As you can see here, it is expected
in the format, street, city, state,
pin code, and country. Let's provide the address in the
same format and
then as you can see here,
we have given the address in a string and
using the set,
it has assigned the address to the corresponding
fields in the address structure.
We can also use flag funk method
to parse and extract the required data from the CLI flag
value. This might come in handy if a custom parsing
function is required. Let's see the code.
Let's go to the slide
number 20. Here we have
defined the flag num which
will take a string and extract
the number value from that and then assign
it to the num which is. This is a custom
parser function that we have written. So let's compile it
and run it with
num,
and we can write any characters,
and in
between the characters we provide the number.
Now that number is parsed and it is extracted
assigned to the num integer.
And then we print the number as promised.
Let's see how we can provide a JSON file as
input and parse it. The JSON file
can have mixed data type for a given field.
So let's look at the JSON file.
So we go to slide number 21 and
this is the input JSON. As you
can see here we have an alias property which
is a string with a comma separated aliases,
and the same aliases property
is an array of strings.
So we will see how we can process
this.
To parse the input JSON, we implement the
flag value interface and unmarshal the
input to JSoN struct using the flag
VAR method. So let's try that's.
Let's walk through the code.
We go to slide number 23.
So here we have a JSON Cli.
And in this JSON Cli, this is the input.
This is the input which accept which will store the JSON values.
And this is the item.
So this corresponds to these values and
it is expected the aliases will be a
slice of strings. Now using
the VAR method, we define
the JSON input flag JSON input flag is defined as
such. We provide the input file as a pointer
and then define the
set method. Here we extract
the JSON, read the JSON file and then parse the
JSON. Unmarshal the JSON to the structure
the input file struct.
So while unmarshalling here, we can see that we create an
alias and unmarshall
the file the JSON data.
We have used the aliases as any in
this case, so that the aliases with
mixed data type is assigned to this
aliases field. And from here
we check if the aliases is a string,
then we split the string using comma separated
values. Otherwise, if it is
a list of strings, then we run
through the list and convert it to
a string and assign it to the aliases
array. So we
can also use alias string likewise
and let's build it and check
json cli. Let's see the
val the input file. We pass the input file using
hyphen I. Let's see the flags
which we can use. As you can see, hyphen I is for
providing the input file. We provide the input file and
it processes the file appropriately without
any errors as
per the print.
Now since you have gone through many parts of the flag
module, let's see how things are implemented
internally. You can find the implementation source code
from the given link.
Let's see the browser.
The parse one is the function which parses
the ClI flag and that is implemented
likewise. And you can see here the formal name.
If it is not there, and if
we have given the health or h
as flag name,
then we will get the usage of the CLI application.
So this is how the
flag model is implemented. You can go through this how the parse
function works and how the parse function for
each argument it parses.
Okay,
now we go through the final part of the session generating
multiple random data.
Let us start with generating a random number between a range
given a max, min and max values from the
CLI argument.
So let's switch to the slide 26.
Here we see we have a min and
max variables,
and those are bound to
the min and max flags. And we have
alias for mean as small m and capital
m for max. And then we have
a check where the max and max should
be greater than min.
We create a random generator
and then create a random value from that and
train the value. Let us build and check random
ClI. Okay,
we need to go mod vendor and then we build it
random cli small m
and capital m. We provide the mean as one and max
as ten so we get a number between one and ten.
If we provide max as 100 and mean as ten,
we get a number between ten and 100.
Now let's try generating random values using a
third party library. We will use go fake
it faker library which generates various types
of valid random data such as address,
name, text, et cetera.
So let's go to the slide 27 and
make a go mod.
So here we have a student structure
with a name, age,
percentage, height, active with the
given data types, string int float 32 int boolean
and in the meta we have a description with a string data type.
Now we will be using go
fake it library and
we seed the library and then we
create a random we create
random values for the fields of the struct using
faker struct method.
So then we
dump the data in the structure.
So let's compile that and seed.
So as you can see here we have the
name random values assigned to the name
and other fields of
the structure.
We saw how we used the faker library to generate
random data, but the data itself was not meaningful. We can
provide some hints to get some meaningful data.
Let us see how.
So let's go to the slide
28 and here we have address
field and in the address field we
have given some hints like
strict street, city, zip, state and country.
This will generate meaningful data for
the given sales considering these hints.
So we do the same. We again run
the faker struct with the address and
we execute that o mod vendor
and run
the build the code and
again run it. So as you can see here we get meaningful
street values, city values, pin code,
state and country.
We can also generate data using custom functions using
the faker API.
Let's see how. So we
go to slide number 29 and here
we have the address structure from
the faker. We generate the address values using
faker address and we assign the
corresponding address values to the fields in
our address struct and then dump the address
value. So let's build
it and
run it. So as you can see again we
got appropriate random values using
the picker functions.
Let's see how the
gofaket library is what the
API of Gofaket library provides.
So we have various functions
and the struct method
that we used.
So there are multiple different functions available for
person generating person addresses
and other miscellaneous things.
So you can go through the
API for learning more about how the
Gofikit works.
Now let's
see here.
We are having some
of the libraries that we used for
debugging the structure so that we can pretty print
the struct and find what is the
data in that. So you can refer this out
of that. We are using spew dump and val
ASD. So this can
help you make a log based debugging.
Let's see how we can embed data in
the CLI app so that we can generate
this embedded data later after the build based
on command line flags.
So let's check out 31
slide 31 and we go
mod. So here
we have a person
schema and we are embedding that
person schema in our go application as
defined here with the person schema variable.
And then we generate
a person.
If the generate flag is present, then we generate a
person schema. We generate the person
schema.
So let's see the person schema
generation here we create a file and then write
the person schema to that file.
So let's see, let's build it.
So here we will be,
we can run the random
and generate so here we generated the person schema file
in the root directory by
providing the command line argument as double dash generate.
And one thing you can note here that the directory
structure that we have is we have a commandline
CMD directory and we have an internal directory in the internal directory.
Whatever we have, it is internal for this application
and no other application outside of
this can import this directory
exported values and CMD
is for having the commandline for
that. And you may also take
a look at the mech file which generates the builds
the command line application.
So we have go build output directory specified
as bin, provide the eligible flags if applicable,
and then provide the structured directory structure
that will be built.
So go generates
the directory, the command line applications
with the same name as the directory where the main file
is located, and hence
we got the random CLI binary
file in the bin directory.
As we saw, we were able to generate the embedded file
using CLI flag.
Previously we have been generating only a single set of data.
Now we will look at how to generate multiple random
sets of data. We will generate a few person details using the
number of persons count from the CLI num
flag. Let's look at the code.
So we will switch to slide
number 33.
So here we
have a, we have the commandline application,
we have a package directory where we have defined
the details of the person
that we need.
And this will also have a new person
factory function which will construct a person
using the faker struct. And we
also have provided hints for the person,
that is the address for the first name, last name,
and then we create a new
person.
So this is the
internal CMD which will generate the persons based
on some number of persons that we want to generate.
And here the command line
application will simply call that method,
call that function and generate function.
Persons and number of persons will be provided from
the CLI. By default, number of persons was one
and then we are printing the persons
which we are generating. So when we print
this then person,
this stringer interface has been implemented and this
is executed and the
value return value is printed in the command line.
So let's build it and
provide the number of person.
We can provide an alias using hyphen n.
So let's generate ten persons.
So as you can see here, we have generated ten
persons with
address values.
We generated the data one after the other. We can also
generate concurrently using go routines.
We make use of weight groups and go routines.
Let's see the code and see how we can generate the
data concurrently. Let's switch to
slide number 34.
And here if we go to the internal
CMD person we are using a weight group
and we are generating the data concurrently. Here we have a
go routine and we have a wait group to wait for
all the coroutines to complete.
So in this case all the coroutines are run simultaneously.
So let's build
it and run it.
So it gives the same ten number of
persons but all these persons
are generated simultaneously.
And the faker
library all supports the simultaneous
generation of person.
Using it supports constructor concurrency.
Sorry.
So previously the concurrent threads
ran all at a time, but we can also restrict the number
of concurrent threads that run at a time which runs the data.
We can make use of buffered channel to achieve
this. Let's see how we go
to slide number 35 and
go to the persons.
Here we
go to main and let's
see the
number of persons we accept using the
num flag and the concurrency using
the concurrency and
we are printing based on this and
we pass the concurrency that we get from
the generate person options through the generate person
options to the gen
persons function. And here
we create a buffer for the
kind of buffer for the channel and
then we fill in the buffer,
adding things in queue till
the buffer is filled. And then we run the go routine as
and when the go routine succeeds the buffer will be cleared
and from here we wait for the go routines, all the go
routines to be finished. And once
there is less than the
maximum buffer amount of data then this
is returned and we get all the persons
from here, so let's build it and
so print the data.
So this will use the concurrency of eight as
the default or we can also specify the concurrency.
As you can see here, we specify the concurrency as
three and we generated ten persons
data.
It now that we have learned how to generate
random data, we will use that to generate
XML file output with random data.
We will have the CLI flags for specifying the output
directory and the name of the
XML file.
Here we also output the embedded schema file
along with the XML. We pass the number of records
to be created using hyphen n, the output
directory using hyphen O, and the name of the XML
file using hyphen capital n, and optionally pass
the concurrency as well using hyphen C like before.
This will create an XML and an XSD schema
file for the XML. Now let's
dive into the code.
So we switch to slide number 37.
So if we go to the command CMD
directory and check the
available flags which are bound,
we have a num flag, we have a concurrency print name
of the XML file, output directory of
the XML file and force if we want to overwrite
the files in the directory.
So we will be outputting it to temporary
directory. So let's delete that and
we have some aliases. So let's first
build it and see how we can generate
XML file and associated XST file.
And then we will look at the code.
So XML
CLI and we use
the number as ten and
the name of the XML as first gen and
the output directory to be temporary directory.
So here you can see we generated
an XML file and
we have the XST file associated with that.
This xst file we have embedded from
here using
go embed and then we
generate the XmL file using the XML
struct that is found here
and the persons are the items of the XML
and that can be added
from here. So persons data is the same
person struct and the
associated person data. So here again we are using the
buffered concurrenced way of generating
the data and creating the XML.
So in this internal CMD file
we create the XML using
generate XML and this will write the data
to the XML. We create an encoder for the XML
and then we encode the XML.
After that we create the XML file.
So this will check if the file is existing and if
it exists and the force is not true then it will
give this error as file already exists.
Otherwise it will create the file and
generate the person generate the XML,
create the schema file and then
return. So schema file is
also specified here, how it is being
written to the disk,
and then the schema
file is after returning it will print the concurrency.
Likewise,
till now we created randomized data,
but we can also pass some predefined data
for a few fields and the rest will be random
data. We can do that using a JSOn file as
input. With the given data we will generate the
placeholder JSON file and the JSON schema for the
file using the hyphen g flag. This will create the files
under the directory passed using the hyphen o
flag and we will input the prefilled
json file using the hyphen
iflag. Now running
our CLI app using the same parameters for generating
the XML as before, we get the XML
and XST files created in the output directory.
Now let's look at the code switch
to slide number 39 it.
So here we
have a few more flags like the input
flag and the generate flag.
The generate flag will generate the input Json and
schema and input flag will input the prefilled
Json data. So this
is the schema input schema and this
is the placeholder
input json. So these two files are
embedded. This is Json schema that is
embedded and then this is the input Json file that is embedded
and those files are generated when
we run hyphen g
when we generate using this and we have a few aliases
as we usually do. So let's first
remove the temporary directory and
we can output generate.
We will go mod vendor,
we'll build the cli application,
see how we can generate a
so input json. So here
we created an input json placeholder file.
So here we can
provide the first name and
rest of the fields will be randomized data.
So let's see if it is generic
input json. If hyphen g command
line argument is provided, then it will create
an input Json file. And if
we go into that we will generate an input
json file in the given directory. And also
we will create a schema file along
with that. So this is the input schema file and this is the input json
file it.
Now if g
is not there and we want to generate
the xml file like before.
So here we provide the predefined
person. So the person is taken from the input
file. If we have an input file
path provided, then it will generate a
refilled person.
It will read the input json file and unmarshall it to the
person info and return that.
So that is added in the person info
and it will pass to the xml generation.
So here in the xml generation everything
is as before. But while generating the
person we also pass the predefined person data.
And if we check the get generate
person here we are copying the predefined data with
the randomized data. This is the new code
change that we have and for copying we have
this code.
Now let
us see, since we have already added
a first name to the prefilled
JSON, so we can input that input JSON
and we can name the XML file as Abhisaic
and create ten records and the
output specified will be temporary.
So this will create an XML file with
the first name prefilled as Abhishek and
other fields are randomized. Now if
we provide the last name as well and
we generate the same thing with hyphen
f so that we can force the overriding of the XML file.
Then if we go to the
XML file as you can see here, we generated
the randomized data with some prefilled data for
the first name and last name for all the records.
So we discussed how we build a CLI
application with randomly generated data input
JSON with mixed data type fills,
embed and generate XML and JSON files
with prefilled data. So here are the key
takeaways. CLA apps are important
tools for developers and Go is a great
language for building them due to its strong support for command line interfaces.
The flag module in Go is a powerful tool for parsing command line
arguments and building CLA apps to generate data
concurrently. You can use Go's powerful concurrency features such
as channels and Go routines. You can override randomized
data with predefined data. When building CLI
apps in Go, it's important to keep the user experience in mind and provide
clear and helpful error messages when commandline arguments are
incorrect. With the techniques covered in this talk,
you'll be able to build powerful and flexible CLI applications in Go
that can generate randomized data quickly and efficiently.
Here are some of the resources that will
be helpful for you in your journey to become the master of CLI.
The first one is the repository for the workshop and
then there is the presentation for the workshop which you can
get from these links and
awesome list of CLI applications are a list of
CLI applications which
will be helpful to
understand how the CLI applications work.
Flag model is the flag model API
provided in the Go documentation and
then there is a flag model source code that you can go
through the GitHub library that we used for aliasing,
the faker library that we use for generating fake data and
if you want to build powerful and more complex
CLis, then framework will come in handy.
If you have any questions, do reach out to me. Kindly forgive
me for any mistakes that I might have made, either verbal or
otherwise. Please scan the code and
provide anonymous feedback. Thank you.