Adventures in Golang

Kwan Lowe (February 18, 2019)

Over the long President's Day weekend, I decided to learn Go. The Go Programming Language (golang) is a relatively recent language designed to improve programming productivity. Highly performant, it has features similar to C or Java without requiring a complex compiler toolchain and build environment. Though intended for multi-core concurrency and massive codebases, these features can make it ideal for infrastructure tasks.

But there's Python. For several years, Python (and an occasional Bash script) has been my de facto language for everything from ad hoc reporting scripts to managing cloud infrastructure to scheduled infrastructure jobs that copy, deploy, audit and maintain the environment. It has performed admirably in this role and has saved me countless hours.

So why look elsewhere?

A little over ten years ago, Perl was my de facto language for everything from ad hoc reporting scripts to managing VMWare infrastructure to scheduled jobs that… You get the idea.

Then last week I was debugging a Python script that moved data from Kafka to Azure Data Lake. There were some issues with an decoding library that broke when trying to deserialize the avro stream. In the end, I punted and used some working Java code to run the deserialization, Python to move the file to ADLS, and a Bash wrapper to tie it all together. In defense of this Frankensteinian melange, these components already existed and needed only minor tweaks to get working. And the solution was just an interim hack until proper development could be started. This is what is known as "famous last words."

A few annoyances occured. The system Python version was 2.7. No big deal. Add a couple from __future__ imports and most of the code just worked. Most of it. One particular library existed solely in Python3. I setup a virtualenv and moved my code over. The ConfigParser library required some minor tweaks. The argparse got replaced with optparse. I checked my code into Bitbucket and wrote an Ansible playbook to deploy it. Alas, it would run as the data transfer user so I had to tweak the code to figure out the Python virtualenv. 

Then there was the Java bit which I had received as a maven project. Arguably, if I were in an IDE the setup would have taken minutes. Alas, my "IDE" is GNU screen session and Vim. I manually edited the POM.XML to include dependencies, installed Larry's Java, and built the Development binary. It worked.

But then came time for the Staging environment. Oh durn, it was an interim hack so there were some hard-coded bits that needed to be changed. OK, managing three sets of binaries for the three (or four) environments was not feasible, even for a hack. I added some Java code to read from a configuration file, re-built and redeployed.

Durn, the Staging environment didn't have the same Java8 environment and would not run ("Write once. Run anywhere!". Yup.). I reworked the configuration to use OpenJDK-8. I re-deployed. I pushed the Bash wrapper script. I pushed the schedule to Jenkins. And yes, it all worked.

But it never sat right with me. The part of me that twitched when a vendor misspelled a blob storage container practically screamed at a solution that involved Java, Python, Bash, Jenkins and even a bit of awk. So I went about manually decoding the avro stream byte by tortuous byte (ok, I exaggerate a bit). I finally got some some working code that built a JSON object from the avro stream.

Oh, but during debugging I'd built a bleeding edge version of the rdkafka library pulled straight from github. This was a C-based avro decoder there only for speed. To get this working required adding a few header libraries and rebuilding. Of course, since it was in my home directory, some LD_LIBRARY_PATH and other path manipulation was needed. So now my stack included Java, Python, Bash, awk, Jenkins and C. And how the heck would I deploy this?

There's Got to Be A Better Way

Go could potentially solve this mess. Versioning would not be a problem. The maintainers of the language and standard libraries seem to go to great pains to ensure compatibility. And because it was compiled, I could ship the binary and it was just run without needing a full environment deploy. The fact that the import statement functioned as a requirements.txt also helped with reproducing an environment. 

To learn Go, I decided to convert a few Python and Bash scripts over. The first hour was spent familiarizing myself with the toolchain, installing the language, setting up an IDE, and getting a "Hello, World!" running. 

My first exercise was to convert a simple Python sorting script to Go. This script read some default parameters from a configuration file, examined the files in a staging directory, and depending on a field in the filename, moved it to a new directory.

The first step was to read the defaults. In Python, I used the ConfigParser library. In Go, the TOML format seemed most accepted and Go-like. Though there were Go INI readers, I decided to go the TOML route for the novelty.

def readConfig(configFile, profileName):

    """ Reads config data from file, returns dictionary """

    import ConfigParser

    import sys

    import os.path


    authConfig={}

    configFile = os.path.expanduser(configFile)

    config = ConfigParser.RawConfigParser()

    try:

        config.read(configFile)

        # print(config.get(profileName, 'environment'))

        return config

    except:

        print("Unexpected error:", sys.exc_info()[0])

        raise

[...]

options = getArgs()

profile = options.profile

conf = readConfig(options.configfile, options.profile)


environment = conf.get(profile, 'environment')

sourcePath = conf.get(profile, 'sourcePath')

companyAPath = conf.get(profile, 'companyAPath')

companyBPath = conf.get(profile, 'companyBPath')

Converted into Go, it looks like this:

package helpers

import (

  "fmt"

  toml "github.com/BurntSushi/toml"

)

type tomlConfig struct {
        Title string
        CasinoConfig casinoConfig `toml:"casino"`
}


type casinoConfig struct {
       StagingDir string
       CompanyAPath string
       CompanyBPath string
       Environment string
}


func GetConfig(configFile string) *tomlConfig {

    config := new(tomlConfig)

    if _, err := toml.DecodeFile(configFile, &config); err != nil {
         fmt.Println(err)
         return config
    }

    return config

}

Next, we had to sort the files by filename. In Python, I had a list of filetypes and each filename was compared against two lists. If there was a match, the file would be moved.

companyB_filetypes = [ "AAA", "BBB", "CCC", "DDD" ]
companyA_filetypes = [ "HHH", "III", "JJJ", "KKK" ]

[...]
sourceFiles = [f for f in listdir(sourcePath) if isfile(join(sourcePath, f))]
for filename in sourceFiles:
    if not fnmatch(filename, "*.bis2"):
        sourceFiles.remove(filename)

for filename in sourceFiles:
    filetype = filename.split(".")[2]
    if filetype in companyB_filetypes:
        print("Moving", filename, "to", companyBPath)
        shutil.move(sourcePath + "/" +  filename, companyBPath + "/" +  filename)
       
    elif filetype in companyA_filetypes:
        print("Moving", filename, "to", companyAPath)

        shutil.move(sourcePath + "/" +  filename, companyAPath + "/" +  filename)

In Go, there is no "in" operator so I grabbed a function from a Google search that does the equivalent. The overall logic is identical otherwise.

func contains(a string, list []string) bool {
    for _, b := range list {
        if b == a {
            return true
        }
    }
    return false
}


[...]
companyBFiletypes := []string{ "AAA", "BBB", "CCC", "DDD" }
companyAFiletypes := []string{ "HHH", "III", "JJJ"m "KKK" }
[...]
files, err := ioutil.ReadDir(StagingDir)
if err != nil {
    log.Fatal(err)
}

      for _, f := range files {
        if contains(f.Name(), companyBFiletypes){
            fmt.Println("Moving to " + companyBPath)
            err := os.Rename(StagingDir + "/" + f.Name(), companyBPath + "/" + f.Name())
                if err != nil {
                    log.Fatal(err)
                }
        } else if contains(f.Name(), companyAFiletypes){
            fmt.Println("Moving to " + companyAPath)
            err := os.Rename(StagingDir + "/" + f.Name(), companyAPath + "/" + f.Name())
                if err != nil {
                    log.Fatal(err)
                }
        } else  {
            fmt.Println("File type not found!")
        }
    }

Even for this trivial example, there were benefits to using Go. For example, having a single binary to deploy, rather than pushing a virtual environment, saves time. For this script it is measured in minutes but potential savings on a more complex task, such as the Kafka reader, is easily in hours. I see what Google is trying to do with the language.

I am not knocking Python. It will continue to play a large role in infrastructure automation and tooling, but Go does impress. In just a couple hours I was able to get something functional and after one day, I was already uploading files to Azure, reading Kafka queues, and even had Dockerized versions ready to upload to our K8S cluster. Go does not appear to have the same level of maturity for data science and other wrangling tasks, so it cannot fully replace our Python code yet.

Anyway, thanks for reading this far. If you know a more Go-like approach to the script above, please do comment. I am thoroughly enjoying the language and want to learn more.