Skip to main content

Curated golang libs and things

I need a goto place where I can track or at least link to my favorite libraries. Since golang is now my complete preference here is my list:

Good article on self referential functions. helpful article for http/REST. List of go tools. And here you can setup a project.

Instructions for setting up for go cross compiling.

gopass - console password prompt

gorename - the article, the source (the source comes from Google and attached to the go tools repo)

gogs - self hosted git service written in Go.

go-bindata - Useful for embedding binary data into a go program. go-bindata-assertfs - Serve embedded files (http handler type funcs)

msgpack seems to be a good and efficient message container. Of course there is JSON and many different implementations with different performance profiles. What is nice about MP is it's support for so many different languages which could provide significant interop.

MQ - gnatsd or here - go implementation of a nats server (client libs are in adjacent projects). This project has a broker. Beanstalkd - is also a strong pub/sub broker although redis can also be used in this capacity. And of course ZeroMQ offers brokered and brokerless topologies. (I'm not a fan of RabbitMQ so no link is given). libchann is also pretty close to a MQ. go-nanomsg and mangos. messagePack code generator. MessagePack home. Kite seems to offer a combination of RPC and netchan/libchan features. (libchan is stalled; netchan is deprecated; new-netchan is in limbo) Que-go is a work queue based on the ruby analog. There is more than one way to do this but it's worth a read.

CLI - there are some new 3rd party libs for processing the command line.  From what I've seen they must be deeply rooted in the psyche of the author. The stdlib does a good job and does not need replacement. Command lines do not need to be complicated and they should not be.

Logging - Lumberjack for it's rolling logs. But remember that Docker like having the logs expressed through stdout. While I'm not a fan of aggregated log files this(log-shuttle) logger might be useful.

Database - this category is tough. It depends on the type of DB [SQL, NoSQL, other] that you might be deploying and then there are a multitude of wrappers to choose from. The latest that caught my eye was a mongo-like document API that used Postgres 9.4 as the backend. In this case if the underpinning or translated structure is "least surprising" then you might get the best of both worlds in one container. (Golang database drivers)

There are a number of ORM libraries that I have used and reused. My favorite is gorp. Since I only use the ORM methods that support proper marshaling I have been looking into other libraries. Ones that are smaller, easier to use, more thoughtful. This article was helpful. SQLX seems like a good option. One important feature is Server support. Many of the ORMs require specifying the dialect and than can be a challenge.
I prefer to write the proper optimized SQL than letting then letting the ORM layer make that decision for me. Early use if "hibernate" which required hand customization via the HQL syntax meant learning two SQL variations in order to code and debug.
The Hood project has gone to great lengths to incorporate the GoLang tags as complete metadata for the DDL. While it's interesting it's a lot of work. It means that teams will have to develop a meaningful style guide in order to use Hood in it's opinionated way. I think the Go Authors did not really want that to happen as evidenced by gofmt and it's motivation(s). And frankly if you need a DDL then implement a DDL rather than manually preprocessing the DDL by embedding DDL meta info in the structure tags. (see go generate)

The model project does not get me to the happy place either; even though it's author is the one that forced me to shift more directly toward the cleaner SQL marshaling (SQLX) rather than the mega-orm.

Authentication

Dependencies - godep, glide and then there is gopkg.in for API versioning.

Load testing with vegeta or boom, written in go, look interesting because they offer a CLI as well as libs that can be integrated into your application. (yes; there is apache bench [ab], loadrunner, seige, and so many others)

Crypto - crypt storing config data in etc (Kelsey Hightower wrote this; I have not been able to find an authoritative link) What makes this interesting is that every application is sharing the same storage but that only the owner (holder of the keys) can retrieve the data. It would be interesting to combine crypt with go-bindata so that the keys might be auto-baked into the application as part of the pipeline. Additionally, one needs to publish the public keys in the etcd engine itself so that others might publish data for your applications consumption across the cluster. There is some interesting ssh APIs too.

I have been following YubiCo for a long time. They have a number of interesting projects and while I like their HSM is does have a few limitations by itself; that being TPS. So there are only two use-cases that make sense. (a) systems with low transaction rates, or (b) use the HSM to decrypt the persisted, encrypted, private keys and leave the transcoding of the cipher/plain-text to a proxy. This increases the TPS but creates a trust issue. This Yubi-KeyServer is worth watching with the only concern being that it's authorship is outside the US.

Instrumentation - "instrumentation through composition" (article), lumbermill, instruments, metrics, go-metrics, Metrics (video) Monitoring from StackExchange (article, home, code)

Web - vulcand looks like a nice reverse proxy and/or HA with interesting ambassador features for docker

IDE - Wide(web based IDE), lightIDE, Atom, Brackets, and so many others.

As a service on Windows

DNS - GeoDNS. A very lightweight DNS server. SkyDNS2.

a go version of gitreceive so that you can trigger hooks.

templates - I really like the golang std library for templating but there is something to be said for mustache if cross platform or shared templates is your thing.

Package templates - requires go 1.4

Go package updater via browser instead of terminal.

libvirt

link to good concurrency slide deck and video

One of the Go Authors has been arguing against frameworks. I do not recall which one or how strongly but then there are some frameworks that me. These might not really be frameworks but templates.... but who knows where the line blurs. This year John Graham-Cumming presented his dotgo project. I cannot agree with him more. His project reads from stdin and write to stdout. The processing takes place in a goroutine so the results may be unordered but could be highly efficient.

Here is my Docker post with a few links to some docker related projects.

Jobber as a replacement for cron

Mailgun is a great toolkit for sending emails. I tried sendgrid but was not able to get it going and mailgun simply worked.

Comments

Popular posts from this blog

Entry level cost for CoreOS+Tectonic

CoreOS and Tectonic start their pricing at 10 servers. Managed CoreOS starts at $1000 per month for those first 10 servers and Tectonic is $5000 for the same 10 servers. Annualized that is $85K or at least one employee depending on your market. As a single employee company I'd rather hire the employee. Specially since I only have 3 servers.

The pricing is biased toward the largest servers with the largest capacities; my dual core 32GB i5 IntelNuc can never be mistaken for a 96-CPU dual or quad core DELL

If CoreOS does not figure out a different barrier of entry they are going to follow the Borland path to obscurity.

UPDATE 2017-10-30: With gratitude the CoreOS team has provided updated information on their pricing, however, I stand by my conclusion that the effective cost is lower when you deploy monster machines. The cost per node of my 1 CPU Intel NUC is the same as a 96 CPU server when you get beyond 10 nodes. I'll also reiterate that while my pricing notes are not currently…

eGalax touch on default Ubuntu 14.04.2 LTS

I have not had success with the touch drivers as yet.  The touch works and evtest also seems to report events, however, I have noticed that the button click is not working and no matter what I do xinput refuses to configure the buttons correctly.  When I downgraded to ubuntu 10.04 LTS everything sort of worked... there must have been something in the kermel as 10.04 was in the 2.6 kernel and 4.04 is in the 3.x branch.

One thing ... all of the documentation pointed to the wrong website or one in Taiwanese. I was finally able to locate the drivers again: http://www.eeti.com.tw/drivers_Linux.html (it would have been nice if they provided the install instructions in text rather than PDF)
Please open the document "EETI_eGTouch_Programming_Guide" under the Guide directory, and follow the Guidline to install driver.
download the appropriate versionunzip the fileread the programming manual And from that I'm distilling to the following: execute the setup.sh answer all of the questio…

Prometheus vs Bosun

In conclusion... while Bosun(B) is still not the ideal monitoring system neither is Prometheus(P).

TL;DR;

I am running Bosun in a Docker container hosted on CoreOS. Fleet service/unit files keep it running. However in once case I have experienced at least one severe crash as a result of a disk full condition. That it is implemented as part golang, java and python is an annoyance. The MIT license is about the only good thing.

I am trying to integrate Prometheus into my pipeline but losing steam fast. The Prometheus design seems to desire that you integrate your own cache inside your application and then allow the server to scrape the data, however, if the interval between scrapes is shorter than the longest transient session of your application then you need a gateway. A place to shuttle your data that will be a little more persistent.

(1) storing the data in my application might get me started more quickly
(2) getting the server to pull the data might be more secure
(3) using a push g…