Skip to main content


Showing posts from September, 2015

a reason to hate docker

Once a quarter I perform some system maintenance on a cluster of Asterisk servers and their Dashboards. (a) backing up log and cdr files, (b) purging some logs, (d) repartitioning the cdr log database tables and the trigger that inserts the records. I happen to use Fabric as the remote execution tool. I also have a 50 line microservice that I use to create the trigger on the fly.

So far so good.

The microservice is running on a Google Compute Engine node, on CoreOS, with Docker. GCE is fine although I needed to punch a hole in the firewall. CoreOS is on release 711 or something. And docker is whatever version it is.

The nice thing is that even though this system is running well and survives reboots it has a number of major flaws. Once a quarter I need to build and run the microservice, however, it never works that way.

previous docker build can consume 100% of available drive spaceI have to make sure the 9090 port is a passthruhave to remember how to build the containerand then how to …

Docker Pricing - WTF

I always knew that Docker was going to charge for it's product. The questions were always; when, how much, and for what? So far most projects that were going to charge for this sort of thing offered the code and binary for free but then charged for support. Granted I had no idea what that support entailed but having been in a corporate environment when even the most expensive subscription service agreements yield less than stellar results; it's just no fun.

So when I tried to download Boot2Docker only to find that it was deprecated and that now docker was offering a non-free TOOLBOX I about lost my lunch. Docker may still offer fragments of their tools slightly crippled or even the FUD that it's not in parity with the open source version... It's just ugly. I suppose they feel the community momentum is in their favor and that the community will continue to test for free. (I'm not so sure).

This also has me concerned about Rancher. They offer a nice package based on …

major cloud-init weakness

Now that you've wet your pants it's not all that bad.

I continue to deep dive into CoreOS, RancherOS, and Docker. I've also been testing ideas with both Google Compute Engine and Digital Ocean. And a lot of things have been going badly.

The most recent hiccup was realizing that any changes to CoreOS' configuration must be accompanied with a complete refresh of the nodes cloud-config file. While I have no experience with it I'm hoping that the enterprise CoreOS experience is better than my standalone.

Doing a complete CoreOS refresh while there is volume sharing etc with the host means that the cloud-init is very complicated. My development machine is configured with both a Bosun and Grafana containers. And then there is my devbox. Since containers have been known to crash from time to time I am sharing a volume with the host. Only some of that is problematic.

In some environments you might have multiple admins... and so the admin's ssh keys would be installed i…

Modern home networking

I work remote.

There I said it and now all those fears you think you've had about QOS, ISP, VPS and so are are all mine. I depend on my network and when I would out of the house I depend on it there to. [I'm selective as to which Starbucks I work from because they are filtering their network]

The actual backstory goes like this.  Last Wednesday there was a severe thunderstorm and historically my ISP loses some equipment which can take several months to convince them to identify and replace. So when my network started to misbehave I knew exactly what to do.

I started making phone calls and was confronted with the same responses. "reboot the modem". "WiFi or wired? WiFi, then connect directly".  "Firewall? Then bypass and surf naked". These recommendations remind me of the old BSOD days from Microsoft ["reboot I say!"]. But I had already tried traceroute and I knew that the problem was in their network. My ISP previously used ATT and was n…

CoreOS missing features

In a recent blog post from the CoreOS team they presented a new feature uses rkt and flannel in order to create an ephemeral network between containers on nodes. I do not completely understand the details but that's coming. What did catch my attention was that the cloud-config file that was demonstrated made it clear that the entire deployment needed to be CI capable with zero downtime. Meaning that each node would have to be replaced in realtime without any downtime.

Since this sort of realtime migration has not been discussed in any of the docs or posts I can only conclude that it's implemented with the paid-for CoreOS tools. This is yet another area that makes selling to managers and stakeholders difficult.

PS: I was told to expect an update on CoreOS pricing but that has not happened yet.
In the last week or so I posted that I had had some success with my CoreOS cluster.

Well, when the cluster is not doing anything except auto updating then that's not really success. I have two service files that I've wanted to use to launch Bosun and Grafana. The problem is they will not launch from the worker. Something is missing in the setup.

When I tried the fleetctl start bosun command I got this error in return:
Error running remote command: SSH_AUTH_SOCK environment variable is not set. Verify ssh-agent is running. See for help. When I followed the link in the message there was nothing about SSH although there were some very vague hints. I went back to the documentation where I pilfered the image above and read it carefully. This stood out:
The cloud-config files provided with each section are valid, but you will need to add SSH keys and other desired configuration options. It's not clear …

cloud storage misconceptions

Here are just a few facts:


if you have a gmail account any additional storage you might purchase it for that one userif you have a free domain account at google then you get the base storage for free but anything after that is for the single user. The extra storage you might get from buying a chrome device directly from google applies to that one accountpaid accounts come in two flavors. The vault option is very promising and I have not seen an equal.Google has always been frugal such that the full size images are in the cloud and the thumbnails are on the device. (I think)Google+ photo at reduced resolution is free. (I like this!!!) APPLE family sharing does not apply to storage (link)Finally added iCloud application storage for iPhoto. I'd say a little too late... with 58K photos it's going to take a few weeks to get sync'd BOX are they a contender DROP BOX APIs are nice but do I care CAMLISTORE nice idea, incomplete, need my own cloud servers, and no client software f…

Advanced Cable Communications in Weston hates the rain

This is what happens when it rains in Weston

The internet is just a failure.

UPDATE: I forgot to mention that when I ran a traceroute the results suggested that the problem was in ATT's network.  The problem, however, is that ACC moved to ComCast for their backhaul and so ATT might actual be right to terminate the connection.

All of this suggests that ACC might have a bad route and some damaged equipment that is redirecting the packets. Over my 15 years with ACC these symptoms have occurred and every time it's hardware. Either damaged, wet, a floating ground, or some other "balance".

why not erlang?

I've developed some highly tolerant applications in erlang. The underlying justification was:
"if erlang drives phone switches why not payments" To this day it still holds. Some of the very basic tenants of erlang and Joe Armstrong's erlang guides still hold. My favorites are "early optimization" and "crashing".

The fact is... erlang is an elite language for the elite programmer. But too many programmers have selected erlang because they want to be elite which is clearly the wrong end of the telescope.

PS: haskell too.

vulcand - quick note

I like vulcand for a number of reasons:

I like the team (mailgun)I like the language and what that means (golang)Could run in a "scratch" container (because it's statically linked)warm config changes (uses etcd) While it's still in BETA the documentation is really weak. The sample docker files demonstrate port forwarding 8181 and 8182 but never really tell you what they do. When I first read this I assumed that I needed to have a port redirect from the firewall or some other proxy in front. WRONG!
It just so happens the documentation is just bad. This doc is also not great, however, they demonstrate forwarding ports 443 and 80 in addition.
Alternatives to vulcand could have been nginx (Russian) and HAProxy (France).  Trust but verify? I'll start with vulcand; thank you.

Trouble in iPhone land

My wife had been having trouble with her iPhone. Several Apple apps would crash immediately after launching. That included Safari and the camera. Additionally the phone was also running hot. Since the phone was a 128GB phone the 5GB free iCloud storage was simply not enough. Finally, plugging the phone into a MacBook did not force a backup.
The three questions that an Apple Genius is going to ask you [a] have you backed it up [b] have you [factory] reset it [c] have you updated the iOS? In my case the backup was not working but I had not spent too much time trying. Also, if the problem is a configuration that has been backed up then the backup may be rendered useless making the process take longer. It's all about risk/reward.
And so I proceeded to factory reset the phone. *sigh* As a result I lost 4 months of pictures. Arguably if I had been able to get the pictures onto iPhoto I would have saved myself a lot of grief. But that's a bit more complicated.
One of the complaints I…

coreos and etcd overhead

I finally managed to get my environment configured in GCE. Ultimately I want to look like:
This configuration is supposed to be pretty standard. The hardest part of the cloud-config was realizing that I was supposed to use $private_ip4 instead of $public_ip4. Many of the examples use the public IP and that is clearly wrong in EVERY case. Using the public IP might leave the system vulnerable to hackers.
Another note about etcd clusters is that he authors recommend that the etcd systems are left only to that function so that all system resources are left to that function. And when I created the workers I simply created etcd proxies. NOTE: if you omit the ?size=3 from he discovery URL then you have to be certain to include the proxy flag. If you include the ?size=3 then the 4th (or n+1) node will automatically become a proxy.
I now have a deployment of 5 machines.  Three in the etcd cluster and two workers. I happened to be looking at the CPU usage and I saw something strange: This gra…

Texting embedded code

In an upcoming post I want to provide several documents for which I'd prefer to embed as code. I've created a gist in github and copied the javascript "embed" code here:

(if you do not see code here then it failed.)

I have no idea if it's going to work. (I tried bitbucket but it does not offer any possibility to embed code.)

Docker Machine Providers - Review

I've been deploying CoreOS and Docker in this configuration:

And while I have had some success I have posted a number of questions and concerns with the Digital Ocean support team and most responses start with "We're sorry" and end with "ask the CoreOS team". I think that there is at least one serious flaw with DO's product and that is that every VM instance received a public IP address and there is no firewall. The side effect being that every system in my cluster has been under attack since it was deployed.

no network drivesno firewalllimited support So I'll be leaving them shortly. But what is interesting is that Digital Ocean supports Docker Machine. Docker Machine is Docker's way of creating Docker instances. Presumably there is some sort of shim between the host OS and the Docker container... While it might work it's an odd feature.
Of course if you have an OpenStack, VMware, or Vagrant then it makes perfect sense.  The shim will give y…

maintainable code

This YouTube video on maintainable code set my hair on fire. These guys usually have something interesting to say but on this subject they are a bit naive.
My comments:
I think we all strive for maintainable code but how we get there is debatable. Given your examples: PEP-8, for python, has become a religious nexus. The only good side to PEP-8 is that there is a tool the will evaluate the code although one can ignore the warnings and continue. Golang has an opinionated view of code formatting but does not go as far as PEP-8. (I've worked on teams where my peers spent more time criticizing peers for PEP-8 adherence that it just delayed execution). Javascript and PHP are going to be the hardest to get concensus from. It just is and I have no nice way to explain it.  Writing the documentation first is also false. While it's nice to say it's impossible in practice. See Knuth's books and essays on programming languages. Finally, the whole thing begins with requirements gath…

Good Enough

Recent hardware failures have made me realize that [a] I/we rely on technology, [b] when bad things happen to tech that I/we are intimately knowledgeable with worse things happen [c] not even Apple can make things simple if you're on a technology budget and even then see [b].

In recent weeks I had a Chromebook Pixel failure. This is an awesome machine with an Intel i7 and 16GB of memory. But when I had my failure I knew I was going to have to solve my problems myself. Even with the online and phone support. In my case both hangouts and google play music failed to start when clicked. The problem is that there are no logs to view and no popups or diagnostics. Not even a BSOD. In the end it took a powerwash+revert and a conversion to dev channel and then back to stable to get the machine back into operating order. Subsequently I emailed Pixel support and they could not help me. (I would have expected someone to ask me for a log or two) I was directed to the individual application fee…

What does a complete modern enterprise container-ship look like?

In a recent rancher labs blog post the author covered ELK (Elasticsearch, Logstash and Kibana). What caught my attention was the number of containers required to deploy the design. As I began to consider the deployment I realized that the 4-5 containers deployed to watch one container is a little overkill but of course you have to start somewhere as you transition from a legacy deployment to containers.

Assuming that converting to containers from baremetal or VM solutions has a net zero overhead cost then converting your enterprise from [a] to [b] should require the same hardware footprint/cost. Agreed?

What does a complete modern enterprise container-ship look like?

did I authorize this?

When did Google ask me for permission to be solicited for donations?

The cost of free

There is something to be said for sweat equity, however, at some point you will need to take a shower. I really like shows like The Profit and Shark Tank. There is a lot of reality distortion going on over there but unless you hit on something yourself it is what it is.

The thing about sweat equity is that it comes at the cost of the equity. What I mean it that if I'm one of those 10x programmers and I work an 16hr day and I'm still taking in an 8hr paycheck then I might only be making pennies on the dollar. This is particularly painful if the equity is not my own.

So while I look at all of the open and free-ness of projects like CoreOS and Kubernetes that free is not free. Building your own system or manually maintaining a CoreOS cluster according to the best practices or managing a Kubernetes cluster while free is still very expensive. When you're a 10x * 16hr accruing equity at pennies on the dollar spending that time on developing infrastructure when it should be vendo…

An iPad for all occasions

I have been a hardware geek as far back as I can remember ... and I have been predicting the iPad 13in for about 4+ years. At the time I was the director of a small software development  department for a payments company headquartered in Alabama. Since I was responsible for everything that had a CPU in it I also contributed to the design of the call center. At the time we were experiencing rapid growth with not much of a disaster recovery plan.  My hope was that Apple was going to expand their tablet offering so that we could (a) deploy a wifi network anywhere we needed it to be including just generally remote with a VPN (b) in-house we could reduce the cost of wiring phone, power, and wired networking (c) the enterprise support for managing users and remote destruction of corporate assets was a huge plus (d) by looping Apple or our hardware leasing company into the DR plan we could have hardware at the ready for on demand deployment to the DR location.
And now here is the iPad Pro in…

Modern Day VPN

I recently read a G+ posting about VPNs that made my skin crawl. It seems clear to me that the unapologetic entitlement crowd has taken and repurposed the RFC. Clearly VPNs have a wide variety of features, however, when it was initially conceived it was about linking private distributed networks. Then with lower cost crypto appliances it became part of the remote workers hardware inventory and then as it made it's way into the mobile device stack it allowed workers to be mobile.

Let's be clear, it was not meant to (a) obfuscate locate network traffic (b) improve QOS (c) bypass regional service restrictions... although this is what each of the VPN service providers in the Google Play store would have you believe. (clearly there is no money in the traditional VPN, and by using a VPN mom and dad won't see that you spend all your time on porn sites.)

And so there is no ambiguity... I did a whois the top 4 VPN providers on google play.

domain registered 2007domain registered 201…

"Where do the well to do buy their kids toys?"

Sitting in the parking lot at the local toy store I'm watch the various families enter and exit the store. The one thing in common is that they/we are all lower and middle income families. Since we spent more on gas than the toy we were exchanging I find myself asking some questions;

(a) where do the well to do buy their toys as to avoid disappointment in their children?
(b) are there any analytics associated with big box toy stores? 
I'm sure there are many more questions to as but it's not my specialty. If anyone knows Malcolm Gladwell it would be great to see him tear this apart.

UPDATE: my wife decided to purchase a coffee mug for my daughter's teacher. In particular since the school's theme was "superheros" it was fitting that the mug she purchased matched the theme.  What arrived was a "super stylist" and not a "superhero" mug. When I inspected the packaging it was clear the retailer had changed the barcode without regard to the …

debugging production problems with git and go

What follows is an accounting of a debug session I just completed. In the end the issue was not in my code but in a 3rd party library that was in turn effected because a service that it depended on was not running... but this is how I go there. (the stack tract stupid)

Logging into my server I realized that CoreOS had updated my Alpha channel server. It's a pain in my ass when that happens... and there are a number of side effects that I have not yet accounted for.

The login:
Welcome to Secure Shell version 0.8.34.
Answers to Frequently Asked Questions:
Connecting to
Loading NaCl plugin... done.'s password: 
Last login: Fri Sep  4 03:40:27 2015 from
CoreOS alpha (794.0.0)
Failed Units: 6

Crap, my chargeback se…

"more plausible than not"

While the NFL has stated that it is "more plausible than not" whether Tom Brady knew about, instructed, approved or participated in the deflating of the footballs used to win the semifinal game last season ... the NFL has reversed course on the punishment it handed out.
What a shame! Though professional athletes have been criticized for all the drugs, performance enhancement, salary caps, domestic abuse, contempt for the fan, fighting with fans.... the one thing that we have collectively agreed upon is that cheating is unacceptable. 
The NFL is not a democracy. You do not have a right to play in the NFL. It is a privilege and an opportunity. One for which you, the pro athlete, are paid very well. 
Besides being "more plausible than not" the New England Patriots have a history of bending the rules and even cheating. Bill Belichik should have been banned from the game for his participation in the taping of the Jet's practices. (or whatever that was). 
The Patriot…

reducing duplicate SQL in Go project

Recently I started writing reports for a client of mine. At first I thought it was going to be just a few reports but over time a number of things have happened. (a) more and more reports (b) even more reports (c) I'm getting lazy so my tools are starting to scale (d) I'm getting lazier and I also have a need to reuse code without cut, copy, paste. (e) the need to share code that the reports generate similar results when based on the same foundation.
While part of the implementation means using CTEs (common template expression) it's not the whole story as I implemented a complete reporting engine that exports to CSV, TSV, text tables, XSLX, JSON, DOT, go templates, and supports it's own DSL including loops and dynamic queries.
In my current implementation I store the SQL in bash, yes bash, shell scripts that export the SQL names like this:
export hello_cte="hello_cte (hello) as (select 'HELLO')" export hello=";with ${hello_cte} select * from hello&…

stop asking for my address book

Either we all know or strongly agree that the like of LinkedIn, Facebook and mySpace find novel ways to make money by marketing to me based on my likes, searches and the possible similarities to people in my circles. So of course they are going to ask me for access to my address book.
But stop fucking asking me. I'm not going to give it to you. And if you turn a phrase that get's me to inadvertently permit you access; not only will you lose my business (who cares right?) but I will also join and support any and all groups that will agree to legislate you into obscurity. As we all know; once they read my address book and slurp the data I will be haunted by my friends likes. 
Just a few days ago I did an Amazon search on small footprint computers like the Asus Chromebook and now everywhere I go I see ads for them. Someone sold me up the river.
The think about my address book is that it container both personal and business related contacts. Of the 12K contacts I only communicate w…

parsing go templates

Given a template file I want to parse out some variables so that I can prompt the user but I do not want to build my own parser or some set of regex although after playing with the go templates and parse tree it might be the best thing to do since I'm already making the syntax simple.

My simple document looks like this:

doc := "fred {{.Fred}} barney"

Here's my sample code (playground):

package main

import (

func main() {
doc := "fred {{.Fred}} barney"
t := template.Must(template.New("sample").Parse(doc))
log.Printf("%#v", t)
log.Printf("%#v", t.Tree.Root)
log.Printf("%#v", t.Tree.Root.Nodes)
for _, n := range t.Tree.Root.Nodes {
log.Printf("%#v", n)
if n.Type() == 1 {
log.Printf("%s", n)


The output was interesting but as I expected and similar to the sort of parsing I was thinking about and was demonstrated by Rob Pike in his Lexer video. The downside of using the tem…

Good reminders for software developers, program managers, and customers

It's not that I hate Agile it's that I hate the "Agile Process". It is worth repeating, the Agile Process shares the same vocabulary with the Agile Manifesto but is a lot more and is filled with the bias of it's supporters who are johnny come latelies who are in it for a buck and not your success.  As soon as they find a "better way" they are going to be knocking on you checkbook again.
While I have not gotten to a point where I have parity with 12-factor apps there seems to be something in the broader strokes that resonates with me. Of course this is yet another cycle (computer history repeating itself). Just as mainframes gave way to the midrange and ultimately the PC so did the frameworks that they operated in. As we dive deeper into PC development and managers want more productivity and reliability from DEV and OPS they are pointed back in the direction of frameworks. And I'm talking about the likes of JCL, CICS and so on. Currently things look …

Bosun and data collection

I like the bosun project but I have to take issue with the guy who packaged the docker version I'm running. Bosun is written in the Go language but has many dependencies that are not. In fact the stackexchange team created their own image that has several java dependencies. And my system, which was lean, is now like a VW Bug pulling a tractor trailer.

Chromebook VPN connection to a Watchguard Firewall

First and foremost the support professionals at WatchGuard have no interest in in taking my calls or emails. My employer is vested in WG firewalls but since the VPN issue is mine and not common to the company I have to deal with it and I'm not likely to get the support contract number or serialno. From that perspective WG is just not my friend.

Chromebook's VPN assumes (a) your certificates have been publicly blessed by a CA (b) using the VPN port and NOT 443 as is most common place. Getting past these limitations means a lot of manual labor. Here are some of the links I have been accumulating:

Chromebook VPN Setup (link)convert PEM to key file (link)Chromebook and OpenVPN server (link) And yet it still does not work.