Skip to main content


Showing posts from February, 2015

Haters hate - Git is in the way

I read an article where a programmer was recommending getting away from git. The strange thing is that it was well thought out and a plausible argument. While I would like to share a link to the original article it's not in my browser history.

I'm sitting here at my desk thinking all things git while I'm waiting for my 'git clone' to complete when it hits me. This guy never used IBM's Orbit. And there is a good chance he's never used RCS or even SourceSafe. The world of VCS was an awful one.

Linus Torvalds made a number of comments in the last few years that Git was meant to be difficult. I feel confident that this was Linus being Linus but if true then he has a point. Git is complicated. Git should be easier. Frankly what we actually do and use Git for should be trivial. The hardest part of Git is rolling back and cherry picking.

Many of the Git GUIs make things easier. Tower, for example, let's you enable "git-workflow" (a standardized git w…

bash one-liner; copy and make unique

As part of the ongoing process of correcting my lost images in iPhoto I offer this one-liner:
find . -name "*.mov" -exec env file={} bash -c 'base="$(basename "$file")";dir="$HOME/Desktop/Movies";pre=$(echo $RANDOM);cp \"$file\" \"$dir/$pre-$base\"' \;  > work . ./work  Here's the problem: I have a folder called "Master" and in that folder there are many sub-folders. Each folder path represents the year/month/day that the images were imported. Now that I am trying to import some MOV files it appears that the iPhoto importer does not like duplicate filenames in the same day. This was evidenced when iPhoto failed to import many MOV files without an explanation.

So I tried to find and copy the files with a simple find command. Unfortunately the source path included spaces and since the copy was into a ingle folder it resulted in duplicate filenames so instead of 2200 files I had about 1000.

The command a…

Better Interviewing Tactics

Job interviews start with the Posting.

Some years ago when I was starting to get into erlang (for the second time; the first time was a failure) I recall reading an article that suggested [prarphrazed] It is better to post a job for an erlang programmer instead of a java as you are (a) going to get fewer respondents (b) of higher quality and aptitude. I cannot find fault with this approach except that posting for an erlang position and then using C#, java or perl and you might actually keep the candidate if you manage to win the day.

In the past I had interview questions which I would say have their origin in the Silicon Valley way of thinking. Of which the author of this article [Phone Interviews] would have you believe is bad; and today I agree as my initial opinion has changed. In the end I realized that I have a good intuition when it came to interviewing. I think I ask questions that cover various median knowledge and aptitude; and some that are meant to gauge the interpersonal s…

iPhoto nuggets

I've been complaining about having to import 54K images into iPhoto and as it happens there are some things I need to remember the next time this happens.

To recap:

old, out of warranty MacBook diedI had copied all of the images to a HDD recently but not 100% currentMy BackBlaze account appears to be currentMy Google+ (aka Picasa lite) is also current So I bought a 2TB HDD from MacSales. Formatted and Carbon Copy Clone'd my 256GB SSD onto the new HDD and booted. Next: I copied all of the "master" images from the manual copy from the external drive to my desktopI imported all of the images into my new iPhoto installation (took 5+ hours)iPhoto appeared to hang but was probably creating thumbnails and rejecting my MOV files But here is the bad news. The "master" folder is partitioned by IMPORT datethe source images were spread over timethe new iPhoto MASTER folder was a single folderiPhoto refused to import 2.6K MOV files that were in the MASTER folderI had to ex…

The Sticky Wrapper class

I was watching a live programming session with Andrew Gerrand and Brad Fitzpatrick where they hacked an http_2 client; when they demonstrated something called a stickyErrWriter. If you've ever written a block of code which calls a group of related functions with potential error return functions then you got gobs of  `if err != nil` after each call. Yuk!

An instance is declared here: = bufio.NewWriter(stickyErrWriter{tconn, &cc.werr}) This is the idiomatic way Go treats readers and writers; and similar packages.

The stickyErrWriter structure overrides the Write method so that when the base function, in this case Flush(), when Flush() calls Write() it's the sticky Write that is called which checks the current error state; and if there is no error then the actual Write() is called.

func (sew stickyErrWriter) Write(p []byte) (n int, err error) { if *sew.err != nil { return0, *sew.err         }         n, err = sew.w.Write(p)         *sew.err = err return }
This is a great exa…

The Achilles heal of micro services

I've been doing the devops thing for as long as I can remember. It's always been a part of the job. Now as I contemplate the new landscape of micro services and it's impact on the teams, business and the clients I'm starting to think about where the value comes from.

Just about 7 years ago I designed and built a POS terminal. It included all of the usual peripherals which were connected via USB. And while micro services were not popular at the time the complexity of identifying and connecting to each different device meant it was well suited for micro services. But the one thing that made this design successful was that each service was built in it's own identical pipeline. That the build pipeline was identical was probably the single most important detail.

Now as teams are moving from monolithic applications to micro services they are also looking at custom build pipelines per project. While CIs like Jenkins, Travis, and Drone offset this by putting the build deta…

CoreOS Fleet vs Docker Swarm

Frankly is there is very little difference between the two. One thing that stands out is the ability to connect to a container instance and ssh into it. Although I'd give ease of use to Docker Swarm; fleet and fleetctl are just as capable. I also like the two features that docker demonstrated. affinity and memory. But then Fleet has a solid config file that fits nicely with cloud-config. Docker's robust CLI options are nice but not very pragmatic. If your goal is automation then config files are your friend. I'm sure that Docker has a swarm config file. (I'm tempted to do a search; but then it seems easy enough.)

Docker Compose or Fig repackaged

Watching the demo on their announcement page I'm led to believe that docker compose is simply a repackaging of fig. Here is a link to the announcement when Orchard/fig was acquired.

Here is the original demo project from the fig home page:
web:build:.command:python app.pylinks:-dbports:-"8000:8000"db:image:postgres
And here is the project from the Compose announcement:
web: build: . links: - redis ports: - "5000:5000" redis: image: redis
It's easy to observe that the tags are the same even though one uses redis and the other postgres. It is sort of silly that the application port numbers are different. In fact as you dig deeper in the fig home page you'll see a sample that is even close that these two.

And then as I looked at the header of the fig site I see the message:
Fig has been replaced by Docker Compose, and is now deprecated. The new documentation is on the Docker website.
I'm not sure whether I'm right and fig was simply r…

Updated Thoughts on Flow Base Programming

My implementation of a flow based framework has been stalled for a few months.  It's nothing serious but a refactoring is planned. I've also run into a few new learnings that have triggered more ideas.

(I've already mentioned FlowHub and NoFlo so I'll leave them out).

GoFlow has a good design and uses some of the same ideas that I published some time ago. In this case I'm referring to anonymous structure attributes. One thing I think might make it more extensible would be moving the channels to the structure instead of hiding that info. There is a strong separation between the node and the node graph. In my implementation I used a JSON file to define the network structure where GoFlow uses straight configuration as code.

Following up I'm also interesting in gnatsd. Gnatsd is a message queue broker and client library. What makes it outstanding is it's implementation of netchan. The netchan was initially a Go core library in the stdlib but was later moved to t…

first impression of rancheros

RancherOS is plenty interesting. What sets them apart from the other micro Linux' like CoreOS is that it's really small. It's nothing more than a kernel with a few containers where the Linux services are running. The rest is meant to be userspace.
At first glance everything is really small. I think the presenter say 19GB. However, since the project is so young it's going to gain some weight as they get to the balance point between size and needed functionality. It's also very incomplete as the docs indicate the upgrade facility is not implemented yet.
One interesting feature or omission is the persistence. The entire OS is run from memory and persisting anything requires formatting the drive. So it's easy to conclude that the documentation and the feature set are also in flux.
The rancher project is interesting and functional. The last time I examined it I was pleased to get things going. It functioned in places that Deis did not. My guess is that Rancher wants…

big numbers of things

I had no idea what to call this post but I knew I wanted to talk about numbers of things. In a previous post I was getting heated about the size of my HDD and the cost in time and money to backup and restore everything I needed. In fact I underestimated the cost.the new drive cost $119I needed a new license for Carbon Copy Cloner $20Moving from my old MB to my backup MB means also upgrading iLife and iWork $20/per In terms of numbers I thought we had about 25-30K raw images in iPhoto. It turn out we have over 54K in images and I know that there were some others in folders that might not actually be part of the "master" since it's been upgraded over the years and the folder structure has changed.
And then there are the emails. I'm looking at my "mail" activity window and the number just keeps climbing. So I logged into my gmail account and I see that I have "sent" 24K emails. I would like to delete the emails because there isn't much point in k…

Fedora 21 - Cloud

The Project Atomic documentation is pretty clear but I had a number of challenges when deploying both Fedora-21 Cloud-Base and Fedora-31 Cloud-Atomic on my vagrant installation on OSX.

First download the files. Notice that the files are .raw.xz files which means that the files are compressed raw. So they need to be unzipped first:
xz -d Fedora-Cloud-Base-20141203-21.x86_64.raw.xz This command is going to replace the file with an uncompressed version of the file and it will be a raw type.

Now you have to convert the file from raw format to VDI for virtualbox:
VBoxManage convertdd Fedora-Cloud-Base-20141203-21.x86_64.raw Fedora-Cloud-Base-20141203-21.x86_64.vdi --format VDI I coped the resulting VDI file to my virtualvm folder but you can always deploy it from wherever you are. Note that this image is NOT an ISO so there is nothing to install. It's already in a base configuration. This also means that the default user account has not had it's password set so you will need to do …

Embedded Lua and/or tcl in golang

I have been working on an installer based on JSON using golang.  The two projects are macroinator and installinator. The macroinatorproject is a refactoring of the original installinator project which will make it's way back into the installinator project very shortly.

In both cases the DSL is very loosely based on Puppet's DSL and implemented in a JSON format.

This design comes at the end of a very long argument with myself about the value of DSLs in the first place. I have been researching both Lua and tcl embedded libraries just to see what was involved. On the one-hand the tcl history document describes the justification for the implementation of the tcl language and then goes on to say that Lua might be a better modern choice even though Lua is incomplete in terms of tcl's scope.

Today I was sitting in my car in a parking lot contemplating all things embedded in golang and as I was considering Lua and tcl I waffled back and forth several times. On the one hand tcl and…

MacBook Air Upgrades

Now that my MacBook Air (MBA) is out of warranty and still humming strong... and after having to repurpose my backup MacBook it might be time for a hardware upgrade. I would prefer a complete hardware refresh but since that's not option let's do the exercise anyway.

11" or 13" that is the question. The price difference between the stock 11" and 13" is just $100. And for that you get an extra 2" diagonal, SD card slot (I like the mini drive). On both the 11" and 13" you can boost the CPU and RAM for $250 but to get to 512GB disk you're going to shell out another $400.

And for comparison... I have my out of warranty MBA with 64GB of SSD. And I'm constantly running out of disk, memory, and horsepower. However, since I'm trying to use my MBA in an editor only mode I could just upgrade the disk. Looking at OWC they offer 1TB of SSD for just over $500. That's amazing!

PS: The new MB's have great battery life. I don't know wh…

Know your stack

Superfish on Friday insisted its own code is safe and said the security flaw was “introduced unintentionally by a third party.” 
Whether it's true or not it's still something to fear when linking libraries or nor vendoring or code reviewing. Etc...

OSX when simple is not simple enough

It happened again. I lost another MacBook.  This was the oldest machine with the largest HDD. The 1TB was toast and so was the machine. I could not pull the drive and mount it as an external drive and I could not put the factory drive back into the MacBook and boot. It was a double loser.

The only thing I can do at this point is repurpose my MacBook unibody. Sadly I had previouslypulled the 500GB drive, placed it with a 1TB drive and then replaced it with a 250GB SDD. Now this 250GB drive is not going to hold enough if the pictures to be useful.

This had me thinking.  What would happen if I had 1TB of pictures and a MacBook Air. Talking to one of the Apple Genius' today he told me that (a) I would have to partition my pictures onto thunderbolt drives and (b) that 4GB was not enough memory for the number of pictures I had.


But there are any number of problems with this strategy:

iCloud does not support this strategyiCloud is expensiveSwitching to multiple libraries is r…

Partitioning Systems and Microservices

When you are building a system from scratch it's easy to build a CI/CD pipeline that builds, tests and deploys the project and it's dependencies in a single pipeline. From this vantage the project is always a fresh install and if you do things right executing migrations are also pretty simple.

But now what happens as some of the systems now become shared systems with their own pipelines and dependencies? Take, for example, databases or message queues. At some point the MQ is going to become a shared bus and the database will no longer scale in the largest VM... so now what?

This partitioning of services and micro services will eventually force you to return to the days when you had to install everything manually and one at a time. While the tools may be upgraded to buttons the risks are no different and the number of moving parts are the same.

The only way this gets better is if the pipeline is truly partitioned and automatic. Automating a rollback could be just as important.

First Impressions Ubuntu Snappy

There are a number of nix based appliance OS' out there. Some of my favorites include CoreOS, NixOS, SmartOS, ProjectAtomic ... I could include the unikernels too but I am not. The latest team to enter the space is Ubuntu Snappy.

Installing snappy on my OSX machine with VirtualBox and Vagrant was a modest success, however, getting to the next step was a serious challenge. Installing docker was easy enough:
sudo snappy install docker however the default system seemed to be missing vim so I was unable to edit the /etc/hosts file in order to correct a host not found error when snappy was referring to itself by hostname. When I tried to install vim I received an error that vim could not be found.

I was pleased with the 5 second boot time and the snappy script felt simple and natural, however,  overall it's still a bit of a square peg. I'm hoping they do a better job in the near future.

G00V00 and G00V00P000

The IBM mainframe OS/JCL implemented a file versioning system named G00V00 or something like that. In today's dotted version numbers they'd need to add a P0000 to keep track of the patch number too. Considering that IBM only allowed for two digits I wonder if that was 00-99 or 00-FF and whether 99 versions of the same file was sufficient before skipping to the next generation? In today's world we commit code whether it's ready or not leaving the build system to determine whether the code is ready or not. In this model patch numbers are ever increasing. Also, with the dotted numbering scheme performing proper numeric ordering is not as simple as a lexical string compare.

If I get to implementing my design for flow based programming by storing the code in a DB then G00V00 and possibly P0000 might makes sense.

QUESTION: Does anyone have some links to the original docs?

The Perfect Programming Language

I've just read a two articles. One talked about learning OCaml instead of Haskell and another talked about switching from Mozilla's Rust to Nim. One in particular said that there was "no perfect language" and while I'm on the eve of presenting at go-Miami with a strong admiration for Google's Go there is nothing like good old-fashioned C.

This post has started to get away from me. 

C is still the perfect language. It has the history and the tools that keep it in the #1 position. C is also the foundation for many other languages and there are interfaces to many more.

And it got away again

In my ideal environment I would be implementing a framework in C and then deferring the task to another language to something like embedded tcl or even lua. One excellent example of embedded tcl is fossil-scm. And an example of embedded lua is Adobe's Lightroom.

And so I like Nim because it generates C code which could then embed tcl or lua; and why not both. Libraries can …

deploy fossil-scm in a docker container on OSX with boot2docker

The title says it all. In this experiment I wanted to create a docker container that had a running fossil-scm instance and the docker container should be as small as possible and therefore should be running from a scratch docker image.

(a trusted scratch image is about as small as an image can get. It's essentially turns a docker run command into an application.)

NOTE: setup boot2docker:
boot2docker init
boot2docker up
boot2docker shellinit | source(There are a few other versions of the shellinit command such as back ticks or $(cmd), however, this version is the most compatible and works with fishshell which the others do not.)

First and foremost I am using OSX version 10.10. Mavericks is nothing special in the equation but I'm just mentioning it. Next I've installed boot2docker 1.5.0 from the distributed binary release for OSX. (I'm not going to show you how to do that here. Some things ya just gotta know.)

Make a working folder for the project and make it the current…

"Compose with functions, not methods"

I was drawn to a link titled "Compose with functions, not methods". It was a talk that Rob Pike (a Go Author) gave in 2012. While I had an idea what he was talking about I looked at the slide and read and read it trying to gleam what he meant by the text.

I then read the method and function definitions in the Golang documentation called "Effective Go". And I still did not understand what the authors were getting to. Then I read the language reference and found the missing pieces.

Function: "A function declaration binds an identifier, the function name, to a function"

Method: "A method is a function with a receiver. A method declaration binds an identifier, the method name, to a method, and associates the method with the receiver's base type."

And now it all made sense. By preferring functions instead of methods one can use interfaces as part of the composition formula in the way that Go implements Read() across different packages. And the same…

If I had enough time to experiment

If I had enough time to experiment then there are a few things I'd want to do:

Build a proper pipeline from NixOS, Nix, NixOps, HydraDesign and build a pipeline to include AppScale so that I can concentrate on application development instead of framework or pipelineBuild a pipeline from's open source offeringBuild a go pipeline with CoreOS and a cluster of Raspberry Pi servers. ($35 a device and 10 servers for a CoreOS Cluster... fun)Design and build a Docker pipeline and framework.. I like the Apcera gnatsd docker/dockerfile build and the Apcera policy engineMicro services are interesting but a bit of a challenge when you consider the guest OS. Phusion is still creating fudetcd 2.0 has been released and CoreOS' rocket, flannel, and appc are around the corner. They need to be investigated as I want to experiment with trust and a design I have for an HSMI like monitoring but there are so many heavyweight tools. Prometheus needs some investigationinvestigate Docker n…

Share this and that

Sadly Douglas Campbell decided to disable comments or I would have responded directly to his post. I don't want to say that he's made a discovery, however, sharethis seems to have a foot planted in the docker trough. I'm only pointing this out because these guys seem to be very devops oriented. Once you look at their career page it's almost immediate.

There is a lot of good to be said for docker.

it is much like a chroot with more partitioning (security)but it supports many different linux flavorsso the host and the guest do not have to be the same brandCoreOS is a perfect host OSdevops is easier if the containers are considered part of the pipeline channel (see CoreOS channels as an example)support for sidekick and ambassador patternsapplications like machine and fig create networks of images through compositionimages are sharable through a registrywith reputation based recommendationsso SMEs can create proper images (i.e.; mongodb) On the downside there there is confus…

Hot code replacement - Erlang, Elixir and so on

I have to make this quick. Hot code replacement is interest, fun, and very complicated. It's so complicated it's not worth the effort and in the few use-cases where it is a requirement might and should be in the far minority.

The first problem with hot code replacement is scope and idempotence. Hot plugging a small module that is only part of the transaction cannot be reproduced. Creating monolithic modules might as well be implemented as blue/green.

Second, when hot replacing code there is challenge to migrate state from the current module to the new one. Since a module can have more than one process maintaining state moving the state from the current to the new and preventing forked transactions... it's all too difficult. (it is possible to store state in a cache like redis but that breaks a few idioms like the benefits of immutability)

Next, while hot code replacement is considered a rockstar level achievement it's complicated and while it can be made to be reliable…

Real Programmers ...

Real programmers can write assembly code in any language. -- Larry Wall Here are just a few links...

The Story of Mel, a Real ProgrammerReal ProgrammerIs it true that “Real programmers can write assembly code in any language.”?  One thing for sure... long before TDD the testing and debugging trend was (a) logging, (b) asserting, (c) single step every line of code (d) look at the assembly version of the code.
A few of the things I really liked about writing in C on DOS systems: the smallest binary executable was only about 4 bytes. In an EXE file there was a 2-byte token, segment index, segment definitions, and then the code segment with it's setting the AX register and a call to interrupt 21h. In a COM file it was just the code segment...Since the only linking you would do were libraries you identified... there was no bloat and the code could be dumped. Reading the code in the assembly version often times made it easier to debug or desk check. With modern programming languages ther…

MongoDB, Partitioning, Sharding, Benchmarks, and application design

10 years ago I ran some benchmarks for an OLTP application I was building. One of the driving design requirements was response time.  The aggregate response time had to be shorter than 250ms. The initial implementation relied on 100% of the work being performed in the application which meant that the network was chatty as the application retrieved data from over 100 tables. The multiple network roundtrips, latency, and transaction lock escalation, not only increased transaction time but also limited the number of concurrent transactions.

The second implementation cached data in the app server. That design also failed. The third implementation, which was elected to production, divided the work between the application server and the database. The application managed the protocol impedance and transaction validation between the payment network and the database. The business logic was implemented in the database  as series of runtime configurable stored procedures. While using T-SQL for e…

Software Architect - Best Job? For how long?

This article popped up again. It was written in 2013 and is mostly valid, however, the winds are shifting.

About 15 years ago I was having a conversation with my stepfather. The topic of conversation was the investing, the stock market, brokers, and the internet. He said: there was no longer any reason to have a broker; with the advent of internet search the investor had access to the same information that the broker did.

As I was reading a few emails this morning I've realized something very similar about architects. I consider myself an architect because I have been architecting greenfield projects for bulk of the last 20-30 years. Any while I try to keep abreast of all the new tech sometimes it's a matter of being the first one to press SEND.

Being a successful architect is more than reading Hacker News, SlashDot, TechCrunch, etc... It's knowing when (a) not to send, (b) when to review and provide a distillation of the cost/benefit. (c) and when to yell "stop the p…

Getting Started with Kubernetes

I'm trying to go through the Kubernetes Getting Started and I'm not getting started. The team has provided a number of ways to get started but none of them are working... let's get started:

GCE - Google's Compute Engine would be a great choice but I'm trying to develop some ideas about how useful Kubernetes is. In that case I'm just hacking so I'm not interested in having to setup accounts and payments just for the sake of experimentation. I will probably come back to this as I will explain...CoreOS was the first strategy I tried. I have had tremendous success with CoreOS and CoreOS clusters. I've distributed fleet clusters, docker and rocket containers. I've also deployed CoreOS clusters using the CoreOS-vagrant tool. So this seemed like a no-brainer. However, it failed. The prerequisites included vagrant, virtualbox, net-tools, and either build or install the kubernetes binary. (a) they were not clear about the source or target of the net tools. I …


UnQLite is the NoSQL variant of SQLite. I suppose if I needed a document store I'd probably go after one of the big boys like RethinkDB, MongoDB, or even Oracle. There are even a plugin or two for Postgres.

And if I needed a key/value store there is Riak, Redis, a number of embedded libraries and of course there also SQLite since k/v stores are just the 5th normal form (or something like that).

But if I needed a small pragmatic storage engine for storing documents or some UnSQLite -like data... I'm not sure I'd ever go embedded with UnQLite. I suppose that's why the SQLite author never completed his UnQLite project.

Things that make me happy in go(lang)

mattbasta pasted that there were some things that he hated about go but that never provided a way to receive comments. Shame. It's an interesting topic.
Project Layout This simply means he does not understand the tools and their potential. And when those tools fail there are others to help like: glide, godeps, golo(mine), goli, make.go and a few others.
Relative Package Imports That's just a stupid idea and is similar to the first topic. Namespaces are important and until someone finds a way to make global namespaces simple then that's just the way it's going to be. It would be interesting to have some sort of local registry but then by the time you do that you end up with a filesystem based on some pattern. and so the argument gets recursively redundant.
The Big Community Mess I'm not sure how the first two items are the communities problems. Frankly, GOPATH, GOROOT, GOARCH, GOOS are the one-stop shop. As far as NSQ having a gnarly makefile... you've never use…

Web apps and frameworks in Go

I'm trying something new. I'm going to write this review while I'm getting acquainted with that which I'm reviewing. Granted this is not an in depth deep dive but it is the first blush. And while they say don't judge a book by it's cover, in this case it's README file,  there are simply too many 3rd party web frameworks out there for go.

The granddaddy of them all is revel. What makes Revel great is that it feels feature complete. The menu in it's manual makes it pretty clear what it supports. The only thing I do not see is some sort of integrated ORM. It's not compulsory but it could be helpful. I've implemented a few projects and while they work they are not very gratifying.

When creating a REST API server it seems obvious that the handlers and documentation should be generated. There are a number of tools for that. For the moment the defacto leader seems to be swagger. From the swagger toolchain (nodejs) there are other tools that capitalize …

The Raspberry Pi 2

The latest Pi looks to be a box I can wrap my ARMs around. pun intended. About 15 years ago I built a cluster of Linux servers that had fanless Intel CPUs that clocked in a around 900Mhz. Maybe that's the sweetspot or material dynamics. If I recall each system cost me about 350-500 to build. The motherboard was probably around $125 and the bulk of the expense was the case, power supply, cdrom, and hard drive. With the new Pi and the state of art injection molding and new plastics I could build the same cluster for a fraction of the price.

One negative attribute is the RAM. The Pi-2 only comes with 1GB and while that might be ok for a small OS like CoreOS, busybox or scratch I don't think it'll handle many containers or something that can act as a POC. However a cost analysis should be performed because maybe an adhoc Google Compute Engine cluster would make more sense.