Saturday, February 28, 2015

Haters hate - Git is in the way

I read an article where a programmer was recommending getting away from git. The strange thing is that it was well thought out and a plausible argument. While I would like to share a link to the original article it's not in my browser history.

I'm sitting here at my desk thinking all things git while I'm waiting for my 'git clone' to complete when it hits me. This guy never used IBM's Orbit. And there is a good chance he's never used RCS or even SourceSafe. The world of VCS was an awful one.

Linus Torvalds made a number of comments in the last few years that Git was meant to be difficult. I feel confident that this was Linus being Linus but if true then he has a point. Git is complicated. Git should be easier. Frankly what we actually do and use Git for should be trivial. The hardest part of Git is rolling back and cherry picking.

Many of the Git GUIs make things easier. Tower, for example, let's you enable "git-workflow" (a standardized git workflow meant to make things easier).

Friday, February 27, 2015

bash one-liner; copy and make unique

As part of the ongoing process of correcting my lost images in iPhoto I offer this one-liner:
find . -name "*.mov" -exec env file={} bash -c 'base="$(basename "$file")";dir="$HOME/Desktop/Movies";pre=$(echo $RANDOM);cp \"$file\" \"$dir/$pre-$base\"' \;  > work
. ./work 
Here's the problem: I have a folder called "Master" and in that folder there are many sub-folders. Each folder path represents the year/month/day that the images were imported. Now that I am trying to import some MOV files it appears that the iPhoto importer does not like duplicate filenames in the same day. This was evidenced when iPhoto failed to import many MOV files without an explanation.

So I tried to find and copy the files with a simple find command. Unfortunately the source path included spaces and since the copy was into a ingle folder it resulted in duplicate filenames so instead of 2200 files I had about 1000.

The command above will:

  • locate the MOV files in the source tree
  • copy the file from the source folder to the destination folder
  • and rename the file by inserting a random number as the filename's prefix
Well, it's not pretty but it works. I wonder if I should have been able to perform this function using the installinator?

Better Interviewing Tactics

Job interviews start with the Posting.

Some years ago when I was starting to get into erlang (for the second time; the first time was a failure) I recall reading an article that suggested [prarphrazed] It is better to post a job for an erlang programmer instead of a java as you are (a) going to get fewer respondents (b) of higher quality and aptitude. I cannot find fault with this approach except that posting for an erlang position and then using C#, java or perl and you might actually keep the candidate if you manage to win the day.

In the past I had interview questions which I would say have their origin in the Silicon Valley way of thinking. Of which the author of this article [Phone Interviews] would have you believe is bad; and today I agree as my initial opinion has changed. In the end I realized that I have a good intuition when it came to interviewing. I think I ask questions that cover various median knowledge and aptitude; and some that are meant to gauge the interpersonal stuff... it is intuition after all.

The Phone Interview also links to two other interview articles that I think I might find interesting

The GitHub Hiring Experience (link)

The post makes an interesting point that other techniques might not. First Contact seems to be "brick and mortar" meaning that it's not your typical internal or 3rd party recruiters that busy "dressing flesh" in order to get candidates through the pipeline. It's an easy way to pre-screen potential candidates.

As for the interview process. That's a bit much too. In this case the candidate indicated that s/he met with 10-12 interviewers. If this is a typical 10a-4p interview day with an hour for lunch that's 10 people over 6 hours if everyone manages to show up on time and is prepared. That's going to work out to an average of 2 interviewers per meeting and a group or just the manager at lunch.

One thing that was not mentioned was "who" were the interviewers? Were they managers, architects, peers in the same or different departments, were any of the interviewers interns, or just learning to give interviews. I'm not sure that any of this works. At some point you have to savvy enough to:

  • stop doing the "ice cream man" dance
  • start selling the company, manager and job 
And that is simply impossible in an hour when there are so many departments, teams, and personalities involved.

Project Based Interviews Instead of GitHub (link)
"No one should ever require a Github profile in favor of a resume"
Resig makes a number of assumptions about "custom projects" for the specific interview. This is something I did when I presented candidates with "the impossible quiz". I suppose if you're interviewing at Google and you want to show your chops then doing a special project might have some value. It's also possible that recording a live pair programming session might have the same outcome. But these are exceptional cases and not the norm and it's not going to demonstrate the candidates interest in you market or stack. There are simply way too many variables for a project to be valuable.


I have been known to provide a few programming assignments, both in advance or at the time of the interview. And from that POV I've asked decision questions. "why a hash instead of a list" etc... That usually leads to a better outcome.

Resig goes on to challenge the pressure of an individual interview. (I blew a few of those with the like of Google and Amazon). Frankly, unless you are simply desperate for a job (been there too) nervousness or lack of confidence can be an indicator for that candidate's abilities. And sadly with a little too much confidence the candidate might not have enough practical experience to know the difference. But a nice balance is good.

iPhoto nuggets

I've been complaining about having to import 54K images into iPhoto and as it happens there are some things I need to remember the next time this happens.

To recap:

  • old, out of warranty MacBook died
  • I had copied all of the images to a HDD recently but not 100% current
  • My BackBlaze account appears to be current
  • My Google+ (aka Picasa lite) is also current
So I bought a 2TB HDD from MacSales. Formatted and Carbon Copy Clone'd my 256GB SSD onto the new HDD and booted. Next:
  • I copied all of the "master" images from the manual copy from the external drive to my desktop
  • I imported all of the images into my new iPhoto installation (took 5+ hours)
  • iPhoto appeared to hang but was probably creating thumbnails and rejecting my MOV files
But here is the bad news.
  • The "master" folder is partitioned by IMPORT date
  • the source images were spread over time
  • the new iPhoto MASTER folder was a single folder
  • iPhoto refused to import 2.6K MOV files that were in the MASTER folder
  • I had to extract the MOV files from the source and drag-n-drop them on iPhoto (not confirmed yet)
  • retrieve the missing files from the BackBlaze backup
  • restart Picasa backup
  • restart BackBlaze backup
One interesting observation. My wife needed to complete her timesheet for her work. The spreadsheet was converted to Apple's Numbers but now that her Mac was in flux she was forced to use Google's spreadsheet app. I thought the experience was better than using Numbers. Numbers is a good spreadsheet program but I'm totally over this PC is the center of my universe thing.

The Sticky Wrapper class

I was watching a live programming session with Andrew Gerrand and Brad Fitzpatrick where they hacked an http_2 client; when they demonstrated something called a stickyErrWriter. If you've ever written a block of code which calls a group of related functions with potential error return functions then you got gobs of  `if err != nil` after each call. Yuk!

An instance is declared here: = bufio.NewWriter(stickyErrWriter{tconn, &cc.werr})
This is the idiomatic way Go treats readers and writers; and similar packages.

The stickyErrWriter structure overrides the Write method so that when the base function, in this case Flush(), when Flush() calls Write() it's the sticky Write that is called which checks the current error state; and if there is no error then the actual Write() is called.

func (sew stickyErrWriter) Write(p []byte) (n int, err error) {
        if *sew.err != nil {
                return 0, *sew.err
        n, err = sew.w.Write(p)
        *sew.err = err

This is a great example.

Another idiomatic example is the NopCloser().
"NopCloser returns a ReadCloser with a no-op Close method wrapping the provided Reader r."
The NopCloser provides a Close() method that does nothing. It's for io.Reader instances that do not provide their own Close() method but where a Close() is required.

The Achilles heal of micro services

I've been doing the devops thing for as long as I can remember. It's always been a part of the job. Now as I contemplate the new landscape of micro services and it's impact on the teams, business and the clients I'm starting to think about where the value comes from.

Just about 7 years ago I designed and built a POS terminal. It included all of the usual peripherals which were connected via USB. And while micro services were not popular at the time the complexity of identifying and connecting to each different device meant it was well suited for micro services. But the one thing that made this design successful was that each service was built in it's own identical pipeline. That the build pipeline was identical was probably the single most important detail.

Now as teams are moving from monolithic applications to micro services they are also looking at custom build pipelines per project. While CIs like Jenkins, Travis, and Drone offset this by putting the build details in the project source the only way to easily refactor back and forth with the pendulum is if that configuration is as similar as possible.

Microservices are a great way to separate concerns, deliver working software, and provide loose coupling. Taking a note from Apcera I also like the BUS or message queue as a way to implement policy. This strategy only works if there is discipline in the build and an underlying services framework.

Thursday, February 26, 2015

CoreOS Fleet vs Docker Swarm

Frankly is there is very little difference between the two. One thing that stands out is the ability to connect to a container instance and ssh into it. Although I'd give ease of use to Docker Swarm; fleet and fleetctl are just as capable. I also like the two features that docker demonstrated. affinity and memory. But then Fleet has a solid config file that fits nicely with cloud-config. Docker's robust CLI options are nice but not very pragmatic. If your goal is automation then config files are your friend. I'm sure that Docker has a swarm config file. (I'm tempted to do a search; but then it seems easy enough.)

Docker Compose or Fig repackaged

Watching the demo on their announcement page I'm led to believe that docker compose is simply a repackaging of fig. Here is a link to the announcement when Orchard/fig was acquired.

Here is the original demo project from the fig home page:
  build: .
  command: python
   - db
   - "8000:8000"
  image: postgres

And here is the project from the Compose announcement:
  build: .
    - redis
    - "5000:5000"
  image: redis

It's easy to observe that the tags are the same even though one uses redis and the other postgres. It is sort of silly that the application port numbers are different. In fact as you dig deeper in the fig home page you'll see a sample that is even close that these two.

And then as I looked at the header of the fig site I see the message:
Fig has been replaced by Docker Compose, and is now deprecated. The new documentation is on the Docker website.

I'm not sure whether I'm right and fig was simply rebranded compose and I suppose that doesn't really matter. But there you have it.

Fig, and now compose, was interesting for simple clusters but that co-dependence makes that cluster less reliable. That's just the facts. Also, these containers need to exist on the same host instance so there is no way to take this configuration to the next level and link remote services. At least I do not see the documentation as yet.

Updated Thoughts on Flow Base Programming

My implementation of a flow based framework has been stalled for a few months.  It's nothing serious but a refactoring is planned. I've also run into a few new learnings that have triggered more ideas.

(I've already mentioned FlowHub and NoFlo so I'll leave them out).

GoFlow has a good design and uses some of the same ideas that I published some time ago. In this case I'm referring to anonymous structure attributes. One thing I think might make it more extensible would be moving the channels to the structure instead of hiding that info. There is a strong separation between the node and the node graph. In my implementation I used a JSON file to define the network structure where GoFlow uses straight configuration as code.

Following up I'm also interesting in gnatsd. Gnatsd is a message queue broker and client library. What makes it outstanding is it's implementation of netchan. The netchan was initially a Go core library in the stdlib but was later moved to the experimental repo and later deprecated. There have been other; similar; implementations but there are two that are notable. (a) from the docker team, (b) from gnatsd.

I have been thinking of experimenting with Go's native RPC structure. Since it runs on the http rails I'm hoping that it's going to be concurrent. But one project that might step up the game a little is gRPC. This project extends RPC with pluggable protocols and a performance gap with similar libraries. While gRPC supports multiple client languages my Flow implementation is limited to a a global namespace that might have minor remote services. It is possible that the services could be implemented in other languages but unlikely.

first impression of rancheros

RancherOS is plenty interesting. What sets them apart from the other micro Linux' like CoreOS is that it's really small. It's nothing more than a kernel with a few containers where the Linux services are running. The rest is meant to be userspace.

At first glance everything is really small. I think the presenter say 19GB. However, since the project is so young it's going to gain some weight as they get to the balance point between size and needed functionality. It's also very incomplete as the docs indicate the upgrade facility is not implemented yet.

One interesting feature or omission is the persistence. The entire OS is run from memory and persisting anything requires formatting the drive. So it's easy to conclude that the documentation and the feature set are also in flux.

The rancher project is interesting and functional. The last time I examined it I was pleased to get things going. It functioned in places that Deis did not. My guess is that Rancher wants to own more of the stack so I wonder whether Rancher is going to get the resources it needs to make progress while they are off capturing the OS market. While Docker is playing in that space Kubernetes and Mesos are not.

CoreOS still seems to be a solid bet.

big numbers of things

I had no idea what to call this post but I knew I wanted to talk about numbers of things. In a previous post I was getting heated about the size of my HDD and the cost in time and money to backup and restore everything I needed. In fact I underestimated the cost.
  • the new drive cost $119
  • I needed a new license for Carbon Copy Cloner $20
  • Moving from my old MB to my backup MB means also upgrading iLife and iWork $20/per
In terms of numbers I thought we had about 25-30K raw images in iPhoto. It turn out we have over 54K in images and I know that there were some others in folders that might not actually be part of the "master" since it's been upgraded over the years and the folder structure has changed.

And then there are the emails. I'm looking at my "mail" activity window and the number just keeps climbing. So I logged into my gmail account and I see that I have "sent" 24K emails. I would like to delete the emails because there isn't much point in keeping them all and since I use the apple Mail application it keep copies locally. Of course this is part of the "keep it simple" problem I've been talking about because the keep it on the server selector never feels like it works properly.

In the analysis there are too many pictures, too many duplicates, too many emails. So I like being able to upload my images to Google+ even if the images are only 2K. Now all I need is a way to move my emails from my active folders to a proper archive folder such that I'm only keeping 30-ish days worth. And so if I need to search I can always use my web interface.

Well, Carbon Copy Cloner is awesome. It performed where Apple's Disk Utility wouldn't. The upgrade price of $20 for a "home" license is plausible. I had forgotten about a feature it had... scheduled backup to network drive.

Monday, February 23, 2015

Fedora 21 - Cloud

The Project Atomic documentation is pretty clear but I had a number of challenges when deploying both Fedora-21 Cloud-Base and Fedora-31 Cloud-Atomic on my vagrant installation on OSX.

First download the files. Notice that the files are .raw.xz files which means that the files are compressed raw. So they need to be unzipped first:
xz -d Fedora-Cloud-Base-20141203-21.x86_64.raw.xz
This command is going to replace the file with an uncompressed version of the file and it will be a raw type.

Now you have to convert the file from raw format to VDI for virtualbox:
VBoxManage convertdd Fedora-Cloud-Base-20141203-21.x86_64.raw Fedora-Cloud-Base-20141203-21.x86_64.vdi --format VDI
I coped the resulting VDI file to my virtualvm folder but you can always deploy it from wherever you are. Note that this image is NOT an ISO so there is nothing to install. It's already in a base configuration. This also means that the default user account has not had it's password set so you will need to do that!!!

The next step is to the virtualbox instance. You'll need a little virtualbox experience to know how to mount the drive you just converted. The GUI makes it easy but I'm certain the same can be performed from the command line.

Before launching / starting the instance you'll need to create an init.iso.  This ISO image contains the necessary cloud-init files that define some basic info. In the real world you'll need some real data for your environment but this is the starting point according to the docs.

  • create a folder: config
  • create a file: config/meta-data
  • create a file: config/user-data
The meta-data file:

instance-id: Atomic01
local-hostname: atomic-host-001

The user-data file:

password: redhat
chpasswd: { expire: False }
ssh_pwauth: True
   - ... ssh-rsa new public key here user@host ...
Now you need to create the init.iso file.
hdiutil makehybrid -o init.iso -hfs -joliet -iso -default-volume-name cidata config/
Finally, going back to the virtualbox GUI you'll need to add the drive to the instance configuration. And then you can start the image. Once the login prompt is displayed you can put the keyboard focus on the screen by clicking.  The default username is 'fedora' and the password is what you set it to.

Here are a few notes:

  • If you need to change any of the config information you'll need to change the instance-id in the meta-data file. This is the only way to signal to the host OS that there was a change. If you do not change it then your changes will not take effect.  (this happened to me and it took hours to determine).
  • If you want to ssh into the box then (a) you need to determine the IP address by logging into the box. There are a few virtualbox commands you can execute to locate the IP but that's another topic. (b) before starting the OS instance you need to enable a second adapter host-only vboxnet0. This will create a local connection between the host and the guest instance.
I have yet to determine the difference between the base and the atomic versions of the cloud image other than their size and docker is not available on the base. I'm certain there are also some reasons to confider the difference between the server and base versions. All are topics for another conversation.

Sunday, February 22, 2015

Embedded Lua and/or tcl in golang

I have been working on an installer based on JSON using golang.  The two projects are macroinator and installinator. The macroinatorproject is a refactoring of the original installinator project which will make it's way back into the installinator project very shortly.

In both cases the DSL is very loosely based on Puppet's DSL and implemented in a JSON format.

This design comes at the end of a very long argument with myself about the value of DSLs in the first place. I have been researching both Lua and tcl embedded libraries just to see what was involved. On the one-hand the tcl history document describes the justification for the implementation of the tcl language and then goes on to say that Lua might be a better modern choice even though Lua is incomplete in terms of tcl's scope.

Today I was sitting in my car in a parking lot contemplating all things embedded in golang and as I was considering Lua and tcl I waffled back and forth several times. On the one hand tcl and Lua are complete grammars and they implemented with very little C code. Something in the range of 10 to 20K lines of C code. So it is very encouraging from that aspect. Without any specific knowledge it feels like go's generate tool could be a good fit for implementing either grammar.

But then there is the counter argument that if the language is already implemented and it's just a matter of cross compiling the tools then so be it. Go that way.

About the only thing I fall back on is that Go's static compilation model makes deploying either embedded language as a DSL so much better when executing in a container like rocket or docker. Picol might be a possible option since it is a tcl-like grammar and it's implemented in go. This might span the gap that I was thinking about such that the commands, verbs, macros, might be expressed and executed in a simpler way. As for Lua, since they just added integer division in the 5.3 release in December 2014 I think tcl/picol is probably the way to go.

MacBook Air Upgrades

Now that my MacBook Air (MBA) is out of warranty and still humming strong... and after having to repurpose my backup MacBook it might be time for a hardware upgrade. I would prefer a complete hardware refresh but since that's not option let's do the exercise anyway.

11" or 13" that is the question. The price difference between the stock 11" and 13" is just $100. And for that you get an extra 2" diagonal, SD card slot (I like the mini drive). On both the 11" and 13" you can boost the CPU and RAM for $250 but to get to 512GB disk you're going to shell out another $400.

And for comparison... I have my out of warranty MBA with 64GB of SSD. And I'm constantly running out of disk, memory, and horsepower. However, since I'm trying to use my MBA in an editor only mode I could just upgrade the disk. Looking at OWC they offer 1TB of SSD for just over $500. That's amazing!

PS: The new MB's have great battery life. I don't know what the real world results are but they feel very appealing.

Finally, I'm echoing some Genius advice. 4GB is not enough RAM for large iPhoto libraries and does any of this feel like we are stepping back into the proprietary hardware mode? I happen to notice that they number of late model parts from OWC is dwindling to a trickle.

Know your stack

Superfish on Friday insisted its own code is safe and said the security flaw was “introduced unintentionally by a third party.” 

Whether it's true or not it's still something to fear when linking libraries or nor vendoring or code reviewing. Etc... 

Saturday, February 21, 2015

OSX when simple is not simple enough

It happened again. I lost another MacBook.  This was the oldest machine with the largest HDD. The 1TB was toast and so was the machine. I could not pull the drive and mount it as an external drive and I could not put the factory drive back into the MacBook and boot. It was a double loser.

The only thing I can do at this point is repurpose my MacBook unibody. Sadly I had previouslypulled the 500GB drive, placed it with a 1TB drive and then replaced it with a 250GB SDD. Now this 250GB drive is not going to hold enough if the pictures to be useful.

This had me thinking.  What would happen if I had 1TB of pictures and a MacBook Air. Talking to one of the Apple Genius' today he told me that (a) I would have to partition my pictures onto thunderbolt drives and (b) that 4GB was not enough memory for the number of pictures I had.


But there are any number of problems with this strategy:

  1. iCloud does not support this strategy
  2. iCloud is expensive
  3. Switching to multiple libraries is risky
  4. moving pictures between libraries is complicated
After all that I realized that what I want is something that is simple and that I do not need to know all of the moving parts. I've become my parents.

The only silver lining is that other world computing sells a 2TB drive that is supposed to work in my unibody MacBook. The better news is that while I'm restoring files from backblaze, decrypting my boot drive, uploading the raw images back to Google+ ... I'm able to write this post on my chromebook.

PS: Three things saved my ass. (1) backup to Google+ (2) backup to backblaze (3) took a local copy of my iPhoto folder just a few weeks before the failure. Both the Google and backblaze saved every single file.

Thursday, February 19, 2015

Partitioning Systems and Microservices

When you are building a system from scratch it's easy to build a CI/CD pipeline that builds, tests and deploys the project and it's dependencies in a single pipeline. From this vantage the project is always a fresh install and if you do things right executing migrations are also pretty simple.

But now what happens as some of the systems now become shared systems with their own pipelines and dependencies? Take, for example, databases or message queues. At some point the MQ is going to become a shared bus and the database will no longer scale in the largest VM... so now what?

This partitioning of services and micro services will eventually force you to return to the days when you had to install everything manually and one at a time. While the tools may be upgraded to buttons the risks are no different and the number of moving parts are the same.

The only way this gets better is if the pipeline is truly partitioned and automatic. Automating a rollback could be just as important.

Tuesday, February 17, 2015

First Impressions Ubuntu Snappy

There are a number of nix based appliance OS' out there. Some of my favorites include CoreOS, NixOS, SmartOS, ProjectAtomic ... I could include the unikernels too but I am not. The latest team to enter the space is Ubuntu Snappy.

Installing snappy on my OSX machine with VirtualBox and Vagrant was a modest success, however, getting to the next step was a serious challenge. Installing docker was easy enough:
sudo snappy install docker
however the default system seemed to be missing vim so I was unable to edit the /etc/hosts file in order to correct a host not found error when snappy was referring to itself by hostname. When I tried to install vim I received an error that vim could not be found.

I was pleased with the 5 second boot time and the snappy script felt simple and natural, however,  overall it's still a bit of a square peg. I'm hoping they do a better job in the near future.

Monday, February 16, 2015

G00V00 and G00V00P000

The IBM mainframe OS/JCL implemented a file versioning system named G00V00 or something like that. In today's dotted version numbers they'd need to add a P0000 to keep track of the patch number too. Considering that IBM only allowed for two digits I wonder if that was 00-99 or 00-FF and whether 99 versions of the same file was sufficient before skipping to the next generation? In today's world we commit code whether it's ready or not leaving the build system to determine whether the code is ready or not. In this model patch numbers are ever increasing. Also, with the dotted numbering scheme performing proper numeric ordering is not as simple as a lexical string compare.

If I get to implementing my design for flow based programming by storing the code in a DB then G00V00 and possibly P0000 might makes sense.

QUESTION: Does anyone have some links to the original docs?

Saturday, February 14, 2015

The Perfect Programming Language

I've just read a two articles. One talked about learning OCaml instead of Haskell and another talked about switching from Mozilla's Rust to Nim. One in particular said that there was "no perfect language" and while I'm on the eve of presenting at go-Miami with a strong admiration for Google's Go there is nothing like good old-fashioned C.

This post has started to get away from me. 

C is still the perfect language. It has the history and the tools that keep it in the #1 position. C is also the foundation for many other languages and there are interfaces to many more.

And it got away again

In my ideal environment I would be implementing a framework in C and then deferring the task to another language to something like embedded tcl or even lua. One excellent example of embedded tcl is fossil-scm. And an example of embedded lua is Adobe's Lightroom.

And so I like Nim because it generates C code which could then embed tcl or lua; and why not both. Libraries can be implemented in any of these languages providing the necessary guard rails, cross OS deployments, and the ability to scale from any of the languages to C and back as needed.

C still has so much potential. It's debuggers, libraries in every corner of programming from operating system kernels, device drivers, to applications and beyond. One last thing. Few languages are bootstrapped in their own language. They all started somewhere and it was probable C.

PS: Several languages are experimenting with translations to javascript. It no simple feat as there are so many impedance issues. This includes threading or concurrency, callbacks, basic language features that have to be simulated, and the differences between the many javascript implementations. While is novel and marginally useful it's not going to replace the existing javascript ecosystem.

UPDATE: I just installed nim-lang.  Compiling 0.10.2 from source took less than 5 minutes. While I only played with hello_world I was able to produce javascript and c code. The compiled C program was only 163K in size as it probably included the stdio lib and maybe a few more. And while the generated C code was almost readable the javascript was unintelligible. I'll also note that nim will output code for Objective-C and C++.

Thursday, February 12, 2015

deploy fossil-scm in a docker container on OSX with boot2docker

The title says it all. In this experiment I wanted to create a docker container that had a running fossil-scm instance and the docker container should be as small as possible and therefore should be running from a scratch docker image.

(a trusted scratch image is about as small as an image can get. It's essentially turns a docker run command into an application.)

NOTE: setup boot2docker:
boot2docker init
boot2docker up
boot2docker shellinit | source
(There are a few other versions of the shellinit command such as back ticks or $(cmd), however, this version is the most compatible and works with fishshell which the others do not.)

First and foremost I am using OSX version 10.10. Mavericks is nothing special in the equation but I'm just mentioning it. Next I've installed boot2docker 1.5.0 from the distributed binary release for OSX. (I'm not going to show you how to do that here. Some things ya just gotta know.)

Make a working folder for the project and make it the current folder
mkdir ${HOME}/fossil_proj
cd ${HOME}/fossil_proj
Download the fossil project zip file and unzip it.
MISTAKE #1: when I downloaded and built my first container everything looked ok until I tried to run it. Then I kept getting "exec format error" messages from docker. I cannot begin to tell you the number of times I tried to build and rebuild my images and containers... subtly tweaking this param and that. I even tried to build the image directly on the boot2docker host. Finally I realized that  I had downloaded the OSX version of fossil. Once I downloaded the linux version I was golden.

Create a Dockerfile. (in the 1.50 release of docker you can name your docker file, however, I'm going with the default for now) Here is my file:

FROM scratch

MAINTAINER Richard Bucker <>

ADD fossil /fossil

VOLUME ["/data"]
CMD []
ENTRYPOINT ["/fossil", "server", "/data/fossil/repos", "--port", "8080"]
  • built on the SCRATCH base image (no need for phusion here)
  • I'm making myself the maintainer of this project
  • I want to expose the default port 8080 to the host
  • ADD will copy the file from the same folder as the Dockerfile to the designated TO location
  • bind the /data folder from the host to the container
  • If there was a CMD then docker would try to run /bin/sh, however, the sh command is not available
  • The entrypoint tells docker to run the fossil as a server and where the fossil repositories are located
I created a small Makefile. It's common for me to build this sort of thing because (a) it's idiomatic for most programming languages to have some sort of element and (b) I use it to store information like dependencies and commands in one location so that I'm not hunting around.

all: del clean build

docker build -rm -t fossil .

docker run --privileged --rm -p 8081:8080 -v /data:/data --name fossil fossil

docker rm fossil

docker ps -a -f status=exited -q | xargs docker rm 
docker images -a -q | xargs docker rmi

It's pretty self explanatory what's going on here. The build command will create the image which I've tagged "fossil". It might be confusing to name the container the same name as the application but for the purpose of the example it will do. The 'rm' option will delete the intermediate layers as the image is composed.

When you run the image (turn it into a container and execute) 'rm' deleting the intermediate layers if there are any, giving privileges to the container(probably unnecessary), linking the host's port 8081 to the container's port 8080, mounting the host's /data folder to the container's folder /data, and naming the container instance "fossil". Again I've named the container the same as the application and the image. This is less than ideal... I'm certain of it.

MISTAKE #2: when I started running the application I had created a repos folder: mkdir -p ${HOME}/data/fossil/repos; only to realize that the folder I was creating was on the OSX host and not the boot2docker host. So the next step was to ssh into the boot2docker host and create the missing folder.
boot2docker ssh
sudo mkdir -p /data/fossil/repos
sudo chown -R docker /data
That was pretty simple... but still not enough. The run command would not execute because I did not have a repo in the repos folder.
boot2docker ssh
./fossil new /data/fossil/repos/first.fossil
Now that I had my first repo I could run the container and launch my browser. But first I needed to know my IP address:
boot2docker ip
I don't have much else to report. I have a working docker/fossil. I suppose there are a few things I could do.
  • choose better names instead of naming the image and container "fossil"
  • the name should be registry friendly
  • and I should publish my container (interesting to note that two other versions of the same idea used ubuntu... so lesson learned use scratch when you can
Good luck!

UPDATE: One thing that I think I missed.... when running the container one should either detach it with the '-d' option or force it to be interactive with the '-ti' options. Using daemon mode means that the --name option makes it easier to address the instance. Using the -ti option means it's easier to kill the daemon from the command line with a ctrl+C.

Wednesday, February 11, 2015

"Compose with functions, not methods"

I was drawn to a link titled "Compose with functions, not methods". It was a talk that Rob Pike (a Go Author) gave in 2012. While I had an idea what he was talking about I looked at the slide and read and read it trying to gleam what he meant by the text.

I then read the method and function definitions in the Golang documentation called "Effective Go". And I still did not understand what the authors were getting to. Then I read the language reference and found the missing pieces.

Function: "A function declaration binds an identifier, the function name, to a function"

Method: "A method is a function with a receiver. A method declaration binds an identifier, the method name, to a method, and associates the method with the receiver's base type."

And now it all made sense. By preferring functions instead of methods one can use interfaces as part of the composition formula in the way that Go implements Read() across different packages. And the same for wrappers where the work can be nested.

Tuesday, February 10, 2015

If I had enough time to experiment

If I had enough time to experiment then there are a few things I'd want to do:

  • Build a proper pipeline from NixOS, Nix, NixOps, Hydra
  • Design and build a pipeline to include AppScale so that I can concentrate on application development instead of framework or pipeline
  • Build a pipeline from's open source offering
  • Build a go pipeline with CoreOS and a cluster of Raspberry Pi servers. ($35 a device and 10 servers for a CoreOS Cluster... fun)
  • Design and build a Docker pipeline and framework.. I like the Apcera gnatsd docker/dockerfile build and the Apcera policy engine
  • Micro services are interesting but a bit of a challenge when you consider the guest OS. Phusion is still creating fud
  • etcd 2.0 has been released and CoreOS' rocket, flannel, and appc are around the corner. They need to be investigated as I want to experiment with trust and a design I have for an HSM
  • I like monitoring but there are so many heavyweight tools. Prometheus needs some investigation
  • investigate Docker networks and VPNs including Docker swarm, machine
  • Fossil instead of Git and hg.
  • I'm tired of reading about languages like Nim, Lua, Ruby, Python
  • I'd like to experiment with Kubernetes, Deis, and mesosphere/marathon
  • TCL is at the core of fossil and SQLite. It's worth experimenting
  • Then there is finishing my installinator and flow based programming projects
There are a few more edge cases so let's see where it all goes from here.

UPDATE: and then there is a lot of plan9 ideas I want to investigate.

Monday, February 9, 2015

Share this and that

Sadly Douglas Campbell decided to disable comments or I would have responded directly to his post. I don't want to say that he's made a discovery, however, sharethis seems to have a foot planted in the docker trough. I'm only pointing this out because these guys seem to be very devops oriented. Once you look at their career page it's almost immediate.

There is a lot of good to be said for docker.

  • it is much like a chroot with more partitioning (security)
  • but it supports many different linux flavors
  • so the host and the guest do not have to be the same brand
  • CoreOS is a perfect host OS
  • devops is easier if the containers are considered part of the pipeline channel (see CoreOS channels as an example)
  • support for sidekick and ambassador patterns
  • applications like machine and fig create networks of images through composition
  • images are sharable through a registry
  • with reputation based recommendations
  • so SMEs can create proper images (i.e.; mongodb)
On the downside there
  • there is confusion as to what a container is supposed to look like
  • which guest is best
  • how to build and deploy you apps
  • proper networking design
  • proper CI/CD pipelines

Today I read the first few paragraphs of the NixOps manual and I was surprised. NixOps, which is also part of the functional family of tools for Nix and NixOS, has a number of features that look like fig and machine had a baby. The best news is that it's just part of the "system". And right now I'm dreaming about all the happiness I can create if I only had some time to devote to Nix and it's tools like containers, NixOps, Nix, Hydra and so on... From my vantage it feels like a complete, self contained, batteries included OS.

UPDATE: I'm building my first NixOS instance using NixOps and it's humming along. Interestingly enough it appears that NixOS is being compiled from scratch in order to make the entire deployment. (whoosh there goes SSL and lot's of perl). I cannot predict if this is a good thing or not but there are a few observations.

  • NixOS as a bare metal OS makes sense so long as the source and packages are mirrored locally and the hardware is capable
  • Once the deploy is completed there must be a way to deploy containers in an way that might resemble chef ... don't do anything if it's already there
  • deploying N+1 is faster than 1
  • deploying N+1 when 1..N has already been deployed does not mean starting over
I would have liked it if the tutorial provided a hello-world type installation. The two that NixOS provided were just slightly more complicated but a toolbox nix would have been nice. And there it is. In less than 15 minutes a complete installation from source. Interesting....

UPDATE: It might not actually be complete.  The welcome file is displaying the version 13.10.XXX which I know is not current (14.12.XXX). That said the terminal window that was running NixOps is still running. It appears that maybe the console output is from the remote VM. Nothing more to say other than I hope the installation is current.

UPDATE:  And then it failed:
building path(s) ‘/nix/store/7ivvsliqfhmrqwj6wjwvb2ni8nf0xiqh-boottime-keymap’
error: a ‘x86_64-linux’ is required to build ‘/nix/store/b97vk6ykglyp1ax71b11kpjscfg8zsxv-boottime-keymap.drv’, but I am a ‘x86_64-darwin’
error: unable to build all machine configurations

Sunday, February 8, 2015

Hot code replacement - Erlang, Elixir and so on

I have to make this quick. Hot code replacement is interest, fun, and very complicated. It's so complicated it's not worth the effort and in the few use-cases where it is a requirement might and should be in the far minority.

The first problem with hot code replacement is scope and idempotence. Hot plugging a small module that is only part of the transaction cannot be reproduced. Creating monolithic modules might as well be implemented as blue/green.

Second, when hot replacing code there is challenge to migrate state from the current module to the new one. Since a module can have more than one process maintaining state moving the state from the current to the new and preventing forked transactions... it's all too difficult. (it is possible to store state in a cache like redis but that breaks a few idioms like the benefits of immutability)

Next, while hot code replacement is considered a rockstar level achievement it's complicated and while it can be made to be reliable and reproducible there is a challenge that the full system might not startup from scratch should it be required. At some point you have to be confident that a restart is possible.

One of the last blockers on the list is parity with the VCS and CI. The continuous integration system is going to build the monolithic application while the hot plug is just a fraction and most CI systems do not address the module as the most atomic component. While partitioning the application along module lines is possible it does seem to be an anti pattern as google seems to like monolithic VCS.

Elixir might be a reasonable gateway from ruby to erlang and while hot code replacements are fun and interesting they create a serious number of operational challenges for the audit sensitive and long transaction applications.

UPDATE: I just started reading this paper. While it starts to talk about DSU, dynamic software updates, I find myself considering compiler versions. The question being; is there parity between the compilers or does the DSU use the compiler of the current version.

Which brings me to another question... what happens when the erlang VM needs to be updated? At that point you're back in the blue/green deploy pipeline. Blue/green is the best way to migrate an application but the transition has to be managed.

Real Programmers ...

Real programmers can write assembly code in any language. -- Larry Wall
Here are just a few links...

One thing for sure... long before TDD the testing and debugging trend was (a) logging, (b) asserting, (c) single step every line of code (d) look at the assembly version of the code.

A few of the things I really liked about writing in C on DOS systems:
  • the smallest binary executable was only about 4 bytes. In an EXE file there was a 2-byte token, segment index, segment definitions, and then the code segment with it's setting the AX register and a call to interrupt 21h. In a COM file it was just the code segment...
  • Since the only linking you would do were libraries you identified... there was no bloat and the code could be dumped.
Reading the code in the assembly version often times made it easier to debug or desk check. With modern programming languages there is TDD (yuk!) but there are all sorts of potential visualizations. I happen to like flow based programming as the logic and function are separated (separation of concerns).

Saturday, February 7, 2015

MongoDB, Partitioning, Sharding, Benchmarks, and application design

10 years ago I ran some benchmarks for an OLTP application I was building. One of the driving design requirements was response time.  The aggregate response time had to be shorter than 250ms. The initial implementation relied on 100% of the work being performed in the application which meant that the network was chatty as the application retrieved data from over 100 tables. The multiple network roundtrips, latency, and transaction lock escalation, not only increased transaction time but also limited the number of concurrent transactions.

The second implementation cached data in the app server. That design also failed. The third implementation, which was elected to production, divided the work between the application server and the database. The application managed the protocol impedance and transaction validation between the payment network and the database. The business logic was implemented in the database  as series of runtime configurable stored procedures. While using T-SQL for every transaction and in such a dynamic structure was not optimum it was more performant than the alternatives.

Once the DBAs exhausted all other options from primary keys, covered indexes, better extents, replication, sharding, partitioning, high availability/load balancing, re-indexing, replication ... we started to evaluate other DB technologies and hardware in order to scale in hardware instead of software. This option never received any traction because all the all of the business intelligence was written in T-SQL and moving to Oracle or other DB meant nearly a complete rewrite.... and version 2 ... even with an army of contractors will never be completed. (mythical man month)

If we had implemented the fourth option (still on the drawing board at the time) we would have had a window of opportunity to get on the new hardware.

The application was written in Java and so implementing the business logic in Java in a separate micro service that would run on the DB would have been the best of both worlds. In today's container world this type of service is referred to as a sidekick or ambassador. This application was going to act as a shim to the DB and perform all the heavy lifting/computation of the application.

In today's application design we are faced with a number of challenges. It's no longer as simple as installing an application on the same core as the database. Now there are CI/CD considerations as well as production discipline when actually pressing the GO-LIVE button. Version management over a cluster of master-master database servers that follow the CAP theorem while trying to update the on-server application ... there are just so many moving parts that continuous operation is a dance between the OPS and the DEV teams.

Software Architect - Best Job? For how long?

This article popped up again. It was written in 2013 and is mostly valid, however, the winds are shifting.

About 15 years ago I was having a conversation with my stepfather. The topic of conversation was the investing, the stock market, brokers, and the internet. He said: there was no longer any reason to have a broker; with the advent of internet search the investor had access to the same information that the broker did.

As I was reading a few emails this morning I've realized something very similar about architects. I consider myself an architect because I have been architecting greenfield projects for bulk of the last 20-30 years. Any while I try to keep abreast of all the new tech sometimes it's a matter of being the first one to press SEND.

Being a successful architect is more than reading Hacker News, SlashDot, TechCrunch, etc... It's knowing when (a) not to send, (b) when to review and provide a distillation of the cost/benefit. (c) and when to yell "stop the presses" and (d) the difference between cutting edge and bleeding edge.

A friend of mine posted a picture of an Arduino project on FaceBook and said that a coworker built a computer. The reality is that the coworker assembled the computer not built. Just because you read an article does not make you a broker or an architect.

Getting Started with Kubernetes

I'm trying to go through the Kubernetes Getting Started and I'm not getting started. The team has provided a number of ways to get started but none of them are working... let's get started:

  • GCE - Google's Compute Engine would be a great choice but I'm trying to develop some ideas about how useful Kubernetes is. In that case I'm just hacking so I'm not interested in having to setup accounts and payments just for the sake of experimentation. I will probably come back to this as I will explain...
  • CoreOS was the first strategy I tried. I have had tremendous success with CoreOS and CoreOS clusters. I've distributed fleet clusters, docker and rocket containers. I've also deployed CoreOS clusters using the CoreOS-vagrant tool. So this seemed like a no-brainer. However, it failed. The prerequisites included vagrant, virtualbox, net-tools, and either build or install the kubernetes binary. (a) they were not clear about the source or target of the net tools. I assume I should already know what that is and it's probably from virtualbox or vagrant... oh well. Building Kubernetes from source is supposed to be simple, however, since I already use boot2docker for other tasks the network addressing does not match the expectation. It's interesting that they are also mapping boot2docker but that it's not mentioned.
  • Fedora/Ansible is not an option. I suppose this could be one of those defining moments when I discover Ansible but not for the moment.
  • Fedora/Manual is even less of an option.
  • Local was meant for someone running Linux as a host; it is not an option for my MacBook.
  • Microsoft Azure could be an option. MS has been doing a lot to capture some of the market. Even their *nix mission seems to have changed. In the meantime it's still a service which is going to require some work and possibly some money. If I were to go the cloud route I'd defiantly go with the GCE since I already have an account there.
  • Rackspace appears to be more of the idempotent structure I was looking for. And even though I have an account at Rackspace the tool is based on OpenStack and so it requires installing Python, virtualenv, Nova and swiftly. It's just not the way I envision the tools to work.
  • The CoreOS option is interesting because there are 6 different strategies. The SingleNode and ClusterNode models seem to be the most idempotent. Any changes would require taking a snapshot and then sharing on a repo someplace. One of the multinode methods required installing fleet and etc on the local host. This is not bad but can get wonky. Another option includes weave which is an open source network framework which I think the CoreOS will trump with their Flannel project. But again there are requirements made upon the host.
I have the unreasonably strong opinion that the host should be pristine and nothing beyond the base operating system and the tools for which version is unimportant. For example one's choice of editor, git, mercurial, boot2docker, virtualization and so on. 

Using boot2docker does have a host component, however, upgrading has not had any effect on the overall system. Sadly tools like rvm and virtualenv can be a serious headache to maintain although development before them was extremely painful. But until someone tells you the stories of development on DOS with a single toolchain you'll never truly appreciate this approach. Just consider the amount of productivity lost just maintaining all of these versions.

So coming full circle... GCE is looking like the right choice again. The trick is going to be managing the cost by destroying non-functioning clusters. Once the cluster has been compiled, tested and the results captured. Dump it. The actual cost for the smallest node is about $.02 per hour. Granted it's severely limited in RAM, Disk and CPU; however; even at a $.50 per hour for a bigger cluster/node you're not paying for downtime and at that pace it's probably very cost effective. (just don't leaving them running over night.)

Thursday, February 5, 2015


UnQLite is the NoSQL variant of SQLite. I suppose if I needed a document store I'd probably go after one of the big boys like RethinkDB, MongoDB, or even Oracle. There are even a plugin or two for Postgres.

And if I needed a key/value store there is Riak, Redis, a number of embedded libraries and of course there also SQLite since k/v stores are just the 5th normal form (or something like that).

But if I needed a small pragmatic storage engine for storing documents or some UnSQLite -like data... I'm not sure I'd ever go embedded with UnQLite. I suppose that's why the SQLite author never completed his UnQLite project.

Things that make me happy in go(lang)

mattbasta pasted that there were some things that he hated about go but that never provided a way to receive comments. Shame. It's an interesting topic.

Project Layout

This simply means he does not understand the tools and their potential. And when those tools fail there are others to help like: glide, godeps, golo(mine), goli, make.go and a few others.

Relative Package Imports

That's just a stupid idea and is similar to the first topic. Namespaces are important and until someone finds a way to make global namespaces simple then that's just the way it's going to be. It would be interesting to have some sort of local registry but then by the time you do that you end up with a filesystem based on some pattern. and so the argument gets recursively redundant.

The Big Community Mess

I'm not sure how the first two items are the communities problems. Frankly, GOPATH, GOROOT, GOARCH, GOOS are the one-stop shop. As far as NSQ having a gnarly makefile... you've never used autotools/automake or you'd have a completely different view, however, if you want to see the right way then look at the gnatsd implementation of it's docker build. It has to be the sexiest build I've ever seen and hits all my touch points.

Obvious Missing Features

Deleting an item from a slice should require arr = append(arr[:idx], arr[idx+1:]...). How is that friendly?
He kinda has a point here, however, the problem is that slices are not really arrays. And for all the gore this is probably the best way to rationalize slicing things. (append slice 1 to slice 2; as simple as that)
There is absolutely zero implicit type casting. Comparison operators could benefit from this the most. int64 and int? Can't compare them.
Thank goodness. It's impossible to reason about implicit casting otherwise everything would be an interface{}. And what fun is that. This simply makes everything explicit so that there are no side effects. And side effects will get you every time. (of course this would make it a dynamic language and that it's not)
Nested function syntax is very awkward. Why can't nested functions optionally use the same syntax as non-nested functions? We may never know.
Sadly, I do not eve know what you're talking about here. This could be anything. Returning a reference to a function or assigning a reference to a function to a variable are pretty sensible.
Integer math. All of the math functions (abs, floor, etc.) operate on float64s. Go needs an imath library to do anything useful with integers at all. Presently, you need to write all of your own commonly used math functions yourself.
I hate all math and I especially hate float. If you've ever done any real math with floats you'll quickly come to the understanding that you have to move to bigint or some such. The reality is that floats based on base10 do not work very well on a base2 system.That's why 2-1=0.999999999 or something like that.
There's a distinct lack of basic data structures. set? Nope. Sugary queues or stacks? Nope. Ordered maps? Nope. Heaps? Nah. While of course it's possible to build all of these ourselves, it's not ideal. And frankly, the ones that are available as libraries in the community are mediocre at best.
It seems clear to me that you do not really understand what you have in front of you. What you seem to be lacking without actually saying it is generics and for that you can fuck off. (sorry). I'm sick to death of all those java programmers that think generics are the only language feature that separates any modern language from the stone-age.

Any thanks for reading. Let me know what you think.

UPDATE: there was another article that went down the same rabbit hole.

UPDATE 2: And then there was a positive article in my reading list. The few weaknesses he mentions are reasonable.

Web apps and frameworks in Go

I'm trying something new. I'm going to write this review while I'm getting acquainted with that which I'm reviewing. Granted this is not an in depth deep dive but it is the first blush. And while they say don't judge a book by it's cover, in this case it's README file,  there are simply too many 3rd party web frameworks out there for go.

The granddaddy of them all is revel. What makes Revel great is that it feels feature complete. The menu in it's manual makes it pretty clear what it supports. The only thing I do not see is some sort of integrated ORM. It's not compulsory but it could be helpful. I've implemented a few projects and while they work they are not very gratifying.

When creating a REST API server it seems obvious that the handlers and documentation should be generated. There are a number of tools for that. For the moment the defacto leader seems to be swagger. From the swagger toolchain (nodejs) there are other tools that capitalize in order to generate multi-lingual client SDKs. Regardless of the streamlined nature of the toolchain I'd prefer to be homogenous for the moment (I've previously written about Chef, Puppet, Ansible and SaltStack in this area). go-restful will use the swagger API docs to create a go webserver but without examples or documentation it's not valuable. A second go-restful project trumped the first with some docs and a few supporting blog posts. Swagger's own codegen is a whole crapload of templates and so it does not benefit from any sort of reflection.

One cool project that feels like it hits the balance between a 3rd party library that makes my life easier without having to be concerned about the cruft is httprouter. It's a small project with a reasonable README file that covers the different aspects of the library. I like that it partitions the routes by method but I'm not sure I like the parameter handling. On the one hand it's nice to have the feature because the stdlib does not support it at all and the maintainer reports that there are no leaks or heap usage... but there has to be a latency in the regex for parsing parameters from the path string. That said it still matches with my sensibilities.

"Goji is a minimalistic web framework that values composability and simplicity." It seems to me that given their examples and documentation that if you're going this way then just use httprouter and do the rest yourself. Goji does not add any noticeable sugar.

"Gin is a web framework written in Golang. It features a Martini-like API..." Anything that is **-like starts to lose my attention. If I needed a like I would probably build a like myself. At least this way when things started to deviate from the norm it would be on me and not on the 3rd parties digression. Gin is marching toward 1.0. There is some activity but not much. The logging is colorized which is useless but it has integrated httprouter suggesting that it's 40 times faster. Faster than what (they do not say)?

Ace, dead and no docs or examples.

Here is a benchmark for a number of frameworks. If you try to read the link from your phone... no real joy. The results favored httprouter which up until this moment I thought was a library and not a framework.

In that spirit I'll include vulcand which is a load balancer. The compelling story here is that it uses etcd in order to configure and update the routes. While mailgun reports that it's still beta code they are running it in production with a modest workload. I do not know what modest is or in what use-case but I know those guys and I believe they are being modest. The mailgun team have a number of other projects... They have their own router which does not work like httprouter and would appear to be too far into the transaction to be anywhere close to the httprouter benchmark. mailgun has a logger however the config seems to be a toml or yaml file.  That's just silly these days. And then they have a webserver that looks like a httprouter and a json config file had a baby. What makes scroll a nice selection is that with just a little reflection it would be possible to create composable routing tables at runtime and from there you might be able to get to something that looked like a flow based programming framework.

swiperproxy has a great name but it's python. I like python but I'll be happy if I can forget the experience. HAProxy, nginx and vulcand are capable replacements.

So I've made it to the bottom of this list. I don't think I hate everything as there are a few gems (not to be confused with ruby-lang) in the ruff. httprouter being one of them.

Five is the new loneliest number

According to the CoreOS team the best node count for an etcd cluster is 5. This way one instance can always be rotated/upgraded at a time and then there is room for one failure too.

5 is the new 1.

Monday, February 2, 2015

The Raspberry Pi 2

The latest Pi looks to be a box I can wrap my ARMs around. pun intended. About 15 years ago I built a cluster of Linux servers that had fanless Intel CPUs that clocked in a around 900Mhz. Maybe that's the sweetspot or material dynamics. If I recall each system cost me about 350-500 to build. The motherboard was probably around $125 and the bulk of the expense was the case, power supply, cdrom, and hard drive. With the new Pi and the state of art injection molding and new plastics I could build the same cluster for a fraction of the price.

One negative attribute is the RAM. The Pi-2 only comes with 1GB and while that might be ok for a small OS like CoreOS, busybox or scratch I don't think it'll handle many containers or something that can act as a POC. However a cost analysis should be performed because maybe an adhoc Google Compute Engine cluster would make more sense.

another bad day for open source

One of the hallmarks of a good open source project is just how complicated it is to install, configure and maintain. Happily gitlab and the ...