Skip to main content

Posts

Showing posts from October, 2014

dotgo - John Graham-Cumming

John did a presentation on small applications which started to sound like "unix programming" at this years dotgo conference in the EU; or so I think. In his presentation he was talking about an application that his boss requested only to be asked to implement many more similar applications. As things progressed he realized that they all shared something in common. That was that there was some input, some output and some processing in the middle.
He presented some code that was 75 lines of exactly that format for the first request. While John offered a github version of the presentation it was lacking in the "process" details.  The z.go file is simply a template. (Interestingly enough it's licensed under MIT and although it's probably the simplest implementation of this formula it's also very obvious so while I hate software patents and encourage open source; any license for something as simple as this is exactly what's wrong with the system)
Create a …

Deis.io version 0.15.0 prerelease

UPDATE: even though the current release of Docker is 1.3.1; this version of Deis, 0.15.0, is using Docker 1.3.0.

Finally!!! Deis' installed and ran the first time.  Now I'll admit it took more than the documented 10 minutes to startup the builder container but when you consider that this container is supposed to identify, classify and build a target container (I think) it's probably bigger than the rest.
deisctl list Will produce a list similar to fleetctl list-units but from the host system. I really like to run the command with form a watcher.
watch deisctl listThis will refresh the output every 2 seconds by default.

After downloading and installing my fork of the example-go application the deisctl program did not add the program to the list. There is a different command:
deis appsThat should display all of the applications that I pushed.  Interestingly I saw 3 applications including mine. Since this was a fresh install I was not expecting to see 3.
$ deis apps
=== Apps
tender-m…

More flow based programming

I just finished watching the kickstarter video for the flowhub/thegrid.io project and I'm still convinced that my FBP framework has great potential. Of course it does not have the GUI that the flowhub project has but it has a different level of self visualization that is still comfortable as most visual programming languages simply fail for not being feature rich or expressive enough. I'm sure flowhub has it's production usecases and I'm curious how big a project can get and how traditional CI/CD and other production migration can be expressed on hundreds or thousands of machines; keeping everything in sync, acid, idempotent and predictable.

PS:  While I was once an erlang programmer and have deployed some non-trivial applications I still find the erlang software development manual inspiring.

Curated golang libs and things

I need a goto place where I can track or at least link to my favorite libraries. Since golang is now my complete preference here is my list:

Good article on self referential functions. helpful article for http/REST. List of go tools. And here you can setup a project.

Instructions for setting up for go cross compiling.

gopass - console password prompt

gorename - the article, the source (the source comes from Google and attached to the go tools repo)

gogs - self hosted git service written in Go.

go-bindata - Useful for embedding binary data into a go program. go-bindata-assertfs - Serve embedded files (http handler type funcs)

msgpack seems to be a good and efficient message container. Of course there is JSON and many different implementations with different performance profiles. What is nice about MP is it's support for so many different languages which could provide significant interop.

MQ - gnatsd or here - go implementation of a nats server (client libs are in adjacent projects). Thi…

Overdue updates

Tweetbot for iOS is overdue to receive the "add to reading list" function directly from the main stream. Currently you have to open the link in Safari and then add it to your reading list.  That's just too many clicks.

Of course the exact same can be said for Google+'s iOS app. It's simply impossibly to "add to reading list" until you get it open in safari. There is a shortcut or two but in the end you are still in the sam place with the same number of clicks.

And of course Google's Chrome browser already sync's bookmarks and such but they have not implemented a reading list as yet... and I'm not going to re-activate those other guys just yet. Safari did it right... and it would be flattery to get even close.

Panic needs to update StatusBoard with some of the 2.x features they talked about a year ago. And while they are at it they need to fix some awful terminal bugs in Diet Coda.

Cell phone manufacturers need to correct their pricing so that pr…

New nexus 6

It looks like one heck of a phone. And I want one. But I'm not going to pay $699 for a phone and I'm not going to get sucked into AT&T Next. I'm even more suspicious of T-Mobile's transfer program.

Cleaning up after Docker

I happen to be scanning my CoreOS boot drive and it's 97% used.  CRAP! I have a number of ways to clean up my drives but in this case I had both active and dormant containers and so the usual way of cleaning this up was not going to work. The usual way was "delete everything that is not running; both containers and images".
Well that was not going to work... but then I was lucky too. I found these two commands here:
docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rmdocker images | grep "<none>" | awk '{print $3}' | xargs docker rmiWhat makes these two commands interesting is actually the first.  It only deletes the containers that were  active "weeks ago". Granted it's possible to delete too much depending on the actual output of the docker command but in this case the data matched the query exactly.
The other thing to note is that while docker seemed to delete all of the images... the…

Watching out for Docker

After an intense 2 weeks of deep dives into everything docker I find myself with a very short list of URLs that I need to monitor:

Docker - of course
CoreOS - naturally
fig - cool
boot2docker - required
deis - sigh
kubernetes - ditto x2
Rancher - used it when it was stampede.io. very promising.
Rancher is making a comeback

Phusion has a lot to say about container contents but it's very scary.

Building good docker images - some ideas about creating good docker images.
Reverse proxy with nginx.
Docker patterns.
Continuous integration with DroneIO
systemd alarm clock
citadel docker API
Project Atomic, Dokku
Vulcand, zero downtime deploy
WeaveDNS is one way to make service discovery easier. Weave is a network partitioning tool that could also be useful. And some boot2docker DNS.

multi-server docker.

Under Review
kitematic - docker on your mac
bowline - build server and UI for docker
artifactory - recently added docker registry to artifactory

UPDATE: flynn - I discarded my interest in flynn. The previous pre…

PaaS critical feature

One critical feature for a PaaS that you're going to run your business on is limiting the number of service outages while upgrading. CoreOS is a good start in that the enterprise toolchain allows the operators to control the rolling reboots, however, looking at the deis instructions for upgrading requires that the entire PaaS fabric is disabled during the upgrade. Before going all "in place" vs "migration" you need to understand that both are just as volatile and the chances that there is going to be an outage is very high.
The only way to manage the potential service interruption is to own the service and the integration points so that the single point of failure between micro-releases is managed.Green/Blue deployments applies ALL aspects of the stack. The OS, the PaaS and the application/micro-services.

UPDATE: deis.io now supports in-place updates.  I need to give this a try.

First pass - web application

A multi-node HA web application based on a minimal 3-node CoreOS installation.
Install CoreOS out of the boxExpanding on my last post this is what I've been thinking about in order to get my stack operational. The incoming event starts with the client browser so there is not much to do there. The transaction also flows through public DNS and into the primary firewall/router with some built-in HA capability. There might be n routes to n nodes in the multi-node installation.
Configure the router(s) to each of the nodesNext I need to implement an HA proxy based on hipache using etcd. There is a feature in the server that will try to detect dead servers and if detected will suspend that service until the TTL has expired.
Deploy the  proxy server and pull the backend configuration from etcd.Implement the backend server. This can be any simple backend Docker micro-service.
Deploy the backend service using Fleet to distribute the service to each of the nodes. The pre/post config in the fleet…

Docker, dockerclient, citadel, fig, multi-node, hipache, etcd, nginx, crypt

It's only a matter of time before the Docker team closes the loop on the multi-node Docker stack and starts to chase the complete PaaS solution. Sure; in the early days of Docker it's open source and the various teams are absorbing the code as quickly as it's available; and the different framework teams are all stitching as much code together as they can. But the one quote that seems to be sticking out in my head is something like:
Build it yourself So while I have been testing all of the PaaS frameworks out there they are still lacking. Whether it's current (Docker <1.3.0, CoreOS <Alpha, Go < 1.3.3) or of it simply does not work or it only supports a limited functional set.

So here's my intuition... and if it were my money looking for a solution in this space:

I'm starting from the fractal dimension. Docker containers are simply just another fractal dimension either in or out from that of virtual machines, mainframes, or J2EE-like enterprise SOA solution…

Apcera's Continuum - works but is costly

I have not made the complete rounds on Continuum, however, I was able to get a VMware host running and interacting with the UI which is a level of success I have not been able to reach with the other PaaS systems. Continuum is similar to Stampede/Cattle in that you can deploy an OS as well as containers. Continuum seems to be tweaking the glossary but the outcome seems to be the same. Docker containers, Full OS, buildpacks... But the pricing is crazy. $2000-$7000 for 32GB to 128GB (I'm not sure what Apcera mean by: "A monthly subscription of Continuum is based on assets under management and consists of a cluster and a standard support package")

Docker + Fig

There is an interesting footnote in the Docker documentation/blog. The core Docker team is merging the features from fig into Docker. While they are at it I hop they are adding multinode and auto ambassadors.

Docker 1.3.0 framework updates

As of today here is the latest docker progress:

Stampede.io - no updates
cattle.io - no updates
fig.sh - current to Docker 1.3.0
kubernetes - current to Docker 1.3.0 on OSX
deis.io - no updates
flynn.io - no updates
CoreOS - updated in the alpha channel only
Dokku - no updates
OctoHost - no updates
Cocaine - not sure. cannot find the source repo
Dawn - no updates
Tsuru - no updates
ProjectAtomic.io - not finding the source or the commit history
OpenShift 3 - no updates
Panamax - using a recent CoreOS but not alpha channel
Shipyard - no updates
spin-docker - very little action at all
Solum.io - no updates
Consul - no updates

I've decided to stop research this issue. It's entirely possible that many of these projects simply do not need to be updated.

** "no updates" means that the project has not indicated whether or not Docker has been updated.

** it's always clear exactly how these projects are implementing Docker and so other than an explicit reference to a Docker…

boot2docker time sync

I'm struggling to keep my docker container and docker host's clock in sync with my host (laptop). The challenge is when I put my MBA to sleep the VM (VirtualBox) stops receiving clock ticks. In turn the VM believes that nothing ever happened and so the clock is late.

There are a number of ways that this is supposed to be addressed. (1) the VM should have installed the VirtualBox tools. This might allow the VM to trigger on wake but who knows for sure. (2) ntpclient appears to be running as a daemon and it also seems to be running... but it's not updating the clock. And while there is a /var/log/ntpclient.log file it's empty.

There are a few choices to make things right.

(i) stop and start docker. This seems to force the ntpclient to do the right thing.
boot2docker stop && boot2docker start (ii) you can just run the ntpclient command manually
boot2docker ssh -- dateboot2docker ssh -- /usr/local/bin/ntpclient -c 1  -q 200  -h pool.ntp.org
I'd prefer that ntpcli…

Apple Yosemite OSX and iOS 8?

I updated my MBA (MacBook Air) to Yosemite. The installation was not as painful as previous releases. In particular I had some FileVault errors that required a complete reinstall. I also watched the iOS 8 and the most recent WWDC where Apple described all the benefits of Yosemite.

Thus far the experience has been relatively painless although not pain free. Different fonts in the menubar, more and more alpha shading, spotlight search is popup instead of a menu-like pulldown, headphone volume was quirky with Chrome until I rebooted a few times, the volume button would chirp when using the keys or mouse. Anyway... lots of changes, not all for the better.

As I watched the Yosemite presentation I found myself adoring many of the features like iCloud integration and continuation. But I still struggle with the cost. Additionally iCloud seems to be sandboxing the data in iCloud so that data cannot be shared between applications. At least it's not clear how the sandboxing is implemented. And…

Docker 1.3.0 is available

The one feature I have been waiting for is trusted containers although I have yet to completely understand the pipeline but I know it's going to be very important.

New and improved Docker 1.3.0

So this was a lot of fun. At first I was thinking that I was going to leave my Docker development to my cloud servers and my work laptop. But that with the latest Docker 1.3.0 release and the associated boot2docker and fig projects I had to install it on my personal laptop.

First some descriptions:

Docker - an open platform for distributed applicationsboot2docker - lightweight linux distribution that runs inside a VirtualBox virtual machine.fig - configuration and orchestration for a single-host deploymentSecond a small piece of advice. Fig and boot2docker are meant for development although fig might work in environments other than boot2docker. There are a number of clues that the docker team and very early adopters (fig was recently acquired by the docker company) have left for the rest of us:
Your build or makefile should use a container to perform the build. Eat your own dogfood. Both fig and boot2docker use a docker container to create the executable tools and boot2docker it's is…

hodgepodge of Docker notes as of today

Updating the latest "box" core image for your virtualbox installation requires one quick command.
vagrant box update --box coreos-alpha  "Yes", CoreOS will update itself based on the LockSmith settings on your system but if you are running a multinode cluster that you might be repeatedly launching and destroying... a normal vagrant box updatewill be processed for each node in the cluster (running or not).

I was tinkering with the latest deis source and I noticed that there were a number of patches that might have effected my experience but since the project only provides binaries of releases I would need to perform the compilation myself. The process was never completed but I did notice that their Makefile depended on boot2docker in order to deploy a working compiler environment. Being a devops build engineer I found this interesting and comforting even though it did not work properly.

Deis is a cool project. It's probably the furthest ahead of all of the Docker P…

Can't live without my speech-to-text

I hate typing on my iPhone.... speech to text has been useful for blogging, tweeting, and all important texting (SMS). While I do not SMS in general public it is the most efficient way to enter text when I'm walking to meetings or waiting for the movie to start. But now that I have destroyed my second iPhone 5 screen and I'm forced to use my early iPhone 2-ish... I find myself wishing that I had my old Motorola StarTac with speech to text.