Skip to main content

vmware, docker-machine, docker swarm, traefik and gitlab

I've been experimenting with my configuration for a while and I have not achieved 100% coverage. Taking lessons learned in this space including configuration as code and lights out DEVOPS and idempotent deploy I'm moving to the next step. One funny thing to note is that I spent 45 minutes looking for my code and notes in every git wiki I had only to recall it was a post and not in the code. RATTS!

Part 1 - Hardware

The system I am running on is a 2017 Intel Skull NUC with 32GB ram and 1TB disk. In 2018 Intel upgraded the hardware and I want to upgrade but for the moment this is what I have. Currently live I'm running a Google WiFi router but that will be replaced with a Ubiquity EdgeRouter-X. The Ubiquity has a unified command center and that makes me happy.

Part 2 - VMware

There is something to be said for running a container OS on bare metal but then there is even more for running VMware on bare metal. Except for some of the overhead one can still allocate 100% of the system resources to a single active VM and you can still have alternates and backups. Also, you can take snapshots and so on. Lastly, with the right license you can do a lot of automation with APIs and in fact docker-machine is a handy tool. And you can mount a VMware filesystem.

Part 3 - RancherOS

There are a number of operating systems that you can choose to operate. I really like CoreOS, however, now that it's been acquired by Red Hat I just do not know the status of the product. Also it's costs are still too high. One thing that CoreOS does, which I thought was a value, is the auto updates. Sure it's only down for 9 seconds and the company is very reliable, the problem is that your system might be in the middle of some critical task or maybe a database write and that quick update causes dataloss or perceived malfunction from the customer perspective. RancherOS is no more or less capable. Since the main OS is actually running in side it's own container almost everything can be updated with no downtime. It's also easy to deploy with scripts and VMware. My only issue is that it requires 4GB to boot and get any sort of meaningful work done.

Beside the docker swarm machines you are going to need a console machine. Since I like immutable and reproducible installations. The console is also going to be RancherOS. I also tend to use Alpine Linux in some cases and inside containers.

Still looking into Docker Moby, Fedora Atomic, Alpine Linux. But I'm staying away from anything that might be too configurable. The console machine is meant to aid in setup and recovery.

Part 4 - Docker

Installed by default. Might need to install docker-compose and docker-machine. RancherOS has some read-only partitions so you have to take care where you put stuff.

Part 5 - Docker/VMware volumes

I need to store or persist my services in my persistent storage. There are several challenges in this configuration. [1] cannot share RW volume with multiple containers on multiple swarm nodes. [2] cannot snapshot these volumes [3] there are essentially hosted volumes meaning that cannot be access on other hardware nodes.

It's undefined right now but it might be possible to put a network filesystem in front of the VMware storage but I'm not there yet.

In this config I'm planning to deploy a traefik server and a gitlab instance.

docker volume create --driver vsphere traefik
docker volume create --driver vsphere gitlab

Verify the volumes

docker volume ls

Part 6 - DNS and nameserver

This is a mixed bag here... register your domain, set your nameserver, get whatever DDNS credentials you need and then continue with the traefik setup.

Part 7 - default traefik config

Traefik configuration is pretty plain. The basics are here and here. Once you know what you're basic configuration is going to look like then mount an alpine container and create the files.

docker run --rm -it -v traefik:/traefik alpine /bin/sh

Head over to the traefik folder and create the 3 require files. Set the chmod correctly and then paste the file fragments.

cd /traefik
touch acme.json docker-compose.yml traefik.toml
chmod 0600 acme.json

And create the traefik network

docker network create --scope swarm --attachable --driver overlay traefik

Part 8 - Launch traefik

docker stack deploy -c docker-compose.yml traefik

a lot more to come....


Popular posts from this blog

Entry level cost for CoreOS+Tectonic

CoreOS and Tectonic start their pricing at 10 servers. Managed CoreOS starts at $1000 per month for those first 10 servers and Tectonic is $5000 for the same 10 servers. Annualized that is $85K or at least one employee depending on your market. As a single employee company I'd rather hire the employee. Specially since I only have 3 servers.

The pricing is biased toward the largest servers with the largest capacities; my dual core 32GB i5 IntelNuc can never be mistaken for a 96-CPU dual or quad core DELL

If CoreOS does not figure out a different barrier of entry they are going to follow the Borland path to obscurity.

UPDATE 2017-10-30: With gratitude the CoreOS team has provided updated information on their pricing, however, I stand by my conclusion that the effective cost is lower when you deploy monster machines. The cost per node of my 1 CPU Intel NUC is the same as a 96 CPU server when you get beyond 10 nodes. I'll also reiterate that while my pricing notes are not currently…

eGalax touch on default Ubuntu 14.04.2 LTS

I have not had success with the touch drivers as yet.  The touch works and evtest also seems to report events, however, I have noticed that the button click is not working and no matter what I do xinput refuses to configure the buttons correctly.  When I downgraded to ubuntu 10.04 LTS everything sort of worked... there must have been something in the kermel as 10.04 was in the 2.6 kernel and 4.04 is in the 3.x branch.

One thing ... all of the documentation pointed to the wrong website or one in Taiwanese. I was finally able to locate the drivers again: (it would have been nice if they provided the install instructions in text rather than PDF)
Please open the document "EETI_eGTouch_Programming_Guide" under the Guide directory, and follow the Guidline to install driver.
download the appropriate versionunzip the fileread the programming manual And from that I'm distilling to the following: execute the answer all of the questio…

Prometheus vs Bosun

In conclusion... while Bosun(B) is still not the ideal monitoring system neither is Prometheus(P).


I am running Bosun in a Docker container hosted on CoreOS. Fleet service/unit files keep it running. However in once case I have experienced at least one severe crash as a result of a disk full condition. That it is implemented as part golang, java and python is an annoyance. The MIT license is about the only good thing.

I am trying to integrate Prometheus into my pipeline but losing steam fast. The Prometheus design seems to desire that you integrate your own cache inside your application and then allow the server to scrape the data, however, if the interval between scrapes is shorter than the longest transient session of your application then you need a gateway. A place to shuttle your data that will be a little more persistent.

(1) storing the data in my application might get me started more quickly
(2) getting the server to pull the data might be more secure
(3) using a push g…