Saturday, January 24, 2015

VirtualBox, Vagrant and NixOS

Most Linux versions, and possibly all Unix variants, provide a customizable welcome message when logging in and one of the things I have come to expect is the IP address in the message. This is especially useful when using VirtualBox or VMware. However, if you're running headless and you're not publishing your guests IP address to a local DNS server then things are going to get a bit challenging.

One of the features I like when using Vagrant is that you can get the IP address of the current guest with a command: vagrant ip. Of course the vagrant config has to be in the cwd so that can be a challenge. And there can be more than one instance too... so bouncing between guest folders is a nuisance.

Here is a NixOS/Vagrant plugin that should be helpful. I also found this article with a couple of sample commands for VirtualBox like:
VBoxManage guestproperty get NixOS "/VirtualBox/GuestInfo/Net/0/V4/IP"
And since I also had a local-only IP address I needed to execute this to get that IP:
VBoxManage guestproperty get NixOS "/VirtualBox/GuestInfo/Net/1/V4/IP"
I suppose there could be a wildcard value to stick in there, however, it's still very difficult to infer which device is hosting which port... but it's still some good info.

And then there is my new favorite command.  Starting the machine in headless mode.
VBoxManage list vms
VBoxHeadless --startvm NixOS
"NixOS" is the name I have my guest and it is in the list from the 'list vms' command.

UPDATE: this is not the polite way to start a guest because it (a) runs in the foreground, and (b) when shutdown inside the guest the command does not terminate. This is less than ideal behavior and requires investigation. There is a way to launch a guest headless from the VirtualBox GUI(shift+double-click).

Thursday, January 22, 2015

Installing NixOS - Short Version

I have been experimenting with NixOS for the last few weeks and every day I find some new reason to like it.

My first experiences were with the VirtualBox version. It was easy to use and the default configuration, with KDE, was pleasant even though I went directly to the Konsole and then with ssh. On the list of interesting features:

  • the Nix installer - it's the foundation of the team's approach across the project
  • related to the installer is the user partitioned installations... i.e. two users can use different versions of the same program since their environments are partitioned
  • and that the installer is transactional
  • It has it's own approach to containers - which I do not fully understand but seems more like chroot or jail than it does Docker. Additional good news is that the container also works like the package manager ... and you can install Docker too.
  • Then I discovered that the project includes a CI, continuous integration, server called hydra. I don't know much about it except that hydra also uses the same Nix ideas and between the two hydra will work with Windows, OS X, and Linux.
Today I installed sshd and started to consider doing a complete install from scratch or at least the closest thing to it. Not knowing how NixOS liked to be installed and that their own manual was a little sparse on the details I found this article that gave me a good head start. I suppose I could just leave the article at that, however, I want my own interpretation of the article in a more concise form; so here we go:
  • download the ISO image, in my case I did the minimal version [no X or desktop]
  • create a VM using VirtualBox and mount the ISO image on the VM's CDROM (I configured it for 512M and 16GB disk)
  • boot the VM
  • run some commands and do some work
    • fdisk /dev/sda
    • press "o" to create a DOS partition
    • press "n, p, 1, ENTER, +2G, t, 82" to create the boot/swap partition
    • press "n, p, 2, ENTER, ENTER" to allocate the rest of the drive for NixOS
    • make the swap partition "mkswap -L swap /dev/sda1"
    • turn the swap on "swapon /dev/sda1"
    • format the partition "mkfs.ext4 -L nixos /dev/sda2"
    • mount the formatted partition "mount /dev/disk/by-label/nixos /mnt"
    • generate the default config file "nixos-generate-config --root /mnt"
    • edit the configuration file "nano /mnt/etc/nixos/configuration.nix"
      • boot.loader.grub.device = “/dev/sda”
      • services.openssh.enable = true
      • I must have janked the user creation because it did not work as expected
    • install NixOS "nixos-install"
    • shutdown the VM "halt"
    • power off the VM from the VirtualBox GUI
    • eject the virtual CDROM
    • add a second network adapter "host-only"
    • re-start the VM
At this point you are pretty close but if you are using VirtualBox you'll have to do at least one more thing in order to be able to ssh into your newly minted system.  Create a new user:
  • login as the root user. Right now the root user does not have a password
  • change the root user's password "passwd"
  • create the new user "useradd -m myusername"
  • change the user's password "passwd myusername"
  • and if the user is an admin user then you need to add this user to the admin group "user mod -G wheel myusername"
Now you should be able to ssh into the VM from your host computer with your newly minted username. I installed vim in my user account because I like it as my preferred editor. 

At this point I shutdown and created a snapshot so that I could return to this state or create clones of my NixOS installation. I hope to find myself going through a number of use-cases shortly:
  • installing packages from the unstable channel
  • using containers
  • installing the Hydra CI
  • trying to determine the Docker play
  • determining the host upgrade workflow
  • physical drive crypto
  • golang and other compiler tech as part of the development stack
  • fossil and/or the hydra storage
It's fun to note that my first 16GB installation left me with 12GB free after the base OS and vim were installed. Based on the way the packages are installed it appears that NixOS is going to take more diskspace than a like Ubuntu or Fedora, however, there may be other economies that are yet to be discovered.

Wednesday, January 21, 2015

"switcher" multiplexing ssh and http over the same port

The Switcher project is an interesting implementation of a dual protocol mux. What is even better is that it's written in go and it's in the same class of solutions as grok. Keep in mind it's irresponsible, immoral and possibly illegal to tunnel through some networks ... after watching this mouse computer promotional video it might be hard for some folks to draw the line or know where the line is drawn.

pub/sub message queues containing blobs

I started off with a story. FAIL. Here's the plan:
  • save the blob to a key-store using a UUID as it's key. The key-store should support a TTL with callback.
  • put the UUID in the MQ
  • The MQ is always going forward
  • the transaction is only every touched by one service at a time
  • If there is a reason to fork the transaction... then each service should replace the UUID and insert a new blob in the key-store.
It is my experience that most services in a transaction only effect a portion a portion of the blob at a time and so copying around is costly.

**Redis has some primitives that allow you to have hashes of array such that you can append to an array as the transaction moves around the system instead of trying to log-aggregate the events after the fact.

Tuesday, January 20, 2015

Docker's machine project is not a CoreOS contender

I'm not sure what Solomon Hykes was thinking when he posted this:



The docker project is strong all by itself and while there appears to be no loveless between CoreOS and Docker... the machine project from the Docker team is no contender to CoreOS. CoreOS is an OS and it stands on it's own as an operating system.  Machine, on the other hand, is a tool for launching containers on different platforms like Vagrant, Google Compute, VMware ... in fact the actual orchestration has been implemented by many 3rd parties.

The statement is clearly inflammatory and not a serious proposition. In fact the opposite is true if you read any of the Phusion posts. The headline on the Phusion website is "Your Docker Image Might be Broken". This tells me is that there is a bigger fundamental problem with Docker.

Saturday, January 17, 2015

DVCS everywhere and everywhere else!!!

Some time ago I wrote about version control systems and storing secrets. And I still think it's a fools errand because you'll never be able to prevent it and education and awareness is the only thing that is going to reduce the incidences.

But now I'm thinking about the footprints left behind... when you cancel or close an email account... cancel or move a public project, team or repository.

For example I have a number of projects that I hosted on both bitbucket and github. In most cases it was more of a land grab than proper use. But in the end it was meaningless.

  1. if I cancelled the project the "brand" remains and someone like a squatter is going to reuse it eventually as there is a google footprint and is likely to make them something
  2. Someone is just as likely to clone the project and use it in a social engineering way to leverage your brand reputation to do bad things
  3. in the case of emails there is a good chance someone is going to receive emails that robots sent you and which might leak personal information, passwords, or grant access to 3rd party systems in the form of "forgot my password" challenges.
I'm fairly certain google cancels an email address once it is canceled or abandoned. It's the sort of thing we should all do.

UPDATE: By DVCS I'm referring to service providers and not self hosted repositories.

Google Support

I was helping my sister in-law recover her Google calendar yesterday when I ran out of options.  I needed Google Support. And the only way you get Google Support is by paying for your Google Apps for Work. Actually getting to support was novel and with just a little friction sprinkled in. While the outcome seems to be a success it's not because of Google but due to a 3rd party app.

Logging in as her Google App admin I found the support button.  No matter how I got there whether it was google search I ended up in the same basic place. There was a moment when I thought I was going to be able to get into the support system and at least ask the question whether or not I could recover her calendar but that never happened. Everything pushed me back. So I finally got her approval to get a paid account... $5 per user per month.

  1. upgrade your account to paid
  2. login as admin
  3. press the support button
  4. press the GET PIN button
  5. the pin is good for 60 minutes
  6. call the support number
  7. enter the pin
  8. when the human answers they will confirm or record contact info that you previous entered...
At this point I recounted my problem. When the support rep repeated what I was asking back to me he said he would consult his peers and get back to me. He returned in less than 5 minutes with the following:
  1. If the calendar was deleted then all hope is lost. The calendar program clearly states that it would not be recoverable.
  2. If someone deleted individual items they could be recovered, however, that would mean using a 3rd party application from the application marketplace... which was free.
  3. Oh, and there is one piece of information missing from your account.... click and click... please select a billing plan.
While I had agreed to the upgrade it was still free for 30 days and Google had not prompted me for billing information. The words he chose for #3 was alarming. I was concerned that while I had been using the free account for years that I had missed a configuration option or something that might put the whole thing at risk... NO! I'll deal with billing later.


While the calendar undelete application worked; it was not flawless. There are calendar entries that would not undelete and without an explanation. Additionally, after completing any task I was prompted for their not-so-free backup service. I'm sure it's a great idea. Clearly Google has not seen fit to provide some of the basic application features that had I a desktop application would likely have been able to recover from.

As for the cost, $5/user/month is not bad and $10/user/month for unlimited disk space is even better. I'm sure that Google has plenty of metrics that prove that it's cost effective for everyone. My overarching concern is that the cloud marketplace is now supplanting my desktop marketplace and the tools are still not in parity.