Skip to main content

My New Python Project Setup

[update 2012-01-18] postgres has been updated to 9.1.2; the latest version as of today.

[update 2012-01-17] feel free to ignore my comments about Lua. While Lua might sit in an interesting place between Python and Java in an embedded/scripting place. The fact that lunatic-python does not compile and lupa depends on LuaJIT2 which is compatible with Lua 5.1 and the current Lua version 5.2 was recently released... and the comment from the LuaJIT team about adoption was a little snarky. I gotta think about something else.

[update 2011-12-29] I forgot to add twitter's bootstrap CSS/JS. I'll cover that in a future post when I also discuss modern-package-template

It's pretty simple to set things up. There are some prerequisites and some basic install packages that need root access but the intent is to get the config in userspace as soon as possible. This article covers VM slices at Rackspace using Ubuntu 11.10.

First: Install and update:

  • allocate the OS

  • select the OS and wait for it to complete.  You're going to receive an email with the root password

  • login, change the root password

  • create an "admin" privileged user (usually my name or "builder")

  • add this user to the sudo

  • change it's password

  • edit /etc/ssh/sshd_config and disable root login

  • update the package definitions (apt-get update)

  • upgrade the packages (apt-get upgrade)

  • reboot

Second: Install the required roo packages

  • postfix - when prompted select the default values

    • apt-get -y install postfix

  • apt-get -y install python-setuptools daemontools daemontools-run python-dev mailutils mutt build-essential uuid-dev python-nose vim htop sysstat dstat ifstat screen locate apache2-utils unzip siege python-virtualenv bwm-ng libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev fail2ban openssl libssl-doc openvpn libssl-dev libgcrypt11-dev lighttpd lighttpd-dev libevent-dev libcurl4-openssl-dev  libreadline6-dev beanstalkd tree

  • apt-get install postgresql-9.1 postgresql-client-9.1 postgresql-doc-9.1 postgresql-plperl-9.1 postgresql-plpython-9.1 postgresql-server-dev-9.1

  • easy_install pip

  • easy_install mercurial

  • easy_install pycurl

  • pip install virtualenvwrapper

That's it, essentially, for the second layer, however, here's an explanation of the modules from a macro perspective:

  • python-setuptools - make the installer, easy_install, available

  • daemontools daemontools-run - there are so many ways to implement a 'daemon' this tools make it simple to make daemon deployment simple

  • python-dev python-nose python-virtualenv - basic prereqs for python development. virtualenv is needed so that packages can be installed in userspace

  • mailutils mutt - generate emails

  • build-essential uuid-dev  - basic developer tools

  • vim screen - editor and console tool

  • htop sysstat dstat ifstat locate unzip bwm-ng - debug /monitoring tools

  • libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev - libs used when rendering usage graphics

  • fail2ban - detect login attempts and put the IP in time-out

  • openssl libssl-doc openvpn libssl-dev libgcrypt11-dev libcurl4-openssl-dev - crypto

  • lighttpd lighttpd-dev - web server that should be in front of the framework

  • libevent-dev - kevent, kpoll libs

  • apache2-utils siege - performance simulation tools

  • beanstalkd - message queue

And finally the third layer, the userspace framework layer. But before you start installing packages you need to create the virtual environment:

  • cd ${HOME}

  • mkdir -p src

  • cd ${HOME}/src

  • virtualenv currentenv

  • . ./currentenv/bin/activate

Now install the third layer.

  • pip install tornado

  • pip install flask

  • pip install flask-rest

  • easy_install pip

  • pip install pycurl

  • pip install simplejson

  • pip install tornado

  • pip install Fabric

  • pip install PasteDeploy

  • pip install PasteScript

  • pip install modern-package-template

  • pip install requests

  • pip install gevent

  • pip install pystache

  • pip install nose

  • pip install redis-py

  • pip install pymongo

  • pip install hoover

  • pip install pyzmq

  • pip install pyyaml

  • pip install beanstalkc

  • pip install django

  • pip install django-redis-cache

  • pip install clint

  • pip install djangorestframework

  • pip install pyparsing

  • pip install flup

I'm hoping that there is a practical use-case for embedding Lua in Python. There are an few interesting projects like lunatic-python and lupa. Normally I would not consider Lua for anything beyond "hello world", however, the redis team is embedding Lua, it seems like a very lightweight codebase, it can be embedded in just about any language (do a google search).

  • cd ${HOME}

  • mkdir -p tmp

  • cd tmp

  • wget

  • tar zxvf lua-5.2.0.tar.gz

  • cd lua-5.2.0

  • make linux

  • sudo make install

NOTE: the lunatic project does not compile under Lua 5.2. So this thread is postponed for now.

  • pip install lunatic-python

Alternatively I tried [lupa] but that requires LuaJIT 2.0 which is currently in beta (version 9)

  • cd ${HOME}

  • mkdir -p tmp

  • cd tmp

  • wget

  • tar zxvf LuaJIT-2.0.0-beta9.tar.gz

  • cd LuaJIT-2.0.0-beta9

  • make

  • sudo make install

  • sudo ldconfig

Then install lupa.

  • pip install lupa

NOTE: hoover is a client library for  You'll need an account if you want to use this service.

In closing, I would like to include a few more libraries, however, the current version in apt-get is too old. I'd prefer installing them from scratch. They are necessary packages so for the time-being I'm just going to list them. They should be installed when installing the first layer and by the root user (or sudo)

  • ZeroMQ - trivial to build and deploy if you follow the instructions

    • cd /tmp

    • wget

    • tar zxvf zeromq-2.1.11.tar.gz

    • ./configure

    • make

    • make install

    • ldconfig

  • MongoDB - (can actually be installed in userspace)

    • cd /tmp

    • wget

    • mkdir -p ${HOME}/bin

    • cd ${HOME}/bin

    • tar zxvf mongodb-linux-x86_64-2.0.2.tgz

    • sudo mkdir -p /data/db

    • sudo chown `id -u` /data/db

    • ./mongodb-linux-x86_64-2.0.2/bin/mongod

    • ... or ...

    • cd ${HOME}/bin

    • find ./mongodb-linux-x86_64-2.0.2/bin/ -type f -exec ln -s {} \;

    • ./mongod

  • Redis - trivial to build and deploy if you follow the instructions

    • cd /tmp

    • wget

    • tar zxvf redis-2.4.6.tar.gz

    • cd redis-2.4.6

    • make

    • make install

  • SQLite - a simple SQL DB

    • cd /tmp

    • wget

    • tar zxvf sqlite-autoconf-3070900.tar.gz

    • cd sqlite-autoconf-3070900

    • ./configure

    • make

    • make install

    • ldconfig

  • ISO8583 - ISO8583 lib

    • # will use virtualenv if it is configured

    • cd /tmp


    • unzip ISO8583\

    • cd ISO8583\ Module-1.2

    • python ./ install

Good luck!

PS: You should consider scripting this installation so that the deploy can be automated.  Specially via Fabris, chef, or puppet.


  1. [...] start our next project. Some of the installation is going to overlap with the python installation (here). You can skip all of the userspace installation but the sudo installation is all you’ll [...]


Post a Comment

Popular posts from this blog

Entry level cost for CoreOS+Tectonic

CoreOS and Tectonic start their pricing at 10 servers. Managed CoreOS starts at $1000 per month for those first 10 servers and Tectonic is $5000 for the same 10 servers. Annualized that is $85K or at least one employee depending on your market. As a single employee company I'd rather hire the employee. Specially since I only have 3 servers.

The pricing is biased toward the largest servers with the largest capacities; my dual core 32GB i5 IntelNuc can never be mistaken for a 96-CPU dual or quad core DELL

If CoreOS does not figure out a different barrier of entry they are going to follow the Borland path to obscurity.

UPDATE 2017-10-30: With gratitude the CoreOS team has provided updated information on their pricing, however, I stand by my conclusion that the effective cost is lower when you deploy monster machines. The cost per node of my 1 CPU Intel NUC is the same as a 96 CPU server when you get beyond 10 nodes. I'll also reiterate that while my pricing notes are not currently…

eGalax touch on default Ubuntu 14.04.2 LTS

I have not had success with the touch drivers as yet.  The touch works and evtest also seems to report events, however, I have noticed that the button click is not working and no matter what I do xinput refuses to configure the buttons correctly.  When I downgraded to ubuntu 10.04 LTS everything sort of worked... there must have been something in the kermel as 10.04 was in the 2.6 kernel and 4.04 is in the 3.x branch.

One thing ... all of the documentation pointed to the wrong website or one in Taiwanese. I was finally able to locate the drivers again: (it would have been nice if they provided the install instructions in text rather than PDF)
Please open the document "EETI_eGTouch_Programming_Guide" under the Guide directory, and follow the Guidline to install driver.
download the appropriate versionunzip the fileread the programming manual And from that I'm distilling to the following: execute the answer all of the questio…

Prometheus vs Bosun

In conclusion... while Bosun(B) is still not the ideal monitoring system neither is Prometheus(P).


I am running Bosun in a Docker container hosted on CoreOS. Fleet service/unit files keep it running. However in once case I have experienced at least one severe crash as a result of a disk full condition. That it is implemented as part golang, java and python is an annoyance. The MIT license is about the only good thing.

I am trying to integrate Prometheus into my pipeline but losing steam fast. The Prometheus design seems to desire that you integrate your own cache inside your application and then allow the server to scrape the data, however, if the interval between scrapes is shorter than the longest transient session of your application then you need a gateway. A place to shuttle your data that will be a little more persistent.

(1) storing the data in my application might get me started more quickly
(2) getting the server to pull the data might be more secure
(3) using a push g…