Friday, September 30, 2011

Agile is still dead and has been since 1991

[updated 2011.09.30] yet another response to Agile is good.
When you have so much of you career invested in something like Agile, XP etc... it can be hard to see the forest for the trees. I had a consulting job in The Haag many years ago. IBM was the incumbent contractor at the customer site (a bank) but after 5 years on the job they had not written a single line of functioning code. In the office there were two teams of software people... both behind closed doors. The first team was the Data team and the second team was Functional. They rarely spoke and they never shared information. I was there for a week, introduced the client to OO and we had a functioning prototype. Smart people do smart things, You cannot make an underachiever exceptional by using Agile. Either they get "it" or they don't.

I just commented on a blog. I'm sure there is some validity to his post beyond observing that Agile Scrum is broken. It certainly is not what it was originally intended but for that you have to go way back to Dave and Andy from PragProg. I have not found the original links and references myself but I recall enough from my reading at the time. Today's Agile does not look anything like what it was.
Scrum deflects from individual accountability. But the failure of agile/scrum is probably more psychology than technology. There are essentially three groups of people. 1) the high achievers that you want everyone to emulate; 2) the average devs; 3) and the low achievers. The high achievers hate these processes because it simply adds friction to their day. The low achievers love it because it deflects individual responsibility (think Survivor or Big Brother; the best do not always win. Floater is a legitimate strategy). And the average achiever is ambivalent and can be tipped either way.

Everyone is different and creative in their own way. You cannot herd cats with agile or scrum.

Agile does not treat people with respect. It dictates with strict rules what and how things are to be done. When the reality is that we have individual and collective responsibility. And most of that is encapsulated in an unwritten "bill of rights" that we carry around with us. At least the high achievers do. Translating that BOR to everyone else with a wide brush is simply too general an approach. Improvement from the under achievers comes from training, education, and more then anything else experience and experience from making mistakes.

The real chalenge here is that businesses are trying to grow their ranks as fast as they can. Many times that means hiring people that they would not hire if there was not such a shortage. This is a different set of problems from Agile and Scrum. This problem can be fixed by being selective, selecting people with aptitude for the work, selecting tools that lower the bar of entry, and managing people rather then allowing them to manage themselves.

Thursday, September 29, 2011

Still managed a Mojo upgrade

Even in my half inebriated state I managed to upgrade my Mojolicious development installation from version 1.98 to 1.99. I cannot wait to restart my latest perl project... with only 2 parts written so far.
$ sudo -s 'cpanm Mojolicious'

CPAN/cpanminus might be the second killer app. (gems and peaks simply do not measure up)

Friday, September 23, 2011

What will be the effect of internet memory?

I thought I would have a deep philosophical viewpoint but after walking the dog and considering the obvious it's not that difficult. The challenge is that the generations that follow me already have a more open viewpoint of the online social experience. They are no less opinionated, fearful, or cliquish than my generation but their memories are much shorter.

I'm not sure if that is because my-gen stores it's knowledge base in local or near "cache" and the next-gen uses google.

So what is going to happen you go to take that all important career making interview and the interviewer pulls out a stack of blog posts or pictures that demonstrate questionable moral character or outright disdain for everything lawful and good.

Will it be easy enough to say "I've changed my opinion since then" or "I was just acting out childhood angst" or my new favorite... "I was publishing without the filter of a professional editor"? We are being judged all the time.. Whether it's because we contributed to a project code tree without checking in comments or PEP-8, submitted a bug report with poor grammer that was fixed anyway, or wrote openly about a crush on a teacher or professor. (these are not my personal experiences).

But what is to come of it all? Specially if you were interviewing for a local fast food establishment or secretary of state. It seems that it might be on a sliding scale.

So as you read my articles... I'm not trying too hard to express too strong of an opinion that I cannot change my mind as I often do. I won't be slamming "the law" as they keep me safe and that job is for the regular press and Assange.

The internet memory is long and potentially unforgiving... and always subjective and subject to interpretation. Good luck to us all.

Wednesday, September 21, 2011

Message Queues and nothing but Message Queues

[Update 2011.09.21] The ink had hardly dried on this post when I decided to quickly evaluate gearman. It's immature and direction internally seems dizzy. So pass on this one too.

I'm reading up on MQs again while I'm waiting for a conference call. I do not want to disrupt my code before the demo (rule #1 of demo-club). So I'm making yet another list of all of the MQs out there.

  • ActiveMQ

  • RabbitMQ

  • Amazon SQS

  • Google Queue/Task

  • Gearman

  • ZeroMQ

  • Sparrow

  • Starling

  • Kestrel

  • RestMQ

  • Oracle Advanced MQ

  • IBM Websphere MQ

  • MicrosoftMQ

  • JBoss Messaging

  • Sun Open Message Queue

  • Apache Qpid


There are several ways to compare these MQs. Core source language, client language support, implementation detail, inspiration or initial design, current activity, licensing, cost, deploy OS, performance/TPS, use-cases, dependencies, deep dependencies, mindshare.

So here is that list again.

  • ActiveMQ - apache and meant for Java JMS although there are libs for other languages it really depends on the actual message payload.

  • RabbitMQ - build with erlang, while it works well and is the MQ for ejabberd it's pretty heavy weight.

  • Amazon SQS - it's not free or at least cost effective. They have had huge outages. For security reasons this has to be onsite.

  • Google Queue/Task - For security reasons this has to be onsite and only supports java, python and possibly go.

  • Gearman - There is potential here. It seems to have been rewritten in C from perl. It is open source and free to use. They appear to be active and there is a CLI making access easy. This is worth looking into. (the only downside is that it's written in Java)

  • ZeroMQ - 0mq leaves a lot to the developer or architect to decide and fill in the blanks. My original implementation used beanstalkd and it was razor fast and trivial to implement. 0mq has plenty of gotchas and the burden is on the developer. I'd use it again but it's not going to be #1.

  • Sparrow - ruby? This is just not going to make the list.

  • Starling - another ruby implementation.

  • Kestrel - argh... this one is implemented in scala. So you get all of the non-functional java components (unverified) and long namespaces.

  • RestMQ - I like the pythonic implementation, however, while twisted is a solid project it appears to be conflicting with my main project based on tornadoweb.

  • Oracle Advanced MQ - not free

  • IBM Websphere MQ - not free

  • MicrosoftMQ - not free

  • JBoss Messaging - too many dependencies. This is akin to J2EE's JMS. And for that I might as well use ActiveMQ.

  • Sun Open Message Queue - They do not live here any more.

  • Apache Qpid - supports AMPQ and trying to get 100% compliant. I suppose this is interesting in that disparate systems will be able to communicate. Looking at the source tree there is just so much code and since I do not want to dedicate this kind of time to it... I think it's an easy PASS.

  • beanstalkd - The last release was over a year ago. I've posted on their group hoping to get some sort of answer. There has been some activity on their website. The APIs have a TTR novelty API that I like. Specially in my fire and delete application.


One of the novel things about the apache team is that when they acquire a technology it's long before there is a wrapper around it. For example Apache Camel.

So here is the state of things.

  1. RestMQ - I'm going to continue my attempt to rewrite RestMQ in perl using Mojolicious. If I get some decent results then I'll publish them/it.

  2. Gearman - deserves some immediate investigation. I'm hoping to make it through the painful documentation.

  3. Beanstalkd - while it has not had much recent action I'm hoping to see some info in my inbox shortly. I like their broker and while they do not specifically support req/resp it's not really needed and can worked around.

  4. ZeroMQ is just an API. My broker is getting more and more complicated. Message routing is also getting complicated.

pydoc - the anti-killer appp

Some time ago I wrote about perldoc being perl's killer app. I still think that's true. Recently I've been going through my python code and attempting to add useful comments that could be picked up by some auto documenting mechanism. Clearly they exist... python.org is full of doc and I'm certain it's rendered.

Included with python, since 2.1, is pydoc. At first glance it appears to be an analog to perldoc; and for the most part it isn't.

  • pydoc will generate text or html - perldoc will convert the poddoc to html, latex, man, text

  • pydoc has a graphic UI embedded - but it requires TK (and x-windows)

  • pydoc has an embedded http server - could leak important data if the pydoc is internal use only.

  • perldoc generates static files - pydoc generates dynamic files

  • pydoc recurses the class dependency tree - if the stack is deep the document is long (there is a lesson in this)


In a way, it's nice that pydoc scans your code looking for your comments in your classes etc. The format makes sense and it's not all that terrible. There are even ways to comment individual functions instead of just one monster perldoc (as typically demonstrated)

So perldoc is still the killer app.

ref: perlpod - perldoc.perl.org http://bit.ly/nNhp58

Tuesday, September 20, 2011

mojolicious – first app (part 2)

I was going to call it a night but I just wanted to make a little more progress on this first app. I'd like to think I'm tired and cranky because I'm not seeing it. Mojolicious generated the following as my first app:
#!/usr/bin/env perl
use Mojolicious::Lite;

# Documentation browser under "/perldoc" (this plugin requires Perl 5.10)
plugin 'PODRenderer';

get '/welcome' => sub {
my $self = shift;
$self->render('index');
};

app->start;
__DATA__

@@ index.html.ep
% layout 'default';
% title 'Welcome';
Welcome to Mojolicious!

@@ layouts/default.html.ep
<!doctype html><html>
<head><title><%= title %></title></head>
<body><%= content %></body>
</html>

That's everything. It's their version of hello world. The app started right away, however, it was my fault.  I pointed my browser to http://127.0.0.1:3000// and the page that popped up was some sort of 404 error page. It was nice enough... but much larger than my screen so I had to scroll to see everything.

Which is when I realized that I was not seeing my project. After a quick code review I changed my browser to load http://127.0.0.1:3000//welcome and my welcome page arrived. However since there was no default page (aka index) or default redirect. I started looking.

Skimming the documentation I found something that looked like:
get '/*' => sub {
. . .
};

This had promise as I added a redirect.
get '/*' => sub {
my $self = shift;
$self->redirect_to('/welcome');
};

My only problem is/was that it would not recognize or redirect http://127.0.0.1:3000/. I had to read a lot more doc and make a few assumptions as to what was going on. The mojo docs indicate that the pattern could have been a regex, however, their pattern matching is more efficient just because it is and based on the way most people use it. And in their pattern matcher '/*' indicates that they are greedy matching placeholders and not a pattern per say. (I think that the * indicates that there has to be at least one byte/char)

The strange thing is that '/' and '/*' are not the same string and both will not match the empty path. Only '/' will.

So in my app I have a '/welcome' in order to say hello. A '/*' to get all of the non-empty paths so that I can redirect them to '/welcome' and I have a '/' in order to catch the empty path. What a pain in the ass! At least I only have a few pages.

ELAPSED TIME: 30min

mojolicious - first app (part 1)

[Update] I should have mentioned that the elapsed time was only about 15 min; maybe less.

I'm a perl programmer from way back and while it has been 2 years since I've written any perl of consequence I've decided that my client's application was going to be implemented in perl... just for the fun of it. (I've been writing a lot of python+tornadoweb+redis+mongodb and I'm very comfortable in this space. Mojolicious is a half step outside my comfort zone because I really do not want to rewrite this app in python if I don't have to. I'm a big believer in DRY)

I'm certain that everything is installed. I used the same installation for documentation's sake recently:
wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.0.tgz
# need to untar and then relocate the files to the path

# MOJOLICIOUS - (as root)
curl -L cpanmin.us | perl - App:cpanminus
curl -L cpanmin.us | perl - Mojolicious
curl -L cpanmin.us | perl - Redis
curl -L cpanmin.us | perl - ZeroMQ
curl -L cpanmin.us | perl - JSON
curl -L cpanmin.us | perl - MongoDB

And we should be ready for the first app.

The application folder structure is going to look like this:
.
|-- COPYRIGHT
|-- INSTALL
|-- LICENSE
|-- README
|-- bin
|-- data
|-- doc
|-- src
`-- test

It's far from perfect and it's missing the Mojolicious application (webapp). So let's create a mojo app (adapted from the docs):

Nope, not yet. First you have to know whether you are going to create a lite_app or a full-blown app.
mojo generate lite_app

or
mojo generate app

The difference is quite extensive. The second selection is probably best suited for anything more that a a few dynamic pages. And the lite app seems suited for all-in-one file deployment. Right now I'm not sure which is best for this design problem, however, I know that I can convert later; I can cut-paste everything and with any luck I won't be repeating myself.

So before I make that decision. What does my application look like.

  • I need a login screen and session state so that the data is private for my client. One user is sufficient.

  • I need to upload a CSV file to a file system and queue that file for processing.

  • I need a background process that looks for queued files and then processes them.

  • The process is as simple as a GET against a URL, storing the result in another CSV and then getting the next record from the input file.

  • Once all of the records have been processed the second file is converted to a PDF of mailing labels.

  • The PDF can be downloaded from inventory.

  • and finally the user should be able to delete the PDF, the original CSV and the intermediate CSV.


So that's pretty simple. I need a few pages:

  • login

  • error

  • queue status

  • upload

  • are you sure (popup??)

  • completed (optional, could always be inline)


So for the time being lite_app it is. This is what I did:
mkdir webapp
cd webapp
mojo generate lite_app lisapp.pl

Notice that I specified my application name (lisapp.pl). Starting the app was a breeze eve though it bugs me that the mojo team misused daemon and does not even want to be open minded about it. Oh well, it's a good platform in spite of 'daemon'. And then I noticed the output:
$ ./lisapp.pl daemon
[Wed Sep 21 00:13:47 2011] [info] Server listening (http://*:3000)
Server available at http://127.0.0.1:3000.

Notice that the output is stating that it's listening on '*' and 127.0.0.1.  What's up with that? When I did a 'netstat -ln' I only had one instance of port 3000.
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN

I'm not certain I understand the motives here. But I have to say that it drop be nuts because I knew that I had to test on 0.0.0.0 since my server is at Rackspace. And at first glance I only saw the 127.0.0.1. So it's on me to pay attention and on them to be more clear.

When I pointed my browser to the URL it popped. As I expected since this is a very lightweight app. So for now I will take a break. Next time I will add the login screen. It's part of the 3rd screencast and pretty simple. I also need to read-up on the under command. It seems to be where all the power comes from. Keep in mind that the home page is going to be a login screen. No data should leak out. Everyone logs in.

Friday, September 16, 2011

Benchmarks - Hello World

If you've ever worked for a credit card company around the winter holiday season then you appreciate, as I do, benchmarking your application; early and often. Usually, however, this sort of benchmarking is meant to identify slow parts of the system as well as potential bugs that might have mad it through development. In this case testing usually centers around a very small set of transactions. Transaction types with the highest volume.

The "hello world" benchmark, on the other hand, is the complete opposite. It tells you some interesting things about your application but only the smallest bit.

Consider the hello_world that the different web frameworks are now benchmarking against. Even the slowest webservers are functioning under 100ms and most of that time is socket negotiation. For example, in tornadoweb, only executes about 35 lines of code in order to get the desired response. Since TW is single threaded all of the speed and performance is pushed to the python implementation and the underlying libs. But the "work" is non-existant and is not being tested at all. At that point we'd be talking about the implementation language and it's performance within the context of the transaction. And that has nothing to do with benchmarking a web framework.

So I'm of the opinion that benchmarking individual webservers is junk. The results are meaningless. How do we get better numbers?

Well, you can use the hello world benchmark to start, however, you need to include similar benchmarks of other webservers that use the same strategy. For example; a hello world benchmark for Mojolicious should be used in conjunction with a benchmark of Tornadoweb. To my knowledge they share enough similarities in their infrastructure/implementation that you might be able to extract the actual performance of the webserver away from the dependencies. (both webservers are single threaded and use [prefork] multi-process in order to scale.)

Of course I could be totally wrong about tornadoweb and mojolicious, however, the ideal design for this type of framework is a very lightweight webserver that forwards requests to an MQ that a worker processes. (I'm not sure if there is a pattern name here.)

Everything is in the cloud - it's all damage control now

There was a time when I was really concerned about what information was "out there". Then I tried to control that information. And finally I realized it was hopeless. Now it's just a matter of trying to keep it clean and ongoing damage control. Not that I've done anything wrong or embarrassing in my life...

The fact remains that the likes of google, yahoo, "social media", advertisers, and so are... while they might really want my demographic information, spending habits, income, neighbors, surfing habits, they will accept my anonymous information just the same. And I give them plenty of both.

The way it works... you visit a website. The site owner has contracted with someone like google for some sort of service. Whether it's advertising, sales or just some general tracking information it's not important. That service drops a cookie on your browser. Later, you visit another site. That site contracts with the same service provider... and looks for a cookie that they might have dropped previously. When they find it they can connect the dots.

This sort of function is everywhere. It's in everything we see and do. From the transponders in our cars, cell phones, the GPS in our cameras. It's everywhere.

So why not put everything in the cloud? What is it that I think I can hide? I'm a lawful person and so I'm not going to advertise that I went through a red light, but some day there is going to be an intersection device that is going to register the infractions... made by the car, not necessarily the person.

I do not care that I use GMail and Google Docs. That all of my music is in iTunes match/iCloud. That I backup everything to Dropbox and CrashPlan. The information privacy war is over. We lost. Now all we can do is damage control.

Thursday, September 15, 2011

How does [mongoDB] concurrency work

This is a short post to call attention to this [mongoDB concurrency]. I do not doubt, for a second, that concurrency is difficult. It's probably very hard. Part of which depends on the overall architecture, resultset size, record version collision, replication, sharding, and conflict resolution; among others.

I do not have a particular concern other than it is an important problem and that it's going to effect write performance and potentially some reads. (This is very similar to the GIL problem in Python). There is a ticket on their system that references this issue and plans to resolve the issue by pushing the lock to the collection instead of the DB. While this is more granular I'm not sure it resolves the real issue with a workable solution. (And I do not have a formal recommendation)

It's not clear to me whether this problem references a lock across all shards or just the one shard. Therefore, could the problem be alleviated by adding shards? (doubt that)

I can say this, however, most RDBMS servers have addressed this problem by setting the locking level via a pragma and that it is tunable. In order to know whether this is an issue for you; you will need to know the distribution of calls [read, insert, update and delete] and the TPS rates too.

I'm less convinced that the day of the RDBMS is over. (Recently PostgreSQL 9.1 was released)

Perl RestMQ

I'm starting a Perl version of the python tool restmq. Restmq is really cool but my only complaint is that it uses cyclone; the tornadoweb server lookalike written for twisted.

Sure, I'd be justified porting it to tornadoweb but after spending time with mojolicious it seems like a better fit and more fun.

Contributors are welcome: http://github.com/rbucker881/perrestmq

Sunday, September 11, 2011

Tweet... What? Huh?

I received a 'being followed' notice from twitter today. Since this does Not happen very often and it's usually a spammer I check them all out. It turns out that this guy is a CEO for a social media company with 15k followers and he follows 15k people.

As my investigation continued I read his tweets for September 9. There must have been over 100 of them and they appeared to be legit. Which got me to thinking. How does he read tweets from 15k people; how does he keep that many conversations straight in his head; why; and what about his day job? If I were on the board of his company I'd be pissed!

And so that got me to thinking. What about the tweeter or blogger ithat tweets about pizza for dinner. Where is the marketing or social capital in that? It feels like smoke and mirrors.

[update 2011.09.11] and then there all the stupid bots that reply to everything. On topic or not.

Friday, September 9, 2011

Quickie : password security

I cannot take credit for the recommendation that we start using words for passwords. The argument I recall reading suggested that a 40-character password made from 4 or 5 dictionary words was a) easy to remember and b) harder to crack.

At first my intuition had me thinking that this guy was nuts. But then I did some rudimentary math. First I assumed that in a strong password or the traditional sense there were a few valid chars:
a-z A-Z 0-9 and !@#$%^&*() ,./<>?;':"[]{}\|= <space>

The problem with this is that even the strongest websites limit the user to alphanumeric plus a few simple special characters. So let's say we have 72 valid chars. That means that the math for a 7 character password looks like:
72^7  = 1.00E13
72^8 = 7.22E14
72^9 = 5.19E16
72^10 = 3.74E18

I'm hoping I have the math right but there is some huge wiggle room. For instance there is a list of invalid words and sequences, dictionary attacks, and then l33t speak. So this will reduce the number of combinations by some amount. If not I'm hoping that it's in proportion to the "word" scenario.

In the proper work scenario we talk about 4 or 5 proper words. Considering the average number of words is between 60,000 and 75,000 [ref] this might actually be reasonable:
60K^4 = 1.29E19
60K^5 = 7.77E23
60K^6 = 4.66E28

WOW! I had no idea that this was the case. Now the real discovery is whether or not I really know 60K words and can I stream them together in a way that I can easily memorize and recite. I suppose it might be better to permit a combination of the strict and worded. This way you get the combined complexity and brute force is even harder to execute.

I don't know where my passwords fall in terms of strength. I know they are random and they they overlap both. But I know that I've had to deal with systems that have rules that limit the valid passwords. This, of course, totally corrupts whatever ruleset I have devised for myself to remember my passwords.

I'm not going to share what I do next. It could be nothing. But if you're going to hack my accounts I want to keep you guessing as long as I can.

Agile Management vs Herding Cats

I recently blogged about agile, building teams and herding cats. It's a topic that's always close at hand. I've had several job interviews over the last 4 months and agile always comes up.

Agile used to mean the tight feedback loop between the client and the producer (the producer can be a programmer). Today it has come to mean so much more.

I asked an agile coach, friend of mine, about the cats and agile thing. His response was that you cannot heard programmers like cats. And that you had to have strong leadership and define clear goals.

So I went back to my Agile Project Management book and started thumbing through. I found a passage where the author said that teams were meant to be self organizing.

All of the contradiction is making my head spin.

Agile Suggests:

  • there are no managers

  • that the teams might be made up of cats, but so what, they will self organize

  • priorities are set by project managers who manage projects and not people

  • producers don't have to make value decisions just do the work


But my friend suggests that:

  • you need a strong manager

  • clear goals (probably priorities)

  • don't bother with the cats, that's so yesterday


hmmm... So I skimmed my Scrumban and Kanban books again; looking for references to teams. The common "self-organized" statement was there... but that was it. And then Eureka.

Agile Project Management is meant to be implemented by strong and seasoned managers. Managers who already heard cats at an intuitive level. And so my blood pressure is resuming it's normal level as I recognize the fact that strong/good managers already herd cats and probably to agile on the fly by intuition; where new PMI, paper dragons, rely on the strictest sense of the "written" agile process instead of a functional blend between project management and people management.

PS: Many years ago I took a class on "root cause analysis" that was being given by a consultant for my employer. Root cause is pretty easy to do; it's essentially a recursive investigation into the mathematical significance in the problem/error space. The instructor was clear to say that noobs should recurse down all paths until the numbers were flat. And more importantly leave intuition out of it. Your intuition will work but later. This was practical advice because "we" were always in the data. Day-in, day-out. This is different than management. There is room for intuition in people and product management. But you need "strong" skills to be there.

Agile is ok. I'm certain it works. But it should not be the only tool in the managers toolbox. Teams, people and projects are much more complicated than this. Otherwise the ten commandments would have been enough.

Monday, September 5, 2011

Incremental improvement for iPhone

When you consider the cost to develop, manufacture and deliver a single iPhone, iPod or iPad you have to realize that there is no incentive to deploy anything more than one killer feature, a few minor visual improvements, and as many bug and security patches at the QA team can deliver.

The reason for this is because; A) they make a bulk of their cash on upgrades and first time purchases. B) why upgrade the calendar or addressbook when there is an app for that? With the sale of the app they make a margin that they would never see if they acquired a better version or if they seriously upgraded all of the apps.

This is not an easy strategy to deal with. The platform is easy to develop for, however, the apple apps have so many advantages over potential replacements. For example; address book cannot be deleted and is integrated into the phone.

The same is true for all of apple's products. I figure I need 5000USD a year to keep up with my wants. Not my needs.

Sunday, September 4, 2011

RSS is a time sucking mistress - go green

Cheng wrote an interesting article: "Why keeping up with RSS is poisonous to productivity, sanity"

In response I promise never to post on tumblr or to syndicate my articles again. I like the idea of syndication, however, I think Cheng was the same as going green.

libev and concurrency

What is libev?

A full-featured and high-performance event loop that is loosely modelled after libevent

Web Servers?

tornado, mojo, lwp, lwpng

What can go Wrong?

blocking on unrelated resources

What to do to correct?

que, req, resp

Benchmarks?

everyone likes a good graph, but hello world sucks, real world too hard to mock, what is a transaction

"Herding cats", building teams, "getting things done"

I'm in the process of refactoring my home office. I've reduced two 7' by 4' shelves, which were filled to capacity with programming language references, OS references, networking references, framework references and now the only topic that remains are my management reading library. And even those have to go.

I plan to give away as many of these books as I can. The rest I will recycle or give to a local university library. If your interested then drop me an email with your street address (sorry US only unless you want to pay postage).

The first book in question is "herding cats: a primer for programmers who lead programmers". It's an interesting book, however, to summarize... there are all sorts of programmers. You need to identity them and their types and then manage to their type. Do not try to treat them all the same.

I have worked for a number of startups and established companies. I have worked in small and large teams. I have also been involved in a mix. So while herding cats makes so much strategic sense for working in a team with so many different personalities the fact is that you are not always going to be confronted with such a diverse group of people.

For example, in most startups with under 10 programmers, we tend to surround ourselves with people just like us. Self-managed we want to have as little friction as possible. We want to get as much work done. We want everyone thinking the same way and solving problems the same way. (commenting the code the same way) All of this synergy reduces friction and at the end of the day we might just complete the assignment. And we won't spend any more cycles placating those that do not fit in.

The next three books are all on the similar topic of project management. "agile project management with scrum", "scrumban", "kanban". I'm a fan of the *ban method of project management. It takes this humongous process and distils it into something that a) beginners can contribute to easily; b) does not add too much friction to project management; c) treats project management as a tool instead of a timesheet; d) allows the stakeholders to get some feedback.

Most startups do not get into this sort of process improvement churn until they start to make some money and then as if manna from heaven has a few disasters. If everything was working great, why add process? That's when companies go out and hire consultants, take classes, hire specialists. However, there is always unmistakable sucking sound as the developers, the objects of the PMI pro, start to feel the squeeze.

So imagine if you will... you are the fourth hire in a team of ten. You've been working 50 to 80 hours a week for the last 12 to 18 months. The project may or may not be live but the team is starting to grow and at some times it seems uncontrollable. Sales and marketing is also growing and let's not forget support. All of these people have a vested interest in the success of the company and so everyone starts making change requests.

Let the process improvement begin...

First there is the ticket system, the "stories", then there is a snowstorm of incoming requests. They come in so fast that the person who wrote it might have already been promoted and no longer cares or possible relocated to a different department. The outcome in the same.... Before I bore you to death with examples. Let's just say that there is CHAOS.

And then you hear that nagging voice in the back of your head. It says: "Do not Panic!". You have PMI pros, black belts, a ticketing system and priority management meeting. And you also have a team of cats. When you were 10 programmers it was easy to be selecting. Not any more. While Larry Wall defines the virtues of a programmer as having hubris and while I agree with his complete description I think he's actually talking about one type of cat. The startup junkie.

Now comes the pain...

The PMI pros are good people but they are not managers. Managers and leads that have been promoted from the programmer ranks are not managers and they are not PMI pros. Just because you heard a lecture on brain surgery does not make you a brain surgeon unless you were already a surgeon and even then I think not.

It is not good enough to be in the same room as a good manager. Good quality management traits do not rub off. They cannot be eaten as a snack. You need accomplished managers who have been there and done that. They need to know your business model, your resources, understand the team dynamics... and if you want to be a good manager then you need to mentor with a great manager; this is a lot more complicated than budgets and performance reports.

And then it's time to think about cutting off your leg...

You've gone from 10 to 400 programmers and you are still not getting things done.  All manner of bugs are creeping into the system. Few new features are making it into development let alone production because everyone is fixing bugs. On top of that management is continuously reorganizing the department and the individual teams. Everything is happening at a rapid pace. Everyone is looking for consensus in order to decide what the next project is and how it's going to work. And then there is the constant complete redesign that is always hanging over everyone's heads.
Susan Powter said it the best. "Stop the maddness!"

To recap...

  • we had a small synergistic team

  • we got things done

  • at some point the team broke the mold

  • added lot's of external pressures (PMI, additional staff, more management)

  • when we hit panic levels do a reorg and wait to see what happens

  • promote from within instead of hiring skilled/professional managers

  • programmers spend a great deal of time grumbling

  • programmers, generally speaking, detest change


How to fix things...

By now you can probably guess the sort of advice I'm going to give you.

  • know your cats

  • adopt a professional attitude and organizational structure

  • hire professional managers first

  • let the managers determine where the real deficiencies are. Not everyone on the team needs to be a rock star. Professional manager know how to develop the right mix.

  • constantly monitor the team dynamics

  • then hire PMI pros if needed

  • make the PMI process as frictionless as possible even if the pros have to do all the work. One should not delegate from the PMI to the programmer.

  • and when all else fails, and long before real chaos... "sometimes you have to fire the rock star" --Steve Maguire (debugging the development process)

  • do not let the rock star upset the team.


Post Script...

Google is a great place to work. There are plenty of examples of other great places to work, however, people do not go to work for Google because they have a great softball team. They go there because the work is hard and interesting, the company mission statement, the executives, the amount of learning. The softball team, xbox, nap room, showers are not a perk for the 10 to 4:30 employees. But the employees that work from 8 to midnight 6 nights a week and come in on Sunday to check operations. If you work from 10-4:30, then you take 1 hour for lunch. You have not earned the privilege of a coffee break or contemplative nap. First you need to reproduce their work ethic, then their results, and then you can buy a ping pong table.

Good Luck!

Mongolab Surprise

I'm still digging into all things NoSQL and I have tried mongohq in the past but now I wanted to try mongolab. I was not expecting anything too super fantastic as I created my user account, my database and then my first collection. But I was.

I had a csv file on my Mac. I had previously installed mongo on my computer... I clicked on the import/export link on the mongolab website and they gave me a list of commands that I could copy/paste to the command line. After I inserted my username and password I executed it. It worked first time. Very nice.

Viewing and editing documents is a snap but subject for some self exploration.

Right now my only criticism is that it's expensive and I cannot determine the value proposition solely based on the webapp. I could install a mongodb server for a small fraction of cost and then clustering and ha become something I can measure.

Saturday, September 3, 2011

time time python time time

I wonder how many bugs there are out there because people use "time" incorrectly. I think I discovered a design flaw, implementation detail or just a really crazy thing to watch out for that I have no idea where to go with it except here.

I've written this code a hundred times:
date = time.strftime('%m/%d/%Y')
time = time.strftime('%H:%M:%S')

This code behaves all the time except!!! At midnight of whatever timezone you are in. Consider that if the strftime() function that is computing/formatting the date took place yesterday and the strftime() for the time variable took place tomorrow. You would get seriously affected results.

While you would think that you could do this:
now = time.time()
date = now.strftime('%m/%d/%Y')
time = now.strftime('%H:%M:%S')

It feels right and it might work in some languages. But not here. When now is assigned time.time(), it is of type float. And therefore you cannot apply the function strftime() to a float. You need an instance of time.

Swinging back around, If there is a date class that returns date and a time class that returns time; why would the time class have a strftime() function that would accept date format params and vice versa?

This works:
now = datetime.datetime.now()
date = now.strftime('%m/%d/%Y')
time = now.strftime('%H:%M:%S')

In order to state the obvious, the first line gets the system time from the clock; then using the copy of the time in order to format the date and time strings.

PS: date and time are the proper names of python modules. Their usage in this exact context is questionable and could cause errors. But it worked... for now.

Thursday, September 1, 2011

Static vs Dynamic Languages

I'm not certain who started the discussion and if it ever really mattered. I started to read this article at TechRepublic and I just could not get into it. I suppose that in the halls of academia there is need to have these sorts of confrontations but in the pragmatic world does it really make sense?

For instance, I'm a programmer of some widget application. Within the application I have code that represents explicit and implicit contracts between objects, classes etc. And in the layer surrounding my widget I have some sort of abstraction or API layer. This layer defines a similar contract(s) between the widget (instance) and the widget's users. Some of these contracts are defined in DOC and others in code and behavior. And when you get really nutty with inheritance static languages start to behave like dynamic ones.

So what is the point of the post? The conversation is really about all of the other features and functions. The fact that we're dwelling on "Static vs Dynamic" probably means that we do not have anything else to talk about or compare. I seriously doubt that this is going to be the make/break topic. "Static vs Dynamic" is getting an unreasonable amount of attention.

Let's move on to feature #2.

another bad day for open source

One of the hallmarks of a good open source project is just how complicated it is to install, configure and maintain. Happily gitlab and the ...