Wednesday, November 30, 2011

There can be only one way!

I recently tweeted:
there is something to be said about a prog language and/or framework that gives you exactly one way to solve a given problem.

And I thought that it might need a little explanation.

 

breadcrumbs, where did we come from?

Some author wrote a column "Moving on from Rails" (I strongly dislike self important authors who attribute their publications to their "handle" instead of their actual person). It's mostly a well written article, however, the author did not not allow for feedback or comments.

My only complaint about the article is that it seems that this person does not have the perspective of having been through the "history repeating itself" or "waterfall" that is our industry.

For example: as servers were more expensive and more powerful "we" shifted the processing to the server. And as memory and personal computers became cheaper and more powerful "we" moved processing to the client-side. Additionally the core of the application has shifted. At one time there was a single database and now there are farms ... and so the core application has shifted and the layers are stacking.

Actually, the layers are not stacking at all. They are the same as they have always been. They are simply more granular in some cases. The OSI network model still stands as a description of what we have architected.

As to why the author is moving on from RoR (Ruby on Rails) has probably more to do with "me too" rather than any real technical justification. This behavior typically happens when a programmer is starting to burn out (either with the framework or the job) and so it is not an inditement of the technology but the person.
me thinks he doth complain to much

As to the conclusion that RoR answers fewer questions is not much of a conclusion. The same can be said of any language or framework and it is terribly subjective. It is also based on the weighted important by the author and not any real metric. Which is why I think the author is unhappy.

As for me and my projects, languages and frameworks. I'm mostly client driven. I like to guide clients toward technology that I like or I'm familiar with, however, they are all general purpose enough that they could complete the job easily enough. For me the question has less to do with the tools than it does the architecture. And with that in mind it's a better idea to have more tools in the toolbox than one tool you think is crappy.

Tuesday, November 29, 2011

New Perl Monger In Weston

I was looking for a PM group in Weston or one of the surrounding cities. Sadly both the Miami and Delray/Deerfield groups seem to be inactive and since I live half way in between I decided to open my own group. I hope this was a good idea. We will see.

Friday, November 25, 2011

Mojolicious or TornadoWeb?

It depends on what you are really trying to accomplish.  In a current project I'm using TornadoWeb to get the transaction through the ZeroMQ broker to a worker. (standard request/response broker implementation).

Since I'm using TornadoWeb I needed to implement about 50 lines of glue code and I needed to hijack TornadoWeb's IOLoop instance with the IOLoop from ZeroMQ. And since the work is only a few dozen lines of code ... it's all up for debate.

In retrospect I think I should have gone with Perl+Mojolicious+beanstalk. It would have completely eliminated my own broker .. another 100 lines of code. Then there was the direct access to redis in tornadoweb.

MEH! While I'm pragmatic that if I did not have to wtote an extra 150 lines of code then I would nto have to debug them either.

This race between the two platforms is so close! (on any given Sunday)

Thursday, November 24, 2011

Django-App-Engine is not Django

I'm not going to give these guys any more press than they deserve.  I tweeted that "django-app-engine development halted". As I tweeted the message I thought they were referring to the DjangoProject, however, I was wrong! Seems the Django project is still going strong, I don't know who these other guys are or what their endgame is...

Anyway, Django is a strong platform. I do not use it much these days because I'm focusing on TornadoWeb and Mojolicious but it is rock solid. It's admin tools can bootstrap just about app you might consider building.

Projects and more projects

I have been working on my resume lately and there are about as many different formats and general approaches. For the time being I'm going with a better narrative and a list of employers. But then I got to thinking about the specific projects I've worked on and why it won't fit on one page. So here is a list, even if it's just for myself:

Florida Freelance IT LLC; System Architect / Programmer, Owner 1/2009-




  • China Union Pay - issuer integration

  • POS - PinPad integration

  • Big data reporting, audit, reconciliation and billing system for a VOIP arbitrage system

  • devops for VOIP system (resource planning and production monitoring)

  • PCI-DSS audit

  • Implement new features (bill payment internals)

  • Online merchant implementation

  • Merchant acquiring system from scratch

  • Personel allocation for outsourced merchant restocking

  • Social media spider for FaceBook and LinkedIn

  • Local Lis Pendens search tools

  • 24x7 operations support


Insight Cards Director of Platform Development 6/2010-8/2010




  • Query optimization and report validation

  • DB schema normalization

  • managed internal and external resources

  • met with VC, executives and clients


MetaVentures System Architect / Programmer 1/2008-6/2009




  • implement new features

  • designed and implemented a POS from scratch

  • designed and implemented an HSM (encryption) from scratch

  • designed and implemented a merchant acquiring system from scratch

  • 24x7 operations support


eDiets Senior Programmer III 2007-1/2008




  • implemented bug fixes

  • implemented client customizations

  • assisted in the ATG conversion


Secure Computing Senior Programmer 2005-2007




  • designed and implemented custom TCL code for SnapGear firewall

  • integrated SnapGear hardware into universal command center tool


Ricoh Corporation Engineering Manager / Programmer 2005




  • managed the QA, Dev, Operations and support teams

  • designed PaaS system upgrade for increased volume

  • designed Desktop application and delivery system

  • SQL performance tuning and optimization

  • presentations to management team and regional superiors

  • presentation at sales meetings


Premier Global Senior Programmer 2004-2005




  • designed and implemented a reporting sub-system

  • rebuilt the build system

  • bug fixes and new feature implementations


WildCard Systems Software Development Manager / Programmer 1998-2004




  • designed and implemented a general purpose authorization system

  • certified with Visa, MasterCard, American Express, Discover

  • designed and implemented 2 SQL Server replication applications

  • designed and implemented association settlement system (Visa, MasterCard, American Express, Discover)

  • designed and implement a reporting script language for special purpose reporting system.

  • 24x7 operations support

  • DBA and SysAdmin

  • visited with clients for customization or integration


First Data Corporation Team Lead 1995-1998




  • Communications TSR for hospitality system

  • designed and implemented a GiftCard authorization system

  • designed and implemented a plastics manufacturing system

  • designed 2 and implemented 1 help desk application

  • designed proprietary changes to the GiftCard magstripe so that the POS did not have to change

  • designed and implemented a "sparse" card manufacturing process

  • worked with sales to prioritize work

  • visited with clients for customization or integration

  • 24x7 operations support

  • DBA and SysAdmin

  • Supported NCR debit system


Misc Contract Programmer, Full-time & Part-time 1983-1994





  • debugged OS/2 video device drivers. Both local and remote applications

  • trained the level 1 and other level 2.5 support teams

  • developed and trained others on strategies for converting OS/2 Presentation Manager (GUI) from 16-bit to 32-bit

  • architected a replacement for the 16-bit thinking layer (dives the entire window architecture)

  • converted 2.5M lines of code including 500K lines of assembly from 16-bit to 32-bit

  • kernel level debugging in order to verify operation

  • I certified a desktop app as CUA compliant

  • designed and implemented a special purpose scripting language for testing special purpose computers

  • built a suite of regression tests for release verification of firmware

  • built a hardware and software solution for simulating signal input for the special purpose computer

  • built second generation of hardware for use in the manufacturing process for FVT(function verification testing)

  • implemented a simulation of the special purpose computer (in assembly language) for testing the desktop app

  • implemented BIOS and POST (power on self test) for the artic coprocessor

  • implemented HDD and Tape device driver for OS/2

  • provided on-site Object-Oriented training

  • developed training materials for Object-Oriented training class

  • implemented and supported commercial copy protection removal

  • file manager for DOS with 256 byte filename support

  • operating room scheduling system

  • vehicle preventative maintenance system

  • produce warehouse inventory tracking system

  • custom MailMerge application for WordStar


I'm pretty sure this is everything.

Monday, November 21, 2011

AnyEvent and memory leaks

Recently twitter kraih posted a gist link that demonstrated a simple memory leak. I say it's simple now because I've spent enough time analyzing it. To say that it's a beginner's error is a bit arrogant but such as it is the bug points out a bigger issue when using AnyEvent (whether inside a Mojolicious app or not) it's own has potential to  cause memory leaks.

In the example:

# Very common leak
my $foo;
$foo = sub {
my ($i, $j) = @_;
return if $i >= $j;
say $i++;
$foo->($i, $j);
};
$foo->(1, 10);

The bug here is caused because $foo is declared and set outside the scope of the virtual function. Then within the function the code access the variable again. That's where the variable reference count is incremented. One way to correct this function looks something like:
# Very common leak
my $foo;
$foo = sub {
my ($foofoo, $i, $j) = @_;
return if $i >= $j;
say $i++;
$foo->($foofoo, $i, $j);
undef $foofoo; #this line is optional
};
$foo->($foo, 1, 10);

The corrected example is nothing special but it's not pretty (so maybe aesthetics should be #5 or #6 on the list).

Where this is a problem for AnyEvent is that it uses a virtual callback function in just about every API. This means that if the function does not push the data you need into the callstack of the callback that you have to get it by triggering access to the variable though the scope.
my $retval = undef;
$ctx->do_somthing_cool(1, 2, 3, cb => sub { $retval = 'done'; }).recv;

The above code is going to leak memory for the same reason that the first example gives. Now there is some good and bad news. The bad news is that there is additional work to do. The good news is that it's pretty standard and not that big of a deal. Just more aesthetics.
my $retval = undef;
$retval = $ctx->do_somthing_cool(1, 2, 3,
cb => sub { my $myretval = 'done';
return $myretval}).recv;

 In AnyEvent, recv() returns the value in the "return" function inside the callback. One place I have not experimented with yet is what happens when you have more than one function. For example:


my $retval2 = undef;
my $retval2 = undef;
$ctx->do_somthing_cool(1, 2, 3,
cb => sub { my $myretval = 'done';
return $myretval});
$ctx->do_somthing_cool(1, 2, 3,
cb => sub { my $myretval = 'really';
return $myretval});
[$retval1, $retval2] = $cv->recv()

In this example I do not know what ->recv() is going to return and if it does return everything; what order will the results be in.

The most important thing is making sure that you do not increase the number of references unnecessarily. Voila.  I'm not a beginner any more.  *sniff*


Sunday, November 20, 2011

Memory Leaks? Really?

I don't know anything about the specifics on this one... but when the project leader makes a statement like:
I have yet to see one non-trivial #perl web application that doesn't leak memory, wish we had cycle detection like #python does since 2000.

Ya gotta wonder whether or not this is a good idea? Is the framework the problem or the language?

When I did a search for "memory leak mojolicious" there was mention of some problems going back to version 1.56... but I'm so disgusted with the state of things that I hope he/they make a correction or statement addressing the issues... I also hope it's not sarcasm.

I cannot wait to update this post!!!

[update #1] "Take this little example, how would you explain what's going on to a #perl beginner? gist.github.com/1381643" [ref: twitter: kraih]

[update #2] "Perl's memory management is well-hidden, but is based on reference counts and garbage collection. It also has mortal variables, whose lifetimes are limited to the current context. It is possible to free(1) the memory(2) assigned to variables (including arrays) explicitly, by undef-ing the only reference to them." (ref: http://bit.ly/tH2XJt)

[update #3] since perl uses a reference counter in order to determine what memory to release, this is not much different than Java and a host of other memory managers. It's easy to cause a leak by having a circular linked list reference and then dropping all of the userspace references to that list. That can easily happen within a function that declares some local memory for a list of some kind and then exits without cleaning up. This is a perfect justification for functional programming style. "no side effects"

Saturday, November 19, 2011

Simple Economics

I'm not an economics major and I hated statistics but I noticed something that I think is interesting and scary at the same time; and I'm not talking alien scary.

  • my grandparents survived the great depression

  • my parents, having learned their lessons, did better than them

  • the baby boomers are entering into retirement

  • US economic policy makers are taxing the crap out of us

  • those same policy makers want us to spend spend spend in order to stimulate the economy and no incentive to save

  • I am clearly not better off than my parents

  • and as I spend over half my income on things that sustain life (food, water, insurance)


I can only hope and prey that my children do better than I am.

Thursday, November 17, 2011

Getting Mojolicious and AnyEvent::Beanstalkd to work

I have been having a hard time getting Mojolicious and AnyEvent::Beanstalkd to integrate. Given the commonality of AnyEvent one would think that it would a) work off the shelf; and b) there would plenty of documentation or examples. Sadly neither are true and I had to test several permutations, give up, and then try one last method (which worked).

It basically comes down to this. $cv->recv() does not work inside a Mojolicious. It causes a recursion error that AnyEvent just barely explains. The first alternative, that worked, was a matter of nesting all of the callback functions and as long as the stack is shallow that would work but it's not pretty and it's certainly unmaintainable.

In the end the code was replaced with $txn->cb(sub { $_[0]->recv }); and then the beanstalkd commands were flattened; and everything seems to be working nicely now. There was no explanation as to why this worked.

After looking at the AnyEvent code I think the code was able to "falsely" detect the $cv->revc() as a recursive call where the other format is undetectable.  I'm not certain whether this is good or bad but it works.
 . . .
my $txn = AnyEvent::condvar;
$txn->begin;
$client->use('restmq', sub {$txn->end;});
$txn->begin;
$client->watch($guid, sub {$txn->end;});
$txn->cb(sub { $_[0]->recv });

$txn = AnyEvent::condvar;
$txn->begin;
my $job2 = $client->put( $msg, sub {$txn->end;});
$txn->begin;
$client->reserve( sub { my $job = shift;
my $decode = $job->decode;
my $uuid = (@$decode)[0][0]->{'uuid'};
my $retval = sprintf "data> %s, %s\n", $job->data, $uuid;
$client->delete($job->id);
$client->ignore($guid);
$client->quit;
$self->render(text => $retval);
$txn->end;
});
$txn->cb(sub { $_[0]->recv });
$self->render_later;

The complete code is here. I'll be posting my scratch code shortly and I'll be updating and commenting the code too.

Benchmark - Hello World - TornadoWeb vs Mojolicious

I was running a simple hello world benchmark. It was meant to be as simple as possible. Mojo clearly wins the LOC comparison but TornadoWeb wins the race.  Mojo produced 20TPS and TW produced 22TPS. I used the basic run commands and I used siege from JoeDogs.

Mojo:


#!/usr/bin/env perl
use Mojolicious::Lite;
# Simple delayed rendering
get '/' => sub {
my $self = shift;
$self->render_text('3 seconds delayed!');
};
app->start;

TornadoWeb:


 
#!/usr/bin/env python
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()

Wednesday, November 9, 2011

6 year old Mac Mini

[uodate] I booted the machine and OSX failed to boot.  The replacement HDD was from an older MacBook (Intel) so I was not expecting great things, however, it was a 20GB upgrade of sorts. I put in my Leopard disk and tried to book via the 'C' key on my bluetooth keyboard. That did not work so I tried a USB keyboard. That did not work either.  Then I tried to eject the disk. That did not work. So it is possible that the HDD was never bad but that the MOBO or related component gave up it's magic smoke. That's enough of that. Now I need a new file server.

My Mac Mini is 6 years old and the HDD finally died. It's been showing signs that it was going to crash for the last 6 months or so but I did not recognize them. I use this machine as my file server; usually headless and recently with a monitor in order to diagnose my problem.

Typically the system would beach-ball when uploading or downloading a file. I initially thought that this was a network stack issue because I throttled network i/o at different times of the day/night. But when the machine hung yesterday I was at a loss. It was my local Genius who confirmed that it was the HDD.

Sadly, Apple categorized this machine as "vintage" so they would not service the machine and they did not have parts. It did not help that I had previously upgraded the memory (myself). So now I was out of warranty too.

So I proceeded to take the machine apart and look for the HDD. It was easy but not simple.

1) the case
2) the 4 screws on the CDROM
3) the 3 screws on the CD/HDD frame
4) the 3 screws on the fan
5) the 4 screws on the HDD
6) the socket that the CD/HDD plugged into
7) the wires that were taped to almost every flat surface

And after I took all of this apart... and put it all back together with no leftovers. I forgot to put the RAM back in. I'm about to boot it for the first time. Let's see what happens. (I've built so many PCs over the years I've lost count. That Apple's computers are less serviceable is probably a good thing.)

Saturday, November 5, 2011

So much input!

According to my step-father the definition of "a diet is when the amount of output exceeds the amount of input." I can't find fault with that description but when applied to RSS, Twitter and Facebook there is a point at which the input is overwhelming... So what is a person supposed to do? Cut the input or be more selective? Current tools do not do that well enough.

If anyone is listening... we need a "genius" [a la iTunes] to help us filter through all this input.

PS: By subscribing to several circles or groups there tends to be a lot of duplication and it's worse still when people change the titles and repost.

Thursday, November 3, 2011

The next startup revolution is going to be the startup

Startups are just too darn simple to startup. All it takes is a reasonable idea, a webapp and an iPhone and/or Android app. Maybe a little social sugar... and some luck.

The good news is that they are cheap to start. You need the right mix of people and skills but if you have the ideas you can knock out multiple version 1.0's until the cows come home.

Don't struggle with the 1.x releases unless the app(s) is a success. When it is then put some modest resources into a 1.x and if that works, keep going. Split the team, hire some maintenance people and move on.

Now all I need is a good idea.

 

OpenBSD 5.0 + Mojolicious + Redis + Beanstalkd

[update 2011-11-06] Recently I installed Mojolicious 2.22 without a hiccup.  Tonight I tried to install the latest 2.24 release.  The challenge tonight was that I needed to upgrade Test::Pod and Test::Pod::Coverage. I just guessed which modules were old based on the output from the build, however, it was 90% luck and 10% intuition. While I hate deep dependencies I wish they would have said something about this before leaving me to fend for myself. This is the sort of thing that make admin/DevOps work dangerous.

[update 2011-11-04] I'm trying to run my test program and I'm finding errors with module dependencies. I'm making the corrections inside the doc.  CRAP! One more thing I found. This time it was my fault. One of the Mojo guys asked me to check the clock/Timezone.  I thought I had. Crap!  I missed it. The clock was wrong and in fact it was 4 days behind. Since the Mojo files were technically 4 days in the future they would not build properly.  When I re-installed NTPD and corrected the clock... Miraculously it installed from CPAN.

[update 2011.11.03] looks like I managed to get everything to install. It's not totally painless but it's also not as involved as making code changes.

I really like OpenBSD (OBSD) and I've used it for years. The only comparison to other OS' that I want to make is that while these guys are pushing development forward with new ideas they truly embrace the "Unix" way by making use that "the sum of the parts is bigger than the whole". What I mean by that is that they build larger applications by incorporating specialized smaller ones. They are also more interested in security and correctness rather than security by constant change or obscurity.

Enough of that.

Armed with a current version of OBSD 5.0 installed in a working VMware instance... I wanted to install The tools I'd need for my next application idea. (skynet-pl). The installation of the primary framework tools was really simple.

Install redis and beanstalk proper (you'll need some basic packages from the CD; not listed here)
cd /tmp
wget https://github.com/downloads/kr/beanstalkd/beanstalkd-1.4.6.tar.gz
tar zxvf beanstalkd-1.4.6.tar.gz
./configure
gmake
gmake install

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/redis-2.2.12.tgz
cd /tmp
wget http://redis.googlecode.com/files/redis-2.2.15.tar.gz
tar zxvf redis-2.2.15.tar.gz
cd redis-2.2.15
gmake
gmake install

(gmake test; is recommended, however, you need to install TCL first)

pkg_add http://ftp.openbsd.org/pub/OpenBSD/5.0/packages/i386/p5-Mojolicious-1.16.tgz
cd /tmp wget http://cpan.metacpan.org/authors/id/S/SR/SRI/Mojolicious-2.22.tar.gz tar zxvf Mojolicious-2.22.tar.gz cd Mojolicious-2.22 /usr/bin/perl Makefile.PL touch Makefile.PL Makefile gmake gmake install

Install the libs from the CPAN (cpan and redis should be running in the background before you try to install the packages.)
curl -L http://cpanmin.us | perl - --sudo App::cpanminus
sudo -s 'cpanm Test::Pod'
sudo -s 'cpanm Test::Pod::Coverage'
sudo -s 'cpanm Mojolicious'
sudo -s 'cpanm AnyEvent::Redis'
sudo -s 'cpanm AnyEvent::Beanstalk'
sudo -s 'cpanm EV'
sudo -s 'cpanm JSON'

Well, that installs everything, however, it is not current.  Redis and Mojolicious are still back level. I hope to resolve this shortly.

PS: I tried to force Mojolicious 2.22 to be installed with the ports collection. FAIL. I also tried to install the latest redis too.  FAIL. I suppose I should try "OpenBSD 5.0 + TornadoWeb + Redis + Beanstalkd" next, however, initial tests are not looking good.

The First Lesson : Disaster Recovery

I do not remember my first program; but I remember the first program I typed into a computer.

My father had a subscription to BYTE magazine and we had a Radio Shack TRS-80.  I wanted to see what this article was talking about so I typed in the 100 lines of BASIC code... before I ran("RUN") it I wanted to save my work. Since I was new to the whole process and I wanted to get it right; I asked my dad to save it for me. Thankfully he executed a ("load") instead of a ("save") and he blew out my program from main memory. My four hours of hunt-n-peck typing were lost.

It was that moment that I learned the most intrinsic lesson of my career that, while intuitive, over the years has resurfaced as a programmer's life lesson.

  1. Know your tools and know your tool's dependencies. (at least one generation)

  2. Have a disaster recover plan for everything. (I use my rule of 2's and sometimes 3's).

  3. Be prepared so that if things go really wrong that you can move to a backup system, temporarily, while the primary is rebuild.


I cannot say that this has worked every time. On one occasion I had ripped my collection of audio CDs. It took weeks to accomplish only to have to re-rip everything because 1) software flaked out and the quality was awful; 2) I managed to destroy a drive on one machine in the cluster and I had no idea what music it had ripped and needed to be re-ripped; 3) and the great ogg/mp3 war.

Good luck.

Wednesday, November 2, 2011

Mojolicious plus Redis equals RestMQ-AEpl

[Update 2012-05-09] Bruno, thank you for pointing out my link was broken.

When I initially abandoned MojoX::Redis in order to use AnyEvent::Redis, it was because the MojoX version was not working properly or as expected. I dug deep into the code and doc and I finally managed to get things working. (documented here).

It still bothered me that the AnyEvent::Redis was not working and I even more upset that the doc was so poor as to not include examples that demonstrated that it worked. (same general complaint that I have for the MojoX version.)

This is what I have to say about that:

  1. I have the code working. It's pretty simple.

  2. I have a project online at bitbucket (here) where I provide all the code. (sorry it's not cleaner; for now)


So here are the bits that are important. Include these libs. Many are probably not necessary, however, I installed them and it works.  Someone else can trim them.
use EV;
use JSON;
use AnyEvent;
use AnyEvent::Redis;
use Mojolicious::Lite;

Make a connection to your redis server. Unlike the MojoX implementation one does not need to overload anything (ioloop). Just get a connection instance. I performed this function in the main body of the Mojolicious::Lite module. Therefore when the module is loaded it will be executed immediately and not when the first transaction is executed.
my $redis = AnyEvent::Redis->new(
host => '127.0.0.1',
port => 6379,
encoding => 'utf8',
on_error => sub { warn @_ },
);

And then within my GET function I do some work, however, there is one small block of code that needs to be asynchronously synchronized. (The update needs to be performed on the DB and the results need to be returned for future updates).
post '/q/:queue' => sub {
my $self = shift;
my $result = undef;
my $queue = $self->param('queue');
my $value = $self->param('value');
if (! defined $queue) {
$self->app->log->debug('queue was not in the URL ('.$queue.')');
$self->render_not_found;
} else {
my $uuid = undef;
my $lkey = undef;
my $q1 = $queue . $QUEUE_SUFFIX;
my $q2 = $queue . $UUID_SUFFIX;

$redis->all_cv(undef);
$redis->all_cv->begin;
$redis->incr($q2, sub{$uuid=shift;$redis->all_cv->end;});
$redis->all_cv->recv;

$self->app->log->debug('the uuid is ('.($uuid||'undefined').')');
$lkey = $queue . ':' . $uuid;
$redis->sadd($QUEUESET, $q1);
$redis->set($lkey, $value);
$redis->lpush($q1, $lkey);
$self->app->log->debug('the uuid q is ('.($q2||'undefined').')');
$self->render(text => '{ok, ' . $lkey . '}');
}
};

And now for the explanation. Very many of the examples look like this:
    $redis->all_cv->begin;
$redis->incr($q2, sub{$uuid=shift;$redis->all_cv->end;});
$redis->all_cv->recv;

And this is what the code is supposed to do:

  1. ->begin(); increment the counter

  2. do some redis work and do it in the background-ish

  3. In some cases the ->recv() will execute first and in others the ->end() will.

  4. If the ->recv() executes first then it will block (nicely) until the count = 0 or ->send() is called

  5. if -> end() executes first then the counter is decremented (in this case back to zero). And and artificial ->send() is executed. This ->send() has memory for any upcoming -recv() and will let the code pass right through onto the next statement.


The trouble with this flow is not whether or not it works. It does.  But what happens on the second pass. So I called the URL once and it worked great. I blocked on the ->recv() and all was well.

On the second pass through the code blew passed the ->recv() and it never waited for the results from the call to the DB. The reason for this is important. Since AnyEvent is using a "condvar" (condition variable)... and the variable is now set to True so that the ->recv() will pass... subsequent calls to ->recv() see the current value of the condvar. And since I only created the connection to the DB once, I was reusing the same instance every time and so the subsequent calls to the condvar worked as they should but not as expected.

Luckily there is a work around:
$redis->all_cv(undef);

By adding this code I forced AnyEvent::Redis to create a new condvar instance for the upcoming call. This meant that the flag was set to it's new-instance default value.

All this junk because AE-condvar does not have a reset or clear method.

Mojolicious plus Redis equals RestMQ-pl

I'm playing with Mojolicious. One of the fun projects I want to implement is a perl version of RestMQ. It was the main reason that I used then I requested a CPAN id. One of my previous complaints was, and remains, that the Python and Ruby versions of this project look elegant and fun. (this also means that a lot of the code is hidden in either the framework or the language; which I hate).

I'll be posting and submitting this project eventually, however, in the meantime I had not been able to get things working properly. There was a timing issue with MojoX::Redis and Mojo. It was not working well at all. I read and read all of the docs over and over again. In fact I had abandoned MojoX::Redis altogether in order to try AnyEvent::Redis. (it also failed).

Finally I reread a page in minute detail. The first piece of example code looked more like a standalone snippet. Which did not make any sense in this context because it was supposed to be a MojoX::Redis implementation in Mojolicious. (The wiki has been wrong more than once before)

But then; there it was:
If you are planning to use MojoX::Redis in Mojolicious' daemon mode, create MojoX::Redis instance with ioloop attr:

my $redis = MojoX::Redis->new(ioloop => Mojo::IOLoop->new);

Once I add this to my code and then added the stop and start calls to my function:
post '/q/:queue' => sub {
my $self = shift;
my $result = undef;
my $queue = $self->param('queue');
my $value = $self->param('value');
if (! defined $queue) {
$self->app->log->debug('queue was not in the URL ('.$queue.')');
$self->render_not_found;
} else {
my $uuid = undef;
my $lkey = undef;
my $q1 = $queue . $QUEUE_SUFFIX;
my $q2 = $queue . $UUID_SUFFIX;
$redis->incr($q2 => sub{
my ($redis, $res) = @_;
$uuid = $res->[0];
$redis->ioloop->stop;
})->start;
$self->app->log->debug('the uuid is ('.($uuid||'undefined').')');
$lkey = $queue . ':' . $uuid;
$redis->execute("sadd" => [$QUEUESET, $q1]);
$redis->execute("set" => [$lkey, $value]);
$redis->execute("lpush" => [$q1, $lkey]);
$self->app->log->debug('the uuid q is ('.($q2||'undefined').')');
$self->render(text => '{ok, ' . $lkey . '}');
}
};

it worked great. I still do not think this code is sexy and it's no where near the appearance of the Python or Ruby code(I should probably remove the debug statements). But it works... and it seems to be pretty fast too.

Tuesday, November 1, 2011

TornadoWeb - web scraping, eventd, recursion

I'm working with TornadoWeb and ZeroMQ at the moment and I was having a heck of a time getting things to work correctly. Especially when I was trying to call-out to another webserver and preserve the async-mode in the system. (TornadoWeb provides a basic Asyncronous HTTP Client library)

First of all, some months ago I found a post that wanted me to replace Tornado's IOLoop with the ZMQ version. They did not state why but they were specific to say that Tornado's was to be replaced with ZMQ and not the other way around. The code was not very interesting:
# override tornado's ioloop with zmq's
ioloop.IOLoop = zmq.eventloop.ioloop.IOLoop

And while I was researching this project I found that ZMQ provides a method that does the work for you. I reviewed the code and there was nothing special in it. It was pretty much the same code as I had originally implemented. I changed the code to use the new method. Not because they did it better but because I am hoping that this might future proof my project incase there is a change that I cannot account for. So the code now looks like:
# override tornado's ioloop with zmq's
#ioloop.IOLoop = zmq.eventloop.ioloop.IOLoop
zmq.eventloop.ioloop.install()

Hey! Nothing spectacular there. Moving on to the next challenge...

The code I'm currently working on is an admin site. Admin in the sense that only one or two users will actually every use this app and it would be very unlikely that more that one user would be online (even in an emergency). While that is true... the synchronous version of the website did not perform very well. Specially when I was deleting large amounts of data from Redis.

In this particular case I'm trying to implement a "test harness". While I like nosetests and it works great I need something that is interactive and ubiquitous. In this use-case the user enters the URL for the test-index-page. The page is drawn from a dictionary of testcases. The user clicks on a testcase and it runs to completion... drawing it's output in the buffer and then going back (recursively) to the admin website and putting in some additional data (a call trace) and appending that to the buffer too. The challenge was several fold. a) the build-in classes were not working asynchronously, b) there were two client calls to make; 1) the authorization 2) the call trace. c) make it all work within the asynchronous framework.

Before I go any further. It works and here is the code. I'm not going to explain it any more than this for now.
# test handler
class TestHandler(tornado.web.RequestHandler):
"""
"""
def initialize(self):
self.server = 'http://myapiserver.local:8882'
self.adminserver = 'http://myadminswerver:8881'
self.path = 'api1.1'
super(TestHandler,self).initialize()

# normally the callback function does not get this decorator, however, this
# was needed in order to make this work. Notice that this is the second handler
@asynchronous
def _handle_request2(self, response):
"""This is the second callback handler.
"""
if response.error:
self.write("Error: %s" % (response.error))
else:
self.write(response.body)
# need the self.finish() because the asynchronous decorator
# disables the auto_finish()
self.finish()

# since this is the first
@asynchronous
def _handle_request1(self, response):
"""This is the first callback handler.
"""
if response.error:
self.write("Error: %s" % (response.error))
else:
# write the output to the buffer but since we are not calling
# finish() the data remains in the buffer.
self.write(response.body)

# make the second call and callback to the second handler.
url = "%s/myfunction_two/" % (self.adminserver)
request = httpclient.HTTPRequest(url)
# it is important to replace the io_loop here (also needed to make it work)
http_client = httpclient.AsyncHTTPClient(io_loop=ioloop.IOLoop.instance())
# going to callback to the 2nd handler
http_client.fetch(request, self._handle_request2)

@asynchronous
def _get(self,other=None):
url = "%s/%s" % (self.server, path or self.path)
pay = self.path
request = httpclient.HTTPRequest(url, body=pay, method='POST')
# it is important to replace the io_loop here (also needed to make it work)
http_client = httpclient.AsyncHTTPClient(io_loop=ioloop.IOLoop.instance())
# going to callback to the 1st handler
http_client.fetch(request, self._handle_request1)

# notice that there is NO decorator here. It will be applied when _get() is called.
def get(self,other=None):
"""display the menu or execute the test
"""
self.guid = str(uuid.uuid1())
if not other:
# display the testcase menu
for k in testcases.keys():
self.write('<a href="/t/%s/">%s</a><br>' % (k, k))
self.finish()
else:
self._get(other)

It would have been nice if there had been a "parallel" task execution as the two queries could be executed at the same time because they are unrelated requests. Granted the callback would have to juggle the results in order to get them in the right order and then display them I suppose it might be possible with a single handler if the handler could inspect the data before calling finish(). It's something worth posting in the future.

I also want to mention that a similar strategy would probably apply to Mojolicious. (assignment for the reader; I home someone will post and link back.)

another bad day for open source

One of the hallmarks of a good open source project is just how complicated it is to install, configure and maintain. Happily gitlab and the ...