Skip to main content

Posts

Showing posts from November, 2011

There can be only one way!

I recently tweeted:
there is something to be said about a prog language and/or framework that gives you exactly one way to solve a given problem.
And I thought that it might need a little explanation.

breadcrumbs, where did we come from?

Some author wrote a column "Moving on from Rails" (I strongly dislike self important authors who attribute their publications to their "handle" instead of their actual person). It's mostly a well written article, however, the author did not not allow for feedback or comments.

My only complaint about the article is that it seems that this person does not have the perspective of having been through the "history repeating itself" or "waterfall" that is our industry.

For example: as servers were more expensive and more powerful "we" shifted the processing to the server. And as memory and personal computers became cheaper and more powerful "we" moved processing to the client-side. Additionally the core of the application has shifted. At one time there was a single database and now there are farms ... and so the core application has shifted and the layers are stacking.

Actually, the layers are not stacking at all. They are the sam…

New Perl Monger In Weston

I was looking for a PM group in Weston or one of the surrounding cities. Sadly both the Miami and Delray/Deerfield groups seem to be inactive and since I live half way in between I decided to open my own group. I hope this was a good idea. We will see.

Mojolicious or TornadoWeb?

It depends on what you are really trying to accomplish.  In a current project I'm using TornadoWeb to get the transaction through the ZeroMQ broker to a worker. (standard request/response broker implementation).

Since I'm using TornadoWeb I needed to implement about 50 lines of glue code and I needed to hijack TornadoWeb's IOLoop instance with the IOLoop from ZeroMQ. And since the work is only a few dozen lines of code ... it's all up for debate.

In retrospect I think I should have gone with Perl+Mojolicious+beanstalk. It would have completely eliminated my own broker .. another 100 lines of code. Then there was the direct access to redis in tornadoweb.

MEH! While I'm pragmatic that if I did not have to wtote an extra 150 lines of code then I would nto have to debug them either.

This race between the two platforms is so close! (on any given Sunday)

Django-App-Engine is not Django

I'm not going to give these guys any more press than they deserve.  I tweeted that "django-app-engine development halted". As I tweeted the message I thought they were referring to the DjangoProject, however, I was wrong! Seems the Django project is still going strong, I don't know who these other guys are or what their endgame is...

Anyway, Django is a strong platform. I do not use it much these days because I'm focusing on TornadoWeb and Mojolicious but it is rock solid. It's admin tools can bootstrap just about app you might consider building.

Projects and more projects

I have been working on my resume lately and there are about as many different formats and general approaches. For the time being I'm going with a better narrative and a list of employers. But then I got to thinking about the specific projects I've worked on and why it won't fit on one page. So here is a list, even if it's just for myself:
Florida Freelance IT LLC; System Architect / Programmer, Owner 1/2009-


China Union Pay - issuer integration
POS - PinPad integration
Big data reporting, audit, reconciliation and billing system for a VOIP arbitrage system
devops for VOIP system (resource planning and production monitoring)
PCI-DSS audit
Implement new features (bill payment internals)
Online merchant implementation
Merchant acquiring system from scratch
Personel allocation for outsourced merchant restocking
Social media spider for FaceBook and LinkedIn
Local Lis Pendens search tools
24x7 operations support

Insight Cards Director of Platform Development 6/2010-8/2010


Query optimizatio…

AnyEvent and memory leaks

Recently twitter kraih posted a gist link that demonstrated a simple memory leak. I say it's simple now because I've spent enough time analyzing it. To say that it's a beginner's error is a bit arrogant but such as it is the bug points out a bigger issue when using AnyEvent (whether inside a Mojolicious app or not) it's own has potential to  cause memory leaks.

In the example:

# Very common leak
my $foo;
$foo = sub {
my ($i, $j) = @_;
return if $i >= $j;
say $i++;
$foo->($i, $j);
};
$foo->(1, 10);
The bug here is caused because $foo is declared and set outside the scope of the virtual function. Then within the function the code access the variable again. That's where the variable reference count is incremented. One way to correct this function looks something like:
# Very common leak
my $foo;
$foo = sub {
my ($foofoo, $i, $j) = @_;
return if $i >= $j;
say $i++;
$foo->($foofoo, $i, $j);
undef $foofoo; #this line is optional
};
$foo->($foo, 1, 10);
The corr…

Memory Leaks? Really?

I don't know anything about the specifics on this one... but when the project leader makes a statement like:
I have yet to see one non-trivial #perl web application that doesn't leak memory, wish we had cycle detection like #python does since 2000.
Ya gotta wonder whether or not this is a good idea? Is the framework the problem or the language?

When I did a search for "memory leak mojolicious" there was mention of some problems going back to version 1.56... but I'm so disgusted with the state of things that I hope he/they make a correction or statement addressing the issues... I also hope it's not sarcasm.

I cannot wait to update this post!!!

[update #1] "Take this little example, how would you explain what's going on to a #perl beginner? gist.github.com/1381643" [ref: twitter: kraih]

[update #2] "Perl's memory management is well-hidden, but is based on reference counts and garbage collection. It also has mortal variables, whose lifetimes are…

Simple Economics

I'm not an economics major and I hated statistics but I noticed something that I think is interesting and scary at the same time; and I'm not talking alien scary.

my grandparents survived the great depression
my parents, having learned their lessons, did better than them
the baby boomers are entering into retirement
US economic policy makers are taxing the crap out of us
those same policy makers want us to spend spend spend in order to stimulate the economy and no incentive to save
I am clearly not better off than my parents
and as I spend over half my income on things that sustain life (food, water, insurance)

I can only hope and prey that my children do better than I am.

Getting Mojolicious and AnyEvent::Beanstalkd to work

I have been having a hard time getting Mojolicious and AnyEvent::Beanstalkd to integrate. Given the commonality of AnyEvent one would think that it would a) work off the shelf; and b) there would plenty of documentation or examples. Sadly neither are true and I had to test several permutations, give up, and then try one last method (which worked).

It basically comes down to this. $cv->recv() does not work inside a Mojolicious. It causes a recursion error that AnyEvent just barely explains. The first alternative, that worked, was a matter of nesting all of the callback functions and as long as the stack is shallow that would work but it's not pretty and it's certainly unmaintainable.

In the end the code was replaced with $txn->cb(sub { $_[0]->recv }); and then the beanstalkd commands were flattened; and everything seems to be working nicely now. There was no explanation as to why this worked.

After looking at the AnyEvent code I think the code was able to "falsely&qu…

Benchmark - Hello World - TornadoWeb vs Mojolicious

I was running a simple hello world benchmark. It was meant to be as simple as possible. Mojo clearly wins the LOC comparison but TornadoWeb wins the race.  Mojo produced 20TPS and TW produced 22TPS. I used the basic run commands and I used siege from JoeDogs.
Mojo:
#!/usr/bin/env perl
use Mojolicious::Lite;
# Simple delayed rendering
get '/' => sub {
my $self = shift;
$self->render_text('3 seconds delayed!');
};
app->start;
TornadoWeb:

#!/usr/bin/env python
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()

6 year old Mac Mini

[uodate] I booted the machine and OSX failed to boot.  The replacement HDD was from an older MacBook (Intel) so I was not expecting great things, however, it was a 20GB upgrade of sorts. I put in my Leopard disk and tried to book via the 'C' key on my bluetooth keyboard. That did not work so I tried a USB keyboard. That did not work either.  Then I tried to eject the disk. That did not work. So it is possible that the HDD was never bad but that the MOBO or related component gave up it's magic smoke. That's enough of that. Now I need a new file server.

My Mac Mini is 6 years old and the HDD finally died. It's been showing signs that it was going to crash for the last 6 months or so but I did not recognize them. I use this machine as my file server; usually headless and recently with a monitor in order to diagnose my problem.

Typically the system would beach-ball when uploading or downloading a file. I initially thought that this was a network stack issue because I thr…

So much input!

According to my step-father the definition of "a diet is when the amount of output exceeds the amount of input." I can't find fault with that description but when applied to RSS, Twitter and Facebook there is a point at which the input is overwhelming... So what is a person supposed to do? Cut the input or be more selective? Current tools do not do that well enough.

If anyone is listening... we need a "genius" [a la iTunes] to help us filter through all this input.

PS: By subscribing to several circles or groups there tends to be a lot of duplication and it's worse still when people change the titles and repost.

The next startup revolution is going to be the startup

Startups are just too darn simple to startup. All it takes is a reasonable idea, a webapp and an iPhone and/or Android app. Maybe a little social sugar... and some luck.

The good news is that they are cheap to start. You need the right mix of people and skills but if you have the ideas you can knock out multiple version 1.0's until the cows come home.

Don't struggle with the 1.x releases unless the app(s) is a success. When it is then put some modest resources into a 1.x and if that works, keep going. Split the team, hire some maintenance people and move on.

Now all I need is a good idea.

OpenBSD 5.0 + Mojolicious + Redis + Beanstalkd

[update 2011-11-06] Recently I installed Mojolicious 2.22 without a hiccup.  Tonight I tried to install the latest 2.24 release.  The challenge tonight was that I needed to upgrade Test::Pod and Test::Pod::Coverage. I just guessed which modules were old based on the output from the build, however, it was 90% luck and 10% intuition. While I hate deep dependencies I wish they would have said something about this before leaving me to fend for myself. This is the sort of thing that make admin/DevOps work dangerous.

[update 2011-11-04] I'm trying to run my test program and I'm finding errors with module dependencies. I'm making the corrections inside the doc.  CRAP! One more thing I found. This time it was my fault. One of the Mojo guys asked me to check the clock/Timezone.  I thought I had. Crap!  I missed it. The clock was wrong and in fact it was 4 days behind. Since the Mojo files were technically 4 days in the future they would not build properly.  When I re-installed NTPD …

The First Lesson : Disaster Recovery

I do not remember my first program; but I remember the first program I typed into a computer.

My father had a subscription to BYTE magazine and we had a Radio Shack TRS-80.  I wanted to see what this article was talking about so I typed in the 100 lines of BASIC code... before I ran("RUN") it I wanted to save my work. Since I was new to the whole process and I wanted to get it right; I asked my dad to save it for me. Thankfully he executed a ("load") instead of a ("save") and he blew out my program from main memory. My four hours of hunt-n-peck typing were lost.

It was that moment that I learned the most intrinsic lesson of my career that, while intuitive, over the years has resurfaced as a programmer's life lesson.

Know your tools and know your tool's dependencies. (at least one generation)
Have a disaster recover plan for everything. (I use my rule of 2's and sometimes 3's).
Be prepared so that if things go really wrong that you can move to a…

Mojolicious plus Redis equals RestMQ-AEpl

[Update 2012-05-09] Bruno, thank you for pointing out my link was broken.

When I initially abandoned MojoX::Redis in order to use AnyEvent::Redis, it was because the MojoX version was not working properly or as expected. I dug deep into the code and doc and I finally managed to get things working. (documented here).

It still bothered me that the AnyEvent::Redis was not working and I even more upset that the doc was so poor as to not include examples that demonstrated that it worked. (same general complaint that I have for the MojoX version.)

This is what I have to say about that:

I have the code working. It's pretty simple.
I have a project online at bitbucket (here) where I provide all the code. (sorry it's not cleaner; for now)

So here are the bits that are important. Include these libs. Many are probably not necessary, however, I installed them and it works.  Someone else can trim them.
use EV;
use JSON;
use AnyEvent;
use AnyEvent::Redis;
use Mojolicious::Lite;
Make a connection to you…

Mojolicious plus Redis equals RestMQ-pl

I'm playing with Mojolicious. One of the fun projects I want to implement is a perl version of RestMQ. It was the main reason that I used then I requested a CPAN id. One of my previous complaints was, and remains, that the Python and Ruby versions of this project look elegant and fun. (this also means that a lot of the code is hidden in either the framework or the language; which I hate).

I'll be posting and submitting this project eventually, however, in the meantime I had not been able to get things working properly. There was a timing issue with MojoX::Redis and Mojo. It was not working well at all. I read and read all of the docs over and over again. In fact I had abandoned MojoX::Redis altogether in order to try AnyEvent::Redis. (it also failed).

Finally I reread a page in minute detail. The first piece of example code looked more like a standalone snippet. Which did not make any sense in this context because it was supposed to be a MojoX::Redis implementation in Mojoliciou…

TornadoWeb - web scraping, eventd, recursion

I'm working with TornadoWeb and ZeroMQ at the moment and I was having a heck of a time getting things to work correctly. Especially when I was trying to call-out to another webserver and preserve the async-mode in the system. (TornadoWeb provides a basic Asyncronous HTTP Client library)

First of all, some months ago I found a post that wanted me to replace Tornado's IOLoop with the ZMQ version. They did not state why but they were specific to say that Tornado's was to be replaced with ZMQ and not the other way around. The code was not very interesting:
# override tornado's ioloop with zmq's
ioloop.IOLoop = zmq.eventloop.ioloop.IOLoop
And while I was researching this project I found that ZMQ provides a method that does the work for you. I reviewed the code and there was nothing special in it. It was pretty much the same code as I had originally implemented. I changed the code to use the new method. Not because they did it better but because I am hoping that this might f…