Thursday, May 30, 2013

cloud quote of the day

adhoc scalable elastic virtual compute cloud using XEN Hypervisor hosts and thin server instances with effective dated code across the entire cloud.
Now the real question. How much time, money and effort?

Moniker is selling XXX domains for $10

I gotta say this is really tempting. But why would I take this domain and what would I do with it? Could XXX be the next COM?

Wednesday, May 29, 2013

I'm reminded how bad DSLs suck!

This evening I was writing a script that was to update a unix hosts file.  While the file is pretty simple to process it is yet another DSL and when added to the list of other filetypes in the various etc folders you realize that there are so many different patterns and DSLs... and while they all have one thing in common - system configuration - they are all different.

I suppose that's when the Windows' admins start talking about the registry... and while it's a key/value store at it's core.... it's not an improvement.

Tuesday, May 28, 2013

Solving Big Tech Problems

I'm more than three quarters the way through this new book "101 Design Ingredients to Solve Big Tech Problems" and I really like it. It's easy to read, truthful, obvious, and reaffirming more than anything else. So the good news is that there plenty of text to skip and plenty to read over and over again. These affirmations are good for all tasks big, small, tech and non-tech.

My only complaint so far is that I'm still looking for the "big tech problem" reference or thread. I had skipped the introduction because in most cases they are just oral resumes so I'll have to read that section shortly.

If there weren't 100 or so affirmations they would make a great 12 month calendar.

Monday, May 27, 2013

listen up: Commission on the Theft of American Intellectual Property

If you permit rootkits on my computers then you are responsible for EVERYTHING that happens. Are you certain you want that responsibility?

Sunday, May 26, 2013

iPhone ePub/PDF reader

Back to current iBooks 3. Scrolling is a great new feature on importing from google drive is convenient. Google's reader is clumsy. This could be a tipping point when iPhone 6 launches. 

Elixir is likened to the Emperor's new clothes.

I looked at Elixir about a year ago. At the time it was not very interesting... but let's face it, the guys at PragProg have me convinced that "if the engine is big enough even a brick will fly." Nether Erlang nor Elixir or the OTP framework are difficult. It's still a club for the one percent'ers. (the one percent who understand it completely and the one percent that know the syntax but nothing else.)

I wouldn't recommend any or either to my customers or employers unless they had a commitment to the languages and access to the one percent that get's it. (neither are truly likely and reserved for startups that are trying to get some instant geek cred.)

Needless to say, I'm going to buy the book anyway.

Google sparrow -well?

Enough time has passed. Push notification should be implemented already. 

Being a programmer

Being a professional programmer is more like a UN Translator than high school conversational Spanish. 

Friday, May 24, 2013

3rd party mailbox apps

The problem with 3rd party mailbox apps is that they do not support enough mail protocols as in exchange etc. so no matter how super fantastic mailbox fr dropbox is (same for sparrow) it's never enough. But apple should step up its game!!

Why is "chrome fast" $1300?

That's the stupidest thing I've ever heard.  The reality is that the 1TB of GoogleDrive storage costs $1200 over the 3 years that they say is free. So the way it really goes... spend $100 on a GooglePixel and $1200 on GoogleDrive. The only thing you can hope and pray for is that Google offers a new machine in 3 years that allows you to continue the model.

Sunday, May 19, 2013

Saturday, May 18, 2013

Angularjs at io13

Google did a nice job presenting Angularjs last week but I have a number of complaints. They showed some advanced features that have not made it into production yet and in general there are very few, well written, examples. The examples are either trivial or meta. Nothing in between or real life. 

Friday, May 17, 2013

Twitter is 140 character RSS

What makes twitter a better RSS feed reader is that the readers are generally very good with little friction making speed reading easy. The only real problem is the robotic retweets. 

Wednesday, May 15, 2013

Google 3x more than pandora

Pandora works on my blue ray DVD player and TiVo too. And there are many 3rd party players. 

Come to think of it. Google I/O was 2/3rds me too. 

Samsung S4 vs Nexus 4

What could be worth $400?

Monday, May 13, 2013

iPhone in-app ads not age appropriate

Why would a game meant for toddlers have ads for dating services? Apple's AppStore is flawed in that it does not offer returns when purchasing real apps which only serves to facilitate the advert ecosystem. 

IRS meets Minority Report

Do conservatives cheat on their taxes more than anyone else? I don't remember political party being on my last tax return. Do we have a privacy problem now?

what is in a codename?

We all know what twitter and tweet mean but it was not too long ago when using those words in a sentence meant having to explain them too. Clearly having a common vocabulary is very important for general understanding as well as making the connection to "prior art". The other benefit is that the term and it meaning suggest meaning and purpose to the recipient without the marketing cost. And then there is the potential that adopters or contributors know exactly how to use, built, or extend "it".

Choosing a name like "Alphabet Land" for a toddler educational toy for learning the alphabet is better than "Buddy's Bets". The same would apply for some ORM, web framework, or crypto library.

Saturday, May 11, 2013

More Zero Downtime

Erlang hot patching is interesting but risky and blue/green deploying works in cases when the application is tightly coupled to the OS but has a heavy hardware requirements.

EDC - Effective Dating Code is the only way to go.

Tuesday, May 7, 2013

Zero Downtime

There are a number of solutions for the Zero Downtime dilema. Many of them include duplicate and partitioned systems. Others use hotpatching of code. On the one hand zero downtime resulting from some for of code and data patching has it's benefits in terms of timing and scale. However, on the other hand there is a certain benefit to performing reboots from time to time.

Monday, May 6, 2013

Bugs are bugs.

Don't cheat the analytics.

Angular + Bootstrap is incomplete

There are two main projects in the works and neither of them are complete. For example, while they both implement tooltip and popover... both do not completely implement hover or template partials.

One other thing about AngularJS -- While it is a paradigm shift of sorts it requires that you begin the project from the beginning. Converting from a standard part to a proper SPA.

Sunday, May 5, 2013

Saturday, May 4, 2013

MQ architecture has many side effects

When it comes down to it all modern day applications have some sort of MQ implementation. Whether it's obvious as in some commercial or free implementation (MSMQ, Rabbit-MQ, ZeroMQ, IronMQ) they are the same... obvious. Compared to the less obvious MQs that are embedded in some library or 3rd party dependency as is common in many database drivers as part of some database backed/integrated application.

MQs are similar to threading in their complexity. If not implemented correctly you can lose transactions, or your mind. SOA is in the family and looks a lot like the Micro-Kernel. Distributed processing with work queues. And nearly impossible to debug.

The biggest challenge is knowing what code to write and what code to buy.

The History of my Payments Experience

[reprinted from Nov 7]

During a phone screen this weekend I was asked to describe all of my payments experience in a 2-3 page cover letter. I quickly wrote an outline and started filling in the blanks and submitted my first draft. This morning I printed the first draft which was now 7 pages. I have since cleaned up the spelling and much of the grammar. It’s not meant to be a memoir and some descriptions are subjectively technical; and I’ve left out details that professionals should already know. Anyway here it is.


The following text represents the many payment systems I designed, implemented, supported, updated, managed, and contributed to in some way. It should be needless to say that I have worked on other projects in other vertical markets and other languages. I trust you will see the value that I bring to the business as well as the technology. One final note. These are my personal accomplishments. Sometimes I was part of a team and sometimes I worked alone it just depended on scheduling, resources, SME, etc.

In 1993 I started working, as a contractor, for NaBanco (acquired by First Data) as a contractor. I designed and implemented a TSR, written in assembler, for their FoxPro/DOS hospitality application. The TSR was designed to connect to each of the property’s Zon terminals and download it’s transactions. It would then post the transactions in the FoxPro database. Later the FoxPro app would send all of the aggregated data to the NaBanco’s host via the TSR. One last thing that the TSR would do (in the days prior to the popular internet) was a trivial email service for HQ to communicate with the properties.

After this project was finished my manager recommended me to the HR department. I interviewed with and was hired to design and develop the ValueLink platform. This was a closed loop stored value system. The First client, BlockBuster Video, needed a working platform ASAP. Once the hardware was selected I went about defining the toolset. Having evaluated Informix, which was currently running on NaBanco’s debit system, I decided on Oracle with PRO*C and a RAD GUI development tool from Computer Associates.

There were a number of tough challenges in designing this system. At the time I did not have any experience on Sun hardware and while I had worked on databases for years I did not know much about SQL other than the evaluation I had just performed. Additionally I had to learn multi-threading, multiplexing transactions over X.25, and everything that comes with OLTP production support. And while I had experience with the Zon terminal there was still a lot more to learn.

The next challenge was the helpdesk. I implemented the first desktop app with a toolset from CA (Computer Associates). The app lacked performance based on the PCs at BlockBuster’s offices in Ft Lauderdale. I used a 2400 baud dial up modem to connect the two locations. Shortly after the project went live I hired a VB programmer to rewrite the application, however, since the application was also going to be used internally we were going to have a lot more users connected than I wanted. So I implemented a REST/SOAP-like server using Java and Java WebServer from Sun. It worked brilliantly and was later used by the IVR subsystem.

Finally, I was introduced to Perl. I used Perl to implement two major systems. The first was the card account creation in order to generate plastics and send them to manufacturing and I also used Perl for generating product performance reports (TPS reports).

In the end I was able to implement a fast, flexible, and reliable system that now transacts over 700 TPS every single day(with plenty of headroom) and hosts thousands of merchants and over 500M accounts.

This platform’s most notable accounts include: BlockBuster Video, Walmart, Starbucks, and the USPS.
WildCard Systems was a client of First Data, however, during the early stages of their discovery it was decided that First Data was not going to be able to deliver. Mostly because they were going in a different direction. Since many of the people who were engaged in the conversation were friends it was easy for me move over.

At WildCard I was tasked with designing a different type of open loop stored value card system. I had implemented the first multi-wallet system that was to be used by insurance companies in order to pay or deliver money to the insured. While WildCard eventually circled back to HSA, FSA and eligibility applications they moved away from direct insurance applications.

The authorization system was implemented in two parts. The first part was a java based front end system that would connect to the association, reformat the transaction (the process of message normalization), adapt to network impedance, and then execute the particular transaction request against a set of T-SQL stored procedures and complex data configuration with rules. This front-end system was eventually certified to work with: Visa, Amex,MasterCard, Discover, First Data Resources. The overall platform replaced Visa’s LAC platform.

Early on it was discovered that the state of the art PC was not going to keep up with our needs so I implemented a rudimentary replication engine in java. This application would sync 4 master-master database servers in different data-centers over a dedicated WAN connection. Eventually others in the department as well as Microsoft tried their hand at replication.

I designed a template language that could emit html, pdf, txt, and csv files. This was written in Perl and was intended to limit the roundtrips to the DB. As a domain specific language it was non-trivial to produce reports and the demand was greater than the staff could produce. Eventually all of the data had to be replicated to a farm of 5 database servers in order to produce the reports.

One of the newer projects I worked on was “WebDog”. This internal-use webapp performed a number of functions supporting the operations staff. (1) it was a production migration management system, where developers wanting to submit code for production would write a ticket that had to be approved and the app managed the workflow. (2) it monitored all of the SQL Server databases. (3) It monitored all of the front end processors. (4) the most important thing it monitored was the approval ratio. When the ratio was out of spec we knew there was a release problem. (5) lastly it was responsible for deciding which SQL Server was the current master.

This killer app was conceived on a beach in Nantucket; modeled after Star Trek, deployed on FreeBSD, used MySQL, written in Perl, receiving requests via apache and mod_perl, and templated responses with Mason.

Notable clients included: AAA, Bank of America as well as the Visa Buxx brand.

After leaving WildCard I decided to work on a side project. One of the last discussions we had at WildCard had to do with TPS rates. The existing system was only working at about 25-TPS at 100% CPU Utilization (8 CPU with 16GB RAM). I posited that (1) there was a problem with our SAN. It has been reported that period SAN drives suffered from brown-outs. (2) there many examples based in truth bashing MicroSoft and SQL Server. Oracle was so much more performant. (3) T-SQL was a pig, all of the code was essentially doing hash lookups O(1) using a relational search O(lg(n)).

So I submitted two papers to SleepyCat, the makers of Berkeley DB. The papers represented payment system designs based on BDB and BDB-XML. I received two honorable mentions. I also implemented one of my designs using Java and BDB. I was able to get 1500TPS on a single core, single spindle drive.

**sidebar** by this time in my professional development I had discovered erlang. The notion that if a language like erlang can offer 9-sigma, if implemented correctly, in a phone switch environment then how different could that be in payments. 9-sigma would be a great platform/language to implement payment.

What attracted me to eDiets was a similarity to a side project I was working on, however, one of the projects I implemented for the company was a prototype erlang merchant gateway. This allowed their internal payment system to connect to different acquirer systems. The first prototype was implemented in erlang and later it was replaced with a java implementation as an ATG plug-in. The team was excited about the erlang potential, however, management steared the company toward more java.

I joined MetaVentures to support their existing CRM platform for Verifone magstripe devices. The Perl application communicated device configuration and transaction details to/from the Verifone devices. Since I had payment knowledge I was tasked to design and implement a complete end to end payment system. This included; POS, HSM, merchant gateway, and PCI compliance. The HSM and merchant gateway were implemented in erlang. The POS is a mix of languages including Perl, C, SQL and bash.

While the erlang systems were interesting to construct it was uneventful. Certifying with multiple acquirers was as simple as changing the message templates. They have been running without interruption since they were installed. There are necessary enhancements, however, none of the current team members really want to spend any time on erlang. (to be continued later)

The POS was interesting in that it needed to support a kiosk mode browser in javascript which used websockets to communicate with local webservice daemons that were connected to barcode scanners, scales, customer facing displays, pin pads, and a magstripe reader.

The gateway was certified with RBS, Global, and First Data. And is PCI compliant.
Insight Card Systems implemented a Ruby/Rails platform for account and card management. At specific times of the day it would perform account balance updates to a service provider and the service provider would send transaction details back. The system suffered from a number of problems including reporting performance and reporting accuracy. Even though I was the director of development I was optimizing the SQL and training the programmers on new ways to get more performance out of their platform, and making production operations decisions. Furthermore I implemented proper release process in order to reduce downtime and improve release quality.

As the director I had a number of other roles and assignments. I needed to hire more staff and bring development in-house. (currently outsourced). I also had to redesign a system that had 5-10x capacity with the same hardware that was currently at 100% capacity. And I had to address client expectations and customization.
I started Florida Freelance because of the economic times we live in. I had a couple of contracts that I knew I could work on. The first was a VOIP arbitrage system that generated about 1M minutes a day in call volume. This was an integrated Asterisk switch and a connected dashboard. I was tasked with redesigning the system because the original system was dropping calls, losing calls, performing badly, and could not handle the volume they needed. While this project is not a payment system is does demonstrate my ability to scale.

My second client, a company in Stockholm Sweden; hired me based on my experience. They wanted me to contribute to their existing platform and help them design new applications in areas I had detailed experience. Their platform is implemented in erlang, however, I built several interfaces in java and C as part of another plan to unify their message passing and logging. I also performed a complete PCI audit of their HQ and operations centers in Stockholm.

**sidebar** One of the interesting features of erlang is hot-code replacement. The erlang core allows developers to replace modules on the fly without interruption. However, while many erlang programmers think this is a cool feature it is actually a detriment to payment systems. Hot-plugging code causes transactions in flight to become unreproducable due to the version mismatch of sub-modules through the transaction. From an operations POV, if you are going to switch master/slave or HA configurations in order to release new versions… then you might as well restart the app. This way you are assured that the app will restart.

A recent client in Portland Oregon, asked me to perform a number of projects. The first was a one-day design and overall roadmap for their future issuing platform and to see whether I was compatible with the CEO. A few months later they asked me to perform a due diligence on a potential payment vendor’s platform. And finally to design a custom issuing system for them in the EU. This was to include to EMV for chip. Shortly after beginning this part of the project I was tasked to design the same for China Union Pay.

Another client in Atlanta Georgia; has decided to rewrite their erlang gateway and HSM. While the system has been running this entire time it still suffers from the inability to enhance the application. Initially they wanted to implement the new platform in C but I convinced them that Python/tornadoweb/redis was a good choice. They recently certified with WorldPay on the first attempt. The entire project took less than a month.

There was a brief moment when I was having second thoughts about Python. The team was made up of Perl programmers, however, their tech lead was not grocking it and wanted a chance to contribute and python was going to be a lot easier for him to learn and easier still for the others to adopt.

So that’s about everything payments. I look forward to fielding any questions you might have.

Thursday, May 2, 2013

AngularJS and Bootstrap

There are several implementations of a bootstrap layer that integrates with AngularJS.  The problem is that they are close enough in name and implementation that I cannot tell them apart and cannot tell which is ready for production and which is not.

Wednesday, May 1, 2013

Zero Downtime? WTF!

When someone says that they want zero downtime; what is it that they really want? An absolute 100% zero downtime is absolutely impossible unless the downstate is the upstate or you live in Bizaro World. Zero Downtime should cover:

  • facilities (the building)
  • network
  • power
  • servers
  • services (database; DNS; NTP; web servers, etc...)
  • applications (userspace applications or REST services)
  • monitoring tools
  • support staff
  • versions, versions, versions...
What most sales people actually mean is completely different.
  • no perceived change in service by any of the end-users
What a systems manager actually means is completely different.
  • database schema
  • database stored procedures
  • application binding to the database - in particular strong binding through shared secrets
  • rolling service availability
What most managers forget:
  • Zero downtime is hard as evidenced by modern H.A. solutions
  • comes with a monster price tag
  • The recovery model is based on tight coupling of components
  • systems are typically master->slave with one-way artifact promotion
  • effective strategizing requires very detailed specific domain knowledge
  • ...
**Clearly perception is king.

** previously I wrote about hot-plugging code in Erlang. It's the worst thing you can do in a credit card, financial or any system with a high transaction consistency requirement.

another bad day for open source

One of the hallmarks of a good open source project is just how complicated it is to install, configure and maintain. Happily gitlab and the ...