Jumping Off a Cliff and Learning To Fly

by Mike Levin SEO & Datamaster, 03/07/2010

I’ve talked some lately about my latest programming project. It’s a re-positioning of myself off of Microsoft software, and onto a combination of Unix/Linux software and generic Web APIs. The Programmable Web is one of my daily news-feeder reads, and I see the world gradually shifting towards better living through short-cuts. Since I’m repositioning my abilities and my very career in this move, I’m aggressively thinking into the issue and have articulated my thoughts in this post.

At first, I was torn between a certain hardcore old-school mentality I’ve developed in dueling sysadmins over the years, in which I believe understanding and controlling every aspect of my application to yield better performance, stability and longevity. I’m the type who believes I should control the network architecture my app sits on, so I can shape traffic. This approach is in opposition to the new short-cut driven way of thinking based on magical incantations that abstract away every implementation detail, letting you be about 10x more productive than old-schoolers, but also lose that ability to optimize at the hardware and network levels. Plus, you add a bunch of new vendor dependencies that could leave you high-and-dry one day.

I have over the past 6 months synthesized a blend of the two approaches in which I simultaneously favor the very-very large, in terms of services provided by billion-dollar datacenter clouds that are not too likely to disappear tomorrow, and the very-very small, in terms of personally administrated servers where I am “root”, and can slam out multiple instances on demand as if I was running my own little cloud. There’s no good name for this second approach, but I’ve heard terms such as micro-servers and plug computing. So I put one foot on the juggernaut’s den, and the other foot in my own silicon sand.

Following that decision, I built my application in raw, but immediately usable form and let it into a semi-wild state–internal company usage. That puts me today at a phase in product development where things are going fast and furious because I use what I develop in day-to-day work. I describe this live deployment and development as jumping off a cliff and learning to fly. Such an approach is possible by piggybacking on top of other products, gradually modifying and customizing. The result is tempered in the fire of live usage.

Jumping off a cliff and learning to fly is sometimes what it takes to move us forward, because it forces matters. Maybe I say that to justify the pressure I put on myself in forcing my career to new height, though it does get me into trouble sometimes over-promising. That’s the motivation for this post, to help me logically think through how to navigate the waters immediately ahead of me. I have deliverables due yesterday based on my new technology, and am running behind.

The trick to being successful is immersing yourself of the task-at-hand, and to not be what Joel Spolsky would call an Architecture Astronaught, with your head up in space while urgent problems are right here on the ground. The last thing the world needs is another framework to solve a general class of problems that sipheons off brainpower from problems that have to be solved immediately today. I’m forgoing large framework building today in favor of small framework. I characterize it as a nimble acrobat living on my micro-servers, swinging on the trapeze of multiple API’s, knitting results together, and storing the data where it’s most readily put to use.

As it happens, this data storage location is common Office software, but in it’s new home on the Cloud. Cloud Office software is cleverly being used by Google to turn average Office software users into unwitting web developers, thanks to the collaboration, permissions and forms features. The Web browser limitations are a bottle-neck in this process, but the browsers themselves are entering a rapid iterative rate of maturation, fueled by the JavaScript performance race triggered by Google. That will be pushed farther and faster once the Chrome browser is the main operating system on a new cheap class of home computer, HTML5 becomes better understood, and native API hooks start allowing in-browser high performance graphics. And even with all that, Google has bought DocVerse, a company that’s helping with the transition off of desktop office software and onto the cloud.

One of the places where developers have invested inordinate amounts of time is programming in the Web browser–the piece of software that changed the world, but also due to its very simple nature, was very unsuitable for application development. It’s gotten better, but you still have to to meld the disjointed worlds of JavaScript, DOM, CSS and HTML with all their finickey nuances, implementation differences, and overall Frankenstein Monster feeling. While there’s no getting away from this monster, it’s also not where to invest your career. Let juggernaut’s like Google deal with it, and instead, endeavor to talk to simple back-end APIs that drive the updates out to the browser.

Of course we have to go through other peoples’ services to get the benefits, creating yet another dependency–but that’s just a transitional stage, because Google is open-sourcing the server components like Wave Server, letting us live in a world of pure abstraction, that they will do the work of re-wiring to tomorrows better browsers, by which time we can hopefully plug the server components onto our own equipment. It has yet to be seen whether Google will ever give us our own open source Google Docs server. If they don’t, you should expect tomorrow’s Open Office to be Open Web Office, reverse-engineering the Google Data API, thereby letting organizations get back the control they give up when moving to the cloud. Just add your own servers and storage.

Old-schoolers (of which I’m one) will usually tell you not to embrace such vendor-specific API abstraction, because you don’t really understand what’s going on behind the scenes and lose opportunites to optimize. Old schoolers will tell you it’s better to do the work directly first without abstraction, then only later use libraries after acquiring a deeper understanding. While this is generally true, it isn’t always the case today. Take for example real-time data push from server to browser, such as chat applications, or the collaboration features in Wave. You waste man-months solving this seemingly straight-forward problem. There is no simple way. Go ahead and research it. It’s a fundamental problem in how the Web works, and Google is going to great expense to try to fix it. They even recently bought a company called AppEngine founded by ex-Google employees, which does real-time better than Google, and they wanted it back. And just last week, they bought another, DocVerse, a Microsoft Office real-time collaboration software company. Get the idea? The push for push is BIG, and it’s just a little bit too early to tackle it as a lone developer, but not too early to get the benefit.

So, I’m really advocating forgoing user interface development during this kooky transitional phase. It’s all going to have to be scrapped and rebuilt anyway. Some knowledge of HTML5, DOM, JavaScript and CSS will always be necessary, but not enough to justify mastering arcane knowledge on a moving target.

Some choose Flash + Flex for this challenge, specifically Adobe LiveCycle Data Services ES2. This framework offers an API abstraction both at the user interface and server levels for real-time datastream. Flex also gives you slick user interface widgets. But you still have to wire it all up, building your own “host apps”. Therefore, it is a short-cut, but not THAT MUCH of a shortcut. Ultimately, you’ve only saved time going from the very-granular to the somewhat granular, and are still stuck with many man-months of development work to have the piece otherwise provided free on a billion-dollar data-center. And still, you’ve introduced a new vendor dependency just the same–only one that’s at odds with the direction of the Web itself (HTML5).

So why not simply piggyback on familiar Office software for a ready-made user interface via web-APIs during this transitional phase? Wouldn’t it be smart to stay away from the brittle bits, and focus instead on timeless server tech, and Web-service calls. I equate this approach to writing Excel Macros, but with with an entire server dedicated to those macros, they are on streoids. You have 100% control over the security context, as opposed to browser programming where you fight same-origin policy with any sort of API-work. Also with my approach, you can use the language of your choice, instead of just JavaScript. And finally, because of the low horse-power required to do this sort of server processing, you can use your own virtual machines and micro-servers to control cost and maintain control. But most importantly, you can focus on just coding the parts that provide the unique competitive advantage, instead of noisy platform overhead. Let the billion-dollar brawlers worry about the expensive brittle pieces. Piggyback on them today, and re-wire your APIs tomorrow.

I once tried explaining this philosophy to a co-worker recently, who smugly dismissed it, tossing out the keyword “Google Closure” as if a JavaScript library and optimizer provided the same boost as piggybacking off a massive multi-billion dollar architecture and set of services. It’s the same argument with Flex/Flash. A grab-bag of components does not bring you as close to the finish-line as already wired-up and working components, that only need a little bit of persuasion to work the way you want.

“All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.” This is a quote by German philosopher Arthur Schopenhauer, and my work of late certainly seems to fall into that catagory, along with the cloud-and-plug computing it’s based on. I therefore decided to race right for the self-evident part, ASAP. Get the productivity boost. Impress some folks. Use it to retain some customers and win some new ones. And then let the benefit flow out to everyone around me–flowing out to my co-workers by making a gift of virtual machines and $100 plug computers.

And that brings us back to where I am today with my work–an ambitious plan to improve my company’s way of working, turning view-source and Powerpoint operators into BattleTech Mech Warriors–enhancing their natural ability to collect and analyze data. The battle-suits are just sitting there ready to be climbed into. Money on the floor. An example of a recent time I saw this happen was when Adobe added video support to the Flash player, and YouTube stepped in to pick up the money. Something very analagous is going on today. Something analogous is ALWAYS going on, in which platform-makers create new potential, and app developers attempt to tap into and unlock that potential, with varying levels of success.

Whenever that potential is unlocked creating new billionaires and empires, it always seem so obvious in hindsight–those smack-in-the-head moments, where you say “I could have thought of that!” Or worse still, saying “I DID think of that!” Or even worse still, “I did think of that, and even IMPLEMENTED it, but it didn’t amount to anything because it was an internal tool, or some implementation nuance wasn’t just-so. This time around, I am determined to be there, doing it right–even if it takes jumping off a cliff and learning to fly.