Jumping Off a Cliff and Learning To Fly
by Mike Levin SEO & Datamaster, 03/07/2010
I’ve talked some lately about my latest programming project. It’s a re-positioning of myself off of Microsoft software, and onto a combination of Unix/Linux software and generic Web APIs. The Programmable Web is one of my daily news-feeder reads, and I see the world gradually shifting towards better living through short-cuts. Since I’m repositioning my abilities and my very career in this move, I’m aggressively thinking into the issue and have articulated my thoughts in this post.
At first, I was torn between a certain hardcore old-school mentality I’ve developed in dueling sysadmins over the years, in which I believe understanding and controlling every aspect of my application to yield better performance, stability and longevity. I’m the type who believes I should control the network architecture my app sits on, so I can shape traffic. This approach is in opposition to the new short-cut driven way of thinking based on magical incantations that abstract away every implementation detail, letting you be about 10x more productive than old-schoolers, but also lose that ability to optimize at the hardware and network levels. Plus, you add a bunch of new vendor dependencies that could leave you high-and-dry one day.
I have over the past 6 months synthesized a blend of the two approaches in which I simultaneously favor the very-very large, in terms of services provided by billion-dollar datacenter clouds that are not too likely to disappear tomorrow, and the very-very small, in terms of personally administrated servers where I am “root”, and can slam out multiple instances on demand as if I was running my own little cloud. There’s no good name for this second approach, but I’ve heard terms such as micro-servers and plug computing. So I put one foot on the juggernaut’s den, and the other foot in my own silicon sand.
Following that decision, I built my application in raw, but immediately usable form and let it into a semi-wild state–internal company usage. That puts me today at a phase in product development where things are going fast and furious because I use what I develop in day-to-day work. I describe this live deployment and development as jumping off a cliff and learning to fly. Such an approach is possible by piggybacking on top of other products, gradually modifying and customizing. The result is tempered in the fire of live usage.
Jumping off a cliff and learning to fly is sometimes what it takes to move us forward, because it forces matters. Maybe I say that to justify the pressure I put on myself in forcing my career to new height, though it does get me into trouble sometimes over-promising. That’s the motivation for this post, to help me logically think through how to navigate the waters immediately ahead of me. I have deliverables due yesterday based on my new technology, and am running behind.
The trick to being successful is immersing yourself of the task-at-hand, and to not be what Joel Spolsky would call an Architecture Astronaught, with your head up in space while urgent problems are right here on the ground. The last thing the world needs is another framework to solve a general class of problems that sipheons off brainpower from problems that have to be solved immediately today. I’m forgoing large framework building today in favor of small framework. I characterize it as a nimble acrobat living on my micro-servers, swinging on the trapeze of multiple API’s, knitting results together, and storing the data where it’s most readily put to use.
Of course we have to go through other peoples’ services to get the benefits, creating yet another dependency–but that’s just a transitional stage, because Google is open-sourcing the server components like Wave Server, letting us live in a world of pure abstraction, that they will do the work of re-wiring to tomorrows better browsers, by which time we can hopefully plug the server components onto our own equipment. It has yet to be seen whether Google will ever give us our own open source Google Docs server. If they don’t, you should expect tomorrow’s Open Office to be Open Web Office, reverse-engineering the Google Data API, thereby letting organizations get back the control they give up when moving to the cloud. Just add your own servers and storage.
Old-schoolers (of which I’m one) will usually tell you not to embrace such vendor-specific API abstraction, because you don’t really understand what’s going on behind the scenes and lose opportunites to optimize. Old schoolers will tell you it’s better to do the work directly first without abstraction, then only later use libraries after acquiring a deeper understanding. While this is generally true, it isn’t always the case today. Take for example real-time data push from server to browser, such as chat applications, or the collaboration features in Wave. You waste man-months solving this seemingly straight-forward problem. There is no simple way. Go ahead and research it. It’s a fundamental problem in how the Web works, and Google is going to great expense to try to fix it. They even recently bought a company called AppEngine founded by ex-Google employees, which does real-time better than Google, and they wanted it back. And just last week, they bought another, DocVerse, a Microsoft Office real-time collaboration software company. Get the idea? The push for push is BIG, and it’s just a little bit too early to tackle it as a lone developer, but not too early to get the benefit.
Some choose Flash + Flex for this challenge, specifically Adobe LiveCycle Data Services ES2. This framework offers an API abstraction both at the user interface and server levels for real-time datastream. Flex also gives you slick user interface widgets. But you still have to wire it all up, building your own “host apps”. Therefore, it is a short-cut, but not THAT MUCH of a shortcut. Ultimately, you’ve only saved time going from the very-granular to the somewhat granular, and are still stuck with many man-months of development work to have the piece otherwise provided free on a billion-dollar data-center. And still, you’ve introduced a new vendor dependency just the same–only one that’s at odds with the direction of the Web itself (HTML5).
“All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.” This is a quote by German philosopher Arthur Schopenhauer, and my work of late certainly seems to fall into that catagory, along with the cloud-and-plug computing it’s based on. I therefore decided to race right for the self-evident part, ASAP. Get the productivity boost. Impress some folks. Use it to retain some customers and win some new ones. And then let the benefit flow out to everyone around me–flowing out to my co-workers by making a gift of virtual machines and $100 plug computers.
And that brings us back to where I am today with my work–an ambitious plan to improve my company’s way of working, turning view-source and Powerpoint operators into BattleTech Mech Warriors–enhancing their natural ability to collect and analyze data. The battle-suits are just sitting there ready to be climbed into. Money on the floor. An example of a recent time I saw this happen was when Adobe added video support to the Flash player, and YouTube stepped in to pick up the money. Something very analagous is going on today. Something analogous is ALWAYS going on, in which platform-makers create new potential, and app developers attempt to tap into and unlock that potential, with varying levels of success.
Whenever that potential is unlocked creating new billionaires and empires, it always seem so obvious in hindsight–those smack-in-the-head moments, where you say “I could have thought of that!” Or worse still, saying “I DID think of that!” Or even worse still, “I did think of that, and even IMPLEMENTED it, but it didn’t amount to anything because it was an internal tool, or some implementation nuance wasn’t just-so. This time around, I am determined to be there, doing it right–even if it takes jumping off a cliff and learning to fly.