Kicking the High-performance Tracking System Habit

by Mike Levin SEO & Datamaster, 03/09/2010

Early on, I fell into the Microsoft web development trap like so many others, because IIS was so widely deployed in the small-office environment, and Active Server Pages were so easy to start coding. Not long after, I discovered SQL Server, and not long after that, I learned how to tweak out it’s performance so the IIS/SQL Server combo could handle unimaginable traffic on a single machine. SQL Server, I believe, is a fabulous product, so I used it to make Web tracking systems. Back then (circa 2000) analytics systems like Urchin were still very expensive.

Years later, In the course of designing HitTail, I was still doing about as much as you can as one person can and should–design the network, configure the hardware, install the OS, and software harden the box, write the apps, admin the database, load test, install the hardware at the hosting site, and ongoing monitoring. I took the same thorough care in optmizing the app–tightly coupling components in a counterintuitive but performance-enhancing fashion. I used various levels of filters along the network stack making sure that each layer in the stack did as much pre-processing and pairing down of data as it could. For example, I used distributed computing in the tracking system’s JavaScript on the browser to pre-crunch the data, so it was ready to show in a report the instant it hit the database. Consequently, at the database layer, it was handling 1/100th the activity as any other similar app with similar requirements and specs. I only deployed features in perfect balance with hardware capacity.

This effectively allowed me to load up 100-times the number of users on each server as anyone else. I was (and still am) getting tremendous power for the dollar out of my equipment. I fell in love with this incredible control and its ability to yield amazing cost/performance ratios. A lot of my identity got tied up in this ability.

But working this way as an application developer is only possible if you are your own sysadmin and netadmin, and have some decent hardware and hosting. But that is not often encountered anymore, with the keys to the kingdom being tightly held by individuals not always ready to share.

In fact, at one point in my career when I had the proper motivation, I decided to fight the fight, locking horns with an an old-school sysadmin to the death. It shook the company to it’s foundations, challenging all it’s core values. In his view, what he saw as “marketdroids” cannot be allowed to challenge the dictates of the engineers that ran the company. I’m convinced I was partly responsible for a subsequent change in management that ordered my equipment “freed”. The sysadmin had to yield, but abdicated all responsibilities for the consequences, which is pretty funny because my work is widely credited today for turning the company around and triggering it’s rise to profitability.

And so it is, I came of age technically on Microsoft technology, embattled in server-politics, and eeking out every last ounce of performance on Wintel-stlye servers. I wrote high-traffic apps like tracking systems serving tens-of-thousands of sites per server. I created every piece of software it took to run a business, shy of accounting and payroll.

And now, I do none of this. My job responsibilities have changed. I’m no longer working somewhere that I care to engage in such politics. My Microsoft-specific high-performance tracking-system expertise could no longer be put to use in day-to-day work. I was a child again, approaching fourty. I had images of my father’s textile industry drying up around him as all the jobs moved overseas, or were 50% travel. Could this happen to me already in tech at the dawn of the Internet? Could an old-school webmaster remain a heavy-hitter in this new environment?

I decided that the answer to these questions would be yes. I am still a young man in a young man’s game. Better still, I’m a battle-seasoned old man who looks young. I’ve dueled nasty sysadmins with deep bags of tricks and won. I’ve deployed Web 2.0 tracking apps combining creativity and technical chops. I continue to have the ability to put forth the occasional extraordinary effort. The results of those extraordinary efforts are usually pretty cool new “platforms” that buy me a good 5-years of competitive advantage.

Now, no longer needing to build real-time tracking systems, I decided to turn it into an advantage. I am going from the very-very large, high-traffic application demanding 100% up-time and zero margin for error, to the very-very small micro-server that con be run anywhere, anytime, for as long or as short as you like, in order to do the types of data-collection investigations that are my responsibility today.

I realized then I could get 100% control back by using these smaller servers that I could run on virtual machines or very cheap micro-servers. And I can eliminate the time I need for coding even on these little servers by piggybacking on top of massive pre-existing cloud infrastructures–thereby doing away with the entire chore of building user interfaces!

So, now I have a formula for re-positioning myself professionally. I’m even thinking ahead 5 to 10 years as the capabilities of these micro-servers increases surpass the capabilities of today’s bigger servers, and things go full-circle again. But by then, I will be fully kicked the Microsoft habit, and maybe even moved onto other more appropriate database technologies than SQL, such as HDF5.