Building Frameworks In The Cloud

by Mike Levin SEO & Datamaster, 01/14/2011

Now that I have my new platform—Linux / Python / Mercurial / VIM, the idea is to drive interesting projects forward now that are valuable to my career, and apply to a broad set of general problems, and hopefully have never been done before. This desire to generalize tools is what leads programmers to create frameworks—just one more layer of abstraction to make some type of work easier. And now there’s countless frameworks to choose from that, so why build my own?

Building Frameworks In The Cloud

It has been my experience that on any given programming project, the developer will tell you it’s 80% done, and all it needs is a user interface. What they really are saying is that it’s 20% done, and they’re underestimating what putting a solid, tested and elegant user interface really requires. I am no different, and now after writing four prior Web-based frameworks with their own UI’s, I have been humbled first by the likes of Ruby on Rails, but then closely followed by Google Docs, and Google Spreadsheets in particular. I have no desire to try to rival the UI of fully-featured Web-based spreadsheets for the type of spreadsheet-centric work that I always need to do.

No one wants to admit it yet, but Google Spreadsheets has killed the need for 2 out of every 3 Web-apps for which developers would normally reach for a framework like Ruby on Rails or Python’s Django. Plain old spreadsheets have got all the basic CRUD operations on records (create, “show” record, update, delete). They’ve got security at least as good as most Web apps. And with the collaboration, revision control, low-maintenance requirements, and perhaps most importantly, a Web API, programmable spreadsheets are frankly a better choice for quickly solving a broad set of problems than either the Office Software or Web frameworks that came before it.

Why fuss with web-forms, when you can just go in and edit the data in rows and cells directly? That’s been sort of a holly grail of Web data manipulation for years, and it’s here. Have you ever noticed that no matter what lengths a developer goes to build a nice Web-based user interface, the first thing a user wants to do is export it to Excel, anyway? Well, why not just make the spreadsheet environment—the place where users ultimately want the data anyway—the place where the data gets pulled in the first place? With the API-equipped cloud-hosted Web spreadsheets that are available today, this is a real option.

Wait! But wouldn’t you rather have control of the UI details? Control of the servers the code runs on? Control of the files and data you create? Control! Control! Control!

Control is an illusion. Show me the person who can burn their CPUs from sand and write their own operating system from scratch, with no compilers, interpreters or text editors to start with, and I will show you someone who has real control. The rest of us are standing on the backs of giants, and millions of man-hours of work. We use pre-built tools that create great potential by magically bypassing most of the complexity it is built on top of, because no one wants to deal with it—beyond understanding it academically to write better code. Doing seemingly simple things should be simple.

Old-school IT folks want to think that Web services are a different and worse sort of dependency than processors or electricity  because of response-time lag, security, and the threat of a remote kill-switch. But electricity can be turned off. Your connection to the Internet can be shut off. And some of those web-services are running on billion-dollar data-centers with reliability those old-schoolers could not hope to rival in their lives. And if it does get turned off, you just connect to a different provider with similar service. We live in a world of interacting systems, and when your application is defined by that interaction, you shouldn’t be shy about embracing one that can eliminate where 80% of your development time gets swallowed up—the UI.

Computing systems are a tower of layers, each layer abstracting the otherwise complex details underneath—from the logic gates in the silicon chips all the way up to all the things a human sees and touches. Today, many of us are capable of doing some rudimentary user interface work at the Web browser layer, using JavaScript and CSS—because we’ve had to. It was easy enough, aside from browser compatibility issues, so that we could cobble together simple things, occasionally helped by JavaScript libraries like JQuery and Dojo.

But entire Office productivity software suites have now been built under those same browser APIs—and even with server-push features that transcend the standard request/response model of TCP/IP. The JavaScript underneath there is now so complex that I wouldn’t want to touch it, and would never even presume to recreate it. What’s further, applications built on Web APIs are now Web app hosts, because they have their own APIs in turn. Another layer of complexity is swept away—namely, the UI. If only you can just deliver a highly useful application in the Office Software interface…

And so, I can. To do it without all sorts of modifications to the Office software, I’m using conventions over customizations. This means that you can be super-productive quickly, so long as you learn and abide by certain standard program behavioral conventions. Developers on my framework get to deal purely with the logic layer, which is where their time is best spent anyway. In order to do more unusual things than what the conventional behavior allows, you do your own customizations, but you’re leaving the “safety rails” of the framework and start leaning more heavily in the underlying programming language. But if Google Docs can provide enough of a UI, why not take the free gift?

It may seem strange to imagine a programming framework living inside of, and piggybacking off of general Office productivity software. Think of tasks you might have normally used external data collection tools for, and then subsequent Excel formulas to massage the data. Now imagine the two wrapped together and reduced to a button-press, like a macro. If you’re sitting on top of a way of manipulating data anyway, why not be sitting on too of a way of pulling and massaging that data in the first place? Think next-generation world-aware spreadsheets.

There may have been similar Products before, but I don’t think they caught on due to their host products being Microsoft Office with automation achieved through Visual Basic for Applications (VBA). The problem here is that the dependencies are costly and local. No one wants to spend money for this stuff, especially when the benefits are difficult to understand until after you’ve used them awhile. And local dependencies break easily, don’t flow between machines, and resist collaboration. Web-based alternatives just keep working anywhere you are—even on your phone.

Google themselves have plugged a few Internet-aware functions into Google Docs, where you can pull such tidbits as world population into a cell with just a function call, but they have not gone far enough. They have a JavaScript-based language built into Google Docs called Google App Script where the code executes on infrastructure very similar to the Google App Engine—but your function capabilities are limited to the components and libraries they provide—a big limitation if your functions perform site-crawls and such. Instead, it’s the Google Data API to Google Spreadsheets that I use, and not App Script. As such, I execute my code on my own servers, communicating with GData just for the user interface.

Yes, this approach creates dependencies on real-time services, which makes some folks uncomfortable. But remember, independence is an illusion—especially on products whose purpose is to pull data over the Internet in the first place. If piggybacking on a free product hosted on the billion-dollar data-center of a company that changed the world is too much dependency for you, you might in time provide your own UI server as folks open source a Web spreadsheet that have GData-equivalent APIs. But for now, there is no viable alternative, so open minded types can enjoy the competitive advantage of shorter product development cycles and astoundingly cool capabilities that old-schoolers are reluctant to realize.