Mike Levin SEO

Future-proof your technology-skills with Linux, Python, vim & git... and me!

Recovering Initial State - The Journaling Life Hack

by Mike Levin SEO & Datamaster, 07/03/2013

This post is not an edited article. It goes back to my raw daily work journal stream of consciousness style, and original intent for this website. It is my notes from throughout the day to keep myself inspired, on-track, and doing the right work for the right reasons.

Don’t forget how competitive things are. Remember the important lessons. How quickly you recover initial state both on a daily basis, and on a switching hardware basis is one of the critical skill-sets of the world you’re going into. You have to have and maintain as many advantages as you possibly can. And that means restarting daily journaling.

Getting yourself generally off of Microsoft and onto Linux - and Linux Server at that - is probably one of the best moves you could have ever made, and you’ll probably spend the rest of your life teaching people how to make the switch, and still make it super-relevant in the modern workplace and world.

And today, I officially proclaim my intent to bootstrap and kickstart my daily work journal, as is the current theme of the site. Sure, I’m developing Levinux, my own distribution (eventually - today, it’s a remix) of Linux, and that will be the page that 99 out of 100 visitors will be here for. They’ll say oh what a clever domain for the Linux, and never realize that there’s a whole rest of the site here. That’s not even the homepage. MikeLev.in is, and that’s where I’m planting this daily work journal.

I’m going to make a point to try to keep the actual journal writing process in vim instead of something much slicker, like Google Docs or an iPhone app, because that is so tied-to the entire theme of the site. It’s not a vim site. But rather, the text editor called vim, represents the rewiring in your brain that I propose for you to get through the Levinux education. And so, I must eat the dogfood - and practice what I preach, if you will. By using vim as my daily journaling method, and using the following line to get it into WordPress, I communicate that I am the genuine article. I am an implementer, and a knower-and-doer of things.

One article compels you to the next and the next, and your ideas refine and improve. You can sometimes go back and change your writing, but that’s a different act. There’s refining the perfect piece of meme communication machinery, or in an actually completely different state that also requires writing, you can be stream of conciousness recording thoughts for processing and state-recovery much as you would combine sleeping on an idea for a fresh view with keeping a precise journal, so you can recover your thoughts, and even your process getting to those thoughts, to play them back like a sort of idea recorder and player device… which they are, of course.

Okay, but in the ultimate act of self-aware matter, what we do is actually change our own programming. Learn vim. You will forever be forced into the mode of programming yourself until such time as you’ve committed everything to muscle memory and spontaneous expertise - where you really don’t have to think about difficult tasks while you’re performing them. The excess capacity that you free up by not having to think about actual tool-usage goes instead into what really needs to fullest capacity of your concious mind to work the sort of magic on it that habit and auto-pilot can’t handle. Mostly creative stuff.

Until that is, until you realize that creativity flows from the unconscious mind. But I get ahead of myself. That’s a story for another day. I want to make the self-programming example because I’m using vim complete. It’s like being born again - being an infant again - just putting 2 and 2 together for the first time. Without getting into to much details, most any task you can do with a mouse and pointer, you can do with keyboard shortcut commands.

You may be skeptical of this statement, except realize that the vim text editor plays a lot like a video game. If you know the magic keystroke combination, you can format that paragraph the way you like. It’s like a Mortal Combat finishing move - for those old enough to remember that. I am both old enough to remember that, and young enough to apply lessons learned. Imagine keyboard shortcuts that can do anything on a screen of text, using a series of keystrokes that actually make sense. In other words, you’re learning to speak a language with syntax - the language of controlling text in the vim editor.

And in that language, I always need to reformat paragraphs, and I’m still doing it the hard way. I happily just learned “visual” select mode, by which you can hit Shift+V and then use the arrow keys to highlight any number if lines. Then you hit: gqgq on the keyboard, and all the line-wrapping is fixed. However, the bummer in this is that you have to move your cursor to the top like of a paragraph, then hit Shift+v, then down-arrow to the last like, then hit gqgq on the keyboard. While this has gotten me by for awhile, it’s getting tiresome when I know there’s just a slightly better spell to cast to reformat the entire paragraph in which my cursor sits. I do this process after I’m done writing every paragraph, just to make sure line wrapping is correct (why this is even an issue is another story). And so, getting the keystrokes down and committing it to habit. See? Self-programming.

So, this is where we get to the discipline part. We have a goal in mind, and stuff we have to do between here and goal. And we have to do things that belong to that set of stuff and not some different set of stuff that won’t move us towards our goal. The only sincere distraction that might be able to get you at this point is a valid nested dependency. In other words, you have to do tiny projects B and C before you get to A because A depends on B and C already being completed. But beware, everything looks like a B and C if you’re trying to avoid doing A. Even this writing. Enter, the 1, 2, 3 method.

1. Go to Google. Google select current paragraph in vim. Ah! It took about 2 minutes to see that the letter combo vip selects current paragraph. So, because gq is the code re-wrap lines, the entire key combo is vipgq. If I can simply remember commit vipgq to muscle-memory for re-wrap the text of the paragraph my cursor’s in, then my productivity subtly goes up everywhere forever into the future. vipgq vipgq vipgq… okay.

2. Okay, I have the actual code I need to improve my life. I have the snippet that I need insert into my own personal overall program in order to improve my life forever forward. And doing so is an example of what I mean by the mental exercise and resetting yourself back to a child-state that I talk about when taking up vim. Achieving true overall mastery in vim represents nothing less than a second adulthood in your own life. You learned a second language and not just some spoken one, but one that makes machines and the information inside them bend to your will.

Okay, but actually MAKING the habit requires a sort of consistency that can sometimes be achieved, per Robert Cialdini’s principles of persuasion, through public commitment. And so, we go full circle back to this journal entry. Why this journal entry? It forces me to do introspection with my own life. But too much self-referentiality can be as bad as none. And so is the decision to make every journal entry actually count for something - something tangibly achieved that you can stand behind to your bosses, family, and future students who will be studying you as a homework assignment. With so few of us on this planet and technology really about to take off, each one of us is going to be someone’s homework assignment in some distant future. We’re the generation that existed at the birth of the information age. Whatever information you leave behind that hasn’t been destroyed or gone forever uncovered will be studied. Hard drives that have crashed but have never been thrown out due to some dream of recovery and are kept generationally in the family will be a rich source of the digital history layer.

But it all won’t go on the public net. A lot will be held back by the family, for reasons varying from avoiding embarrassment to maintaining some unique competitive advantage for the family. This is not isolated to just information archives, but it could actually apply to running servers with continuously running tasks whose value is inherent in its continuously running state. This is thee world we are going into. Things are really going to get crazy.

But I tell you here, you take great advantage of that. You predispose things to your own advantage, and eventually your own descendent’s advantage. Think of it in terms of treasure maps. Say someone from this generation knows the location of a secret wealth of treasure, and wants to leave a map. But instead of parchment and ink, you have the entire faculty of the modern state of technology and electronics at your disposal. Even mere electronics sounds too fragile for this potentially trans-generational code.

But let’s come back to earth. The specific life-hack I’m talking about is public commitment to create consistency. That’s Peter Cialdin’s observation that going on the record in a place that you really think is going to be seen by people and you will be judged by has major force to motivate you to do stuff. So, it’s time to re-birth a section of my site that I always keep retiring, called revelations or headsmackers or some variation on the image of realizing something critical that changes things forever forward. Self-programming should be a series of headsmackers. If you’re not experience them all the time, you’re not maintaining your competitive edge in the information economy.

Such a long entry! And it’s only going to get longer, until I reach a natural “cutting” point in the journal entry. So, I have to make this day amount to something. I need to keep keenly aware of time now. I had lunch already. It’s not quite 1:00 PM yet. I’m keeping a Windows widget out on my desktop now both to remind me about time and about desktop widgets. On the Mac, it would be Dashboard widgets. Windows and Mac vary a bit here, with the mac choosing to make a sort of virtual space for these things. On Mac, you can use “Web clips” as a dashboard widget, which is good information indeed. On Windows, they’re called Desktop Gadgets, but they are not in Windows 8. Hmmmm. Though if anyone is really on Windows 8, I assume it would just be one of those panels. Windows 7 Desktop Gadgets are lightweight HTML, and I’m thinking whatever I do for them, I could adapt to a Windows 8 Panel easily enough.

Okay, I think the existing FAQ system is how I should add these revelations or head smackers to my site, under each respective page. And so, the vim realizations will go on: http://mikelev.in/computerprogramming/vim/

Start thinking in vim. http://stackoverflow.com/questions/1218390/

Okay, other things I did today was to continue my switch-over from using PuTTY under Windows as my SSH terminal windows to using MinTTY, the new default console program under Cygwin. While it’s better than just PuTTY, it still takes getting used to and needs customization. Namely, I had to add the old Mac Monaco font to get a usable programming font that would allow 3 readable 80-column text windows on a 1920 x 1080 resolution screen. All those other fonts like Lucida and Console are a just a little too wide to allow 3 80-column terminal windows. I had to set it to emulate xterm-256 color, and be 69 rows high to reach the top and bottom of the screen. I get no tabs, but that’s a small trade-off. Ctrl+Tab cycles me between the terminal windows without having to go out to the bloated confusing OS switcher with Alt+Tab. Okay, so now I’m comfortable on Windows again with Unix-y terminal work.

Now, it’s after 1:30 PM, and I want to announce something today - probably getting Pinterest functions working again in Tiger. It’s been challenging, because Pitnerest is blocking both screen scraping and the JSON calls to the API that powers those “number of times pinned” chicklets. I have a solution in place, but it requires discovering, testing, and hitting proxy servers. And doing that was not purely on a per-function basis, or it would have been relatively quick to fix. But because the way the Tiger system recognizes the use of a “url” column the first time it encounters it, and make the HTML that was fetched for that url available to every other function on that row that takes url or html as input. See? There’s never any reason to call html twice for a given URL if you’re just going to parse this and that out of the same HTML. The efficient way to do that is to fetch HTML once per row.

So, if anything calls for html as input, then it uses what is fetched for the URL column without proxying. But say that call needs to be proxied. The function that needs that html has to turn the proxy on, make the request, throw the cached html into the global reuse (per row) location, and then toggle the global proxy off, because it should be off nearly all the time to not slow down other functions that don’t need to go through the proxy server.

Now, there’s a nuanced issue here. The proxy server setting technique I’m currently using is global. That means once you set it, everything uses it. I don’t have enough knowledge yet to make a custom opener that is just on a particular function. It’s very easy to just call the proxyme() function within another function and throw the global urllib2 object into a proxied state. But all the other functions slow down. And because Tiger is sequential, I actually can be toggling a global object this way and that, making it have whatever behavior it needs to have for whatever’s “in focus”. But in doing so, we have to assume whatever need the function had for the proxy, it is over after that row is complete. However, we are faced with the issue of the next row, which will again need to be proxied for that particular function.

Also, the ezscrape functionality of Tiger is in the picture. This is where most everything that looks like a simple screen scraping or json query is not really its own standalone function until runtime. Instead, it is just an entry in a Python list-of-dicts object until the bookmarklet is run. Then, they become functions. So there is nowhere convenient to insert the proxyme function. It seems like another name/value pair in the list-of-dicts is whether or not the screen scraping item is meant to be proxied or not - with the default not. This is the most difficult thing, I think, about supporting proxy servers in Tiger.

Okay, I see three proxy behaviors. I have to make sure they don’t conflict with each other, producing buggy, annoying behavior.

1. Any given function can be proxied. All you have to do is make a call to the proxyme() function anywhere in the function needing it. It will reach into the global urllib2 object and make it work through a proxy - but only for the duration of that function running. The overridden urllib2 object will be automatically overwritten back to default every time Tiger moves from one column to the next in its processing. But if global HTML were fetched with a proxy, that global HTML remains accessible to the rest of the row, so the efficiencies remain. There are conceivably some undesired side effects here, but for the most part, that should work spot-on. I should also point out that the proxy server that was discovered is plugged somewhere that has memory across rows. So every time a proxyme() is subsequently encountered (on the same proxied column of the next row), proxyme() will check for the existence of a verified working proxy before trying to discover and test a new one. In this way, the same proxy is generally used for the entire worksheet. I should add what it is to the config tab. Turn it into one of those config values instead of just reused across rows. It will make it a lot more clear what’s going on, and help with debugging.

2. I was intending to auto-proxy. The problem with that is that the symptoms indicating that you need to use a proxy is not the same across all API’s. Therefore, I don’t want to put all the checking in against every single indication that the API has reached a quota. Functions, and even ezscrape records, should specify whether or not they are to be proxied. And there’s also the question of how much control you give the function over the selection of the proxy. Should it be USA only, or western hemisphere only? Or are originating requests from China and Russia okay, because if so, it makes a whole bunch more proxies available. Okay, so turn off the global proxy ability. If a function wants/needs to be proxied, it explicitly has to ask for it. That’s going to be a principle within Tiger. No proxy unless asked-for.

3. And the third way would be deliberate global proxy’ing. This would presumably be a different entry in the config file. I am tempted to just support proxy=TRUE in there, so proxy selection is totally up to Tiger. But there will inevitably be record-keeping of which proxy was used, so why not let that be the usage toggle? The answer is, they have different purposes. One showed you which proxy was used, and the other told you that a proxy was to be used globally across the board for every function. Soooo… how about… two new entries in the config file:

proxy=TRUE proxyused=

So, there are a few states to consider:

proxy not present proxy present and set to TRUE proxy present and set to FALSE proxyused not present proxyused present but no value proxyused present and value set

And there is the intersection of the first and second set, making 3x3 possibilities. But I can rein in the behavior so just the most frequent use cases ever really can occur.

Okay, the global proxy auto-attempt was ripped out. Now, it’s completely up to the functions to invoke proxyme. Also, proxyme has to use the new config tab based record keeping. So, both the proxy and proxyused config entries are created as-needed. If proxyme is encountered at all, then a proxy entry is made in the config tab and set to true. After a proxy is discovered and tested, the proxyused entry is made and set to the IP and port.

Okay, the day is winding down. Going to cut the journal entry here. But I will be able to recover initial state very easily when next I sit down with this project.

I’ll spend my time on the subway using today’s learning to accelerate what I actually get done when next I’m back at the office. And so on, until I am spontaneously expert at all the Linux short stack tools I’m using, and all excess goes into tackling this or challenge.