Experience

Especially with a code base that is mature, assumptions correctly made years ago can be terribly difficult to deal with later when the current and new assumptions reign. Writing code that lasts even 5 years and "is robust" is what everyone wants to do for sure, but the recipe for doing just that is not easy to learn and even harder to actually do. For example, you might know what changes need to be made to conform to even the most obvious object oriented design principles, but the business requirements and time to market needs dictate leaving once again old crusty code alone and racking up yet another round of technical debt. Over time, this technical debt will demand payment and the effects on your ability to hire, employee morale, design changes possible, speed of delivery, testing burden, marketing message etc. become very real and very painful. I wonder, can these concepts can be fully learned without actually experiencing pain? Could you even attempt to learn these concepts experientially in college? My experience so far makes me think that one may know something intellectually, without really knowing it. It seems like for so many, one may talk about design patterns or abstraction or low coupling, but until you actually try to build something the other way, the painful way, you just don't appreciate what you are avoiding. What's worse, junior developers who, for no fault of their own lack experience with "the hard way" have a difficult time understanding why one must "go the long way around" to do what seems like such a direct solution. Passing on the stories of the past and their consequences and lessons seems to be an unending challenge.

From my economics book comes this fictional story which I think illustrates the point:

"One Pepsi plant is managed by an economics major with an MBA and has a labor force with an average of 10 years of experience. This plant produces a larger output than does an otherwise identical plant that is managed by someone with no business training or experience and that has a young labor force that is new to bottling."

I definitely fall into the "young labor force that is new to bottling" camp. In the technology industry there is a tendency to glorify the young and bright rather than those who have the experience and have paid the price to learn the lessons that matter in the long run. Whenever I talk to some developer and we talk about where they learned some valuable lesson about software development, rarely do they refer to a book. Almost invariably, they say, "I worked with Joe on this project and he showed me x, y and z. I learned so much working with him." Smart developers, experienced developers who know how to teach and share important lessons to junior programmers seem like a key to the experience problem. Sadly, junior developers who have the presence of mind to ask, listen and apply what they hear are hard to find, and senior developers who are both willing and able to teach effectively are even more rare.

Perhaps the answer to all this is simple. Perhaps it's just he or she who writes the most code, wins. What I mean by this is simply when one writes a lot of code that increases the probability that he or she will make more of the key mistakes needed to learn how to write software with longevity. Just getting more exposure to "how bad things can get" helps bring a sober reality to each line of code written thereafter.

What's remarkable about all this, is that failures in teaching and learning are what keep us "discovering" new ideas that are 40 years old. Truly, "there is no new thing under the sun."

Metacognitive Miscalibration

A few tweets ago, (I always feel weird referring to Twitter in the past tense) I posted: Why are the unintelligent or uninformed so arrogantly confident while the intelligent and well informed so often unsure and apprehensive? There is something very human to thinking you know more than you really do about a subject or issue. While, this problem can be seen in many areas generally it's particularly acute in software development. For example...

AppleScript in Office X

Back when Mac OS X was just about to go 10.0.0 we were busy working on getting Mac Office working on the new platform. There was a lot of pain involved with the transition. Many APIs had been removed and alternatives needed to be provisioned (and tested), the new Aqua interface guidelines had to be applied to the whole of Office and a host of other issues needed to be addressed. I was at the time on the team that wrote the tools for automated testing and I was pushing hard to get Office wide AppleScript support on the list of features we'd commit to doing for Office X. I wanted this not only for our customers, but to augment our testing efforts. With an API to drive the applications we could automate many "smoke tests" on a daily basis as well as set the stage for long term AppleScript based test suites. Automation testing benefits are often not fully realized, especially static automation, until the version after you setup the automation, so I was especially anxious that we get our API in Office X, so we could reap the return on investment for Office 2004.

I remember talking with Jim Murphy about getting AppleScript support like it was yesterday. We were in building 44 and his window office was near the 2nd story walkway that connected our building to the cafeteria. In my naivete I asked why we couldn't just build a some kind of layer to call into the engine code of Office and presto, AppleScript dictionaries for Office would be complete. I pointed to some improvements Apple had made to make AppleScript a better inter-application communication protocol and ask, "How hard could it be?" Jim turned to me and said, "Hard? There's nothing easy about it. It's all pain. All pain." Then as was typical, turned back to his work, leaving me to think.

In this case, I was the unintelligent and uninformed, but very enthusiastic novice. Jim turned out to be right. For Office X, we did try to do the AppleScript work, but it turned out to be much more difficult a problem to solve. After months of work and many dead ends, we pulled the feature from the Office X feature list. For Office 2004, we tried again. In this case, Jim himself, one of our best developers, ended up spending the better part of a year getting AppleScript to work with Office, which is probably worth another post in and of itself. I thought I knew more than I did, I thought the problem was simpler than it was. My confidence and thinking was miscalibrated. The root cause was my lack of understanding and experience.

Wicked Problems

Sometimes, even the best developers underestimate the scope, breadth and depth of a problem. In a drive for simplicity, I observed in myself and others another kind of design time problem where thinking was miscalibrated. The cycle I have observed looks like this:

1. Developer looks at a problem A, and thinks he or she fully understands the problem.

2. Developer designs a simple solution to problem A.

3. Developer codes the simple solution.

4. Developer or tester or marketing or customer use simple solution for problem A.

5. Bugs trickle in and solution A is modified, bit by bit, bug by bug, until it is patched in a thousand ways and no longer looks or acts like the simple solution. The solution is complex.

Some will say, "Hey! Why didn't that lame developer take the time to really understand the problem? Then he could have taken the time to fully comprehend the complexities of the problem, and then design a simple, elegant solution!" The problem with this is that many subtleties to the problem do not manifest themselves until very late in the development process which make re-architecting the solution for such a small issue un-reasonable. But this is a "death by a thousand paper cuts" issue. What's worse, many architectural issues can only be comprehended after years in a problem space, and as promotions or attrition happen, the so called simple solutions, with their complex instantiations proliferate in code.

Adam Richardson spoke to the challenges of wicked problems when he said:

Wicked problems are very difficult to understand by staring straight into them and looking for clear detail, however. They need to be approached from the edges, sort of like doing a jigsaw puzzle where you find the edge pieces first. Having peripheral vision that is trained to be sensitive to the edges is a key capability (this applies both to product teams and to business units - wherever wicked problems occur).

Normally this causes the developer to tack on simple solution to solutions with patch upon patch applied with the Hippocratic Oath echoing in their ears to "do no harm", but deceptively the bandage is not big enough for he wound, and no one knows.

The Desire to Learn

Another reason why some can feel "informed" when in fact they are not is that they have lost the hunger to learn. They've lost the desire to learn and grow. René Descartes said it this way:

Good sense is the most equitably distributed of all things because no matter how much or little a person has, everyone feels so abundantly provided with good sense that he feels no desire for more than he already possesses.

There is a reason for diversity of opinion, it provides the natural tension that keeps us from thinking we've got the problem solved. When I left Microsoft to go back to school and finish my degree, many asked, "Why are you doing that? What are they going to be able to teach you?" I'll admit that this partially resonated with me. I wanted to be the sage of wisdom, but as I've spent time with teachers and students, many of whom do have less "industry experience", I have learned an enormous amount! To put it bluntly, I have been humbled. I've learned things technically, intellectually, physically and even spiritually. I feel so blessed. I'm beginning to think that what any given situation can teach you depends largely on the person experiencing the situation and very little on the experience itself. Put another way, there's a great difference between 50 years of experience and 1 years worth of experience repeated 50 times.

Personal Pride - the Anti-change Agent

Closely related to the lack of desire to learn is the desire to avoid being wrong. So much in school is focused on being right, knowing the right answer to a test or the correct proof or solution. In sports, no one likes to lose. Everyone likes a winner! But this desire can work into our minds in a limiting way. Leo Tolstoy wrote:

I know that most men, including those at ease with problems of the highest complexity, can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.

At work I would say, "We don't fail half as much as we need to." I still think this way, but now I've found a new reason: Failing keeps our mental muscles and joints from stiffing with pride and forming into the arthritis of the mind.

The Well Intended Deception

Another reason for metacognitive miscalibration I think stems from the very basis of good object oriented abstraction and encapsulation. When I write some simple app with Cocoa's AppKit and Foundation libraries, I am, as they say, standing on the shoulders of giants. I've wondered how many actual lines of code are called when I double click a simple app like TextEdit. Who wrote these lines of code? When did they actually get written? What were the discussions behind their design and implementation? Even Apple couldn't fully answer these questions. It is all this code that even the simplest of Cocoa applications depends upon. But still we continue to have demos around how "few lines of code" were needed to accomplish something. This is a well intended deception, because really, more code is being executed and written, but only now how this code works is opaque. It's well intended because who wants to write, maintain and debug more code? Why not offload this to Apple? I expect these demos to continue, because less code for a developer to write is the canonical example of efficiency, but there's something wrong that comes from this. It's the feeling that you actually really understand what is going on to make your application tick. These assumptions and abstractions can be benign, but they can also make one feel and think that they know more than they do and can do more than easily possible.

In the process of detail management that is software development, making things simpler and more tractable is an excellent goal, but there's a limit to how "simple and easy" things can get and still be valuable. Recently I overheard two students talking about building an iPhone application. The one said to the other, "Dude, they showed this awesome demo of Spore, where in only 2 weeks, 2 weeks! they got this awesome iPhone game running! It's so easy. We can totally build an awesome iPhone app!" First, I love the enthusiasm and I hope they do in fact build an awesome iPhone application. Apple was trying to demo how easy it was to build iPhone applications and I think we've never seen a mobile platform that's better for developers, however:

Taking a senior developer who as spent the last several years of his life building games and developing Spore, bringing him into Apple with full and uninterrupted access to Apple's staff of iPhone engineers that have been developing the iPhone SDK and iPhone applications hardly equates to a college student being successful at all the complexity involved with boot-strapping a company and building a successful iPhone application from scratch. Building great software is still hard, hard work. It's tedious, often un-glamorous and takes someone who doesn't mind getting down and dirty with the details to make something great. Most will simply give up, and even those with the passion and stamina to continue are hardly assured of success. I suppose I'm arguing simply that a healthy dose of Steve Job's famed "Reality Distortion Field" do not a great developer, company or software make.

There are probably other reasons we tend to think we know more than we actually do. But the lesson for me is this: I need to take some time to ponder and reflect regularly. Am I in the "unintelligent or uninformed and arrogantly confident" camp? If so why and how can I get humble? Am I part of the "intelligent and well informed but unsure and apprehensive" group? If so why and what can I do increase my tolerance for risk and decrease my fear of being wrong? A little meta I know, but the title should have warned you. :-)

On Knowing Stuff

I just heard a funny story about a little boy in Columbus, Ohio and it goes like this: A traveler came along one day and said to the little boy, "My boy, where's the road to Cleveland?" The little boy says, "I don't know." "Well, where's the road to Cincinnati?" "I don't know that either." "Well, could you tell me how to get to Toledo?" "No", says the little boy, "I don't know that either." Then the man looks at him rather disgustedly and says "Well you are an ignorant little boy aren't you?" To which the boy replies, "I may be ignorant, but I'm not lost!" One of the great things about children is their willingness to communicate and try stuff out. They are not afraid to say, "I don't know" and yet they seem to have such a clear view of things. It's refreshing, frankly. They seem to be born with this right from the start. In so many ways they are superior to us older folk in their ability to candidly interact with the world around them. I've been watching the TED presentations that, little by little, are being put online and one of the best talks I've seen so far is the presentation on education by Sir Ken Robinson. It's only 20 minutes long, so take a lunch break and "give it a go." If learning is important to you, I think you'll really enjoy it.

My First QTVR: Petra, Jordan

A few years back, my sister and I went on a travel study trip to Egypt, Jordan, and Israel. It was fantastic! I had a digital camera, a wide angle lens, a Kaidan mount adjusted for the focal point of the lens on my Dad's Kodak, DC265 Digital Camera, and enough storage for about 64 "high quality" digital pictures.

Side Note: If in the future, you are going to bring a digital camera to a place where you will most likely never visit again, make sure you have enough batteries and storage to take as many pictures as you can, even if it costs a lot.

Well, I did get a few good pictures, but one of the most interesting places we visited was Petra, Jordan It is was one of the high lights of the trip. I set up the tripod and 12 or 13 portrait pictures later, I had enough to stitch together in Apple's QTVR creation software. It was super fun and I thought you might enjoy the look around. You can also check out the Google Maps images here.

Curiosity and Exploration

From the beginning I've liked what's going on over at Make magazine. I love their video podcast and all the crazy projects they put on display. What they show are regular people, just like you and me, doing stuff, making things, being creative. I was just reading an article titled, Life is Not a Spectator Sport by Darla Isackson in which she makes a very interesting point:

Today’s children are following the example of many adults to become watchers instead of doers, consumers instead of creators, observers instead of active participants. ... Instead of singing, they watch others sing. Instead of making up stories, they watch or listen to stories someone else made up. Instead of figuring out how to do something and developing a new skill they watch someone else perform.

This is so true! There are many reasons for this, but one that stands out to me is the very focus on math and science, these left-brain, less active, and somewhat less creative activities. What's funny to me, is that while the whole schooling infrastructure is supposed to produce excellent employees, what the post industrial revolution market needs is "makers", people who creatively produce.

Paul Graham recently published an essay about the The Power of the Marginal. Near the end he makes this summary comment:

If I had to condense the power of the marginal into one sentence it would be: just try hacking something together. That phrase draws in most threads I've mentioned here. Hacking something together means deciding what to do as you're doing it, not a subordinate executing the vision of his boss. It implies the result won't be pretty, because it will be made quickly out of inadequate materials. It may work, but it won't be the sort of thing the eminent would want to put their name on. Something hacked together means something that barely solves the problem, or maybe doesn't solve the problem at all, but another you discovered en route. But that's ok, because the main value [of] that initial version is not the thing itself, but what it leads to. Insiders who daren't walk through the mud in their nice clothes will never make it to the solid ground on the other side.

The word "try" is an especially valuable component. I disagree here with Yoda, who said there is no try. There is try. It implies there's no punishment if you fail. You're driven by curiosity instead of duty. Which means the wind of procrastination will be in your favor: instead of avoiding this work, this will be what you do as a way of avoiding other work. And when you do it, you'll be in a better mood. The more the work depends on imagination, the more that matters, because most people have more ideas when they're happy.

If I could go back and redo my twenties, that would be one thing I'd do more of: just try hacking things together. Like many people that age, I spent a lot of time worrying about what I should do. I also spent some time trying to build stuff. I should have spent less time worrying and more time building. If you're not sure what to do, make something.

So what should you do? Don't just watch, do something! Don't be afraid to try. Fear kills creativity. Just do something, anything and you'll be surprised how much you'll learn and how well it will go. It's the doing that matters.