Project Planning: Art and Science
In an earlier column of this guide entitled "Why projects are not as successful as they can be, " we proposed that part of the reason for project failure is too much forward planning, and that deciding what decisions to make first and knowing when its time to make a decision are the most import plans.When we said there is a science to this, we meant that there are a couple of guiding principles that you can follow that will get you a long way. The art of it is being able to take these principles and make calculated risks based on them even in a new environment. A lot of the art really has to do with experience and just how you as a person organize thoughts.
Introduction to the Darwinian Evolutionary approach to Software Project Management
The project planning environment resembles the backdrop of evolution in that you are forced to make decisions as to what will be viable in the short-term and long-term given changing climates (technology, corporate and business structure, human dynamics). Mother nature has done a surprisingly good job of this and in a surprisingly simplistic and yet efficient way. Given that, it makes sense to look at this and to see how it applies to our similar problem of project planning in a constantly changing environment.The Darwinian Theory of Survival of the Fittest has been applied to various areas in software practice and engineering areas. Some common ones are Job Cost Planning, financial market predictions, game theory, and database query plans.
The evolutionary approach can be summarized in one sentence. At any given point in time - choose the path that maximizes on possibility of useful options and gets you closer to your goal in the shortest amount of time.
Implicit in the above statement are two main points
- Maximize useful options – it is sometimes hard to tell what will be useful in the future, so you concentrate on what is most useful and possible now and only a little bit about what is useful later.
- Shortest amount of time - because since dynamics are constantly changing - an elegant solution that takes a long time to build is not viable if the problem has changed
Decisions, Decisions, Decisions: What to do First?
A decision choice is a double-sided coin - one decision choice can open up a whole realm of possibilities and at the same time kill others.Decision to do the Project
In an earlier section, we said the most important decision to make is to decide whether or not to do a project. If you take a close analysis of this decision we see - well first a decision to do a project, sprouts a whole well of useful options --- If successful our goal of eradicating redundancy will be achieved
- Increased knowledge
- The possibility that the work we do on this project can be reused in later projects.
It also kills some options because in order to do this project we have utilized resources that could have been applied to potentially other worthwhile or even more worthwhile endeavors.
Balance of Complexity
A decision choice that is complex on the one hand may satisfy your goal better than a simpler one, but will cost more too, take more time to implement, and is usually not as nimble to change should your goals change or your technical environment change abruptly. On the upshot, selective complexity - may increase nimbleness in certain areas while sacrificing others.Case in Point - suppose you are making tax calculation software. If I can predict that the tax rate will never change or I only need the tax-rate in one part of my calculation, then hard-coding the tax rate in my program makes sense for a couple of reasons.
- Its easier to understand - the tax rate is right there in the code
- No messy having to keep track of another file or reading in logic of file manipulation libraries needs - so its easier to debug should something go wrong
- Less dependency on external libraries so its more likely to work in more environments because I only depend on my little program
- My little program is much smaller without the file manipulation code - so I can fit it on my little floppy disk and carry it everywhere
- Its much faster to write this and much cheaper too because I don't need to know anything about file manipulation
But what if you knew that it is likely that the tax rate will change yearly, it is unlikely that I will ever use anything but a windows operating system to access this program, and I'm never going to be carrying this on a floppy because I've got my laptop.
Suddenly all these options you get from minimal complexity are irrelevant to you, but you really yearn for having the ease of changing the tax rate at your whim by changing a little text file that has nothing but the tax rate in it - no searching for the line in your code where the tax rate is.
The Evolution of Complexity
The above example is interesting in that in many cases if you were to start out with the simple approach of hard-coding you would eventually arrive at the more complex solution and most likely this path would be more lucrative than if you had started off trying to do it the more complex way.Think about molecular biologists, they start off experimenting on viruses, work their way up to bacteria, then insects, finally mammals. If you look at evolution - it works the same way too. In software engineering there is a temptation to try to short-circuit this evolutionary process. Perhaps it is because in software it is not quite as clear as it is in other science realms that this is a recipe for disaster. Most people have their eyes on the prize and can't see the tidbits of fruit lying on the floor.
This reminds me of what one of my co-workers said when I criticized his code - pointing out this flaw and that and why don't you just do it the right way the first time. His answer was This is good enough for now. That simple statement really stuck with me. The reason is that the right way that took into consideration all the features that I knew and he knew with a 99% confidence would be needed within the next two months would have been too difficult to test, debug and much longer to program. So this is why we start with viruses when in the end we want to end up with mammals.
Use Foresight Cautiously
Most people think that foresight is a good thing and if you have it you should use this knowledge to preempt disaster. In the above example I said that we knew with a 99% confidence level that we were not covering all our bases and in fact going down the path we chose would require a fair amount of rewriting. Despite this fact, we deliberately made this mistake.Conventional wisdom would tell you this is a mistake, but lets look at the other great mistakes made in the software industry by one of the most hated and respected companies in the world.
In one of Larry Ellison’s (CEO of Oracle) speeches, he basically inferred that Bill Gates (Chief Software Architect of Microsoft) was an idiot – always missing these boats - e.g. he was late for the database revolution, he was late for the web revolution and he is late for this new Grid Revolution that we are moving into.
Now the first thing that any inquisitive reader would think is hmm, yes it seems that Microsoft has missed the boat on many occasions;no doubt about that, yet why is Microsoft so successful? The answer to that question I think is they made a deliberate decision to miss the boat. It was not a result of short-sightedness.
Bill Gates is no dummy – he probably saw these things coming as did many people in the software industry and made a calculated risk as my best friend made a calculated risk and said, This is good enough for now. He deliberately chose not to use his foresight and to miss the boat as we chose to make our mistake.
Let us look at another stupid mistake that Microsoft has made.
While Apple had this great windows computer with the environmentally and health friendly black and white monitor, Microsoft had this dinky DOS system with that crappy color monitor (that lead to a large number of people needing glasses) and that could run on the junkiest of hardware.
Surely Microsoft realized that everyone wants a windows system and that people don’t like their eyes radiated, but they decided perhaps that being accessible to more people is more important than having a better more expensive system that few could afford.
So yes Microsoft did eventually have a windows computer system when it became economical to do so, that was not quite as good as the Macintosh, but was just good enough for now. They had made the decision that an inferior system that is accessible to many is better than a polished system that is accessible to few and that people value color more than their eyes.
E-Week had an interesting tidbit of a speech that Larry Ellison made when introducing the Grid.
"I think what happened was Bill [Gates] sent a team
of people out to IBM to figure out what was new," Ellison said.
"Somehow the Microsoft intelligence-gathering team held their Yahoo
map upside down. Instead of making a left turn into IBM Research, they made
a right turn into IBM's museum."
(quoted from eWeek article)
Contrast the above quote to one from Bill Gates (see Bill Gates Speaks)
Interviewer: Is studying computer science the best way to prepare to be a programmer?
Gates: No, the best way to prepare is to write programs, and to study
great programs that other people have written. In my case, I went to
the garbage cans at the Computer Science Center and I fished out listings of
their operating system.
-- Bill Gates, from "Programmers at Work" by Microsoft Press, interview with Bill
I laughed when I saw the quote above because that's the way I learned how to program.
So while the joke is probably true (We all know that some people prefer to read their maps upside down), I think the Microsoft intelligence-gathering team knew very well where they were going. The museum is froth with carcasses swept aside because they were just too far ahead of their time to be viable. Of course I suspect that Microsoft was a little too late on one occasion. An occasion that Larry perhaps knows well.
Take the invention of the Web - a very simple idea of navigating from link to link; so simple that someone must have thought about it before its existence, but yet perhaps it just did not have the resources at that time to make it viable.
It is said that Leonardo Da Vinci invented the first design for a helicopter, but poor Leo was too far ahead of his time for such an advanced concept to flourish.
The Future can be changed
Part of the reason why foresight can be such a dangerous thing is that by preparing for the future, you often reinforce your worst fears. Knowing when it makes sense to make the change and when not is really an art.Take a war for example, political, technical or otherwise. Imagine if you could read minds and you know exactly what your enemy is thinking. He is thinking that he wants to kill you. Your instinct tells you - got to protect myself - put up the defenses. Your enemy sees this and says -- yes he has proven that he wants to kill me too so I want to kill him.
So in the slew of things -- you either both kill each other, or the stronger opponent wins.
Now if you couldn't read minds, you would simply judge your opponent by his actions or look around and see that you have battles on other sides that are much closer and easier to win. If your enemy doesn't see you raising arms, he may conclude -- hmm I want to kill him because I think he wants to kill me, but he's not trying to kill me maybe I should rethink this.
So in this situation even if you are the weaker opponent, you may not die because as your enemy comes closer - he is no longer your enemy. Again if you are the weaker opponent and your enemy is still bent on killing you he may put all his resources into his defenses against you and not see the enemy behind him about to cut of his head. So though you were weaker, you didn't really need to kill your enemy because someone else did it for you.
Look at Linux and Open-Source. Many people think that these two are a natural enemy to Microsoft and have fantasies about how Microsoft is plotting on all sides to destroy it. Microsoft is probably half-heartedly plotting, but also thinking - I've got more lucrative fish to fry. Meanwhile all these people are heating up with their hatred of Microsoft - getting ready to explode in their own pot thinking that Microsoft is doing the same - pouring all their defenses into this battle. Novell was an enemy too of Microsoft and now seems to be coming back. If you look closer at Novell for example -- on the one side they have SUSE Linux and on the other they have Mono - an open source initiative whose directive is to bring .NET to Linux and beyond. So it seems very strange to bring your enemy into your camp unless your enemy is not your enemy anymore.
You look at another enemy of Microsoft, Apple. Didn't Microsoft give them some money to sustain them; doesn't Microsoft still produce a lot of software for Apple? Gosh maybe they were worried that if they completely killed off Apple they wouldn't have any more enemies to make life interesting.
You look at Red Hat and they are competing with Novell. In one statement they said something to the effect "We are abandoning our desktop version because home users are better off with Windows." - huh Why is Red Hat pushing people to the arch enemy Microsoft when they are arch enemies? Seems like a rather odd battle gesture.
The fact of the matter is that your enemy is only your enemy sometimes and your friend in other situations. In situations where it is a WIN-WIN or a LOSE-LOSE proposition to work together or to work apart, then your enemy is your friend. In situations when it’s a WIN-LOSE then your enemy is your enemy. In the Prisoner's Dilemma, a situation we often find ourselves in, you don't know for sure how trustworthy your bedfellow is, you just have to learn to read his body language or decide that you've got better things to do with your time than worrying about it.
When is it no longer good enough?
Through out a project life-cycle you will often face the decision "Is this still good enough or do we need more and can we have more?"While foresight can be a dangerous thing if used inappropriately, it can be also beneficial. When we started off we only did client-side windows development. We saw all the problems and benefits of it and then realized that sometimes a desktop application is just not good enough. It's too difficult to change, too difficult to maintain and too difficult to deploy especially when you are trying to do all of this for a client 500 miles away from you. In the distance we saw this thing called "The Web." We were already using the internet. It was a rather immature creature, but it was beginning to mature. Browsers were becoming better and more powerful, development suites and languages were sprouting up for it, and most importantly high-speed internet access was becoming cheaper and more plentiful. "The Web" seemed to be leading in the direction we wanted to go, but we realized then that the infrastructure just wasn't quite there yet -- not quite as rich as a desktop app and not fast enough. So we just stuck our proverbial feet in the icy lake testing the waters to feel if the ice had hardened enough to support our goal. We discovered that for many projects, it was sturdy enough and for others it was not, but that the ice kept on getting harder to the point that projects we wouldn't consider doing as web-based last year, we would only do them web-based this year.
I also look back at all these great projects and technologies and how they failed or were marginally successful because while the development was sound, the thought was sound, and the reasoning was sound, the infrastructure they needed to survive was not there.
Here is another amusing quote from Larry Ellison- quoted from In Larry's Own words
Most people are so in love with their own ideas that it confines their thinking—creates boundaries and limits their ability to solve problems. Bill, however, has this Asian-like ability to manage his intellectual vanity and take ideas, regardless of where they come from, and put them to work for Microsoft.* The terrifying thing about Bill is that he's smart enough to understand what ideas are good—what's worth replicating—and he has the discipline and resources to get on with it and make it just a little bit better. That's very Japanese. That's very scary.
| Readings |
| A unified theory of software evolution |
| EWeek: Ellison: Microsoft's Out of Step with Grid |
| Bill Gates Speaks |
| In Larry's Own Words |
| Prisoner's Dilemma |
| Red Hat: Stick with Windows at home |
| Famous Bill Gates Quotes |
| Michael Stonebraker Speaks Out Here is an interesting interview with Michael Stonebraker - the chief lead of INGRES and POSTGRES DBMS and Informix and who is considered to be by many one of the great fathers (or uncles) of relational databases. What is very interesting about his comments was that he states if he knew then what he knows now, he would never have undertaken the Ingres project, which turned out to be one of the most successful database projects of its time as well as his biggest success. He also has some other interesting observations on general development. |
Del.icio.us | 
Reddit | 
Digg | 
BlinkList | 
Furl
Spurl | 
Simpy