Cykod Blog

Note: this post takes the iterative nature of software development for granted. As a consultant on dozens upon dozens of Web projects, I take it for granted. If you have had great experiences in complicated projects that go from specification to completion without change, I've never met you but I would love to as you sound like a mythically perfect client.

So you've got to get a project done that needs outside help. Whether it's for a corporate site or your next change-the-world idea, the normal sequence of events from this step is probably:

1. Write up a Specification
2. Put out a Request for Proposal
3. Look over the responses
4. Of the responses from reputable looking companies, pick the cheapest

Number four might get more complicated if you are working with companies you already have a relationship with, but the bottom line usually comes down to finding the right balance of price and security. How much are you willing to wager on your project's success to get a cheaper price.

Now let's take a look at this from the perspective of the consultant. I assume most consultants work off a standard hourly rate, estimate the number of hours and other costs (software, hardware, etc) and come up with a quote.

For inexperienced consultants, the number of estimated hours is usually a vast underestimate of the actual time it takes to complete the project. For experienced consultants, it's usually a significant overestimate of the time.

In the former case (and I've been guilty of this) the estimate is based on a best-case-scenario that fails to account for the risks of the project. It's a if-everything-goes-right number. Developers are a cocky bunch and tend to see the world through rose-colored glasses right up until the first bug. In the real world rarely does everything go right, and if the project is done at too low of a number, disaster will ensue.

The consultant will be motivated to triage features on the project because once they have gone over the number of hours estimated, they will essentially be losing money for every additional hour of work time. Since all good software development is iterative, it is a given that there will be new requests from the client based on feedback from prototypes that weren't part of the spec, but are ESSENTIAL to the project being useful when it's all said and done. These will be refused or re-quoted and features that are no longer necessary, but are in the Spec, will be added-in.

Barring a mid-project renegotiation, one of two things will happen: first the project might degrade into an acrimonious battle of wills back and forth as both the developer and client becomes less and less flexible, the former trying to do the minimum as outlined exactly in the spec while the latter holding the developer to features that have no reason to exist except for their presence in the original spec. Alternatively and much more seldom, the developer will eat the costs of the additional time to provide a quality product and a happy client. While this looks like a win from the client's perspective, future projects from the client will either be turned down or vastly overestimated to prevent a repeat. Any changes to the system will need to be done by a new party (and remember, software is iterative so there will be changes), bringing a whole new set of risks into the mix.

Experienced consultants, i.e. ones who have gone through this disaster-cycle a couple of times, will see risks and dangers hiding in every crack of the specification and will base their estimate on Murphy's law, making sure their ass is covered for any eventuality. What consultant, after all, wants to lose money on a project? The risks on the consultant's side: finicky client, non-payment, etc, are just as real on the consultant's side as the clients side.

To sum up, as it stands now, someone who needs a project done has two options:

1. Underpay an inexperienced consultant and count on their goodwill professionalism to complete the project.
2. Overpay an experienced consultant

Neither of those two options should sound particularly good, so what's the solution?

Most projects, from a risk/completion versus time standpoint operate something like this:

The risk is enormous at the start of the project. It falls quickly, however, as any developer worth their salt will try to solve the projects unknowns first and foremost. If you are building real-time stock viewer, you don't build a whole website, membership and payment system before you figure out how to get and the real-time data. Thus a couple weeks into a project, a lot of risk as already fallen by the wayside. Things will either be able to work the way anticipated, or they won't. If you pay a developer hourly for the first couple of weeks, you'll be able to remove a tremendous amount of risk from the process and won't force the developer to quote a number that covers that risk. Since you don't win if the developer underquotes the project (see above), why should you lose if they overquote it?

What if, as an alternative to asking for a quote in response to an RFP, you asked potential consultants for their hourly rate and an estimate on the number of man-hours each of the chunks of the project might take along with a confidence measure of each of those chunks?

While these early estimates will be a little all over the board, you should see some convergence on the number of hours from the reputable shops and you should be able to pick out the outliers on both ends to stay away from.

Now pick a firm on the same criteria you would have originally, but make sure to request a number of smaller milestones with working prototypes. Each of these milestones, coming in at a specific date represents a certain amount of developer time. If the developer is ahead of schedule, they can add features to the milestone, if they are behind schedule then features will need to be removed, but the date for a working prototype is the sticking point.

Provided that you have workable milestones, you, as a client, will have a much better idea of how the project is progressing, and, since you are paying hourly, can iterate on changes as necessary to get the best outcome for the project. If the developer can't deliver the first milestone, it's better to know that immediately and get out of the arrangement as early as possible rather than hang onto that sinking ship all the way down to the bottom of the ocean.

At first glance it might appear that you've shifted the risk of the project from yourself to the developer, but the truth is that the risk was always on you. If the project was under-quoted, you're set up for failure (again, see above), while if it was over-quoted, you'll never see that money back (do you think a consultant who was forced to quote a flat-fee will invoice for less?)

While I've plugged it in other posts, the first few chapters of Head First Software Development from Oreilly gives a great overview of why you should do iterative software development and is very readable by anyone interested in software consulting, not just developers. If you are in charge of hiring or managing software consultants I'd very much recommend you spend a couple hours on it.

So, like I said, you don't want a fixed quote for that project. A fixed quote is heartbreak on a stack of paper. You want a great developer with an estimate, an hourly rate and a number of measurable milestones to take you to that happy place in the ether where software is delivered on budget and on time.

When developing a web app, your app's capabilities are far more important than it's features. Aren't they the same thing? Not in my mind: a capability is something you can do with the product while a feature is something a product can do.

Capabilities are awesome. Features for features sake suck.

The first iPhone was a hit because even though it lacked in the feature department (No 3G, MMS, etc) it let people do things with their phone they couldn't or wouldn't have before. Same with the first iPod. In what has to be one of the funniest-in-retrospect single line reviews of the first iPod, alpha-geek CmdrTaco, the creator of Slashdot wrote:

No wireless. Less space than a nomad. Lame.

He couldn't have been more wrong. We as geeks tend to focus on the features because we think they are important to us, but the truth is that we are users too; what we can achieve with something is orders of magnitude more important than what the product itself can do.

The good news is that there seems to be a strong trend in software development to focus less on features first and rather on user stories.

On of the nice things about user story driven design is that you say "A user should be able to do YYY" and then figure out the simplest way to make that happen.

Feature driven design might say "we should have a web 2.0 interface" and so you spend two weeks creating the perfect developer framework for whatever web 2.0 means - and then the minute you try to add in an additional capability into the site you realize your framework doesn't support what you want to do. It's true at launch time you will get to check the checkbox "Web 2.0" on the back of the metaphorical box but all you really ended up with is code that will need to be supported but isn't at all essential to your users.

Microsoft word is the starkest example of this I've seen. The hell that the Microsoft programmers must go through supporting all the features that no one uses must be a very dark place indeed. I need my word processor to have a very limited set of capabilities:

1. Let me coherently format and style text
2. Let me share documents with other people
3. Let me spell check my work so I don't look like an idot.
4. Let me print my work.

 

After that anything else that gets added in is a feature I probably won't use. OpenOffice and Google Docs are slowly supplanting Word because they make the user experience better than Word (I'm counting not having to shell out $500 for useless features as part of that user experience) The first version I used of OpenOffice was awesome because it had, right there in the File menu a "Export as PDF". This was a feature - but it made the sharing of documents so much easier that it greatly added to the "sharing" capability - I could email a document and be sure it would look that same way on the person on the other end of tube's computer.

The poster boys for the "less features" community are 37Signals' applications - BaseCamp, HighRise and Backpack in each case they dramatically underfeatured any of the other project management / CRM / intranet suites out there but had all the capabilities that people needed. They also added the capability of "usable by moms and/or CEO's" because their products were so simple and built-on standard web user interface concepts. That's a capability that you don't ever get by adding features - it's something you only get by having solid UX coherency and a lack of confusing and overwhelming features.

If capabilities are so great, then why do we as programmers tend to spend our times focused on features?

Reason one: features are fun. I love coding in new whiz-bang features and I'd bet other developers are the same way. It means I get to walk over to another developer on the project and say "Did you see the sweet new realtime Ajax notification panel I put in?" and they say "Cool" and I feel like I earned my nerd points for the day. Unfortunately if that notification panel doesn't do anything besides add in some eye-candy it's not a win. It's going to be a weight on the codebase that will need to be carried along without adding in real value.

Reason two: far to often project descriptions are in terms of features rather than user stories. Features are easier to describe, more concrete and more estimatable than user stories. This is something that's easily solved though by making sure every feature relates back to a user story. If each feature you're adding in relates back to a user story (e.g. "Sally B. User should be able to login even if she forgot her password" = "Password Reset Feature") then the creepy features won't get in there.

For a great analysis of user stories and how use them to drive your development process I recommend "Head first software Development" from the wildly entertaining O'Reilly "Head First" serious (yes I just called a set of computer books wildly entertaining - once you've read one of the series you'll understand) Despite the title, it's by no means a basic book on programming but actually a great read on the higher level processes involved in making great software.

Remember, focus on capabilities driven by user stories, letting the features generate from the stories and everything will be ok. By the way, did I show you the cool new totally user-driven HTML5 websockets event mash-up stuff I'm working on?

Until I published this blog post I didn't actually believe that twitter was going to destroy our country, unfortunately now I do.

If that last sentence doesn't make sense, let me explain by way of anecdote: I remember having a conversation with a former schoolmate a few years ago at a reunion that went something like this...

Me: Are you still into Dave Matthews as much as you used to be?
Him: Dave Matthews, Naw, I never liked them all that much.
Me: What - that's all you talked about senior year - you literally wouldn't shut up about them.
Him: I think you have me confused with someone else.

Now, assuming my increasingly disappointing memory hadn't failed me - I'm pretty sure he was the obnoxious, over the top DMB fan I remember. Of course a decade plus later, I can't be 100% sure it was him - it really might have been someone else.

If only we hadn't gone to high-school in the bad old days before everyone tweeted, facebook'd (or now buzz'd I guess) their every daily minutia and fleeting opinion, I could have nailed that kid to his over-the-top-pyschofan opinions. What would he have done then? We'll since the "deny it" option had flown the coop he would have been left with two choices:

a. Stay consistent with his opinion from years ago in high-school
b. Cop to his former obsession but admit he has moved on.

The obvious choice would be b - he clearly doesn't feel the same way about the band as he used to - but unfortunately we as human beings have a hard time being inconsistent. We find it repulsive. Hobgoblins notwithstanding, our hatred of inconsistency in our actions and beliefs is explained by a well known theory called Cognitive Dissonance. I've seen a friend stick to provably false "facts" they've written about on Facebook simply because they don't want to be seen as inconsistent.

"The person whose beliefs, words and deeds don't match is seen as confused, two-faced, even mentally ill. On the other side, a high degree of consistency is normally associated with personal and intellectual strength." (p.53, Influence, Cialdini, 2001)

If you can recall back to the 2004 election, one of the prime strikes against senator Kerry (ok, besides his wooden demeanor) was that he was a flip-flopper. This fact was contrasted with GW "stay-the-course" Bush. While in the abstract we might prefer our politicians to be able to take new evidence into consideration and change their minds, in reality we prefer them not to change their minds to often (Remember Doonesbury's reader-chosen characterization of Bill Clinton?) 

If I can get you to make a commitment (that is, to take a stand, to go on the record), I will have set the stage for your automatic and ill-considered consistency with that earlier commitment. (p.59, Influence, Cialdini, 2001)

What is microblogging but an easily-posted, bite-sized, on-the-record commitment? For many of us it's no longer enough to have opinions - we now need to scream them to the entire world. This effectively carves those originally transient opinions into stone and makes us less likely to consider evidence contrary to our stated positions on policy, politics, or whatever.

Since micro-blogging is so accessible - people now post just about anything from just about anywhere - it creates a particularly dangerous combination: a medium that encourages us to make a public commitment based on something we often have only experienced shallowly & fleetingly.  It's less important to a politician that your 100 followers know you "Stand with [Candidate]", than the fact that you made the commitment to post that in first place (and are now more susceptible to requests for money and help.)

By taking public positions too early (sometimes years early - e.g. "OMG [Politician Name Here] is the greatest [s]he'll be prez in 2012!!!!!" ) we deny ourselves the chance to take in more evidence before settling on a final position for possibly important decisions,  reducing what should be lively well-reasoned debates into dogmatic fear-mongering flinging of FUD to maintain our committed positions against any opposition.

To get back to the divide and destroy our country thing - clearly I wouldn't title a public blog post that way unless I wholly believed the thesis, so now I'm going to have to spend the rest of 2010 railing on the evils of microblogging. Oh well, should be fun. 

Think before your tweet because you will end up thinking what you tweet.

 

Some thoughts on property and the direction we're moving in the twenty-tens. It's not completely insane that sometime in the not to distant future a parent will hear their progeny ask a question like:

"Dad, what does it mean to buy something?"

and the following conversation might ensue:

Dad: "Well Billy, it's like licensing something, but you can do whatever you want with it"
Child: "But that's crazy, no company would ever let someone do WHATEVER they want with their product"
Dad: "Actually it used to happen all the time, you would buy something, and then you could do what you wanted - use it, disassemble it, loan it, even sell it again"
Child: "But isn't that completely anti-capitalist? It you could sell stuff you didn't need anymore then how would companies make money. Why would be people keep buying new stuff if the old stuff worked just as well?"
Dad: "I don't know it used to just work itself out...You know free market and all that"
Child: "Dad are you a Commie? Do I need to call the thought police?"

Ok, that last piece was a little over the top, but the rest isn't all that far from where we are now.

Anything that's delivered in a digital form is already licensed and not sold. Music, Movies, Video Games, Books.

Physical goods are moving in that direction too, helped by embedded microcontrollers and software. You can't take ownership of a computer without accepting a whole bunch of restrictions on what you can do, but it gets worse. Want to sell ink cartridges for someone elses printer? Get slapped with a DMCA lawsuit for reverse engineering their faux-DRM. Want to sell an unlicensed game on console? Same thing.

Intellectual Property rights are slowly encroching into all walks of life. Want to post a video of yourself dancing like an idiot to a pop song? Good luck.

Pretty soon baseball bats will come shrink-wrapped with End User License Agreements allowing them to sue you if you use their bat in an unauthorized manner.

Combine the licensing phenomen with the credit phenomenon, and suddenly we don't actually own anything. That house you bought? Technically the bank owns it, and you owe them more than it's worth. That car? Actually, you traded in your last car for less than it was worth to upgrade to this years model, so technically you own less than 0% of that too.

Ownership just isn't for the little people anymore. It's so 2009.

This never quite made it out of the backlog back in October, but I thought the implications of a case discussed on Volokh a couple of months ago were pretty staggering. To highlight the article's quote of the ruling (significantly chopped down):

The Fourth Amendment protects our homes from unreasonable searches and seizures, requiring that, absent special circumstances, the government obtain a search warrant based on probable cause before entering. This is strong privacy protection for homes and the items within them in the physical world.

When a person uses the Internet, however, the user’s actions are no longer in his or her physical home; in fact he or she is not truly acting in private space at all. The user is generally accessing the Internet with a network account and computer storage owned by an ISP like Comcast or NetZero. All materials stored online, whether they are e-mails or remotely stored documents, are physically stored on servers owned by an ISP. When we send an e-mail or instant message from the comfort of our own homes to a friend across town the message travels from our computer to computers owned by a third party, the ISP, before being delivered to the intended recipient. Thus, “private” information is actually being held by third-party private companies.

...[snip]...

Thus subscribers are, or should be, aware that their personal information and the contents of their online communications are accessible to the ISP and its employees and can be shared with the government under the appropriate circumstances. Much of the reluctance to apply traditional notions of third party disclosure to the e-mail context seems to stem from a fundamental misunderstanding of the lack of privacy we all have in our e-mails. Some people seem to think that they are as private as letters, phone calls, or journal entries. The blunt fact is, they are not.

This is a pretty significant statement - what it means is that if you transmit over a public line (read: the internet) anything that could be read by a third party, you shouldn't have an expectation of privacy about it. In essence, this is what the tinfoil-hats have been saying all along,  and the only solution is to encrypt-encrypt-encrypt. Why? Because if you encrypt your transmissions and storage then suddenly you do have an expectation of privacy.

If the government wanted to read an encrypted email that was encrypted against your own private key, then following the above logic - even if it was stored on a third party server - they would need to get a warrant and they couldn't just read what they wanted.

Now one of the problems here is that the data would need to be encrypted in a way only accessible to you along each step of it's journey: sending, transmission, reception, storage and retrieval, otherwise since the ISP could have logs of any of those steps, your expectation of privacy would fly back out the window. 

The good news - the securing transmission part is getting easier. While very few people outside of the enterprise use certificates to encode and sign their emails (unless they like wearing the aforementioned metal-headpieces), a good portion of the email being sent is starting to be transported via SSL.  What this means is that if you only leave the ISP the transmission part, and host your own email servers, then your expectation of privacy would hopefully return since you're not relying on a third-party for reception, storage and retrieval.

The bad news - until Google releases a GMail appliance, there's no way to use an ISP for other services and be protected. Now while the 4th amendment only applies to governmental search and seizure, this has further implications for private enterprise moving to SaaS models. If the court has stated that you never should have had any legal expectation of privacy about any of your data in the first place then you need to be very dilligent in reading your terms of service. Could you sue a SaaS provider if they released some valuable statistics about your data to a competitor? Most likely their ToS doesn't explicitly mention all meta-data and derived data, so I don't know, but until it gets decided in court, I wouldn't throw out all those servers and cloudify all your business needs just yet. 

Given the way computers have come to completely dominate our society over the past 30 years it's sometimes surprising when the digital fingerprints of living, breathing humans appear in places that they really shouldn't.

It's easy as a small web shop to imagine that everyone has by now completely integrated all their separate parts into one seamless, magical ESB - after all if we can do it with almost no budget, a larger company that can throw million of dollars at the problem should be able to do it as well. This is obviously an opinion that greatly discounts how difficult corporate inertia is too change - one client of a e-procurement system I worked on is just finally finishing integrating the system into their processes, 3 years after it was finished. But more than just corporate inertia, legacy systems and huge installed user bases make it extremely difficult to smoothly and completely integrate different pieces of a company's computer systems together - for example completely tying a website to a company's the back-end system.

We recently got back from a two-week vacation in Germany, and having been given some excellent advice from my father that international data roaming charges are exorbitant, I signed up for AT&T's "international data-roaming plan" for about $20/month which gets you 20 MB of emergency-email-checking and where-the-heck-are-we-GPS'ing on my phone while we traveled around the country. At least that's what I thought I did - when I got back I and opened my AT&T phone bill it said I owed them a whole lot money. That can't be right I thought, so I logged onto the website and verified that the international data-roaming plan was on the phone and read the fine print (expecting to see something like "except in southern Bavaria") - but no, everything looked hunky-dory.

I called up and talked to what turned out to be a amazingly friendly AT&T customer service representative that issued an apology immediately and said that the data roaming had been added the wrong phone - Martha and I share a family-plan for our business phones. I immediately assumed she had misspoken so I asked:

Me: Oh, so I added the data-roaming to the wrong phone?
AT&T Rep: No sir, we did.
Me: Wait, how is that possible, I did it on the website.
AT&T Rep: I'm not sure, it's billed correctly but someone here added it to the wrong phone.

They immediately issued a credit but the fact that something like this could happen speaks to a larger issue - AT&T most likely hasn't completely integrated the various parts of their wireless operation. I can only engage in wild, over-the-top speculation - but I'd guess that anything you do on their nice, modern website probably gets printed out on an old-school line printer fed with special printer paper, put in a small metal cylinder and then delivered via pneumatic tubes to the 9th floor, the "Account Modification and Liquidation Department" where a overworked, chain-smoking employee in a non-descript gray suit sitting at a small wooden desk manually enters the change request into a completely different system running on an old mainframe via a green phosphorescent terminal interface (In fact, in my head AT&T corporate looks and operates exactly like the office building in the Hudsucker Proxy )

As a small company, not having to deal with legacy systems and huge installed user bases is one of the reasons that we can do a lot more with a lot less. We don't have millions of users demanding certain existing functionality or hundreds of employees entrenched in their ways. We have the ability to be a lot more agile and a lot more competitive by working of a new technology picked specifically for the task rather than dealing with a system built on COBOL because it was the only language available at the time.

For another example let's take my name - I'm pretty sure it's Pascal Rettig - and I'm pretty sure that I usually spell it correctly. What is surprising however, is the amount of mail that shows up at our door addressed to "Pascal Retting." People really like to put in an extra "N" into my last name and there's nothing I can do to stop them.

It's enough of a problem that I always verify the spelling of my last name when I set up some service over the phone, but invariably a good portion of the time the first bill will show up and there it will be: "Pascal Retting."

What this means is that just like the AT&T snafu above - people are taking my information, entering it into a computer (unless their typing is just an IM to a friend), and then at some point along the line it's getting retyped in by someone who makes the inadvertant subconscious addition.

When National Grid acquired energy provider Keyspan in our area, the name on my account suddenly changed from Pascal Rettig to Pascal Retting (I can't be 100% sure that's exactly when it happened but it's there now). What that means to me is that during the acquisition, at some point, someone re-typed in my name, and by inference a couple of million other customers names into a new computer system. That is insane. It speaks to a corporate culture playing serious catchup in the digital age - a liability for them but an opportunity for small businesses not encumbered by the shakles of human error and inefficiency to make their mark. That being said, having a few million to throw at a project probably doesn't hurt.