Cykod Blog

Have you ever had a conversation with a fellow developer that went something like the following:

Q: Why is this so complicated?
A: Because I didn't have enough time to make it simple.

To someone outside of the field, it might seem odd, but to a programmer, it's pretty straightforward. Simplicity is hard to do the first time around. During development you need to make guesses about how your code and end product will be used well above and beyond what the requirements say. This leads to over-complication as without knowing the exact use cases you can't make add the right abstractions (and the wrong abstractions often do worse things to complexity than none at all.) Writing a blog post is the same way - editing and reworking tends to chop and simplify rather than add.

In the pre-DVD days when people actually recorded shows onto these huge VHS tape things (I know, crazy, right?) a common joke was how hard it was to correctly set a VCR to properly record a show taking place at a future date. Entire sitcom episodes were written around messing up the recording of some important show or event.

Technology was something to be afraid of, something that only the geeks and 7-year olds could figure out. For all intents it looked like we were heading towards a bureaucracy-led eventual meltdown of society as things got more and more complicated to the point where nothing worked anymore. Terry Gilliam captured this idea perfectly in his fantastically over-the-top movie Brazil. In the movie, one justification for the film's military state is a series of constant terrorist explosions, but while it's never outright explained, those explosions are arguably just caused by a society overtaken by bad duct-work. I couldn't find the intro clip with the malfunctioning apartment, so here's a younger De Niro as a rogue heating engineer:

It's understandable, however, that things have to get worse before they get better. Every advance forward is preceded by a minor step back as the kinks in a new technology get worked out. The first TV tuner card I bought in the mid 90's was a complete disaster - my computer didn't have enough power to run it correctly and so my dreams of coding while watching the game turned out to be a disaster of frustration. A cheap $30 portable TV would have gotten the job done 10 times better.

Instead I futzed around with getting drivers to work for Windows95 and tried to use a UI that should have been taken out back and shot. But because of idiots like me who kept buying these crappy cards and validating the market, people kept developing the technology to the point where someone came up with TiVo.

At that point not only could my parents record TV shows, but they could do it in without reading a manual. TiVo simplified the interface to the point where it was usable without needing to think about it. TiVo was a hundred times more powerful than programming a VCR yet it could be done without even glancing at the instructions.

A couple years earlier, Google launched it's search engine and overwhelmed people by underwhelming them. I still remember the first time I saw the Google home page with none of that "Portal" junk that was popular at the time, just search. It was an epiphany. Turns out that's what people wanted. Yes, Google eventually launched iGoogle, but it kept it off by default and that's the way it stays for most users.

When simplicity gets the job done, people like it and they will stick with it. Like like the saying goes: KISS.

Which brings us to Apple. When the iPod launched, it really only had one thing going for it: simplicity. A simple, elegant interface and a simple, elegant way to get music onto the it via iTunes. It lacked any compatibility with other software and has less storage than competitors, but since you could use it without having to "learn" anything it quickly won people over (I'm air-quoting learn because yes, you did learn to use it but it wasn't a struggle). Apple followed with the iPhone using the same formula of making things that were hard to figure out on other phones easy on the iPhone, and then took it to a whole new level with the iPad. Say what you will about the lack of features or restrictions (and I have), the most common review of the iPad by a member of the tech press goes something like this: "My [Dad/Mom/Husband/Wife/Kid] picked it up and just started using it and I couldn't get it back from them."

Companies had been trying and failing to make a "simple" computer for years (Anyone remember Microsoft's Bob?) However once the technology caught up to Steve Job's obsessive vision and delivered thin screens, SSD's, multi-touch and fast enough processors, it became became physically possible to build a simple reasonably-general-purpose computer that you could just pick up and use.

To get back to programming, the 90s poster children for complexity are C++ and Java. If C was the two steps forward of a previous era, a simple well-defined language that forms the backbone of GNU and L(unix), C++ was the step backwards as we hurtled ourselves forward into a new OOP phase. C++ was the promise of the great next thing but was hindered by the complexity of not knowing which parts were needed. I'm not implying C++ was at all a failure, just that it's high level of complexity was a result of treading into unknown waters, and so people coped. To quote Joshua Bloch from Peter Seibel's Coder's at work:

I think C++ was pushed well beyond its complexity threshold and yet there are a lot of people programming it. But what you do is you force people to subset it. So almost every shop that I know of that uses C++ says, “Yes, we’re using C++ but we’re not doing multiple-implementation inheritance and we’re not using operator overloading.” There are just a bunch of features that you’re not going to use because the complexity of the resulting code is too high. And I don't think it's good when you have to start doing that. You lose this programmer portability where everyone can read everyone else's code, which I think is such a good thing.

A next step in the same space to follow C++ was Java - and the primary thing that Java did was simplify coding and remove features from C++. Memory leaks? Gone with Garbage Collection. Multiple inheritance? No go, but you can have a simplified version called Interfaces. Operator overloading? Gone. People figured out the parts of OOP that were necessary to get the job done and dropped the rest.

Unfortunately Java focused on simplifying the language and forgot about simplifying the development eco-system (although they did nail deployment with build-once run anywhere) While the Java language is simplified, the steps to setting up and building it or any Java projects are anything but - and anyone who argues otherwise should be required to write an Ant build.xml from scratch.

What if we wanted something simpler than Java? What could be simpler than what might be the following "Hello World!" program (written in PHP):

        Hello World!

(As linked to by this Reddit post technically even without any <?php .. ?> tags in, it's still a valid .php file)

PHP filled the need for simplicity and quickly became the de-facto open-source web development language, but as things started to get complicated as project size grew, the same impulse to mitigate complexity with OOP pushed people towards other solutions. As Einstein's famous saying goes "make everything as simple as possible, but not simpler." Out of that need for more expressiveness but still keeping the simplicity came David Heinemeier Hansson's decision to try Ruby and create what would eventually be Rails.

Rails was simple because it was opinionated. It came with a set belief system about how you should go about your business. This was something that had never been expressed so openly by a programming framework, essentially: "we know we can't make everyone happy and keep it simple, so instead let's just make most people happy" (Caveat, I work on a Rails CMS - so I'm more than a little biased)

Simplicity and a strong opinion usually go hand in hand. When we know what we're doing, we usually are willing to make the hard, initially unpopular decisions for our users about what they don't need. When things get muddled, we have to go down multiple tracks as we don't know the destination.

So many different things have cropped up recently that seem just, well, simpler than what came before them that I'm starting to come to the conclusion that we must be getting better at this whole computer thing as we're finally comfortable with what we don't need in our software.

I once assumed that the graph of technological complexity was rising infinitely up and to the right, but now I think there's hope. Computers arrived and took over so quickly in the grand scheme of things that it just took us a few decades to figure out a way to simplify and still end up where we needed to be. That's not to say that everything is getting simpler right away, once a system is in production it's going to be installed and supported for a long time and unfortunately this generation's simplicity is necessarily built on the back of last generation's complexity, but I think we're headed in the right direction.

So here's to the Simplicity Era, may we stay simple and happy with our two steps forward until we find the next big step back to take.

[ As an aside, I've intentionally mostly ignored Microsoft in this discussion - from a programming standpoint they exist in a world adjacent to the one I've been paying attention to for the past 10 years, but I think their (now-waning) might made them somewhat oblivious to the forces of simplicity - and no, I don't think Visual Basic is simple, Basic or not. Now with Windows, I told them they needed to make it simpler, and so Windows 7, that was my idea. ]

Let me know your thoughts on any other technologies or trends that are getting the simplification bug, or feel free to burst my tiny, happy bubble and share your irrefutable evidence that we're headed into a Brazil-like complexity meltdown. Do you think we are entering a new Era of Simplicity or is it just a temporary trend, like drop shadows, rounded corners and Lady Gaga?

With any codebase there are set of conventions that should be followed For example, CONTANTS_SHOULD_BE_UPPER_CASE, functionsShouldBeCamelCase, _private_variables_should_be_underscored, spaces not tabs, etc. Making the decision early on as to what conventions to follow will give you a consistent codebase. The big benefit, though it's more than just aesthetic appeal. There are real, solid benefits to having consistent conventions in a project.

Reading, as a mental process is really hard for your eyes - the human eye can only focus on a very small portion of it's entire field of view to actually read the individual words in a line of text, however we are able to pick up a lot more information than just the small piece we're focusing on. By giving your entire codebase a constant structure, your giving you're brain a distinct advantage in understanding what's going on in the code in front of you. Similar to syntax highlighting (find me one programmer who isn't a fan) giving your overworked brain additional cues can lead to significant advantages in scanning over code to find bugs and make changes. Two of my favorite languages take this one step further. Ruby (and Rails specifically) pretty much defines one standard set of conventions to use through out your code - classes are UpperCase, methods_are_underscored, etc. Python makes indentation (one of the oldest visual cues programmer use to structure code) a feature of the language - ensuring a consistent usage across all python code.

The flip side is also true: code that has no consistent structure or conventions is much harder to grok at a higher level. One of the major differences between novices and masters, the ability to quickly scan and understand code, flies out the window if the code isn't structured consistently and readable. Being lazy about coding conventions does more than just look unprofessional, it makes it more difficult to understand what's going on and leads to more bugs in the codebase.

This doesn't mean you have to recode your entire codebase in hungarian notation and implement a 100 page style guide for every project - but make sure you pick conventions early on and stick with them. It's more than just an aesthetic advantage but will lead to better software in the long run.

I made the mistake yesterday of reading through the comments section on a Boston.com article. I should know better and I quickly regretted my decision.

The comment sections on websites these days suck either because they are completely devoid of any sort of intelligent thought (poster child: YouTube) or because people really enjoy expressing strong opinions tangential to the subject matter just to make themselves feel smarter (poster child: Slashdot)

For the former problem there's not much you can do - to have a good conversation you need people willing to write coherent sentences. For the latter problem, unfortunately, it's not a question of being able to write the comment, it's being willing to take the time to read the article and add something productive to the conversation.

Now, the problem may stem from us internet-friendly millennials who think we're terriffic on account of our extensive collection of after-school sports trophies and refrigeratored A+'s, and so are convinced that the world really, REALLY needs to hear what we have to say, regardless of what we're saying. Or maybe it's a universal issue and people just like griefing. It's tough to tell.

In any case, for our purposes people who read blog posts fall into three overlapping categories: people who want to write comments, people with something intelligent to say, and people who actually RTFA. Ideally, you'd like the comments on your blog to come from only the intersection of all three of those groups.

What often happens, however, is that you just end up with the difference of the first group minus the other two. People who have RTFA will scroll down to the comments, see all the comments that say "You are an idiot and should never have children" and decide to move on. The other group - people with something intelligent to say, will often make a great comment that adds nothing to the subject matter because they can't be bothered to RTFA to the end. Furthermore, Austrian goats are my favorite animals!

As a partial solution to this problem we've come up with a concept called the Tilder filter that works like this: somewhere in the blog post, preferably near the middle, there needs to exist a complete non sequitur. Something that's really out of place and will catch the attention of anyone who reads the post.

Next, as soon as someone tries to submit a comment the system sets a cookie via javascript (more on this in a second) and darkens the screen with a lightbox-like popover obscuring the entire page. On this popover is a multiple choice question with 8 or so answers which the user has twenty seconds to answer. The answer is of course the non sequitur mentioned earlier.

If they get it wrong - too bad. No comment for them. Of course they could remove their site cookies and try again but at that point they are going to be so angry that their comment will be easily discernible and moderated (e.g. "Your site sucks, I just lost my f#$@%ing comment you worthless pile of ..." and so on)

Now regarding the cookie, since we set a cookie the minute they press submit, the system can black out the page if they bring it up in another browser and disable the comments form after they failed to answer the question correctly the first time. This could all be done very simply in javascript as a proof of concept, while a real implementation would need some server side support.

I'd love to hear feedback on the idea, provided, of course, you've taken the time to read the full post.

It was the famous Jesse Schell video from DICE 2010 that finally convinced me that Game Mechanics on websites were here to stay. I had hoped once people got tired of mayoring-up and badges it would die down, but that video opened my eyes to the fact that it's here to stay.

Why wasn't I convinced before that? I think it might have something to do with the fact that currently implemented game mechanics don't particularly engage me. I love video games and play on Steam and Xbox but have never gotten into achievements, so I idiotically assumed that since I wasn't that interested most other people weren't either. Some recent conversations I've had have clearly indicated that's not true and that a significant portion of the population can be manipulated to change their behavior via achievements.

Two things someone said in particular amazed me:
1) People will buy crappy movie knock-off's game because they intentionally make achievements easier than normal games
2) People will pay to skip portions of the game to keep up with their friends (Farmville, WoW, etc) but don't think their friends do the same.
(Notice neither of these are actually "fun." In fact, they both constitute, in a loose moral sense "cheating")

So I now very much believe, as someone poignantly wrote, Game mechanics are the new black. We are already heading into the overhype stage. Just like "niche social networks" were the big thing a few years ago game mechanics are the new hotness. The idea of "Game Mechanics" has filtered down to the client level and clients are asking for them. However, just like the "niche social networks" few of the multitude of projects are going to bring real innovation to the space for a period of time. Why? They don't need to - clients are going to be asking for plain-jane points and badges because that's what they've seen work - so most developers aren't going to push back and try to innovate because that's not what client want.

Simple game mechanics are going to sprout up everywhere soon and it's going to get painful. If you think constant foursquare twitter updates are already a pain - extrapolate that exact same idea to every area of your life. To use the example from the Schell talk, want you get your teeth-brushing-points? You'll have to let your your wifi enabled toothbrush tweet every time you brush your teeth. "Just made my teeth extra clean with #colgate!"

At an abstract level, one of the reasons for the success of points, badges, etc is that they guide our efforts. They place a clear valuation on our time and say "You can do A or B or C, but B is worth 2x as much as A and C doesn't get you anywhere." They are an effective way in our mentally-exhausting media saturated lifestyles to cut through the noise to a clear quantitative signal and feel rewarded for our efforts with tangible results. Once they are everywhere, however, their apparent value is going to decrease as the noise increases.

The argument for implementing basic game mechanics is that whether everyone's overdoing them doesn't matter because they will help engagement, expansion and retention on my site. That's unfortunately not necessarily true and let me try to explain why:

We call these things game mechanics but they aren't really. They are meta-game mechanics. Mechanics that operate on a level outside the game. For example in a 3D FPS, the game mechanics would be the running around and the shooting of aliens in the head. Your score and achievements are meta-mechanics outside of the game. Shooting aliens is fun. Getting points for doing so isn't, in and of itself, at all fun.

In a similar fashion, points and badges on Web sites operate outside of the core feature of the website. You could argue that on a site like FourSquare things get a little muddled - but even there the "Game" is the tips and location aware check-ins - the badges or mayor-ships you get for doing so are a meta-mechanic.

(As an aside has anyone come across a meta-meta game mechanics website? A site that gives you points for other points that you get on other sites - I'm sure there's at least one out there already)

To get back to why a focus only on the meta-mechanics might be bad, let's take the example of MySpace and facebook. The Game Mechanic, aka the "meta" is the number of friends you have in your social network. For a lot of MySpace users that was all that there was to it - get as many friends on your account as possible (well, besides ugly wallpaper and writing idiotic shout-outs on people's walls.) Because that was the game, MySpace grew like crazy, but once people got tired of adding friends it was done [a gross over-simplification I know]. Facebook was much the same at the beginning, with a focus on adding friends, but then they innovated the hell out of the game. Making it actually social by pushing other user updates to your wall and adding third-party applications kept users engaged and Facebook growing like crazy.

So a game mechanic can work to your advantage, but only if you don't let it dominate the show.

Sachin Agarwall made a great point in his "Don't be a douche" Barcamp Boston presentation. The Slides are sort of hard to follow but here's the gist: companies shouldn't really care about Game Mechanics, they should care about Viral Mechanics. How the game mechanics promote viral expansion of the user base is what is actually important to the bottom line.  Just throwing a couple of points into the system amounts to Cargo Cult Game Design. As Agarwall pointed out, the issue is that at some point notching the viral mechanic up in favor of the company will make it annoying enough that users will turn it off or leave.

To look at it another way, the fact that the simple game mechanics most people are talking about put the focus on the meta means that when it's all said and done they actually take away from the core of the game. As Jeff Atwood wrote a while back: Meta is murder. Even though he was talking about online forums and discussions, I think he had it right.

The tagline from FourSquare is: "Foursquare on your phone gives you & your friends new ways of exploring your city. Earn points & unlock badges for discovering new things."

Yet I never hear people discussing the great things they found or discovered on FourSquare. I have seen and heard plenty about that second part. Go to http://foursquare.com/ and watch the "recent activity." What is the Tip to Badge/Mayor ratio you see? The times I looked I saw a lot fewer Tips than other stuff. FourSquare is more about the meta than the game. That's obviously working well currently, but I wouldn't count on that continuing if they don't provide real value. People are always quick to move on to the next flash in the pan.

To go back to the Atwood blog: "If you don't control it, meta-discussion, like weeds run amok in a garden, will choke out a substantial part of the normal, natural growth of a healthy community." And yet with Game Mechanics we are intentionally adding meta-elements into our systems and making the conversation revolve around them.

Now as businesses we like the meta-elements because they take advantage of people's strong innate desire for personal-validation and self-aggrandizement, but more than just muddying the waters, they also unfortunately bring some bad behavior into the mix.

To put it simply: if it's set up like a game, people will play (and cheat) to win.

When users are playing against themselves, you don't care that much if they cheat, as engagement is king on the web and if plays are taking the time to cheat you're actually probably ok with it as a site owner. When users are competitive with other users however, bad things can happen. I stay away from Digg these days because the game part of it has skewed to dramatically favor the all-powerful Digg superusers and that means that it no longer feels like a balanced, social environment.

Summary / TL;DR:

Game Mechanics - at least the simple ones that people are talking about like points and badges, aren't actually the mechanics of the game at all. They are the meta-mechanics that surround the game built to induce behavior inside the game. But there's no reason it has to be that way, and so where you should focus your efforts to avoid launching a dud as we roll through the game mechanics hype cycle is on making the core of whatever system you are building more game-like (put simply, fun) and not just toss a thin-veneer of simple game mechanics around the outside. Read the literature and blogs on the subject and brush up on what makes a game fun.

I love the Mega-man games because unlike some other games that just pat you on the back when you complete a level, in Mega-man you get a kick-ass new weapon. It adds something of value inside the game. FourSquare's push to have mayorships translate to perks in real life is a great piece of value getting added, but unfortunately, because it's an out-of-game perk that is also a scarce resource it encourages cheating that has negative effects (think FourSquare wants to play check-in cop?) My opinion would be to try to make sure your app is a "game" worth playing in-and-of itself before resorting to those types of tricks.

And in any case, you'll have plenty of chances to do in-real-life rewards next year when we roll through an augmented-reality overhype cycle.

[ Update 5/20: Case in point "Badge" support on HuffPost ]

One of the mistakes that I was (and probably still am - but being aware of the problem I think has helped) guilty of was the crime of trying to make one piece of code do too many things.

Like a vacuum cleaner with too many useless attachments that don't work correctly and keep getting lost, expanding on one function or piece of code by adding too many different conditional branches leads to a quick and ugly code death. At first glance, adding options might seem like vintage DRY - if you already have code that "almost" performs a certain function, why not add a couple of lines and a parameter or two to make it do a little bit more. In reality though, a hundred and one conditional branches are the quickest way to take an ugly-stick to your code. The next guy forced to look at the spaghetti that you've written will probably just end up quitting so that he doesn't have to maintain what you've written.

The better option, which is still DRY but also allows for separation of concerns is to extract the shared functionality separately and then invoke both your old function and a new function to get job done. To give a simplified example (extracted from some old PHP code I looked at recently), what's better:

function generate_table(&$data,$wrap_in_a_div = false) {
   $table_info = ... // Generate your table
   if($wrap_in_a_div} {
      return "<div>" . $table_info . "</div>";
   }
   else {
      return $table_info;
   }  
}

Or:

function generate_table(&$data) { 
    $table_info = ... // Generate your table
   return $table_info;   
 }     
function generate_table_in_div(&$data) {       
   return "<div>" . generate_table($data) . "</div>";   
} 

Ignoring the triviality of the example, for my money, the second one is liquid gold compared to the first one. There may be some perfectly valid need to have a table in a <div> tag sometimes and not sometimes, but the table generating code shouldn't worry about it because it's not it's job. If it turns out you need a whole bunch of junk occassionally wrapped in a <div>, you can down the road extract a generic wrap_in_div method and refactor generate_table_in_div to us that method.

When developing a web app, your app's capabilities are far more important than it's features. Aren't they the same thing? Not in my mind: a capability is something you can do with the product while a feature is something a product can do.

Capabilities are awesome. Features for features sake suck.

The first iPhone was a hit because even though it lacked in the feature department (No 3G, MMS, etc) it let people do things with their phone they couldn't or wouldn't have before. Same with the first iPod. In what has to be one of the funniest-in-retrospect single line reviews of the first iPod, alpha-geek CmdrTaco, the creator of Slashdot wrote:

No wireless. Less space than a nomad. Lame.

He couldn't have been more wrong. We as geeks tend to focus on the features because we think they are important to us, but the truth is that we are users too; what we can achieve with something is orders of magnitude more important than what the product itself can do.

The good news is that there seems to be a strong trend in software development to focus less on features first and rather on user stories.

On of the nice things about user story driven design is that you say "A user should be able to do YYY" and then figure out the simplest way to make that happen.

Feature driven design might say "we should have a web 2.0 interface" and so you spend two weeks creating the perfect developer framework for whatever web 2.0 means - and then the minute you try to add in an additional capability into the site you realize your framework doesn't support what you want to do. It's true at launch time you will get to check the checkbox "Web 2.0" on the back of the metaphorical box but all you really ended up with is code that will need to be supported but isn't at all essential to your users.

Microsoft word is the starkest example of this I've seen. The hell that the Microsoft programmers must go through supporting all the features that no one uses must be a very dark place indeed. I need my word processor to have a very limited set of capabilities:

1. Let me coherently format and style text
2. Let me share documents with other people
3. Let me spell check my work so I don't look like an idot.
4. Let me print my work.

After that anything else that gets added in is a feature I probably won't use. OpenOffice and Google Docs are slowly supplanting Word because they make the user experience better than Word (I'm counting not having to shell out $500 for useless features as part of that user experience) The first version I used of OpenOffice was awesome because it had, right there in the File menu a "Export as PDF". This was a feature - but it made the sharing of documents so much easier that it greatly added to the "sharing" capability - I could email a document and be sure it would look that same way on the person on the other end of tube's computer.

The poster boys for the "less features" community are 37Signals' applications - BaseCamp, HighRise and Backpack in each case they dramatically underfeatured any of the other project management / CRM / intranet suites out there but had all the capabilities that people needed. They also added the capability of "usable by moms and/or CEO's" because their products were so simple and built-on standard web user interface concepts. That's a capability that you don't ever get by adding features - it's something you only get by having solid UX coherency and a lack of confusing and overwhelming features.

If capabilities are so great, then why do we as programmers tend to spend our times focused on features?

Reason one: features are fun. I love coding in new whiz-bang features and I'd bet other developers are the same way. It means I get to walk over to another developer on the project and say "Did you see the sweet new realtime Ajax notification panel I put in?" and they say "Cool" and I feel like I earned my nerd points for the day. Unfortunately if that notification panel doesn't do anything besides add in some eye-candy it's not a win. It's going to be a weight on the codebase that will need to be carried along without adding in real value.

Reason two: far to often project descriptions are in terms of features rather than user stories. Features are easier to describe, more concrete and more estimatable than user stories. This is something that's easily solved though by making sure every feature relates back to a user story. If each feature you're adding in relates back to a user story (e.g. "Sally B. User should be able to login even if she forgot her password" = "Password Reset Feature") then the creepy features won't get in there.

For a great analysis of user stories and how use them to drive your development process I recommend "Head first software Development" from the wildly entertaining O'Reilly "Head First" serious (yes I just called a set of computer books wildly entertaining - once you've read one of the series you'll understand) Despite the title, it's by no means a basic book on programming but actually a great read on the higher level processes involved in making great software.

Remember, focus on capabilities driven by user stories, letting the features generate from the stories and everything will be ok. By the way, did I show you the cool new totally user-driven HTML5 websockets event mash-up stuff I'm working on?

Roll-your-own or Not-built-here syndrome is a mentality that rightly takes a fair amount of abuse. After all, with more than half a century of software development under our belt as a species do we really need yet another XYZ? Chances are it's been built before, and with a little bit of research you could find an existing workable alternative.

For a now-fairly-large ecosystem like Ruby/Rails, there's generally a plugin or a gem that gets close to what you need it to do, and with ruby's infinite flexbility, it's easy to make a couple of modifications and get what you need. Contrast `gem install XYZ` and `ri XYZ` with taking the time to really learn the problem domain of what your trying to solve, come up with a workable architecture and interface and implement.So we roll-our-own, throwing together some half-assed implementation that'll get the job done but will probably end being more trouble than it's worth.

It should be a no brainer - but often we take the latter path because we fear what we don't understand, and integrating a whole bunch of code, the pedigree of which isn't known is an understandly scary task. So we roll-our-own, throwing together some half-assed implementation that'll get the job done but will probably end being more trouble than it's worth.

That's not the time to RYO. If there's an available library that looks like it does the job, your time would be much better spent going through the code, online docs, tutorials and blog posts about it then jumping the gun and trying to write your own. However, don't necessarily just will-nilly throw the library into your project without giving it a solid once over, both at the API and at the code level.

We spend hours and hours coding, but when's the last time you took a couple of hours to read through the actual code of an auxiliary library that will perform an important function in your app? I'm getting better, but oftentimes the first time I look at the code is when something doesn't work right. It think that's a problem that should be fixed.

So, let's say you've looked at the library and it looks like a winner that's going to meet most of your needs. There's still now one more question that you need to ask yourself:

"Am I completely, overly and 100% happy with the interface that this library provides?"

You might be, but there are a number of example of circumstance where you could have a reason for not being completely thrilled. If the library provides more features than you need, you might not want your code to have to worry about overly complicated method calls. If the library provides too few features, and you need to call some other library's methods to get the output you need, you also have a reason for trepidation. Even something as basic as an unhappyness with the naming convention could cause some doubt - take fpdf for example - a 100% Ruby port of PHP's FPDF that keeps the same not-ruby-friendly name convention and a php method style.

Whatever it is, if you're not 100% happy with the interface or have some doubts about the library itself (is it being maintained, is it well written?), it's a great time to layer something between your code and the library. Technically it's called the Adapter pattern, but putting a layer of interfacing code is something that developers write all the time so you don't have feel like a pattern-snob.

Doing so actually a great way to figure out exactly what parts of the XYZ library you actually need and now much effort it is to make them work. If you're unit testing, you could even stub out calls to the adapter for the time being to make sure the interface you think you want to use makes sense.

(Brief interlude: I know what your thinking - "Who the $#@%@ is this guy, he titles his blog post 'When it's ok to RYO' and now is talking about the complete opposite - integrating libraries and the adapter pattern. What a jerk." That's ok, I've been called worse)

Just bear with me for one more second. Let's say you've got the XYZ library integrated behind a nice clean interface and your project is chugging along for a couple months. As we all know, change happens. Invariably something will pop up that make the XYZ library less of a perfect match than it used to be. Maybe the project is getting more advanced and you need to do more than the library was intended for. Maybe your realize that you don't need full XYZ support, just X+Z and the library is bloating your codebase for no reason.

Whatever the reason. Now's the time to RYO.

What's changed from way back when you originally need the functionality implemented in the XZY library? Well, three things:

1. You know more about the XYZ problem domain - being forced to examine different options and actually integrate the library forced you to learn about it.
2. You know a lot more about your application's needs for XYZ's functionality, not just from the requirements but from actual use.
3. You having a working implementation of XYZ and have an idea of what they've done right and what they haven't. You are effectively building the second version of something without have had to build the first one.

For me, all three of these are big wins. We've now done this a number of times inside of Webiva. The rails file_column plugin is a great example - it was easy to get installed and working, but a couple of years down the road we need both less and more functionality and it wasn't being maintained. Luckily, since the Webiva codebase didn't have file_column's everywhere but instead had abstracted them in the DomainFile class - changing out that plugin for some custom code ended up not being too big of a deal. Authorization is another example - we used a great authorization plugin that we determined did a lot more than we needed, and a couple years down the line were able to easily extract the core functions we needed and roll-our-own very simply with a nice clean syntax (the main class SiteAuthorizationEngine clocks in under 200 loc)

Now sometimes the right library just isn't out there and you don't have a choice, but for the other 95% of the time, it's worth taking a look at the library that fulfills 96% of your requirements and writing the other 4% than the other way around. After all, you can always come back and write those 96% later on and you'll have whole lot of a better idea of what you're doing. 

One of my favorite things about working in a higher level language is what I like to call "Low Ceremony Objects" (If there's another more popular term for them please let me know and excuse my ignorance)  - a.k.a. arbitrary data structures built off of combinations of general purpose containers like arrays and hashs. They are an effective way to quickly create and manipulate data that has a short lifespan but can be counter productive both in terms code readability and maintainability when over-used and sometimes more structured data or traditional objects are much more effective.

When used correctly - LCO's (Yup, tired of typing Low Ceremony Object already) Data structures generally exist for only brief chunks of time that are only defined insomuch as they are used. The existence of general purpose containers in higher level languages and the minimal amount of code needed to create and access them means that data that would otherwise sit in a predefined data structure now often ends up sitting in a combination of Hashes and Arrays ( or Lists and Dictionaries if you swing that way, or PHP's bastard stepchild of both)

As for the name - why low ceremony? Well, like a Vegas shotgun wedding, these generally don't come with a lot of planning -  no design documents or even a set structure - so there's generally not a lot of set guidelines involved. Now why Low Ceremony Objects and not Low Ceremony Data or Low Ceremony Structures? Because a large part of the value of using these objects built out of general purpose containers is the large and easy-to-use toolkit of methods either in the objects themselves (ruby, python, java) or the standard accessor functions (php, lisp) which aid greatly in manipulation of the data - adding, removing, searching, sorting etc.

Reading a book on Clojure got me thinking about this again (after struggling with it a couple of years ago - see the footnote) as in the Lisp variant I first learned - Scheme - pretty much every piece of data is a Low Ceremony Object that you can car, cdr, or caaaaadr to your hearts content, but without some additional structure or abstraction added on top of the language complex data quickly becomes difficult to work with.

When used correctly - Low Ceremony Objects are a great boon to development both in programmer productivity and in code cohesion and DRY philosophy - since they are defined instead of declared - their definition is always close in the code base to their usage. If you make a change to the creation of the LCO you have effectively changed it's declaration. You don't need to dig up a header file or separate class source file to make a modification. Want a quick data structure to hold a menu? Two lines and it's done:

[ { :title => 'Item 1', :url => '/item1' },
  { :title => 'Item 2', :url => '/item2', :selected => true } ]

If that menu is going to be created and digested during a portion of 1 Web request then you don't really want to go through the effort of creating a class, especially if that class is just going to be used as a data structure and isn't going to have any of it's own methods. What does the following actually get you:

class MenuItem
    attr_reader :title, :url

   def initialize(title,url)
      @title = title
      @url = url
   end    

end

class MenuItemList
 def addItem(item)
   @items ||= []
   @items << item
 end

 def item(idx)
   @items[idx]
 end
end

lst = MenuItemList.new()
lst.addItem(MenuItem.new('item1','/item1')
lst.addItem(MenuItem.new('item2','/item2')

Not a whole lot (Ignoring that no one in their right mind wouldn't just use an Array for MenuItemList unless some more functionality was added). Ruby provides the Struct construct for just this reason - but I'm not sure that using Struct gets you all that much more than just using a Hash. In particular I'm not a fan of passing in a huge parameter list to the constructor as you need to remember the exact order of your properties every time you read code using the initializer otherwise you'll have problems. For my money:

menu_item = { :title => 'Item 1', :url => '/item1',
   :selected => true, :dropdown => false, :green => true }

Is more readable than:

menu_item Struct::MenuItem.new("Item 1","/item1",true,false,true)

There are a lot of situations where LCO's are great, but there's two guidelines I now try to follow:

1. Only use a Low Ceremony Object if it is going to be created and consumed in relatively close proximity code-wise.
2. Once a LCO gets too complicated to understand easily by looking at it's definition or a debug output of the data, it's time move on to something else.

The reason for the first rule is that as soon as you are moved away from the definition of the object errors are going to creep in since there's no help from the interpreter or compiler in properly generating and consuming the LCO.

The second guideline should be pretty self evident - since you only have a definition of the data and not a declaration of the type, once the data gets too difficult to understand you are going to make mistakes using it because you don't have a declaration to fall back on.

LCO's can also be limiting because data can be hard to extend when a small piece of custom code could achieve the same effect. Let's go back to our menu item - what if we made the menu item responsible for displaying itself? Suddenly the whole menu system could be a lot more powerful by overloading the base class (or just duck-typing some other type in there):

class MenuItem
  ...Previous Definition...
  def display; "<li><a href='#{@url}'>#{@title}</a></li>"; end
end

class BlinkingMenuItem < MenuItem
  def display;  "<li><a href='#{@url}' style='text-decoration:blink;'>#{@title}</a></li>"; end
end

class MenuItemList
  ...Previous Definition...
  def display; "<ul>" + @items.map { |itm| itm.display }.join + "</ul>"; end
end

menu = MenuItemList.new
menu.addItem(MenuItem.new('Item1','/item1'))
menu.addItem(BlinkingMenuItem.new('Item2','/item2'))
print menu.display

Because who doesn't like blinking menu items? Achieving the same sort of functionality with just a data structure would mean adding in a conditional branch for each added option - to the point where your code can degenerate into if/elsif/else spaghetti.

Because of how easy LCO's are to create, they may tend to get overused when some additional design level decisions take more effort than just throwing together an Array of Hashes - don't lose all the benefits of years of work in OOP design just because high level languages make LCO's so easy to create and consume.

One of my least favorite examples of an LCO is the form system in the Drupal Content Management System - the way to create forms is to generate an enormous associative array where different pound-sign prefixed keys have different meaning and different nested arrays create different functional groups in the form.

This fails both of the LCO tests - most people generating drupal forms never look at the drupal code that actually uses them (I took a couple of looks and while it's nice, modular code, it's also very far away from what we're generating) - and since there's no strong typing it's impossible to know what went wrong when your form doesn't show up correctly (this may have been fixed with additional error checking in newer releases.)  Secondly with huge forms with dozens of items, it's hard to look at the data that you're generating and say with any certainty whether or not there's a mistake. Lastly, let's say I want to add a special widget to a form (like a slider for example) - I wouldn't even know where to start since the data that I'm passing into the form just gets magically transformed into HTML output on the other end, I don't have any control over it (other than just putting HTML directly into the form).

Because of the lack of meta-programming at the class level PHP code in general can suffer from a the downsides of lots of complicated LCO's, let's take the Cake PHP framework. From their tutorial, here's an example of a model:

class User extends AppModel {
   var $name = 'User';
   var $validate = array(
     'login' => 'alphaNumeric',
     'email' => 'email',
     'born' => 'date'
    );
 }

Compare this to an example in Rails:

class User < DomainModel
  table_name 'users'
  validates_as_alphanum :login
  validates_as_email :email
  validates_date :born
end

Wait you say - validates_as_alphanum, validates_as_email and validates_date don't exist in Rails - except that they do in the super class:

class DomainModel < ActiveRecord::Base
    def self.validates_as_alphanum(field,options={})
       validates_format_of field, options.merge({ :with => ..REG_EXP..})
    end
    ...
end   

The same thing is doable in CakePHP, but you end up adding instance methods instead of being able to meta program at the class level since the use of a data structure instead of a method forces the implementation to rely on conditional branching and dispatching instead of letting the language handle that part itself. The advantage of using code instead of a LCO in this case is that there's a lot more help from the language compiler/interpreter than when you try to create a Domain Specific Language solely out of data. You effectively need to write an interpreter for the DSL while on the other hand building it out of meta-constructs allows you to use the development language itself as the interpreter (in which case, you'll probably end up fullfilling some variation on Greenspun's Tenth Rule [via Proggit]).

So, in conclusion: LCO's are great where you need to quickly create data types to be consumed just as quickly, but can become a drag on a project when they get too complicated or are used as complicated interfaces as they don't self-document and can make it difficult to track down bugs when they become overly complex. Of course given the apparent rise of Schema-free databases (Like CouchDB ) and the NoSQL movement, I might soon be in the minority arguing against the limiting overuse of LCO's.
---------
Footnote:

As an aside, in the development of Webiva (Our newly-released, newly-open-sourced Rails CMS) we came across the problem that CMS's require a boatload of customizable features so that modules can be developed effectively and generally. This customization must be easy to add into the system from a developer perspective and easier to extend later on as more options are needed. The former cried out for the support and validation offered by ActiveRecord while the later made more sense using an LCO - after all who wants to update a bunch of models or the database every time a new option is added to the thousands of existing ones.

I started out using just Hash's as that's what comes in via the params object with some custom-inline validation. But that quickly became painful and repetitious, so finally what we ended up with  was the HashModel - a hybrid between using an LCO and using a full model that allows easy standard ActiveRecord usage in forms but is easy to create, update and use and store in generic DB text fields. Here's an made-up example usage:

class BlogDisplayOptions < HashModel
  attributes :blog_id => nil, :per_page => 20, :category => '', :active=>true

  integer_options :per_page
  boolean_options :active
 
  def blog
   @blog ||= Blog.find_by_id(self.blog_id)
  end   

  def validate
   # we need a valid blog
   self.errors.add(:blog_id,:invalid) unless self.blog
  end
end   

Usage - storing values as a hash in a serialized column:

@options = BlogDisplayOptions.new(params[:blog])
if @options.valid?
  paragraph.data = @options.to_hash
  paragraph.save
end     

Or:

@options = BlogDisplayOptions.new(paragraph.data)
if(@options.blog)
 ...Do something..
end

Switching from Hash's to HashModel's was a huge win in reusability and simplicity. Now I just need to fix all the other places in the system where I ignored the two rules from above.

Nothing. Done, shortest blog post ever.

Ok - but let's step back, I like BDD (Behavior Driven Development ) a lot and the benefits at the end of the day are definitely there to see, but I have noticed a shift in how I develop now that I'm focusing on BDD - while I used to take a very high level attack on problems, the use of BDD is somewhat subversively shifting me to a bottom up approach instead of top down. Instead of adding functionality horizontally on a whole bunch of different parts of project, I end up working vertically and locally on individual classes because it's easier to move forward from a testing perspective.

While that might seem normal or desired on larger shops (where developers are assigned smaller pieces)  and fully spec'd out projects - we usually count pretty heavily on the feedback loop that develops with clients by getting usable prototypes out as early as possible. Working vertically on individual pieces of a project instead of horizontally across the breadth of the project makes it harder to have a working prototype to show at any given time, so the shift that's happening because of BDD isn't necessarily beneficial.

Occasionally I need to make sure to remember to look for the forest through the trees instead of overworking certain interfaces. I end up adding more functionality than needed at a certain stage of the project because BDD makes it both incredibly easy to add additional functionality to interfaces that you are already testing and gives you a warm fuzzy feeling with each additional test that passes and will now proceed to sit in a state permanent watchfulness over the correctness of your code.

Testing of new Controllers, Models, etc classes (In Rails case) that require a fair amount of bootstrapping and tear down state around them - whether it be with Mocks or filters or whatever end up getting put off while others that are easier to test effectively get further along earlier in the process even if the classes themselves aren't that complicated.

There's no easier solution around this - tests are always going to add some additional friction in creating new classes that need to be unit tested separately. I've found that making note of the issue and stubbing out a bunch of tests across a couple different pieces of the project before writing any code - don't even write the actual test right away (In RSpec - the it "should .." without the actual test block acts as a nice stub) makes it easier from a mental standpoint to jump around to different classes and unit tests as the code gets developed.

Further I think there is real value in writing in some test abstractions in the form of helper methods even if you end up pulling your tests to a slightly higher level as long as what you're testing is still clear from test code. RSpec helps with this be making it easy to write additional matchers that you can then use very naturally throughout the rest of your code without a lot of ceremony.

In the process of updating Webiva to the newest version of Rails, on the obstacles we had to overcome was adding in support for CSRF protection throughout the system. This is an essential protection for any system but an absolute must for a system like an open-source CMS where anyone can study the code and make an educated guess about what users and content to attack (user id #1 for example)

Working with standard forms in Webiva code base proved to be not that difficult as the form_tag function automatically attaches the required CSRF token, however dealing with hand-coded Ajax calls in the prototype library and seemed like it was going to be a major pain, with the worst case scenario being manually attaching a authenticity_token= parameter to each request (there were probably a couple hundred of them). Luckily there was an easy workaround. We ended up just adding the following code to the top of each layout:

 <script> var AUTH_TOKEN = "<%= form_authenticity_token.to_s %>";</script>

And then just added the following to the bottom of application.js:

try {  if (!AUTH_TOKEN)   AUTH_TOKEN = 'DummyToken'; }
catch (e) {  AUTH_TOKEN = 'DummyToken'; }

Object.toQueryString = function (object) {
                 var result =  $H(object).toQueryString();
                 if (!result.include ("authenticity_token"))
                 {
                   result += "&authenticity_token=" + encodeURIComponent (AUTH_TOKEN);
                 }
                 return result;
               };

Since Prototype calls Object.toQueryString on any parameters passed to Ajax.Request or Ajax.Update the authenticity token should be added in automatically. For testing purposes I added in the try / catch block just to make sure something is set so that if AUTH_TOKEN isn't set we get a server error that can be tracked more easily than just a javascript error on the client side.