Welcome to Cykod. We are a fully-integrated, self-funded web-development startup located in Boston, MA.

Cykod Web Development and Consulting Blog

Great Takes Trying

I've never really considered myself to be much of a "superfan" - you know the type of people who are obsessed with one particular movie, actor, video game, or whatever, and then go and constantly relate everything that happens in their life to that obsession. Somehow my attention span never lasts quite that long. However as close as I've ever come was watching Zefrank's  "The Show "  - a vlog that ran (intentionally) exactly for 1 year starting on March 17 2006.

By the time I began watching, a few months after he started, each couple minute episode was a sparkling two-minute gem of humor, news and general goofiness that was a highlight of each workday. I remember sitting on the site hitting the refresh button waiting for that new episode to post.

Even while traveling and hungover, Ze would still manage to produce a nicely edited and humorous clip for the day:

Wanting more than just the daily episode I started watching backwards in time and came to a startling realization: the first few episodes of the show were, pretty "Eh". While they still were wacky and entertaining they were slightly stilted, sometimes slow and boring.

This absolutely boggled my mind - not because I was surprised that they early shows weren't that good - but because they became good so quickly. Ze is obviously an exceptional individual and performing for the camera came pretty naturally, but nonetheless, over the course of 1 month Ze went from a passable but sort of average video blog to a nicely edited, superbly paced, fall-over-funny nugget of Internet gold:

From (3/17/06):

To (4/20/06):

Better vocalization, better lighting, better facial expressions, better cuts - all around a much improved presentation.

I'd imagine there's plenty of people who do something similar - record a youtube video, do a screencast or write a blog - and then get discouraged because their first time around was barely passable. They look around at all the great stuff out there and think to themselves, why even bother? Well, the answer is that doing something great takes both trying it and spending some time trying to get better at it. You mileage may vary, but if it could really only take a month, isn't it worth giving it a shot?

Posted Wednesday, Sep 30 2009 08:55 AM by Pascal

Don't be a commodity

If you asked me whether I prefer to shop at a small, locally-owned and operated store or a large chain/box store - I would tell you that I prefer the small store, but the truth of the matter is that I shop at a number of chain stores all the time, places like: Home Depot, Costco, Stop & Shop, Dunkin Donuts, Target, Macy's.

Some of what's sold at those larger stores don't have alternatives in small locally run stores. Some do but are much pricier in smaller stores or don't have the same selection.

The interesting thing is that in my own experience sometimes we are willing to accept a limited selection or higher prices and sometimes we aren't.

Take food for example - by the end of the day we need to have put a certain number of calories into our body to make it to the next one. For any given meal, there's a lot of options for how we do that - All the way from McDonald's and Subway to Locke-Ober and Grill 23 & Bar - where we go depends a lot on what we're looking for. If we just want to put some food in our belly, we don't really care where we end up, we just want something decent, quick and cheap. If we're not just looking for food but for an "experience" the sky's the limit for how far we'll go for a bite for how much we're willing to pay. Are we really that much better off in a physical sense at the end of the day paying 20x as much for a couple thousand calories? Of course not, but there's obviously more to eating than just satisfying a physical need.

The same can be said for just about any other product or service. If all we're looking for is the end result - let's take a specific model of Refrigerator in this example - then what we will do is scour around for the best price and delivery options and use those as the guidelines for where to make our purchase. However, if we're looking for more than just a specific fridge - if instead you're looking for someone to help you decide which fridge to buy - suddenly where you go to buy your product matters because of the specific knowledge of the person you end up talking to. In our case when we went to Best Buy to ask a couple of questions about appliances we quickly realized that the person trying to help us (after we waited 10 minutes to successfully flag someone down) knew less than we did.

The contrast couldn't have been more visible when we later went to Yale Appliance and had someone help us decide what to buy. Yes - what we ended up buying was a commodity - but the process to get there highlighted how a company can add value in their approach to the business which people are willing to pay for.

Great businesses have the ability to turn commodities into experiences with the way they treat their customers. 

Great businesses have the ability to turn commodities into experiences with the way they treat their customers. These great businesses aren't always small locally run businesses - sometimes there are franchises that people love to go to because of the people that work there - but many times it seems that the type of people that are engaged and motivated in what they are doing end up at smaller businesses because of the more positive culture that seems to permeate the good ones. There's a company in the book Small Giants that is in the Records Storing business - CitiStorage Inc - that manages to do just that for a business sector that's almost the definition of a commodity service industry. Yes, they do have to compete somewhat on price, but what makes them great has more to do with their business atmosphere and methodology than the price tag that they offer.

All those larger stores that I go to: Target, Home Depot, Costco, etc sell commodities and nothing else. If I can find the same product somewhere else for less - they've lost me as customer. So their only options are to compete on price and try to lower their cost of doing business. This usually results in removing perks and employees that cost too much (usually the good ones) - further pushing the store into commodity-land.

The alternative is to embrace the things that make your operation less of a commodity - the differences in how you run your company and the employees that are great at their job (and who probably cost you a little more) - and suddenly you don't have to compete just on price. You're customers will come back to you because of the value that you add - the particular things that make your company unique.

If you're a Web Developer and all you're doing is trying to compete on price, there's a whole bunch of people overseas who are going to beat you at that game. However, if you are can differentiate yourself - be it by you reputation, service, responsiveness or skill set you'll be able to charge a lot more for what you do and be a lot better at it.

Posted Monday, Sep 28 2009 11:20 AM by Pascal | Consulting

Outrage in Waiting

It seems that outrage has become the new pastime here in America. Accuse first and do the research and ask questions later

Scribblenauts' is a game for the Nintendo DS that lets you write words and have the object appear on the screen which can be used to solve puzzles in creative ways. Did the game's develoepr 5th Cell really think they could let people use an estimated 20,000 words and not figure out a way to outrage themselves somehow?

Sure enough within a week after release a post appears: And yet "WASP" produces some sort of insect

Entering the word "Sambo" results in something, that from the blogger's perspective looks like a watermelon. And since "Sambo" is a racial slur (albeit one that I hadn't heard used in a long time), and watermelons are a stereotype, it must be a hidden piece of easter-egg-racism. (  Ignoring the fact that it could just be a "Sambo"  )

Cue the overreaction by a number of blogging sites (Kotaku , Consumerist , Joystiq ) the first of which, I believe posted their original story before hearing back from the developer - luckily, however, while the internet never forgets, it's easy to amend - and all the stories carry the developer's response.

Now, I don't actually believe that we as americans get outraged that often. Given all the stuff going on in the world, I would really be surprised if things like this actually generated an emotional response of outrage. I think we have become such attention hounds that people will write anything they think could get them the page views - my favorite is the fact that as bloggers we view ourselves as just "raising the question," not actually responsible for the result of our words or for a modicum of research to prevent the resulting cacophony. Just like the birthers are "raising the question" of whether president Obama is a US citizen or not. [ And in response, people are "raising the question" of whether or not Glenn Beck Raped and Murdered a young girl in 1990 - I personally don't think he did, but I haven't heard him refute it ]

Regardless of the reason, the bottom line is - even if you don't say or think anything racist / sexist / controversial on the internet - as long as there's one person out there looking to get angry or make a name for themselves, all they have to do is take what you've written, put a personal racist / sexist / classist interpretation on it and you'll end up spending the rest of the week trying to convince people you don't run around the office in white hats - 5th cell's need to resort to writing that they have a number of black employees, also known as the "black friend" defense was a little sad.

Posted Wednesday, Sep 23 2009 04:04 AM by Pascal | Troll

Fixed Quotes: The Consultant's White Whale

(A Prequel-ish post to Specs: The Consultant's MacGuffin )

One of the largest challenges you'll face as a consultant is in generating quotes for potential projects. Prospective clients will come to you and ask - How much will it cost to make X? And you will be expected to respond with a hard and fast number that you are willing stick to as the project progresses.

As most clients don't know exactly what they want until much later in the process, asking for a fixed quote is akin to asking a car dealer to commit to a price before you've decided on a sedan vs. a SUV.

In my experience, coming up with that perfect hard-and-fast number is a impossible proposition for you and a irrational expectation on the part of the client for the majority of projects - pretty much anything without perfect specifications that's not a carbon copy of a previous project that you've done. As most clients don't know exactly what they want until much later in the process, asking for a fixed quote is akin to asking a car dealer to commit to a price before you've decided on a sedan vs. a SUV.

As Venkat and Andy write about their book Practices of an Agile Developer in the section "Fixed Prices Are Broken Promises":

A software project is subject to all the simple mistakes [as building construction] plus fundamental changes to the requirements (no, not a shed, I want a skyscraper!), huge variability in individual and team performance (20X or more, depending on which studies you believe), and of course, the constant inrush of new technology (from now on, the nails are circular) (Practices of an Agile Developer, p.75) .

They suggest an alternative - Ask the client to pay for one iteration of an agile development cycle. start with a small set of features and a shorter development cycle to create a deliverable with most of those features that is useful in some way to the client. The client can then choose to go forward or not with the project but at most they are only out the cost of an initial iteration.

This is a great idea and probably a win for everyone involved - but in my experience it's a tough sell to a client unfamiliar to agile processes. If you need the project, sometimes your are going to have to come in at a fixed cost or the client will go with someone who does.

Your (potential) client is not a software developer. They don't understand the intricacies of what's involved nor do they understand that there are vastly varying levels of project quality depending on the developer (See: Software is a Craft ). I'm not saying that they should understand those things - it's not their business - but the bottom line is that your client will need to make a decision on who to use and will generally use some very simple metrics to choose who to go with. I've found they generally go by:

1. Were you recommended by someone they trust
2. What does your existing body of work look like
3. What is your price.

There's also a 4:

4. What are the details of your proposal

But that, I've found, is less important than they other three. This is why - when your company is just starting out and #1 and #2 are not working in your favor you need to compete on price. You will probably need to save the creative proposal Venkat and Andy describe for a little while down the road when you have a little more clout behind your endeavor.

So - back to that impossible quote. Before we get into the details, I think it's important to keep a definition of what a quote is in mind and here's mine:

Quote (n) - the amount of money you are willing to do a project for.

That's all it is. It's not a perfect calculation of the number of man-hours involved with precisely added-in expenses for overhead and coffee. You can pretend it is - but in reality, it's just the baseline number for which you are willing and able to complete the project - obviously including any fixed hardware, software, contractor costs added in.

Now, you do probably have an hourly rate that you've set either explicitly or implicitly - and you should work off of that rate when generating your quote because that's the only way to get a number that's in the correct ballpark. Going back to that definition above, however, you need to acknowledge to yourself that some projects are going to be more essential to your business than others and you need to have a little bit of flexibility in your numbers to give yourself a better chance at projects that you actually need and want. If you need the project to make ends meet and put food on the table, than it's perfectly reasonable to push your final number down a little bit to get a better shot at landing the project. Alternatively, if a project is only tangentially interesting and you're already booked pretty solid with work, put a bigger multiplier on your final number.

You are not beholden to your self-set hourly rate and the RFP process is a pure a form of the free market so there's no need to feel guilty adjusting your number up or down to whatever works for you. That being said, I do feel that once you are working with an existing client or on extending existing projects, there is a level of trust built up so that your client can count on some consistency in their costs.

Now however you do it, you will most likely fail miserably the first few times at connecting your estimated hours to reality, at least that's been my experience, and there's only 1 important thing to remember: once you get a project, track your time versus the numbers you came up with. If you do that consistently over a few projects you'll be able to develop some insight and start to converge your quotes with reality.

Now for how to come up with that imaginary number that is supposed to represent the Total Hours X Hourly rate. One solution is to scan over the request quickly, think about it for a couple of seconds, and then just pull a number out of thin air based on your gut feeling. I'd recommend against that, but it'll probably take a couple of times making that mistake as a new consultant before you are willing to commit yourself to some sort of process.

Here's what we do (and I'm sure the MBA-types will shake their head's disapprovingly at the lack of ISO-9000 certification on our processes ):

Write the quote out in a bullet point, line item form generating only as much detail as you need to figure out how long a piece will take to complete. (See: Spec's the Consultants MacGuffin for a little more on specs )

Now put a time number (either in hours or days depending on the size of the project) on each of those bullet points and add those up to get a base number . That's your: If I had all the details I needed and could develop in a sense-deprivation-chamber number. It's only bearing on reality is as a starting off point. You now need to take into consideration:

  • Time spend talking about the project
  • On-site meetings and travel
  • Small preference changes (green v. red, radio button vs. select list)
  • Features the client will assume are in there even if they didn't make them explicit
  • Bug Fixes, both during development and after
  • Larger included functionality changes
  • Training
  • Documentation (outside of the code itself)
  • Dealing with corporate hierarchy (waiting for approvals up and down the chain)
  • Anything else that will make you spend time working on that project

If you forget to take these into consideration you'll be unable to come up with a realistic quote and will instead just focus on development time - and if you're anything like me you'll view it as a challenge to your prowess (I've unfortunately said: "That's not that complicated - I could bang that code out in a couple hours" more than a couple of times and always regretted it.)

You also need to decide on a couple of important features that your project agreement will need to determine explicitly - will this project be done work for hire? i.e. who own the copyright on the developed code - you or the client - if you own the code it might be worth doing the project for less as you can reuse that code in the future, if not - then make sure you take that into consideration. Also, make clear how long are you responsible for bug fixes. If you don't make this explicit most clients will assume "forever" - and provided your prepared for that, that's OK. Make sure you and the client are clear on both of those issues - don't try to gloss over either ( and please don't take the previous paragraph as any sort of legal advice, talk to a lawyer and get yourself a legit development contract - See: Know your IP Basics for a little more on the above.)

All of these together need to be added into your quote in some way or another - either as explicit line items, or as a multiplier.

If you use a multiplier then the magic number is: 2.65. If you multiply your initial quote by 2.65 you'll come up with the perfect number to represent exactly how much time you will take on the project....that's complete BS of course, the multiplier is highly dependent on the initial estimator, the developers and the client and is by no means a magical, fixed number.

The multiplier will vary project to project depending on how much work the client has done to figure out what they want ahead of time and how involved in every small decision they need to be. It'll vary on how engaged you are in the project; how much the client will be involved in testing and a hundred other things. You'll have to figure out your own multiplier, but don't feel guilty using one (and by no means is "2" high - but, again, your mileage will vary)

I've generating quotes now for around 7 years and I still fail miserably at it from time to time. It's heartening to read something like "Fixed Prices are Broken Promises" to know that I'm not the only one. I try whenever possible to stay away from fixed quotes for larger projects, but fixed quotes are still the norm for the industry and until that changes we will have to keep pulling numbers out of the air to keep the work coming.

Posted Monday, Sep 21 2009 06:13 AM by Pascal | Consulting

Incomplete integration and the mystery "N"

Given the way computers have come to completely dominate our society over the past 30 years it's sometimes surprising when the digital fingerprints of living, breathing humans appear in places that they really shouldn't.

It's easy as a small web shop to imagine that everyone has by now completely integrated all their separate parts into one seamless, magical ESB - after all if we can do it with almost no budget, a larger company that can throw million of dollars at the problem should be able to do it as well. This is obviously an opinion that greatly discounts how difficult corporate inertia is too change - one client of a e-procurement system I worked on is just finally finishing integrating the system into their processes, 3 years after it was finished. But more than just corporate inertia, legacy systems and huge installed user bases make it extremely difficult to smoothly and completely integrate different pieces of a company's computer systems together - for example completely tying a website to a company's the back-end system.

We recently got back from a two-week vacation in Germany, and having been given some excellent advice from my father that international data roaming charges are exorbitant, I signed up for AT&T's "international data-roaming plan" for about $20/month which gets you 20 MB of emergency-email-checking and where-the-heck-are-we-GPS'ing on my phone while we traveled around the country. At least that's what I thought I did - when I got back I and opened my AT&T phone bill it said I owed them a whole lot money. That can't be right I thought, so I logged onto the website and verified that the international data-roaming plan was on the phone and read the fine print (expecting to see something like "except in southern Bavaria") - but no, everything looked hunky-dory.

I called up and talked to what turned out to be a amazingly friendly AT&T customer service representative that issued an apology immediately and said that the data roaming had been added the wrong phone - Martha and I share a family-plan for our business phones. I immediately assumed she had misspoken so I asked:

Me: Oh, so I added the data-roaming to the wrong phone?
AT&T Rep: No sir, we did.
Me: Wait, how is that possible, I did it on the website.
AT&T Rep: I'm not sure, it's billed correctly but someone here added it to the wrong phone.

They immediately issued a credit but the fact that something like this could happen speaks to a larger issue - AT&T most likely hasn't completely integrated the various parts of their wireless operation. I can only engage in wild, over-the-top speculation - but I'd guess that anything you do on their nice, modern website probably gets printed out on an old-school line printer fed with special printer paper, put in a small metal cylinder and then delivered via pneumatic tubes to the 9th floor, the "Account Modification and Liquidation Department" where a overworked, chain-smoking employee in a non-descript gray suit sitting at a small wooden desk manually enters the change request into a completely different system running on an old mainframe via a green phosphorescent terminal interface (In fact, in my head AT&T corporate looks and operates exactly like the office building in the Hudsucker Proxy )

As a small company, not having to deal with legacy systems and huge installed user bases is one of the reasons that we can do a lot more with a lot less. We don't have millions of users demanding certain existing functionality or hundreds of employees entrenched in their ways. We have the ability to be a lot more agile and a lot more competitive by working of a new technology picked specifically for the task rather than dealing with a system built on COBOL because it was the only language available at the time.

For another example let's take my name - I'm pretty sure it's Pascal Rettig - and I'm pretty sure that I usually spell it correctly. What is surprising however, is the amount of mail that shows up at our door addressed to "Pascal Retting." People really like to put in an extra "N" into my last name and there's nothing I can do to stop them.

It's enough of a problem that I always verify the spelling of my last name when I set up some service over the phone, but invariably a good portion of the time the first bill will show up and there it will be: "Pascal Retting."

What that means to me is that during the acquisition, at some point, someone re-typed in my name, and by inference a couple of million other customers names into a new computer system.  That is insane.

What this means is that just like the AT&T snafu above - people are taking my information, entering it into a computer (unless their typing is just an IM to a friend), and then at some point along the line it's getting retyped in by someone who makes the inadvertant subconscious addition.

When National Grid acquired energy provider Keyspan in our area, the name on my account suddenly changed from Pascal Rettig to Pascal Retting (I can't be 100% sure that's exactly when it happened but it's there now). What that means to me is that during the acquisition, at some point, someone re-typed in my name, and by inference a couple of million other customers names into a new computer system. That is insane. It speaks to a corporate culture playing serious catchup in the digital age - a liability for them but an opportunity for small businesses not encumbered by the shakles of human error and inefficiency to make their mark. That being said, having a few million to throw at a project probably doesn't hurt.

Posted Thursday, Sep 17 2009 10:55 AM by Pascal | Business

The Entitlement Culture

You worked hard, you did your homework on time, got straight A's, and graduated from college with honors. What do you deserve? What are you entitled to? Many students graduating this past spring think they are entitled to a job that pays them at least as well as their peers from a couple of years previously are being paid; they think they are entitled to flex time, to email and surf the web whenever they are at work; to call in sick with a hangover whenever the weekend stretches an extra night or two. They worked hard during college and believe that they should be entitled to a pay off for all that hard work. Some even take it one ridiculous step further (and are then rewarded by some idiots )

Unfortunately the truth of the matter is that the the hard work has just begun. It doesn't get easier once college is over, I've found real life is actually a whole lot harder than 4 years at what our european peers like to derogatively call "Country Clubs." In real life, no one gives you an A for doing something well, in fact half the time your not even sure if you did it well or not. That environment is difficult for some people (especially those graduating in down economic times) to deal with.

I can empathize with the shock of coming into the real world because I know the feeling - My wife and I both graduated in 2002 during the fallout from the dot-com bust. Neither of us had a job lined up after college because we arrogantly thought we had earned some time off and there would be plenty of jobs for two eager and engaged young professionals when we decided it was time to look. We couldn't have been more wrong. By the time we got around to it in the fall of 2002, the job landscape for both developers and graphic designers was pretty much destroyed.

Coming out of school we effectively got hit over the with the "You Suck" stick enough that we took a step back and realized: College doesn't really mean crap in the real world.

Coming out of school we effectively got hit over the with the "You Suck" stick enough that we took a step back and realized: college doesn't really mean crap in the real world. In good times, the name of the college on your diploma is enough to get an interview. In bad times, it might not even do that.

So what did we do? I worked for myself doing home IT (read: removing porn-related spyware from people's computers) and small business IT consulting (read: plugging in Ethernet cables, and, yup, still removing porn from the boss's computer) for a couple of years before starting to land development gigs off of web job boards and finally landing with a couple of larger jobs with clients that stuck. Martha stuck to her guns and temped for 6 months before finding someone who actually needed and wanted a junior graphic designer. Many design job postings at the time were getting 50-100 responses, with experienced out-of-work designers willing to take pay cuts to get a job.

Over the next couple of years, we slowly progressed upwards - I used to the freedom of working for myself to try to immerse myself as much as possible into web development - I figured the knowledge was very useful but temporary until I could get back to "real development" - and get up to speed on as many of technologies out there as possible. Eventually I progressed onto larger and more complicated systems while Martha steadily advanced in her trade and changed jobs a couple of times.

Two and half years ago Martha and I joined forces to create Cykod and we haven't looked back. We kicked and scratched our way up from small websites to larger sites with more and more features and are now moving to the next stage, from Web Consultants to Internet Startup, managing to stay small and independent. We just signed the lease on our first office space. To be completely honest, we're pretty happy with our situation, at the same time understanding that we're just one small notch up from the bottom on the life scale and still have a long we to go to be successful in the long term.

We also recognize that the past 7 years really don't mean jack for guarenteeing what's going to happen now. We're not entitled to any success just because we dug ourselves out of the first hole that we got ourselves into and we're more than likely going to fall right into the next one. That hard work we've done over the past couple of years doesn't mean we're going to end up with the next Facebook or Twitter. I'm am now optimistic that we're going to make it, but I understand that success is going to come as a struggle and I'm definitely not entitled to it just for showing up every day to that 8 o'clock class back in junior year.

I hate it when people look retrospectively back over hard times and say they wouldn't trade those difficult times for anything - so I won't lie and do that - I would have loved to have been successful right off the bat, but I will say that going through a little bit of hardship has made Martha and me a little less arrogant, and little more willing to consider other viewpoints and much more appreciative when good things come our way.

Posted Wednesday, Sep 16 2009 04:02 AM by Pascal | Troll

Simple, cheap temporary scaling: Using Amazon S3 to host a static copy of your site

Update 2010: Check out our new post on scaling Wordpress on Amazon's EC2

Recently we were tasked with the following problem: one of the websites we host was going to be featured prominently on a national television program. They are not a huge-traffic website, averaging probably a couple thousand hits a month and are hosted on a shared server. As we also haven't seen what the tv program was going to be and what sort of mention the website was going get, we essentially had no idea how much traffic to expect.

A couple of previous times, under normal circumstances the game plan would be something along the lines of: throw up a bunch of x-large amazon instances and move their site, which contains a number of static and dynamic elements onto the cloud. Unfortunately as this isn't something that we do very often so it's a somewhat painful manual process that takes a fair number of hours to do (and is not something I look forward to as it's a human-error-prone process) and we don't have stock EC2 images up-to-date with all the required software. So, by the time the sites are up and tested and load balanced there is a significant expense to the client - both in the cost of the instances and the consulting time to get them up and functioning properly. This client both didn't have the budget to do that and as we didn't know how much traffic was going to come in, we didn't want to waste their money if the rampaging horde of internet users never arrived.

So we can up with a solution that would be cheap but, for the most part at least, would be able to handle whatever traffic came their way - put the static pages of the site on Amazon S3 and run the website from there, but bring users back to the DB driven back-end for the pages that needed it (logins, user accounts, contributions, volunteering, etc) - all of which we were able to do with a just a couple of hours of time and a couple of quick and dirty scripts. Here's what we did, assuming a linux/apache2 hosting environment:

First the Preliminary steps:

Get yourself an s3 account and download the s3sync.rb scripts and get them configured properly. See the included README - it's pretty simple and mostly involves configuring ~/.s3conf/s3config.yml

Then, move all the file and image assets on the live site to S3 (how you do this depends on your setup - many CMS's have some S3 module or support, on a static site s3sync.rb and sed should do the trick) This isn't actually necessary for the rest of what's below but as the point is scaling to handle increased traffic it'll do it's part to reduce the load on the live server as well by greatly reducing the number of requests that even hit the server.

Now the meat of the process:

1. Download a static copy of your site. There's an export script built into our CMS Webiva but I believe using wget to mirror the site should get you there too:

$ wget -m http://sample.com

Note: Depending on how your site downloads - if the pages aren't named .html you'll have to modify the commands and scripts that follow - there's no reason S3 can't serve pages ending with a .php extension with a text/html mime type.

Note 2: This is more of a tutorial than a set of drop-in-place scripts - don't expect to follow these steps exactly as your own site may be set up differently. 

2. We need to replace any references to the root homepage with index.html - a couple of quick sed calls searching for '/' and "/" did the trick for us:

$ find . -name "*.html" -exec sed -i s/\'\\/\'/\'\\/index.html\'/g {} \;
$ find . -name "*.html" -exec sed -i s/\"\\/\"/\"\\/index.html\"/g {} \;


This is necessary because amazon s3 doesn't support index.html files. We can get around this issue for other pages on the site but not for the homepage.

3. Create an s3 bucket with s3cmd.rb with the full sub-domain that you will be hosting on (in our case example static.sample.com):

$ s3cmd.rb createbucket static.sample.com

4. Drop the static javascripts and stylesheets onto our s3 bucket - this applies for files that are hosted locally that your static site will need:

$ cd public/
$ s3sync.rb -v -p -r javascripts/ static.sample.com:javascripts/
$ s3sync.rb -v -p -r stylesheets/ static.sample.com:stylesheets/

5. Next I wrote a quick script (url_replace.rb) to replace links to specific pages that have forms or dynamic content to point all the way back to the real site. This will allow any relative links to something like the /contribute page to always go back to the live site. The script just searches for certain single - ' - or double - " - quoted relative urls and replaces them with full http://sitename.com/url urls. This worked well enough for us but depending on your site's content you might need to modify. This script needs to be called once for each page with dynamic content with the domain to redirect to and the relative url of the page.

Here's the script:

#!/usr/bin/env ruby
domain = ARGV[0]; url = ARGV[1]
url = url.gsub("/","\\\\\\/")

`find . -name \"*.html\" -exec sed -i s/\\'#{url}\\'/\\'http:\\\\/\\\\/#{domain}#{url}\\'/g {} \\;`
`find . -name \"*.html\" -exec sed -i s/\\\"#{url}\\\"/\\\"http:\\\\/\\\\/#{domain}#{url}\\\"/g {} \\;`

Now here's a sample usage:

$ ./url_replace.rb static.sample.com /email_signup
$ ./url_replace.rb static.sample.com /create_event
$ ./url_replace.rb static.sample.com /contribute
$ ./url_replace.rb static.sample.com /contact

6. Finally I wrote another quick script (s3put.rb) to copy all the exported html files over to S3. Since S3 doesn't support index files (in fact, s3 doesn't really support directories, everything is just a key) it needs to just give them directory-like names but make sure they get sent to the client as text/html files (this is why I couldn't just use s3sync.rb to do the heavy lifting)::

Here's the script:

#!/usr/bin/env ruby
bucket = ARGV[0]; file = ARGV[1..-1].join(' ')

# Files are coming in from a find command, so they are all prefixed with a ./ (i.e. ./dir/filename.html)

# Special case for index.html files - let them work as /directoryname and /directoryname/
puts("Handling: #{file}")
if file =~ /^\.\/(.+)\/index.html$/
  filebase = $1
  `s3cmd.rb put \"#{bucket}:#{filebase}\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`
  `s3cmd.rb put \"#{bucket}:#{filebase}/\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`
end

# copy the file normally
filepath = file[2..-1]
`s3cmd.rb put \"#{bucket}:#{filepath}\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`

And then a find command piped to xargs should do the trick (note the script above is counting on a "find ." command so all files paths coming in have a leading ./ you can adjust as necessary).

$ find . -name "*.html" -print | xargs -L 1 ./s3put.rb static.sample.com

7.  Now do s3sync for any other static files that need to be hosted (in my case, all static files were in the system/ directory)

$ s3sync.rb -v -p -r system/ static.sample.com:system/

8.  Now to take advantage of the fact that any sub-domain prefix to the s3.amazonaws.com domain is treated as a bucket name, we need to add in a quick CNAME to our site's DNS to get a nice looking static domain - in our case we want static.sample.com would serve the static site. Add something like the following to your BIND setup or configure your DNS with whatever you use:

static.sample.com CNAME static.sample.com.s3.amazonaws.com

Once that propogates, http://static.sample.com/index.html will take you to the homepage of the static site. 

You can test your site out on http://static.example.com/index.html and make sure everything is good to go. Any links to dynamic pages should take you back to the live site.

Once it's time to deploy the static site:

Add some rewrite rules to your the apache2 config of the live site to kick it over to static.sample.com on the appropriate pages. For example to kick the homepage and anything under /about, /news /blog, /partners you can use:

RewriteRule /$ http://static.sample.com/index.html [NC,QSA,L]
RewriteRule ^/(about|news|blog|partners)(.*)$ http://static.sample.com/$1$2 [NC,QSA,L]
...Any other rules you need...

And were done - the majority of users will see only the static site unless they to do something active on the site.

Update 9/20:  As someone pointed out, pushing from S3 to Cloudfront will give you much better latency and is a good idea for public-facing hosted files at the cost of being unable to modify your site instantly (if you discover a typo in some html for example). A good compromise would be to turn cloud front on in your bucket, and then do a sed replacementfor any static media files (leaving your .html files on s3). Since the url for the media files is much less important, leaving them on the deafult cloudfront domain name, as ugly as it is, won't really hurt.

Update 2010: Cloudfront now supports default objects - so you can actually skip the index.html hack and point your whole www.sample.com site at the cloudfront distribution, only pointing back to dynamic site as needed (under a site.sample.com CNAME for example) - just set your index.html page as the default object for the distribution.

Now what happens to as soon as the rush is over and people have linked to the static site - but the site contains old content? Not a problem, because we used the cname feature of s3 we can kick the s3 bucket to the curb and just add in another virtual host on your live server to redirect people to the real site:

<virtualhost>
  ServerName static.sample.com
  RewriteEngine On
  RewriteRule /(.*)$ http://sample.com/$1 [R=permanent,NC,QSA,L]
  ...
</virtualhost> 

Now point that cname back to you live local server and all the static links will redirect to the live site.

Overall, not a huge deal and we saved the client a bunch of money. Now we just have to hope too many people don't try to contribute all at once on the live server.

=====================

Followup - the TV spot was cut to a quick 5 minute spot without a mention of the website and we didn't even need the extra bandwidth of the static hosted site. Oh well, one more for the bag of tricks, I'm sure the experience will come in handy.

Posted Monday, Sep 14 2009 02:45 AM by Pascal | IT

No Comment

Comments in your code are bad. There - I went ahead and said it. If you need to comment the majority of your code you're doing something wrong.

Now, before I get attacked by an angry lynch mob of pitchfork wielding programmers who have just spent the last week digging through a uncommented pile of junk code let me put in a couple of caveats and go over a little history. First of all, I'm talking about code along the lines of something written for Web development - higher level stuff that's usually relatively straightforward (or should be). I'm not talking about device driver code in the Linux Kernel.

Now jump back 15 years ago - When CGI on the Web was just starting to hit it's stride, and there was a good chance that you were writing in C or C++, and going through the occasional loop construct:

 for(k=0;k<<len;k++) *ptr++ = (int) *ptr2++ * 100;

I've seen a lot of code written that way, and short variable names, pointer arithmetic and i,j,k iterators were pretty much par for the course.

These days (and in a different language), I'd write the above something like the following:

prices_in_cents = prices.map { |price| (price * 100).to_i )

Both of these snippets (on in C one in Ruby) do more or less the same thing. I would probably want a comment before the first one, but an extra comment in front of the second one just takes up space.

The newest round of languages have become much more expressive and thus can be much more compact, and so you can do a lot more in fewer lines of code and still do it more clearly. This and a better set of tools (e.g. autocomplete) has sort of made it ok for developers to start using longer and more expressive variable and method names - if you only need to type a variable a couple of times and your IDE types most of the word for you anyway, a 40 character method name is much_easier_to_understand (vs. _metu).

... I would really rather the developer spend whatever time they would have spent on the comments on improving the readability of or refactoring the actual code.

All this has gotten me to the point where most of the time, I would really rather the developer spend whatever time they would have spent on the comments on improving the readability of or refactoring the actual code.

When written with an eye towards correctly using the expressiveness of the language and keeping methods small and to the point, I like code better than comments, especially as many programmers don't write expressive high level comments but rather just end up writing some pseudo-code to comment their own code. For the C snippet above, I'd love to see a comment like:

/* Create a list of prices in cents (rather than dollars) */

but chances are, the comment will be something like:

/* Go over each element of ptr2, multiply it by 100 and put it in ptr */

That second one doesn't tell me anything the code doesn't - assuming you understand the programming language you're reading - so there's not a huge advantage to having that comment there anyway.

A couple more caveats - this is only for well-written code in a newer language that an average programmer could understand in a glance. If it's anything more complicated than that or you're writing closer to the metal in a lower level language - comment away, but ask yourself first if the code be made clearly with some extra work instead. The next person in is going to primarily be changing the code, not the comments.

Clarifying the code directly circumvents the other major issue with comments, that they can be out of date while the code never can be (or you have bigger issues). This doesn't happen that much, but comments about the specific blocks or lines of functionality are the ones that are most likely to be out of date as bug fixes and modifications are likely to be verified before a developer comes back and spends time to update the comment, and let's be honest, they (and you) will sometimes forget.

Last caveat - I'm talking about code comments here, not comments used to generate documentation (like RDoc or pydoc), those are fantastic ways to generate useful documentation for a project that are much more likely to stay up to date given their proximity to the code they are discussing. Also As these comments are generally at the method and class level, they force the developer to think about the abstraction created by the code as opposed to the code itself and so will tend to be more useful. That being said, I have also seen:

# send an order given and order id and a cart
def send_order(order_id,cart)

So your mileage may vary.

So there you have it - I'd rather see good, compact, refactored code without comments than one-and-done code and some quickly written comments. Not that controversial after all, right?

Update 9/14: Before I get skewered for saying something idiotically obvious (i.e. "Bad comments are bad; film at 11."  ) - this post was written in response to an angry email I got after delivering a project a couple months back, with full Rdoc,but, I was told a lack of comments in the code...So yes, many people subscribe to the philosophy of limiting code comments and focusing on refactoring, but at many levels (especially the corporate level), there is still an expectation of a large number of comments in your code.

Update 9/23: Of course,  "self-documenting" code can be done wrong as well

Posted Friday, Sep 11 2009 04:09 PM by Pascal | Development

Know when to punt: Deferring intelligently

Larger project generally have a large set features that need to be developed and large set of features that could benefit from existing functionality available in libraries and off the shelf components.

Large projects furthermore all suffer from one issue whether you acknowledge it or not: incomplete specs.

No matter how good you are at what developing specifications and software, if you're building something you've never built before, 99% of the time your not going to have a perfect design at the moment when development commences.

The Agile methodology is built on this concept, but even if you don't necessarily subscribe to "the whole agile thing" - experience in software development should give some truth to the above paragraph. If not, well you are either lying to yourself, were recently hit over the head with a large object, or you are just that good in which case you can stop reading as the rest doesn't apply to you.

For the rest of us, the reasons we have incomplete specs are pretty varied, here's some examples
- You never wrote complete specs to begin with
- The client doesn't know what they want
- Feedback from early prototypes has led to changes
- The client modifies the requirements
- Different 3rd party libraries have different features and you don't know which to use

And there's plenty more (e.g. the US economy crashes in the worst recession in a century and the client slices the budget in half)

Regardless, should any of these happen, the specs you have are effectively incomplete (even if you don't know it). What this means is that even if you could through all of Google's crack developer staff at your project, you couldn't finish it right away because there are still design decisions to be made.

What this means is that there are certain parts of the project whose development you are going to need to put off for the time being. This can be a problem if other parts of the project need to use that component or if you need to get a quick prototype up that uses features of that component.

... figure out a way to defer a making a hard decision on that component while still doing what needs to be done to keep the project going forward.

You have two options: One, even knowing that the specs for that component might change, go full steam ahead! and get something in there. Or two, figure out a way to defer a making a hard decision on that component while still doing what needs to be done to keep the project going forward.

What's wrong with 1? If you don't have all the information necessary to make the decision, you really shouldn't be making it if there's another way.

Let's say your project needs to incorporate video but you don't know what types yet. If you go ahead and integrate a 3rd party library that only supports quicktime, then you are going to be in a for some painful extraction process down the road when you need to support AVI and OGG down the road. If you went ahead and tightly coupled your code to 1 specific library, that extraction process is going to be even more painful.

You'd like to put off deciding on a video library but if video is a big part of your project, your boss will want to see some video in there sooner rather than later.

Here's where the Adapter Pattern comes into great use. Instead of coupling to a specific video library, ask yourself what do you need a video library to do, maybe all you need is:

Video#new(filename)
Video#stop()
Video#start()
Video#seek(offset_from_start)

Now lets say the video library you are looking at offers the following functions:

XYZVideo#new(read_type,base_bath,file_type,filename,option_A,option_B)
XYZVideo#play(offset)
XYZVideo#pause(remember_location=true)
XYZVideo#reset
XYZVideo#fastForward(time)
XYZVideo#rewind(time)
...And 50 other methods you don't actually need


Creating a wrapper class around the XYZVideo library will give you the simple interface that you need, but you'll be able to quickly integrate video into you project - and your project won't be peppered with implementation-specific details about accessing your video.

Even better, write some unit tests on your Video class, and then in two months when you invariably switch libraries, you'll already have tests written that will let you determine if you're new video library is working the same way as the old one.

Going back to the idea of deferring - if you really don't have the specs for video nailed down, you could even skip the library for now and just output a static frame instead of actually linking up to a real library. This will give the rest of the project time to fill out around the video requirements and any changes to you imagined video interface will be extremely easy to implement.

A couple months down the line - the other code in the project will have defined exactly the interface that you'll need for you video library and you'll be able to drop in the perfect 3rd party library that fits your need and budget.

Two caveats to this:

using an adapter pattern has some downsides. First, it's more project-specific code to maintain and secondly it's more code that gets run during execution so it can slow your project down some. Written correctly, the first problem is not a big deal as it reduces coupling between components and third party libraries, so the effect of a 3rd party library with different conventions and coding styles will be minimized and your code base should look more uniform for the most part. The performance issue is also not usually a big deal, except when you're using an adapter pattern at too fine grain a detail - if a co-worker comes in and says - What if we will want a different for(;;) loop implemention later in the project? Easiest solution: throw something (light) at them. Used correctly, most of the time the total performance hit is a couple of extra stack pushs and jumps, but performance issues are something to keep in mind.

Bottom line, if some parts of a project are unclear, put them off. It's better to put off writing code if you can when you don't know what you want to write. 

Second caveat, make sure you aren't overusing the Adapter pattern for the wrong reasons. If your project is called KSuperApplication, it's probably ok to use the KDE libraries as that's part of the point. Also you shouldn't really be using Adapter's to adapt your own code base to your own code base. If that's the case you probably have some bigger architectures issues you need to work out (this is sometimes necessary with legacy code)

Bottom line, if some parts of a project are unclear, put them off. It's better to put off writing code if you can when you don't know what you want to write. This isn't always possible, but sometimes you can get away with deferring decisions by focusing on the features you do know that you need and generating a stub or an adapter class to expose just those features and hide the rest of the details from the rest of the code.

 

Posted Friday, Sep 04 2009 05:49 AM by Pascal | Development

Know Where to Get It Right

With a lot of businesses, there are, depending on your type of business a number of ancillary aspects of your business that you need to get right otherwise people won't take you seriously.

That thought was made clear to me recently when I received a duplicate bill from our accountant recently. It came 4 months after we had already paid the first one and was for a different amount. This, coming from the people we are entrusting not to screw up our accounting, doesn't inspire a lot of confidence.

On the web, it's easy to put up a professional front even if you're nothing more than someone working out of a basement. Put up a nice website, get some nice-looking business cards and you're in business.

The converse, however, is also true - you can easily destroy your credibility by doing thing that a legit web shop shouldn't be doing.

Your website shouldn't say 'under construction' when a generic picture of a bulldozer. It should be a nice solid example of your work and not require that everyone view it 800x600 on IE6 (No one will think oh wow they've been in business for a while, instead they will just say, wow they look dated).

You also shouldn't use superwebdesign@gmail.com* as your contact email.

If you can't do elements of your core business correctly for yourself, how can your clients count on your being able to do them correctly for them?

Bottom line - even if you're swamped with work, make sure you put your our house in order and do the things that make it look like you actually know what you're doing

-----------

*I'm not picking on anyone in particular, that's not a real email as far as I know.

Posted Wednesday, Sep 02 2009 05:36 AM by Pascal | Consulting