The Seven Deadly Sins of Software Application Development – Part 2 of 2

by Lorne Cooper, CEO, AccuRev 

 

A Top 25 WordPress Blog Post of the Day

 

In Part 1, we looked at the sins of Building a Weak Team and Ignoring the Future.  Once you’re committed to hiring talent and building a forward-looking process, there are two more sins to avoid.

 

Sin #6: A Long Tail After Development is Complete

 

We constantly see organizations where the time from “requirements-in” to “feature freeze” is half the time to “solutions-out.”  That 50% “tail” on development provides no direct value to customers, but is there because of limitations in our development process.

 

Sometimes that tail is twice the length of the initial development. 

 

Ironically, while the long heavy tail is grown to protect the organization from delivering bad product, it eventually decreases response time until it becomes an instrument in the organization’s death.

 

But many industries have a long tail, don’t they?  Surely it doesn’t hurt the construction of bridges that the plans are completed long before the bridge is built.

 

For forty years people have tried to make analogies between software development and more mature engineering disciplines, say, like building buggy whips.  The designer of the buggy whip might make a sample or two to try it out on his children, then send it off to the factory, where a slew of Industrial Buggy Whip Engineers would spend months figuring out how to sew the damn things, buying up horse-hair, and hiring immigrants to sit at sewing machines. A year or so late, Pony Express delivers the new buggy whip to general stores all over the Grand Trunk Railway line. 

 

We can all certainly learn a lot from that.

 

Now try telling your customer that the problem she’s been working around is well understood, and in fact fixed, but she’s not going to get it for three months.  She’ll be looking for alternative suppliers before the screen door has hit your butt. 

 

Everything moves too fast to use that tired old process.  Frankly, I use to blame computers.  Now I blame the Internet. 

 

When you take a hard look at the time it takes to get completed product out the door, you’ll probably find it dominated by two things: testing, and fixing bugs that the testers find.  Both stem from the problem that you’re not doing enough testing during the development process, and you’re trying to test too much at one time.

 

I’m no fan of Hawthorn-effects, or what the politically incorrect might call gimmicks, like pair programming or stand-up meetings, but projects that use an Agile development with automated regression test processes are certainly doing more right than wrong. 

 

Is it really so hard to manage by tasks/issues, normalize them to a small unit of work, merge in just one set of changes into each build, and test just that perturbation?  What if you knew you would consistently get better code out the back of the process, and therefore make your organization more responsive to your customers?

 

Your development process must minimize latency, instead of trying to maximize bandwidth.  Don’t miss the changes in requirements and environments that produce opportunities for better products, or sooner or later you’ll end up locked in a suicide pact with a project no one wants.

 

Sin #7: Complacency

 

Calais’s stone walls were so thick the city had no fear of invaders, until they showed up with cannons.  Napoleon’s Old Guard was undefeated in battle after battle, until they were cut down by Wellington at Waterloo.  Big Blue’s OS/2 would crush little Microsoft’s Windows product.  Big Microsoft’s Live Search would crush little Google’s search engine. 

 

Hubris on ice will leave you with a heck of a hangover.

 

That brilliant team using that wonderful process you put together last year, is just a few months of complacency away from dysfunction.  People move and, thanks to the 13^th Amendment to the US Constitution, may leave their jobs and there’s little you can do about it.  Environments change and architectures become outdated.  The marketplace changes and requirements change with them.

 

One VP Engineering I know told me he asked his off-shore application development partner to give him 110%.  They did, but it turned out to be turnover.

 

Your process needs to be constantly monitored and upgraded, because everything in the process is changing.  If your offshore partner has high turnover, their part of the process might require more code inspection and knowledge transfer.  If your biggest customer can’t use your product until it supports their new environment, you need to put a special team on building a custom version of the product, and then merge that back into main development.  If you have to replace an old veteran with a rookie, then you might create new integration points and a more rigorous test protocol.

 

Recognize that everything will change, and the assumptions under which you built your last success are no longer valid.  Make small process changes early and often, or in the long run the big changes will be made by your replacement.

 

Stirring Conclusion

 

Managing a software development organization is a very challenging task, as tough as playing a violin with your toes, and less fun. 

 

I don’t mean to trivialize important management functions such as customer management, budget and project tracking, running a good meeting, and designing effective compensation structures. And by all means, hiring someone to spend their time doing all of that.

 

But that won’t protect you from the seven deadly sins.  Only you can do that.

 

I’ll finish with the Four Virtues, which are the antidote to the Seven Sins:

1. Hire great people or don’t hire.
2. Plan to build a product that will have a long life.
3. Keep your process nimble, and deliver less, faster.
4. Keep improving everything.

The Seven Deadly Sins of Software Application Development – Part 1 of 2

by Lorne Cooper, CEO, AccuRev

A Top 25 WordPress Blog Post of the Day 

Here at AccuRev our business requires we dive into the development process of dozens of software development organizations each month seeking our expertise. While reasons for success differ, there seems to be a pattern in ways organizations fail to deliver on their goals.

Here are seven pitfalls and tarballs to avoid when managing a software development organization.

Sin #1: Building a Weak Team

Great people build great products that generate great value. The rest of them fill up meeting rooms discussing whether they can get that bug fixed by Friday.

Sure, sooner or later someone will show you a great product developed by a committee of average engineers. Until then, do what you have to do to get great talent, and demand exceptional things from them.

Not sure if they’re good or great? Ask them to do the impossible: the good ones will tell you why they can’t do it, the bad ones won’t understand the problem, and the great ones will smile, and then ask for a little more time.

What kind of people answered (what is claimed to be) Sir Ernest Shackleton’s ad for his expedition to Antarctica?

Men Wanted: For hazardous journey. Small wages, bitter cold, long months of complete darkness, constant danger, safe return doubtful. Honour and recognition in case of success.

Sin #2: Blaming the Workers for their Bad Management

Is disaster a function of the people or the process? If great products come from great people, do failures signify you’ve hired too many chumps, and it’s time to play French Revolution with your development team? Can’t be the manager’s fault, after all, since the managers are the smart ones, or they wouldn’t have been made managers! Right?

Leadership and Management are the steering wheel and the accelerator. Development and QA are the engine. Why blame the engine for hitting the tree?

Can a project succeed despite poor management? Absolutely, and it happens all the time. Sometimes the needs are so dramatic, the architecture so obvious, and the team so experienced, that success is as certain as a Russian election.

But project failures are always the fault of management or leadership. If the engineers couldn’t do the job, good managers would swap them out, and good leaders would motivate them.

With added power comes added responsibility. Good leadership tolerates and anticipates engineering errors, but has little tolerance for management failings.

Sin #3: Short Term Thinking

If they paid you by how long it took to replace your software, you’d be a rich man, and could pay someone else to spend their time reading my blog.

When do you replace your underwear? When a new color comes out? When there’s a new innovation in cotton? If you’re an engineer, of any gender, you wait until that underwear is worn ragged.

And that’s the problem with software. It doesn’t wear out. In fact, it probably runs faster now than it did ten years ago when it was shipped. And it has more reports, more adapters, and more integrations than it did then, too. Ask any CIO: it’s hard to get rid of working software.

If you admitted your product was going to go through ten major revisions over eight years, and be used by 100,000 people, would you do anything differently? My guess is you would invest in a strong architecture, design code for re-use, and develop tons of automated tests. How about if it were to go through three revisions in two years and be used by 100 people? Same answer!

Investment in architecture is what determines whether your crack development team spends its time designing tomorrow’s skyscrapers, or on spreading spackle in the cracks in your old foundation.

Bad architecture is the answer to the question of “Where did all that money go?”

Good architecture is the answer to “How did someone so young get that job?”

Sin #4: No Investment in Code Re-Use

Good implementation recognizes that code’s going to be there for a long time, and what we build should be re-usable components. Ok, they don’t build bridges that way. But then again, the cost of building the second copy is a little higher.

If there’s one thing we can learn from the TQM experts, and even one might be optimistic, it’s that code reuse is the single most important thing you can do to improve your development organization.

If there’s one thing we can learn from experience (which might be even more ambitious, seeing how people drive in snow), it’s that Code Reuse is hard. Hard, and expensive.

Getting code that works properly is tough. It sure is nice to use something that already works. Tends to decrease the bug rate, increase development effectiveness, and best of all, gets the product to QA with fewer bugs.

Don’t forget that the amount of time it will take to get the product _out_ of QA is a function of the number of bugs it brought with it _into_ QA in the first place. An exponential function. Which, for those of you with Art History backgrounds, is a bad thing.

Code re-use is like changing the oil in your car. Slows you down, costs you money, but in the long run will save you a bundle.

Sin #5: Manual Processes

If you are going to have to do the same thing five hundred times, you’d certainly automate it. Same if you were going to have to do the same thing fifty times. How about five? With testing, the answer is yes, even if it takes ten times as long, as to do it once, you should automate it, because you’re very likely to be wrong and you will end up running the test twenty times.

This sin is pretty much the sin of “sloth” of which my grandmother used to accuse me every time she saw me pull off my shoes. Every development manager who has put on the stripes knows that good architecture code re-use, and automated testing is big. But each time we have short term project pressure, we find it hard to spend the extra time, and it’s not a little extra time; to invest in the infrastructure, streamline the functionality, expand the testing, and document the function, so we’re ready to meet tomorrow.

Part 2 conclusion

Team of One Pattern

We all know that AccuRev is well suited for large enterprises with teams of developers spread across the globe. But what about the crack team of one developer? You know, those of use who go solo because we know we can do it faster, better, and leaner than any contrived dream team.

Can AccuRev really work for small teams including a team of one? … You bet your ASCII it does!

In fact, I use AccuRev for my own personal projects. They happen to be AccuRev integrations, but are software projects nonetheless [Vim Plugin, GNU Bash, etc]. At this point, critics may proclaim, “For one developer, you just need to commit to trunk and label for each release.” Well… that worked OK for the first and second release, but then came the need to maintain multiple versions in parallel with patch releases (due to wholescale refactoring per major release) as well as compatibility between corresponding versions of AccuRev (since these are plugins). The compatibility matrix completely obliterates any suggestion of linear branching/labeling… so fuh-get-abot-it. Time to graduate from the traditional branch-based tools.

click to enlarge

The above picture shows my stream structure containing projects including vim-plugin and bash-completion. I’ll use the bash-completion project as a reference example to discuss my pattern of development. Even as a single developer, I found it critical to maintain a strict develop -> test -> release pattern simply because my development activities change day-to-day and typically turn into “It’s been a month since I looked at this code… time to roll up the sleeves and figure out what the heck I was thinking!” If I was forced to a single commit bucket (branch), I’d go nuts — trying to manage multiple patches, new development, updates to documentation, etc and then being forced to deliver it all because pulling out changes is about as fun as filling sandbags with a pitch fork… I’d rack my brains trying to keep it all straight especially since I have multiple projects going on concurrently!

My development process is as follows (annotated as steps 1-6 on the picture):

1. Develop — After working in my private workspace on a unit of work for days, weeks, months, I promote to the “-test “stream. Then, continue working in the private workspace on the next set of work.
2. Test — After unit testing and performing a clean-room functional test of all changes in my “-test” stream, I deem all changes “release ready” and promote to the top bash-completion stream.
3. Release Candidate — The changes in the base stream represent a configuration that is good-enough to start a new codeline. I do NOT snapshot an official release X.Y (just yet). I first create a “dash-x” line to start the codeline (e.g. bash-completion-3.x for the 3.0, 3.1, 3.2, etc versions).
4. Maintenance — In anticipation for even minor patch work, I proactively create a “-maint” stream to collect any upcoming maintenance changes based on the starting codeline. Initially, this stream will just be empty and identical in configuration to the parent ‘dash-x’ stream.
5. Official Release — At this point, I’ve immediately created the”dash-x” and “-maint” streams in succession so they are all identical in contents – namely, all containing the next release. So NOW I create my first official “dot oh” release (e.g. bash-completion-3.0).
6. Publish — With the official configuration under snapshot, I ‘pop’ the code, archive, and ship to my web server. La commedia e finite!

With the visual nature of the StreamBrowser and the ease of creating streams, managing multiple versions of multiple products with AccuRev is priceless. I use a simple, repeatable development pattern that lets me separate ongoing development work (workspaces) from upcoming changes being tested (-test stream) all separate from previous releases (dash-x) and patch development (-maint). And the best part about all of this is I can (and have, MANY times) come back to any project even months later and quickly ascertain the current state of the union — what’s in development, what’s in test, which releases are published, etc. Sweet.

Lastly, while I use this ‘dot oh’ pattern for my own projects, I even recommend it for large team development. It’s a great pattern and I hope you find it useful for your own stream management.

/happy releasing/ – dave

Three Absolute Requirements for Successful Offshore Application Development, Part 3

 

Requirement #3: Match the Project to the Team

You can put lipstick on a pig but it doesn’t do much good. It’s not much to look at, and it annoys the pig. You can assign any old set of tasks to the offshore team, but if they CAN’T succeed at the project, or DON’T CARE about succeeding at the project, you won’t get much value and you’ll have ended up annoying the team.

Agile methodologies have taught us all the benefits of keeping the customer tied into the development process. We know the offshore team will have less of a context for managing changes and usually will average less years of experience than the corporate team.

That doesn’t mean you can try to do the “thinking” for them. There are as many smart people in Moscow and Bangalore as there are in Silicon Valley, just fewer Starbucks.

There are two common failure modes in defining projects for the offshore team:

1. Asking a remote team to solve a complex problem the customer might not understand; or
2. Asking the remote team to grind through endless bug fixes to a stable product line.

In the first, it’s easy to see that the customer isn’t getting any happier with each release. In the second, the failure will be the Dutch Elm Disease of application development: developer turnover.

Developer turnover comes from multiple sources, many of which you don’t control. Given the heavy cost of recruiting and training new hires, it pays to stay on top of the factors you do influence. Everybody wants to be successful, and will stay on a successful project longer than on an unsuccessful one. So pick the project wisely.

Great projects for remote teams are ones that have very well defined deliverables but ambitious long-term goals. Projects like cloning capabilities of one system to another, changing platforms or improving performance of an app, all benefit from clarity of definition and sufficient technical challenge to keep the remote team productive and engaged.

Epilog

In my humble experience, meeting these three requirements eliminates the additional risk created by an offshore development group, and reduces the problem to the same set of risks as an onshore development group. Not that onshore development is easy.

Are three requirements really all it takes? Yes, because all three are meta-requirements that cover the most important challenges of offshoring. In fact, a superstar remote team lead (Requirement #1) can be enough in itself to deliver success. Superstars are so hard to find we can’t count on finding them. That’s why you need to maximize your success rate by meeting requirements 2 and 3 as well.

Bonne Chance, Bueno Suerte, Удачи, and Good luck!

AccuRev TimeSafe makes retrieving code from “anywhen” easy

One of the problems frequently faced in traditional branch-and-label version control systems is how to retrieve a particular configuration of code from the past if it wasn’t tagged or labeled at the time. You forgot to label the release-candidate-6 branch from last month before building it, and now you have no way to accurately pull the correct files.

AccuRev, through a combination of its TimeSafe architecture and the power of its streams, makes this easy. Say you have a development process where code follows a promotion model from individual contributors to an Integration area, to a QA area, to Production. The release candidates are based on the QA stream. The QA team built and released RC5 on Friday, June 7th at 11:00 pm, but unfortunately neglected to snapshot it. In the following weeks, additional changes were promoted to the QA stream.

To get guaranteed access to the code configuration as it existed for RC5, you could do the following:
* create a stream off of QA and assign it a time basis of June 7th, 11:00 pm
* run “accurev pop” to retrieve the contents to disk

It’s as easy as that! Furthermore, if you want to automate this process, how about creating a stream for all your source pulls, call it something like <depot>_source_pull (or equivalent, doesn’t really matter). Then, whenever you need source files, use the “accurev chstream” command to first move it to whatever parent stream you want the files from and second to change the time-basis on it to the correct time or transaction. Lastly, you pop from that time-basis stream to place the files in whatever location you want:

“accurev chstream -s <depot>_source_pull -b <depot>_target_stream”
“accurev chstream -s <depot>_source_pull -t 356”
“accurev pop -R -O -v <depot>_source_pull -L c:\build_dir .”

There are other ways to skin this cat in AccuRev, as there usually are, but this is a quick way to get guaranteed results.