How We Integrated our Offshore Development Team

by Tahir Hussein, Alterian

This diagram shows one key aspect of our development process i.e. 3^rd Line Support. It is critical to our operations because we support anywhere from the last 3 to 5 release steams. This includes monthly patch releases which address key customer defects and enhancements.

3rd Line Support

Prior to 2006 our development operation was UK based (in a single location). Everything was straight forward! However, when we opened up a development centre in Bangalore, that situation changed. Once we had recruited and trained the developers it was time to get them using our SCM tool, Serena Dimensions. This has served us well since 2000 but the performance of the connection to Bangalore meant that the PC and Web clients were totally unusable for shared development work e.g.

* It was taking over 5 minutes for them to browse change documents

* Editing the attributes e.g. to specify the fix times etc was taking around 10 minutes

* Making any kind of source code changes on a change document, especially one with a lot of additional related items or change documents was even longer.

* To fetch all 9000 items from the repository for a sandbox development and build was an overnight job (if it worked at all!)

Hence we had to put our thinking caps on and come up with another solution.

We tried making use of Subversion and associated tools and initially this looked very promising. However, when we started putting our Development processes on top of Subversion, in particular “Branching” and “Merging”, it became apparent that the underlying SVN functionality was not going to be mature enough for the tools to be available to cater for this. The solution we were looking at was Subversion running in UK and Bangalore, with the replication of data taken care of by a product called WANdisco and the Application Lifecycle management by Polarion. We had a number of meetings and discussions with all of the above parties, including the Subversion developers, who gave us an insight into the future plans. In the end the overall solution was not going to be elegant and satisfy all our needs. We carried on looking and eventually came across AccuRev.

At first we were reticent to go down this route as it was a proprietary system. However, when we were demoed the system by the AccuRev guys it seemed to fit in exactly with what we were looking for. Even with the addition of merge tracking in SVN, AccuRev is still head and shoulders above with its merge, change and namespace tracking.

We then obtained a demo license and spent a month evaluating the product against our list of requirements. The sort of checks we carried out were:

* How long does it take to create something e.g. stream, workspace in the UK and see it reflected in Bangalore and vice versa

* Can you create/delete hundreds or steams without affecting the system performance

* The critical one of can a number of developers on the UK and Bangalore work together on a set of changes in parallel and easily have those merged back together.

The whole thing ran so smoothly that we could no longer delay the decision to purchase. Last year around this time we purchased the required licences and then started the process of moving the development over to AccuRev.

So, where are we currently?

The intention has been to move over to using AccuRev and Jira combination (as Serena Dimensions covers both Issue Tracking and Source Control). However, as we have a limited number of resources available, and the fact that we require quite a lot of work to implement the appropriate workflow and associated build process, we have not had the time to do this. We are currently using AccuRev for ALL development and 3^rd Line support work but still use Dimensions for the Issue tracking and build process. Given that none of the developers in Bangalore or UK are impacted by this and can get on with their day-day work then I am cool with this. The only person affected is me as I have an additional step to perform by moving changes between AccuRev and Dimensions and vice-versa. I am confident that in the near future we will fully move onto the AccuRev/JIRA system and gain even more benefits.

I hope this will be of help to others and I would be more than happy to answer specific queries.

The Seven Deadly Sins of Software Application Development – Part 2 of 2

by Lorne Cooper, CEO, AccuRev 

 

A Top 25 WordPress Blog Post of the Day

 

In Part 1, we looked at the sins of Building a Weak Team and Ignoring the Future.  Once you’re committed to hiring talent and building a forward-looking process, there are two more sins to avoid.

 

Sin #6: A Long Tail After Development is Complete

 

We constantly see organizations where the time from “requirements-in” to “feature freeze” is half the time to “solutions-out.”  That 50% “tail” on development provides no direct value to customers, but is there because of limitations in our development process.

 

Sometimes that tail is twice the length of the initial development. 

 

Ironically, while the long heavy tail is grown to protect the organization from delivering bad product, it eventually decreases response time until it becomes an instrument in the organization’s death.

 

But many industries have a long tail, don’t they?  Surely it doesn’t hurt the construction of bridges that the plans are completed long before the bridge is built.

 

For forty years people have tried to make analogies between software development and more mature engineering disciplines, say, like building buggy whips.  The designer of the buggy whip might make a sample or two to try it out on his children, then send it off to the factory, where a slew of Industrial Buggy Whip Engineers would spend months figuring out how to sew the damn things, buying up horse-hair, and hiring immigrants to sit at sewing machines. A year or so late, Pony Express delivers the new buggy whip to general stores all over the Grand Trunk Railway line. 

 

We can all certainly learn a lot from that.

 

Now try telling your customer that the problem she’s been working around is well understood, and in fact fixed, but she’s not going to get it for three months.  She’ll be looking for alternative suppliers before the screen door has hit your butt. 

 

Everything moves too fast to use that tired old process.  Frankly, I use to blame computers.  Now I blame the Internet. 

 

When you take a hard look at the time it takes to get completed product out the door, you’ll probably find it dominated by two things: testing, and fixing bugs that the testers find.  Both stem from the problem that you’re not doing enough testing during the development process, and you’re trying to test too much at one time.

 

I’m no fan of Hawthorn-effects, or what the politically incorrect might call gimmicks, like pair programming or stand-up meetings, but projects that use an Agile development with automated regression test processes are certainly doing more right than wrong. 

 

Is it really so hard to manage by tasks/issues, normalize them to a small unit of work, merge in just one set of changes into each build, and test just that perturbation?  What if you knew you would consistently get better code out the back of the process, and therefore make your organization more responsive to your customers?

 

Your development process must minimize latency, instead of trying to maximize bandwidth.  Don’t miss the changes in requirements and environments that produce opportunities for better products, or sooner or later you’ll end up locked in a suicide pact with a project no one wants.

 

Sin #7: Complacency

 

Calais’s stone walls were so thick the city had no fear of invaders, until they showed up with cannons.  Napoleon’s Old Guard was undefeated in battle after battle, until they were cut down by Wellington at Waterloo.  Big Blue’s OS/2 would crush little Microsoft’s Windows product.  Big Microsoft’s Live Search would crush little Google’s search engine. 

 

Hubris on ice will leave you with a heck of a hangover.

 

That brilliant team using that wonderful process you put together last year, is just a few months of complacency away from dysfunction.  People move and, thanks to the 13^th Amendment to the US Constitution, may leave their jobs and there’s little you can do about it.  Environments change and architectures become outdated.  The marketplace changes and requirements change with them.

 

One VP Engineering I know told me he asked his off-shore application development partner to give him 110%.  They did, but it turned out to be turnover.

 

Your process needs to be constantly monitored and upgraded, because everything in the process is changing.  If your offshore partner has high turnover, their part of the process might require more code inspection and knowledge transfer.  If your biggest customer can’t use your product until it supports their new environment, you need to put a special team on building a custom version of the product, and then merge that back into main development.  If you have to replace an old veteran with a rookie, then you might create new integration points and a more rigorous test protocol.

 

Recognize that everything will change, and the assumptions under which you built your last success are no longer valid.  Make small process changes early and often, or in the long run the big changes will be made by your replacement.

 

Stirring Conclusion

 

Managing a software development organization is a very challenging task, as tough as playing a violin with your toes, and less fun. 

 

I don’t mean to trivialize important management functions such as customer management, budget and project tracking, running a good meeting, and designing effective compensation structures. And by all means, hiring someone to spend their time doing all of that.

 

But that won’t protect you from the seven deadly sins.  Only you can do that.

 

I’ll finish with the Four Virtues, which are the antidote to the Seven Sins:

1. Hire great people or don’t hire.
2. Plan to build a product that will have a long life.
3. Keep your process nimble, and deliver less, faster.
4. Keep improving everything.

Three Absolute Requirements for Successful Offshore Application Development, Part 3

 

Requirement #3: Match the Project to the Team

You can put lipstick on a pig but it doesn’t do much good. It’s not much to look at, and it annoys the pig. You can assign any old set of tasks to the offshore team, but if they CAN’T succeed at the project, or DON’T CARE about succeeding at the project, you won’t get much value and you’ll have ended up annoying the team.

Agile methodologies have taught us all the benefits of keeping the customer tied into the development process. We know the offshore team will have less of a context for managing changes and usually will average less years of experience than the corporate team.

That doesn’t mean you can try to do the “thinking” for them. There are as many smart people in Moscow and Bangalore as there are in Silicon Valley, just fewer Starbucks.

There are two common failure modes in defining projects for the offshore team:

1. Asking a remote team to solve a complex problem the customer might not understand; or
2. Asking the remote team to grind through endless bug fixes to a stable product line.

In the first, it’s easy to see that the customer isn’t getting any happier with each release. In the second, the failure will be the Dutch Elm Disease of application development: developer turnover.

Developer turnover comes from multiple sources, many of which you don’t control. Given the heavy cost of recruiting and training new hires, it pays to stay on top of the factors you do influence. Everybody wants to be successful, and will stay on a successful project longer than on an unsuccessful one. So pick the project wisely.

Great projects for remote teams are ones that have very well defined deliverables but ambitious long-term goals. Projects like cloning capabilities of one system to another, changing platforms or improving performance of an app, all benefit from clarity of definition and sufficient technical challenge to keep the remote team productive and engaged.

Epilog

In my humble experience, meeting these three requirements eliminates the additional risk created by an offshore development group, and reduces the problem to the same set of risks as an onshore development group. Not that onshore development is easy.

Are three requirements really all it takes? Yes, because all three are meta-requirements that cover the most important challenges of offshoring. In fact, a superstar remote team lead (Requirement #1) can be enough in itself to deliver success. Superstars are so hard to find we can’t count on finding them. That’s why you need to maximize your success rate by meeting requirements 2 and 3 as well.

Bonne Chance, Bueno Suerte, Удачи, and Good luck!