Pattern for Floating Labels with Streams

I’ve encountered a number of use cases in the field where people enjoy the use of  ‘floating’ labels.  Implemented correctly, floating labels are a good example of convention-over-configuration  usually eliminating repetitive, manual build system updates.   Implemented incorrectly, they can be a traceability curse.   In this writeup I’ll show two examples of how to implement a floating label pattern in AccuRev.

Deprecated Concept? If you’re used to using floating labels as ‘markers’ on a branch [in another CM system] for stable code or feature complete state, you’ll find that the AccuRev stream hierarchy and associated workflow is a much better solution – in some cases, eliminating the need for floating labels altogether.  In my experience, floating labels have been used because the traditional CM system didn’t support process – so you had to roll your own – ultimately leading to lots of scripting and a solution that doesn’t scale.

Stories of Misuse. Every now and then I hear stories about how an urgent quick fix or missing doc change -needed- to be included in the “release-4.0.1″ that just got labeled in source control a few days prior. In some traditional CM systems, you have the ability to commit new file changes, un-label the old version and re-label the new version. As easy as this sounds, this anti-pattern instantly breaks traceability [do your labels maintain history?] and can quickly lead to confusion as “release 4.0.1″ has multiple meanings depending on when the label was pulled, either before or after the re-label. Example. I once asked someone why they routinely re-label individual files and their answer was – “because creating a new full label takes about 30 minutes.”  Why so long? Because most CM systems label the individual files (or references) in a given configuration; the more files you have, the longer it will take to label the entire configuration.  I’ve even heard stories about creating labels (and branches!) that take upwards of an hour each run.  Unless you are using a time/txn-based CM system, labeling build events is required if you need to track and compare changes between builds over time.  If you are waiting 30-min per label, forget about the agile “10-minute” build – you haven’t even gotten to the compile step yet!

Valid Cases. In practice there are  cases where having a floating label makes sense, typically related to your build system.  Most cases have to do with marking transient build configurations.  For example, when doing nightly or continual integration builds, let the build script or CI server simply pull/poll a single label and let the developers or build engineer determine what content is bound to the label. This way, the build server can be “dumb” and just get whatever the developer or build engineer say it should have.  This eliminates the need to configure your build system with each-and-every new build label – especially painful for those build systems requiring manual edits of config xml files – ugh.

The Pattern.   Two examples are presented below showing how floating labels can be used to support continuous integration with streams.

Static Floating Label.  In this example, continuous builds are performed on the single stream App_Integration.  No need to change your CI configuration. New builds are kicked off whenever changes are  promoted [by team leads, etc] from the child project streams.

Static Floating Label

The CI/build tool is configured to monitor the single App_Integration stream.  As the configuration changes, the name of the build stream remains unchanged – hence, the App_Integration stream is the ‘floating label’.

When doing very-frequent builds (e.g. per-promote or even nightly) this pattern works well as long as you have the discipline to immediately fix any broken configurations.

Dynamic Floating Label.

In this example, individual build configurations for App_Integration are marked with snapshots.  New snapshots are created whenever changes are promoted [by team leads, etc] from child project  streams, or on schedule (e.g. Nightly).

Dynamic Floating Label

The stream Int_Good_Build is dynamically reparented to either the latest snapshot (e.g. latest build) or reparented to an alternate (e.g. prior) snapshot representing the last known good build.  The ability to move this stream, on-demand and as needed, represents a dynamic ‘floating label’ pattern.  Simply reparent the stream, update the attached workspace and the files on disk will mirror the current release snapshot configuration.  The same pattern can be applied to production releases where internal snapshots capture detailed per-configuration changes, but the reparented floating label can represent the latest public release.

Summary. In all, I’m a big fan of creating snapshots for each build, and removing old[er] ones along the way.  There are two camps on the subject, and this will be left for another blog.  Thus, I like the visibility of knowning which snapshots have been created (implying a purpose), and having a ‘marker’ stream as a floating label (e.g. context) to identify the current build being tested and/or delivered to an Int or Test machine.   Lastly, having recent snapshots immediately available assist with debugging why the latest build is broken – diff by snapshot.  sweet.

/Happy Coding/ dave

Persuading Your CFO to Buy in a Recession

If you are having trouble convincing your CFO to spend money on capital expenditures in this challenging environment, you are not alone.  Forecasters are projecting a significant decrease in capital spending for 2009, which is making buying new software and hardware very difficult to get approved by senior finance executives.

So how does a software development manager convince a CFO to spend money on new software and equipment?  CFO’s are seeking ways to increase the efficiency of how organizations deploy resources as well as how to control costs.  So focus on the Return on Investment (ROI), or net cost savings from the increased efficiencies, when you try to convince your CFO to approve a purchase.

The following is an example in which a company is determining whether to hire a new engineer or buy new software and hardware for the team whose productivity needs to increase 20%:

Assumptions

• Cost of software and hardware to be acquired – $50,000
• Size of team – 5 engineers
• Increased productivity from acquisition – 20%
• Average fully loaded cost of an engineer – $125,000

Cost Benefit or ROI

Based on the assumptions above, if the team were to increase its productivity by 20% without purchasing new software or hardware, they would need to add one engineer (20% x 5 engineers) at a fully loaded annual cost of $125,000.  Accordingly, if it is expected that the workload of the team is going to increase by 20%, it becomes an easy decision: spend $50,000 on new software and hardware instead of hiring a new engineer at a cost of $125,000, for a net cost savings of $75,000.

The typical CFO is also likely to question whether demands on the QA team will really increase and when the timing for the purchase of new equipment has to be made.  When this question comes up, and it will, you can now argue that if demand actually stays the same or decreases, you will be in a position where you can reduce headcount by 20% (a savings of $125,000) without losing any capacity.  This is the beauty of focusing on the efficiencies created from deploying new software and hardware

Summary

Engineering budgets are dominated by personnel costs.  If you focus on the efficiencies that will be created from deploying new software and hardware and tie it to either being able to cut headcount or not having to hire new staff, you will almost always be able to convince your CFO to open up the purse strings!

About The Author

Peter Dreifus was invited to contribute to our Distinguished Lecturer Series because of his experience as a CFO and COO managing finance and operations at software and technology companies.  Mr. Dreifus is a CPA with over 20 years experience and has held a variety of senior finance positions at Escher Group, Ltd.,  Sequence Design, Inc., Avery Dennison Corporation and the accounting, tax and consulting firm Deloitte.

Pattern for Versioning Generated Objects

After building your software, do you check-in your generated binary  files? How about the output from test runs? If your software runs on multiple platforms or your test runs take hours/days to execute, you may want to consider storing the output — especially if binary reproducibility is critical.

Example. Consider shipping an application to a customer who 2 years later reports a defect. Can you reproduce their build “today”? Surely you have the exact versions of source files. But are you using the exact build file? Probably. How about the original version of the compiler? Maybe. But probably not. Don’t forget that your compilers get upgraded too — their optimization algorithms or bugfixes can change the binary execution format of your application. Thus, compiling source from 2 years ago may result in an equally functioning application at the user-level, but at the byte-level, things may have changed dramatically — and at a level where runtime defects (performance/memory) rear their ugly heads.

Myth #1: Committing generated  files results in longer checkout times. No developer wants to checkout source code and wait for or be inundated with megabytes of .o, .class, .jar, .war files that they are either never used or are going to be rebuilt anyway.  The AccuRev Truth: Use include/exclude rules on streams and workspaces to control which streams have access to generated objects and who will receive them during checkout.

Myth #2: Committing binary files slows down your CM system. Traditional SCM systems combine both meta data and content resulting in slower performance over time as the number of files increase (think labeling).  The AccuRev Truth: AccuRev stores meta-data separate from file contents and uses indexes to lookup and retrieve contents.  For example, transactions are labeled not files.  Using a card catalog (index lookup) to find your books is always faster than walking the isles (linear scan).

Myth #3: Storing generated artifacts will bloat the repository. Back in the day of wild-west coding, there was little rhyme or reason for where files were saved in the source tree.  The build system would simply compile the files it found, save the generated output right next to the source file, and as long as everything linked & compiled — it worked.  But in todays complex world of multi-layer software architectures, tiered deployments, mixed technologies, and sophisticated build tools, following a convention is almost a necessity (think  ruby on rails, maven, etc). The AccuRev Truth: Organizing the top-level source tree and configuring your build tool can make it very easy to carve out source vs. binary vs. tests vs. scripts, etc.  Using include/exclude rules, end-users can decide at the stream or workspace level what parts of the file tree need to be visible.

The Pattern. In this pattern for versioning generated artifacts, I’ll show how streams can be used to store generated files only in the appropriate stage of development and prevent unwanted exposure to developers.  Two options are present that can also be used in combination.

Option #1: sub-configurations

Option #1: Store and track generated artifacts as sub-configurations isolated from the mainline.   From a baseline snapshot such as a test build or release candidate, create a new child stream to store the generated artifacts.  Then create a second snapshot that represents both source code and generated artifacts. For a single “configuration” you now have two snapshots – one for source only and a second for source + binary.  Furthermore, you can diff these two snapshots to know exactly how the binary configuration is different from the source configuration.  You might also consider storing compiler files, debugging output, test output,the compilers themselves (!), etc.

Option #2: include/exclude rules

Option #2: Store and track generated artifacts directly in mainline but exclude them from downstream access using stream-level exclude rules.   The top-most streams that need access to both source and binary will include the majority or entire filesystem footprint in their configurations.   The first stream that does not need access to generated objects will likely be the candidate to set an exclude rule on the folder(s) that contain those files.  The exclude rule is inherited to all children and grandchildren.

When using exclude rules, it is easiest to set a single rule on a top-level ‘./build’ or ‘./generated’ folder rather than creating a rule for each sub-folder in a large source tree.  Traditionally, make based build systems would generate the compiled files in-line with the source code.  Lately, ant based build systems would package all generated artifacts in a separate sub-tree off the root.  Regardless of your build tool, it’s best to have all generated artifacts in their own tree – it makes it easier to exclude as well as safer to clean!

In practice I see both patterns in use and both have equal merit depending simply on the situation at hand.  Option #1 is commonly used when generated artifacts are not to be included in the official release.  For example, transient or secondary artifacts such as test cases, debugging output, reports, etc.  These files are not promoted up to the release stream.  Option #2 is usually used when the generated artifacts are expected to be included in the official release snapshot.  Thus, they are promoted up through the test/build/release streams.  The build system for these types of ‘uber’ configurations may have multiple release targets creating different levels of release packages such as ‘minimal’, ‘app’ , ‘app-with-tests’ and ‘full’.  That is to say, the CM system may have all possible files but you can choose what actually gets deployed.  Ultimately, storing everything in the CM system may likely be the right choice for audit and reproducibility.

/Happy Coding/

Agile Software Development For No Apparent Reason

The always informative Brad Appleton has a post on Software Agility which defines some fascinating terminology right out of Yogi Maharishi’s mantra.

How can you not love a process that includes:

• Sense the Problem/Opportunity
• See the Problem in the Context of the “Whole”
• Socialize the Goals and Constraints
• Swarm the Solution
• Show the Results
• Share the Knowledge Learned
• Life Goes on Within You and Without You

[Ok, I admit I added the last one from The Beatles, not Brad Appleton.]

I don’t really know what “Swarm the Solution” means, or “See the Problem in the Context of the Whole”, and I’m the poor dumb bastard that actually read the post.

The biggest challenge with “Agility” or “Agile” is one you can learn from the taxi business: if you don’t know where they want to go, it’s hard to get the passengers to pay you for taking them there.

Having run an Agile (Scrum) process here for almost two years, and having almost 100 Agile customers, there’re three things you need to know about Agile development before you can get it right:

1. Figuring out what to do next is the hardest part.  Kudos to those organizations who have a single customer they need to satisfy, but most of us have to make sense of often contradictory feedback coming from many sources.  Agile doesn’t make this easier.  It may, however, make your mistakes more like scratching a parked car than bouncing off a road grader.  Beware the process that can’t trace it’s stories back to actual customer statements!  Beware!

2. Time boxing means you have to spend the energy testing that you always told your mother you were going to do, but never quite go around to it.  If you start getting all soft on those developers and letting them make excuses for why it can’t be verified in the time box, you’re going to be in Faux Agile land, an evil place where developers run amok, dates come and go without meaningful results, and customers wander off for lack of adult supervision. Joel Spolsky wrote about Netscape entering this land many years ago.  I read it in my Chrome browser.

3. Self-Managed Teams can save you a lot of money by increasing your management branching-factor.  But don’t let the inmates run the asylum: people still go through bad marriages, get mad at their co-workers, and generally screw up, just like they did before.  The larger the team, the more likely your average team member is to be no smarter than the beer drinking, C+ grading, problem set copying, frat brother who peed on your car last night.  Management still has to provide the whack on the head needed to keep the good ones motivated and the bad ones headed to someone else’s frat party.

“Agility” is about getting somewhere faster.  If your process doesn’t get you going the right way, your doing Agile for No Apparent Reason.

Lorne Cooper

Agile is a Step on the Path to Business Agility

Brad Appleton’s series of posts on What Is Agility? bring up the right questions, but they don’t dive into the muck, like a BBQ eater in a white suit.  Muck diving is right up my alley, as the stuff I find interesting is usually at the bottom of the pile.

There are several fundamental problems to be solved, but the most interesting, by which I mean really hard, problem is understanding what is the most important thing to do next.

In The Agility Cycle, Brad quotes a host of luminaries, including himself, in defining what that thing is.   I like Gartner’s the best, and here I’ll have to quote Brad, as Gartner won’t let me read their original:

1. Sense the need for change in the environment (includes the proactive initiation of change)
2. Strategize the available options and develop alternatives
3. Decide which path to take and commit to the approach
4. Communicate internally and externally to everyone who needs to know
5. Act to produce results and follow-through efficiently

1. Sensing the Need for Change in the Environment

Let’s start with “Sense the need for change in the environment”.  Beware of management aphorisms that sound like something you’d hear at a Yoga workshop!

Not only is it hard to tell what the need for change is, it’s as hard to agree on it as it is to get two people to agree to toppings on a pizza.  By the time your company has it’s second employee, you’re in trouble.  When you get your second customer, it’s pretty much over.

Practically speaking, there are four avenues in which we business people “Sense the need for change in the environment”:

1. Strategic Pressure.  “The future is in Cloud Computing on the i-Phone.  Get me there in six months.”  Most engineering organizations are willing to set fire to the old application and do a little resume polishing on the new one.  Nothing like starting from scratch, unless you actually want to make money.

2. Competitive Pressure.  “Your App needs to do foobar, just like that other app does.”  Now that’s going to happen every day after you ship version 0.9.  It’s unlikely that any one request requires real change, but maybe that sound you hear is an out of control semi, and it might be nice to change direction before slamming into the grill of the truck.

3. Customer Pressure.  “I’m not loading any more of your crappy releases until the .2 version, at least.”  Hey, you Windows XP users, you know who you are.  And who I am.

4. Financial Pressure.  Sales drop.  New Account sales drop.  The CFO takes away our CapEx budget.  We have to reduce headcount [or "Free people to pursue other opportunities" as they say in Hollywood], reduce hours, reduce pay.  Doesn’t take a highly developed set of Senses to know things have changed.  But what do we do about it?

It’s relatively easy to deal with Strategic Pressure: unless the pressure is coming from your CEO’s who’s an ex-Marine Karate instructor with a Meth habit: slow down amigo, and study it. Engineering is a discipline where things take time.  The worst thing we can do is change priorities faster than we can get them to market.  Change needs to require a good case.

Market of any size don’t have windows that  close in six months.  More likely people will learn that Prolog wasn’t really all that valuable, or you could like without a MS-Bob plug-in.  SOAs and the ASP business model have been around for ten years.  Before you imagine yourself as the next “i-Fart” author, ask yourself, what percentage of the apps on the i-Phone have made over $100K in revenue?

Competitive Pressure and Customer Pressure are not the same.  Competitive pressure shows up when your organization is losing business to a competitor.  Real Customer Pressure is when you can’t live up to your value proposition without making changes.  These two are the realm of Product Management.

The difference between a great product manager and all the rest is whether they can determine which customer or sales requests require change, versus the ones that go to triage and get prioritized with all the others.  Signals that you might have to really change what you’re doing are when you get demands for 10x performance improvements, or to support workflows that just don’t fit.  Patches (“now 30% faster!”) just aren’t going to make those requirements go away.  Now you’re going to have to think about how to address a new market environment.

Finally, Financial Pressure is the easiest to sense, and the least likely to activate change.  Financial Pressure tends to make organizations to freeze up.  Whether the reason was lack of product competitiveness, global recession, or a controller with a cocaine problem, the result is the same competitive and customer pressure, fewer resources to deal with it, and (unless you’ve been investing in those employee meditation session) lower morale.

Two mistakes managers typically make: trying to use the same development process, and freezing CapEx.

Improving the development process is your way out of this mess, not something you can’t afford to do.  That’s like the teenager who knows he’s going the wrong way, so he drives faster.  And, if you have the cash (or if the government is giving it out like candy on Halloween), Capital Expenditure is the cheapest way to get onto a better process.

Lorne Cooper

Security Primer – Anatomy of the AccuRev Admin Trigger

by Rob Mohr

Do you have the “Admin Trigger” installed and running in your AccuRev environment?  I hope so!

The “Admin Trigger” is the best way for you to restrict those non-ACE’d (AccuRev Certified Engineer) users from wreaking havoc on the process you’ve meticulously designed and implemented for your organization.  Make sure you lock it down!  It’s really simple to do!

Now, I’m not talking about taking flexibility away from your developers, they’ll need to control certain aspects of the process too.  It’s up to you where the line is drawn in the stream hierarchy and the Admin Trigger is the chalk.

The equator is commonly established on the integration stream.  Since the globe in this case is AccuRev, the line of demarcation runs north and south.  To the West (upstream from integration), ACE’rs set the rules on workflow, code promotion, stream configurations, and general access control.  To the East (downstream from integration), developers and development teams are free to make their own decisions to best support their process, products, projects, components, patches, bug fixing and development activities.

Have you read the private prototyping or stream-per-task blogs by Dave Thomas?  These are good examples of how dev teams and developers control how their activities are organized using streams while adhering to the overall enterprise software development life cycle (SDLC).

The Admin Trigger is a simple if-then-else perl script that fires on the server whenever certain commands are  executed.  Out-of-the-box, the script restricts “admin type” commands such as creating users, groups, depots, etc without needing additional customization.

 @cmdlist = qw/mkuser chref chdepot chslice lsacl addmember
                  rmmember mkgroup mkdepot mktrig rmtrig
                  setacl write_schema/;
    # is the user command in the above list?
    if (grep $_ eq $command, @cmdlist) {
    ...

The admin type commands are typically global in nature, meaning, that a single Admin group is granted permission for these commands.  Stream creation has a more granular scope allowing different groups to control their development process and stream management capabilities.

Simply list the streams in the trigger that only Admins have the ability to “manage.”  By default, other streams not listed are managed by the development teams themselves.  There are 2 sections in the trigger to set this up depending upon the commands to control.

Restricts: lock, unlock, chstream, incl, excl, incldo

    $admin_stream{"replace_with_admin_stream"} = 1;
    $admin_stream{"replace_with_admin_stream"} = 1;
    $admin_stream{"replace_with_admin_stream"} = 1;
    ...

Restricts: mkstream, mkws

    $basis_stream_deny{"replace_with_basis_stream_to_deny"} = 1;
    $basis_stream_deny{"replace_with_basis_stream_to_deny"} = 1;
    $basis_stream_deny{"replace_with_basis_stream_to_deny"} = 1;
    ...

Inside the trigger logic, each command is evaluated and will allow the operation to complete or not.

For example, the following section validates the “mkstream” command:

if ($command eq "mkstream") {
...
 # only a user listed as an administrator can create a new stream
 # based on an existing stream in the "basis_stream_deny" list
 if ( defined($basis_stream_deny{$stream2}) and `$::AccuRev ismember $principal "$admingrp"` == 0 ) {
   print TIO "Basing a new stream on existing stream '$stream2' disallowed:\n";
   print TIO "server_admin_trig: You are not in the $admingrp group.\n";
   close TIO;
   exit(1);
  }
}

There are other facilities in AccuRev to control the process and workflow too. Stream Locks grant users/groups the ability to promote to and from streams and Access Control Lists (ACLs) grant access to entire depots and subhierarchies.  Setting up these security measures combined with the Admin Trigger provide your organization with the flexible and granular security model it needs for the optimum development process.

Drop me a note and let me know the creative ways you’re using the Server Admin Trigger.

Pattern for Private Prototype Development

by Dave Thomas

Your team has been coding a feature for a few days, across dozens of files, and everyone is excited with progress.  Deep into development, you find an interesting 3rd party library that may simplify your work and possibly take the team’s feature to the next level.  But it will require adding new files, moving some directories, and refactoring some critical code.  However, you’re not ready to commit your current changes because they are unfinished and will surely break everyone else.  Your changes need to be saved before prototyping in case of a rollback but also be integrated with prototype development.  And what if the new library is a bust?  If you start co-mingling the library integration with unfinished code, the potential revert process will be a complete nightmare and waste of time.

Private Versioning. With AccuRev’s private workspace, you can always commit your changes early, often, and safely without sharing amongst your peers.  But in this case you have two logical development efforts, the 2nd effort depends on the first and both need commits to preserve evolving changes.   How do you keep them cleanly separated and continue to work on both in parallel?

Stream Inheritance.  AccuRev streams have an intriguing and very powerful feature called inheritance.   Similar to how an OO subclass implicitly inherits methods from parent and grandparent classes, a child stream implicitly inherits versions of files & directories from parent and grandparent streams.   Taking advantage of inheritance, we can use streams to independently manage logical changes and cleanly maintain change dependencies without physically co-mingling files.

The Pattern.  Prototyping changes without co-mingling files can be done by simply creating a series of ‘personal’ or ‘private’ development streams (though, they are just regular dynamic streams). This pattern will create a “Feature”, “MyDevelopment”, and “MyPrototype” stream sub-hierarchy.  See Picture.

click to enlarge

From your Integration stream (or equivalent), start by creating a “Feature” stream that will collect all changes from your entire team.  [See related blog, Stream-per-Task Pattern].  The majority of team members may have their workspaces directly from here.  Next, create a child stream called “MyDevelopment” that will track your personal ongoing development activity.  Finally, create a grandchild stream called “MyPrototype” to track changes that will be discarded or retained depending on the level of success.

The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and private development.

The “Feature” stream is the collection point of all in-progress development activities for the given feature from the entire team.  Developers will promote here frequently possibly kicking off an automated build with success/fail notification.   The “MyDevelopment” stream provides a collection point for all of your personal development changes.  This stream may be considered a “private” development stream simply because no other developers will likely use it – set a stream lock to be sure.  A lightweight Continuous Integration build (i.e. compilation only) may be performed on “MyDevelopment” for sanity sake or just compile and promote as a practice.  The “MyPrototype” stream is a collection point for all prototype changes. Even as new changes are promoted to “Feature” and “MyDevelopment”, the “MyPrototype” stream will automatically incorporate those changes (via inheritance) and the prototype developers will merge changes as necessary. The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and personal development. Also, by using a stream for prototype work, multiple developers can contribute and collaborate. If the prototyping work is deemed successful, the files can be promoted to “MyDevelopment”. If the prototyping efforts don’t work out – no problem – just remove the “MyPrototype” stream and re-purpose the workspaces.

The beauty of this pattern is that it isolates development activity by purpose without co-mingling physical file changes.   It lets the prototype developers go nuts and shoot from the hip while the regular feature developers (with the deadline!) work unimpeded and without fear of rampant changes — and everyone stays up-to-date.   And with no limit to stream depth, teams can perform prototyping efforts in parallel and/or perform prototyping based on existing prototypes by adding another child stream!  Furthermore, the pattern works for any size development activity or team, even for us Team-of-One developers with tons of ideas, fast fingers, and a few green screen x-terms (2 space, 80-char wrapping of course – <chuckle>).

This is a perfect example of how AccuRev empowers the developer to take control of their own development.  Creating streams is extremely easy and the stream browser provides unprecedented visibility into the entire development process.  With the right amount of security in place (Locks, ACLs, Triggers), the critical release streams (left side of the stream structure) can be locked down by the CM Admins, but the development related streams (right side) can be fair game for developers to create an environment that suits their purpose, such as prototyping.

Does anyone have a good story to tell about how this pattern (or equivalent) helped with a major refactoring effort or library upgrade?

/happy prototyping/ – dave

How We Manage Continuous Integration 2.0

I work for a large software company, and we’ve used AccuRev to facilitate using a large scale distributed Continuous Integration model.

AccuRev makes this possible with the Stream approach to managing different codebases.  Developers run builds using the same build scripts used by the core team for production builds that ultimately are packaged and shipped out of Engineering.

These build scripts not only build and package and kit the product, they also run thousands of xUnit tests written to run fast and fail fast.  Developers that encounter failures immediately know where to fix the code to pass the tests.  We also use test driven development.

Each day, developers promote their changes to a task stream.  We use Scrum, so a task stream correlates in most cases to a Scrum team.  This team runs automated builds / tests at their task stream level and when stories are done and accepted and passing, promote the appropriate changes to the integration stream.

The integration stream is built every afternoon, and any test failures run during the build are quickly addressed by the team.  Our Continuous Integration software provides a failure email with the modifications made that day with AccuRev user names.  Developers can then go into AccuRev using the StreamBrowser and the Version Browser and determine the root cause.

Fixes are then promoted back to the integration stream, and the full nightly build in most cases runs successfully.  We fail all integration builds on test failure as we believe in Continuous Integration.

Each week our qa level stream is built and we repeat the same process. Developers handle the promotes, the central release team does not promote code for teams.  As code promotes up the hierarchy from task to integration to qa the frequency of broken builds, due to test errors or compilation, decreases.

This Multi-stage Continuous Integration approach is easy with AccuRev due to stream inheritence.  If you used a branch / merge solution you would need to staff a central team just to manage the commits to source control, and manage code that is “done”.

Use Case: Fixing the Broken Build

by Rob Mohr, AccuRev

In one of many travels and customer visits, I came across a very cool way that AccuRev was helping to improve the way development teams do their work. To be more specific, this group was using Change Packages integrated with the JIRA Issue Tracking system to manage changes across their various product releases. They also used CruiseControl for continuous integration that would kick off nightly builds and notify the team with the results of the build.

From what they told me, the success of builds has significantly improved since they started using AccuRev because of the ability for the developers to work in their own private workspaces where they can integrate and unit test before promoting their changes for the rest of the team. Although broken builds are, for the most part, a thing of the past, they will still occur once in a while and need to be fixed ASAP.

Here is how they do it with AccuRev

The stream structure below is a simpler view of their overall software development process, but will be sufficient to show the use case.

Promoting to the Integration Stream

To start, the 4 developers below have made changes in their workspaces that will be promoted and associated to 4 different issues.

As you can see below, the integration stream (EntSoft_Client_Int) is “aware” of which issues are active in the stream. These are the new “change packages” introduced in the stream to be included in the next nightly build.

Build Fails in the Integration Stream

The next morning, the team is notified that last nights build failed via an email notification from CruiseControl. They have also scripted CruiseControl to automatically enable a time based stream called the “Temp_Fix_Build” stream and assign the appropriate transaction number to rollback the change packages from last night.

Assign the Developer to Fix the Build

One of the developers creates a workspace on the Temp_Fix_Build and “change palettes” over each change package one at a time.  This gives them the ability to mix and match change packages together to determine which one of them is the problem.

Problem Solved

After the culprit is fixed, the repaired change package(s) are promoted back into the integration stream for all to share.

Free Webinar on Continuous Integration

We had a Webinar covering Continuous Integration Using AccuRev, CruiseControl and VisualStudio.NET. Learn how to take advantage of Continuous Integration using AccuRev with CruiseControl.NET in a VisualStudio.NET environment.

Our own Jack Flynn presented, along with Ryan LaNeve, an AccuRev customer with Audio Visual Innovations, now AVI-SPL.

Ryan shows off some great stuff with users checking in code that automatically kicks off a build and even updates the product version number to reflect the latest transaction number from AccuRev. Ryan recently blogged about some of the cool things he’s done using the AccuRev command-line interface (CLI) and its ability to output results formatted as XML: The AccuRev CLI, Going Beyond SCM.

Click here to view the recorded session.

To obtain Ryan’s custom labeller for AccuRev that retrieves a stream’s highest transaction number, visit his blog at http://weblogs.asp.net/rlaneve