Pattern for Floating Labels with Streams

I’ve encountered a number of use cases in the field where people enjoy the use of  ‘floating’ labels.  Implemented correctly, floating labels are a good example of convention-over-configuration  usually eliminating repetitive, manual build system updates.   Implemented incorrectly, they can be a traceability curse.   In this writeup I’ll show two examples of how to implement a floating label pattern in AccuRev.

Deprecated Concept? If you’re used to using floating labels as ‘markers’ on a branch [in another CM system] for stable code or feature complete state, you’ll find that the AccuRev stream hierarchy and associated workflow is a much better solution – in some cases, eliminating the need for floating labels altogether.  In my experience, floating labels have been used because the traditional CM system didn’t support process – so you had to roll your own – ultimately leading to lots of scripting and a solution that doesn’t scale.

Stories of Misuse. Every now and then I hear stories about how an urgent quick fix or missing doc change -needed- to be included in the “release-4.0.1” that just got labeled in source control a few days prior. In some traditional CM systems, you have the ability to commit new file changes, un-label the old version and re-label the new version. As easy as this sounds, this anti-pattern instantly breaks traceability [do your labels maintain history?] and can quickly lead to confusion as “release 4.0.1” has multiple meanings depending on when the label was pulled, either before or after the re-label. Example. I once asked someone why they routinely re-label individual files and their answer was – “because creating a new full label takes about 30 minutes.”  Why so long? Because most CM systems label the individual files (or references) in a given configuration; the more files you have, the longer it will take to label the entire configuration.  I’ve even heard stories about creating labels (and branches!) that take upwards of an hour each run.  Unless you are using a time/txn-based CM system, labeling build events is required if you need to track and compare changes between builds over time.  If you are waiting 30-min per label, forget about the agile “10-minute” build – you haven’t even gotten to the compile step yet!

Valid Cases. In practice there are  cases where having a floating label makes sense, typically related to your build system.  Most cases have to do with marking transient build configurations.  For example, when doing nightly or continual integration builds, let the build script or CI server simply pull/poll a single label and let the developers or build engineer determine what content is bound to the label. This way, the build server can be “dumb” and just get whatever the developer or build engineer say it should have.  This eliminates the need to configure your build system with each-and-every new build label – especially painful for those build systems requiring manual edits of config xml files – ugh.

The Pattern.   Two examples are presented below showing how floating labels can be used to support continuous integration with streams.

Static Floating Label.  In this example, continuous builds are performed on the single stream App_Integration.  No need to change your CI configuration. New builds are kicked off whenever changes are  promoted [by team leads, etc] from the child project streams.

Static Floating Label

The CI/build tool is configured to monitor the single App_Integration stream.  As the configuration changes, the name of the build stream remains unchanged – hence, the App_Integration stream is the ‘floating label’.

When doing very-frequent builds (e.g. per-promote or even nightly) this pattern works well as long as you have the discipline to immediately fix any broken configurations.

Dynamic Floating Label.

In this example, individual build configurations for App_Integration are marked with snapshots.  New snapshots are created whenever changes are promoted [by team leads, etc] from child project  streams, or on schedule (e.g. Nightly).

Dynamic Floating Label

The stream Int_Good_Build is dynamically reparented to either the latest snapshot (e.g. latest build) or reparented to an alternate (e.g. prior) snapshot representing the last known good build.  The ability to move this stream, on-demand and as needed, represents a dynamic ‘floating label’ pattern.  Simply reparent the stream, update the attached workspace and the files on disk will mirror the current release snapshot configuration.  The same pattern can be applied to production releases where internal snapshots capture detailed per-configuration changes, but the reparented floating label can represent the latest public release.

Summary. In all, I’m a big fan of creating snapshots for each build, and removing old[er] ones along the way.  There are two camps on the subject, and this will be left for another blog.  Thus, I like the visibility of knowning which snapshots have been created (implying a purpose), and having a ‘marker’ stream as a floating label (e.g. context) to identify the current build being tested and/or delivered to an Int or Test machine.   Lastly, having recent snapshots immediately available assist with debugging why the latest build is broken – diff by snapshot.  sweet.

/Happy Coding/ dave

Free Webinar: Emerging SCM Best Practices for Agile Development

More Agile resources

AccuRev is hosting a free Webinar on “Emerging SCM Best Practices for Agile Development” on Thursday, February 5 from 1:00 – 2:00 PM EST. The webinar will introduce the unique demands that agile processes place on legacy SCM tools and ways to build and automate an efficient Agile development process. Damon Poole, AccuRev CTO, and Uttam Narsu will be presenting.

Mr. Narsu, an industry expert on software configuration management (SCM) best practices and former Forrester/Giga analyst, will discuss the most important aspects to consider when applying SCM best practices to an agile world.  Mr. Poole will discuss how Multi-stage Continuous Integration can solve some of the underlying impediments to a successful Agile development environment.

“SCM is critical to Agile success. If your SCM tools don’t provide integrated and seamless support for Agile, you won’t get widespread adoption of Agile development,” said Narsu. “Development teams require brutally efficient SCM tools, but the tools must still be issue-based, must have support for flexible process models, and enable efficient branching, merging, and refactoring. The SCM tool should also provide private workspaces and assist continuous integration.”

Mr. Narsu has worked with hundreds of clients using wildly differing software development environments. While clients who succeed with agile development have strong SCM practices, the best practiced agile SCM: transparently integrated, flexible, and collaborative.

Specific topics discussed will include:

• Typical SCM obstacles to Agile success and how to avoid them;
• Key Agile Process requirements for SCM products and specific use case scenarios;
• Challenges with continuous integration, and how Multi-stage Continuous Integration delivers value and how to adopt it today; and
• Key SCM metrics for delivering on Agile development goals.

Register here for this free webinar taking place on Thursday, February 5th from 1:00-2:00 EST: Emerging SCM Best Practices for Agile Development.

AccuRev is a 2009 Jolt Award Finalist

AccuRev is once again pleased to inform everyone that its flagship product, AccuRev, has been selected as a Jolt Award Finalist for Dr. Dobb’s 19th Annual Jolt Product Excellence Awards in the Change and Configuration Management category. 

For the past 18 years, the Jolt Product Excellence Awards have been presented annually to showcase products that have “jolted” the industry with their significance and made the task of creating software faster, easier, and more efficient. The awards presentation is sponsored by JOLT, the fabled soft drink quaffed by software developers for sustenance during project development marathons.

“The Jolt judges have selected these finalists from the nearly 300 qualified nominations that were submitted online. The submissions represent the many innovative tools available for every phase of the software development lifecycle,” said Amber Ankerholz, Jolt Award Project Manager. “This year’s finalist round was extremely competitive and we appreciate the effort all of the participants put into showcasing their products’ features for the judges.”

In the next round of the Jolt Award process, the judges will examine the finalists according to the standard criteria of audience suitability, productivity, innovation, quality, ROI, risk, and flexibility. They focus on products that are ahead of the curve, universally useful, rich in functionality or that were solutions to problems in their product space.

Winners are announced during the Jolt Awards ceremony that takes place on March 11 at SD West 2009 Conference & Expo at the Santa Clara Convention Center.

Pattern for Private Prototype Development

by Dave Thomas

Your team has been coding a feature for a few days, across dozens of files, and everyone is excited with progress.  Deep into development, you find an interesting 3rd party library that may simplify your work and possibly take the team’s feature to the next level.  But it will require adding new files, moving some directories, and refactoring some critical code.  However, you’re not ready to commit your current changes because they are unfinished and will surely break everyone else.  Your changes need to be saved before prototyping in case of a rollback but also be integrated with prototype development.  And what if the new library is a bust?  If you start co-mingling the library integration with unfinished code, the potential revert process will be a complete nightmare and waste of time.

Private Versioning. With AccuRev’s private workspace, you can always commit your changes early, often, and safely without sharing amongst your peers.  But in this case you have two logical development efforts, the 2nd effort depends on the first and both need commits to preserve evolving changes.   How do you keep them cleanly separated and continue to work on both in parallel?

Stream Inheritance.  AccuRev streams have an intriguing and very powerful feature called inheritance.   Similar to how an OO subclass implicitly inherits methods from parent and grandparent classes, a child stream implicitly inherits versions of files & directories from parent and grandparent streams.   Taking advantage of inheritance, we can use streams to independently manage logical changes and cleanly maintain change dependencies without physically co-mingling files.

The Pattern.  Prototyping changes without co-mingling files can be done by simply creating a series of ‘personal’ or ‘private’ development streams (though, they are just regular dynamic streams). This pattern will create a “Feature”, “MyDevelopment”, and “MyPrototype” stream sub-hierarchy.  See Picture.

click to enlarge

From your Integration stream (or equivalent), start by creating a “Feature” stream that will collect all changes from your entire team.  [See related blog, Stream-per-Task Pattern].  The majority of team members may have their workspaces directly from here.  Next, create a child stream called “MyDevelopment” that will track your personal ongoing development activity.  Finally, create a grandchild stream called “MyPrototype” to track changes that will be discarded or retained depending on the level of success.

The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and private development.

The “Feature” stream is the collection point of all in-progress development activities for the given feature from the entire team.  Developers will promote here frequently possibly kicking off an automated build with success/fail notification.   The “MyDevelopment” stream provides a collection point for all of your personal development changes.  This stream may be considered a “private” development stream simply because no other developers will likely use it – set a stream lock to be sure.  A lightweight Continuous Integration build (i.e. compilation only) may be performed on “MyDevelopment” for sanity sake or just compile and promote as a practice.  The “MyPrototype” stream is a collection point for all prototype changes. Even as new changes are promoted to “Feature” and “MyDevelopment”, the “MyPrototype” stream will automatically incorporate those changes (via inheritance) and the prototype developers will merge changes as necessary. The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and personal development. Also, by using a stream for prototype work, multiple developers can contribute and collaborate. If the prototyping work is deemed successful, the files can be promoted to “MyDevelopment”. If the prototyping efforts don’t work out – no problem – just remove the “MyPrototype” stream and re-purpose the workspaces.

The beauty of this pattern is that it isolates development activity by purpose without co-mingling physical file changes.   It lets the prototype developers go nuts and shoot from the hip while the regular feature developers (with the deadline!) work unimpeded and without fear of rampant changes — and everyone stays up-to-date.   And with no limit to stream depth, teams can perform prototyping efforts in parallel and/or perform prototyping based on existing prototypes by adding another child stream!  Furthermore, the pattern works for any size development activity or team, even for us Team-of-One developers with tons of ideas, fast fingers, and a few green screen x-terms (2 space, 80-char wrapping of course – <chuckle>).

This is a perfect example of how AccuRev empowers the developer to take control of their own development.  Creating streams is extremely easy and the stream browser provides unprecedented visibility into the entire development process.  With the right amount of security in place (Locks, ACLs, Triggers), the critical release streams (left side of the stream structure) can be locked down by the CM Admins, but the development related streams (right side) can be fair game for developers to create an environment that suits their purpose, such as prototyping.

Does anyone have a good story to tell about how this pattern (or equivalent) helped with a major refactoring effort or library upgrade?

/happy prototyping/ – dave

How We Manage Continuous Integration 2.0

I work for a large software company, and we’ve used AccuRev to facilitate using a large scale distributed Continuous Integration model.

AccuRev makes this possible with the Stream approach to managing different codebases.  Developers run builds using the same build scripts used by the core team for production builds that ultimately are packaged and shipped out of Engineering.

These build scripts not only build and package and kit the product, they also run thousands of xUnit tests written to run fast and fail fast.  Developers that encounter failures immediately know where to fix the code to pass the tests.  We also use test driven development.

Each day, developers promote their changes to a task stream.  We use Scrum, so a task stream correlates in most cases to a Scrum team.  This team runs automated builds / tests at their task stream level and when stories are done and accepted and passing, promote the appropriate changes to the integration stream.

The integration stream is built every afternoon, and any test failures run during the build are quickly addressed by the team.  Our Continuous Integration software provides a failure email with the modifications made that day with AccuRev user names.  Developers can then go into AccuRev using the StreamBrowser and the Version Browser and determine the root cause.

Fixes are then promoted back to the integration stream, and the full nightly build in most cases runs successfully.  We fail all integration builds on test failure as we believe in Continuous Integration.

Each week our qa level stream is built and we repeat the same process. Developers handle the promotes, the central release team does not promote code for teams.  As code promotes up the hierarchy from task to integration to qa the frequency of broken builds, due to test errors or compilation, decreases.

This Multi-stage Continuous Integration approach is easy with AccuRev due to stream inheritence.  If you used a branch / merge solution you would need to staff a central team just to manage the commits to source control, and manage code that is “done”.

Use Case: Fixing the Broken Build

by Rob Mohr, AccuRev

In one of many travels and customer visits, I came across a very cool way that AccuRev was helping to improve the way development teams do their work. To be more specific, this group was using Change Packages integrated with the JIRA Issue Tracking system to manage changes across their various product releases. They also used CruiseControl for continuous integration that would kick off nightly builds and notify the team with the results of the build.

From what they told me, the success of builds has significantly improved since they started using AccuRev because of the ability for the developers to work in their own private workspaces where they can integrate and unit test before promoting their changes for the rest of the team. Although broken builds are, for the most part, a thing of the past, they will still occur once in a while and need to be fixed ASAP.

Here is how they do it with AccuRev

The stream structure below is a simpler view of their overall software development process, but will be sufficient to show the use case.

Promoting to the Integration Stream

To start, the 4 developers below have made changes in their workspaces that will be promoted and associated to 4 different issues.

As you can see below, the integration stream (EntSoft_Client_Int) is “aware” of which issues are active in the stream. These are the new “change packages” introduced in the stream to be included in the next nightly build.

Build Fails in the Integration Stream

The next morning, the team is notified that last nights build failed via an email notification from CruiseControl. They have also scripted CruiseControl to automatically enable a time based stream called the “Temp_Fix_Build” stream and assign the appropriate transaction number to rollback the change packages from last night.

Assign the Developer to Fix the Build

One of the developers creates a workspace on the Temp_Fix_Build and “change palettes” over each change package one at a time.  This gives them the ability to mix and match change packages together to determine which one of them is the problem.

Problem Solved

After the culprit is fixed, the repaired change package(s) are promoted back into the integration stream for all to share.

Free Webinar on Continuous Integration

We had a Webinar covering Continuous Integration Using AccuRev, CruiseControl and VisualStudio.NET. Learn how to take advantage of Continuous Integration using AccuRev with CruiseControl.NET in a VisualStudio.NET environment.

Our own Jack Flynn presented, along with Ryan LaNeve, an AccuRev customer with Audio Visual Innovations, now AVI-SPL.

Ryan shows off some great stuff with users checking in code that automatically kicks off a build and even updates the product version number to reflect the latest transaction number from AccuRev. Ryan recently blogged about some of the cool things he’s done using the AccuRev command-line interface (CLI) and its ability to output results formatted as XML: The AccuRev CLI, Going Beyond SCM.

Click here to view the recorded session.

To obtain Ryan’s custom labeller for AccuRev that retrieves a stream’s highest transaction number, visit his blog at http://weblogs.asp.net/rlaneve

Agile 2008 – Agile Skeptic?

By Josh Sherwood, AccuRev engineer

I had the opportunity to co-present a talk on Continuous Integration at the Agile 2008 conference this past week. While there I took the opportunity to attend a variety of sessions. Workshops discussing Business Value, Experience reports describing transitions from Waterfall to Agile, Aspects of Leadership in and Agile Environment and many others. And okay, I’ll admit that I have been a skeptic of a variety of the Agile Processes. Our CTO, Damon Poole, has been an Agile advocate for some time now. He and I have had ongoing discussions about the value of Agile practices. We would talk about the iterative model, going back and forth about the value and how to describe it to different groups. We’d eventually come to an understanding, but there were still many elements that I have been skeptical of and found difficult to see their value.

In addition to the elements of Agile practices that Damon and I would discuss, I started taking a look at other Agile topics. I looked at things like estimation, use of 3×5 cards and even pair programming. I didn’t see the value in the sticky note processes, I mean using a paper process to discuss a digital product? I didn’t see the value in pair programming, especially for consulting companies where your development hours translate directly into billable hours.

But then I came to the conference. I listened to people like Richard Sheridan talk about his experience with Pair Programming, why and how it worked for him. I listened to experience reports from companies like Healthwise, and heard about the difficulties they overcame to improve prior processes and move into Agile. And I spoke with Scrum trainers, who were encouraging their customers and helping them overcome the hurdles of understanding, while remaining flexible about sharing Agile ideas.

What was great about things like the Pair Programming was the team participation. Not only was Richard there talking about their factory model, team members were in the audience, talking about what they take away from the model. They talked about some of the things an individual developer would run into. You know those times, where you are working on a particular problem and just can’t quite figure out how to break the bottleneck. You wander around, get some coffee, or browse the web checking the Olympics. With their pair model, now that bottleneck has two developers working on it. No they’re both not checking their email, they are floating ideas back and forth, bringing to bare the problem and coming to some resolution. With a close proximity to other pairs they are also propogating these ideas farther out amongst the group. Yes there is training and not everybody can manage to work in pairs, but by using this model and actively rotating the pairs they ensure there are no knowledge silos. If someone is out one day there is more than one person who can tackle the problem at hand.

It has been a community of ideas this week. Some people use Scrum but don’t perform retrospectives. Some people model agile practices, but don’t follow a specific practice. Some people are taking the team based processes and developing large scale models for larger development tasks.

As I sit here building out the story for some of our future ideas, I’m encouraged that we can adopt more of the Agile practices than we have in the past, work through some of the confusion those ingrained in waterfall have with new processes, and further the practices that have allowed us to deliver innovation at a faster rate than we could with older models.

What Can Developers Learn from their Surgeon?

I recently read a story that the World Health Organization (W.H.O.) issued a new safety checklist for surgical teams. Checklists sound great and frankly it was kind of surprising they weren’t already in practice. Here at our company we go through a number of checklists, and I can only assume you do as well. There are testing checklists to determine what test coverage exists, process checklists when products are about to ship, and integration points themselves could be considered a checklist of activities that need to occur for success.

What I found particularly interesting were the topics covered in the checklist. There are fundamental items like marking the location of the surgery, but I found the following to be the most topical:

*    Confirm that all team members have introduced themselves both by name and by their role on the surgical team.

*   The surgeon, anesthesia professional, and nurse should verbally confirm the patient’s identity, surgical site, and procedure to be performed.

*    Anticipated critical events to be reviewed by the surgeon are any critical or unexpected steps, estimated operative duration, and anticipated blood loss.

Keep it High-Level

Blood loss aside, what I find interesting about these items is how high level they are. You have team members participating as a team, identifying the areas of focus and activities. Staff executing risk analysis. Providing estimates. This is great.

SCRUM

We’ve been using Scrum. This means we have a 15 minute meeting once a day to go over our activities. We talk about what we did yesterday, what we are doing now, how long it should take and any risks to success. Sound familiar?

SCRUM HURDLES to HURDLE

Although Agile has been going well here for a while, there may be a number of initial hurdles to watch out for if you make this kind of change. Having to coordinate a day and time for all team members, then trying to keep the meeting to 15 minutes can be a challenge. It’s far too easy for development staff to lapse into traditional practices. When we first went to Scrum some of us would begin to discuss specific issues during the meeting (a Scrum no-no), and people would obviously start tuning out. This then translates into not knowing the more high-level team goals and activities.

Our first attempt to fix this, having everyone send a daily status email, at first just went to the team lead. The team lead knew what was going on, but it still didn’t solve the larger goal of having people be aware of their role and its impact on the team. We then tried to fix this, by group consensus, by everyone emailing the group with their daily status. This had the advantage of getting the format of the data consistent amongst the team, and getting information to everybody in the team. But it doesn’t get people to read the email. I now have a filter that automatically drops these into a directory that makes them easier to find, open, and read, or scan depending on my level of caffeine.

NOT LOSING SIGHT OF THE GOAL

So what I see as part of the goal, as well as what the W.H.O. was trying to achieve could be lost. Without the physical meeting everyone remains caught up in their own particular goals.

These personal goals are necessary, but they are not always conducive to team productivity.

Here we use Continuous Integration. We’ve been progressively converting more of our attended tests to unattended, increasing our product coverage and productivity. At the same time, staff members have been changing the test infrastructure to allow us to test varying configurations. Both are great goals, but in the beginning they ended up in conflict. One engineer was making more configuration settings required while another staffer was trying to make more of the configuration settings dynamic for portablity. The newly unattended tests were hanging or exiting early because configuration items unexpectedly became required.

Without a common goal, or at the least an understanding of everyones goals we run into conflict.

What else can be done?

When developers are used to being siloed, and their work feels isolated, they are more likely to tune out as to what is being done around them. But integration has to happen. And it has to work. Integration should no longer be thought of as somebody else’s problem, or as something that happens after the fact. Yes I was one of the people who didn’t like yet another meeting. I was one of the people who sometimes drifted into the mundane details of my work that I thought others would want to know, eating up meeting time. Now I champion the 15 minute meeting and keep people quick and to the point. And if everyone can start feeling like integration is the goal, then people will start thinking about how their work impacts others. Sharing that information in a timely manner will smooth out the bumps, and ultimately get us more focused on the larger goals. This isn’t “brain surgery”…or is it??

Continuous Integration: Begin Again

We’ve been using CruiseControl productively for a number of years. First as a stop gap in ensuring build success on multiple operating systems, later as a quick check of tests. More recently we’ve added reporting, and tests as part of a larger memory analysis platform.

But, being a bunch of software developers means we are continuously craving something different, like switching to Haskell and using Ruby on Rails. Joking aside (and nobody panic) it is more that we look at areas of our build processes to identify the pain points, and what can be done to improve them. I mean, that’s how we provide value right? By continuously improving processes and features.

This time we took a look at things we don’t change often, but end up injecting a lot of pain into the process each time they are done. And these pain points become more obvious as we increase our agile usage, as they occur more frequently than they did in the past.

How often do you go through your “official” release process? In my past life it used to be this occurred every six months at best, a year or so on a bad year and a long time ago (in a far far away land) I had the opportunity to work on a project that lasted 7 years. Now granted I certainly wasn’t on it for seven years, people tended to drift on and off the project but it took seven years to ship. That was a long time ago.

Today we are looking at monthly cycles, or in the case of very specific changes even two week cycles. Those painful processes that were blotted out with 6 months between them now happen pretty frequently. We used to not bother with the problems, just moan a little and trudge through it. But agile changes that. Continuous Integration changes that. These processes should work, be easy to change, and adapt quickly. Anything that doesn’t adapt you want to drop as quickly as possible.

To do that we’ve started looking at new tools for our processes. Because new tools solve all your problems right? As soon as you started compiling in C++ from C all your problems went away right? Okay not really. Tools are nice, tools make a good impetus for change, but change solves your problems. Yes moving to C++ provides you with access to new functionality, but only until you actually implement and use exception handling in a structured manner do you really get value out of the tool.

Sometimes it’s felt that the tool is enough. I mean when you switched SCM products you kept your old processes. Sure, maybe they didn’t quite fit the new tool and you had to write a lot of scripts, but it was what you were used to. You may even be nodding your head right now. Nothing like shoe horning a process you created 5 years ago into a tool adopted last year. If you didn’t change your process how did the new tool provide value? Performance and scalability may get you some value, but new features and usage patterns get you a lot farther.

As part of examining our build and integration tools, we looked at processes. What areas of our process are not formal? Is there a specific person who knows the incantations to make it work? What happens when machines fail, do we have alternative machines that can take up the slack? Can we run multiple builds on the same machine? Multiple tests? How do we get and review our results? What are the top 3 pain points for each person involved?

Not only does asking these questions help us understand the process better, but it identifies the areas of change. And with these changes we can leverage both our existing tools as well as the new tools we’ve chosen to acquire. It helps our existing team, and it helps those who join us in the future. They’ll be more productive faster. We’ll have time to work on the features our customers want, and instead of just getting by we’ll be able to use forward looking technologies.