The most of project management methods assume projects in the project stage, what means for me this happy time of development, when you just have requirements, deadlines and team and you can slice time into iterations and play. You can use different project management methodologies and different approaches of branches management (like Trunk Based or Feature Branch) and finally come to conclusion that with a good team you can effectively get it done in whatever you’ve currently chosen.

Going to production changes the game, though. In my opinion a lot of project management tools are focused only on the project stage and don’t really support production stage. On the other hand techniques related to branches management seem to fight agains each other for being the best and only reasonable. I personally don’t believe in this. Here I’d like to show our development process in mature web application. Especially how we can mix different branching approaches with our favourite ticketing tool YouTrack to have a successful development process.

Prerequisites

In few words what are preliminary conditions and what we want to achieve:

1. Project is almost four years old and almost two on production.
2. Fast changing business environment, i.e. we discover the market on the fly and need to quickly adjust the software.
3. We don’t have fixed-time iterations because we don’t know what will be important next week.
4. We don’t have daily meetings, however we discuss features before development starts, and whenever feature owner feels he needs to discuss something with the rest of the team.
5. We do code review before sending anything to tests. It is not said who makes the code review - anyone can, excluding original feature developer.
6. We don’t estimate tasks precisely, although we split things to tasks (with a length from hours to days) and features (with a length from days to weeks).
7. We release when we have consistent set of changes developed and tested. The release time is not planned. Looking on the last year we had average release time every 3.5 days.
8. We need to manage backlog. We have more things reported than we can deliver, and the list is constantly growing. We need to able to manage backlog to choose the most important things to be done at the time.
9. We need to keep clean master to be able to release immediately, when a bug or anything else is required to be fixed at the moment. We call it a hotfix.

Branching rules

We use GIT as a VCS system and we assume following branches workflow. Each change starts its life in integration branch, then it goes to release candidate branch and finally finishes in master. We release only from master:

All changes in these three branches need to pass automatic tests to be promoted to the next one. In integration and release candidate branches we additionally do all manual tests of new features relased.

The main continuous integration branch is called integration. However, we don’t directly commit to this branch. There are separated branches with things under development derived from this branch, which are then merged back. Things we split to following kinds: task and bugs which have very short life (hours to days, with usually single developer involved) and features which can live longer (days to weeks, with usually more than one developer involved).

There’s a simple explanation why we do blue things in branches - this is because on the blue level we do code review. All changes need to be reviewed before they are merged back to integration and it’s well supported in GIT to make a quick review while changes are still in branches.

Release candidate

Now it’s time to explain what is release candidate branch for. It’s obvious that we don’t merge changes to master because we need to keep it clean in case of a hotfix. But after all we could merge changes from integration directly to master, when they are ready for release.

And “when they are ready for release” is a key word here. We do code review in ticket branches, but we test and stabilize changes in integration branch. Let me show what can happen during such stabilization:

All tickets have been integrated to integration and started their stabilization stage. Stabilization of TICKET-3 is in progress, what causes new commits to integration. However, developers can freely merge their other finished tasks to integration branch as well, and the influx of changes is uncontrollable. For example during stabilization on TICKET-3, new changes (TICKET-4 and TICKET-5) can come to integration, and they also need to be tested and stabilized.

This way we can partially apply continuous integration idea, but on the other hand we can’t determine the exact moment when we can release. Thus, to finish and stabilize work from integration we move everything to release candidate branch at arbitrary moment:

To release candidate branch we can move partially unstable changes, and developers can’t freely integrate new things here. This branch contains all work we decided to release in a little while. Usually it still needs to be tested and adjusted. When it’s done, we can merge stable set of changes to master and release, while the same set of changes we need to merge back to integration. The previous situation but with release candidate branch involved looks as follows:

Finally, the stable set of changes is merged to master to be relased, and at the same time is present in integration branch to be easily integrated with incoming changes from ticket branches:

Then we tag master with version tag, and all this stuff goes to production.

Hotfix workflow

The last thing in this picture is a hotfix workflow. This is simple. Hotfix is something needs to be fixed immediately, so it starts and ends its life in master. Hotfixes are usually very quickly deployed, frequently on the same day as they are delivered. Sometimes the deployment consists of release candidate just merged to master and few hotfixes made on the same day.

For the record here is the diagram of hotfix workflow:

Issue tracking prerequisites

Here ends the easy part: planning of branches workflow. But now how to reflect this process in project management or issue tracking system? Of course it depends on the application, but we tested few solutions and it’s not really obvious and rather difficult. Here is the short list of what we want to have:

1. We want to have a backlog management view, where we can easily select tickets to be developed from growing backlog list.
2. We want to make a quick review of tickets before they go to the backlog, to be able to identify hotfixes.
   
3. We want to have no more than few new things chosen for development at the same time.
4. We want to see whole work currently in progress split into branches on the main project management dashboard.
5. We want to assign and perform three basic kinds of task: development, code review and testing.
6. We want to have a view where a team member can see only his work to do (regardless of it’s development, code review or testing) and can easily find a branch where he should look for a given change.
7. We want to be able to manage features (big tasks), which should be then split to subtasks. We don’t want to see these subtasks on the main dashboard, though. There we want to see only features progress. However, feature subtasks should be assignable to different team members and they should see them on their individual dashboards.
8. We want to have separate ticket workflow for hotfixes.
9. We want to be able to see and easy reflect changes from GIT branches in project management dashboards. For example developer should be able to move closed tickets from ticket branch to integration. Project master should be able to decide when we want to move set of changes from integration to release candidate, and finally when we move stable changes from release candidate to master to be released.
10. We want to be able to mark released tickets with version number and we want to keep full version history.

YouTrack

After reviewing capabilities of few different tools, we finally returned back to our favourite YouTrack. This issue tracking tool is very dynamic, has very nice and configurable UI, nice and configurable dashboards, configurable custom fields, and horrible but flexible workflow editor. About the latest one, it’s provided in separate desktop application and to achieve something using it you need to be equipped with a lot of patience and send your children to holidays, to not let them hear how you swear. But finally after hours of struggling you can apparently achieve what you want, and it counts.

YouTrack configuration

The majority of the next part of this article will be rather boring. I’ll show detailed YouTrack configuration, because I want to use it as my own reference for my further projects. But you can scroll down to see the final results.

Custom fields configuration

At the beginning we narrowed the list of possible task types to keep it clean. Bugs and tasks are our fine grained ticket types, feature is a big one and hotfix is additionally marked with red color:

Then we customized list of states to what we wanted to use in the project, i.e.:

Where:

• Submitted - our pre-backlog stage when we can still decide whether the ticket is a hotfix or not.
• Accepted - a ticket in backlog waiting for good times (however it still can be classified as a hotfix from here).
• Open - a ticket moved from the backlog to development stage.
• In progress - cannot really remember what is this state for.
• Code revision - finished tickets which should be reviewed, thay are still in their ticket branches and they might be accepted (move to finished state) or returned to development (moved to in progress state).
• Fixed - this is a special state, usually it means that ticket is done, but isn’t tested yet. However in one instance, when the change is still in the ticket branch, this state means that code revision passed and ticket can be merged to the integration branch.
  

Next field is our custom field, indicating branch where the ticket is located at the moment:

When the ticket is in the Ticket branch, it means it is under development in its own branch (no matter if it’s a task, bug or feature). Ticket a blue branch from previous GIT diagrams. All ticket branches have the same name as the ticket they represent (eg. T-80).

Mysterious Parent branch will be discussed in a while. It’s related to feature lifecycle and indicates that this change doesn’t have its own branch, but is always located in the parent feature branch.

Finally, we have not one but three users involved in ticket development: Assignee, Code reviewer and Tester. We will also heavily use Fix version field:

Simple workflow rules

Together with custom fields setup we prepared a few simple workflows to have a control over important things. But the first step was to disable default YouTrack Subtasks workflow which is not useful in our config:

First two rules are simple and only turning on ticket watching for people involved in the development. Assignee watches the ticket by default, but our custom role users don’t:

rule Watch on code reviewer 
 
when Code reviewer.changed && Code reviewer != null { 
  Code reviewer.watchIssue(issue); 
  addComment("Code reviewer assigned: " + Code reviewer.fullName); 
}

rule Watch on tester 
 
when Tester.changed && Tester != null { 
  Tester.watchIssue(issue); 
  addComment("Tester assigned: " + Tester.fullName); 
}

The next rule is important from the point of view of deployments and version management. When the ticket is deployed in a release, we mark it with the version number in Fix version field. We never allow these tickets to be reopened again after they are relased:

rule Do not reopen deployed tickets 
 
when resolved.changed && Fix versions.isNotEmpty { 
  assert !(resolved.oldValue != null && resolved == null): "Do not reopen deployed tickets. Please create new related ticket instead."; 
}

Last two rules show how we manage features. Feature, which is the biggest amount of work we assume, can be split to smaller tickets and connected to the original feature ticket using subtask of relation. This is the only allowed usage of subtask of relation and feature subtickets can’t have their own branches. We indicate this setting Parent branch on all subtickets:

rule Feature subtickets workflow 
 
when !(subtask of.isEmpty) && (subtask of.changed || becomesReported()) { 
  assert subtask of.first.Type == {Feature}: "Tickets can be only subtasks of feature."; 
  assert Type == {Task} || Type == {Bug}: "Only bug and task tickets can have a parent." + Type; 
  assert subtask of.first.Fix versions.isEmpty: "This feature is already released. You cannot add more tickets to it. In case of a bug or new related task you need to report it to standard backlog."; 
   
  Branch = {Parent}; 
}

When feature is released, and gets its own Fix version number, we propagate this version to all feature subtickets to mark them released as well:

rule Feature fix version propagation 
 
when Fix versions.changed { 
  for each child in parent for { 
    child.Fix versions.clear; 
    for each fixVersion in Fix versions { 
      child.Fix versions.add(fixVersion); 
    } 
  } 
}

Project dashboards

The most powerful feature of YouTrack are dashboards, and they can be customized in very flexible way. We use five dashboards to manage the whole project. To configure all dashboards we use similar rules:

First dashboard is for backlog management, and its base query is:

State: Submitted , Accepted has: -{Subtask of}

It shows tickets in Accepted and Submitted states, but doesn’t show feature subtickets (only main features - this is default behavior for almost all subsequent dashboards):

Second dashboard is used by a project master for general project management, and its base query is:

State: Open, {In Progress}, {Code revision}, Fixed, Verified has: -{Subtask of}

It shows tickets progress on kanban board, but we can easily see in which branch the ticket is present at the moment, and what is its status:

The next one is an auxiliary project management dashboard for assignment management, and its base query is:

((State: Open, {In Progress} and Assignee: Unassigned ) 
    or (State: {Code revision} and Code reviewer: Unassigned ) 
    or (State: Fixed and Tester: Unassigned )) and (has: -{Subtask of})

On this dashboard project master can catch all unassigned tickets. This means tickets should be developed, but there’s no Assignee, tickets should be reviewed but there’s no Code reviewer assigned and tickets should be tested but there’s no Tester assigned:

Penultimate dashboard is the only one for a team member, and its base query is:

(State: Open, {In Progress} and Assignee: me) 
    or (State: {Code revision} and Code reviewer: me) 
    or (State: Fixed and Tester: me) 
    or (State: Fixed and Assignee: me and Branch: Ticket)

It shows in one view all work that should be done by specific team member, i.e:

• tickets in development, where he is assigned as Assignee and should work on
• tickets required code review, where he is assigned as Code reviewer and should review
• fixed tickets in Ticket branch which should be merged to Integration branch, where he is assigned as Assignee and thus is responsible for integrating this change
• fixed tickets in other branches, where he is assigned as Tester and should test them

The last but not least dashboard enables features management, and its base query is:

has: {Subtask of} or Type: Feature 

This dashboard is accessible by all team members and is intended both for feature planning and feature development (what will be shown in the next chapter):

This is the only dashboard where swimlanes are made by parent tickets of Feature type, identified by subtask of relation:

Statemachines

We could work just on this configuration but … people do mistakes. To reduce them and to prevent the resulting mess it’s worthwhile to define state workflow rules, allowing to perform only the right steps. We have two statemachines defining what can happen next (and what can’t) for State and Branch field. To see how it works you just need to navigate to the next chapter.

Here is the State field statemachine:

statemachine State workflow for field State { 
 
  initial state Submitted { 
    on send to backlog[always] do {<define statements>} transit to Accepted 
      
    on start hotfix[always] do { 
      if (subtask of.isNotEmpty) { 
        // feature subtasks are created directly in open state on feature dashboard and this workflow is executed 
        // we just don't want to mark them as hotfixes 
      } else { 
        // otherwise from submitted we can only open hotfixes 
        Type = {Hotfix}; 
        if (Assignee == null) { 
          project.leader.notify("[ASSIGNEE] " + getId() + ": " + summary, getUrl()); 
        } 
      } 
    } transit to Open 
      
    on can't reproduce[always] do {<define statements>} transit to Can't Reproduce 
      
    on won't fix[always] do {<define statements>} transit to Won't fix 
      
    on duplicate[always] do {<define statements>} transit to Duplicate 
  } 
    
  state Accepted { 
    on start hotfix[always] do { 
      Type = {Hotfix}; 
      if (Assignee == null) { 
        project.leader.notify("[ASSIGNEE] " + getId() + ": " + summary, getUrl()); 
      } 
    } transit to Open 
      
    on send to development[always] do { 
      if (Assignee == null) { 
        project.leader.notify("[ASSIGNEE] " + getId() + ": " + summary, getUrl()); 
      } 
    } transit to Open 
      
    on can't reproduce[always] do {<define statements>} transit to Can't Reproduce 
      
    on won't fix[always] do {<define statements>} transit to Won't fix 
      
    on duplicate[always] do {<define statements>} transit to Duplicate 
  } 
    
  state Can't Reproduce { 
    on  reopen [always] do {<define statements>} transit to Submitted 
  } 
    
  state Won't fix { 
    on  reopen [always] do {<define statements>} transit to Submitted 
  } 
    
  state Duplicate { 
    on  reopen [always] do {<define statements>} transit to Submitted 
  } 
    
  state Open { 
    on start work[always] do { 
      assert loggedInUser == Assignee: "Only assignee can start work on this ticket."; 
      if (Type == {Hotfix}) { 
        message("Don't forget to create ticket branch from master for this ticket."); 
      } else if (Branch == {Ticket}) { 
        message("Don't forget to create ticket branch from integration for this ticket."); 
      } 
    } transit to In Progress 
      
    on can't reproduce[always] do {<define statements>} transit to Can't Reproduce 
      
    on won't fix[always] do {<define statements>} transit to Won't fix 
      
    on duplicate[always] do {<define statements>} transit to Duplicate 
  } 
    
  state In Progress { 
    on send to code revision[always] do { 
      if (Branch != {Parent}) { 
        assert loggedInUser == Assignee: "Only assignee can finish work on this ticket."; 
        assert Tester == null: "This ticket is already in testing stage and shouldn't be sent to code revision anymore. You should send it for test."; 
        assert Branch == {Ticket}: "Only tickets in ticket branch can be send to code revision."; 
        if (Code reviewer == null) { 
          project.leader.notify("[CODE REVIEWER] " + getId() + ": " + summary, getUrl()); 
        } 
      } else { 
        // workflow within the feature is free 
        if (Code reviewer == null) { 
          subtask of.first.Assignee.notify("[CODE REVIEWER] " + getId() + ": " + summary, getUrl()); 
        } 
      } 
    } transit to Code revision 
      
    on fix it [always] do { 
      if (Branch != {Parent}) { 
        assert loggedInUser == Assignee: "Only assignee can finish work on this ticket."; 
        if (Type != {Hotfix}) { 
          assert Branch != {Ticket}: "Tickets in ticket branch are not prepared for testing yet. Please merge this ticket to integration branch " + "and change Branch field of this ticket."; 
          if (Tester == null) { 
            project.leader.notify("[TESTER] " + getId() + ": " + summary, getUrl()); 
          } 
        } else { 
          Branch = {Master}; 
          message("Done hotfixes are moved instantly to master to be released. Please merge ticket branch to master immediately!"); 
        } 
        // workflow within the feature is free 
      } 
    } transit to Fixed 
      
    on can't reproduce[always] do {<define statements>} transit to Can't Reproduce 
      
    on won't fix[always] do {<define statements>} transit to Won't fix 
      
    on duplicate[always] do {<define statements>} transit to Duplicate 
  } 
    
  state Code revision { 
    on  reopen [always] do { 
      if (Branch != {Parent}) { 
        assert loggedInUser == Code reviewer: "Only code reviewer can reopen this ticket."; 
        assert Branch == {Ticket}: "Only tickets in ticket branch can be sent back from code revision."; 
      } 
      // workflow within the feature is free 
    } transit to In Progress 
      
    on send to integration[always] do { 
      if (Branch != {Parent}) { 
        assert loggedInUser == Code reviewer: "Only code reviewer can finish this ticket."; 
        assert Branch == {Ticket}: "Only tickets in ticket branch can be sent to integration."; 
        if ((Type != {Hotfix})) { 
          addComment("Code revision passed, please merge this ticket to integration branch and change branch to Integration."); 
        } else { 
          addComment("Code revision passed, please merge this ticket to master branch and change branch to Master."); 
        } 
      } 
      // workflow within the feature is free 
    } transit to Fixed 
  } 
    
  state Fixed { 
    on reopen[always] do { 
      if (Branch != {Parent}) { 
        assert loggedInUser == Tester: "Only tester can reopen this ticket."; 
        assert Branch != {Ticket}: "Fixed tickets in ticket branch should be merged to integration, not reopened."; 
      } 
      // workflow within the feature is free 
    } transit to In Progress 
      
    on tests passed[always] do { 
      if (Branch != {Parent}) { 
        if ((Type != {Hotfix})) { 
          assert loggedInUser == Tester: "Only tester can mark this ticket verified."; 
        } 
        assert Branch != {Ticket}: "Tickets in ticket branch are not sent for tests yet, so they cannot be marked as verified."; 
      } 
      // workflow within the feature is free 
    } transit to Verified 
  } 
    
  state Verified { 
     
  } 
 
}

And this one controls how the Branch field can change:

statemachine Branch workflow for field Branch { 
 
  initial state Ticket { 
    on merge to integration[always] do { 
      assert State == {Fixed}: "Only fixed tickets can be sent to integration branch for testing."; 
      assert Code reviewer != null: "Only reviewed tickets can be sent to integration branch for testing."; 
      assert Type != {Hotfix}: "Hotfixes don't go through integration and rc workflow. Please merge it directly to master and change branch in YT to master."; 
      assert loggedInUser == Assignee || loggedInUser == project.leader: "Only assignee or project leader can merge this ticket to integration."; 
      message("Ticket moved to integration branch in YT, you should now merge it from ticket to integration branch in GIT."); 
      if (Tester == null) { 
        project.leader.notify("[TESTER] " + getId() + ": " + summary, getUrl()); 
      } 
    } transit to Integration 
      
    on merge to master[always] do { 
      assert Type == {Hotfix}: "Only hotfixes can be sent to master directly from ticket branches."; 
      assert State == {Fixed}: "Only fixed hotfixes can be sent to master to be released."; 
      assert loggedInUser == Assignee || loggedInUser == project.leader: "Only assignee or project leader can merge this ticket to master."; 
      message("Ticket moved to master branch in YT, you should now merge it from ticket to master branch in GIT."); 
      project.leader.notify("[HOTFIX TO DEPLOY] " + getId() + ": " + summary, getUrl()); 
    } transit to Master 
  } 
    
  state Integration { 
    on merge to release candidate[always] do { 
      assert loggedInUser == project.leader: "Only project leader can merge integration tickets to release candidate branch."; 
    } transit to Release candidate 
  } 
    
  state Release candidate { 
    on merge to master[always] do { 
      assert loggedInUser == project.leader: "Only project leader can merge release candidate tickets to master."; 
      assert State == {Verified}: "Only verified tickets can be merged to master and released."; 
    } transit to Master 
  } 
    
  state Master { 
     
  } 
 
}

YouTrack issues workflow

This chapter covers the results of previous assumptions and configurations, and shows how we can work with our process using YouTrack.

Backlog management

New tickets come from different sources. I won’t discuss them here. The only thing worth mentioning is that ticket starts its life in the Submitted state, which is our pre-backlog state. All these tickets are swept by the project master daily either by moving them to backlog or by classyfing as hotfixes.

Project master can move the ticket to the backlog by dragging it to Accepted state on backlog dashboard:

From time to time (weekly, usually) we try to catch business people and discuss which things are the most important. This is a different kind of art, because for the business all things are the most important. Although, we can manage it to choose no more than 10 tasks weekly and move them to development stage:

The backlog management process is still most wanting and I’m thinking how to improve it.

Tasks workflow

When task is sent to development, it is moved from backlog to the project management dashboard. In next few screens I’ll show how it travels through states and branches. It isn’t usually done here - team members do the same activities on their own boards. But it’s easy to show the concept on this single board.

When task is assigned, task owner moves it to the next column what means it is in progress. Because it’s the ticket branch, he also has to create new separated branch from Integration:

Previously shown statemachines control if the ticket state changes comply to the workflow rules. Making illegal move ends up with following message:

When developer ends his work he has to move the ticket to code review state:

Code reviewer has two options: either to send the ticket back to development or accept it by moving to Fixed state:

Fixed state in Ticket branch means the developer should now merge his ticket branch to Integration. Afterwards the ticket should be moved one level down:

Fixed ticket in Integration branch should be tested. Tester sees it on his dashboard and can send it back to development or mark the ticket as Verified, what means the ticket passed tests:

Let’s consider an example where ticket is sent back to development. It’s of course shown on Assignee dashboard as In progress ticket. But the important thing here is what happens when we want to move the work to Release candidate branch. We can do it on arbitrary moment, even for unfinished or unstable tickets:

Release candidate branch is for stabilization of existing changes before they are released. The only difference from Integration is that we don’t allow to have an influx of new changes here, but the workflow is exactly the same as in Integration branch:

Tester is still involved in the workflow and he has the same responsibility as on the previous stage: to push the ticket back to the development stage or mark it as Verified:

When ticket is Verified it can be released. Project master waits to have Release candidate branch containing only Verified tickets. When it happens, it means that the release is stable and can be moved level down:

Feature workflow

Now, few words about features. Feature is the same kind of ticket as the others and it goes through the same workflow. It can, for example, wait in the backlog for better times:

The only difference between Feature and other ticket types is that Feature can have subtickets. On Features dashboard we can easily add them:

In fact features can live in the backlog for a longer time, during which we can thoroughly split them to subtasks:

Feature subtasks are never shown on the backlog board, but we can see here main Feature ticket. When we decide to start work on it, we just do the same things as for general tickets:

Feature ticket travels through states and branches in exactly the same way as Task and Bug:

In the meanwhile the feature dashboard is managed by the feature master in his own way:

We don’t enforce any rules here. Developers assigned to feature subtasks can see them on their own dashboard, but from the main project dashboard point of view we are not interested in what happens in these subtasks. We only want to see overall feature progress, which finally ends its life as Verified in Master:

Hotfix workflow

The last one kind of change is a Hotfix. Issue can be classified as a hotfix in Submitted or Accepted stage:

Hotfix starts its workflow in the same way as different tickets. The only difference here is that Ticket branch is not derived from Integration, but directly from Master:

Depending on developer decision, work on hotfix can be ended in two ways:

But when it’s finished it should be directly merged to Master, because it represents a hot change to be deployed ASAP:

Releases and history

All kinds of changes finally end their lifecycles in the bottom right corner of the main project dashboard. Because we use Fix versions field to indicate sprints, we can easily remove released tickets from operational dashboard by setting the version number:

And because we use Fix versions field to indicate sprints, we can have easy access to project versions history on the main board, just by changing the sprint here:

Moreover, because of Fix versions propagation from Feature down to its subtickets, we can also have access to detailed feature history on Features dashboard:

Final words

The article above shows how we manage work in one of my projects. I must admit the methodology was very different in almost all of them. It depends on many things. Here for example I discuss mature project in the production stage, what will completely be different from the project in early development stage. But there are many different factors. Do we have full stack developers can do anything, or we have people experienced only in single layers? Do we have and office or we make the project remotely? Do we know well the realm or we need to discover it by small iterations of releases? Etc.

What is surprising here is that I recently reviewed a lot of issue tracking / project management tools (whatever you name it), and it looks most of them are stuck to concrete methodology they believe is the best, usually scrum. Even YouTrack discussed here - we just needed to hack over its own approach to achieve what we wanted. YouTrack turned out the most flexible from those we tested, but the question is why do we have dozens of kanban/scrum tools with the same features and almost nothing really configurable? Scrum is good but in my development history it was really applicable in maybe 30% of projects. Is the market still opened for something really adjustable to the environment in which the project is made?

Here you can download all YouTrack workflows mentioned in the article. This package is ready to be imported to YouTrack 7.0 without neccessity of using YouTrack Workflow Editor.