Git/Workflow/Topic

Introduction

This workflow is based on the branchy development workflow documented by git help workflows.

Motivation

The primary goal of this workflow is to make release preparation and maintenance easier. We set the following requirements, of which some are themselves worthwhile goals:

• Amortize the effort of release preparation throughout development
• Support granular selection of features for release
• Allow immature features to be published without delaying release
• Keep unrelated development paths (topics) independent of one another
• Maintain a clean shape of history

Design

The design of this workflow is based on the observation that meeting the highlighted goal makes the other goals easy. It is based on branchy development in which each branch has a well-defined purpose.

We define two branch types:

• Topic Branch
  • Commits represent changes (real work)
  • Distinguished by feature (one topic per feature or fix)
  • Named locally by each developer (describe purpose of work)
  • Heads not published (no named branch on server)
• Integration Branch
  • Commits represent merges (merge topics together)
  • Distinguished by stability (release maintenance, release preparation, development edge)
  • Named everywhere
  • Heads published on server

Notation

This document uses ascii-art to depict commit history:

Branch name	master
Current branch	*master
Commit with parent in same branch	----o
Commit with parent in another branch	\ o
Commit with two parents (merge)	----o /
Commit with arbitrary ancestry	...o

Topic branches generally consist of a linear sequence of commits forked off an integration branch:

...o  master
    \
     o----o----o----o  topic

Integration branches generally consist of a sequence of merge commits:

        ...o      ...o
            \         \
...o----o----o----o----o  master
       /         /
   ...o      ...o

Published Branches

We publish an integration branch for each stage of development:

• main: Release maintenance (high stability). Only bug fixes should be published here. Only the release manager can push here.
• master: Release preparation (medium stability). Only mature features and bug fixes should be published here.
• next: Development (low stability). New features should be published here.

Topic branches are not published directly; their names exist only in each developer's local repositories.

Development

We cover below the steps to take during each phase of development.

Initial Setup

These instructions generally provide all arguments to "git push" commands. Some people prefer to use "git push" with no additional arguments to push the current tracking branch. Run the command

$ git config --global push.default tracking

to establish this behavior. See the git config man-page for details.

New Topic

Create a new topic branch for each separate feature or bug fix. Always start the topic from a stable integration branch, usually master. If the topic fixes a bug in the current release, use maint. Never start a topic from next! In the following table we review the steps and commands to create, develop, and publish a topic branch based on master.

Actions	Results	Troubleshooting
Update master to base work on the most recently integrated features.
$ git checkout master	...o *master
$ git pull	...o----o *master
Create the local topic branch. Use a meaningful name for "topic".
$ git checkout -b topic	...o----o master ^ *topic
This is where the real work happens. Edit, stage, and commit files repeatedly as needed for your work. During this step, avoid the "urge to merge" from an integration branch. Keep your commits focused on the topic at hand.
$ edit files $ git add -- files $ git commit	...o----o master \ o *topic
$ edit files $ git add -- files $ git commit	...o----o master \ o----o *topic
When the topic is ready for publication it must be merged into next. Do not merge from next! It should be the current local branch when you merge. Switch to next and update it. Use "git checkout -b next origin/next" to create a local next the first time.
$ git checkout next $ git pull	...o----o master . \ . o----o topic . ........o *next
Merge the topic and test it.
$ git merge topic	...o----o master . \ . o----o topic . \ ........o----o *next	conflicts?
Finally, publish the change.
$ git push origin next	...o----o master . \ . o----o topic . \ ........o----o *next (and origin/next)	remote end hung up unexpectedly? non-fast-forward?

Mature Topic

When a topic is ready for inclusion in the next release, we merge it into master.

Actions	Results	Troubleshooting
Update master to get the latest work by others. We will merge the topic into it.
$ git checkout master	...o----o *master . \ . o----o topic . \ ........o----o next
$ git pull	.......... . \ ...o----o----o *master . \ . o----o topic . \ ........o----o next
Merge the topic and test it.
$ git merge topic	.......... . \ ...o----o----o---o *master . \ / . o----o topic . \ ........o----o next	conflicts?
Delete the local branch.
$ git branch -d topic	.......... . \ ...o----o----o---o *master . \ / . o----o . \ ........o----o next
Finally, publish the change.
$ git push origin master	.......... . \ ...o----o----o---o *master, origin/master \ / o----o	remote end hung up unexpectedly? non-fast-forward?

Note that master sees only the topics that have been merged into it. It cannot reach any of the merges into next:

  ..........
 .          \
...o----o----o---o *master
         \      /
          o----o (topic)

Old Topic

Sometimes we need to continue work on an old topic that has already been merged to an integration branch and for which we no longer have a local topic branch. To revive an old topic, we create a local branch based on the last commit from the topic (this is not one of the merges into an integration branch).

First we need to identify the commit by its hash. It is an ancestor of the integration branch into which it was once merged, say next. Run git log with the --first-parent option to view the integration history:

$ git log --first-parent next
commit 9057863...
Merge: 2948732 a348901
...
    Merge branch topicA

commit 2948732...
Merge: 1094687 b235725
...
    Merge branch topicB

commit 1094687...
Merge: 8267263 c715789
...
    Merge branch topicC

Locate the merge commit for the topic of interest, say topicB. It's second parent is the commit from which we will restart work (b235725 in this example).

Actions	Results	Troubleshooting
Create a local topic branch starting from the commit identified above.
$ git checkout -b topic b235725	.......... . \ ...o----o----o---o master . \ / . o----o *topic (b235725) . \ ........o----o----o next / / ...o ...o (c715789) (a348901)
Continue development on the topic.
$ edit files $ git add -- files $ git commit	.......... . \ ...o----o----o---o master . \ / . o----o----o *topic . \ ........o----o----o next / / ...o ...o
$ edit files $ git add -- files $ git commit	.......... . \ ...o----o----o---o master . \ / . o----o----o----o *topic . \ ........o----o----o next / / ...o ...o
When the new portion of the topic is ready, merge it into next and test.
$ git checkout next $ git pull $ git merge topic	.......... . \ ...o----o----o---o master . \ / . o----o----o----o topic . \ \ ........o----o----o----o *next / / ...o ...o
Publish next.
$ git push origin next
Optionally repeat the above, publishing to next, to continue development based on feedback from initial publication. Finally, when the new changes are mature, merge to master and publish.
$ git checkout master $ git pull $ git merge topic	.......... . \ ...o----o----o---o---------o *master . \ / / . o----o----o----o topic . \ \ ........o----o----o----o next / / ...o ...o
$ git branch -d topic $ git push origin master	.......... . \ ...o----o----o---o---------o *master . \ / / . o----o----o----o . \ \ ........o----o----o----o next / / ...o ...o

Again, note that master sees only the topics that have been merged into it. It cannot reach any of the merges into next:

  ..........
 .          \
...o----o----o---o---------o *master
         \      /         /
          o----o----o----o (topic)

Dependent Topic

Occasionally you may realize that you need the work from another topic to complete work on your topic. In this case your topic depends on the other topic, so merging the other topic into yours is legitimate. Do not merge an integration branch that has the other topic. Use the instructions below to merge only the other topic without getting everything else.

Actions	Results	Troubleshooting
Fetch the upstream integration branch that has the other-topic branch, say master.
$ git fetch origin	...o (extra-topic) \ \ ...o----o-----o----o origin/master \ \ /^ "Merge branch 'other-topic'" \ o------o 0a398e5 \ o----o *topic
Use git log --first-parent origin/master to find the commit that merges other-topic. The commit message gives you the name of the other topic branch (we use "other-topic" here as a placeholder). The second parent of the commit (0a398e5 in this example) is the end of the other-topic branch. Create a local branch from that commit.
$ git branch other-topic 0a398e5	...o (extra-topic) \ \ ...o----o-----o----o origin/master \ \ / \ o------o other-topic \ o----o *topic
Merge the other branch into your topic.
$ git merge other-topic $ git branch -d other-topic	...o (extra-topic) \ \ ...o----o-----o----o origin/master \ \ / \ o------o \ \ o----o------o *topic ^ "Merge branch 'other-topic' into topic"

(It is also possible to run git merge 0a398e5 and then use git commit --amend to write a nice commit message.)

The topic branch now looks like this:

       \
...o----o
    \    \
     \    o------o  (other-topic)
      \           \
       o----o------o *topic

Note that after the merge, the other-topic is reachable from your topic but the extra-topic has not been included. By not merging from the integration branch we avoided bringing in an unnecessary dependency on the extra-topic. Furthermore, the message "Merge branch 'other-topic' into topic" is very informative about the purpose of the merge. Merging the whole integration branch would not be so clear.

Merge Integration Branches

Each published integration branch has a defined level of stability. Express this relationship by merging more-stable branches into less-stable branches to ensure that they do not diverge. After merging a mature topic to master, we merge master into next:

Actions	Results	Troubleshooting
Update master and then next:
$ git checkout master $ git pull $ git checkout next $ git pull	.......... . \ ...o----o----o---o master . \ / . o----o . \ ........o----o *next
Merge master into next:
$ git merge master	.......... . \ ...o----o----o---o master . \ / \ . o----o \ . \ \ ........o----o---o *next
Finally, publish the change.
$ git push origin next	.......... . \ ...o----o----o---o master, origin/master . \ / \ . o----o \ . \ \ ........o----o---o *next, origin/next	remote end hung up unexpectedly? non-fast-forward?

Discussion

History Shape

The history graphs produced by this workflow may look complex compared to the fully linear history produced by a rebase workflow (used by CVS and Subversion):

  ..........
 .          \
...o----o----o---o---------o  master
 .       \      /         /
  .       o----o----o----o  topic
   .            \         \
    ........o----o----o----o  next
           /         /
       ...o      ...o

However, consider the shape of history along each branch. We can view it using Git's --first-parent option. It traverses history by following only the first parent of each merge. The first parent is the commit that was currently checked out when the git merge command was invoked to create the merge commit. By following only the first parent, we see commits that logically belong to a specific branch.

Command	View
$ git log --first-parent topic	... \ o----o----o----o topic
$ git log --first-parent master	\ ...o----o----o---o---------o master / /
$ git log --first-parent next	\ \ ...o----o----o----o next / /

Each branch by itself looks linear and has only commits with a specific purpose. The history behind each commit is unique to that purpose. Topic branches are independent, containing only commits for their specific feature or fix. Integration branches consist of merge commits that integrate topics together.

Note that achieving the nice separation of branches requires understanding of the above development procedure and strict adherence to it.

Naming Topics

This document uses the italicized placeholder "topic" in place of a real topic name. In practice, substitute a meaningful name. Name topics like you might name functions: concise but precise. A reader should have a general idea of the feature or fix to be developed given just the branch name.

Note that topic names are not published as branch heads on the server, so no one will ever see a branch by your topic name unless they create it themselves. However, the names do appear in the default merge commit message:

$ git checkout next
$ git merge topic
$ git show
...
    Merge branch 'topic' into next
...

These merge commits appear on the integration branches and should therefore describe the changes they integrate. Running git log --first-parent as described here will show only these merge commits, so their messages should be descriptive of the changes made on their topics. If you did not choose a good branch name, or feel that the merge needs more explanation than the branch name provides, amend the commit to update the message by hand:

$ git commit --amend
Merge branch 'topic' into next
(edit the message)

Urge to Merge

Avoid the "urge to merge" from an integration branch into your topic. Keep commits on your topic focused on the feature or fix under development.

Habitual Merges

Merge your work with others when you are finished with it by merging into an integration branch as documented above. Avoid habitual merges from an integration branch; doing so introduces unnecessary dependencies and complicates the shape of history.

Many developers coming from centralized version control systems have trained themselves to regularly update their work tree from the central repository (e.g. "cvs update"). With those version control systems this was a good habit because they did not allow you to commit without first integrating your work with the latest from the server. When integrating the local and remote changes resulted in conflicts, developers were forced to resolve the conflicts before they could commit. A mistake during conflict resolution could result in loss of work because the local changes might have been lost. By regularly updating from the server, developers hoped to avoid this loss of work by resolving conflicts incrementally.

Developers using Git do not face this problem. Instead, one should follow a simple motto: "commit first, integrate later". There is no risk that your work will be lost during conflict resolution because all your changes have been safely committed before attempting to merge. If you make a mistake while merging, you always have the option to throw away the merge attempt and start over with a clean tree.

Legitimate Merges

One reason to merge other work into your topic is when you realize that your topic depends on it. See above for help with this case.

Occasionally one may merge directly from master if there is a good reason. This is rare, so bring up the reason on your project mailing list first. Never merge next into a topic under any circumstances!!!

Troubleshooting

Here we document problems one might encounter while following the workflow instructions above. This is not a general Git troubleshooting page.

Trouble Merging

TODO: Write this sub-section and link sub-sub-sections from git merge commands above.

Trouble Pushing

remote end hung up unexpectedly

Pushing may fail with this error:

$ git push
fatal: The remote end hung up unexpectedly

This likely means that you have set a push URL for the remote repository. You can see the URL to which it tries to push using -v:

$ git push -v
Pushing to git://public.kitware.com/Project.git
fatal: The remote end hung up unexpectedly

The git:// repository URL may not be used for pushing; it is meant for efficient read-only anonymous access only. Instead you need to configure a ssh-protocol URL for pushing:

$ git config remote.origin.pushurl git@public.kitware.com:Project.git

(Note that 'pushurl' requires Git >= 1.6.4. Use just 'url' for Git < 1.6.4.) The URL in the above example is a placeholder. In practice, use the push URL documented for your repository.

The above assumes that you want to push to the same repository that you originally cloned. To push elsewhere, see help for git push and git remote.

non-fast-forward

When trying to publish new merge commits on an integration branch, perhaps next, the final push may fail:

$ git push origin next
To ...
 ! [rejected]        next -> next (non-fast-forward)
error: failed to push some refs to '...'
To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes before pushing again.  See the 'Note about
fast-forwards' section of 'git push --help' for details.

This means that the server's next refers to a commit that is not reachable from the next you are trying to push:

...o----o  master
 .       \
  .       o----o  topic
   .            \
    .       .----o *next
     .     .
      ....o----o  origin/next
              /
          ...o  (other-topic)

This is the Git equivalent to when cvs commit complains that your file is not up-to-date, but now it applies to the whole project and not just one file. Git is telling you that it cannot update next on the server to point at your merge commit because that would throw away someone else's work (such as other-topic). There are a few possible causes, all of which mean you have not yet integrated your work with the latest from upstream:

• You forgot to run git pull before git merge so you didn't have everything from upstream
• Someone else managed to merge and push something into next since you last ran git pull

Some Git guides may tell you to just git pull again to merge upstream work into yours. That approach is not compatible with the goals of this workflow. We want to preserve a clean shape of history.

The solution is to throw away your previous merge and try again, but this time start from the latest upstream work:

Actions	Results
$ git reset --hard origin/next	...o----o master . \ . o----o topic . ...o----o *next, origin/next / ...o (other-topic)
$ git merge topic	...o----o master . \ . o----o topic . \ ...o----o----o *next / ...o (other-topic)
Now your next can reach the upstream work as well as yours. Publish it.
$ git push origin next	...o----o master . \ . o----o topic . \ ...o----o----o *next, origin/next / ...o (other-topic)

See git rerere to help avoid resolving the same conflicts on each merge attempt.

first-parent sequence not preserved

One goal of this workflow is to preserve a clean shape of history. This means that a --first-parent traversal of an integration branch, such as master, should see only the merge commits that integrate topics into the branch:

  \         \
...o----o----o----o  master
       /         /

The commits on the individual topic branches are not included in the traversal. This provides a medium-level overview of the development of the project.

We enforce the shape of history on the server's integration branches using an update hook at push-time. Each update must point its branch at a new commit from which a first-parent traversal reaches the old head of the branch:

                       master@{1}
  \         \         \v
...o----D----C----B----A----M  master
       /         / \       /
                    U-----T  (topic)

A first-parent traversal of master from before the update (master@{1}) sees A B C D:

  \         \         \
...o----D----C----B----A  master@{1}
       /         /

A first-parent traversal of master from after the update sees M A B C D:

  \         \         \
...o----D----C----B----A----M  master
       /         /         /

The above assumes correct history shape. Now, consider what happens if merge M is incorrectly made on the topic branch:

                            master@{1}
  \         \              \v
...o----D----C----B---------A
       /         / \         \
                    U----T----M'  master
                         ^ (topic)

Now a first-parent traversal of master from after the update sees M' T U B C D:

  \         \
...o----D----C----B
       /         / \         \
                    U----T----M'  master
                         ^ (topic)

This not only shows details of the topic branch, but skips over A altogether! Our update hooks will reject the push in this case because the new master cannot see the old one in a first-parent traversal.

There are a few possible causes and solutions to the above problem, but all involve non-strict compliance with the workflow instructions. A likely cause is that you did not create a local topic branch but instead committed directly on master and then pulled from upstream before pushing:

(Wrong) Actions	Results
wrong$ git checkout master	\ \ ...o----D----C----B *master, origin/master / /
wrong$ edit files wrong$ git add files wrong$ git commit	\ \ ...o----D----C----B origin/master / / \ U *master
wrong$ edit files wrong$ git add files wrong$ git commit	\ \ ...o----D----C----B origin/master / / \ U----T *master
wrong$ git push origin master	Rejected as non-fast-forward.
wrong$ git pull	\ \ ...o----D----C----B---------A origin/master / / \ \ U----T----M' *master
wrong$ git push origin master	Rejected with this error.

The solution in this case is to recreate the merge on the proper branch.

Actions	Results
First, create a nicely-named topic branch starting from the first-parent of the incorrect merge.
$ git branch topic 'master^1'	\ \ ...o----D----C----B---------A origin/master / / \ \ U----T----M' *master ^ topic
Then reset your local master to that from upstream.
$ git reset --hard origin/master	\ \ ...o----D----C----B----A *master, origin/master / / \ U-----T topic
Now create the correct merge commit as described in the workflow instructions above.
$ git merge topic	\ \ ...o----D----C----B----A----M *master / / \ / U-----T topic
$ git push origin master $ git branch -d topic	\ \ ...o----D----C----B----A----M *master, origin/master / / \ / U-----T

topics must be merged

TODO

Note: I was referred to this documentation when my merge to master failed a pre-commit hook. The fix was to add "--no-ff" arg when merging the topic.