704 lines
		
	
	
		
			30 KiB
		
	
	
	
		
			Plaintext
		
	
	
			
		
		
	
	
			704 lines
		
	
	
		
			30 KiB
		
	
	
	
		
			Plaintext
		
	
	
| git-filter-branch(1)
 | |
| ====================
 | |
| 
 | |
| NAME
 | |
| ----
 | |
| git-filter-branch - Rewrite branches
 | |
| 
 | |
| SYNOPSIS
 | |
| --------
 | |
| [verse]
 | |
| 'git filter-branch' [--setup <command>] [--subdirectory-filter <directory>]
 | |
| 	[--env-filter <command>] [--tree-filter <command>]
 | |
| 	[--index-filter <command>] [--parent-filter <command>]
 | |
| 	[--msg-filter <command>] [--commit-filter <command>]
 | |
| 	[--tag-name-filter <command>] [--prune-empty]
 | |
| 	[--original <namespace>] [-d <directory>] [-f | --force]
 | |
| 	[--state-branch <branch>] [--] [<rev-list-options>...]
 | |
| 
 | |
| WARNING
 | |
| -------
 | |
| 'git filter-branch' has a plethora of pitfalls that can produce non-obvious
 | |
| manglings of the intended history rewrite (and can leave you with little
 | |
| time to investigate such problems since it has such abysmal performance).
 | |
| These safety and performance issues cannot be backward compatibly fixed and
 | |
| as such, its use is not recommended.  Please use an alternative history
 | |
| filtering tool such as https://github.com/newren/git-filter-repo/[git
 | |
| filter-repo].  If you still need to use 'git filter-branch', please
 | |
| carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land
 | |
| mines of filter-branch, and then vigilantly avoid as many of the hazards
 | |
| listed there as reasonably possible.
 | |
| 
 | |
| DESCRIPTION
 | |
| -----------
 | |
| Lets you rewrite Git revision history by rewriting the branches mentioned
 | |
| in the <rev-list-options>, applying custom filters on each revision.
 | |
| Those filters can modify each tree (e.g. removing a file or running
 | |
| a perl rewrite on all files) or information about each commit.
 | |
| Otherwise, all information (including original commit times or merge
 | |
| information) will be preserved.
 | |
| 
 | |
| The command will only rewrite the _positive_ refs mentioned in the
 | |
| command line (e.g. if you pass 'a..b', only 'b' will be rewritten).
 | |
| If you specify no filters, the commits will be recommitted without any
 | |
| changes, which would normally have no effect.  Nevertheless, this may be
 | |
| useful in the future for compensating for some Git bugs or such,
 | |
| therefore such a usage is permitted.
 | |
| 
 | |
| *NOTE*: This command honors `.git/info/grafts` file and refs in
 | |
| the `refs/replace/` namespace.
 | |
| If you have any grafts or replacement refs defined, running this command
 | |
| will make them permanent.
 | |
| 
 | |
| *WARNING*! The rewritten history will have different object names for all
 | |
| the objects and will not converge with the original branch.  You will not
 | |
| be able to easily push and distribute the rewritten branch on top of the
 | |
| original branch.  Please do not use this command if you do not know the
 | |
| full implications, and avoid using it anyway, if a simple single commit
 | |
| would suffice to fix your problem.  (See the "RECOVERING FROM UPSTREAM
 | |
| REBASE" section in linkgit:git-rebase[1] for further information about
 | |
| rewriting published history.)
 | |
| 
 | |
| Always verify that the rewritten version is correct: The original refs,
 | |
| if different from the rewritten ones, will be stored in the namespace
 | |
| 'refs/original/'.
 | |
| 
 | |
| Note that since this operation is very I/O expensive, it might
 | |
| be a good idea to redirect the temporary directory off-disk with the
 | |
| `-d` option, e.g. on tmpfs.  Reportedly the speedup is very noticeable.
 | |
| 
 | |
| 
 | |
| Filters
 | |
| ~~~~~~~
 | |
| 
 | |
| The filters are applied in the order as listed below.  The <command>
 | |
| argument is always evaluated in the shell context using the 'eval' command
 | |
| (with the notable exception of the commit filter, for technical reasons).
 | |
| Prior to that, the `$GIT_COMMIT` environment variable will be set to contain
 | |
| the id of the commit being rewritten.  Also, GIT_AUTHOR_NAME,
 | |
| GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL,
 | |
| and GIT_COMMITTER_DATE are taken from the current commit and exported to
 | |
| the environment, in order to affect the author and committer identities of
 | |
| the replacement commit created by linkgit:git-commit-tree[1] after the
 | |
| filters have run.
 | |
| 
 | |
| If any evaluation of <command> returns a non-zero exit status, the whole
 | |
| operation will be aborted.
 | |
| 
 | |
| A 'map' function is available that takes an "original sha1 id" argument
 | |
| and outputs a "rewritten sha1 id" if the commit has been already
 | |
| rewritten, and "original sha1 id" otherwise; the 'map' function can
 | |
| return several ids on separate lines if your commit filter emitted
 | |
| multiple commits.
 | |
| 
 | |
| 
 | |
| OPTIONS
 | |
| -------
 | |
| 
 | |
| --setup <command>::
 | |
| 	This is not a real filter executed for each commit but a one
 | |
| 	time setup just before the loop. Therefore no commit-specific
 | |
| 	variables are defined yet.  Functions or variables defined here
 | |
| 	can be used or modified in the following filter steps except
 | |
| 	the commit filter, for technical reasons.
 | |
| 
 | |
| --subdirectory-filter <directory>::
 | |
| 	Only look at the history which touches the given subdirectory.
 | |
| 	The result will contain that directory (and only that) as its
 | |
| 	project root. Implies <<Remap_to_ancestor>>.
 | |
| 
 | |
| --env-filter <command>::
 | |
| 	This filter may be used if you only need to modify the environment
 | |
| 	in which the commit will be performed.  Specifically, you might
 | |
| 	want to rewrite the author/committer name/email/time environment
 | |
| 	variables (see linkgit:git-commit-tree[1] for details).
 | |
| 
 | |
| --tree-filter <command>::
 | |
| 	This is the filter for rewriting the tree and its contents.
 | |
| 	The argument is evaluated in shell with the working
 | |
| 	directory set to the root of the checked out tree.  The new tree
 | |
| 	is then used as-is (new files are auto-added, disappeared files
 | |
| 	are auto-removed - neither .gitignore files nor any other ignore
 | |
| 	rules *HAVE ANY EFFECT*!).
 | |
| 
 | |
| --index-filter <command>::
 | |
| 	This is the filter for rewriting the index.  It is similar to the
 | |
| 	tree filter but does not check out the tree, which makes it much
 | |
| 	faster.  Frequently used with `git rm --cached
 | |
| 	--ignore-unmatch ...`, see EXAMPLES below.  For hairy
 | |
| 	cases, see linkgit:git-update-index[1].
 | |
| 
 | |
| --parent-filter <command>::
 | |
| 	This is the filter for rewriting the commit's parent list.
 | |
| 	It will receive the parent string on stdin and shall output
 | |
| 	the new parent string on stdout.  The parent string is in
 | |
| 	the format described in linkgit:git-commit-tree[1]: empty for
 | |
| 	the initial commit, "-p parent" for a normal commit and
 | |
| 	"-p parent1 -p parent2 -p parent3 ..." for a merge commit.
 | |
| 
 | |
| --msg-filter <command>::
 | |
| 	This is the filter for rewriting the commit messages.
 | |
| 	The argument is evaluated in the shell with the original
 | |
| 	commit message on standard input; its standard output is
 | |
| 	used as the new commit message.
 | |
| 
 | |
| --commit-filter <command>::
 | |
| 	This is the filter for performing the commit.
 | |
| 	If this filter is specified, it will be called instead of the
 | |
| 	'git commit-tree' command, with arguments of the form
 | |
| 	"<TREE_ID> [(-p <PARENT_COMMIT_ID>)...]" and the log message on
 | |
| 	stdin.  The commit id is expected on stdout.
 | |
| +
 | |
| As a special extension, the commit filter may emit multiple
 | |
| commit ids; in that case, the rewritten children of the original commit will
 | |
| have all of them as parents.
 | |
| +
 | |
| You can use the 'map' convenience function in this filter, and other
 | |
| convenience functions, too.  For example, calling 'skip_commit "$@"'
 | |
| will leave out the current commit (but not its changes! If you want
 | |
| that, use 'git rebase' instead).
 | |
| +
 | |
| You can also use the `git_commit_non_empty_tree "$@"` instead of
 | |
| `git commit-tree "$@"` if you don't wish to keep commits with a single parent
 | |
| and that makes no change to the tree.
 | |
| 
 | |
| --tag-name-filter <command>::
 | |
| 	This is the filter for rewriting tag names. When passed,
 | |
| 	it will be called for every tag ref that points to a rewritten
 | |
| 	object (or to a tag object which points to a rewritten object).
 | |
| 	The original tag name is passed via standard input, and the new
 | |
| 	tag name is expected on standard output.
 | |
| +
 | |
| The original tags are not deleted, but can be overwritten;
 | |
| use "--tag-name-filter cat" to simply update the tags.  In this
 | |
| case, be very careful and make sure you have the old tags
 | |
| backed up in case the conversion has run afoul.
 | |
| +
 | |
| Nearly proper rewriting of tag objects is supported. If the tag has
 | |
| a message attached, a new tag object will be created with the same message,
 | |
| author, and timestamp. If the tag has a signature attached, the
 | |
| signature will be stripped. It is by definition impossible to preserve
 | |
| signatures. The reason this is "nearly" proper, is because ideally if
 | |
| the tag did not change (points to the same object, has the same name, etc.)
 | |
| it should retain any signature. That is not the case, signatures will always
 | |
| be removed, buyer beware. There is also no support for changing the
 | |
| author or timestamp (or the tag message for that matter). Tags which point
 | |
| to other tags will be rewritten to point to the underlying commit.
 | |
| 
 | |
| --prune-empty::
 | |
| 	Some filters will generate empty commits that leave the tree untouched.
 | |
| 	This option instructs git-filter-branch to remove such commits if they
 | |
| 	have exactly one or zero non-pruned parents; merge commits will
 | |
| 	therefore remain intact.  This option cannot be used together with
 | |
| 	`--commit-filter`, though the same effect can be achieved by using the
 | |
| 	provided `git_commit_non_empty_tree` function in a commit filter.
 | |
| 
 | |
| --original <namespace>::
 | |
| 	Use this option to set the namespace where the original commits
 | |
| 	will be stored. The default value is 'refs/original'.
 | |
| 
 | |
| -d <directory>::
 | |
| 	Use this option to set the path to the temporary directory used for
 | |
| 	rewriting.  When applying a tree filter, the command needs to
 | |
| 	temporarily check out the tree to some directory, which may consume
 | |
| 	considerable space in case of large projects.  By default it
 | |
| 	does this in the `.git-rewrite/` directory but you can override
 | |
| 	that choice by this parameter.
 | |
| 
 | |
| -f::
 | |
| --force::
 | |
| 	'git filter-branch' refuses to start with an existing temporary
 | |
| 	directory or when there are already refs starting with
 | |
| 	'refs/original/', unless forced.
 | |
| 
 | |
| --state-branch <branch>::
 | |
| 	This option will cause the mapping from old to new objects to
 | |
| 	be loaded from named branch upon startup and saved as a new
 | |
| 	commit to that branch upon exit, enabling incremental of large
 | |
| 	trees. If '<branch>' does not exist it will be created.
 | |
| 
 | |
| <rev-list options>...::
 | |
| 	Arguments for 'git rev-list'.  All positive refs included by
 | |
| 	these options are rewritten.  You may also specify options
 | |
| 	such as `--all`, but you must use `--` to separate them from
 | |
| 	the 'git filter-branch' options. Implies <<Remap_to_ancestor>>.
 | |
| 
 | |
| 
 | |
| [[Remap_to_ancestor]]
 | |
| Remap to ancestor
 | |
| ~~~~~~~~~~~~~~~~~
 | |
| 
 | |
| By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the
 | |
| set of revisions which get rewritten. However, positive refs on the command
 | |
| line are distinguished: we don't let them be excluded by such limiters. For
 | |
| this purpose, they are instead rewritten to point at the nearest ancestor that
 | |
| was not excluded.
 | |
| 
 | |
| 
 | |
| EXIT STATUS
 | |
| -----------
 | |
| 
 | |
| On success, the exit status is `0`.  If the filter can't find any commits to
 | |
| rewrite, the exit status is `2`.  On any other error, the exit status may be
 | |
| any other non-zero value.
 | |
| 
 | |
| 
 | |
| EXAMPLES
 | |
| --------
 | |
| 
 | |
| Suppose you want to remove a file (containing confidential information
 | |
| or copyright violation) from all commits:
 | |
| 
 | |
| -------------------------------------------------------
 | |
| git filter-branch --tree-filter 'rm filename' HEAD
 | |
| -------------------------------------------------------
 | |
| 
 | |
| However, if the file is absent from the tree of some commit,
 | |
| a simple `rm filename` will fail for that tree and commit.
 | |
| Thus you may instead want to use `rm -f filename` as the script.
 | |
| 
 | |
| Using `--index-filter` with 'git rm' yields a significantly faster
 | |
| version.  Like with using `rm filename`, `git rm --cached filename`
 | |
| will fail if the file is absent from the tree of a commit.  If you
 | |
| want to "completely forget" a file, it does not matter when it entered
 | |
| history, so we also add `--ignore-unmatch`:
 | |
| 
 | |
| --------------------------------------------------------------------------
 | |
| git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD
 | |
| --------------------------------------------------------------------------
 | |
| 
 | |
| Now, you will get the rewritten history saved in HEAD.
 | |
| 
 | |
| To rewrite the repository to look as if `foodir/` had been its project
 | |
| root, and discard all other history:
 | |
| 
 | |
| -------------------------------------------------------
 | |
| git filter-branch --subdirectory-filter foodir -- --all
 | |
| -------------------------------------------------------
 | |
| 
 | |
| Thus you can, e.g., turn a library subdirectory into a repository of
 | |
| its own.  Note the `--` that separates 'filter-branch' options from
 | |
| revision options, and the `--all` to rewrite all branches and tags.
 | |
| 
 | |
| To set a commit (which typically is at the tip of another
 | |
| history) to be the parent of the current initial commit, in
 | |
| order to paste the other history behind the current history:
 | |
| 
 | |
| -------------------------------------------------------------------
 | |
| git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD
 | |
| -------------------------------------------------------------------
 | |
| 
 | |
| (if the parent string is empty - which happens when we are dealing with
 | |
| the initial commit - add graftcommit as a parent).  Note that this assumes
 | |
| history with a single root (that is, no merge without common ancestors
 | |
| happened).  If this is not the case, use:
 | |
| 
 | |
| --------------------------------------------------------------------------
 | |
| git filter-branch --parent-filter \
 | |
| 	'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD
 | |
| --------------------------------------------------------------------------
 | |
| 
 | |
| or even simpler:
 | |
| 
 | |
| -----------------------------------------------
 | |
| git replace --graft $commit-id $graft-id
 | |
| git filter-branch $graft-id..HEAD
 | |
| -----------------------------------------------
 | |
| 
 | |
| To remove commits authored by "Darl McBribe" from the history:
 | |
| 
 | |
| ------------------------------------------------------------------------------
 | |
| git filter-branch --commit-filter '
 | |
| 	if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ];
 | |
| 	then
 | |
| 		skip_commit "$@";
 | |
| 	else
 | |
| 		git commit-tree "$@";
 | |
| 	fi' HEAD
 | |
| ------------------------------------------------------------------------------
 | |
| 
 | |
| The function 'skip_commit' is defined as follows:
 | |
| 
 | |
| --------------------------
 | |
| skip_commit()
 | |
| {
 | |
| 	shift;
 | |
| 	while [ -n "$1" ];
 | |
| 	do
 | |
| 		shift;
 | |
| 		map "$1";
 | |
| 		shift;
 | |
| 	done;
 | |
| }
 | |
| --------------------------
 | |
| 
 | |
| The shift magic first throws away the tree id and then the -p
 | |
| parameters.  Note that this handles merges properly! In case Darl
 | |
| committed a merge between P1 and P2, it will be propagated properly
 | |
| and all children of the merge will become merge commits with P1,P2
 | |
| as their parents instead of the merge commit.
 | |
| 
 | |
| *NOTE* the changes introduced by the commits, and which are not reverted
 | |
| by subsequent commits, will still be in the rewritten branch. If you want
 | |
| to throw out _changes_ together with the commits, you should use the
 | |
| interactive mode of 'git rebase'.
 | |
| 
 | |
| You can rewrite the commit log messages using `--msg-filter`.  For
 | |
| example, 'git svn-id' strings in a repository created by 'git svn' can
 | |
| be removed this way:
 | |
| 
 | |
| -------------------------------------------------------
 | |
| git filter-branch --msg-filter '
 | |
| 	sed -e "/^git-svn-id:/d"
 | |
| '
 | |
| -------------------------------------------------------
 | |
| 
 | |
| If you need to add 'Acked-by' lines to, say, the last 10 commits (none
 | |
| of which is a merge), use this command:
 | |
| 
 | |
| --------------------------------------------------------
 | |
| git filter-branch --msg-filter '
 | |
| 	cat &&
 | |
| 	echo "Acked-by: Bugs Bunny <bunny@bugzilla.org>"
 | |
| ' HEAD~10..HEAD
 | |
| --------------------------------------------------------
 | |
| 
 | |
| The `--env-filter` option can be used to modify committer and/or author
 | |
| identity.  For example, if you found out that your commits have the wrong
 | |
| identity due to a misconfigured user.email, you can make a correction,
 | |
| before publishing the project, like this:
 | |
| 
 | |
| --------------------------------------------------------
 | |
| git filter-branch --env-filter '
 | |
| 	if test "$GIT_AUTHOR_EMAIL" = "root@localhost"
 | |
| 	then
 | |
| 		GIT_AUTHOR_EMAIL=john@example.com
 | |
| 	fi
 | |
| 	if test "$GIT_COMMITTER_EMAIL" = "root@localhost"
 | |
| 	then
 | |
| 		GIT_COMMITTER_EMAIL=john@example.com
 | |
| 	fi
 | |
| ' -- --all
 | |
| --------------------------------------------------------
 | |
| 
 | |
| To restrict rewriting to only part of the history, specify a revision
 | |
| range in addition to the new branch name.  The new branch name will
 | |
| point to the top-most revision that a 'git rev-list' of this range
 | |
| will print.
 | |
| 
 | |
| Consider this history:
 | |
| 
 | |
| ------------------
 | |
|      D--E--F--G--H
 | |
|     /     /
 | |
| A--B-----C
 | |
| ------------------
 | |
| 
 | |
| To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use:
 | |
| 
 | |
| --------------------------------
 | |
| git filter-branch ... C..H
 | |
| --------------------------------
 | |
| 
 | |
| To rewrite commits E,F,G,H, use one of these:
 | |
| 
 | |
| ----------------------------------------
 | |
| git filter-branch ... C..H --not D
 | |
| git filter-branch ... D..H --not C
 | |
| ----------------------------------------
 | |
| 
 | |
| To move the whole tree into a subdirectory, or remove it from there:
 | |
| 
 | |
| ---------------------------------------------------------------
 | |
| git filter-branch --index-filter \
 | |
| 	'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
 | |
| 		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
 | |
| 			git update-index --index-info &&
 | |
| 	 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
 | |
| ---------------------------------------------------------------
 | |
| 
 | |
| 
 | |
| 
 | |
| CHECKLIST FOR SHRINKING A REPOSITORY
 | |
| ------------------------------------
 | |
| 
 | |
| git-filter-branch can be used to get rid of a subset of files,
 | |
| usually with some combination of `--index-filter` and
 | |
| `--subdirectory-filter`.  People expect the resulting repository to
 | |
| be smaller than the original, but you need a few more steps to
 | |
| actually make it smaller, because Git tries hard not to lose your
 | |
| objects until you tell it to.  First make sure that:
 | |
| 
 | |
| * You really removed all variants of a filename, if a blob was moved
 | |
|   over its lifetime.  `git log --name-only --follow --all -- filename`
 | |
|   can help you find renames.
 | |
| 
 | |
| * You really filtered all refs: use `--tag-name-filter cat -- --all`
 | |
|   when calling git-filter-branch.
 | |
| 
 | |
| Then there are two ways to get a smaller repository.  A safer way is
 | |
| to clone, that keeps your original intact.
 | |
| 
 | |
| * Clone it with `git clone file:///path/to/repo`.  The clone
 | |
|   will not have the removed objects.  See linkgit:git-clone[1].  (Note
 | |
|   that cloning with a plain path just hardlinks everything!)
 | |
| 
 | |
| If you really don't want to clone it, for whatever reasons, check the
 | |
| following points instead (in this order).  This is a very destructive
 | |
| approach, so *make a backup* or go back to cloning it.  You have been
 | |
| warned.
 | |
| 
 | |
| * Remove the original refs backed up by git-filter-branch: say `git
 | |
|   for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git
 | |
|   update-ref -d`.
 | |
| 
 | |
| * Expire all reflogs with `git reflog expire --expire=now --all`.
 | |
| 
 | |
| * Garbage collect all unreferenced objects with `git gc --prune=now`
 | |
|   (or if your git-gc is not new enough to support arguments to
 | |
|   `--prune`, use `git repack -ad; git prune` instead).
 | |
| 
 | |
| [[PERFORMANCE]]
 | |
| PERFORMANCE
 | |
| -----------
 | |
| 
 | |
| The performance of git-filter-branch is glacially slow; its design makes it
 | |
| impossible for a backward-compatible implementation to ever be fast:
 | |
| 
 | |
| * In editing files, git-filter-branch by design checks out each and
 | |
|   every commit as it existed in the original repo.  If your repo has
 | |
|   `10^5` files and `10^5` commits, but each commit only modifies five
 | |
|   files, then git-filter-branch will make you do `10^10` modifications,
 | |
|   despite only having (at most) `5*10^5` unique blobs.
 | |
| 
 | |
| * If you try and cheat and try to make git-filter-branch only work on
 | |
|   files modified in a commit, then two things happen
 | |
| 
 | |
|   ** you run into problems with deletions whenever the user is simply
 | |
|      trying to rename files (because attempting to delete files that
 | |
|      don't exist looks like a no-op; it takes some chicanery to remap
 | |
|      deletes across file renames when the renames happen via arbitrary
 | |
|      user-provided shell)
 | |
| 
 | |
|   ** even if you succeed at the map-deletes-for-renames chicanery, you
 | |
|      still technically violate backward compatibility because users
 | |
|      are allowed to filter files in ways that depend upon topology of
 | |
|      commits instead of filtering solely based on file contents or
 | |
|      names (though this has not been observed in the wild).
 | |
| 
 | |
| * Even if you don't need to edit files but only want to e.g. rename or
 | |
|   remove some and thus can avoid checking out each file (i.e. you can
 | |
|   use --index-filter), you still are passing shell snippets for your
 | |
|   filters.  This means that for every commit, you have to have a
 | |
|   prepared git repo where those filters can be run.  That's a
 | |
|   significant setup.
 | |
| 
 | |
| * Further, several additional files are created or updated per commit
 | |
|   by git-filter-branch.  Some of these are for supporting the
 | |
|   convenience functions provided by git-filter-branch (such as map()),
 | |
|   while others are for keeping track of internal state (but could have
 | |
|   also been accessed by user filters; one of git-filter-branch's
 | |
|   regression tests does so).  This essentially amounts to using the
 | |
|   filesystem as an IPC mechanism between git-filter-branch and the
 | |
|   user-provided filters.  Disks tend to be a slow IPC mechanism, and
 | |
|   writing these files also effectively represents a forced
 | |
|   synchronization point between separate processes that we hit with
 | |
|   every commit.
 | |
| 
 | |
| * The user-provided shell commands will likely involve a pipeline of
 | |
|   commands, resulting in the creation of many processes per commit.
 | |
|   Creating and running another process takes a widely varying amount
 | |
|   of time between operating systems, but on any platform it is very
 | |
|   slow relative to invoking a function.
 | |
| 
 | |
| * git-filter-branch itself is written in shell, which is kind of slow.
 | |
|   This is the one performance issue that could be backward-compatibly
 | |
|   fixed, but compared to the above problems that are intrinsic to the
 | |
|   design of git-filter-branch, the language of the tool itself is a
 | |
|   relatively minor issue.
 | |
| 
 | |
|   ** Side note: Unfortunately, people tend to fixate on the
 | |
|      written-in-shell aspect and periodically ask if git-filter-branch
 | |
|      could be rewritten in another language to fix the performance
 | |
|      issues.  Not only does that ignore the bigger intrinsic problems
 | |
|      with the design, it'd help less than you'd expect: if
 | |
|      git-filter-branch itself were not shell, then the convenience
 | |
|      functions (map(), skip_commit(), etc) and the `--setup` argument
 | |
|      could no longer be executed once at the beginning of the program
 | |
|      but would instead need to be prepended to every user filter (and
 | |
|      thus re-executed with every commit).
 | |
| 
 | |
| The https://github.com/newren/git-filter-repo/[git filter-repo] tool is
 | |
| an alternative to git-filter-branch which does not suffer from these
 | |
| performance problems or the safety problems (mentioned below). For those
 | |
| with existing tooling which relies upon git-filter-branch, 'git
 | |
| filter-repo' also provides
 | |
| https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],
 | |
| a drop-in git-filter-branch replacement (with a few caveats).  While
 | |
| filter-lamely suffers from all the same safety issues as
 | |
| git-filter-branch, it at least ameliorates the performance issues a
 | |
| little.
 | |
| 
 | |
| [[SAFETY]]
 | |
| SAFETY
 | |
| ------
 | |
| 
 | |
| git-filter-branch is riddled with gotchas resulting in various ways to
 | |
| easily corrupt repos or end up with a mess worse than what you started
 | |
| with:
 | |
| 
 | |
| * Someone can have a set of "working and tested filters" which they
 | |
|   document or provide to a coworker, who then runs them on a different
 | |
|   OS where the same commands are not working/tested (some examples in
 | |
|   the git-filter-branch manpage are also affected by this).
 | |
|   BSD vs. GNU userland differences can really bite.  If lucky, error
 | |
|   messages are spewed.  But just as likely, the commands either don't
 | |
|   do the filtering requested, or silently corrupt by making some
 | |
|   unwanted change.  The unwanted change may only affect a few commits,
 | |
|   so it's not necessarily obvious either.  (The fact that problems
 | |
|   won't necessarily be obvious means they are likely to go unnoticed
 | |
|   until the rewritten history is in use for quite a while, at which
 | |
|   point it's really hard to justify another flag-day for another
 | |
|   rewrite.)
 | |
| 
 | |
| * Filenames with spaces are often mishandled by shell snippets since
 | |
|   they cause problems for shell pipelines.  Not everyone is familiar
 | |
|   with find -print0, xargs -0, git-ls-files -z, etc.  Even people who
 | |
|   are familiar with these may assume such flags are not relevant
 | |
|   because someone else renamed any such files in their repo back
 | |
|   before the person doing the filtering joined the project.  And
 | |
|   often, even those familiar with handling arguments with spaces may
 | |
|   not do so just because they aren't in the mindset of thinking about
 | |
|   everything that could possibly go wrong.
 | |
| 
 | |
| * Non-ascii filenames can be silently removed despite being in a
 | |
|   desired directory.  Keeping only wanted paths is often done using
 | |
|   pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.
 | |
|   ls-files will only quote filenames if needed, so folks may not
 | |
|   notice that one of the files didn't match the regex (at least not
 | |
|   until it's much too late).  Yes, someone who knows about
 | |
|   core.quotePath can avoid this (unless they have other special
 | |
|   characters like \t, \n, or "), and people who use ls-files -z with
 | |
|   something other than grep can avoid this, but that doesn't mean they
 | |
|   will.
 | |
| 
 | |
| * Similarly, when moving files around, one can find that filenames
 | |
|   with non-ascii or special characters end up in a different
 | |
|   directory, one that includes a double quote character.  (This is
 | |
|   technically the same issue as above with quoting, but perhaps an
 | |
|   interesting different way that it can and has manifested as a
 | |
|   problem.)
 | |
| 
 | |
| * It's far too easy to accidentally mix up old and new history.  It's
 | |
|   still possible with any tool, but git-filter-branch almost
 | |
|   invites it.  If lucky, the only downside is users getting frustrated
 | |
|   that they don't know how to shrink their repo and remove the old
 | |
|   stuff.  If unlucky, they merge old and new history and end up with
 | |
|   multiple "copies" of each commit, some of which have unwanted or
 | |
|   sensitive files and others which don't.  This comes about in
 | |
|   multiple different ways:
 | |
| 
 | |
|   ** the default to only doing a partial history rewrite ('--all' is not
 | |
|      the default and few examples show it)
 | |
| 
 | |
|   ** the fact that there's no automatic post-run cleanup
 | |
| 
 | |
|   ** the fact that --tag-name-filter (when used to rename tags) doesn't
 | |
|      remove the old tags but just adds new ones with the new name
 | |
| 
 | |
|   ** the fact that little educational information is provided to inform
 | |
|      users of the ramifications of a rewrite and how to avoid mixing old
 | |
|      and new history.  For example, this man page discusses how users
 | |
|      need to understand that they need to rebase their changes for all
 | |
|      their branches on top of new history (or delete and reclone), but
 | |
|      that's only one of multiple concerns to consider.  See the
 | |
|      "DISCUSSION" section of the git filter-repo manual page for more
 | |
|      details.
 | |
| 
 | |
| * Annotated tags can be accidentally converted to lightweight tags,
 | |
|   due to either of two issues:
 | |
| 
 | |
|   ** Someone can do a history rewrite, realize they messed up, restore
 | |
|      from the backups in refs/original/, and then redo their
 | |
|      git-filter-branch command.  (The backup in refs/original/ is not a
 | |
|      real backup; it dereferences tags first.)
 | |
| 
 | |
|   ** Running git-filter-branch with either --tags or --all in your
 | |
|      <rev-list-options>.  In order to retain annotated tags as
 | |
|      annotated, you must use --tag-name-filter (and must not have
 | |
|      restored from refs/original/ in a previously botched rewrite).
 | |
| 
 | |
| * Any commit messages that specify an encoding will become corrupted
 | |
|   by the rewrite; git-filter-branch ignores the encoding, takes the
 | |
|   original bytes, and feeds it to commit-tree without telling it the
 | |
|   proper encoding.  (This happens whether or not --msg-filter is
 | |
|   used.)
 | |
| 
 | |
| * Commit messages (even if they are all UTF-8) by default become
 | |
|   corrupted due to not being updated -- any references to other commit
 | |
|   hashes in commit messages will now refer to no-longer-extant
 | |
|   commits.
 | |
| 
 | |
| * There are no facilities for helping users find what unwanted crud
 | |
|   they should delete, which means they are much more likely to have
 | |
|   incomplete or partial cleanups that sometimes result in confusion
 | |
|   and people wasting time trying to understand.  (For example, folks
 | |
|   tend to just look for big files to delete instead of big directories
 | |
|   or extensions, and once they do so, then sometime later folks using
 | |
|   the new repository who are going through history will notice a build
 | |
|   artifact directory that has some files but not others, or a cache of
 | |
|   dependencies (node_modules or similar) which couldn't have ever been
 | |
|   functional since it's missing some files.)
 | |
| 
 | |
| * If --prune-empty isn't specified, then the filtering process can
 | |
|   create hoards of confusing empty commits
 | |
| 
 | |
| * If --prune-empty is specified, then intentionally placed empty
 | |
|   commits from before the filtering operation are also pruned instead
 | |
|   of just pruning commits that became empty due to filtering rules.
 | |
| 
 | |
| * If --prune-empty is specified, sometimes empty commits are missed
 | |
|   and left around anyway (a somewhat rare bug, but it happens...)
 | |
| 
 | |
| * A minor issue, but users who have a goal to update all names and
 | |
|   emails in a repository may be led to --env-filter which will only
 | |
|   update authors and committers, missing taggers.
 | |
| 
 | |
| * If the user provides a --tag-name-filter that maps multiple tags to
 | |
|   the same name, no warning or error is provided; git-filter-branch
 | |
|   simply overwrites each tag in some undocumented pre-defined order
 | |
|   resulting in only one tag at the end.  (A git-filter-branch
 | |
|   regression test requires this surprising behavior.)
 | |
| 
 | |
| Also, the poor performance of git-filter-branch often leads to safety
 | |
| issues:
 | |
| 
 | |
| * Coming up with the correct shell snippet to do the filtering you
 | |
|   want is sometimes difficult unless you're just doing a trivial
 | |
|   modification such as deleting a couple files.  Unfortunately, people
 | |
|   often learn if the snippet is right or wrong by trying it out, but
 | |
|   the rightness or wrongness can vary depending on special
 | |
|   circumstances (spaces in filenames, non-ascii filenames, funny
 | |
|   author names or emails, invalid timezones, presence of grafts or
 | |
|   replace objects, etc.), meaning they may have to wait a long time,
 | |
|   hit an error, then restart.  The performance of git-filter-branch is
 | |
|   so bad that this cycle is painful, reducing the time available to
 | |
|   carefully re-check (to say nothing about what it does to the
 | |
|   patience of the person doing the rewrite even if they do technically
 | |
|   have more time available).  This problem is extra compounded because
 | |
|   errors from broken filters may not be shown for a long time and/or
 | |
|   get lost in a sea of output.  Even worse, broken filters often just
 | |
|   result in silent incorrect rewrites.
 | |
| 
 | |
| * To top it all off, even when users finally find working commands,
 | |
|   they naturally want to share them.  But they may be unaware that
 | |
|   their repo didn't have some special cases that someone else's does.
 | |
|   So, when someone else with a different repository runs the same
 | |
|   commands, they get hit by the problems above.  Or, the user just
 | |
|   runs commands that really were vetted for special cases, but they
 | |
|   run it on a different OS where it doesn't work, as noted above.
 | |
| 
 | |
| GIT
 | |
| ---
 | |
| Part of the linkgit:git[1] suite
 |