In a simple test, this brings down the CPU time from 47 sec to 22 sec.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The d_ino field is only used for performance reasons in
fsck-objects. On a typical filesystem, i-number tends to have a
strong correlation with where the actual bits sit on the disk
platter, and we sort the entries to allow us scan things that
ought to be close together together.
If the platform lacks support for it, it is not a big deal.
Just do not use d_ino for sorting, and scan them unsorted.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Store pointers to referenced objects in a variable sized array instead
of linked list. This cuts down memory usage of utilities which use
object references; e.g., git-fsck-objects --full on the git.git
repository consumes about 2 MB of memory tracked by Massif instead of
7 MB before the change. Object refs are still the biggest consumer of
memory (57%), but the malloc overhead for a single block instead of a
linked list is substantially smaller.
Signed-off-by: Sergey Vlasov <vsu@altlinux.ru>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The Massif tool of Valgrind revealed that parsed tree entries occupy
more than 60% of memory allocated by git-fsck-objects. These entries
can be freed immediately after use, which significantly decreases
memory consumption.
Signed-off-by: Sergey Vlasov <vsu@altlinux.ru>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds the counterpart of git-update-ref that lets you read
and create "symbolic refs". By default it uses a symbolic link
to represent ".git/HEAD -> refs/heads/master", but it can be compiled
to use the textfile symbolic ref.
The places that did 'readlink .git/HEAD' and 'ln -s refs/heads/blah
.git/HEAD' have been converted to use new git-symbolic-ref command, so
that they can deal with either implementation.
Signed-off-by: Junio C Hamano <junio@twinsun.com>
Actually report what exactly is wrong with the object, instead of an
ambiguous 'bad sha1 file' or such. In places where we already do, unify
the format and clean the messages up.
Signed-off-by: Petr Baudis <pasky@suse.cz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
We have deprecated the old environment variable names for quite a
while and now it's time to remove them. Gone are:
SHA1_FILE_DIRECTORIES AUTHOR_DATE AUTHOR_EMAIL AUTHOR_NAME
COMMIT_AUTHOR_EMAIL COMMIT_AUTHOR_NAME SHA1_FILE_DIRECTORY
Signed-off-by: Junio C Hamano <junkio@cox.net>
As promised, this is the "big tool rename" patch. The primary differences
since 0.99.6 are:
(1) git-*-script are no more. The commands installed do not
have any such suffix so users do not have to remember if
something is implemented as a shell script or not.
(2) Many command names with 'cache' in them are renamed with
'index' if that is what they mean.
There are backward compatibility symblic links so that you and
Porcelains can keep using the old names, but the backward
compatibility support is expected to be removed in the near
future.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Solaris 8 doesn't have the newer unsetenv() and setenv()
functions, so replace them with putenv(). The one use of
unsetenv() in fsck-cache.c now sets GIT_ALTERNATE_OBJECT_
DIRECTORIES to the empty string. Every place that var
is used, NULLs are also replaced with empty strings, so
it's ok.
Signed-off-by: Jason Riedy <ejr@cs.berkeley.edu>
Omitting the first branch in ?: is a GNU extension. Cute,
but not supported by other compilers. Replaced mostly
by explicit tests. Calls to getenv() simply are repeated
on non-GNU compilers.
Signed-off-by: Jason Riedy <ejr@cs.berkeley.edu>
It was a mistake to use GIT_ALTERNATE_OBJECT_DIRECTORIES
environment variable to specify what alternate object pools to
look for missing objects when working with an object database.
It is not a property of the process running the git commands,
but a property of the object database that is partial and needs
other object pools to complete the set of objects it lacks.
This patch allows you to have $GIT_OBJECT_DIRECTORY/info/alternates
whose contents is in exactly the same format as the environment
variable, to let an object database name alternate object pools
it depends on.
Signed-off-by: Junio C Hamano <junkio@cox.net>
In particular, warn about things like zero-padding of the mode bits,
which is a big no-no, since it makes otherwise identical trees have
different representations (and thus different SHA1 numbers).
Also make the warnings more regular.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
We historically used to be very careful in fsck-cache, but when it was
re-written to use "parse_object()" instead of parsing everything by
hand, it lost a bit of the checks. This, together with the previous
commit, should make it do more proper commit object syntax checks.
Also add a "--strict" flag, which warns about the old-style "0664" file
mode bits, which shouldn't exist in modern trees, but that happened
early on in git trees and that the default git-fsck-cache thus silently
accepts.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
The location alt_odb[j].name[0..] is filled with ??/?{38} to form a sha1
filename to try, but I was too lazy to allocate a copy, so while
fsck_object_dir() is running for the directory, the filenames ??/?{38}
are filled after NUL (usually and always the location should have '/'),
making them "not found".
This should fix it.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
In particular, check that it's a symlink, and points to refs/heads/. We
depend on that these days not only for "git checkout", but also because
fsck and others only check for references in the .git/refs/
subdirectory, not things like HEAD itself.
Nico pointed out that having verify_pack.c and verify-pack.c was
confusing. Rename verify_pack.c to pack-check.c as suggested,
and enhances the verification done quite a bit.
- Built-in sha1_file unpacking knows that a base object of a
deltified object _must_ be in the same pack, and takes
advantage of that fact.
- Earlier verify-pack command only checked the SHA1 sum for the
entire pack file and did not look into its contents. It now
checks everything idx file claims to have unpacks correctly.
- It now has a hook to give more detailed information for
objects contained in the pack under -v flag.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Given a list of <pack>.idx files, this command validates the
index file and the corresponding .pack file for consistency.
This patch also uses the same validation mechanism in fsck-cache
when the --full flag is used.
During normal operation, sha1_file.c verifies that a given .idx
file matches the .pack file by comparing the SHA1 checksum
stored in .idx file and .pack file as a minimum sanity check.
We may further want to check the pack signature and version when
we map the pack, but that would be a separate patch.
Earlier, errors to map a pack file was not flagged fatal but led
to a random fatal error later. This version explicitly die()s
when such an error is detected.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e. found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).
Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.
This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.
Another new flag, --full, makes git-fsck-cache to check not only the
current GIT_OBJECT_DIRECTORY but also objects found in alternate object
pools and packed GIT archives.a
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
The fsck-cache complains if objects referred to by files in .git/refs/
or objects stored in files under .git/objects/??/ are not found as
stand-alone SHA1 files (i.e. found in alternate object pools
GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives stored under
.git/objects/pack).
Although this is a good semantics to maintain consistency of a single
.git/objects directory as a self contained set of objects, it sometimes
is useful to consider it is OK as long as these "outside" objects are
available.
This commit introduces a new flag, --standalone, to git-fsck-cache.
When it is not specified, connectivity checks and .git/refs pointer
checks are taught that it is OK when expected objects do not exist under
.git/objects/?? hierarchy but are available from an packed archive or in
an alternate object pool.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Packed delta files created by git-pack-objects seems to be the
way to go, and existing "delta" object handling code has exposed
the object representation details to too many places. Remove it
while we refactor code to come up with a proper interface in
sha1_file.c.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
We used to ignore unreachable tags, which just causes problems: it makes
"git prune" leave them around, but since we'll have prune everything
that tag points to, the tag object really should be removed too.
So remove the code that made us think tags were always reachable.
This adds knowledge of delta objects to fsck-cache and various object
parsing code. A new switch to git-fsck-cache is provided to display the
maximum delta depth found in a repository.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Fixes all in-code names that leaved during "big name change".
Signed-off-by: Alexey Nezhdanov <snake@penza-gsm.ru>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
explicit references for reachability analysis.
We already had that as separate logic in git-prune-script, so this
is not a new special case - it's an old special case moved into
fsck, making normal usage be much simpler.
This is a follow-up fix to the earlier "Notice index that has
path and path/file and refuse to write such a tree" patch.
With this fix, git-fsck-cache complains if a tree object stores
more than one entries with the same name.
Signed-off-by: Junio C Hamano <junkio@cox.net>
SHA1_FILE_DIRECTORIES environment variable is a colon separated paths
used when looking for SHA1 files not found in the usual place for
reading. Creating a new SHA1 file does not use this alternate object
database location mechanism. This is useful to archive older, rarely
used objects into separate directories.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The tree object parsing used to get the executable bit wrong,
and didn't know about symlinks. Also, fsck really wants the
full mode value so that it can verify the other bits for sanity,
so save it all in struct tree_entry.
(updated from the version posted to GIT mailing list).
When a new blob is registered with update-cache, and before the cache
is written as a tree and committed, git-fsck-cache will find the blob
unreachable. This patch adds a new flag, "--cache" to git-fsck-cache,
with which it keeps such blobs from considered "unreachable".
The git-prune-script is updated to use this new flag. At the same time
it adds .git/refs/*/* to the set of default locations to look for heads,
which should be consistent with expectations from Cogito users.
Without this fix, "diff-cache -p --cached" after git-prune-script has
pruned the blob object will fail mysteriously and git-write-tree would
also fail.
Signed-off-by: Junio C Hamano <junkio@cox.net>
fsck_tag() failes to notice that the parsing of the tag may
have failed in the parse_object() call on the object that it
is tagging.
Noticed by Junio.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
We check the ordering of the entries, and we verify that none
of the entries has a slash in it (this allows us to remove the
hacky "has_full_path" member from the tree structure, since we
now just test it by walking the tree entries instead).
This improves the cold-cache behaviour on most filesystems,
since it makes the fsck access patterns more regular on
the disk, rather than seeking back and forth.
Note the "most". Not all filesystems have any relationship
between inode number and location on disk.
This allows the programs to use various simplified versions of
the SHA1 names, eg just say "HEAD" for the SHA1 pointed to by
the .git/HEAD file etc.
For example, this commit has been done with
git-commit-tree $(git-write-tree) -p HEAD
instead of the traditional "$(cat .git/HEAD)" syntax.
Show the types of objects involved in broken links, and don't bother
warning about unreachable tag files (if somebody cares about tags,
they'll use the --tags flag to see them).
With support for parse_object() and tags in the core, fsck_cache can just
call them, and can be simplified a bit.
Signed-Off-By: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Johannes Schindelin