kernel
/
git
mirror of https://git.kernel.org/pub/scm/git/git.git
1
0
Fork 0
Code Releases Activity

Merge branch 'jt/subprocess-handshake' into maint 

Code cleanup.

* jt/subprocess-handshake:
  sub-process: refactor handshake to common function
  Documentation: migrate sub-process docs to header
  convert: add "status=delayed" to filter process protocol
  convert: refactor capabilities negotiation
  convert: move multiple file filter error handling to separate function
  convert: put the flags field before the flag itself for consistent style
  t0021: write "OUT <size>" only on success
  t0021: make debug log file name configurable
  t0021: keep filter log files on comparison
maint
Junio C Hamano 2017-08-23 14:33:52 -07:00
parent de55703672 fa64a2fdbe
commit df2dd28316
14 changed files with 810 additions and 293 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

69
Documentation/gitattributes.txt
View File

@ -425,8 +425,8 @@ packet: git< capability=clean
packet: git< capability=smudge packet: git< capability=smudge
packet: git< 0000 packet: git< 0000
------------------------ ------------------------
Supported filter capabilities in version 2 are "clean" and Supported filter capabilities in version 2 are "clean", "smudge",
"smudge". and "delay".


Afterwards Git sends a list of "key=value" pairs terminated with Afterwards Git sends a list of "key=value" pairs terminated with
a flush packet. The list will contain at least the filter command a flush packet. The list will contain at least the filter command
@ -512,12 +512,73 @@ the protocol then Git will stop the filter process and restart it
with the next file that needs to be processed. Depending on the with the next file that needs to be processed. Depending on the
`filter.<driver>.required` flag Git will interpret that as error. `filter.<driver>.required` flag Git will interpret that as error.


After the filter has processed a blob it is expected to wait for After the filter has processed a command it is expected to wait for
the next "key=value" list containing a command. Git will close a "key=value" list containing the next command. Git will close
the command pipe on exit. The filter is expected to detect EOF the command pipe on exit. The filter is expected to detect EOF
and exit gracefully on its own. Git will wait until the filter and exit gracefully on its own. Git will wait until the filter
process has stopped. process has stopped.


Delay
^^^^^

If the filter supports the "delay" capability, then Git can send the
flag "can-delay" after the filter command and pathname. This flag
denotes that the filter can delay filtering the current blob (e.g. to
compensate network latencies) by responding with no content but with
the status "delayed" and a flush packet.
------------------------
packet: git> command=smudge
packet: git> pathname=path/testfile.dat
packet: git> can-delay=1
packet: git> 0000
packet: git> CONTENT
packet: git> 0000
packet: git< status=delayed
packet: git< 0000
------------------------

If the filter supports the "delay" capability then it must support the
"list_available_blobs" command. If Git sends this command, then the
filter is expected to return a list of pathnames representing blobs
that have been delayed earlier and are now available.
The list must be terminated with a flush packet followed
by a "success" status that is also terminated with a flush packet. If
no blobs for the delayed paths are available, yet, then the filter is
expected to block the response until at least one blob becomes
available. The filter can tell Git that it has no more delayed blobs
by sending an empty list. As soon as the filter responds with an empty
list, Git stops asking. All blobs that Git has not received at this
point are considered missing and will result in an error.

------------------------
packet: git> command=list_available_blobs
packet: git> 0000
packet: git< pathname=path/testfile.dat
packet: git< pathname=path/otherfile.dat
packet: git< 0000
packet: git< status=success
packet: git< 0000
------------------------

After Git received the pathnames, it will request the corresponding
blobs again. These requests contain a pathname and an empty content
section. The filter is expected to respond with the smudged content
in the usual way as explained above.
------------------------
packet: git> command=smudge
packet: git> pathname=path/testfile.dat
packet: git> 0000
packet: git> 0000 # empty content!
packet: git< status=success
packet: git< 0000
packet: git< SMUDGED_CONTENT
packet: git< 0000
packet: git< 0000 # empty list, keep "status=success" unchanged!
------------------------

Example
^^^^^^^

A long running filter demo implementation can be found in A long running filter demo implementation can be found in
`contrib/long-running-filter/example.pl` located in the Git `contrib/long-running-filter/example.pl` located in the Git
core repository. If you develop your own long running filter core repository. If you develop your own long running filter

59
Documentation/technical/api-sub-process.txt
View File

@ -1,59 +0,0 @@
sub-process API
===============

The sub-process API makes it possible to run background sub-processes
for the entire lifetime of a Git invocation. If Git needs to communicate
with an external process multiple times, then this can reduces the process
invocation overhead. Git and the sub-process communicate through stdin and
stdout.

The sub-processes are kept in a hashmap by command name and looked up
via the subprocess_find_entry function. If an existing instance can not
be found then a new process should be created and started. When the
parent git command terminates, all sub-processes are also terminated.

This API is based on the run-command API.

Data structures
---------------

* `struct subprocess_entry`

The sub-process structure. Members should not be accessed directly.

Types
-----

'int(*subprocess_start_fn)(struct subprocess_entry *entry)'::

User-supplied function to initialize the sub-process. This is
typically used to negotiate the interface version and capabilities.


Functions
---------

`cmd2process_cmp`::

Function to test two subprocess hashmap entries for equality.

`subprocess_start`::

Start a subprocess and add it to the subprocess hashmap.

`subprocess_stop`::

Kill a subprocess and remove it from the subprocess hashmap.

`subprocess_find_entry`::

Find a subprocess in the subprocess hashmap.

`subprocess_get_child_process`::

Get the underlying `struct child_process` from a subprocess.

`subprocess_read_status`::

Helper function to read packets looking for the last "status=<foo>"
key/value pair.

3
builtin/checkout.c
View File

@ -358,6 +358,8 @@ static int checkout_paths(const struct checkout_opts *opts,
state.force = 1; state.force = 1;
state.refresh_cache = 1; state.refresh_cache = 1;
state.istate = &the_index; state.istate = &the_index;

enable_delayed_checkout(&state);
for (pos = 0; pos < active_nr; pos++) { for (pos = 0; pos < active_nr; pos++) {
struct cache_entry *ce = active_cache[pos]; struct cache_entry *ce = active_cache[pos];
if (ce->ce_flags & CE_MATCHED) { if (ce->ce_flags & CE_MATCHED) {
@ -372,6 +374,7 @@ static int checkout_paths(const struct checkout_opts *opts,
pos = skip_same_name(ce, pos) - 1; pos = skip_same_name(ce, pos) - 1;
} }
} }
errs |= finish_delayed_checkout(&state);


if (write_locked_index(&the_index, lock_file, COMMIT_LOCK)) if (write_locked_index(&the_index, lock_file, COMMIT_LOCK))
die(_("unable to write new index file")); die(_("unable to write new index file"));

3
cache.h
View File

@ -1500,6 +1500,7 @@ struct checkout {
struct index_state *istate; struct index_state *istate;
const char *base_dir; const char *base_dir;
int base_dir_len; int base_dir_len;
struct delayed_checkout *delayed_checkout;
unsigned force:1, unsigned force:1,
quiet:1, quiet:1,
not_new:1, not_new:1,
@ -1509,6 +1510,8 @@ struct checkout {


#define TEMPORARY_FILENAME_LENGTH 25 #define TEMPORARY_FILENAME_LENGTH 25
extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath); extern int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath);
extern void enable_delayed_checkout(struct checkout *state);
extern int finish_delayed_checkout(struct checkout *state);


struct cache_def { struct cache_def {
struct strbuf path; struct strbuf path;

223
convert.c
View File

@ -501,6 +501,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t le


#define CAP_CLEAN (1u<<0) #define CAP_CLEAN (1u<<0)
#define CAP_SMUDGE (1u<<1) #define CAP_SMUDGE (1u<<1)
#define CAP_DELAY (1u<<2)


struct cmd2process { struct cmd2process {
struct subprocess_entry subprocess; /* must be the first member! */ struct subprocess_entry subprocess; /* must be the first member! */
@ -512,69 +513,49 @@ static struct hashmap subprocess_map;


static int start_multi_file_filter_fn(struct subprocess_entry *subprocess) static int start_multi_file_filter_fn(struct subprocess_entry *subprocess)
{ {
int err; static int versions[] = {2, 0};
static struct subprocess_capability capabilities[] = {
{ "clean", CAP_CLEAN },
{ "smudge", CAP_SMUDGE },
{ "delay", CAP_DELAY },
{ NULL, 0 }
};
struct cmd2process *entry = (struct cmd2process *)subprocess; struct cmd2process *entry = (struct cmd2process *)subprocess;
struct string_list cap_list = STRING_LIST_INIT_NODUP; return subprocess_handshake(subprocess, "git-filter", versions, NULL,
char *cap_buf; capabilities,
const char *cap_name; &entry->supported_capabilities);
struct child_process *process = &subprocess->process; }
const char *cmd = subprocess->cmd;


sigchain_push(SIGPIPE, SIG_IGN); static void handle_filter_error(const struct strbuf *filter_status,

struct cmd2process *entry,
err = packet_writel(process->in, "git-filter-client", "version=2", NULL); const unsigned int wanted_capability) {
if (err) if (!strcmp(filter_status->buf, "error"))
goto done; ; /* The filter signaled a problem with the file. */

else if (!strcmp(filter_status->buf, "abort") && wanted_capability) {
err = strcmp(packet_read_line(process->out, NULL), "git-filter-server"); /*
if (err) { * The filter signaled a permanent problem. Don't try to filter
error("external filter '%s' does not support filter protocol version 2", cmd); * files with the same command for the lifetime of the current
goto done; * Git process.
*/
entry->supported_capabilities &= ~wanted_capability;
} else {
/*
* Something went wrong with the protocol filter.
* Force shutdown and restart if another blob requires filtering.
*/
error("external filter '%s' failed", entry->subprocess.cmd);
subprocess_stop(&subprocess_map, &entry->subprocess);
free(entry);
} }
err = strcmp(packet_read_line(process->out, NULL), "version=2");
if (err)
goto done;
err = packet_read_line(process->out, NULL) != NULL;
if (err)
goto done;

err = packet_writel(process->in, "capability=clean", "capability=smudge", NULL);

for (;;) {
cap_buf = packet_read_line(process->out, NULL);
if (!cap_buf)
break;
string_list_split_in_place(&cap_list, cap_buf, '=', 1);

if (cap_list.nr != 2 || strcmp(cap_list.items[0].string, "capability"))
continue;

cap_name = cap_list.items[1].string;
if (!strcmp(cap_name, "clean")) {
entry->supported_capabilities |= CAP_CLEAN;
} else if (!strcmp(cap_name, "smudge")) {
entry->supported_capabilities |= CAP_SMUDGE;
} else {
warning(
"external filter '%s' requested unsupported filter capability '%s'",
cmd, cap_name
);
}

string_list_clear(&cap_list, 0);
}

done:
sigchain_pop(SIGPIPE);

return err;
} }


static int apply_multi_file_filter(const char *path, const char *src, size_t len, static int apply_multi_file_filter(const char *path, const char *src, size_t len,
int fd, struct strbuf *dst, const char *cmd, int fd, struct strbuf *dst, const char *cmd,
const unsigned int wanted_capability) const unsigned int wanted_capability,
struct delayed_checkout *dco)
{ {
int err; int err;
int can_delay = 0;
struct cmd2process *entry; struct cmd2process *entry;
struct child_process *process; struct child_process *process;
struct strbuf nbuf = STRBUF_INIT; struct strbuf nbuf = STRBUF_INIT;
@ -603,12 +584,12 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
} }
process = &entry->subprocess.process; process = &entry->subprocess.process;


if (!(wanted_capability & entry->supported_capabilities)) if (!(entry->supported_capabilities & wanted_capability))
return 0; return 0;


if (CAP_CLEAN & wanted_capability) if (wanted_capability & CAP_CLEAN)
filter_type = "clean"; filter_type = "clean";
else if (CAP_SMUDGE & wanted_capability) else if (wanted_capability & CAP_SMUDGE)
filter_type = "smudge"; filter_type = "smudge";
else else
die("unexpected filter type"); die("unexpected filter type");
@ -630,6 +611,14 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
if (err) if (err)
goto done; goto done;


if ((entry->supported_capabilities & CAP_DELAY) &&
dco && dco->state == CE_CAN_DELAY) {
can_delay = 1;
err = packet_write_fmt_gently(process->in, "can-delay=1\n");
if (err)
goto done;
}

err = packet_flush_gently(process->in); err = packet_flush_gently(process->in);
if (err) if (err)
goto done; goto done;
@ -645,14 +634,73 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
if (err) if (err)
goto done; goto done;


err = strcmp(filter_status.buf, "success"); if (can_delay && !strcmp(filter_status.buf, "delayed")) {
string_list_insert(&dco->filters, cmd);
string_list_insert(&dco->paths, path);
} else {
/* The filter got the blob and wants to send us a response. */
err = strcmp(filter_status.buf, "success");
if (err)
goto done;

err = read_packetized_to_strbuf(process->out, &nbuf) < 0;
if (err)
goto done;

err = subprocess_read_status(process->out, &filter_status);
if (err)
goto done;

err = strcmp(filter_status.buf, "success");
}

done:
sigchain_pop(SIGPIPE);

if (err)
handle_filter_error(&filter_status, entry, wanted_capability);
else
strbuf_swap(dst, &nbuf);
strbuf_release(&nbuf);
return !err;
}


int async_query_available_blobs(const char *cmd, struct string_list *available_paths)
{
int err;
char *line;
struct cmd2process *entry;
struct child_process *process;
struct strbuf filter_status = STRBUF_INIT;

assert(subprocess_map_initialized);
entry = (struct cmd2process *)subprocess_find_entry(&subprocess_map, cmd);
if (!entry) {
error("external filter '%s' is not available anymore although "
"not all paths have been filtered", cmd);
return 0;
}
process = &entry->subprocess.process;
sigchain_push(SIGPIPE, SIG_IGN);

err = packet_write_fmt_gently(
process->in, "command=list_available_blobs\n");
if (err) if (err)
goto done; goto done;


err = read_packetized_to_strbuf(process->out, &nbuf) < 0; err = packet_flush_gently(process->in);
if (err) if (err)
goto done; goto done;


while ((line = packet_read_line(process->out, NULL))) {
const char *path;
if (skip_prefix(line, "pathname=", &path))
string_list_insert(available_paths, xstrdup(path));
else
; /* ignore unknown keys */
}

err = subprocess_read_status(process->out, &filter_status); err = subprocess_read_status(process->out, &filter_status);
if (err) if (err)
goto done; goto done;
@ -662,29 +710,8 @@ static int apply_multi_file_filter(const char *path, const char *src, size_t len
done: done:
sigchain_pop(SIGPIPE); sigchain_pop(SIGPIPE);


if (err) { if (err)
if (!strcmp(filter_status.buf, "error")) { handle_filter_error(&filter_status, entry, 0);
/* The filter signaled a problem with the file. */
} else if (!strcmp(filter_status.buf, "abort")) {
/*
* The filter signaled a permanent problem. Don't try to filter
* files with the same command for the lifetime of the current
* Git process.
*/
entry->supported_capabilities &= ~wanted_capability;
} else {
/*
* Something went wrong with the protocol filter.
* Force shutdown and restart if another blob requires filtering.
*/
error("external filter '%s' failed", cmd);
subprocess_stop(&subprocess_map, &entry->subprocess);
free(entry);
}
} else {
strbuf_swap(dst, &nbuf);
}
strbuf_release(&nbuf);
return !err; return !err;
} }


@ -699,7 +726,8 @@ static struct convert_driver {


static int apply_filter(const char *path, const char *src, size_t len, static int apply_filter(const char *path, const char *src, size_t len,
int fd, struct strbuf *dst, struct convert_driver *drv, int fd, struct strbuf *dst, struct convert_driver *drv,
const unsigned int wanted_capability) const unsigned int wanted_capability,
struct delayed_checkout *dco)
{ {
const char *cmd = NULL; const char *cmd = NULL;


@ -709,15 +737,16 @@ static int apply_filter(const char *path, const char *src, size_t len,
if (!dst) if (!dst)
return 1; return 1;


if ((CAP_CLEAN & wanted_capability) && !drv->process && drv->clean) if ((wanted_capability & CAP_CLEAN) && !drv->process && drv->clean)
cmd = drv->clean; cmd = drv->clean;
else if ((CAP_SMUDGE & wanted_capability) && !drv->process && drv->smudge) else if ((wanted_capability & CAP_SMUDGE) && !drv->process && drv->smudge)
cmd = drv->smudge; cmd = drv->smudge;


if (cmd && *cmd) if (cmd && *cmd)
return apply_single_file_filter(path, src, len, fd, dst, cmd); return apply_single_file_filter(path, src, len, fd, dst, cmd);
else if (drv->process && *drv->process) else if (drv->process && *drv->process)
return apply_multi_file_filter(path, src, len, fd, dst, drv->process, wanted_capability); return apply_multi_file_filter(path, src, len, fd, dst,
drv->process, wanted_capability, dco);


return 0; return 0;
} }
@ -1058,7 +1087,7 @@ int would_convert_to_git_filter_fd(const char *path)
if (!ca.drv->required) if (!ca.drv->required)
return 0; return 0;


return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN); return apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL);
} }


const char *get_convert_attr_ascii(const char *path) const char *get_convert_attr_ascii(const char *path)
@ -1096,7 +1125,7 @@ int convert_to_git(const struct index_state *istate,


convert_attrs(&ca, path); convert_attrs(&ca, path);


ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN); ret |= apply_filter(path, src, len, -1, dst, ca.drv, CAP_CLEAN, NULL);
if (!ret && ca.drv && ca.drv->required) if (!ret && ca.drv && ca.drv->required)
die("%s: clean filter '%s' failed", path, ca.drv->name); die("%s: clean filter '%s' failed", path, ca.drv->name);


@ -1122,7 +1151,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,
assert(ca.drv); assert(ca.drv);
assert(ca.drv->clean || ca.drv->process); assert(ca.drv->clean || ca.drv->process);


if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN)) if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))
die("%s: clean filter '%s' failed", path, ca.drv->name); die("%s: clean filter '%s' failed", path, ca.drv->name);


crlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe); crlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);
@ -1131,7 +1160,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,


static int convert_to_working_tree_internal(const char *path, const char *src, static int convert_to_working_tree_internal(const char *path, const char *src,
size_t len, struct strbuf *dst, size_t len, struct strbuf *dst,
int normalizing) int normalizing, struct delayed_checkout *dco)
{ {
int ret = 0, ret_filter = 0; int ret = 0, ret_filter = 0;
struct conv_attrs ca; struct conv_attrs ca;
@ -1156,22 +1185,30 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
} }
} }


ret_filter = apply_filter(path, src, len, -1, dst, ca.drv, CAP_SMUDGE); ret_filter = apply_filter(
path, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);
if (!ret_filter && ca.drv && ca.drv->required) if (!ret_filter && ca.drv && ca.drv->required)
die("%s: smudge filter %s failed", path, ca.drv->name); die("%s: smudge filter %s failed", path, ca.drv->name);


return ret | ret_filter; return ret | ret_filter;
} }


int async_convert_to_working_tree(const char *path, const char *src,
size_t len, struct strbuf *dst,
void *dco)
{
return convert_to_working_tree_internal(path, src, len, dst, 0, dco);
}

int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst) int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)
{ {
return convert_to_working_tree_internal(path, src, len, dst, 0); return convert_to_working_tree_internal(path, src, len, dst, 0, NULL);
} }


int renormalize_buffer(const struct index_state *istate, const char *path, int renormalize_buffer(const struct index_state *istate, const char *path,
const char *src, size_t len, struct strbuf *dst) const char *src, size_t len, struct strbuf *dst)
{ {
int ret = convert_to_working_tree_internal(path, src, len, dst, 1); int ret = convert_to_working_tree_internal(path, src, len, dst, 1, NULL);
if (ret) { if (ret) {
src = dst->buf; src = dst->buf;
len = dst->len; len = dst->len;

26
convert.h
View File

@ -4,6 +4,8 @@
#ifndef CONVERT_H #ifndef CONVERT_H
#define CONVERT_H #define CONVERT_H


#include "string-list.h"

struct index_state; struct index_state;


enum safe_crlf { enum safe_crlf {
@ -34,6 +36,26 @@ enum eol {
#endif #endif
}; };


enum ce_delay_state {
CE_NO_DELAY = 0,
CE_CAN_DELAY = 1,
CE_RETRY = 2
};

struct delayed_checkout {
/*
* State of the currently processed cache entry. If the state is
* CE_CAN_DELAY, then the filter can delay the current cache entry.
* If the state is CE_RETRY, then this signals the filter that the
* cache entry was requested before.
*/
enum ce_delay_state state;
/* List of filter drivers that signaled delayed blobs. */
struct string_list filters;
/* List of delayed blobs identified by their path. */
struct string_list paths;
};

extern enum eol core_eol; extern enum eol core_eol;
extern const char *get_cached_convert_stats_ascii(const struct index_state *istate, extern const char *get_cached_convert_stats_ascii(const struct index_state *istate,
const char *path); const char *path);
@ -46,6 +68,10 @@ extern int convert_to_git(const struct index_state *istate,
struct strbuf *dst, enum safe_crlf checksafe); struct strbuf *dst, enum safe_crlf checksafe);
extern int convert_to_working_tree(const char *path, const char *src, extern int convert_to_working_tree(const char *path, const char *src,
size_t len, struct strbuf *dst); size_t len, struct strbuf *dst);
extern int async_convert_to_working_tree(const char *path, const char *src,
size_t len, struct strbuf *dst,
void *dco);
extern int async_query_available_blobs(const char *cmd, struct string_list *available_paths);
extern int renormalize_buffer(const struct index_state *istate, extern int renormalize_buffer(const struct index_state *istate,
const char *path, const char *src, size_t len, const char *path, const char *src, size_t len,
struct strbuf *dst); struct strbuf *dst);

132
entry.c
View File

@ -137,6 +137,105 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,
return result; return result;
} }


void enable_delayed_checkout(struct checkout *state)
{
if (!state->delayed_checkout) {
state->delayed_checkout = xmalloc(sizeof(*state->delayed_checkout));
state->delayed_checkout->state = CE_CAN_DELAY;
string_list_init(&state->delayed_checkout->filters, 0);
string_list_init(&state->delayed_checkout->paths, 0);
}
}

static int remove_available_paths(struct string_list_item *item, void *cb_data)
{
struct string_list *available_paths = cb_data;
struct string_list_item *available;

available = string_list_lookup(available_paths, item->string);
if (available)
available->util = (void *)item->string;
return !available;
}

int finish_delayed_checkout(struct checkout *state)
{
int errs = 0;
struct string_list_item *filter, *path;
struct delayed_checkout *dco = state->delayed_checkout;

if (!state->delayed_checkout)
return errs;

dco->state = CE_RETRY;
while (dco->filters.nr > 0) {
for_each_string_list_item(filter, &dco->filters) {
struct string_list available_paths = STRING_LIST_INIT_NODUP;

if (!async_query_available_blobs(filter->string, &available_paths)) {
/* Filter reported an error */
errs = 1;
filter->string = "";
continue;
}
if (available_paths.nr <= 0) {
/*
* Filter responded with no entries. That means
* the filter is done and we can remove the
* filter from the list (see
* "string_list_remove_empty_items" call below).
*/
filter->string = "";
continue;
}

/*
* In dco->paths we store a list of all delayed paths.
* The filter just send us a list of available paths.
* Remove them from the list.
*/
filter_string_list(&dco->paths, 0,
&remove_available_paths, &available_paths);

for_each_string_list_item(path, &available_paths) {
struct cache_entry* ce;

if (!path->util) {
error("external filter '%s' signaled that '%s' "
"is now available although it has not been "
"delayed earlier",
filter->string, path->string);
errs |= 1;

/*
* Do not ask the filter for available blobs,
* again, as the filter is likely buggy.
*/
filter->string = "";
continue;
}
ce = index_file_exists(state->istate, path->string,
strlen(path->string), 0);
errs |= (ce ? checkout_entry(ce, state, NULL) : 1);
}
}
string_list_remove_empty_items(&dco->filters, 0);
}
string_list_clear(&dco->filters, 0);

/* At this point we should not have any delayed paths anymore. */
errs |= dco->paths.nr;
for_each_string_list_item(path, &dco->paths) {
error("'%s' was not filtered properly", path->string);
}
string_list_clear(&dco->paths, 0);

free(dco);
state->delayed_checkout = NULL;

return errs;
}

static int write_entry(struct cache_entry *ce, static int write_entry(struct cache_entry *ce,
char *path, const struct checkout *state, int to_tempfile) char *path, const struct checkout *state, int to_tempfile)
{ {
@ -179,11 +278,34 @@ static int write_entry(struct cache_entry *ce,
/* /*
* Convert from git internal format to working tree format * Convert from git internal format to working tree format
*/ */
if (ce_mode_s_ifmt == S_IFREG && if (ce_mode_s_ifmt == S_IFREG) {
convert_to_working_tree(ce->name, new, size, &buf)) { struct delayed_checkout *dco = state->delayed_checkout;
free(new); if (dco && dco->state != CE_NO_DELAY) {
new = strbuf_detach(&buf, &newsize); /* Do not send the blob in case of a retry. */
size = newsize; if (dco->state == CE_RETRY) {
new = NULL;
size = 0;
}
ret = async_convert_to_working_tree(
ce->name, new, size, &buf, dco);
if (ret && string_list_has_string(&dco->paths, ce->name)) {
free(new);
goto finish;
}
} else
ret = convert_to_working_tree(
ce->name, new, size, &buf);

if (ret) {
free(new);
new = strbuf_detach(&buf, &newsize);
size = newsize;
}
/*
* No "else" here as errors from convert are OK at this
* point. If the error would have been fatal (e.g.
* filter is required), then we would have died already.
*/
} }


fd = open_output_fd(path, ce, to_tempfile); fd = open_output_fd(path, ce, to_tempfile);

19
pkt-line.c
View File

@ -171,25 +171,6 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...)
return status; return status;
} }


int packet_writel(int fd, const char *line, ...)
{
va_list args;
int err;
va_start(args, line);
for (;;) {
if (!line)
break;
if (strlen(line) > LARGE_PACKET_DATA_MAX)
return -1;
err = packet_write_fmt_gently(fd, "%s\n", line);
if (err)
return err;
line = va_arg(args, const char*);
}
va_end(args);
return packet_flush_gently(fd);
}

static int packet_write_gently(const int fd_out, const char *buf, size_t size) static int packet_write_gently(const int fd_out, const char *buf, size_t size)
{ {
static char packet_write_buffer[LARGE_PACKET_MAX]; static char packet_write_buffer[LARGE_PACKET_MAX];

2
pkt-line.h
View File

@ -25,8 +25,6 @@ void packet_buf_flush(struct strbuf *buf);
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3))); void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int packet_flush_gently(int fd); int packet_flush_gently(int fd);
int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3))); int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
LAST_ARG_MUST_BE_NULL
int packet_writel(int fd, const char *line, ...);
int write_packetized_from_fd(int fd_in, int fd_out); int write_packetized_from_fd(int fd_in, int fd_out);
int write_packetized_from_buf(const char *src_in, size_t len, int fd_out); int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);



104
sub-process.c
View File

@ -105,3 +105,107 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
hashmap_add(hashmap, entry); hashmap_add(hashmap, entry);
return 0; return 0;
} }

static int handshake_version(struct child_process *process,
const char *welcome_prefix, int *versions,
int *chosen_version)
{
int version_scratch;
int i;
char *line;
const char *p;

if (!chosen_version)
chosen_version = &version_scratch;

if (packet_write_fmt_gently(process->in, "%s-client\n",
welcome_prefix))
return error("Could not write client identification");
for (i = 0; versions[i]; i++) {
if (packet_write_fmt_gently(process->in, "version=%d\n",
versions[i]))
return error("Could not write requested version");
}
if (packet_flush_gently(process->in))
return error("Could not write flush packet");

if (!(line = packet_read_line(process->out, NULL)) ||
!skip_prefix(line, welcome_prefix, &p) ||
strcmp(p, "-server"))
return error("Unexpected line '%s', expected %s-server",
line ? line : "<flush packet>", welcome_prefix);
if (!(line = packet_read_line(process->out, NULL)) ||
!skip_prefix(line, "version=", &p) ||
strtol_i(p, 10, chosen_version))
return error("Unexpected line '%s', expected version",
line ? line : "<flush packet>");
if ((line = packet_read_line(process->out, NULL)))
return error("Unexpected line '%s', expected flush", line);

/* Check to make sure that the version received is supported */
for (i = 0; versions[i]; i++) {
if (versions[i] == *chosen_version)
break;
}
if (!versions[i])
return error("Version %d not supported", *chosen_version);

return 0;
}

static int handshake_capabilities(struct child_process *process,
struct subprocess_capability *capabilities,
unsigned int *supported_capabilities)
{
int i;
char *line;

for (i = 0; capabilities[i].name; i++) {
if (packet_write_fmt_gently(process->in, "capability=%s\n",
capabilities[i].name))
return error("Could not write requested capability");
}
if (packet_flush_gently(process->in))
return error("Could not write flush packet");

while ((line = packet_read_line(process->out, NULL))) {
const char *p;
if (!skip_prefix(line, "capability=", &p))
continue;

for (i = 0;
capabilities[i].name && strcmp(p, capabilities[i].name);
i++)
;
if (capabilities[i].name) {
if (supported_capabilities)
*supported_capabilities |= capabilities[i].flag;
} else {
warning("external filter requested unsupported filter capability '%s'",
p);
}
}

return 0;
}

int subprocess_handshake(struct subprocess_entry *entry,
const char *welcome_prefix,
int *versions,
int *chosen_version,
struct subprocess_capability *capabilities,
unsigned int *supported_capabilities)
{
int retval;
struct child_process *process = &entry->process;

sigchain_push(SIGPIPE, SIG_IGN);

retval = handshake_version(process, welcome_prefix, versions,
chosen_version) ||
handshake_capabilities(process, capabilities,
supported_capabilities);

sigchain_pop(SIGPIPE);
return retval;
}

51
sub-process.h
View File

@ -6,41 +6,88 @@
#include "run-command.h" #include "run-command.h"


/* /*
* Generic implementation of background process infrastructure. * The sub-process API makes it possible to run background sub-processes
* See: Documentation/technical/api-sub-process.txt * for the entire lifetime of a Git invocation. If Git needs to communicate
* with an external process multiple times, then this can reduces the process
* invocation overhead. Git and the sub-process communicate through stdin and
* stdout.
*
* The sub-processes are kept in a hashmap by command name and looked up
* via the subprocess_find_entry function. If an existing instance can not
* be found then a new process should be created and started. When the
* parent git command terminates, all sub-processes are also terminated.
*
* This API is based on the run-command API.
*/ */


/* data structures */ /* data structures */


/* Members should not be accessed directly. */
struct subprocess_entry { struct subprocess_entry {
struct hashmap_entry ent; /* must be the first member! */ struct hashmap_entry ent; /* must be the first member! */
const char *cmd; const char *cmd;
struct child_process process; struct child_process process;
}; };


struct subprocess_capability {
const char *name;

/*
* subprocess_handshake will "|=" this value to supported_capabilities
* if the server reports that it supports this capability.
*/
unsigned int flag;
};

/* subprocess functions */ /* subprocess functions */


/* Function to test two subprocess hashmap entries for equality. */
extern int cmd2process_cmp(const void *unused_cmp_data, extern int cmd2process_cmp(const void *unused_cmp_data,
const struct subprocess_entry *e1, const struct subprocess_entry *e1,
const struct subprocess_entry *e2, const struct subprocess_entry *e2,
const void *unused_keydata); const void *unused_keydata);


/*
* User-supplied function to initialize the sub-process. This is
* typically used to negotiate the interface version and capabilities.
*/
typedef int(*subprocess_start_fn)(struct subprocess_entry *entry); typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);

/* Start a subprocess and add it to the subprocess hashmap. */
int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd, int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
subprocess_start_fn startfn); subprocess_start_fn startfn);


/* Kill a subprocess and remove it from the subprocess hashmap. */
void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry); void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);


/* Find a subprocess in the subprocess hashmap. */
struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd); struct subprocess_entry *subprocess_find_entry(struct hashmap *hashmap, const char *cmd);


/* subprocess helper functions */ /* subprocess helper functions */


/* Get the underlying `struct child_process` from a subprocess. */
static inline struct child_process *subprocess_get_child_process( static inline struct child_process *subprocess_get_child_process(
struct subprocess_entry *entry) struct subprocess_entry *entry)
{ {
return &entry->process; return &entry->process;
} }


/*
* Perform the version and capability negotiation as described in the "Long
* Running Filter Process" section of the gitattributes documentation using the
* given requested versions and capabilities. The "versions" and "capabilities"
* parameters are arrays terminated by a 0 or blank struct.
*
* This function is typically called when a subprocess is started (as part of
* the "startfn" passed to subprocess_start).
*/
int subprocess_handshake(struct subprocess_entry *entry,
const char *welcome_prefix,
int *versions,
int *chosen_version,
struct subprocess_capability *capabilities,
unsigned int *supported_capabilities);

/* /*
* Helper function that will read packets looking for "status=<foo>" * Helper function that will read packets looking for "status=<foo>"
* key/value pairs and return the value from the last "status" packet * key/value pairs and return the value from the last "status" packet

180
t/t0021-conversion.sh
View File

@ -28,7 +28,7 @@ file_size () {
} }


filter_git () { filter_git () {
rm -f rot13-filter.log && rm -f *.log &&
git "$@" git "$@"
} }


@ -42,10 +42,10 @@ test_cmp_count () {
for FILE in "$expect" "$actual" for FILE in "$expect" "$actual"
do do
sort "$FILE" | uniq -c | sort "$FILE" | uniq -c |
sed -e "s/^ *[0-9][0-9]*[ ]*IN: /x IN: /" >"$FILE.tmp" && sed -e "s/^ *[0-9][0-9]*[ ]*IN: /x IN: /" >"$FILE.tmp"
mv "$FILE.tmp" "$FILE" || return
done && done &&
test_cmp "$expect" "$actual" test_cmp "$expect.tmp" "$actual.tmp" &&
rm "$expect.tmp" "$actual.tmp"
} }


# Compare two files but exclude all `clean` invocations because Git can # Compare two files but exclude all `clean` invocations because Git can
@ -56,10 +56,10 @@ test_cmp_exclude_clean () {
actual=$2 actual=$2
for FILE in "$expect" "$actual" for FILE in "$expect" "$actual"
do do
grep -v "IN: clean" "$FILE" >"$FILE.tmp" && grep -v "IN: clean" "$FILE" >"$FILE.tmp"
mv "$FILE.tmp" "$FILE"
done && done &&
test_cmp "$expect" "$actual" test_cmp "$expect.tmp" "$actual.tmp" &&
rm "$expect.tmp" "$actual.tmp"
} }


# Check that the contents of two files are equal and that their rot13 version # Check that the contents of two files are equal and that their rot13 version
@ -342,7 +342,7 @@ test_expect_success 'diff does not reuse worktree files that need cleaning' '
' '


test_expect_success PERL 'required process filter should filter data' ' test_expect_success PERL 'required process filter should filter data' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
test_config_global filter.protocol.required true && test_config_global filter.protocol.required true &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
@ -375,7 +375,7 @@ test_expect_success PERL 'required process filter should filter data' '
IN: clean testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK] IN: clean testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK]
STOP STOP
EOF EOF
test_cmp_count expected.log rot13-filter.log && test_cmp_count expected.log debug.log &&


git commit -m "test commit 2" && git commit -m "test commit 2" &&
rm -f test2.r "testsubdir/test3 '\''sq'\'',\$x=.r" && rm -f test2.r "testsubdir/test3 '\''sq'\'',\$x=.r" &&
@ -388,7 +388,7 @@ test_expect_success PERL 'required process filter should filter data' '
IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK] IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


filter_git checkout --quiet --no-progress empty-branch && filter_git checkout --quiet --no-progress empty-branch &&
cat >expected.log <<-EOF && cat >expected.log <<-EOF &&
@ -397,7 +397,7 @@ test_expect_success PERL 'required process filter should filter data' '
IN: clean test.r $S [OK] -- OUT: $S . [OK] IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


filter_git checkout --quiet --no-progress master && filter_git checkout --quiet --no-progress master &&
cat >expected.log <<-EOF && cat >expected.log <<-EOF &&
@ -409,7 +409,7 @@ test_expect_success PERL 'required process filter should filter data' '
IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK] IN: smudge testsubdir/test3 '\''sq'\'',\$x=.r $S3 [OK] -- OUT: $S3 . [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r && test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r && test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
@ -419,7 +419,7 @@ test_expect_success PERL 'required process filter should filter data' '


test_expect_success PERL 'required process filter takes precedence' ' test_expect_success PERL 'required process filter takes precedence' '
test_config_global filter.protocol.clean false && test_config_global filter.protocol.clean false &&
test_config_global filter.protocol.process "rot13-filter.pl clean" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean" &&
test_config_global filter.protocol.required true && test_config_global filter.protocol.required true &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
@ -439,12 +439,12 @@ test_expect_success PERL 'required process filter takes precedence' '
IN: clean test.r $S [OK] -- OUT: $S . [OK] IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP STOP
EOF EOF
test_cmp_count expected.log rot13-filter.log test_cmp_count expected.log debug.log
) )
' '


test_expect_success PERL 'required process filter should be used only for "clean" operation only' ' test_expect_success PERL 'required process filter should be used only for "clean" operation only' '
test_config_global filter.protocol.process "rot13-filter.pl clean" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean" &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
( (
@ -462,7 +462,7 @@ test_expect_success PERL 'required process filter should be used only for "clean
IN: clean test.r $S [OK] -- OUT: $S . [OK] IN: clean test.r $S [OK] -- OUT: $S . [OK]
STOP STOP
EOF EOF
test_cmp_count expected.log rot13-filter.log && test_cmp_count expected.log debug.log &&


rm test.r && rm test.r &&


@ -474,12 +474,12 @@ test_expect_success PERL 'required process filter should be used only for "clean
init handshake complete init handshake complete
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log test_cmp_exclude_clean expected.log debug.log
) )
' '


test_expect_success PERL 'required process filter should process multiple packets' ' test_expect_success PERL 'required process filter should process multiple packets' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
test_config_global filter.protocol.required true && test_config_global filter.protocol.required true &&


rm -rf repo && rm -rf repo &&
@ -514,7 +514,7 @@ test_expect_success PERL 'required process filter should process multiple packet
IN: clean 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK] IN: clean 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
STOP STOP
EOF EOF
test_cmp_count expected.log rot13-filter.log && test_cmp_count expected.log debug.log &&


rm -f *.file && rm -f *.file &&


@ -529,7 +529,7 @@ test_expect_success PERL 'required process filter should process multiple packet
IN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK] IN: smudge 3pkt_2+1.file $(($S*2+1)) [OK] -- OUT: $(($S*2+1)) ... [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


for FILE in *.file for FILE in *.file
do do
@ -539,7 +539,7 @@ test_expect_success PERL 'required process filter should process multiple packet
' '


test_expect_success PERL 'required process filter with clean error should fail' ' test_expect_success PERL 'required process filter with clean error should fail' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
test_config_global filter.protocol.required true && test_config_global filter.protocol.required true &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
@ -558,7 +558,7 @@ test_expect_success PERL 'required process filter with clean error should fail'
' '


test_expect_success PERL 'process filter should restart after unexpected write failure' ' test_expect_success PERL 'process filter should restart after unexpected write failure' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
( (
@ -579,7 +579,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f
git add . && git add . &&
rm -f *.r && rm -f *.r &&


rm -f rot13-filter.log && rm -f debug.log &&
git checkout --quiet --no-progress . 2>git-stderr.log && git checkout --quiet --no-progress . 2>git-stderr.log &&


grep "smudge write error at" git-stderr.log && grep "smudge write error at" git-stderr.log &&
@ -588,14 +588,14 @@ test_expect_success PERL 'process filter should restart after unexpected write f
cat >expected.log <<-EOF && cat >expected.log <<-EOF &&
START START
init handshake complete init handshake complete
IN: smudge smudge-write-fail.r $SF [OK] -- OUT: $SF [WRITE FAIL] IN: smudge smudge-write-fail.r $SF [OK] -- [WRITE FAIL]
START START
init handshake complete init handshake complete
IN: smudge test.r $S [OK] -- OUT: $S . [OK] IN: smudge test.r $S [OK] -- OUT: $S . [OK]
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK] IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r && test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r && test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
@ -609,7 +609,7 @@ test_expect_success PERL 'process filter should restart after unexpected write f
' '


test_expect_success PERL 'process filter should not be restarted if it signals an error' ' test_expect_success PERL 'process filter should not be restarted if it signals an error' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
( (
@ -634,12 +634,12 @@ test_expect_success PERL 'process filter should not be restarted if it signals a
cat >expected.log <<-EOF && cat >expected.log <<-EOF &&
START START
init handshake complete init handshake complete
IN: smudge error.r $SE [OK] -- OUT: 0 [ERROR] IN: smudge error.r $SE [OK] -- [ERROR]
IN: smudge test.r $S [OK] -- OUT: $S . [OK] IN: smudge test.r $S [OK] -- OUT: $S . [OK]
IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK] IN: smudge test2.r $S2 [OK] -- OUT: $S2 . [OK]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r && test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.r &&
test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r && test_cmp_committed_rot13 "$TEST_ROOT/test2.o" test2.r &&
@ -648,7 +648,7 @@ test_expect_success PERL 'process filter should not be restarted if it signals a
' '


test_expect_success PERL 'process filter abort stops processing of all further files' ' test_expect_success PERL 'process filter abort stops processing of all further files' '
test_config_global filter.protocol.process "rot13-filter.pl clean smudge" && test_config_global filter.protocol.process "rot13-filter.pl debug.log clean smudge" &&
rm -rf repo && rm -rf repo &&
mkdir repo && mkdir repo &&
( (
@ -673,10 +673,10 @@ test_expect_success PERL 'process filter abort stops processing of all further f
cat >expected.log <<-EOF && cat >expected.log <<-EOF &&
START START
init handshake complete init handshake complete
IN: smudge abort.r $SA [OK] -- OUT: 0 [ABORT] IN: smudge abort.r $SA [OK] -- [ABORT]
STOP STOP
EOF EOF
test_cmp_exclude_clean expected.log rot13-filter.log && test_cmp_exclude_clean expected.log debug.log &&


test_cmp "$TEST_ROOT/test.o" test.r && test_cmp "$TEST_ROOT/test.o" test.r &&
test_cmp "$TEST_ROOT/test2.o" test2.r && test_cmp "$TEST_ROOT/test2.o" test2.r &&
@ -697,8 +697,124 @@ test_expect_success PERL 'invalid process filter must fail (and not hang!)' '


cp "$TEST_ROOT/test.o" test.r && cp "$TEST_ROOT/test.o" test.r &&
test_must_fail git add . 2>git-stderr.log && test_must_fail git add . 2>git-stderr.log &&
grep "does not support filter protocol version" git-stderr.log grep "expected git-filter-server" git-stderr.log
) )
' '


test_expect_success PERL 'delayed checkout in process filter' '
test_config_global filter.a.process "rot13-filter.pl a.log clean smudge delay" &&
test_config_global filter.a.required true &&
test_config_global filter.b.process "rot13-filter.pl b.log clean smudge delay" &&
test_config_global filter.b.required true &&

rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.a filter=a" >.gitattributes &&
echo "*.b filter=b" >>.gitattributes &&
cp "$TEST_ROOT/test.o" test.a &&
cp "$TEST_ROOT/test.o" test-delay10.a &&
cp "$TEST_ROOT/test.o" test-delay11.a &&
cp "$TEST_ROOT/test.o" test-delay20.a &&
cp "$TEST_ROOT/test.o" test-delay10.b &&
git add . &&
git commit -m "test commit"
) &&

S=$(file_size "$TEST_ROOT/test.o") &&
cat >a.exp <<-EOF &&
START
init handshake complete
IN: smudge test.a $S [OK] -- OUT: $S . [OK]
IN: smudge test-delay10.a $S [OK] -- [DELAYED]
IN: smudge test-delay11.a $S [OK] -- [DELAYED]
IN: smudge test-delay20.a $S [OK] -- [DELAYED]
IN: list_available_blobs test-delay10.a test-delay11.a [OK]
IN: smudge test-delay10.a 0 [OK] -- OUT: $S . [OK]
IN: smudge test-delay11.a 0 [OK] -- OUT: $S . [OK]
IN: list_available_blobs test-delay20.a [OK]
IN: smudge test-delay20.a 0 [OK] -- OUT: $S . [OK]
IN: list_available_blobs [OK]
STOP
EOF
cat >b.exp <<-EOF &&
START
init handshake complete
IN: smudge test-delay10.b $S [OK] -- [DELAYED]
IN: list_available_blobs test-delay10.b [OK]
IN: smudge test-delay10.b 0 [OK] -- OUT: $S . [OK]
IN: list_available_blobs [OK]
STOP
EOF

rm -rf repo-cloned &&
filter_git clone repo repo-cloned &&
test_cmp_count a.exp repo-cloned/a.log &&
test_cmp_count b.exp repo-cloned/b.log &&

(
cd repo-cloned &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay11.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay20.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.b &&

rm *.a *.b &&
filter_git checkout . &&
test_cmp_count ../a.exp a.log &&
test_cmp_count ../b.exp b.log &&

test_cmp_committed_rot13 "$TEST_ROOT/test.o" test.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay11.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay20.a &&
test_cmp_committed_rot13 "$TEST_ROOT/test.o" test-delay10.b
)
'

test_expect_success PERL 'missing file in delayed checkout' '
test_config_global filter.bug.process "rot13-filter.pl bug.log clean smudge delay" &&
test_config_global filter.bug.required true &&

rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.a filter=bug" >.gitattributes &&
cp "$TEST_ROOT/test.o" missing-delay.a
git add . &&
git commit -m "test commit"
) &&

rm -rf repo-cloned &&
test_must_fail git clone repo repo-cloned 2>git-stderr.log &&
cat git-stderr.log &&
grep "error: .missing-delay\.a. was not filtered properly" git-stderr.log
'

test_expect_success PERL 'invalid file in delayed checkout' '
test_config_global filter.bug.process "rot13-filter.pl bug.log clean smudge delay" &&
test_config_global filter.bug.required true &&

rm -rf repo &&
mkdir repo &&
(
cd repo &&
git init &&
echo "*.a filter=bug" >.gitattributes &&
cp "$TEST_ROOT/test.o" invalid-delay.a &&
cp "$TEST_ROOT/test.o" unfiltered
git add . &&
git commit -m "test commit"
) &&

rm -rf repo-cloned &&
test_must_fail git clone repo repo-cloned 2>git-stderr.log &&
grep "error: external filter .* signaled that .unfiltered. is now available although it has not been delayed earlier" git-stderr.log
'

test_done test_done

230
t/t0021/rot13-filter.pl
View File

@ -2,8 +2,9 @@
# Example implementation for the Git filter protocol version 2 # Example implementation for the Git filter protocol version 2
# See Documentation/gitattributes.txt, section "Filter Protocol" # See Documentation/gitattributes.txt, section "Filter Protocol"
# #
# The script takes the list of supported protocol capabilities as # The first argument defines a debug log file that the script write to.
# arguments ("clean", "smudge", etc). # All remaining arguments define a list of supported protocol
# capabilities ("clean", "smudge", etc).
# #
# This implementation supports special test cases: # This implementation supports special test cases:
# (1) If data with the pathname "clean-write-fail.r" is processed with # (1) If data with the pathname "clean-write-fail.r" is processed with
@ -17,6 +18,16 @@
# operation then the filter signals that it cannot or does not want # operation then the filter signals that it cannot or does not want
# to process the file and any file after that is processed with the # to process the file and any file after that is processed with the
# same command. # same command.
# (5) If data with a pathname that is a key in the DELAY hash is
# requested (e.g. "test-delay10.a") then the filter responds with
# a "delay" status and sets the "requested" field in the DELAY hash.
# The filter will signal the availability of this object after
# "count" (field in DELAY hash) "list_available_blobs" commands.
# (6) If data with the pathname "missing-delay.a" is processed that the
# filter will drop the path from the "list_available_blobs" response.
# (7) If data with the pathname "invalid-delay.a" is processed that the
# filter will add the path "unfiltered" which was not delayed before
# to the "list_available_blobs" response.
# #


use strict; use strict;
@ -24,9 +35,19 @@ use warnings;
use IO::File; use IO::File;


my $MAX_PACKET_CONTENT_SIZE = 65516; my $MAX_PACKET_CONTENT_SIZE = 65516;
my $log_file = shift @ARGV;
my @capabilities = @ARGV; my @capabilities = @ARGV;


open my $debug, ">>", "rot13-filter.log" or die "cannot open log file: $!"; open my $debug, ">>", $log_file or die "cannot open log file: $!";

my %DELAY = (
'test-delay10.a' => { "requested" => 0, "count" => 1 },
'test-delay11.a' => { "requested" => 0, "count" => 1 },
'test-delay20.a' => { "requested" => 0, "count" => 2 },
'test-delay10.b' => { "requested" => 0, "count" => 1 },
'missing-delay.a' => { "requested" => 0, "count" => 1 },
'invalid-delay.a' => { "requested" => 0, "count" => 1 },
);


sub rot13 { sub rot13 {
my $str = shift; my $str = shift;
@ -64,7 +85,7 @@ sub packet_bin_read {


sub packet_txt_read { sub packet_txt_read {
my ( $res, $buf ) = packet_bin_read(); my ( $res, $buf ) = packet_bin_read();
unless ( $buf =~ s/\n$// ) { unless ( $buf eq '' or $buf =~ s/\n$// ) {
die "A non-binary line MUST be terminated by an LF."; die "A non-binary line MUST be terminated by an LF.";
} }
return ( $res, $buf ); return ( $res, $buf );
@ -99,6 +120,7 @@ packet_flush();


( packet_txt_read() eq ( 0, "capability=clean" ) ) || die "bad capability"; ( packet_txt_read() eq ( 0, "capability=clean" ) ) || die "bad capability";
( packet_txt_read() eq ( 0, "capability=smudge" ) ) || die "bad capability"; ( packet_txt_read() eq ( 0, "capability=smudge" ) ) || die "bad capability";
( packet_txt_read() eq ( 0, "capability=delay" ) ) || die "bad capability";
( packet_bin_read() eq ( 1, "" ) ) || die "bad capability end"; ( packet_bin_read() eq ( 1, "" ) ) || die "bad capability end";


foreach (@capabilities) { foreach (@capabilities) {
@ -109,88 +131,142 @@ print $debug "init handshake complete\n";
$debug->flush(); $debug->flush();


while (1) { while (1) {
my ($command) = packet_txt_read() =~ /^command=(.+)$/; my ( $command ) = packet_txt_read() =~ /^command=(.+)$/;
print $debug "IN: $command"; print $debug "IN: $command";
$debug->flush(); $debug->flush();


my ($pathname) = packet_txt_read() =~ /^pathname=(.+)$/; if ( $command eq "list_available_blobs" ) {
print $debug " $pathname"; # Flush
$debug->flush(); packet_bin_read();


if ( $pathname eq "" ) { foreach my $pathname ( sort keys %DELAY ) {
die "bad pathname '$pathname'"; if ( $DELAY{$pathname}{"requested"} >= 1 ) {
} $DELAY{$pathname}{"count"} = $DELAY{$pathname}{"count"} - 1;

if ( $pathname eq "invalid-delay.a" ) {
# Flush # Send Git a pathname that was not delayed earlier
packet_bin_read(); packet_txt_write("pathname=unfiltered");

}
my $input = ""; if ( $pathname eq "missing-delay.a" ) {
{ # Do not signal Git that this file is available
binmode(STDIN); } elsif ( $DELAY{$pathname}{"count"} == 0 ) {
my $buffer; print $debug " $pathname";
my $done = 0; packet_txt_write("pathname=$pathname");
while ( !$done ) { }
( $done, $buffer ) = packet_bin_read();
$input .= $buffer;
}
print $debug " " . length($input) . " [OK] -- ";
$debug->flush();
}

my $output;
if ( $pathname eq "error.r" or $pathname eq "abort.r" ) {
$output = "";
}
elsif ( $command eq "clean" and grep( /^clean$/, @capabilities ) ) {
$output = rot13($input);
}
elsif ( $command eq "smudge" and grep( /^smudge$/, @capabilities ) ) {
$output = rot13($input);
}
else {
die "bad command '$command'";
}

print $debug "OUT: " . length($output) . " ";
$debug->flush();

if ( $pathname eq "error.r" ) {
print $debug "[ERROR]\n";
$debug->flush();
packet_txt_write("status=error");
packet_flush();
}
elsif ( $pathname eq "abort.r" ) {
print $debug "[ABORT]\n";
$debug->flush();
packet_txt_write("status=abort");
packet_flush();
}
else {
packet_txt_write("status=success");
packet_flush();

if ( $pathname eq "${command}-write-fail.r" ) {
print $debug "[WRITE FAIL]\n";
$debug->flush();
die "${command} write error";
}

while ( length($output) > 0 ) {
my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
packet_bin_write($packet);
# dots represent the number of packets
print $debug ".";
if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
}
else {
$output = "";
} }
} }

packet_flush(); packet_flush();

print $debug " [OK]\n"; print $debug " [OK]\n";
$debug->flush(); $debug->flush();
packet_txt_write("status=success");
packet_flush(); packet_flush();
} }
else {
my ( $pathname ) = packet_txt_read() =~ /^pathname=(.+)$/;
print $debug " $pathname";
$debug->flush();

if ( $pathname eq "" ) {
die "bad pathname '$pathname'";
}

# Read until flush
my ( $done, $buffer ) = packet_txt_read();
while ( $buffer ne '' ) {
if ( $buffer eq "can-delay=1" ) {
if ( exists $DELAY{$pathname} and $DELAY{$pathname}{"requested"} == 0 ) {
$DELAY{$pathname}{"requested"} = 1;
}
} else {
die "Unknown message '$buffer'";
}

( $done, $buffer ) = packet_txt_read();
}

my $input = "";
{
binmode(STDIN);
my $buffer;
my $done = 0;
while ( !$done ) {
( $done, $buffer ) = packet_bin_read();
$input .= $buffer;
}
print $debug " " . length($input) . " [OK] -- ";
$debug->flush();
}

my $output;
if ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{"output"} ) {
$output = $DELAY{$pathname}{"output"}
}
elsif ( $pathname eq "error.r" or $pathname eq "abort.r" ) {
$output = "";
}
elsif ( $command eq "clean" and grep( /^clean$/, @capabilities ) ) {
$output = rot13($input);
}
elsif ( $command eq "smudge" and grep( /^smudge$/, @capabilities ) ) {
$output = rot13($input);
}
else {
die "bad command '$command'";
}

if ( $pathname eq "error.r" ) {
print $debug "[ERROR]\n";
$debug->flush();
packet_txt_write("status=error");
packet_flush();
}
elsif ( $pathname eq "abort.r" ) {
print $debug "[ABORT]\n";
$debug->flush();
packet_txt_write("status=abort");
packet_flush();
}
elsif ( $command eq "smudge" and
exists $DELAY{$pathname} and
$DELAY{$pathname}{"requested"} == 1
) {
print $debug "[DELAYED]\n";
$debug->flush();
packet_txt_write("status=delayed");
packet_flush();
$DELAY{$pathname}{"requested"} = 2;
$DELAY{$pathname}{"output"} = $output;
}
else {
packet_txt_write("status=success");
packet_flush();

if ( $pathname eq "${command}-write-fail.r" ) {
print $debug "[WRITE FAIL]\n";
$debug->flush();
die "${command} write error";
}

print $debug "OUT: " . length($output) . " ";
$debug->flush();

while ( length($output) > 0 ) {
my $packet = substr( $output, 0, $MAX_PACKET_CONTENT_SIZE );
packet_bin_write($packet);
# dots represent the number of packets
print $debug ".";
if ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {
$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );
}
else {
$output = "";
}
}
packet_flush();
print $debug " [OK]\n";
$debug->flush();
packet_flush();
}
}
} }

2
unpack-trees.c
View File

@ -380,6 +380,7 @@ static int check_updates(struct unpack_trees_options *o)
if (should_update_submodules() && o->update && !o->dry_run) if (should_update_submodules() && o->update && !o->dry_run)
reload_gitmodules_file(index, &state); reload_gitmodules_file(index, &state);


enable_delayed_checkout(&state);
for (i = 0; i < index->cache_nr; i++) { for (i = 0; i < index->cache_nr; i++) {
struct cache_entry *ce = index->cache[i]; struct cache_entry *ce = index->cache[i];


@ -394,6 +395,7 @@ static int check_updates(struct unpack_trees_options *o)
} }
} }
} }
errs |= finish_delayed_checkout(&state);
stop_progress(&progress); stop_progress(&progress);
if (o->update) if (o->update)
git_attr_set_direction(GIT_ATTR_CHECKIN, NULL); git_attr_set_direction(GIT_ATTR_CHECKIN, NULL);