Allow frontends to bidirectionally communicate with fast-import

The existing checkpoint command is very useful to force fast-import
to dump the branches out to disk so that standard Git tools can
access them and the objects they refer to.  However there was not a
way to know when fast-import had finished executing the checkpoint
and it was safe to read those refs.

The progress command can be used to make fast-import output any
message of the frontend's choosing to standard out.  The frontend
can scan for these messages using select() or poll() to monitor a
pipe connected to the standard output of fast-import.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Documentation/git-fast-import.txt

@@ -298,6 +298,11 @@ and control the current import process.  More detailed discussion
 	This command is optional and is not needed to perform
 	an import.
 
+`progress`::
+	Causes fast-import to echo the entire line to its own
+	standard output.  This command is optional and is not needed
+	to perform an import.
+
 `commit`
 ~~~~~~~~
 Create or update a branch with a new commit, recording one logical
@@ -775,6 +780,31 @@ explicit checkpointing may not be necessary.
 
 The `LF` after the command is optional (it used to be required).
 
+`progress`
+~~~~~~~~~~
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream.  The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+
+....
+	'progress' SP <any> LF
+	LF?
+....
+
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`.  The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+
+====
+	frontend | git-fast-import | sed 's/^progress //'
+====
+
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
+
 Tips and Tricks
 ---------------
 The following tips and tricks have been collected from various
@@ -867,6 +897,15 @@ This will take longer, but will also produce a smaller packfile.
 You only need to expend the effort once, and everyone using your
 project will benefit from the smaller repository.
 
+Include Some Progress Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Every once in a while have your frontend emit a `progress` message
+to fast-import.  The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
+
 
 Packfile Optimization
 ---------------------

fast-import.c

@@ -8,6 +8,7 @@ Format of STDIN stream:
         | new_tag
         | reset_branch
         | checkpoint
+        | progress
         ;
 
   new_blob ::= 'blob' lf
@@ -53,6 +54,9 @@ Format of STDIN stream:
   checkpoint ::= 'checkpoint' lf
     lf?;
 
+  progress ::= 'progress' sp not_lf* lf
+    lf?;
+
      # note: the first idnum in a stream should be 1 and subsequent
      # idnums should not have gaps between values as this will cause
      # the stream parser to reserve space for the gapped values.  An
@@ -2125,6 +2129,14 @@ static void cmd_checkpoint(void)
 	skip_optional_lf();
 }
 
+static void cmd_progress(void)
+{
+	fwrite(command_buf.buf, 1, command_buf.len - 1, stdout);
+	fputc('\n', stdout);
+	fflush(stdout);
+	skip_optional_lf();
+}
+
 static void import_marks(const char *input_file)
 {
 	char line[512];
@@ -2235,6 +2247,8 @@ int main(int argc, const char **argv)
 			cmd_reset_branch();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			cmd_checkpoint();
+		else if (!prefixcmp(command_buf.buf, "progress "))
+			cmd_progress();
 		else
 			die("Unsupported command: %s", command_buf.buf);
 	}

t/t9300-fast-import.sh

@@ -885,4 +885,35 @@ test_expect_success \
 	 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
 	 git diff expect actual'
 
+cat >input <<INPUT_END
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zstring
+COMMIT
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zof
+COMMIT
+progress Two commits down, 2 to go!
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zempty
+COMMIT
+progress Three commits down, 1 to go!
+commit refs/heads/O4
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+data <<COMMIT
+zcommits
+COMMIT
+progress I'm done!
+INPUT_END
+test_expect_success \
+	'O: progress outputs as requested by input' \
+	'git-fast-import <input >actual &&
+	 grep "progress " <input >expect &&
+	 git diff expect actual'
+
 test_done