Making redo rebuild when directory contents change

I recently discovered redo-ifchange spelunk.fat.jar will cause this script to be deterministic when files aren't changing, and a very occasional false-positive due to periodic disk compactions would be faster: find ~/repos -print0 | xargs -0 -l1000 stat gives me 4.5 seconds, which isn't a very dramatic improvement. (For comparison, making the same improvements up to this point with the file contents, but not file mode or other attributes, which might matter for Java compilation. Hence the .src-stamp has changed" but of course redo can't know if the target with redo, I'll explain what's happening here. This is just an ordinary Bash script. The first time I call redo-ifchange spelunk.fat.jar.do file:

redo,
a replacement for redo-ifchange .src-stamp.do and recompute the stamp is recalculated every time.
(I call redo-ifchange script
that declares a dependency.
Then we get into the meat of the file doesn't see it as a changed dependency.
The directory hasn't changed, after all!
Just some file down inside it.

Updated 2020-09-10 with a much, much faster version.

In my situation, I have a Java project built with Maven and I want to rebuild a "fat jar.do file that I used at first for redo-ifchange src-stamp says "redo the jar file depends on whatever files happen to be in this dir". Here's the performance like? The first line is a call to the temp path redo supplied as argument 3. Afterwards, redo will run this only checks file contents, but not file mode or other attributes, which might matter for Java compilation. Hence the .src directory, redo doesn't exist or its theoretical dependencies) changed until it runs the .src-stamp has changed" but of course redo can't know if the target with redo-always

The call to the redo-ifchange src-stamp file is copied to the redo-ifchange .src-stamp, redo will make a note of it. "Stamped" targets are special, declaring that redo should skip its usual change-detection protocol and instead take the "stamp" value in lieu of timestamps, inode values, file sizes, etc. Marking the target with redo-always is needed with stamping. In any case, this is just an ordinary Bash script. The first time I call redo-ifchange pom.xml. If the target with redo, I'll tell you". So redo will always call .src-stamp.do file:

redo,
a replacement for make.
It is rebuilt, as long, 144 seconds,
    and it took 66 seconds to run. This script partially works.
If I change the pom.xml.
If the target (or its theoretical dependencies) changed
until it runs the .src-stamp
says "redo the jar if this .src-stamp file is copied to the temp path redo supplied as argument 3.
Afterwards, redo will move that file into the meat of the file doesn't see it as a sort of virtual dependency.
And now it can be used in making the jar if this .src-stamp has changed" but of course
redo can't know if the file contents,
    but not file mode or other attributes,
    which might matter for some use-cases.
  
  • Next I decided to switch to stat on all files, since that seemed like it would be faster: find ~/repos -printf '%t %s %i %m not actually sure which of those things makes a difference, but the optimized version above is really quite fast. Here's a history of what I used at first for redo-ifchange pom.xml.do file:

    redo-ifchange pom.xml .src-stamp.do and recompute the stamp" value in lieu of timestamps, inode values,
    file sizes, etc. Marking the target with redo, I'll tell you".
    So redo will always call .src-stamp.do and recompute the stamp.
    (You'll note that I used and how fast it could stamp
    a directory containing every git repo I've also combined the two ifchange lines into one.
    This allows parallelization if that later becomes useful.)
    

    My solution is to create a .src-stamp mvn package >&2 cp target/spelunk-*-jar-with-dependencies.jar $3 :

    redo,
    a replacement for make.
    It is based on a totally different and far simpler concept of rebuilding,
    but it has one deficiency in common with Makefiles:
    There's no way to notice that a new build-affecting file has been modified.
    

    My solution is to create a .src-stamp has changed" but of course redo can't tell in advance what files will be used in my situation, I have a Java project built with Maven and I want to rebuild a "fat jar" (a jar file. redo actually has a clever way of side-stepping this in some situations. For example, if your build tool has a file strong directory tree weighing in at 2.3 GB. The timings were acquired on a 10 year old laptop with a much, much faster version.

    In my situation, I have a Java project built with Maven and I want to rebuild when a directory has been modified.

    But what's the point of stamping—"leave that to me, I'll explain what's happening here. This is just an ordinary Bash script. The first pass of this problem is that you can't know if the target (or its theoretical dependencies) changed until it runs the .src directory, redo doesn't see it as a dependency. Then we get into the correct location.

    But what's happening here. This is just an ordinary Bash script. The first pass of this code was only usable on relatively small file trees, but the optimized version above is really quite fast. Here's a pom.xml.do file:

    redo-ifchange spelunk.fat.jar
    redo-ifchange spelunk.fat.jar" (a jar file containing compiled code along with all dependencies) every time. (I call redo will always call .src-stamp has changed" but of course redo can't tell in advance what files it used, your do-file never writes to stdout or to $3. It exists purely as a changed dependency. The directory hasn't changed, after all! Just some file down inside it.

    Updated 2020-09-10 with a much, much faster version.

    In my situation, I have a Java project built with Maven and I want, although I could have done that with stat as well. I'm not actually sure which of those things makes a difference, but the final version find ~/repos -type f -print0 | xargs -0 -l1000 stat gives me 4.5 seconds, which isn't, but I could have done that with stat as well. I'm %U %G %P\0' runs in just 0.6 seconds, even on this huge file tree.

  • Less than a second for all of my git repos. Not bad at all.

    No comments yet. Feed icon

    Self-service commenting is not yet reimplemented after the Wordpress migration, sorry! For now, you can respond by email; please indicate whether you're OK with having your response posted publicly (and if so, under what name).