As initially designed, `lib.packagesFromDirectoryRecursive` allowed
passing a string for the `directory` argument. This is necessary for
several reasons:
- `outPath` on derivations and Flake inputs is not a path.
- Derivations can be coerced to their `outPath` in string interpolation,
but that produces strings, not paths.
- `builtins.path`, bizarrely, returns a string instead of a path (not
that the documentation makes this clear).
If a path is used instead of a string here, then Nix will dutifully copy
the entire directory into a new path in the Nix store (ignored as
WONTFIX by Eelco in https://github.com/NixOS/nix/issues/9428). For
industrial use cases, this can result in an extra 10-15 seconds on every
single eval just to copy files from one spot in the Nix store to another
spot in the Nix store.
In #361424, this was changed so that `directory` must be a path,
breaking these use-cases.
I'm not really sure what happened here -- #361424 has very little
justification for why it exists, only a reference to a previous version
of the PR (#359941), which itself had very little justification given.
The description on #359941 explained that it would "Shrink the
function's code by ~2/3rd 🎉", but 60% of the reduction in size was just
deleting comments (!) and bindings like `directoryEntryIsPackage` that
helped clarify the intent of the implementation. As a result, the new
implementation is (to my eyes) more challenging to read and understand.
I think the whole thing was in service of #392800, which adds a
`newScope` argument in order "to create nested scopes for each
(sub)directory (not just the top-level one) when `newScope` is given."
Nobody noticed this regression until after the commit was merged. After
@phanirithvij pointed out the regression, @nbraud said they would
"shortly prepare a PR to fix this" [1] but did not. Later, they would
explain that they were "quite ill the last month(s)" [2], which explains
why this got forgotten about. @nbraud also requested a review from
@Gabriella439 [3], as she had reviewed the original PR adding
`lib.packagesFromDirectoryRecursive`, but not from me, the original
author of that PR. @Gabriella439 did not review the "refactor" PR, and
no attempt to contact her or myself was made after that initial request.
This behavior is admittedly rather subtle, so I'm not sure either
Gabriella or myself would have noticed the change (especially since the
relevant PR restructures the entire implementation).
While I find this a bit frustrating, I should have added a test for this
use-case in my original PR; if there was a test that relied on passing
paths in as a string, perhaps the authors modifying this code would have
noticed that the implementation was not an accident.
[1]: https://github.com/NixOS/nixpkgs/pull/361424#discussion_r1912407693
[2]: https://github.com/NixOS/nixpkgs/pull/359984#issuecomment-2775768808
[3]: https://github.com/NixOS/nixpkgs/pull/361424#issuecomment-2521308983
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
Fixes a bug preventing `recurseIntoDirectory` from changing the `directory` argument.
Moreover, `processDir` now cannot capture arguments from the `packagesFromDirectoryRecursive` call,
entirely preventing this class of bug from reoccurring should new parameters be added etc.
No functional changes.
- Centralize the logic classifying files/directories of interest, instead of
being spread between `directoryEntryIsPackage` and `directoryEntryToAttrPair`.
- Replace a composition of `mapAttrs'` and `filterAttrs` with `concatMapAttrs`.
- Simplify future improvements, such as creating nested scopes for subdirs,
or ignoring unsupported files.
* doc: migrate lib.filesystem to doc-comment format
* defintion list fixes lib/filesystem.nix
Co-authored-by: Daniel Sidhion <DanielSidhion@users.noreply.github.com>
---------
Co-authored-by: Daniel Sidhion <DanielSidhion@users.noreply.github.com>
Previously it would fail with
error: attribute 'nonexistent' missing
at nixpkgs/lib/filesystem.nix:29:10:
28| if dirOf path == path then "directory"
29| else (readDir (dirOf path)).${baseNameOf path};
| ^
30|
Previously this function couldn't handle / being passed, it would throw
an error:
error: attribute '' missing
at nixpkgs/lib/filesystem.nix:24:20:
23| */
24| pathType = path: (readDir (dirOf path)).${baseNameOf path};
| ^
25|
Consequently this also fixes the
lib.filesystem.{pathIsDirectory,pathIsRegularFile} functions.
Add a friendly function to easily return a flattened list of files
within a directory.
This is useful if you want to easily iterate or concatSep the list of
files all found within a directory.
(i.e. when constructing Java's CLASSPATH)
Style improvements
Co-authored-by: Silvan Mosberger <github@infinisil.com>
This does break the API of being able to import any lib file and get
its libs, however I'm not sure people did this.
I made this while exploring being able to swap out docFn with a stub
in #2305, to avoid functor performance problems. I don't know if that
is going to move forward (or if it is a problem or not,) but after
doing all this work figured I'd put it up anyway :)
Two notable advantages to this approach:
1. when a lib inherits another lib's functions, it doesn't
automatically get put in to the scope of lib
2. when a lib implements a new obscure functions, it doesn't
automatically get put in to the scope of lib
Using the test script (later in this commit) I got the following diff
on the API:
+ diff master fixed-lib
11764a11765,11766
> .types.defaultFunctor
> .types.defaultTypeMerge
11774a11777,11778
> .types.isOptionType
> .types.isType
11781a11786
> .types.mkOptionType
11788a11794
> .types.setType
11795a11802
> .types.types
This means that this commit _adds_ to the API, however I can't find a
way to fix these last remaining discrepancies. At least none are
_removed_.
Test script (run with nix-repl in the PATH):
#!/bin/sh
set -eux
repl() {
suff=${1:-}
echo "(import ./lib)$suff" \
| nix-repl 2>&1
}
attrs_to_check() {
repl "${1:-}" \
| tr ';' $'\n' \
| grep "\.\.\." \
| cut -d' ' -f2 \
| sed -e "s/^/${1:-}./" \
| sort
}
summ() {
repl "${1:-}" \
| tr ' ' $'\n' \
| sort \
| uniq
}
deep_summ() {
suff="${1:-}"
depth="${2:-4}"
depth=$((depth - 1))
summ "$suff"
for attr in $(attrs_to_check "$suff" | grep -v "types.types"); do
if [ $depth -eq 0 ]; then
summ "$attr" | sed -e "s/^/$attr./"
else
deep_summ "$attr" "$depth" | sed -e "s/^/$attr./"
fi
done
}
(
cd nixpkgs
#git add .
#git commit -m "Auto-commit, sorry" || true
git checkout fixed-lib
deep_summ > ../fixed-lib
git checkout master
deep_summ > ../master
)
if diff master fixed-lib; then
echo "SHALLOW MATCH!"
fi
(
cd nixpkgs
git checkout fixed-lib
repl .types
)
