`tsm-client` uses a global configuration
file that must contain coordinates for each
server that it is supposed to contact.
This configuration consists of text
lines with key-value pairs.
In the NixOS module, these servers may be declared
with an attribute set, where the attribute name
defines an alias for the server, and the value
is again an attribute set with the settings for
the respective server.
This is organized as an option of type `attrsOf submodule...`.
Before this commit:
Important settings have their own option within
the submodule. For everything else, there is
the "catch-all" option `extraConfig` that may
be used to declare any key-value pairs.
There is also `text` that can be used to
add arbitrary text to each server's
section in the global config file.
After this commit:
`extraConfig` and `text` are gone,
the attribute names and values of each server's attribute
set are translated directly into key-value pairs,
with the following notable rules:
* Lists are translated into multiple lines
with the same key, as such is permitted by
the software for certain keys.
* `null` may be used to override/shadow a value that
is defined elsewhere and hides the corresponding key.
Those "important settings" that have previously been
defined as dedicated options are still defined as such,
but they have been renamed to match their
corresponding key names in the configuration file.
There is a notable exception:
"Our" boolean option `genPasswd` influences the "real"
option `passwordaccess', but the latter one is
uncomfortable to use and might lead
to undesirable outcome if used the wrong way.
So it seems advisable to keep the boolean option
and the warning in its description.
To this end, the value of `getPasswd` itself is
later filtered out when the config file is generated.
The tsm-backup service module and the vm test are adapted.
Migration code will be added in a separate
commit to permit easy reversal later, when the
migration code is no longer deemed necessary.
With the tsm-client 8.1.19.0 release,
IBM renamed the product brand from
"IBM Spectrum Protect" to "IBM Storage Protect":
https://www.ibm.com/support/pages/node/6964770 .
The package already got updated in commits
5ff5b2ae4c and
a4b7a62532 .
The commit at hand updates the modules accordingly.
The --rsyncable option changes the behavior of gzip/zstd so that the
resulting files can be incrementally backed up easily. Tools like Borg,
rsync and xdelta can make use their deduplication/diff mechanisms more
easily.
In my local testing, this resulted in a 2% size increase for backup
files.
Signed-off-by: Sefa Eyeoglu <contact@scrumplex.net>
The sshKey options do not need to be a valid path at build time. Using
string instead allow use case when the path is not known at build time
such as when using systemd credentials (e.g. `sshKey =
"\${CREDENTIALS_DIRECTORY}/zfs-replication_ed25519";`).
As described in the release lifecycle docs from postgresql[1], v11 will
stop receiving fixes as of Nov 9 2023. This means it's EOL throughout
the entire lifetime of 23.11, so let's drop it now.
A lot of examples are also referencing postgresql_11. Where it's
sensible, use postgresql_15 as example now to avoid confusion.
This is also handy because the LLVM 16 fix for postgresql is not
available for postgresql 11 ;-)
[1] https://www.postgresql.org/support/versioning/
Also, add a test to verify that it works.
This change also removes the part of custom package test that verifies
that the correct paths are provided. This is already tested by restore
tests.
Before this change, setting both paths and dynamicFileFrom would cause
paths to be silently ignored. Making that actually apply the obvious
interpretation seems to me to be strictly better than prohibiting the
two from being set at the same time.
The module falsely disabled postgresql completely when the bacula-sd was not enabled.
Quotation marks are not necessary and only useful in `name` fields.
To reduce the danger of accidentally exposing sensitive files processed
by a restic backup to other services/users, enable the `PrivateTmp=`
feature of restic service units, which provides a per service isolation
of `/tmp` and `/var/tmp`.
Co-authored-by: Daniel Nagy <danielnagy@posteo.de>
The restic repository cache location defaults to ~/.cache/restic when
not overwritten either by the --cache-dir command line parameter or the
universal RESTIC_CACHE_DIR environment variable.
Currently, the --cache-dir variable is set to only some restic commands,
but, e.g., not to the unit's preStart command for the module's
initialize option. This results in two distinct cache locations, one at
~/.cache/restic for the initialize commands and one at the configured
--cache-dir location for the restic backup command.
By explicitly setting RESTIC_CACHE_DIR for the unit, only one cache at
the correct location will be used.
https://restic.readthedocs.io/en/v0.15.1/manual_rest.html#caching
this converts meta.doc into an md pointer, not an xml pointer. since we
no longer need xml for manual chapters we can also remove support for
manual chapters from md-to-db.sh
since pandoc converts smart quotes to docbook quote elements and our
nixos-render-docs does not we lose this distinction in the rendered
output. that's probably not that bad, our stylesheet didn't make use of
this anyway (and pre-23.05 versions of the chapters didn't use quote
elements either).
also updates the nixpkgs manual to clarify that option docs support all
extensions (although it doesn't support headings at all, so heading
anchors don't work by extension).
This provides an easy way to specify exclude patterns in config. It was
already possible via extraBackupOptions; this change creates a simpler,
similar to other backup services, way to specify them.
makes sure that program listing tags are separated from their contents
by exactly a newline character. this makes the markdown translation
easier to verify (since no new newlines need to be inserted), and
there's no rendering difference anyway.
Invoke `install` separately for each directory to get ownership right --
i.e. not always owned by root. When owned by root, user sessions break
as no user processes are allowed to create directores there. On normal
systems the directories already exist, but in clean environments / NixOS
test VMs, the bug shows.
Before:
$ namei -l /home/user1/.cache/borg
f: /home/user1/.cache/borg
drwxr-xr-x root root /
drwxr-xr-x root root home
drwx------ user1 users user1
drwxr-xr-x root root .cache
drwxr-xr-x user1 users borg
After:
$ namei -l /home/user1/.cache/borg
f: /home/user1/.cache/borg
drwxr-xr-x root root /
drwxr-xr-x root root home
drwx------ user1 users user1
drwxr-xr-x user1 users .cache
drwxr-xr-x user1 users borg
Adds a new option for backup jobs `inhibitsSleep` which prevents
the system from going to sleep while a backup is in progress.
Uses `systemd-inhibit`, which holds a "lock" that prevents the
system from sleeping while the process it invokes is running.
This did require wrapping the existing backup script using
`writeShellScript` so that it could be run by `systemd-inhibit`.
Configures the `--cache-dir` parameter for the prune and check commands run after backing up. For `check`, also adds a `checkOpts` flag to enable using the cache, since that is disabled by default.
conversions were done using https://github.com/pennae/nix-doc-munge
using (probably) rev f34e145 running
nix-doc-munge nixos/**/*.nix
nix-doc-munge --import nixos/**/*.nix
the tool ensures that only changes that could affect the generated
manual *but don't* are committed, other changes require manual review
and are discarded.
this mostly means marking options that use markdown already
appropriately and making a few adjustments so they still render
correctly. notable for nftables we have to transform the md links
because the manpage would not render them correctly otherwise.
now nix-doc-munge will not introduce whitespace changes when it replaces
manpage references with the MD equivalent.
no change to the manpage, changes to the HTML manual are whitespace only.
make (almost) all links appear on only a single line, with no
unnecessary whitespace, using double quotes for attributes. this lets us
automatically convert them to markdown easily.
the few remaining links are extremely long link in a gnome module, we'll
come back to those at a later date.
we can't embed syntactic annotations of this kind in markdown code
blocks without yet another extension. replaceable is rare enough to make
this not much worth it, so we'll go with «thing» instead. the module
system already uses this format for its placeholder names in attrsOf
paths.
the conversion procedure is simple:
- find all things that look like options, ie calls to either `mkOption`
or `lib.mkOption` that take an attrset. remember the attrset as the
option
- for all options, find a `description` attribute who's value is not a
call to `mdDoc` or `lib.mdDoc`
- textually convert the entire value of the attribute to MD with a few
simple regexes (the set from mdize-module.sh)
- if the change produced a change in the manual output, discard
- if the change kept the manual unchanged, add some text to the
description to make sure we've actually found an option. if the
manual changes this time, keep the converted description
this procedure converts 80% of nixos options to markdown. around 2000
options remain to be inspected, but most of those fail the "does not
change the manual output check": currently the MD conversion process
does not faithfully convert docbook tags like <code> and <package>, so
any option using such tags will not be converted at all.
That way the `backupCleanupCommand` can also run when the backup service
failed for some reason.
Fixes: #182089.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Transform exit handlers of the form
trap cleanup EXIT [INT] [TERM] [QUIT] [HUP] [ERR]
(where cleanup is idempotent)
to
trap cleanup EXIT
This fixes a common bash antipattern.
Each of the above signals causes the script to exit. For each signal,
bash first handles the signal by running `cleanup` and then runs
`cleanup` again when handling EXIT.
(Exception: `vscode/*` prevents the second run of `cleanup` by removing
the trap in cleanup`).
Simplify the cleanup logic by just trapping exit, which is always run
when the script exits due to any of the above signals.
Note: In case of borgbackup, the exit handler is not idempotent, but just
trapping EXIT guarantees that it's only run once.
The backupPrepareCommand and backupCleanupCommand options offer a way to
run a script to prepare for backup and then cleanup it once finish.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Allow providing the repository as a file, useful when we don't want it
being stored in the Git repository as plain text.
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
automysqldump passes the --events flag, but without the EVENTS permission a error occures:
> mysqldump: Couldn't execute 'show events': Access denied for user 'automysqlbackup'@'localhost' to database 'mysql' (1044)
This allows btrbk instances without a triggering timer by setting
`onCalendar` to `null`.
This is useful for manual-starting only btrbk backup settings.
fixes#158802
Sets the mysql backup systemd service type to "oneshot" to ensure the
service is marked as started after the backup script fully proceeds. This
allows to reliably depend on completing of this service by other services.
The module option type `nonEmptyStr` was introduced in commit
a3c5f0cba8
The tsm modules previously simply used
`strMatching ".+"` to prevent empty option strings,
but the new type is more thorough as
it also catches space-only strings.
This enables some systemd sandboxing
options for the `tsm-backup.service`.
Those settings have been determined by expermentation.
This commit tries hard to protect the filesystem from
write access, but not to hide anything from read access,
so users can backup all files they choose to backup.
An exception are API filesystems (`/dev`, `/proc`, `/sys`):
As their "files" are not stored on persistent storage,
they are sandboxed away as much as possible.
Note that the service still has to run with root
privileges to reach files with limited access permissions.
The obvious alternative to use a dedicated user account and
the `CAP_DAC_READ_SEARCH` capability to permit system-wide
read access while blocking write access does not work.
Experiments have shown that `dsmc` verifies access permissions
for each file before attempting to open it for reading.
Hence `dsmc` refuses to copy files where the file permission
mode blocks read access -- even if process capabilities
would allow it to proceed irrespective of permissions.
When `privateRepos = true`, the service will not start if the `.htpasswd` does not exist.
Use `systemd-tmpfiles` to autocreate an (empty) file to ensure the service can boot
before actual `htpasswd` contents are registered.
This is safe as restic-rest-server will deny all entry if the file is empty.
most modules can be evaluated for their documentation in a very
restricted environment that doesn't include all of nixpkgs. this
evaluation can then be cached and reused for subsequent builds, merging
only documentation that has changed into the cached set. since nixos
ships with a large number of modules of which only a few are used in any
given config this can save evaluation a huge percentage of nixos
options available in any given config.
in tests of this caching, despite having to copy most of nixos/, saves
about 80% of the time needed to build the system manual, or about two
second on the machine used for testing. build time for a full system
config shrank from 9.4s to 7.4s, while turning documentation off
entirely shortened the build to 7.1s.
Other services such as minecraft-server and plex allow configuration of
the dataDir option, allowing the files stored by each service to be in a
custom location.
Co-authored-by: Aaron Andersen <aaron@fosslib.net>
unfortunately we don't have a good way to represent defaults that
reference other values of the current submodule, so we just use the
relative path of the referenced value and assume that the submodule was
declared as `rec`.
This is done as the s3CredentialsFile specifies the environmentFile
for the systemd service, which can be used for more than just s3.
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
borg is able to process stdin during backups when backing up the special path -,
which can be very useful for backing up things that can be streamed (eg database
dumps, zfs snapshots).
This is to address a regression introduced in #131118.
When syncing the first dataset, syncoid expects that the target
dataset doesn't exist to have a clean slate to work with. So during
runtime we'll check if the target dataset does exist and if it doesn't
- delegate the permissions to the parent dataset instead.
But then, on unallow, we do the unallow on both the target and the
parent since the target dataset should have been created at this
point, so the unallow can't know which dataset that got permissions
just by which datasets exists.
I noticed this minor grammar mistake when running update.nix, and then
while grepping to find the source I noticed we had it a few times in
Nixpkgs. Just as easy to fix treewide as it was to fix the one
occurrence I noticed.
This option allows basic configuration of the compression technique
used in the backup script. Specifically it adds `none` and `zstd` as
new alternatives, keeping `gzip` as the default.
When sending or receiving datasets with the old implementation it
wouldn't matter which dataset we were sending or receiving, we would
always delegate permissions to the entire pool.
Previously, a failed backup would always overwrite ${db}.sql.gz,
because the bash `>` redirect truncates the file; even if the
backup was going to fail.
On the next run, the ${db}.prev.sql.gz backup would be
overwritten by the bad ${db}.sql.gz.
Now, if the backup fails, the ${db}.in-progress.sql.gz is in an
unknown state, but ${db}.sql.gz will not be written.
On the next run, ${db}.prev.sql.gz (our only good backup) will
not be overwritten because ${db}.sql.gz does not exist.
Or … none! Because forcing a string always results in an OnCalender=
setting, but an empty string leads to an empty value.
> postgresqlBackup-hass.timer: Timer unit lacks value setting. Refusing.
or
> postgresqlBackup-miniflux.timer: Cannot add dependency job, ignoring: Unit postgresqlBackup-miniflux.timer has a bad unit file setting.
I require the postgresqlBackup in my borgbackup unit, so I don't
strictly need the timer and could previously set it to an empty list.
Current module add backups forever, with no way to prune old ones.
Add an option to remove backups after n full backups or after some
amount of time.
Also run duplicity cleanup to clean unused files in case some previous
backup was improperly interrupted.
As the only consequence of isSystemUser is that if the uid is null then
it's allocated below 500, if a user has uid = something below 500 then
we don't require isSystemUser to be set.
Motivation: https://github.com/NixOS/nixpkgs/issues/112647
By default, restic determines the location of the cache based on the XDG
base dir specification, which is `~/.cache/restic` when the environment
variable `$XDG_CACHE_HOME` isn't set.
As restic is executed as root by default, this resulted in the cache being
written to `/root/.cache/restic`, which is not quite right for a system
service and also meant, multiple backup services would use the same cache
directory - potentially causing issues with locking, data corruption,
etc.
The goal was to ensure, restic uses the correct cache location for a
system service - one cache per backup specification, using `/var/cache`
as the base directory for it.
systemd sets the environment variable `$CACHE_DIRECTORY` once
`CacheDirectory=` is defined, but restic doesn't change its behavior
based on the presence of this environment variable.
Instead, the specifier [1] `%C` can be used to point restic explicitly
towards the correct cache location using the `--cache-dir` argument.
Furthermore, the `CacheDirectoryMode=` was set to `0700`, as the default
of `0755` is far too open in this case, as the cache might contain
sensitive data.
[1] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers
