I'm a little annoyed at myself that I only realized this _after_ #392030
got merged. But I realized that if something else is using AF_VSOCK or
you simply have another interactive test running (e.g. by another user
on a larger builder), starting up VMs in the driver fails with
qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=3: vhost-vsock: unable to set guest cid: Address already in use
Multi-user setups are broken anyways because you usually don't have
permissions to remove the VM state from another user and thus starting
the driver fails with
PermissionError: [Errno 13] Permission denied: PosixPath('/tmp/vm-state-machine')
but this is something you can work around at least.
I was considering to generate random offsets, but that's not feasible
given we need to know the numbers at eval time to inject them into the
QEMU args. Also, while we could do this via the test-driver, we should
also probe if the vsock numbers are unused making the code even more
complex for a use-case I consider rather uncommon.
Hence the solution is to do
sshBackdoor.vsockOffset = 23542;
when encountering conflicts.
Closes#180089
I realized that the previous commits relying on `sys.exit` for dealing
with `MachineError`/`RequestedAssertionFailed` exit the interactive
session which is kinda bad.
This patch uses the ipython driver: it seems to have equivalent features
such as auto-completion and doesn't stop on SystemExit being raised.
This also fixes other places where this happened such as things calling
`log.error` on the CompositeLogger.
After a discussion with tfc, we agreed that we need a distinction
between errors where the user isn't at fault (e.g. OCR failing - now
called `MachineError`) and errors where the test actually failed (now
called `RequestedAssertionFailed`).
Both get special treatment from the error handler, i.e. a `!!!` prefix
to make it easier to spot visually.
However, only `RequestedAssertionFailed` gets the shortening of the
traceback, `MachineError` exceptions may be something to report and
maintainers usually want to see the full trace.
Suggested-by: Jacek Galowicz <jacek@galowicz.de>
When doing `machine.succeed(...)` or something similar, it's now clear
that the command `...` was issued on `machine`.
Essentially, this results in the following diff in the log:
-(finished: waiting for unit default.target, in 13.47 seconds)
+machine: (finished: waiting for unit default.target, in 13.47 seconds)
(finished: subtest: foobar text lorem ipsum, in 13.47 seconds)
I think it's reasonable to also have this kind of visual distinction
here between test failures and actual errors from the test framework.
A failing `machine.require_unit_state` now lookgs like this for
instance:
!!! Traceback (most recent call last):
!!! File "<string>", line 3, in <module>
!!! machine.require_unit_state("postgresql","active")
!!!
!!! RequestedAssertionFailed: Expected unit 'postgresql' to to be in state 'active' but it is in state 'inactive'
Co-authored-by: Benoit de Chezelles <bew@users.noreply.github.com>
Replaces / Closes#345948
I tried to integrate `pytest` assertions because I like the reporting,
but I only managed to get the very basic thing and even that was messing
around a lot with its internals.
The approach in #345948 shifts too much maintenance effort to us, so
it's not really desirable either.
After discussing with Benoit on Ocean Sprint about this, we decided that
it's probably the best compromise to integrate `unittest`: it also
provides good diffs when needed, but the downside is that existing tests
don't benefit from it.
This patch essentially does the following things:
* Add a new global `t` that is an instance of a `unittest.TestCase`
class. I decided to just go for `t` given that e.g.
`tester.assertEqual` (or any other longer name) seems quite verbose.
* Use a special class for errors that get special treatment:
* The traceback is minimized to only include frames from the
testScript: in this case I don't really care about anything else and
IMHO that's just visual noise.
This is not the case for other exceptions since these may indicate a
bug and then people should be able to send the full traceback to the
maintainers.
* Display the error, but with `!!!` as prefix to make sure it's
easier to spot in between other logs.
This looks e.g. like
!!! Traceback (most recent call last):
!!! File "<string>", line 7, in <module>
!!! foo()
!!! File "<string>", line 5, in foo
!!! t.assertEqual({"foo":[1,2,{"foo":"bar"}]},{"foo":[1,2,{"bar":"foo"}],"bar":[1,2,3,4,"foo"]})
!!!
!!! NixOSAssertionError: {'foo': [1, 2, {'foo': 'bar'}]} != {'foo': [1, 2, {'bar': 'foo'}], 'bar': [1, 2, 3, 4, 'foo']}
!!! - {'foo': [1, 2, {'foo': 'bar'}]}
!!! + {'bar': [1, 2, 3, 4, 'foo'], 'foo': [1, 2, {'bar': 'foo'}]}
cleanup
kill machine (pid 9)
qemu-system-x86_64: terminating on signal 15 from pid 6 (/nix/store/wz0j2zi02rvnjiz37nn28h3gfdq61svz-python3-3.12.9/bin/python3.12)
kill vlan (pid 7)
(finished: cleanup, in 0.00 seconds)
Co-authored-by: bew <bew@users.noreply.github.com>
The current setup is really weird and definitely wrong for many cases
because it inverts the colors of the image, which is never a good idea
for GUIs. So, try to OCR three different times: once on the source image,
once with processing, and once with processing but no negation.
This should hopefully make things work at least somewhat better for GUIs.
Specifies the "last try" parameter in all methods called by `retry`.
Doing this clarifies its presence, and makes it easier to use it in the
future if needed.
I don't know why it doesn't want to do TIFF now, but there's also
absolutely no reason for it to be TIFF anyway, so let's just use
an image format that is actually sane.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
The previous commit removed the handling of `dict` arguments, but
didn’t adjust the type, leading to the following type-checking error:
test_driver/driver.py:216: error: Argument 1 to "NixStartScript" has incompatible type "str | dict[Any, Any]"; expected "str" [arg-type]
It also left an unused import that Ruff is unhappy about:
build/lib/test_driver/driver.py:11:22: F401 [*] `colorama.Fore` imported but unused
…
build/lib/test_driver/driver.py:11:28: F401 [*] `colorama.Style` imported but unused
Fixes: 71306e6b36
(cherry picked from commit d490680530)
(cherry picked from commit ff31b814b6)
Scheduled for removal in 24.11, so let's follow through.
Added in #291544.
(cherry picked from commit 71306e6b36)
(cherry picked from commit 8427b6f640)
Using __del__ is somewhat unsound resource cleanup in our clase the
logger already closed its logfile and therefor fails with exception
before the rest of the resources can be cleaned up.
We remove the global rootlog in favor of instantiating the logger as
required in the __init__.py and pass it down as a parameter (of our
AbstractLogger type).
Previously, the XML logging was always present and only created an
output file if a special environment variable was present. We now only
create the XML logger if the environment variable is present, saving us
from logging to XML internally if it is not required.
We add a new logger that allows generating a junit-xml compatible report
listing the subtests used in the nixos integration test. Junit-xml is a
widely used standard for test reports. The report can be used for quick
evaluation of which subtest failed.
We use the newly AbstractLogger class and separate the XML and Terminal
logging that is currently mixed into one class. We restore the old
behavior by introducing a CompositeLogger that takes care of logging
both to terminal and XML.
We do not use the generic "nested" function but introduce a separate
subtest log call. This will later allow us to track subtests and account
logs to specific subtests.
As the TODO says, this is already included by the script.
If adding a device, including this again here would result in either
two devices being added, or, if they were explicitly named, an error
due to reuse of the name.
