#
# This file and its contents are supplied under the terms of the
# Common Development and Distribution License ("CDDL"), version 1.0.
# You may only use this file in accordance with the terms of version
# 1.0 of the CDDL.
#
# A full copy of the text of the CDDL should have accompanied this
# source.  A copy of the CDDL is also available via the Internet at
# http://www.illumos.org/license/CDDL.
#

#
# Copyright 2019 Joyent. Inc.
# Copyright 2024 MNX Cloud, Inc.
#

Strictly speaking, this is not a set of tests. Rather, it is a wrapper that
automates the configuration of a SmartOS system to prepare it for test
execution, optionally running those tests.

The smartos-test script should be extracted from the test archive to ensure
the correct version is being executed. If you download it, download it
directly, as the test archive .tgz file is needed by the smartos-test script
itself.  Furthermore, the directory you extract it in should NOT be a
system-level directory (e.g. /tmp/.) but in its own subdirectory
(e.g. /tmp/test/.)

For example:

[root@kura /var/tmp]# mkdir test ; cd test
[root@kura /var/tmp/test]# tar zvxf tests-test_archive-master-20191001T134222Z.tgz ./opt/smartos-test
Decompressing 'tests-test_archive-master-20191001T134222Z.tgz' with '/usr/bin/gzcat'...
x ./opt/smartos-test, 0 bytes, 0 tape blocks
x ./opt/smartos-test/README, 958 bytes, 2 tape blocks
x ./opt/smartos-test/bin, 0 bytes, 0 tape blocks
x ./opt/smartos-test/bin/smartos-test, 10062 bytes, 20 tape blocks
[root@kura /var/tmp/test]# ./opt/smartos-test/bin/smartos-test -h
Usage: smartos-test [-h] [-c] [-e] [-r] [-w] <path to tests.tgz>

At least one of -c, -e, -r is required.

  -h       print usage
  -c       configure the system for testing
  -e       execute known tests
  -f       skip the check to ensure platform version == test version
  -r       snapshot or rollback to zones/opt@system-test-smartos-test
           before doing any system configuration or test execution
  -w       when mounting the lofs /usr, make it writable


Specifically, the script will:

* verify that the user has declared that no production data exists on the
  test system (by checking /lib/sdc/.sdc-test-no-production-data)
* verify we're running in the global zone
* verify the test archive version matches the version of the platform we're
  running
* optionally snapshot or rollback /opt to "zones/opt@system-test-smartos-test"
* setup loopback mounts for any files from the smartos "tests-[stamp].tgz"
  file that need to be installed to a normally read-only location, and
  extract the portions of test archive that must appear in /usr
* extract the remaining test archive contents to /opt and /kernel
* install pkgsrc-tools (or update existing GZ pkgsrc if installed)
* create a `ztest` user for ZFS tests (if DISKS is set... see below)
* enable per-process coredumps
* put /etc/shadow on to /etc itself to allow tests like the ZFS one to
  add/remove users.
* install required test packages
* execute tests that should all pass

Over time, we hope to add to the set of tests that are executed.

After configuring the system for testing, you may choose to run individual
test suites, for example:

    # /opt/util-tests/bin/utiltest
or
    # /opt/os-tests/bin/ostest


Notes on NVMe testing:

NVMe is tested by unit tests and by non-destructive component tests.  The
latter must specify NVME_TEST_DEVICE from nvme* names.  E.g. "nvme0".  If
NVME_TEST_DEVICE is not set, the non-destructive component tests will not run.

Notes on ZFS testing:

At present, this wrapper will execute the ZFS tests IF the DISKS environment
variable is set to a list of disks that are available for testing. The
wrapper will run those by changing to the 'ztest' user if the proper
environment variables are set.

In this example, DISKS is the list of three unused disks that will be used
to run the tests. KEEP is a list of zpools that the tests should *not*
destroy:

    # su ztest
    $ export DISKS='c2t1d0 c2t2d0 c2t3d0'
    $ export KEEP='zones'
    $ /opt/zfs-tests/bin/zfstest

Beware that data loss will occur for any data residing on DISKS and failure to
specify a KEEP pool may also lead to data loss.


A note for SmartOS developers:

The test archive .tgz file is built by the 'tests-tar' Makefile target in
smartos-live.git.

The manifest of files included in the archive is generated at build-time by
parsing the IPS package manifests from $SRC/pkg/manifests that are listed in
$SRC/Makefile.testarchive in this repository.

It is important when adding or removing tests that the IPS manifests are
updated. If new test packages are created, they should be added to
$SRC/Makefile.testarchive.


How We Use This:

Every release we pull the release ISO and the matching test .tgz file down.
For example:

kebe(~/ws/ij-cr)[141]% mls -lt /Joyent_Dev/public/SmartOS/20230323T000605Z | egrep "iso|tests-"
-rwxr-xr-x 1 Joyent_Dev     606420992 Mar 22 22:43 smartos-20230323T000605Z.iso
-rwxr-xr-x 1 Joyent_Dev      12946513 Mar 22 22:43 tests-release-20230323-20230323T000605Z.tgz
kebe(~/ws/ij-cr)[0]% 

We then use a test machine (currently a VMware VM, which does cause some
inconsistencies in the BHYVE/VMM results) which has:

	16GB of RAM (though 8 would probably suffice)

	A 100GB disk for `zones` (a smaller disk can cause ENOSPC problems
	sometimes)

	Six 30GB disks for setting up in DISKS for ZFS testing.  (fewer
	can cause ZFS test failures)

	(optional) a Virtual NVMe drive of 20GB (or greater size).

	An NIC that can reach the public Internet.

We then perform a fresh installation.  The installation needs to have a bit
of manual intervention.

	The pool should not include the three 30GB disks. We do this by
	selecting "manual" at pool creation time, and create the pool with
		"zpool create -f -B zones c1t0d0"

We also select "zones" as the bootable pool and we say yes to pkgsrc in the
global zone.

Once installed, we detach the ISO and boot from the `zones` pool.  After
that, we pull the tests tarball and put it into /tmp/test/.  Inside /tmp/test
we run:

	gtar -xzf tests-release-20230323-20230323T000605Z.tgz
		(Extracts tests)

	touch /lib/sdc/.sdc-test-no-production-data
		(let smartos-test know we're okay with this)

	export DISKS="c1t2d0 c1t3d0 c1t4d0 ..."
		(your disk names may vary)

	(optional)
	export NVME_TEST_DEVICE=nvme0

	./opt/smartos-build/bin/smartos-build -c ./tests-release-20230323-20230323T000605Z.tgz
		(Configure for testing)

	/opt/smartos-build/bin/smartos-build -e ./tests-release-20230323-20230323T000605Z.tgz
		(Run the tests!
		 You can also use ./opt/ as /opt is copied from here)

Now the tests are running, and with ZFS tests also executing, this will take
a long time. All of the test results will end up in /var/tmp/test_results
under multiple timestamped directories.

The results of these directories contain "log" file that summarize passes and
fails.  Detailed test output is also in the test_results directory.  Shell
scripts can process and compare runs from prior releases to the one that is
under test now.
