Skip to content

Regression References and the Test Registry

OpenQP's example suite (openqp --run_tests all) is also its regression harness: each example's committed .json is the reference its numbers are checked against. What gets checked is defined by a single registry, so the suite stays honest as features are added.

The registry (single source of truth)

pyoqp/oqp/utils/regression.py declares every regression quantity once as a RegKey row — its JSON key, the runtypes/methods/properties it applies to, whether it is required, where it lives (primary JSON or the .hess.json sidecar), and flags for sign/phase ambiguity. That one registry drives three things so they cannot drift apart:

  • comparison — check_ref compares exactly the registered keys for the run;
  • lean references — save_data(lean=True) keeps only registry keys (plus identity/metadata), dropping the large internal OQP:: arrays;
  • the gate — openqp --validate_examples fails if any example reference is missing a value its runtype requires.

Adding a new regression value

This is the one place you touch:

  1. Add a RegKey(...) row in regression.py (key, runtypes, required, source, and any needs_excited / needs_prop / phase_invariant gates).
  2. If it is a newly computed quantity, emit it under the same key from Molecule.get_results() (and, if it originates in Fortran, store it to a tagarray as in OQP::nmr_shielding).
  3. Regenerate the affected references (see below). lean_keep keeps the new key automatically.

From then on the value is compared, kept in lean references, and enforced: an example missing it fails the gate.

Generating and validating references

Use the supported commands — do not hand-edit references:

# (Re)write lean references for one or more inputs (registry keys only)
openqp --generate_reference path/to/example.inp [more.inp ...]

# Verify every reference carries the values its runtype requires
openqp --validate_examples            # defaults to $OPENQP_ROOT/share/examples

--generate_reference runs each input and writes only the registry keys next to the .inp, so committed references stay small and consistent. It validates what it wrote before returning.

What is enforced in CI

  • openqp --validate_examples — every reference is complete (blocking).
  • openqp --run_tests all — the numbers still match (blocking; the step uses set -o pipefail so a failing example fails the build).
  • An agent review flags a new computed value added without a RegKey, an example added without a reference, and code/manual drift.

Coverage notes

  • Hessian references compare freqs, infrared_intensities (IR) and raman_activities (Raman) from the .hess.json sidecar, not just the matrix.
  • NAC compares the NACME derivative-coupling matrix by magnitude (sign/phase ambiguous between builds).
  • IRC is regression-tested on energy only; the final IRC geometry is a non-stationary path point that is not reproducible across BLAS/compilers.
  • Properties (dipole, mulliken_charges, lowdin_charges, resp_charges) are opt-in via [properties] scf_prop and tested only when requested.