Changelog¶
Release notes for Quantum Pipeline, starting from version 1.4.0.
For earlier versions, see the GitHub releases.
2.4.0¶
Breaking changes¶
Before 2.4.0, a run without --shots sampled 1024 shots unless you passed
--exact. From 2.4.0, a run without --shots is exact: expectation values are
computed from the statevector, with no shot sampling. To sample, pass --shots N.
The default optimizer, L-BFGS-B, needs gradients. With sampled energies the
gradient would come from comparing energies at nearby points, and the sampling
error is larger than those differences.
The gradient would be noise, so the optimizer now refuses sampled shots.
To replicate pre 2.4.0 behavoiur, add --shots 1024 and switch to a
gradient-free optimizer such as --optimizer COBYLA.
Two cases differ. --ibm without --shots still samples 1024 shots, so it
needs a gradient-free optimizer for the same reason. --noise now needs either
--shots N or --simulation-method density_matrix.
These gradient optimizers refuse sampled shots with an error and exit
code 2: L-BFGS-B, BFGS, CG, TNC, SLSQP, Newton-CG and trust-constr. COBYLA and
the other gradient-free optimizers accept sampled shots, but they now sample
only when you pass --shots.
Without it they run exact like everything else. Separately, a file passed
with --load overrides the CLI flags, including whether the run is exact or
sampled, so if the loaded file says to sample (or to be exact), that wins
over what was typed.
qiskit-algorithms>=0.4.0 is a new runtime dependency.
Analytic-gradient fast path¶
There is a new fast path for exact runs with a gradient optimizer. The "analytic" path computes energies and exact gradients (derivatives) directly on the statevector, with no sampling. Before, these runs went through Aer (the Qiskit simulator) and estimated gradients by finite differences.
Energies from the new path agree with Aer exact to about 1e-13 Ha (chemical accuracy is about 1.6e-3 Ha, so the difference is negligible).
On the benchmark, one optimizer step (one energy and gradient update) is roughly
4x (H2) to 250x (NH3) faster than on Aer, from step times measured in
scripts/benchmark_fast_path.py.
The run log announces the path, and the backend saved in the run's output data
is statevector_analytic.
Problems above 16 qubits stay on the Aer exact path. The analytic gradient builds a large matrix form of the Hamiltonian (the molecule's energy operator) on every call, and a larger H2 setup with 20 qubits (cc-pVDZ basis set) ran out of memory.
Optimizers¶
Newton-CG and trust-constr are added, for ten supported optimizers in total.
Newton-CG only works on the analytic path (the exact-derivative fast path
described above). That means the run must be exact, local (not --ibm),
without noise, without GPU, and at most 16 qubits.
If those conditions are not met, Newton-CG says so and does not run. trust-constr has no such restriction: when the analytic path is not available, it falls back to finite differences.
Hartree-Fock start and the stationary-point trap¶
With --init-strategy hf at the default 2 repetitions a gradient optimizer can stop at the Hartree-Fock energy; see Hartree-Fock initialization.
Benchmark script and tests¶
New script: scripts/benchmark_fast_path.py compares the analytic path against
Aer. It takes --output and --molecules, and it also runs outside a git
checkout. Its results are committed under benchmarks/fast_path/.
Because the default is now exact, the ML batch builder
(scripts/generate_ml_batch.py) and the smoke script
(scripts/smoke_test_ml_pipeline.sh) now pass --exact explicitly.
Fixes¶
L-BFGS-B no longer passes the deprecated disp option.
Solver configuration errors now print the usage and the reason and exit with
code 2, instead of showing a traceback. This covers a gradient optimizer
combined with --shots, and Newton-CG used off the analytic path.
2.3.0¶
Breaking changes¶
pyspark is no longer a core dependency. It moved to the dev extra
(Spark job tests) and the ml extra (the data-processing notebook).
The Docker images no longer contain conda or pip, so images built FROM them
that install packages at build time need their own Python tooling.
Docker images¶
The GPU image compiles Aer on the CUDA toolchain and ships it on plain Ubuntu
24.04 in a venv, without conda, cuDNN or the CUDA base.
883 MiB instead of 16 GiB, and cleared some security findings. It also runs on
GeForce cards, which the CUDA base's forward-compat libcuda broke.
A CUDA_VERSION build arg selects the toolkit. The CPU image moved to Debian
Trixie and drops git.
Both images run under any --user, ship without pip, and install their
dependencies in a layer of their own.
Published GPU image is now scanned weekly.
The GPU image is built for compute capabilities 6.1 7.0 8.0 9.0+PTX, so the
published image runs on every NVIDIA GPU from Pascal on.
The example compose files in examples/ run against the current images, with
apache/kafka and the schema registry --kafka needs, in plaintext, TLS and
SASL over TLS variants. docker/scripts/ssl-gen.sh now writes the credential
files apache/kafka expects.
2.2.0¶
Breaking changes¶
Existing Iceberg tables must be rebuilt from fresh data. The Spark jobs now generate different experiment and child ids (see Data processing), so old and new rows cannot be mixed.
The monitoring metric qp_vqe_accuracy_score is renamed to
qp_vqe_hf_deviation_score. Custom Grafana dashboards and alert rules that
reference the old name must be updated; the bundled ones already are.
to_avro_bytes now raises KeyError when the schema has no registry id,
instead of writing bytes without the Confluent header and logging a warning.
VQERunner takes kafka_config and backend_config objects in place of the
loose kafka_* and backend_* keyword arguments.
Configuration and CLI¶
--convergence always overrides --max-iterations, with a warning when both
are given. Values from a --load file override the CLI values.
KAFKA_SERVERS is applied only with --kafka, SCHEMA_REGISTRY_URL is read
from the environment, and the security config reads the ssl_* keys.
A negative --molecule-index is rejected instead of silently selecting a
molecule from the end of the file.
--optimizer now lists only the eight optimizers that can run (L-BFGS-B,
COBYLA, SLSQP, Nelder-Mead, Powell, BFGS, CG, TNC); the others failed after the
molecule was loaded. --simulation-method no longer offers stabilizer,
extended_stabilizer, unitary and superop, which cannot run a
parameterized circuit. --max-iterations must be positive. --ssl uses the
individual --ssl-* files when the --ssl-dir directory does not exist
instead of failing with a traceback. --dump names the file local or api
the right way round.
Simulation¶
The stored init_strategy is the one that actually ran; hf falls back to
random when Hartree-Fock data is missing. The hf initialization now also
works for RealAmplitudes, and ExcitationPreserving has an hf arm (small
jitter, 0.01) next to its random arm (0.5).
--exact now honours the configured backend: noise model, simulation method
and GPU. --exact --noise requires --simulation-method density_matrix.
One failing molecule no longer aborts the batch. The remaining molecules run, and the run fails at the end with every failure reported. Reports are generated once after the loop rather than once per molecule. Metrics derived from the Hartree-Fock reference are now empty instead of 0 when no reference exists.
Data processing¶
The Spark jobs build experiment_id as a sha256 over an explicit field list,
so Avro and JSON inputs produce the same id. Child ids are positional
(_iter_N, _init_N, _opt_N, _term_N) rather than hash buckets, and
duplicates within a batch are removed. Appends merge schemas and skip rows
whose keys already exist.
The exact_estimator flag and performance_start timestamp are now carried
through to the tables, and mean_param_change is fixed. The ML feature job
checks each target table separately, so a crash between writes is repaired on
the next run.
Monitoring¶
--performance-export-format json no longer pushes VQE metrics to the
PushGateway.
qp_vqe_hf_deviation_score is a 0-100 log-scaled deviation from the
Hartree-Fock reference. It is absent when the run has no reference. The
dashboard and the alert rule use the new name.
ML package (experimental)¶
quantum_pipeline/ml/ was rewritten for ongoing thesis work. It is
experimental, not part of the core simulation flow, and its API may change
without notice.
The package is split into convergence and energy subpackages with shared
schema, model registry (XGBoost, RandomForest, LogisticRegression, Ridge),
fitting and trajectory modules. The old convergence_predictor,
energy_estimator and circuit feature modules are removed, and the package
no longer re-exports the predictor, estimator or tracker.
Maintenance¶
Type annotations were tightened, mypy passes and CI now runs it. The schema registry client was rewritten. Docs, code comments and the README were cleaned up.
2.1.0¶
This version includes fixes for issue #52 reported by @avnerbensoussan.
ExcitationPreserving ansatz¶
The ExcitationPreserving ansatz now works instead of getting stuck near
the Hartree-Fock energy. It prepends the Hartree-Fock state as the
circuit's initial state.
Full entanglement is used and optimization starts from a small jitter off zero parameters. The jitter matters because zero parameters sit exactly on the HF reference, a stationary point where the optimizer stalls; noise lets it descend into the correlation energy.
Full entanglement is required because adjacent-only gates can only slide electrons between neighbouring orbitals (a Slater determinant in, a Slater determinant out). It leaves the correlated states unreachable. With these changes it reached chemical accuracy on H2 in quick checks and improved with more reps.
Error is now raised when Hartree-Fock data is not provided, rather than silently producing a meaningless result.
Reproducible seeding¶
A fixed --seed now reaches the Aer simulator (seed_simulator) and the
transpiler (seed_transpiler). Two runs with the same seed now produce
identical results.
Exact statevector mode¶
A new --exact flag computes exact expectation values through the Aer
EstimatorV2 (default_precision=0.0) with no shot noise. It overrides
--shots.
Minimum energy reporting¶
The reported minimum is now the best evaluated iteration: the energy and the parameters come from the same step, rather than pairing the lowest observed energy with whatever parameters happened to be stored alongside it. This applies to every run, not only exact mode.
Noise model fix¶
--noise now reaches the backend. The flag was previously dropped before
the backend configuration was built, so noisy runs silently ran noiseless
and were recorded as if they were not.
Spark incremental processing¶
Incremental processing now derives a deterministic experiment_id from the
experiment's own fields instead of a fresh value on each run, so
re-processing no longer duplicates the dataset every day.
Batch generation¶
The batch run key now includes the basis set.
Tiers differ by basis set while sharing molecule, optimizer, init strategy, seed, and ansatz, so without the basis in the key a finished Tier 1 (sto3g) run marked the higher-basis tiers as done and they were silently skipped.
Kafka delivery¶
A failed Kafka send is no longer dropped silently. Undelivered results are
spooled to disk (gen/undelivered/) as JSON for manual re-submission and the
run exits non-zero, so a downed broker cannot discard computed output while
the job still reports success.
2.0.0¶
Complete rewrite of the simulation module, DAGs, tests, and infrastructure. Adds batch generation, multiple ansatze and optimizers, a full observability stack, and a storage migration from MinIO to Garage. Removes legacy code paths and simplifies metrics.
Removals¶
quantum-pipeline.pyroot script removed. The package is now invoked viaquantum_pipeline.cli:mainentry point.- Chemical accuracy metric removed from simulation output and tests.
- Scientific literature references replaced with simpler PySCF-derived metrics (HF energy, nuclear repulsion) throughout the codebase.
Batch generation¶
A batch system for generating ML training data across many configurations.
Four tiers sweep combinations of basis set, optimizer, init strategy, seed,
and ansatz. Three hardware lanes (two GPU, one CPU) run concurrently via
ThreadPoolExecutor, and state is persisted to gen/ml_batch_state.json
for idempotent resume. Progress metrics (qp_batch_*) are pushed to the
PushGateway per tier and lane.
Multiple ansatze¶
The --ansatz flag selects between EfficientSU2 (default),
RealAmplitudes, and ExcitationPreserving.
Extended optimizer support¶
Eight optimizers are used in batch generation: L-BFGS-B, COBYLA, SLSQP, Nelder-Mead, Powell, BFGS, CG, and TNC. Additional scipy optimizers (Newton-CG, COBYQA, trust-constr, dogleg, trust-ncg, trust-exact, trust-krylov) are available but less tested.
Observability¶
PerformanceMonitor collects system and experiment metrics and exports
them to the Prometheus PushGateway. Monitoring environment variables
renamed from QUANTUM_PERFORMANCE_* to shorter names:
MONITORING_ENABLED, MONITORING_INTERVAL, PUSHGATEWAY_URL,
MONITORING_EXPORT_FORMAT. Metrics use the qp_ prefix (qp_vqe_* for
experiment metrics, qp_sys_* for system metrics, qp_batch_* for batch
progress).
Grafana dashboards added for VQE experiment tracking, batch progress,
and system resource usage (monitoring/grafana/dashboards/).
Full exporter stack in the Docker Compose environment: statsd-exporter, postgres-exporter, redis-exporter, nvidia-gpu-exporter, and Garage S3 metrics. Prometheus scrape configuration covers all exporters and the PushGateway.
Storage migration¶
MinIO replaced by Garage (dxflrs/garage) as the S3-compatible object
store. Environment variables: S3_ENDPOINT (default http://garage:3901),
S3_REGION, S3_ACCESS_KEY, S3_SECRET_KEY. Feature tables are synced
to Cloudflare R2 via the R2 sync DAG.
Docker¶
- GPU image updated to CUDA 12.6.3, builds qiskit-aer from source with
cuQuantum.
CUDA_ARCHbuild argument controls the target GPU architecture (default8.6). DOCKER_GIDbuild argument for Airflow container to access the Docker socket (default970).
Simulation module¶
Runner, Solver, and Mapper base classes now use proper ABCs.
Hardcoded constants (HF fidelity threshold, per-optimizer default
iterations) were moved to configs/constants.py.
The L-BFGS-B optimizer config had a workaround that set ftol and
gtol to 1e-15 when max_iterations was active, to prevent early
convergence. This was redundant - compute_energy() already enforces
the iteration limit by raising MaxFunctionEvalsReachedError. The hack
also prevented natural convergence, which is arguably worse for data
collection. Removed.
DAGs¶
Configuration that was duplicated across 4 DAG files and 2 Spark scripts
(S3 paths, default args, Spark session setup) now lives in
docker/airflow/common/. Retries use exponential backoff. Spark tasks
have SLA targets. A dedicated batch generation DAG orchestrates ML data
collection runs, including container image builds. Airflow logic
simplified by removing overly complex scheduling and retry paths.
Tests¶
Solver tests were spread across 8 files with overlapping coverage (4
files testing VQESolver, 3 testing optimizer config). Consolidated to
4 files. Deleted 6 tests that were testing scipy.optimize.minimize
behavior rather than project itself.
Added pytest-xdist for parallel test execution and @pytest.mark.slow
for a fast inner loop (pytest -m "not slow" runs in ~38s).
Verification¶
Ran the refactored simulation across a range of configurations. All combinations completed correctly on CPU (Aer statevector, seed 42, EfficientSU2 2 reps unless noted). The table below is a partial sample; more combinations are still being verified.
| Optimizer | Basis | Init | Iters | H\(_2\) (Ha) | HeH\(^+\) (Ha) | Notes |
|---|---|---|---|---|---|---|
| COBYLA | sto-3g | random | 30 | -1.555 | -4.004 | Completed naturally |
| L-BFGS-B | sto-3g | random | 30 | -0.895 | -3.283 | Hard limit |
| SLSQP | sto-3g | random | 50 | -1.111 | -3.562 | RealAmplitudes 3r |
| Nelder-Mead | sto-3g | random | 50 | -0.553 | -3.200 | |
| Powell | sto-3g | random | 30 | -0.005 | -0.011 | ExcitationPreserving |
| COBYLA | sto-3g | random | 235/221 | -1.785 | -4.338 | Convergence 1e-4 |
| COBYLA | sto-3g | hf | 50 | -1.836 | -4.267 | |
| BFGS | sto-3g | hf | 40 | -1.838 | -4.224 | |
| L-BFGS-B | sto-3g | hf | 30 | -1.838 | -4.218 | |
| L-BFGS-B | 6-31g | random | 1029/1372 | +2.101 | -0.956 | H\(_2\) stuck in local min |
| L-BFGS-B | 6-31g | hf | 50 | -1.857 | -4.294 | HF avoids local min |
The last two rows are a good example of why initialization matters. L-BFGS-B with random init on 6-31g spent 1029 iterations to arrive at a positive energy for H\(_2\) - a barren plateau. The same setup with HF init reached -1.857 Ha in 50 iterations.
1.4.0¶
Hartree-Fock Initialization¶
The random parameter initialization used in prior versions (uniform \([0, 2\pi)\)) was identified as a major contributor to poor convergence (see Experimental Observations).
Version 1.4.0 adds --init-strategy hf, which attempts to start VQE
from the classical Hartree-Fock solution instead.
Prepending a HartreeFock circuit to EfficientSU2 and setting all parameters to zero - turned out not to work. The fixed CX entangling gates in EfficientSU2 are not parameterized and always act, regardless of rotation angles.
At zero parameters the rotation gates become identity, but the CX gates still alter the HF state.
The current approach runs a short classical pre-optimization that finds
EfficientSU2 parameters which directly prepare the HF state through the
ansatz, by maximizing state fidelity via Statevector simulation. This
partially resolves the initialization problem - for H\(_2\) (reps=1,
16 parameters) the pre-optimization reliably achieves fidelity of 1.0,
while for H\(_2\)O (reps=2, 72 parameters) it reaches approximately 0.993.
Early results comparing v1.3.x (random init) against v1.4.0 (HF init) under the same conditions (sto-3g, COBYLA, reps=2, statevector, GPU, max 300 iterations):
H\(_2\) (4 qubits, 32 parameters, HF reference: -1.117 Ha):
| Version | Init | Best electronic | Total energy | Iterations | Time |
|---|---|---|---|---|---|
| v1.3.x | Random | -1.73 Ha | n/a | 300 | 23 s |
| v1.4.0 | HF | -1.837 Ha | -1.122 Ha | 187 | 15 s |
The random run oscillated around -1.68 to -1.73 Ha without settling, while HF init converged in 187 iterations (below the 300 limit) and found energy within 5 mHa of the HF reference. Pre-optimization fidelity: 1.000.
H\(_2\)O (12 qubits, 72 parameters, HF reference: -74.963 Ha):
| Version | Init | Best electronic | Total energy | Iterations | Time |
|---|---|---|---|---|---|
| v1.3.x | Random | ~-75.95 Ha | n/a | 300 | 965 s |
| v1.4.0 | HF | -84.209 Ha | -75.016 Ha | 300 | 660 s |
The random run showed large oscillations at the end (costs jumping between -73 and -76 Ha between iterations), indicating the optimizer was struggling with the landscape. The HF run produced a stable result 53 mHa below the HF reference, capturing some correlation energy. Pre-optimization fidelity: 0.993.
Note
v1.3.x did not account for nuclear repulsion energy, so total energy is not available for those runs. The electronic energies are not directly comparable between versions - the v1.4.0 total energy column is what should be compared against literature values.
These results are encouraging but there is a lot more work to be done - particularly around scaling the pre-optimization to larger molecules and higher ansatz reps, where fidelity drops and the approach becomes slower. Alternative strategies such as ADAPT-VQE or layer-by-layer parameter fitting have not been explored yet.
Nuclear Repulsion Energy¶
VQE optimizes the electronic Hamiltonian, which does not include nuclear repulsion energy. Previous versions reported only the raw VQE minimum, making it difficult to compare results against literature values directly.
The pipeline now captures nuclear repulsion energy from PySCF and adds it
to the VQE result. VQEResult exposes a total_energy property
(electronic + nuclear repulsion), and both components are logged:
Best energy: -1.83805736 Ha
Total energy (electronic + nuclear repulsion): -1.12295302 Ha
(nuclear repulsion: 0.71510434 Ha)
Code Organization¶
quantum_pipeline/circuits/package -HFDataandbuild_hf_initial_statemoved fromsolvers/into a dedicated module, since initial state construction is circuit-level work rather than solver logic.- Mapper wrapper -
Mapper.get_qiskit_mapper()provides access to the underlying qiskit-nature mapper, so that circuit construction does not depend on direct qiskit imports.
Bug Fixes¶
- Timer context manager - accessing
Timer.elapsedinside awithblock before__exit__ran causedValueErrorwhenMaxFunctionEvalsReachedErrorwas raised. The exception is now handled after the timer block exits. - Convergence vs max-iterations - the
--convergenceflag no longer conflicts with the defaultmax_iterations=100. When convergence mode is enabled,max_iterationsis cleared unless explicitly set. - Simulation method validation - GPU-only methods now produce a clear
error when
--gpuis not enabled. - Ansatz reps parsing -
--ansatz-repsis correctly parsed as an integer.
Known Limitations¶
The items below are either known to be incomplete or are being actively worked on:
- HF pre-optimization scaling - for molecules with many qubits and high ansatz reps, the fidelity pre-optimization may not converge to a satisfactory starting state. This is a known weakness of the current approach.
- EfficientSU2 ansatz - the hardware-efficient ansatz still does not preserve particle number or spin symmetry. The anomalies documented in the thesis (e.g. HeH\(^+\) sub-FCI energy) remain possible.