Frequently Asked Questions¶

This page addresses the most common questions about the Quantum Pipeline, organized by topic. Click on any question to expand the answer.

General¶

What is Quantum Pipeline?

Quantum Pipeline is an extensible framework for exploring Variational Quantum Eigensolver (VQE) algorithms backed by a modern data stack. It combines quantum chemistry simulations using Qiskit with a streaming data platform built on Apache Kafka, Spark, Airflow, and Iceberg.

The system supports both CPU and GPU-accelerated simulation, and all results are automatically serialized, streamed, and stored for analysis. It was developed as part of a thesis project to demonstrate effects of GPU acceleration on quantum computing workloads.

What is the Variational Quantum Eigensolver (VQE)?

VQE is a hybrid quantum-classical algorithm that iteratively optimizes a parameterized quantum circuit to find the ground state energy of a molecular system. See the VQE Algorithm page for details on how Quantum Pipeline implements it.

What molecules are supported?

The system supports any molecule that can be described by the PySCF molecular specification format. The thesis benchmark set (data/molecules.thesis.json) includes:

H\(_2\) (Hydrogen) - 4 qubits with sto3g
HeH\(^+\) (Helium hydride cation) - 4 qubits with sto3g
LiH (Lithium hydride) - 12 qubits with sto3g
BeH\(_2\) (Beryllium hydride) - 14 qubits with sto3g
H\(_2\)O (Water) - 14 qubits with sto3g
NH\(_3\) (Ammonia) - 16 qubits with sto3g
CO\(_2\) (Carbon dioxide)
N\(_2\) (Nitrogen)

The qubit counts above are the full Jordan-Wigner counts. The pipeline applies no active-space or frozen-core reduction, so each count equals twice the number of spatial orbitals for the basis set.

The general-purpose file (data/molecules.json) contains 9 molecules including He2, BeH, BH, and CH\(_4\). You can define custom molecules by specifying atomic symbols and 3D coordinates. See Basic Usage for examples of custom molecule definitions.

Can Quantum Pipeline connect to real quantum hardware?

The system was designed with the capability to integrate with real quantum computers via IBM Quantum backends. However, due to access costs, testing uses the Qiskit Aer simulator exclusively.

The architecture allows for swapping in real quantum backends with minimal code changes - the backend configuration is abstracted behind a consistent interface.

What license is Quantum Pipeline released under?

Quantum Pipeline is open-source software under MIT license.

GitHub · Codeberg

Who developed Quantum Pipeline?

Quantum Pipeline was developed by Piotr Krzysztof Lis as part of a thesis project. The project demonstrates the integration of quantum computing simulations with data engineering using streaming and batch processing technologies.

GitHub · Codeberg

Installation¶

Which Python version is required?

Quantum Pipeline requires Python 3.12.

During development uv was extensively used to ensure proper versioning. Alternatives include conda, pyenv and similar tools to ensure correct Python version. See the Installation guide for detailed setup instructions.

Can I use Quantum Pipeline on Windows?

The recommended environment is Linux. Windows users can run simulations through WSL2 with Docker Desktop. MacOS supports CPU-only simulations (no GPU acceleration due to the NVIDIA CUDA requirement). Native Windows execution is not officially supported and only Linux is tested.

How do I install GPU support?

Install NVIDIA drivers and the NVIDIA Container Toolkit on the host, then use the GPU image (straightchlorine/quantum-pipeline:gpu). From 2.3.0 the published image runs on Pascal (GTX 10xx) and newer GPUs (RTX 20xx/30xx/40xx, A100, H100, Blackwell), so you do not need to rebuild it. The host needs an NVIDIA driver supporting CUDA 12 (525 or newer).

See GPU Acceleration for complete setup instructions.

Can I install without Docker?

Yes. The core simulation module can be installed as a Python package using pip install quantum-pipeline. This allows you to run VQE simulations directly on your machine.

What are the required system dependencies?

Python 3.12 for the package. Docker and Docker Compose are only needed for the container images and the full platform. For GPU support: NVIDIA drivers and the NVIDIA Container Toolkit. All scientific computing dependencies are managed through pip.

Usage¶

How do I choose the right optimizer?

The most commonly used are COBYLA and L-BFGS-B.

The pipeline supports 10 optimizers:

L-BFGS-B - limited-memory BFGS, good general default
COBYLA - gradient-free, handles noisy landscapes, so it works with --shots (gradient optimizers refuse sampled shots)
SLSQP - sequential least squares, gradient-based
Nelder-Mead - simplex method, gradient-free
Powell - directional set method, gradient-free
BFGS - full quasi-Newton method
CG - conjugate gradient
TNC - truncated Newton with bounds
Newton-CG - Newton conjugate gradient, needs the analytic path: an exact run (no --shots) that is local and noiseless, without --gpu, with at most 16 qubits
trust-constr - trust-region method with constraints

See the Optimizers page for a comparison.

What basis set should I use?

Start with sto3g for rapid prototyping (fewest qubits, fastest execution). Use cc-pvdz for high-accuracy results (GPU speedup up to 4x). See Basis Sets for a detailed comparison.

Which ansatz should I use?

Three ansatze are available via the --ansatz flag. Start with the default unless you have a reason not to - it is the one the thesis experiments and most of the verification runs used.

Ansatz	What it is	Where it starts
EfficientSU2 (default)	Hardware-efficient RY/RZ rotations with reverse-linear CNOT entanglement. The ansatz of the thesis experiments	All-zero state, parameters uniform [0, 2pi) - or a Hartree-Fock pre-optimization under `--init-strategy hf`; at 2 repetitions it can stall at the Hartree-Fock energy, so prefer `--ansatz-reps 3` with `hf`
RealAmplitudes	RY rotations only, so half the parameters and only real amplitudes are reachable. Simpler circuit	All-zero state, parameters uniform [0, 2pi) - or the same pre-optimization under `--init-strategy hf`
ExcitationPreserving	XX+YY rotations with all-to-all entanglement, conserving particle number, at a parameter count that grows quadratically in qubits	The Hartree-Fock determinant, always prepended, with parameters jittered off zero

The entanglement pattern is fixed per ansatz and is not a command-line option. Control the circuit depth with --ansatz-reps (default: 2). More repetitions increase expressiveness at the cost of more parameters to optimize. See Ansatz Construction for the full comparison.

What initialization strategies are available?

Two strategies are available via the --init-strategy flag:

random (default) - uniform random parameters in [0, 2pi). Simple but prone to local minima, especially for larger molecules or higher basis sets.
hf (Hartree-Fock) - runs a classical pre-optimization to find parameters that prepare the Hartree-Fock state through the ansatz. The pre-optimization fidelity decreases for larger molecules.

The pre-optimization runs for EfficientSU2 and RealAmplitudes. It falls back to random, with a warning, only when no Hartree-Fock data is available. ExcitationPreserving always begins from the Hartree-Fock determinant, so for it hf only narrows the jitter on the parameters. What the strategies did in the runs so far is on the --init-strategy page; see also Hartree-Fock Initialization.

With hf at the default 2 repetitions a gradient optimizer can stop at the Hartree-Fock energy; see Hartree-Fock initialization.

For data collection or benchmarking, run both strategies across multiple seeds.

Why do I get different results with the same --seed?

With a fixed --seed the initial parameters, the simulator and the transpiler are seeded, so two runs with the same settings give the same result. Results can still differ across Qiskit and Aer versions. A run without --shots is exact: expectation values are computed from the statevector, with no sampling, so there is no shot noise. Gradient optimizers such as L-BFGS-B need that, so a sampled run (--shots N) must use a gradient-free optimizer such as COBYLA, or the pipeline exits with code 2.

Without --seed, shot sampling (only with --shots, for example --optimizer COBYLA --shots 1024) makes each run follow a slightly different trajectory. To collect meaningful statistics, run multiple seeds and compare the distributions rather than individual values.

How do I run multiple simulations in parallel?

The batch generation system is the recommended way to run many simulations. It defines 4 tiers of increasing complexity and distributes work across 3 hardware lanes that run concurrently: two GPU lanes for the larger molecules (data/molecules.gpu1.json and data/molecules.gpu2.json) and one CPU lane for the smallest ones (data/molecules.cpu.json).

Batch generation is triggered via the vqe_batch_generation Airflow DAG (manual trigger with {"tier": N} conf). The underlying script is scripts/generate_ml_batch.py, which handles 3-lane parallel Docker execution with idempotent resume.

Each tier varies the basis set, optimizer set, init strategy, seeds, and ansatz. Tier 1 (sto3g, all 8 optimizers, 25 seeds) is the broadest sweep; Tier 4 (cc-pvdz, 3 optimizers, 10 seeds) targets the hardest configurations.

How do I interpret the simulation output?

The key output values are:

Minimum energy: The ground state energy found, measured in Hartree (Ha). Lower values indicate a better approximation of the true ground state.
Total iterations: Number of optimizer steps taken to complete optimization.
Timing breakdown: Hamiltonian construction time, qubit mapping time, VQE optimization time, and total execution time.
Convergence status: Whether the optimizer reached the convergence threshold or hit the iteration limit.

Data Platform¶

Do I need Kafka to run simulations?

No. The simulation module can run in standalone mode without Kafka. Results will be printed to the console or saved to local files.

Can I use the pipeline without Spark?

Yes. Kafka and Garage still store your simulation results in the object storage bucket, as JSON files by default (Avro with the Kafka Connect alternative). Spark is only needed for the incremental processing step that transforms raw results into structured feature tables in Apache Iceberg format.

You can read the raw files with any JSON or Avro tool, including Python's json module, the fastavro library or Apache Spark in standalone mode.

How do I access stored simulation data?

There are several ways to access the data:

S3-compatible client: Any S3 SDK (boto3, rclone, etc.) can read from Garage using the S3_ENDPOINT, S3_ACCESS_KEY, and S3_SECRET_KEY environment variables.
Spark SQL: Query Iceberg feature tables with SQL through Spark.

Feature tables are stored under features/warehouse/quantum_features/ in Garage, and synced to Cloudflare R2 (qp-data/features/) via the R2 sync DAG.

You can also use configured aws tool to browse through the S3 storage.

What serialization format is used?

Apache Avro for wire serialization between the producer and Kafka, with schemas managed by the Confluent Schema Registry. Data at rest is JSON (via Redpanda Connect) or Avro (via Kafka Connect). See Serialization for details.

How does schema evolution work?

When a field is added to a result dataclass, its Avro schema gets a nullable default, so older and newer messages stay decodable by the same consumer. See Serialization for details.

What happens if Kafka goes down during a simulation?

The producer retries failed deliveries and waits for full acknowledgement; the settings are in defaults.py. If Kafka is temporarily unavailable, it will retry message delivery.

Simulation results are not lost as long as Kafka retains the messages (Kafka's default retention is 7 days unless overridden). Redpanda Connect persists each message to object storage, which is the durable boundary.

Monitoring¶

How do I enable monitoring?

Set the MONITORING_ENABLED environment variable to true, or pass --enable-performance-monitoring on the command line.

The environment-variable path reads its configuration from:

PUSHGATEWAY_URL - PushGateway address (default: http://localhost:9091)
MONITORING_INTERVAL - collection interval in seconds (default: 10)
MONITORING_EXPORT_FORMAT - export format (prometheus, json, or both)
CONTAINER_TYPE - label for the container (e.g. GPU_GTX1060_6GB, CPU)

The CLI path has matching flags: --performance-pushgateway for the URL, --performance-interval for the collection interval (default 30 seconds), and --performance-export-format. A value given on the command line takes priority over the environment variable, which takes priority over the built-in default.

What metrics are collected?

Metrics use the qp_ prefix and fall into three groups:

qp_vqe_* - per-experiment VQE metrics (timing, energy, accuracy, iterations, convergence)
qp_sys_* - system resource metrics (CPU, memory, uptime)
qp_batch_* - batch generation progress (total, done, failed, in-progress, pending per tier and lane)

See Performance Metrics for the full list and label descriptions.

Deployment¶

Can I run Quantum Pipeline on a cloud provider?

Yes. The Docker Compose deployment runs on any cloud VM with Docker installed.

How do I scale the processing layer?

The Spark processing layer scales horizontally by adding more worker nodes. Update the Docker Compose file to add additional spark-worker services.

Kafka scales by adding partitions to topics and additional broker nodes. Garage can be deployed in a multi-node layout for storage scaling.

How do I back up my data?

Garage data is stored in Docker volumes. Use docker volume commands or host-mounted volumes for backups. The R2 sync DAG automatically replicates feature tables to Cloudflare R2 as an off-site backup. Iceberg's metadata layer provides point-in-time snapshots with time-travel query support.

How do I update to a newer version?

Pull the latest Docker images and restart the services:

docker compose pull
docker compose down
docker compose up -d

For pip installations, upgrade with pip install --upgrade quantum-pipeline.

Performance¶

What speedup can I expect with GPU acceleration?

Speedup depends on the basis set and molecule complexity:

sto3g basis set: 1.7-1.8x average speedup
cc-pvdz basis set: 3.5-4.1x speedup due to larger matrix operations

Small molecules (4 qubits) may see no benefit or slight slowdown due to GPU overhead. The greatest speedup is observed for molecules of medium complexity where the computation-to-overhead ratio is most favorable. See Benchmarking for detailed results.

How do I speed up optimization?

Try --init-strategy hf (with EfficientSU2 only at --ansatz-reps 3 or more) to start from a better initial point (with ExcitationPreserving the flag only narrows the parameter jitter), increase --ansatz-reps for expressiveness, run multiple simulations to avoid local minima, use GPU acceleration, start with sto3g, and enable --convergence with a looser --threshold (e.g. 1e-4) for faster results.

Why are my energy values different from reference literature?

Common causes: random parameter initialization converging on local minima, limited ansatz expressiveness, or early termination from convergence tolerance. Run multiple simulations and select the best result, or try --init-strategy hf --ansatz-reps 3 for a better starting point with EfficientSU2 or RealAmplitudes. See Benchmarking for detailed analysis.

For issues not covered here, see the Troubleshooting guide or open an issue on GitHub.