heterodyne.analysis.core

Core Analysis Engine for Heterodyne Scattering Analysis

High-performance heterodyne scattering analysis with configuration management.

This module implements the complete analysis pipeline for heterodyne XPCS data with separate reference and sample scattering components, based on He et al. PNAS 2024.

Physical Theory - Heterodyne Model

The heterodyne scattering model describes the two-time correlation function c₂(t₁,t₂,φ) for X-ray photon correlation spectroscopy (XPCS) measurements of two-component systems (reference + sample) under nonequilibrium conditions.

The heterodyne correlation function (He et al. PNAS 2024, Equation S-95):

c₂(q⃗,t₁,t₂,φ) = 1 + (β/f²)[
    [xᵣ(t₁)xᵣ(t₂)]² exp(-q²∫ₜ₁^ₜ₂ Jᵣ(t)dt) +
    [xₛ(t₁)xₛ(t₂)]² exp(-q²∫ₜ₁^ₜ₂ Jₛ(t)dt) +
    2xᵣ(t₁)xᵣ(t₂)xₛ(t₁)xₛ(t₂) exp(-½q²∫ₜ₁^ₜ₂[Jₛ(t)+Jᵣ(t)]dt) cos[q cos(φ)∫ₜ₁^ₜ₂ v(t)dt]
]
where f² = [xₛ(t₁)² + xᵣ(t₁)²][xₛ(t₂)² + xᵣ(t₂)²]

Two-time correlation structure: - xₛ(t₁), xₛ(t₂): Sample fraction at time t₁ and t₂ (each in [0,1]) - xᵣ(t₁) = 1 - xₛ(t₁): Reference fraction at time t₁ - xᵣ(t₂) = 1 - xₛ(t₂): Reference fraction at time t₂ - All integrals: From t₁ to t₂ - Normalization f²: Uses fractions at BOTH times - Angle φ: Relative angle = φ₀ - φ_scattering (flow minus scattering direction) - Baseline: 1 (uncorrelated limit) - Contrast: β (absorbed in experimental measurements)

Transport Coefficients (separate for reference and sample):

Jᵣ(t) = J0_ref * t^(alpha_ref) + J_offset_ref Jₛ(t) = J0_sample * t^(alpha_sample) + J_offset_sample

Velocity Coefficient (shared between components):

v(t) = v0 * t^β + v_offset

Sample Fraction Function:

fₛ(t) = f0 * exp(f1 * (t - f2)) + f3

Note: Parameters labeled “D” are transport coefficients J following He et al. For equilibrium: J = 6D where D is traditional diffusion coefficient.

Parameter Model (Heterodyne, 14 parameters): Reference Transport (3): - D0_ref: Reference transport coefficient J₀_ref [nm²/s] - alpha_ref: Reference power-law exponent [-] - D_offset_ref: Reference baseline transport J_offset_ref [nm²/s]

Sample Transport (3): - D0_sample: Sample transport coefficient J₀_sample [nm²/s] - alpha_sample: Sample power-law exponent [-] - D_offset_sample: Sample baseline transport J_offset_sample [nm²/s]

Velocity (3): - v0: Velocity amplitude [nm/s] - beta: Velocity power-law exponent [-] - v_offset: Baseline velocity [nm/s]

Fraction (4): - f0: Fraction amplitude [0-1] - f1: Exponential decay rate [s⁻¹] - f2: Time offset [s] - f3: Baseline fraction [0-1]

Flow Angle (1): - phi0: Angular offset parameter [degrees]

Experimental Parameters: - q: Scattering wavevector magnitude [Å⁻¹] - φ: Scattering angle [degrees] - dt: Time step between frames [s/frame]

Features

  • JSON-based configuration management

  • Experimental data loading with intelligent caching

  • Parallel processing for multi-angle calculations

  • Performance optimization with Numba JIT compilation

  • Comprehensive parameter validation and bounds checking

  • Memory-efficient matrix operations and caching

Performance Optimizations (v0.6.1+)

This version includes significant performance improvements:

Core Optimizations: - Chi-squared calculation: 38% performance improvement (1.33ms → 0.82ms) - Memory access patterns: Vectorized operations using reshape() instead of list comprehensions - Configuration caching: Cached validation and chi-squared configs to avoid repeated dict lookups - Least squares optimization: Replaced lstsq with solve() for 2x2 matrix systems - Memory pooling: Pre-allocated result arrays to avoid repeated allocations

Algorithm Improvements: - Static case vectorization: Enhanced broadcasting for identical correlation functions - Precomputed integrals: Cached shear integrals to eliminate redundant computation - Vectorized angle filtering: Optimized range checking with np.flatnonzero() - Early parameter validation: Short-circuit returns for invalid parameters

Performance Metrics: - Chi-squared to correlation ratio: Improved from 6.0x to 1.7x - Memory efficiency: Reduced allocation overhead through pooling - JIT compatibility: Maintained Numba acceleration while improving pure Python paths

Usage

>>> from heterodyne.analysis.core import HeterodyneAnalysisCore
>>> analyzer = HeterodyneAnalysisCore('config.json')
>>>
>>> # 14-parameter heterodyne model
>>> params = [100.0, -0.5, 10.0,   # Reference transport
...           100.0, -0.5, 10.0,   # Sample transport (initially = reference)
...           0.1, 0.0, 0.01,      # Velocity
...           0.5, 0.0, 50.0, 0.3, # Fraction
...           0.0]                 # Flow angle
>>>
>>> c2 = analyzer.calculate_heterodyne_correlation(params, phi_angle=0.0)
>>> chi2 = analyzer.calculate_chi_squared_optimized(params, phi_angles, c2_experimental)

Migration from 11-Parameter Model

Existing configurations can be automatically migrated:

>>> from heterodyne.core.migration import HeterodyneMigration
>>> migrated = HeterodyneMigration.migrate_config_file('old_config.json', 'new_config.json')

The migration initializes sample parameters equal to reference parameters for backward compatibility. During optimization, they can diverge.

References

He, H., Chen, W., et al. (2024). “Heterodyne X-ray Photon Correlation Spectroscopy.” PNAS, Equation S-95 (Heterodyne Correlation Function). https://doi.org/10.1073/pnas.2315354121

Authors: Wei Chen, Hongrui He Institution: Argonne National Laboratory

class heterodyne.analysis.core.HeterodyneAnalysisCore(config_file='heterodyne_config.json', config_override=None, config_path=None, config=None)[source]

Bases: object

Core analysis engine for heterodyne scattering data.

Implements Equation S-95 (general time-dependent two-component form) from He et al. PNAS 2024 (https://doi.org/10.1073/pnas.2401162121), using time-dependent transport coefficients J(t) for nonequilibrium dynamics.

The model captures heterodyne scattering between reference and sample components, where transport coefficients J(t) evolve with time to describe aging, yielding, and shear banding in soft matter systems.

Implementation Notes: - Uses transport coefficients J(t) directly (not traditional diffusion coefficients D) - For equilibrium Wiener processes: J = 6D - Parameters labeled “D” (D₀, α, D_offset) are transport coefficient parameters (J₀, α, J_offset) - Implements S-95 with J_r = J_s (single transport coefficient for both components)

Key capabilities: - 11-parameter heterodyne model with time-dependent fraction mixing - Configuration-driven parameter management - Experimental data loading with intelligent caching - Optimized correlation function calculations (Numba JIT-compiled) - Time-dependent transport, velocity, and fraction dynamics

Parameters:
__init__(config_file='heterodyne_config.json', config_override=None, config_path=None, config=None)[source]

Initialize the core analysis system.

Parameters:

  • config_file (str) – Path to JSON configuration file

  • config_override (dict, optional) – Runtime configuration overrides

  • config_path (str, optional) – Alias for config_file (for backward compatibility)

  • config (dict, optional) – Alias for config_override (for backward compatibility)

get_effective_parameter_count()[source]

Get the effective number of parameters for heterodyne analysis.

Returns:

Always returns 14 for the heterodyne model: - Reference transport coefficients (3): D0_ref, alpha_ref, D_offset_ref - Sample transport coefficients (3): D0_sample, alpha_sample, D_offset_sample - Velocity coefficients (3): v0, beta, v_offset - Fraction coefficients (4): f0, f1, f2, f3 - Flow angle (1): phi0

Return type:

int

get_effective_parameters(parameters)[source]

Extract effective parameters for laminar flow analysis.

Parameters:

parameters (np.ndarray) – Full 14-parameter array for heterodyne model: [D0_ref, alpha_ref, D_offset_ref, D0_sample, alpha_sample, D_offset_sample, v0, beta, v_offset, f0, f1, f2, f3, phi0]

Returns:

All 14 parameters as provided for heterodyne model

Return type:

np.ndarray

load_experimental_data[source]

Load experimental correlation data with caching.

Returns:

(c2_experimental, time_length, phi_angles, num_angles)

Return type:

tuple

calculate_diffusion_coefficient_optimized(params)[source]

Calculate time-dependent transport coefficient J(t).

Note: Method name retained for API compatibility. Calculates transport coefficient J(t) following He et al. PNAS 2024 Equation S-95.

Ensures J(t) > 0 always by applying a minimum threshold.

Special handling for negative alpha: - For alpha < 0, J(t) diverges as t→0 - Physical limit: J(0) = J_offset (labeled D_offset in code) - For t > threshold: J(t) = J₀ * t^alpha + J_offset

Parameters:

params (ndarray)

Return type:

ndarray

calculate_shear_rate_optimized(params)[source]

Calculate time-dependent shear rate.

Ensures γ̇(t) > 0 always by applying a minimum threshold.

Special handling for negative beta: - For beta < 0, γ̇(t) diverges as t→0 - Physical limit: γ̇(0) = offset - For t > threshold: γ̇(t) = γ̇₀ * t^beta + offset

Parameters:

params (ndarray)

Return type:

ndarray

create_time_integral_matrix_cached[source]

Create cached time integral matrix with optimized algorithm selection.

calculate_c2_single_angle_optimized(parameters, phi_angle, precomputed_D_t=None)[source]

Calculate heterodyne correlation function for a single angle.

Uses the 2-component heterodyne scattering model.

Parameters:

  • parameters (np.ndarray) – 14-parameter array for heterodyne model

  • phi_angle (float) – Scattering angle in degrees

  • precomputed_D_t (ndarray | None)

Returns:

Correlation matrix c2(t1, t2)

Return type:

np.ndarray

validate_heterodyne_parameters(parameters)[source]

Validate physical constraints on heterodyne parameters.

Parameters:

parameters (np.ndarray) – 14-parameter array for heterodyne model with structure: reference transport (3), sample transport (3), velocity (3), fraction (4), flow angle (1). Note: Transport coefficients labeled “D0”, “D_offset” in code.

Raises:

ValueError – If parameters violate physical constraints

Return type:

None

calculate_heterodyne_correlation(parameters, phi_angle, precomputed_D_t=None, precomputed_v_t=None)[source]

Calculate 2-component heterodyne two-time correlation function.

Implements Equation S-95 from He et al. PNAS 2024, using separate transport coefficients for reference and sample components.

Theoretical Equation S-95::

c₂(q⃗,t₁,t₂,φ) = 1 + β/f² [
    [xᵣ(t₁)xᵣ(t₂)]² exp(-q²∫ₜ₁^ₜ₂ Jᵣ(t)dt) +
    [xₛ(t₁)xₛ(t₂)]² exp(-q²∫ₜ₁^ₜ₂ Jₛ(t)dt) +
    2xᵣ(t₁)xᵣ(t₂)xₛ(t₁)xₛ(t₂) exp(-½q²∫ₜ₁^ₜ₂[Jₛ(t)+Jᵣ(t)]dt) cos[q cos(φ)∫ₜ₁^ₜ₂ v(t)dt]
]
where f² = [xₛ(t₁)² + xᵣ(t₁)²][xₛ(t₂)² + xᵣ(t₂)²]

Two-Time Correlation Structure:

The correlation function is computed as a matrix where each element (i,j) represents the correlation between times t₁[i] and t₂[j]:

  • Fractions: xₛ(t₁), xₛ(t₂) evaluated at each time (meshgrid)

  • Reference: xᵣ(t) = 1 - xₛ(t) at each time

  • Normalization: f² computed from fractions at BOTH times

  • Integrals: ∫ₜ₁^ₜ₂ computed over time interval for each (t₁,t₂) pair

  • Angle: φ in cos(φ) = φ₀ - φ_scattering (relative angle between flow and scattering)

Implementation Using Field Correlations:

The implementation uses:

g₁_r(t₁,t₂) = exp(-q²/2 ∫ₜ₁^ₜ₂ Jᵣ(t)dt) # Reference field correlation g₁_s(t₁,t₂) = exp(-q²/2 ∫ₜ₁^ₜ₂ Jₛ(t)dt) # Sample field correlation

Note: g₁² = exp(-q²∫ Jdt) and g₁_r·g₁_s = exp(-½q²∫[Jₛ+Jᵣ]dt)

Transport Coefficient Model: - Separate transport: Jᵣ(t) and Jₛ(t) for reference and sample - Power-law form: J(t) = J₀·t^α + J_offset - Equilibrium limit: J = 6D (Wiener process) - Legacy naming: Parameters labeled “D” are transport coefficients J

Parameters:

  • parameters (np.ndarray) – 14-parameter array with structure: reference transport (3), sample transport (3), velocity (3), fraction (4), flow angle (1). Note: Transport coefficients labeled “D0”, “D_offset” in code

  • phi_angle (float) – Scattering angle in degrees

  • precomputed_D_t (np.ndarray, optional) – Pre-computed transport coefficient array (labeled “D” for legacy compatibility)

  • precomputed_v_t (np.ndarray, optional) – Pre-computed velocity array

Returns:

Heterodyne correlation matrix c2(t1, t2)

Return type:

np.ndarray

calculate_velocity_coefficient(velocity_params)[source]

Calculate time-dependent velocity coefficient v(t).

Model: v(t) = v₀ × t^β + v_offset

Special handling for negative beta: - For beta < 0, v(t) diverges as t→0 - Physical limit: v(0) = v_offset - For t > threshold: v(t) = v₀ * t^beta + v_offset

Parameters:

velocity_params (np.ndarray) – [v0, beta, v_offset]

Returns:

Velocity array v(t)

Return type:

np.ndarray

calculate_fraction_coefficient(fraction_params)[source]

Calculate time-dependent fraction coefficient f(t).

Model: f(t) = f₀ × exp(f₁ × (t - f₂)) + f₃

Physical constraint: 0 ≤ f(t) ≤ 1 (enforced by clipping)

Parameters:

fraction_params (np.ndarray) – [f0, f1, f2, f3]

Returns:

Fraction array f(t), clipped to [0, 1]

Return type:

np.ndarray

calculate_c2_heterodyne_parallel(parameters, phi_angles)[source]

Calculate 2-component heterodyne correlation function for all angles with parallel processing.

Uses the heterodyne scattering model with 14 parameters for reference and sample components with independent transport coefficients.

Parameters:

  • parameters (np.ndarray) – 14-parameter array for heterodyne model with structure: reference transport (3), sample transport (3), velocity (3), fraction (4), flow angle (1). Note: Transport coefficients labeled “D0”, “D_offset” in code

  • phi_angles (np.ndarray) – Array of scattering angles in degrees

Returns:

3D array of correlation matrices [angles, time, time]

Return type:

np.ndarray

calculate_chi_squared_optimized(parameters, phi_angles, c2_experimental, method_name='', return_components=False, filter_angles_for_optimization=False)[source]

Calculate chi-squared goodness of fit with per-angle analysis and uncertainty estimation.

This method computes the reduced chi-squared statistic for model validation, with optional detailed per-angle analysis and uncertainty quantification. The uncertainty in reduced chi-squared provides insight into the consistency of fit quality across different angles.

Performance Optimizations (v0.6.1+): - Configuration caching: Cached validation and chi-squared configs to avoid repeated lookups - Memory optimization: Pre-allocated arrays with reshape() instead of list comprehensions - Least squares optimization: Replaced lstsq with solve() for 2x2 matrix systems - Vectorized operations: Improved angle filtering and array operations - Early validation: Short-circuit returns for invalid parameters - Result: 38% performance improvement (1.33ms → 0.82ms)

Parameters:

  • parameters (np.ndarray) – Model parameters [D0, alpha, D_offset, gamma_dot_t0, beta, gamma_dot_t_offset, phi0]

  • phi_angles (np.ndarray) – Scattering angles in degrees

  • c2_experimental (np.ndarray) – Experimental correlation data with shape (n_angles, delay_frames, lag_frames)

  • method_name (str, optional) – Name of optimization method for logging purposes

  • return_components (bool, optional) – If True, return detailed results dictionary with per-angle analysis

  • filter_angles_for_optimization (bool, optional) – If True, only include angles in optimization ranges [-10°, 10°] and [170°, 190°] for chi-squared calculation

Returns:

If return_components=False, returns reduced chi-squared value (float). If return_components=True, returns dictionary with keys: chi_squared, reduced_chi_squared, reduced_chi_squared_uncertainty, reduced_chi_squared_std, n_optimization_angles, degrees_of_freedom, angle_chi_squared, angle_chi_squared_reduced, angle_data_points, phi_angles, scaling_solutions, valid

Return type:

float or dict

Notes

The uncertainty calculation follows standard error of the mean:

reduced_chi2_uncertainty = std(angle_chi2_reduced) / sqrt(n_angles)

Interpretation of uncertainty:

  • Small uncertainty (< 0.1 * reduced_chi2): Consistent fit across angles

  • Large uncertainty (> 0.5 * reduced_chi2): High angle variability, potential systematic issues or model inadequacy

The method uses averaged (not summed) chi-squared for better angle weighting:

reduced_chi2 = mean(chi2_reduced_per_angle) for optimization angles only

Quality assessment guidelines: - Excellent: reduced_chi2 ≤ 2.0 - Acceptable: 2.0 < reduced_chi2 ≤ 5.0 - Warning: 5.0 < reduced_chi2 ≤ 10.0 - Poor/Critical: reduced_chi2 > 10.0

analyze_per_angle_chi_squared(parameters, phi_angles, c2_experimental, method_name='Final', save_to_file=True, output_dir=None)[source]

Comprehensive per-angle reduced chi-squared analysis with quality assessment.

This method performs detailed analysis of chi-squared values across different scattering angles, providing quality metrics, uncertainty estimation, and angle categorization to identify systematic fitting issues.

Parameters:

  • parameters (np.ndarray) – Optimized model parameters [D0, alpha, D_offset, gamma_dot_t0, beta, gamma_dot_t_offset, phi0]

  • phi_angles (np.ndarray) – Scattering angles in degrees

  • c2_experimental (np.ndarray) – Experimental correlation data with shape (n_angles, delay_frames, lag_frames)

  • method_name (str, optional) – Name of the analysis method for file naming and logging

  • save_to_file (bool, optional) – Whether to save detailed results to JSON file

  • output_dir (str, optional) – Output directory for saved results (defaults to current directory)

Returns:

Comprehensive analysis results dictionary with keys: method, overall_reduced_chi_squared, reduced_chi_squared_uncertainty, quality_assessment, angle_categorization, per_angle_analysis, statistical_summary, recommendations

Return type:

dict[str, Any]

Notes

Quality Assessment Criteria:

  • Overall reduced chi-squared uncertainty indicates fit consistency

  • Small uncertainty (< 10% of chi2): Consistent fit across angles

  • Large uncertainty (> 50% of chi2): High variability, investigate systematically

Angle Classification:

  • Good angles: reduced_chi2 ≤ acceptable_threshold (default 5.0)

  • Unacceptable angles: reduced_chi2 > acceptable_threshold

  • Statistical outliers: reduced_chi2 > mean + 2.5*std

The method uses configuration-driven thresholds from validation_rules.fit_quality for consistent quality assessment across the package.

Note: Per-angle chi-squared results are included in the main analysis results. No separate file is saved.

See also

calculate_chi_squared_optimized

Underlying chi-squared calculation

save_results_with_config(results, output_dir=None)[source]

Save optimization results along with configuration to JSON file.

This method ensures all results including uncertainty fields are properly saved with the configuration for reproducibility.

Parameters:

  • results (dict[str, Any]) – Results dictionary from optimization methods

  • output_dir (str, optional) – Output directory for saving results file (default: current directory)

Return type:

None

fit(c2_data, angles=None, t1_array=None, t2_array=None)[source]

Fit model parameters to experimental data.

Parameters:

  • c2_data (np.ndarray) – Experimental correlation data

  • angles (np.ndarray, optional) – Phi angles

  • t1_array (np.ndarray, optional) – Time array for first time point

  • t2_array (np.ndarray, optional) – Time array for second time point

Returns:

Optimization result

Return type:

Any