Data-Driven Modeling
Data to Dynamics
Chemical processes
- Chemical processes are complex systems that transform raw materials into valuable products.
- Optimal process operations require efficient and effective control
- Data-driven approaches enable better understanding of process behavior and inform control strategies
About me
Ranjeet Utikar
Associate Professor, Chemical Engineering, Curtin University, Australia
SMILE Lab (Sustainable Manufacturing and Intelligent Process Engineering)
Research interests
- Sustainable energy and bioprocesses
- Process modeling and scale-up
- Process intensification and multiphase flows
Experience
- 100+ peer-reviewed publications
- Industrial experience: chemical, oil & gas, resources, biotechnology
- Entrepreneur: Tridiagonal Solutions, Vivira Process Technologies
Linear systems
- Deterministic and predictable
- Output proportional to input (V = IR, F = kx)
- Doubled input \Rightarrow doubled response
- Superposition: effects simply add up
- Design complex systems with high confidence
Nonlinear systems
- Stochastic nature
- Inherent process complexity
- Sources of non-linearity
- Multivariable & Coupled
- One variable change affects multiple outputs
- Difficult to isolate variables (e.g., distillation)
Time and length scales
- Time-varying behavior
- Catalyst sintering (~years)
- Heat exchanger fouling (~months)
- Ambient temp swings (daily/seasonal)
- Vast scales involved
- Molecular (Å, femtoseconds)
- Unit operations (meters, hours)
- Plant lifecycle (km, years)
Chemical systems
- Inherently complex.
- Non-linear relationships between variables
- Tight coupling
- Dynamic behavior that evolves over time
- Phenomena occurring across multiple scales
\Rightarrow Challenging to design, difficult to predict, hard to control, and complex to optimize
Computational models
- Models enable
- simulation of process behavior,
- prediction of responses to disturbances,
- design of effective control strategies, and
- optimization of operations
- Classification
- Mechanistic or first-principles models
- Empirical or data-driven models
- Hybrid approaches
First principles models
- Formalize conservation laws
- Mass, energy, momentum
- Constitutive equations
- Transport rates, kinetics, thermodynamics
- Mathematical structure
- Algebraic, ODEs, or PDEs
- Physical constraints
- Enforce limits (e.g., non-negativity)
- Outcome
- Encode evolution of process variables
First principles models
Advantages
- Parameters have physical meaning
- Process behavior is interpretable
- Valid outside training range (extrapolatable)
Limitations
- Difficult to develop (high expertise)
- Limited understanding of complex mechanisms
- Computationally expensive to solve
- Plant-model mismatch common
- Challenging for online control/optimization
First principles models
- Essential when data is scarce or expensive
- Required for process development and scale-up
- Enables extrapolation beyond historical data
- Analysis of rare/unsafe events (e.g., runaways)
- Provides physical bounds for data-driven models
Data-driven models
- Based on statistical relationships between measured variables
- Exploits large volumes of operational data (DCS, sensors, historians)
- Constructs input-output mappings without mechanistic knowledge
- Effective when:
- Phenomena are too complex for first principles
- Mechanisms are poorly understood
- Rapid development is required
Traditional empirical modeling
- Statistical regression (Linear, PLS)
- Captures linear relationships
- Handles high-dimensional data
- Time-series models (ARMAX, State-space)
- Captures temporal dynamics and feedback loops
- Benefits
- Computational efficiency
- Theoretical guarantees (confidence intervals)
- Interpretability via coefficients
Modern machine learning
- Learns relationships directly from data (no functional form assumed)
- Neural Networks: Approximate arbitrary nonlinear relationships
- Tree-based ensembles (Random Forests, Gradient Boosting):
- Handle regime transitions
- Automatic feature selection
- Support Vector Machines: Classification and nonlinear regression
- Trade-off: High flexibility vs. limited interpretability
Classification dimensions
- Static vs. Dynamic
- Time dependence (differential equations vs. direct mapping)
- Linear vs. Nonlinear
- Superposition vs. interactions/thresholds
- Learning Paradigm
- Supervised (labeled data)
- Unsupervised (latent structure)
- Reinforcement (trial and error)
- Task
- Regression (continuous) vs. Classification (discrete)
Practical advantages
- Minimal domain expertise required
- Rapid model development (minutes/hours vs. weeks/months)
- Captures complex interactions & nonlinearities naturally
- Modest computational cost (real-time capable)
- Reproduces plant behavior with sufficient data
Fundamental limitations
- Lack physical interpretation (black-box)
- Unreliable extrapolation beyond training data
- Susceptible to model degradation (drift/fouling)
- High data requirements (high dimensionality)
- Correlation not equal to Causality
- May violate physical constraints (mass/energy balances)
Hybrid models
- Combine mechanistic and data-driven approaches
- Structure
- Physical equations for well-understood parts
- ML/Stats for difficult-to-model phenomena
- Benefits
- Extrapolation capability (via physical backbone)
- Handles theoretical knowledge gaps (via data)
- Reduced parameter learning burden
Hybrid models architectures
- Serial configurations
- FP model predicts nominal \rightarrow ML corrects residuals
- Parallel structures
- Output = Mechanistic term + Empirical term
- Nested formulations
- ML embedded in mechanistic eqns (e.g. rate laws)
- Selection depends on available knowledge and model accuracy
Hybrid models synergistic advantages
- Lower data requirements (inductive bias)
- Better extrapolation validity
- Targeted experimentation
- Root-cause analysis enablement
- Balanced computational effort
Hybrid models challenges
- High expertise demanded (Process + ML)
- Complex parameter identification
- Risk of structural bias from FP component
- Optimization convergence issues
- Complex maintenance (update FP + recalibrate ML)
Engineering data hierarchy
- Level 4 (Enterprise): Business planning (ERP)
- Level 3 (MES/LIMS): Manufacturing operations, scheduling, QA
- Level 2 (Control): Real-time control (DCS, PLC)
- Level 1 (Sensing): Instrumentation, high-freq data
- Level 0 (Process): Physical equipment
Engineering data hierarchy
- Levels 0-2 (Lower tier):
- High volume/velocity
- Real-time stability focus
- Short retention (hours-weeks)
- Levels 3-4 (Upper tier):
- Lower volume, higher context
- Production metrics
- Evolution:
- Unified Name Space (UNS)
- Breaking data silos
Dimensions of engineering data
- Volume: GBs of sensor logs to singular design docs
- Velocity: Millisecond control signals to static reference data
- Variety: Structured time-series, logs, images, documents
- Veracity: Calibrated sensors vs. soft sensors (uncertainty)
Dimensions of engineering data
- Value: Context-dependent (single failure report > routine logs)
- Lifecycle: Transient errors discarded; regulatory records archived decades
- Ownership: Fragmented (corporate, personal, public databases)
Anomalies in time-series data
- Spike: Transient excursion (interference)
- Missing data: Gaps (outages)
- Drift: Slow shift (fouling)
- Noise: Rapid fluctuations (turbulence)
- Bias: Constant offset (calibration)
Data-driven modeling workflow
Workflow: Phase 1-4
- Problem framing: Objective, prediction horizon, KPIs
- Data collection: Signals, sources, context tags
- QA/QC: Tag mapping, timestamps, bad data removal
- Feature construction: Resampling, dataset splitting
Workflow: Phase 5-8
- Development: Occam’s razor (simplest effective model)
- Validation: Time-based splitting, backtesting
- Deployment: Integration into production
- Monitoring: Track health, trigger improvement loops
Visualizing time-series structure
Key EDA plots
- Run sequence: Shifts in mean/noise, outliers, frozen sensors.
- Pareto: Rank factors by importance (80/20 rule).
- Correlation matrix: Pairwise dependencies, collinearity.
- Lag plots: Detect autocorrelation (Y_i vs Y_{i-1}).
- Autocorrelation (ACF): Quantify process memory.
- Box plots: Distributions, outliers > 1.5 IQR.
- Q-Q plots: Test Gaussian assumption (normality).
CSTR Data Inspection
- Nominal start: Steady operation.
- Step changes:
- t=50: Cooling temp drop (T_c \downarrow \rightarrow T \downarrow \rightarrow C_A \uparrow)
- t=120: Feed conc drop (C_{A,in} \downarrow \rightarrow C_A \downarrow)
- Data quality issues:
- High-frequency noise (needs filtering)
- Data gap at t=80 (needs imputation)
- Outlier spikes (needs removal)
CSTR scatter matrix features
- Bimodality: KDE shows two operating regimes.
- Cluster separation: Three clusters from distinct phases.
- Step signature: Vertical bands at T_c = 290, 300 K.
Parallel coordinates plot
- Maps each row as a line through vertical axes.
- Excellent for visualizing operating regimes (4+ variables).
- Bundle structure identifies regime transitions:
- Low cooling: High T_c, T; low C_A
- High cooling: Opposite pattern
Missing value imputation
- Sources: Sensor maintenance, network latency, compression.
- Strategies:
- Forward-fill: Zero-order hold (creates steps).
- Backward-fill: Non-causal (uses future data).
- Linear interpolation: Preferred for continuous processes.
- Caution: If gap > time constant, discard segment.
Spike removal methods
- Spikes bias model parameters and error statistics.
- Two approaches: Clipping (boundary) vs Replacement (local value).
Spike removal comparison
- Fixed thresholding:
- Clips to global bounds.
- Fails during regime shifts (misses new-regime spikes).
- May truncate legitimate process data.
- Rolling Z-score:
- Accounts for local variation.
- Susceptible to masking: Large outliers inflate std.
- At t=90 min: Outlier masks itself.
- Hampel filter:
- Uses median and MAD (robust to outlier magnitude).
- Precise pruning without spurious artifacts.
- Best for high-noise environments.
Noise filtering methods
- Moving average (MA): Suppresses noise, introduces lag.
- EWMA: Recursive, weights recent data, balances smoothing and responsiveness.
- Median filter: Robust for non-Gaussian noise and spikes.
- Savitzky-Golay: Local polynomial, preserves peaks and derivatives.
- Butterworth: Precise frequency cutoff control.
- Kalman: Optimal when process model available.
- Wavelet: Multi-resolution for non-stationary signals.
Filter comparison
- EWMA:
- Phase lag (shifts signal right).
- Real-time recursive implementation.
- Savitzky-Golay:
- Zero lag, preserves higher-order moments.
- Sensitive to local outliers.
- Butterworth (filtfilt):
- Sharp cutoff, zero phase shift.
- Requires batch processing.
Example: Heat exchanger energy balance
- Counter-current heat exchanger (Hot/Cold streams).
- Measurements: Flow (m), Temp (T_{in}, T_{out}).
- Faults:
- Stuttering T_{h,out} (samples 130-170).
- Sudden bias in m_h (sample 300+).
- Visual check: T_{out} flat despite m_h jump? Suspicious.
Gross error reconstruction
- Identify fault: High residual contribution.
- Set weight \sigma_i \to \infty.
- Reconstruct: Solve optimization with healthy sensors.
- Result:
- Filters stuttering (T_{h,out}).
- Reconciles bias (m_h) to physics.
Steady-state detection example
- Step change: T_c drops (300 K to 290 K).
- Process settles to new steady state.
- Gain (K): Sensitivity of reactor temp. K = \frac{\Delta T}{\Delta T_c} = \frac{T_{new} - T_{old}}{290 - 300}
- Positive gain: Strong coupling to jacket.
Challenge: High noise
- Issue: High noise masks steady state.
- Rolling std dev exceeds threshold.
- Result: False positives (unsteady).
- Fix:
- Increase window size (averaging).
- Filter data (EWMA/Butterworth).
Challenge: Process drift
- Issue: Slow degradation (fouling/poisoning).
- Low variance (appears steady).
- Non-zero slope (drifting).
- Fix: Monitor both variance and slope.
- True steady state: Low variance + Zero slope.
Example: First-order system
- Physics: Capacity/Inertia (Heating, Filling).
- Response: Smooth rise to steady state (no overshoot).
- AR(1) Model: y_t = \phi y_{t-1} + (1-\phi) K u_{t-1}
- \phi = e^{-\Delta t / \tau}: Autocorrelation coefficient.
- High \phi \to Slow response (high inertia).
- Low \phi \to Fast response.
Example: Second-order system
- Physics: Momentum + Capacity (Fluid oscillation).
- Response: Overshoot and oscillation (\zeta < 1).
- AR(2) Model: y_t = \phi_1 y_{t-1} + \phi_2 y_{t-2} + \beta u_{t-1}
- Roots: Complex conjugates for underdamped systems.
- \phi_2: Controls decay rate (damping).
- \phi_1: Controls frequency.
AR(1) vs AR(2) comparison
- AR(1) cannot capture overshoot.
- AR(2) matches oscillatory dynamics.
- Metric: Integral Absolute Error (IAE). \text{IAE} = \sum |y_{true} - y_{model}| \Delta t
- AR(1) IAE: 4.79
- AR(2) IAE: 1.0 (Better fit).
Comparing dead time approaches
- AR(1): Ignores delay \to Poor initial fit.
- ARX (Correct k): Physically accurate.
- High-order ARX: learns delay via zero parameters.
- Risk: Overfitting, ill-conditioned.
- IAE values: 100, 1.0, 0.0
ARX vs. ARMAX results
- Drift impact: Unmeasured non-stationary disturbance.
- Process Model: Tracks initially, then fails (IAE = 249.08).
- ARX: Biased by drift (IAE = 194.87).
- ARMAX:
- Captures drift (C(q) term).
- Fits dynamics (IAE = 31.08).
Multi-variable interactions
- Industrial systems are interconnected networks.
- Coupling: Shared utilities, feeds, constraints.
- Patterns:
- Independent: Isolated loops.
- One-way: Upstream \to Downstream.
- Recycle: Circular interactions.
Visualizing process coupling
- Heatmap: Visualizes coefficient matrix \mathbf{A}.
- Interpretation:
- Off-diagonal color \to Interaction strength.
- Zero \to Decoupled.
- Reveals network structure (one-way vs two-way).
Example: Identifying topology from data
- Scenario: 2-variable one-way interaction (y_2 \to y_1).
- Goal: Recover structure from data alone (no physics).
- Method: Fit Vector Autoregressive (VAR) model.
- Inspect off-diagonal coefficients.
- Result:
- y_2 \to y_1 coefficient is significant.
- y_1 \to y_2 coefficient is zero.
- Topology successfully identified.
Granger Causality Matrix
- Heatmap of p-values.
- Interpretation:
- Low p-value (< 0.05): Causal link.
- High p-value: Coincidence.
- Example:
- y_2 \to y_1 (p < 0.05): Confirmed coupling.
- y_1 \to y_2 (p \approx 1.0): No feedback.
- Distinguishes physical coupling from correlation.
Sampling frequency & Aliasing
- Nyquist Theorem: Sample at least 2x max frequency.
- Rule of thumb: 5-10 \times faster than dominant time constant \tau.
- Aliasing: Slow sampling creates fake low-freq trends.
- Example: 1 Hz sine sampled at 0.8 Hz appears as 0.2 Hz wave.
- Danger: Controller fights phantom dynamics.
ADC Resolution & Quantization
- Cause: Low bit depth or aggressive compression.
- Effect: Smooth signals turn into steps.
- Derivative dy/dt becomes series of infinite spikes.
- Mitigation:
- Low-pass / Savitzky-Golay filtering.
- Kalman filtering (treat as measurement noise).
Measurement lag (Sensor dynamics)
- Reality: Sensors have their own dynamics (e.g., thermowells).
- Acts as a low-pass filter.
- Consequence:
- Process appears slower/more stable than reality.
- Underestimates variability.
- Risk: Controller designed too slow for real disturbances.
