Uncovering Hidden Equations: The Magic of Equation Discovery#

Key Themes We Will Explore#

1. Understanding the basics: Variables, domains, and the difference between empirical and analytical approaches.
2. Approaches to discovery: From traditional symbolic regression to AI-driven methods.
3. Practical examples in code: Tools and libraries to guide you through your first experiments.
4. Beyond the basics: Advanced techniques, domain knowledge integration, multi-objective optimization, dimensional analysis, and more.
5. Real-world case studies: Examples drawn from physics, biology, finance, and beyond.

Real-World Applications#

• Physics: Certain physical constants and laws (like Kepler’s laws or Newton’s law of gravitation) have been discovered based on data-driven insights and tested systematically to create universal theories.
• Biology: From enzyme kinetics to population growth models, many biological phenomena are governed by equations that can be discovered and refined by analyzing experimental data.
• Finance: Trading strategies, risk models, and econometric relationships can emerge from analyzing large financial datasets and discovering consistent patterns.
• Engineering: PID control loops, signal processing filters, and robotics algorithms often arise from equation-based models derived from system identification processes.

By revealing the mechanics behind the data, equation discovery helps engineers and scientists design systems, make predictions, and innovate effectively.

Understanding Variables, Parameters, and Domains#

An equation fundamentally relates one or more variables within a given domain. For instance, we might have:

• Independent variables (predictors): x, y, or t (time), representing input features or the domain.
• Dependent variables (targets): The quantities we hope to model, such as f(x, y) or y(t).
• Parameters (constants or coefficients): Numeric values (like a, b, c) that define specific relationships and can be fine-tuned to best fit data.

Extracting an equation from data means determining both the functional form (e.g., linear, polynomial, exponential, etc.) and the values of the parameters.

Distinguishing Empirical from Analytical Models#

• Empirical Models: Derived purely from data observations, may or may not have a theoretical underpinning, but often provide a good fit for the observed dataset.
• Analytical Models: Built upon theoretical insights, such as physical laws, chemical reactions, or established theorems, and refined with data.

Equation discovery can blur the boundary between empirical and analytical models, especially when sophisticated algorithms are used to find relationships that align with known physical or theoretical constraints.

Symbolic Regression#

Symbolic regression is a key term in equation discovery. Instead of starting with a predefined model form (e.g., linear regression: y = ax + b), symbolic regression attempts to construct an expression from a space of possible mathematical primitives (e.g., +, -, ×, ÷, sin, cos) and constants. The aim is to find the functional structure that best fits the data in terms of accuracy and complexity.

Symbolic regression typically:

1. Randomly assembles candidate functions from available operations.
2. Evaluates how well each candidate fits the data (fitness function).
3. Refines these candidates iteratively, often using evolutionary algorithms (like genetic programming).

Data Mining vs. Machine Learning#

Equation discovery can be considered a subset of data mining or machine learning, but it differs in that the output is not just a decision but a structured, interpretable expression. This leads to a synergy with scientific research, as the derived equation can reveal new phenomena rather than merely predicting outcomes.

Classic Algorithmic Methods#

• Polynomial fitting: Expanding a polynomial to a certain degree and using least squares or other techniques to fit parameters.
• Fourier expansions: Using trigonometric series to model periodic signals.
• Rational function fitting: Combining polynomials in numerator and denominator.

These are systematic but can be limited in their capacity to scale with complexity. If you only allow polynomials up to a certain degree, for example, you may miss non-polynomial relationships.

Modern AI-Driven Methods#

• Genetic programming: Populations of symbolic expressions evolve through crossover and mutation.
• Neuro-symbolic approaches: Combine neural networks (e.g., recurrent architectures or transformers) with symbolic manipulation to propose equations.
• Bayesian symbolic regression: Uses probabilistic methods to search the space of expressions.

These methods aim to account for complexity and noise while maintaining some degree of interpretability.

Data Collection#

Suppose the “true�?function is:

y = 2.0 × sin(x) + 0.5x² - 3.0

We generate some synthetic data:

1. Create an array of x-values (for example, 100 points between -5 and 5).
2. Compute y-values using the above expression.
3. Optionally add some noise to simulate measurement errors.

Exploratory Data Analysis#

Quick checks:

• Plot x against y to see the shape.
• Note that it clearly has polynomial and sinusoidal components.

These steps are crucial in a real problem, where you cannot rely on knowing the underlying function. Even if you are not sure about the exact structure, exploratory visualizations often hint at periodicity, linearity, or other traits.

Setting Up a Symbolic Regression Experiment#

Below is a short example in Python using a minimal symbolic regression library. You can adapt this to more sophisticated libraries like PySR, gplearn, or TensorFlow-based solutions.

1
import numpy as np
2
import matplotlib.pyplot as plt
3
from sklearn.model_selection import train_test_split
4

5
# Synthetic data generation
6
np.random.seed(42)
7
x = np.linspace(-5, 5, 100)
8
y_true = 2.0 * np.sin(x) + 0.5 * x**2 - 3.0
9
noise = np.random.normal(0, 0.5, size=x.shape)
10
y_noisy = y_true + noise
11

12
# Split data
13
X_train, X_test, y_train, y_test = train_test_split(
14
    x.reshape(-1, 1), y_noisy, test_size=0.2, random_state=42
15
)
16

17
# Placeholder for symbolic regression approach
18
# You might use gplearn or PySR here:
19
# Example with gplearn:
20
from gplearn.genetic import SymbolicRegressor
21

22
est = SymbolicRegressor(
23
    population_size=500,
24
    generations=20,
25
    stopping_criteria=0.01,
26
    p_crossover=0.7,
27
    p_subtree_mutation=0.1,
28
    p_hoist_mutation=0.05,
29
    p_point_mutation=0.1,
30
    max_samples=0.9,
31
    verbose=1,
32
    parsimony_coefficient=0.01,
33
    random_state=42
34
)
35
est.fit(X_train, y_train)
36

37
print("Best expression found:", est._program)
38

39
# Evaluate on test data
40
y_pred = est.predict(X_test)
41
mse = np.mean((y_test - y_pred)**2)
42
print("Mean Squared Error on test data:", mse)
43

44
# Visualization
45
plt.scatter(X_test, y_test, label='Test Data')
46
plt.scatter(X_test, y_pred, label='Prediction', alpha=0.7)
47
plt.legend()
48
plt.show()

In this snippet:

• We generate synthetic data.
• We split it into training and testing sets.
• We use a simple symbolic regression library (gplearn) to evolve an expression.
• We report the resulting expression and its mean squared error.

While this is trivial because we have artificially constructed the function, it illustrates how you might set up a symbolic regression experiment in practice.

Constraints and Regularization#

• Complexity penalization: It is sometimes beneficial to penalize overly complex solutions to favor simpler, more interpretable expressions.
• Domain constraints: If you know, for example, that a certain function must remain positive, or must be periodic, incorporate such constraints into the search.

Incorporating Domain Knowledge#

If you have physical understanding about your system, you might already suspect certain terms (like sin(x), x², or e^x). You can feed those as candidate building blocks rather than all possible operators. This guides the search toward more plausible results and speeds up convergence.

Handling Noise and Outliers#

Most real-world datasets contain noise. Symbolic regression algorithms often handle noise through robust fitness measures (like mean absolute error rather than mean squared error). Outlier detection and elimination (or robust weighting) can improve the stability of the discovered equation.

Validation and Cross-Checking#

Always reserve a portion of your data for validation. If possible, collect new data not used in the training process to see how your discovered equations generalize. Another cross-check is dimensional consistency: if your variables have physical units, ensure the discovered equation does not violate basic dimensional analysis.

Dimensional Analysis in Symbolic Regression#

Dimensional or unit-aware symbolic regression ensures that any proposed combination of variables respects the dimensional constraints. This significantly reduces the search space and can produce physically meaningful models more quickly. Tools to incorporate dimensional analysis might require you to label each feature with a unit and define valid operators.

Multi-Objective Optimization#

A single measure of an equation’s quality (like mean squared error) may not be adequate. Sometimes you want to balance accuracy, simplicity, and even interpretability or computation cost. This leads to a multi-objective approach, often guided by Pareto optimality. Evolutionary strategies can track separate objectives (e.g., error vs. complexity) and produce a Pareto front of solutions.

Genetic Algorithms and Grammar-Based Approaches#

While a typical symbolic regression approach uses evolutionary operations on syntax trees, more advanced heuristics may impose grammar rules (e.g., certain functional forms are allowed or disallowed). This can accelerate the discovery of relevant equations by limiting the search space to feasible syntactic forms.

Deep Learning for Equation Discovery#

Neuro-symbolic approaches combine the representational power of deep networks (such as sequence-to-sequence or transformer models) with symbolic reasoning. For instance:

1. A neural network might compress or encode numeric data into a latent representation.
2. A symbolic decoder or search mechanism then translates that latent representation into candidate equations.

Deep learning can accelerate discovery for large datasets or complex phenomena, but interpretability and reliability remain active research challenges.

Example: Physics-Informed Equation Discovery#

In many areas of physics—classical mechanics, quantum mechanics, and electromagnetism—scientists rely on known equations. However, there are still areas (e.g., fluid dynamics with turbulence) where the underlying equations are known only in approximation. Advanced symbolic regression can be used alongside partial differential equation solvers, generating corrections to known models or entirely new equation forms. Crucially, consistency with dimensional and physical constraints guides the search away from spurious solutions.

Example: Biological Systems#

Biological systems often involve nonlinearities, feedback loops, and limited data. Equation discovery can help identify small sets of core variables. For instance, in metabolic pathways, you might suspect Michaelis-Menten kinetics but are not fully sure about the parameter form or additional side reactions. Symbolic regression with domain-appropriate operators (logistic functions, saturating functions, etc.) can yield new insights.

Example: Economics and Finance#

Financial markets are notoriously complex, and purely data-driven methods (like ML-based forecasting) often yield black-box models. Combining domain insights, such as known economic indicators and established relationships, with symbolic regression helps discover interpretable risk models or market relationships. However, high noise levels and non-stationarity in financial data demand robust, adaptive methods.

Sympy#

Sympy is an extensive Python library for symbolic math. While it does not perform symbolic regression on its own, it can be used to:

• Validate discovered expressions.
• Simplify them symbolically.
• Differentiate or integrate expressions once discovered.

PySR#

PySR stands for “Python Symbolic Regression�?but uses a high-performance Julia backend. It features GPU acceleration and can explore enormous search spaces. The library integrates well with Python’s data ecosystem, letting you pre-process data with libraries like pandas and scikit-learn.

Eureqa#

Eureqa popularized modern approaches to equation discovery and symbolic regression. It provides a guided user interface, integrates domain knowledge constraints, and uses powerful search heuristics behind the scenes. For large commercial or proprietary tasks, it can be an excellent choice, though it is not free for all use cases.

Open Source vs. Proprietary Tools#

Open-source solutions are typically more transparent, easily automated, and can be customized. Proprietary systems may offer more features out of the box as well as robust commercial support. The choice may hinge on budget, domain constraints, and the need for specialized capabilities (e.g., large-scale distributed computing).

Theoretical Extensions#

You can combine domain-specific theory with data-driven methods. For instance, in PDE (partial differential equation) discovery, you might define a space of candidate operators (d/dx, d/dt, etc.) and systematically search for PDE forms that best describe observations. Such hybrid approaches remain at the forefront of scientific discovery.

Hybrid Approaches#

Hybrid approaches often fuse existing knowledge with data-driven exploration:

1. Physics-informed neural networks (PINNs): Neural architectures that incorporate differential equation constraints.
2. Semi-symbolic expansions: Start with a partial known structure (like linear combination of certain known basis functions) and allow symbolic expansions in other parts.

By merging domain knowledge and machine learning, you maintain interpretability while improving generalization.

Large-Scale Data Considerations#

Big projects can involve tens of thousands or even millions of data points. In such scenarios, efficiency and scalability are crucial:

1. Parallelization: Genetic programming can be parallelized by evaluating populations on multiple nodes or GPUs.
2. Batch sampling: Rather than evaluating each candidate on the entire dataset, sample subsets repeatedly.
3. Memory optimization: Keep large data on disk or use streaming methods to avoid running out of RAM.

Professional practitioners often combine multiple optimization and machine learning techniques to handle these challenges.