From Data to Differential From Data to Differential EquationsEquations

Jim RamsayJim RamsayMcGill UniversityMcGill University

With inspirations from With inspirations from Paul Speckman and Chong GuPaul Speckman and Chong Gu

The themesThe themes

Differential equations are powerful Differential equations are powerful tools for modeling data.tools for modeling data.

We have new methods for estimating We have new methods for estimating differential equations directly from differential equations directly from data.data.

Some examples are offered, drawn Some examples are offered, drawn from chemical engineering and from chemical engineering and medicine.medicine.

Differential Equations as Differential Equations as ModelsModels

DIFE’S make DIFE’S make explicit the relation explicit the relation between one or between one or more derivatives more derivatives and the function and the function itself.itself.

An example is the An example is the harmonic motion harmonic motion equation:equation:

2 2( ) ( )D x t x t

Why Differential Equations?Why Differential Equations?

The behavior of a derivative is often of The behavior of a derivative is often of more interest than the function itself, more interest than the function itself, especially over short and medium time especially over short and medium time periods.periods.

How rapidly a system responds rather than How rapidly a system responds rather than its level of response is often what matters.its level of response is often what matters.

Velocity and acceleration can reflect Velocity and acceleration can reflect energy exchange within a system. Recall energy exchange within a system. Recall equations like equations like f = maf = ma and and e = mce = mc2.2.

Natural scientists often provide theory Natural scientists often provide theory to biologists and engineers in the form to biologists and engineers in the form of DIFE’s.of DIFE’s.

Many fields such as pharmacokinetics Many fields such as pharmacokinetics and industrial process control routinely and industrial process control routinely use DIFE’s as models.use DIFE’s as models.

Especially for input/output systems, Especially for input/output systems, and for systems with two or more and for systems with two or more functional variables mutually functional variables mutually influencing each other.influencing each other.

DIFE’s arise when feedback systems DIFE’s arise when feedback systems must be developed to control the must be developed to control the behavior of systems.behavior of systems.

The solution to an The solution to an mmth order linear DIFE th order linear DIFE is an is an mm-dimensional function space, and -dimensional function space, and thus the equation can model thus the equation can model variation variation over replications as well as over replications as well as averageaverage behavior.behavior.

A DIFE requires that derivatives behave A DIFE requires that derivatives behave smoothly, since they are linked to the smoothly, since they are linked to the function itself.function itself.

Nonlinear DIFE’s can provide compact Nonlinear DIFE’s can provide compact and elegant models for systems and elegant models for systems exhibiting exceedingly complex exhibiting exceedingly complex behavior.behavior.

The Rössler EquationsThe Rössler Equations

This nearly linear system exhibits chaotic This nearly linear system exhibits chaotic behavior that would be virtually impossible behavior that would be virtually impossible to model without using a DIFE:to model without using a DIFE:

( ) ( ) ( )

( ) ( ) ( )

( ) ( ( ) ) ( )

Dx t y t z t

Dy t x t ay t

Dz t b x t c z t

Stochastic DIFE’sStochastic DIFE’s

We can introduce randomness into We can introduce randomness into DIFE’s in many ways:DIFE’s in many ways:

Random coefficient functions.Random coefficient functions. Random forcing functions.Random forcing functions. Random initial, boundary, and other Random initial, boundary, and other

constraints.constraints. Time unfolding at a random rate.Time unfolding at a random rate.

DeliverablesDeliverables

If we can model data on functions or If we can model data on functions or functional input/output systems, we will functional input/output systems, we will have a modeling tool that greatly have a modeling tool that greatly extends the power and scope of existing extends the power and scope of existing nonparametric curve-fitting techniques. nonparametric curve-fitting techniques.

We may also get better estimates of We may also get better estimates of functional parameters and their functional parameters and their derivatives. derivatives.

A simple input/output A simple input/output systemsystem

We begin by looking at a first order We begin by looking at a first order DIFE for a single output function DIFE for a single output function x(t)x(t) and a single input function and a single input function u(t). u(t). (SISO)(SISO)

But our goal is the linking of multiple But our goal is the linking of multiple inputs to multiple outputs (MIMO) by inputs to multiple outputs (MIMO) by linear or nonlinear systems of linear or nonlinear systems of arbitrary order arbitrary order mm..

( ) ( ) ( ) ( ) ( )Dx t t x t t u t •u(t)u(t) is often called the is often called the forcing functionforcing function, , and and is an exogenous functional is an exogenous functional independent independent variablevariable..•Dx(t) = -Dx(t) = -ββ(t)x(t) (t)x(t) is called the is called the homogeneoushomogeneous part of the equationpart of the equation..•αα(t) (t) and and ββ(t) (t) are the are the coefficient coefficient functionsfunctions that define the DIFE.that define the DIFE.•The system is The system is linearlinear in these in these coefficient coefficient functions, and in the input functions, and in the input u(t)u(t) and and output output x(t).x(t).

In this simple case, an analytic solution is In this simple case, an analytic solution is possible:possible:

0 0( ) ( )

0( ) [ (0) ( ) ( ) ]

t sts ds r dr

x t e x t e u s ds

However, it is necessary to use However, it is necessary to use numerical numerical methods to find the solution to most methods to find the solution to most DIFE’S. DIFE’S.

A simpler constant coefficient A simpler constant coefficient exampleexample

We can see more clearly what happens when We can see more clearly what happens when the coefficients the coefficients αα and and ββ are constants, are constants, αα = 1, = 1, xx00 = 0, and = 0, and u(t)u(t) is a step function, stepping from 0 to 1 is a step function, stepping from 0 to 1

at time at time tt11::

1( )1

1( ) [1 ],t tx t e t t

Constant Constant αα//ββ is the is the gaingain in in the system.the system.

Constant Constant ββ controls the controls the responsivityresponsivity of the of the system to a system to a change in change in input. input.

A Real Example: Lupus A Real Example: Lupus treatmenttreatment

Lupus is an incurable auto-immune Lupus is an incurable auto-immune disease that mainly afflicts women.disease that mainly afflicts women.

It flares unpredictably, inflicting wide It flares unpredictably, inflicting wide damage with severe symptoms.damage with severe symptoms.

The treatment is The treatment is prednisoneprednisone, an immune , an immune system suppressant used in transplants.system suppressant used in transplants.

But prednisone has serious short- and But prednisone has serious short- and long-term side affects, and exposure to it long-term side affects, and exposure to it must be controlled.must be controlled.

0 1 2 3 4 5

0

5

10

15

20

25

30

Year

SLE

DA

I (R

/G),

Pre

dnis

one

(B)

Patient 13, Nobs = 70, NSLEDAI = 40, Nflare = 13

How to Estimate a Differential How to Estimate a Differential Equation from Raw DataEquation from Raw Data

A previous method, A previous method, principal principal differential analysisdifferential analysis, first smoothed the , first smoothed the data to get functions data to get functions x(t)x(t) and and u(t),u(t), and and then estimated the coefficient functions then estimated the coefficient functions defining the DIFE.defining the DIFE.

This two-stage procedure is inelegant This two-stage procedure is inelegant and probably inefficient. Going directly and probably inefficient. Going directly from data to DIFE would be better.from data to DIFE would be better.

Profile Least SquaresProfile Least Squares

The idea is to replace the function fitting The idea is to replace the function fitting the raw data, the raw data, x(t),x(t), by the equations defining by the equations defining the fit to the data conditional on the DIFE.the fit to the data conditional on the DIFE.

Then we optimize the fit with respect to Then we optimize the fit with respect to only the unknown parameters defining the only the unknown parameters defining the DIFE itself. DIFE itself.

The fit The fit x(t)x(t) is defined as a by-product of the is defined as a by-product of the process, but does not itself require process, but does not itself require additional parameters.additional parameters.

This This profiling processprofiling process is often used in nonlinear is often used in nonlinear least squares problems where some parameters least squares problems where some parameters are easily solved for given other parameters.are easily solved for given other parameters.

There we express the conditional estimates of There we express the conditional estimates of the these easy-to-estimate parameters as the these easy-to-estimate parameters as functions of the unknown hard-to-estimate functions of the unknown hard-to-estimate parameters, and optimize only with respect to parameters, and optimize only with respect to the hard parameters.the hard parameters.

This saves both computational time and degrees This saves both computational time and degrees of freedom.of freedom.

An alternative strategy is to integrate over the An alternative strategy is to integrate over the easy parameters, and optimize with respect to easy parameters, and optimize with respect to the hard ones; this is the M-step in the EM the hard ones; this is the M-step in the EM algorithm.algorithm.

The DIFE as a linear differential The DIFE as a linear differential operatoroperator

We can re-express the first order DIFE as a We can re-express the first order DIFE as a linear differential operator:linear differential operator:

( ) ( ) ( ) ( ) ( ) ( ) 0Lx t t x t Dx t t u t

More compactly, suppressing “(t)”, and making More compactly, suppressing “(t)”, and making explicit the dependency of explicit the dependency of LL on on αα and and ββ,,

L x x Dx u

Smoothing data with the Smoothing data with the operator operator LL

If we know the differential equation, then the differential operator If we know the differential equation, then the differential operator LL defines a data smoother (Heckman and Ramsay, 2000). defines a data smoother (Heckman and Ramsay, 2000).

The fitting criterion is:The fitting criterion is:

2

2

1

( ) ( )N

i ii

PENSSE y x t L x t dt

The larger The larger λλ is, the more the fitting function is, the more the fitting function x(t)x(t) is forced to be a solution of the differential equation is forced to be a solution of the differential equation LLαβαβx(t) = 0x(t) = 0..

Let Let x(t)x(t) be expanded in terms of a be expanded in terms of a set set K K basis functions basis functions φφkk(t),(t),

1

( )K

k kk

x t c t

• Let Let NN by by KK matrix matrix ZZ contain the values contain the values of these basis functions at time points of these basis functions at time points ttii , and, and• Let Let yy be the vector of raw data. be the vector of raw data.

Then the smooth values have the Then the smooth values have the expression expression Zc,Zc,

where where cc is the vector of coefficients. is the vector of coefficients. But these coefficients are easy parameters But these coefficients are easy parameters

to estimate given operator to estimate given operator LLαβαβ . The . The expression for them isexpression for them is1( ' ) ( ' )

( )( ) ' ( )

c Z Z R Z y s

R L L s L u

GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

• We therefore remove parameter vector We therefore remove parameter vector cc by by replacing it with the expression above.replacing it with the expression above.

How to estimate How to estimate LLLL is a function of weight coefficients is a function of weight coefficients αα(t)(t) and and ββ(t).(t).

If these have the basis function expansionsIf these have the basis function expansions

( ) ;J L

j j l lj l

t a t t b t then we can optimize the profiled error sum of then we can optimize the profiled error sum of squaressquares

2

( , )N

i ii

SSE a b y y

with respect to coefficient vectors with respect to coefficient vectors aa and and bb..

It is also a simple matter to:It is also a simple matter to: constrain some coefficient functions to be constrain some coefficient functions to be

zero or a constant.zero or a constant. force some coefficient functions to be force some coefficient functions to be

smooth, employing specific linear smooth, employing specific linear differential operators to smooth them differential operators to smooth them towards specific target spaces. We do this towards specific target spaces. We do this by appending penalties to SSE(a,b), such asby appending penalties to SSE(a,b), such as

22ˆ( , ) ( )

N

i ii

PENSSE a b y y M t dt

where where MM is a linear differential operator for is a linear differential operator for penalizing the roughness of penalizing the roughness of ββ..

And more …And more …

This approach is easily generalizable to:This approach is easily generalizable to: DIFE’s and differential operators of DIFE’s and differential operators of

any order.any order. Multiple inputs Multiple inputs uujj(t)(t) and outputs and outputs xxii(t).(t). Replicated functional data.Replicated functional data. Nonlinear DIFE’s and operators.Nonlinear DIFE’s and operators.

Adaptive smoothingAdaptive smoothing

We can also use this approach to have the We can also use this approach to have the level of smoothing vary. We modify the level of smoothing vary. We modify the differential operator as follows:differential operator as follows:

( )( ) ( ) ( ) ( ) ( ) ( )tLx t t x t e Dx t t u t The exponent function The exponent function κκ(t)(t) plays the role of a plays the role of a log log λλ that varies with that varies with tt..

Choosing the smoothing parameter Choosing the smoothing parameter λλ is is always a delicate matter.always a delicate matter.

The right value of The right value of λλ will be rather large if the will be rather large if the data can be well-modeled by a low-order data can be well-modeled by a low-order DIFE.DIFE.

But it should not so large as to smooth away But it should not so large as to smooth away additional functional variation that may be additional functional variation that may be important.important.

Estimating Estimating λλ by generalized cross-validation by generalized cross-validation seems to work reasonably well, at least for seems to work reasonably well, at least for providing a tentative value to be explored providing a tentative value to be explored furtherfurther..

A First ExampleA First Example

The first example simulates replicated The first example simulates replicated data where the true curves are a set of data where the true curves are a set of tilted sinusoids.tilted sinusoids.

The operator The operator LL is of order 4 with constant is of order 4 with constant coefficients. coefficients.

How precisely can we estimate these How precisely can we estimate these coefficients?coefficients?

How accurately can we estimate the How accurately can we estimate the curves and first two derivatives?curves and first two derivatives?

For replications For replications i=1,…,Ni=1,…,N and time values and time values j=1,…,nj=1,…,n, let, let

1 2 3 4sin 6 cos 6ij i i j i j i j ijy c c t c t c t

where the where the ccikik’s’s and the and the εεijij’s’s are are N(0,1);N(0,1); and and t = 0(0.01)1t = 0(0.01)1..The functional variation satisfies the differential The functional variation satisfies the differential equationequation

2 2 4( ) (6 ) ( ) ( ) 0Lx t D x t D x t where where ββ00(t) = (t) = ββ11(t) = (t) = ββ33(t)=0(t)=0 and and ββ22(t)(t) = (6 = (6ππ))2 2 = 355.3.= 355.3.

For simulated data with For simulated data with N = 20N = 20 replications and replications and constant bases for constant bases for ββ00(t) ,…, (t) ,…, ββ33(t), (t), we getwe get

L = DL = D44: : best results are forbest results are for λλ=10=10-10-10 and the and the RIMSE’s for derivatives 0, 1 and 2 are 0.32, 9.3 RIMSE’s for derivatives 0, 1 and 2 are 0.32, 9.3 and 315.6, respectively.and 315.6, respectively.

LL estimated estimated: best results are for: best results are for λλ=10=10-5-5 and and the RIMSE’s are 0.18, 2.8, and 49.3, the RIMSE’s are 0.18, 2.8, and 49.3, respectively.respectively.

giving precision ratios of 1.8, 3.3 and 6.4, giving precision ratios of 1.8, 3.3 and 6.4, resp.resp.

ββ22 was estimated as 353.6 whereas the true was estimated as 353.6 whereas the true value was 355.3.value was 355.3.

ββ33 was 0.1, with true value 0.0. was 0.1, with true value 0.0.

In addition to better curve estimates and In addition to better curve estimates and much better derivative estimates, note that much better derivative estimates, note that the derivative RMSE’s do not go wild at the the derivative RMSE’s do not go wild at the end points, which is usually a serious end points, which is usually a serious problem with polynomial spline smoothing.problem with polynomial spline smoothing.

This is because the DIFE ties the This is because the DIFE ties the derivatives to the function values, and the derivatives to the function values, and the function values are tamed at the end points function values are tamed at the end points by the data.by the data.

A decaying harmonic with a A decaying harmonic with a forcing functionforcing function

Data from a second order equation Data from a second order equation defining harmonic behavior with defining harmonic behavior with decay, forced by a step function, is decay, forced by a step function, is generated bygenerated by

ββ0 0 = 4.04, = 4.04, ββ1 1 = 0.4, = 0.4, αα = -2.0. = -2.0. u(t) = 0, t < 2u(t) = 0, t < 2ππ, u(t) = 1, t ≥ 2, u(t) = 1, t ≥ 2ππ.. Adding noise with std. dev. 0.2.Adding noise with std. dev. 0.2.

0 2 4 6 8 10 12-1.5

-1

-0.5

0

0.5

1

1.5

2

t

datax(t)u(t)

ParameterParameter True True ValueValue

Mean Mean EstimateEstimate

Std. ErrorStd. Error

ββ00 4.0404.040 4.0414.041 0.0730.073

ββ11 0.4000.400 0.3970.397 0.0480.048

αα -2.000-2.000 -1.998-1.998 0.0880.088

With only one replication, using minimum With only one replication, using minimum generalized cross-validation to choose generalized cross-validation to choose λλ, the , the results estimated for 100 trials are:results estimated for 100 trials are:

An oil refinery exampleAn oil refinery example

The single input is “reflux flow” and The single input is “reflux flow” and the output is “tray 47” level in a the output is “tray 47” level in a distillation column.distillation column.

There were 194 sampling points.There were 194 sampling points. 30 B-spline basis functions were used 30 B-spline basis functions were used

to fit the output, and a step function to fit the output, and a step function was used to model the input.was used to model the input.

After some experimentation with first After some experimentation with first and second order models, and with and second order models, and with constant and varying coefficient constant and varying coefficient models, the clear conclusion seems models, the clear conclusion seems to be the constant coefficient model:to be the constant coefficient model:

( ) 0.02 ( ) 0.19 ( )Dx t x t u t

The standard errors of The standard errors of ββ and and αα in this model, in this model, as estimated by parametric bootstrapping, as estimated by parametric bootstrapping, were 0.0004 and 0.0023, respectively. were 0.0004 and 0.0023, respectively. The delta method yielded 0.0004 and 0.0024, The delta method yielded 0.0004 and 0.0024, respectively. Pretty much the same. respectively. Pretty much the same.

Monotone smoothingMonotone smoothing

Some constrained functions can be Some constrained functions can be expressed as DIFE’s.expressed as DIFE’s.

A smooth strictly monotone function A smooth strictly monotone function can be expressed as the second order can be expressed as the second order DIFEDIFE

2 ( ) ( ) ( )D x t t Dx t

We can monotonically smooth data by We can monotonically smooth data by estimating the second order DIFE estimating the second order DIFE directly.directly.

We constrain We constrain ββ00(t) = 0(t) = 0, and give , and give ββ11(t)(t) enough flexibility to smooth the data.enough flexibility to smooth the data.

In the following artificial example, the In the following artificial example, the smoothing parameter was chosen by smoothing parameter was chosen by generalized cross-validation. generalized cross-validation. ββ11(t)(t) was was expanded in terms of 13 B-splines. expanded in terms of 13 B-splines.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

t

x(t

)

DataEstimateTrue

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

2

4

6

8

10

12

t

Dx(

t) DataEstimateTrue

Analyzing the Lupus dataAnalyzing the Lupus data

Weight function Weight function ββ(t)(t) defining an defining an order 1 DIFE for symptoms estimated order 1 DIFE for symptoms estimated with and without prednisone dose as with and without prednisone dose as a forcing function.a forcing function.

Weight expanded using B-splines Weight expanded using B-splines with knots at every observation time.with knots at every observation time.

Weight Weight αα(t)(t) for prednisone is for prednisone is constant.constant.

The forced DIFE for lupusThe forced DIFE for lupus

0 1 2 3 4 5

-20

0

20

40

60

80

0(t

)

0 1 2 3 4 5 6-7

-6.5

-6

-5.5

-5

-4.5

1(t

)

The data fitThe data fit

0 1 2 3 4 5 6-10

-5

0

5

10

15

20

25

30

35

t

x(t

)

DataNonhomogeneousHomogeneous

Adding the forcing function halved the Adding the forcing function halved the LS fitting criterion being minimized.LS fitting criterion being minimized.

We see that the fit improves where the We see that the fit improves where the dose is used to control the symptoms, dose is used to control the symptoms, but not where it is not used.but not where it is not used.

These results are only suggestive, and These results are only suggestive, and much more needs to be done.much more needs to be done.

We want to model treatment and We want to model treatment and symptom as mutually influencing each symptom as mutually influencing each other. This requires a system of two other. This requires a system of two differential equations. differential equations.

SummarySummary We can estimate differential equations We can estimate differential equations

directly from noisy data with little bias and directly from noisy data with little bias and good precision.good precision.

This gives us a lot more modeling power, This gives us a lot more modeling power, especially for fitting input/output functional especially for fitting input/output functional data.data.

Estimates of derivatives can be much Estimates of derivatives can be much better, relative to smoothing methods.better, relative to smoothing methods.

Functions with special properties such as Functions with special properties such as monotonicity can be fit by estimating the monotonicity can be fit by estimating the DIFE that defines them.DIFE that defines them.