Embed Size (px)
Citation preview
Hyperproperties of Real-Valued SignalsLuan Viet Nguyen
University of Texas at [email protected]
James KapinskiToyota Motor North America R&D
Xiaoqing JinToyota Motor North America R&D
Jyotirmoy V. DeshmukhToyota Motor North America R&[email protected]
Taylor T. JohnsonVanderbilt University
ABSTRACTA hyperproperty is a property that requires two or more executiontraces to check. is is in contrast to properties expressed usingtemporal logics such as LTL, MTL and STL, which can be checkedover individual traces. Hyperproperties are important as they areused to specify critical system performance objectives, such asthose related to security, stochastic (or average) performance, andrelationships between behaviors. We present the rst study of hy-perproperties of cyber-physical systems (CPSs). We introduce a newformalism for specifying a class of hyperproperties dened overreal-valued signals, called HyperSTL. e proposed logic extendssignal temporal logic (STL) by adding existential and universal tracequantiers into STL’s syntax to relate multiple execution traces.Several instances of hyperproperties of CPSs including stability,security, and safety are studied and expressed in terms of HyperSTLformulae. Furthermore, we propose a testing technique that allowsus to check or falsify hyperproperties of CPS models. We present adiscussion on the feasibility of falsifying or verifying various classesof hyperproperties for CPSs. We extend the quantitative semanticsof STL to HyperSTL and show its utility in formulating algorithmsfor falsication of HyperSTL specications. We demonstrate howwe can specify and falsify HyperSTL properties for two case studiesinvolving automotive control systems.
ACM Reference format:Luan Viet Nguyen, James Kapinski, Xiaoqing Jin, Jyotirmoy V. Deshmukh,and Taylor T. Johnson. 2017. Hyperproperties of Real-Valued Signals. InProceedings of MEMOCODE ’17, Vienna, Austria, September 29-October 2,2017, 10 pages.DOI: 10.1145/3127041.3127058
1 INTRODUCTIONHyperproperties were rst proposed by Clarkson and Schneiderto characterize properties of security policies that cannot be de-ned over individual traces, such as service level agreements andinformation-ow properties [15]. In this work, we extend the no-tion of hyperproperties to cover a broad range of requirements for
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prot or commercial advantage and that copies bear this notice and the full citationon the rst page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permied. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specic permission and/or afee. Request permissions from [email protected] ’17, Vienna, Austria© 2017 ACM. 978-1-4503-5093-8/17/09. . .$15.00DOI: 10.1145/3127041.3127058
cyber-physical systems (CPSs), and we present a taxonomy of hy-perproperties used to address security and control design concernsfor CPSs. Also, we provide practical techniques for automating theprocess of testing hyperproperties for CPSs.
In contrast to trace properties expressed over individual execu-tion traces, hyperproperties are dened over multiple executiontraces. For example, one execution of a system cannot be checkedagainst a service level agreement property such as “the averagetime elapsed between a user’s request and response over all executionsshould be less than 1 second”; the property can only be evaluatedover all system execution traces. Moreover, we can consider aninformation-ow policy of noninterference specied as “for all pairsof traces of a system that have the same low-level security inputs,they will also have the same low-level security output” [22, 41]. isnoninterference property is a hyperproperty as it is expressed overall pairs of traces of a system.
Hyperproperties generalize more traditional formal properties byspecifying relationships between disparate execution traces, insteadof behaviors of individual execution traces. Traditional logics thatconsider traces individually, such as LTL, cannot be used to specifyhyperproperties, and thus, hyperproperties are more expressive.Logics such as CTL and CTL* allow properties over multiple pathsof a computation tree, but they do not permit comparisons betweenthe paths themselves. Instead, to express and eciently checkhyperproperties, Clarkson et al., introduced notions of HyperLTLand HyperCTL* [14]. Both logics directly extend LTL and allow usto reason about more than one execution trace at a time. e maindierence between HyperLTL and HyperCTL* is that the formerrequires trace quantiers appearing at the beginning of a formula,but the laer allows us to specify them within a formula.
Although hyperproperties are well studied in the context ofsecurity policies for soware systems, hyperproperties have notbeen explored for CPSs. For a CPS that includes stochastic factorssuch as noise, environment disturbance, or transducer inaccuracies,it is realistic for design engineers to expect that the system hassome acceptable performance in a probabilistic sense rather thanrequiring an absolute performance limit be met for all individualbehaviors. Acceptable performances dened over the averages ofseling time, overshoot, undershoot, or error bounds cannot bespecied and checked using individual execution traces; they mustbe quantied over all execution traces.
Recently, security-aware function modeling of CPSs has emergedas an important research topic in computer science and systemverication. A CPS, which is an integration between cyber andphysical subcomponents, can be vulnerable to both cyber-basedand physical-based aacks [5, 19, 39, 48]. For instance, consider a
MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria Nguyen et al.
modern automobile, which is a complex CPS composed of manycomputer units such as an Engine Control Unit (ECU), the Trans-mission Control Module (TCM), and an Electronic Brake ControlModule (EBCM), all interacting with the physical world via sensorsand actuators. Cyber-based aackers can gain access to the com-munication channels of a modern automobile through wireless orin-vehicle networks. As a result, aackers can inltrate an ECUor EBCM to stall the engine or disable the brake system [30, 45].An alternative method of aack involves gaining physical accessto the system, for example by manipulating the signals processedby the sensors (known as sensor spoong), to compromise secureinformation or to alter system behaviors [5, 46]. Instances of ac-tive physical-based aacks include vehicle braking system aacks,where faulty data is injected into the wheel speed sensor of a vehi-cle to disrupt the braking function [48], and insulin delivery deviceaacks, where glucose level sensor data is corrupted to compromisethe function of the insulin delivery service [31]. A passive physical-based aack, also called a side-channel aack, is based on physicallyobserving the system behavior and using leaked information togain insights into the system implementation [26, 28, 42]. Somewell-known side channel aacks are power analysis aacks [27],timing aacks [29], electromagnetic aacks [43] and dierentialfault analysis aacks [10].
Designing a safety-critical CPS that is entirely secure from bothcyber-based and physical-based aacks is challenging or impossi-ble. A reasonable approach is to iteratively improve a CPS controldesign using a falsication technique. Falsication is an automatedbest-eort approach to identify system behaviors that violate agiven formal specication [40]. e design approach would be torst formally specify safety properties of a CPS that protect thesystem against possible cyber-based and physical-based aacksusing formalisms such as temporal logic and to then iterativelyimprove the design using falsication, which would automaticallyidentify vulnerabilities in the design. Despite the aractiveness offalsication techniques, aacks for CPSs oen need to be denedover multiple execution traces of the system, which is somethingthat cannot be expressed or falsied using existing temporal log-ics such as LTL, MTL, and STL. us we propose an extension tothese logics that would be compatible with the appropriate spec-ications. In this work, we present a study of hyperpropertiesincluding stability, security and safety, as applied to CPSs. Weintroduce several instances of hyperproperties capturing relation-ships (e.g input-output relationships) between multiple traces ofa CPS. We extend the syntax and semantics of STL [17] to specifyhyperproperties over dense-time real-valued signals, which resultsin a new logic called HyperSTL. Basically, we add quantiers atthe beginning of an STL formula to express relationships betweenmultiple traces. We also introduce a testing algorithm based ona fragment of HyperSTL and apply it to nd falsifying traces forhyperproperties of industrial Simulink models. Moreover, we pro-vide a discussion on the feasibility of falsifying or verifying variousclasses of hyperproperties for CPSs.Related work. e study of hyperproperties for CPSs evaluatedin this paper was inspired by the previous work of Clarkson andSchneider, who introduced hyperproperties to express security
policies such as secure information ows and service level agree-ments [15]. In [13], Bryans et. al. presented a general formalizationof opacity policies that prevent observers from deducing the truthvalue of a predicate; those opacity policies require behaviors tobe specied over multiple paths of a system. In earlier work [37],McLean showed that some “possibilistic” security properties likerestrictiveness [35], noninterference[22] and nondeducibility [49]are closure properties that cannot be expressed by individual exe-cution traces. In [37], those properties are specied with respectto dierent sets of trace contractors called selective interleavingfunctions.
Following the introduction of hyperproperties [15], Clarksonet al. introduced HyperLTL and HyperCTL*, which are exten-sions to existing temporal logics, to express and check classes ofinformation-ow hyperproperties [14]. ese logics extended LTLand CTL* by adding the path quantiers that permit specicationsinvolving multiple paths in the system. Model checking algorithmsand complexity of fragments of HyperLTL and HyperCTL* werealso given in [14], which were then further exploited and appliedto check some classes of information-ow hyperproperties in [41].
Prototype implementations of model checkers for HyperLTLand HyperCTL*, which assume the system is modeled as a Kripkestructure, can verify some information-ow hyperproperties of adiscrete-time system, but extending that work to check hyperprop-erties dened over continuous traces is a challenging endeavor.For complex CPS models or for models built in frameworks withproprietary or otherwise obfuscated semantics, such as Simulink®,formal verication of hyperproperties is eectively impossible, asno corresponding Kripke structure may be obtained from thosemodels1. Alternatively, an easier but still dicult task is to developan ecient testing framework, which could be used to check hy-perproperties for nite collections of traces or could be used tofalsify hyperproperties of a CPS model; this is the contribution ofthe work presented herein.
In [50], Xu et al. introduced a notion of CensusSTL that utilizesSTL by adding an outer logic to quantify the number of individualagents of a multiagent system whose behaviors satisfy an inner STLformula. CensusSTL is similar to the HyperSTL proposed in thispaper; however, the former is only able to specify group behaviorsfrom dierent components of an individual trace while the laerallows us to express relationships between multiple traces.
e remainder of the paper is organized as follows. Section 2 re-views relevant background. Section 3 introduces several examples ofhyperproperties of CPSs including stability, security and safety. Sec-tion 4 presents the syntax and semantics of HyperSTL. Section 5and Section 6 describe the testing algorithm for two fragments ofHyperSTL. Section 7 applies the proposed approach to nd falsify-ing traces for some hyperproperties of industrial Simulink models,and Section 8 concludes the paper.
1Some have created their own translation of Simulink models to modeling languageswith well-dened formal semantics (for example, see [3, 52]), but these translationsnecessarily only handle a subset of the Simulink/Stateow modeling language. isis due to the fact that some Simulink constructs correspond to behaviors that cannotbe modeled using standard frameworks for hybrid systems. One such construct isthe Variable Transport Delay block, which, roughly speaking, corresponds to a delaydierential equation, a construct that is not handled by standard modeling frameworksfor hybrid systems.
Hyperproperties of Real-Valued Signals MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria
2 PRELIMINARIESIn this section, we review the concepts of signal, system, traceproperty, falsication, and verication.Signal. We dene a signal w as a function w : T → D, whereT ⊆ R≥0 is the time domain. If D = B, w is a Boolean signal whosevalue is either true or false, and ifD = R, then we say that the signalis real-valued. A trace, w : T→ D1 × . . . × Dn , is a collection of nsignals, where ∀t ∈ T,w(t) ∆
= (w1(t),w2(t), ...,wn (t)). Intuitively,we can consider w as one execution trace of a continuous-timesystem with n variables that describes an evolution of the system.In what follows, we reserve the use of bold leers like w, w′ fortraces (i.e., tuples of signals), while we use lowercase italicizedleers such as wi to represent signals.System. We dene a deterministic or nonstochastic2 cyber-physicalsystem Σ as a function mapping a given input trace in (T→ Dm )to an output trace in (T→ Dn ). We denote by JΣK the set of tracesw such that the rstm components of w correspond to them inputsignals for JΣK, and the next n components correspond to the noutput signals.Trace properties. A trace property φ is a nite or innite set ofindividual traces. A trace property is either satised or violated byany given set of traces [6, 41]. A set of tracesW satises the traceproperty φ ifW ⊆ φ. As noted above, an individual trace can haveseveral components, for example, a trace could contain m inputsignals and n output signals of a given system Σ. We say that thetrace property φ holds for a system Σ (denoted as Σ |= φ) if the setof input-output traces compatible with the system description iscontained in the trace property, i.e., JΣK ⊆ φ.Falsication. Given a trace property φ and a CPS Σ, the falsi-cation problem is to nd a non-empty set W ⊆ JΣK such thatW * φ.Verication. Given a trace property ϕ, the verication problem ofa CPS Σ with respect to ϕ is to show that JΣK ⊆ ϕ.
3 HYPERPROPERTIES OF REAL-VALUEDSIGNALS
Hyperproperties generalize formal properties of a system by con-sidering sets of sets of execution traces, instead of only sets ofexecution traces.
Denition 3.1 (Hyperproperty). Let S denote the set of all traces. Letthe power set of S be wrien as P ∆
= P(S). A hyperproperty is anysubset of P(S).
We say a set of tracesW satises a hyperpropertyϕ ⊆ P ifW ∈ ϕ.Given a hyperproperty ϕ and a system Σ, the falsication task is tond a non-empty setW ⊆ JΣK such thatW < ϕ. Similarly, given ahyperproperty ϕ and a system Σ, the verication task is to showthat JΣK ∈ ϕ.
2Note the contrast with stochastic systems. In stochastic systems, one or more partsof the system have randomness associated with them; for instance, the value of aparticular system parameter may be drawn from a probability distribution. e keydierence is that the stochastic system may not produce the same output for a giveninput. Unless otherwise specied, all the systems that we consider in this paper aredeterministic.
In this section, we introduce hyperproperties for determinis-tic systems to characterize properties such as security, safety, andstability. We focus on a class of hyperproperties capturing relation-ships (e.g., the input-output relationship) between multiple traces ofa system, and we show several examples of hyperproperties relatedto stability and security for CPSs. In rest of this section, we usedsup (w,w′) to denote the sup-norm distance between traces w andw′, where dsup (w,w′) = supt ∈R≥0 | |w(t) −w
′(t)| |.• Robust behavior is a requirement that guarantees that small dif-
ferences in system inputs result in small dierences in systemoutputs. Consider the following property: “For all pairs of tracesof a system with an input dierence less than ϵ1, the output dier-ence should be bounded by ϵ2”. Such a property is a hyperpropertyas it requires at least two execution traces to check. is hyper-property can be formally wrien as:
ϕ1∆= W ∈ P | ∀w,w′ ∈W : dsup (win ,w′in ) ≤ ϵ1
=⇒ dsup (wout ,w′out ) ≤ ϵ2. (1)is type of property is related to certain stability notions, such asbounded input, bounded output (BIBO) stability and the L2 gain,as these notions also bound the variation in the output, basedon bounded variation in the input. We note, however, that therobust behavior hyperproperty diers from BIBO stability andthe L2 gain, as the robust behavior hyperproperty is speciedover all pairs of execution traces while the BIBO and L2 proper-ties are dened based on individual traces. e robust behaviorhyperproperty is also related to bisimulation relations [18] andconformance-closeness [2] for a dynamical system, as all three ofthese properties are based on some constraints on the distancesbetween multiple traces. In fact, we may specify bisimulationor conformance-closeness functions in terms of hyperproperties.Lastly, we note that the robust behavior hyperproperty is per-haps most closely related to Lipschitz Robustness of systems [23],which bounds dierences in output behaviors based on boundeddierences in input behaviors, though Lipschitz Robustness wasoriginally developed for timed input/output systems as opposedto general CPS models.
• Side-channel aacks are aacks against cryptographic devicesbased on studying leaking information about the operationsthey process, such as power consumption, heat generation, andexecution time. e side channel aack is an instance of aninactive physical-based aack that can be used against a CPS inwhich some physical behaviors are observable. Aackers candeduce the working principle of a system without either accessto the system itself or an understanding of the internal operationof the system. For example, aackers can analyze an abnormalchange in the power consumption of an integrated circuit whilean encryption process is being executed and then reconstructthe encryption key to access secret data [27, 28]. e followingproperty permits side-channel aacks:
ϕ2∆= W ∈ P | ∃w ∈W : ∀w′ ∈W : (dsup (w,w′) > 0∧ Power(w(t)) > c1) =⇒ Power(w′(t)) < c2, (2)
where Power(w(t)) represents the power consumption corre-sponding to w over time, and c1, c2 are arbitrary constants suchthat c1 > c2. A system that satises this property allows an
MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria Nguyen et al.
aacker to detect that a particular behavior has occurred (w inFormula 2) by monitoring the power associated with the behav-ior. e property is a hyperproperty as it is expressed in termsof multiple traces. To ensure the safety of a system from thepower-monitoring aack, the system should satisfy ¬ϕ2. Wenote that other classes of side-channel aacks such as timingaacks, electromagnetic aacks, and dierential fault analysisaacks can be specied using properties similar to Formula 2.
• Robust control invariance is a property that can be used to syn-thesize safe controllers, or more to the point, can be utilized todetermine whether a safe controller exists for systems with dis-turbances [11]. Informally, the property states that, for a givenset of behaviors that is deemed safe, a control action exists, suchthat the system remains within the safe set for any allowabledisturbance input. is can be stated formally as follows:
ϕ3∆= W ∈ P | ∃w ∈W : ∀w′ ∈W : (w,w′) |= ϕ, (3)
where (w,w′) |= ϕ means that the pair (w,w′) satises someproperty ϕ. In this formulation, wu (t) is the component of wthat represents the controller action,wd (t) is a disturbance input,wy (t) is a system output, and (w,w′) |= ϕ enforces both thatwu = w ′u and w ′y (t) ∈ Ω, where Ω is the set of safe behaviors.e robust control invariance property is related to fault datainjection (FDI) aacks, which are active physical-based aackswhere aackers try to input faulty data into a system to corruptthe behavior of the controller. For example, aackers can spoofthe sensors of DC microgrids by injecting false data such asthe past outputs of the sensors at previous time instants. isinstance of FDI aack is also well known as a replay aack [8,31, 48]. FDI aacks have been studied widely for CPS, and manytechniques have been proposed to eciently detect those aacksin the early stages [8, 32, 34]. However, the optimal solution isto design a system that can defend itself against FDI aacks [38].To guarantee that a system can defend against a sensor aack,given a specication ϕ, it must be possible to choose a controllerthat ensures that the output of the system always satises ϕ, i.e.ϕ3 must hold.
3.1 Beyond Hyperproperties?A hyperproperty is more expressive than a trace property as it isdened over a set of sets of traces and requires multiple tracesto check. If a system is modeled as trace sets, one interestingquestion to ask is whether there are system properties inexpressibleas hyperproperties. For security policies, all properties of tracesets can be considered as hyperproperties, so the answer may benegative [6, 15]. For CPSs, there may exist some properties that arechallenging to classify.
Consider the following property specifying the Lyapunov stabil-ity of a dynamical control system:
ϕLy∆= ∀ϵ ∈ [0,∞),∃δ ∈ [0, ϵ),∀w ∈W :| |w(0)| | < δ =⇒ (t > 0 ∧ ||w(t)| | < ϵ). (4)
Intuitively, this property indicates that a system is Lyapunov stableif for any ϵ-ball around the origin, there exists a δ -ball around theorigin (δ < ϵ) such that if the system starts within the δ -ball, then
𝜖
𝛿
Figure 1: Illustration of a Lyapunov stable system.
it will never leave the ϵ-ball [9]. e illustration of a Lyapunovstable system is shown in Figure 1.
Lyapunov stability is specied over the space of parameters andexecution traces, and involves two alternations between universaland existential quantiers. As we cannot check the Lyapunovstability with individual traces, it is not a trace property; so is ita hyperproperty? Consider the parameters δ and ϵ as constantsignals, and then rewrite Lyapunov stability as follows:
ϕ ′Ly∆= W ∈ P | ∀w ∈W : ∃w′ ∈W : ∀w′′ ∈W :
| |w ′′out (0)| | < w ′δ (0) =⇒ (t > 0 ∧ ||w ′′out (t)| | < wϵ (t)), (5)
where a trace w is composed of two constant input signals wδ , wϵand an output signal wout . By mapping parameters into constantsignals, we can express interesting properties of the system ashyperproperties. en Lyapunov stability is a hyperproperty thatrequires multiple traces to check; and it can be formally speciedusing the HyperSTL introduced in the next section. As to theoriginal question of whether all system properties of interest canbe specied as hyperproperties, we leave this open.
Remark 3.2 Although we focus on describing hyperpropertiesdened over real-valued signals, we note that there are other hyper-properties that can be specied in the context of CPSs as well. Forinstance, the nondeducibility property is an important information-ow security policy that prevents a low-level observer with su-cient knowledge of a target CPS from deducing high-level (con-dential) information. e nondeducibility property is dened suchthat for each low-level input trace, there are more than one possiblehigh-level input traces that produce the same output. Intuitively,an aacker should not be able to distinguish between permissiblehigh-level behaviors based on low-level behaviors [20, 36]. Onthe other hand, the noninterference property is another importantinformation-ow security policy that requires that high-level secu-rity users should not interfere with low-level security users. Intu-itively, the outputs observed by the low-level security users remainunchanged despite the actions of the high-level security users [22].Other variants of the noninterference property such as noninfer-ence [37], observational determinism [51], declassication [44], andquantitative nonterinference [47] are also hyperproperties that needto be specied over multiple traces. ough the nondeducibilityand noninterference properties are relevant for CPS, in many casestheir impact on and from real-valued signals is tenuous, and so wedo not treat them further herein.
Hyperproperties of Real-Valued Signals MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria
4 HYPERSTLIn this section, we introduce HyperSTL, a temporal logic that canbe used to specify a class of hyperproperties of real-valued signals.e syntax and semantics of HyperSTL are naturally extended fromthose of STL by adding existential and universal trace quantiersinto STL’s syntax to relate multiple execution traces [17].
Syntax. Let v be a trace variable from an innite set of tracevariablesV . e syntax of HyperSTL is then dened as follows:
ϕ := ∃v.ϕ | ∀v.ϕ | φφ := true | µv | ¬φ | φ ∧ φ | φUIφ
Here, we add a universal quantier ∀ and an existential quantier∃ to the syntax to indicate whether we want to specify that aformula holds over all traces or over at least one trace, respectively.For instance, ∀v.∃v′.ϕ means that for any trace w assigned totrace variable v , there exists a trace w′ that can be assigned totrace variable v′ such that ϕ holds on these two traces. We deneΠ : V → S as a trace assignment (i.e., a valuation), which is apartial function mapping trace variables to traces, and S is a set ofall innite traces. Letvi be the projection of a trace variable v alongits ith component, the projection of a trace assignment Π(vi ) mapsvi to the ith component of a trace w (i.e., wi ). Also, we abuse thesubscript notation of a trace’s component to write its correspondingtrace variable’s component in a HyperSTL formula, e.g., wout isrepresented by vout . A trace w can be Booleanized through atomicpredicates of the form µw
∆= f (w1(t),w2(t), ...,wn (t)) > 0, where
f is a real-valued function. en, µv = f (Π(v)(t)) > 0 representsa Booleanized atomic predicate µw if v is instanced by w. Also, Iis an interval over R≥0 such as [a,b), (a,b), (a,b], [a,b], (a,+∞),or [a,+∞), where a, b are real numbers and 0 ≤ a < b. If I isnot specied, we assume that I = [0,∞). We also allow Booleanoperators ∨ and =⇒ with their standard meaning. Temporaloperators used in HyperSTL formulas include always (), eventually(^), and until (U), respectively, where ^Iφ = trueUIφ, and Iφ =¬^I¬φ. Note that we use trace variables such as v, v′ to expressHyperSTL formula and the corresponding traces represented bythese trace variables like w w′ to interpret the formula. Considerthe HyperSTL formula ϕ := ∃v.∀v′.[0,1](| |v − v′ | | < 1). isproperty says that there is always a trace w, such that for all timesin the interval [0, 1], every other trace w′ is at a bounded distanceof 1 from w.
Boolean Semantics. A HyperSTL formula satised by a set oftracesW at a time t is wrien as Π, t |=W ϕ, e validity judgmentof a HyperSTL formula at a given time t is specied according tothe following recursive semantics:
Π, t |=W ∃v.ϕ i exists w ∈W : w |= ϕ and Π(v) = w
Π, t |=W ∀v.ϕ i forall w ∈W : w |= ϕ and Π(v) = w
Π, t |=W µv i f (Π(v)(t)) > 0Π, t |=W ¬φ i Π, t 6 |=W φ
Π, t |=W φ1 ∧ φ2 i Π, t |=W φ1 and Π, t |=W φ2
Π, t |=W φ1UIφ2 i ∃t1 ∈ t + I s.t Π, t1 |=W φ2
and ∀t2 ∈ [t , t1] s.t Π, t2 |=W φ1
Using HyperSTL, we can express the hyperproperties describedin Section 3 over some time interval [t1, t2] as follows3.• e robust behavior in Formula 1 can be specied as:
ϕ ′1∆= ∀v.∀v′. [t1,t2](dsup (vin , v
′in ) ≤ ϵ1
=⇒ dsup (vout , v′out ) ≤ ϵ2). (6)
• e power-monitoring aack in Formula 2 can be wrien as:
ϕ ′2∆= ∃v.∀v′. [t1,t2]((dsup (v, v
′) > 0∧ Power (v) > c1) =⇒ Power (v′) < c2). (7)
Furthermore, we can rewrite the Lyapunov stability specied inFormula 5 as the following HyperSTL formula
ϕ ′′Ly∆= ∀v.∃v′.∀v′′. (v ′′out < v ′δ =⇒ (0,∞)v
′′out < vϵ ). (8)
According to the possible alternation of quantiers in a Hyper-STL’s syntax, we classify the above HyperSTL formulae into twofragments:
(a) alternation-free HyperSTL formulae including one type ofquantier, and
(b) k-alternation HyperSTL formulae that have k number ofalternations between existential and universal quantiers.
us, the robust behavior property can be expressed using alternation-free HyperSTL while the power-monitoring aack property canbe specied using 1-alternation HyperSTL. e Lyapunov stabil-ity property is more complex as it must be expressed using 2-alternation HyperSTL.Falsication or Verication of Hyperproperties? We have in-troduced several classes of hyperproperties for CPSs and a temporallogic approach to express them. Next, we investigate whether wecan falsify or verify those hyperproperties using existing methods.Hyperproperties are more complex and expressive than traditionalproperties, and performing falsication and verication for hyper-properties is harder, in many cases. Despite this, we observe thatcertain classes of hyperproperties can be falsied or veried. Forinstance, we can falsify an alternation-free HyperSTL formula thatcontains a universal quantier (e.g., the robust behavior hyperprop-erty), and we can verify an alternation-free HyperSTL formula thatcontains an existential quantier. For the class of hyperpropertiesthat includes alternating quantiers, falsication or vericationare oen undecidable unless we impose some assumption aboutthe sets of execution traces (e.g., quantied over some nite set oftraces with bounded time).
4.1 t-HyperSTLWe introduce t-HyperSTL as a fragment of HyperSTL in which anesting structure of temporal logic formulas involving dierenttraces is not allowed. For example, a formula ∀v.∃v′.[0,2]v >1 =⇒ ^[1,2]v′ > 2 is allowed but a formula ∀v.∃v′.[0,2](v >1 =⇒ ^[1,2]v′ > 2) is not allowed. Also, t-HyperSTL restrictsthe until operator to be specied over an individual trace, e.g., t-HyperSTL does not allow the formula ∀v.∃v′.(v > 1)U[0,1](v′ > 2).
Inherited from the syntax of HyperSTL, t-HyperSTL formulaeare also classied into alternation-free and k-alternation types.3 For a robust control invariance hyperproperty, an instance of the correspondingHyperSTL formula will be shown in Section 7.2.
MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria Nguyen et al.
t-HyperSTL suces to express the class of hyperproperties formu-lated in Section 3, and its corresponding semantics, which is morerestrictive than that of HyperSTL, allow us to perform falsicationfor these hyperproperties.
antitative Semantics. e quantitative semantics of t-HyperSTLreects the robustness satisfaction of a t-HyperSTL formula. It is anatural extension of those for STL [17, 33]. Given χ is a real-valuedfunction of a formula φ, a trace assignment Π, a trace variable v,and a time t , the quantitative semantics of t-HyperSTL is denedinductively as follows:
χ (φ,Π,∃v, t) = maxw∈W
χ (φ,Π(v) = w, t)
χ (φ,Π,∀v, t) = minw∈W
χ (φ,Π(v) = w, t)
χ (µv > 0,Π, v, t) = µvχ (¬φ,Π, v, t) = −χ (φ,Π, v, t)
χ (φ1 ∧ φ2,Π, v, t) = min (χ (φ1,Π, v, t), χ (φ2,Π, v, t))χ (φ1UIφ2,Π, v, t) = sup
t1∈t+Imin (χ (φ2,Π, v, t1),
inft2∈[t,t1]
χ (φ1,Π, v, t2))
5 FALSIFYING ALTERNATION-FREET-HYPERSTL
We rst consider the falsication of alternation-free t-HyperSTLformulae. is fragment of HyperSTL is expressive enough tocapture a broad range of hyperproperties specifying input-outputrelationships over all pairs of execution traces. We use a translationscheme called self-composition [7], which allows us to falsify analternation-free t-HyperSTL formula that includes only universalquantiers using a robust testing method for a normal STL formula.en, given an alternation-free t-HyperSTL that includes universalquantiers, we aempt to nd a set of falsifying traces for CPSscorresponding to this formula.Falsication algorithm. e procedure that addresses the falsi-cation problem of a system Σ with respect to a given hyperpropertyφh over a time duration T is shown in Algorithm 1, and furtherinterpreted as follows. We rst transform the alternation-free t-HyperSTL formula φh
into the equivalent STL formula φST L . We then call a function NewSystemGen to generate a new
model that contains copies of the original system. e numberof copies is equal to the number of quantiers of the formulaφh .
en, we apply existing falsication mechanisms for an STL for-mula such as Breach4 [16] to compute the minimum robustnessvalue χmin of the system Σ′ according to φST L . Breach allowsus to parametrically generate dierent input signals over a pa-rameter space. For example, parameters can represent controlpoints, and an input signal can be created using interpolationbetween these points. If χmin is negative we return the optimalset of parameters Θf ∈ Θ that produces a falsifying behavior.
4Breach [16] is a tool that applies a best-eort approach to automatically check whethera system satises a given STL formula.
Algorithm 1 Falsication of alternation-free t-HyperSTL
1 Require: a system Σ, a parameter space Θ,
a t-HyperSTL formula φh , a time duration T ,
3 a maximum number of simulations N
begin5 φST L ← HyperSTL2STL(φh ) / / t r a n s f o rm s p e c i f i c a t i o n
Σ′ ← NewSystemGen(Σ, φh ) / / t r a n s f o rm model7 χmin, Θf ← FalsifySTL(Σ′, φST L, Θ, T , N )
if χmin < 0 then9 return Θf
end11 end
We note that, unlike formal verication, performing falsicationcannot ensure a system is always safe; even if falsication fails toidentify a falsifying behavior, a counter-example may still exist.
Example 5.1. Consider a mechanical mass-spring damper systemwhose dynamics are dened by the second-order ordinary dieren-tial equation:
Üx(t) + 2 Ûx(t) + 5x(t) = 3F (t), (9)
where x is the vertical position of the mass, and F is the randomexternal force. e robust behavior hyperproperty of the systemis specied as follows: for all pairs of traces of the system withthe external force dierence less than ϵ1 , the output dierenceshould be bounded by ϵ2; here ϵ1 = 0.2 and ϵ2 = 0.3. We apply theAlgorithm 1 to falsify the robust behavior hyperproperty for thesystem with a duration T = 10 seconds. Formula 6 can be reducedto the normal STL formula as follows:
ϕM∆= [0,10](ρin ≤ ϵ1 =⇒ ρout ≤ ϵ2), (10)
where a trace p ∆= (ρin , ρout ) of the system Σ′ captures the input-
output dierence between two traces w,w′ of the original systemΣ′, e.g., ρin (t) = | |win (t) − w ′in (t)| |. Here, the system Σ′ con-tains two copies of the mechanical mass-spring damper system Σ.e falsication result shown in Figure 2 illustrates the inductivechecking procedure for the satisfaction of Formula 10 using Breach,where alw[0,10] is equivalent to [0,10], and the le y-axis denotesrobustness degree. Here, we observe that the violation of the robustbehavior hyperproperty of the mechanical mass-spring dampersystem occurs during the overshoot period of the outputs of thesystem.
Remark 5.2 ere is a duality between addressing the falsica-tion problem of an alternation-free t-HyperSTL that only containsuniversal quantiers and solving the verication problem of analternation-free t-HyperSTL that only contains existential quan-tiers. Given an alternation-free t-HyperSTL such as ∃v.∃v′.ϕe ,our purpose is to extensively simulate a system and nd a singlepair of execution traces of the system that satises ϕe . Here, wedo not aempt to falsify the system, but verify the system. us,this process is dual to nding the falsifying traces of the systemcorresponding to the formula ∀v.∀v′.¬ϕe .Also, we note that we can leverage Algorithm 1 such that it includesa parameter synthesis approach to mine hyperproperties for CPSs,as in [24, 25]. For instance, we could use a requirement mining
Hyperproperties of Real-Valued Signals MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria
0 2 4 6 8 10-0.05
00.05
false
trueQuant. satBool. sat
0 2 4 6 8 10-0.15
00.15
false
true
0 2 4 6 8 10-0.15
00.15
false
true
0 2 4 6 8 10
Time (seconds)
-0.020
0.02
false
true
Figure 2: Falsication result of the mass-spring damper sys-tem. e counterexample pair of traces found by Breach forthe robust behavior hyperproperty.
approach to automatically infer appropriate values for the ϵ1 andϵ2 variables in Formula 10.
6 FALSIFYING K-ALTERNATION T-HYPERSTLFalsifying k-alternation t-HyperSTL formulas is a challenging task,as it requires us to examine all execution traces of a system. Con-sider a 1-alternation t-HyperSTL formula such as ∃v.∀v′.ϕ; falsify-ing a system for this property is as hard as verifying the system,since we need to show that for all traces w ∈ S , there exists a tracew′ that the formula ϕ is violated, where S is an innite set of traces.It is even more dicult to perform falsication for CPSs whosedynamics evolve continuously over time. Furthermore, if a hyper-property contains more than one alternation of quantiers (e.g. theLyapunov stability property), the falsifying algorithm may sueran exponential growth in complexity. Despite this, if we assumea CPS can be modeled by a nite set of traces, we can develop afalsifying algorithm for the system that can prove or disprove ϕ.
In general, there may not exist a unique answer to the questionof whether we can verify or falsify a system with respect to theformula ∃v.∀v′.ϕ using nite simulations. We can consider severalpossible answers for that question as follows. Case 1: if both w,w′ belong to some innite set of traces, then
we can neither verify nor falsify ϕ. Case 2: if w belongs to an innite set of traces and w′ belongs
to a nite set of traces, then we cannot falsify but we can verifyϕ.
Case 3: if w belongs to a nite set of traces and w′ belongsto an innite set of traces, then we cannot verify but we canfalsify ϕ.
Case 4: If both w and w′ belong to a nite set of n traces, weare able to verify the system with n simulations as well as falsifythe system with n(n−1)
2 simulations.We note that in all of the cases that we are able to falsify the systemcorresponding to the formula ∃v.∀v′.ϕ with nite simulations, wecan apply Algorithm 1 to transform the falsication problem toanother equivalent problem that uses a traditional STL specication.
Table 1: Feasibility of solving the falsication and verica-tion problems for properties and hyperproperties expressedusing STL and k-alternation t-HyperSTL under two assump-tions: A1) using nite simulation and A2) applying a veri-cation oracle that can do reachability analysis with respectto the last quantier.
TypeA1: Finite Simulation A2 : Verication Oracle
on the Last antierFalsication Verication
∀ Yes No -
∃ No Yes -
∀∃ No No ∀∃∀ No No ∃∀∃∀ No No ∀∃∃∀∃ No No ∃∀
e falsication procedure is similar to solving the falsicationproblem of alternation-free t-HyperSTL.
For the case that both execution traces of a system, w and w′,belong to some innite sets, and if we have a verication oracle toaddress the last quantier (e.g., by conservatively estimating theset of possible system behaviors, under certain conditions), we caneither falsify or verify the system. Given a set of initial states, averication oracle can be a method that mathematically overap-proximates the reachable set of the system or a simulation-basedtechnique [1, 21] that may verify the system with nite simulations.
Alternatively, for a hyperproperty that requires two or morealternations of quantiers to express, even if we have a vericationoracle corresponding to the last quantier, we can neither falsifynor verify a system. Using a verication oracle, the feasibility ofaddressing the falsication and verication problems associatedwith a k-alternation t-HyperSTL formula is equivalent to that of a(k − 1)-alternation t-HyperSTL formula; this is shown in Table 1.We emphasize that any hyperproperties for general CPSs that areas complex as, or more complicated than Lyapunov stability, arenot veriable or falsiable without reasonable restrictions on setsof execution traces.
7 CASE STUDYIn this section, we introduce two proof-of-concept case studies inthe domain of automotive control systems: a) an industrial-scaleSimulink model of a closed-loop airpath control (APC) system andb) a Simulink model of a fault-tolerant fuel (FTF) control system. Wewill demonstrate how to apply the testing framework of HyperSTLbuilt on top of Breach to falsify the robust behavior hyperpropertyof the APC system, and the robust control invariance hyperpropertyof the FTF system under FDI aacks.
7.1 Airpath Control ModelWe use a prototype APC system to evaluate the capability of ourproposed method on an industrial control system. e APC is a keysubsystem for a hydrogen Fuel-Cell (FC) vehicle powertrain. e
MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria Nguyen et al.
0 2 4 6 8 100
0.5
1
;te
mp
0 2 4 6 8 100
0.5
1
;F
CI
0 2 4 6 8 10
Time (seconds)
0
0.5
1
;A
FR
Violation
Figure 3: Falsication result of the APC system. e coun-terexample pair of traces found by Breach for the robust be-havior hyperpropperty.
purpose of the APC is to regulate the air ow rate into the FC stackusing multiple actuators. e FC stack generates electrical powerfor the vehicle using a mixture of air and hydrogen. e FC stackonly operates under restricted conditions, such as temperature,pressure and moisture level within the stack. An excess of moisturein the stack will impede the performance while moisture deciencycould permanently damage the FC stack. us, to achieve highperformance while still operating the system in a safe regime, thecontroller is required to accurately regulate the air ow rate.
e closed-loop Simulink model of the APC system is complex;it contains more than 7,000 Simulink blocks such as integrators, sat-urations, S-Function blocks, lookup tables, and data store memoryblocks. e model has two input signals including i) the ambienttemperature and ii) the fuel cell current request (FCI). Details of thesystem, such as units and expected signal ranges, are suppressed dueto proprietary concerns. Intuitively, an FCI value is proportionalto the desired torque requested by the driver, which is ultimatelybased on the accelerator pedal angle. e output of the APC systemis an air ow rate (AFR). e purpose of the controller model is toregulate the AFR to some desirable reference value. To ensure theAPC system works properly, for some small perturbations of theambient temperature and FCI values, the dierences in AFR valuesshould be bounded within a desirable range. In other words, toavoid unexpected changes in the air ow rate at the inlet of an FCstack, which may cause undesirable behavior, the system shouldsatisfy the robust behavior hyperproperty. e robust behaviorhyperproperty of the APC system can be formalized as follows,
ϕAPC∆= W ∈ P | ∀w,w′ ∈W :(dsup (wtemp ,w
′temp ) ≤ ϵ1 ∧ dsup (wFCI ,w
′FCI ) ≤ ϵ2)
=⇒ dsup (wAFR ,w′AFR ) ≤ ϵ3), (11)
which can be translated to the following STL formula using Algo-rithm 1 to perform the falsication task,
ϕ ′APC∆= [0,T ]((ρtemp ≤ ϵ1 ∧ ρFCI ≤ ϵ2) =⇒ ρAFR ≤ ϵ3),
(12)
where a trace w is composed of the temperature and FCI inputsignals wtemp and wFCI respectively, and the AFR output signalwAFR . Here, we create a new model including two copies of theoriginal APC system; and a trace p ∆
= (ρtemp , ρFCI , ρAFR ) of thenew model captures the input-output dierence between two tracesw,w′ of the original model, for instance, ρtemp (t) = | |wtemp (t) −w ′temp (t)| |.
e result of falsication of the robust behavior hyperpropety ofthe APC system is shown in Figure 3, where the blue lines presentthe distance signals ρtemp , ρFCI , ρAFR respectively, and the redlines demonstrate their corresponding bounds. Here, the parametervalues selected by a design engineer are normalized to 0.5. atis, ϵ1 = 0.5, ϵ2 = 0.5, and ϵ3 = 0.5. e sampling time is 0.001024seconds and the simulation time T is 10 seconds. For proprietaryreasons, we normalize the quantities and suppress the units forthe data shown in the gure. e counterexample pairs of tracesreported by Breach demonstrate a behavior where the output dif-ference exceeds its allowed bounds when the input dierences arestill less than their given thresholds, which is a violation of For-mula 12. Finding this counter-example is signicant, as it can helpautomotive control engineers to improve the controller design toeliminate such an undesirable behavior of the APC system.
7.2 Fault-tolerant Fuel ModelWe consider a fault-tolerant fuel (FTF) model that includes bothSimulink blocks and Stateow charts5. e model has two externalinput signals, engine speed and throle command, and one outputsignal, which is the eective air-fuel ratio inside the combustionchamber. e model also contains four sensors measuring throleangle, engine speed, the amount of residual oxygen in the exhaustgas (EGO), and the manifold absolute pressure (MAP). e controllerhas three dierent control strategies: a normal operation mode,which is used when no sensor faults are present, a fault mitigationmode, which is used when one sensor fault has occurred, and amode that disables fuel control, which is used when two or moresensor faults are detected. We only consider the normal and faultmitigation modes for this example. e goal of the controller is toregulate the air-fuel ratio output, denoted as λ, so that it remainswithin a desirable range, despite a failure in at most one sensor.
In this case study, we evaluate the ability of the FTF controllerto tolerate an engine speed sensor fault. In the original version ofthe model, a speed sensor fault consists of the speed sensor outputbeing set to 0.0 rad/sec; the controller detects the fault when thesensor reading equals 0.0. In the modied version that we use, wedo not x the controller mode based on the sensor reading, butinstead we evaluate the controller performance when either thenormal or fault mitigation modes are selected. In the modiedversion of the model that we use, a speed sensor fault consists of asensor output producing a xed but randomly selected value in thesensor range [0, 620] rad/sec. is kind of sensor fault could occurwhen an aacker uses a sensor spoong approach to inject incorrectmeasurements into the sensor readings or when a real fault occursin the speed sensor. We use the robust control invariance propertyto specify desired controller performance in the presence of the
5We use a modied version of the FTF model available at hps://www.mathworks.com/help/simulink/examples/modeling-a-fault-tolerant-fuel-control-system.html
Hyperproperties of Real-Valued Signals MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria
indicated class of sensor faults:
ϕFT F∆= ∃v.∀v′.[τ ,∞](dsup (vu ,v ′u ) = 0=⇒ 0.8λr ef ≤ v ′λ ≤ 1.2λr ef ), (13)
where λr ef is the reference value of the air-fuel ratio λ, and τ is theseling time. Here, a trace variable v can be mapped to a trace wcomposed of the controller input wu corresponding to a controllermode decision, a disturbance wd representing the xed randomsensor input injected into the speed sensor, and an output wλ .Ingeneral, we cannot falsify Formula 13 according to the discussionshown in Table 1; however, for systems like the FTF model thathave a nite set of control strategies, we can eectively performfalsication by creating a new model that contains copies of theoriginal system, one copy for each control mode (two copies, in thiscase). e external input (the speed sensor reading) is connected toeach of the copies of the model. e specication ϕFT F is convertedto the following equivalent formula in standard STL:
ϕFT F∆= ∀wd .[τ ,∞](0.8λr ef ≤ wλ1 ≤ 1.2λr ef∨ 0.8λr ef ≤ wλ2 ≤ 1.2λr ef ), (14)
where wλ1 and wλ2 are the air-fuel ratios of the rst and secondcopies of the model. We note that Formula 14 is arrived at by apply-ing the quantitative semantics provided in Sec. 4; the disjunctionin Formula 14 appears due to the ∃ quantier in Formula 13, whicheectively applies a max operator over the two available controlmodes. e formula ϕFT F can be tested using the falsicationmethods for traditional STL available in Breach.
Figure 4 illustrates the falsication result of the FTF model. eblue lines correspond to a simulation trace representing the falsi-fying behavior, the green line illustrates an instance of the correctspeed, and the red lines represent the error bound of λ, whereτ = 10 seconds, T = 50 seconds, and λr ef = 14.6. Based on theresults, we can conclude that there exists a trace, which includesoutputs wλ1 and wλ2 that both evolve beyond the tolerance boundregardless of whether the controller operates in the normal modeor the fault mitigation mode (i.e., the performance requirementis violated despite which control mode is used). is experimentdemonstrates the capability of using a falsication approach toautomatically test hyperproperties for CPSs.
8 CONCLUSION AND FUTUREWORKIn this paper, we represented the rst study of the hyperpropertiesof CPSs. We dened a new temporal logic, called HyperSTL, toexpress several hyperproperties including stability, security, andsafety for CPSs. HyperSTL allows us to eectively specify moregeneral requirements of CPS rather than STL as it can express therelationships between multiple execution traces. e testing frame-work of t-HyperSTL, a fragment of HyperSTL, was also given andapplied to falsify the robust behavior hyperproperty of a hydrogenfuel-cell powertrain model, and the robust control invariance hy-perproperty of the fuel control model under a fault data injectionaack. We also discuss the feasibility of performing the falsicationand verication for various classes of hyperproperties for CPSs.
0 10 20 30 40 500
300
600
Spe
ed
0 10 20 30 40 500
10
20
30
w6
1
= = 10
0 10 20 30 40 50
Time (seconds)
0
10
20
30
w6
2
= = 10
Figure 4: Apair of falsifying traces found byBreach illustrat-ing the FTF model cannot tolerate the fault under a speedsensor fault.
Future Work. We rst plan to introduce a library of HyperSTLfomulae that encapsulates dierent general classes of hyperprop-erties of CPS including those presented in this paper. Second, thefalsication algorithm of HyperSTL proposed in the paper is in-complete as it relies on self-composition (i.e. making copies of asystem) and only falsies a restricted class of hyperproperties. us,extending the falsication algorithm to bypass self-compositionto falsify more interesting hyperproperties is planned. Also, themonitoring algorithms of HyperLTL recently proposed in [4, 12]could be applied to HyperSTL.
9 ACKNOWLEDGMENTSe authors would like to acknowledge Borzoo Bonakdarpour andhis research group for insightful comments that helped us rene ourdenition of HyperSTL. We would also like to thank the anonymousreviewers for their feedback. e material presented in this paperis based upon work supported by the National Science Founda-tion (NSF) under grant numbers CNS 1464311, EPCN 1509804, andSHF 1527398, the Air Force Research Laboratory (AFRL) throughcontract numbers FA8750-15-1-0105, and FA8650-12-3-7255 via sub-contract number WBSC 7255 SOI VU 0001, and the Air Force Oceof Scientic Research (AFOSR) under contract numbers FA9550-15-1-0258 and FA9550-16-1-0246. e U.S. government is authorizedto reproduce and distribute reprints for Governmental purposesnotwithstanding any copyright notation thereon. Any opinions,ndings, and conclusions or recommendations expressed in thispublication are those of the authors and do not necessarily reectthe views of AFRL, AFOSR, or NSF.
REFERENCES[1] Houssam Abbas, Bardh Hoxha, Georgios Fainekos, and Koichi Ueda. 2014.
Robustness-guided temporal logic testing and verication for stochastic cyber-physical systems. In Cyber Technology in Automation, Control, and IntelligentSystems (CYBER), 2014 IEEE 4th Annual International Conference on. IEEE, 1–6.
[2] Houssam Abbas, Hans Mielmann, and Georgios Fainekos. 2014. Formal prop-erty verication in a conformance testing framework. In Formal methods andmodels for codesign (memocode), 2014 twelh acm/ieee international conference on.IEEE, 155–164.
MEMOCODE ’17, September 29-October 2, 2017, Vienna, Austria Nguyen et al.
[3] Aditya Agrawal, Gyula Simon, and Gabor Karsai. 2004. Semantic Translation ofSimulink/Stateow Models to Hybrid Automata Using Graph Transformations.Electronic Notes in eoretical Computer Science 109 (2004), 43 – 56.
[4] Shreya Agrawal and Borzoo Bonakdarpour. 2016. Runtime verication of k-safetyhyperproperties in HyperLTL. In Computer Security Foundations Symposium(CSF), 2016 IEEE 29th. IEEE, 239–252.
[5] Mohammad Al Faruque, Francesco Regazzoni, and Miroslav Pajic. 2015. Designmethodologies for securing cyber-physical systems. In Proceedings of the 10thInternational Conference on Hardware/Soware Codesign and System Synthesis.IEEE Press, 30–36.
[6] Mack W Alford, Jean-Pierre Ansart, Gunter Hommel, Leslie Lamport, BarbaraLiskov, Geo P Mullery, and Fred B Schneider. 1985. Distributed systems: methodsand tools for specication. An advanced course. Springer-Verlag New York, Inc.
[7] Gilles Barthe, Pedro R D’Argenio, and Tamara Rezk. 2004. Secure informationow by self-composition. In Computer Security Foundations Workshop, 2004.Proceedings. 17th IEEE. IEEE, 100–114.
[8] Omar Beg, Taylor Johnson, and Ali Davoudi. 2017. Detection of False-data Injec-tion Aacks in Cyber-Physical DC Microgrids. IEEE Transactions on IndustrialInformatics (2017).
[9] Nam Parshad Bhatia and Giorgio P Szego. 2002. Stability theory of dynamicalsystems. Springer Science & Business Media.
[10] Eli Biham and Adi Shamir. 1997. Dierential fault analysis of secret key cryp-tosystems. In Annual International Cryptology Conference. Springer, 513–525.
[11] F. Blanchini. 1999. Survey Paper: Set Invariance in Control. Automatica 35, 11(Nov. 1999), 1747–1767.
[12] Noel Bre, Umair Siddique, and Borzoo Bonakdarpour. 2017. Rewriting-BasedRuntime Verication for Alternation-Free HyperLTL. In International Conferenceon Tools and Algorithms for the Construction and Analysis of Systems. Springer,77–93.
[13] Jeremy W Bryans, Maciej Koutny, Laurent Mazare, and Peter YA Ryan. 2008.Opacity generalised to transition systems. International Journal of InformationSecurity 7, 6 (2008), 421–435.
[14] Michael R Clarkson, Bernd Finkbeiner, Masoud Koleini, Kristopher K Micinski,Markus N Rabe, and Cesar Sanchez. 2014. Temporal logics for hyperproperties.In International Conference on Principles of Security and Trust. Springer, 265–284.
[15] Michael R Clarkson and Fred B Schneider. 2010. Hyperproperties. Journal ofComputer Security 18, 6 (2010), 1157–1210.
[16] Alexandre Donze. 2010. Breach, a toolbox for verication and parameter synthe-sis of hybrid systems. In Computer Aided Verication. Springer, 167–170.
[17] Alexandre Donze, omas Ferrere, and Oded Maler. 2013. Ecient robust moni-toring for STL. In Computer Aided Verication. Springer, 264–279.
[18] Georgios E Fainekos, Antoine Girard, and George J Pappas. 2006. Temporal logicverication using simulation. In International Conference on Formal Modelingand Analysis of Timed Systems. Springer, 171–186.
[19] oshitha T Gamage, Bruce M McMillin, and omas P Roth. 2010. Enforcinginformation ow security properties in cyber-physical systems: A generalizedframework based on compensation. In Computer Soware and Applications Con-ference Workshops (COMPSACW), 2010 IEEE 34th Annual. IEEE, 158–163.
[20] oshitha T. Gamage, Bruce M. McMillin, and omas P. Roth. 2010. EnforcingInformation Flow Security Properties in Cyber-Physical Systems: A GeneralizedFramework Based on Compensation.. In COMPSAC Workshops. IEEE ComputerSociety, 158–163.
[21] Antoine Girard and George J Pappas. 2006. Verication using simulation. InInternational Workshop on Hybrid Systems: Computation and Control. Springer,272–286.
[22] Joseph A Goguen and Jose Meseguer. 1982. Security policies and security models.In Security and Privacy, 1982 IEEE Symposium on. IEEE, 11–11.
[23] omas A. Henzinger, Jan Otop, and Roopsha Samanta. 2016. Lipschitz Robustnessof Timed I/O Systems. Springer Berlin Heidelberg, Berlin, Heidelberg, 250–267.
[24] Bardh Hoxha, Adel Dokhanchi, and Georgios Fainekos. [n. d.]. Mining parametrictemporal logic properties in model-based design for cyber-physical systems.International Journal on Soware Tools for Technology Transfer ([n. d.]), 1–15.
[25] Xiaoqing Jin, Alexandre Donze, Jyotirmoy V Deshmukh, and Sanjit A Seshia.2015. Mining requirements from closed-loop control models. Computer-AidedDesign of Integrated Circuits and Systems, IEEE Transactions on 34, 11 (2015),1704–1717.
[26] John Kelsey, Bruce Schneier, David Wagner, and Chris Hall. 1998. Side channelcryptanalysis of product ciphers. In European Symposium on Research in ComputerSecurity. Springer, 97–110.
[27] Paul Kocher, Joshua Jae, and Benjamin Jun. 1999. Dierential power analysis.In Annual International Cryptology Conference. Springer, 388–397.
[28] Paul Kocher, Ruby Lee, Gary McGraw, Anand Raghunathan, and SrivathsModerator-Ravi. 2004. Security as a new dimension in embedded system design.In Proceedings of the 41st annual Design Automation Conference. ACM, 753–760.
[29] Paul C Kocher. 1996. Timing aacks on implementations of Die-Hellman, RSA,DSS, and other systems. In Annual International Cryptology Conference. Springer,104–113.
[30] Karl Koscher, Alexei Czeskis, Franziska Roesner, Shwetak Patel, TadayoshiKohno, Stephen Checkoway, Damon McCoy, Brian Kantor, Danny Anderson, Ho-vav Shacham, et al. 2010. Experimental security analysis of a modern automobile.In Security and Privacy (SP), 2010 IEEE Symposium on. IEEE, 447–462.
[31] Chunxiao Li, Anand Raghunathan, and Niraj K Jha. 2011. Hijacking an insulinpump: Security aacks and defenses for a diabetes therapy system. In e-HealthNetworking Applications and Services (Healthcom), 2011 13th IEEE InternationalConference on. IEEE, 150–156.
[32] Lanchao Liu, Mohammad Esmalifalak, Qifeng Ding, Valentine A Emesih, andZhu Han. 2014. Detecting false data injection aacks on power grid by sparseoptimization. IEEE Transactions on Smart Grid 5, 2 (2014), 612–621.
[33] Oded Maler and Dejan Nickovic. 2004. Monitoring temporal properties of con-tinuous signals. In Formal Techniques, Modelling and Analysis of Timed andFault-Tolerant Systems. Springer, 152–166.
[34] Kebina Manandhar, Xiaojun Cao, Fei Hu, and Yao Liu. 2014. Detection of faultsand aacks including false data injection aack in smart grid using kalman lter.IEEE transactions on control of network systems 1, 4 (2014), 370–379.
[35] Daryl McCullough. 1987. Specications for multi-level security and a hook-up.In Security and Privacy, 1987 IEEE Symposium on. IEEE, 161–161.
[36] John McLean. 1990. Security models and information ow. In Research in Securityand Privacy, 1990. Proceedings., 1990 IEEE Computer Society Symposium on. IEEE,180–187.
[37] John McLean. 1994. A general theory of composition for trace sets closedunder selective interleaving functions. In Research in Security and Privacy, 1994.Proceedings., 1994 IEEE Computer Society Symposium on. IEEE, 79–93.
[38] Shaunak Mishra, Yasser Shoukry, Nikhil Karamchandani, Suhas Diggavi, andPaulo Tabuada. 2015. Secure state estimation: optimal guarantees against sen-sor aacks in the presence of noise. In Information eory (ISIT), 2015 IEEEInternational Symposium on. IEEE, 2929–2933.
[39] Yilin Mo, Tiany Hyun-Jin Kim, Kenneth Brancik, Dona Dickinson, Heejo Lee,Adrian Perrig, and Bruno Sinopoli. 2012. Cyber–physical security of a smartgrid infrastructure. Proc. IEEE 100, 1 (2012), 195–209.
[40] Truong Nghiem, Sriram Sankaranarayanan, Georgios Fainekos, Franjo Ivancic,Aarti Gupta, and George J. Pappas. 2010. Monte-carlo Techniques for Falsicationof Temporal Properties of Non-linear Hybrid Systems. In Proceedings of the 13thACM International Conference on Hybrid Systems: Computation and Control (HSCC’10). ACM, New York, NY, USA, 211–220.
[41] Markus N Rabe. 2016. A temporal logic approach to information-ow control.(2016).
[42] Srivaths Ravi, Anand Raghunathan, and Srimat Chakradhar. 2004. Tamper resis-tance mechanisms for secure embedded systems. InVLSI Design, 2004. Proceedings.17th International Conference on. IEEE, 605–611.
[43] Pankaj Rohatgi. 2009. Electromagnetic aacks and countermeasures. In Crypto-graphic Engineering. Springer, 407–430.
[44] Andrei Sabelfeld and David Sands. 2005. Dimensions and principles of declassi-cation. InComputer Security Foundations, 2005. CSFW-18 2005. 18th IEEEWorkshop.IEEE, 255–269.
[45] Florian Sagsteer, Martin Lukasiewycz, Sebastian Steinhorst, Marko Wolf,Alexandre Bouard, William R Harris, Somesh Jha, omas Peyrin, AxelPoschmann, and Samarjit Chakraborty. 2013. Security challenges in automo-tive hardware/soware architecture design. In Proceedings of the Conference onDesign, Automation and Test in Europe. EDA Consortium, 458–463.
[46] Yasser Shoukry, Paul Martin, Paulo Tabuada, and Mani Srivastava. 2013. Non-invasive Spoong Aacks for Anti-lock Braking Systems. In Proceedings of the15th International Conference on Cryptographic Hardware and Embedded Systems(CHES’13). Springer-Verlag, Berlin, Heidelberg, 55–72.
[47] Georey Smith. 2009. On the foundations of quantitative information ow. InInternational Conference on Foundations of Soware Science and ComputationalStructures. Springer, 288–302.
[48] Jiang Wan, Arquimedes Canedo, and Mohammad Abdullah Al Faruque. 2015.Security-aware functional modeling of cyber-physical systems. In EmergingTechnologies & Factory Automation (ETFA), 2015 IEEE 20th Conference on. IEEE,1–4.
[49] J Todd Wibold and Dale M Johnson. 1990. Information ow in nondetermin-istic systems. In Research in Security and Privacy, 1990. Proceedings., 1990 IEEEComputer Society Symposium on. IEEE, 144–161.
[50] Zhe Xu and A Agung Julius. 2016. Census signal temporal logic inference formultiagent group behavior analysis. IEEE Transactions on Automation Scienceand Engineering (2016).
[51] Steve Zdancewic and Andrew C Myers. 2003. Observational determinism forconcurrent program security. In Computer Security Foundations Workshop, 2003.Proceedings. 16th IEEE. IEEE, 29–43.
[52] Liang Zou, Naijun Zhan, Shuling Wang, and Martin Franzle. 2015. Formal Veri-cation of Simulink/Stateow Diagrams. In Automated Technology for Vericationand Analysis - 13th International Symposium, Shanghai, China. 464–481.
