arX

iv:1

308.

5053

v1 [

cs.E

T]

23 A

ug 2

013

1

Delay Optimal Schedulingfor Energy Harvesting Based Communications

Juan Liu, Huaiyu Dai,Senior Member, IEEE, and Wei Chen,Senior Member, IEEE

Abstract—Green communications attract increasing researchinterest recently. Equipped with a rechargeable battery, asourcenode can harvest energy from ambient environments and rely onthis free and regenerative energy supply to transmit packets. Dueto the uncertainty of available energy from harvesting, however,intolerably large latency and packet loss could be induced,ifthe source always waits for harvested energy. To overcome thisproblem, one Reliable Energy Source (RES) can be resorted tofor a prompt delivery of backlogged packets. Naturally, thereexists a tradeoff between the packet delivery delay and powerconsumption from the RES. In this paper, we address the delayoptimal scheduling problem for a bursty communication linkpowered by a capacity-limited battery storing harvested energytogether with one RES. The proposed scheduling scheme givespriority to the usage of harvested energy, and resorts to theRESwhen necessary based on the data and energy queueing processes,with an average power constraint from the RES. Through two-dimensional Markov chain modeling and linear programmingformulation, we derive the optimal threshold-based schedulingpolicy together with the corresponding transmission parameters.Our study includes three exemplary cases that capture someimportant relations between the data packet arrival processand energy harvesting capability. Our theoretical analysis iscorroborated by simulation results.

Index Terms—Energy harvesting, packet scheduling, Markovchain, queueing delay, delay-power tradeoff.

I. I NTRODUCTION

Energy harvesting can provide renewable free energy supplyfor wireless communication networks. With the help of solarcells, thermoelectric and vibration absorption devices, and thelike, communication devices are able to gather energy fromsurrounding environments. Energy harvesting can also helpreduce carbon emission and environmental pollution, as wellas the consumption of traditional energy resources [1]–[3]. Inpractice, harvested energy arrives in small units at randomtimes and the storage battery usually has limited capacity [4].Hence, wireless communication systems exclusively poweredby energy harvesting devices may not guarantee the users’quality of service. To provide dependable communication ser-vice, reliable energy resources can serve as backup in the caseof energy shortage. In this way, efficient mixed usage of theharvested energy and reliable energy provides a key solutionto robust wireless green communications [5], an emerging areaof critical importance to future wireless development.

J. Liu and H. Dai are with the Department of Electrical and ComputerEngineering, NC State University, Raleigh, NC 27695 (Email: hdai@ncsu.edu,jliu23@ncsu.edu). J. Liu and W. Chen are with State Key Laboratoryon Microwave and Digital Communications, Tsinghua National Labora-tory for Information Science and Technology (TNList) and Department ofElectronic Engineering, Tsinghua University, Beijing, China (e-mail: eeliu-juan@tsinghua.edu.cn, wchen@tsinghua.edu.cn).

In wireless networks, energy efficient transmission hasbeen an ever-present important issue [6]–[8]. Subject to therandomness and causality of energy harvesting, the optimaltransmission problem has been investigated for an energyharvesting wireless link with batteries of either finite or infinitecapacity in [4], [9], [10]. In these works, the authors assumedthat the energy harvesting profile (i.e., the arrival times andassociated amount of harvested energy) is known before thetransmission starts. This line of work has been extended towireless fading channels [11], broadcast channels [12] andtwo-hop networks [13].

Some other recent works have focused on developing ef-ficient transmission and resource allocation algorithms withdifferent objectives and energy recharging models. For ex-ample, a save-then-transmit protocol was proposed in [14] tominimize the delay constrained outage probability by usingtwo alternating batteries, where the battery charging rateismodeled as a random variable. In [15], a cross-layer resourceallocation problem was studied for wireless networks poweredby rechargeable batteries, where the amount of replenishedenergy is assumed to be independent and identically distributedin each time slot. In [16], an optimal energy allocation problemwas studied for a wireless link with time varying channelconditions and energy sources. A line of work pertinent toour study focuses on the queueing performance analysis foroptimal energy management policies. In particular, differentsleep/wake-up strategies in a solar-powered wireless sensornetwork were studied in [17]. Energy management policieswere proposed in [18] to maximize the stable throughput andminimize the mean delay for energy harvesting sensor nodes.

While a node can harvest an infinite amount of energyin the long run, harvested energy actually arrives at randomtimes. Due to the energy causality constraint, the node shouldaccumulate a sufficient amount of energy before each packettransmission. Hence, the waiting time could be undesirablylong and some packets might be dropped due to delay vi-olation. Intuitively, this situation can be greatly relieved ifone Reliable Energy Source (RES) can be used to transmitbacklogged packets when needed. At the other extreme, theproblem becomes trivial if the system can always transmit us-ing the reliable energy. Hence, there exists a tradeoff betweenthe packet delivery delay and the energy consumption fromthe reliable source.

In this paper, we investigate the delay optimal schedulingpolicy for a communication link powered by a capacity-limitedbattery storing harvested energy and one RES. In our system,the source will first seek energy supply from the capacity lim-ited energy harvesting battery whenever available, and resort to

2

data queue

switch

control

packet scheduling

´1

reliable energy source

q1

battery virtual queue

´2

data queue and

battery storage status

q2

Q2

solar panel

Q1

Fig. 1. System model.

the RES when necessary, but with an average power constraint.In particular, subject to the bursty energy harvesting profile, ittransmits with one of the energy supplies according to the dataqueue status and the energy storage status at the battery. Underthe constraint of the average power consumption from theRES, we study the delay optimal scheduling problem, takinginto account the match and mismatch between the energyharvesting capabilities and data packet arrival.

To analyze the proposed scheme, we formulate a two-dimensional discrete-time Markov chain and derive the steady-state probabilities. Based on the Markov chain modeling, wecan derive the average delay and the average power consumedfrom the RES as functions of the steady-state probabilities.Then, by formulating a Linear Programming (LP) problemand analyzing its properties, we are able to characterize thestructure of the optimal solution. Moreover, we can obtain anelegant closed-form expression for the optimal solution inthecase where each unit of harvested energy can support one datapacket transmission. We also develop an algorithm to find theoptimal solutions in other cases. From the optimal solution,we can determine the optimal probabilistic transmission pa-rameters. It is found that in the face of a depleted battery, theoptimal transmission strategy depends on a critical thresholdfor the data queue length. In particular, the source relies onthe harvested energy supply if the data queue length is belowthe critical threshold, and resorts to the RES otherwise. Ourtheoretical analysis is verified by simulations.

The rest of this paper is organized as follows. Section IIintroduces the system model and the stochastic schedulingscheme. In Section III, a two-dimensional Markov chain modelis constructed for the data and energy packet queueing system.Section IV formulates an LP problem for our schedulingobjective. By analyzing the properties of the LP problem,we derive the optimal steady-state probabilities and thendetermine the optimal transmission parameters in Section V.Section VI demonstrates the simulation results and SectionVIIconcludes this paper. For better illustration and in the interestof space, most proofs for our results are put in the appendices.

II. SYSTEM MODEL

A. System Description

We consider a communication link which is powered mainlyby a battery storing the harvested energy and further by theRES when necessary, as shown in Fig. 1. The RES refers toany reliable energy source, either traditional (such as powergrid) or newly developed. The source node (e.g. base station)employs a buffer to store the backlogged packets randomlygenerated from higher-layer applications. Suppose that the datapackets arrive at the source buffer according to a Bernoulliarrival process [19] with probabilityη1. This simple yet widelyadopted traffic model allows tractable analysis, and providesinsights for further study. The system is assumed to be time-slotted, and at the beginning instant of each slot,k1 ∈ N

data packets arrive at the data queue with capacityQ1. In thiswork, Q1 is treated as sufficiently large (so no data overflowwill incur) and fixed. Letq1[t] ∈ Q1 = 0, 1, 2, · · · , Q1 bethe length of the data queue at the end of slott, updated as

q1[t] = minq1[t− 1] + a1[t]− v1[t], Q1, (1)

wherea1[t] ∈ k1, 0 and v1[t] ∈ 1, 0 denote the numberof data packets arriving and served in each time slott,respectively. Without loss of generality, it is assumed thatat most one packet is transmitted in each slot due to thecapacity limitation of the communication link. Extension tomulti-packet transmission will be considered in future work.

The harvested energy is generally sporadically and ran-domly available, and we adopt a probabilistic energy harvest-ing model similar to [20]. Assume thates Joule harvestedenergy arrives at the beginning of a time slot with proba-bility η2, which can be used to transmitk2 packets. That ises = k2es, where es (Joule) denotes the amount of energyneeded for transmission of one data packet, andk2(≥ 1)is rounded down to the nearest integer. We will considerseveral interesting combinations ofk1 and k2 in this study,and leave the casek2 < 1 to future study. The harvestedenergy is stored in the battery with the maximum capacityE Joule, and discarded when the battery is full. The batterystorage is modeled as an energy queue with a finite capacityQ2 = ⌊E/es⌋, where one unit of transmission energyesis viewed as one energy packet. Leta2[t] and v2[t] be thenumber of energy packets received and consumed in each slott, respectively. At the end of time slott, the length of theenergy queueq2[t] ∈ Q2 = 0, 1, 2, · · · , Q2 is updated as

q2[t] = minq2[t− 1] + a2[t]− v2[t], Q2. (2)

It is assumed that the packet and energy arrival processes areindependent, and the newly harvested energy can be used fordata transmission in the same slot. For notational convenience,we set q[t] = (q1[t], q2[t]) to be the buffer status in thetime slot t. Similarly, the arrival and service processes canbe characterized by the vectorsa[t] = (a1[t], a2[t]) andv[t] = (v1[t], v2[t]), respectively.

B. Stochastic Scheduling

As we mentioned above, the source node is encouraged toexploit the harvested energy whenever available, and resort to

3

the backup RES when necessary. To this end, the source shouldalways transmit using the energy stored in the battery or newlyarriving energy packet when possible, which corresponds tothe caseq2[t − 1] > 0 or a2[t] > 0. When the harvestedenergy is not available,i.e., q2[t − 1] = a2[t] = 0, thesource schedules the transmission of data packets with theRES energy according to the data queue statusq1[t−1] and thedata packet arrival statusa1[t]. For generality, we define twosets of parameters:gi andfi in our scheduling scheme.In particular, withq1[t − 1] = i, if there is new data packetarrival in this slot,i.e., a1[t] > 0, the source node transmitsone data packet with probabilitygi with the RES energy andholds from transmission with probability1− gi, respectively;If no new data packet arrives,i.e., a1[t] = 0, it transmits withprobability fi and holds with probability1− fi, respectively.As discussed later, these parametersgi and fi shall beoptimized to achieve the minimum average queueing delay indifferent cases.

According to the proposed scheduling policy, the serviceprocessv[t] depends on the queue statusq[t − 1] and thearrival processa[t], as described below.

1) Case 1:q[t− 1] = (0, j) (j > 0)In this case, the source can transmit a newly arrivingdata packet using the harvested energy from the batteryin the current time slott, and the service process can beexpressed as

v[t] =

(1, 1) w.p.1, a[t] = (k1, ·),

(0, 0) a[t] = (0, ·),(3)

wherew.p. means ’with the probability’. The notationof (k1, ·) is used to denote both(k1, k2) and (k1, 0).

2) Case 2:q[t− 1] = (i, j) (i > 0, j > 0)In this case, the source can transmit a backloggedpacket with the harvested energy. The service processis expressed as

v[t] = (1, 1) w.p.1, a[t] = (·, ·). (4)

3) Case 3:q[t− 1] = (0, 0)In this case, when both data and energy packets arrive,the source will transmit with the energy harvested;When new data packets arrive in the absence of energyharvesting, the source shall use the energy from theRES to transmit with probabilityg0. Hence, the serviceprocess can be expressed as

v[t] =

(1, 1) w.p. 1 a[t] = (k1, k2)

(1, 0) w.p. g0 a[t] = (k1, 0)

(0, 0) otherwise.

(5)

4) Case 4:q[t− 1] = (i, 0) (i > 0)In this case, the source will transmit definitely usingthe harvested energy if it is available in the currentslot t. Otherwise, it will transmit using the RES energywith probability gi if a1[t] = k1 and with probabilityfi if a1[t] = 0, respectively. The service process is

characterized as

v[t] =

(1, 1) w.p. 1, a[t] = (·, k2)

(1, 0) w.p. gi a[t] = (k1, 0)

(1, 0) w.p. fi a[t] = (0, 0).

(6)

The above four cases include all possible scenarios.

C. Average Delay and Power Consumption

In a queueing system, the average queuing delay is an im-portant metric [21]. From the above description, the queueingsystem can be modeled as a discrete-time Markov chain, whereeach state represents the buffer status. Let(i, j) be the statethat the data queue length isi and the energy queue length isj, andπ(i,j) denote the steady-state probability of state(i, j).By the Little’s law, the average queueing delay is related tothe average buffer occupancy, and can be computed as

D =1

k1η1

∑Q1

i=1iπi =

1

k1η1

∑Q1

i=1i∑Q2

j=0π(i,j), (7)

whereπi =∑Q2

j=0 π(i,j) (i, j ≥ 0).The average transmission power is also an important perfor-

mance metric in wireless green communication systems. In thiswork, we focus on the average power consumption from theRES. Denote byc[t] the power consumed in thetth time slot. Ifthe source transmits using the energy from the RES in time slott, c[t] = p := 1

Tses, whereTs denotes the transmission time.

Otherwise,c[t] = 0. As will discussed below, the source drawsone energy packet from the RES depending on the currentqueueing statusq[t]. Let ωq(x) = Prc[t] = x|q[t] denotethe probability that the power consumptionc[t] is equal tox(x ∈ 0, p) conditioned on the queue stateq[t]. Using the lawof total probability, we obtain the normalized average powerconsumption (with respect top) as

P =∑

q∈Qp

πq · ωq(p), (8)

whereQp is the set of states conditioned on which the sourcemay draw the RES energy to transmit one data packet. Thisnormalized quantity can be interpreted as the proportion ofthe number of time slots in which the source transmits usingthe power from the RES. From (7) and (8), both the averagequeueing delay and power consumption are functions of thesteady-state probabilities. In this work, we aim to study thedelay optimal scheduling policy which minimizesD subjectto the average power constraintP ≤ pmax by determiningthe optimal transmission parametersg∗i andf∗

i . As a keystep, we will develop two-dimensional Markov chain modelsfor different combinations ofk1 andk2 in the next section.

III. T WO-DIMENSIONAL MARKOV CHAIN MODELING

To analyze the proposed scheduling scheme, we formulate atwo-dimensional discrete-time Markov chain for the queueingsystem, as shown in Fig. 2.

1The subfigure Fig.2(a) is intended for the general case ofk1 ≥ 1 andk2 ≥ 1 (so the dashed lines are used for transitions); butk1 = k2 = 2 canbe assumed when checking the transition probabilities given in Section III.

4

¹0 1;Q

1+

¹3

(1,0) (0,1)(Q1-1,0)(Q1,0) (0,Q2-1) (0,Q2)

¸1;Q1¡2¸1;Q1¡1

~¹1;Q1

´1

¹0¹0 ¹0

¹2 ¹2

¸0

1;0 + ¹1 + ¹3 ¹1 + ¹3 ¹1 + ¹3

(0,0)

(1,0)

(2,0)

(0,1) (0,2) (0,Q2-1) (0,Q2)

¸1;0 ~¹1;1

¹2

¹0

~¹1;2¸1;1

(Q1-1,0)

(Q1,0)

¸1;2 ~¹1;3

¸1;Q1¡2

~¹1;Q1¡1

¸1;Q1¡1 ~¹1;Q1

(1,1) (1,2) (1,Q2-1)

(2,1) (2,2) (2,Q2-1)

(Q1-1,1) (Q1-1,2)(Q1-1,

Q2-1)

(Q1,1) (Q1,2)(Q1,

Q2-1)

¹0¹1

¹2

¹0

¹0¹1

¹2

¹0

¹0¹1

¹2

¹0

¹0

¹2

¹0

¹0

¹2

¹0

¹0

¹0

¹0

¹0

¹0

(1,Q2)

(2,Q2)

(Q1-1,

Q2)

(Q1,Q2)

¹1

¹1 ¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹1

¹2

¹0

¹2

¹0

¹2

¹0

´1

¹0

´1

¹1

´1

¹1

´1

¹1

´1

¹1

´1

¹1

¹1 + ¹3

¹3

¹3

¹3¹3

¹3 ¹3

¹3 ¹3

¹3 ¹3

¹3

¹3

¸0

1;0 + ¹1 + ¹3 ¹1 + ¹3 ¹1¹1 + ¹3¸0

1;1 + ¹0

1;1 + ¹3¸

0

1;Q1¡1+ ¹

0

1;Q1¡1+ ¹3

¹0

1;Q1+ ¹3

¸0 1;1

+¹

0 1;1

+¹

3¸

0 1;2

+¹

0 1;2

+¹

3¸

0 1;Q

1¡

1+

¹0 1;Q

1¡

1+

¹3

¢¢¢

¸1;0

~¹1;1

¸0

1;0 + ¹1

(0,0)

(1,0)

(2,0)

(0,1) (0,2) (0,Q2-1) (0,Q2)

(Q1-

1,0)

(Q1,0)

(1,1) (1,2) (1,Q2)(1,Q2-1)

(2,1) (2,2) (2,Q2)(2,Q2-1)

(Q1-

1,1)

(Q1-

1,2)(Q1-

1,Q2)

(Q1-1,

Q2-1)

(Q1,1) (Q1,2) (Q1,Q2)(Q1,Q2-

1)

¹1¹1

¹2

¹1

¹1¹1

¹1

¹1

¹1

¹1

´1¹2

¹3

¹2

¹0 ¹0

¹2¹2

¹3¹3

¹3

¹2

¹3

¹2

¹3

¹2

¹0 ¹1

¹2

¹3

¹3

¹2

¹3

¹2

¹1

¹3

¹2

¹3

¹2

¹3

¹2

¹3

¹2

¹3

¹2

¹3

¹2

¹0 ¹0

¹1 ¹1 ¹1

¹1¹0¹0 ¹0

¹0

¹0

¸1;0 ¹1;1

¹0

¹0

¹1;2¸1;1

¹0

¹1¹0

¸1;2

¸1;Q1¡2

¹1;Q1¡1

¸1;Q1¡1¹1;Q1

¹0¹0

¹0

¹0

¹3

¹3¹3 ¹3 ¹3

¹0

¹0

¹1;3

¹0

¹0

¹0

¹0

¹2 ¹2 ¹2

¹1

¹3

¹2

¹3

¹2

¹3

¹2

¹3

¹2

¹1¹1 ¹1

¹1¹0¹0

¹0¹0

¹0

´1

¹1

´1

¹1

´1

¹1

´1

¹1

´1

¹1

¹3

¹3

¹3

¹3

¹3

¹1¹1

¸0 1;1

+¹

0 1;1

¸0 1;2

+¹

0 1;2

¸0 1;Q

1¡

1+

¹0 1;Q

1¡

1¢¢¢

¹0 1;Q

1¡

1

¹1 + ¹3

¹0

¹0

1;Q1

¹0

1;Q1¡1

¹0

1;1

¹0

1;2

¹0

1;3

(1,0)

(Q1-1,0)

(Q1,0)

~¹1;2~

1;1

~1;2

~1;Q1¡2

~¹1;Q1¡1

~1;Q1¡1

~¹1;Q1

~¹1;3

¸1;1

¸1;2

¸1;Q1¡2

(1,1) (1,2) (1,Q2-1) (1,Q2)

(Q1-1,1) (Q1-1,2)(Q1-1,

Q2-1)

(Q1-1,

Q2)

(Q1,1) (Q1,2)(Q1,

Q2-1)(Q1,Q2)

¹2 ¹2¹1 ¹1

¹0¹3

¹2

¹0¹3 ¹0

¹3 ¹0¹3

¹2

¹2 ¹2¹1 ¹1

¹0¹3 ¹2

¹0¹3 ¹0

¹3 ¹0¹3

¹2

¹2 ¹2¹1 ¹1

¹0¹3 ¹2

¹0¹3 ¹0

¹3 ¹0¹3

¹2

¹2 ¹2¹1 ¹1

¹0¹3 ¹2

¹0¹3 ¹0

¹3 ¹0¹3

¹2

¹1

¹1

¹1

¹1

¹1

¹1

¹1

¹1

(0,0) (0,1) (0,2) (0,Q2-1) (0,Q2)

~1;0 ~¹1;1

¹0

¹2 ¹2¹1 ¹1 ´1 ¹1

¹0¹3 ¹2

¹0¹3 ¹0

¹3 ¹0¹3

¹2

¸1;0

(2,0) (2,1) (2,2) (2,Q2-1) (2,Q2)

¸1;Q1¡1

¹0 ¹0¹0 ¹0

´1 ¹1

´1 ¹1

´1 ¹1

´1 ¹1

¹1¹1 ¹1 ¹1¹1¹1

(1,0)

(Q1-1,0)

(Q1,0)

¹1;2¸0

1;1

¸0

1;2

¸0

1;Q1¡2

¹1;Q1¡1

¸0

1;Q1¡1

¹1;Q1

¹1;3

¸1;1

¸1;2

1;Q1¡2

(1,1) (1,2) (1,Q2-1) (1,Q2)

(Q1-1,1) (Q1-1,2)(Q1-1,

Q2-1)

(Q1-1,

Q2)

(Q1,1) (Q1,2)(Q1,

Q2-1)(Q1,Q2)

(0,0) (0,1) (0,2) (0,Q2-1) (0,Q2)

¸0

1;0¹1;1

¸1;0

(2,0) (2,1) (2,2) (2,Q2-1) (2,Q2)

¸1;Q1¡1

¹0 ¹0¹0 ¹0

¹0

¹1

¹3 ¹3

¹1

¹3

¹1

¹3 ¹3¹2

¹0

¹1 ¹1

¹2

¹0

¹2

¹0

¹2

¹0 ¹0

¹1´1

¹3 ¹3

¹1

¹3

¹1

¹3 ¹3¹2

¹0

¹1 ¹1

¹2

¹0

¹2

¹0

¹2

¹0 ¹0

¹1´1

¹3 ¹3

¹1

¹3

¹1

¹3 ¹3¹2

¹0

¹1 ¹1

¹2

¹0

¹2

¹0

¹2

¹0 ¹0

¹1´1

¹3¹3

¹1

¹3

¹1

¹3 ¹3¹2

¹0

¹1¹1

¹2

¹0

¹2

¹0

¹2

¹0 ¹0

¹1´1

¹3 ¹3

¹1

¹3

¹1

¹3 ¹3¹2

¹0

¹1 ¹1

¹2

¹0

¹2

¹0

¹2

¹0 ¹0

¹1´1

¹1 ¹1 ¹1¹1¹1

¹0

1;1

¹0

1;2

¹0

1;3

¹0

1;Q1¡1

¹0

1;Q1

(a) Two-dimensional discrete-time Markov chain for the general case

(b) Two-dimensional discrete-time Markov chain for Case I with k1 = k2 = 1

(c) Trimmed Markov chain for Case I with k1 = k2 = 1

k1 = k2 = 1

(d) Two-dimensional discrete-time Markov chain for Case II with k1 = 1; k2 > 1

(e) Two-dimensional discrete-time Markov chain for Case III with k1 > 1; k2 = 1

k1 = 1

k2 = 1

(0,0)

¹0

¹2

~¹1;2

¸1;1

~¹1;Q1¡1

Fig. 2. Two-dimensional discrete-time Markov chain1.

Let Prq[t + 1]|q[t] denote the one-step transition prob-ability of the Markov chain, which is homogeneous by thescheme description. For ease of expression, we define fourconstants as

µ0 = (1− η1)η2, µ1 = (1− η1)(1− η2),

µ2 = η1(1− η2), µ3 = η1η2.(9)

We further define two subsets ofQi as:QLi = 0, · · · , Qi−1,

QRi = 1, · · · , Qi, and setηi = 1− ηi, for i = 1, 2.We now describe the one-step transition probabilities in

Fig. 2(a) in detail, by grouping them into several types. Westart with the four transitions among each square unit, for ex-ample, those among(2, 1), (2, 2), (1, 2) and(1, 1) in Fig. 2(a).First, let us examine the transition from(2, 1) to (1, 2), moregenerally, from(i, j) to (i − 1,minj + k2, Q2 − 1). Thiscorresponds to the case that there is no data but energy packetarrival, and one backlogged data packet is delivered, so clearlythe corresponding probability isµ0. When neither data norenergy packets arrive, one data packet stored in the buffercan be transmitted using one energy packet from the battery

5

if there exists. In this case, the state will transfer from(i, j)to (i − 1, j − 1) (e.g., from(2, 2) to (1, 1) in Fig. 2(a)) withprobability µ1 for all i > 0 andQ2 > j > 0. Whenk1 datapackets arrive while no energy is harvested, one data packetwill be transmitted using one energy packet if there is energystored in the battery. That is, the state will transfer from(i, j)to (i + k1 − 1, j − 1) (e.g., from(1, 2) to (2, 1) in Fig. 2(a))with probabilityµ2 for all Q2 > j > 0. When data and energypackets arrive simultaneously, one data packet is transmittedusing one energy packet. In this case, the state will transferfrom (i, j) to (i + k1 − 1,minj + k2, Q2 − 1) (e.g., from(1, 1) to (2, 2) in Fig. 2(a)) with probabilityµ3 for j ∈ QL

2 .The casej = Q2 requires special treatment, as the battery

is full and the newly harvested energy has to be discardedanyway. With the capacity limit in mind, we havePr(i −1, Q2−1)|(i, Q2) = η1 for i > 0, Pr(0, Q2)|(0, Q2) = η1,andPr(i+ k1 − 1, Q2 − 1)|(i, Q2) = η1 for all i.

We then consider the first row in Fig. 2(a). When nodata packets arrive andk2 energy packets newly arrive, thestate (0, j) will transfer to (0,minj + k2, Q2) with thecorresponding transition probabilityµ0 for j ∈ QL

2 . Wehave mentioned thatPr(0, Q2)|(0, Q2) = η1 is due to thecapacity limitation of the battery. The state(0, j) remainsthe same with probabilityµ1 (when neither data nor energypackets arrive).

We now focus our attention on the group of transitionprobabilities on the first column of Fig. 2,λ1,i andλ

′

1,i

(i ∈ QL1 ), µ1,i andµ

′

1,i (i ∈ QR1 ), which corresponds to

the case that there is no storage of harvested energy in thebattery, and can be obtained as

λ1,i = Pr(i+ k1, 0)|(i, 0) = µ2(1− gi) (i ∈ QL1 ),

λ′

1,i = Pr(i+ k1 − 1, 0)|(i, 0) = µ2gi (i ∈ QL1 ),

µ1,i = Pr(i− 1, 0)|(i, 0) = µ1fi (i ∈ QR1 ),

µ′

1,i = Pr(i, 0)|(i, 0) = µ1(1− fi) (i ∈ QR1 ).

(10)

In particular, whenk1 data packets arrive while no energy isharvested (which happens with probabilityµ2), λ1,i andλ

′

1,i

denote the transition probabilities from state(i, 0) to (i+k1, 0)and(i+k1−1, 0), respectively, depending on whether one datapacket is delivered with the reliable energy in this slot (withprobability gi). When neither data nor energy packets arrive(which happens with probabilityµ1), µ1,i and µ

′

1,i denotethe transition probabilities from state(i, 0) to (i − 1, 0) and(i, 0), respectively, depending on whether one data packet istransmitted using the reliable energy (with probabilityfi).

We order theN = (1 +Q1)(1 + Q2) states as(0, 0), · · · ,(0, Q2), (1, 0), · · · , (1, Q2), · · · (Q1, 0), · · · , (Q1, Q2), and letP denote theN×N transition matrix. We denote byπ the1×N column vector containing steady-state probabilities, andbye theN ×1 column vector with all the elements equal to one.For notational convenience, we also define two sub-vectors ofπ as: πi = [π(i,0); · · · ;π(i,Q2)] and πi = [π0; · · · ;πi], anddenote byei a 1 × (i + 1)(1 + Q2) row vector with all theelements equal to one. Given a set of parametersgi andfi, the steady-state probabilitiesπ(i,j) can be obtained bysolving the linear equationsπP = π andπe = 1. Note thatthe transmission parametersgi andfi only influence the

transition probabilities from the states(i, 0), i ∈ Q1. We thusconsiderPs, a submatrix ofP = P − I, to exclude the statetransition starting from states(i, 0). In this way,πPs = 0

present the local balance equations at the states(i, j) (i ≥0, j > 0). For ease of expression, we also denote byPi theleft-top submatrix of(i+ 1)Q2 dimensions fromPs.

In the general case withk1 ≥ 1 and k2 ≥ 1, thecorresponding Markov chain seems not amenable to analysis.In this paper, we mainly focus on three cases: Case I withk1 = 1 and k2 = 1, Case II withk1 = 1 and k2 > 1, andCase III with k1 > 1 and k2 = 1, respectively. These threeexemplary cases nonetheless capture some important relationsbetween the data and energy arrival processes, and serve asthe basis for further extensions. In the following, we illustratethe Markov chain for each of the three cases.

A. Case I:k1 = 1 and k2 = 1

In this case, one data packet and one energy packet arrivein each slot with probabilitiesη1 and η2, respectively. Ac-cordingly, the simplified Markov chain is shown in Fig. 2(b).Essentially all expressions in the general case carry over withthe substitution ofk1 = k2 = 1. For example, the transitionfrom (i, j) to (i−1,minj+k2, Q2−1) in Fig. 2(a) becomesthat from(i, j) to (i−1, j) in Fig. 2(b), again with probabilityµ0. This applies to the states in the first column as well, andas a result, a new notation is needed for the transition from(i, 0) to (i− 1, 0), which combinesµ0 and the previousµ1,i:

µ1,i = Pr(i− 1, 0)|(i, 0) = µ0 + µ1fi (11)

for all i ∈ QR1 . Also, it is worth noting that in the dashed

square, neither queue length can ever increase regardless ofthe arrival processes, as one data packet transmission happensfor sure. As a result, the statesq[t] with q1[t] · q2[t] > 0 aretransient in the following lemma.

Lemma 1. In Case I withk1 = k2 = 1 when η1 < 1 orη2 < 1, the queue status satisfyingq1[t] ·q2[t] > 0 is transient.

Proof: Let f (n)q denote the probability that the queue state

q[t] will return to itself for the first time aftern steps. Asshown in Fig. 2(b), wheni · j > 0, f (1)

q = Pr(i, j)|(i, j) =

µ3 and f(n)q = 0 for n > 1. Hence,

∑∞

n=1 f(n)q = µ3 =

η1η2 < 1, whenη1 < 1 or η2 < 1. From [22], the stateq[t]with q1[t] · q2[t] > 0 is a transient state.

This implies that either the data queue or the energyqueue will be exhausted, even if they are not empty initially.Hence, when calculating the steady-state probabilitiesπ(i,j),the two-dimensional Markov chain can be reduced to the one-dimensional one, as plotted in Fig. 2(c), which consists of thestates(i, 0) and (0, j) for all i ∈ Q1 andj ∈ Q2.

B. Case II and Case III

In Case II,k2 energy packets (k2 > 1) arrive at the batterywith probability η2 per slot. Hence, the length of the energyqueue may increase byk2 or k2 − 1 (when one energy packetis consumed in the current slot) each time. The resulting two-dimensional Markov chain is shown in Fig. 2(d). In Case III,

6

k1 > 1 data packets arrive with the probabilityη1 at eachslot, and the two-dimensional Markov chain is illustrated inFig. 2(d), where the data queue length could increase byk1 ork1 − 1 (when one data packet is transmitted using an energypacket harvested or drawn from the RES in the current slot).

As shown in Fig. 2(d), the solid lines present the fixed statetransitions while the dotted lines indicate state transitions thatvary with differentk2. In particular, the state(i, j) transfersto (i− 1,minj+k2, Q2− 1) with the probabilityµ0 and to(i,minj+k2, Q2− 1) with the probabilityµ3, respectively.Similarly, the state(0, j) transfers to(0,minj+k2, Q2) withthe probabilityµ0, and to(0,minj + k2, Q2 − 1) with theprobabilityµ3, respectively. Note that the states(i, Q2) for alli > 0 are transient.

Similarly in Fig. 2(e), solid and dotted lines are used topresent the fixed state transitions and state transitions thatvary with different k1, respectively. Similar to Case I, thestate transfers from(i, 0) to (i − 1, 0) with the combinedtransition probabilityµ1,i = µ0 + µ1fi. For the same reason,the transition probability from(i, 0) to (i + k1 − 1, 0) is

λ1,i = µ3 + µ2gi = µ3 + λ′

1,i = η1 − λ1,i. (12)

And the states(i, Q2) for all i > 0 are transient.

IV. LP PROBLEM FORMULATION

As discussed above, both the average delay and power con-sumption from the RES are functions of the steady-stateprobabilities of the corresponding Markov chains, which inturn depend on the transmission parametersgi andfi tobe designed. To seek the optimal scheduling policy, we adopta two-step procedure [23]: first we formulate an LP problemonly depending on the steady-state probabilities, and obtainthe corresponding solution; then from the optimal solutionof the LP problem, we determine the optimal transmissionparameters.

Our objective is to minimize the average queueing delaysubject to the maximum average power constraint from theRES. The corresponding LP problem can be formulated as

min D =1

k1η1

∑

Q1

i=1i∑

Q2

j=0π(i,j)

s.t.

P =Q1∑

i=0

ξi · π(i,0) −Q1∑

i=0

ζi · π(i,1) ≤ pmax, (a)

Θl(i, πi−1) ≤Q2∑

j=0

π(i,j) ≤ Θu(i, πi)(i > 0), (b)

π(i,j) ≥ 0, (∀i, j), (c)∑Q1

i=0

∑Q2

j=0π(i,j) = 1, (d)

πPs = 0. (e)

(13)

From the properties of a Markov chain, the last three con-straints (c)-(e) are straightforward. The original definition ofP (c.f. (8)) in constraint (a) does depend on the transmissionparameters; to facilitate derivation, we will give a new ex-pression forP in Lemma 2 below that is only a functionof the steady-state probabilitiesπ(i,0) and π(i,1), i ∈ Q1.The influence of the transmission parameters on the problem

is encapsulated in the constraint (b), which represents therelationship between the steady-state probabilitiesπ(i,j) dueto the varying transmission parametersgi and fi, asdiscussed later in Lemma 3. The optimal solution to (13) isdenoted byπ∗

(i,j) and the minimum average delay byD∗.

Lemma 2. In Cases I, II and III, the normalized averagepower consumption from the RES can be expressed as

P =∑

Q1

i=0ξi · π(i,0) −∑

Q1

i=0ζi · π(i,1), (14)

where the coefficientsξi and ζi are presented in Table I.

Proof: The proof is deferred to Appendix A.Remark: By exploiting the local balance equations of states

(i, 0) (i ∈ QL1 ), we can replace all the itemsπ(i,0)µ2gi (i ∈

Q1) andπ(i,0)µ1fi (i ∈ QR1 ) of P with the itemsξiπ(i,0) and

ζiπ(i,1) (i ∈ Q1). In this way, the average power consumptionP becomes a linear function of the steady-state probabilitiesπ(i,0) and π(i,1). Thus, the direct dependence ofP on thetransmission parametersgi andfi is removed.

Then, we discuss the constraint (13.b). The basic idea isto vary the transmission parametersgi and fi in thefull range of [0, 1], so as to obtain an upper and lowerbound for eachπi. In this way, we transform the constraintson gi and fi into the relationship between the steady-state probabilities themselves, which allows us to obtain theoptimal solution to (13) in terms ofπ(i,j) first. For ease ofillustration, we define several constants asτ = η1

1−η1, φ = µ2

µ0,

andφ1 = η1

µ0. Let us define[x]+ = max0, x.

Lemma 3. In Cases I, II and III, the probabilityπi satisfies

Θl(i, πi−1) ≤ πi =∑

Q2

j=0π(i,j) ≤ Θu(i, πi) (i > 0), (15)

whereΘu(·) andΘl(·) are presented in Table II.

Proof: The proof is deferred to Appendix B.Remark: From the proof of Lemma 3, we haveπi =

Θu(i, πi) at gi−k1 = fi = 0, and πi = Θl(i, πi−1) atgi−k1 = fi = 1, respectively, in all the three cases2.This lies in the fact that the transmission parametersgiand fi determine the relationship between the steady-stateprobabilitiesπ(i,j), and vice versa. As listed in Table II,Θu(i, πi) is a linear function of the steady-state probabilitiesπ(i−k1,0), · · · , π(i−1,Q2), π(i,0), and Θl(i, πi−1) is a linearfunction of π([i−k1+1]+,0), · · · , π(i−1,Q2).

From Lemmas 2 and 3,P , Θu(i, πi) andΘl(i, πi−1) areall linear functions of the steady-state probabilitiesπ(i,j).Hence, we can represent them in the form ofP (π) = πa0,∑Q2

j=0 π(i,j) − Θu(i, πi) = πaui (i > 0), andΘl(i, πi−1) −

∑Q2

j=0 π(i,j) = πali (i > 0), wherea0, au

i andali areN × 1

column vectors collecting corresponding coefficients.

V. DELAY OPTIMAL SCHEDULING UNDER POWER

CONSTRAINT

In this section, we discuss the optimal solution to Problem(13) by studying its structure with respect to the steady-stateprobabilities of the corresponding Markov chains.

2More rigorously, in Case I,πi = Θl(i, πi−1) holds just whengi−1 = 1andfi can be arbitrary.

7

Table ITHE COEFFICIENTSξi AND ζi FOR CASES I, II, AND III.

Case I withk1 = k2 = 1 Case II withk1 = 1 andk2 > 1 Case III withk1 > 1 andk2 = 1

ξiξ0 = µ2

ξi = µ2 + η2(Q1 − i) (i ∈ Q1)

ξ0 = µ0Q1 − µ3 + k1η1

ξi = µ2 − µ0(i ∈ QR1 ) ξi = k1η1 − η2 (1 ≤ i ≤ Q1 − k1)

ξi = η1(Q1 − i) − η2 (Q1 − k1 + 1 ≤ i ≤ Q1)

ζiζi = 0 (i ∈ Q1)

ζ0 = µ2Q1 ζ0 = µ2(Q1 − k1 + 1)

ζi = η2(Q1 − i) + µ1 (i ∈ QR1 )

ζi = (µ1 + µ2)(Q1 + 1− i)− µ2k1 (1 ≤ i ≤ Q1 − k1)

ζi = µ1(Q1 + 1− i) (Q1 − k1 + 1 ≤ i ≤ Q1)

Table IIΘu(i, πi) AND Θl(i, πi−1) FOR CASES I, II, AND III.

Case I withk1 = k2 = 1 Case II withk1 = 1, k2 > 1 Case III withk1 > 1, k2 = 1

Θu(i, πi) φπ(i−1,0) τ η2π(i−1,0) + π(i,0)η2i < k1 π(i,0)η2 +

i−1∑

m=0

Q2∑

j=0τπ(m,j)

i ≥ k1 τ η2π(i−k1,0) + π(i,0)η2 +i−1∑

m=i−k1+1

Q2∑

j=0τπ(m,j)

Θl(i, πi−1) 0 0i−1∑

m=[i−k1+1]+

Q2∑

j=0τπ(m,j)

A. Structure of The Optimal Solution

For ease of discussion, we first consider a scheduling policystrictly based on the thresholdm: the source waits for theharvested energy when the number of backlogged data packetsis less than or equal toa certain thresholdm and transmitsusing the reliable energy when the data queue length exceedsm. According to the thresholdm, we usepm to measure theamount of power drawn from the RES. Sincepm is sufficientfor the application of the scheduling policy based on thethresholdm + 1, but not vise versa,pm is non-increasingwith the thresholdm. We will show that the threshold basedscheduling policy turns out to be the optimal and the optimalthreshold is determined by the power thresholdspm.

Theorem 4. The optimal threshold isi∗ = 0 whenpmax ≥ p0,and i∗ > 0 whenk1η1 − k2η2 < pmax < p0, respectively.

Proof: The proof is deferred to Appendix C.We notice that the average queueing delayD =1

k1η1

∑Q1

i=1 iπi is a weighted summation of the steady-stateprobabilities πi. Thus, D can be reduced, if we assign alarger value toπi with a smaller indexi and vice versa.Based on this intuition, we can reveal that the optimal solutionto the LP problem (13) corresponds to a threshold basedscheduling policy with the optimal thresholdi∗ determinedby the maximum allowable power consumption from the RESpmax.

Theorem 5. The optimal solutionπ∗ satisfies

π∗a0 ≤ pmax,

π∗aui = 0 (i = 1, · · · , i∗ − 1),

π∗ali = 0 (i = i∗ + 1, · · · , Q1),

(16)

where the optimal threshold is obtained as

i∗ = arg minpm≤pmax

m. (17)

Proof: The proof is deferred to Appendix D.Remark: According to Lemma 3, we haveπi = Θu(i, πi)

or πaui = 0 when gi−k1 = fi = 0, and πi = Θl(i, πi−1)

whengi−k1 = fi = 1, respectively. Therefore, associated with(16) is a threshold based scheduling policy that waits for theharvested energy when the number of backlogged data packetsis less thana certain thresholdi∗, and draws the reliable energydefinitely when the harvested energy is not available while thenumber of backlogged data packets exceeds the threshold (i∗

if there is no new data packet arrival, andi∗ − k1 if there isnew data packet arrival).

Note that the LP problem (13) has an optimal solution onlywhen the queueing system is stable,i.e., when the service rateis greater than the arrival rate, according to Loynes’s theorem[24]. Throughout this paper, the service rate is specialized asthe total amount of energy that can be drawn either fromthe RES or from the battery,pmax + k2η2. Hence, we willdiscuss the optimal solution to the LP problem (13) under theassumption thatpmax > k1η1 − k2η2.

B. The Optimal Solution

By exploiting the result in Theorem 5, we continue to derivethe optimal steady-state probabilities for Case I, and developan algorithm to obtain the optimal solutions for Case II andCase III, respectively.

1) Case I: In this case, the two-dimensional Markov chainis reduced to a one-dimensional one, where transitions takesplace only between adjacent states, as shown in Fig. 2(c).We only need to discuss the optimal steady-state probabilitiesπ∗(i,0) and π∗

(0,j) for all i ∈ Q1 and j ∈ Q2. In the sequel,we first show that the optimalπ∗

(0,j) is a function ofπ∗(0,0) in

Lemma 6 and then present the optimalπ∗(i,0) in Corollary 7.

Lemma 6. In Case I, the optimal steady-state probability

8

π∗(0,j) is related toπ∗

(0,0) as

π∗(0,j) =

π∗(0,0)φ

−j , 1 ≤ j ≤ Q2 − 1,

π∗(0,0)φ

−(Q2−1)φ−11 , j = Q2.

(18)

Proof: From the proof of Theorem 5, the optimal proba-bility π∗

(0,j) is a function ofπ∗(0,0), as given by (18).

From (18), we getπ∗0 =

∑Q2

j=0 π∗(0,j) = απ∗

(0,0), where

α =

Q2−1∑

i=0

φ−i + φ−(Q2−1)φ−11 =

(Q2 + φ−11 ), φ = 1,

φ1φQ2+φ−φ1−1

φQ2−1(φ−1)φ1, φ 6= 1.

From the results obtained in Theorem 5, we show that theoptimal π∗

(i,0) for all i > 0 are functions ofπ∗(0,0). Further,

taking advantage of the dependance ofP on π∗(0,0), we can

derive the closed-form optimal solutionπ∗(i,0) in Corollary 7.

Corollary 7. In Case I, whenpmax ≥ p0 = µ2α−1, we have

π∗(0,0) = α−1 and π∗

(i,0) = 0 for all i > 0, respectively. When

η1 − η2 < pmax < p0, π∗(0,0) = pmax−(µ2−µ0)

µ2−α(µ2−µ0), and π∗

(i,0)

(i > 0) is given by

π∗(i,0) =

π∗(0,0)φ

i, i ≤ i∗ − 1,

1− απ∗(0,0) − π∗

(0,0)

∑i∗−1i=1 φi, i = i∗,

0, i > i∗,

(19)

where the optimal thresholdi∗ is obtained as

i∗ = Ωφ(π∗(0,0), 1− απ∗

(0,0)) (20)

with the functionΩφ(a, b) defined as

Ωφ(a, b) := max

ai−1∑

m=1φm≤b

i =

⌊ ba⌋+ 1, φ = 1,

⌊logφ(a+b)φ−b

a⌋, φ 6= 1.

Proof: The proof is deferred to Appendix E.From Eqs. (18), (19) and (20), one can see that the optimal

steady-state probabilitiesπ∗(i,0) and π∗

(0,j), and the optimalthresholdi∗ are solely determined by the maximum averagepower pmax for given η1, η2 and Q2. We also show thatπ∗(i,0) = 0 for all i > i∗. This indicates that the length of

the packet queue never exceeds the thresholdi∗. Hence, nopacket loss will be induced as long as the queue capacityQ1

is larger thani∗.2) Case II and Case III:In Case II withk1 = 1 andk2 > 1

and Case III withk1 > 1 andk2 = 1, it is challenging to derivea closed-form optimal solution to the LP problem (13). Basedon the result in Theorem 5, we then develop an algorithm tofind the optimal solutions for these two cases.

In Theorem 5, we show that the optimal solution cor-responds to the threshold based transmission scheme. Theoptimal threshold can be determined by comparing the powerconstraintpmax to the power thresholdspm (m ≥ 0). Inparticular,pm can be computed as

pm = π′

ma0, (21)

Algorithm 1 Finding the optimal solution for Cases II and III.1: Initialization: setQ1 to be a large constant.2: if pmax ≤ k1η1 − k2η2 then3: The optimal solution and parameters do not exist.4: else5: Computeπ

′

0 = b′

(A′

0)−1 and p0 = π

′

0a0.6: if pmax ≥ p0 then7: Setπ∗ = π

′

0.8: else9: Setm = 1.

10: repeat11: Computeπ

′

m = b′

(A′

m)−1 and pm = π′

ma0.12: if pm ≤ pmax < pm−1 then13: Set i∗ = m. Computeπ∗ = bA−1. Exit.14: end if15: until m > Q1

16: Setπ∗ = π′

Q1, and seti∗ = ∞.

17: end if18: end if

whereπ′

m is the solution to the following linear equations

πaui = 0 (i = 1, · · · ,m),

πali = 0 (i = m+ 1, · · · , Q1),

πPs = 0,

πe = 1.

(22)

Let b′

= [0, · · · , 0, 1] be a 1 × N row vector, andA′

m =[au

1 , · · · ,aum,al

m+1, · · · ,alQ1

, Ps, e] be anN×N matrix. Thesolution to (22) can be expressed asπ

′

m = b′

(A′

m)−1. Thus,the power threshold is equal topm = π

′

ma0 = b′

(A′

m)−1.Once obtaining the power thresholdspm, we can compute

the optimal solutionπ∗ as follows.

Corollary 8. The optimal solution to (13) for Cases II andIII can be computed as

π∗ =

π′

0, if pmax ≥ p0,

bA−1, if pi∗ ≤ pmax < pi∗−1,(23)

whereA = [a0,au1 , · · · ,a

ui∗−1,a

li∗+1, · · · ,a

lQ1

, Ps, e] is anN×N matrix, andb = [pmax, · · · , 0, 1] is a1×N row vector.

Proof: The proof is deferred to Appendix F.Remark: By exploiting the structure of the optimal solution,

we can compute the optimal solutionπ∗ by solving (1 +Q1)(1 +Q2) independent linear equations. Based on the def-inition of the power thresholdspm, π∗ can be alternativelyobtained by solving linear equations (22) whenpmax = pi∗ ,i.e., π∗ = π

′

m. In Case II, sinceπi = Θl(i, πi−1) = 0, wehaveπ∗

(i,j) = 0 for all i > i∗. And π∗i∗ can be obtained by

solving (1 + i∗)(1 + Q2) linear equations:πi∗a0 = pmax,πi∗a

ui = 0 (i = 1, · · · , i∗ − 1), πi∗Pi∗ = 0 and πi∗ei∗ = 1.

Based on the above discussion, we develop an algorithm,i.e., Algorithm 1, to show how to find the optimal solutionπ∗(i,j) to the LP problem for Case II and Case III. The optimal

thresholdi∗ is sought iteratively by comparing the maximumallowable power consumptionpmax with the power thresholds

9

Table IIITHE OPTIMAL TRANSMISSION PARAMETERSg∗i AND f∗

i FOR CASES I, II, AND III.

Case I withk1 = k2 = 1 Case II withk1 = 1 andk2 > 1 Case III withk1 > 1 andk2 = 1

0 ≤ i < i∗ − k1 0

g∗i i = i∗ − k1 1−1−απ∗

(0,0)−π∗

(0,0)

∑i∗−1i=1 φi

π∗

(0,0)φi∗

1−π∗

(i∗,0)µ0+η1∑Q2

j=1 π∗

(i∗,j)

µ2π∗

(i∗−1,0)1−

η1π∗

i∗−π∗

(i∗,0)µ1−η1

i∗−1∑

m=i∗−k1+1

π∗

m

µ2π∗

(i∗−k1,0)

i > i∗ − k1 1

Case I and Case II Case III withk1 > 1 andk2 = 1

0 < i < i∗

0

0

f∗

i i = i∗0 (i∗ ≥ k1)

η1

i∗−1∑

m=0π∗

m−η1π∗

i∗+π∗

(i∗,0)µ1

π∗

(i∗,0)µ1

(i∗ < k1)

i > i∗ 1

pm. Once locating the thresholdi∗, the optimal steady-stateprobabilitiesπ∗

(i,j) can be obtained by solving an LP problem.There are two exceptions: (1) whenpmax ≤ k1η1 − k2η2,the queueing system is not stable and the optimal solutiondoes not exist; and (2) when the iteration number exceeds thesufficiently large data queue lengthQ1, we regard the optimalthreshold i∗ as the infinity. GivenQ1, the algorithm runsat mostQ1 iterations, and in each iteration the computationcomplexity of solvingN linear equations isO(N3). Hence,the computation complexity of this algorithm can be roughlyestimated asO(Q1(1+Q1)

3(1+Q2)3). For a relatively small

Q2, the complexity can be approximated asO(Q41).

By comparison of Case I and Case II, we notice thatchanging the number of energy packets arriving each time,k2, does not change the property of the optimal results. Fromthis perspective, it is feasible to deal with the case whenk1 > 1 and k2 > 1 using the same method as in Case III.Firstly, we formulate a concrete Markov chain for a pair ofsuch k1 and k2 and find the mutual relations between thestates. Secondly, we construct an LP problem under the powerconsumption constraint, which manifests as the constant linearcombination of the steady-state probabilities. Finally, we canadopt Algorithm 1 to find the optimal solution and the optimaltransmission parameters.

C. The Optimal Transmission Parameters

By exploiting the local equilibrium equations and the corre-sponding optimal solutionπ∗, we then obtain the optimaltransmission parametersg∗i and f∗

i for Cases I, II andIII.

Corollary 9. When pmax ≥ p0, the optimal transmissionparameters are given byg∗i = 1 (i ≥ 0) and f∗

i = 0 (i > 0);Whenk1η1 − k2η2 < pmax < p0, the optimal transmissionparametersg∗i and f∗

i are listed in Table III.

Proof: The proof is deferred to Appendix G.Remark: Note thatpmax ≥ p0 indicates that the allowable

backup energy supply is sufficient so that the source canuse the reliable energy whenever it needs. In this scenario,packet delivery is guaranteed in each slot, and there will be

no backlogged packets in Case I and Case II, while in Case III,the data queue may still accumulate as each data arrival bringsin multiple packets while at most one packet is delivered ineach slot. Whenk1η1−k2η2 < pmax < p0, the source shouldtransmit according to the optimal thresholdi∗. For Case Iand Case II, the utilization of power from the RES couldhappen only in two scenarios: when a new packet arrives butno harvested energy can be used, and the data packet queuelength is equal toi∗ − 1 and i∗, respectively. In the formercase, the source transmits using the power from the RES withthe probabilityg∗i∗−1 < 1, while in the latter case, the sourcewill transmit using the energy from the RES definitely withg∗i∗ = 1. In Case III, the source should transmit using thereliable energy as soon as the data queue length exceeds thethresholdi∗, no matter whether there is a new data arrival. Thatis, g∗i−k1

= 1 andf∗i = 1 are set for alli > i∗. Once getting

the optimalg∗i and f∗i , we can compute the optimal steady-

state probabilitiesπ∗(i,j) and the corresponding minimum aver-

age delayD∗ = 1k1η1

∑Q1

i=1 i∑Q2

j=0 π∗(i,j), which depends on

the allowable reliable energypmax. Hence,D∗ is an implicitfunction of pmax. As shown by simulation results in the nextsection, the average queuing delay monotonically decreaseswith the increase of the powerpmax.

VI. SIMULATION RESULTS

In this section, simulation results are presented to demonstratethe performance of the proposed scheduling scheme andvalidate our theoretical analysis.

In simulations, the packet and energy arrival processes aremodeled by generating two Bernoulli random variables withthe parametersη1 andη2, respectively, at the beginning of eachtime slot. The packet transmissions are scheduled according toour proposed policy. And we apply the optimal transmissionparametersf∗

i and g∗i listed in Table III to get the optimaldelay-power tradeoff curves. Each simulation runs over106

time slots. In the figures, the lines and the marks ’o’ indicatetheoretical and simulation results, respectively. One canseethat theoretical and simulation results match well.

Fig. 3 plots the optimal delay-power tradeoff performancefor Case I. It is observed from Fig. 3(a) that the minimum aver-

10

0 0.02 0.04 0.06 0.08 0.1 0.12 0.140

2

4

6

8

10

12

14

16

18

20

The normalized average power

The

opt

imal

ave

rage

del

ay

Theoretical, Q2=1

Theoretical, Q2=3

Theoretical, Q2=10

Theoretical, Q2=100

Simulation results

(a) η1 = η2 = 0.3

0 0.05 0.1 0.15 0.2 0.25 0.30

2

4

6

8

10

12

14

16

18

20

The normalized average power

The

opt

imal

ave

rage

del

ay

Theoretical, η2=0.3

Theoretical, η2=0.5

Theoretical, η2=0.6

Simulation results

(b) η1 = 0.5, Q2 = 1

Fig. 3. The delay-power curve for Case I.

age queueing delay monotonically decreases with the increaseof the maximum power consumptionpmax, which contributesto the growth of the service rate. That is, when more power canbe drawn from the RES, the packets will be transmitted morequickly and the queueing delay is reduced. One can see thatthe minimum average queueing delay decreases from infinityto zero, whenpmax grows from zero top0, which is equalto µ2

Q2+φ−11

in the case ofη1 = η2. Hence, the decreasingrate grows with the increase of the battery capacityQ2. Thismeans that a largerQ2 leads to a much smaller queueing delay,since less harvested energy is wasted due to the limitationof the battery capacity. Fig. 3(b) demonstrates the optimaldelay-power performance for different energy arrival rates η2.When η2 ≤ η1, the average delay is infinite atpmax = 0,since the arrival rate is greater than or equal to the servicerate. Therefore, the source should exploit extra energy fromthe RES to transmit backlogged packets, corresponding to apositivepmax. While the source can rely only on the harvestedenergy to transmit,i.e., pmax = 0, whenη2 > η1.

Fig. 4 shows the optimal delay-power tradeoff performanceof the proposed scheme in Case II withk2 = 2, · · · , 6. Theoptimal delay-power curve of the case withk1 = k2 = 1 isalso plotted for comparison. In this experiment, we setη1 =

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.450

5

10

15

20

25

30

The normalized average power

The

min

imum

ave

rage

del

ay

k2=1

k2=2k

2=3k

2=4

k2=5

k2=6

Fig. 4. The optimal delay-power performance for Case II withη1 = 0.5,η2 = 0.1 andQ2 = 5.

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

0

2

4

6

8

10

12

14

16

18

20

The normalized average power

The

min

imum

ave

rage

del

ay

k1=1,2,3,4,5,6

Fig. 5. The delay-power performance for Case III withη1 = 0.1, η2 = 0.3,andQ2 = 5.

0.5, η2 = 0.1 and Q2 = 5. From this figure, one can seethat there exists an optimal delay-power tradeoff for eachk2.The average queueing delay monotonically decreases with theincrease of the maximum allowable power consumptionpmax

from the RES due to the enhanced service rate. For the samereason, a largerk2 means a higher amount of energy harvestedeach time, and leads to a much better delay-power tradeoff. Itis also observed the delay-power curves ofk2 = 5 andk2 = 6are almost identical to each other. This owes to the fact thatin the case ofk2 = 6, a part of harvest energy is wasted whenrecharging the battery with capacityQ2 = 5.

Similarly, we plot the optimal delay-power curves of theproposed scheme for Case III with differentk1 in Fig. 5. Wesetη1 = 0.1, η2 = 0.3, andQ2 = 5. Similar to Case II shownin Fig. 4, a higherpmax induces reduced average queueingdelay thanks to the enhanced service rate. The only differencebetween them is the behavior of the minimum average delayD∗. In Case I withk1 = k2 = 1, the average queueing delayis equal to zero if there exists sufficient energy whether fromthe battery or the RES, since one newly arriving data packetcan always be delivered immediately. In Case III, however,at most one ofk1 data packets that newly arrive at this slot

11

can be delivered, and the other packets shall wait for the nexttransmission opportunity. And more packets are queued whenthe data arrival rate is increased due to the growth ofk1 orη1. As shown in Fig. 5,D∗ increases with the increase ofk1.

VII. C ONCLUSIONS

In this paper, we investigated the delay optimal schedulingproblem over a communication link. The source node can relyon energy supply either from an energy harvesting batteryof finite capacity or from the RES subject to a maximumpower consumption from the RES. Using the two-dimensionalMarkov chain modeling, we formulated an LP problem andstudied the structure of the optimal solution. As a result,we obtained the optimal scheduling policy through rigorousderivation and algorithm design.

It is found that the source should schedule packet transmis-sions according to a critical threshold on the data queue length.Specifically, the source should always wait for the harvestedenergy when the data queue length is below the optimalthreshold i∗, and resort to the RES when the data queuelength exceeds the thresholdi∗ while no harvested energy canbe exploited. The optimal thresholdi∗ is determined by themaximum allowable power from the RESpmax. Simulationresults confirmed our theoretical analysis. It was shown thatthere always exists an optimal delay-power tradeoff and itsdecreasing rate depends on the energy arrival rate and thebattery capacity.

In this work, we assume that the Bernoulli data and energyarrival processes generate integral packets probabilistically,and only one data packet is transmitted in each slot. In thefuture, we will extend the study to the scenario where rate-flexible physical-layer transmissions are scheduled basedonthe randomly available amount of harvested energy and time-varying wireless channel conditions.

APPENDIX

A. Proof of Lemma 2

By applying the stochastic scheduling scheme described inSection II.2, the source shall resort to the RES only when theharvested energy is not available in the current slot. Thus,thestate setQp (c.f. (8)) is given by(0, 0), (1, 0), · · · , (Q1, 0).Recall that the source will draw the reliable energy totransmit with probabilitygi if new data packets arrive andwith probability fi if no data packets arrive, respectively.Hence, the reliable energy consumption at state(i, 0) can beexpressed asω(0,0)(p) = µ2g0 and ω(i,0)(p) = µ2gi + µ1fi(i > 0), respectively. Consequently, the normalized averagepower consumed from the RES can be obtained asP =∑Q1

i=0 π(i,0)ω(i,0)(p) =∑Q1

i=0 π(i,0)µ2gi +∑Q1

i=1 π(i,0)µ1fi.The following result eliminates the dependence ofP on thetransmission parameters and presents a unified expression forall three cases.

1) In Case I, from Fig. 2(c), the local equilibrium equationof the Markov chain can be expressed as

π(i,0)λ1,i = π(i+1,0)µ1,i+1, π(0,j)µ0 = π(0,j+1)µ2, (24)

for all i ∈ 0, · · · , Q1 − 1 and j ∈ 0, · · · , Q2 − 2, andπ(0,Q2−1)µ0 = π(0,Q2)η1. From (24), we haveπ(0,0)λ1,0 =∑Q1

i=1 π(i,0)(µ1,i−λ1,i), whereπ(Q1,0)λ1,Q1 = 0 is introducedfor notational convenience. Withλ1,i = µ2(1− gi) (c.f. (10))andµ1,i = µ0+µ1fi (c.f. (11)), the normalized average powerconsumption from the RESP can also be expressed as

P =π(0,0)µ2g0 +∑Q1

i=1π(i,0)(µ2gi + µ1fi)

=π(0,0)(µ2 − λ1,0) +∑Q1

i=1π(i,0)(µ2 − λ1,i + µ1,i − µ0)

=π(0,0)µ2 + (µ2 − µ0)∑Q1

i=1π(i,0).

Hence, we getξ0 = µ2, ξi = µ2 − µ0 for all i ∈ QR1 and

ζi = 0 for all i ∈ Q1.2) From Fig. 2(d) for Case II, the local equilibrium equation

at state(0, 0) can be expressed asπ(1,0)µ1,1 − π(0,0)λ1,0 =η2π(0,0) −µ2π(0,1) −µ1π(1,1). Following a similar procedure,the local equilibrium equation at state(i, 0) (i ≥ 1) is thus

π(i+1,0)µ1,i+1 − π(i,0)λ1,i = π(i,0)µ1,i − π(i−1,0)λ1,i−1

+ (µ0 + µ3)π(i,0) − µ2π(i,1) − µ1π(i+1,1)

=η2∑i

m=0π(m,0) − µ2

∑i

m=0π(m,1) − µ1

∑i+1

m=1π(m,1),

where the second equality is obtained through recursion of(π(i,0)µ1,i − π(i−1,0)λ1,i−1). Hence, we can compute thenormalized average power consumption as

P =∑Q1

i=0π(i,0)(µ2 − λ1,i) +

∑Q1

i=1π(i,0)µ1fi

=µ2

∑Q1

i=0π(i,0) +

∑Q1

i=1

(

π(i,0)µ1,i − π(i−1,0)λ1,i−1

)

=∑Q1

i=0π(i,0)(µ2 + η2(Q1 − i))

− µ2Q1π(0,1) −∑Q1

i=1(η2(Q1 − i) + µ1)π(i,1),

where the second equality is due to the fact thatµ1,i = µ1fi,and the third equality is obtained through the summation ofπ(i,0)µ1,i − π(i−1,0)λ1,i−1 over all i > 0. In this way, weget ξi = µ2 + η2(Q1 − i) for all i ∈ Q1, ζ0 = µ2Q1, andζi = η2(Q1 − i) + µ1 for all i ∈ QR

1 .3) In Case III, the corresponding Markov chain is depicted

in Fig. 2(e). Wheni < k1 − 1, the local balance equation atstate(i, 0) is given by

µ1,i+1π(i+1,0) = π(i,0)µ1,i + π(i,0)(λ1,i + λ1,i)− µ1π(i+1,1)

=π(0,0)µ0 +∑i

m=0π(m,0)η1 −

∑i+1

m=1µ1π(m,1),

whereλ1,i+ λ1,i = η1 (c.f. (12)) is applied. Wheni ≥ k1−1,the local balance equation at state(i, 0) can be expressed as

π(i−k1+1,0)λ1,i−k1+1 + µ1,i+1π(i+1,0)

=π(i,0)µ1,i − λ1,i−k1π(i−k1,0) + π(i,0)(λ1,i + λ1,i)

− µ2π(i−k1+1,1) − µ1π(i+1,1) = η1

i∑

m=i−k1+1

π(m,0)

+ π(0,0)µ0 − µ1

∑i+1

m=1π(m,1) − µ2

∑i−k1+1

m=0π(m,1).

12

Then, we can compute the normalized average power con-sumption as

P =∑Q1

i=0π(i,0)µ2gi +

∑Q1

i=1π(i,0)µ1fi

=∑Q1−1

i=k1−1(π(i−k1+1,0)λ1,i−k1+1 + µ1,i+1π(i+1,0))

+∑k1−2

i=0µ1,i+1π(i+1,0) − µ3π(0,0) − η2

∑Q1

i=1π(i,0)

=∑Q1

i=0ξiπ(i,0) −

∑Q1

i=0ζiπ(i,1).

where the second equality is due to the fact thatλ1,i = µ3 +µ2gi (c.f. (12)) andµ1,i = µ0+µ1fi (c.f. (11)), the last equal-ity is obtained via the summation ofπ(i−k1+1,0)λ1,i−k1+1 +µ1,i+1π(i+1,0) and µ1,i+1π(i+1,0) over i ≥ k1 − 1 andi < k1 − 1, respectively. As a result, we can compute thecoefficientsξi andζi as listed in Table I.

B. Proof of Lemma 3

We will prove that for eachi, the probabilityπi satisfies theinequality (15) for Cases I, II and III, respectively.

1) In Case I, we haveλ1,i = µ2(1−gi) andµ1,i = µ0+µ1fi,which satisfy0 ≤ λ1,i ≤ µ2, µ0 ≤ µ1,i ≤ µ0 + µ1 = η1,since0 ≤ gi ≤ 1 and0 ≤ fi ≤ 1. From the local equilibriumequation (24), we have0 ≤ πi = π(i,0) ≤ φπ(i−1,0), since0 ≤π(i,0)

π(i−1,0)=

λ1,i−1

µ1,i≤ µ2

µ0= φ. In this case, we haveΘu(i, πi) =

φπ(i−1,0) when gi−1 = 0 and fi = 0, andΘl(i, πi−1) = 0whengi−1 = 1.

2) From Fig. 2(d) in Case II, the local balance equationπ(i−1,0)λ1,i−1 = π(i,0)(µ1,i+µ0)+(µ0+µ1)

∑Q2−1j=1 π(i,j)+

η1π(i,Q2) holds for alli > 0, thus leading toπ(i−1,0)λ1,i−1 =

π(i,0)(µ1,i + µ0) + η1∑Q2

j=1 π(i,j) because ofµ0 + µ1 = η1.

Hence, we haveη1πi = η1∑Q2

j=0 π(i,j) = π(i−1,0)λ1,i−1 −π(i,0)(µ1,i+µ0)+ η1π(i,0). Since0 ≤ gi ≤ 1 and0 ≤ fi ≤ 1,we get0 ≤ λ1,i = µ2(1 − gi) ≤ µ2 andµ0 ≤ µ1,i + µ0 =µ1fi+µ0 ≤ η1. With the varying parameterλ1,i andµ1,i,we further have0 ≤ πi ≤ Θu(i, πi), whereπi = Θu(i, πi) =τ η2π(i−1,0) + π(i,0)η2 whenλ1,i−1 = µ2 andµ1,i + µ0 = µ0

(gi−1 = fi = 0), andπi = Θl(i, πi−1) = 0 whenλ1,i−1 = 0andµ1,i + µ0 = η1 (gi−1 = fi = 1), respectively.

3) In Case III, from Fig. 2(e), the local equilibriumequation between states(i − 1, j) and (i, j) (j ∈ Q2)can be expressed as

∑i−1m=0 π(m,0)(λ1,m + λ1,m) + (µ2 +

µ3)∑i−1

m=0

∑Q2−1j=1 π(m,j) + η1

∑i−1m=0 π(m,Q2) = π(i,0)µ1,i +

(µ0 + µ1)∑Q2−1

j=1 π(i,j) + η1π(i,Q2) for all i < k1. Withλ1,i−1 + λ1,i−1 = µ2 + µ3 = η1 andµ0 + µ1 = η1, it can berewritten asη1

∑i−1m=0 πm = π(i,0)µ1,i + η1

∑Q2

j=1 π(i,j).Sinceµ0 ≤ µ1,i = µ0 + µ1fi ≤ η1, we getΘl(i, πi−1) ≤

η1∑Q2

j=0 π(i,j) ≤ Θu(i, πi). Thus, πi = Θu(i, πi) =µ1

η1π(i,0) + η1

η1

∑i−1m=0 πm when µ1,i = µ0 (fi = 0), and

πi = Θl(i, πi−1) =η1

η1

∑i−1m=0 πm when µ1,i = η1 (fi = 0),

respectively.Similarly, when i ≥ k1, the corresponding local equilib-

rium equation between states(i − 1, j) and (i, j) is givenby η1

∑i−1m=i−k1+1 πm = π(i,0)µ1,i − π(i−k1,0)λ1,i−k1 +

η1∑Q2

j=1 π(i,j). Since 0 ≤ λ1,i = µ2(1 − gi) ≤ µ2 and

µ0 ≤ µ1,i = µ0 + µ1fi ≤ η1, we getη1∑i−1

m=i−k1+1 πm ≤

η1∑Q2

j=0 π(i,j) ≤ µ2π(i−k1,0)+µ1π(i,0)+η1∑i−1

m=i−k1+1 πm.Thus, πi = Θu(i, πi) = µ2

η1π(i−k1,0) + µ1

η1π(i,0) +

η1

η1

∑i−1m=i−k1+1 πm when λ1,i−k1 = µ2 and µ1,i = µ0

(gi−k1 = 0, fi = 0), πi = Θl(i, πi−1) =η1

η1

∑i−1m=0 πm when

λ1,i−k1 = 0 and µ1,i = η1 (gi−k1 = 1, fi = 1), respectively.Combining the above two cases:i < k1 andi ≥ k1, we get

Θu(i, πi) andΘl(i, πi−1) as listed in Table II.

C. Proof of Theorem 4

Subject to the constraint (13.b), we haveD =1

k1η1

∑Q1

i=1 iπi ≥ 1k1η1

∑Q1

i=1 iΘl(i, πi−1). This meansthat the average queuing delayD, as a weighted summationof πi, can be minimized, if eachπi chooses its lower boundΘl(i, πi−1) for all i ≥ 1. From Lemma 3, this happenswhen all the transmission parameters satisfygi−k1 = fi = 1for all i > 0. This corresponds to the scheduling policybased on the optimal thresholdi∗ = 0 and the correspondingaverage power consumption from the RES isp0. Hence,when pmax ≥ p0, the minimum average queueing delayD∗ is obtained ifπ∗

i = Θl(i, π∗i−1) or π∗al

i = 0 for alli = 1, · · · , Q1, and the optimal thresholdi∗ is zero. Whenk1η1 − k2η2 < pmax < p0, gi−k1 = fi = 1 does not hold forall i > 0 and hence there must existi∗ > 0.

D. Proof of Theorem 5

Subject to the constraint∑Q1

i=0 πi = 1, The average queueingdelay D = 1

k1η1

∑Q1

i=1 iπi can be minimized, if eachπi

chooses its lower boundΘl(i, πi−1) for all i ≥ 1, corre-sponding to the case whenpmax ≥ p0 shown in the proof ofTheorem 4. Here, we focus on studying the optimal solutionin the case whenk1η1 − k2η2 < pmax < p0. In this case,P =

∑Q1

i=0 ξiπ∗(i,0)−

∑Q1

i=0 ζi·π∗(i,1) = pmax is straightforward.

This lies in the fact that the data queue length becomessmaller when more reliable energy can be exploited to transmitbacklogged data packets.

In the sequel, we will show that whenk1η1 − k2η2 <pmax < p0, the average queueing delay can also be minimized,if the optimal solutionπ∗ satisfies (16) (π0, π1, · · · , πi∗−1

are assigned to their upper bounds andπi∗+1, · · · , πQ1 areassigned to their lower bounds) for Cases I, II, and III,respectively.

1) Case I: From Lemma 1, we only need to consider thesteady-state probabilitiesπ(i,0) andπ(0,j) for all i ∈ Q1 andj ∈ Q2. The constraint

∑Q2

j=0 π(i,j) ≥ Θl(i, πi−1) = 0naturally holds sinceπ(i,j) ≥ 0, and hence we do not considerthe corresponding constraints

∑Q2

j=0 π(i,j) ≥ 0 (i > 0).From the Markov chain in Fig. 2(c), the optimal solution

π∗(0,j) satisfiesµ2π

∗(0,j) = µ0π

∗(0,j−1) for j = 1, · · · , Q2 − 1

and η1π∗(0,Q2)

= µ0π∗(0,Q2−1) for j = Q2. Hence, the

probability that the data queue length is zero is obtained

as π∗0 = π∗

(0,0) + π∗(0,0)

Q2−1∑

j=1

φ−j +π∗

(0,0)

φQ2−1φ1= απ∗

(0,0),

where α =∑Q2−1

i=0 φ−i + φ−(Q2−1)φ−11 . Thus, we have

∑Q1

i=1 π∗(i,0) = 1 − π∗

0 = 1 − απ∗(0,0). And the normalized

13

average power consumption from the RES isP = µ2π∗(0,0) +

(µ2 − µ0)∑Q1

i=1 π∗(i,0) = µ2π

∗(0,0) + (µ2 − µ0)(1 − απ∗

(0,0)),

from which we obtainπ∗(0,0) =

pmax−(µ2−µ0)µ2−α(µ2−µ0)

. Hence,π∗(0,0)

depends only onpmax. As shown in the proof of Theorem4, whenpmax ≥ p0, we haveπ∗

0 = απ∗(0,0) = 1 and thus

π∗(0,0) = 1

α. When pmax < p0, there must existπ∗

(0,0) < 1α

and∑Q1

i=1 π∗(i,0) > 0.

By contradiction, we will show that the optimal solutionπ∗(i,0) satisfies (16):π∗

(i,0) = φπ∗(i−1,0) for i < i∗, 0 <

π∗(i,0) < φπ∗

(i−1,0) for i = i∗, and π∗(i,0) = 0 for i > i∗,

since it leads to the minimum average queueing delay. Supposethat there exists another set of steady-state probabilitiesπ(i,0):π(i,0) = π∗

(i,0) = π∗(0,0)φ

i for 0 ≤ i < m, and0 < π(m,0) <

φπ(m−1,0) for m ≤ i ≤ i1. Subject toπ0 +∑i1

i=1 π(i,0) =

π∗0 +

∑i∗

i=1 π∗(i,0) = 1, there must existπ(i,0) < π∗

(i,0) for

m ≤ i < i∗ and∑i1

i=i∗ π(i,0)−π∗(i∗,0) =

∑i∗−1i=m (π∗

(i,0)−π(i,0))

with i1 ≥ i∗. Thus, we havei∗(∑i1

i=i∗ π(i,0) − π∗(i∗,0)) =

i∗∑i∗−1

i=m (π∗(i,0) − π(i,0)). Hence, the corresponding average

queueing delayD = 1η1

∑i1i=1 i · π(i,0) satisfies

D = 1η1

∑i∗

i=1 i · π∗(i,0) −

1η1

∑i∗−1i=1 i · (π∗

(i,0) − π(i,0))

+ 1η1(∑i1−1

i=i∗ iπ(i,0) − i∗π∗(i∗,0)) > D∗.

As a result, we can obtain the minimum average queueingdelay when the optimal solutionπ∗ satisfies (16).

2) Case II: Similar to Case I, we haveΘl(i, πi−1) = 0,as listed in Table II. From the corresponding Markov chainshown in Fig. 2(d), we haveπi = Θl(i, πi−1) = 0 andπ(i,j) = 0 for all i > m, if πm = Θl(m, πm−1) holds. Wefirst show thatπi = Θl(i, πi−1) does not hold fori ≤ i∗. Ifthe solutionπ satisfies0 < πi ≤ Θu(i, πi) for 1 < i < i1andπi1 = Θl(i1, πi1−1) for somei1 ≤ i∗, the correspondingpower consumption from the RESP will be larger thanpmax,since it satisfiesP ≥ pi1−1 ≥ pi∗−1 > pmax ≥ pi∗ (pmdecreases with the thresholdm). This violates the constraint(13.a). Hence, the solutionπ∗ should satisfy0 < π∗

i ≤Θu(i, π

∗i ) for 0 < i ≤ i1, and π∗

i = Θl(i, π∗i−1) = 0 for

i ≥ i1 > i∗, respectively. Then, we show that among allthe candidate solutions, the solutionπ∗ satisfying (16) leadsto the minimum queueing delay. According to the condition,we have0 < πi ≤ τ η2π(i−1,0) + π(i,0)η2 for 0 < i ≤ i1.In this case, the solutionπ∗ satisfiesπ∗

i =∑Q2

j=0 π∗(i,j) =

τ η2π∗(i−1,0) + η2π

∗(i,0) for 0 < i < i∗ and the minimum

average queueing delay isD∗ =∑i∗

i=1 iπ∗i . Suppose that

there is a solutionπ that satisfiesπi = τ η2π(i−1,0) + η2π(i,0)

for 0 < i < m, 0 < πm < τη2π(m−1,0) + η2π(m,0) forsomem ≤ i∗ − 1, and0 < πi ≤ τ η2π(i−1,0) + η2π(i,0) form < i ≤ i1. Subject to

∑i∗i=0 π

∗i =

∑i1i=0 πi = 1, the average

queueing delayD = 1η1

∑i1i=1 i · πi satisfies

D = 1η1

∑i∗

i=1 i · π∗i + 1

η1(∑i∗−1

i=1 i · (πi − π∗i )

+(∑i1

i=i∗ iπi − i∗π∗i∗)) > D∗ + 1

η1

∑i∗−1i=0 (i∗ − i)(π∗

i − πi)

= D∗ + 1η1

∑i∗−1i=0 (i∗ − i)η2(π

∗(i,0) − π(i,0))

+ 1η1

∑i∗−2i=0 (i∗ − i− 1)τ η2(π

∗(i,0) − π(i,0)) ≥ D∗

where the first inequality holds since∑i1

i=i∗ iπi − i∗π∗i∗ =

∑i∗−1i=0 i∗(π∗−πi)+

∑i1i=i∗+1(i− i∗)πi >

∑i∗−1i=0 i∗(π∗−πi),

the last two inequalities hold since we obtainπ0 > η2π(0,0)

and∑i∗−1

i=0 π∗(i,0) ≥

∑i∗−1i=0 π(i,0) based on the property of the

Markov chain. Hence, the optimal solutionπ∗ should satisfy(16) to minimize the average queueing delay.

3) Case III: In this case, we haveΘl(i, πi−1) =τ∑i−1

m=[i−k1+1]+ πm > 0. Note that Q1 should be suffi-ciently large to avoid buffer overflow. To decrease the averagequeueing delayD = 1

k1η1

∑Q1

i=1 iπi, πi with large index

should be assigned its lower boundτ∑i−1

m=[i−k1+1]+ πm. Inthis sense, there exists an integeri1 that satisfiesπi =τ∑i−1

m=[i−k1+1]+ πm for all i ≥ i1. For the same reason asstated in Case II,i1 > i∗ should be satisfied to meet thepower consumption (from the RES) constraintP ≤ pmax.

In the same way, we will compare the delay performancesbetween the optimal solutionπ∗ satisfying (16) and a can-didate solutionπ. Suppose thatπ satisfiesπi = Θu(i, πi)for 0 < i < m, Θl(i, πi−1) < πm < Θu(i, πi) for somem ≤ i∗−1, and0 < πi ≤ Θu(i, πi) for m < i ≤ i1. We noticethat π∗

i = τ∑i−1

m=[i−k1+1]+ π∗m (i > i∗) is a decreasing

sequence (otherwise the data queue will be unstable). So doesthe sequenceπi (i > i1). And π∗

i = Θu(i, π∗i ) increases

with the growth of i for 0 ≤ i ≤ i∗. Then, subject to theconstraint

∑Q1

i=0 π∗i =

∑Q1

i=0 πi = 1, we haveπ∗i ≥ πi

for 0 ≤ i ≤ i∗ and π∗i ≤ πi for i∗ < i ≤ Q1, and

∑i∗

i=0(π∗i −πi) =

∑Q1

i=i∗+1(πi−π∗i ). As a result, the average

queueing delayD satisfies

D = 1k1η1

Q1∑

i=1

iπ∗i + 1

k1η1(i∗∑

i=1

i(πi − π∗i ) +

Q1∑

i=i∗+1

i(πi − π∗i ))

> D∗ + 1k1η1

(−i∗∑

i=1

i∗(πi − π∗i ) +

Q1∑

i=i∗+1

i(πi − π∗i )) > D∗.

In this way, we show that the optimal solutionπ∗ shouldsatisfy (16) in Case III.

Note that the optimal solutionπ∗ corresponds to the thresh-old based scheduling policy. Naturally, a larger thresholdmleads to a larger queueing delay. Meanwhile, a lower powerpm is consumed from the RES. The optimal threshold canbe obtained by comparing the maximum allowable powerconsumptionpmax with the power thresholdspm as: i∗ =argminpm≤pmax

m.

E. Proof of Corollary 7

From Theorem 5 and Lemma 3, whenpmax ≥ p0, π∗i =

π∗(i,0) = 0 for all i > 0. Then, by substituting (18) into the

equationπ∗(0,0) +

∑Q2

j=1 π∗(0,j) = 1, we obtainπ∗

(0,0) = 1α

.Accordingly, the power thresholdp0 = µ2π

∗(0,0) = µ2α

−1

because ofg0 = 1. Then, we discuss the optimal solution forthe case whenη1− η2 < pmax < p0. From Theorem 5 and itsproof, we know that there exists an optimal thresholdi∗ > 0so thatπ∗

(i,0) = φπ∗(i−1,0) = π∗

(0,0)φi for i < i∗ andπ∗

(i,0) = 0

for i > i∗, whereπ∗(0,0) =

pmax−(µ2−µ0)µ2−α(µ2−µ0)

. Sinceπ∗0 = απ∗

(0,0)

and thus∑i∗

i=1 π∗(i,0) = 1 − απ∗

(0,0), we obtainπ∗(i∗,0) = 1 −

απ∗(0,0) −

∑i∗−1i=1 π∗

(i,0) = 1− π∗(0,0)(α+

∑i∗−1i=1 φi). From the

14

property of the optimal solutionπ∗(i,0), the optimal threshold

i∗ can be evaluated asi∗ = Ωφ(π∗(0,0), 1 − απ∗

(0,0)), which

is the integer that satisfiesπ∗(0,0)

∑i∗−1m=1 φ

m ≤ 1 − απ∗(0,0) <

π∗(0,0)

∑i∗

m=1 φm.

F. Proof of Corollary 8

From Theorem 5 and its proof, the optimal threshold isi∗ = 0 when pmax ≥ p0 and hence the optimal solutionπ∗

can be obtained by solving(1 + Q1)(1 + Q2) independentlinear equations:πau

i = 0 (∀i > 0), πPs = 0, andπe = 1. In this case, we getπ∗ = π

′

0 according to(22). When pi∗ ≤ pmax < pi∗−1, the optimal solutionπ∗

satisfiesπ∗a0 = pmax, π∗aui = 0 (i = 1, · · · , i∗ − 1),

π∗ali = 0 (i = i∗+1, · · · , Q1), πPs = 0, andπe = 1. Hence,

we can obtainπ∗ = bA−1 by solving(1+Q1)(1+Q2) linearequations.

G. Proof of Corollary 9

We will discuss the optimal transmission parameters inCases I, II and III, respectively.

1) Case I: When pmax ≥ p0, from the local equilib-rium equationπ∗

(0,0)λ∗1,0 = π∗

(1,0)µ∗1,1 = 0, we must have

λ∗1,0 = µ2(1 − g∗0) = 0 andg∗0 = 1, sinceπ∗

(0,0) = α−1 > 0and π∗

(i,0) = 0 for all i > 0. Also, since π∗(i,0)λ

∗1,i =

π∗(i+1,0)µ

∗1,i+1 = 0 always holds fori > 0, we can set

g∗i = 1 for all i ≥ 0 and f∗i = 0 for i > 0. When

η1 − η2 < pmax < p0, from (19), π∗(i,0) = π∗

(0,0)φi for all

i < i∗. Thus,π∗

(i+1,0)

π∗

(i,0)=

λ∗

1,i

µ∗

1,i+1= φ. On the other hand,

0 ≤ λ∗1,i ≤ µ2 andµ0 ≤ µ∗

1,i ≤ η1. Hence, we haveλ∗1,i = µ2

and µ∗1,i+1 = µ0 for i < i∗ − 1. Substitutingλ∗

1,i and µ∗1,i+1

into (11) givesg∗i = 0 for 0 ≤ i ≤ i∗ − 2 and f∗i = 0 for

1 ≤ i ≤ i∗ − 1.From π∗

(i−1,0)λ1,i−1 = π∗(i,0)µ1,i, we can getλ∗

1,i∗ = 0and g∗i∗ = 1, sinceπ∗

(i,0) = 0 for all i > i∗. Without loss ofgenerality, we can setgi = 1 andfi = 0 for i > i∗ to maintainconsistency. Fromπ∗

(i∗−1,0)λ1,i∗−1 = π∗(i∗,0)µ1,i∗ , we have

µ2(1−g∗i∗−1) =π∗

(i∗,0)

π∗

(i∗−1,0)(µ0+µ1fi∗), which is satisfied when

fi∗ = 0 andg∗i∗−1 = 1− µ0

µ2

π∗

(i∗,0)

π∗

(0,0)φi∗−1 = 1−

π∗

(i∗,0)

π∗

(0,0)φi∗ . In this

way, we getf∗i = 0 for all i > 0 andg∗i as listed in Table III.

2) Case II: Similar to Case I, by exploiting the local equi-librium equationπ∗

(i−1,0)λ∗1,i−1 = π∗

(i,0)µ∗1,i + η1

∑Q2

j=1 π∗(i,j)

andπ∗(i,j) = 0 for all i > i∗, we can obtaing∗i = 1 (i ≥ 0)

and f∗i = 0 (i > 0) when pmax ≥ p0, and f∗

i = 0 and g∗ilisted in Table III whenη1 − k2η2 < pmax < p0, respectively.

3) Case III: Similar to the above two cases, whenpmax ≥p0, we haveg∗i = f∗

i+1 = 1 for all i ≥ 0, sincei∗ = 0. Whenk1η1 − η2 < pmax < p0, the local equilibrium equation at thestate(i∗, 0) is given byπ∗

(i∗,0)(µ0+µ1f∗i∗)−π∗

(i∗−k1,0)µ2(1−

g∗i∗−k1) = η1

∑i∗−1m=i∗−k1+1

∑Q2

j=0 π∗(m,j) − η1

∑Q2

j=1 π∗(i∗,j)

when i∗ ≥ k1. When 0 ≤ i∗ < k1, the local equilibriumequation at the state(i∗, 0) can be expressed asπ∗

(i∗,0)(µ0 +

µ1f∗i∗) = η1

∑i∗−1m=0

∑Q2

j=0 π∗(m,j) − η1

∑Q2

j=1 π∗(i∗,j). From the

local equilibrium equations, we can computeg∗i∗−k1and f∗

i∗

for i∗ ≥ k1 and i∗ < k1, respectively, as listed in Table III.

REFERENCES

[1] A. Kansal and M. B. Srivastava, “An environmental energyharvestingframework for sensor networks,” inProc. International Symposium onLow Power Electronics and Design (ISLPED), 2003, pp. 481–486.

[2] M. Rahimi, H. Shah, G. Sukhatme, J. Heideman, and D. Estrin, “Study-ing the feasibility of energy harvesting in a mobile sensor network,”in Proc. IEEE International Conference on Robotics and Automation(ICRA), Sept. 2003, pp. 19–24.

[3] W. K. G. Seah, Z. A. Eu, and H.-P. Tan, “Wireless sensor networkspowered by ambient energy harvesting (WSN-HEAP) - Survey andchallenges,” inProc. Wireless VITAE, May 2009, pp. 1–5.

[4] J. Yang and S. Ulukus, “Transmission completion time minimizationin an energy harvesting system,” inProc. CISS, Princeton, New Jersey,March 2010.

[5] Z. Niu, S. Zhou, Y. Hua, Q. Zhang, and D. Cao, “Energy-aware networkplanning for wireless cellular system with inter-cell cooperation,” IEEETrans. Wireless Commun., vol. 11, no. 4, pp. 1412–1423, April 2012.

[6] E. Uysal-Biyikoglu, B. Prabhakar, and A. El Gamal, “Energy-efficientpacket transmission over a wireless link,”IEEE/ACM Trans. Networking,vol. 10, no. 4, pp. 487–499, Aug. 2002.

[7] M. A. Zafer and E. Modiano, “A calculus approach to minimum energytransmission policies with quality of service guarantees,” in Proc. IEEEINFOCOM, March 2005.

[8] W. Chen, U. Mitra, and M. Neely, “Energy-efficient scheduling withindividual delay constraints over a fading channel,” inProc. WiOpt,April 2007.

[9] J. Yang and S. Ulukus, “Optimal packet scheduling in an energyharvesting communication system,”IEEE Trans. Commun., vol. 60,no. 1, pp. 220–230, Jan. 2012.

[10] K. Tutuncuoglu and A. Yener, “Optimum transmission policies for bat-tery limited energy harvesting nodes,”IEEE Trans. Wireless Commun.,vol. 11, no. 3, pp. 1180–1189, March 2012.

[11] O. Ozel, K. Tutuncuoglu, J. Yang, S. Ulukus, and A. Yener, “Transmis-sion with energy harvesting nodes in fading wireless channels: Optimalpolicies,” IEEE J. Sel. Areas Commun., vol. 29, no. 8, pp. 1732–1743,Sep. 2011.

[12] J. Yang, O. Ozel, and S. Ulukus, “Optimal packet scheduling in abroadcast channel with an energy harvesting transmitter,”in Proc. IEEEICC, June 2011.

[13] O. Orhan and E. Erkip, “Optimal transmission policies for energy har-vesting two-hop networks,” inProc. of 2012 Conference on InformationSciences and Systems (CISS), Princeton, March 2012.

[14] S. Luo, R. Zhang, and T. J. Lim, “Optimal save-then-transmit protocolfor energy harvesting wireless transmitters,” inProc. IEEE ISIT, Cam-bridge, MA, July 2012.

[15] M. Gatzianas, L. Georgiadis, and L. Tassiulas, “Control of wirelessnetworks with rechargeable batteries,”IEEE Trans. Wireless Commun.,vol. 9, no. 3, pp. 581–593, Feb. 2010.

[16] C. K. Ho and R. Zhang, “Optimal energy allocation for wirelesscommunications with energy harvesting constraints,”IEEE Trans. SignalProcessing, vol. 60, no. 9, pp. 4808–4818, Sep. 2012.

[17] D. Niyato, E. Hossain, and A. Fallahi, “Sleep and wakeupstrategies insolar-powered wireless sensor/mesh networks: Performance analysis andoptimization,” IEEE Trans. Mobile Comput., vol. 6, no. 2, pp. 221–236,Feb. 2007.

[18] V. Sharma, U. Mukherji, V. Joseph, and S. Gupta, “Optimal energymanagement policies for energy harvesting sensor nodes,”IEEE Trans.Wireless Commun., vol. 9, no. 4, pp. 1326–1336, April 2010.

[19] T. G. Robertazzi,Computer Networks and Systems: Queueing Theoryand Performance Evaluation, 3rd ed. Springer-Verlag New York, Inc.,2000.

[20] J. Lei, R. Yates, and L. Greenstein, “A generic model foroptimizingsingle-hop transmission policy of replenishable sensors,” IEEE Trans.Wireless Commun., vol. 8, no. 2, pp. 547–551, Feb. 2009.

[21] R. A. Berry and R. G. Gallager, “Communication over fading channelswith delay constraints,”IEEE Trans. Inform. Theory, vol. 48, no. 5, pp.1135–1149, May 2002.

[22] A. Shiryaev,Probability, volume 95 of Graduate Texts in Mathematics,2nd ed. Springer-Verlag, 1996.

[23] W. Chen, K. B. Letaief, and Z. Cao, “Buffer-aware network coding forwireless networks,”IEEE/ACM Trans. Networking, vol. 20, no. 5, pp.1389–1401, Oct. 2012.

[24] R. M. Loynes, “The stability of a queue with non-independent inter-arrival and service times,”Proc. Cambridge Philos. Soc., vol. 58, pp.497–520, 1962.