行政院國家科學委員會專題研究計畫 成果報告

運用切割分析於驗證不可分割性之研究

研究成果報告(精簡版)

計 畫 類 別 ：個別型

計 畫 編 號 ： NSC 94-2213-E-006-091-

執 行 期 間 ： 94年 08 月 01 日至 95年 10 月 31 日

執 行 單 位 ：國立成功大學會計學系（所）

計 畫主持人：徐立群

計畫參與人員：碩士班研究生-兼任助理：黃義和、陳韋翰、葉光仁

報 告 附 件 ：出席國際會議研究心得報告及發表論文

處 理 方 式 ：本計畫可公開查詢

中 華 民 國 96年 01 月 24 日

行政院國家科學委員會補助專題研究計畫成果報告 ※※※※※※※※※※※※※※※※※※※※※※※※※

※ ※

※ 運用切割技術於驗證不可切割性之研究 ※ ※ ※ ※※※※※※※※※※※※※※※※※※※※※※※※※

計畫類別：■個別型計畫 □整合型計畫 計畫編號：NSC 94－2213－E－006－091 執行期間： 94 年 08 月 01 日至 95 年 10 月 31 日

計畫主持人：徐立群

本成果報告包括以下應繳交之附件：

□赴國外出差或研習心得報告一份 □赴大陸地區出差或研習心得報告一份 ■出席國際學術會議心得報告及發表之論文各一份 □國際合作研究計畫國外研究報告書一份

執行單位：國立成功大學會計學系

中 華 民 國 96 年 1 月 25 日

1

行政院國家科學委員會專題研究計畫成果報告

運用切割技術於驗證不可切割性之研究 Verifying Atomicity using Chopping Analysis

計畫編號：NSC 94-2213-E-006-091 執行期限：94 年 08 月 01 日至 95 年 10 月 31 日 主持人：徐立群 國立成功大學會計學系

shulc@mail.ncku.edu.tw

一、中文摘要

近年來確保同步程式碼能夠不被切割

的執行愈來愈受到重視。然而，驗證一組

methods 一定會不被切割的執行不是一件容易的工作。本研究利用並改造資料庫領

域發展出之異動處理切割技術來驗證不可

切割性。我們所設計的方法較型態推論技

術產出較準確的結果，但較模式檢驗代價

為低。 關鍵辭: 多執行緒、不可切割性、異動處理切割、至多一次特性。 Abstract

Ensuring atomic execution for concurrent code segments is gaining more and more attention in the past few years. However, verifying that a set of methods always execute atomically is not an easy task. The work reported here makes use of and adapts a transaction chopping technique originally developed in the database area in verifying atomicity. The goal is to design a technique that produces more accurate results than those generated by type inference, and incurs less cost than those incurred by model checking.

Key Words ： Multithread programs 、Atomicity、Transaction Chopping、At Most Once Property.

二、緣由與目的

驗證多執行緒程式的正確性是一項相

當困難的工作，因為多執行緒可以有太多

不同的執行組合，吾人很難去考量所有可

能情形。然而如果一組多執行緒彼此有所

謂不可切割性(atomicity)，驗證的工作將大大的簡化，這是因為 atomicity 代表一組

多執行緒雖然同步執行，效果卻與順序

(sequential)執行相同。因此對一組 atomic執行的多執行緒，我們僅需要驗證每一執

行緒是否符合其 specification，而無需考量多執行緒間可能產生之干擾，因此大大簡

化驗證工作之困難度。 Atomicity 在資料庫領域稱為 isolation

或 serializability，有許多成熟之技術被發展出來，本計畫與九十二年度之計畫⎯「並行程式分析與並行控制」，以及九十三年

度之計畫⎯「植基於切割技術之程式分析」即是探討源自於資料庫之異動處理

(transaction)切割技術，如何可以正確並合理代價用來驗證多執行緒之 atomicity 特性。 三、結果與討論

利用並行技術來滿足系統需求在軟體

設計上可說是相當普遍。在過去，「軟體

分析」與「資料庫系統」兩大領域各自進

行並行技術相關課題的研究，甚少有交

集。前者稱此項研究為「並行程式分析」，

而後者稱為「並行控制」。值得注意的是

2

兩者的研究模式、欲解決的問題，以及各

自發展出的解決方案皆有一些相似處，但

也有各自的獨特處。我們在九十二年度執

行「並行程式分析與並行控制」計畫，即

在探索二者之關連性，藉以激發出綜效性

的效果，目標是對單一領域產出新的且具

衝擊力的成果。我們得到以下的成果：

1.深入了解並行程式分析與並行控制之相

似處與相異處。 2.運用並行程式分析的典

型方法⎯assertional approach ，去驗證我們

過去為即時系統所設計的一個並行控制協

定[7]之實現方法的正確性。 3.運用並行控

制之 serializability theory ，將由 Owicki 及

Gries 提出針對並行程式分析使用多年的

atomicity criterion ⎯ at-most-once

property[6]加以清楚定位，我們也運用

serializability theory 指出 concurrent

programming 資深學者 Gregory Andrews

在其著作[2,3]中對 at-most-once property

不正確的詮釋。 4. 雖然 at-most-once

property 在 syntax 上相當容易檢測，但卻排

除了許多原本為atomic的actions(亦即對一

些 atomic action 會發出 false alarm)。有鑑

於此，我們將原本設計做為改進資料庫

transaction processing 效率的 chopping

analysis 技術，運用到並行程式分析上。我

們提出以 chopping analysis 之根基──

“SC-Cycle-Freedom＂做為 criterion for

atomicity。我們證明了它較 at-most-once

property 更 general，同時也能明確指明程

式中那些 statement 需要 synchronization，

俾確保 atomicity。儘管檢測

SC-Cycle-Freedom 會較 at-most-once

property 付出較高成本，我們提出一些重要

的 observations，可以將檢測成本壓低。 5.

我們將 SC-Cycle-Freedom運用到數個在[3]

中用到的範例程式。其中一個範例

(Peterson’s two-process tie-breaker

algorithm)，藉由 chopping analysis 我們確

信程式無需額外的 synchronization，即可以

保證 action atomicity。反之，如果不用

chopping analysis，我們就必須非常清楚該

演算法之邏輯，才能推論在無額外

synchronization 之下之 action atomicity。我

們另用 chopping 方法分析[3]中提到的

array copy 問題，我們證明其中四個

statement 皆無需額外的 synchronization 便

可保證 action atomicity。 6. 我們比較另一

學者所提出的 criterion for atomicity─

“Single-Phase-Condition＂[1]。我們的分析

發現 SC-Cycle-Freedom 較 Single-Phase-

Condition 更 general。

在九十二年度計畫執行過程中我們發

覺 chopping analysis 在分析並行程式之

code segment 是否為 atomic 時相當有潛

力，較 model checking 的代價為小，但較

type system ，如[4,5]，可獲得較準確的分

析結果，我們希望深入探究並期望得到具

影響力的結果，因此我們接續提出九十三

年度的計畫⎯「植基於切割技術之程式分

析」。

九十三年度我們專注於將 chopping

analysis 技術由 single action level 推向

procedure level，以便驗證一組 Java methods

是否為 atomic，我們的成果包括：

1. 我們證明了一組 Java methods，如果它

3

們的 chopping graph 沒有 SC-cycle 的話，

則每一個 method 皆是 atomic。 2. 我們克

服了一個應用 chopping analysis 難題：一個

Java method 可以同時被許多其它 thread

invoke，我們無法在 chopping graph 中對每

一個 method 繪製無限多 instance。我們證

明了以下的定理： In doing chopping

analysis, each method needs at most two

instances to be represented in the chopping

graph. 3. 探討如何在 chopping graph 中表

示 typical program constructs，如 conditional

branches，loop iterations，procedure calls

與 recursions 等。 4. 探討如何 summarize

effects of processes. 5. 撰寫工具程式針對

Java library classes 進行 atomicity 分析。

由於前兩年計畫成果漸臻成熟，我們繼

之提出九十四年度的計畫⎯「運用切割技

術於驗證不可切割性之研究」。我們解決

了 chopping analysis 應用於驗證 Java

method 之剩餘但依然重要的課題，同時進

行了廣泛的實驗評估。除了前述已有結果

外，本年度新的成果為：1. Representing

typical program constructs in the chopping

graph：我們確定了如何在 chopping graph

中表達 conditional branches, loop iterations,

procedure calls 等 常 見 的 program

constructs.

2.將理論結果納入 chopping analysis工具程

式之撰寫，着手進行廣泛實驗評估。我們

已經跟另一頗受學術界重視的方法 type

system approach [4,5]作了比較。我們發覺

chopping analysis 較 type system 分析為準

確，一些 chopping analysis 確認為 atomic

的 examples ， type system 卻 錯 認 為

non-atomic，亦即發出了 false alarm。這是

因為 type system 是屬於 local analysis，為

了避免錯誤，它只能做較保守的推估。反

之 chopping analysis 屬於 global analysis，

因此較可以看到全貌。當然 global analysis

付出的分析代價一定比 local analysis 為

大，不過 chopping analysis 用到一些 good

property，使其分析的 cost 相當合理。3. 除

了驗證 Java library classes，我們也針對其

它相關論文所使用的實例做驗證，得到了

正面的結果。

四、計畫成果自評

本計畫最重要的目標是研究結果必須

具有實用價值，因此進行之工作除了包含

理論的推導與原則的分析，我們也探討推

導出之理論與原則應用在目前主要並行軟

體案例時，所獲得實際效益有多大。 利用並行物件導向語言如 Java 所撰寫

之並行程式，要推論其正確性時，必須考

量指令間之交錯執行，這個問題一直是研

究人員認為極富挑戰之議題。Atomicity 是建構並行程式之重要根基，當一個 Java method 被驗証為 atomic 後，則該 method與其它method任何交錯執行的結果與沒有任何交錯執行之結果相同。這意味我們推

論一個 atomic method 之正確性時，可將其看成一段 sequential 程式碼，驗証的複雜度將大為降低。我們經過三年的研究，克服

將 chopping analysis 運用到程式分析的一些關鍵問題，實驗結果也看出這個方法的

潛力。我們目前正在將成果整理並將投稿

至適當的會議。 另外，我們也在發展 compositional

chopping analysis 之技術，chopping graph analysis 是一種以單一分析步驟來廣域分析所有的的程式中的 process。如果當

4

process 的數目很大時，一個 compositional並且 incremental 的分析是必須的。一個scalable 的 compositional analysis 需要一種能擷取“剛好足夠＂的流程細節及

subsystem 之表達，而組合許多 subsystem之表示式進而產生一個新的 subsystem 的關係，我們運用 congruence relation 來最小化或是至少簡化中間的結果。

如果我們的目標是分析整個軟體系

統，那麼一個 compositional and incremental analysis 是值得去實行的。首先，我們希望能避免任何考慮到的process數目上之固有的限制，而且一個 compositional analysis可能更具 scalability，若能提供此一分析，將可有助於軟體系統的階層化組織，進而

在 subsystem 限制中省去一些細節並簡化中間的結果。第二，當系統的重新分析只

需 循 著 單 一 subsystem 的 改 變 時 ，compositional analysis也能遞增式的降低重新工作的次數。

一個有效的 compositional analysis，即為一個有利於軟體系統的 modular structure ── 必須基於一種能擷取“剛好足夠＂的流程細節及 subsystem 之表達。在軟體系統中，modular structure 會在 interface of a module 後隱含內部的實作細節。相對地，compositional analysis只須維護在中間結果中具相關性的介面細節。我們將嘗試設計

具 明 確 效 益 的 compositional chopping analysis 技術，並以實驗來驗証其價值。 五、參考文獻 [1] J. Anderson and M. Gouda. “A Criterion

for Atomicity, ” Formal Aspects of Computing: The International Journal of Formal Methods , 4(3):273-298, May 1992.

[2] G.R. Andrews. “Concurrent Programming: Principles and Practice, ” Benjamin/Cummings Publishing Company, 1991.

[3] G.R. Andrews. “Foundations of Multithreaded, Parallel, and Distributed Programming,” Pearson Addison Wesley,

1999. [4] Cormac Flanagan , Shaz Qadeer “Types

for Atomicity” TLDI’03, January 18, 2003, New Orleans, Louisiana, USA.

[5]Cormac Flanagan , Shaz Qadeer “A Type and Effect System for Atomicity” PLDI’03, June 9–11, 2003, San Diego, California, USA.

[6]Susan Owicki and David Gries. “An Axiomatic Proof Technique for Parallel Programs I,” Acta Informatica, 6, 319-340, 1976.

[7]LihChyun Shu and Michal Young.

“Versioning Concurrency Control for Hard Real-Time System,” Journal of Systems and Software, 63(3):201-218, Sept. 2002 (SCI).

5

出席國際學術會議心得報告

計畫編號 NSC 94-2213-E-006 -091

計畫名稱 運用切割分析於驗證不可分割性之研究

出國人員姓名 服務機關及職稱

徐立群 國立成功大學會計學系

會議時間地點 2006 年 8 月 30 日至 2006 年 9 月 1 日

Grand Canyon, Arizona

會議名稱 第十二屆分散式多媒體系統國際會議出國報告

發表論文題目 Frame Dropping Control by Video Content Characteristics under Limited Bandwidth

一、參加會議經過

多媒體應用在我們日常生活愈來愈普遍，包括隨選視訊、視訊監控、影像電

話與即時新聞等。然而建構多媒體應用系統依然有許多問題待解決。本會議主要

專注在分散式多媒體系統 (distributed multimedia systems) 之相關技術、理論，

與應用，它提供學界與產業界分享最新研究成果很好的平台。主要的議題包括:

audio and video compression, MPEG, Quicktime, Windows API standards, image, video,

audio content analysis and indexing/retrieval, image, video and audio watermark, 3D audio

and video, computer graphics and animation, modeling and analysis of distributed

multimedia systems, OS support for distributed multimedia systems, distributed

multimedia databases and computing, Large Real-Time Multimedia Systems, Human-GIS

interaction, Media Streaming , Multimedia Representation and Indexing.

本次會議已是DMS的第十二屆，參加的學者來自全球有美國、義大利、日本、

印度、南韓、中國與台灣等國，主要以美國與亞洲居多，這個會議投稿的接受率

在近 4 年都維持在 50%以下，並具有獨立審查制度確保其論文品質，三天的會議

除了DMS主題外，還包含Visual Languages and Computing, Mobile Systems，

E-Commerce and Agent Technology，以及Distance Education Technology其他領域

的Workshop，發表的論文主要分兩種，其一是regular paper可有 6 pages空間發表

時間 25 分鐘，另一是short paper可有 4 pages空間發表時間 15 分鐘，而本人被接

受的論文屬regular paper。

這次大會安排兩場 keynote speech, 第一場由 Dr. Catherine Plaisant 主講，演講

主題為“Reaching Our Users＂。他提到他的研究團隊正發展一些有效的 visual

tools 以解決處理使用者的差異，比方年齡、語言，使用多樣的技術去解決與 user

之間知識落差，這是一項具挑戰但非常重要的研究。

第二場 keynote speech 由 Dr.Erland Jungert 主講，演講主題為“Using

Multi-Media to Spport Command and Control in Crisis Management Systems”。今天

人們居住的環境充斥著各種不同的危機，災難和緊急事件。因此我們需要一些工

具可以用來支援和模擬我們的 crisis management system。同時，為了形成更值得

信賴的命令和操作的系統架構，我們必須透過大量的 sensor 擷取 spatial/temporal

type (空間型態) 的資料。並分析，蒐集，處理，和 visualization。對於各種多媒

體的資料，都是吾人發展 crisis management system 所必須重視的。

Crisis management 通常用來處理實體設施的保護，而 crisis management 的

建置是起因於不同種類的要脅，傳統上，這類系統可視之為簡單的預警系統，比

方說，要脅成真的時候警告的機制就啟動，大部分的時候，預警對於要脅發生時

採取的可能動作仍然是有其限制和無能為力的，基於此種原因，一定要經過一段

較長的時間來形成足夠的 knowledge 以便提高採取可能的行動的能力， 而多媒

體正可幫助實踐此系統，並提高其實際模擬安全程度，比方說，多媒體可幫助我

們偵測異常行為的判斷。

本次會議有幾篇論文讓我們較印象深刻，以下我們作一報告。DMS session

I-A是有關於Coding的議題，其中一篇名為“H.264 Fast Encoder with Adaptive

Interpolation Based on Motion Detection Algorithm,” 由Gianluca Balio, Massimo

Bariani, Marco Raggio, Riccardo Stagnaro合著。H.264/MPEG-4 AVC 是目前最新

http://www.dis.unina.it/congress

的video編碼標準，是由ITU-T視頻編碼專家組（VCEG）和ISO/IEC運動圖像專家

組（MPEG）聯合組成的聯合視頻組（JVT，Joint Video Team）提出的高度壓縮數

位視頻編解碼器標準。而H.264 較現存已知的video編碼標準省下 50%的bit-rate，

但是需要額外的計算複雜度，而此篇提出一motion detection 演算法可以減少寫

入時間，其目的是依本篇提出的motion detection 演算法為基礎，結合ISO/IEC於

H.264 提出的 1/4 pel displacement resoiution 技術。

DMS session I-B 是有關於 Wireless Networking 的議題，其中一篇名為“An

Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless

Networks,” 由 Shahram Ghandeharizadeh, Tooraj Helmi, Taehee Jung, Shyam

Kapadia, Shahin Shayandeh 合著。本篇著墨在網狀網路的點對點設施下，對於

stream continuous 多媒體，audio、video clip 的 greedy data 置換策略。此 greedy data

置換策略，是要最大化被本地端服務的參照(reference)數目，作者分析二種方法

來實現此策略，分別為 frequency-based 和 byte-hit。此二種方法擁有相同的複雜

度和幾乎相同的建置，根據模擬的結果，byte-hit 比 frequency-based 表現的好，

其理由有二個，第一它可以最大化同時展現參照的 clips 的數目，第二它在存取

經常性上的表現有較低的錯誤，就我們研究來說，byte-hit 可以說是最佳的 greedy

data 置換策略。

本人所發表的論文被安排在第二天下午 3:15 DMS session VI，主題為

“Frame Dropping Control by Video Content Characteristics under The Limited

Bandwidth＂。目前較先進的多媒體視頻編碼為 MPEG-4 (H.264)，主要是以呈

現虛擬世界的影像處理為目標的技術，MPEG-4 規格將目標放在多種不同的傳

輸架構（區域存取，遠端互動，廣播及多點傳輸）及其資料傳輸技術。但考量

網路傳送的不確定性和頻寬的受限制性，如何在不影響播放的品質前提下

(Quality of Service)，可針對特定的視訊框(frame)採取捨棄的方法，是我們此次

發表論文的主要焦點。

DMS session VI 另一篇論文名為“Packet and Frame Rate Control Methods

http://zh.wikipedia.org/w/index.php?title=ITU-T&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=VCEG&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=ISO&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=IEC&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=%E6%95%B0%E5%AD%97%E8%A7%86%E9%A2%91&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=%E6%95%B0%E5%AD%97%E8%A7%86%E9%A2%91&variant=zh-cnhttp://zh.wikipedia.org/w/index.php?title=%E7%BC%96%E8%A7%A3%E7%A0%81%E5%99%A8&variant=zh-cn

for Continuous Media over Heterogeneous Environment by Wired and Wireless

Networks,” 由 Noriki Uchida, Xuanrui Xiong, kazoo Takahata , Yoshitaka Shibata 合

著。在無線網路和有線網路環境的架構下，探討 frame 和 packet 的遺失和延遲

比率的計算和控制機制，在無線網路採用 FEC 方法和 Reed-Solomon coding 來降

低 frame 和 packet 的遺失率，另一方面，當 frame 和 packet 的遺失和延遲比率

變高時，frame 傳送速率會改變，而我們可以選擇要的 frame 來傳送，如果將 packet

的遺失率保持在一個水準之下，我們可以確保維持整個 End-to-End 產出不會受

到太大影響。

DMS session VI 另一篇論文名為“Virtual Theater Network: Enabling

Large-Scale Peer-to-Peer Streaming Service,” 由 Masaru Okuda 與 Taieb Znati 合

著。對於大範圍、點對點的 streaming video distribution network 而言，必須要考

量三項問題(1)受限制的 peer 傳送頻寬 (2)有時間限制的 video frame 有效性 (3)

設計 p2p 連結的結構。本研究著眼於上述三個問題，而提出一個 video distribution

model 的混合架構，該架構介於 client-server 和 peer-to-peer 之間計算，在該 model

下，一個 video 可以被切割為一連串的 small segments ，它還採用一個 scheduling

scheme 來記錄不同來源的傳送速率，可以分配多餘的頻寬和空間給予那些低於

正常傳送速率的來源。

二、與會心得

分散式多媒體系統國際會議從 1994 年第一次在臺北舉行，當時主要參加者

大都為亞洲人士。直到今日歐、美、亞各國皆有研究者與會，受重視程度逐年遞

增，這反映出建構高品質多媒體系統吸引許多人高度的興趣，個人必須加緊努力

才能在激烈競爭的環境下作出成績。

三、建議

多媒體系統領域之研究有許多的華人參與，臺灣在此領域如要扮演主導角

色，有賴年輕有潛力的學者持續投入，並能作出夠份量的作品。另外更多國內產

官學研界的一同投入，才能在此重要領域達到國際領先的地位。

Frame Dropping Control by Video ContentCharacteristics under The Limited Bandwidth

Huey-Min Sun

Dept. of Information Management,Chang Jung Christian University,Tainan County, Taiwan 711, ROC

Email: prince@mail.cju.edu.tw

LihChyun Shu

Dept. of Accounting,National Cheng Kung University,Tainan City, Taiwan 701, ROCEmail: shulc@mail.ncku.edu.tw

Abstract— Due to the limited bandwidth and the unpredictablenetwork, the MPEG-4 video coding stream with Fine GranularityScalability (FGS) can be flexibly dropped by very fine granularityto adapt to the available network bandwidth. However, this is atrade-off problem between to drop the partial frames to reduceframe rate and to drop the partial packets of frames to keepframe rate. We propose the weighted assignment scheme inorder to dynamically control frame dropping by our schedulingalgorithm when the bandwidth is not sufficient. Our proposedscheme not only can integrate with our optimal schedulingalgorithm to maximize total important packets by the availablebandwidth but can also control frame dropping by video contentcharacteristics. In our experimental results, we analyze the framerate preference for video content characteristics under the limitedbandwidth. Based on the prior information for each test sequence,our frame dropping control scheme can be better than other fixedframe rate.

I. INTRODUCTIONDecoding and re-encoding a stream at a different frame rate

in real-time is not easy, particularly for MPEG and H.263schemes which employ motion estimation techniques. Motionestimation is an extremely time-consuming process, particu-larly if a full motion search algorithm is employed. Whenthe bandwidth is not sufficient, the sender cannot arbitrarilydrop frames in the compressed domain because decoding ofsubsequent frames at the receiver depends on them. If thesender is to re-encode the video stream at a different framerate, it would have to recompute the motion vectors.

One solution is the adaptation approaches which offer sub-optimal solutions to the problem of selecting the applicationsettings that better fulfill the users expectations. These schemesencode and store the sequence at different frame rates (say 30,24, 21, 18 fps and so on) to adapt the bandwidth requirement.This will take up a lot of space on the server. The secondsolution is the dropping approaches which sacrifice the unim-portant data to save the bandwidth resource. The method storesonly the largest frame rate (say 30 fps) to save a lot of spaceon the server, but it increases the complexity of scheduling todrop some frames under the insufficient bandwidth network.

In MPEG-4 Fine Granularity Scalability (FGS) framework,the basic video packets for the base layer are not alloweddata dropping because the data may be needed during the

decoding for the enhancement layer. The FGS framework withboth the temporal and the SNR scalabilities requires threeencoders: one for the base-layer and two for the enhancementlayer. The base layer can ensure the basic acceptable QoSrequirements of the user, while the enhancement layer canrefine the result. Therefore, the important data to user mustbe satisfied the acceptable requirement. The FGS with hybridtemporal-SNR scalability has been added to the MPEG-4video coding standard[1] in order to increase the flexibility ofvideo streaming. The temporal scalability can be cut anywherebefore transmission. The received part of the stream can besuccessfully decoded thus improving upon the basic videoquality. The SNR scalability[2] can code a video sequenceinto two layers at the same frame rate and the same spatialresolution, but with different quantization accuracy. A higheraccuracy discrete cosine transform (DCT) coefficient is ob-tained by adding the base layer reconstructed DCT coefficientand the enhancement layer DCT residue. The DCT coefficientswith a higher accuracy are given to the inverse DCT unitto produce reconstructed image domain residues that are tobe added to the motion-compensated block from the previousframe. When the amount of the data for the previous frame isconsiderably less than the predicted frame, the decoding willfail.

Our proposed method employs the dropping approach whichdiscards the packets of the lower weight by a greedy schedul-ing algorithm. We survey some real-time scheduling research,most of them consider the measurement of miss ratio thatcomputes the number of missing deadline tasks over thenumber of all tasks. The imprecise computation model inthis field is similar to MPEG-4 FGS framework. The FASTalgorithm proposed by Shih and Liu [3], [4] in the modelhas been proved with the optimization for the measurementof miss ratio. In our previous study [5], we found that thescheduling based on the earliest deadline first (EDF) policydoes not work during the decoding in MPEG-4 FGS becauseit does not consider whether the dropping data will affect thedecoding. Unfortunately, the FAST algorithm is based on theEDF. This results in the decoding failure. On the other hand,their algorithm does not consider the problem of adjusting the

frame rate when the bandwidth is not sufficient for transmittingall the packets. Our scheduling algorithm will dynamicallycontrol frame dropping by our weighted assignment schemefor the MPEG-4 FGS framework.

The paper is organized as follows. Section 2 presents therelated work. Section 3 states our system model and problems.We make use of the unit time tasks scheduling algorithmto solve the problem for maximizing total weights includingthe packets of the base layer in Section 4. Results of theexperiment are shown in Section 5. The conclusions are drawnin Section 6.

II. RELATED WORKS

A network with limited bandwidth may not be able todeliver all the multimedia requests and fulfill all the QoSrequirements. The dropping policy adopted by most researchdrops some data packets in order to satisfy the schedula-bility requirements. The dropping mechanism saves up thebandwidth utilization to satisfy the requests with the real-time constraints [6], [7], [8]. These researchers discover thatdiscarding the frames of incurring peak rate can save thebandwidth.

Furini and Towsley[6] propose several frame droppingmechanisms to reduce bandwidth consumption subject to aQoS constraint. The proposed algorithms are evaluated bythe JPEG and MPEG videos under the proposed bandwidthallocation mechanism.

Zhang et al.[7] propose an efficient selective frame discardalgorithm for stored video delivery across resource constrainednetworks. They attempt to minimize the discarded frames tomeet the constraints in JPEG videos.

Besides, reducing the frame rate can also save the bandwidthutilization. Pejhan et al.[9] propose a dynamic frame ratecontrol mechanism. The scheme is to encode and store thesequence at different frame rates. It stores only the motionvectors for the lower frame rates, but this will take up a lot ofspace on the server. This way all motion estimation is doneoff-line. When re-encoding, motion vectors can be read fromthe motion files instead of being computed. The advantageof the scheme is that the motion files are much smallerthan the corresponding compressed streams. The dynamicframe rate control, used in conjunction with dynamic bit-ratecontrol, allows clients to solve the rate mismatch between thebandwidth available to them and the bit-rate of the pre-encodedbitstream.

Song and Chun[10] present a virtual frame rate controlalgorithm and a bit allocation algorithm at frame level forefficient MPEG-2 video encoding. The proposed frame ratecontrol scheme is composed of three steps. At the first stage,the scan format of an input video sequence is convertedinto progressive scan format before video encoding. At thesecond stage, an average motion activity of the frames withina previous temporal window of a pre-defined size is examined,and a proper frame rate of a current temporal window isadaptively determined based on the computed average motionactivity. At the final stage, the frames located at particular

positions in a current temporal window are virtually skippedaccording to the determined frame rate. The scheme can skipthe selected frames by deliberately fixing the coding types ofall the macroblocks in those frames to skipped macroblocks.

Chan et al.[11] propose a structured Rate-Quantization baserate-control framework for low-delay video coding which con-tains three processing stages: the control of encoding frame-rate, bit-allocation for frame-level and the decision of quan-tization parameters for macroblock-level. At the first stage,the proposed framework decides the encoding frame-rate pergroup of pictures (GOPS) for better motion continuity. Atthe second stage, target-bit per frame is effectively estimatedwith the frame texture and buffer fullness. At the final stage,based on an adaptive Rate-Quantization model modified bythe Kalman filter for each cluster of macroblocks, a suitablequantization parameter of each macroblock can be confirmed.

Song et al.[12] propose a new H.263+ rate control schemewhich supports the variable bit rate (VBR) channel throughthe frame rate adjustment. In particular, a fast realizationof encoding frame rate control based on motion informationwithin a sliding window is developed to efficiently determinethe tradeoff between spatial and temporal qualities.

Yang and Hemami[13] propose a frame rate-control schemewith a MINMAX distortion criterion in the framework of arate-distortion (RD) optimized motion compensated embeddedwavelet coder. The MINMAX criterion aims to minimizethe maximum distortion in a group of pictures (GOP) fora given bit rate. The proposed frame level bit allocationscheme combined with RD optimized within frame allocationallows precise rate control up to the exact bit. A simplifyingfast algorithm is developed using an initial prediction ofthe operating distortion and adaptive adjustment during GOPcoding.

Lee and Kim[14] propose an adaptive video frame ratecontrol method for the network of time-varying rate channelwith explicit rate feedback. It consists of a prediction moduleof future channel rate and an adaptive frame discarding andskipping module. They derive an encoder buffer constraintwhich guarantees an end-to-end delay bound of video frames.Recursive Least-Squares(RLS) method is used as a tool topredict the low frequency component of channel rate. Theadaptive frame discarding method prevents delay violation offrames due to the channel rate prediction error. Also, the frameskipping method adapt the encoder output rate to the channelrate while keeping the constant level of video quality.

III. SYSTEM MODEL

A. System Model

Our system model based on MPEG-4 FGS framework is asFig. 1. The hybrid temporal-SNR scalability with an all FGSstructure supports both the temporal and the SNR scalabili-ties through a single enhancement layer[1]. Each multimediastreaming is made up of encoded frames such as Intra frame (Iframe), Bidirectional frame (B frame) and Predicted frame (Pframe). At the encoding time, the frame is coded using DCTfor compressing the base layer and the enhancement layer.

The encoded data for the enhancement layer form a numberof bitplanes. Every bitplane consists of 16× 16 macroblocks,and each macroblock includes four 8×8 luminance blocks andtwo chroma blocks. In this stage, the frame rate is set to 30fps. The sampling rate of the base layer is 3, i.e., one frame perthree frames is encoded. The sampling rate of the enhancementlayer is 1. The Group of Pictures (GoP) structure is setIBBPBBPBBPBB. The sampling rate of the enhancementfor the SNR scalability must depend on the base layer, sincethe frames I and P are with SNR scalability enhancement.The frame B is with temporal scalability enhancement.

Fig. 1. The system model based on MPEG-4 FGS framework

At the transmission time, the compressed bitstream inMPEG-4 is divided the frames into some video packets withthe same bit size in order to overcome the effect of atransmission error occurred in group of blocks. These packetsare classified into three parts illustrated by Fig. 2. A packetof the base layer is regarded as a mandatory task, and apacket of the enhancement layer for both the the temporaland the SNR scalabilities is regarded as an optional task.In the model, performing a trade-off between the SNR andthe temporal enhancement has to depend on an adaptivescheduling algorithm. We propose the weighted assignmentscheme which employs the characteristics for the sequencecontent such as the motion and the texture. Fig. 2 illustratesthe assignment example for keeping the original frame rate.The scheme can provide the priority of the packets for thescheduling algorithm to drop partial packets of the lowerpriority when the bandwidth is not sufficient.

At the decoding time, the packets of the base layer mustbe completely received. The packets of the enhancement layerfor the temporal scalability can smooth the consecutive frames,while the packets for the SNR scalability can refine the imagequality.

B. The problem of frame dropping control

We observe the relationship between the average bits forthe motion characteristic in group of pictures and the framerate. The higher motion characteristic the sequence suits withthe larger frame rate. Fig. 3 illustrates the relationship. Wealso observe the relationship between the average bits for the

Fig. 2. The weighted assignment principle

texture characteristic in group of pictures and the bit rate. Thehigher texture characteristic the sequence suits with the largerbit rate. Fig. 4 illustrates the relationship. We aim at adjustingthe adaptive frame rate according to our weighted assignmentscheme when the motion and the texture characteristics havebeen known in advance. In our system model, the frame ratefor each sequence is initially set to 30 fps. When the bandwidthis not sufficient, we drop the unimportant packets by ourscheduling algorithm to adapt to the available bandwidth. Weassign the weights to the packets for the video sequencesof the different content characteristics by the mean opin-ion scores(MOS) of the subjective measurement in advance.The scheduling problem consider not only maximizing totalweight, i.e, maximizing the number of important packets,but also controlling frame dropping under the constraint ofsatisfying the schedulability of the base layer.

Fig. 3. The relationship between the motion characteristic and the frame rate

Fig. 4. The relationship between the texture characteristic and the bit rate

IV. SCHEDULING ALGORITHM

A. The weighted assignment scheme

According to the prior information of the characteristicsfor the sequences, we can assign the weight to packets soas to control the frame dropping during the transmission. Forexample, we assign the weight to the packets of the framesshown in Fig. 5 when the sequence prefers 20 fps duringsome consecutive frames. The smaller number represents thehigher priority. To illustrate the scheme, we consider 12 framesas a GOP and denote 6 frames as a half of GOP, calledthe HGOP. Then, 30 frames are divided into 5 HGOPs. Ouroriginal frame rate is set to 30 fps, so the number of thedecoded frames is usually 20 fps if drops 2 frames per HGOP.When the bandwidth is not sufficient for transmitting most ofdata, the packets of the lower priority are always dropped.Therefore, the scheme can control the frame rate according todrop frames. Based on the weighted assignment scheme, theframe rate can be controlled by 30, 25, 20, 15, and 10 fps ifdrops 0, 1, 2, 3, and 4 frames per HGOP, respectively.

Fig. 5. The weighted assignment scheme for the preference of 20 fps

B. Satisfying the schedulability of the base layer

To solve the packets schedule problem, we employ a knownscheduling algorithm, called the unit time tasks scheduling.The problem of scheduling unit time tasks with deadlines andweights has the following inputs: a set T = T1, T2, . . . , Tnof n unit-time tasks; A set of n integer deadlines such thateach di satisfies 1 ≤ di ≤ n ; And a set of n nonnegativeweights w1, w2, . . . , wn such that wi is offered for task Tionly if it is finished before the time di. This well-definedproblem can find a schedule for T that maximizes the totalweight and meets deadlines. This has been proved using thegreedy algorithm of running time O(n2) to find the optimalschedule [15]. Table I shows an example of the unit time tasksscheduling. We use the unit time tasks scheduling algorithmto get the optimal schedule. Therefore, we can select tasks1, 2, 3, 4 and 6 according to the algorithm, and then rejecttask 5. This optimal schedule has a total weight of 275.

Table I. An example for the unit time tasks schedulingtask i 1 2 3 4 5 6

di 4 2 4 3 1 5wi 80 70 60 50 40 15

However, our problem is both to maximize the total weightand to satisfy the schedulability of the base layer. We definethe problem as follows.

Problem 1. Follow the above definitions, we consider thatthe set of tasks is classified into the mandatory set of tasks andthe optional set of tasks. This problem is to find a scheduleincluding all the mandatory tasks for T that maximizes thetotal weight and meets deadlines.

We make use of the unit time tasks scheduling algorithmto solve Problem 1 by the theorems which have been provedin our previous study[5]. Therefore, the set of the optimalsolution, S, found by the algorithm including the given set Mif there exists at least a solution including the given set M andthe weights of all the elements in M is added by a constant y.This problem is proved in Theorem 1. We define a new set T ′

based on the set T . The weights set T ′ is similar to the weightsset T except that the weights of all the elements in a givenset M , M ⊆ T , are added by a constant y where y is definedas

∑

t∈T

W (t) . The weight of tasks in M can be expressed as

W ′(t) = W (t) + y, if t ∈M , and W ′(t) = W (t), otherwise.Secondly, Problem 1 for the set T is equivalent to Problem

1 for the set T ′ in Theorem 2. This means OPT ′ = OPT +y ∗ |M | where OPT ′ and OPT are an optimal solution forProblem 1 in the set T ′ and T , respectively.

Finally, Problem 1 for the set T ′ is equal to Problem 1 inTheorem 3. Therefore, according to Theorem 2 and 3 underthe condition that there exists at least a solution including thegiven set M , this optimal solution of Problem 1 in the set T isequal to subtract a constant value, y ∗ |M |, from the solutionof Problem 1 in the set T ′.

Theorem 1: A set of the optimal schedule solution, S,found by the algorithm for the set T ′ exist that the given setM is belong to S if there exists at least a solution includingthe given set M in the set T .

Theorem 2: Problem 1 for set T is equivalent to Problem1 for set T ′, where W ′(t) = W (t) + y if t ∈ M otherwiseW ′(t) = W (t).

Theorem 3: The unit time scheduling problem for the setT ′ is equal to Problem 1 for the set T ′.

Theorem 4: The unit time scheduling problem for the setT ′ is equivalent to Problem 1 for the set T .

V. EXPERIMENT RESULTS

In our experiments, we use the Microsoft MPEG-4 softwareencoder/decoder with FGS functionality[16]. We encode thevideos using 30 frames per second with the CIF(352×288 pix-els) format. Every sequence has 300 frames, then the encodedstream of a frame is divided into hundreds of tasks(packets).The data size of each task is set to 64 bytes. The videos areprocessed in the Y UV format(Y is the luminance component,U and V are color components of a frame). The test sequenceafter encoding and scheduling under the limited bandwidth ismade up of the truncated encoded file. We compute the PSNRof luminance for each test sequence. The sequences employedin our evaluation are well-known MPEG-4 test sequences.

The sequences adopted for the analysis contain variousdegrees with both the motion and the texture characteristics.In the ”Akiyo” sequence, the motion characteristic is staticbackground and talking head, and the texture characteristicis easy. In the ”Foreman” sequence, the motion characteristiccontains high motion in the first part of the sequence andlow motion in the second part, and the texture characteristiccontains also easy in the first part and detailed in the secondpart. In the ”Stefan” sequence, the motion characteristic isvery high and the texture characteristic is relatively detailed.In the ”Mobile” sequence, the motion characteristic is slowand constant movement, and the texture characteristic is verydetailed.

We compute the average bits of all frames for the motioncharacteristic shown in Fig 6 and the average bits of Intraframes for the texture characteristic shown in Fig 7, respec-tively. In our encoded data, the ratios of the base layer forthe ”Akiyo”, the ”Foreman”, the ”Stefan”, and the ”Mobile”sequences are 0.03, 0.03, 0.07, and 0.04, respectively. Accord-ing to the information, our weighted assignment scheme canrecognize the video content characteristics. In our previousstudy[17], we evaluated the perceived quality by the doublestimulus continuous quality scale (DSCQS) method[18]to getthe preference frame rate for each test sequence. The mea-surement is based on the mean opinion scores(MOS) of therespondents. Fig. 8 illustrates that the sequences ”Akiyo” and”Mobile” prefer 10 fps and the sequences ”Foreman” and”Stefan” prefer 30 fps under the insufficient bandwidth.

0

1000

2000

3000

4000

5000

6000

Motion characteristic

Th

eav

era

ge

bit

so

ffr

am

es_

Akiyo

Foreman

Stefan

Mobile

Fig. 6. The average bits of all frames for motion characteristic

0

20000

40000

60000

80000

100000

120000

140000

Texture characteristic

Th

eav

era

ge

bit

so

fIn

tra-f

ram

es

_

Akiyo

Foreman

Stefan

Mobile

Fig. 7. The average bits of Intra frames for texture characteristic

We use the earliest deadline first (EDF) algorithm and theFAST algorithms as base lines to evaluate the performance

Fig. 8. The average mean opinion scores(MOS) by varying the frame ratefor each sequence

of transmitted packets. We implement the FAST algorithmby [3], [4] and our scheduling algorithm to analyze the lostpackets of the base layer. These algorithms can maximize thetotal number of packets. However, the EDF algorithm cannotdeal with the problem for satisfying the schedulability of allthe mandatory tasks. The FAST algorithm has been provedthe optimal scheduling for CPU in the imprecise computationmodel, but it may fail to decode in MPEG-4 FGS by ourprevious study[5]. Our optimal scheduling algorithm, namedOPT , has the optimal packets scheduling and guaranteesuccessfully decoding in MPEG-4 FGS. Table II, III, IV, andV show the lost ratios and the lost packets of the base layerby various available bandwidth for each sequence.

Table II. The lost packets of the base layer for the ”Akiyo”sequence

bandwidth(bps) 0.9M 1.0M 1.1M 1.2M 1.3Mlost ratio 0.75 0.72 0.70 0.67 0.64

EDF 0 0 0 0 0FAST/OPT 0 0 0 0 0

Table III. The lost packets of the base layer for the”Foreman” sequence

bandwidth(bps) 1.2M 1.4M 1.6M 1.8M 2.0Mlost ratio 0.83 0.80 0.78 0.75 0.72

EDF 161 79 16 0 0FAST/OPT 0 0 0 0 0

Table IV. The lost packets of the base layer for the ”Stefan”sequence

bandwidth(bps) 2.4M 2.6M 2.8M 3.0M 3.2Mlost ratio 0.76 0.74 0.72 0.69 0.67

EDF 328 111 53 0 0FAST/OPT 0 0 0 0 0

Table V. The lost packets of the base layer for the ”Mobile”sequence

bandwidth(bps) 3.8M 4.0M 4.2M 4.4M 4.6Mlost ratio 0.72 0.71 0.70 0.68 0.67

EDF 218 3 0 0 0FAST/OPT 0 0 0 0 0

To understand the effects of image quality for varying theframe rate, we show the relationship for each sequence in

30

33

36

39

42

45

48

10 15 20 25 30

Frame rate (fps)

PS

NR

(dB

) Akiyo

Foreman

Stefan

Mobile

Fig. 9. The average PSNR by varying the frame rate for each sequence

Fig. 9. The figure illustrates that reducing the frame ratecan increase the image quality when the bandwidth is notsufficient. However, the smoothing quality of consecutiveframes cannot be measured by the PSNR.

We adopt the subjective video quality assessment to measureour performance. Table VI illustrates that the perceived qualitybased on the subjective measurement for our scheme can bebetter than other fixed frame rates when the bandwidth is notsufficient. This is because our scheme can control the framedropping to adapt the preferred frame rate of video content.

Table VI. The analysis of effectiveness for our schemesequences Akiyo Foreman Stefan Mobile

MOS for our scheme 67.5 49.5 43.2 52.3MOS for fixed 15 fps 65.2 42.7 35.6 49.9MOS for fixed 20 fps 60.6 43.2 38.5 46.8MOS for fixed 25 fps 60.6 46.6 40.5 45.2

VI. CONCLUSION

We have proposed the weighted assignment scheme tocontrol frame dropping for MPEG-4 FGS framework when thebandwidth is not sufficient. Our scheduling algorithm not onlycan guarantee the schedulability of the base layer in MPEG-4FGS framework but can also maximize total important packetswhen the available bandwidth is sufficient for all the packetsof the base layer.

Our experimental result shows that our scheme can controlthe frame dropping to correspond to the preference for eachtest sequence. The mean opinion scores for our frame droppingscheduling algorithm can be better than other scheduling forthe fixed frame rate.

Our weighted assignment scheme and the scheduling algo-rithm can drop the partial frames or keep all the frames as faras possible to adapt to various video content characteristics.The frame dropping scheduling mechanism contributes toMPEG-4 FGS framework under the limited bandwidth.

ACKNOWLEDGEMENT

This work was supported by the National Science Councilof the R.O.C under Contract NSC 94-2213-E-309-008.

REFERENCES[1] Radha, H.M., Schaar, V.D.M., and Chen, Yingwei. (2001) The MPEG-

4 fine-grained scalable video coding method for multimedia streamingover IP. IEEE Trans. on Multimedia, Vol. 3, No.1, Page(s):53-68.

[2] Li, Weiping.(2001) Overview of fine granularity scalability in MPEG-4video standard, IEEE Transactions on Circuits and Systems for VideoTechnology, 2001, vol 11, issue 3, Page(s): 301-317.

[3] Liu, J.W.-S., Lin, K.-J., Shih,W.-K., Yu, A.C.-S., Chung, C., Yao,J., and Zhao, W., (1991) Fast Algorithms for Scheduling ImpreciseComputations, Computer, Volume:24, No. 5, May 1991, Page(s).58-68.

[4] Shih, W.-K., Liu, J.W.S., and Chung, J.-Y. (1989) Fast algorithms forscheduling imprecise computations ,Real Time Systems Symposium,1989,Page(s): 12 -19.

[5] Huey-Min Sun, Lychun Shu, and Chia-Mei Chen; (2005) OptimizingVideo Stream Scheduling Based on MPEG-4 FGS Framework, The 18thIPPR conference on computer vision, graphics, and image processing,2005, 21-23 Aug., Taiwan, Page(s):1475-1482.

[6] Furini, M. and Towsley, D.F. (2001) Real-time traffic transmissions overthe Internet. IEEE Transactions on Multimedia, Volume: 3 Issue: 1, p.33- 40.

[7] Zhi-Li Zhang, Nelakuditi, S., Aggarwal, R., and Tsang, R.P.(1999)Efficient selective frame discard algorithms for stored video deliveryacross resource constrained networks. Proceedings of INFOCOM ’99,Volume: 2, p.472 -479.

[8] Tsou,Fu-Ming., Chiou, Hong-Bin., Tsai, and Zsehong. (2000) A novelATM traffic scheduler for real-time multimedia data transport withimproved packet level QOS. Proceedings of ICME 2000, Volume: 2, p.1047 -1050.

[9] Sassan Pejhan, Ti-hao Chiang, Ya-Qin Zhang. (1999) Dynamic FrameRate Control for Video Streams, Oct 30 - Nov 5, 1999, Orlando, Florida,ACM Multimedia, Electronic Proceedings

[10] Byung Cheol Song; Kang Wook Chun;(2003) A virtual frame rate con-trol algorithm for efficient MPEG-2 video encoding, IEEE Transactionson Consumer Electronics, Volume 49, Issue 2, May 2003 Page(s):460 -465.

[11] Din-Yuan Chan; Chun-Yuan Chang; Shou-Jen Lin; Chung-WeiLin;(2004) A rate-control scheme using fast frame-rate controlling andadaptive rate-quantization modeling for H.263, 2004 IEEE InternationalSymposium on Consumer Electronics, Sept. 1-3, 2004, Page(s):390 -395.

[12] Hwangiun Song; Jongwon Kim; Jay Kuo, C.-C.; (1999) Real-timeH.263+ frame rate control for low bit rate VBR video, Proceedingsof the 1999 IEEE International Symposium on Circuits and Systems,ISCAS ’99. Volume 4, 30 May-2 June 1999 Page(s):307 - 310.

[13] Yan Yang; Hemami, S.S.; (1999) MINMAX frame rate control using arate-distortion optimized wavelet coder, International Conference onIm-age Processing, 1999. ICIP 99. Volume 3, 24-28 Oct. 1999, Page(s):551- 555.

[14] Myeong-Jin Lee; Jae-Kyoon Kim; (2000) Video frame rate controlfor non-guaranteed network services with explicit rate feedback, IEEETelecommunications Conference, 2000. GLOBECOM ’00. Volume 1,27 Nov.-1 Dec. 2000 Page(s):293 - 297.

[15] Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest,(1993) Introduction to Algorithms, The MIT Press, McGraw-Hill BookCompany, 1993.

[16] Microsoft.(2004)ISO/IEC 14496 MPEG-4 Video Reference Software,Version: Microsoft-FDAM1-2.5-040207.

[17] Huey-Min Sun, Yung-Chuan Lin.(2005) The Effect of Video Qualityfor Different Contents, Frame Rates and Bit Rates, National ComputerSymposium 2005, Tainan, Taiwan, Dec. 15-16, 2005.

[18] M. Pinson and S. Wolf. (2003) Comparing Subjective Video QualityTesting Methodologies, SPIE Video Communications and Image Pro-cessing Conference, Lugano, Switzerland, July, 2003.