S03 hybrid app_and_gae_datastore_v1.0

Smart TV

http://www.softwareinlife.com

Hybrid Appand

GAE DataStore

대표이사 장선진@jangsunjin

- Mobile Phone: (+82) 010-4585-1770- E-Mail: jangsunjin@softwareinlife.com

MapReduce

• MapReduce is a patented software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers.

http://www.cs.berkeley.edu/~ballard/cs267.sp11/hw0/results/htmls/Muzaffar.html

CAP and ACID• CAP 

– 정합성 (Consistency). The client perceives that a set of opera-tions has occurred all at once.

– 가용성 (Availability). Every operation must terminate in an in-tended response.

– 단절내성 (Partition tolerance). Operations will complete, even if individual components are unavailable.

• ACID– 원자성 (Atomicity). All of the operations in the transaction will

complete, or none will.– 일관성 (Consistency). The database will be in a consistent state

when the transaction begins and ends.– 고립성 (Isolation). The transaction will behave as if it is the only

operation being performed upon the database.– 지속성 (Durability). Upon completion of the transaction, the op-

eration will not be reversed.http://queue.acm.org/detail.cfm?id=1394128

Google BigTable System

Tablet Server #1

MapReduce Worker #1

GFS #1

Tablet Server #2

MapReduce Worker #2

GFS #2

BigTable Master Server

Chubby Server

Google BigTable Architecture

GAE Application

DataNucleus ORM(JDO or JPA Implementation)

Low Level API

Datastore

GFS(Google File System)

Big

Table

Meta 0 tablet

Meta 1 tablet Meta 1 tablet Meta 1 tablet

Tab 1 Tab 2 Tab 3 Tab 4 Tab n

Google BigTable• Dynamic control over

Data Layout and Format• Not support a full Rela-

tional Model• Dynamic control over

serving data from mem-ory or disk

• Column oriented• The map is indexed by

– A row key– A column name– A timestamp

BigTable

Key

TimeStamp / Version

Columns

Index

LDAP(Lightweight Directory Access Protocol)

• The Lightweight Directory Access Protocol (LDAP) is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) net-work.

• Directory services may provide any organized set of records, often with a hierarchical structure, such as a corporate electronic mail directory. Similarly, a telephone directory is a list of sub-scribers with an address and a phone number.

Google App Engine Product Roadmap• Features on Deck

– App Engine out of Preview– SSL access on custom domains– Full-text search over Datastore– Support for Python 2.7– Support for running MapReduce jobs across App En-

gine datasets– Bulk Datastore Import and Export tool– Improved monitoring and alerting of application serving– Logging system improvements to remove limits on size and

storage– Integration with Google Storage for Developer

DataStore 의 제약사항• Write 가 느림• 검색 가능한 Text 사이즈 : 500 자• Entity 최대 사이즈 : 1Mb• 최대 Batch (save/delete) : 500 Entity• SQL 을 사용할 수 없음• 데이터베이스의 제약조건을 적용할 수 없음• Aggregation 함수를 지원하지 않음 (count, avg)• 쿼리가 최대 조회 레코드 : 2000• 쿼리에서 Equal 연산 이외의 조건은 1 개만 추가 가능• 트랜잭션의 단위는 Entity Group 에 한정 됨

Datastore API and ORM API

Data-store API

Java ORM API

(JDO, JPA)

Datastore API(Low Level API)• 모든 Entity 는 Kind 가 지정• Entity 는 하나이상의 Property 를 포함• 특정 Kind 에 포함된 Entity 는 동일한 Property 를 갖지 않음• Entity 의 Property 는 이름은 동일하지만 데이터 유형이 다를수 있음• Schemaless 의 특성• ID 는 지정하지 않을 경우 자동 할당 됨

Object-Oriented RDBMS Datastore

Class Table Kind

Object Record Entity

Attribute Column Property

Entity• App Engine Datastore 에서 관리하는 객체는 Entity• Entity 는 하나의 Key 를 포함

– 모든 Entity 중 유일한 구분자• Key 의 구성

– Path– Parent Entity 의 key– Entity 의 Kind– Entity 에 할당된 이름

• App 이 할당한 값• Datastore 가 지정한 numeric ID

• 지원 데이터 타임– integers, floating point values, strings, dates, binary data 등

• 각 엔티티는 하나이상의 Property 를 포함– 하나의 Property 에 복수의 데이터 타입 저장 가능– 하나의 Property 에는 복수의 값이 저장 가능– 복수의 값들의 데이터 유형은 다를 수 있음

Entity Model• Settings

– Kind : Table– Key : Primary Key– Entity Group : Partitioning– Property : column

Kind Person

key /Person:you

Entity Group /Person:you

Name helloworld

age 25

Entity Example(Create)• import java.util.Date;

import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;

// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();

Entity employee = new Entity("Employee");employee.setProperty("firstName", "Antonio");employee.setProperty("lastName", "Salieri");Date hireDate = new Date();employee.setProperty("hireDate", hireDate);employee.setProperty("attendedHrTraining", true);

datastore.put(employee);

Entity Example(Scheme Free)• import java.util.Date;

import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;

// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();

Entity employee = new Entity("Employee");employee.setProperty("firstName", “Sun-Jin");

datastore.put(employee);

Entity Group• Entity Group 는 부모 Entity 를 갖는 Entity 의 Group

• 하나의 트랜잭션에서 여러 Entity 를 조회 , 생성 , 수정 , 삭제 할 수 가능하지만 그 범위는 단일 Entity Group 에 국한

• 부모 Entity 를 가지 않은 Entity 를 Root Entity 라고 함

• Entity Group 은 하나의 Root Entity 와 Child Entity 로 구성

• 하나의 Entity Group 은 동일한 분산 네트웍 서버에 저장됨 (Tablet 서버 )

Entity Group

Root Entity(xxxx)

Child Entity(xxxxyyyy)

Child Entity(xxxxyyyyzzzz)

Entity Group

Entity Example(Read)• import java.util.Date;

import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;

// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();

• Key rootKey = KeyFactory.createKey("Employee", “CEO");

• Entity employee = datastore.get(rootKey);

Entity Example(Update)• import java.util.Date;

import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;

// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();

• Key rootKey = KeyFactory.createKey("Employee", “CEO");

• Entity employee = datastore.get(rootKey);

employee.setProperty("firstName", “Sun-Jin");employee.setProperty("lastName", “Jang");

datastore.put(employee);

Entity Example(Delete)• import java.util.Date;

import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;

// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();

• Key rootKey = KeyFactory.createKey("Employee", “CEO");

• Entity employee = datastore.get(rootKey);

datastore.delete(employee);

Entity Example(Query)•

// ...DatastoreService datastore = DatastoreServiceFactory.getDatastore-Service();

• Query q = new Query(”employee") • q.addFilter("lastName", Query.FilterOperator.EQUAL,

lastNameParam); • q.addFilter("height", Query.FilterOperator.LESS_THAN, maxHeight-

Param);

• PreparedQuery pq = datastore.prepare(q);

• for (Entity result : pq.asIterable()) { • String firstName = (String) result.getProperty("firstName"); • String lastName = (String) result.getProperty("lastName"); • Long height = (Long) result.getProperty("height"); • }

Java ORM API

JDO(Java Data

Object)

JPA(Java Per-sistence

API)

DataNucleus

• DataNucleus attempts to make the whole process of persisting data a transparent process.

• The idea revolves around the developer having a series of Java classes that need persisting.

http://www.datanucleus.org/products/accessplatform/development_process.html

JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;

@PersistenceCapablepublic class Employee {    @PrimaryKey    @Persistent(valueStrategy = IdGeneratorStrategy.I-DENTITY)    private Key key;

•     @Persistent    private String firstName;

    @Persistent    private String lastName;

    @Persistent    private Date hireDate; }

JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;

// …

@PersistenceCapablepublic class Employee {    @PrimaryKey    @Persistent(valueStrategy = IdGeneratorStrategy.IDEN-TITY)    private Key key;

•     @Persistent    private String firstName;

    @Persistent    private String lastName;

    @Persistent    private Date hireDate; }

JDO Model(Create)•         PersistenceManager pm =

PMF.get().getPersistenceManager();

        Employee e = new Employee("Alfred", "Smith", new Date());

        try {            pm.makePersistent(e);        } finally {            pm.close();        }

JDO Model(Update)• public void updateEmployeeTitle(User user, String

newTitle) {    PersistenceManager pm = PMF.get().getPersistenceManager();    try {        Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);        if (titleChangeIsAuthorized(e, newTitle) {            e.setTitle(newTitle);        } else {            throw new UnauthorizedTitleChangeExcep-tion(e, newTitle);        }    } finally {        pm.close();    }}

JDO Model(Delete)• public void updateEmployeeTitle(User user, String

newTitle) {    PersistenceManager pm = PMF.get().getPersistenceManager();    try {        Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);

•        pm.deletePersistent(e);     } finally {        pm.close();    }}

JDO Model(Query)• import java.util.List;

import javax.jdo.Query;

// ...

    Query query = pm.newQuery(Employee.class);    query.setFilter("lastName == lastNameParam");    query.setOrdering("hireDate desc");    query.declareParameters("String lastNameParam");

    try {        List<Employee> results = (List<Employee>) query.execute("Smith");        if (!results.isEmpty()) {            for (Employee e : results) {                // ...            }        } else {            // ... no results ...        }    } finally {        query.closeAll();    }

Demo

감사합니다 .

http://code.google.com/p/devmento-gae-seminar