Upload
sun-jin-jang
View
5.790
Download
0
Embed Size (px)
DESCRIPTION
Google BigTable and GAE Datastore
Citation preview
Smart TV
http://www.softwareinlife.com
Hybrid Appand
GAE DataStore
대표이사 장선진@jangsunjin
- Mobile Phone: (+82) 010-4585-1770- E-Mail: [email protected]
MapReduce
• MapReduce is a patented software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers.
http://www.cs.berkeley.edu/~ballard/cs267.sp11/hw0/results/htmls/Muzaffar.html
CAP and ACID• CAP
– 정합성 (Consistency). The client perceives that a set of opera-tions has occurred all at once.
– 가용성 (Availability). Every operation must terminate in an in-tended response.
– 단절내성 (Partition tolerance). Operations will complete, even if individual components are unavailable.
• ACID– 원자성 (Atomicity). All of the operations in the transaction will
complete, or none will.– 일관성 (Consistency). The database will be in a consistent state
when the transaction begins and ends.– 고립성 (Isolation). The transaction will behave as if it is the only
operation being performed upon the database.– 지속성 (Durability). Upon completion of the transaction, the op-
eration will not be reversed.http://queue.acm.org/detail.cfm?id=1394128
Google BigTable System
Tablet Server #1
MapReduce Worker #1
GFS #1
Tablet Server #2
MapReduce Worker #2
GFS #2
BigTable Master Server
Chubby Server
Google BigTable Architecture
GAE Application
DataNucleus ORM(JDO or JPA Implementation)
Low Level API
Datastore
GFS(Google File System)
Big
Table
Meta 0 tablet
Meta 1 tablet Meta 1 tablet Meta 1 tablet
Tab 1 Tab 2 Tab 3 Tab 4 Tab n
Google BigTable• Dynamic control over
Data Layout and Format• Not support a full Rela-
tional Model• Dynamic control over
serving data from mem-ory or disk
• Column oriented• The map is indexed by
– A row key– A column name– A timestamp
BigTable
Key
TimeStamp / Version
Columns
Index
LDAP(Lightweight Directory Access Protocol)
• The Lightweight Directory Access Protocol (LDAP) is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) net-work.
• Directory services may provide any organized set of records, often with a hierarchical structure, such as a corporate electronic mail directory. Similarly, a telephone directory is a list of sub-scribers with an address and a phone number.
Google App Engine Product Roadmap• Features on Deck
– App Engine out of Preview– SSL access on custom domains– Full-text search over Datastore– Support for Python 2.7– Support for running MapReduce jobs across App En-
gine datasets– Bulk Datastore Import and Export tool– Improved monitoring and alerting of application serving– Logging system improvements to remove limits on size and
storage– Integration with Google Storage for Developer
DataStore 의 제약사항• Write 가 느림• 검색 가능한 Text 사이즈 : 500 자• Entity 최대 사이즈 : 1Mb• 최대 Batch (save/delete) : 500 Entity• SQL 을 사용할 수 없음• 데이터베이스의 제약조건을 적용할 수 없음• Aggregation 함수를 지원하지 않음 (count, avg)• 쿼리가 최대 조회 레코드 : 2000• 쿼리에서 Equal 연산 이외의 조건은 1 개만 추가 가능• 트랜잭션의 단위는 Entity Group 에 한정 됨
Datastore API and ORM API
Data-store API
Java ORM API
(JDO, JPA)
Datastore API(Low Level API)• 모든 Entity 는 Kind 가 지정• Entity 는 하나이상의 Property 를 포함• 특정 Kind 에 포함된 Entity 는 동일한 Property 를 갖지 않음• Entity 의 Property 는 이름은 동일하지만 데이터 유형이 다를수 있음• Schemaless 의 특성• ID 는 지정하지 않을 경우 자동 할당 됨
Object-Oriented RDBMS Datastore
Class Table Kind
Object Record Entity
Attribute Column Property
Entity• App Engine Datastore 에서 관리하는 객체는 Entity• Entity 는 하나의 Key 를 포함
– 모든 Entity 중 유일한 구분자• Key 의 구성
– Path– Parent Entity 의 key– Entity 의 Kind– Entity 에 할당된 이름
• App 이 할당한 값• Datastore 가 지정한 numeric ID
• 지원 데이터 타임– integers, floating point values, strings, dates, binary data 등
• 각 엔티티는 하나이상의 Property 를 포함– 하나의 Property 에 복수의 데이터 타입 저장 가능– 하나의 Property 에는 복수의 값이 저장 가능– 복수의 값들의 데이터 유형은 다를 수 있음
Entity Model• Settings
– Kind : Table– Key : Primary Key– Entity Group : Partitioning– Property : column
Kind Person
key /Person:you
Entity Group /Person:you
Name helloworld
age 25
Entity Example(Create)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
Entity employee = new Entity("Employee");employee.setProperty("firstName", "Antonio");employee.setProperty("lastName", "Salieri");Date hireDate = new Date();employee.setProperty("hireDate", hireDate);employee.setProperty("attendedHrTraining", true);
datastore.put(employee);
Entity Example(Scheme Free)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
Entity employee = new Entity("Employee");employee.setProperty("firstName", “Sun-Jin");
datastore.put(employee);
Entity Group• Entity Group 는 부모 Entity 를 갖는 Entity 의 Group
• 하나의 트랜잭션에서 여러 Entity 를 조회 , 생성 , 수정 , 삭제 할 수 가능하지만 그 범위는 단일 Entity Group 에 국한
• 부모 Entity 를 가지 않은 Entity 를 Root Entity 라고 함
• Entity Group 은 하나의 Root Entity 와 Child Entity 로 구성
• 하나의 Entity Group 은 동일한 분산 네트웍 서버에 저장됨 (Tablet 서버 )
Entity Group
Root Entity(xxxx)
Child Entity(xxxxyyyy)
Child Entity(xxxxyyyyzzzz)
Entity Group
Entity Example(Read)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
Entity Example(Update)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
employee.setProperty("firstName", “Sun-Jin");employee.setProperty("lastName", “Jang");
datastore.put(employee);
Entity Example(Delete)• import java.util.Date;
import com.google.appengine.api.datastore.DatastoreService;import com.google.appengine.api.datastore.DatastoreService-Factory;import com.google.appengine.api.datastore.Entity;
// ...DatastoreService datastore = DatastoreServiceFactory.get-DatastoreService();
• Key rootKey = KeyFactory.createKey("Employee", “CEO");
• Entity employee = datastore.get(rootKey);
datastore.delete(employee);
Entity Example(Query)•
// ...DatastoreService datastore = DatastoreServiceFactory.getDatastore-Service();
• Query q = new Query(”employee") • q.addFilter("lastName", Query.FilterOperator.EQUAL,
lastNameParam); • q.addFilter("height", Query.FilterOperator.LESS_THAN, maxHeight-
Param);
• PreparedQuery pq = datastore.prepare(q);
• for (Entity result : pq.asIterable()) { • String firstName = (String) result.getProperty("firstName"); • String lastName = (String) result.getProperty("lastName"); • Long height = (Long) result.getProperty("height"); • }
Java ORM API
JDO(Java Data
Object)
JPA(Java Per-sistence
API)
DataNucleus
• DataNucleus attempts to make the whole process of persisting data a transparent process.
• The idea revolves around the developer having a series of Java classes that need persisting.
http://www.datanucleus.org/products/accessplatform/development_process.html
JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;
@PersistenceCapablepublic class Employee { @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.I-DENTITY) private Key key;
• @Persistent private String firstName;
@Persistent private String lastName;
@Persistent private Date hireDate; }
JDO Model Define(Annotation)• import javax.jdo.annotations.PersistenceCapable;
// …
@PersistenceCapablepublic class Employee { @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.IDEN-TITY) private Key key;
• @Persistent private String firstName;
@Persistent private String lastName;
@Persistent private Date hireDate; }
JDO Model(Create)• PersistenceManager pm =
PMF.get().getPersistenceManager();
Employee e = new Employee("Alfred", "Smith", new Date());
try { pm.makePersistent(e); } finally { pm.close(); }
JDO Model(Update)• public void updateEmployeeTitle(User user, String
newTitle) { PersistenceManager pm = PMF.get().getPersistenceManager(); try { Employee e = pm.getObjectById(Employee.class, “Sun-Jin”); if (titleChangeIsAuthorized(e, newTitle) { e.setTitle(newTitle); } else { throw new UnauthorizedTitleChangeExcep-tion(e, newTitle); } } finally { pm.close(); }}
JDO Model(Delete)• public void updateEmployeeTitle(User user, String
newTitle) { PersistenceManager pm = PMF.get().getPersistenceManager(); try { Employee e = pm.getObjectById(Employee.class, “Sun-Jin”);
• pm.deletePersistent(e); } finally { pm.close(); }}
JDO Model(Query)• import java.util.List;
import javax.jdo.Query;
// ...
Query query = pm.newQuery(Employee.class); query.setFilter("lastName == lastNameParam"); query.setOrdering("hireDate desc"); query.declareParameters("String lastNameParam");
try { List<Employee> results = (List<Employee>) query.execute("Smith"); if (!results.isEmpty()) { for (Employee e : results) { // ... } } else { // ... no results ... } } finally { query.closeAll(); }
Demo
감사합니다 .
http://code.google.com/p/devmento-gae-seminar