Embed Size (px)
Citation preview
Making Postgres Central in Your Data Center
BRUCE MOMJIAN
January, 2015
This talk explores why Postgres is uniquely capable of functioningas a central database in enterprises. Title concept from Josh BerkusCreative Commons Attribution License http://momjian.us/presentations
1 / 44
PostgreSQL the database…
◮ Open Source Object Relational DBMS since 1996
◮ Distributed under the PostgreSQL License
◮ Similar technical heritage as Oracle, SQL Server & DB2
◮ However, a strong adherence to standards (ANSI-SQL 2008)
◮ Highly extensible and adaptable design
◮ Languages, indexing, data types, etc.◮ E.g. PostGIS, JSONB, SQL/MED
◮ Extensive use throughout the world for applications andorganizations of all types
◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOSand Amazon Linux
Making Postgres Central in Your Data Center 2 / 44
PostgreSQL the community…
◮ Independent community led by a Core Team of six
◮ Large, active and vibrant community
◮ www.postgresql.org
◮ Downloads, Mailing lists, Documentation
◮ Sponsors sampler:
◮ Google, Red Hat, VMWare, Skype, Salesforce, HP andEnterpriseDB
◮ http://www.postgresql.org/community/
Making Postgres Central in Your Data Center 3 / 44
EnterpriseDB the company…
◮ The worldwide leader of Postgres based products andservices founded in 2004
◮ Customers include 50 of the Fortune 500 and 98 of theForbes Global 2000
◮ Enterprise offerings:
◮ PostgreSQL Support, Services and Training◮ Postgres Plus Advanced Server with Oracle Compatibility◮ Tools for Monitoring, Replication, HA, Backup & Recovery
Community
◮ Citizenship
◮ Contributor of key features: Materialized Views, JSON, &more
◮ Nine community members on staff
Making Postgres Central in Your Data Center 4 / 44
EnterpriseDB the company…
Making Postgres Central in Your Data Center 5 / 44
Outline
1. Object-Relational (extensibility)
2. NoSQL
3. Data analytics
4. Foreign data wrappers (database federation)
5. Central role
Making Postgres Central in Your Data Center 6 / 44
1. Object-Relational (Extensibility)
Object-relational databases like Postgres support support classesand inheritance, but most importantly, they define databasefunctionality as objects that can be easily manipulated.http://en.wikipedia.org/wiki/Object-relational_database
Making Postgres Central in Your Data Center 7 / 44
How Is this Accomplished?
starelid
staattnum
staop
pg_statistic
oprleft
oprright
oprresult
oprcom
oprnegate
oprlsortop
oprrsortop
oprcode
oprrest
oprjoin
pg_operator
typrelid
typelem
typinput
typoutput
typbasetype
pg_type
prolang
prorettype
pg_proc
pg_rewrite
ev_class
datlastsysoid
pg_database
tgfoid
tgrelid
pg_trigger
inhrelid
pg_inherits
inhparent
pg_language
pg_namespacepg_depend pg_shadow
pg_aggregate
aggfinalfn
aggtransfn
aggfnoid
aggtranstype
castsource
casttarget
pg_cast
castfunc
pg_description
pg_constraint
contypid
pg_conversion
conproc
amopopr
amopclaid
pg_attribute
indexrelid
attnum
amopclaid
atttypid
indrelid
pg_attrdef
pg_group
adrelid
pg_index
adnum
pg_am
pg_amop
amgettuple
reltoastidxid
aminsert
reltoastrelid amcostestimate
amproc
ambeginscan
pg_amproc
amrescan
relfilenode
amendscan
relam
ammarkpos
reltype amrestrpos
pg_class
ambuild
opcdeftype
ambulkdelete
pg_opclass
attrelid
http://www.postgresql.org/docs/current/static/catalogs.html
Making Postgres Central in Your Data Center 8 / 44
Example: ISBN Data Type
CREATE EXTENSION isn;
\dTList of data types
Schema | Name | Description--------+--------+--------------------------------------------------public | ean13 | International European Article Number (EAN13)public | isbn | International Standard Book Number (ISBN)public | isbn13 | International Standard Book Number 13 (ISBN13)public | ismn | International Standard Music Number (ISMN)public | ismn13 | International Standard Music Number 13 (ISMN13)public | issn | International Standard Serial Number (ISSN)public | issn13 | International Standard Serial Number 13 (ISSN13)public | upc | Universal Product Code (UPC)
http://www.postgresql.org/docs/current/static/isn.html
Making Postgres Central in Your Data Center 9 / 44
ISBN Behaves Just Like Built-In Types
\dTS…pg_catalog | integer | -2 billion to 2 billion integer, 4-byte storage…public | isbn | International Standard Book Number (ISBN)
Making Postgres Central in Your Data Center 10 / 44
The System Catalog Entry for INTEGERSELECT * FROM pg_type WHERE typname = ’int4’;-[ RECORD 1 ]--+---------typname | int4typnamespace | 11typowner | 10typlen | 4typbyval | ttyptype | btypcategory | Ntypispreferred | ftypisdefined | ttypdelim | ,typrelid | 0typelem | 0typarray | 1007typinput | int4intypoutput | int4outtypreceive | int4recvtypsend | int4sendtypmodin | -typmodout | -typanalyze | -typalign | itypstorage | ptypnotnull | ftypbasetype | 0 Making Postgres Central in Your Data Center 11 / 44
The System Catalog Entry for ISBNSELECT * FROM pg_type WHERE typname = ’isbn’;-[ RECORD 1 ]--+---------------typname | isbntypnamespace | 2200typowner | 10typlen | 8typbyval | ttyptype | btypcategory | Utypispreferred | ftypisdefined | ttypdelim | ,typrelid | 0typelem | 0typarray | 16405typinput | isbn_intypoutput | public.isn_outtypreceive | -typsend | -typmodin | -typmodout | -typanalyze | -typalign | dtypstorage | ptypnotnull | ftypbasetype | 0 Making Postgres Central in Your Data Center 12 / 44
Not Just Data Types, Languages
CREATE EXTENSION plpythonu;
\dLList of languages
Name | Owner | Trusted | Description-----------+----------+---------+------------------------------------------plpgsql | postgres | t | PL/pgSQL procedural languageplpythonu | postgres | f | PL/PythonU untrusted procedural language
http://www.postgresql.org/docs/current/static/plpython.html
Making Postgres Central in Your Data Center 13 / 44
Available Languages
◮ PL/Java
◮ PL/Perl
◮ PL/pgSQL (like PL/SQL)
◮ PL/PHP
◮ PL/Python.
◮ PL/R (like SPSS)
◮ PL/Ruby
◮ PL/Scheme
◮ PL/sh
◮ PL/Tcl
◮ SPI (C)
http://www.postgresql.org/docs/current/static/external-pl.html
Making Postgres Central in Your Data Center 14 / 44
Specialized Indexing Methods
◮ BTree
◮ Hash
◮ GiST (generalized search tree)
◮ SP-GiST (space-partitioned GiST)
◮ GIN (generalized inverted index)
http://www.postgresql.org/docs/current/static/indexam.html
Making Postgres Central in Your Data Center 15 / 44
Index Types Are Defined in the System Catalogs Too
SELECT amname FROM pg_am;amname--------btreehashgistginspgist
http://www.postgresql.org/docs/current/static/catalog-pg-am.html
Making Postgres Central in Your Data Center 16 / 44
Operators Have Similar Flexibility
Operators are function calls with left and right arguments of specified types:
\doSSchema | Name | Left arg type | Right arg type | Result type | Description
…
pg_catalog | + | integer | integer | integer | add
\dfSSchema | Name | Result data type | Argument data types | Type
…pg_catalog | int4pl | integer | integer, integer | normal
Making Postgres Central in Your Data Center 17 / 44
Other Extensibility
◮ Casts are defined in pg_cast, int4(float8)
◮ Aggregates are defined in pg_aggregate, sum(int4)
Making Postgres Central in Your Data Center 18 / 44
Externally Developed Plug-Ins
◮ PostGIS (Geographical Information System)
◮ PL/v8 (server-side JavaScript)
◮ experimentation, e.g. full text search was originallyexternally developed
Making Postgres Central in Your Data Center 19 / 44
Offshoots of Postgres
◮ AsterDB
◮ Greenplum
◮ Informix
◮ Netezza
◮ ParAccel
◮ Postgres XC
◮ Redshift (Amazon)
◮ Truviso
◮ Vertica
◮ Yahoo! Everest
https://wiki.postgresql.org/wiki/PostgreSQL_derived_databaseshttp://de.slideshare.net/pgconf/elephant-roads-a-tour-of-postgres-forks
Making Postgres Central in Your Data Center 20 / 44
Offshoots of Postgres
https://raw.github.com/daamien/artwork/master/inkscape/PostgreSQL_timeline/timeline_postgresql.png
Making Postgres Central in Your Data Center 21 / 44
Plug-In Is Not a Bad Word
Many databases treat extensions as special cases, with seriouslimitations. Postgres built-ins use the same API as extensions, so llextensions operate just like built-in functionality.
Making Postgres Central in Your Data Center 22 / 44
Extensions and Built-In FacilitiesBehave the Same
ExtensionsPL/R
ISN
PostGIS
Postgres System Tables
int4
btree
sum()
PL/pgSQL
Making Postgres Central in Your Data Center 23 / 44
2. NoSQL
SQL
Making Postgres Central in Your Data Center 24 / 44
NoSQL Types
There is no single NoSQL technology. They all take differentapproaches and have different features and drawbacks:
◮ Key-Value stores, e.g. Redis
◮ Document databases, e.g. MongoDB (JSON)
◮ Columnar stores: Cassandra
◮ Graph databases: Neo4j
Making Postgres Central in Your Data Center 25 / 44
Why NoSQL Exists
Generally, NoSQL is optimized for:
◮ Fast querying
◮ Auto-sharding
◮ Flexible schemas
Making Postgres Central in Your Data Center 26 / 44
NoSQL Sacrifices
◮ A powerful query language
◮ A sophisticated query optimizer
◮ Data normalization
◮ Joins
◮ Referential integrity
◮ Durability
Making Postgres Central in Your Data Center 27 / 44
Are These Drawbacks Worth the Cost?
◮ Difficult Reporting Data must be brought to the client foranalysis, e.g. no aggregates or data analysis functions.Schema-less data requires complex client-side knowledge forprocessing
◮ Complex Application Design Without powerful querylanguage and query optimizer, the client software isresponsible for efficiently accessing data and for dataconsistency
◮ Durability Administrators are responsible for data retention
Making Postgres Central in Your Data Center 28 / 44
When Should NoSQL Be Used?
◮ Massive write scaling is required, more than a single servercan provide
◮ Only simple data access pattern is required
◮ Additional resource allocation for development is acceptable
◮ Strong data retention or transactional guarantees are notrequired
◮ Unstructured duplicate data that greatly benefits fromcolumn compression
Making Postgres Central in Your Data Center 29 / 44
When Should Relational Storage Be Used?
◮ Easy administration
◮ Variable workloads and reporting
◮ Simplified application development
◮ Strong data retention
Making Postgres Central in Your Data Center 30 / 44
The Best of Both Worlds: Postgres
Postgres has many NoSQL features without the drawbacks:
◮ Schema-less data types, with sophisticated indexing support
◮ Transactional schema changes with rapid additional andremoval of columns
◮ Durability by default, but controllable per-table orper-transaction
◮ Postgres XC and PL/Proxy allow auto-sharding for writescaling
Making Postgres Central in Your Data Center 31 / 44
Schema-Less Data: JSON
CREATE TABLE customer (id SERIAL, data JSON);
INSERT INTO customer VALUES (DEFAULT, ’{"name" : "Bill", "age" : 21}’);
SELECT data->’name’ FROM customer WHERE (data->’age’)::text = ’21’;?column?----------"Bill"
Making Postgres Central in Your Data Center 32 / 44
Easy Relational Schema Changes
ALTER TABLE customer ADD COLUMN status CHAR(1);BEGIN WORK;ALTER TABLE customer ADD COLUMN debt_limit NUMERIC(10,2);ALTER TABLE customer ADD COLUMN creation_date TIMESTAMP WITH TIME ZONE;ALTER TABLE customer RENAME TO cust;COMMIT;
Making Postgres Central in Your Data Center 33 / 44
3. Data Analytics
◮ Aggregates
◮ Optimizer
◮ Server-side languages, e.g. PL/R
◮ Window functions
◮ Bitmap heap scans
◮ Tablespaces
◮ Data partitioning
◮ Materialized views
◮ Common (recursive) table expressions
◮ Min/Max indexes (coming in 2015)
Data warehouse-specific solutions are required for paralleloperations across servers.http://www.slideshare.net/PGExperts/really-big-elephants-postgresql-dw-15833438http://wiki.postgresql.org/images/3/38/PGDay2009-EN-Datawarehousing_with_PostgreSQL.pdf
Making Postgres Central in Your Data Center 34 / 44
Read-Only Slaves for Analytics
Network
Data WarehouseMaster Server
/pg_xlog/pg_xlog
Making Postgres Central in Your Data Center 35 / 44
4. Foreign Data Wrappers (Database Federation)
Foreign data wrappers (SQL MED) allow queries to read and writedata to foreign data sources. Foreign database support includes:
◮ CouchDB
◮ Informix
◮ MongoDB
◮ MySQL
◮ Neo4j
◮ Oracle
◮ Postgres
◮ Redis
The transfer of joins, aggregates, and sorts to foreign servers is not yetimplemented.http://www.postgresql.org/docs/current/static/ddl-foreign-data.htmlhttp://wiki.postgresql.org/wiki/Foreign_data_wrappers
Making Postgres Central in Your Data Center 36 / 44
Foreign Data Wrappers to Interfaces
◮ JDBC
◮ LDAP
◮ ODBC
Making Postgres Central in Your Data Center 37 / 44
Foreign Data Wrappers toNon-Traditional Data Sources
◮ Files
◮ HTTP
◮ AWS S3
Making Postgres Central in Your Data Center 38 / 44
Foreign Data Wrapper Example
CREATE SERVER postgres_fdw_test
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host ’localhost’, dbname ’fdw_test’);
CREATE USER MAPPING FOR PUBLIC
SERVER postgres_fdw_test
OPTIONS (password ’’);
CREATE FOREIGN TABLE other_world (greeting TEXT)
SERVER postgres_fdw_test
OPTIONS (table_name ’world’);
\detList of foreign tablesSchema | Table | Server--------+-------------+-------------------public | other_world | postgres_fdw_test(1 row)
Foreign Postgres server name in red; foreign table name in blue
Making Postgres Central in Your Data Center 39 / 44
Read and Read/Write Data Sources
Postgres
ora_tab
tw_tab
mon_tabMongoDB
Oracle
Making Postgres Central in Your Data Center 40 / 44
5. Postgres Centrality
Postgres can rightly take a central place in the data center withits:
◮ Object-Relation flexibility and extensibility
◮ NoSQL-like workloads
◮ Powerful data analytics capabilities
◮ Access to foreign data sources
No other database has all of these key components.
Making Postgres Central in Your Data Center 41 / 44
Postgres’s Central Role
Extensions
NoSQL
Postgres
WarehouseData
Foreign DataWrappers
Window Functions
Data Paritioning
Bitmap Scans Sharding
Oracle
MongoDB
Easy DDL
JSON
PL/R
ISN
PostGIS
Making Postgres Central in Your Data Center 42 / 44
Additional Resources…
◮ Postgres Downloads:
◮ www.enterprisedb.com/downloads
◮ Product and Services information:
Making Postgres Central in Your Data Center 43 / 44
Conclusion
http://momjian.us/presentations http://flickr.com/photos/vpickering/3617513255
Making Postgres Central in Your Data Center 44 / 44
