Upload
enterprisedb
View
102
Download
4
Embed Size (px)
DESCRIPTION
Quite often, developers and managers find themselves in a situation where their database isn't performing well. Due to a lack of understanding of inner workings of the databases, they try to solve the problem using unrelated solutions. In this talk we will walk through the various operations a database system (Postgres, Oracle, and others) perform while serving the application queries. We will also see how those operations relate to the operations done by OS, filesystem, and the storage disks.
Citation preview
© 2013 EDB All rights reserved. 1
How Databases Work
• Gurjeet Singh | 09.18.14
© 2013 EDB All rights reserved. 2
Database Architect @ EDB– Index Advisor
– Optimizer Hints
– Postgres Enterprise Manager
– Postgres Plus Cloud Database
Postgres Contributor– Postgres Hibernator and others
Learnt From Mistakes
Who Am I
© 2013 EDB All rights reserved. 3
Impart some new knowledge, that helps you do yourjob better.
(The “Full Monty” slides is where the meat of the talkis)
Goal
© 2013 EDB All rights reserved. 4
• ACID
• Application Interaction• Backend Operations• Concurrent/Cooperating Operations
• OS/Disk Interaction• NoSQL
Agenda
© 2013 EDB All rights reserved. 5
ACID
© 2013 EDB All rights reserved. 6
ACID
Traits of a transaction in an RDBMS• Atomicity
– Changes done by a transaction are one unit; all ornothing.
• Consistency– Transaction end leaves database in a consistent
state; constraints cannot be violated.• Isolation
– Transactions behave as if only one transaction isin progress at a time.
• Durability– Transactions, once committed, will not be lost.
© 2013 EDB All rights reserved. 7
NoSQL
© 2013 EDB All rights reserved. 8
NoSQL
• We are not going to discuss NoSQL databases.• They compromise on at least one of ACID properties
© 2013 EDB All rights reserved. 9
Application Interaction
© 2013 EDB All rights reserved. 10
Application Interaction• Applications use a database driver• The driver uses a custom protocol (language)• Enables Client-Server/n-Tier architectures
Java
C/C++
JavaScript
Python
Ruby
© 2013 EDB All rights reserved. 11
Backend Operations
© 2013 EDB All rights reserved. 12
Backend Operations
• Query Processing– Tokenize/Parse
– Parse Analysis
– Plan/Optimize
– Execute
© 2013 EDB All rights reserved. 13
Backend Operations – Tokenize/Parse
SELECT e.id, d.name
FROM emp e JOIN dept d
ON e.dept = d.id
WHERE emp.name = 'Agent Smith';
SELECT e . id , d . name
FROM emp e JOIN dept d
ON e . dept = d . id
WHERE emp . name = 'Agent Smith' ;
© 2013 EDB All rights reserved. 14
Backend Operations – Tokenize/Parse
'Agent Smith'e.name
WHERE ListFROM ListSELECT List
=deptempd.namee.id
Parse Tree
ON
d.ide.dept
=
© 2013 EDB All rights reserved. 15
Backend Operations – Parse Analysis
• Ensure that– The objects (tables, columns, functions, etc.) exist
– Resolve/expand views• Type Coercion
© 2013 EDB All rights reserved. 16
Backend Operations – Plan/OptimizeGenerate all possible plans to access objects
• Compare cost of– Accessing tables using various methods
● Full Table Scan, Index Scan, Bitmap Index, etc.– Joining tables using various methods
● Nested Loop join, Hash Join, Sort-Merge Join– Other operations like distinct, ordering, grouping.
• Choose the best plan
• Best may mean
– Fastest (e.g. LIMIT clause)
– Cheapest
© 2013 EDB All rights reserved. 17
Backend Operations – Plan/OptimizePlan/Execution Tree
Index Scan - emp
Nested Loop
Result
Condition: name ='Agent Smith'
Full Table Scan - dept
Filter: e.dept =d.id
To Client
© 2013 EDB All rights reserved. 18
Backend Operations - Execute
• Traverse the Execution Tree recursively• Pull rows from each execution node, until exhausted• Insert/Update rows
– Delete in Postgres is just an update of row header
– Update is similar to Delete + Insert
– Multi-Version Concurrency Control (MVCC)• Send result/rows to the client
© 2013 EDB All rights reserved. 19
Concurrent/CooperatingOperations
© 2013 EDB All rights reserved. 20
Concurrent/Cooperating Operations
• Read data from disk• Share data
• Locks to prevent concurrent update• Write data to disk
© 2013 EDB All rights reserved. 21
Block Structure
Free Space
Block Header
Row Pointers
Row Header
© 2013 EDB All rights reserved. 22
MVCC
Isolation; A transaction will:• Ignore changes done by newer transactions• Ignore changes done by running transactions
• Ignore changes done by rolled-back transactions• Only “see” changes by older and committed
transactions
Effectively:
• Readers don't block writers• Writers don't block readers
© 2013 EDB All rights reserved. 23
MVCC
Provides• Atomicity; with the help of commit log (clog)• Isolation
Consistency
• Ensured by the individual backends
© 2013 EDB All rights reserved. 24
Full MontyClient 1 Client 2 Client 3 Client n
Network
Backend 1 Backend 2 Backend n Postmaster
Checkpointer
Autovacuum
RAM
Disk Cache
Shared Buffers
WAL Buffers
Controller Cache Controller Cache
BGWriter
© 2013 EDB All rights reserved. 25
Full Monty
• WAL – Write Ahead Log– Record change before the actual change
– Sequential, always growing infinite log• Checkpoint
– Flush dirty data blocks to database
– Truncate WAL; prevent infinite growth• Autovacuum
– Clean up dead row versions• BGWriter
– Write dirty blocks so backends don't have to
© 2013 EDB All rights reserved. 26
Be Curious
Thank You