In-memory Database

Nam.DinhSecurity Researcher / Developer

Agenda

1. NoSQL

2. NoSQL Type

3. Alternative PHP cache

4. Memcached

5. Redis

6. Demo

7. References

1. NoSQL

● It may not require fixed table schemas, their data models are simpler

● Easy to make changes in real-time, development is faster

● Less database administrator time is needed.

● High Availability

● Vertical scalable

● Horizontal scalable

● Distributed Data

2.NoSQL Type

● Key-value stores.

A key-value store is a system that stores values indexed for retrieval by keys.

2.NoSQL Type (cont..)● Column- oriented databases.

Column-oriented databases contain one extendable column of closely related data

It is recommended for analytical systems

● Document-based stores.

These databases store and organize data as collections of documents

● Graph databases.

http://en.wikipedia.org/wiki/Graph_database

More: http://nosql-database.org/

3. Alternative PHP Cache ( APC )

● By implementing APC you cut down on repeat PHP script executions, skipping the parsingand compiling steps. APC stores the opcode,compiled php code and that is simply executedeach time the script is called again.

● Opcode discarded affter page has been executed.

● APC allow you to cache variables for speed and can cache files.

● APC is great when you need local caching of objects for your PHP scripts that are relativelysmall and frequently accessed.

API: http://php.net/manual/en/book.apc.php

Install: http://sam.bitmorse.com/posts/install-and-enable-alternative-php-cache-apc-on-ubuntu-12.04-lts/

4. What is Memcached?● Memcached is developed by Brad Fitzpatrick for LiveJournal in 2003 and now used by

Netlog, Facebook, Flickr, Wikipedia, Twitter, YouTube etc.

● Free & open source, high-performance, distributed memory object caching system, generic innature, but intended for use in speeding up dynamic web applications by alleviating databaseload.

● It will use its own memory manager system called the slab allocator.

● A distributed memory object caching system with hash table.

● Use non-blocking network I/O.

● Least Recently Used will be removed.(LRU)

● Not a persistent data store .

● It does not provide redundancy.

● Data is not replicated across the cluster.

● It doesn’t handle failover.

● Scale-out : machine, process.

● Key : max 250 chars.

● Values: max 1MB.

API : http://php.net/manual/en/book.memcached.php

Install: apt-get install php-memcached memcached

4. What is Memcached? (cont...)

5. REDIS

Remote Dictionary Server

Agenda• About Redis

• Redis data structures

• Key operations

• String operations

• Hash operations

• List operations

• Sort operations

• Sorted set operations

• Sort

• Transaction

• Persistent

• Pattern

Redis• Redis is a very fast non-relational database and often described as an in-

memory store, high performance.

• Redis supports in-memory persistent storage on disk.

• Like memcached, Redis can also store a mapping of keys to values and caneven achieve similar performance levels as memcached. But Redis supportmore 4 data structures: hash, list, set and sorted set.

• Support for atomic operations.

• Support for transactions, replication.

• Single threaded.

API:

https://github.com/phpredis/phpredis

Redis data structures

Key operatons• KEYS pattern : Find all keys matching the given pattern

• EXISTS key : Determine if a key exists

• TYPE key : Determine the type stored at key

• EXPIRE key seconds : Set a key's time to live in seconds

• EXPIREAT key timestamp : Set the expiration for a key as a UNIX timestamp

• TTL key : Get the time to live for a key

• PEXPIRE key milliseconds : Set a key's time to live in milliseconds

• PEXPIREAT key milliseconds-timestamp : Set the expiration for a key as a UNIX timestampspecified in milliseconds

• PTTL key : Get the time to live for a key in milliseconds

• PERSIST key : Remove the expiration from a key

• RENAMENX key newkey : Rename a key, only if the new key does not exist

• DEL key : Delete a key

String operatons • SET key value : Set the string value of a key.(A value can't be bigger than 512 MB)

• GET key : Get the value of a key

• MGET key [key ...] : Get the values of all the given keys

• MSET key value [key value ...] : Set multiple keys to multiple values

• APPEND key value : Append a value to a key

• STRLEN key : Get the length of the value stored in a key

• GETRANGE key start end : Get a substring of the string stored at a key

• INCR key : Increment the integer value of a key by one

• DECR key : Decrement the integer value of a key by one

• DECRBY key decreament : Decrement the integer value of a key by the given number

• INCRBY key increment : Increment the integer value of a key by the given amount

• INCRBYFLOAT key increment : Increment the float value of a key by the given amount

List operatons

• LPUSH key value [value ...] : Prepend one or multiple values to a list

• RPUSH key value [value ...] : Append one or multiple values to a list

• LSET key index value : Set the value of an element in a list by its index

• LINSERT key BEFORE|AFTER pivot value : Insert an element before or afteranother element in a list

• LLEN key : Get the length of a list

• LRANGE key start stop : Get a range of elements from a list

• LINDEX key index : Get an element from a list by its index

• LPOP key : Remove and get the first element in a list

• RPOP key : Remove and get the last element in a list

• LTRIM key start stop : Trim a list to the specified range

• LREM key count value : Remove elements from a list

Hash operatons• HSET key field value : Set the string value of a hash field

• HGET key field : Get the value of a hash field

• HMSET key field value [field value ...] : Set multiple hash fields to multiple values

• HMGET key field [field ...] : Get the values of all the given hash fields

• HEXISTS key field : Determine if a hash field exists

• HLEN key : Get the number of fields in a hash

• HKEYS key : Get all the fields in a hash

• HVALS key : Get all the values in a hash

• HGETALL key : Get all the fields and values in a hash

• HINCRBY key field increment : Increment the integer value of a hash field by the given number

• HINCRBYFLOAT key field increment : Increment the float value of a hash field by the givenamount

• HDEL key field [field ...]: Delete one or more hash fields

Set operatons• SADD key member [member ...] : Add one or more members to a set

• SCARD key : Get the number of members in a set

• SMEMBERS key : Get all the members in a set

• SISMEMBER key member : Determine if a given value is a member of a set

• SMOVE source destination member : Move a member from one set to another

• SINTER key [key ...] : Intersect multiple sets

• SUNION key [key ...] : Add multiple sets

• SDIFF key [key ...] : Subtract multiple sets

• SREM key member [member ...] : Remove one or more members from a set

Sorted set operatons 1• ZADD key score member [score member ...] : Add one or more members to a sorted set, or

update its score if it already exists

• ZCARD key :Get the number of members in a sorted set

• ZRANK key member : Determine the index of a member in a sorted set

• ZREVRANK key member : Determine the index of a member in a sorted set, with scoresordered from high to low

• ZRANGE key start stop [WITHSCORES] : Return a range of members in a sorted set, byindex

• ZREVRANGE key start stop [WITHSCORES] : Return a range of members in a sorted set,by index, with scores ordered from high to low

• ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] : Return a rangeof members in a sorted set, by score

Sorted set operatons 2• ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] : Return a

range of members in a sorted set, by score, with scores ordered from high to low

• ZREMRANGEBYRANK key start stop : Remove all members in a sorted set within thegiven indexes

• ZREMRANGEBYSCORE key min max : Remove all members in a sorted set within thegiven scores

• ZINTERSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]][AGGREGATE SUM|MIN|MAX] : Intersect multiple sorted sets and store the resultingsorted set in a new key

• ZUNIONSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]][AGGREGATE SUM|MIN|MAX] : Add multiple sorted sets and store the resulting sorted setin a new key

• ZREM key member [member ...] : Remove one or more members from a sorted set

Example Sorted set

Sort• Returns or stores the elements contained in the list, set or sorted set at key

- SORT mylist [ALPHA] DESC | ASC LIMIT offset count

• Sorting by external keys.

• With the STORE option, the result will be stored as a list at the specified key instead of being returnedto the client.

- SORT …. STORE resultkey

Transactons• All the commands in a transaction are serialized and

executed sequentially, atomically.

• It can never happen that a request issued by another client isserved in the middle of the execution of a Redis transaction

• MULTI : Mark the start of a transaction block

• EXEC : All commands are executed after EXEC, block andreturn values for the commands as a list.

• DISCARD : Transactions can be discard by this command

• WATCH : Watched keys are monitored in order to detectchanges against them. If at least one watched key is modifiedbefore the EXEC command, the whole transaction aborts,and EXEC returns a Null multi-bulk reply to notify that thetransaction failed.

Persistent

1. The RDB persistence

- Point-in-time snapshots at specified intervals

- Disaster recovery, not good to minimize the chance of data loss

2. The AOF persistence

- Log every operations, played again at startup.

Patterns

1.Simple, Object Store

- Very fast, high write throughput.

- Atomic manipulation of object members:counter...

2. Each object is saved as a HASH.

•

3. Object Indexing

7. Demo● APC

● Memcached and Redis

8. References

1.http://php.net/manual/en/book.apc.php

2.http://www.mongodb.com/nosql-explained

3.https://www.adayinthelifeof.nl/2011/02/06/memcache-internals/

4.http://memcached.org/

5.http://redis.io/

6.http://stackoverflow.com/questions/2377273/how-does-an-interpreter-compiler-work

7.http://www.intracto.com/nl/blog/testdriving-zend-optimizer

Q&A

● What ?● How ?● When ?● Why ?