1/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Experiences running sks-keyservers.netNorwegian Unix User’s Group

Kristian Fiskerstrand0x0B7F8B60E3EDFAE3

9 September 2014

2/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Table of Contents

1 A brief introduction to OpenPGP

2 sks-keyservers.net

3 Code base

4 Keyserver network and community

3/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Introduction I

B PGP is an abbreviation for “Pretty Good Privacy” and wasoriginally created by Phil Zimmermann in 1991.

B OpenPGP is a framework for cryptographic operationsdefined as a standard by IETF and described in RFC4880and RFC6637 (Elliptic Curves)

B The most popular implementation of OpenPGP forGNU/Linux environments is the GNU Privacy Guard(GnuPG), but alternatives exists both as free softwareand commercially (e.g. PGP).

4/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Introduction II

B OpenPGP is a hybrid scheme using– Asymmetric keys for integrity, authentication and

encryption of a random session key per message.– Symmetric encryption is used to ensure confidentiality

of the data itself.B A default key generated by GnuPG today contains:

– A primary key capable of Certification and Signing– An encryption capable subkey.– At least one User ID (e.g. “Kristian Fiskerstrand<kf@sumptuouscapital.com>”)

5/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Table of Contents

1 A brief introduction to OpenPGP

2 sks-keyservers.net

3 Code base

4 Keyserver network and community

6/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Introduction I

B http://sks-keyservers.net provide a convenient way forend users of OpenPGP to retrieve and update keys fromsynchronised and responsive HKP keyservers

B The project was started in 2006B The servers are mainly based on SKS (p. 9).B Several pools of keyservers are available

pool. Main pool (on port 11371)hkps.pool. TLS enabled pool (HKPS) (more

on p. 14){NA,EU,OC}.pool. Geographical pools (more on p. 13)p80.pool. Firewall friendly, servers

responding on port 80.

7/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Introduction II

subset.pool. Supports latest functionalities.Currently SKS 1.1.5 only; ensuringsupport for e.g. Elliptic Curve keysc.f. RFC6637.

ha.pool. High availability. Historicallyincluded only instances that wererunning behind reverse proxies, butsince making that a hardrequirement for the main pool it iscurrently not different - howeverkeeping it and intend to re-use forclustered setups.

8/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Introduction III

B These pools are functioning as a DNS Round RobinB Currently ∼ 110 servers included in the main pool and

another ∼ 60 detected but rejected (inclusion criteria onp. 12)

B The pools are accessible using both IPv4 and IPv6 (withsome special pools limited to a single stack, e.g.ipv{4,6}.pool.sks-keyservers.net)

B GnuPG defaults to keys.gnupg.net which is a DNSCNAME to pool.sks-keyservers.net

9/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Quick intro to keyservers, protocols etc I

B Synchronizing Key Server (SKS) is the most usedkeyserver today.

B One notable feature of SKS is the implementation of thefast reconciliation algorithms that were part of YaronMinksy’s PhD thesis.

B SKS replaced PKS that synchronized using email and didnot include the updated features of OpenPGP presented inRFC4880.

B SKS is written in objective caml, and the maindevelopment team is Yaron, John Clizbe and myself.Contributions are welcome.

10/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Quick intro to keyservers, protocols etc II

B The keyservers in the pool is accessed based on theHorowitz Key Protocol (HKP). Although alternativeimplementations exists (e.g. PGP Corp’s keyserver is usingLDAP)

B HKP is a layer on top of HTTP defining how to access thekeyserver. It was never formally accepted as a standardbut the basis is found in http://tools.ietf.org/html/draft-shaw-openpgp-hkp-00.

B Developments since the initial draft is based on communityconsensus, mainly between GnuPG and SKS as referenceimplementations.

B Alternative keyservers still exists, including Hockeypuckthat is written in Go and that has preliminary support forSKS synchronization.

11/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Quick intro to keyservers, protocols etc III

B Some HKP examples http://pool.sks-keyservers.net:11371Stats request:/pks/lookup?op=stats − operationVerbose index:/pks/lookup?op=vindex&search=kf@sumptuouscapital.com − searchparametersGet:/pks/lookup?op=get&options=mr − Machine readable output&search=0x0b7f8b60e3edfae3

12/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Inclusion criteria

B Server responds to HKP queriesB Number of keys can’t contain less than median number of

keys – ∆ where ∆ is calculated during a two-stage process.First excluding outliers (diff exceeding ±0.5σ) then∆ = Max (0.5σ, 300) of the recalculated σ

B Upgraded to a compatibility level of SKS 1.1.3 or higher.B Server name reported is a fully qualified domain nameB HTTP/1.1 POST does not result in HTTP status 417B Reverse Proxy (providing via header or based on server

header)B Server is not in the global exclude listB Responds to port 80 as well as 11371 (for p80 pool)B Not vulnerable to CVE-2014-3207 (for HKPS pool)

13/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Geographical pools

(1) During an update run, determinethe response time from multipleclient measuring stations (groupedby pool) as the mean of N attemptsand the bandwidth capacity foreach individual server.

(2) Calculate the mean (µ) andstandard deviation (σ) of theaforementioned parameters acrossthe servers in the pool.

(3) Exclude outliers to the results basedon N (σ) rejection criteria.

(4) Calculate new µ and σ for theremainder of the servers.

(5) Calculate the SRV weights for theindividual servers.

(6) Sort the servers by weights and picktop 10 for each individual pool

SRV weights are defined as:

weight = α

+ Min(βR

( 1(R − (µR − 2σR))y

), φ

)

+ βB

Bm∑

j=1Bj

+ βP ·DP · ρ

(1)

where:α is a constant to provide a basis weightβi is the loading (weight) of the respective factor indetermining the SRV weightDP is a dummy variable that is 1 if a reverse proxy existsand 0 otherwiseρ is the additional weight given for a reverse proxy enabledservery is a constant (even number) that define penalization fordeviationφ is the ceiling of weight to be added for R

Further information is available at http://kfwebs.com/sks-keyservers-SRV.pdf

14/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

HKPS Pool I

B HKPS is regular HKP over TLS (port 443)B In order to achieve such a setup across multiple

independent servers, all participants in the pool needs acertificate signed by the sks-keyservers.net CA (SCA)

B The certificates contains the CN of the individual serversand include the pool hostnames as subjectAltNames.

B The SCA issued certificate needs to be issued whenever thehkps SNI is presented, but several operators also maintaintheir own certificates from the global PKIX system fortheir individual hostname.

15/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

HKPS Pool II

B The public SCA cert is available at https://sks-keyservers.net/sks-keyservers.netCA.pem andis signed by my OpenPGP key in a detached signature with.pem.asc suffix.

B The Cetificate Revocation List (CRL) is published athttps://sks-keyservers.net/ca/crl.pem

B the SCA cert is also included in GnuPG:http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=dirmngr/sks-keyservers.netCA.pem;hb=refs/heads/master

16/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

HKPS Pool III

B More info onhttps://sks-keyservers.net/verify_tls.php

B To use the HKPS pool require the following configuration:

∼/.gnupg/gpg.conf for GnuPG 1.4 and 2.0 brancheskeyserver hkps://hkps.pool.sks-keyservers.netkeyserver-optionsca-cert-file=/path/to/CA/sks-keyservers.netCA.pem

17/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Global reachDespite not having any data on actual usage, as this is handledby the slave DNS servers, visitors on the website indicate aglobal reach (based on last months visitors, the number of usersof the service is expected to be higher than the number ofvisitors of the website)

18/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

SRV compatibility issues

B However, some issues still remain in GnuPG:– Issue 1446: hkps SRV lookup discards port from SRV– Issue 1447: TLS hostname selection uses insecure SRV

dataB Until these are fixed and been present in standard versions

for a while, a large-scale use of SRV records will not beused in the pool (it is currently disabled for everythingexcept for the geographical pools)

B This restricts e.g. the possibility of using HKPS pool onnon-standard port (restricted to 443)

B Note: Even when clients are expected to handle it, it mightmake sense to keep the restriction, e.g. due to firewalls.

19/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Table of Contents

1 A brief introduction to OpenPGP

2 sks-keyservers.net

3 Code base

4 Keyserver network and community

20/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

The code base

B The code base consists of PHP (∼ 14k LOC (excludingembedded libraries)) and bash

B Open Source:https://code.google.com/p/sks-keyservers-pool/

B A crawler is executed once an hour to update the data andis performing a recursive walk across servers detected inthe peering mesh from the other servers.

B The primary DNS server is updated every 15 minutesB Slave DNS servers are provided by a voluntary community,

several of which are AnyCasted clusters.

21/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Scalability I

B When I started in 2006 the number of servers was about15–20

B As the number of tests performed as well as the number ofservers increased, so did crawler execution time

B The linear increase in time resulted in a need to rewritecore components to parallel execution.

B The rewrite reduced the execution time for a full crawlfrom 14.5 minutes to 3.5 minutes (most of which is networkdelay and intended sleep() procedures to avoid DoSconditions)

22/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Scalability II

B The crawler is now executed using libevent– a stand-alone PHP script is executed for each server

lookup that returns data in json format.– peering servers not already checked are added to a

pool of servers to check– iterates until the pool is empty

B This approach also allows for direct testing against specificservers for debugging purposes.

23/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Customizations I

B In particular setting up the HKPS pool required tweaksoutside of PHP.

B At the time of initial writing, PHP did not supportCURL RESOLVE, as such execution would not accuratelydetect the presence of a X.509 Certificate of the pool dueto connecting to IP address with Host: header of the pool.

B This issue was solved by patching curl directly to use thevalue of the HTTP Host header as value for Server NameIndication.http://curl.haxx.se/mail/lib-2012-11/0104.html

B CURL RESOLVE was added in PHP5.5. This permitted acode rewrite to remove the need for a custom patch of curl.

24/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Customizations II

B But PHP did not yet allow for specifying a CertificateRevocation List (CRL). As such PHP itself had to bepatched: https://bugs.php.net/bug.php?id=65575

B Support for CRL files were added in the mainline PHPsource in 5.5.4.

B The crawler now operate without need for custom patchesin any external component.

25/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Table of Contents

1 A brief introduction to OpenPGP

2 sks-keyservers.net

3 Code base

4 Keyserver network and community

26/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Number of keys

B I started recordingstatistics on the number ofkeys in 2011.

B Since then there are about800,000 new keys available

B There has been a sustainedincrease in the growth ratesince the Snowdenrevelations.

B A more detailed breakdownof the keys (key length,algorithms etc) is availableat http://blog.sumptuouscapital.com/2014/01/openpgp-key-statistics/

27/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

Network analysis

B With a growing number ofservers, keys and otherupdates the time toreplicate the data acrossthe network increase

B In order to optimize this,bottlenecks in the networkhave to be detected andfixed.

B This is mostly a manualoperation aided by toolsfor network analysis.

28/28

A brief introduction to OpenPGP sks-keyservers.net Code base Keyserver network and community

B The keyserver community is generally friendly and is activein providing support and responding to new requests

B Wiki at https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Home

B Mailing list at sks-devel@nongnu.org, both for SKSquestions and general keyserver operational procedures