Integrating Keystone with Large-scale Centralized Authentication

Chris JaniszewskiOpenStack Solutions Architectchris.openstack@redhat.comblog: chrisj.cloud

Ken HoldenOpenStack Solutions Architectkholden@redhat.comblog: holdenthecloud.com

● Level set● Benefits● Issues and Bottlenecks● Best Practices● Tuning and Results

Agenda

● Authentication and Authorization● Keystone● Tokens● LDAP● Domains● Kerberos and SSO

Level Set

● Keystone (OpenStack Identity Service)● Methods of Authentication

● Authorization tokens● Transport Layer Security (TLS) with X.509

Authentication and Authorization

.. provides identity, token, catalog, and policy services for use specifically by services in the OpenStack family

Features:● External authentication● Federation● Multi-factor authentication

Considerations:● Invalid login attempts ● No enforcement policies on password strength, expiration

Keystone

● generated for authorization and access

● default value for expiry is one hour

● often passed within the structure of a larger context

● Token types:○ UUID○ PKI and PKIZ○ FERNET

Tokens

● The Lightweight Directory Access Protocol (LDAP) - is an open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network

● LDAP simplifies integration of Identity authentication

● Identity service MUST NOT be allowed to write to LDAP services

LDAP

Domains

Keystone V2 Keystone V3

*Pictures found on Matt’s Dorn Blog - http://madorn.com/keystone-v3-api.html#.WwM2hXWUvgk

*

● Keystone can now be executed in a web server like Apache HTTPD

● The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service

● Web server handle authentication is not exclusive

● Web server can provide support for X.509 or Kerberos authentication

● Keystone provides support for password authentication (with SQL or LDAP)

Kerberos

Carnival Ticket Booth vs. Kerberos (server) Identity Service: Authentication Mechanisms

TGT● Grants a ticket called a

Ticket GrantingTicket (TGT) which says which services youhave access to. This ticket also has an expiration called a Time-To-Live (TTL).

● You present the TGT to the service you want access to and the service will use that to determine if you can access the service.

● Consolidated repository● Simplified management● Ideal for multiple OpenStack Deployments with

shared resources● Performance of LDAP vs SQL● SSL/TLS● Back-end database flexibility● Well-defined client API● Decoupling of the roles

LDAP Benefits

● Few people understand LDAP

● No writes allowed

● Is OpenStack a front end for your users?

● Very strict schema

● Performance considerations

LDAP Considerations

● Token Performance● LDAP Size● Database● Horizon Dashboard● LDAP response

Issues and Bottlenecks

● Enable Debug and Verbosity - Impact can be significant● Use slow hard drives for Controller’s OS and Databases● Use non-secured APIs, Dashboard, and LDAP queries

Generally a good idea NOT to...

● The faster you can request tokens, the faster OpenStack responds● Fernet tokens offer superior performance over UUID

Tokens

Who remembers the good old days when OpenStack started to crawl ?

# du -sh /var/lib/mysql/keystone149G /var/lib/mysql/keystone

MariaDB [keystone]> select count(*) from token;+----------+| count(*) |+----------+| 20717136 | Number of UUID tokens+----------+

Just say No to UUID

● Active Directory DNS: Use Domain Nameurl = ldaps://example.com

Or obtain a list of Domain Controllers from DNS via SRV Recordsdig -t SRV _ldap.tcp.example.com

● LDAP: list multiple LDAP serversurl = ldaps://ldap1,ldaps://ldap2

Where do I query?

● Do all 100,000+ users REALLY need access to OpenStack?● Keystone doesn’t always cleanup after itself

MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number of LDAP Accounts without filtering +----------+

MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number doesn’t decrease when adding filtering later +----------+

Limit number of LDAP records

● Filtering users and groupsuser_filter = (&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users)))

group_filter = (&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))

● Using AND / OR(&(|(memberOf=foo)(memberOf=bar)))(&(memberOf=for)(memberOf=bar))

● Active Directory Nested groups(memberOf:1.2.840.113556.1.4.1941:=cn=OpenStack)

LDAP - Filtering

● Member of group1 OR group2ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users)))'

● Member of group1 AND group2ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users))'

● Members of Active Directory Nested Groupldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(objectClass=organizationalPerson)(sAMAccountName=*)(memberOf: 1.2.840.113556.1.4.1941:=cn=OpenStack))'

ldapsearch

Default value is 0 which disables paging

$ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' \ -w PASSWORD |grep sAMAccountName |wc -lSize limit exceeded (4)825

$ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -E pr=1/noprompt -b 'dc=domain,dc=com' \ -D 'DOMAIN\USERNAME' \ -w PASSWORD |grep sAMAccountName |wc -l10052

LDAP Tuning - Pagination

● View how keystone queries LDAP by enabling debugcrudini --set keystone.conf ldap debug_level 3crudini --set keystone.conf DEFAULT insecure_debug truecrudini --set keystone.conf DEFAULT debug true# restart keystone

● Watch the magic...tail -f /var/log/keystone/keystone.log | grep LDAP\ search

filterstr=(&(sAMAccountName=kholden)(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=example,DC=com)))(objectClass=person)) attrs=['sAMAccountName', 'userPassword', 'userAccountControl', 'mail', 'description'] attrsonly=0

● Disable debug when done

LDAP Debug

● Keystone Pooling○ Decrease load on Domain Controller especially with TLS○ Separate pool for User Authentication○ use_auth_pool requires use_pool enabled

crudini --set keystone.DOMAIN.conf ldap use_pool Truecrudini --set keystone.DOMAIN.conf ldap pool_size 200crudini --set keystone.DOMAIN.conf ldap pool_retry_max 20crudini --set keystone.DOMAIN.conf ldap pool_retry_delay 0.1crudini --set keystone.DOMAIN.conf ldap pool_connection_timeout -1crudini --set keystone.DOMAIN.conf ldap pool_connection_lifetime 600crudini --set keystone.DOMAIN.conf ldap use_auth_pool Truecrudini --set keystone.DOMAIN.conf ldap auth_pool_size 1000crudini --set keystone.DOMAIN.conf ldap auth_pool_connection_lifetime 60# restart keystone

Keystone Pooling

● Enable Keystone Caching with memcached backendsystemctl enable memcached && systemctl start memcachedcrudini --set keystone.conf cache enabled truecrudini --set keystone.conf cache backend dogpile.cache.memcachedcrudini --set keystone.conf cache backend_argument url: LOCAL_IP:11211crudini --set keystone.conf catalog caching truecrudini --set keystone.conf domain_config caching truecrudini --set keystone.conf federation caching truecrudini --set keystone.conf revoke caching truecrudini --set keystone.conf role caching truecrudini --set keystone.conf token caching truecrudini --set keystone.conf token cache_on_issue truecrudini --set keystone.conf identity caching truecrudini --set keystone.conf identity cache_time 600# restart keystone

Keystone Caching

Results

https://docs.google.com/file/d/17S2CoXRnRJ3lPlav_uzjM1KxPcrTts_u/preview

https://docs.google.com/file/d/12QXkHoN-XqCiBTdpM1dv5Wc32M6dTXg2/preview

● Challenges with Implementation○ Proxies in front of Domain Controllers○ Firewalled Domain Controllers○ Omitted Distinguished Names to save licensing costs ○ Service Account Password Change○ Certificate / CA change○ Changes require keystone restart○ Fernet Token rotation○ The Defiant Windows Admin

Meanwhile, in the real world...

● Multiple OpenStack Environments with One LDAP○ Unique IDs for roles, Domains, and Projects are unique to each

site○ Determine Project, Domain and Role IDs from one site and

update other sites with same IDs (BACKUP DB FIRST)

select * from project where name='PROJECT_NAME';select * from project where name='DOMAIN_NAME';select * from project where name='ROLE_NAME';

update project set id='SITE1_PROJECT_ID' where name='PROJECT_NAME';update project set id='SITE1_DOMAIN_ID' where name='DOMAIN_NAME';update project set id='SITE1_ROLE_ID' where name='ROLE_NAME';

Multi-site