Effective Integration of OpenStack and EGI Check-In: Challenges and Solutions

EODC Integrating OpenStack and EGI Check-In Tom Clark tom.clark@eodc.eu Openstack and EGI Check-In - 2023-06-21 1

Where are we now? Fully redundant integration with EGI Check Fully redundant integration with EGI Check- -In In Supporting multiple OIDC providers Supporting multiple OIDC providers IaC IaC driven cluster deployment driven cluster deployment Containerised Openstack operated with Containerised Openstack operated with Kolla Kolla- -Ansible Ansible Automated controller/node provisioning via Ansible Automated controller/node provisioning via Ansible Project creation scripts handle cross Project creation scripts handle cross- -mapping between IdPs mapping between IdPs Openstack and EGI Check-In - 2023-06-21 2

Challenges Containerised deployment and Containerised deployment and IaC IaC EGI Check EGI Check- -in is great. in is great. We need to be able to rotate OIDC secrets We need to be able to rotate OIDC secrets Add additional OIDC providers Add additional OIDC providers Recommended solution for multiple OIDC Providers ESACO Proxy Limited documentation Discrepancies between existing documentation and deployment Integration with existing project handling scripts and toolsets Uncertainties around certificate based auth Openstack and EGI Check-In - 2023-06-21 3

What Handles AuthN, AuthZ, and OIDC? Our HA deployment Keepalived provides public facing virtual IP HAProxy listening on VIP and distributing to relevant services Apache Web server gateway interface Keystone Memcache cluster Mod_auth_openidc Opoenstack keystone mapping Which users belong to which groups. Which groups have access to which projects Openstack and EGI Check-In - 2023-06-21 4

Apache Limitations and Workarounds Working OIDC integration with Apache, Keycloak, and Keystone since 2018 Mod_auth_openidc Works very well within its design contraints Allows for metadata from multiple IdPs Single introspection endpoint for OAuth2 CLI access is a must Alternatives or invent something new? We know mod_auth_openidc and apache well Openstack and EGI Check-In - 2023-06-21 5

Enter ESACO Configure ESACO proxy as a workload. Config for IdPs Point and configure mod_auth_openidc towards ESACO proxy OIDCMetadataDir /etc/httpd/oidc/metadata OIDCResponseType code OIDCScope "openid profile email eduperson_entitlement" OIDCRedirectURI https://cloud.eodc.eu:5000/redirect_uri OIDCOAuthIntrospectionEndpoint https://cloud.eodc.eu:5000/introspect <LocationMatch /v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth> AuthType oauth20 Require claim iss:https://aai.egi.eu/auth/realms/egi/ Require valid-user </LocationMatch> <LocationMatch /v3/OS-FEDERATION/identity_providers/bouncer/protocols/openid/auth> Require claim iss:https://bouncer.eodc.eu/auth/realms/EODC/ Require valid-user AuthType oauth20 </LocationMatch> Openstack and EGI Check-In - 2023-06-21 6

ESACO Issues We never reached out in any way As principle, prefer not to expose things to the internet when not strictly required Didn t see a requirement here What does ESACO need? Stateless, outgoing connectivity preserved, reachable internally How to shelter ESACO Bind to loopback address? Not an option kolla-ansible? Zun? Kuryr? Gvisor? Apache: require IP directive? Limiting to openstack tunnel network should protect Everything comes via tunnel network! Solution: New endpoint - Reverse proxy via apache Limit public access via HAProxy ACL Openstack and EGI Check-In - 2023-06-21 7

HAProxy frontend keystone_external_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 193.170.252.200:5000 ssl crt /etc/letsencrypt/haproxy.pem default_backend keystone_external_back acl network_allowed src 193.170.252.200 acl restricted_page path,url_dec -m beg -i /introspect http-request deny if restricted_page !network_allowed backend keystone_external_back mode http balance source server node1 1.1.1.1:5000 check inter 2000 rise 2 fall 5 server node2 1.1.1.2:5000 check inter 2000 rise 2 fall 5 server node3 1.1.1.3:5000 check inter 2000 rise 2 fall 5 Openstack and EGI Check-In - 2023-06-21 8

Certificates Adding GRID/IGTF Certificate Authorities Clear requirement Simple principles EGI provide a comfortable repo and packages for host installs We are containerised Add CAs to all containers? Restart containers to ensure certificates are loaded Not required - Inconsistency concerns Rebuild containers with updated CAs More effort than benefit Openstack and EGI Check-In - 2023-06-21 9

Certificates Restarts when updating CAs? How about CRLs? If an identity is invalid We do not want the user having access Solution Schedule CA updates using EGI tools Restart Apache after CA updates Initially, CA + CRL handling in HAProxy Better to offload to apache Can easily restart apache instances Native format support Bind in folders instead of creating and passing PEMs Offset between controllers HAProxy maintains service availability eventual consistency Openstack and EGI Check-In - 2023-06-21 10

Result Project setup handled via scripts Options defined in json file Existing field is friendly name Create group in keycloak with relevant name Create group in keystone with relevant name Update mapping between keystone and OIDC that keycloak group is mapped to keystone group Introduction of a new field in our definitions - VO project name Mappings are per IdP Field omission supported Not every project has an EGI VO Updates mapping for EGI Checkin IdP Also performs all steps above Entry point shouldn t matter Ties well with disaster recovery and IaC concepts IdP mappings version controlled Comfortable to expand further with additional providers Openstack and EGI Check-In - 2023-06-21 11

Looking Ahead? Happy with current state Nice to have? Public key attribute and import Next iteration likely to be Openstack on Kubernetes Handling CA + CRLs further reworking Openstack and EGI Check-In - 2023-06-21 12

Questions? Hopefully answers too Openstack and EGI Check-In - 2023-06-21 13