Authentication/Authorization Extension for ElasticSearch and Kibana

Authentication/Authorization Extension for ElasticSearch and Kibana
Slide Note
Embed
Share

This project focuses on enhancing authentication and authorization capabilities for the ElasticSearch and Kibana platform, particularly leveraging Kerberos 5 for API restrictions and user-specific querying. Development of plugins like Search Guard and the AppendFilter plugin are discussed to ensure secure access control and personalized query results.

  • Security
  • Authentication
  • Authorization
  • ElasticSearch
  • Kibana
kger
kger Follow

Uploaded on Mar 07, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

Download Presentation

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

E N D

Presentation Transcript

  1. Authentication/Authorization extension for ElasticSearch and Kibana platform Wataru Takase Computing Research Center, KEK 10thFebruary, 2016 FJPPL Computing Project Annual Workshop 1

  2. Our Goals Kerberos 5 authentication ElasticSearch API restrictions based on: Kerberos 5 principals Index and/or alias names Index types Cluster update operations only for certain K5 principals Kibana 4 tweaking .kibana.%{krb5_princ} (personal index based on Kerberos 5 principal) read + write Common .kibana.%{team} (common team index based on K5 principal) read + write (restricted) 2

  3. Search Guard makes us happy Search Guard is an ElasticSearch plugin https://github.com/floragunncom/search-guard Flexible REST layer access control (User/Role; based on aliases, indices and types) Also last year I developed an ElasticSearch plugin to limit user s request My plugin restricts based on user All users refer same index. Plugin intercepts user s request from Kibana and appends user specific query Search Guard restricts based on user, role, indices, types, etc. It s better to use Search Guard rather than to extend my plugin 3

  4. My plugin Development of AppendFilter plugin Plugin appends user specific query tenant_id= xxxxxxxxx User gets result related to own tenants Tenant id query + Searc h Query JSON query JSON query ES CERN SSO ES Kibana Plugin User ID #userid wtakase 0 ... admin_flag tenant_id fcb04d9a-d9f4-4bb7-8581-8c987ce569b9 domain_id NULL 4

  5. My plugin Original JSON query { "sort": [{"_score": { "order": "desc", "ignore_unmapped": true }}], "query": {"filtered": { "query": {"bool": {"should": [{"query_string": {"query": "*"}}]}}, "filter": {"bool": {"must": [{"range": {"@timestamp": { "to": 1406215934612, "from": 1405611134612 }}}]}} }}, "highlight": { "pre_tags": ["@start-highlight@"], "post_tags": ["@end-highlight@"], "fragment_size": 2147483647, "fields": {} }, "size": 500 }

  6. My plugin Appended Filter to JSON query { "sort": [{"_score": { "order": "desc", "ignore_unmapped": true }}], "query": {"filtered": { "query": {"bool": {"should": [{"query_string": {"query": "*"}}]}}, "filter": {"bool": {"must": [{"range": {"@timestamp": { "to": 1406215934612, "from": 1405611134612 }}}, {"fquery": {"query": {"query_string": {"query": "@fields.project:\"fcb04d9a-d9f4-4bb7-8581-8c987ce569b9\" OR @fields.tenant_id:\"fcb04d9a-d9f4-4bb7-8581-8c987ce569b9\""}}}} ]}} }}, "highlight": { "pre_tags": ["@start-highlight@"], "post_tags": ["@end-highlight@"], "fragment_size": 2147483647, "fields": {} }, "size": 500 }

  7. My plugin @fields.project:"fcb04d9a-d9f4-4bb7-8581-8c987ce569b9" {"cell": "", "domain": "Personal", "project": "fcb04d9a-d9f4-4bb7-8581-8c987ce569b9", "project_name": "Personal_wtakase", "source": "nova", "type": "instances", "key": "active", "timestamp": 1407332557004, "val_int": 3} {"cell": "", "domain": "Personal", "project": "fcb04d9a-d9f4-4bb7-8581-8c987ce569b9", "project_name": "Personal_wtakase", "source": "nova", "type": "instances", "key": "error", "timestamp": 1407332557004, "val_int": 0} {"cell": "", "domain": "Personal", "project": "fcb04d9a-d9f4-4bb7-8581-8c987ce569b9", "project_name": "Personal_wtakase", "source": "nova", "type": "instances", "key": "building", "timestamp": 1407332557004, "val_int": 0} {"cell": "", "domain": "Personal", "project": "fcb04d9a-d9f4-4bb7-8581-8c987ce569b9", "project_name": "Personal_wtakase", "source": "nova", "type": "instances", "key": "total", "timestamp": 1407332557004, "val_int": 3} {"cell": "", "domain": "wwww", "project": "wwwwwwww-wwww-wwww-wwww-wwwwwwwwwwww", "project_name": "wwwwwww", "source": "nova", "type": "instances", "key": "active", "timestamp": 1407332557004, "val_int": 5} {"cell": "", "domain": "xxxx", "project": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "project_name": "xxxxxxx", "source": "nova", "type": "instances", "key": "error", "timestamp": 1407332557004, "val_int": 0} {"cell": "", "domain": "yyyy", "project": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy", "project_name": "yyyyyyy", "source": "nova", "type": "instances", "key": "building", "timestamp": 1407332557004, "val_int": 0} {"cell": "", "domain": "zzzz", "project": "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz", "project_name": "zzzzzzz", "source": "nova", "type": "instances", "key": "total", "timestamp": 1407332557004, "val_int": 5} ES returns filtered information

  8. Solutions Kerberos 5 authentication Apache mod_auth_kerb module + reverse proxy ElasticSearch API restrictions based on: Kerberos 5 principals Search Guard supports it Use REMOTE_USER variable and role Index and/or alias names Search Guard supports it Index types Search Guard supports it Cluster update operations only for certain K5 principals Search Guard may support it Kibana 4 tweaking .kibana.%{krb5_princ} (personal index based on Kerberos 5 principal) read + write Apache configuration + Search Guard Common .kibana.%{team} (common team index based on K5 principal) read + write (restricted) Apache configuration + Search Guard 8

  9. Kerberos 5 authentication Apache mod_auth_kerb module + reverse proxy RewriteRule ^/?es/?(.*)$ http://localhost:9200/$1 [L,P] kerberos elasticsearch authenticatio n reverse proxy kerberos auth https httpd reverse proxy Kibana 4 container <Location /> AuthType Kerberos . . . Require valid-user </Location> RewriteRule ^/(.*)$ http://localhost:15601/$1 [L,P] Only authenticated users can access to ES and Kibana 9

  10. ElasticSearch API restrictions Search Guard + REMOTE_USER + LDAP kerberos ldap role elasticsearch authorization authenticatio n search guard REMOTE_USER reverse proxy with REMOTE_USER Flexible REST layer access control (User/Role; based on aliases, indices and types) kerberos auth https httpd reverse proxy Kibana 4 container Another authorization backend is supported: https://github.com/floragunncom/search-guard/blob/master/searchguard_config_template.yml 10

  11. ElasticSearch API restrictions Kerberos 5 principals Index and/or alias names Index types Cluster update operations only for certain K5 principals * Any user with es_admin role can do anything Index restriction role public-* Everyone can read/write es_admin cloud grid REMOTE_USER cloud-* Any user with cloud role can read/write es_admin o grid-* Any user with grid role can read/write takase o takase-* User takase can read/write test1 o test1-* User test1 can read/write test2-* User test2 can read/write test2 o 11

  12. /etc/elasticsearch/elasticsearch.yaml ##################################################### # Configure the actionrequestfilter to allow or forbid action searchguard.actionrequestfilter.names: ["readonly", "noaccesible"] searchguard.actionrequestfilter.readonly.allowed_actions: ["indices:data/read/*", "*monitor*"] searchguard.actionrequestfilter.readonly.forbidden_actions: ["cluster:admin*", "indices:admin*", "indices:data/write*"] searchguard.actionrequestfilter.noaccesible.allowed_actions: [] searchguard.actionrequestfilter.noaccesible.forbidden_actions: ["cluster:*", "indices:*"] 12

  13. ElasticSearch API restrictions curl -XPUT 'http://localhost:9200/searchguard/ac/ac' -d '{ > "acl": [ > { > "__Comment__": "Default is to execute all filters", > "filters_bypass": [], > "filters_execute": ["*"] > }, > { > "__Comment__": "Any authenticated user on the 'public' index - no filter will be executed", > "indices": ["public-*"], > "filters_bypass": [], > "filters_execute": [] > }, > { > "__Comment__": "This means any user with the role cloud can do anything with the cloud index", > "roles" : ["cloud"], > "indices": ["cloud-*"], > "filters_bypass": ["*"], > "filters_execute": [] > }, > { > __Comment__ : This means that for the user test2 on index grid only the actionrequestfilter.readonly will be executed, no other", > users : [ test2"], > "indices": ["grid-*"], > "filters_bypass": [], > "filters_execute": ["actionrequestfilter.readonly"] > } >} 13

  14. Kibana 4 tweaking Problem: 1 Kibana instance has only 1 index for storing dashboards .kibana All users share same Kibana index User can read/write others dashboards Our goal: UserA only can read/write UserA dashboards Users belonged to GroupB can read/write GroupB dashboards 14

  15. Initial Idea: Overview of Test Environment kerberos ldap Flexible REST layer access control (User/Role; based on aliases, indices and types) role elasticsearch authorization authenticatio n search guard reverse proxy with REMOTE_USER REMOTE_USER kerberos auth https httpd reverse proxy based on REMOTE_USER Kibana 4 container port and kibana_index are different Kibana 4 REMOTE_USER kibana4 port kibana_index container takase 15601 .kibana.takase test1 15602 .kibana.test1 ... test2 15603 .kibana.test2 Kibana 4 es_admin 25601 .kibana.es_admin container Doesn t scale!! cloud 35601 .kibana.cloud grid 35602 .kibana.grid

  16. Kibana 4 tweaking For user based restriction Each user has his/her own index for storing Kibana dashboards Ex) .kibana.takase Modify Kibana 4 s request based on REMOTE_USER on httpd Kibana always tries to store dashboard to .kibana kerberos ldap role elasticsearch authorization authenticatio n search guard REMOTE_USER REMOTE_USER kibana_index reverse proxy with REMOTE_USER takase .kibana.takase kerberos auth test1 .kibana.test1 modify .kibana index request based on REMOTE_USER https httpd test2 .kibana.test2 reverse proxy Kibana 4 16 container

  17. Kibana Request Specify ES index in URL or POST data: $ curl https://xxxx/es/.kibana/xxxxx $ curl XPOST https://www/es/xxxx --data-binary { index : { _index : .kibana }} URL modification mod_rewrite POST data modification mod_ext_filter 17

  18. Kibana 4 tweaking For user based restriction Modify URL RewriteCond %{LA-U:REMOTE_USER} (.+)@TGPU01.KEK.JP RewriteRule .* - [E=TRIMMED_REMOTE_USER:%1] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/(.*)/\.kibana/(.*)$ http://localhost:9200/$1/.kibana.%{ENV:TRIMMED_REMOTE_USER}/$2 [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/(.*)/\.kibana$ http://localhost:9200/$1/.kibana.%{ENV:TRIMMED_REMOTE_USER} [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/\.kibana/(.*)$ http://localhost:9200/.kibana.%{ENV:TRIMMED_REMOTE_USER}/$1 [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/\.kibana$ http://localhost:9200/.kibana.%{ENV:TRIMMED_REMOTE_USER} [L,P] This can be simpler 18

  19. Kibana 4 tweaking For user based restriction Modify POST data if REMOTE_USER= takase, then set takase=1 RewriteCond %{LA-U:REMOTE_USER} takase RewriteRule .* - [E=takase:1] RewriteCond %{LA-U:REMOTE_USER} test1 RewriteRule .* - [E=test1:1] if takase=1, then execute sed command ExtFilterDefine takase_kibana_index mode=input cmd="/bin/sed s/\".kibana\"/\".kibana.takase\"/g" enableenv=takase ExtFilterDefine test1_kibana_index mode=input cmd="/bin/sed s/\".kibana\"/\".kibana.test1\"/g" enableenv=test1 SetInputFilter takase_kibana_index;test1_kibana_index Call defined filters 19 This can be simpler

  20. Kibana 4 tweaking For group based restriction Each group has own index for storing Kibana dashboards Ex) .kibana.cloud Modify Kibana 4 s request based on HTTP_REFERER on httpd User needs to access group specific URL Ex) https://kef01.tgpu01.kek.jp/cloud Avobe URL access is restricted by Apache and LDAP configuration <Location /cloud> AuthType Kerberos . . . AuthLDAPUrl ldap://ldap01.tgpu01.kek.jp:389/ou=People,dc=tgpu01,dc=kek,dc=jp?krbPrincipalName Require ldap-group cn=cloud,ou=Groups,dc=tgpu01,dc=kek,dc=jp </Location> You can use AuthGroupFile instead of AuthLDAPUrl 20

  21. Kibana 4 tweaking For group based restriction Request URI Proxy to https://kef01.tgpu01.kek.jp/ http://localhost:15601 (kibana4) https://kef01.tgpu01.kek.jp/cloud/ http://localhost:15601 (kibana4) https://kef01.tgpu01.kek.jp/grid/ http://localhost:15601 (kibana4) <Location /cloud> AuthType Kerberos . . . AuthLDAPUrl ldap://ldap01.tgpu01.kek.jp:389/ou=People,dc=tgpu01,dc=kek,dc=jp? krbPrincipalName Require ldap-group cn=cloud,ou=Groups,dc=tgpu01,dc=kek,dc=jp </Location> <Location /> AuthType Kerberos . . . Require valid-user </Location> <Location /grid> AuthType Kerberos . . . AuthLDAPUrl ldap://ldap01.tgpu01.kek.jp:389/ou=People,dc=tgpu01,dc=kek,dc=jp? krbPrincipalName Require ldap-group cn=grid,ou=Groups,dc=tgpu01,dc=kek,dc=jp </Location> 21

  22. Kibana 4 tweaking For group based restriction RewriteCond %{HTTP_REFERER} ^https?://kef01.tgpu01.kek.jp/cloud.*$ RewriteRule .* - [E=TRIMMED_REMOTE_USER:cloud] RewriteCond %{HTTP_REFERER} ^https?://kef01.tgpu01.kek.jp/grid.*$ RewriteRule .* - [E=TRIMMED_REMOTE_USER:grid] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/(.*)/\.kibana/(.*)$ http://localhost:9200/$1/.kibana.%{ENV:TRIMMED_REMOTE_USER}/$2 [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/(.*)/\.kibana$ http://localhost:9200/$1/.kibana.%{ENV:TRIMMED_REMOTE_USER} [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/\.kibana/(.*)$ http://localhost:9200/.kibana.%{ENV:TRIMMED_REMOTE_USER}/$1 [L,P] RewriteCond %{LA-U:REMOTE_USER} !^$ RewriteRule ^/?es/\.kibana$ http://localhost:9200/.kibana.%{ENV:TRIMMED_REMOTE_USER} [L,P] 22

  23. Kibana 4 tweaking ES index access control is done by Search Guard Index restriction .kibana* Kibana management user can read/write .kibana.cloud Any user with cloud role can read/write .kibana.grid Any user with grid role can read/write .kibana.takase User takase can read/write .kibana.test1 User test1 can read/write .kibana.test2 User test2 can read/write 23

  24. Demo role es_admin cloud grid REMOTE_USER es_admin o kdc01 ldap01 takase o kerberos ldap test1 o test2 o role REMOTE_USER authorization authenticatio n Flexible REST layer access control (User/Role; based on aliases, indices and types) group elasticsearch reverse proxy with REMOTE_USER search guard kerberos auth https modify .kibana index request based on REMOTE_USER or HTTP_REFERER httpd REMOTE_USER kibana_index reverse proxy Kibana 4 takase .kibana.takase container kef01 test1 .kibana.test1 test2 .kibana.test2 HTTP_REFERER kibana_index https://xxxxx/cloud .kibana.cloud 24 https://xxxxx/grid .kibana.grid

  25. Need investigation Data flow from Flume to ElasticSearch under Kerberos/search-guard control Flume ElasticSearch Kibana ElasticSearchSink doesn t support Kerberos auth Kerberos auth can be avoidable by accessing to http://kef01.tgpu01.kek.jp:9300 But search-guard may block the request Access to Kibana/ElasticSearch from command line: # curl -k --krb --negotiate -u takase https://kef01.tgpu01.kek.jp/es Enter host password for user 'takase : How to access by using existing Kerberos ticket 25

Related


New trends in agriculture extension

New trends in agriculture extension

murphy54
TNAES & TNHS-Agriculture and Horticulture jobs in Tamil Nadu

TNAES & TNHS-Agriculture and Horticulture jobs in Tamil Nadu

OBCRIGHTS
Managed Elasticsearch for Dispatch Services

Managed Elasticsearch for Dispatch Services

harold
Challenges in DUNE Computing: Addressing Authentication and Authorization Needs

Challenges in DUNE Computing: Addressing Authentication and Authorization Needs

novalee
Security and Authentication in Electronic Filing Systems: Arkansas Subcommittee Report

Security and Authentication in Electronic Filing Systems: Arkansas Subcommittee Report

harri
Evolution of Sevier Orogeny and Cretaceous Extension in North American Cordillera

Evolution of Sevier Orogeny and Cretaceous Extension in North American Cordillera

taliesin
Enhancing Authorization in Alpaca: A Decade-old Approach Revisited

Enhancing Authorization in Alpaca: A Decade-old Approach Revisited

jream
Evolution of Agricultural Extension Services in India

Evolution of Agricultural Extension Services in India

selia
DAVA Drugs Authentication & Verification Application by National Informatics Centre

DAVA Drugs Authentication & Verification Application by National Informatics Centre

ruxt_322
Authentication Mechanisms and Security Vulnerabilities

Authentication Mechanisms and Security Vulnerabilities

lev_tre
Secure Usage of Kibana & ElasticSearch in Multi-User Environments

Secure Usage of Kibana & ElasticSearch in Multi-User Environments

thoennes_s
Authentication and Authorization in Astronomy: A Deep Dive into ASTERICS

Authentication and Authorization in Astronomy: A Deep Dive into ASTERICS

anna_ght
Enhancing Goodput with HTCSS and Adstash in High Throughput Computing

Enhancing Goodput with HTCSS and Adstash in High Throughput Computing

keel539
Evolution of User Authentication Practices: Moving Beyond IP Filtering

Evolution of User Authentication Practices: Moving Beyond IP Filtering

rin_fi
Authorization in Modern Applications

Authorization in Modern Applications

obermeier_y
Kerberos Authentication in Network Security

Kerberos Authentication in Network Security

pcapp
Kerberos Authentication Protocol

Kerberos Authentication Protocol

bri_pr
Managing BEAST Alarm System: Enhancing Alarm Monitoring and Analysis

Managing BEAST Alarm System: Enhancing Alarm Monitoring and Analysis

nahzi
Secure Authentication Using ISO 15693 RFID Tags

Secure Authentication Using ISO 15693 RFID Tags

rtrez
Authentication and Authorization in Information Assurance

Authentication and Authorization in Information Assurance

zam_nus
Authentication and Message Authentication in Network Security

Authentication and Message Authentication in Network Security

lein_aye
Authentication in Networking Environments

Authentication in Networking Environments

bolander_j

More Related Content