Note: This document is evolving and is in draft state.
Plugin architecture enables extending core features of OpenSearch. There are various kinds of plugins which are supported. But, the architecture has significant problems for OpenSearch customers. Importantly, plugins can fatally impact the cluster i.e., critical workloads like ingestion/search traffic would be impacted because a non-critical plugin like s3-repository failed with an exception.
This problem exponentially grows when we would like to run a third party plugin from the community. As OpenSearch and plugins run in the same process, it increases security risk, compounds dependency conflicts, and reduces the velocity of releases. Introducing extensions, a simple and easy way to extend features of OpenSearch. Extensions would support all plugin features and enable them to run in a separate process or on another node via OpenSearch SDK for Java (other SDKs will be developed). Meta Issue: Steps to make OpenSearch extensible Sandboxing: Step towards modular architecture in OpenSearch Security: Security for extensions
- Support for Extension REST Handlers
- Rest-Layer Authorization
- Reserved (System) Indices
- Asynchronous Operations for Extensions
- Identity Plugin / Identity Service
- Appendix
A common way that plugins extend the capabilities of OpenSearch is via adding APIs to add new routes to OpenSearch. In the plugin architecture, the Security plugin wraps around the API’s REST Handler to authenticate the request and enrich the request’s ThreadContext with information about the authenticated user. When a plugin is running in-process, it has access to the same ThreadContext and any transport action a plugin tries to execute will run within this same context. The Security plugin will block unauthorized requests before the transport action is executed on a node in the cluster.
With extensions, the Security Plugin will require an extension to present an Auth Token with every request to the OpenSearch cluster. The Auth Token will be used to authenticate and authorize the request. The Auth Token can be one of two types of tokens:
- On-behalf-of tokens - Access Token issued on-behalf-of a user that an extension can utilize to make REST requests against the cluster. Authorization of a request using this token ensures that 1) The request from this extension is permitted (Service Gate) and 2) The request from the original user is permitted (User Gate) . See section on On-behalf-of Tokens Authorization for more details
- Service Account Tokens - See section on Reserved (System) Indices for more details.
When a User initiates a REST request, for example:
curl -XGET https://admin:admin@localhost:9200/_extensions/_hw/hello --insecure
The trace of the request follows:
- The Security Plugin authenticates the request
- If the request is bound for an extension, the IdentityService issues an Access Token (JWT) on-behalf-of the authenticated user. See section on Identity Plugin / Identity Service for more details.
- The REST Request is forwarded to the extension including the Access Token
- (Optional) The extension can make requests back to the OpenSearch cluster using this Access Token in the window that the token is valid for
- When the extension is done handling the request, it returns results back to the user who initiated the request
Note: In many instances, an extension may want to write to indices reserved by the extension to store data needed to run the extension. When an extension writes to its reserved indices it would use a Service Account Token to write to the reserved index. See section on Reserved (System) Indices for more details.
On-behalf-of tokens are tokens issued on-behalf-of a user that allow a service to make requests on the user’s behalf - with restrictions! See section on On-behalf-of Tokens Authorization which touches on defining how requests can be further restricted based on the extension the request originates from.
On-behalf-of tokens are issued just-in-time (JIT) in core which means that they are issued just before they are needed. These tokens have a very short window, 5 minutes by default, and cannot be revoked.
See the example below of claims in the payload of an on-behalf-of token.
{
"iss": "<clustername>",
"iat": 1681318260981, // Time in epoch that token was issued
"sub": "admin", // Username
"aud": "hw", // Extension Unique ID
"nbf": 1681318260981, // Time in epoch that token will start being valid
"exp": 1681318560981, // Time in epoch that token will cease being valid
"enc_r": "F1GW4L84Lfgdn4IcHShhtyGtu4VOSbwaSHN40CSb3Dc=" // Encrypted roles
}
The claims in this token are very similar to what the existing JWT backend expects except for the following key differences:
- The
audclaim - The audience claim specifies which extension the token is issued for and is used on receipt of the token to further restrict how the request can interact with the OpenSearch cluster - Encryption of sensitive claims - It is necessary to include roles and backend_roles in the token in order to perform privilege evaluation on requests that include this token. Requests in OpenSearch are stateless meaning that it is not possible to lookup session information. As a result, the cluster cannot determine the current user nor for the cluster to lookup roles information from the sub claim alone. Extensions are third-party code and security will forbid the sharing of sensitive data with an extension without explicit authorization. In order to be able to authorize requests with these tokens, the roles and backend_roles will be encrypted to hide the sensitive information from that extension that is still needed by the security plugin. The Security plugin will know how to encrypt and decrypt these sensitive claims
- Many plugins rely on
backend_rolesto protect resources created by the plugin. The Security team will provide an extension-specific setting to turn on backwards compatibility mode which would allow the transmission ofrolesandbackend_rolesunencrypted to an extension as claims in the token.
See section in Appendix on On-behalf-of Tokens Authorization
In the plugin architecture of OpenSearch, plugins will create TransportActions with a name that can be referred to in the Security plugin. The action name will be used to permit a user assigned a role the ability to perform the corresponding action. In most cases, the action is permitted based on the name of the action alone, but in some instances the requests require index patterns to authorize the request.
See below for 2 examples of plugins that have reserved roles in the Security plugin that will be available to cluster admins to map to users to use the respective plugins.
anomaly_read_access:
reserved: true
cluster_permissions:
- 'cluster:admin/opendistro/ad/detector/info'
- 'cluster:admin/opendistro/ad/detector/search'
- 'cluster:admin/opendistro/ad/detectors/get'
- 'cluster:admin/opendistro/ad/result/search'
- 'cluster:admin/opendistro/ad/tasks/search'
- 'cluster:admin/opendistro/ad/detector/validate'
- 'cluster:admin/opendistro/ad/result/topAnomalies'
index_management_full_access:
reserved: true
cluster_permissions:
- "cluster:admin/opendistro/ism/*"
- "cluster:admin/opendistro/rollup/*"
- "cluster:admin/opendistro/transform/*"
- "cluster:admin/opensearch/notifications/feature/publish"
index_permissions:
- index_patterns:
- '*'
allowed_actions:
- 'indices:admin/opensearch/ism/*'
In the plugin architecture, the Security plugin will apply the SecurityFilter on a node right before the action is executed on that node. This is what the term Transport-Layer authorization refers to; the TransportAction is blocked right before it is executed on a node.
With extensions, the REST request is forwarded to an extension before ever hitting the Transport-Layer. In order to provide similar behavior to plugins where requests can be blocked, it is necessary to build Authorization into the REST-Layer (SecurityRestFilter) of the Security Plugin to block a request before its forwarded to an extension if the request is unauthorized.
In order to accomplish this, there are a few key steps:
- Introduce a concept of a ProtectedRoute which has a human-readable name associated with it that can be referenced in a role definition
- Allow legacy naming of the ProtectedRoute to allow backwards compatibility with plugins, but also introduce a strategic naming convention
- Build authorization into the REST-Layer to block based on the route name and index patterns
- Add audit logging into the new Authz layer
- Allow extensions to register roles without requiring a PR against the Security repo
In Support for Extension REST Handlers, a mechanism for extensions to interact with an OpenSearch cluster on-behalf-of a user was outlined. There are other instances where an extension would want to interact with OpenSearch on its own behalf and needs a mechanism for that interaction. Reserved (System) Indices are a great example of an extension interacting with OpenSearch on its own behalf. In order to facilitate this operation, the Security plugin will be altered to introduce the concept of a Service Account.
A Service Account is a special type of account that represents a service. Its important to note that many users of OpenSearch may already create special internal users that they configure an OpenSearch client to connect with and operate as a service, but we are more formally introducing the concept now for the extensions use-case.
A Service Account Token is an Auth Token that the service can utilize to authenticate and authorize requests to the OpenSearch cluster.
Some additional notes:
- Service Account Tokens are created like a HTTP Basic Auth Header with name and secret Base64 encoded to create the token. Service Account Tokens can be rotated by updating the password associated with the Service Account
- The Security Plugin’s protections on System Indices will be relaxed to allow interacting from service accounts that have reserved the indices
In the plugin architecture, many plugins store data required by the plugin in OpenSearch indices. In many instances, plugins want assurances that these indices have special protections as they are required for the plugin to operate as expected. The Security plugin offers special protections to these indices via settings in opensearch.yml
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ...]
The protections the security plugin offers these indices is that it will forbid anyone, including admin, from meddling with these indices. There are only 2 ways to interact with these indices:
- Via Admin Cert Authentication
- As a plugin by stashing the ThreadContext
Many plugin use-cases involve running a job on a schedule. In the plugin architecture, JobScheduler is an ExtensiblePlugin meaning that other plugins can extend JobScheduler in order to register and schedule jobs to run periodically. In the plugin architecture, Security is challenging because the job is run asynchronously outside of a user context as was described in the section on Support for Extension REST Handlers.
In order to create a context for the job to run in, plugins store the User object from the thread context with the Job definition. Plugin’s will take the User object from the job definition, read the roles and then utilize Roles Injection to inject the roles into the thread context that they wish to evaluate the privileges of a request with. Another item to note, is that plugins will impersonate the original user and use a stand-in user called plugin when evaluating privileges. Lastly, the roles that are injected into the ThreadContext are the roles at time of job creation—if a user’s roles were changed after job creation then they are not propagated to the job scheduler to update. This is a limitation because of the Security plugin’s support for external Identity Providers where roles cannot be looked up at runtime.
Job Definitions are stored in an index controlled by the plugin defining the job and JobScheduler is aware of these indices.
For extensions, the User object will not be stored in the Job Definition. In it’s place, the Security plugin will introduce the concept of a Refresh Token which is stored with the Job Definition and utilized by Job Scheduler to request a new access token to pass to an extension on Job invocation. A Refresh Token is similar to the Access Token defined in On-behalf-of Tokens, but it has a longer expiry, contains a special type claim which designates the token as a refresh_token, and it cannot be used to authenticate or authorize a request. The only thing a refresh token is good for is allowing JobScheduler to request a new access token.
Refresh tokens are rotated after each use and the latest valid refresh token will be stored with the job definition and parsed out by JobScheduler’s ScheduleParser.
JobScheduler trades the Refresh Token for a new Refresh Token and Access Token and this Access Token will behave exactly the same as discussed in Support for Extension REST Handlers.
Code for extensions registration, extension request forwarding and extension management all reside in core, but all authentication and authorization related code lives in the Security plugin. A bridge needs to be built in order for Core to invoke APIs in the security plugin to fulfill features described above like passing an Access Token to an extension.
As part of this project, the Security team will introduce a new extension point in OpenSearch called the IdentityPlugin. At most one IdentityPlugin can be installed at any time in OpenSearch. Initially, this interface will supply methods needed for Security for Extensions, but will be built out to support more use-cases as more of Security is moved into the core of OpenSearch.
The initial interface will include:
/**
* Plugin that provides identity and access control for OpenSearch
*
* @opensearch.experimental
*/
public interface IdentityPlugin {
/**
* Get the current subject
* */
public Subject getSubject();
/**
* Get the token manager
* */
public TokenManager getTokenManager();
}
The Subject will be an interface containing methods to interact with the current Subject. See snippet below for some of the methods:
/**
* An individual, process, or device that causes information to flow among objects or change to the system state.
*
* @opensearch.experimental
*/
public interface Subject {
/**
* Get the application-wide uniquely identifying principal
* */
Principal getPrincipal();
/**
* Method that returns whether the current subject of the running thread is authenticated
*/
boolean isAuthenticated();
}
And the TokenManager will contain methods to issue tokens. See snippet below for some of the methods:
/**
* Interface for a token manager to issue Auth Tokens for a User
*
* @opensearch.experimental
*/
public interface TokenManager {
/**
* Issue an access token on-behalf-of authenticated user for a service to utilize to interact with
* the OpenSearch cluster on-behalf-of the original user
* */
AuthToken issueAccessTokenOnBehalfOfAuthenticatedUser(String extensionUniqueId) throws OpenSearchSecurityException;
/**
* Issue a refresh token on-behalf-of authenticated user for a service to refresh access tokens
* */
AuthToken issueRefreshTokenOnBehalfOfAuthenticatedUser(String extensionUniqueId) throws OpenSearchSecurityException;
}
In Support for Extension REST Handlers, it briefly touched upon the notion of a 1) Service Gate and 2) User Gate. The Security team has been debating how to evaluate the privileges on receipt of a request using an On-behalf-of Token and so far there are two proposals:
- Policies
See opensearch-project/security#2587 (comment) for in depth proposal on policies which re-use the notion of a role and allow the same expressiveness of a role definition
- Scopes
See opensearch-project/security#2644 for details on Scopes
opensearch-project/security#2638
In order to expedite the development process and reach the milestone for the experimental release of extensions, the Security team has chosen to make sensible defaults and defer adding configurability until the initial milestone is met.
Some of the list of sensible defaults so far include:
- HS512 Symmetric Key encryption for signing and verifying the signature on tokens generated for extensions
- AES Encryption for the encryption of sensitive claims in the payload of the tokens
There have been a couple of areas that the team has determined where there are suitable alternative approaches to solving a problem that help meet the target experimental release if the design in this document proves to take longer than expected.
- Instead of Service Account / Service Account Token, the extension can be registered with a username/password combo similar to how OpenSearch Dashboards is configured today. The limitation with this approach is that the index that the extension would like to interact with cannot be given special protections via the Security plugin.
- Defer On-behalf-of Tokens Authorization to P1. Since there is no current consensus on how to implement authorization at the Service Gate, this decision can be deferred to a release after the initial experimental release. Without Service Gate authorization, an extension will be able to perform any action that the original user can without any additional restrictions. This is in line with how plugins behave, but ultimately the Security team would like to impose additional restrictions.
This document represents the current state of thinking for Security for Extensions and the Security team is actively seeking comments from the community on the design. Extensions is an exciting project for OpenSearch as the organization is striving to grow the OpenSearch community and foster innovation from anyone seeking to make a contribution. This design is proposed by the Security team, but ultimately its the community that will determine the direction of where extensions go in the future.