Beta-elastic docs

api-reference.md

API Reference

The DC/OS Elastic Service implements a REST API that may be accessed from outside the cluster. The <dcos_url> parameter referenced below indicates the base URL of the DC/OS cluster on which the Elastic Service is deployed.

REST API Authentication

REST API requests must be authenticated. This authentication is only applicable for interacting with the Elastic REST API directly. You do not need the token to access the Elasticsearch nodes themselves.

If you are using Enterprise DC/OS, follow these instructions to create a service account and an authentication token. You can then configure your service to automatically refresh the authentication token when it expires. To get started more quickly, you can also get the authentication token without a service account, but you will need to manually refresh the token.

If you are using open source DC/OS, follow these instructions to [pass your HTTP API token to the DC/OS endpoint]https://dcos.io/docs/1.9/security/iam-api/).

Once you have the authentication token, you can store it in an environment variable and reference it in your REST API calls:

$ export auth_token=uSeR_t0k3n

The curl examples in this document assume that an auth token has been stored in an environment variable named auth_token.

If you are using Enterprise DC/OS, the security mode of your installation may also require the --ca-cert flag when making REST calls. Refer to Obtaining and passing the DC/OS certificate in cURL requests for information on how to use the --cacert flag. If your security mode is disabled, do not use the --ca-cert flag.

Plan API

The Plan API provides endpoints for monitoring and controlling service installation and configuration updates.

$ curl -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/plans/deploy

Pause Installation

The installation will pause after completing installation of the current node and wait for user input.

$ curl -X POST -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/plans/deploy/interrupt

Resume Installation

The REST API request below will resume installation at the next pending node.

$ curl -X PUT <dcos_surl>/service/elastic/v1/plans/deploy/continue

Nodes API

The pods API provides endpoints for retrieving information about nodes, restarting them, and replacing them.

List Nodes

A list of available node ids can be retrieved by sending a GET request to /v1/pods:

CLI Example

$ dcos elastic pods list

HTTP Example

$ curl  -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/pods

Node Info

You can retrieve node information by sending a GET request to /v1/pods/<node-id>/info:

$ curl  -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/pods/<node-id>/info

CLI Example

$ dcos elastic pods info journalnode-0

HTTP Example

$ curl  -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/pods/journalnode-0/info

Replace a Node

The replace endpoint can be used to replace a node with an instance running on another agent node.

CLI Example

$ dcos elastic pods replace <node-id>

HTTP Example

$ curl -X POST -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/pods/<node-id>/replace

If the operation succeeds, a 200 OK is returned.

Restart a Node

The restart endpoint can be used to restart a node in place on the same agent node.

CLI Example

$ dcos elastic pods restart <node-id>

HTTP Example

$ curl -X POST -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/pods/<node-id>/restart

If the operation succeeds a 200 OK is returned.

Configuration API

The configuration API provides an endpoint to view current and previous configurations of the cluster.

View Target Config

You can view the current target configuration by sending a GET request to /v1/configurations/target.

CLI Example

$ dcos elastic config target

HTTP Example

$ curl -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/configurations/target

List Configs

You can list all configuration IDs by sending a GET request to /v1/configurations.

CLI Example

$ dcos elastic config list

HTTP Example

$ curl -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/configurations

View Specified Config

You can view a specific configuration by sending a GET request to /v1/configurations/<config-id>.

CLI Example

$ dcos elastic config show 9a8d4308-ab9d-4121-b460-696ec3368ad6

HTTP Example

$ curl -H "Authorization:token=$auth_token" <dcos_url>/service/elastic/v1/configurations/9a8d4308-ab9d-4121-b460-696ec3368ad6

Service Status Info

Send a GET request to the /v1/state/properties/suppressed endpoint to learn if Elastic is in a suppressed state and not receiving offers. If a service does not need offers, Mesos can "suppress" it so that other services are not starved for resources.

You can use this request to troubleshoot: if you think Elastic should be receiving resource offers, but is not, you can use this API call to see if Elastic is suppressed.

curl -H "Authorization: token=$auth_token" "<dcos_url>/service/elastic/v1/state/properties/suppressed"

backup_restore.md

Backup and Restore

You interact with the Elasticsearch cluster directly to perform snapshot and restore operations. Elasticsearch's built in snapshot and restore module allows you to create snapshots of individual indices or an entire cluster into a remote repository like a shared file system, S3, or HDFS.

changelog.md

Changelog

elastic-1.0.11-5.4.0-beta

Breaking Changes

• Kibana has been removed from the Elastic package, along with the proxylite helper service. Please see the 'Connecting Clients' section for instructions on how to provision and connect Kibana on DC/OS.

Improvements/Features

• Added an option to toggle installation of commercial X-Pack plugin (disabled by default).
• Increased ingest node default RAM to 2GB (issue: #908).
• Added a configurable health check user/password to use as Elastic credentials during readiness/health checks.

Upgrades

• Upgraded to Elastic 5.4.0.
• Upgraded to Support Diagnostics Version 5.12.
• Upgraded to dcos-commons-0.16.0.

configure.md

Changing Configuration at Runtime

You can customize your cluster in-place when it is up and running. These are the general steps to follow:

1. View your DC/OS dashboard at http://$DCOS_URI/#/services/overview
2. In the list of Applications, click the name of the Elastic service to be updated.
3. Within the Elastic instance details view, click the Configuration tab, then click Edit.
4. In the dialog that appears, expand the Environment Variables section and update any field(s) to their desired value(s). For example, to increase the number of data nodes, edit the value for DATA_NODE_COUNT. Do not edit the value for FRAMEWORK_NAME, MASTER_NODE_TRANSPORT_PORT, or any of the disk type/size fields.
5. Click Change and deploy configuration to apply any changes and cleanly reload the Elastic service scheduler. The Elastic cluster itself will persist across the change.

Configuration Guidelines

• Service name: This needs to be unique for each instance of the service that is running. It is also used as your cluster name.
• Service user: This must be a non-root user that already exists on each agent. The default user is nobody.
• X-Pack is not installed by default, but you can enable it. X-Pack comes with a 30-day trial license.
• Health check credentials: If you have X-Pack enabled, the health check will use these credentials for authorization. We recommend you create a specific Elastic user/password for this with minimal capabilities rather than using the default superuser elastic.
• Plugins: You can specify other plugins via a comma-separated list of plugin names (e.g., “analysis-icu”) or plugin URIs.
• CPU/RAM/Disk/Heap: These will be specific to your DC/OS cluster and your Elasticsearch use cases. Please refer to Elastic’s guidelines for configuration.
• Node counts: At least 1 data node is required for the cluster to operate at all. You do not need to use a coordinator node. Learn about Elasticsearch node types here. There is no maximum for node counts.
• Master transport port: You can pick whichever port works for your DC/OS cluster. The default is 9300. If you want multiple master nodes from different clusters on the same host, specify different master HTTP and transport ports for each cluster. If you want to ensure a particular distribution of nodes of one task type (e.g., master nodes spread across 3 racks, data nodes on one class of machines), specify this via the Marathon placement constraint.
• Serial vs Parallel deployment. By default, the DC/OS Elastic Service tells DC/OS to install and update everything in parallel. You can change this to serial in order to have each node installed one at a time.

Immutable settings (at cluster creation time via Elastic package UI or JSON options file via CLI)

These setting cannot be changed after installation.

• Service name (aka cluster name). Can be hyphenated, but not underscored.
• Master transport port.
• Disk sizes/types.

Modifiable settings (at runtime via Marathon env vars):

• Plugins
• CPU
• Memory
• JVM Heap (do not exceed ½ available node RAM)
• Node count (up, not down)
• Health check credentials
• X-Pack enabled/disabled
• Deployment/Upgrade strategy (serial/parallel). Note that serial deployment does not yet wait for the cluster to reach green before proceeding to the next node. This is a known limitation.

Any other modifiable settings are covered by the various Elasticsearch APIs (cluster settings, index settings, templates, aliases, scripts). It’s possible that some of the more common cluster settings will get exposed in future versions of the Elastic DC/OS Service.

Viewing Plans via the CLI

You can view the deploy plan for the DC/OS Elastic Service via the service URL: http://$DCOS_URL/service/$SERVICE_NAME/v1/plans

Topology

Each task in the cluster performs one and only one of the following roles: master, data, ingest, coordinator.

The default placement strategy specifies no constraint except that all the master nodes are distributed to different agents. You can specify further Marathon placement constraints for each node type. For example, you can specify that data nodes are never colocated, or that ingest nodes are deployed on a rack with high-CPU servers.

No matter how big or small the cluster is, there will always be exactly 3 master-only nodes with minimum_master_nodes = 2.

Default Topology (with minimum resources to run on 3 agents)

• 3 master-only nodes
• 2 data-only nodes
• 1 ingest-only node
• 1 coordinator node

The master/data/ingest/coordinator nodes are set up to only perform their one role. That is, master nodes do not store data, and ingest nodes do not store cluster state.

Minimal Topology

You can set up a minimal development/staging cluster without ingest nodes, or coordinator nodes. You’ll still get 3 master nodes placed on 3 separate hosts. If you don’t care about replication, you can even use just 1 data node. By default, Elasticsearch creates indices with a replication factor of 1 (i.e., 1 primary shard + 1 replica), so with 1 data node, your cluster will be stuck in a ‘yellow’ state unless you change the replication factor.

Note that with X-Pack installed, the default monitoring behavior is to try to write to an ingest node every few seconds. Without an ingest node, you will see frequent warnings in your master node error logs. While they can be ignored, you can turn them off by disabling X-Pack monitoring in your cluster, like this:

$ curl -XPUT -u elastic:changeme master.elastic.l4lb.thisdcos.directory:9200/_cluster/settings -d '{
    "persistent" : {
        "xpack.monitoring.collection.interval" : -1
    }
}'

connecting.md

Connecting Clients

The Elasticsearch REST APIs are exposed using JSON over HTTP. You simply send HTTP requests to the appropriate Named VIP, which is essentially a load-balanced name-based service address. By default, the Elastic framework creates an Elasticsearch cluster with one coordinator node. Send your requests to this service address as shown in the following examples:

1. Create an index.

curl -s -u elastic:changeme -XPUT 'coordinator.elastic.l4lb.thisdcos.directory:9200/customer?pretty'

1. Store and retrieve data.

curl -s -u elastic:changeme -XPUT 'coordinator.elastic.l4lb.thisdcos.directory:9200/customer/external/1?pretty' -d '
{
  "name": "John Doe"
}'
curl -s -u elastic:changeme -XGET 'coordinator.elastic.l4lb.thisdcos.directory:9200/customer/external/1?pretty'

Note: If you did not install any coordinator nodes, you should direct all queries to your data nodes instead:

curl -s -u elastic:changeme 'data.elastic.l4lb.thisdcos.directory:9200/_cat/nodes?v'

The service address varies based on the name you assigned the framework when you installed it.

<node type>.<framework name>.l4lb.thisdcos.directory:9200

The default framework name is elastic. If you customized the name your framework, you need to adjust the service address accordingly.

Kibana

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack. You can install Kibana like any other DC/OS package via the Universe > Packages tab of the DC/OS web interface or the DC/OS CLI:

$ dcos package install kibana

This command launches a new Kibana application with the default name kibana and the default Elasticsearch URL http://coordinator.elastic.l4lb.thisdcos.directory:9200. Two Kibana application instances cannot share the same name, so installing additional Kibana applications beyond the default one requires customizing the name at install time for each additional instance.

Access Kibana

1. Log into your DC/OS cluster so that you can see the Dashboard. You should see your Elastic service and your Kibana service running under Services.

2. Make sure Kibana is ready for use. Depending on your Kibana node’s resources and whether or not you are installing X-Pack, it can take ~10 minutes to launch. If you look in the stdout log for Kibana, you will see this line takes the longest when installing X-Pack:

Optimizing and caching browser bundles...

Then you’ll see this:

{"type":"log","@timestamp":"2016-12-08T22:37:46Z","tags":["listening","info"],"pid":12263,"message":"Server running at http://0.0.0.0:5601"}

1. If you installed X-Pack, go to

http://<dcos_url>/service/kibana/login

and log in with elastic/changeme.

Otherwise go to

http://<dcos_url>/service/kibana

Configuration Guidelines

• Service name: This needs to be unique for each instance of the service that is running.
• Service user: This must be a non-root user that already exists on each agent. The default user is nobody.
• The Kibana X-Pack plugin is not installed by default, but you can enable it. See the X-Pack documentation to learn more about X-Pack in the Elastic package. This setting must match the corresponding setting in the Elastic package (i.e., if you have X-Pack enabled in Kibana, you must also have it enabled in Elastic).
• Elasticsearch credentials: If you have X-Pack enabled, Kibana will use these credentials for authorization. The default user is kibana.
• Elasticsearch URL: This is a required configuration parameter. The default value http://coordinator.elastic.l4lb.thisdcos.directory:9200 corresponds to the named VIP that exists when the Elastic package is launched with its own default configuration.

Custom Installation

You can customize the Kibana installation in a variety of ways by specifying a JSON options file. For example, here is a sample JSON options file that installs X-Pack and customizes the service name and Elasticsearch URL:

{
    "service": {
        "name": "another-kibana"
    },
    "kibana": {
          "elasticsearch_url": "http://my.elasticsearch.cluster:9200",
          "xpack_enabled": true
    }
}

The command below installs Kibana using a options.json file:

$ dcos package install kibana --options=options.json 

index.md

DC/OS Elastic Service

DC/OS Elastic Service is an automated service that makes it easy to deploy and manage Elastic on Mesosphere DC/OS, eliminating nearly all of the complexity traditionally associated with managing an Elasticsearch cluster. Elasticsearch is a distributed, multitenant-capable, full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch clusters are highly available, fault tolerant, and durable. For more information on Elasticsearch, visit the Elastic site. Multiple Elasticsearch clusters can be installed on DC/OS and managed independently, so you can offer Elasticsearch as a managed service to your organization with or without the commercial X-Pack capabilities.

Benefits

DC/OS Elastic offers the following benefits of a semi-managed service:

• Easy installation
• Elastic scaling of nodes
• Replication for high availability
• Elasticsearch cluster and node monitoring

Features

DC/OS Elastic provides the following features:

• Single-command installation for rapid provisioning
• Multiple clusters for multiple tenancy with DC/OS
• High availability runtime configuration and software updates
• Storage volumes for enhanced data durability, known as Mesos Dynamic Reservations and Persistent Volumes
• Automatic reporting of Elasticsearch metrics to DC/OS statsd collector

install.md

Default Installation

To start a basic cluster with three master nodes, two data nodes, one ingest node, and one coordinator node, run the following command on the DC/OS CLI:

$ dcos package install elastic

This command creates a new Elasticsearch cluster with the default name elastic. Two clusters cannot share the same name, so installing additional clusters beyond the default cluster requires customizing the name at install time for each additional instance.

Note: You can also install Elastic from the Universe > Packages tab of the DC/OS web interface. If you install Elastic from the web interface, you must install the Elastic DC/OS CLI subcommands separately. From the DC/OS CLI, enter:

dcos package install elastic --cli

Custom Installation

You can customize the Elastic cluster in a variety of ways by specifying a JSON options file. For example, here is a sample JSON options file that customizes the service name, master transport port, and plugins:

{
    "service": {
        "name": "another-cluster"
    },
    "master_nodes": {
          "transport_port": 19300
    },
    "elasticsearch": {
        "plugins": "analysis-icu,analysis-kuromoji"
    }
}

The command below creates a cluster using a options.json file:

$ dcos package install elastic --options=options.json 

Multiple Elastic Cluster Installation

Installing multiple Elastic clusters is identical to installing Elastic clusters with custom configurations as described above. The only requirement on the operator is that a unique name is specified for each installation.

Sample JSON options file named another-cluster.json:

{
    "service": {
        "name": "another-cluster"
    }
}

The command below creates a cluster using another-cluster.json:

$ dcos package install elastic --options=another-cluster.json

See the Configuring section for a list of fields that can be customized via an options JSON file when the Elastic cluster is created.

Overlay networks

Elastic supports deployment on the dcos overlay network, a virtual network on DC/OS that allows each container (task) to have its own IP address and not use the ports resources on the agent. This can be specified by passing the following configuration during installation:

{
    "service": {
        "virtual_network": true
    }
}

As mentioned in the developer guide once the service is deployed on the overlay network, it cannot be updated to use the host network.

limitations.md

Limitations

Managing Configurations Outside of the Service

The DC/OS Elastic Service core responsibility is to deploy and maintain the deployment of an Elasticsearch cluster whose configuration has been specified. In order to do this, the service makes the assumption that it has ownership of node configuration. If an end-user makes modifications to individual nodes through out-of-band configuration operations, the service will almost certainly override those modifications at a later time. If a node crashes, it will be restarted with the configuration known to the scheduler, not with one modified out-of-band. If a configuration update is initiated, all out-of-band modifications will be overwritten during the rolling update.

Nodes

The maximum number of deployable nodes is constrained by the DC/OS cluster's resources. Each Elasticsearch node has specified required resources, so nodes may not be placed if the DC/OS cluster lacks the requisite resources.

Upgrade and Configuration Rolling Updates Do Not Wait for Health

The serial deploy strategy does wait for the cluster to reach green before proceeding to the next node.

Overlay networks

When elastic is deployed on the dcos overlay network, the configuration cannot be updated to use the host network.

managing.md

Managing

Add a Data/Ingest/Coordinator Node

Increase the DATA_NODE_COUNT/INGEST_NODE_COUNT/COORDINATOR_NODE_COUNT value from the DC/OS dashboard as described in the Configuring section. This creates an update plan as described in that section. An additional node will be added as the last step of that plan.

Node Info

Comprehensive information is available about every node. To list all nodes:

dcos elastic --name=<service-name> pods list

To view information about a node, run the following command from the CLI.

$ dcos elastic --name=<service-name> pods info <node-id>

For example:

$ dcos elastic pods info master-0

Node Status

Similarly, the status for any node may also be queried.

$ dcos elastic --name=<service-name> pods info <node-id>

For example:

$ dcos elastic pods info data-0

private-node-by-vip.png

private-nodes-by-agent.png

quick-start.md

Quick Start

Add a Data/Ingest/Coordinator Node

Increase the DATA_NODE_COUNT/INGEST_NODE_COUNT/COORDINATOR_NODE_COUNT value from the DC/OS dashboard as described in the Configuring section. This creates an update plan as described in that section. An additional node will be added as the last step of that plan.

Node Info

Comprehensive information is available about every node. To list all nodes:

dcos elastic --name=<service-name> pods list

To view information about a node, run the following command from the CLI.

$ dcos elastic --name=<service-name> pods info <node-id>

For example:

$ dcos elastic pods info master-0

Node Status

Similarly, the status for any node may also be queried.

$ dcos elastic --name=<service-name> pods info <node-id>

For example:

$ dcos elastic pods info data-0

troubleshooting.md

Troubleshooting

Replacing a Permanently Failed Node

The DC/OS Elastic Service is resilient to temporary node failures. However, if a DC/OS agent hosting an Elastic node is permanently lost, manual intervention is required to replace the failed node. The following command should be used to replace the node residing on the failed server.

$ dcos elastic --name=<service-name> pods replace <node_id>

Restarting a Node

If you must forcibly restart a node, use the following command to restart the node on the same agent node where it currently resides. This will not result in an outage or loss of data.

$ dcos elastic --name=<service-name> pods restart <node_id>

uninstall.md

Uninstall

To uninstall the service, replace name with the name of the Elastic instance to be uninstalled.

dcos package uninstall --app-id=<name> elastic

Then, use the framework cleaner script to remove your Elastic instance from Zookeeper and destroy all data associated with it. The script requires several arguments. The default values to be used are:

• framework_role is <service-name>-role.
• framework_principal is <service-name>-principal.
• zk_path is dcos-service-<service-name>.

These values may vary if you customized them during installation. For instance, if you changed the Elastic service name to customers, then instead of

• framework_role is elastic-role.
• framework_principal is elastic-principal.
• zk_path is dcos-service-elastic.

you would use

• framework_role is customers-role.
• framework_principal is customers-principal.
• zk_path is dcos-service-customers.

If you are using the Enterprise Edition of DC/OS with authentication enabled, you will need to include the token in the GET command.

AUTH_TOKEN=$(dcos config show core.dcos_acs_token)
dcos node ssh --master-proxy --leader
docker run mesosphere/janitor /janitor.py -r elastic-role -p elastic-principal -z dcos-service-elastic --auth_token=$AUTH_TOKEN

upgrade.md

Upgrade

Overview

We support upgrade/rollback between adjacent versions only. Concretely, to upgrade from version 2 to version 4, you must upgrade from 2 -> 3, then from 3 ->4.

Upgrade Instructions

1. In the DC/OS web interface, destroy the Elastic instance to be updated. (This will not kill Elasticsearch node tasks).
2. Verify that you no longer see the Elastic instance in the DC/OS web interface.
3. From the DC/OS CLI, install the N+1 version of Elastic (where N is your current version) with any customizations you require in a JSON options file:

$ dcos package install elastic --options=/path/to/options.json

The command above will trigger the install of the new Elastic version. You can follow the upgrade progress by making a REST request identical to the one used to follow the progress of a configuration upgrade. See the Configuring section for more information.

Note: The upgrade process will cause all of your Elastic node processes to restart.

version-policy.md

Version Policy

The DC/OS Elastic Service is engineered and tested to work with a specific release of the Elastic Stack and X-Pack. We select stable versions of the base technology in order to promote customer success. We have selected the latest stable version from Elastic for new releases.

x-pack.md

Elastic X-Pack

X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, and graph capabilities into one easy-to-install package. X-Pack is a commercial product of Elastic that requires a license. By default, X-Pack is not installed as part of the DC/OS Elastic service. However, it's easy to enable X-Pack as part of the service configuration:

You can toggle this setting at any time. This gives you the option of launching an Elastic cluster without X-Pack and then later enabling it. Or, you can run a cluster with X-Pack enabled to try out the commercial features and, if at the end of the 30-day trial period you don't wish to purchase a license, you can disable it without losing access to your data.

License Expiration

If you uninstall the X-Pack plugin via the service configuration or you let your license expire, remember these two important points:

1. Your data is still there.
2. All data operations (read and write) continue to work.

Graph, Machine Learning, Alerting and Notification, Monitoring, and Security all operate with reduced functionality when X-Pack becomes unavailable.

Click here to learn more about how X-Pack license expiration is handled.

x-pack.png