# OpenSearch Query Input Plugin

This [OpenSearch](https://opensearch.org/) plugin queries endpoints
to derive metrics from data stored in an OpenSearch cluster.

The following is supported:

- return number of hits for a search query
- calculate the `avg`/`max`/`min`/`sum` for a numeric field, filtered by a query,
  aggregated per tag
- `value_count` returns the number of documents for a particular field
- `stats` (returns `sum`, `min`, `max`, `avg`, and `value_count` in one query)
- extended_stats (`stats` plus stats such as sum of squares, variance, and standard
  deviation)
- `percentiles` returns the 1st, 5th, 25th, 50th, 75th, 95th, and 99th percentiles

## OpenSearch Support

This plugins is tested against OpenSearch 2.5.0 and 1.3.7.

## Global configuration options <!-- @/docs/includes/plugin_config.md -->

In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.

[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins

## Configuration

```toml @sample.conf
# Derive metrics from aggregating OpenSearch query results
[[inputs.opensearch_query]]
  ## OpenSearch cluster endpoint(s). Multiple urls can be specified as part
  ## of the same cluster.  Only one successful call will be made per interval.
  urls = [ "https://node1.os.example.com:9200" ] # required.

  ## OpenSearch client timeout, defaults to "5s".
  # timeout = "5s"

  ## HTTP basic authentication details
  # username = "admin"
  # password = "admin"

  ## Skip TLS validation.  Useful for local testing and self-signed certs.
  # insecure_skip_verify = false

  [[inputs.opensearch_query.aggregation]]
    ## measurement name for the results of the aggregation query
    measurement_name = "measurement"

    ## OpenSearch index or index pattern to search
    index = "index-*"

    ## The date/time field in the OpenSearch index (mandatory).
    date_field = "@timestamp"

    ## If the field used for the date/time field in OpenSearch is also using
    ## a custom date/time format it may be required to provide the format to
    ## correctly parse the field.
    ##
    ## If using one of the built in OpenSearch formats this is not required.
    ## https://opensearch.org/docs/2.4/opensearch/supported-field-types/date/#built-in-formats
    # date_field_custom_format = ""

    ## Time window to query (eg. "1m" to query documents from last minute).
    ## Normally should be set to same as collection interval
    query_period = "1m"

    ## Lucene query to filter results
    # filter_query = "*"

    ## Fields to aggregate values (must be numeric fields)
    # metric_fields = ["metric"]

    ## Aggregation function to use on the metric fields
    ## Must be set if 'metric_fields' is set
    ## Valid values are: avg, sum, min, max, sum
    # metric_function = "avg"

    ## Fields to be used as tags.  Must be text, non-analyzed fields. Metric
    ## aggregations are performed per tag
    # tags = ["field.keyword", "field2.keyword"]

    ## Set to true to not ignore documents when the tag(s) above are missing
    # include_missing_tag = false

    ## String value of the tag when the tag does not exist
    ## Required when include_missing_tag is true
    # missing_tag_value = "null"
```

### Required parameters

- `measurement_name`: The target measurement to be stored the results of the
  aggregation query.
- `index`: The index name to query on OpenSearch
- `query_period`: The time window to query (eg. "1m" to query documents from
  last minute). Normally should be set to same as collection
- `date_field`: The date/time field in the OpenSearch index

### Optional parameters

- `date_field_custom_format`: Not needed if using one of the built in date/time
  formats of OpenSearch, but may be required if using a custom date/time
  format. The format syntax uses the [Joda date format][joda].
- `filter_query`: Lucene query to filter the results (default: "\*")
- `metric_fields`: The list of fields to perform metric aggregation (these must
  be indexed as numeric fields)
- `metric_function`: The single-value metric aggregation function to be performed
  on the `metric_fields` defined. Currently supported aggregations are "avg",
  "min", "max", "sum", "value_count", "stats", "extended_stats", "percentiles".
  (see the [aggregation docs][agg]
- `tags`: The list of fields to be used as tags (these must be indexed as
  non-analyzed fields). A "terms aggregation" will be done per tag defined
- `include_missing_tag`: Set to true to not ignore documents where the tag(s)
  specified above does not exist. (If false, documents without the specified tag
  field will be ignored in `doc_count` and in the metric aggregation)
- `missing_tag_value`: The value of the tag that will be set for documents in
  which the tag field does not exist. Only used when `include_missing_tag` is
  set to `true`.

[joda]: https://opensearch.org/docs/2.4/opensearch/supported-field-types/date/#custom-formats
[agg]: https://opensearch.org/docs/2.4/opensearch/aggregations/

### Example configurations

#### Search the average response time, per URI and per response status code

```toml
[[inputs.opensearch_query.aggregation]]
  measurement_name = "http_logs"
  index = "my-index-*"
  filter_query = "*"
  metric_fields = ["response_time"]
  metric_function = "avg"
  tags = ["URI.keyword", "response.keyword"]
  include_missing_tag = true
  missing_tag_value = "null"
  date_field = "@timestamp"
  query_period = "1m"
```

#### Search the maximum response time per method and per URI

```toml
[[inputs.opensearch_query.aggregation]]
  measurement_name = "http_logs"
  index = "my-index-*"
  filter_query = "*"
  metric_fields = ["response_time"]
  metric_function = "max"
  tags = ["method.keyword","URI.keyword"]
  include_missing_tag = false
  missing_tag_value = "null"
  date_field = "@timestamp"
  query_period = "1m"
```

#### Search number of documents matching a filter query in all indices

```toml
[[inputs.opensearch_query.aggregation]]
  measurement_name = "http_logs"
  index = "*"
  filter_query = "product_1 AND HEAD"
  query_period = "1m"
  date_field = "@timestamp"
```

#### Search number of documents matching a filter query, returning per response status code

```toml
[[inputs.opensearch_query.aggregation]]
  measurement_name = "http_logs"
  index = "*"
  filter_query = "downloads"
  tags = ["response.keyword"]
  include_missing_tag = false
  date_field = "@timestamp"
  query_period = "1m"
```

#### Search all documents and generate common statistics, returning per response status code

```toml
[[inputs.opensearch_query.aggregation]]
  measurement_name = "http_logs"
  index = "*"
  tags = ["response.keyword"]
  include_missing_tag = false
  date_field = "@timestamp"
  query_period = "1m"
```

## Metrics

All metrics derive from aggregating OpenSearch query results.  Queries must
conform to appropriate OpenSearch
[Aggregations](https://opensearch.org/docs/latest/opensearch/aggregations/)
for more information.

Metric names are composed of a combination of the field name, metric aggregation
function, and the result field name.

For simple metrics, the result field name is `value`, and so getting the `avg`
on a field named `size` would produce the result `size_value_avg`.

For functions with multiple metrics, we use the resulting field.  For example,
the `stats` function returns five different results, so for a field `size`,
we would see five metric fields, named `size_stats_min`,
`size_stats_max`, `size_stats_sum`, `size_stats_avg`, and `size_stats_count`.

Nested results will build on their parent field names, for example, results for
percentile take the form:

```json
{
  "aggregations" : {
  "size_percentiles" : {
    "values" : {
      "1.0" : 21.984375,
      "5.0" : 27.984375,
      "25.0" : 44.96875,
      "50.0" : 64.22061688311689,
      "75.0" : 93.0,
      "95.0" : 156.0,
      "99.0" : 222.0
    }
  }
 }
}
```

Thus, our results would take the form `size_percentiles_values_1.0`.  This
structure applies to `percentiles` and `extended_stats` functions.

Note: `extended_stats` is currently limited to 2 standard deviations only.

## Example Output

```toml
[[inputs.opensearch_query.aggregation]]
    measurement_name = "bytes_stats"
    index = "opensearch_dashboards_sample_data_logs"
    date_field = "timestamp"
    query_period = "10m"
    filter_query = "*"
    metric_fields = ["bytes"]
    metric_function = "stats"
    tags = ["response.keyword"]
```

```text
bytes_stats,host=localhost,response_keyword=200 bytes_stats_sum=22231,doc_count=4i,bytes_stats_count=4,bytes_stats_min=941,bytes_stats_max=9544,bytes_stats_avg=5557.75 1672327840000000000
bytes_stats,host=localhost,response_keyword=404 bytes_stats_min=5330,bytes_stats_max=5330,bytes_stats_avg=5330,doc_count=1i,bytes_stats_sum=5330,bytes_stats_count=1 1672327840000000000
```