[7.x] Conditionally sets ignore_throttled only when search:includeFrozen is true

[jasonrhodes]

Fixes #113197

Summary

The "ignore_throttled" ES option is deprecated, as are frozen indices in general.. ES queries behave as though this flag is set to true by default, meaning they will ignore frozen indices.

When the Kibana setting for "include frozen indices" is set to true, we still need to set this flag to false explicitly to continue to respect that Kibana setting. (Exact deprecation strategy for that Kibana setting is still TBD.)

However, the current code was always setting the ignore_throttled option to be the !-negated value of whatever the search:includeFrozen setting was. Whenever this value is included in a query, an ES deprecation log is produced. When a user hasn't indicated that they are even trying to interact with the deprecated frozen index feature, we shouldn't be including this flag and setting it to its default value. We should leave the flag out altogether and avoid the deprecation log warning.

To Test / Reproduce

1. Spin up this branch which should run Kibana in 7.16
2. Make sure the "include frozen" setting in "Advanced Settings" is off:

3. Visit the Upgrade Assistant and turn on deprecation logging:

4. Click "View deprecation logs in Observability" (Note: it will open the Logs UI with ?sourceId=deprecation_logs, if you visit the Logs UI directly, you won't see them)
5. Live stream the logs
6. You shouldn't see any deprecation logs about "ignore_throttled"
7. Go back to "Advanced Settings" and turn on the "include frozen" setting
8. Return to the Upgrade Assistant and click on "View deprecation logs in Observability"
9. Live stream the logs
10. You should see deprecation logs about "ignore_throttled"

[weltenwort]

seems to work as advertised, thank you 👍 just left a small style question below

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[weltenwort]

@elasticmachine merge upstream

[weltenwort]

@elasticmachine merge upstream

[cjcenizal]

@elasticmachine merge upstream

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request

• Commit: 81ec436

• Storybooks Preview

• Flaky suites:

  • xpack-kibana-ciGroup6

---

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/ml/jobs/categorization_field_examples·ts.apis Machine Learning jobs Categorization example endpoint - invalid, too many tokens.

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]     │
[00:00:00]       └-: apis
[00:00:00]         └-> "before all" hook in "apis"
[00:09:28]         └-: Machine Learning
[00:09:28]           └-> "before all" hook in "Machine Learning"
[00:09:28]           └-> "before all" hook in "Machine Learning"
[00:09:28]             │ debg creating role ft_ml_source
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_source]
[00:09:28]             │ debg creating role ft_ml_source_readonly
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_source_readonly]
[00:09:28]             │ debg creating role ft_ml_dest
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_dest]
[00:09:28]             │ debg creating role ft_ml_dest_readonly
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_dest_readonly]
[00:09:28]             │ debg creating role ft_ml_ui_extras
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_ml_ui_extras]
[00:09:28]             │ debg creating role ft_default_space_ml_all
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_all]
[00:09:28]             │ debg creating role ft_default_space1_ml_all
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space1_ml_all]
[00:09:28]             │ debg creating role ft_all_spaces_ml_all
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_all_spaces_ml_all]
[00:09:28]             │ debg creating role ft_default_space_ml_read
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_read]
[00:09:28]             │ debg creating role ft_default_space1_ml_read
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space1_ml_read]
[00:09:28]             │ debg creating role ft_all_spaces_ml_read
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_all_spaces_ml_read]
[00:09:28]             │ debg creating role ft_default_space_ml_none
[00:09:28]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [ft_default_space_ml_none]
[00:09:28]             │ debg creating user ft_ml_poweruser
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser]
[00:09:28]             │ debg created user ft_ml_poweruser
[00:09:28]             │ debg creating user ft_ml_poweruser_spaces
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_spaces]
[00:09:28]             │ debg created user ft_ml_poweruser_spaces
[00:09:28]             │ debg creating user ft_ml_poweruser_space1
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_space1]
[00:09:28]             │ debg created user ft_ml_poweruser_space1
[00:09:28]             │ debg creating user ft_ml_poweruser_all_spaces
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_poweruser_all_spaces]
[00:09:28]             │ debg created user ft_ml_poweruser_all_spaces
[00:09:28]             │ debg creating user ft_ml_viewer
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer]
[00:09:28]             │ debg created user ft_ml_viewer
[00:09:28]             │ debg creating user ft_ml_viewer_spaces
[00:09:28]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_spaces]
[00:09:28]             │ debg created user ft_ml_viewer_spaces
[00:09:28]             │ debg creating user ft_ml_viewer_space1
[00:09:29]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_space1]
[00:09:29]             │ debg created user ft_ml_viewer_space1
[00:09:29]             │ debg creating user ft_ml_viewer_all_spaces
[00:09:29]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_viewer_all_spaces]
[00:09:29]             │ debg created user ft_ml_viewer_all_spaces
[00:09:29]             │ debg creating user ft_ml_unauthorized
[00:09:29]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_unauthorized]
[00:09:29]             │ debg created user ft_ml_unauthorized
[00:09:29]             │ debg creating user ft_ml_unauthorized_spaces
[00:09:29]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [ft_ml_unauthorized_spaces]
[00:09:29]             │ debg created user ft_ml_unauthorized_spaces
[00:13:32]           └-: jobs
[00:13:32]             └-> "before all" hook in "jobs"
[00:13:32]             └-: Categorization example endpoint - 
[00:13:32]               └-> "before all" hook for "valid with good number of tokens"
[00:13:32]               └-> "before all" hook for "valid with good number of tokens"
[00:13:32]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Loading "mappings.json"
[00:13:32]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Loading "data.json.gz"
[00:13:32]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [ft_categorization] creating index, cause [api], templates [], shards [1]/[0]
[00:13:32]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Created index "ft_categorization"
[00:13:32]                 │ debg [x-pack/test/functional/es_archives/ml/categorization] "ft_categorization" settings {"index":{"number_of_replicas":"0","number_of_shards":"1"}}
[00:13:33]                 │ info [x-pack/test/functional/es_archives/ml/categorization] Indexed 1501 docs into "ft_categorization"
[00:13:33]                 │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:13:33]               └-> valid with good number of tokens
[00:13:33]                 └-> "before each" hook: global before each for "valid with good number of tokens"
[00:13:33]                 └- ✓ pass  (133ms)
[00:13:33]               └-> invalid, too many tokens.
[00:13:33]                 └-> "before each" hook: global before each for "invalid, too many tokens."
[00:13:33]                 │ info [r.suppressed] [node-01] path: /_analyze, params: {}
[00:13:33]                 │      org.elasticsearch.transport.RemoteTransportException: [node-01][127.0.0.1:63121][indices:admin/analyze[s]]
[00:13:33]                 │      Caused by: java.lang.IllegalStateException: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.increment(TransportAnalyzeAction.java:397) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.access$100(TransportAnalyzeAction.java:387) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.simpleAnalyze(TransportAnalyzeAction.java:229) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:204) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:122) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:110) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:99) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.0-SNAPSHOT.jar:7.16.0-SNAPSHOT]
[00:13:33]                 │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
[00:13:33]                 │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
[00:13:33]                 │      	at java.lang.Thread.run(Thread.java:833) [?:?]
[00:13:33]                 └- ✖ fail: apis Machine Learning jobs Categorization example endpoint -  invalid, too many tokens.
[00:13:33]                 │       Error: expected 'partially_valid' to sort of equal 'invalid'
[00:13:33]                 │       + expected - actual
[00:13:33]                 │ 
[00:13:33]                 │       -partially_valid
[00:13:33]                 │       +invalid
[00:13:33]                 │       
[00:13:33]                 │       at Assertion.assert (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:13:33]                 │       at Assertion.eql (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:13:33]                 │       at Context.<anonymous> (test/api_integration/apis/ml/jobs/categorization_field_examples.ts:302:44)
[00:13:33]                 │       at runMicrotasks (<anonymous>)
[00:13:33]                 │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
[00:13:33]                 │       at Object.apply (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:13:33]                 │ 
[00:13:33]                 │ 

Stack Trace

Error: expected 'partially_valid' to sort of equal 'invalid'
    at Assertion.assert (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/ml/jobs/categorization_field_examples.ts:302:44)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (/dev/shm/workspace/parallel/12/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16) {
  actual: 'partially_valid',
  expected: 'invalid',
  showDiff: true
}

---

Metrics [docs]

✅ unchanged

History

• 💔 Build #162178 failed 5a93dba
• 💚 Build #159886 succeeded ea3f54e
• 💛 Build #159205 was flaky ddb291c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream