Aggregating _ignored field values

[aclowkey]

When dealing with lots of data and lots of users, having the ignore_malformed option is great!
And using it in combination with _ignored field can give a lot of information about what is wrong with the data.

But unfortunately we can't aggregate on it, so it's hard to give an overview of which fields had issues in the last X hours.

I think it would be very useful to add .keyword for _ignored field, or make the _ignored field tokenised.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[jtibshirani]

We had an initial discussion and agreed that it seems useful to be able to aggregate over the _ignored field. We didn't reach a final conclusion but plan to continue the discussion. Other notes:

• We'd like to avoid adding another option to the mapping like "doc_values": true. However we weren't sure if it was worth the cost to always enable doc_values.
• One idea was to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values.

[aclowkey]

Thanks for responding! What's the cost difference for enabling doc_values? I mean at scale there could be lots of _ignored fields, could the cost be amplified hugely?

…

On Mon, 3 Aug 2020 at 23:50, Julie Tibshirani ***@***.***> wrote: We had an initial discussion and agreed that it seems useful to be able to aggregate over the _ignored field. We didn't reach a final conclusion but plan to continue the discussion. Other notes: - We'd like to avoid adding another option to the mapping like "doc_values": true. However we weren't sure if it was worth the cost to always enable doc_values. - One idea was to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#59946 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPXFWN4F6HGRMMLISJO7XDR64PJNANCNFSM4PDHS2KQ> .

[davishmcclurg]

We recently started looking at using ignore_malformed in Enterprise Search. Being able to aggregate on the _ignored field would be super helpful for showing users what field types make sense for their data. Let me know if you're interested and I can explain a little bit more about our use case.

[jimczi]

We discussed internally and agreed to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values as Julie proposed. @aclowkey would you still be interested to address this in your PR ?