[Logs Explorer][Meta] Data selector improvements

[ruflin]

In Logs Explorer, that data selector supports integrations, datasets and data views. A first version has been released and several improvements were discussed. This issue is to track the list of potential improvements. For each, separate issues should be created. Potentially some of the issues listed below can be solved in one go with design changes.

Rename uncategorized to dataset and show all datasets

Currently under uncategorized, only the datasets which don't belong to an integration are shown. Instead, all datasets should be shown including the ones belonging to an integration to have a single place to browse all of them. In addition, the title "Uncategorized" should be renamed to "Datasets".

Titles of datasets

Each dataset belonging to an integration has a title defined in the dataset manifest. This title should be shown in the user to make dataset names less abstract. Ideally there is a way to also see somehow the raw name.

Counts of docs in dataset

There is currently no info available if a dataset contains any data or not. For preselection, an estimate (or exact) should be shown of the number of docs in the dataset. The number of docs is also an indicator to the user, how "active" a dataset is.

#171394

Treat empty dataset special

Selecting a dataset that has no data inside and then see an empty screen is not ideal. Instead, empty dataset should be treated in a special way. For integrations, in many cases not even the data stream itself exists before data is shipped. This means there are 2 scenarios that could be treated differently: data stream does not exist, data stream exists but is empty

Improve selection of data views

Logs users with log data views should stay in Logs Explorer. More details can be found here: #172469

Support for CCS for data selection

CCS must be supported for integration and dataset selection. More details can be found here: #172905

Edit integration / datasets

Some ideas were floating around how the integration selector can also be used to allow users to jump to the dataset quality page / add an edit option and similar. More details needed.

Support multi select

Users cannot only select a single dataset but select multiple or a full integration. More details here (private link)

Support for full text search

Searching for integrations currently only takes into account the raw name. e.g. when searching for "System" the system integration is not returned even though it is displayed as such in the dropdown menu.

Highlight search term

The search term should be highlighted in the list of results so that users understand why an integration or dataset matched.

Ability to search for datasets across all integrations

Search currently only returns matches from the active view. For example when entering a search term that should match an existing dataset on the Integrations view no results are returned. Users have to manually drill down and select the correct integration first before they can search for datasets.
We should create a combined view when searching where we can show all matching datasets across all integrations.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[maryam-saeidi]

Suggestion: Improving data selector for no integrations found scenario. (comment)

1. @isaclfreire 's input: adding add data button:

   

2. Not underlining No integrations found(1) and Integrations(2) on hover to avoid giving the impression that they are clickable.
   

[isaclfreire]

Regarding this section:

Treat empty dataset special
Selecting a dataset that has no data inside and then see an empty screen is not ideal. Instead, empty dataset should be treated in a special way. For integrations, in many cases not even the data stream itself exists before data is shipped. This means there are 2 scenarios that could be treated differently: data stream does not exist, data stream exists but is empty

If a data stream doesn't exist yet before the data is shipped, would it appear in the dataset list anyway? We may want it to be disabled for clicking and mark it as in progress somehow to allow users to know that it is 'loading'.

data stream exists but is empty

In this scenario, what are the main reasons why a data stream would be empty? And what would be the main actions users can take from an empty data stream to change this scenario? Do we want them to do checks to the ingestion? This could be an opportunity to 'fix' the flow if empty data stream is caused by some error, but we need to list them, if we know.

[tonyghiani]

@isaclfreire showing the last activity indicator as "Empty" next to a dataset and disabling it from being selectable I think is the most explicit way to tell the user what's going on.
It communicates this dataset can exist if data are ingested, but it simply doesn't have anything to show at the moment.

[isaclfreire]

I started to address the main topics of this issue in this Figma page, feel free to comment.

When it comes to point 3 (Counts of docs in a dataset, or mainly showing the dataset activity) please refer to this Figma page. I have started working on point 4 (Treat empty dataset special), and will share the explorations later on this week.

I haven't address yet these use cases:

• Support for CCS for data selection
• Highlight search term
• Ability to search for datasets across all integrations

They seem like big topics, so we might need to split the issue and add more context.

[ruflin]

I don't expect we can solve all the issues at once and most prioritise the list.

[tonyghiani]

2. Not underlining No integrations found(1) and Integrations(2) on hover to avoid giving the impression that they are clickable.
   

@maryam-saeidi this was fixed here on the EUI side, thanks for reporting that!