Smaller state representation to fix IE errors due URL length

[rashidkpc]

Currently we rison encode the full state of the app in the URL. This can cause problems with very complex dashboards and IE as IE limits URLs to 2,083 characters.

Opening this ticket for a discussion of options. In theory this should all be pretty modifiable via the state service. We could start pushing state into localStorage, or sessionStorage, though we would need to continue providing support for RISON at least for initializing state and for backwards compat

Needs to go hand in hand with #1553 to allow for sharing.

[rcrezende]

+1

[yehosef]

perhaps you could use something like http://pieroxy.net/blog/pages/lz-string/index.html using compressToEncodedURIComponent

It won't be editable by hand but I'm not sure if that matters.

[simianhacker]

Do you have to be able to still share a URL and receive the full state just by coping it from the URL bar? Can it be a more explicit action like a button to share a link (similar to how the URL shortener PR works)? I could imagine moving all the state to localStorage, then having to click a share button to create a unique URL that will load the state from ES and restore it in localStorage. Then from that point on you've essentially forked the state. The way codepen or gists work come to mind.

How will something like this work with the back button?

[spalger]

@simianhacker I personally think we should use history.pushState() and history.state to manage it if we are going to move it from the URL. That is what they are for after all.

[rashidkpc]

@stormpython are you actively working on this?

[stormpython]

@rashidkpc, I dropped it for a while in favor of work for alpha 5.0. I planned to start working on it again this week.

[stormpython]

It looks like I won't get to this again this week. So I will drop off of this and pick it up if no one else adopts it later.

[pemontto]

👍

[burgerzoid]

We have recently migrated from Kibana 3 to 4 and the URL length immediately became a serious issue. Many of our customers are on SOE builds and have no option but to use IE so we are having to reduce dashboards to the bare minimum number of visualizations. Would be great to see this resolved.

[SJovic777]

Any progress on this Issue?
I cannot put my Elasticstack into production, because only IE is allowed browser in our company.

[djontra]

+1

[markharwood]

The Graph workspace is a complex bundle of entirely transient state right now (vertices, edges, selections, groupings, blacklists, undo/redo action lists....). It would nice to be able to retain it while bouncing around into other areas of Kibana without requiring an explicit save operation.

Is there a security risk associated with local storage to consider? Some of the information in the workspace might be confidential.

[gmoskovicz]

@rashidkpc @epixa

This is a very recurrent issue. It looks like it is useless Kibana with IE11 when there are a bunch of pinned filters on it, which are included in the URL. Do we have any workarounds for this, other than remove the filters and include that in the query_string from the search box in kibana (manual workaround)?

Are we expecting to fix this anytime soon?

[epixa]

There is no workaround for it, and we are planning on tackling this post 5.0.

On Jun 9, 2016, at 7:27 PM, Gabriel Moskovicz notifications@github.com wrote:

@rashidkpc @epixa

This is a very recurrent issue. It looks like it is useless Kibana with IE11 when there are a bunch of pinned filters on it, which are included in the URL. Do we have any workarounds for this, other than remove the filters and include that in the query_string from the search box in kibana (manual workaround)?

Are we expecting to fix this anytime soon?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[SJovic777]

Well, it looks like i will have to delay rollout of Elasticstack (or even drop internal project all togehter), because company policy does not allow using different browser than IE11/Edge.

Sorry, but this is showstopper for me, and many many other people...

[eamonngryan]

+1 Please address this. The URL does need to remain editable/human-readable however.

[Na2l]

+1,000,000

[cjcenizal]

@eamonngryan Can you elaborate on your use case(s) for the human-readable/editable URLs? More information will help us craft a solution that can meet your needs.

[cjcenizal]

Internet Explorer URL limit exploration

I set up an experiment where I visited a link containing a 3002-character query-string in a couple different ways. Here are the results.

Clicking the link

Clicking the link opens a new tab with the long URL. Copy-pasting the URL into Notepad reveals the URL has been truncated to 2083 characters, but logging the query string in the console reveals that it is 3002 characters long.

So JavaScript does have access to the original query string, though the user doesn't. And only if you access a URL by clicking a link.

Copy/pasting the link

When I copy/pasted this link into a new tab, the URL was truncated to 2048 characters, and logging the query string in the console revealed it had been truncated to 1971 characters.

So JavaScript doesn't have access to the original query string if the link is copy/pasted by hand.

Next steps

So where does this leave us? Here are my thoughts.

1. We have the ability to access a long query string when the IE user has clicked a link to open Kibana. In situations where the user copy/pastes the URL into IE, we need to detect this edge case (e.g. by detecting an improperly-encoded app state), and surface a message explaining the problem and directing the user to click the link instead.
2. On load, we should decode the long query string, remove it from the URL, and store the decoded query string in our app state services. If there is no query string, we should try to load, decode, and store a similar value from storage.
3. Whenever the app state services have a new app state set (e.g. in this onload step and when the user makes changes to it by changing a filter or what-have-you), the app state is encoded and stored in a "serialized app state" service, which persists the value in storage.
4. This serialized app state can then be surfaced as a link via "Share" or "Deep-link" buttons in the UI. Clicking such a link would bring you back to step 1.

Notes

• This can be applied to all browsers, not just IE.
• Maybe only certain parts of the app state should be serialized and some other parts can remain in the URL, e.g. time range? I'll look into this.
• This solution involves removing the state from the URL entirely, so that clicking the back button will only affect route changes. This seems like a better UX. Thoughts?

[epixa]

I think clicking the back button should still walk through UI state as it is now. It's a big change, and if at all possible, I'd like to roll this out for 4.6.0... but we can't do that if we break BC in behaviors as fundamental as browser navigation.

[trevan]

What about having a custom rison encoder/decoder or a pre-pass that minimizes the json object?

As an example, each panel adds the following to the string:

(col:1,id:visualization_id,panelIndex:1,row:1,size_x:1,size_y:2,type:visualization)

This is a conversion of the object:

{
col: 1
id: 'visualization_id',
panelIndex: 1,
row: 1,
size_x: 1,
size_y: 2,
type: 'visualization'
}

You could modify that object before sending it into rison as:

{
i: 'visualization_id',
t: 'v',
p: '1,1,1,2'
}

Then the rison format of that is:
(i:visualization_id,t:v,p'1,1,1,2')

Then, after parsing the rison string, you would then reconvert the output back into the normal format before the rest of the system gets it.

That's 84 characters -> 36 characters.

[epixa]

@trevan That would get us shorter urls, but it would still mean that urls are not linkable since there's no way to guarantee that the converted properties would be the same from machine to machine.

In the end, a 50% or even 100% improvement isn't enough. Our state capacity for non-ie browsers is 30 times larger than url limits in IE, which is the level we need to establish in IE unless we want this to continue to be a problem for us. That rules out minification or compression.

@cjcenizal Having back/forward capabilities on state should still be doable using the html5 pushstate api. We can throw arbitrary values in the url to differentiate requests, and then we can actually persist the ui/app state in the history api itself. All of this should be doable in a way that's either browser-specific or opt-in.

[trevan]

@epixa, I don't understand how it would mean that the urls are not linkable? The only problem with my proposal is that a link from Kibana5 won't work in Kibana4. But a link from Kibana4 would actually work in Kibana5.

Its sounding like the direction planned is to make the url useless for sharing between people; that you'll require people to use the share link to get a url that they can share to others. Am I mistaken? Because I think that route will cause confusion since people naturally assume they can copy the url and send it to others.

[epixa]

@trevan How would we definitively and universally convert a property name into a consistent short name? How would, for example, we know that "visualization_id" should always be shortened to "i" whereas "someother_id" should always be shortened to "s" while guaranteeing that no other properties will conflict with that logic? And keep in mind that we also store non-trivial json structures in the rison, so the same logic would need to apply to their properties as well. The challenge isn't doing that on one machine, the challenge is consistently reversing the conversion on another machine, which would mean that there would need to be some set of universal logic that we could apply to all possible properties (in any combination) to do the conversion.

And again, even if we did that, it would only be a bandaid for IE. You'd still have an upper limit that from a practical perspective will be hit by IE users on things like complex dashboards.

[trevan]

@epixa, you only convert the parts that you know Kibana controls and that is universally the same. As I said, every panel in the dashboard creates that object. So, when doing the rison conversion, you know that if you see that object, you know that you are dealing with a panel and you can shrink it under a deterministic method; keys/objects that are not known don't change. I'm not talking about a universal shrinker. I'm talking about a Kibana specific shrinker. It shrinks the keys/objects that it manages.

And yes, this would be a bandaid. But it sounds like the direction is to just give up on the url and force users to use the share link. Am I misunderstanding?

[epixa]

@trevan A solution based on a complete understanding of all properties that could possibly be in a kibana url isn't really practical since plugins can add any data they want to state at any time.

You're partially right about the sharing url thing, but that's why we're considering things like making this IE-only or an opt-in behavior. Someone that has to use IE is probably happier with limitations with url sharing than with not being able to use Kibana at all. Plus, as @cjcenizal mentioned, there's the possibility that certain key state data might still be able to exist in the url (like time range). Things like saved dashboard names are stored in the route itself, so that isn't even a part of the "state". So it's possible we can store the vast majority of state with pushstate and still people would be able to do things like copy their current url for the sake of linking to a saved dashboard and such.

[eamonngryan]

@cjcenizal - the reason we need the human-readable/editable URLs is because we create webpages and emails that contain links to Kibana dashboards with certain filters applied. While it was not very easy to figure out what to change in the URL for the filter value, we were able to figure it and it is now a key part of our offering to users. Without human-readable/editable URLs, I don't think we could use Kibana. We are paying gold level subscription customers btw.

[cjcenizal]

Thanks you @eamonngryan ! We'll take your use case into consideration as we address this problem.

[epixa]

@eamonngryan We're not going to be able to simultaneously solve this and allow all of the existing state to be modified in the URL, so to be clear: you specifically are modifying filters in the url?

[eamonngryan]

@epixa - We only care about modifying the filters (and search field contents) in the URL.