Will Sandcastle gist creation break because of this?

[pjcozzi]

https://github.com/blog/2503-deprecation-notice-removing-anonymous-gist-creation

Or is Sandcastle not anonymous? I think we made a key at one point.

[mramato]

I believe the key is only there to allow us to create/retrieve more gists-per-day than would be allowed from their non-authenticated API. Unfortunately, it looks like the gists we create are actually anonymous. I found this out by creating a simple gist in Sandcastle and then going directly to the GitHub API for it which gets you to https://gist.github.com/27329c6ff6deb5a09ef406d3b56cf593

There are a few options here:

1. Simply remove the ability to create gists (keep the loading code for existing ones) and add back the feature at a TBD time.
2. Create an account specifically for gists (not sure how well this would work security-wise).
3. Remove and replace with a different method (such as the data uri feature I recently added combined with a URL shortener, pastebin, etc). (requires a little research)

We should probably do something for March 1, since current functionality will break on the 19th.

[pjcozzi]

All - do you use the "Share" feature in Sandcastle?

[ggetz]

I use this all the time for the forum, and that's how about 50% of the community shares their code examples.

[pjcozzi]

OK, we should try to do something for March 1 then.

Does anyone want to look into this?

Perhaps first evaluate if this is reasonable:

2. Create an account specifically for gists (not sure how well this would work security-wise).

Then, if needed:

3. Remove and replace with a different method (such as the data uri feature I recently added combined with a URL shortener, pastebin, etc). (requires a little research)

[kring]

You guys might consider persisting to S3. It's easy to rig up an EC2 instance so it automatically has access to a private S3 bucket. Then you can use code like this in a simple back-end service to save/load to/from S3:
https://github.com/TerriaJS/terriajs-server/blob/master/lib/controllers/share.js#L106

Is Sandcastle on cesiumjs.org still hosted in IIS these days, or has it moved to Node.js?

[pjcozzi]

We could build something into ion for this with S3/etc, but not for March 1. Could also punt a bit past March 1 if the other suggests do not pan out.

[kring]

In a pinch I could probably open a PR to make Sandcastle use our S3-based persistence service on https://map.terria.io. Long term you'll probably want to control it, though.

[pjcozzi]

Thanks, but I don't think we'll want the terria persistence. If you have time, feel free to implement or rule out (2) and (3) above, #6232 (comment)

[hpinkos]

We have a bunch of issues that have code examples stored in sandcastle gists like #5371
Will those all disappear? We should go through and grab the code snippets and paste them in the issues if that's the case.

[ggetz]

@hpinkos The blog post says:

Current anonymous gists will always remain accessible

So it sounds like they will remain available.

[kring]

Thanks, but I don't think we'll want the terria persistence. If you have time, feel free to implement or rule out (2) and (3) above, #6232 (comment)

I'd rule out 2 right away. You'd need a server-side piece to protect your GitHub API key, anyway. If you're doing that, might as well use S3; it's not any harder. Plus we considered the authenticated gist approach for Terria, emailed GitHub to see if they were ok with gists being used in that way, and they told us flat-out no.

I don't think a URL shortener will fly. We used to use the Google shortener, and it worked fine for awhile, until one day Google quietly put a limit on the URL length and broke everything. Not sure about pastebin or other services, but I suspect you'll run into similar problems: these kinds of services aren't meant to be a data store for you app, so at best you'd be getting away with it.

The Terria share service is at least meant for a very similar purpose, and isn't going to randomly go away or start blocking you without warning, so IMO it's better than any of these options. But really I'd only recommend it as a stopgap before you can put a persistence service in Ion or whatever. If the alternative is turning off sharing entirely for awhile, definitely go with the Terria thing. :)

[pjcozzi]

Thanks @kring, will look into what is involved for ion.

[hpinkos]

@mramato @pjcozzi since this feature will break in the middle of the next month, should we remove the share button before the next release goes out?

[pjcozzi]

I was OK with it or potentially updating Sandcastle on the website in the middle of the month.

If you two really want to remove it for now, OK.

[hpinkos]

It's a feature people seem to use fairly often. I think we'll get a bunch of forum questions asking why it's broken if we leave the button there after the 19th. I'm okay leaving it alone until then, as long as it's easy to update the website mid-month. @ggetz will it be easy to just update Sandcastle on the website on March 19th?

[ggetz]

Yep! We just need to update the cesiumjs.org branch and redeploy. I'm in favor of leaving the button there until then.

[shunter]

Since we're fast approaching doomsday here I started prototyping storing code in the URL directly, without a server. Using zlib compression and storing the compressed result base64-encoded in the URL fragment looks like this:

http://localhost:8080/Apps/Sandcastle/#cc=eJy1VV1v2jAU/StWXkil1ECh7UZptX5M66ShdaXay5gq17kQC8dGtkPpKv77rvMBIZ22vWDlwb4+Pvfc44+8BlzHEAyCJTNkKeAZDDknCp7JNViRpfR7HgtbPB9ea+WYUGBaEXmdKIJNqKm+0isyIFMmLURF1IIE7oRWn1UsOHPaNAEJi/Wzxagz2TaoMxlfKpEyB+XURK0PziZqoqaZyhkJN4DTI9Qtw8zIiCQgZok7qBQVZVBQTjgBlhpI9RIupQwLohyD5S60FTnheVXsNTMOe0z16NTo9AZmBsCGh92jHu2c9vsn3fcR6fdp57jTO+2cbDKfbUkTYLFQsy3niLmEOn2PYaZs2O0d1+EL4XiC4E4tZrSUjVCyaGzLbZHmzi+/R3xY5o0KxignqSfSRqAhbLfeB4OSptqkliYNwm8ZemwUwsPKp8jL2LUw9/gFCZueszgOy+3wTbHU76ffrW1w4/9g063N1gUP6qMaJvVnAGdrmXzLjHiTLIcLPFlZeidWIMfil1fUPXrXBLGVB405kx5w1MG2RayL7npjbVG3M4zPIf5Y2VH4gpi1dyv3f+G1W5z7Uap1sHKYoHUpDDds6lqlEETll8dfmPLIhwf1Euvnv0VpG78xSxcSbphj7dwT2y52GLnL3iN26Uw+4cU9xopop6oAFa43l7nS9MnoTMVkCYngEvahrMhQiStGpb6/S7vVjmAt5IpJqbXah7aSendUiuv+27yRkHP/dPH5PsR59gdP3hwfzlP+Xw6O50IpiAlPGB5b5x/zPchklX+P2P2Trp/5SzJmKubMOgn+zXjQWj4xMwKVheWN8UuCKEhcKvE3NbTuRcJFwfJBpAttnL/pIUpxgFJQn20/oR3gKLf56mG7WjSMxZKI+HwSNH5nk4BwyazFmWkm88dhElwM24jfWSZ1/kp+XYKR7MVDku7FlyJIKR22cfh2lSuqqjEG698lIGo+

This is the existing "3D Models" example (1884 bytes) compressed then encoded to a 1014 byte URL. Large but works. The largest existing example is CZML Path which is 85439 bytes and produces a 28942 byte URL. It does load correctly in Chrome at least. I believe that because I'm using the fragment instead of a query parameter, then URL length limits do not apply (the fragment is not sent to the server).

Improvements would be to use base85 instead of 64, and using a better compression algorithm than zlib.

[mramato]

Given the main reason for this feature is to post to the forum and GitHub issues, I very much prefer the url approach, even if it's a little verbose.

@kring to answer your question, cesiumjs.org is hosted on S3 now (via CloudFront). I did look into using lambda or a small shim server to allow for storing of gists directly into S3, but I think it's extra infrastructure that we don't necessarily need.

[mramato]

@shunter what library are you using for the compression? https://github.com/rotemdan/lzutf8.js looks promising and is pretty small (9kb minified).

[pjcozzi]

url approach is OK with me. If it gets out of control, we can re-evaluate in the future. I think most code snippets we share are pretty small - which is exactly what we want for bug reports.

[mramato]

https://github.com/pieroxy/lz-string/ may also be a good choice.

[shunter]

I tested several other compression algorithms and surprisingly zlib (via pako) provides the best compression of anything I tried (LZMA, bzip2, smaz, all of the algorithms from https://www.npmjs.com/package/compressjs)

base85 / ascii85 didn't pan out because it turns out that 14 out of the total 95 printable ASCII characters are not permitted in a URL fragment - "#%< >[\]^`{|} - leaving only 81 characters. Percent-encoding overhead seems like it would counteract the 8% gain of b85 (5/4) over b64 (4/3)

[mramato]

Thanks @shunter, I really appreciate you handling this. I'll review a PR as soon as you have it and we'll get this deployed ASAP.