website: long TTFB wait for /pkg/*

[mark-rushakoff]

There was a previous mailing list post at https://groups.google.com/forum/#!topic/golang-nuts/fFEuIHf6NWE about this issue. @broady fixed it once back in June but it's happening again, as @stevenh mentioned about a week ago.

Just now, with Chrome's network profiler open, when I load https://golang.org/pkg/net/http/, I see waiting TTFB at 5.94s; loading https://golang.org/pkg/encoding/json/ spent 3.62s waiting TTFB. But then if I load another package's documentation in the same tab, it seems to load acceptably fast. Sometimes other packages in a new tab load quickly, so I don't understand if it's actually intermittent, or if Chrome is doing something that speeds up subsequent connections, or something else altogether.

Is there other information I or anyone else can gather to help with this issue?

[mark-rushakoff]

@gopherbot, please remove label documentation

[agnivade]

@andybons

[benhoyt]

I've definitely been noticing this too. I don't know what powers the Go documentation, but it seems to me it could be a very simple and super-fast static site hosted via S3+Cloudfront (or the Google Cloud equivalents).

[chtzvt]

I'd imagine the package docs on golang.org are powered by the godoc tool's http server, no?

[andybons]

@ctrezevant that’s correct.

We are hitting OOM errors on GAE Classic. I’m working on the issue. Moving to flex will help with this (#27205).

[benhoyt]

Isn't the documentation very static? Seems like the fewer moving parts here the better. Could it be output to HTML once and then hosted on a static store (eg: S3) behind a CDN (eg: Cloudfront)? This would make it really stable and fast.

[chtzvt]

@andybons Ok, awesome!

Also, the other commentors here make a good point- you could realize significant performance gains if you were using static pages for docs right now, though I have another suggestion that would allow you to keep your current godoc workers and leverage some features from GAE/GCP to improve performance. This would also buy you some time, since these improvements wouldn't be contingent on finalizing #27205.

Since you're already running on GAE (and leveraging GLB, which I'm assuming based on golang.org's response headers), my recommendation would be to leverage the Cloud CDN integration of Google Load Balancer. This will allow GLB to automatically begin caching common requests from your load balancer, and serve the responses to those from Cloud CDN.

You can set this up fairly easily from your backend load balancer definition for golang.org, by ticking the “Enable Cloud CDN” box. From then on GLB/CDN should handle all the caching transparently. Ultimately, this configuration would allow you to continue running godoc workers upstream in GAE, but significantly improve performance while reducing load on your actual godoc servers.

The only caveat I can see is that you'd most likely have to make some patches to godoc to ensure that the responses it generates suit the Cacheability criteria for Cloud CDN (as responses from Godoc currently lack any Content-Length, Transfer-Encoding, Cache-Control, or Content-Range headers). However, godoc's web server should really be responding with some caching headers, anyways (or at least an etag!), since these are vital to any caching web infrastructure and I'd imagine are the primary reason that none of Godoc's responses are getting cached (I may open an issue/PR for this at a later time).

Some more useful info about GLB+Cloud CDN can be found here: https://medium.com/@duhroach/removing-the-need-for-caching-servers-with-gcps-load-balancers-ae516497c7fb

Thanks again!

[agnivade]

It is not static.

• There is a search functionality.
• Template logic which fills up different templates which finally serve the pages.
• And other routing logic which needs Go code.

The advantage of having everything in a single binary is simplicity. You can locally run godoc very easily. Separating out html and assets, and having a separate binary for business logic is going to unnecessarily complicate things.

[benhoyt]

Fair enough @agnivade. What about @ctrezevant's suggestion about putting a caching CDN in front for maximum performance?

[andybons]

https://tip.golang.org is an example of how well godoc can perform on its own. Let’s start with getting there via moving to flex and then we can explore further optimizations afterward.

[bcmills]

What about @ctrezevant's suggestion about putting a caching CDN in front for maximum performance?

The usual fix for OOM issues is flow control, not (just) caching.

(Caching helps with overall throughput, which reduces average load, but OOMs are caused by instantaneous load.)

[broady]

Should be fast again now.

FWIW, the new App Engine flex instances are using about 4 GB memory. I believe this includes the operating system and sidecar containers (e.g., for logging).