Allow non-Git indices

[jdemilledt]

Describe the problem you are trying to solve

Currently, indices are downloaded via Git. For public repositories, this is not an issue. However, for private repositories, this could be. The authentication issue is part of #7091, but sometimes, the repository owner may wish to filter who sees what crates. Automating this would be quite difficult due to the nature of Git.

Describe the solution you'd like

I propose to allow non-Git indices where a separate URL can indicate updates, and a tarball of the index is downloaded. Authentication info is also supplied for this request.

Notes

[ehuss]

I'm going to close as I believe this is mostly resolved by #9069.