Skip to content

Commit

Permalink
Storage/Merge STG95 to main (#5885)
Browse files Browse the repository at this point in the history
* Storage/STG95 Files OAuth - full data plane support (#5759)

* File OAuth

* Remove comment

* update test record

* [Storage STG95 Branch] Merge Main to fix cmake build error (#5796)

* Update vcpkg SHA (#5757)

Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>

* Sync eng/common directory with azure-sdk-tools for PR 8528 (#5755)

* Fix default value for env vars in build-test-resource-config template

* Add empty pool condition

---------

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 8516 (#5752)

* Ensure subConfigFiles is not an empty string

* Skip instances where $file is an empty string

---------

Co-authored-by: Daniel Jurek <djurek@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 8549 (#5762)

* commit the file changes so that we can see them running

* use standalone tool

---------

Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>

* Disable codeql as it is causing build issues (#5761)

All the other languages have disabled codeql and we are now hitting blocking issues when running codeql on macos (https://dev.azure.com/twcdot/Data/_workitems/edit/138580) and even on the other OS's the configuration causes the builds to run for a long time.

* Delete eng/common/InterdependencyGraph.html which is unused. (#5763)

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Add missing include (#5766)

* Update GitHubEventProcessor version to 1.0.0-dev.20240708.1 (#5768)

Co-authored-by: James Suplizio <jasupliz@microsoft.com>

* Provide a walkthrough for how to install the Azure SDK packages in a VS (#5751)

MSBuild project via vcpkg.

* Sync eng/common directory with azure-sdk-tools for PR 8558 (#5764)

* Support storage network access and worm removal in remove test resources script

* Move storage network access script to common resource helpers file

* Improve storage container deletion resilience

* Plumb through pool variable to live test cleanup template

* Add sleep for network rule application

---------

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* add the ability to override default succeeded() conditioning by parameter (#5780)

Co-authored-by: Scott Beddall <scbedd@microsoft.com>

* Support regex/negative regex filters for stress test discovery. Add storage env defaults (#5779)

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Update vcpkg SHA (#5772)

* Show-FailureLogs to include vcpkg build failure logs (#5776)

* Show-FailureLogs to include vcpkg build failure logs

* Add comment

* Add proper array syntax

* Use proper syntax to create an array even if there's only a single element

Co-authored-by: Ben Broderick Phillips <ben@benbp.net>

---------

Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>

* Turn federated auth on for Identity tests. (#5785)

* Turn federated auth on for Identity tests.

* Update test resources json.

* Move tests to use azure pipeline credentials  (#5754)

* test1

* hgdfchg

* remove the remnants of azure client secret

* test KV with federated auth

* UseFederatedAuth

* fdsa

* kv template with managed

* try try again

* retry permissions

* add net acls

* blunt force replace the resource json

* put back stuff

* trey again with new method

* attempt

* missed something

* flip if else

* Temporarily use empty sub config file path for preview cloud

* remove client secret

* try to fix the identity tests

* live skip failing tests and return in samples

* samples for identity fix

* disable failing samples in identity

* fix winhttp failing test

* comment out code

* remove managed identity

* restore version from main

* revert readme changes

* PR comments

* test 2

* clang

* attempt default creds with pipeline chanined

* clangs

* identity test and clangs

* oops

* live

* cleanup

* reter

* test

* revert the DAC change

* missed one

* taking the samples to a farm upstate

* PR comments

* Fix bad merge

---------

Co-authored-by: Daniel Jurek <djurek@microsoft.com>
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension
point hidden.

* Mark test helper virtual functions private, so they aren't
accessible/callable by callers.

* Update the changelog.

* Update CL.

* Azure Core July GA Release (#5792)

Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>

* Revert commits related to the new RetryPolicy method (#5793)

* Revert "Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)"

This reverts commit 9ccd206.

* Revert "Update the RetryPolicy and ShouldRetry customization logic to allow loosening the retry condition. (#5656)"

This reverts commit f1d9552.

* Do not remove changelog entry from a previous beta release

* Revert "Add a virtual ShouldRetry method to the RetryPolicy for customization. (#5584)"

This reverts commit ab90ef6.

---------

Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>

* Sync eng/common directory with azure-sdk-tools for PR 8598 (#5777)

* Set storage account test resources to disable blob public access

* Skip adding network rules to storage accounts that don't need them during cleanup

* Add succeeded check to set pipeline subnet info step

* Disable network firewall by default in resource creation/removal

---------

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Increment package version after release of azure-core (#5794)

* Acknowledge community contribution in the changelog (#5797)

* Mention community contribution in the changelog

* cspell

* Remove double spaces

---------

Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>

---------

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Robert Schulze <robert@clickhouse.com>
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
Co-authored-by: George Arama <50641385+gearama@users.noreply.github.com>

* Storage/STG95 Premium Files Paid Burst (#5795)

* paid burst

* update record

* Storeage/STG95 Files Binary ACE (#5824)

* feature code

* update test case to playback only

* update records

* fix cspell

* update record

* fix cspell

* fix cspell

* fix comments

* Remove copy opreation rest code.

* Storage/STG95 Support Sas StringToSign for debugging (#5830)

* StringToSign

* Update deprecate

* Update Func name

* ignore deprecated warning

* Add missing ignore

* fix clang format

* Storage/STG95 Resolve Archboard Review Comments (#5856)

* fix comments

* fix comments

* Add Create dir and upload from support (#5866)

* merge test recordings

* Remove other changes

* Remove deprecated (#5873)

* Remove others

---------

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Robert Schulze <robert@clickhouse.com>
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
Co-authored-by: Ahson Khan <ahson_ahmedk@yahoo.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
Co-authored-by: George Arama <50641385+gearama@users.noreply.github.com>
  • Loading branch information
13 people authored Aug 7, 2024
1 parent 7d5c76c commit 999709a
Show file tree
Hide file tree
Showing 34 changed files with 1,816 additions and 751 deletions.
2 changes: 2 additions & 0 deletions .vscode/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,8 @@
"Schulze",
"scus",
"SDDL",
"Sddl",
"sddl",
"sdpath",
"serializers",
"Seriot",
Expand Down
2 changes: 1 addition & 1 deletion sdk/storage/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "cpp",
"TagPrefix": "cpp/storage",
"Tag": "cpp/storage_c3a4a21ce9"
"Tag": "cpp/storage_90767ef070"
}
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,29 @@ namespace Azure { namespace Storage { namespace Sas {
const Blobs::Models::UserDelegationKey& userDelegationKey,
const std::string& accountName);

/**
* @brief For debugging purposes only.
*
* @param credential
* The storage account's shared key credential.
* @return Returns the string to sign that will be used to generate the signature for the SAS
* URL.
*/
std::string GenerateSasStringToSign(const StorageSharedKeyCredential& credential);

/**
* @brief For debugging purposes only.
*
* @param userDelegationKey UserDelegationKey returned from
* BlobServiceClient.GetUserDelegationKey.
* @param accountName The name of the storage account.
* @return Returns the string to sign that will be used to generate the signature for the SAS
* URL.
*/
std::string GenerateSasStringToSign(
const Blobs::Models::UserDelegationKey& userDelegationKey,
const std::string& accountName);

private:
std::string Permissions;
};
Expand Down
610 changes: 213 additions & 397 deletions sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/rest_client.hpp

Large diffs are not rendered by default.

80 changes: 80 additions & 0 deletions sdk/storage/azure-storage-blobs/src/blob_sas_builder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -323,4 +323,84 @@ namespace Azure { namespace Storage { namespace Sas {
return builder.GetAbsoluteUrl();
}

std::string BlobSasBuilder::GenerateSasStringToSign(const StorageSharedKeyCredential& credential)
{
std::string canonicalName = "/blob/" + credential.AccountName + "/" + BlobContainerName;
if (Resource == BlobSasResource::Blob || Resource == BlobSasResource::BlobSnapshot
|| Resource == BlobSasResource::BlobVersion)
{
canonicalName += "/" + BlobName;
}
std::string protocol = _detail::SasProtocolToString(Protocol);
std::string resource = BlobSasResourceToString(Resource);

std::string snapshotVersion;
if (Resource == BlobSasResource::BlobSnapshot)
{
snapshotVersion = Snapshot;
}
else if (Resource == BlobSasResource::BlobVersion)
{
snapshotVersion = BlobVersionId;
}

std::string startsOnStr = StartsOn.HasValue()
? StartsOn.Value().ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate)
: "";
std::string expiresOnStr = Identifier.empty()
? ExpiresOn.ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate)
: "";

return Permissions + "\n" + startsOnStr + "\n" + expiresOnStr + "\n" + canonicalName + "\n"
+ Identifier + "\n" + (IPRange.HasValue() ? IPRange.Value() : "") + "\n" + protocol + "\n"
+ SasVersion + "\n" + resource + "\n" + snapshotVersion + "\n" + EncryptionScope + "\n"
+ CacheControl + "\n" + ContentDisposition + "\n" + ContentEncoding + "\n" + ContentLanguage
+ "\n" + ContentType;
}

std::string BlobSasBuilder::GenerateSasStringToSign(
const Blobs::Models::UserDelegationKey& userDelegationKey,
const std::string& accountName)
{
std::string canonicalName = "/blob/" + accountName + "/" + BlobContainerName;
if (Resource == BlobSasResource::Blob || Resource == BlobSasResource::BlobSnapshot
|| Resource == BlobSasResource::BlobVersion)
{
canonicalName += "/" + BlobName;
}
std::string protocol = _detail::SasProtocolToString(Protocol);
std::string resource = BlobSasResourceToString(Resource);

std::string snapshotVersion;
if (Resource == BlobSasResource::BlobSnapshot)
{
snapshotVersion = Snapshot;
}
else if (Resource == BlobSasResource::BlobVersion)
{
snapshotVersion = BlobVersionId;
}

std::string startsOnStr = StartsOn.HasValue()
? StartsOn.Value().ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate)
: "";
std::string expiresOnStr = ExpiresOn.ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate);
std::string signedStartsOnStr = userDelegationKey.SignedStartsOn.ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate);
std::string signedExpiresOnStr = userDelegationKey.SignedExpiresOn.ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate);

return Permissions + "\n" + startsOnStr + "\n" + expiresOnStr + "\n" + canonicalName + "\n"
+ userDelegationKey.SignedObjectId + "\n" + userDelegationKey.SignedTenantId + "\n"
+ signedStartsOnStr + "\n" + signedExpiresOnStr + "\n" + userDelegationKey.SignedService
+ "\n" + userDelegationKey.SignedVersion + "\n\n\n\n"
+ (IPRange.HasValue() ? IPRange.Value() : "") + "\n" + protocol + "\n" + SasVersion + "\n"
+ resource + "\n" + snapshotVersion + "\n" + EncryptionScope + "\n" + CacheControl + "\n"
+ ContentDisposition + "\n" + ContentEncoding + "\n" + ContentLanguage + "\n" + ContentType;
}

}}} // namespace Azure::Storage::Sas
2 changes: 1 addition & 1 deletion sdk/storage/azure-storage-blobs/swagger/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ package-name: azure-storage-blobs
namespace: Azure::Storage::Blobs
output-folder: generated
clear-output-folder: true
input-file: https://raw.github.com/Azure/azure-rest-api-specs/main/specification/storage/data-plane/Microsoft.BlobStorage/stable/2021-12-02/blob.json
input-file: https://raw.github.com/Azure/azure-rest-api-specs/main/specification/storage/data-plane/Microsoft.BlobStorage/stable/2024-08-04/blob.json
```
## ModelFour Options
Expand Down
78 changes: 78 additions & 0 deletions sdk/storage/azure-storage-blobs/test/ut/blob_sas_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -771,4 +771,82 @@ namespace Azure { namespace Storage { namespace Test {
EXPECT_TRUE(e.AdditionalInformation.count("ExtendedErrorDetail") != 0);
}
}

TEST(SasStringToSignTest, GenerateStringToSign)
{
std::string accountName = "testAccountName";
std::string accountKey = "dGVzdEFjY291bnRLZXk=";
std::string blobUrl = "https://testAccountName.blob.core.windows.net/container/blob";
auto keyCredential = std::make_shared<StorageSharedKeyCredential>(accountName, accountKey);
auto sasStartsOn = std::chrono::system_clock::now() - std::chrono::minutes(5);
auto sasExpiresOn = std::chrono::system_clock::now() + std::chrono::minutes(60);

// Account Sas
{
Sas::AccountSasBuilder accountSasBuilder;
accountSasBuilder.Protocol = Sas::SasProtocol::HttpsAndHttp;
accountSasBuilder.StartsOn = sasStartsOn;
accountSasBuilder.ExpiresOn = sasExpiresOn;
accountSasBuilder.Services = Sas::AccountSasServices::Blobs;
accountSasBuilder.ResourceTypes = Sas::AccountSasResource::All;
accountSasBuilder.SetPermissions(Sas::AccountSasPermissions::Read);
auto sasToken = accountSasBuilder.GenerateSasToken(*keyCredential);
auto signature = Azure::Core::Url::Decode(
Azure::Core::Url(blobUrl + sasToken).GetQueryParameters().find("sig")->second);
auto stringToSign = accountSasBuilder.GenerateSasStringToSign(*keyCredential);
auto signatureFromStringToSign = Azure::Core::Convert::Base64Encode(_internal::HmacSha256(
std::vector<uint8_t>(stringToSign.begin(), stringToSign.end()),
Azure::Core::Convert::Base64Decode(accountKey)));
EXPECT_EQ(signature, signatureFromStringToSign);
}

// Blob Sas
{
Sas::BlobSasBuilder blobSasBuilder;
blobSasBuilder.Protocol = Sas::SasProtocol::HttpsAndHttp;
blobSasBuilder.StartsOn = sasStartsOn;
blobSasBuilder.ExpiresOn = sasExpiresOn;
blobSasBuilder.BlobContainerName = "container";
blobSasBuilder.BlobName = "blob";
blobSasBuilder.Resource = Sas::BlobSasResource::Blob;
blobSasBuilder.SetPermissions(Sas::BlobSasPermissions::Read);
auto sasToken = blobSasBuilder.GenerateSasToken(*keyCredential);
auto signature = Azure::Core::Url::Decode(
Azure::Core::Url(blobUrl + sasToken).GetQueryParameters().find("sig")->second);
auto stringToSign = blobSasBuilder.GenerateSasStringToSign(*keyCredential);
auto signatureFromStringToSign = Azure::Core::Convert::Base64Encode(_internal::HmacSha256(
std::vector<uint8_t>(stringToSign.begin(), stringToSign.end()),
Azure::Core::Convert::Base64Decode(accountKey)));
EXPECT_EQ(signature, signatureFromStringToSign);
}

// Blob User Delegation Sas
{
Blobs::Models::UserDelegationKey userDelegationKey;
userDelegationKey.SignedObjectId = "testSignedObjectId";
userDelegationKey.SignedTenantId = "testSignedTenantId";
userDelegationKey.SignedStartsOn = sasStartsOn;
userDelegationKey.SignedExpiresOn = sasExpiresOn;
userDelegationKey.SignedService = "b";
userDelegationKey.SignedVersion = "2020-08-04";
userDelegationKey.Value = accountKey;

Sas::BlobSasBuilder blobSasBuilder;
blobSasBuilder.Protocol = Sas::SasProtocol::HttpsAndHttp;
blobSasBuilder.StartsOn = sasStartsOn;
blobSasBuilder.ExpiresOn = sasExpiresOn;
blobSasBuilder.BlobContainerName = "container";
blobSasBuilder.BlobName = "blob";
blobSasBuilder.Resource = Sas::BlobSasResource::Blob;
blobSasBuilder.SetPermissions(Sas::BlobSasPermissions::Read);
auto sasToken = blobSasBuilder.GenerateSasToken(userDelegationKey, accountName);
auto signature = Azure::Core::Url::Decode(
Azure::Core::Url(blobUrl + sasToken).GetQueryParameters().find("sig")->second);
auto stringToSign = blobSasBuilder.GenerateSasStringToSign(userDelegationKey, accountName);
auto signatureFromStringToSign = Azure::Core::Convert::Base64Encode(_internal::HmacSha256(
std::vector<uint8_t>(stringToSign.begin(), stringToSign.end()),
Azure::Core::Convert::Base64Decode(accountKey)));
EXPECT_EQ(signature, signatureFromStringToSign);
}
}
}}} // namespace Azure::Storage::Test
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,16 @@ namespace Azure { namespace Storage { namespace Sas {
*/
std::string GenerateSasToken(const StorageSharedKeyCredential& credential);

/**
* @brief For debugging purposes only.
*
* @param credential
* The storage account's shared key credential.
* @return Returns the string to sign that will be used to generate the signature for the SAS
* URL.
*/
std::string GenerateSasStringToSign(const StorageSharedKeyCredential& credential);

private:
std::string Permissions;
};
Expand Down
46 changes: 46 additions & 0 deletions sdk/storage/azure-storage-common/src/account_sas_builder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -144,4 +144,50 @@ namespace Azure { namespace Storage { namespace Sas {
return builder.GetAbsoluteUrl();
}

std::string AccountSasBuilder::GenerateSasStringToSign(
const StorageSharedKeyCredential& credential)
{
std::string protocol = _detail::SasProtocolToString(Protocol);

std::string services;
if ((Services & AccountSasServices::Blobs) == AccountSasServices::Blobs)
{
services += "b";
}
if ((Services & AccountSasServices::Queue) == AccountSasServices::Queue)
{
services += "q";
}
if ((Services & AccountSasServices::Files) == AccountSasServices::Files)
{
services += "f";
}

std::string resourceTypes;
if ((ResourceTypes & AccountSasResource::Service) == AccountSasResource::Service)
{
resourceTypes += "s";
}
if ((ResourceTypes & AccountSasResource::Container) == AccountSasResource::Container)
{
resourceTypes += "c";
}
if ((ResourceTypes & AccountSasResource::Object) == AccountSasResource::Object)
{
resourceTypes += "o";
}

std::string startsOnStr = StartsOn.HasValue()
? StartsOn.Value().ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate)
: "";
std::string expiresOnStr = ExpiresOn.ToString(
Azure::DateTime::DateFormat::Rfc3339, Azure::DateTime::TimeFractionFormat::Truncate);

return credential.AccountName + "\n" + Permissions + "\n" + services + "\n" + resourceTypes
+ "\n" + startsOnStr + "\n" + expiresOnStr + "\n"
+ (IPRange.HasValue() ? IPRange.Value() : "") + "\n" + protocol + "\n" + SasVersion + "\n"
+ EncryptionScope + "\n";
}

}}} // namespace Azure::Storage::Sas
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,29 @@ namespace Azure { namespace Storage { namespace Sas {
const Files::DataLake::Models::UserDelegationKey& userDelegationKey,
const std::string& accountName);

/**
* @brief For debugging purposes only.
*
* @param credential
* The storage account's shared key credential.
* @return Returns the string to sign that will be used to generate the signature for the SAS
* URL.
*/
std::string GenerateSasStringToSign(const StorageSharedKeyCredential& credential);

/**
* @brief For debugging purposes only.
*
* @param userDelegationKey UserDelegationKey returned from
* BlobServiceClient.GetUserDelegationKey.
* @param accountName The name of the storage account.
* @return Returns the string to sign that will be used to generate the signature for the SAS
* URL.
*/
std::string GenerateSasStringToSign(
const Blobs::Models::UserDelegationKey& userDelegationKey,
const std::string& accountName);

private:
std::string Permissions;
};
Expand Down
Loading

0 comments on commit 999709a

Please sign in to comment.