Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue-2734 - Clamp buffer to maximum upon large write operation #2745

Merged
merged 18 commits into from
Sep 18, 2024
Merged

Issue-2734 - Clamp buffer to maximum upon large write operation #2745

merged 18 commits into from
Sep 18, 2024

Conversation

ali-ahsan-ali
Copy link
Contributor

Fix Issue 2734 by clamping storage size after large write operation

Motivation:

Fix Issue 2734

Modifications:

  • Added a function to clamp storage by copying bytes and setting new capacity of storage
  • Adding a function to clamp the capacity of ByteBuffer
  • Added the ability to specify he maxBufferCapacity to MessageToByteHandler

Result:

Once a write message that is larger than the capacity of the MessageToByteHandler's maxBufferCapacity, it clamps the byteBuffer down.

@ali-ahsan-ali
Copy link
Contributor Author

Addressing #2734

@Lukasa Lukasa added the semver/patch No public API change. label Jun 20, 2024
Lukasa
Lukasa requested changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@Lukasa Lukasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! I've left a few notes in the diff.

Tests/NIOCoreTests/ByteBufferTest.swift Outdated Show resolved Hide resolved
Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
FranzBusch
FranzBusch reviewed Jun 26, 2024
View reviewed changes
Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
@ali-ahsan-ali ali-ahsan-ali force-pushed the Issue-2734-shrink-buffer-after-write branch from ebb1d9c to 825a6b3 Compare July 4, 2024 08:19
FranzBusch
FranzBusch reviewed Jul 10, 2024
View reviewed changes
Copy link
Member

@FranzBusch FranzBusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except the doc nit this look good to me now. I would like @Lukasa to also take a look before merging.

Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
Lukasa
Lukasa reviewed Jul 22, 2024
View reviewed changes
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
@ali-ahsan-ali ali-ahsan-ali force-pushed the Issue-2734-shrink-buffer-after-write branch from c64d301 to 180b41f Compare July 27, 2024 10:18
@ali-ahsan-ali ali-ahsan-ali force-pushed the Issue-2734-shrink-buffer-after-write branch from 180b41f to a48b322 Compare July 27, 2024 10:18
@weissi weissi requested a review from Lukasa August 5, 2024 09:14
@ali-ahsan-ali ali-ahsan-ali force-pushed the Issue-2734-shrink-buffer-after-write branch from 3c4c678 to 3da1d3a Compare August 6, 2024 10:09
Lukasa
Lukasa approved these changes Aug 9, 2024
View reviewed changes
Copy link
Contributor

@Lukasa Lukasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I'm happy with this if @weissi is.

weissi
weissi requested changes Aug 9, 2024
View reviewed changes
Copy link
Member

@weissi weissi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks for the patch. I think I found a UB issue and I'm not sure if the reader/writerIndices are handled correctly either

Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
Sources/NIOCore/Codec.swift Outdated Show resolved Hide resolved
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
@ali-ahsan-ali
Copy link
Contributor Author

Hey @weissi LMK when you find time to take a look at this. No rush!

weissi
weissi reviewed Sep 10, 2024
View reviewed changes
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
Sources/NIOCore/ByteBuffer-core.swift Outdated Show resolved Hide resolved
@weissi
Copy link
Member

weissi commented Sep 10, 2024

Hey @weissi LMK when you find time to take a look at this. No rush!

Commented, looks 99% complete, just three small things

weissi
weissi approved these changes Sep 13, 2024
View reviewed changes
Copy link
Member

@weissi weissi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you!

@weissi weissi enabled auto-merge (squash) September 13, 2024 13:41
@ali-ahsan-ali
Copy link
Contributor Author

Will cry tears of joy once this gets merged 👍

Lukasa
Lukasa approved these changes Sep 16, 2024
View reviewed changes
Copy link
Contributor

@Lukasa Lukasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thank you very much for your hard work and persistence with this!

auto-merge was automatically disabled September 17, 2024 02:03

Head branch was pushed to by a user without write access

@Lukasa Lukasa enabled auto-merge (squash) September 17, 2024 12:58
@Lukasa Lukasa merged commit 8307ad6 into apple:main Sep 18, 2024
27 of 29 checks passed
cgrindel-self-hosted-renovate bot referenced this pull request in cgrindel/rules_swift_package_manager Sep 25, 2024
@cgrindel-self-hosted-renovate
chore(deps): update dependency apple/swift-nio to v2.73.0 (#1258)
a2d42d0 
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [apple/swift-nio](https://redirect.github.com/apple/swift-nio) | minor
| `2.72.0` -> `2.73.0` |

---

### Release Notes

<details>
<summary>apple/swift-nio (apple/swift-nio)</summary>

###
[`v2.73.0`](https://redirect.github.com/apple/swift-nio/releases/tag/2.73.0)

[Compare
Source](https://redirect.github.com/apple/swift-nio/compare/2.72.0...2.73.0)

<!-- Release notes generated using configuration in .github/release.yml
at main -->

#### What's Changed

##### SemVer Minor

- Make `ByteBuffer`'s description more useful by
[@&#8203;supersonicbyte](https://redirect.github.com/supersonicbyte) in
[https://github.com/apple/swift-nio/pull/2864](https://redirect.github.com/apple/swift-nio/pull/2864)
- Expose `UDP_MAX_SEGMENTS` via System by
[@&#8203;rnro](https://redirect.github.com/rnro) in
[https://github.com/apple/swift-nio/pull/2891](https://redirect.github.com/apple/swift-nio/pull/2891)
- Add new `ChannelOption` to get the amount of buffered outbound data in
the Channel by
[@&#8203;johnnzhou](https://redirect.github.com/johnnzhou) in
[https://github.com/apple/swift-nio/pull/2849](https://redirect.github.com/apple/swift-nio/pull/2849)
- Add an `AcceptBackoffHandler` to the async server bootstraps by
[@&#8203;FranzBusch](https://redirect.github.com/FranzBusch) in
[https://github.com/apple/swift-nio/pull/2782](https://redirect.github.com/apple/swift-nio/pull/2782)

##### SemVer Patch

- Adding a nicer description for `WebSocketFrame` by
[@&#8203;supersonicbyte](https://redirect.github.com/supersonicbyte) in
[https://github.com/apple/swift-nio/pull/2862](https://redirect.github.com/apple/swift-nio/pull/2862)
- Improving `description` and adding `debugDescription` to `NIOAny` by
[@&#8203;supersonicbyte](https://redirect.github.com/supersonicbyte) in
[https://github.com/apple/swift-nio/pull/2866](https://redirect.github.com/apple/swift-nio/pull/2866)
- Make FileChunk sendable by
[@&#8203;ali-ahsan-ali](https://redirect.github.com/ali-ahsan-ali) in
[https://github.com/apple/swift-nio/pull/2871](https://redirect.github.com/apple/swift-nio/pull/2871)
- Make `ByteBuffer.debugDescription` suitable for structural display by
[@&#8203;dnadoba](https://redirect.github.com/dnadoba) in
[https://github.com/apple/swift-nio/pull/2495](https://redirect.github.com/apple/swift-nio/pull/2495)
- Add support for WASILibc by
[@&#8203;MaxDesiatov](https://redirect.github.com/MaxDesiatov) in
[https://github.com/apple/swift-nio/pull/2671](https://redirect.github.com/apple/swift-nio/pull/2671)
- `NIOSingleStepByteToMessageDecoder` reentrancy safety by
[@&#8203;rnro](https://redirect.github.com/rnro) in
[https://github.com/apple/swift-nio/pull/2881](https://redirect.github.com/apple/swift-nio/pull/2881)
- Adopt `NIOThrowingAsyncSequenceProducer` by
[@&#8203;rnro](https://redirect.github.com/rnro) in
[https://github.com/apple/swift-nio/pull/2879](https://redirect.github.com/apple/swift-nio/pull/2879)
- Clamp buffer to maximum upon large write operation by
[@&#8203;ali-ahsan-ali](https://redirect.github.com/ali-ahsan-ali) in
[https://github.com/apple/swift-nio/pull/2745](https://redirect.github.com/apple/swift-nio/pull/2745)
- Revert "Adopt `NIOThrowingAsyncSequenceProducer`
([#&#8203;2879](https://redirect.github.com/apple/swift-nio/issues/2879))"
by [@&#8203;rnro](https://redirect.github.com/rnro) in
[https://github.com/apple/swift-nio/pull/2892](https://redirect.github.com/apple/swift-nio/pull/2892)
- Add concrete description for `EmbeddedEventLoop` by
[@&#8203;aryan-25](https://redirect.github.com/aryan-25) in
[https://github.com/apple/swift-nio/pull/2890](https://redirect.github.com/apple/swift-nio/pull/2890)
- Conditionally include linux/udp.h by
[@&#8203;rnro](https://redirect.github.com/rnro) in
[https://github.com/apple/swift-nio/pull/2894](https://redirect.github.com/apple/swift-nio/pull/2894)
- Work around a type checking error when using the Static Linux SDK by
[@&#8203;euanh](https://redirect.github.com/euanh) in
[https://github.com/apple/swift-nio/pull/2898](https://redirect.github.com/apple/swift-nio/pull/2898)

##### Other Changes

- \[CI] Run tests on push to main by
[@&#8203;FranzBusch](https://redirect.github.com/FranzBusch) in
[https://github.com/apple/swift-nio/pull/2868](https://redirect.github.com/apple/swift-nio/pull/2868)
- \[CI] License header support `.in` and `.cmake` files by
[@&#8203;FranzBusch](https://redirect.github.com/FranzBusch) in
[https://github.com/apple/swift-nio/pull/2870](https://redirect.github.com/apple/swift-nio/pull/2870)
- Include nanoseconds in assertion of timestamp for NIOFileSystem tests
by [@&#8203;gjcairo](https://redirect.github.com/gjcairo) in
[https://github.com/apple/swift-nio/pull/2869](https://redirect.github.com/apple/swift-nio/pull/2869)
- Correct the link of sswg-security at SECURITY.md by
[@&#8203;lamtrinhdev](https://redirect.github.com/lamtrinhdev) in
[https://github.com/apple/swift-nio/pull/2872](https://redirect.github.com/apple/swift-nio/pull/2872)
- Speculative fix for flakey AsyncTestingEventLoop test by
[@&#8203;simonjbeaumont](https://redirect.github.com/simonjbeaumont) in
[https://github.com/apple/swift-nio/pull/2873](https://redirect.github.com/apple/swift-nio/pull/2873)
- ci: Install shellcheck if not present in CI runner by
[@&#8203;simonjbeaumont](https://redirect.github.com/simonjbeaumont) in
[https://github.com/apple/swift-nio/pull/2882](https://redirect.github.com/apple/swift-nio/pull/2882)
- ci: Use ${GITHUB_BASE_REF} as treeish for checking API break by
[@&#8203;simonjbeaumont](https://redirect.github.com/simonjbeaumont) in
[https://github.com/apple/swift-nio/pull/2883](https://redirect.github.com/apple/swift-nio/pull/2883)
- ci: Refer to nested reusable workflows using remote variant by
[@&#8203;simonjbeaumont](https://redirect.github.com/simonjbeaumont) in
[https://github.com/apple/swift-nio/pull/2884](https://redirect.github.com/apple/swift-nio/pull/2884)
- \[CI] Fix pull request label workflow by
[@&#8203;FranzBusch](https://redirect.github.com/FranzBusch) in
[https://github.com/apple/swift-nio/pull/2885](https://redirect.github.com/apple/swift-nio/pull/2885)

#### New Contributors

- [@&#8203;ali-ahsan-ali](https://redirect.github.com/ali-ahsan-ali)
made their first contribution in
[https://github.com/apple/swift-nio/pull/2871](https://redirect.github.com/apple/swift-nio/pull/2871)
- [@&#8203;aryan-25](https://redirect.github.com/aryan-25) made their
first contribution in
[https://github.com/apple/swift-nio/pull/2890](https://redirect.github.com/apple/swift-nio/pull/2890)
- [@&#8203;johnnzhou](https://redirect.github.com/johnnzhou) made their
first contribution in
[https://github.com/apple/swift-nio/pull/2849](https://redirect.github.com/apple/swift-nio/pull/2849)
- [@&#8203;euanh](https://redirect.github.com/euanh) made their first
contribution in
[https://github.com/apple/swift-nio/pull/2898](https://redirect.github.com/apple/swift-nio/pull/2898)

**Full Changelog**:
apple/swift-nio@2.72.0...2.73.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC45NC4xIiwidXBkYXRlZEluVmVyIjoiMzguOTQuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->

Co-authored-by: cgrindel-self-hosted-renovate[bot] <139595543+cgrindel-self-hosted-renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FranzBusch FranzBusch FranzBusch left review comments

@weissi weissi weissi approved these changes

@Lukasa Lukasa Lukasa approved these changes

Assignees
No one assigned
Labels
semver/patch No public API change.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ali-ahsan-ali @weissi @Lukasa @FranzBusch