Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OutOfMemoryError happens when try to decode the wrong ByteArray #2649

Closed
sang-eun opened this issue Apr 23, 2024 · 2 comments
Closed

OutOfMemoryError happens when try to decode the wrong ByteArray #2649

sang-eun opened this issue Apr 23, 2024 · 2 comments

Comments

@sang-eun
Copy link

sang-eun commented Apr 23, 2024

Describe the bug
Sometimes our server get wrong ByteArray. We hoped the server throw wrong format error immediately when we try to decode the message. However, protobuf decoder blocks until java heap space out of memory error happens.

From our debugging, message is keep pushBacked by reader.pushBackTag() and index never moves forward from this method.

private fun decodeTaggedListIndex(): Int {
        val protoId = if (index == -1) {
            // For the very first element tag is already read by the parent
            reader.currentId
        } else {
            reader.readTag()
        }

        return if (protoId == tagOrSize.protoId) {
            ++index
        } else {
            // If we read tag of a different message, push it back to the reader and bail out
            reader.pushBackTag()
            CompositeDecoder.DECODE_DONE
        }
    }

To Reproduce

// class 
@Serializable
@SerialName("FrontRequest")
data class FrontRequest (
    val requests: List<String>,
) {
    companion object {
        @OptIn(ExperimentalSerializationApi::class)
        fun fromProtobuf(bytes: ByteArray): FrontRequest
            = ProtoBuf.decodeFromByteArray(bytes)
    }
}


//test 
val payloadBase64 = "+kCbAII+8m6eL8J/nzj6GfwQwzzAa6gzw2uhO8FJ/AGbOv9vr2y/P8l1o1SWPfcylGOnaMFroz7BOak6xGykapQ6qG6Sa/M5lD+jb5RupzvFbvBvxGmnbZVt9WiSaqY6yWL1bpQ/8z3EbvdpwG6pacRj9GrEbfI8xmr1OpM+o2nHPvBqwT7wb8Fup2g="
val payload = Base64.getDecoder().decode(payloadBase64)
//keep working until java heap space error
val frontRequestReceived = FrontRequest.fromProtobuf(payload)

// result

Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3609)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.expand(ProtobufTaggedBase.kt:40)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.pushTag(ProtobufTaggedBase.kt:34)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedBase.access$pushTag(ProtobufTaggedBase.kt:17)
	at kotlinx.serialization.protobuf.internal.ProtobufTaggedDecoder.decodeSerializableElement(ProtobufTaggedDecoder.kt:110)
	at com.dunamu.quot.realtime.front.request.FrontRequest$$serializer.deserialize(FrontRequest.kt:13)

Expected behavior
throw wrong format error immediately.

Environment

  • Kotlin version: 1.8.21
  • Library version: 1.5.1
  • Kotlin platforms: JVM
  • Gradle version: 7.5.1
@sang-eun sang-eun changed the title Infinite loop happens to deserialize the ByteArray Infinite loop happens when decode the wrong ByteArray Apr 23, 2024
@sang-eun sang-eun changed the title Infinite loop happens when decode the wrong ByteArray OutOfMemoryError happens when decode the wrong ByteArray Apr 23, 2024
@sang-eun sang-eun changed the title OutOfMemoryError happens when decode the wrong ByteArray OutOfMemoryError happens when try to decode the wrong ByteArray Apr 23, 2024
@sandwwraith
Copy link
Member

@sandwwraith sandwwraith self-assigned this Apr 24, 2024
@sandwwraith
Copy link
Member

Thanks, reproduced on 1.9.21 + 1.6.3, too

@sandwwraith sandwwraith assigned shanshin and unassigned sandwwraith Jul 29, 2024
shanshin added a commit that referenced this issue Aug 8, 2024
…criptor

Also optimized skipping of size delimited fields - removed the creation of an byte array in case of skipping

Fixes #2649
shanshin added a commit that referenced this issue Aug 23, 2024
…criptor

Also optimized skipping of size delimited fields - removed the creation of an byte array in case of skipping

Fixes #2649
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants