Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add basic benchmarking #275

Draft
wants to merge 11 commits into
base: keyring
Choose a base branch
from
Draft

chore: add basic benchmarking #275

wants to merge 11 commits into from

Conversation

mattsb42-aws
Copy link
Member

Description of changes:

This adds some tests that benchmark the performance of the client under different configurations.

Results from running this on an m5a.xlarge EC2 instance:

---------------------------------------------------------------------------------------------------------------------------------- benchmark: 54 tests -----------------------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                                                          Min                 Max                Mean            StdDev              Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_end2end_performance[decrypt only-single frame-cache and only miss once-AWS KMS master key provider]                2.8315 (1.0)        3.7074 (1.22)       2.9646 (1.0)      0.2633 (9.59)       2.8719 (1.0)      0.0644 (2.98)          1;1  337.3181 (1.0)          10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw RSA master key]                          2.8418 (1.00)       3.3037 (1.09)       3.0307 (1.02)     0.1612 (5.87)       3.0182 (1.05)     0.2963 (13.69)         4;0  329.9582 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw AES master key]                          2.8661 (1.01)       4.1578 (1.37)       3.0152 (1.02)     0.4016 (14.62)      2.8940 (1.01)     0.0216 (1.0)           1;1  331.6482 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-AWS KMS master key provider]                 2.9036 (1.03)       3.7566 (1.24)       3.0272 (1.02)     0.2578 (9.39)       2.9545 (1.03)     0.0577 (2.66)          1;1  330.3333 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw RSA keyring]                             2.9234 (1.03)       3.0458 (1.00)       2.9720 (1.00)     0.0367 (1.34)       2.9756 (1.04)     0.0417 (1.93)          3;0  336.4700 (1.00)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw RSA master key]                         2.9325 (1.04)       3.1961 (1.05)       3.0515 (1.03)     0.0879 (3.20)       3.0451 (1.06)     0.1453 (6.71)          3;0  327.7041 (0.97)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw AES master key]                         2.9329 (1.04)       4.2640 (1.40)       3.1076 (1.05)     0.4070 (14.82)      2.9892 (1.04)     0.0373 (1.72)          1;1  321.7916 (0.95)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-AWS KMS keyring]                            2.9513 (1.04)       4.3001 (1.41)       3.2733 (1.10)     0.4726 (17.21)      3.0528 (1.06)     0.2174 (10.05)         2;2  305.5039 (0.91)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-AWS KMS master key provider]                2.9515 (1.04)       6.3232 (2.08)       3.4855 (1.18)     1.0961 (39.91)      3.0233 (1.05)     0.0720 (3.33)          1;2  286.9019 (0.85)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw AES keyring]                             2.9519 (1.04)       3.0399 (1.0)        2.9925 (1.01)     0.0308 (1.12)       2.9969 (1.04)     0.0605 (2.79)          5;0  334.1644 (0.99)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-AWS KMS keyring]                             2.9543 (1.04)       5.3401 (1.76)       3.2993 (1.11)     0.7188 (26.17)      3.0880 (1.08)     0.0655 (3.03)          1;2  303.0971 (0.90)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw RSA keyring]                            2.9677 (1.05)       3.1697 (1.04)       3.0521 (1.03)     0.0616 (2.24)       3.0329 (1.06)     0.0711 (3.29)          3;0  327.6395 (0.97)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw AES master key]                         2.9687 (1.05)       4.3998 (1.45)       3.1586 (1.07)     0.4379 (15.94)      3.0413 (1.06)     0.0896 (4.14)          1;1  316.5959 (0.94)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw RSA master key]                         2.9696 (1.05)       4.3351 (1.43)       3.1453 (1.06)     0.4198 (15.28)      3.0185 (1.05)     0.0698 (3.23)          1;1  317.9318 (0.94)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw AES keyring]                            3.0199 (1.07)       3.1493 (1.04)       3.0767 (1.04)     0.0409 (1.49)       3.0693 (1.07)     0.0512 (2.37)          4;0  325.0214 (0.96)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-AWS KMS keyring]                            3.0994 (1.09)       6.3785 (2.10)       3.4788 (1.17)     1.0192 (37.11)      3.1612 (1.10)     0.0224 (1.03)          1;2  287.4553 (0.85)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw AES keyring]                            3.1026 (1.10)       3.3538 (1.10)       3.2067 (1.08)     0.0719 (2.62)       3.1940 (1.11)     0.0457 (2.11)          3;3  311.8437 (0.92)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw RSA keyring]                            3.1123 (1.10)       3.2192 (1.06)       3.1422 (1.06)     0.0350 (1.27)       3.1319 (1.09)     0.0505 (2.33)          1;0  318.2455 (0.94)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw AES master key]                          3.1880 (1.13)       3.2767 (1.08)       3.2296 (1.09)     0.0275 (1.0)        3.2315 (1.13)     0.0432 (1.99)          3;0  309.6329 (0.92)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw RSA keyring]                             3.3358 (1.18)       4.6336 (1.52)       3.5028 (1.18)     0.3986 (14.51)      3.3682 (1.17)     0.0427 (1.97)          1;1  285.4823 (0.85)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw AES keyring]                             3.3393 (1.18)       3.4384 (1.13)       3.3837 (1.14)     0.0350 (1.27)       3.3823 (1.18)     0.0668 (3.09)          4;0  295.5379 (0.88)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw RSA master key]                          3.3598 (1.19)       3.4824 (1.15)       3.4288 (1.16)     0.0447 (1.63)       3.4377 (1.20)     0.0866 (4.00)          5;0  291.6510 (0.86)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw AES keyring]                                            4.3160 (1.52)       5.7662 (1.90)       4.6448 (1.57)     0.4091 (14.89)      4.5205 (1.57)     0.1211 (5.60)          1;1  215.2967 (0.64)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw AES master key]                                         4.3175 (1.52)       4.4212 (1.45)       4.3650 (1.47)     0.0318 (1.16)       4.3632 (1.52)     0.0408 (1.88)          3;0  229.0965 (0.68)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw AES master key]                                         4.6765 (1.65)       4.7871 (1.57)       4.7278 (1.59)     0.0323 (1.18)       4.7291 (1.65)     0.0331 (1.53)          3;1  211.5170 (0.63)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw AES keyring]                                            4.7414 (1.67)       6.1184 (2.01)       4.9875 (1.68)     0.4014 (14.61)      4.8880 (1.70)     0.0801 (3.70)          1;1  200.5015 (0.59)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw RSA keyring]                                            4.7542 (1.68)       6.0527 (1.99)       4.9270 (1.66)     0.3972 (14.46)      4.7965 (1.67)     0.0658 (3.04)          1;1  202.9649 (0.60)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-AWS KMS master key provider]       6.1176 (2.16)      10.2705 (3.38)       6.8095 (2.30)     1.2904 (46.98)      6.3088 (2.20)     0.5383 (24.87)         1;2  146.8546 (0.44)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw RSA master key]                6.3366 (2.24)       8.1186 (2.67)       6.7200 (2.27)     0.5168 (18.81)      6.6344 (2.31)     0.3505 (16.19)         1;1  148.8095 (0.44)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw RSA master key]                                         6.3942 (2.26)       7.8302 (2.58)       6.6359 (2.24)     0.4263 (15.52)      6.5032 (2.26)     0.1329 (6.14)          1;1  150.6965 (0.45)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-AWS KMS keyring]                   6.4201 (2.27)      10.5999 (3.49)       7.0723 (2.39)     1.3027 (47.43)      6.5317 (2.27)     0.4896 (22.62)         1;2  141.3960 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw AES master key]                 6.4928 (2.29)       8.7652 (2.88)       6.9987 (2.36)     0.6680 (24.32)      6.8791 (2.40)     0.5484 (25.34)         1;1  142.8832 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw AES master key]                6.6806 (2.36)       8.4195 (2.77)       6.9204 (2.33)     0.5300 (19.30)      6.7670 (2.36)     0.1381 (6.38)          1;1  144.5011 (0.43)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw AES keyring]                    6.7439 (2.38)       8.5047 (2.80)       7.3200 (2.47)     0.4683 (17.05)      7.2874 (2.54)     0.3691 (17.05)         2;1  136.6125 (0.40)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw RSA keyring]                   6.8581 (2.42)       7.1733 (2.36)       6.9659 (2.35)     0.0851 (3.10)       6.9623 (2.42)     0.0587 (2.71)          2;1  143.5571 (0.43)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw AES keyring]                   6.9545 (2.46)       7.3742 (2.43)       7.1091 (2.40)     0.1218 (4.43)       7.0802 (2.47)     0.1478 (6.83)          3;0  140.6640 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw RSA master key]                 7.1794 (2.54)       8.5197 (2.80)       7.3895 (2.49)     0.4025 (14.65)      7.2523 (2.53)     0.1358 (6.27)          1;1  135.3266 (0.40)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw RSA keyring]                    7.1805 (2.54)       8.2461 (2.71)       7.7462 (2.61)     0.4923 (17.93)      7.7774 (2.71)     0.9414 (43.49)         2;0  129.0961 (0.38)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw AES master key]                                8.9643 (3.17)      10.7343 (3.53)       9.3247 (3.15)     0.5125 (18.66)      9.2113 (3.21)     0.2976 (13.75)         1;1  107.2416 (0.32)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw AES keyring]                                   9.1484 (3.23)      10.6542 (3.50)       9.4189 (3.18)     0.4414 (16.07)      9.2794 (3.23)     0.1723 (7.96)          1;1  106.1698 (0.31)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw RSA master key]                                        11.2803 (3.98)      12.7799 (4.20)      11.5652 (3.90)     0.4365 (15.89)     11.4336 (3.98)     0.1646 (7.61)          1;1   86.4666 (0.26)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-AWS KMS master key provider]                11.3130 (4.00)      13.6719 (4.50)      11.7862 (3.98)     0.7034 (25.61)     11.4614 (3.99)     0.4695 (21.69)         1;1   84.8450 (0.25)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw RSA keyring]                                           11.5758 (4.09)      11.7989 (3.88)      11.6848 (3.94)     0.0755 (2.75)      11.6830 (4.07)     0.0991 (4.58)          4;0   85.5812 (0.25)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-AWS KMS keyring]                            11.6115 (4.10)      14.5919 (4.80)      12.4075 (4.19)     0.8815 (32.10)     12.2145 (4.25)     0.6012 (27.78)         1;1   80.5965 (0.24)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw RSA keyring]                                  16.2953 (5.75)      18.4330 (6.06)      16.8372 (5.68)     0.5949 (21.66)     16.6533 (5.80)     0.2282 (10.54)         1;1   59.3921 (0.18)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw RSA master key]                               17.8754 (6.31)      19.5608 (6.43)      18.2335 (6.15)     0.5187 (18.89)     18.0378 (6.28)     0.2469 (11.41)         1;2   54.8440 (0.16)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-AWS KMS keyring]                   22.6073 (7.98)      27.0973 (8.91)      24.2102 (8.17)     1.3239 (48.20)     24.1547 (8.41)     1.6790 (77.57)         2;0   41.3049 (0.12)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-AWS KMS master key provider]       22.9630 (8.11)      26.5351 (8.73)      24.2309 (8.17)     1.1369 (41.39)     23.9474 (8.34)     1.6900 (78.08)         3;0   41.2696 (0.12)         10         100
test_end2end_performance[decrypt only-single frame-no cache-AWS KMS master key provider]                               85.9302 (30.35)     97.9246 (32.21)     92.6231 (31.24)    4.2571 (155.00)    92.3338 (32.15)    6.3445 (293.13)        5;0   10.7964 (0.03)         10         100
test_end2end_performance[decrypt only-single frame-no cache-AWS KMS keyring]                                           86.0042 (30.37)     98.2628 (32.32)     92.0426 (31.05)    4.0520 (147.53)    91.9101 (32.00)    3.2239 (148.95)        4;0   10.8645 (0.03)         10         100
test_end2end_performance[encrypt only-single frame-no cache-AWS KMS keyring]                                           87.1333 (30.77)    100.9260 (33.20)     94.6055 (31.91)    4.0725 (148.28)    93.4407 (32.54)    5.4352 (251.12)        3;0   10.5702 (0.03)         10         100
test_end2end_performance[encrypt only-single frame-no cache-AWS KMS master key provider]                               88.1433 (31.13)    101.6990 (33.46)     93.3860 (31.50)    4.0176 (146.28)    93.2626 (32.47)    4.9964 (230.85)        3;0   10.7082 (0.03)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-AWS KMS master key provider]                     180.7815 (63.85)    194.7949 (64.08)    186.9117 (63.05)    4.7156 (171.69)   186.0659 (64.79)    6.3366 (292.76)        3;0    5.3501 (0.02)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-AWS KMS keyring]                                 184.4317 (65.13)    195.0076 (64.15)    188.8441 (63.70)    3.6701 (133.63)   188.4976 (65.64)    5.0244 (232.14)        4;0    5.2954 (0.02)         10         100
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

  • Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

@mattsb42-aws
Copy link
Member Author

Looks like breaking CI is due to pypa/virtualenv#1857

@texastony
Copy link
Contributor

Putting in Draft as this is for keying which has not shipped and is not currently on our roadmap

@texastony texastony marked this pull request as draft September 16, 2022 16:57
@mattsb42-aws
Copy link
Member Author

You do you, but IMO having benchmarking data in the tests would be useful without keyrings. I'd be happy to take a look at stripping the keyring benchmarks out of this.

@texastony
Copy link
Contributor

Oh... @mattsb42-aws it appears the mattsb42-aws:benchmarking has been deleted... maybe I can download this source code via gh?

@texastony
Copy link
Contributor

I have recreated this PR here.
I need to replace the Keyring references with Master Key Provider references.
@mattsb42-aws If you want, you can pitch in for that work.
Otherwise, I will get to it eventually...

@mattsb42-aws
Copy link
Member Author

Whoops; I guess I got a bit overzealous deleting forks when I changed teams. I'll see about dusting off my memory of how I put this together to re-arrange it to use MKPs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants