We seem to be leaking coroutines on error

[dimaqq]

Description

These kind of back traces are often printed out in at least a couple of circumstances:

• there's an error talking to juju (?)
• sometimes, even when the charm integration test completes successfully

asyncio:base_events.py:1758 Task exception was never retrieved
future: <Task finished name='Task-2309' coro=<Connection.reconnect() done, defined at /home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py:779> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py", line 788, in reconnect
    res = await connector(
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py", line 868, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py", line 817, in _connect
    result = await task
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py", line 806, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/juju/client/connection.py", line 427, in _open
    return (await websockets.connect(
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/websockets/legacy/client.py", line 647, in __await_impl_timeout__
    return await self.__await_impl__()
  File "/home/runner/work/spark-integration-hub-k8s-operator/spark-integration-hub-k8s-operator/.tox/integration-charm/lib/python3.10/site-packages/websockets/legacy/client.py", line 651, in __await_impl__
    _transport, _protocol = await self._create_connection()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1103, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1119, in _create_connection_transport
    sock.setblocking(False)

There are two distinct problems with this:

1. we seem to be leaking coroutines (exception was not awaited)
2. our use of webssocket library, or the library itself messes up library internal state (bad file descriptor)

Urgency

Annoying bug in our test suite

Python-libjuju version

latest

Juju version

any latest

Reproduce / Test

run any integration test

to make sure this shows up, run an integration test with some kind of error (e.g. wrong container image name to ensure the pod and application are never ready)