[Asyncio] Support for async → sync → async call chains

[omerXfaruq]

Feature or enhancement
Support for running async functions in sync functions
Pitch
Currently running async functions inside sync functions is only available with
asyncio.run(func(x)) or loop.run_until_complete(); however when there is a running loop, these calls cannot be used.
The only way is running the function in background, and using callback functions(which is very limiting)

event_loop = asyncio.get_event_loop()
task = event_loop.create_task(func(*args))
task.add_done_callback(callback_func)

See example PR

I think there should be function to handle all cases of running async funcs in sync funcs

• when there is no event_loop
• when there is already running event_loop
  Suggested syntax: synchronize_async(func(*args))

[omerXfaruq]

It looks like I found a similar solution in fsspec, thanks to @isidentical!

Would be great to have something similar in asyncio as well!

[arhadthedev]

Could you change the title to something like Support for async → sync → async call chains please? Reading the title alone gives the wrong impression that you suggest implicit loop creation.

[omerXfaruq]

Was able to create a solution to this in this PR, using fsspec.

[gvanrossum]

Previous discussion
#93462

Is that a typo? It just links back to this same issue.

Separately, I note that the solution in fsspec (linked from the previous comment) just runs the whole thing in a separate thread, using a separate event loop. That's pretty heavy machinery, not something I'd like to enable with a quick call or decorator -- it'd just encourage a bad architecture.

[omerXfaruq]

Yeah it seems like a typo, removed it.

I definitely agree that fsspec's solution is a hacky solution, there were a lot of occassions I needed this feature in the past, and think this is a critical feature.

[kumaraditya303]

I have needed something like this myself when wrapping an async library with a sync interface, the solution I came up with is very complicated for an average user so would be nice to have support for this given that https://github.com/erdewit/nest_asyncio library which allows something like this is downloaded 17 million per month so in high demand.

[kumaraditya303]

This would require an event loop which supports reentrancy which does has it downsides so even if this were to be supported it should be behind a user configurable flag and should definitely be not the default behavior.

[gvanrossum]

Millions of downloads may be the result of being the dependency of one other very popular package; nest_asyncio only has modest numbers of starts/forks/issues.

Nevertheless, it seems to just patch a bunch of asyncio functions to allow re-entrancy, which looks much lighter weight than running in a thread (pool).

What would it take to just incorporate those patches in the stdlib asyncio? I guess we'd have to reimplement them in C for the C accelerator to keep working (nest_asyncio disables that), but that's a matter of work. If those patches don't introduce other bugs. My post from 7 years ago makes it sound rather dubious though.

But I'd be happy to see someone try this experiment and report back how it went. (Try hard to come up with scenarios that break it, that's the interesting part of the experiment.)

[omerXfaruq]

WoW what an ancient issue :D

[kumaraditya303]

Please continue the discussion on #66435