What is the main thread doing when calling runtime.block_on?

[eladm-ultrawis]

I am new to rust and tokio, and trying to rewrite a grpc server as a learning experience. I wish to gain a better understanding into how tokio manages its threads and what each thread is doing. I am running on Windows in this case.

    let runtime = Builder::new_multi_thread()
        .worker_threads(1) // Set the number of threads
        .enable_all() // Enable all Tokio features (timer, I/O, etc.)
        .build()?;

    runtime.block_on(async {
        Server::builder()
            .add_service(TelemetryServiceServer::new(telemetry_service))
            .serve(addr)
            .await
    })?;

When pausing the application, I see 2 threads that are executing rust code.

1. A thread that blocks on GetQueuedCompletionStatusEx. This is what I expect from other frameworks that process IO completions on Windows.

2. A thread that waits on WaitOnAddress. This seems like the main thread is participating in processing tasks for tokio. If I increase the number of threads, I only get more threads that sit on this function and not others.

So my questions are:

1. Is my understanding regarding the role of the main thread correct? Is it processing tasks as well?
2. How can I allocate more threads to sit on the IO completion port?
3. Are other threads simply part of the general pool? Does WaitOnAddress act as some sort of a condition variable?

[Darksonn]

1. When using multi-thread runtime, the thread calling block_on will only execute the single future you passed to it. No other tasks execute on that thread.
   • With a multi-thread runtime, you should generally move your server listener into a tokio::spawn task rather than block_on so that it can run within the general pool, rather than being pinned to the block_on thread.
   • When using the current-thread runtime, this is different. That runtime spawns no worker threads, and runs all tasks inside the block_on call. The current-thread runtime does not execute tasks at all unless there is a call to block_on.
2. A Tokio runtime never has more than one thread waiting on the IO completion port. Other threads wait on a condition variable instead. Note that it is much cheaper to wake up a thread waiting on WaitOnAddress than threads waiting on the IO completion port.
   • The thread in the pool waiting on IO can change. There will always be one, but if one thread waits on IO and then becomes busy running a task, then another thread will take over waiting for IO.
   • You can spawn multiple runtimes, and each runtime will have a thread waiting on IO, but tasks will not migrate between runtimes.
3. Yes, whenever tasks spawned using tokio::spawn have work to do, they will run on one of the worker threads in the pool. They will never run on the block_on thread.

[eladm-ultrawis]

Ok cool, thanks!

So assume I will now replace the block_on call with tokio::spawn, my main thread will not be doing anything.
Can I somehow have my main thread participate in task processing as well (as part of the general pool)?

[Darksonn]

Yes. Just be careful. If you return from main, that kills the process. No, you can't have the main thread be part of the pool.