psutil is never installed on Windows

[gvanrossum]

On Windows, it turns out pyperf (and hence pyperformance) will always print a warning when psutil is not installed. The warning is:

WARNING: unable to increase process priority

It is printed by _process_priority(). (Note the if not MS_WINDOWS: return right before.)

The function set_priority() silently returns None, but the caller prints the warning in that case.

So now for every benchmark run by pyperformance it calls this _process_priority() function for each run, printing the warning each time (maybe 10 times per run).

The solution of course is to install psutil. But no matter what I do, because the way venvs used to run the benchmark are created, those venvs do not get psutil installed, even if it is installed in site-packages. (At least, when running from a dev environment.)

• I tried adding "psutil" to the install_requires line in pyperformance's setup.py, but that doesn't seem to help.
• psutil is already listed in requirements.in, so that doesn't seem to make a difference.
• It is also listed in pyperformance/data-files/requirements.txt, apparently with no effect.

So how do I add this dependency to every package? Surely there's a more elegant way than adding it to every benchmark-specific requirements.txt file in pyperformance/data-files/benchmarks?

[corona10]

@vstinner cc @gvanrossum @ericsnowcurrently
IMO, it should be solved from pyperf side instead of pyperformance project.
WDYT?

[gvanrossum]

No, I think this has to do with how pyperformance manages venvs. @ericsnowcurrently

[vstinner]

pyperformance does install psutil in VirtualEnvironment._install_req():

        # install optional requirements
        for req in requirements.optional:
            cmd = pip_program + ['install', '-U', req]
            exitcode = self.run_cmd_nocheck(cmd)
            if exitcode:
                print("WARNING: failed to install %s" % req)
                print()

But if the installation fails, it only logs a warning.

psutil is marked as an "optional" dependency because I failed issues sometimes to install it on psutil or on development versions of Python:

        # parse requirements
        filename = os.path.join(PERFORMANCE_ROOT, 'requirements.txt')
        requirements = Requirements(filename,
                                    # FIXME: don't hardcode requirements
                                    ['psutil', 'dulwich'])

It's listed in pyperformance/requirements.in:

# Optional dependencies
# ---------------------
#
# The list of optional dependencies is hardcoded in pyperformance/venv.py

psutil
dulwich                            # bm_dulwich_log

[vstinner]

I tried adding "psutil" to the install_requires line in pyperformance's setup.py, but that doesn't seem to help.

Currently, psutil is hardcoded as an "optional dependency" in venv.py: it tries to install it, but if pip fails, it only logs a warning.

Do you see this warning when you just create a venv using the python3 -m pyperformance venv create command?

[gvanrossum]

Victor, it looks like you're looking at an earlier version of pyperformance. I suspect that @ericsnowcurrently accidentally broke this (perhaps only for Windows).

[gvanrossum]

Do you see this warning when you just create a venv using the python3 -m pyperformance venv create command?

No, I don't see that warning, and psutil is not present in the created venv.

[ericsnowcurrently]

#170 should fix it. I suspect that I originally thought psutil was in pyperf's requirements.txt (it isn't), but didn't check.

[gvanrossum]

Still no dice. :-(

Now it does indeed try to install psutil, but since there's no wheel, it tries to build from source, and fails in the same way I've described elsewhere. And since apparently in this case there's no way to get a psutil wheel to end up in pip's wheel cache I still cannot get this working.

UPDATE: There is a way to get a psutil wheel in the pip cache (same as before), and then it does work. Not sure what I did wrong the first time I tried.

[ericsnowcurrently]

Ah, so now the remaining issue is "psutil wheels are not available for 3.11".

[gvanrossum]

I would rather summarize it as "on Windows, building extensions from source doesn't work in a venv created using a python.exe built in the cpython source repo". Having psutil wheels for 3.11 available is only one of the many ways to mask this problem. :-)

[ericsnowcurrently]

got it

would it help if psutil released a stable ABI wheel for the extension?

[ericsnowcurrently]

Just in case, I created giampaolo/psutil#2089.

[gvanrossum]

See #171, it's a more general issue.

[vstinner]

I would rather summarize it as "on Windows, building extensions from source doesn't work in a venv created using a python.exe built in the cpython source repo".

On Linux, the pyperformance compile comand installs the newly build Python since an installed Python and Python run from its source tree behave a little bit differently, especially for sys.path and startup performance.

https://pyperformance.readthedocs.io/usage.html#compile-python-to-run-benchmarks

Maybe you should try to do something similar on Windows?

[gvanrossum]

Yes, @zooba showed me how to do this:

.\python.bat -m PC.layout --preset-default --copy installed -v

[vstinner]

It would be nice to put that command in pyperformance compile, since it's non-trivial and it's always pleasant when things are automated. Running a benchmark is hard enough ;-)