[Bug] Tensorboard logging not logging every log_interval timesteps

[YannBerthelot]

🐛 Bug

The documentation of DQN agent (https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) specifies that log_interval parameter is "The number of timesteps before logging". However, when set to 1 (or any other value) the logging is not made at that pace but is instead made every log_interval episode (and not timesteps). In the example below this is made every 200 timesteps.

To Reproduce

from stable_baselines3 import DQN

env = gym.make("MountainCar-v0")
model = DQN("MlpPolicy", env,tensorboard_log="logs")
model.learn(total_timesteps=2000, log_interval=1)

Expected behavior

I would have expected to see the logging every timestep and not every episode. Either the behavior should be switched (but logging every n episode is useful too ...) or the doc should be updated.

### System Info

OS: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic #1 SMP Tue Dec 7 09:58:10 PST 2021
Python: 3.7.12
Stable-Baselines3: 1.3.0
PyTorch: 1.10.0+cu111
GPU Enabled: False
Numpy: 1.19.5
Gym: 0.17.3

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have read the documentation (required)
• I have provided a minimal working example to reproduce the bug (required)

[Miffyli]

Thanks for reporting this! As per Discord chats, we should update the documentation to indeed reflect this behaviour (potentially same for other off-policy algos as well). I can try to fix this quickly, but if you want to get some github experience we would love to accept a PR from you that fixes this :)

[m1kol]

Yeah, the same experience with the DDPG algorithm on the Cartpole environment. And not only with TensorBoard, but logging in general. I used HumanOutputFormat and MLflowCustomFormat, provided in the documentation.

[theSquaredError]

I want to solve this issue as my first open-source contribution. Please help me with further instructions.

[qgallouedec]

Hi @theSquaredError, I would be happy to help you with your first open-source contribution. I advise you to follow the step described in https://github.com/DLR-RM/stable-baselines3/blob/master/CONTRIBUTING.md. Don't hesitate to contact me if necessary!

[theSquaredError]

Hi @qgallouedec, I can understand the issue here. what should i try for here, adding a new arguement for logging for each time interval. Can you tell me

[qgallouedec]

This one is just about updating the documentation, from "log every n timeteps" to "log every n episodes".

[theSquaredError]

Okay thank you very much i will do that

[theSquaredError]

I have corrected that statement, but I cannot push it. It says
remote: Permission to DLR-RM/stable-baselines3.git denied to theSquaredError.
fatal: unable to access 'https://github.com/DLR-RM/stable-baselines3.git/': The requested URL returned error: 403

[qgallouedec]

This is because you are trying to push to the master branch, which is protected. Here is a step-by-step process you can follow https://github.com/firstcontributions/first-contributions.

[theSquaredError]

Hi @qgallouedec , I have updated the doc and submitted a pull request. Can you check once please