"System.Console.Clear()" does not clear the scroll buffer on Linux.

[rougemeilland]

Description

In this document, scroll buffer refers to "an area that is normally outside the display range of the console screen and is not displayed, but can be displayed by scrolling".

The spec notes for System.Console.Clear() says:

Using the Clear method is equivalent invoking the MS-DOS cls command in the command prompt window. When the Clear method is called, the cursor automatically scrolls to the top-left corner of the window and the contents of the screen buffer are set to blanks using the current foreground background colors.

In Linux, the clear command is equivalent to the MS-DOS cls command. But in Linux System.Console.Clear() only clears the console screen and not the scroll buffer. This is a different behavior than the clear command.

Reproduction Steps

1. Display more lines than there are lines on the console screen.
2. Call System.Console.Clear().

Expected behavior

Like the clear command, both the scroll buffer and the screen are cleared.

Actual behavior

Unlike the 'clear' command, it clears the screen but not the scroll buffer.

Regression?

No response

Known Workarounds

Execute System.Console.Out.Write("\u001b[3J") after executing System.Console.Clear(). (unless standard output is redirected)
This workaround is not perfect. Because the escape sequence "\u001b[3J" is not supported by all terminals.

Configuration

OS: Ubuntu 20.04.6 LTS (Linux on Windows)
Architecture: x64
.NET Runtime: .NET 6.0.15 / .NET 7.0.4
Terminal: ubuntu (Windows terminal version)

Other information

1. Findings of source code

After examining the source code of the UNIX version of the System.ConsolePal class, I found the following.

• System.Console.Clear() outputs the value of the terminfo string capability clear_screen to standard output. (I was wondering why the output destination is fixed to standard output, but I won't touch on that in this document.)
• The size of the console buffer == the size of the console window and WindowLeft == WindowTop == 0.

2. Findings on a real machine (ubuntu on "Linux on Windows")

• When I checked with man terminfo, the description of the capability clear_screen said "clear screen and home cursor".
• On ubuntu (Windows terminal version) he checked the value of the TERM environment variable and it seems to be emulating a terminal called "xterm-256color".
• When I checked the contents of the terminfo file of "xterm-256color" on ubuntu, the value of the string capability clear_screen is "\u001b[H\u001b[2J". The action of this escape sequence is "move the cursor to the upper left corner of the console window and clear the console window".

3. My thoughts on this issue

3.1 About the implementation of System.Console.Clear() in .NET

The current .NET implementation prints the value of the string capability clear_screen to the terminal.
The terminfo manual defines the behavior of the string capability clear_screen as "clear the screen and move the cursor to the home position", and doesn't say anything about scroll buffers.
On my real machine the string capability clear_screen value is "\u001b[H\u001b[2J". This value means "move the cursor to the upper left corner and clear the screen".

This is probably due to the lack of a general way to determine the size of the scroll buffer in the first place.
I guess that's why .NET implements it like this:

• It is regarded as "console buffer (screen + scroll buffer) == "console window (screen)".
• The role of System.Console.Clear() is to "clear the console buffer", but since "console buffer = = console window", it only clears the console window (screen).

3.2. Problems with current System.Console.Clear()

Admittedly, the above logic is correct when it comes to working within a .NET application.
However, many virtual consoles such as the Windows terminal allow the screen to be scrolled by operating the scrollbar or mouse, allowing the user to view the contents of the scroll buffer before System.Console.Clear() is called.
Text that the application should have cleared by calling System.Console.Clear() is visible to the user. I find this very inconvenient in some cases.

Am I stating something I can't deal with? I do not think so. Because the Linux clear command can actually clear the scroll buffer and screen. This is exactly the behavior I would expect.

3.3. Countermeasures

So why is the clear command capable of such behavior? The clue was in the online manual for the clear command.
Some terminals define the extended capability E3. According to the confirmation on the actual machine (Ubuntu 20.04.6 LTS), the extended capability E3 is defined in the terminfo of the terminal below.

• alacritty
• alacritty-direct
• alacritty+common
• linux
• linux-16color
• linux-koi8
• linux-koi8r
• linux-lat
• linux-m1
• linux3.0
• mintty
• mintty-direct
• putty
• putty-256color
• putty-noapp
• putty-sco
• putty-vt100
• screen.linux-m2
• screen.putty
• screen.putty-256color
• tmux
• tmux-256color
• vscode
• vscode-direct
• xterm
• xterm-1002
• xterm-1003
• xterm-1005
• xterm-1006
• xterm-16color
• xterm-256color
• xterm-88color
• xterm-basic
• xterm-debian
• xterm-direct
• xterm-direct2
• xterm-hp
• xterm-new
• xterm-nrc
• xterm-rep
• xterm-sco
• xterm-sun
• xterm-utf8
• xterm-x10mouse
• xterm-x11hilite
• xterm-x11mouse
• xterm-xmc
• xterm.js
• xterms-sun

I suggest the following method.

1. If the current terminal's terminfo defines the extended capability E3, System.Console.Clear() uses it to clear the console.
2. If E3 is not defined, System.Console.Clear() clears the console in the traditional way using only the capability clear_screen.

3.4. How to implement countermeasures

On most terminals, with a few exceptions, the extended capability E3 has the value "\u001b[3J", and the "xterm" manual describes this escape sequence as "Erase Saved Lines".

When I checked the operation of extended capability E3 on the actual machine, only the scroll buffer was cleared, and the screen was not cleared. In other words, even when using the extended capability E3, it must be used in conjunction with the capability clear_screen.

I think it's good to output the escape sequence in the following procedure.

1. Check if extended capability E3 is defined.
2. If E3 is defined, print the value of the capability clear_screen followed by the value of E3.
3. If E3 is not defined, print the value of the capability clear_screen.

Note that you must print E3 after clear_screen¹.

April 12th, 2023 postscript: I found the source code for the clear command on debian. Below is an excerpt.

int
clear_cmd(bool legacy)
{
    int retval = tputs(clear_screen, lines > 0 ? lines : 1, putch);
    if (!legacy) {
	/* Clear the scrollback buffer if possible. */
	char *E3 = tigetstr("E3");
	if (E3)
	    (void) tputs(E3, lines > 0 ? lines : 1, putch);
    }
    return retval;
}

According to this, it seems that the value of the capability clear_screen is obtained from terminfo and output, and if the capability E3 exists in terminfo, its value is obtained and output.
The clear command can suppress clearing of the scroll buffer by specifying the -x option. I think that's what the legacy parameter in the above source code is for.

Footnotes

1. I think a better way to find out about this is to examine the source code of the Linux clear command (which is actually the tput command). However, I couldn't find the source code for it, so I checked it with a real machine.
   In many terminals including ubuntu (Windows terminal) the capability clear_screen is defined as "\u001bH\u001b[2J".
   At least on a real machine (Ubuntu 20.04.6 LTS), outputting "\u001b[3J" followed by "\u001b[2J" does not clear the scroll buffer completely. ↩

[ghost]

Tagging subscribers to this area: @dotnet/area-system-console
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

In this document, scroll buffer refers to "an area that is normally outside the display range of the console screen and is not displayed, but can be displayed by scrolling".

The spec notes for System.Console.Clear() says:

Using the Clear method is equivalent invoking the MS-DOS cls command in the command prompt window. When the Clear method is called, the cursor automatically scrolls to the top-left corner of the window and the contents of the screen buffer are set to blanks using the current foreground background colors.

In Linux, the clear command is equivalent to the MS-DOS cls command. But in Linux System.Console.Clear() only clears the console screen and not the scroll buffer. This is a different behavior than the clear command.

Reproduction Steps

1. Display more lines than there are lines on the console screen.
2. Call System.Console.Clear().

Expected behavior

Like the clear command, both the scroll buffer and the screen are cleared.

Actual behavior

Unlike the 'clear' command, it clears the screen but not the scroll buffer.

Regression?

No response

Known Workarounds

Execute System.Console.Out.Write("\u001b[3J") after executing System.Console.Clear(). (unless standard output is redirected)
This workaround is not perfect. Because the escape sequence "\u001b[3J" is not supported by all terminals.

Configuration

OS: Ubuntu 20.04.6 LTS (Linux on Windows)
Architecture: x64
.NET Runtime: .NET 6.0.15 / .NET 7.0.4
Terminal: ubuntu (Windows terminal version)

Other information

1. Findings of source code

After examining the source code of the UNIX version of the System.ConsolePal class, I found the following.

• System.Console.Clear() outputs the value of the terminfo string capability clear_screen to standard output. (I was wondering why the output destination is fixed to standard output, but I won't touch on that in this document.)
• The size of the console buffer == the size of the console window and WindowLeft == WindowTop == 0.

2. Findings on a real machine (ubuntu on "Linux on Windows")

• When I checked with man terminfo, the description of the capability clear_screen said "clear screen and home cursor".
• On ubuntu (Windows terminal version) he checked the value of the TERM environment variable and it seems to be emulating a terminal called "xterm-256color".
• When I checked the contents of the terminfo file of "xterm-256color" on ubuntu, the value of the string capability clear_screen is "\u001b[H\u001b[2J". The action of this escape sequence is "move the cursor to the upper left corner of the console window and clear the console window".

3. My thoughts on this issue

3.1 About the implementation of System.Console.Clear() in .NET

The current .NET implementation prints the value of the string capability clear_screen to the terminal.
The terminfo manual defines the behavior of the string capability clear_screen as "clear the screen and move the cursor to the home position", and doesn't say anything about scroll buffers.
On my real machine the string capability clear_screen value is "\u001b[H\u001b[2J". This value means "move the cursor to the upper left corner and clear the screen".

This is probably due to the lack of a general way to determine the size of the scroll buffer in the first place.
I guess that's why .NET implements it like this:

• It is regarded as "console buffer (screen + scroll buffer) == "console window (screen)".
• The role of System.Console.Clear() is to "clear the console buffer", but since "console buffer = = console window", it only clears the console window (screen).

3.2. Problems with current System.Console.Clear()

Admittedly, the above logic is correct when it comes to working within a .NET application.
However, many virtual consoles such as the Windows terminal allow the screen to be scrolled by operating the scrollbar or mouse, allowing the user to view the contents of the scroll buffer before System.Console.Clear() is called.
Text that the application should have cleared by calling System.Console.Clear() is visible to the user. I find this very inconvenient in some cases.

Am I stating something I can't deal with? I do not think so. Because the Linux clear command can actually clear the scroll buffer and screen. This is exactly the behavior I would expect.

3.3. Countermeasures

So why is the clear command capable of such behavior? The clue was in the online manual for the clear command.
Some terminals define the extended capability E3. According to the confirmation on the actual machine (Ubuntu 20.04.6 LTS), the extended capability E3 is defined in the terminfo of the terminal below.

• alacritty
• alacritty-direct
• alacritty+common
• linux
• linux-16color
• linux-koi8
• linux-koi8r
• linux-lat
• linux-m1
• linux3.0
• mintty
• mintty-direct
• putty
• putty-256color
• putty-noapp
• putty-sco
• putty-vt100
• screen.linux-m2
• screen.putty
• screen.putty-256color
• tmux
• tmux-256color
• vscode
• vscode-direct
• xterm
• xterm-1002
• xterm-1003
• xterm-1005
• xterm-1006
• xterm-16color
• xterm-256color
• xterm-88color
• xterm-basic
• xterm-debian
• xterm-direct
• xterm-direct2
• xterm-hp
• xterm-new
• xterm-nrc
• xterm-rep
• xterm-sco
• xterm-sun
• xterm-utf8
• xterm-x10mouse
• xterm-x11hilite
• xterm-x11mouse
• xterm-xmc
• xterm.js
• xterms-sun

I suggest the following method.

1. If the current terminal's terminfo defines the extended capability E3, System.Console.Clear() uses it to clear the console.
2. If E3 is not defined, System.Console.Clear() clears the console in the traditional way using only the capability clear_screen.

3.4. How to implement countermeasures

On most terminals, with a few exceptions, the extended capability E3 has the value "\u001b[3J", and the "xterm" manual describes this escape sequence as "Erase Saved Lines".

When I checked the operation of extended capability E3 on the actual machine, only the scroll buffer was cleared, and the screen was not cleared. In other words, even when using the extended capability E3, it must be used in conjunction with the capability clear_screen.

I think it's good to output the escape sequence in the following procedure.

1. Check if extended capability E3 is defined.
2. If E3 is defined, print the value of the capability clear_screen followed by the value of E3.
3. If E3 is not defined, print the value of the capability clear_screen.

Note that you must print E3 after clear_screen¹.

Author:	rougemeilland
Assignees:	-
Labels:	area-System.Console, untriaged
Milestone:	-

Footnotes

1. I think a better way to find out about this is to examine the source code of the Linux clear command (which is actually the tput command). However, I couldn't find the source code for it, so I checked it with a real machine.
   In many terminals including ubuntu (Windows terminal) the capability clear_screen is defined as "\u001bH\u001b[2J".
   At least on a real machine (Ubuntu 20.04.6 LTS), outputting "\u001b[3J" followed by "\u001b[2J" does not clear the scroll buffer completely. ↩

[adamsitnik]

Hi @rougemeilland

Big thanks for a very detailed analysis of the problem.

I think it's good to output the escape sequence in the following procedure.

It sounds very reasonable to me. Would you like to send a PR with a fix?

[rougemeilland]

Does "PR" mean "pre-release"?
If so, you don't need to send me the pre-release version. The reason is as follows.

• Conditions for occurrence of this problem and confirmation method are simple.
• I have very few types of test environments. (only for ubuntu on Widnows terminal)

I'm sorry if I misunderstood.
My main language is Japanese, and I am not good at reading and writing English. I rely on Google Translate to read and write this post. Google Translate is excellent, but sometimes it translates strangely. Especially not good at translating abbreviations :_(

[danmoseley]

Hi @rougemeilland , PR means Pull Request, that is, offer a fix to this repo if you would like. We are always happy to have new contributors.