ls: File not found exception with wildcard (expression) is not caught in Windows.

[hanbings]

Coreutils: 0.0.27 Release
Windows 11 23H2 (OS Build 22631.4169)
Debian 12

Windows:

PS C:\Users\hanbings\test> coreutils ls
a.txt
PS C:\Users\hanbings\test> coreutils ls "a.txtt"
ls: cannot access 'a.txtt': No such file or directory
PS C:\Users\hanbings\test> coreutils ls "a.txtt*"
ls: unknown io error: '"a.txtt*"', 
'Os { 
    code: 123, 
    kind: InvalidFilename, 
    message: "The filename, directory name, or volume label syntax is incorrect." 
}'
PS C:\Users\hanbings\test>

Debian:

root@application:~/test# ./coreutils ls
a.txt  coreutils
root@application:~/test# ./coreutils ls "a.txtt"
ls: cannot access 'a.txtt': No such file or directory
root@application:~/test# ./coreutils ls "a.txtt*"
ls: cannot access 'a.txtt*': No such file or directory
root@application:~/test# ./coreutils ls a.txtt
ls: cannot access 'a.txtt': No such file or directory
root@application:~/test# ./coreutils ls a.txt
a.txt

[rulu158]

Hi!

It also happens in more situations.

In Windows, with '?', '<', '>', ':', '|', '?', '*', new line ('\n'), and I'm assuming more ASCII non-printable characters. This is happens because a call to the Windows implementation of File::open(...) returning (unstable) InvalidFilename.

In Windows AND Unix, with long filenames:
Ubuntu:

rulus@rgarcial-portat:~$ coreutils ls "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
ls: unknown io error: '"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"', 'Os { code: 36, kind: InvalidFilename, message: "File name too long" }'

Windows:

PS C:\Users\rgarcial> coreutils.exe ls "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
ls: unknown io error: '"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"', 'Os { code: 123, kind: InvalidFilename, message: "The filename, directory name, or volume label syntax is incorrect." }'

This is more problematic. In Linux GNU coreutils, this is the result:

rulus@rgarcial-portat:~$ ls "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
ls: cannot access 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa': File name too long

Which I'm not sure it's the correct way of working for ls, as I think it should print "No such file or directory." in that case (you can see a little more here). I'm not sure about that though, as I couldn't find much about it.

Note that this is somewhat unstable territory, as InvalidFilename is itself part of unstable io_error_more (there is a ticket about it from just two weeks ago in Rust-lang community). InvalidFilename:

Currently corresponds to ENAMETOOLONG on Unix, ERROR_FILENAME_EXCED_RANGE & ERROR_INVALID_NAME on Windows.

This means we have to differentiate the ErrorKind, and maybe expect future changes in Rust implementation.

If you don't mind, I will take on this and start working on wildcard and special chars.

About the filename too long, I'm not sure if it should print "File name too long" or "No such file or directory." I would like to see more takes on this.

[hanbings]

Hi, I'm not sure if these issues all stem from the exception being caught in the same place.
But thanks for taking the time to continue looking into it. :)

[rulu158]

All these issues stem from the same place (calling to get_metadata -> get_metadata_with_deref_opt -> Path::symlink_metadata() -> ... -> File::open(...)).

And there are two more cases to consider.

The first, trying to ls a file as if it was a directory. Windows and Unix Rust coreutils fail, for different reasons (Linux returns ErrorKind::NotADirectory on File::open("existing_file/")):
Linux (GNU coreutils):

rulus@rgarcial-portat:~$ coreutils ls non_existing_file/
ls: cannot access 'non_existing_file/': No such file or directory
rulus@rgarcial-portat:~$ coreutils ls existing_file/
ls: cannot access 'existing_file/': Not a directory

Linux (uutils/coreutils):

rulus@rgarcial-portat:~$ coreutils ls non_existing_file/
ls: cannot access 'non_existing_file/': No such file or directory
rulus@rgarcial-portat:~$ coreutils ls existing_file/
ls: unknown io error: '"existing_file/"', 'Os { code: 20, kind: NotADirectory, message: "Not a directory" }'

Windows (uutils/coreutils):

PS C:\Users\rgarcial> coreutils.exe ls non_existing_file/
ls: cannot access 'non_existing_file/': No such file or directory
PS C:\Users\rgarcial> coreutils.exe ls existing_file/
ls: unknown io error: '"existing_file/"', 'Os { code: 123, kind: InvalidFilename, message: "The filename, directory name, or volume label syntax is incorrect." }'

The second, which only affects Windows, is when trying to ls a reserved keyword (CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9), doesn't matter if the file exists or not (this is specific of std::sys::pal::windows::fs::File::open(..), also called from Path::symlink_metadata()):

PS C:\Users\rgarcial> coreutils.exe ls CON
ls: unknown io error: '"existing_file/"', 'Os { code: 87, kind: InvalidInput, message: "The parameter is incorrect." }'

In Unix filesystems, only '/' char is an illegal filename char, while in Windows there are some of them ('?', '<', '>', ':', '|', '?', '*', non-printable 0-31 ASCII control characters), so illegal filenames are mostly a Windows problem, in which Rust implementation, as of right now, maps the error to ErrorKind::InvalidFilename. For long filenames, both Windows and Linux map the error to ErrorKind::InvalidFilename. Windows take a '/' char as an invalid filename char when trying to use it in a file it knows is not a folder, while Linux just interpretes the name with a '/' char as directly a folder, returning (also unstable) ErrorKind::NotADirectory on File::open("existing_file/") because it knows existing_file is not a directory.

About the wildcard ('*'), it should be noted that the expand functionality is solely provided by the shell, not the process itself. The shell tries to expand all folders that, for example, start with 'a' (for a*); if none exist, it just passes the string a* as the filename (as it is done when, for example, calling ls with argument a* from another process). In those cases, Unix will say "No such file or directory" because '*' is a valid filename char, but Windows complains because it is not.

Also we have to have in mind that both ErrorKind::InvalidFilename and ErrorKind::NotADirectory are unstable, and will maybe change in the future.

Taking all of that into consideration, the best solution seems to be the simplest one.

In other uutils/coreutils utilities, like mkdir or join, the IO error is shamelessly printed, which makes sense because this is not an error of the utility itself, but of something implementation-related:
Windows:

PS C:\Users\rgarcial> coreutils.exe mkdir "CON"
mkdir: Invalid input
PS C:\Users\rgarcial> coreutils.exe mkdir "asd*"
mkdir: The filename, directory name, or volume label syntax is incorrect.

To maintain consistency and handle this kind of errors, the ls utility should probably print ls: cannot access '{path}': {error} instead of ls: unknown io error: '{path}', '{error}'. This is the expected way of working for ls.

I have it changed in a branch, producing this results:
Windows:

PS C:\Users\rgarcial> coreutils.exe ls "asd*"
ls: cannot access '"asd*"': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "asd?"
ls: cannot access '"asd?"': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "asd:"
ls: cannot access '"asd:"': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "asd<"
ls: cannot access '"asd<"': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "asd>"
ls: cannot access '"asd>"': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
ls: cannot access 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls non_existing_file/
ls: cannot access 'non_existing_file/': No such file or directory
PS C:\Users\rgarcial> coreutils.exe ls existing_file/
ls: cannot access 'existing_file/': The filename, directory name, or volume label syntax is incorrect.
PS C:\Users\rgarcial> coreutils.exe ls "CON"
ls: cannot access 'CON': The parameter is incorrect.

Linux:

rulus@rgarcial-portat:~$ coreutils ls "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
ls: cannot access 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa': File name too long
rulus@rgarcial-portat:~$ coreutils ls non_existing_file/
ls: cannot access 'non_existing_file/': No such file or directory
rulus@rgarcial-portat:~$ coreutils ls existing_file/
ls: cannot access 'existing_file/': Not a directory

That is should be the expected behaviour I believe. The error for Windows reserved filenames (ErrorKind::InvalidInput) could be refactored to print Invalid input, like mkdir, but I think the system error is preferable.

I will leave a PR created for this.