Removing console window

Jul 15, 2023

Introduction #

So, for the people of the world who are using scoop as their package manager (instead of chocolatey), you should’ve come across a weird console window while opening a shim which links to a GUI application.

For instance,

scoop install brave # brave
scoop install vim # gvim, gvimdiff
scoop install notepadplusplus # notepad++

is a nice way to install programs and localize their data to the scoop directory. This is the main reason I use scoop. Unlike chocolatey, it doesn’t make its packages spill all their data in the PC, everything is localized in a directory (SCOOP_DIR environment variable).

But, opening brave is a pain because, it also starts a console window along with it. GUI Programs do not need an additional console window. If it needed one, it would be compiled in a different way already.

I decided to fix this issue.

Approach #

First, I examined the shim’s source code.

In the src/shim.cs file:

if(!CreateProcess(null, cmd, IntPtr.Zero, IntPtr.Zero,
    bInheritHandles: true,
    dwCreationFlags: 0,
    lpEnvironment: IntPtr.Zero, // inherit parent
    lpCurrentDirectory: null, // inherit parent
    lpStartupInfo: ref si,
    lpProcessInformation: out pi)) {

    var error = Marshal.GetLastWin32Error();
    if(error == ERROR_ELEVATION_REQUIRED) {
        // handle this error (we don't need this)
    }
    return error;
}

WaitForSingleObject(pi.hProcess, INFINITE);

uint exit_code = 0;
GetExitCodeProcess(pi.hProcess, out exit_code);

// Close process and thread handles.
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);

return (int)exit_code;

In the build.ps1 file:

& "$PSScriptRoot\packages\Microsoft.Net.Compilers\tools\csc.exe" /deterministic /platform:anycpu 
    /nologo /optimize /target:exe /out:"$build\shim.exe" "$src\shim.cs"

Let’s see how the installer prepares the shim (for .exe/.com builds):

It gets the target path of the application which it should shim.
It copies a default pre-compiled shim executable (which has been compiled with the above command) to the destination shim directory with the same name of the executable.
It prepares a .shim file which matches the base name of the target application with content in the following format:
```
path = <path to executable without quotes>
args = <arguments>
```

You can guess, the shim starts the application with the path and the required args using the CreateProcess API, which is really asynchronous. But, we are waiting for the handle returned by CreateProcess API using the WaitForSingleObject function.

Now, I didn’t want to introduce a lot of changes in the source code.

I want to discuss how executables are compiled in windows. We know that, for some executables, we don’t see a console window. And this is due to the Subsystem field in the PE file format. I had a lot of pain reading it, maybe you should skip the link, but the key takeaway is that the operating system checks this value to determine the way the file should be treated while running. This is the list of values are found in the table.

If you looked at the table, you should have found the following:

Awesome. We can deduce that (or by some experimentation) if the subsystem value is 2, windows disables the command line. Or if it is 3, windows creates a console window. It is fascinating that, no matter how much abstract things become, these little handy values have a lot of power in them and can do anything under the hood.

Remember, binary editing in these fields is 100% OK, but you should be careful when you are inserting more data than it requires. If we are trying to replace an integer, lets insert a 4-byte value. Inserting something larger than the allocated space will corrupt the next subsequent fields in the executable. We don’t have much flexibility inside an executable.

Alright, I liked the nature of the solution as it doesn’t need any change in the shim source code, or the nature of the manifests. There are pull requests requesting a change in the manifests, but it should be done by every developer contributing an application to the repository. I already wrote that I don’t like changes like these. Minimal effort, Maximum outcome is my way to go.

Implementation #

First, I forked the Scoop repository. Then I introduced a few powershell functions (in src/core.ps1) to scrap or change the subsystem value off the target application.

function Get-PESubsystem($filePath) {
    try {
        $fileStream = [System.IO.FileStream]::new($filePath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
        $binaryReader = [System.IO.BinaryReader]::new($fileStream)

        $fileStream.Seek(0x3C, [System.IO.SeekOrigin]::Begin) | Out-Null
        $peOffset = $binaryReader.ReadInt32()

        $fileStream.Seek($peOffset, [System.IO.SeekOrigin]::Begin) | Out-Null
        $fileHeaderOffset = $fileStream.Position

        $fileStream.Seek(18, [System.IO.SeekOrigin]::Current) | Out-Null
        $fileStream.Seek($fileHeaderOffset + 0x5C, [System.IO.SeekOrigin]::Begin) | Out-Null

        return $binaryReader.ReadInt16()
    } catch {
        return -1
    } finally {
        $binaryReader.Close()
        $fileStream.Close()
    }
}

function Set-PESubsystem($filePath, $targetSubsystem) {
    try {
        $fileStream = [System.IO.FileStream]::new($filePath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::ReadWrite)
        $binaryReader = [System.IO.BinaryReader]::new($fileStream)
        $binaryWriter = [System.IO.BinaryWriter]::new($fileStream)

        $fileStream.Seek(0x3C, [System.IO.SeekOrigin]::Begin) | Out-Null
        $peOffset = $binaryReader.ReadInt32()

        $fileStream.Seek($peOffset, [System.IO.SeekOrigin]::Begin) | Out-Null
        $fileHeaderOffset = $fileStream.Position

        $fileStream.Seek(18, [System.IO.SeekOrigin]::Current) | Out-Null
        $fileStream.Seek($fileHeaderOffset + 0x5C, [System.IO.SeekOrigin]::Begin) | Out-Null

        $binaryWriter.Write([System.Int16] $targetSubsystem)
    } catch {
        return $false
    } finally {
        $binaryReader.Close()
        $fileStream.Close()
    }
    return $true
}

Now, I modified the shim function where it installs the shim for .exe or .com files:

$target_subsystem = Get-PESubsystem $resolved_path
if ($target_subsystem -eq 2) { # we only want to make shims GUI
    Write-Output "Making $shim.exe a GUI binary."
    Set-PESubsystem "$shim.exe" $target_subsystem | Out-Null
}

So, note that, the shim function is called during installation to install an application, so you should force a reinstallation/update of your package.

scoop update -f brave  # Don't use the old version!

It doesn’t always redownload the package, scoop maintains app cache in the cache folder, so don’t worry. So far, while patching the scoop shim, I didn’t experience any lag during the installation. It would be worth noting that we only read a couple of bytes and jump a few times in the file, which is pretty fast.

This PR was merged into develop. See: #5559