Convert winget output to PowerShell objects

ConvertFrom-WingetStdout.ps1

# this is one of Stéphane BARIZIEN's public domain scripts
# the most recent version can be found at:
# https://gist.github.com/sba923/7924b726fd44af91d18453ee595e6548#file-convertfrom-wingetstdout-ps1

#requires -version 7

# This crude script converts the output of the winget.exe executable into an array of PowerShell objects
# usage: winget <args> | ConvertFrom-WingetStdout.ps1
#
# examples of application:
#
# 1. Upgrade everything except some apps (e.g. managed by your employer's IT, 
#    or you know winget doesn't handle them properly yet)
#
#    winget upgrade | ConvertFrom-WingetStdout.ps1 | ? { $_.Id -notin ('VideoLAN.VLC', 'Microsoft.Office', 'Kitware.CMake') } | % { winget upgrade --id $_.Id }
#
#
# If this code doesn't work, I dunno who wrote it.
#
# Stéphane BARIZIEN <github.nospam4sba@xoxy.net>
#

param([string] $DebugCmd = 'upgrade')

# winget now outputs UTF-8 e.g. for '…' in the 'Available' column, we need to account for this
[Console]::InputEncoding = [Console]::OutputEncoding = $InputEncoding = $OutputEncoding = [System.Text.Utf8Encoding]::new()
# get rid of PSSA warning
$null = $InputEncoding

$index = 0
$fieldnames = @()
$fieldoffsets = @()
$offset = 0
$re = ""
$objcount = 0

# regex for matching progress information such as
# ██████████████████▒▒▒▒▒▒▒▒▒▒▒▒  2.00 MB / 3.20 MB
# ████████████████████████████▒▒  3.00 MB / 3.20 MB
# ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒  0%
# ΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûêΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆΓûÆ  1024 KB / 3.49 MB
# add U+2589..U+258F to account for https://github.com/microsoft/winget-cli/pull/2046
# 
# first part of $progress_re within first () is 
# U+00d4 U+00fb U+00ea U+007c 
# U+00d4 U+00fb U+00c6 U+007c 
# U+0393 U+00fb U+00ea U+007c 
# U+0393 U+00fb U+00c6 U+007c 
# U+005b U+2588 U+2589 U+258a U+258b U+258c U+258d U+258e U+258f U+2592 U+005d
$progress_re = '(Ôûê|ÔûÆ|Γûê|ΓûÆ|[█▉▊▋▌▍▎▏▒])+\s+([\d\.]+\s+.B\s+/\s+[\d\.]+\s+.B|[\d\.]+%)'

# logfile for debugging
$logfile = Join-Path -Path $env:TEMP -ChildPath ($MyInvocation.MyCommand.Name -replace '\.ps1', '.log')

# for debugging within VScode
if ($Host.Name -eq 'Visual Studio Code Host')
{
    Write-Debug ("Debugging with output from 'winget {0}'" -f $DebugCmd)
    $data = & winget $DebugCmd
}
else
{
    $data = $input
}

function DumpString([string] $string)
{
    $result = "hex: "
    for ($index = 0; $index -lt $string.Length; $index++)
    {
        $result += ("{0:x2} " -f [int]$string[$index])
    }
    return ($result -replace '\s+$', '')
}

foreach ($line in $data)
{
    Write-Debug("index={0}, fieldcount={1}, fieldnames={3}, re='{2}'" -f `
            $index, `
            $fieldnames.Count, `
            $re, `
        ($fieldnames -join ':') `
    )
    Write-Debug ("line='{0}'" -f ($line -replace '[\x01-\x1F]', '.'))
    # skip lines before the column headers
    if ($line -notmatch '^\s+\x08' -and $line -notmatch $progress_re -and $line -notmatch '^\s*$')
    {
        # build regex from line with field names
        if ($index -eq 0)
        {
            $line0 = $line
            while ($line -ne '')
            {
                if ($line -match '^(\S+)(\s+)(.*)')
                {
                    $fieldnames += $Matches[1]
                    $fieldoffsets += $offset
                    $offset += $Matches[1].Length + $Matches[2].Length
                    $line = $Matches[3]
                }
                else
                {
                    $fieldnames += $line
                    $fieldoffsets += $offset
                    $line = ''
                }
            }

            $re = '^'
            for ($fieldindex = 0; $fieldindex -lt ($fieldnames.Count - 1); $fieldindex++)
            {
                $re += ('(.{{{0},{0}}})' -f ($fieldoffsets[$fieldindex + 1] - $fieldoffsets[$fieldindex]))
            }
            $re += '(.*)'
        }
        # skip separator line
        elseif ($index -eq 1)
        {
            if ($line -notmatch '^-+$')
            {
                if ($line -match $progress_re -or $line0 -match $progress_re) # progress info, skip and reset index
                {
                    $msg = ("Skipping:`n{0}`n{1}" -f $line0, $line)
                    $msg | Out-File -Encoding utf8BOM -Append -LiteralPath $logfile
                    $index = -1
                }
                else
                {
                    $msg = ("Unexpected input:`n{0}`n{1}" -f $line0, $line)
                    Write-Host -ForegroundColor Red $msg
                    $msg | Out-File -Encoding utf8BOM -Append -LiteralPath $logfile
                    Exit(1)
                }
            }
        }
        else
        {
            # if line matches regex, turn into object and output said object to pipeline
            if ($line -match $re)
            {
                $obj = New-Object -TypeName PSObject
                for ($fieldindex = 0; $fieldindex -lt ($Matches.Count - 1); $fieldindex++)
                {
                    Add-Member -InputObject $obj -MemberType NoteProperty -Name $fieldnames[$fieldindex] -Value ($Matches[$fieldindex + 1] -replace '\s+$', '')
                }
                $obj
                $objcount++
            }
            else
            {
                Write-Debug ("Cannot match input based on field names: '{0}' 're='{1}')" -f $line, $re)
            }
        }

        $index++
    }
    else
    {
        # skip
        Write-Debug ("Skipped '{0}' ({1})" -f ($line -replace '[\x01-\x1F]', '.'), (DumpString -string $line))
    }
}

Write-Debug("Output {0} object(s)" -f $objcount)

I need to find a way to avoid this transient issue:

There is a five-minute TTL (by default) for the PreIndexed cache. This is happening when the package manager is downloading a new copy of the source package.

Was my guess.

I'm building a workaround for this.

Is there a way to force this short of waiting 5 minutes between tests? 😝

You can set the interval in winget settings you can set it to 1 minute. 0 minutes means don't update the cache.

The latest code improves on the progress things, but still sometimes fails with:

Added some debugging logic that hopefully will help me track down the remaining parsing corner cases.

For anyone running into errors on the regex lines, make sure to save this script with UTF-8 encoding with Byte Order Mark (BOM). With Notepad++, simply go to 'Encoding' -> 'UTF-8-BOM', and save again.

@denelon something has changed today (in winget? In Windows Terminal?) , I now get this that my current code doesn't handle:

Any clue?

We haven't released anything new in the last few weeks. I'm not sure when Windows Terminal was last updated. It looks like it might be the progress bar characters when the source is being updated or an installer is being downloaded.

The plot thickens...

I sometimes get:

Instead of the expected:

(got both in the same PowerShell instance running in the same Windows Terminal tab on the same machine, a few minutes apart...).

Seems to be linked to piping (or redirecting the output with >) in PowerShell (5.1 or 7.2.2, within Windows Terminal or standalone in conhost)...

winget list with no piping outputs (as displayed within PowerShell within Windows Terminal):

but if I pipe the output I get:

I can't repro this with piping or redirection in CMD.

@denelon, any clue?

@sba923,

I haven't seen this before. Let me ask around.

@sba923,

I haven't seen this before. Let me ask around.

Great.

FWIW I have tested with PowerShell 7.3.0-preview.3, and with -noprofile. Problem occurs everywhere.

By the way, this is coming in the next release, and it may impact what you're doing with the progress bar:
microsoft/winget-cli#2046

By the way, this is coming in the next release, and it may impact what you're doing with the progress bar: microsoft/winget-cli#2046

Thanks for the warning. Is there a pre-built binary I can already experiment with, or do I need to clone and build?

I think @jedieaston may have a fork with a binary, but I'm not sure. He's been staying on the bleeding edge 😊

Based on the PR's code, the change I just made should cope with the new progress bar.

If you want the bleeding edge: https://github.com/jedieaston/winget-build/releases/latest

If you want the bleeding edge: https://github.com/jedieaston/winget-build/releases/latest

Thanks.

Still looking for a way to force winget list (or another non-install-related command) to produce a (slow-growing) progress bar for testing...

@sba923,

I haven't seen this before. Let me ask around.

@denelon, any news? I'm still seeing that:

At first I was confused because of the output width, but the answer is that that the Available column has output a …, and by default PowerShell doesn't like UTF-8.

I don't know enough to tell you exactly which you need, but I did this in the example for Completion:

[Console]::InputEncoding = [Console]::OutputEncoding = $OutputEncoding = [System.Text.Utf8Encoding]::new()

Thanks for the hint.

The current code now correctly copes with winget's output. @denelon this use of UTF-8 must've been introduced at some point.

But now I'm wondering why winget doesn't size the columns according to the terminal's width, in order to avoid … when not needed:

I believe we check the width and truncate at narrower sizes, but I'm not sure if we have logic taking longer widths into consideration. @JohnMcPMS can you confirm?

We've been using UTF-8 output forever, there just isn't a lot that we generate outside of ASCII if you have your language set to English.

We do adjust for width, I just tried it with a maximized Terminal window and it used the entire width. The problem is that we only buffer so many lines (default 50) before we fix the size and start outputting. If data beyond the buffer has columns that are too wide, they still get cropped.

Thanks for the clarification. Indeed, the first entries in the Available column are quite narrow, resulting in everything wider (and that's in my current case after 74 entries) being truncated with an ellipsis.

What we really need IMVHO is an option resulting in the same behavior as piping into Format-Table -AutoSize in PowerShell so that no data is lost.

That is likely to be a part of a feature request. We've been considering some kind of a "--json" argument to specify the output should be in a format other than optimized for screen width (in this case JSON formatted).

Sure. Will file a feature request for that (something like a --autocolumnwidth option).

Created microsoft/winget-cli#2161

Hero level stuff here.

I found this script and it gave me joy.

I use it with Out-ConsoleGridView:

 winget list | ./ConvertFrom-WingetStdout.ps1 | ? { $_.Source -in ('winget') } | ocgv | % { winget uninstall --id $_.Id }

This example just selected one package, but OCGV and the command line supports multiple. Great way to select all the sh*t you want to uninstall or upgrade or whatever.

Hero level stuff here.

Thanks @tig for the praise! I'm glad this is helpful to you (and hopefully to others!).

As you probably know, this is a temporary solution until winget sports a mechanism that outputs structured data instead of stdout text.

@denelon do you want to comment on this / share info on where you stand?