Testing concurrent processing of piped objects and comparing globbing versus match performance in reply to https://www.reddit.com/r/PowerShell/comments/35ua3n/running_into_massive_memory_consumption_using/cr89zuo

Get-ChildItem-pipelining.ps1

Function Get-Fibonacci {
    # Function yoinked from http://blogs.technet.com/b/heyscriptingguy/archive/2010/03/07/hey-scripting-guy-march-7-2010.aspx
    Param([int]$max)
    For($i = $j = 1; $i -lt $max) {
        $i
        $i,$j = ($i + $j),$i
    }
}

Function Get-Stats {
    param (
        $MyStartTime,
        $MyCount
    )
    $Now = Get-Date
    $Stats = Get-Process -Id $PID
    New-Object psobject -Property @{
        Time = $Now
        Elapsed = ($Now - $MyStartTime)
        Count = $MyCount
        Handles = $Stats.Handles
        NPM = $Stats.NPM
        PM = $Stats.PM
        WS = $Stats.WS
        VM = $Stats.VM
        CPU = $Stats.CPU
    }
}

$FilePath = "\\SERVER\SHARE"
$Count = 0

$Fibonacci = Get-Fibonacci 10000000
$StartTime = Get-Date

#Get-ChildItem -Recurse -Path $FilePath\*.jpg |
Get-ChildItem -Recurse -Path $FilePath | Where-Object { $_.name -match '\.jpg$'} |
     ForEach-Object -Begin {
             Write-Verbose "Begin Foreach-Object Loop"
         } -Process {
             $Count ++
             if ($Fibonacci -contains $Count) {
                Get-Stats $StartTime $Count
             }
         } -End {
             Write-Verbose "End Foreach-Object Loop"
         }

Get-Stats $StartTime $Count

Synopsis: Trying to demonstrate two things:

• Pipelining this way does not create bottlenecks of arrays
• Using a wildcard in Get-ChildItem is slower for large sets of files than piping to Where-Object and matching the name.

What the script does: Sets up a few variables including a set of fibonacci numbers because they're cool and they let the script report status in reasonable increments on any data set size. I've limited my patience to occasional status updates on up to 10,000,000 items.

There are two Get-ChildItem lines. Run with one commented, observe the efficiency and speed and then switch which one is commented and run again. In my trials the match method is 3-5 times faster than the wildcard method.