Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Splits a large log file using PowerShell. Tested against very large multi-terabyte log files.
# Split Large Log File
$inputFile = "C:\temp\VeryLargeLogFile.log"
$outputDirectory = "C:\temp\logFileSplit"
$outputFileName = "LogFile"
$outputFileExtension = "log"
$upperBound = 500MB
New-Item -ItemType Directory -Force -Path $outputDirectory
$fromFile = [io.file]::OpenRead($inputFile)
$buff = new-object byte[] $upperBound
$count = $idx = 0
try {
do {
"Reading $upperBound"
$count = $fromFile.Read($buff, 0, $buff.Length)
if ($count -gt 0) {
$to = "{0}\{1}.{2}.{3}" -f ($outputDirectory, $outputFileName, $idx, $ext)
$toFile = [io.file]::OpenWrite($to)
try {
"Writing $count to $to"
$tofile.Write($buff, 0, $count)
} finally {
$tofile.Close()
}
}
$idx ++
} while ($count -gt 0)
}
finally {
$fromFile.Close()
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment