Skip to content

Instantly share code, notes, and snippets.

@smarkwell
Last active December 12, 2023 22:45
Show Gist options
  • Save smarkwell/60159fb1f56498f9a53d to your computer and use it in GitHub Desktop.
Save smarkwell/60159fb1f56498f9a53d to your computer and use it in GitHub Desktop.
Loading large JSON data files into Powershell
# Work around on
# ConvertFrom-Json : Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property.
$json = New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer
$json.MaxJsonLength = 104857600 #100mb as bytes, default is 2mb
$filedata = [System.IO.File]::ReadAllText($JSONDataFile) #Using default encoding
$data = $json.DeserializeObject($filedata, [System.Object])
$filedata = $null
$json = $null
@kbbgl
Copy link

kbbgl commented Oct 18, 2018

It fails for me on line 8:

Exception calling "Deserialize" with "2" argument(s): "Invalid JSON primitive: ."

@ntarakaram1980
Copy link

Its not working getting Invalid primitive as said by kbbgl

@SpeedmaxX
Copy link

SpeedmaxX commented May 22, 2019

@sitarv
Copy link

sitarv commented Apr 25, 2020

May be it is too late here as i have come across searching for similar references... The solution provided works very well. But we need to make sure that the content is saved without conversion by Invoke-RestMethod which will automatically does the conversion if the stream size is less than default max Json length otherwise the content is written to the file as is. The problems mentioned by kbbgl and ntarakaram1980 are observed in case where the payload size is less. So it is always better to save the content with (Invoke-WebRequest -Uri 'Uri').Content instead of Invoke-RestMethod to ensure that it doesn't convert the payload automatically based on the size.

@smarkwell
Copy link
Author

works like a charm when usint $json.DeserializeObject($string) (see also https://docs.microsoft.com/de-de/dotnet/api/system.web.script.serialization.javascriptserializer.deserializeobject?view=netframework-4.8)

Updated the Gist based on this. I haven't tested this. I assume this comes back to Powershell/.NET version at the time I originally wrote the script.

May be it is too late here as i have come across searching for similar references... The solution provided works very well. But we need to make sure that the content is saved without conversion by Invoke-RestMethod which will automatically does the conversion if the stream size is less than default max Json length otherwise the content is written to the file as is. The problems mentioned by kbbgl and ntarakaram1980 are observed in case where the payload size is less. So it is always better to save the content with (Invoke-WebRequest -Uri 'Uri').Content instead of Invoke-RestMethod to ensure that it doesn't convert the payload automatically based on the size.

Not sure if my original issues was loading data out of a file or off a web request, thanks for the additional note.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment