I recently found myself enraptured in a pet project to periodically check PS5 stock across the major U.S. retailers and send a Discord message to a set of recipients when one is found. I set my sights on the following retailers:
- Amazon
- Target
- Walmart
- BestBuy
- GameStop
- Playstation Direct
GameStop and Playstation Direct do not do bot checks, CATCHPA, or any other security method so I simply used Java's HTTPRequest system and split the response with certain HTML tags to check for "out of stock" related text (interestingly enough, most of these retailers use different terms for "out of stock"). BestBuy provides a free consumer API, making the process significantly easier. Walmart has their Walmart IO API, but during the holiday season unfortunately they have disabled the creation of new applications or accounts. I intend to add Walmart support once the holiday season is over, seeing as the bot has been running for the entirety of Black Friday and (at the time of this writing) has not gotten a single hit. Amazon also has an API but it requires you to be a part of their program that requires referrals and sales, which I cannot participate in due to my lack of a platform to do either.
However, this leaves a lone target... Target! I originally tried to use my HTTPRequest method but came across an interesting peculiarity. While I wasn't getting any CATCHPAs, the .html files the program was generating was procuring pages, except for the product area being blank:
Note: I'm using a basket for this example because I wrote a simple example app to demonstrate and when I plugged in the link for the PS5 it said the product was unavailable...
Target accessed by a browser
Target accessed by java.net.http.HttpResponse
Now, while I experimented with Selenium on different storefronts later and I believe that might've alleviated the issue, at the time, I didn't know of much of a solution. So, I looked into the webpage to see what made it tick. What script was being loaded into the page after the fact to show this information? I went to inspect element. Seeing as I was looking for stock, I inspected the delivery/pick-up information. Something about this might pop out to you as it did for me:
For the purposes of this, I've plugged in a random ZIP Code... mileage may vary!
The word that stuck out to me was fulfillment. So, I went to network sources to see what I could find.
Based on the fact that it is highlighted and also the name of this gist, you may suspect my target (heh) of interest is that redsky.target.com domain, and you would be correct! The big kicker, however, is the response to that request:
{
"data": {
"product": {
"__typename": "Product",
"tcin": "14758453",
"notify_me_enabled": false,
"store_positions": [
{
"aisle": 40,
"block": "C"
}
],
"fulfillment": {
"product_id": "14758453",
"is_out_of_stock_in_all_store_locations": false,
"store_options": [
{
"location_name": "Lakeland North",
"location_address": "4005 US Highway 98 N,Lakeland,FL,33809-3815",
"location_id": "1859",
"search_response_store_type": "PRIMARY",
"location_available_to_promise_quantity": 5,
"order_pickup": {
"pickup_date": "2021-11-26",
"guest_pick_sla": 120,
"availability_status": "IN_STOCK"
},
"in_store_only": {
"availability_status": "IN_STOCK"
}
}
],
"shipping_options": {
"availability_status": "OUT_OF_STOCK",
"loyalty_availability_status": "OUT_OF_STOCK",
"available_to_promise_quantity": 0,
"reason_code": "INVENTORY_UNAVAILABLE",
"minimum_order_quantity": 1,
"services": []
}
}
}
}
}Aha! Stock information! The first point of interest here is the amount of information given. There are variables for the product being out of stock in all locations, information for quantities, and information for by what venues there is stock (shipping, in-store, etc.). The second point of interest is the fact that Target uses the ZIP code saved for me (shown by the request URL) and gets distribution information for that area from the store that I have picked (in this case, a store in Lakeland, FL) down to the latitude/longitude. Lastly is how the request URLs are constructed:
https://redsky.target.com/redsky_aggregations/v1/web/pdp_fulfillment_v1
?key=ff457966e64d5e877fdbad070f276d18ecec4a01
&tcin=14758453
&store_id=1859
&store_positions_store_id=1859
&has_store_positions_store_id=true
&zip=98801
&state=WA
&latitude=47.430
&longitude=-120.320
&pricing_store_id=1859
&has_pricing_store_id=true
&is_bot=false
?key=ff457966e64d5e877fdbad070f276d18ecec4a01
This key is reminiscent of a UUID or hash. I assume it is a static verification key to make a valid request, as when you change a digit in it and make a request, you get the following JSON:
{
"message": "Invalid Key",
"errors": [
{
"resource": "Go-Proxy",
"field": "",
"reason": "key is inactive or otherwise disabled in PROD environment"
}
]
}Fascinatingly enough, this infers the existence of alternate keys for the Redsky API, but at the very least this one key appears to have been hardcoded in. The key has remained the same across different browsers, VPNs, and products I've tried and has continued to work for requests over the last week.
NOTE: I've been informed that this is "less of a key and more of a header", makes sense due to the fact that this is a request URL 👍
&tcin=14758453
The TCIN value is the product ID. It is contained in every Target product URL preceded by A-:
https://www.target.com/p/y-weave-medium-decorative-storage-basket-room-essentials-153/-/A-53816617?preselect=52853977#lnk=sametab
Note: For products with selectable options like colors, TCINs get tricky both in Redsky requests, responses, and different types of requests made. For instance, with our Y-Weave Medium basket example, a distinctly different redsky.target.com fulfillment request was made with a tcins field and multiple product IDs put into the request.
&store_id=1859
&store_positions_store_id=1859
&pricing_store_id=1859
These lines all refer to the store ID of the store I input for fulfillment information. I am not sure why different values would be needed for general, pricing, and stock information, but the three values are distinctly separate.
&has_store_positions_store_id=true
&has_pricing_store_id=true
These booleans appear to refer to the availability of in-store positions or pricing for specific stores. For instance, I assume if an aisle number could not be found for a product then the has_store_positions_store_id key would be set to false.
&zip=98801
&state=WA
&latitude=47.430
&longitude=-120.320
These values refer to a location I believe is set the first time you visit the site without cookies (I presume the first time I visited my VPN was set to Washington!). This data was present even though at the time of the request my VPN was set to Silicon Valley. This ZIP code + the store ID can both be edited from the product page, subsequently modifying the data.
&is_bot=false
Last but not least, this valiant yet entirely useless flag... I presume the intention of this is if Target CATCHPA-flags the client then the request is not sent, maybe as a bandwidth saving measure? However, in practice this flag is demonstrably useless:
Pictured: Valiant software engineer's effort decimated by the scourge of all programmers: Java (circa 2021)
Target isn't outwardly friendly to scraping or automated inventory status collection outside of their corporate + partners sanctioned API. Otherwise, their API would be open like Best Buy's.
One of these makes hobbyist software engineering student me happy... the other makes me sad and also impressed on how well the Target logo fits into a gear.
However, I do think that it is super nice of them to leave this API publicly available and easily usable for anyone that is savvy enough to utilize it. I personally intend to use this where possible in my program knowing that it is on the up and up. All this does beg the question though: if one of the United States' most prominent big-box retailers has their distribution API open for those who know how to use it: what else is out there to tinker with and explore?
~ Luma
NOTE: This was able to get passed along to some really awesome and important people over at Target who took time out of their day to explain that the Redsky API is public and that there are some other ones they intentionally keep open. Because the data is considered publicly available they also keep the API publicly accessible. A giant thank you to Mike McNamara and Nancy King over at Target, they're amazing!
Have you had any issues getting your IP banned? It seems like mine is cooked.