Skip to content

Instantly share code, notes, and snippets.

@yifanlu yifanlu/psv.h Secret
Last active Jul 17, 2019

Embed
What would you like to do?
/**
* Motivation: One unified .psv format for archiving (preserving) Vita games.
* The goal is to preserve as much of the original game structure while ensuring
* the all the information needed to decrypt and extract data can be derived
* from just the file and a hacked Vita.
*
* We want something akin to .nds or .3ds/.cia or .iso but for Vita games. The
* unique challenge is that Vita cart games require a per-cart key to decrypt
* and digital games require a similar key from activation. With just the raw
* game image, it is not possible to extract the game data.
*
* What's wrong with using .vpk? VPK is designed for homebrew. The patches to
* enable homebrew strips out a lot of the game executable metadata as well as
* change the system state to be different than a Vita running an original game.
* This leads to many subtle as well as major bugs (saves not working, some
* games require additional patches to run, saves are not compatible with
* non-hacked Vitas, etc).
*
* Why not just ZIP the original files? Why not strip PFS as well to make data
* mining/emulation easy? Why not make a compressed format? One reason is that
* by stripping more than necessary (like, for example PFS), we might be losing
* information that we currently do not think is important. An example of this
* is when SNES games are first dumped and Earthbound was not dumped properly
* and people did not know about the anti-piracy checks until much later. There
* may be, for example, games that do timing checks or checks on the file
* modification time or something. Either explicitly for anti-piracy or
* implicitly due to bad programming (a lot of older consoles are infamous for
* the latter case). By preserving as much of the original structure as
* possible, we ensure that we can somehow play these games in a future where no
* more Vitas exist.
*
* Different tools (data extraction, backup loaders, archival storage, etc)
* might require different use cases. Someone might for example want to strip
* PFS and compress the game data for more efficient storage. We invite them to
* extend this format though flags BUT just as you shouldn't store all your
* photos in level-9 compressed JPEG, your code in executables, or any data you
* care about in a lossy format, you should archive your games in its original
* form. You can easily go from a RAW image to a JPEG but you cannot go back.
*/
typedef struct {
uint32_t magic; // 'PSV\0'
uint32_t version; // 0x00 = first version
uint32_t flags; // see below
uint8_t key1[0x10]; // for klicensee decryption
uint8_t key2[0x10]; // for klicensee decryption
uint8_t signature[0x14]; // same as in RIF
uint8_t hash[0x20]; // optional consistancy check. sha256 over complete data (including any trimmed bytes) if cart dump, sha256 over the pkg if digital dump.
uint64_t image_size; // if trimmed, this will be actual size
uint64_t image_offset_sector; // image (dump/pkg) offset in multiple of 512 bytes. must be > 0 if an actual image exists. == 0 if no image is included.
opt_header_t headers[]; // optional additional headers as defined by the flags
} psv_file_header_t;
#define PSV_MAGIC (0x00565350) // 'PSV\0'
#define FLAG_TRIMMED (1 << 0) // if set, the file is trimmed and 'image_size' is the actual size
#define FLAG_DIGITAL (1 << 1) // if set, RIF is present and an encrypted PKG file follows
#define FLAG_COMPRESSED (1 << 2) // undefined if set with `FLAG_TRIMMED` or `FLAG_DIGITAL`. if set, the data must start with a compression header (not currently defined)
#define FLAG_LICENSE_ONLY (FLAG_TRIMMED | FLAG_DIGITAL) // if set, the actual PKG is NOT stored and only RIF is present. 'image_size' will be size of actual package.
typedef struct {
uint32_t type; // 0x1 indicates header for digital content
uint32_t flags; // 1 == game, 2 == DLC, etc (not yet specified)
uint64_t license_size; // size of RIF that follows
uint8_t rif[]; // rif file
} digital_header_t;
typedef struct {
uint32_t type; // 0x2 indicates header for compression
uint32_t compression_algorithm; // not yet specified
uint64_t uncompressed_size;
} compression_header_t;
typedef union {
uint32_t type;
digital_header_t;
compression_header_t;
} opt_header_t;
/**
* Sample Usage 1: Game Cart Archival
* flag = 0, rif_size = 0, image_size = size of game dump, header is
* followed by raw dump of game cart
* Sample Usage 2: Save space of dump
* flag = FLAG_TRIMMED, rif_size = 0, image_size = size of game dump,
* header is followed by trimmed dump (trailing zeros are not included)
* Sample Usage 3: Digital content archival
* flag = FLAG_DIGITAL, rif_size = 0x200 (size of rif), image_size =
* size of PKG from PSN servers, header is followed by RIF followed
* by the game PKG
* Sample Usage 4: Backup of license for digital content
* flag = FLAG_DIGITAL | FLAG_TRIMMED, rif_size = 0x200, image_size =
* size of PKG from PSN servers, header is followed by RIF
**/
@motoharu-gosuto

This comment has been minimized.

Copy link

commented Sep 8, 2017

I have fully read the description. Using flag and adding rif_size field is a very good idea. I agree on extending the format.
Archiving can also save quite a lot of space actually, so this definitely should be done. It is not that hard to fix my code.
You have also mentioned that it is preferred to store klicensee as well because we want to decouple the dump from act.dat.
This is a good idea as well. So should we add klicensee field to the structure?

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 8, 2017

I would prefer just storing the key to decrypt klicensee than klicensee itself. Correct me if I'm wrong but for digital games, klicensee key decryption key is derived from act.dat, right? And for cart, derived from CMD56? So just as we don't want to depend on CMD56 for carts, we don't want to depend on act.dat for digital... but we want to keep most of the DRM intact (RIF here). For carts, the RIF is inside the game dump, but for digital, the RIF is downloaded separately from the PKG.

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 8, 2017

Archiving can also save quite a lot of space actually, so this definitely should be done.

I think you are talking about trimming. When you "compress" a raw dump, the only thing that's getting removed are the padding since all game data is encrypted. So I think it's easier to just call it trimming and for the loader to assume all bytes after the end of the file are 00 than to deal with compression that way.

@devnoname120

This comment has been minimized.

Copy link

commented Sep 8, 2017

The proposal looks good to me, but I would like to make a few comments.

So I think it's easier to just call it trimming and for the loader to assume all bytes after the end of the file are 00 than to deal with compression that way.

Trimming is a form of compression, even though it's a primitive one. Should we have one flag per compression type (FLAG_COMPRESSION_TRIMMED, FLAG_COMPRESSION_DEFLATE, etc.) or add an enum in a new field? The latter looks more tidy to me, although it wastes space.

I think that it would make sense to keep some metadata about the game that is contained in the dump. For example the title ID, the game name (optional), the release date (optional), and the developer (optional). I know that this information is already contained in the GC image or PKG, but I don't think it's easily accessible.

I don't see any checksum to make sure that the dump is correct, and that it hasn't been tampered with. Could we add one, for instance using SHA-256?

Although we might want to keep it for a later version, we need to keep in mind that it's not always possible to obtain an official PKG file for some games. For instance, many users have PSM games installed on their PS Vita (that they could dump), yet PKG files cannot be obtained anymore (as far as I know).

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 8, 2017

I agree with the checksum--this would be especially useful for FLAG_LICENSE_ONLY type with an expected pkg checksum.

I think trim != compression, but we can have a flag FLAG_COMPRESSED (1 << 2) AND a new "compressed format header" following the file header (we may also want additional info just for compressed files).

I also agree with keeping some metadata but let's say it is optional? But I don't want to include too much metadata (like title, release date, etc) because that makes it feel too specific for a particular use case for a general format.

@motoharu-gosuto

This comment has been minimized.

Copy link

commented Sep 8, 2017

Correct me if I'm wrong

Yes. This is correct. For digital games 2 keys are derived from act.dat. For game cards 2 keys are derived from CMD56. You then use exactly same function to decrypt klicensee using the keys.

I dont actually follow the statement

we don't want to depend on CMD56 for carts, we don't want to depend on act.dat for digital

You mean we still want to store CMD56 data (which is a good decision, I agree, since we want to preserve most of DRM), but we should not use these keys?
Or by "don't want to depend" you mean dependance on hardware in the game card that handles CMD56, so we store the keys in the dump?

I think you are talking about trimming.

Yes I was talking about trimming. Compression is useless since data is encrypted.

@devnoname120

This comment has been minimized.

Copy link

commented Sep 8, 2017

I also agree with keeping some metadata but let's say it is optional? But I don't want to include too much metadata (like title, release date, etc) because that makes it feel too specific for a particular use case for a general format.

The reasoning behind this is that tools and emulators will need this information to inform the user about the dump. If we don't enforce some information to be provided, it won't be.

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2017

@motoharu I mean just what's currently in the format--it was just a statement of what is already being done.

@devnoname120, I think we can just store the titleid and tools can have their own database (release date, cover, etc). The only tricky part is that dlc/addon have the same titleid but I think whatever tool is organizing can do some work to see if it's a game vs. other content.

@CelesteBlue-dev

This comment has been minimized.

Copy link

commented Sep 18, 2017

What is key2 ? There is normally only 1 klicensee...
signature is completely useless for PSN games because it is the signature of the RIF and there are millions of different RIF for the same game.
By the way it could be a way to know if the game is digital or cratridge : if (signature == null) type=digital;
In the case you only used the signature for PSN games, I told you : it's useless.
For cartridge it could allow to check if the rif file embedded in the .psv is not broken.
I remind you that to get klicensee for PSN games you need : act.dat (per account / per console) + .rif(per account / per game) + idps (per console) through NpDrm algorithm.
For cartridge games : per console not needed (act.dat and idps) and used a different algo and a special kmodule and F00D SELF.

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 18, 2017

@CelesteBlue-dev please read the wiki. Signature isn't for our benefit, it's used in the rif decryption process. What you cite is reveals less knowledge than what's currently on the wiki. For example, you don't need act.dat. You only need it when you don't know how act.dat works. If you extract the right klicensee key decryption key from act.dat, you don't need it or idps.

@devnoname120

This comment has been minimized.

Copy link

commented Sep 19, 2017

I've put a little tool together to help view the structure of .psv files: https://gist.github.com/devnoname120/47da5187b29863d09793a373718d421d

@yifanlu

This comment has been minimized.

Copy link
Owner Author

commented Sep 21, 2017

Proposal for an extension that allows backing up digital games in a pseudo-gamecard format (for compatibility).

What's needed is a fake exFAT image with the required files in the correct path. (If it's a digital game, it would be in app/titleid, if it's DLC, it would be in app/patch, if it's PSM it would be psm/titleid, and so on). A patched RIF shall reside in the correct license directory as well.

Header changes: key1/key2/signature shall be zero indicating pre-patched content. Maybe a new flag FLAG_PATCHED shall indicate that files have been changed. SHA hash shall still be used for integrity.

The main use-case is to have a common format for when emulators come in play.

All current NoNpDrm dumps can be converted to this "patched" format.

@frangarcj

This comment has been minimized.

Copy link

commented Sep 22, 2017

original psm games pkg can be retrieved from web so I prefer to store header + digital header + pkg

template:
http://zeus.dl.playstation.net/psm/np/NPPA/NPPA00010_00_D09A76B4/2.00/NPPA00010_00.pkg

NPPA = first 4 chars from titleid
D09A76B4 = FIRST 4 bytes from HMACSha1 of NPPA00010_00
2.00 = version (unknown)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.