File Magic Numbers

file_magic_numbers.md

File Magic Numbers

Magic numbers are the first bits of a file which uniquely identify the type of file. This makes programming easier because complicated file structures need not be searched in order to identify the file type.

For example, a jpeg file starts with ffd8 ffe0 0010 4a46 4946 0001 0101 0047 ......JFIF.....G ffd8 shows that it's a JPEG file, and ffe0 identify a JFIF type structure. There is an ascii encoding of "JFIF" which comes after a length code, but that is not necessary in order to identify the file. The first 4 bytes do that uniquely.

This gives an ongoing list of file-type magic numbers.

Image Files

File type	Typical extension	Hex digits xx = variable	Ascii digits . = not an ascii char
Bitmap format	.bmp	42 4d	BM
FITS format	.fits	53 49 4d 50 4c 45	SIMPLE
GIF format	.gif	47 49 46 38	GIF8
Graphics Kernel System	.gks	47 4b 53 4d	GKSM
IRIS rgb format	.rgb	01 da	..
ITC (CMU WM) format	.itc	f1 00 40 bb	....
JPEG File Interchange Format	.jpg	ff d8 ff e0	....
NIFF (Navy TIFF)	.nif	49 49 4e 31	IIN1
PM format	.pm	56 49 45 57	VIEW
PNG format	.png	89 50 4e 47	.PNG
Postscript format	.[e]ps	25 21	%!
Sun Rasterfile	.ras	59 a6 6a 95	Y.j.
Targa format	.tga	xx xx xx	...
TIFF format (Motorola - big endian)	.tif	4d 4d 00 2a	MM.*
TIFF format (Intel - little endian)	.tif	49 49 2a 00	II*.
X11 Bitmap format	.xbm	xx xx
XCF Gimp file structure	.xcf	67 69 6d 70 20 78 63 66 20 76	gimp xcf
Xfig format	.fig	23 46 49 47	#FIG
XPM format	.xpm	2f 2a 20 58 50 4d 20 2a 2f	/* XPM */

Compressed files

File type	Typical extension	Hex digits xx = variable	Ascii digits . = not an ascii char
Bzip	.bz	42 5a	BZ
Compress	.Z	1f 9d	..
gzip format	.gz	1f 8b	..
pkzip format	.zip	50 4b 03 04	PK..

Archive files

File type	Typical extension	Hex digits xx = variable	Ascii digits . = not an ascii char
TAR (pre-POSIX)	.tar	xx xx	(a filename)
TAR (POSIX)	.tar	75 73 74 61 72	ustar (offset by 257 bytes)

Excecutable files

File type	Typical extension	Hex digits xx = variable	Ascii digits . = not an ascii char
MS-DOS, OS/2 or MS Windows		4d 5a	MZ
Unix elf		7f 45 4c 46	.ELF

##Miscellaneous files

File type	Typical extension	Hex digits xx = variable	Ascii digits . = not an ascii char
pgp public ring		99 00	..
pgp security ring		95 01	..
pgp security ring		95 00	..
pgp encrypted data		a6 00	¦.

Source

Hi
I've also published another gist but all data is in JSON format:
https://gist.github.com/Qti3e/6341245314bf3513abb080677cd1c93b

What you tell over JPEG files is not true. There also exist JPEG files with a ff d8 ff e1 magic number and an Exif data block, which come without the JFIF string.

What you tell over JPEG files is not true. There also exist JPEG files with a ff d8 ff e1 magic number and an Exif data block, which come without the JFIF string.

I have tested for the file formats of JPEG, JPG, and JFIF. All the three have the same result ff d8 ff e0. kindly share your image file to test it.

@BathriNathan see here: https://stackoverflow.com/questions/1975172/iphone-jpg-image-has-non-standard-magic-bytes-ff-d8-ff-e1

There are two different metadata formats for JPEG files: JFIF and EXIF. JFIF starts with ff d8 ff e0 and EXIF starts with ff d8 ff e1.

There are two different metadata formats for JPEG files: JFIF and EXIF. JFIF starts with ff d8 ff e0 and EXIF starts with ff d8 ff e1.

thanks for sharing the link for an explanation, I can now understand that we have some other metadata formats for JPEG however I could not still get the image file from the link. do you have any solution for this. can we add ff d8 ff e0 ff d8 ff e1 ff d8 ff e2 ff d8 ff e8 for JPEG validation?

@BathriNathan Here are two example files.
JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=13, manufacturer=Panasonic, model=DMC-TZ10, orientation=upper-left, xresolution=196, yresolution=204, resolutionunit=2, software=Ver.1.1 , datetime=2014:07:03 16:26:13], baseline, precision 8, 4000x3000, components 3
JPEG image data, JFIF standard 1.01, resolution (DPI), density 180x180, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=11, manufacturer=Panasonic, model=DMC-TZ10, orientation=upper-left, software=Ver.1.1 , datetime=2014:07:15 13:17:54], baseline, precision 8, 2597x1948, components 3

I would consider all headers from ff d8 ff e0 to ff d8 ff ef as valid JPEG files. I could find further examples on the Internet:
e2 Canon
e3 Samsung
e4 (maybe a corruption, so no 100 % evidence, see espressif/esp32-camera#4 )
e8 SPIFF (Still Picture Interchange File Format)
ee unknown, mentioned on Wikipedia

The JPEG specification can be found here. If you look at page 32, you can see that ff e0 to ff ef are used for application segments. If there is no application segment at all (which is totally fine, since application metadata are optional), you could expect ff db (define quantization table(s)) after ff d8 (start of image), which is also mentioned as valid on the Wikipedia article.

@spixi thanks for sharing the image. so now to be in the safe side, I will also consider the other number as you said in the validation. do we have this problem only in the JPEG or also in PNG and GIF?

@BathriNathan Although PNG allows a wide range of header fields, the magic word .PNG always appears in the first four bytes. So everything is fine. GIF always starts with GIF87a or GIF89a, which have both GIF8 in common. Everything is fine. I don’t know much about TIFF, because it has many variants like Exif, sDCF, TIFF/EP, TIFF/IT an GeoTIFF and also may come with different byte orders in the header.

@spixi I have all the mandatory hex numbers in my validation for JPEG, PNG, and GIF as per our discussion. It is working fine as expected.

What about for .pdf file what are the magic numbers in that case ? Is it <Buffer 25 50 44 46> ?

@racg0092 If you check out the Wikipedia list of file signatures it looks like the signature for a .pdf is 25 50 44 46 2D OR DF BF 34 EB CE

@racg0092 here is the like for all magic numbers https://www.garykessler.net/library/file_sigs.html and https://en.wikipedia.org/wiki/List_of_file_signatures

@leommoore @BathriNathan

Thanks for feedback ! Much appreciated it.

My JPEG has several ffd8. One at the start and several throughout the image. What are the middle ones supposed to be?

Edit: these not markers. But rather 'ff' may be from one marker and 'd8' from another and they would be adjacent by chance.
And no, these aren't thumbnails.

Great information

My JPEG has several ffd8. One at the start and several throughout the image. What are the middle ones supposed to be?

Edit: these not markers. But rather 'ff' may be from one marker and 'd8' from another and they would be adjacent by chance. And no, these aren't thumbnails.

@merajhasan88 ffd8 is the start of the image. IIRC JPEG container may contain more than one image, not only thumbnails, but also preview images for the web. A web browser could show a low-quality image while the high-quality image is loading. TIFF has a similar feature.

@qti3e There is no Magic number for plain Text files. As I debugged Text file changes the magic number every time. Other magic numbers are working but for the Text File I am getting new Magic numbers every time. Can anyone help me here!

@qti3e There is no Magic number for plain Text files. As I debugged Text file changes the magic number every time. Other magic numbers are working but for the Text File I am getting new Magic numbers every time. Can anyone help me here!

Plain text files have no magic number. However, they may contain a byte order mark. You can consider a file as a plain text file when it only contains characters valid for a certain codepage like Windows-1252 or Unicode UTF-8 and is no source code (e. g. Java, C++) / script (e. g. bash, PowerShell) / markup (e. g. XML, TeX) / config file (e. g. INI, YAML).

What you tell over JPEG files is not true. There also exist JPEG files with a ff d8 ff e1 magic number and an Exif data block, which come without the JFIF string.

I have tested for the file formats of JPEG, JPG, and JFIF. All the three have the same result ff d8 ff e0. kindly share your image file to test it.

You should test the Exif format, now Exif is more popular, Exif starts with FF E1 byte

Hi,
Please help me to convert from .bmp/.jpg to ff d8 ff e0 bitmap format.

Which tool need to use for conversion from .jpg to ff d8 ff e0 bitmap format.

please provide the solution.

Thanks,  

Hi, Please help me to convert from .bmp/.jpg to ff d8 ff e0 bitmap format.

Which tool need to use for conversion from .jpg to ff d8 ff e0 bitmap format.

please provide the solution.

Thanks,

Try ImageMagick

hi, sorry for possibly a silly question, you are talking about the start bytes, but how to detect the end of the file? where to find the end byte signature for a specific file extension?

hi, sorry for possibly a silly question, you are talking about the start bytes, but how to detect the end of the file? where to find the end byte signature for a specific file extension?

The magic numbers refer to the starting bytes of a file. End bytes signatures are not very useful, because that would require you to read the entire file. Some file types end with checksums and/or padding (which is usually filling up with 0x00).

where can i have magic number for csv file or how can i make changes to magic.mgc to identify csv file

where can i have magic number for csv file or how can i make changes to magic.mgc to identify csv file

Short answer: You can't.
Explanation: CSV files are just plaintext files. They have no special header. You can detect an XML file, because it always begins with <?xml (Unicode plaintext files may start with a Byte Order Mark), but there is no way to detect a CSV file. Only heuristic (like only usage of ASCII characters and a regular pattern of semicolons)

The magic numbers refer to the starting bytes of a file. End bytes signatures are not very useful, because that would require you to read the entire file. Some file types end with checksums and/or padding (which is usually filling up with 0x00).

I suppose the different tools for file recovery from a corrupt USB or hard drive work like they do a byte-by-byte reading with the file start and end byte signature detection. correct me if I'm wrong. If I'm not, so they somehow know when the file ends...

The magic numbers refer to the starting bytes of a file. End bytes signatures are not very useful, because that would require you to read the entire file. Some file types end with checksums and/or padding (which is usually filling up with 0x00).

I suppose the different tools for file recovery from a corrupt USB or hard drive work like they do a byte-by-byte reading with the file start and end byte signature detection. correct me if I'm wrong. If I'm not, so they somehow know when the file ends...

The size of a file is saved in the directory information of the file system. When this information is corrupt, recovery tools can try to find the boundary of files (e. g. by scanning for magic numbers), but this is not 100 % reliable, especially when a file is fragmented over multiple data block ranges.