Skip to content

Instantly share code, notes, and snippets.

@jimrubenstein
Last active September 15, 2023 19:04
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save jimrubenstein/9827718c508b2b7f5bad578bae421bfc to your computer and use it in GitHub Desktop.
Save jimrubenstein/9827718c508b2b7f5bad578bae421bfc to your computer and use it in GitHub Desktop.
Restore files from .git/objects

This script will restore all the objects in your .git/objects folder.

To start, you must identify what the "root object" is. This will be a tree object that lists out the files that are in the root of your git repository.

Once you have that object identified, this script will recursively walk through the tree and restore all the files that git knows/knew about.

background

I started working on a new project and got quite a bit of work done locally before initializing a git repo or creating any commits.

When I decided it was time to commit, I ran git init && git add . && git commit. At this point, I realized that there were some files I wanted to add to the .gitignore, so I cancelled the commit and composed my .gitignore file. At this point, I decided I'd re-add everything (now that I was ignoring stuff I didn't need) and ran git reset --hard.

YIKES

This wiped out all the code I had written.

No attempts to recover would work (git reflog was empty, git fsck couldn't find any objects, git rev-log had nothing). All I had what was left of the .git/objects directory.

After some reading and understanding of how the git object data is stored and written, it became clear that I could recover my data by parsing these files by hand.

The attached script does exactly that.

What a Monday.

#!/usr/local/bin/php
<?php
define('DIR', 'tree');
define('FILE', 'blob');
function main() {
$root_object = 'f3c8ec92f83e2a87c1d00a5681598126e23ab356';
restore_object($root_object, 'restored');
}
function restore_object($object, $dest) {
$type = get_object_type($object);
if ($type == FILE) {
echo 'restoring ' . $dest . "\n";
shell_exec('git cat-file -p ' . $object . ' > ' . $dest);
} else {
echo 'restoring ' . $dest . "\n";
mkdir($dest);
restore_directory($object, $dest);
}
}
function restore_directory($object, $base_dest) {
$files = parse_tree_object($object);
foreach ($files as $file) {
restore_object($file['object'], $base_dest . '/' . $file['name']);
}
}
function parse_tree_object($object) {
if (DIR != get_object_type($object)) {
throw new Exception("Trying to decode TREE object, but BLOB given: {$object}");
}
$contents = read_object($object);
$tree = [];
$lines = explode("\n", trim($contents));
foreach ($lines as $branch) {
$leaves = preg_split("#\s+#", $branch, 4);
$tmp = [
'mode' => $leaves[0],
'type' => $leaves[1],
'object' => $leaves[2],
'name' => $leaves[3],
];
array_map('trim', $tmp);
$tree[] = $tmp;
}
return $tree;
}
function read_object($object) {
$contents = trim(shell_exec('git cat-file -p ' . $object));
return $contents;
}
function get_object_type($object) {
$type = shell_exec("git cat-file -t {$object}");
return trim($type);
}
main();
@jimrubenstein
Copy link
Author

Hi Jim. It happen to me likewise. Still try to find a way restore my files.
I'm trying to run your script using a XAMP server. Question: where should I locate the git object files so the php script can read it>
Thanks for any help.

What a Monday!

Wow, it's been a while since I looked at this!

Sucks that this happened to you -- I hope you can recover!

I wrote the script to be run from the root of the git repo (eg, in the same folder that your .git folder resides).

I also wrote it to be run from the command line. There's probably not a huge issue with running it from the browser, though whichever user your webserver/php process is running as will have to have permission to read data from the .git folder and write to disk.

Hope that helps!

Good luck!

@john-auld
Copy link

Alternatively and for the same scenario.

$ git reflog show
93567ad HEAD@{0}: reset: moving to HEAD@{6}    
203e84e HEAD@{1}: reset: moving to HEAD@{1}    
9937a76 HEAD@{2}: reset: moving to HEAD@{2}
203e84e HEAD@{3}: checkout: moving from master to master
203e84e HEAD@{4}: reset: moving to HEAD~1
9937a76 HEAD@{5}: reset: moving to HEAD~1
d5bb59f HEAD@{6}: reset: moving to HEAD~1
9300f9d HEAD@{7}: commit: fix-bug

# said the commit to be recovered back is on 9300f9d (with commit message fix-bug)
$ git reset HEAD@{7}

Source: SO

Or, make a new branch from the "lost" commit.

git branch recovery  9300f9d
git checkout recovery

Panic over!

Copy link

ghost commented Jul 19, 2020

Thanks a lot! you made my day! :)

@annamalaiarunachalam
Copy link

this soln didnt work for me. I'd the same problem. I tried to push my local folders (250 folders, 950 files) to github after creating a empty repository. followed web examples, this is my first time with github. git terminal keot throwing error that few files are above 100MB, I have added, committed and I got this error when I pushed. I moved the three files to another folder. and it still threw. I realised I have to remove them from the commit. I used reset. added, committed, tried pushing, no matter how many times, i tried those 3 alrge files appeared though they didnt even exist in my folders. after 1 hr of various commands, all of a sudden I lost all my files. they were not in windows recycle. after some research found out it is in .git folder under objects but not in original format. How do I recover now? I dont have a backup and these files are not pushed to github yet. if I cannot recover, I will loose close to 950 files. please help

@annamalaiarunachalam
Copy link

I should not have installed git tool. a big blunder. can someone help me to recover my files please?

@john-auld
Copy link

I should not have installed git tool. a big blunder. can someone help me to recover my files please?

try my steps above: https://gist.github.com/jimrubenstein/9827718c508b2b7f5bad578bae421bfc#gistcomment-3095016

@annamalaiarunachalam
Copy link

I tried your steps before I posted the above message. it didnt work

@annamalaiarunachalam
Copy link

ran git reflog show
identified the first commit and git reset HEAD@{11}
I tried branch recovery and checkout recovery as well.
nothing worked

@john-auld
Copy link

Show the output of git reflog here

@annamalaiarunachalam
Copy link

$ git reflog show
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{0}: reset: moving to HEAD
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{1}: checkout: moving from master to recovery
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{2}: reset: moving to HEAD@{11}
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{3}: reset: moving to HEAD
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{4}: reset: moving to HEAD
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{5}: reset: moving to HEAD1
90c9091 HEAD@{6}: reset: moving to HEAD
90c9091 HEAD@{7}: reset: moving to HEAD
1
1d07234 HEAD@{8}: reset: moving to HEAD
1d07234 HEAD@{9}: commit: pyprojects all files
90c9091 HEAD@{10}: reset: moving to HEAD
90c9091 HEAD@{11}: commit: adding all folders and files from pyprojects
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{12}: Branch: renamed refs/heads/master to refs/heads/master
b66e468 (HEAD -> recovery, origin/master, master) HEAD@{14}: commit (initial): first commit

@annamalaiarunachalam
Copy link

image

@john-auld
Copy link

john-auld commented Sep 10, 2020

Using text in first column, try checking out commits until you find something recent and useful. See example below.

git checkout 1d07234

There are only a small number of commits, so it won't take long. You will get a message about having a detached head, but ignore that and keep trying each commit hash until you find your code.

@annamalaiarunachalam
Copy link

should I try from the bottom b663468?

@annamalaiarunachalam
Copy link

90c9091 - when I tried this
image

@annamalaiarunachalam
Copy link

I started with the bottom row first
image

@annamalaiarunachalam
Copy link

$ git checkout 1d07234
error: cannot stat 'Ex_Files_DSF_DataMining/Ex_Files_DSF_DataMining/Exercise Files/Ch7/07_04': Invalid argument
error: cannot stat 'Ex_Files_DSF_DataMining/Ex_Files_DSF_DataMining/Exercise Files/Ch7/07_04': Invalid argument

@annamalaiarunachalam
Copy link

then the rest are the same numbers... (first 6 rows have the same number >> b66e468)

@annamalaiarunachalam
Copy link

If you can provide me a solution to this, it would be helpful. thank you for your help. I will check again tomorrow.

@john-auld
Copy link

Try each unique commit hash in any order.

@annamalaiarunachalam
Copy link

there are just 3 unique hashes and I have tried all - have pasted my results above
the ones starting with 1d0 and 90c threw error - scrn shot shows that err
the 0nly hash that returned without error was b66 >> but my files were not recovered from .git/objects folder.
I see 180MB+ as size of .git/ and objects folder have several sub-folders with 2 digit folder names. why is this GIT so complicated. what kind of technology that moves my files into whatever folder without my permission.

@annamalaiarunachalam
Copy link

Please help. Its my mistake for not backing up the folder before running these git commands. all my program files will be lost if I dont recover.
please help

@john-auld
Copy link

john-auld commented Sep 11, 2020

Since you are having trouble with your local copy, try

git checkout -

Create a new repo on github "example-repo" and be in the directory containing your code.

git remote add origin https://github.com/your-user-name/example-repo.git
git branch -M master
git push -u origin master

Navigate to https://github.com/your-user-name/example-repo/commits/master

The <> icon on the right of the commit number contains a list of your files.

@annamalaiarunachalam
Copy link

first command worked
$ git checkout -
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

aa1e82f Revert "first commit"

If you want to keep it by creating a new branch, this may be a good time
to do so with:

git branch aa1e82f

Switched to branch 'recovery'

@annamalaiarunachalam
Copy link

second command threw fatal error
$ git remote add origin https://github.com/annamalaiarunachalam/recoverfiles-repo.git
fatal: remote origin already exists.

@annamalaiarunachalam
Copy link

I created a new repo called recoverfiles-repo as per your advisc. I just named it as recoverfiles-repo instead of example-repo

@annamalaiarunachalam
Copy link

I strongly believe the .git/objects/folder has all my files
image

@annamalaiarunachalam
Copy link

there are 258 folders in objects, with several sub-folders. the properties of 'object' folder shows 950 files.
git bash tool's commit operation has encrypted/compressing all my files and moved them to object folder. but whatever command I have tried so far doesnt seem to decrypt/decompress and restore it back into main workspace. besides, it doesnt push it to github either. all my files are stuck in the objects folder. its quite baffling, how this entire git tool is designed. can you help me with the decompress/decrypt of my files?

@3d-illusions
Copy link

how do you identify the root object, and how do you modify the php script accordingly please?

@jimrubenstein
Copy link
Author

@3d-illusions

You can use this bash script to output a list of the objects that are trees (directories) and the files they point to:

#!/bin/bash

for f in .git/objects/*/*; 
  do prefix=$(echo $f | cut -d / -f 3); 
  obj=$(echo $f | cut -d / -f 4); 
  type=$(git cat-file -t $prefix$obj); 

  if [[ $type == 'tree' ]]; 
  then 
    echo $prefix$obj; 
    git cat-file -p $prefix$obj; 
  fi; 
done;

this will output (potentially) a lot of stuff. You'll want to look through it to find the object that contains the files on the root of the directory.

Here's an example of one of mine:

1e9fd13ba91a602176ec2ed1b7616465d93c9abb
100644 blob 1671c9b9d94ae80b2d39c6b6a64d154b0ac6cb65	.editorconfig
100644 blob 44853cd59a7f52746b03239cbdac581d529e7c03	.env.example
100644 blob 967315dd3d16d50942fa7abd383dfb95ec685491	.gitattributes
100644 blob e45547285c62b5c739341255efca090152ddcdd6	.gitignore
100644 blob 877ea701dbf43c9a22cd57f70d876e69a3cb42c4	.styleci.yml
100644 blob 49c38be935606513ed8118401a1707a588dadebd	README.md
040000 tree be46bffe506f151d04b6a007875c426ec4a34a15	app
100755 blob 67a3329b183e042b14516122b5d470bc337a5a90	artisan
040000 tree fa579600b150dfe96277f923c509bc473517b32a	bootstrap
100644 blob 5e3dc6aaca598d6e750d8938b8ca3bfb93a63e33	composer.json
100644 blob f98cd7596615f0125e6d525203163cad4a8f5504	composer.lock
040000 tree 98261ed53ce2c6f11fb8554bfdaa803302c1020d	config
040000 tree caeda1d15bea1531e6c3dbb07444c37dc73b5aab	database
100644 blob 1447c30508375c3b5e106b0684ae22b7619cb57f	package-lock.json
100644 blob 304e26707c6a2794546a517a52d73c02e5a68105	package.json
100644 blob 4ae4d979d1ecc9bb45aabf0ff7071ce0e63bd4e2	phpunit.xml
040000 tree f82e1c58e9e62d5c241538ef419ec18f467d3a0b	public
040000 tree ed583017fccdfc4ce370762756a4c41481eced40	resources
040000 tree 131b325669c6284caa0111245124c5eaff8eab46	routes
100644 blob 5fb6379e71f275a1784e6638adc96420aaa5c5b2	server.php
040000 tree a9b549189653697bdcc2597e2a81e93fae10cea6	storage
100644 blob cbf5b8313a92e74c4b679b6fa519b38e0a5c2513	tailwind.config.js
040000 tree 4e9c4c557d2a1efe665c8b303db5c704da6294be	tests
100644 blob 71d071d37d860b3d993fbb246696dd91bac27a7e	webpack.mix.js

The root object identifier in this case is: 1e9fd13ba91a602176ec2ed1b7616465d93c9abb

Worth noting, the directories listed in ./git/objects are the prefix for each object within those directories. So, in the case of the root object, above, it's actually located in: .git/objects/1e/9fd13ba91a602176ec2ed1b7616465d93c9abb. This is worth making note of, if you try to run the git cat-file commands directly.

Hope this helps.

@3d-illusions
Copy link

Thanks Jim, I've sorted it now though. did an init git repo in empty folder, copy in objects folder from the repoistory where everything has been set to 0kb except the objects folder. Then git fsck, and then merged the sha1 of the dangling commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment