Skip to content

Instantly share code, notes, and snippets.

@edgarriba
Created February 22, 2016 15:10
Show Gist options
  • Save edgarriba/38b3fa347107f0fbaf2e to your computer and use it in GitHub Desktop.
Save edgarriba/38b3fa347107f0fbaf2e to your computer and use it in GitHub Desktop.
External features extraction
/* Reimplementation of `AddImagesToReconstructionBuilder`
https://github.com/sweeneychris/TheiaSfM/blob/master/applications/build_reconstruction.cc#L334
Using an external local features extractor. In this case AKAZE from OpenCV.
*/
void AddImagesToReconstructionBuilder(
ReconstructionBuilder* reconstruction_builder) {
std::vector<std::string> image_files;
CHECK(theia::GetFilepathsFromWildcard(FLAGS_images, &image_files));
// Create the feature matcher.
theia::FeatureMatcherOptions matching_options;
theia::VerifyTwoViewMatchesOptions verification_options;
SetMatchingOptions(&matching_options, &verification_options);
const theia::MatchingStrategy matching_strategy =
StringToMatchingStrategyType(FLAGS_matching_strategy);
std::unique_ptr<theia::FeatureMatcher<theia::L2> > matcher =
CreateFeatureMatcher(matching_strategy, matching_options);
cv::Mat img, opencv_descs;
std::vector<cv::KeyPoint> opencv_kpts;
cv::Ptr<cv::Feature2D> extractor = cv::AKAZE::create();
std::vector<theia::Keypoint> theia_kpts;
std::vector<Eigen::VectorXf> theia_descs;
for (size_t i = 0; i < image_files.size(); ++i) {
img = cv::imread(image_files[i], cv::IMREAD_COLOR);
extractor->detectAndCompute(img, cv::Mat(), opencv_kpts, opencv_descs);
// conversion between OpenCV/Theia
opencv_kpts_to_theia(opencv_kpts, &theia_kpts);
opencv_descriptors_to_theia(opencv_descs, &theia_descs);
std::string image_name;
theia::GetFilenameFromFilepath(image_files[i], true, &image_name);
matcher->AddImage(image_name, theia_kpts, theia_descs);
LOG(INFO) << "Added Image " << image_name << " with "
<< theia_kpts.size() << " Keypoints, "
<< theia_descs.size() << " Descriptors";
}
LOG(INFO) << "Perform MATCHING!";
std::vector<theia::ImagePairMatch> image_matches;
//matcher->MatchImages(&image_matches);
// Or, with geometric verification:
matcher->MatchImagesWithGeometricVerification(
verification_options, &image_matches);
// Optionally read the intrinsics from a calibration file.
std::vector<theia::CameraIntrinsicsPrior> camera_intrinsics_prior;
ReadIntrinsicsFromCalibrationFile(image_files, &camera_intrinsics_prior);
// Write the matches out.
LOG(INFO) << "Writing matches to file: " << FLAGS_output_matches_file;
CHECK(theia::WriteMatchesAndGeometry(FLAGS_output_matches_file,
image_files,
camera_intrinsics_prior, image_matches))
<< "Could not write the matches to " << FLAGS_output_matches_file;
// Add all the views.
for (int i = 0; i < image_files.size(); i++) {
reconstruction_builder->AddImage(image_files[i]);
}
// Add the matches.
for (const auto& match : image_matches) {
CHECK(reconstruction_builder->AddTwoViewMatch(match.image1,
match.image2,
match));
}
}
@edgarriba
Copy link
Author

ERROR LOG


I0222 16:06:37.516746 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0003.png
I0222 16:06:37.516894 29987 build_reconstruction.cpp:401] Added Image 0003.png with 3482 Keypoints, 3482 Descriptors
I0222 16:06:38.835705 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0008.png
I0222 16:06:38.835736 29987 build_reconstruction.cpp:401] Added Image 0008.png with 5637 Keypoints, 5637 Descriptors
I0222 16:06:40.055903 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0002.png
I0222 16:06:40.055943 29987 build_reconstruction.cpp:401] Added Image 0002.png with 3275 Keypoints, 3275 Descriptors
I0222 16:06:41.372514 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0009.png
I0222 16:06:41.372551 29987 build_reconstruction.cpp:401] Added Image 0009.png with 7236 Keypoints, 7236 Descriptors
^[[AI0222 16:06:42.606714 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0001.png
I0222 16:06:42.606752 29987 build_reconstruction.cpp:401] Added Image 0001.png with 3513 Keypoints, 3513 Descriptors
I0222 16:06:43.916947 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0007.png
I0222 16:06:43.916992 29987 build_reconstruction.cpp:401] Added Image 0007.png with 4190 Keypoints, 4190 Descriptors
I0222 16:06:45.163768 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0005.png
I0222 16:06:45.163811 29987 build_reconstruction.cpp:401] Added Image 0005.png with 3204 Keypoints, 3204 Descriptors
I0222 16:06:46.431634 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0006.png
I0222 16:06:46.431666 29987 build_reconstruction.cpp:401] Added Image 0006.png with 4660 Keypoints, 4660 Descriptors
I0222 16:06:47.701047 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0004.png
I0222 16:06:47.701086 29987 build_reconstruction.cpp:401] Added Image 0004.png with 3599 Keypoints, 3599 Descriptors
I0222 16:06:48.970335 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0000.png
I0222 16:06:48.970387 29987 build_reconstruction.cpp:401] Added Image 0000.png with 3273 Keypoints, 3273 Descriptors
I0222 16:06:50.425819 29987 cascade_hashing_feature_matcher.cc:72] Created the hashed descriptors for image: 0010.png
I0222 16:06:50.425850 29987 build_reconstruction.cpp:401] Added Image 0010.png with 8433 Keypoints, 8433 Descriptors
I0222 16:06:50.425858 29987 build_reconstruction.cpp:406] Perform MATCHING!
I0222 16:06:50.450451 30017 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.486928 30017 feature_matcher.h:448] Images 0002.png and 0004.png were matched with 146 verified matches and 90 homography matches out of 238 putative matches.
I0222 16:06:50.536309 30015 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.609216 30005 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.655319 30006 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.676648 30008 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.722546 30017 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.876929 30008 feature_matcher.h:448] Images 0003.png and 0006.png were matched with 76 verified matches and 23 homography matches out of 199 putative matches.
I0222 16:06:50.906653 30005 feature_matcher.h:448] Images 0003.png and 0002.png were matched with 308 verified matches and 167 homography matches out of 394 putative matches.
I0222 16:06:50.908435 30011 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.940737 30009 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.947199 30014 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.978847 30017 feature_matcher.h:448] Images 0002.png and 0000.png were matched with 85 verified matches and 33 homography matches out of 190 putative matches.
I0222 16:06:50.985059 30020 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:50.994726 30008 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.006784 30010 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.093911 30014 feature_matcher.h:448] Images 0002.png and 0001.png were matched with 310 verified matches and 140 homography matches out of 387 putative matches.
I0222 16:06:51.141115 30006 feature_matcher.h:448] Images 0003.png and 0001.png were matched with 132 verified matches and 82 homography matches out of 251 putative matches.
I0222 16:06:51.205926 30011 feature_matcher.h:448] Images 0007.png and 0005.png were matched with 129 verified matches and 54 homography matches out of 264 putative matches.
I0222 16:06:51.283207 30005 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.284070 30020 feature_matcher.h:448] Images 0009.png and 0010.png were matched with 88 verified matches and 37 homography matches out of 413 putative matches.
I0222 16:06:51.309803 30007 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.325556 30011 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.424381 30005 feature_matcher.h:448] Images 0006.png and 0004.png were matched with 152 verified matches and 30 homography matches out of 287 putative matches.
I0222 16:06:51.429401 30008 feature_matcher.h:448] Images 0003.png and 0004.png were matched with 334 verified matches and 153 homography matches out of 432 putative matches.
I0222 16:06:51.455874 30006 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.501765 30009 feature_matcher.h:448] Images 0001.png and 0004.png were matched with 68 verified matches and 40 homography matches out of 175 putative matches.
I0222 16:06:51.542135 30009 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.543455 30015 feature_matcher.h:448] Images 0005.png and 0006.png were matched with 424 verified matches and 166 homography matches out of 516 putative matches.
I0222 16:06:51.580240 30011 feature_matcher.h:448] Images 0007.png and 0006.png were matched with 366 verified matches and 121 homography matches out of 493 putative matches.
I0222 16:06:51.593257 30006 feature_matcher.h:448] Images 0003.png and 0005.png were matched with 126 verified matches and 65 homography matches out of 235 putative matches.
I0222 16:06:51.598353 30015 detect_structure.cc:107] Schur complement static structure <2,4,-1>.
I0222 16:06:51.633323 30009 feature_matcher.h:448] Images 0001.png and 0000.png were matched with 192 verified matches and 93 homography matches out of 317 putative matches.
I0222 16:06:51.691997 30010 feature_matcher.h:448] Images 0008.png and 0007.png were matched with 179 verified matches and 58 homography matches out of 344 putative matches.
I0222 16:06:51.699502 30015 feature_matcher.h:448] Images 0005.png and 0004.png were matched with 375 verified matches and 139 homography matches out of 463 putative matches.
I0222 16:06:51.831341 30007 feature_matcher.h:448] Images 0008.png and 0009.png were matched with 275 verified matches and 107 homography matches out of 458 putative matches.
I0222 16:06:51.831524 29987 feature_matcher.h:371] Matched 18 image pairs out of 55 possible image pairs.
E0222 16:06:51.831562 29987 read_calibration.cc:52] Cannot read the list file from
I0222 16:06:51.831574 29987 build_reconstruction.cpp:420] Writing matches to file: /home/eriba/datasets/fountain_dense/urd/matches
I0222 16:06:51.862475 29987 track_builder.cc:124] 2288 tracks were created. 61 features were dropped because they formed inconsistent tracks, and 0 features were dropped because they formed singleton tracks.
I0222 16:06:51.863436 29987 reconstruction_builder.cc:317] Attempting to reconstruct 11 images from 18 two view matches.
I0222 16:06:51.863489 29987 global_reconstruction_estimator.cc:170] Filtering the intial view graph.
I0222 16:06:51.863528 29987 global_reconstruction_estimator.cc:180] Calibrating any uncalibrated cameras.
I0222 16:06:51.863554 29987 global_reconstruction_estimator.cc:186] Estimating the global rotations of all cameras.
I0222 16:06:51.866338 29987 robust_rotation_estimator.cc:268] IRLS Converged in 1 iterations.
I0222 16:06:51.866389 29987 global_reconstruction_estimator.cc:197] Filtering any bad rotation estimations.
I0222 16:06:51.866425 29987 filter_view_pairs_from_orientation.cc:132] Removed 0 view pairs by rotation filtering.
I0222 16:06:51.866456 29987 global_reconstruction_estimator.cc:204] Optimizing the pairwise translation estimations.
I0222 16:06:51.871439 29987 global_reconstruction_estimator.cc:211] Filtering any bad relative translations.
I0222 16:06:51.871474 29987 global_reconstruction_estimator.cc:407] Filtering relative translations with 1DSfM filter.
I0222 16:06:51.872452 29987 filter_view_pairs_from_relative_translation.cc:294] Removed 0 view pairs by relative translation filtering.
I0222 16:06:51.872485 29987 global_reconstruction_estimator.cc:218] Estimating the positions of all cameras.
E0222 16:06:51.872550 29987 solver.cc:484] Terminating: Can't use SPARSE_NORMAL_CHOLESKY with SUITESPARSE because SuiteSparse was not enabled when Ceres was built.
I0222 16:06:51.872586 29987 nonlinear_position_estimator.cc:154]
Solver Summary (v 1.10.0-lapack-no_openmp)

                                 Original                  Reduced

Parameter blocks -1 -1
Parameters -1 -1
Residual blocks -1 -1
Residual -1 -1

Minimizer TRUST_REGION

Sparse linear algebra library SUITE_SPARSE
Trust region strategy LEVENBERG_MARQUARDT

                                    Given                     Used

Linear solver SPARSE_NORMAL_CHOLESKY SPARSE_NORMAL_CHOLESKY
Threads -1 -1
Linear solver threads -1 -1

Cost:
Initial -1.000000e+00

Minimizer iterations -2
Successful steps -1
Unsuccessful steps -1

Time (in seconds):
Preprocessor -1.0000

Residual evaluation -1.0000
Jacobian evaluation -1.0000
Linear solver -1.0000
Minimizer -1.0000

Postprocessor -1.0000
Total -1.0000

Termination: FAILURE (Can't use SPARSE_NORMAL_CHOLESKY with SUITESPARSE because SuiteSparse was not enabled when Ceres was built.)
W0222 16:06:51.872666 29987 global_reconstruction_estimator.cc:221] Position estimation failed!
F0222 16:06:51.872675 29987 build_reconstruction.cpp:495] Check failed: reconstruction_builder.BuildReconstruction(&reconstructions) Could not create a reconstruction.
*** Check failure stack trace: ***
@ 0x7f49a7661daa (unknown)
@ 0x7f49a7661ce4 (unknown)
@ 0x7f49a76616e6 (unknown)
@ 0x7f49a7664687 (unknown)
@ 0x476695 main
@ 0x7f49a6448ec5 (unknown)
@ 0x474b26 (unknown)
@ (nil) (unknown)
./run_theia.sh: line 15: 29987 Aborted

@avanindra
Copy link

Hi,

It seems like you have not built ceres with suite sparse. The problem is not with feature detector , it's the bundle adjustment which is failing. You could give linear solver as DENSE_SCHUR. Or build ceres with one of the sparse libraries such as Suitesparse , CXSparse or Eigen Sparse.

Correction:

Ignore the bundle adjustment part. Just build ceres with sparse library. It's the position estimator , where ceres is asking for suitesparse.

@sweeneychris
Copy link

@avanindra is correct. You should build Ceres with SuiteSparse (the failing module is for position estimation) then this problem will be fixed.

@edgarriba
Copy link
Author

Suitesparse was detected during Ceres installation. Just tried to reinstall everything again to be sure but got the same log. I'm using the Global solver which fails, however with Incremental got an approximated result of the real solution.

@shenw000
Copy link

edgarriba,

Would you mind to share your code for the following functions?
opencv_kpts_to_theia(opencv_kpts, &theia_kpts);
opencv_descriptors_to_theia(opencv_descs, &theia_descs);
I need to use these functions but don't want to reinvent the wheels.... Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment