Skip to content

Instantly share code, notes, and snippets.

@kezhuw

kezhuw/rotate.after.log

Created May 18, 2016 08:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kezhuw/44a1850543b66a8643c8aa127337b866 to your computer and use it in GitHub Desktop.
Save kezhuw/44a1850543b66a8643c8aa127337b866 to your computer and use it in GitHub Desktop.
Download ZIP
Rotate LevelDB compaction outputs at user key boundary
Raw
rotate.after.log
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 18 [I am key!(24) ==> I am key!(5)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 42 [I am key!(72) ==> I am key!(25)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 68 [I am key!(120) ==> I am key!(73)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 94 [I am key!(168) ==> I am key!(121)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 120 [I am key!(216) ==> I am key!(169)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 146 [I am key!(264) ==> I am key!(217)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 172 [I am key!(312) ==> I am key!(265)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 198 [I am key!(360) ==> I am key!(313)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 224 [I am key!(408) ==> I am key!(361)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
=========================================
level 1 compaction: smallest snapshot: 1
file 250 [I am key!(456) ==> I am key!(409)](compacting)
=========================================
level 0 compaction: smallest snapshot: 1
Raw
rotate.before.log
=========================================
level 0 compaction: smallest snapshot: 1
level 1: compacted file 18 ==> file 23, overlapping at user key: I am key![23 ==> 22]
level 1: compacted file 23 ==> file 28, overlapping at user key: I am key![21 ==> 20]
level 1: compacted file 28 ==> file 33, overlapping at user key: I am key![19 ==> 18]
level 1: compacted file 33 ==> file 36, overlapping at user key: I am key![17 ==> 16]
level 1: compacted file 36 ==> file 37, overlapping at user key: I am key![15 ==> 14]
level 1: compacted file 37 ==> file 38, overlapping at user key: I am key![13 ==> 12]
level 1: compacted file 38 ==> file 39, overlapping at user key: I am key![11 ==> 10]
level 1: compacted file 39 ==> file 40, overlapping at user key: I am key![9 ==> 8]
level 1: compacted file 40 ==> file 41, overlapping at user key: I am key![7 ==> 6]
=========================================
level 1 compaction: smallest snapshot: 1
file 18 [I am key!(24) ==> I am key!(23)](compacting)
file 23 [I am key!(22) ==> I am key!(21)](compacting)
file 28 [I am key!(20) ==> I am key!(19)](compacting)
file 33 [I am key!(18) ==> I am key!(17)](compacting)
file 36 [I am key!(16) ==> I am key!(15)](compacting)
file 37 [I am key!(14) ==> I am key!(13)](compacting)
file 38 [I am key!(12) ==> I am key!(11)](compacting)
file 39 [I am key!(10) ==> I am key!(9)](compacting)
file 40 [I am key!(8) ==> I am key!(7)](compacting)
file 41 [I am key!(6) ==> I am key!(5)](compacting)
level 2: compacted file 44 ==> file 49, overlapping at user key: I am key![23 ==> 22]
level 2: compacted file 49 ==> file 52, overlapping at user key: I am key![21 ==> 20]
level 2: compacted file 52 ==> file 53, overlapping at user key: I am key![19 ==> 18]
level 2: compacted file 53 ==> file 54, overlapping at user key: I am key![17 ==> 16]
level 2: compacted file 54 ==> file 55, overlapping at user key: I am key![15 ==> 14]
level 2: compacted file 55 ==> file 56, overlapping at user key: I am key![13 ==> 12]
level 2: compacted file 56 ==> file 57, overlapping at user key: I am key![11 ==> 10]
level 2: compacted file 57 ==> file 58, overlapping at user key: I am key![9 ==> 8]
level 2: compacted file 58 ==> file 59, overlapping at user key: I am key![7 ==> 6]
level 2: compacted file 59 ==> file 60, overlapping at user key: I am key![5 ==> 4]
level 2: compacted file 60 ==> file 61, overlapping at user key: I am key![3 ==> 2]
=========================================
level 0 compaction: smallest snapshot: 1
level 1: compacted file 62 ==> file 63, overlapping at user key: I am key![71 ==> 70]
level 1: compacted file 63 ==> file 64, overlapping at user key: I am key![69 ==> 68]
level 1: compacted file 64 ==> file 65, overlapping at user key: I am key![67 ==> 66]
level 1: compacted file 65 ==> file 66, overlapping at user key: I am key![65 ==> 64]
level 1: compacted file 66 ==> file 67, overlapping at user key: I am key![63 ==> 62]
level 1: compacted file 67 ==> file 68, overlapping at user key: I am key![61 ==> 60]
level 1: compacted file 68 ==> file 69, overlapping at user key: I am key![59 ==> 58]
level 1: compacted file 69 ==> file 70, overlapping at user key: I am key![57 ==> 56]
level 1: compacted file 70 ==> file 71, overlapping at user key: I am key![55 ==> 54]
level 1: compacted file 71 ==> file 72, overlapping at user key: I am key![53 ==> 52]
level 1: compacted file 72 ==> file 73, overlapping at user key: I am key![51 ==> 50]
level 1: compacted file 73 ==> file 74, overlapping at user key: I am key![49 ==> 48]
level 1: compacted file 74 ==> file 75, overlapping at user key: I am key![47 ==> 46]
level 1: compacted file 75 ==> file 76, overlapping at user key: I am key![45 ==> 44]
level 1: compacted file 76 ==> file 77, overlapping at user key: I am key![43 ==> 42]
level 1: compacted file 77 ==> file 78, overlapping at user key: I am key![41 ==> 40]
level 1: compacted file 78 ==> file 79, overlapping at user key: I am key![39 ==> 38]
level 1: compacted file 79 ==> file 80, overlapping at user key: I am key![37 ==> 36]
level 1: compacted file 80 ==> file 81, overlapping at user key: I am key![35 ==> 34]
level 1: compacted file 81 ==> file 82, overlapping at user key: I am key![33 ==> 32]
level 1: compacted file 82 ==> file 83, overlapping at user key: I am key![31 ==> 30]
level 1: compacted file 83 ==> file 84, overlapping at user key: I am key![29 ==> 28]
level 1: compacted file 84 ==> file 85, overlapping at user key: I am key![27 ==> 26]
=========================================
level 1 compaction: smallest snapshot: 1
file 62 [I am key!(72) ==> I am key!(71)](compacting)
file 63 [I am key!(70) ==> I am key!(69)]
file 64 [I am key!(68) ==> I am key!(67)]
file 65 [I am key!(66) ==> I am key!(65)]
file 66 [I am key!(64) ==> I am key!(63)]
file 67 [I am key!(62) ==> I am key!(61)]
file 68 [I am key!(60) ==> I am key!(59)]
file 69 [I am key!(58) ==> I am key!(57)]
file 70 [I am key!(56) ==> I am key!(55)]
file 71 [I am key!(54) ==> I am key!(53)]
file 72 [I am key!(52) ==> I am key!(51)]
file 73 [I am key!(50) ==> I am key!(49)]
file 74 [I am key!(48) ==> I am key!(47)]
file 75 [I am key!(46) ==> I am key!(45)]
file 76 [I am key!(44) ==> I am key!(43)]
file 77 [I am key!(42) ==> I am key!(41)]
file 78 [I am key!(40) ==> I am key!(39)]
file 79 [I am key!(38) ==> I am key!(37)]
file 80 [I am key!(36) ==> I am key!(35)]
file 81 [I am key!(34) ==> I am key!(33)]
file 82 [I am key!(32) ==> I am key!(31)]
file 83 [I am key!(30) ==> I am key!(29)]
file 84 [I am key!(28) ==> I am key!(27)]
file 85 [I am key!(26) ==> I am key!(25)]
level 2: compacted file 88 ==> file 93, overlapping at user key: I am key![71 ==> 24]
level 2: compacted file 93 ==> file 98, overlapping at user key: I am key![23 ==> 22]
level 2: compacted file 98 ==> file 103, overlapping at user key: I am key![21 ==> 20]
level 2: compacted file 103 ==> file 108, overlapping at user key: I am key![19 ==> 18]
level 2: compacted file 108 ==> file 113, overlapping at user key: I am key![17 ==> 16]
level 2: compacted file 113 ==> file 116, overlapping at user key: I am key![15 ==> 14]
level 2: compacted file 116 ==> file 117, overlapping at user key: I am key![13 ==> 12]
level 2: compacted file 117 ==> file 118, overlapping at user key: I am key![11 ==> 10]
level 2: compacted file 118 ==> file 119, overlapping at user key: I am key![9 ==> 8]
level 2: compacted file 119 ==> file 120, overlapping at user key: I am key![7 ==> 6]
level 2: compacted file 120 ==> file 121, overlapping at user key: I am key![5 ==> 4]
level 2: compacted file 121 ==> file 122, overlapping at user key: I am key![3 ==> 2]
=========================================
level 1 compaction: smallest snapshot: 1
file 63 [I am key!(70) ==> I am key!(69)](compacting)
file 64 [I am key!(68) ==> I am key!(67)]
file 65 [I am key!(66) ==> I am key!(65)]
file 66 [I am key!(64) ==> I am key!(63)]
file 67 [I am key!(62) ==> I am key!(61)]
file 68 [I am key!(60) ==> I am key!(59)]
file 69 [I am key!(58) ==> I am key!(57)]
file 70 [I am key!(56) ==> I am key!(55)]
file 71 [I am key!(54) ==> I am key!(53)]
file 72 [I am key!(52) ==> I am key!(51)]
file 73 [I am key!(50) ==> I am key!(49)]
file 74 [I am key!(48) ==> I am key!(47)]
file 75 [I am key!(46) ==> I am key!(45)]
file 76 [I am key!(44) ==> I am key!(43)]
file 77 [I am key!(42) ==> I am key!(41)]
file 78 [I am key!(40) ==> I am key!(39)]
file 79 [I am key!(38) ==> I am key!(37)]
file 80 [I am key!(36) ==> I am key!(35)]
file 81 [I am key!(34) ==> I am key!(33)]
file 82 [I am key!(32) ==> I am key!(31)]
file 83 [I am key!(30) ==> I am key!(29)]
file 84 [I am key!(28) ==> I am key!(27)]
file 85 [I am key!(26) ==> I am key!(25)]
level 2: compacted file 123 ==> file 124, overlapping at user key: I am key![71 ==> 70]
level 2: compacted file 124 ==> file 125, overlapping at user key: I am key![69 ==> 24]
level 2: compacted file 125 ==> file 126, overlapping at user key: I am key![23 ==> 22]
level 2: compacted file 126 ==> file 127, overlapping at user key: I am key![21 ==> 20]
level 2: compacted file 127 ==> file 128, overlapping at user key: I am key![19 ==> 18]
level 2: compacted file 128 ==> file 129, overlapping at user key: I am key![17 ==> 16]
level 2: compacted file 129 ==> file 130, overlapping at user key: I am key![15 ==> 14]
level 2: compacted file 130 ==> file 131, overlapping at user key: I am key![13 ==> 12]
level 2: compacted file 131 ==> file 132, overlapping at user key: I am key![11 ==> 10]
level 2: compacted file 132 ==> file 133, overlapping at user key: I am key![9 ==> 8]
level 2: compacted file 133 ==> file 134, overlapping at user key: I am key![7 ==> 6]
level 2: compacted file 134 ==> file 135, overlapping at user key: I am key![5 ==> 4]
level 2: compacted file 135 ==> file 136, overlapping at user key: I am key![3 ==> 2]
=========================================
level 1 compaction: smallest snapshot: 1
file 64 [I am key!(68) ==> I am key!(67)](compacting)
file 65 [I am key!(66) ==> I am key!(65)]
file 66 [I am key!(64) ==> I am key!(63)]
file 67 [I am key!(62) ==> I am key!(61)]
file 68 [I am key!(60) ==> I am key!(59)]
file 69 [I am key!(58) ==> I am key!(57)]
file 70 [I am key!(56) ==> I am key!(55)]
file 71 [I am key!(54) ==> I am key!(53)]
file 72 [I am key!(52) ==> I am key!(51)]
file 73 [I am key!(50) ==> I am key!(49)]
file 74 [I am key!(48) ==> I am key!(47)]
file 75 [I am key!(46) ==> I am key!(45)]
file 76 [I am key!(44) ==> I am key!(43)]
file 77 [I am key!(42) ==> I am key!(41)]
file 78 [I am key!(40) ==> I am key!(39)]
file 79 [I am key!(38) ==> I am key!(37)]
file 80 [I am key!(36) ==> I am key!(35)]
file 81 [I am key!(34) ==> I am key!(33)]
file 82 [I am key!(32) ==> I am key!(31)]
file 83 [I am key!(30) ==> I am key!(29)]
file 84 [I am key!(28) ==> I am key!(27)]
file 85 [I am key!(26) ==> I am key!(25)]
level 2: compacted file 137 ==> file 138, overlapping at user key: I am key![71 ==> 70]
level 2: compacted file 138 ==> file 139, overlapping at user key: I am key![69 ==> 68]
level 2: compacted file 139 ==> file 140, overlapping at user key: I am key![67 ==> 24]
level 2: compacted file 140 ==> file 141, overlapping at user key: I am key![23 ==> 22]
level 2: compacted file 141 ==> file 142, overlapping at user key: I am key![21 ==> 20]
level 2: compacted file 142 ==> file 143, overlapping at user key: I am key![19 ==> 18]
level 2: compacted file 143 ==> file 144, overlapping at user key: I am key![17 ==> 16]
level 2: compacted file 144 ==> file 145, overlapping at user key: I am key![15 ==> 14]
level 2: compacted file 145 ==> file 146, overlapping at user key: I am key![13 ==> 12]
level 2: compacted file 146 ==> file 147, overlapping at user key: I am key![11 ==> 10]
level 2: compacted file 147 ==> file 148, overlapping at user key: I am key![9 ==> 8]
level 2: compacted file 148 ==> file 149, overlapping at user key: I am key![7 ==> 6]
level 2: compacted file 149 ==> file 150, overlapping at user key: I am key![5 ==> 4]
level 2: compacted file 150 ==> file 151, overlapping at user key: I am key![3 ==> 2]
=========================================
level 1 compaction: smallest snapshot: 1
file 65 [I am key!(66) ==> I am key!(65)](compacting)
file 66 [I am key!(64) ==> I am key!(63)]
file 67 [I am key!(62) ==> I am key!(61)]
file 68 [I am key!(60) ==> I am key!(59)]
file 69 [I am key!(58) ==> I am key!(57)]
file 70 [I am key!(56) ==> I am key!(55)]
file 71 [I am key!(54) ==> I am key!(53)]
file 72 [I am key!(52) ==> I am key!(51)]
file 73 [I am key!(50) ==> I am key!(49)]
file 74 [I am key!(48) ==> I am key!(47)]
file 75 [I am key!(46) ==> I am key!(45)]
file 76 [I am key!(44) ==> I am key!(43)]
file 77 [I am key!(42) ==> I am key!(41)]
file 78 [I am key!(40) ==> I am key!(39)]
file 79 [I am key!(38) ==> I am key!(37)]
file 80 [I am key!(36) ==> I am key!(35)]
file 81 [I am key!(34) ==> I am key!(33)]
file 82 [I am key!(32) ==> I am key!(31)]
file 83 [I am key!(30) ==> I am key!(29)]
file 84 [I am key!(28) ==> I am key!(27)]
file 85 [I am key!(26) ==> I am key!(25)]
level 2: compacted file 152 ==> file 153, overlapping at user key: I am key![71 ==> 70]
level 2: compacted file 153 ==> file 154, overlapping at user key: I am key![69 ==> 68]
level 2: compacted file 154 ==> file 155, overlapping at user key: I am key![67 ==> 66]
level 2: compacted file 155 ==> file 156, overlapping at user key: I am key![65 ==> 24]
level 2: compacted file 156 ==> file 157, overlapping at user key: I am key![23 ==> 22]
level 2: compacted file 157 ==> file 158, overlapping at user key: I am key![21 ==> 20]
level 2: compacted file 158 ==> file 159, overlapping at user key: I am key![19 ==> 18]
level 2: compacted file 159 ==> file 160, overlapping at user key: I am key![17 ==> 16]
level 2: compacted file 160 ==> file 161, overlapping at user key: I am key![15 ==> 14]
level 2: compacted file 161 ==> file 162, overlapping at user key: I am key![13 ==> 12]
level 2: compacted file 162 ==> file 163, overlapping at user key: I am key![11 ==> 10]
level 2: compacted file 163 ==> file 164, overlapping at user key: I am key![9 ==> 8]
level 2: compacted file 164 ==> file 165, overlapping at user key: I am key![7 ==> 6]
level 2: compacted file 165 ==> file 166, overlapping at user key: I am key![5 ==> 4]
level 2: compacted file 166 ==> file 167, overlapping at user key: I am key![3 ==> 2]
=========================================
level 1 compaction: smallest snapshot: 1
file 66 [I am key!(64) ==> I am key!(63)](compacting)
file 67 [I am key!(62) ==> I am key!(61)]
file 68 [I am key!(60) ==> I am key!(59)]
file 69 [I am key!(58) ==> I am key!(57)]
file 70 [I am key!(56) ==> I am key!(55)]
file 71 [I am key!(54) ==> I am key!(53)]
file 72 [I am key!(52) ==> I am key!(51)]
file 73 [I am key!(50) ==> I am key!(49)]
file 74 [I am key!(48) ==> I am key!(47)]
file 75 [I am key!(46) ==> I am key!(45)]
file 76 [I am key!(44) ==> I am key!(43)]
file 77 [I am key!(42) ==> I am key!(41)]
file 78 [I am key!(40) ==> I am key!(39)]
file 79 [I am key!(38) ==> I am key!(37)]
file 80 [I am key!(36) ==> I am key!(35)]
file 81 [I am key!(34) ==> I am key!(33)]
file 82 [I am key!(32) ==> I am key!(31)]
file 83 [I am key!(30) ==> I am key!(29)]
file 84 [I am key!(28) ==> I am key!(27)]
file 85 [I am key!(26) ==> I am key!(25)]
level 2: compacted file 168 ==> file 169, overlapping at user key: I am key![71 ==> 70]
level 2: compacted file 169 ==> file 170, overlapping at user key: I am key![69 ==> 68]
level 2: compacted file 170 ==> file 171, overlapping at user key: I am key![67 ==> 66]
level 2: compacted file 171 ==> file 172, overlapping at user key: I am key![65 ==> 64]
level 2: compacted file 172 ==> file 173, overlapping at user key: I am key![63 ==> 24]
Raw
rotate.diff
diff --git a/Makefile b/Makefile
index 299da38..d58baf5 100644
--- a/Makefile
+++ b/Makefile
@@ -138,7 +138,10 @@ $(SHARED_OUTDIR)/$(SHARED_LIB3): $(SHARED_LIBOBJECTS)
endif # PLATFORM_SHARED_EXT
-all: $(SHARED_LIBS) $(SHARED_PROGRAMS) $(STATIC_OUTDIR)/libleveldb.a $(STATIC_OUTDIR)/libmemenv.a $(STATIC_PROGRAMS)
+all: rotate $(SHARED_LIBS) $(SHARED_PROGRAMS) $(STATIC_OUTDIR)/libleveldb.a $(STATIC_OUTDIR)/libmemenv.a $(STATIC_PROGRAMS)
+
+rotate: $(STATIC_LIBOBJECTS) rotate.cpp #$(STATIC_OUTDIR)/libleveldb.a
+ $(CXX) $(CXXFLAGS) $^ -o $@ $(LIBS)
check: $(STATIC_PROGRAMS)
for t in $(notdir $(TESTS)); do echo "***** Running $$t"; $(STATIC_OUTDIR)/$$t || exit 1; done
diff --git a/db/db_impl.cc b/db/db_impl.cc
index 60f4e66..7a4ef1d 100644
--- a/db/db_impl.cc
+++ b/db/db_impl.cc
@@ -72,6 +72,12 @@ struct DBImpl::CompactionState {
uint64_t total_bytes;
Output* current_output() { return &outputs[outputs.size()-1]; }
+ Output* previous_output() {
+ if (outputs.size() <= 1) {
+ return nullptr;
+ }
+ return &outputs[outputs.size()-2];
+ }
explicit CompactionState(Compaction* c)
: compaction(c),
@@ -856,6 +862,25 @@ Status DBImpl::FinishCompactionOutputFile(CompactionState* compact,
compact->compaction->level(),
(unsigned long long) current_entries,
(unsigned long long) current_bytes);
+ if (compact->previous_output() != nullptr) {
+ ParsedInternalKey largest, smallest;
+ if (!ParseInternalKey(compact->previous_output()->largest.Encode(), &largest)) {
+ abort();
+ }
+ if (!ParseInternalKey(compact->current_output()->smallest.Encode(), &smallest)) {
+ abort();
+ }
+ if (strcmp(largest.user_key.data(), smallest.user_key.data()) >= 0) {
+ fprintf(stderr, "level %d: compacted file %d ==> file %d, overlapping at user key: %s[%d ==> %d]\n",
+ compact->compaction->level()+1,
+ int(compact->previous_output()->number),
+ int(compact->current_output()->number),
+ largest.user_key.data(),
+ int(largest.sequence),
+ int(smallest.sequence)
+ );
+ }
+ }
}
}
return s;
@@ -902,6 +927,26 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
compact->smallest_snapshot = snapshots_.oldest()->number_;
}
+ int level = compact->compaction->level();
+ fprintf(stderr, "\n\n\n=========================================\n");
+ fprintf(stderr, "level %d compaction: smallest snapshot: %d\n",
+ level, int(compact->smallest_snapshot));
+ if (level != 0) {
+ const auto& files = compact->compaction->input_version_->files_[level];
+ ParsedInternalKey smallest, largest;
+ for (auto f : files) {
+ ParseInternalKey(f->smallest.Encode(), &smallest);
+ ParseInternalKey(f->largest.Encode(), &largest);
+ const char *suffix = "";
+ auto it = std::find(begin(compact->compaction->inputs_[0]), end(compact->compaction->inputs_[0]), f);
+ if (it != end(compact->compaction->inputs_[0])) {
+ suffix = "(compacting)";
+ }
+ fprintf(stderr, "file %d [%s(%d) ==> %s(%d)]%s\n", int(f->number), smallest.user_key.data(), int(smallest.sequence), largest.user_key.data(), int(largest.sequence), suffix);
+ }
+ fprintf(stderr, "\n");
+ }
+
// Release mutex while we're actually doing the compaction work
mutex_.Unlock();
@@ -925,7 +970,11 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
imm_micros += (env_->NowMicros() - imm_start);
}
+#define ROTATE_AT_USER_KEY
+#undef ROTATE_AT_USER_KEY
+
Slice key = input->key();
+#if !defined(ROTATE_AT_USER_KEY)
if (compact->compaction->ShouldStopBefore(key) &&
compact->builder != NULL) {
status = FinishCompactionOutputFile(compact, input);
@@ -933,9 +982,11 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
break;
}
}
+#endif
// Handle key/value, add to state, etc.
bool drop = false;
+ bool first_occurence = false;
if (!ParseInternalKey(key, &ikey)) {
// Do not hide error keys
current_user_key.clear();
@@ -947,6 +998,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
Slice(current_user_key)) != 0) {
// First occurrence of this user key
current_user_key.assign(ikey.user_key.data(), ikey.user_key.size());
+ first_occurence = last_sequence_for_key != kMaxSequenceNumber;
has_current_user_key = true;
last_sequence_for_key = kMaxSequenceNumber;
}
@@ -986,6 +1038,16 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
if (!status.ok()) {
break;
}
+#if defined(ROTATE_AT_USER_KEY)
+ } else if (first_occurence &&
+ (compact->compaction->ShouldStopBefore(key) ||
+ compact->builder->FileSize() >=
+ compact->compaction->MaxOutputFileSize())) {
+ status = FinishCompactionOutputFile(compact, input);
+ if (!status.ok()) {
+ break;
+ }
+#endif
}
if (compact->builder->NumEntries() == 0) {
compact->current_output()->smallest.DecodeFrom(key);
@@ -993,6 +1055,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
compact->current_output()->largest.DecodeFrom(key);
compact->builder->Add(key, input->value());
+#if !defined(ROTATE_AT_USER_KEY)
// Close output file if it is big enough
if (compact->builder->FileSize() >=
compact->compaction->MaxOutputFileSize()) {
@@ -1001,6 +1064,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
break;
}
}
+#endif
}
input->Next();
diff --git a/db/version_set.h b/db/version_set.h
index 1dec745..c3fcad9 100644
--- a/db/version_set.h
+++ b/db/version_set.h
@@ -113,7 +113,7 @@ class Version {
// Return a human readable string that describes this version's contents.
std::string DebugString() const;
- private:
+// private:
friend class Compaction;
friend class VersionSet;
@@ -362,7 +362,7 @@ class Compaction {
// is successful.
void ReleaseInputs();
- private:
+// private:
friend class Version;
friend class VersionSet;
diff --git a/rotate.cpp b/rotate.cpp
new file mode 100644
index 0000000..ea8f12e
--- /dev/null
+++ b/rotate.cpp
@@ -0,0 +1,35 @@
+#include "leveldb/db.h"
+
+class Slice {
+public:
+ Slice(const char *s) : _data(s), _size(strlen(s)+1) {}
+
+ operator leveldb::Slice() const {
+ return leveldb::Slice(_data, _size);
+ }
+
+private:
+ const char *_data;
+ size_t _size;
+};
+
+int main() {
+ leveldb::DB *db = nullptr;
+ leveldb::Options options;
+ options.create_if_missing = true;
+ leveldb::Status status = leveldb::DB::Open(options, "d2", &db);
+ if (!status.ok()) {
+ printf("open error: %s\n", status.ToString().c_str());
+ return -1;
+ }
+ Slice key("I am key!");
+ size_t n = 1024*1024+1024;
+ leveldb::Slice value(new char[n], n);
+ leveldb::WriteOptions woptions;
+ while (status.ok()) {
+ status = db->Put(woptions, key, value);
+ db->GetSnapshot();
+ }
+ printf("put error: %s\n", status.ToString().c_str());
+ return 0;
+}
Raw
rotate.filesize.log
total 531104
-rw-r--r-- 1 kezhuw kezhuw 312M May 18 16:08 000175.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000252.ldb
-rw-r--r-- 1 kezhuw kezhuw 144M May 18 16:08 000253.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000255.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000257.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000259.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000261.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000263.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000265.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000267.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000269.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000271.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000273.ldb
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000274.log
-rw-r--r-- 1 kezhuw kezhuw 4.0M May 18 16:08 000275.ldb
-rw-r--r-- 1 kezhuw kezhuw 9.0M May 18 16:08 000276.ldb
-rw-r--r-- 1 kezhuw kezhuw 16B May 18 16:07 CURRENT
-rw-r--r-- 1 kezhuw kezhuw 0B May 18 16:07 LOCK
-rw-r--r-- 1 kezhuw kezhuw 47K May 18 16:08 LOG
-rw-r--r-- 1 kezhuw kezhuw 9.9K May 18 16:08 MANIFEST-000002
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment