Skip to content

Instantly share code, notes, and snippets.

@aeyakovenko

aeyakovenko/memory leak

Last active February 21, 2020 16:58
Show Gist options
  • Save aeyakovenko/976bbae3cf7e022c17e4bac64a4f021a to your computer and use it in GitHub Desktop.
Save aeyakovenko/976bbae3cf7e022c17e4bac64a4f021a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
memory leak
```
1599 temporary allocations of 4095 allocations in total (39.05%) from:
WriteBatch
at rocksdb/db/write_batch.cc:146
in /home/ubuntu/solana/bin/solana-validator
rocksdb::DB::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)
at rocksdb/db/db_impl_write.cc:1553
in /home/ubuntu/solana/bin/solana-validator
rocksdb::DBImpl::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)
at rocksdb/db/db_impl_write.cc:25
in /home/ubuntu/solana/bin/solana-validator
rocksdb_put_cf
at rocksdb/db/c.cc:787
in /home/ubuntu/solana/bin/solana-validator
rocksdb::db::_$LT$impl$u20$rocksdb..DB$GT$::put_cf_opt::hf5c89755940e3384
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.13.0/src/ffi_util.rs:39
in /home/ubuntu/solana/bin/solana-validator
rocksdb::db::_$LT$impl$u20$rocksdb..DB$GT$::put_cf::hdd5dd65c587a5c5d
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.13.0/src/db.rs:1374
solana_ledger::blockstore_db::Rocks::put_cf::hb1871eaec63743c7
at ledger/src/blockstore_db.rs:187
in /home/ubuntu/solana/bin/solana-validator
solana_ledger::blockstore_db::LedgerColumn$LT$C$GT$::put::h8ef7c741cb323da2
at ledger/src/blockstore_db.rs:709
solana_ledger::blockstore::Blockstore::set_dead_slot::h43fbb2148fcdb123
at ledger/src/blockstore.rs:1637
in /home/ubuntu/solana/bin/solana-validator
solana_core::replay_stage::ReplayStage::mark_dead_slot::h4c01396c565cea06
at core/src/replay_stage.rs:601
in /home/ubuntu/solana/bin/solana-validator
solana_core::replay_stage::ReplayStage::replay_blockstore_into_bank::h5df0d02422dec79b
at core/src/replay_stage.rs:593
solana_core::replay_stage::ReplayStage::replay_active_banks::hdbe568154ad5b656
at core/src/replay_stage.rs:762
solana_core::replay_stage::ReplayStage::new::_$u7b$$u7b$closure$u7d$$u7d$::hd8e3146ebcd20b3a
at core/src/replay_stage.rs:257
in /home/ubuntu/solana/bin/solana-validator
```
@aeyakovenko
Copy link
Author 

                                                                                                                                                                                                                                                                                                                                                                            
5523902 temporary allocations of 9533543 allocations in total (57.94%) from                                                                                                                                                                                                                                                                                                 
std::__cxx11::basic_string<>::reserve(unsigned long)                                                                                                                                                                                                                                                                                                                        
  in /usr/lib/x86_64-linux-gnu/libstdc++.so.6                                                                                                                                                                                                                                                                                                                               
5522303 temporary allocations of 8715791 allocations in total (63.36%) from:                                                                                                                                                                                                                                                                                                
    WriteBatch                                                                                                                                                                                                                                                                                                                                                              
      at rocksdb/db/write_batch.cc:146                                                                                                                                                                                                                                                                                                                                      
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::DB::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)                                                                                                                                                                                                                                              
      at rocksdb/db/db_impl_write.cc:1553                                                                                                                                                                                                                                                                                                                                   
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::DBImpl::Put(rocksdb::WriteOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&)                                                                                                                                                                                                                                          
      at rocksdb/db/db_impl_write.cc:25                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb_put_cf                                                                                                                                                                                                                                                                                                                                                          
      at rocksdb/db/c.cc:787                                                                                                                                                                                                                                                                                                                                                
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::db::_$LT$impl$u20$rocksdb..DB$GT$::put_cf_opt::hf5c89755940e3384                                                                                                                                                                                                                                                                                               
      at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.13.0/src/ffi_util.rs:39                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::db::_$LT$impl$u20$rocksdb..DB$GT$::put_cf::hdd5dd65c587a5c5d                                                                                                                                                                                                                                                                                                   
      at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/rocksdb-0.13.0/src/db.rs:1374                                                                                                                                                                                                                                                                         
    solana_ledger::blockstore_db::Rocks::put_cf::hb1871eaec63743c7                                                                                                                                                                                                                                                                                                          
      at ledger/src/blockstore_db.rs:187                                                                                                                                                                                                                                                                                                                                    
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    solana_ledger::blockstore_db::LedgerColumn$LT$C$GT$::put::hb2ec78265231d424                                                                                                                                                                                                                                                                                             
      at ledger/src/blockstore_db.rs:709                                                                                                                                                                                                                                                                                                                                    
    solana_ledger::blockstore::Blockstore::write_transaction_status::h0dd9aea7bbbed9de                                                                                                                                                                                                                                                                                      
      at ledger/src/blockstore.rs:1397                                                                                                                                                                                                                                                                                                                                      
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    solana_core::transaction_status_service::TransactionStatusService::write_transaction_status_batch::h31b00e18fe1899e6                                                                                                                                                                                                                                                    
      at core/src/transaction_status_service.rs:76                                                                                                                                                                                                                                                                                                                          
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    solana_core::transaction_status_service::TransactionStatusService::new::_$u7b$$u7b$closure$u7d$$u7d$::ha6cc23823433cd81                                                                                                                                                                                                                                                 
      at core/src/transaction_status_service.rs:33                                                                                                                                                                                                                                                                                                                          
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    std::sys_common::backtrace::__rust_begin_short_backtrace::h183ead9c0b79413d                                                                                                                                                                                                                                                                                             
      at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/sys_common/backtrace.rs:136                                                                                                                                                                                                                                                                             
    std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::he98db34f01a709a1                                                                                                                                                                                                                                                    
      at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/thread/mod.rs:469                                                                                                                                                                                                                                                                                       
      in /home/ubuntu/solana/bin/solana-validator

@aeyakovenko
Copy link
Author 

241865 temporary allocations of 251042 allocations in total (96.34%) from:                                                                                                                                                                                                                                                                                                  
    std::__cxx11::basic_string<>::_M_append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                     
      in /usr/lib/x86_64-linux-gnu/libstdc++.so.6                                                                                                                                                                                                                                                                                                                           
    std::__cxx11::basic_string<>::append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                        
      at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/basic_string.h:1244                                                                                                                                                                                                                                                                  
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::AppendInternalKey(std::__cxx11::basic_string<>*, rocksdb::ParsedInternalKey const&)                                                                                                                                                                                                                                                                            
      at rocksdb/db/dbformat.cc:78                                                                                                                                                                                                                                                                                                                                          
    rocksdb::InternalKey::SetMinPossibleForUserKey(rocksdb::Slice const&)                                                                                                                                                                                                                                                                                                   
      at rocksdb/db/dbformat.h:218                                                                                                                                                                                                                                                                                                                                          
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::SomeFileOverlapsRange(rocksdb::InternalKeyComparator const&, bool, rocksdb::LevelFilesBrief const&, rocksdb::Slice const*, rocksdb::Slice const*)                                                                                                                                                                                                              
      at rocksdb/db/version_set.cc:838                                                                                                                                                                                                                                                                                                                                      
    rocksdb::VersionStorageInfo::OverlapInLevel(int, rocksdb::Slice const*, rocksdb::Slice const*)                                                                                                                                                                                                                                                                          
      at rocksdb/db/version_set.cc:2683                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionStorageInfo::RangeMightExistAfterSortedRun(rocksdb::Slice const&, rocksdb::Slice const&, int, int)                                                                                                                                                                                                                                                      
      at rocksdb/db/version_set.cc:3200                                                                                                                                                                                                                                                                                                                                     
    rocksdb::VersionStorageInfo::GenerateBottommostFiles()                                                                                                                                                                                                                                                                                                                  
      at rocksdb/db/version_set.cc:2623                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::ProcessManifestWrites(std::deque<>&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                                                                                                  
      at rocksdb/db/version_set.cc:3567                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::LogAndApply(rocksdb::autovector<> const&, rocksdb::autovector<> const&, rocksdb::autovector<> const&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                                 
      at rocksdb/db/version_set.cc:3819                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::autovector<> const&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                               
      at rocksdb/db/version_set.h:795                                                                                                                                                                                                                                                                                                                                       
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::MemTableList::TryInstallMemtableFlushResults(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::autovector<> const&, rocksdb::LogsWithPrepTracker*, rocksdb::VersionSet*, rocksdb::InstrumentedMutex*, unsigned long, rocksdb::autovector<>*, rocksdb::Directory*, rocksdb::LogBuffer*)                                                    
      at rocksdb/db/memtable_list.cc:404                                                                                                                                                                                                                                                                                                                                    
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*)                                                                                                                                                                                                                                                                                           
      at rocksdb/db/flush_job.cc:243

@aeyakovenko
Copy link
Author 

242962 temporary allocations of 251723 allocations in total (96.52%) from:                                                                                                                                                                                                                                                                                                  
    std::__cxx11::basic_string<>::_M_append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                     
      in /usr/lib/x86_64-linux-gnu/libstdc++.so.6                                                                                                                                                                                                                                                                                                                           
    std::__cxx11::basic_string<>::append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                        
      at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/basic_string.h:1244                                                                                                                                                                                                                                                                  
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::AppendInternalKey(std::__cxx11::basic_string<>*, rocksdb::ParsedInternalKey const&)                                                                                                                                                                                                                                                                            
      at rocksdb/db/dbformat.cc:78                                                                                                                                                                                                                                                                                                                                          
    rocksdb::InternalKey::SetMinPossibleForUserKey(rocksdb::Slice const&)                                                                                                                                                                                                                                                                                                   
      at rocksdb/db/dbformat.h:218                                                                                                                                                                                                                                                                                                                                          
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::SomeFileOverlapsRange(rocksdb::InternalKeyComparator const&, bool, rocksdb::LevelFilesBrief const&, rocksdb::Slice const*, rocksdb::Slice const*)                                                                                                                                                                                                              
      at rocksdb/db/version_set.cc:838                                                                                                                                                                                                                                                                                                                                      
    rocksdb::VersionStorageInfo::OverlapInLevel(int, rocksdb::Slice const*, rocksdb::Slice const*)                                                                                                                                                                                                                                                                          
      at rocksdb/db/version_set.cc:2683                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionStorageInfo::RangeMightExistAfterSortedRun(rocksdb::Slice const&, rocksdb::Slice const&, int, int)                                                                                                                                                                                                                                                      
      at rocksdb/db/version_set.cc:3200                                                                                                                                                                                                                                                                                                                                     
    rocksdb::VersionStorageInfo::GenerateBottommostFiles()                                                                                                                                                                                                                                                                                                                  
      at rocksdb/db/version_set.cc:2623                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::ProcessManifestWrites(std::deque<>&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                                                                                                  
      at rocksdb/db/version_set.cc:3567                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::LogAndApply(rocksdb::autovector<> const&, rocksdb::autovector<> const&, rocksdb::autovector<> const&, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                                 
      at rocksdb/db/version_set.cc:3819                                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, rocksdb::InstrumentedMutex*, rocksdb::Directory*, bool, rocksdb::ColumnFamilyOptions const*)                                                                                                                                                      
      at rocksdb/db/version_set.h:778                                                                                                                                                                                                                                                                                                                                       
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::CompactionJob::InstallCompactionResults(rocksdb::MutableCFOptions const&)                                                                                                                                                                                                                                                                                      
      at rocksdb/db/compaction_job.cc:1411                                                                                                                                                                                                                                                                                                                                  
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::CompactionJob::Install(rocksdb::MutableCFOptions const&)                                                                                                                                                                                                                                                                                                       
      at rocksdb/db/compaction_job.cc:728

@aeyakovenko
Copy link
Author 

4010551 temporary allocations of 320383398 allocations in total (1.25%) from:                                                                                                                                                                                                                                                                                               
    std::__cxx11::basic_string<>::_M_append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                     
      in /usr/lib/x86_64-linux-gnu/libstdc++.so.6                                                                                                                                                                                                                                                                                                                           
    std::__cxx11::basic_string<>::append(char const*, unsigned long)                                                                                                                                                                                                                                                                                                        
      at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/bits/basic_string.h:1244                                                                                                                                                                                                                                                                  
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::PutFixed64(std::__cxx11::basic_string<>*, unsigned long)                                                                                                                                                                                                                                                                                                       
      at rocksdb/util/coding.h:225                                                                                                                                                                                                                                                                                                                                          
    rocksdb::InternalKeyComparator::FindShortestSeparator(std::__cxx11::basic_string<>*, rocksdb::Slice const&) const                                                                                                                                                                                                                                                       
      at rocksdb/db/dbformat.cc:143                                                                                                                                                                                                                                                                                                                                         
    rocksdb::ShortenedIndexBuilder::AddIndexEntry(std::__cxx11::basic_string<>*, rocksdb::Slice const*, rocksdb::BlockHandle const&)                                                                                                                                                                                                                                        
      at rocksdb/table/index_builder.h:145                                                                                                                                                                                                                                                                                                                                  
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)                                                                                                                                                                                                                                                                                      
      at rocksdb/table/block_based_table_builder.cc:528                                                                                                                                                                                                                                                                                                                     
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)                                                                                                                                                                                                                                                                          
      at rocksdb/db/compaction_job.cc:938                                                                                                                                                                                                                                                                                                                                   
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::CompactionJob::Run()                                                                                                                                                                                                                                                                                                                                           
      at rocksdb/db/compaction_job.cc:600                                                                                                                                                                                                                                                                                                                                   
      in /home/ubuntu/solana/bin/solana-validator                                                                                                                                                                                                                                                                                                                           
    rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)                                                                                                                                                                                                                  
      at rocksdb/db/db_impl_compaction_flush.cc:2684

@aeyakovenko
Copy link
Author

@sakridge

828410 temporary allocations of 843414 allocations in total (98.22%) from:                                                                                                                                                                                                                                                                                                  
    std::sys::unix::os::getenv::h15ecf502d02814f6                                                                                                                                                                                                                                                                                                                           
      at src/libstd/sys/unix/os.rs:470                                                                                                                                                                                                                                                                                                                                      
      in /home/ubuntu/solana/bin/deps/libsolana_vote_program.so                                                                                                                                                                                                                                                                                                             
    std::env::_var_os::h1568f8b19da9efde                                                                                                                                                                                                                                                                                                                                    
      at src/libstd/env.rs:250                                                                                                                                                                                                                                                                                                                                              
    std::env::var_os::h57e91f49931f748f                                                                                                                                                                                                                                                                                                                                     
      at src/libstd/env.rs:246                                                                                                                                                                                                                                                                                                                                              
      in /home/ubuntu/solana/bin/deps/libsolana_vote_program.so                                                                                                                                                                                                                                                                                                             
    std::env::_var::h7295174216fa55ca                                                                                                                                                                                                                                                                                                                                       
      at src/libstd/env.rs:216

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment