Skip to content

Instantly share code, notes, and snippets.

@marcelmaatkamp

marcelmaatkamp/gist:123e8793e07a72a382d8d0e8d66bbd8f

Last active March 26, 2024 10:13
Show Gist options
  • Save marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f to your computer and use it in GitHub Desktop.
Save marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f to your computer and use it in GitHub Desktop.
Download ZIP
Fix Sonatype Nexus OrientDB
Raw
gistfile1.txt
$ ls /opt/sonatype/sonatype-work/nexus3/db/
OSystem accesslog analytics audit component config model.properties security
$ docker-compose exec nexus java -jar ./lib/support/nexus-orient-console.jar
CONNECT PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component admin admin
REBUILD INDEX *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
DISCONNECT
@xtorrentgorjon
Copy link

Thanks for leaving this snippet, this helped us fix our database which was broken after storage got full.

Nexus error was showing:
javax.servlet.ServletException: com.orientechnologies.orient.core.exception.OPageIsBrokenException: Following files and pages are detected to be broken ['assetdownloadcount_node_id_repository_name_asset_name_date_type_date_idx.sbt' :10230;], storage is switched to 'read only' mode. Any modification operations are prohibited. To restore database and make it fully operational you may export and import database to and from JSON. DB name="component"

Instead of performing that database export/import stated in the error message, we did the steps mentioned in your fix, and after restarting the Nexus pod (running in Kubernetes), problem was solved.

@yobert
Copy link

yobert commented Jul 12, 2019

Another note about this: With nexus OSS 3.15.1, when it ran out of disk space, it created WAL files with zero file size. These caused orientdb to croak with null pointer exceptions on startup. I just had to remove the zero size files, and then the above commands worked perfectly.

@aaaia
Copy link

aaaia commented Jul 25, 2019

Thanks, worked perfectly. (nexus 3.12.1)

@marcelmaatkamp
Copy link
Author

Very nice to hear this worked and thanks for the positive feedback, that will help other people as well!

@chanjarster
Copy link

For people who cannot solve the problem, try this: https://stackoverflow.com/a/45986526

@bigmyx
Copy link

bigmyx commented Nov 15, 2019

I was trying to execute REBUILD INDEX * and got error:

Rebuilding index(es)...$ANSI{green {db=component}} Error during index rebuild
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
        DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
        at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2498)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
        at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:940)
        at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1040)
        at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:539)
        at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:500)
        at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:403)
        at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.rebuild(OIndexAbstractDelegate.java:167)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLRebuildIndex.execute(OCommandExecutorSQLRebuildIndex.java:91)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.sqlCommand(OConsoleDatabaseApp.java:3076)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.rebuildIndex(OConsoleDatabaseApp.java:1273)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
        at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
        at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
        at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
        DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671

Please advise.

@bansalrajat
Copy link

any update?

@Janjko
Copy link

Janjko commented Apr 30, 2020

I get this:

Error creating history file java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at com.orientechnologies.common.console.TTYConsoleReader.getHistoryFile(TTYConsoleReader.java:554) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:89) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) History file not found java.io.FileNotFoundException: .orientdb_history (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileReader.<init>(FileReader.java:72) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:92) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) Exception in thread "main" com.orientechnologies.common.exception.OSystemException: Cannot access to the input stream. Check permissions of running process at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:112) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

@flapouille
Copy link

any update on this @bigmyx ?

@rkrhlikar
Copy link

I get this:

Error creating history file java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at com.orientechnologies.common.console.TTYConsoleReader.getHistoryFile(TTYConsoleReader.java:554) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:89) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) History file not found java.io.FileNotFoundException: .orientdb_history (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileReader.<init>(FileReader.java:72) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:92) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) Exception in thread "main" com.orientechnologies.common.exception.OSystemException: Cannot access to the input stream. Check permissions of running process at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:112) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

In case anyone else encounters this issue, try running the $ docker-compose exec nexus java -jar ./lib/support/nexus-orient-console.jar command with the root user specified:

$ docker-compose exec -u root nexus java -jar ./lib/support/nexus-orient-console.jar

After running the commands from the original snippet, make sure to restore ownership of all database files back to the nexus user and nexus group, as the modified files will now belong to the root user!

@ojawel
Copy link

ojawel commented Sep 8, 2020 •
edited

If you are getting bigmyx's error:

Rebuilding index(es)...$ANSI{green {db=component}} Error during index rebuild
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671
at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2498)
at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:940)
at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1040)
at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:539)
at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:500)
at com.orientechnologies.orient.core.index.OIndexAbstra

You can truncate the record as described here: https://mariosk.wordpress.com/2019/09/07/how-a-nexus-repository-manager-corruption-led-to-a-mini-odyssey/
After i fixed all the dups to fix index: asset_bucket_component_name_idx i encountered a lot more appeared in the browse_node_repository_name_parent_path_name_idx index, after fixing a couple of dozes manually i dropped the class browse_node as mentioned here: https://issues.sonatype.org/browse/NEXUS-16419 this should only have affect on the webui and after restart you the indexes are repopulated.

@SilentTalentsSolutions
Copy link

Another note about this: With nexus OSS 3.15.1, when it ran out of disk space, it created WAL files with zero file size. These caused orientdb to croak with null pointer exceptions on startup. I just had to remove the zero size files, and then the above commands worked perfectly.

Many thanks

@microyahoo
Copy link 

~/go/src/etcd-io/etcd/tools/etcd-dump-db (main ✔) docker exec -u 0 -ti af9b0952e351 bash
                              
[root@af9b0952e351 /]# java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar 
                                                                      
OrientDB console v.2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x) https://www.orientdb.com
Type 'help' to display all the supported commands.    
orientdb> CONNECT PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component admin admin
                                   
Connecting to database [PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component] with user 'admin'...
2023-08-21 08:11:10:770 WARNI {db=component} Storage 'component' was not closed properly. Will try to recover from write ahead log... [OLocalPaginatedStorage]                                                     
2023-08-21 08:11:10:779 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRec
ord{lsn=LSN{segment=23, position=160518}} will be skipped during data restore [OLocalPaginatedStorage]
2023-08-21 08:11:10:779 WARNI {db=component} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=23, position=160525}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null,previousCheckpoint=LSN{segment=23, position=160478}} will be skipped during data restore [OLocalPaginatedStorage]
2023-08-21 08:11:10:780 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=23, position=160565}} will be skipped during data restore [OLocalPaginatedStorage]OK
orientdb {db=component}> REBUILD INDEX *        
                                   

Rebuilding index(es)...
Rebuilt index(es). Found 108 link(s) in 10.375000 sec(s).

Index(es) rebuilt successfully
orientdb {db=component}> REPAIR DATABASE --fix-graph

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> REPAIR DATABASE --fix-links

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> 
orientdb {db=component}> REPAIR DATABASE --fix-ridbags

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> REPAIR DATABASE --fix-bonsai 

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> disconnect  

Disconnecting from the database [component]...OK
orientdb>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment