Reconciling Chef Server database and Solr index inconsistencies

gistfile1.md

Sometimes a database object can get deleted but the Solr index entry will remain.

This can cause problems such as infinite Chef Search API request loops (for knife and I believe for Manage).

The following procedure identifies remaining index entries and provides a way to delete them.

# set the name of the organization you are interested in
export ORG_NAME=demo

# Get the organizaton's ID
export ORG_ID=`echo "select id from orgs where name = '$ORG_NAME' LIMIT 1;" | chef-server-ctl psql opscode_chef --options '-tA'`

# confirm that you got the organization's ID
echo $ORG_ID

# set the index name you are interested in from the following options:
# client, data_bag_item, environment, node or role
export INDEX_NAME=node

# get a list of entry IDs from the database for the $ORG_NAME organization
echo "select id from ${INDEX_NAME}s where org_id = '${ORG_ID}';" | chef-server-ctl psql opscode_chef --options '-tA' | sort > db_${INDEX_NAME}s.txt

# set the hostname and port for the Solr server
# if you are on the backend server that is running Solr then localhost:8983 should work
export SOLR_HOST=localhost:8983

# get a list of entry IDs from the Index for the $ORG_NAME organization
curl -s "http://${SOLR_HOST}/solr/select?q=X_CHEF_database_CHEF_X:chef_${ORG_ID}%20AND%20X_CHEF_type_CHEF_X:${INDEX_NAME}&rows=2000&wt=json" | python -m json.tool | awk -F'"' '/X_CHEF_id_CHEF_X/ {print $4}' | sort > index_${INDEX_NAME}s.txt

# compare the database results with the index results
diff -u db_${INDEX_NAME}s.txt index_${INDEX_NAME}s.txt

# if there are entries in the database that are not in the index then they need to be reindexed

# if there are entries in the index that are not in the database then they need to be deleted from the index
# delete an entry by its ID
export OBJECT_ID=<pick an object ID to delete>
curl -s "http://${SOLR_HOST}/solr/update?stream.body=<delete><query>X_CHEF_database_CHEF_X:chef_${ORG_ID}%20AND%20X_CHEF_type_CHEF_X:${INDEX_NAME}%20AND%20X_CHEF_id_CHEF_X:${OBJECT_ID}</query></delete>&commit=true"

Alternatively, you could verify ALL entries in an index by not filtering by the ORG_ID

# get a list of all entry IDs from the database

echo "select id from ${INDEX_NAME}s;" | chef-server-ctl psql opscode_chef --options '-tA' | sort > all_db_${INDEX_NAME}s.txt

# get a list of all entry IDs from the index

curl -s "http://${SOLR_HOST}/solr/select?q=X_CHEF_type_CHEF_X:${INDEX_NAME}&rows=2000&wt=json" | python -m json.tool | awk -F'"' '/X_CHEF_id_CHEF_X/ {print $4}' | sort > all_index_${INDEX_NAME}s.txt