Skip to content

Instantly share code, notes, and snippets.

@dlmarion

dlmarion/updated_drop_behind_test.md Secret

Last active December 12, 2022 15:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04 to your computer and use it in GitHub Desktop.
Save dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04 to your computer and use it in GitHub Desktop.
Download ZIP
Updated setDropBehind test with pcstat
Raw
updated_drop_behind_test.md

I found the pcstat utility, which allows you to check how much of a file is in the page cache. To use this, we need to find the underlying HDFS files (blocks) associated with the file that we write into HDFS.

I updated Keith's test code, located here and then I installed pcstat. This required that go is installed. Once that was done, I did the following:

export GOPATH=${HOME}/go
export PATH=$GOPATH/bin:${PATH}
go install github.com/tobert/pcstat@latest

You can test the pcstat utility to cat'ing a file, then running pcstat <file>. I wrote the following script which runs the java test code to create the file in HDFS, finds the block locations for the file, then runs pcstat to report how much of the file is in the page cache.

#!/usr/bin/env bash

HADOOP_HOME="${HOME}/Software/hadoop-3.3.4"
PCSTAT="${HOME}/go/bin/pcstat"
NAMENODE="hdfs://127.0.0.1:9000"
DATA_DIR="/data/hadoop/dfs/data"
ACCUMULO_BIN="${HOME}/Software/accumulo-3.0.0-SNAPSHOT/bin"
HDFS_FILE="/test.bin"

SYNC_BLOCK_ON_OS="n"
SET_DROP_BEHIND="n"
CALL_HSYNC_ON_OS="n"

#
# Write file to HDFS
#
function write_file_to_hdfs {
  OUTPUT_FILE=$1
  BUFFER_SIZE="4096"
  head -c 1000000000 /dev/urandom > $OUTPUT_FILE
  cat $OUTPUT_FILE | ${ACCUMULO_BIN}/accumulo org.apache.accumulo.core.WriteDropBehindTest \
       ${NAMENODE} ${HDFS_FILE} ${BUFFER_SIZE} ${SYNC_BLOCK_ON_OS} ${SET_DROP_BEHIND} ${CALL_HSYNC_ON_OS}
}

#
# Run FSCK on the file, save block names to output file
#
function get_block_locations {
  OUTPUT_FILE=$1
  $HADOOP_HOME/bin/hdfs fsck ${HDFS_FILE} -files -blocks -replicaDetails | grep -v "Under" | grep "BP-" | awk '{ print $2 }' | awk -F':' '{ print $2 }' | awk -F'_' '{ print $1"_"$2 }' > $OUTPUT_FILE
}

function run_pcstat {
  BLOCK_LOCATIONS=$1
  declare filenames
  idx=0
  for block in $(cat $BLOCK_LOCATIONS); do
    file_name=$(find $DATA_DIR -name $block)
    filenames[$idx]="$file_name"
    ((idx=idx+1))
  done
  ${PCSTAT} -bname ${filenames[*]}
}

#main

$HADOOP_HOME/bin/hadoop fs -rm ${HDFS_FILE}

TMP_FILE=$(mktemp)
write_file_to_hdfs $TMP_FILE
rm -f $TMP_FILE

BLOCK_LOCATIONS=$(mktemp)
get_block_locations $BLOCK_LOCATIONS
run_pcstat $BLOCK_LOCATIONS
rm -f $BLOCK_LOCATIONS

Running this with different options generated some interesting results.

Test 1: All options disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742820 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742821 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742822 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742823 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742824 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742825 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742826 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742827 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742828 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742829 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742830 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742831 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742832 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742833 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742834 | 60475904       | 14765      | 14765     | 100.000 |
+----------------+----------------+------------+-----------+---------+

Test 2: SYNC_BLOCK on OutputStream enabled, setDropBehind and hsync disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742835 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742836 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742837 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742838 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742839 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742840 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742841 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742842 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742843 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742844 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742845 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742846 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742847 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742848 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742849 | 60475904       | 14765      | 14765     | 100.000 |
+----------------+----------------+------------+-----------+---------+

Test 3: hsync enabled, SYNC_BLOCK on OutputStream and setDropBehind disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742850 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742851 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742852 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742853 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742854 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742855 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742856 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742857 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742858 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742859 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742860 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742861 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742862 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742863 | 67108864       | 16384      | 16384     | 100.000 |
| blk_1073742864 | 60475904       | 14765      | 14765     | 100.000 |
+----------------+----------------+------------+-----------+---------+

Test 4: setDropBehind enabled, SYNC_BLOCK on OutputStream and hsync disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742865 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742866 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742867 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742868 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742869 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742870 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742871 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742872 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742873 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742874 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742875 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742876 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742877 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742878 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742879 | 60475904       | 14765      | 6664      | 045.134 |
+----------------+----------------+------------+-----------+---------+

Test 5: SYNC_BLOCK on OutputStream and setDropBehind enabled, hsync not enabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742880 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742881 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742882 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742883 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742884 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742885 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742886 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742887 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742888 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742889 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742890 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742891 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742892 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742893 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742894 | 60475904       | 14765      | 6664      | 045.134 |
+----------------+----------------+------------+-----------+---------+

Test 6: hsync and setDropBehind enabled, SYNC_BLOCK on OutputStream disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742895 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742896 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742897 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742898 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742899 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742900 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742901 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742902 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742903 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742904 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742905 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742906 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742907 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742908 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742909 | 60475904       | 14765      | 6664      | 045.134 |
+----------------+----------------+------------+-----------+---------+

Test 7: All options enabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073742910 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742911 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742912 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742913 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742914 | 67108864       | 16384      | 6228      | 038.013 |
| blk_1073742915 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742916 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742917 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742918 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742919 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742920 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742921 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742922 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742923 | 67108864       | 16384      | 8283      | 050.555 |
| blk_1073742924 | 60475904       | 14765      | 6664      | 045.134 |
+----------------+----------------+------------+-----------+---------+

HDFS-16864

I applied HDFS-16864 to the Hadoop 3.3.5 branch and I updated the test class to move the hsync call to right before the call to DFSOutpuStream.close(). I reran the tests above.

Test 8: setDropBehind enabled, SYNC_BLOCK on OutputStream and hsync disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073743010 | 67108864       | 16384      | 4271      | 026.068 |
| blk_1073743011 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743012 | 67108864       | 16384      | 6227      | 038.007 |
| blk_1073743013 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743014 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743015 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743016 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743017 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743018 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743019 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743020 | 67108864       | 16384      | 6227      | 038.007 |
| blk_1073743021 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743022 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743023 | 67108864       | 16384      | 4278      | 026.111 |
| blk_1073743024 | 60475904       | 14765      | 4615      | 031.256 |
+----------------+----------------+------------+-----------+---------+

Test 9: SYNC_BLOCK on OutputStream and setDropBehind enabled, hsync not enabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073743025 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743026 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743027 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743028 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743029 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743030 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743031 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743032 | 67108864       | 16384      | 186       | 001.135 |
| blk_1073743033 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743034 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743035 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743036 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743037 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743038 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743039 | 60475904       | 14765      | 175       | 001.185 |
+----------------+----------------+------------+-----------+---------+

Test 10: hsync and setDropBehind enabled, SYNC_BLOCK on OutputStream disabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073743070 | 67108864       | 16384      | 4271      | 026.068 |
| blk_1073743071 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743072 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743073 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743074 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743075 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743076 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743077 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743078 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743079 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743080 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743081 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743082 | 67108864       | 16384      | 6234      | 038.049 |
| blk_1073743083 | 67108864       | 16384      | 4271      | 026.068 |
| blk_1073743084 | 60475904       | 14765      | 175       | 001.185 |
+----------------+----------------+------------+-----------+---------+

Test 11: All options enabled

+----------------+----------------+------------+-----------+---------+
| Name           | Size (bytes)   | Pages      | Cached    | Percent |
|----------------+----------------+------------+-----------+---------|
| blk_1073743025 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743026 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743027 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743028 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743029 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743030 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743031 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743032 | 67108864       | 16384      | 186       | 001.135 |
| blk_1073743033 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743034 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743035 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743036 | 67108864       | 16384      | 175       | 001.068 |
| blk_1073743037 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743038 | 67108864       | 16384      | 168       | 001.025 |
| blk_1073743039 | 60475904       | 14765      | 175       | 001.185 |
+----------------+----------------+------------+-----------+---------+
@dlmarion
Copy link
Author

dlmarion commented Dec 6, 2022

The results for Test 4 (setDropBehind enabled, others disabled) are not different from other tests which use hsync and/or SYNC_BLOCK on the output stream. I think Test 4 shows that my host had flushed some of the dirty pages to disk on its own because the test did not use hsync or SYNC_BLOCK. Tests 5, 6, and 7 should have sync'd all of the dirty pages to disk, but the OS allowed some of the pages to remain in the page cache because setDropBehind just provides advice. I'm sure that the results may be different based on the memory size and concurrent workload of the host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment