Robot Framework Sample Code For Testing Log Follow

test_suites.robot

*** Settings ***

Library                 RobotUtilities
Library                 OperatingSystem
Library                 SSHLibrary
Library                 String
Library                 Collections
Library                 DateTime
Force Tags              data_flow
Suite Setup             run keyword    Load API Context

*** Test Cases ***

TC01 Test Component_1 Receives Logs From Various Log Sources
   [Documentation]    This test verifies that the Component_1 is able to receives logs from various external sources
   [Tags]        component_1  

   ${raw_log}   Use existing raw log for clusters
   ...    directory=Use_test_id    file_name=sampleraw_log.txt
   
   # Here test_1_logs.txt is the local logs you have decided to test on
   Create file     path=${DIR}/resources/remote_robot_resources/test_1_logs.txt    content=${raw_log}
   
   Copy remote resources test folder to your node    test_subdirectory=crules
   
   ${nc_injection_cmd}    Set Variable    cat /tmp/test_1_logs.txt | nc localhost 514
   
   Run The NC Command To Inject The Raw Log    ${COMPONENT_1_IP}    ${nc_injection_cmd}
   
   Poll rsyslog file for pattern     host=${COMPONENT_1_IP}    file_path=/opt/data/input/rsyslog/    
   ...    pattern=Testing Purpose String

   [Teardown]    No Operation
   

TC02 Test Component_2 Receives Logs From Component_1 and apply the alert detection model
   [Documentation]    This test verifies that the Component_2 is able to receives logs from component 1, and apply the alert
   ...    models on the logs received
   [Tags]        component_2  

   ${path}    Set Variable    /opt/data/input/rsyslog/
   Log To Console     Verify the component_2 is able to receives logs from component_1 through syslog forwarding   
   Poll rsyslog file for pattern     host=${COMPONENT_2_IP}    file_path=${path}    pattern=Testing Purpose String

   Log To Console    Checking if the logs has pattern to identify as Alert
   Poll rsyslog file for pattern     host=${COMPONENT_2_IP}    file_path=${path}    pattern=location_flag_set:true
 
   Log To Console    Applying the Alert Rule Model on a log
   ${add_alert_flag_cmd}    Set Variable    sed -i 's/location_flag_set/alert_type location_flag_set/' ${path}
   ${nc_cmd_succeed}    Run a generic command on a host and return return code only
   ...    host=${COMPONENT_2_IP}    cmd_string=${add_alert_flag_cmd}  
   Run Keyword If    ${nc_cmd_succeed} == False    fail    
   ...     msg=Not Able to Modify the file to add alert flag once alert condition meet

   [Teardown]    No Operation


Test Component_3 Receives the Alert Logs From Component_2 and create the incident
   [Documentation]    This test verifies that the Component_3 is able to receives alert logs from component 2, and able to
   ...    create the incident id ticket for the alert
   [Tags]        component_3

   ${path}    Set Variable    /opt/data/input/rsyslog/
   Log To Console     Verify the component_3 is able to receives alert log from component_2 through syslog forwarding
   Poll rsyslog file for pattern     host=${COMPONENT_3_IP}    file_path=${path}    pattern=alert_type

   Log To Console           Creating the Incident ID by adding a ID number in the logs
   ${path}                  /opt/data/input/rsyslog/Incident_Responder.log
   ${add_incident_cmd}      Set Variable    sed -i 's/alert_type/IR-ID-3243 alert_type/' + log_path ${path}
   ${nc_cmd_succeed}    Run a generic command on a host and return return code only
   ...    host=${COMPONENT_3_IP}    cmd_string=${add_incident_cmd}
   Run Keyword If    ${nc_cmd_succeed} == False    fail    msg=Not Able to Create Incident ID

   [Teardown]    No Operation


*** Keywords ***

Successfully connect through ssh
   [Arguments]        ${host}    ${second_host}=None
  
   run keyword if    ${IS_SAAS_ENV}    Successfully connect to cloud service    ${host}    ${second_host}
   ...               ELSE    Successfully connect to On-prem    ${host}


Run The NC Command To Inject The Raw Log
   [Documentation]    To Inject the Raw Log on Remote Host
   [Arguments]        ${host}    ${nc_injection_cmd}
  
   ${nc_cmd_succeed}    Run a generic command on a host and return return code    
   ...    host=${host}    cmd_string=${nc_injection_cmd}
   Run Keyword If    ${nc_cmd_succeed} == False    fail    msg=netcat injection operation failed
  

Run a generic command on a host and return return code
   [Arguments]        ${host}    ${cmd_string}    ${second_host}=None
   
   Successfully connect through ssh        ${host}    ${second_host}
   ${rc}    execute command     ${cmd_string}  timeout=${SSH_TIMEOUT}    return_stdout=False    return_rc=True
   ${command_succeed}    Run Keyword and Return Status    Should Be Equal As Integers     ${rc}  0   
   [Return]      ${command_succeed}
   [Teardown]    Close Connection
  

Copy remote resources test folder to node machine
   [Arguments]    ${test_subdirectory}    ${host}=${CLUSTER_IP}    ${second_host}=None
   
   Successfully connect through ssh    ${host}    ${second_host}
   Put directory      source=${DIR}/resources/remote_resources/${test_subdirectory}    destination=/tmp/    scp=ON
   [Teardown]    Close All Connection


Use existing raw log for clusters
   [Arguments]    ${directory}    ${file_name}
   
   Log To Console    Using previously recorded raw log at ${DIR}/resources/remote_resources/${directory}/${file_name}
   ${raw_log}    Operating System.Get File    path=${DIR}/resources/remote_resources/${directory}/${file_name}
   [Return]    ${raw_log}


Poll rsyslog file for a specific pattern   
   [Documentation]    This keyword will do a grep operation on the rsyslog on  nodes for verifying the patterns has been
   ...    populated inside the logs.
   [Arguments]    ${host}    ${file_path}    ${pattern}    ${max_tries}=5    ${polling_interval}=1 min


   ${latest_filename}    Set Variable    Internal.syslog.log
   ${grep_cmd_string}    set variable    cat ${file_path}/${latest_filename} | grep '${pattern}'   

   Log to console    Polling ${file_path}/${latest_filename} up to ${max_tries} times with an interval of ${polling_interval}
   
   # Override global variable here as, it requires to wait longer in the event of grep. If this keyword fails,
   # revert this global change in the [TearDown] section of this keyword
   Set Global Variable    ${SSH_TIMEOUT}    200s
  
   FOR     ${i}    IN RANGE    1    ${max_tries}
       ${alert_pattern_found}    Run a generic command on a host and return return code    
	   ...     host=${host}    cmd_string=${grep_cmd_string}
       exit for loop if    ${alert_pattern_found}        
       Log to Console    Attempt number ${i} failed, sleeping for ${polling_interval}
       Sleep    ${polling_interval}
   END
  
   Run Keyword If    ${alert_pattern_found?} == False    Fail    msg=Did not find pattern: ${pattern}
   Log to Console    Found pattern! ${pattern}
   [TearDown]    Run Keywords    Close All Connections    AND    Set Global Variable    ${SSH_TIMEOUT}    30s