Skip to content

Instantly share code, notes, and snippets.

@Soothly
Last active September 6, 2023 06:38
Show Gist options
  • Save Soothly/c7dba5e626fd4f1b6b172940ea6ce9a6 to your computer and use it in GitHub Desktop.
Save Soothly/c7dba5e626fd4f1b6b172940ea6ce9a6 to your computer and use it in GitHub Desktop.
[Not working] Attempting to upload a file using custom Ansible HttpApi plugin
ansible/module_utils/connection.py", line 149, in _exec_jsonrpc
raise ConnectionError(
fatal: [device_1]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"filename": "file.bin",
"state": "present"
}
},
"msg": "Connection error occurred: Failed to encode some variables as JSON for communication with ansible-connection. The original exception was: Object of type BufferedReader is not JSON serializable"
}
...ignoring
PLAY RECAP ******************************************************************************************************************************************************************************
device_1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
2023-09-06 08:09:26,576 - root - [request:get:71] - INFO - Starting GET:/api/data/swdb/file_coll call with args: {}
2023-09-06 08:09:26,576 - root - [request:_request_handler:40] - INFO - Sending GET request to: /api/data/swdb/file_coll with payload: {}, with headers: {}
2023-09-06 08:09:26,577 - root - [http_api:send_request:62] - INFO - Request: GET:/api/data/swdb/file_coll -> {}
2023-09-06 08:09:26,577 - root - [http_api:send_request:63] - DEBUG - Cookie: None
2023-09-06 08:09:26,577 - root - [http_api:send_request:66] - DEBUG - Headers: {}
2023-09-06 08:09:26,577 - root - [http_api:login:40] - INFO - Logging in started
2023-09-06 08:09:26,578 - root - [http_api:send_request:62] - INFO - Request: POST:/api/login -> {}
2023-09-06 08:09:26,578 - root - [http_api:send_request:63] - DEBUG - Cookie: None
2023-09-06 08:09:26,578 - root - [http_api:send_request:66] - DEBUG - Headers: {}
2023-09-06 08:09:26,932 - root - [http_api:send_request:78] - DEBUG - [POST:/api/login]Response: <http.client.HTTPResponse object at 0x7f5306b56730>
2023-09-06 08:09:26,932 - root - [http_api:send_request:79] - DEBUG - [POST:/api/login]Response code: 200
2023-09-06 08:09:26,932 - root - [http_api:send_request:80] - DEBUG - [POST:/api/login]Response content: <!DOCTYPE html><html>API session key created.</html>
2023-09-06 08:09:26,932 - root - [http_api:login:48] - INFO - Logging in ended
2023-09-06 08:09:27,272 - root - [http_api:send_request:78] - DEBUG - [GET:/api/data/swdb/file_coll]Response: <http.client.HTTPResponse object at 0x7f5306c10cd0>
2023-09-06 08:09:27,272 - root - [http_api:send_request:79] - DEBUG - [GET:/api/data/swdb/file_coll]Response code: 200
2023-09-06 08:09:27,272 - root - [http_api:send_request:80] - DEBUG - [GET:/api/data/swdb/file_coll]Response content: {"data": {...}}
2023-09-06 08:09:27,273 - root - [request:_request_handler:56] - INFO - [GET:/api/data/swdb/file_coll] Parsing response content
2023-09-06 08:09:27,273 - root - [request:_parse_content:62] - DEBUG - Parsing: {"data": {...}}
2023-09-06 08:09:27,273 - root - [request:_request_handler:58] - INFO - [GET:/api/data/swdb/file_coll] Parsing response finished
2023-09-06 08:09:27,273 - root - [request:get:71] - INFO - Starting GET:/api/data/swdb/file_coll?depth=1 call with args: {}
2023-09-06 08:09:27,274 - root - [request:_request_handler:40] - INFO - Sending GET request to: /api/data/swdb/file_coll?depth=1 with payload: {}, with headers: {}
2023-09-06 08:09:27,274 - root - [http_api:send_request:62] - INFO - Request: GET:/api/data/swdb/file_coll?depth=1 -> {}
2023-09-06 08:09:27,274 - root - [http_api:send_request:63] - DEBUG - Cookie: SESID=810vVgZ82WQzzjLDm9cwfb; Path=/
2023-09-06 08:09:27,274 - root - [http_api:send_request:66] - DEBUG - Headers: {'Cookie': 'SESID=810vVgZ82WQzzjLDm9cwfb; Path=/'}
2023-09-06 08:09:27,618 - root - [http_api:send_request:78] - DEBUG - [GET:/api/data/swdb/file_coll?depth=1]Response: <http.client.HTTPResponse object at 0x7f5306c10790>
2023-09-06 08:09:27,619 - root - [http_api:send_request:79] - DEBUG - [GET:/api/data/swdb/file_coll?depth=1]Response code: 200
2023-09-06 08:09:27,619 - root - [http_api:send_request:80] - DEBUG - [GET:/api/data/swdb/file_coll?depth=1]Response content: {"data": {...}}
2023-09-06 08:09:27,619 - root - [request:_request_handler:56] - INFO - [GET:/api/data/swdb/file_coll?depth=1] Parsing response content
2023-09-06 08:09:27,620 - root - [request:_parse_content:62] - DEBUG - Parsing: {"data": {...}}
2023-09-06 08:09:27,620 - root - [request:_request_handler:58] - INFO - [GET:/api/data/swdb/file_coll?depth=1] Parsing response finished
2023-09-06 08:09:27,621 - root - [request:put:75] - INFO - Starting PUT:/upload/file.bin call with args: {'data': <_io.BufferedReader name='/home/me343/ansible_collections/company/collection/software/file.bin'>, 'headers': {'Content-Type': 'application/octet-stream'}}
2023-09-06 08:09:27,621 - root - [request:_request_handler:40] - INFO - Sending PUT request to: /upload/file.bin with payload: <_io.BufferedReader name='/home/me343/ansible_collections/company/collection/software/file.bin'>, with headers: {'Content-Type': 'application/octet-stream'}
2023-09-06 08:09:27,737 - root - [http_api:logout:51] - INFO - Logging out started
2023-09-06 08:09:27,737 - root - [http_api:send_request:62] - INFO - Request: POST:/api/logout -> {}
2023-09-06 08:09:27,737 - root - [http_api:send_request:63] - DEBUG - Cookie: SESID=810vVgZ82WQzzjLDm9cwfb; Path=/
2023-09-06 08:09:27,737 - root - [http_api:send_request:66] - DEBUG - Headers: {'Cookie': 'SESID=810vVgZ82WQzzjLDm9cwfb; Path=/'}
2023-09-06 08:09:28,095 - root - [http_api:send_request:78] - DEBUG - [POST:/api/logout]Response: <http.client.HTTPResponse object at 0x7f5306d0e070>
2023-09-06 08:09:28,095 - root - [http_api:send_request:79] - DEBUG - [POST:/api/logout]Response code: 200
2023-09-06 08:09:28,095 - root - [http_api:send_request:80] - DEBUG - [POST:/api/logout]Response content:
2023-09-06 08:09:28,096 - root - [http_api:logout:55] - INFO - Logging out ended
from __future__ import absolute_import, division, print_function
__metaclass__ = type
DOCUMENTATION = """
---
author: ...
name: ...
short_description: ...
description:
- ...
version_added: "1.0.0"
"""
from ansible.module_utils.basic import to_text
from ansible.errors import AnsibleAuthenticationFailure
from ansible.module_utils.six.moves.urllib.error import HTTPError
from ansible_collections.ansible.netcommon.plugins.plugin_utils.httpapi_base import (
HttpApiBase,
)
from pathlib import Path
import logging
log_path = Path("http_api.log")
log_format = "%(asctime)s - %(name)s - [%(module)s:%(funcName)s:%(lineno)d] - %(levelname)s - %(message)s"
logging.basicConfig(
filename=log_path.resolve(), filemode="a", level=logging.DEBUG, format=log_format
)
class HttpApi(HttpApiBase):
def __init__(self, connection):
super().__init__(connection)
self.cookie = None
def login(self, username, password):
logging.info(f"Logging in started")
login_path = "/api/login"
# credentials sent as BasicAuth
self.send_request(method="POST", path=login_path)
try:
self.connection._auth = {"Cookie": self.cookie}
except KeyError:
raise AnsibleAuthenticationFailure(message="Failed to acquire login token.")
logging.info(f"Logging in ended")
def logout(self):
logging.info(f"Logging out started")
logout_path = "/api/logout"
self.send_request(method="POST", path=logout_path)
self.connection._auth = None
logging.info(f"Logging out ended")
def update_auth(self, response, response_text):
# cookie is only updated when POST /api/login is called
# not on every request
pass
def send_request(self, method, path, data={}, headers={}):
logging.info(f"Request: {method}:{path} -> {data}")
logging.debug(f"Cookie: {self.cookie}")
if self.cookie:
headers["Cookie"] = self.cookie
logging.debug(f"Headers: {headers}")
try:
# send() requires path & data to be Unicode
response, response_content = self.connection.send(
to_text(path), to_text(data), method=method, headers=headers
)
except HTTPError as exc:
return exc, exc.read()
if not self.cookie:
self.cookie = response.getheader("Set-Cookie")
content = to_text(response_content.getvalue())
logging.debug(f"[{method}:{path}]Response: {response}")
logging.debug(f"[{method}:{path}]Response code: {response.status}")
logging.debug(f"[{method}:{path}]Response content: {content}")
return response.status, content
def handle_httperror(self, exc):
# The method defined in ansible.plugins.httpapi
# We need to override it to avoid endless re-tries
# if HTTP authentication fails
if exc.code == 401:
return False
return exc
from pathlib import Path
from json import dumps
from time import perf_counter
from ansible_collections.company.collection.plugins.module_utils.request import (
Request,
)
from ansible_collections.company.collection.plugins.module_utils.exceptions import (
UnknownApiException,
SoftwareBinaryMissingException,
SoftwareUploadFailedException,
)
from ansible_collections.company.collection.plugins.module_utils.status_codes import (
HTTP_200_OK,
HTTP_201_CREATED,
HTTP_204_NO_CONTENT,
)
from ansible_collections.company.collection.plugins.module_utils.elements import (
get_json_data_from_path,
)
import logging
log_path = Path("http_api.log")
log_format = "%(asctime)s - %(name)s - [%(module)s:%(funcName)s:%(lineno)d] - %(levelname)s - %(message)s"
logging.basicConfig(
filename=log_path.resolve(), filemode="a", level=logging.DEBUG, format=log_format
)
def upload_software(connection: Request, sw_file_path: Path) -> None:
upload_url = f"/upload/{sw_file_path.name}"
if not sw_file_path.exists():
raise SoftwareBinaryMissingException(sw_file_path)
headers = {"Content-Type": "application/octet-stream"}
t_upload_start = perf_counter()
upload_response = connection.put(
url=upload_url, data=open(sw_file_path.resolve(), "rb"), headers=headers
)
t_upload_end = perf_counter()
response_time = t_upload_end - t_upload_start
logging.debug(f"SW upload time: {response_time}")
if upload_response.code not in (HTTP_201_CREATED, HTTP_204_NO_CONTENT):
raise UnknownApiException(
"PUT", upload_url, upload_response.code, upload_response
)
link_url = f"/api/data/swdb/upload"
data = dumps({"file_name": sw_file_path.name})
headers = {"Content-Type": "application/json"}
t_link_start = perf_counter()
link_response = connection.post(link_url, data=data, headers=headers)
t_link_stop = perf_counter()
link_time = t_link_stop - t_link_start
logging.debug(f"SW link time: {link_time}")
if link_response.code == HTTP_200_OK:
try:
result_code: int = get_json_data_from_path(
link_response.content, "data.swdb.result.code"
)
messages = get_json_data_from_path(
link_response.content, "data.swdb.result.message_coll"
)
except KeyError:
raise UnknownApiException(
"POST", link_url, link_response.code, link_response
)
if result_code < 0:
raise SoftwareUploadFailedException(sw_file_path, result_code, messages)
from __future__ import absolute_import, division, print_function
__metaclass__ = type
from typing import Dict, Union
import json
from dataclasses import dataclass
from ansible.module_utils.urls import CertificateError
from ansible.module_utils.connection import (
ConnectionError,
Connection,
)
from pathlib import Path
import logging
log_path = Path("http_api.log")
log_format = "%(asctime)s - %(name)s - [%(module)s:%(funcName)s:%(lineno)d] - %(levelname)s - %(message)s"
logging.basicConfig(
filename=log_path.resolve(), filemode="a", level=logging.DEBUG, format=log_format
)
@dataclass
class ApiResponse:
# this is returned from all API requests made by Request class
url: str
method: str
code: int
content: Union[str, Dict]
class Request(object):
# The modules instantiate this class and use its public methods to call the API
def __init__(self, module):
self.module = module
self.connection = Connection(self.module._socket_path)
def _request_handler(self, method, uri, **kwargs):
data = kwargs.get("data", {})
headers = kwargs.get("headers", {})
logging.info(
f"Sending {method} request to: {uri} with payload: {data}, with headers: {headers}"
)
try:
code, content = self.connection.send_request(method, uri, **kwargs)
except ConnectionError as e:
self.module.fail_json(msg=f"Connection error occurred: {e}")
except CertificateError as e:
self.module.fail_json(msg=f"Certificate error occurred: {e}")
except ValueError as e:
try:
self.module.fail_json(msg=f"Certificate not found: {e}")
except AttributeError:
pass
except Exception as e:
self.module.fail_json(msg=f"Unknown error occurred: {e}")
logging.info(f"[{method}:{uri}] Parsing response content")
parsed = self._parse_content(content)
logging.info(f"[{method}:{uri}] Parsing response finished")
return ApiResponse(code=code, content=parsed, url=uri, method=method)
def _parse_content(self, response_content):
logging.debug(f"Parsing: {response_content}")
try:
return json.loads(response_content) if response_content else {}
except Exception as e:
logging.error(f"Parsing to JSON failed: {response_content}")
logging.info(f"Falling back to text response")
return response_content
def get(self, url, **kwargs):
logging.info(f"Starting GET:{url} call with args: {kwargs}")
return self._request_handler("GET", url, **kwargs)
def put(self, url, **kwargs):
logging.info(f"Starting PUT:{url} call with args: {kwargs}")
return self._request_handler("PUT", url, **kwargs)
def patch(self, url, **kwargs):
logging.info(f"Starting PATCH:{url} call with args: {kwargs}")
return self._request_handler("PATCH", url, **kwargs)
def post(self, url, **kwargs):
logging.info(f"Starting POST:{url} call with args: {kwargs}")
return self._request_handler("POST", url, **kwargs)
def delete(self, url, **kwargs):
logging.info(f"Starting DELETE:{url} call with args: {kwargs}")
return self._request_handler("DELETE", url, **kwargs)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment