Pridhi Outreachy

gistfile1.txt

# Pridhi Outreachy



Send this to Máirín Duffy
"Máirín Duffy" <duffy@fedoraproject.org>



Hi Máirín

I am Pridhi, from India. Currently an Outreachy candidate. I am writing this email to express interest in one of the projects that you have listed. I am particulary interested in the project **Improve how plugin instance logs are stored in the backend**. I have done a little bit of my part on the research side. This project requires storing logs in object storage. By object storage, right now Amazon s3 and it's equivalent Minio comes to my mind. Though, I would be needing some time to get a grasp on both of these technologies, but I am willing to make my best effort. What do you think about the technologies I have mentioned?

Also, please let me know how can I start my contribution? I have started listening to the talks that are mentioned on the project details. Also, I have started setting up ChRIS. I have some knowledge of python, docker and kubernetes. If there is something specific you would like me to take up as a part of my contribution; I would highly appreciate it as it will provide me a proper path to start my work and prove my dedication.

Thanks
Regards
Pridhi Arora

Errors I encountered.

docker network create -d overlay --attachable remote                          │▒
│Error response from daemon: This node is not a swarm manager. Use "docker swarm │▒
│init" or "docker swarm join" to connect this node to swarm and try again.

Fixed this by running docker swarm init

Error: couldn't start pfcon container on swarm │▒
│ This script will now terminate with exit code '2'.

Solution: stopped and delted the docker containers for pfcon

docker ps -a | grep -i pfcon

4f5bed994065   fnndsc/pfcon:latest                                                    "gunicorn -w 5 -b 0.…"   57 minutes ago   Up 57 minutes               5005/tcp                                                                  pfcon_stack_pfcon.1.o52yn6p2p6f59k88tb94mbl95
e8deb77561f8   fnndsc/pman:latest                                                     "gunicorn --bind 0.0…"   58 minutes ago   Up 57 minutes               5010/tcp                                                                  pfcon_stack_pman.1.ljln8191gqcd80c5n5aqup04m

docker stop 4f5bed994065
docker rm 4f5bed994065

Now restart ./make coomand

Errors while running chris UI

sirv-cli/bin.js:33:3)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:14) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/home/localuser/build'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! chris_ui@0.0.0 start: `sirv build --quiet --host --port 8080 --single --etag`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the chris_ui@0.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/localuser/.npm/_logs/2022-10-09T07_55_56_628Z-debug.log

Hi! I am Pridhi and a potential Outreachy intern. I am rooting for the project `Improve how plugin instance logs are stored in the backend.
I am trying to understand what possible object storage we can use. Right now, Amazon s3 and its open-source alternative Minio come to my mind. Please shed some light on this topic.

Also, I have tried to install ChRIS backend and UI for the same.
Following is the progress that I made while setting up both the services.

docker network create -d overlay --attachable remote                          │▒
│Error response from daemon: This node is not a swarm manager. Use "docker swarm │▒
│init" or "docker swarm join" to connect this node to swarm and try again.

I fixed the above by running docker swarm init

Next, I faced another problem, because of the previous errors, some containers such as pfcon were in limbo state. So, I had to delete them before moving ahead.

Error: couldn't start pfcon container on swarm │▒
│ This script will now terminate with exit code '2'.

I used
docker ps -a | grep -i pfcon

4f5bed994065   fnndsc/pfcon:latest                                                    "gunicorn -w 5 -b 0.…"   57 minutes ago   Up 57 minutes               5005/tcp                                                                  pfcon_stack_pfcon.1.o52yn6p2p6f59k88tb94mbl95
e8deb77561f8   fnndsc/pman:latest                                                     "gunicorn --bind 0.0…"   58 minutes ago   Up 57 minutes               5010/tcp                                                                  pfcon_stack_pman.1.ljln8191gqcd80c5n5aq

docker stop 4f5bed994065
docker rm 4f5bed994065

After the above steps. I was able to make the backend work.

The next step is to start the UI service. I have limited knowledge on UI, so I am not sure how to proceed. But I am facing the following error.

found 10 vulnerabilities (1 moderate, 9 high)
run npm audit fix to fix them, or npm audit for details

chris_ui@0.0.0 start /home/localuser
sirv build --quiet --host --port 8080 --single --etag

internal/fs/utils.js:314
    throw err;
    ^

Error: ENOENT: no such file or directory, scandir '/home/localuser/build'
    at readdirSync (fs.js:1043:3)
    at totalist (/home/localuser/node_modules/totalist/sync/index.js:6:12)
    at module.exports (/home/localuser/node_modules/sirv/build.js:149:3)
    at module.exports (/home/localuser/node_modules/sirv-cli/index.js:45:11)
    at o.parse (/home/localuser/node_modules/sade/lib/index.js:1:3402)
    at Object.<anonymous> (/home/localuser/node_modules/sirv-cli/bin.js:33:3)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:14) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/home/localuser/build'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! chris_ui@0.0.0 start: `sirv build --quiet --host --port 8080 --single --etag`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the chris_ui@0.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

I am running it via docker, I don't think I need to do docker run --rm -it -v $(pwd):/home/localuser -p 3000:3000 -u $(id -u):$(id -g) --name chris_ui fnndsc/chris_ui:dev
Looks like from Readme of chris UI, the docker build command was missing form the website.

Looks like we needed to run the build command.

It was missing from website of Chris. We updated the command to website and created a pull request.

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCbF0U0+pl9L64j3cTwFqBbQN1yqYv8qhDqKwaBDwQcpCnR3zFQRBLtzs403TfQ6q8RPVcwW8P7Ua1DX8QFLRBtk5mHxu0x9g5FR/k7KV5U8ryKrkJCUcI1E8Y+QkRtIFFB3fh3uysu1Gse9iiKXpJwCsQvm+YkAPrqAcVzoNu4xRxQTgg50o/u/7ScK7GWmkFILnCuy8hdCHddlQn0Oocl8G9WpbscSxY+TG3FnngISfZsaBT15nNmYdun8Xv5wqxDb3ylQJMLqRhqDdla2mCrbluHnxEoiGpWC6iV0zulqrkeJz6146uZRPJMCJACQt8gm1/v61p9rxh+3IIGiciVglcmrr01UOWsnW6mcY+uG74mUWVK5sDU+A2wZBuZ9fQJrbaxuUkIHFMD60OIadfAy3C4vLccSi5Tm61Flt+Wjo339THXE30T3ILbW2VywKjy4U7lMBR94DLi2X2cMCMiiQiFLfKnMOsm3S9qhDOakuDoHojOIFFBJ08gXveGrLhk48/dBvFDlDJZQ9dTBej9J5POvfjbft51SnquzDtvtc01Mneaf09tgGl5uOSNz1qQAz/s716YpbONsDiikVfdRs3/7D0eZrZksNc+ajJjIy6Z32sAHoDjLCqU8kcxYBf8G3kaneRTMRj7eeMXfj6MNcf7t+0EPFsnObQ+5vkpQw== pridhiarora17@gmail.com

Store ChRIS Logs in Object Storage

---
title: Store ChRIS Logs in Object Storage
authors:
  - "@PridhiArora"
reviewers:
  - 
  
creation-date: 2022-10-14
last-updated: 2022-10-14
status: implementable

Table of Contents

• Store ChRIS Logs in Object Storage
  • Table of Contents
  • Glossary
  • Summary
  • Motivation
    • Goals
  • Proposal
    • User Stories
      • Story 1 -Decreasing the latency in ChRIS backend.
    • Requirements
      • Functional Requirements
      • Non-Functional Requirements
    • Implementation Details/Notes/Constraints
      • Proposed Changes
    • Test Plan
      • Subtasks
    • Graduation Criteria
  • Implementation History

Glossary

• S3 - Amazon S3 is an object storage service that stores data as objects within buckets
• Minio - Minio is an open-source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality

Summary

ChRIS project currently stores log messages as strings in Postgres DB, and this causes Postgres DB to be populated with relatively large strings. The large size causes the backend to slow down.
One quick fix is to truncate the log string that is already implemented and use only a limited string. Right now, a substring until 3000 words is populated into the database, and the rest is truncated. But this devoids us of a lot of information. The decoration of log strings is done to make messages aesthetically pleasing. Removing unnecessary characters, such as emojis and Chinese characters, reduces the string size from 4000B to 2000B. Both the above fixes are still not enough as log size increases daily, and it is essential to find a better solution, and this is where object storage comes into the picture.

Motivation

The absence of a proper storage system for logs is slowing the backend, causing many users to be frustrated. The idea is to have an appropriate design so that ChRIS system is seamless.

Goals

1. To allow implementation of object storage for storage of logs.
2. To increase performance as well by using asynchronous data streams instead of buffering full responses.

Proposal

User Stories

Story - Decreasing the latency in ChRIS backend.

Stacy works at an IT department of an organization where ChRIS backend is used; she intends to use it regularly and needs logs to refer to. Storing logs in object storage would give her more readable logs.

Requirements

Functional Requirements

• FR1: Run Docker container with Minio
• FR2: Connect Docker container to ChRIS' backend.
• FR3: Storge logs inside of Minio

Non-Functional Requirements

• NFR1: Unit tests MUST exist for the support.

Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints

Proposed Changes

Currently, the summary gets saved in Postgres.

When object storage will be implemented, the following lines would need to be changed.

 self.c_plugin_inst.status = 'started'
self.c_plugin_inst.summary = self.get_job_status_summary()  # initial status
self.c_plugin_inst.raw = json_zip2str(d_resp)
self.c_plugin_inst.save()

Following is a code snippet taken from the official Minio Docs that can be used to implement the functionality.

from minio import Minio
from minio.error import S3Error


def main():
    # Create a client with the MinIO server playground, its access key
    # and secret key.
    client = Minio(
        "play.min.io",
        access_key="Q3AM3UQ867SPQQA43P2F",
        secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
    )

    # Make 'asiatrip' bucket if not exist.
    found = client.bucket_exists("asiatrip")
    if not found:
        client.make_bucket("asiatrip")
    else:
        print("Bucket 'asiatrip' already exists")

    # Upload '/home/user/Photos/asiaphotos.zip' as object name
    # 'asiaphotos-2015.zip' to bucket 'asiatrip'.
    client.fput_object(
        "asiatrip", "asiaphotos-2015.zip", "/home/user/Photos/asiaphotos.zip",
    )
    print(
        "'/home/user/Photos/asiaphotos.zip' is successfully uploaded as "
        "object 'asiaphotos-2015.zip' to bucket 'asiatrip'."
    )


if __name__ == "__main__":
    try:
        main()
    except S3Error as exc:
        print("error occurred.", exc)
Run File Uploader
$ python file_uploader.py
'/home/user/Photos/asiaphotos.zip' is successfully uploaded as object 'asiaphotos-2015.zip' to bucket 'asiatrip'.

Test Plan

• Unit tests.

Graduation Criteria

• Docker-supported Minio
• Integration of ChRIS with Minio
• Unit tests to test the functionality

pylint command
pylint **/*.py
autopep8 command
autopep8 --in-place --aggressive --aggressive **/*.py