steniowagner

Context

I was explaining to a non-technical stakeholder, in the context of Artificial Inteligence, how the process of divide a large text into smaller texts (chunks) works. We used a library called LangChain to make this process, so the explanation was about the internals of how LangChain create the chunks from the input text.

This investigation started because we wanted to understand (and, if possible, to control) how this library was breaking down the text to make the chunks.

PS: this conversation happened in a slack-channel on october 26th, 2023, and the content here is simply a copy-and-paste of the conversation.

PS²: Pinecone, mentioned in the end of the conversation, is a vector-database. We use it to store the text-chunks, so we could query it by giving some piece of text, and it would return similar chunks that it has stored.

---- the conversation starts here ----

steniowagner

// MockedError Provider

import React from 'react';
import { ApolloProvider } from '@apollo/react-hooks';
import { InMemoryCache } from 'apollo-cache-inmemory';
import { ApolloLink, Observable } from 'apollo-link';
import { ApolloClient } from 'apollo-client';
import { GraphQLError } from 'graphql';

type Props = {

steniowagner

import React, { useState } from "react";

import LoginComponent from "./LoginComponent";

const usersTest = Array(8)
  .fill({ name: "", cns: "" })
  .map((_, index) => ({
    name: `User ${index}`,
    cns: `123.456.${index}`,
  }));

steniowagner

import React from "react";
import { TouchableOpacity, Text, StyleSheet, View } from "react-native";
import TVEventHandler from "TVEventHandler";
import Video from "react-native-video";

const ITEMS = [
  {
    color: "#f0f",
    text: "Rosa"
  },

steniowagner

import { PermissionsAndroid } from 'react-native';

export const PERMISSIONS_TYPES = {
  WRITE_EXTERNAL_STORAGE: 'WRITE_EXTERNAL_STORAGE',
  READ_EXTERNAL_STORAGE: 'READ_EXTERNAL_STORAGE',
};

const PERMISSIONS = {
  [PERMISSIONS_TYPES.WRITE_EXTERNAL_STORAGE]: {
    type: 'WRITE_EXTERNAL_STORAGE',

steniowagner

const parseDegreeToRadians = (degrees) => degrees * (Math.PI / 180);

const calculateDistanceBetweenTwoCoordinates = (firstCoordinate, secondCoordinate) => {
  const EARTH_RADIUS = 6371;

  const deltaLatitude = parseDegreeToRadians(secondCoordinate.latitude - firstCoordinate.latitude);
  const deltaLongitude = parseDegreeToRadians(secondCoordinate.longitude - firstCoordinate.longitude);

  const haversinesInnerTerm = Math.sin(deltaLatitude / 2) * Math.sin(deltaLatitude / 2) +
    Math.cos(parseDegreeToRadians(firstCoordinate.latitude)) * Math.cos(parseDegreeToRadians(secondCoordinate.longitude)) * 

steniowagner

// @flow

import { Dimensions, Platform } from 'react-native';

const screenWidth = Dimensions.get('window').width;
const screenHeight = Dimensions.get('window').height;

const IPHONEX_WIDTH = 375;
const IPHONEX_HEIGHT = 812;

steniowagner

// @flow

import { Platform } from 'react-native';
import RNFetchBlob from 'rn-fetch-blob';
import shorthash from 'shorthash';

const BASE_DIRECTORY_PATH = `${RNFetchBlob.fs.dirs.DocumentDir}/bon_appetit`;
const FILE_PREFIX = Platform.OS === 'ios' ? '' : 'file://';

const verifyResourceAlreadyExists = async (path: string): any => {

steniowagner

// @flow

import React, { Component } from 'react';
import { View, StyleSheet, Animated } from 'react-native';
import styled from 'styled-components';

const Wrapper = styled(View)`
  background-color: #e1e4e8;
`;

steniowagner

// @flow

import React, { Fragment } from 'react';

import { BrowserRouter as ApplicationRouter } from 'react-router-dom';

import { connect } from 'react-redux';

import NavigationMenu from './navigation-menu';
import HeaderBar from './header-bar';