Skip to content

Instantly share code, notes, and snippets.

View mightyhorst's full-sized avatar
🎯
Focusing

Nick Mitchell mightyhorst

🎯
Focusing
View GitHub Profile
I see where the confusion might have come from, especially with the introduction of a query value "2" without a clear explanation. Let's clarify the concept of a query in the context of the attention mechanism and adjust the example for better understanding.
In the context of attention mechanisms, a **query** (\(\mathbf{q}\)) represents the current item or piece of information the model is focusing on or trying to relate to other pieces of information. The **keys** (\(\mathbf{k}_i\)) represent aspects or features of other items in a dataset, and the **values** (\(\mathbf{v}_i\)) represent the content or information of those items. The goal of the attention mechanism is to determine how relevant each item (key-value pair) in the dataset is to the query and to produce a weighted combination of these items' values based on their relevance.
Let's correct and simplify the example without introducing arbitrary numbers for the query:
Imagine we have a dataset of items, each with a key (\(\mathbf{k}_i\)) and a val
Environment - dynamics of the problem
Reward - specifies the goal
Agent -
* agent state
* policy
* value function estimate - optional
* model - optional
## 1. Inside the agent: state
@mightyhorst
mightyhorst / MIT 6.S191 - Lecture 2: Recurrent Neural Networks
Last active July 29, 2023 20:38
MIT 6.S191: Intro to Deep Learning MIT notes
t is single timestep
y hat of t = function of input vector x at time t
y hat is predicted
temporal dependence
recurence relationship
h is internal state of memeory passed timestep to timestep
<!DOCTYPE html>
<html>
<head>
<style>
.editor { font-family: 'Roboto Mono', monospace; font-size: 12px; outline: none; overflow-y: auto; padding-left: 48px; counter-reset: line; }
.editor div { display: block; position: relative; white-space: pre-wrap; }
.editor div::before { content: counter(line); counter-increment: line; position: absolute; right: calc(100% + 16px); opacity: 0.5; }
</style>
</head>
@mightyhorst
mightyhorst / scrollinto.jsx
Last active October 29, 2021 05:28 — forked from giventofly/scroll into.js
scroll to element vanilla js
//from https://css-tricks.com/snippets/jquery/smooth-scrolling/
// Scroll to specific values
// scrollTo is the same
window.scroll({
top: 2500,
left: 0,
behavior: 'smooth'
});
@mightyhorst
mightyhorst / index.css
Created October 19, 2021 02:14 — forked from stereokai/index.css
Trigonometry in CSS
//----------------------------------*\
// TRIGONOMETRY FUNCTIONS
//----------------------------------*/
// # Trigonometry in CSS
//
// - Through Taylor/Maclaurin polynomial representation: http://people.math.sc.edu/girardi/m142/handouts/10sTaylorPolySeries.pdf
// - Useful if you don't want to use JS.
// - With CSS Variables.
// - `calc()` can't do power (x ^ y) so I used multiplication instead.
apiVersion: v1
kind: Service
metadata:
name: hello-kubernetes-first
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
selector:
import {JWK, JWS} from 'node-jose';
import {join} from 'path';
import {writeFileSync, readFileSync} from 'fs';
import fetch from 'node-fetch';
/**
* @requires Models
*/
import {IdTokenPayload, AccessTokenPayload} from '../models';
@mightyhorst
mightyhorst / index.ts
Created August 31, 2020 10:59
4. yarn workspaces - example imported files
/**
* @namespace @monorepo/common
* @desc barrel file to export the common files
*/
export * from './models';
@mightyhorst
mightyhorst / nestjs.package.json
Last active August 31, 2020 10:39
3. yarn workspaces - sharing workspace packages
{
"name": "@monorepo/nestjs",
"version": "1.0.0",
"private": true,
"dependencies": {
"@monorepo/common": "1.0.0"
}
}