RNN shape?

gistfile1.txt

I worked with this model during the EthDenver https://github.com/spro/char-rnn.pytorch . I did not manage to get the right shape for https://github.com/zkonduit/pyezkl/tree/main/examples/tutorial . (loading the model to convert to *.onnx)
I learned a bit about ML though. :P
You
11:48 AM
https://gist.github.com/bshambaugh/af01e7366ac7f43abd8295533703d990 (other scratch work)
....
You
11:51 AM
Here is my colab python sketches: https://colab.research.google.com/drive/1mi2Rcehz-YWOeXGBNjCe-COR5XhnfPTB

review this: https://github.com/zkonduit/ezkl

RNN clearly explained: https://www.youtube.com/watch?v=AsNTP8Kwu80 .

A guide on character level RNN: https://edumunozsala.github.io/BlogEms/fastpages/jupyter/rnn/lstm/pytorch/2020/09/03/char-level-text-generator-pytorch.html

Language Model for RNN: https://ppasumarthi-69210.medium.com/language-model-using-char-rnn-1df53f735880

Generating text with RNN: http://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf

RNN and Vanish

Perhaps this gives you the shape???
https://github.com/eliben/deep-learning-samples/blob/master/min-char-rnn/min-char-lstm.py

For reference, reworded:

1. https://github.com/spro/char-rnn.pytorch (see train.py and model.py) [Model we want to use]
2. https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm (# Size: [batch_size, seq_len, input_size] [shape that the model needs to be]
3. https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html, https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html, https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru#torch.nn.GRU, https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=linear#torch.nn.Linear [supporting documents from pyTorch]

2a) Shape needs to fit into a script like this: https://github.com/zkonduit/ezkl/blob/main/examples/onnx/1l_relu/gen.py#L13
following this: https://github.com/zkonduit/pyezkl/tree/main/examples/tutorial
2b) Workflow needs to be:
"ezkl is a library and command-line tool for doing inference for deep learning models and other computational graphs in a zk-snark. It enables the following workflow:

Define a computational graph, for instance a neural network (but really any arbitrary set of operations), as you would normally in pytorch or tensorflow.
Export the final graph of operations as an .onnx file and some sample inputs to a .json file.
Point ezkl to the .onnx and .json files to generate a ZK-SNARK circuit with which you can prove statements such as:

" https://github.com/zkonduit/ezkl/blob/main/README.md

Brent Shambaugh, [3/2/23 11:02 AM]
Jason's comments about RNN Model from before:

• 2 Inputs
• What are the shapes of the inputs

Brent Shambaugh, [3/2/23 11:07 AM]

1. Is the Model that we want to use.
2. Is the likely shape of the Model that we want to use that will be processed by ezkl/pyezkl
3. is additional information to explain the model that we want to use

2a) Example of an export script for ezkl, but it may be the wrong shape.
2b) process for creating a custom shape for ezkl

Brent Shambaugh, [3/2/23 11:09 AM]
2b) [x,y,z] should be [x,y]

Brent Shambaugh, [3/2/23 11:15 AM]
https://gist.github.com/bshambaugh/af01e7366ac7f43abd8295533703d990

Brent Shambaugh, [3/2/23 11:14 AM]
this is the likely script to create a shape

Brent Shambaugh, [3/2/23 11:15 AM]
I mean ONNX file

The models for https://github.com/nikhilbarhate99/Char-RNN-PyTorch and https://github.com/spro/char-rnn.pytorch are similar, so look at the documentation.

https://github.com/spro/char-rnn.pytorch#training

Brent Shambaugh, [3/2/23 11:55 AM]
https://gist.github.com/karpathy/d4dee566867f8291f086 (shortest version of char-rnn, use for inspiration as it is written in numpy, but not for implementation)
cross reference with https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/ —> https://www.youtube.com/watch?v=56TYLaQN4N8 (lecture 12) &&

Brent Shambaugh, [3/2/23 12:19 PM]
Train on this: https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html (LSTMs in Pytorch. used in char-rnn)

Brent Shambaugh, [3/2/23 1:19 PM]
Exporting a file from PyTorch and Using it in ONNX Runtime: https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html

Brent Shambaugh, [3/2/23 1:27 PM]
—-> additional ref: training LSTM: https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/practicals/practical6.pdf

Brent Shambaugh, [3/2/23 1:28 PM]
—-> https://github.com/oxford-cs-ml-2015/practical6

Brent Shambaugh, [3/4/23 8:54 PM]
https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html

Brent Shambaugh, [3/4/23 8:55 PM]
https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru

Brent Shambaugh, [3/4/23 8:55 PM]
https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm

Jason, [2/24/23 1:01 PM]
https://github.com/zkonduit/ezkl

Jason, [2/26/23 3:34 PM]
https://github.com/nikhilbarhate99/Char-RNN-PyTorch/blob/master/CharRNN.py

Jason, [2/24/23 1:01 PM]
https://github.com/zkonduit/ezkl

Jason, [2/26/23 3:34 PM]
https://github.com/nikhilbarhate99/Char-RNN-PyTorch/blob/master/CharRNN.py

Jason, [2/26/23 3:34 PM]
This might be doable

Jason, [2/26/23 3:50 PM]
The original is here: https://github.com/karpathy/char-rnn

Brent Shambaugh, [3/1/23 5:10 PM]
https://gist.github.com/karpathy/d4dee566867f8291f086 (minimal RNN)

Brent Shambaugh, [3/1/23 5:36 PM]
https://github.com/spro/char-rnn.pytorch (also try tis one

Brent Shambaugh, [3/1/23 11:50 PM]
https://github.com/zkonduit/ezkl/blob/4a444b249396b6a09e4c323143f6daf1b71b4c92/README.md#onnx-examples

onnx examples

This repository includes onnx example files as a submodule for testing out the cli.

If you want to add a model to examples/onnx, open a PR creating a new folder within examples/onnx with a descriptive model name. This folder should contain:

an input.json input file, with the fields expected by the ezkl cli.
a network.onnx file representing the trained model
a gen.py file for generating the .json and .onnx files following the general structure of examples/tutorial/tutorial.py.

TODO: add associated python files in the onnx model directories.

---

It looks like the export function is failing because the shape of the input tensor is not matching the provided input_shape argument.

When calling the export function, you need to make sure that the input_array argument has the correct shape for the model input. In this case, the model input size is (batch_size, input_size), where batch_size is the number of input sequences and input_size is the length of each input sequence. So if you want to export a single input sequence with length 4, you should pass a 2D tensor with shape (1, 4).

Here's an example of how to do this:

lua

input_array = torch.tensor([[0, 1, 2, 3]])
hidden = rnn.init_hidden(1)
export(rnn, [1, 4], input_array, hidden)

This will export the model with an input tensor of shape (1, 4) and a batch size of 1. Note that you need to wrap the input values in a PyTorch tensor before passing them to the export function.

---

https://github.com/zkonduit/pyezkl/tree/main/examples/tutorial

As noted above this graph takes in 3 inputs and produces 2 outputs. The main function instantiates an instance of Circuit and saves it to an Onnx file.

def main():
torch_model = Circuit()
# Input to the model
shape = [3, 2, 2]
x = 0.1*torch.rand(1,shape, requires_grad=True)
y = 0.1torch.rand(1,shape, requires_grad=True)
z = 0.1torch.rand(1,*shape, requires_grad=True)
torch_out = torch_model(x, y, z)
# Export the model
torch.onnx.export(torch_model, # model being run
(x,y,z), # model input (or a tuple for multiple inputs)
"network.onnx", # where to save the model (can be a file or file-like object)
export_params=True, # store the trained parameter weights inside the model file
opset_version=10, # the ONNX version to export the model to
do_constant_folding=True, # whether to execute constant folding for optimization
input_names = ['input'], # the model's input names
output_names = ['output'], # the model's output names
dynamic_axes={'input' : {0 : 'batch_size'}, # variable length axes
'output' : {0 : 'batch_size'}})

d = ((x).detach().numpy()).reshape([-1]).tolist()
dy = ((y).detach().numpy()).reshape([-1]).tolist()
dz = ((z).detach().numpy()).reshape([-1]).tolist()


data = dict(input_shapes = [shape, shape, shape],
            input_data = [d, dy, dz],
            output_data = [((o).detach().numpy()).reshape([-1]).tolist() for o in torch_out])

# Serialize data into file:
json.dump( data, open( "input.json", 'w' ) )

if name == "main":
main()

Running the file generate an .onnx file. Note that this also create the required input json file, whereby we use the outputs of the pytorch model as the public inputs to the circuit.

Brent Shambaugh, [3/2/23 12:04 AM]
here is what I am working with loading the .onnx

Brent Shambaugh, [3/2/23 1:37 AM]
maybe this says something about shapes: https://machinelearningmastery.com/reshape-input-data-long-short-term-memory-networks-keras/

Brent Shambaugh, [3/2/23 1:44 AM]
https://github.com/spro/char-rnn.pytorch/blob/master/model.py

Brent Shambaugh, [3/2/23 2:06 AM]
https://ppasumarthi-69210.medium.com/language-model-using-char-rnn-1df53f735880 (shapes?)

Brent Shambaugh, [3/2/23 2:11 AM]
https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm (# Size: [batch_size, seq_len, input_size])

Brent Shambaugh, [3/2/23 2:14 AM]
https://jvns.ca/blog/2020/11/30/implement-char-rnn-in-pytorch/ (another thing that might help find shapes) ... hint .... look at the training python code

Use this for inspiration: https://github.com/hunter-z-hunter/hunter-z-hunter/blob/main/hunter.ipynb

https://github.com/hunter-z-hunter/hunter-z-hunter/blob/main/setup/trainer.py (maybe this is useful?)