RNN shape?

gistfile1.txt

I worked with this model during the EthDenver https://github.com/spro/char-rnn.pytorch . I did not manage to get the right shape for https://github.com/zkonduit/pyezkl/tree/main/examples/tutorial . (loading the model to convert to *.onnx)
I learned a bit about ML though. :P
You
11:48 AM
https://gist.github.com/bshambaugh/af01e7366ac7f43abd8295533703d990 (other scratch work)
....
You
11:51 AM
Here is my colab python sketches: https://colab.research.google.com/drive/1mi2Rcehz-YWOeXGBNjCe-COR5XhnfPTB

Jason, [2/24/23 1:01 PM]
https://github.com/zkonduit/ezkl

Jason, [2/26/23 3:34 PM]
https://github.com/nikhilbarhate99/Char-RNN-PyTorch/blob/master/CharRNN.py

Jason, [2/24/23 1:01 PM]
https://github.com/zkonduit/ezkl

Jason, [2/26/23 3:34 PM]
https://github.com/nikhilbarhate99/Char-RNN-PyTorch/blob/master/CharRNN.py

Jason, [2/26/23 3:34 PM]
This might be doable

Jason, [2/26/23 3:50 PM]
The original is here: https://github.com/karpathy/char-rnn

Brent Shambaugh, [3/1/23 5:10 PM]
https://gist.github.com/karpathy/d4dee566867f8291f086 (minimal RNN)

Brent Shambaugh, [3/1/23 5:36 PM]
https://github.com/spro/char-rnn.pytorch (also try tis one

Brent Shambaugh, [3/1/23 11:50 PM]
https://github.com/zkonduit/ezkl/blob/4a444b249396b6a09e4c323143f6daf1b71b4c92/README.md#onnx-examples

onnx examples

This repository includes onnx example files as a submodule for testing out the cli.

If you want to add a model to examples/onnx, open a PR creating a new folder within examples/onnx with a descriptive model name. This folder should contain:

an input.json input file, with the fields expected by the ezkl cli.
a network.onnx file representing the trained model
a gen.py file for generating the .json and .onnx files following the general structure of examples/tutorial/tutorial.py.

TODO: add associated python files in the onnx model directories.

---

It looks like the export function is failing because the shape of the input tensor is not matching the provided input_shape argument.

When calling the export function, you need to make sure that the input_array argument has the correct shape for the model input. In this case, the model input size is (batch_size, input_size), where batch_size is the number of input sequences and input_size is the length of each input sequence. So if you want to export a single input sequence with length 4, you should pass a 2D tensor with shape (1, 4).

Here's an example of how to do this:

lua

input_array = torch.tensor([[0, 1, 2, 3]])
hidden = rnn.init_hidden(1)
export(rnn, [1, 4], input_array, hidden)

This will export the model with an input tensor of shape (1, 4) and a batch size of 1. Note that you need to wrap the input values in a PyTorch tensor before passing them to the export function.

---

https://github.com/zkonduit/pyezkl/tree/main/examples/tutorial

As noted above this graph takes in 3 inputs and produces 2 outputs. The main function instantiates an instance of Circuit and saves it to an Onnx file.

def main():
torch_model = Circuit()
# Input to the model
shape = [3, 2, 2]
x = 0.1*torch.rand(1,shape, requires_grad=True)
y = 0.1torch.rand(1,shape, requires_grad=True)
z = 0.1torch.rand(1,*shape, requires_grad=True)
torch_out = torch_model(x, y, z)
# Export the model
torch.onnx.export(torch_model, # model being run
(x,y,z), # model input (or a tuple for multiple inputs)
"network.onnx", # where to save the model (can be a file or file-like object)
export_params=True, # store the trained parameter weights inside the model file
opset_version=10, # the ONNX version to export the model to
do_constant_folding=True, # whether to execute constant folding for optimization
input_names = ['input'], # the model's input names
output_names = ['output'], # the model's output names
dynamic_axes={'input' : {0 : 'batch_size'}, # variable length axes
'output' : {0 : 'batch_size'}})

d = ((x).detach().numpy()).reshape([-1]).tolist()
dy = ((y).detach().numpy()).reshape([-1]).tolist()
dz = ((z).detach().numpy()).reshape([-1]).tolist()


data = dict(input_shapes = [shape, shape, shape],
            input_data = [d, dy, dz],
            output_data = [((o).detach().numpy()).reshape([-1]).tolist() for o in torch_out])

# Serialize data into file:
json.dump( data, open( "input.json", 'w' ) )

if name == "main":
main()

Running the file generate an .onnx file. Note that this also create the required input json file, whereby we use the outputs of the pytorch model as the public inputs to the circuit.

Brent Shambaugh, [3/2/23 12:04 AM]
here is what I am working with loading the .onnx

Brent Shambaugh, [3/2/23 1:37 AM]
maybe this says something about shapes: https://machinelearningmastery.com/reshape-input-data-long-short-term-memory-networks-keras/

Brent Shambaugh, [3/2/23 1:44 AM]
https://github.com/spro/char-rnn.pytorch/blob/master/model.py

Brent Shambaugh, [3/2/23 2:06 AM]
https://ppasumarthi-69210.medium.com/language-model-using-char-rnn-1df53f735880 (shapes?)

Brent Shambaugh, [3/2/23 2:11 AM]
https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm (# Size: [batch_size, seq_len, input_size])

Brent Shambaugh, [3/2/23 2:14 AM]
https://jvns.ca/blog/2020/11/30/implement-char-rnn-in-pytorch/ (another thing that might help find shapes) ... hint .... look at the training python code

Use this for inspiration: https://github.com/hunter-z-hunter/hunter-z-hunter/blob/main/hunter.ipynb

https://github.com/hunter-z-hunter/hunter-z-hunter/blob/main/setup/trainer.py (maybe this is useful?)