gsoc25-warm-and-fuzzy.md

Warm and Fuzzy

Background

The RGW's frontend is an S3 REST API server, and in this project we would like to use a REST API fuzzer to test the RGW for security issues (and other bugs). Would recommend exploring the Restler tool. Very good intro in this video. Feed it with the AWS S3 OpenAPI spec, and see what happens when we let it connect to the RGW.

Project

Initial (evaluation) Phase

• run Ceph with a radosgw. you can use cephadm to install and run ceph in containers or build it from source and run it a vstart cluster
• build and run the fuzzer against a demo service
• use the fuzzer to compile the s3 spec and run it agianst the radosgw (note that this will mostly fail due to lack of support with s3 authentication)
• feel free to suggest other fuzzing tools if
  • they have good stateful fuzzing support (this is must for S3)
  • they support S3 header signing, or it is esy to add it to them

Test Phase

• add S3 authntication to the fuzzer (mainly signing the S3 header)
  • see this issue
  • this tool has a python callback for getting a token, for token based authentication, however this won't help since S3 requires signing of the header
  • note that signing the header may require code changes to the tool (written in F#)
• make sure that the fuzzer knows the right order of operations (e.g. create the bucket before performing operations on it)
• fix issues of inconsistency between the AWS S3 dialect and the Ceph S3 dialect

Fuzzing Phase

• run the fuzzer and try to find as many issues as possible in the radosgw
• try to fix some of the issues

Test Frameework Phase (stretch goal)

• add fuzzing tests to the teuthology test automation framework

Hi Yuval,

I’m currently exploring ways to enhance the fuzzing process for radosgw, specifically handling S3 authentication in Restler. Given that Restler is written in F#, would it be more effective to integrate an external signing mechanism ( a Python signing service) rather than modifying its core to support S3 header signing?

Hi Yuval,

I’m currently exploring ways to enhance the fuzzing process for radosgw, specifically handling S3 authentication in Restler. Given that Restler is written in F#, would it be more effective to integrate an external signing mechanism ( a Python signing service) rather than modifying its core to support S3 header signing?

not sure if this is possible. i would imagine that the code that build the http header (where the signight happens) is at the code of the F# app.
there exists a python callback for fetching a session token, but this does not help withj the signing.

@VinayakTiwari1103 seems like this should actually be done in python (phew...) see: microsoft/restler-fuzzer#942 (comment)

I’ll take a closer look at microsoft/restler-fuzzer#942 and explore how we can integrate S3 signing within the Python layer of Restler.

Hi @yuvalif, I’m working on integrating S3 signing within Restler’s Python layer but facing issues with incorrect/missing signed headers. The RESTler engine fails during testing, likely due to auth handling.

Checking logs in restler_working_dir/Test, I found formatting issues in the signed headers. I’m debugging this by modifying the Python callback and verifying request construction.

I’ve also opened a PR in the Microsoft Restler repo Check PR Any insights on handling S3 signing in Restler would be helpful!

@VinayakTiwari1103 you can see the signing code example here: https://github.com/aws-samples/sigv4-signing-examples/blob/main/no-sdk/python/main.py
the idea is to take the user's access and secret key and sign the payload, and then insert the value into a specific HTTP header.
from the perspective of the restler code (in which i have no experience...) i think you should just be able to call a callback that has access to the message, and can return HTTP header fields that will be added to the HTTP messahe the restler code is sending.

@yuvalif Just to make sure I understand correctly—are you saying that the callback in Restler should handle signing the request before it gets sent, and then return the modified headers? I was under the impression that Restler might modify the request structure before sending it out. Would that affect the validity of the signature? Let me know if I’m missing something here.

@yuvalif Just to make sure I understand correctly—are you saying that the callback in Restler should handle signing the request before it gets sent, and then return the modified headers? I was under the impression that Restler might modify the request structure before sending it out. Would that affect the validity of the signature? Let me know if I’m missing something here.

if restler is changing the content after signing the signature will not be valid anymore.
this means that the callback has be invoked just before the actual sending.

Hi @yuvalif ,

That makes sense. To ensure the signature remains valid, the callback must be executed as the final step before the request is sent. Does Restler provide a mechanism to guarantee that no modifications occur after the callback has signed the request?

Additionally, I’m integrating the AWS SigV4 signing example within the Python layer of Restler. Given that signing requires a precise request structure, I’m considering adding validation logs right before transmission to ensure correctness. Let me know if you have any insights on this approach or any example to understand.

Thanks!

Hi @yuvalif , I really like the project and would love to add any value possible. I am good with C++ and Python, still figuring some stuff out, but I am ready to fix bugs and make things work better. I would love to sit in on the RGW daily standup and weekly refactoring meetings to learn more and pitch in where I can.
The project sounds cool to me. Fixing whatever it finds would be satisfying. Is there anything sir that you would like to suggest me or what me to study some specific documents before starting.

I really like this project—it's been an exciting journey so far. I've successfully generated the configuration, compiled the grammar, and run initial fuzzing sessions. However, the current fuzzing run isn’t producing valid requests because of S3 authentication issues.

Further Plan:

Develop a Signing Callback:
Create a Python function (using the AWS SigV4 signing example as reference) to compute and add the proper Authorization header based on the complete request.

Integrate the Callback:
Register this callback in RESTler’s Python layer so it’s invoked as the very last step before the request is sent, ensuring no modifications are made afterward.

Validate & Test:
First, test the signing functionality against a demo S3 service to confirm that the signature is correct, then use it for fuzzing against RGW.

Looking forward to any feedback or suggestions on this direction!

Hey @Aadik1ng , one thing that might help - maybe add some debug logging right before the signing happens? That way when a signature fails, you can check if RESTler is messing with the request after your callback runs. I've seen it tweak headers unexpectedly sometimes.

Hi @yuvalif ,

I’ve been looking into alternatives to Restler for fuzzing RGW, especially given the S3 authentication challenges. Boofuzz seems promising since it allows more control over request sequences, making it easier to handle stateful fuzzing. It’s Python-based, so integrating our signing mechanism should be smoother.

Would love to hear your thoughts!

Best,
Vinayak

Hi @VinayakTiwari1103 the message sending code in restler is also done in python, so, i dont see any specific issue there.
Had a quick look into boofuzz and it seems like int is more low level than restler.
IMO, the added value with restler is the fact tha tyou can feed it the S3 API schema, and define its statefullness at the S3 API level (e.g. create a bucket and then do bucket operations on it).
Coding the entire S3 API in boofuzz seems like lots of effort. However, it might be interestign to use it in order to fuzz out HTTP layer code (baseed on boost::beast + boost::asio)

Hello @yuvalif ,

Thank you for clarifying. Indeed, Boofuzz’s lower-level approach requires much more work to model the entire S3 API, while RESTler’s built-in schema and stateful support align better with our current S3 fuzzing goals. If we pursue deeper HTTP-layer fuzzing in the future, Boofuzz could be worth revisiting. I appreciate your guidance.

Hi @yuvalif ,
I’ve opened a PR for adding AWS SigV4 signing to RESTler (from #942). It would be great to get your input when you have a moment.

Thanks!
Vinayak

Hi @yuvalif ,
Following up on the SigV4 signing PR—would appreciate your feedback when convenient.
I've also created a PR for the Ceph RGW task (#70786) and would be grateful for any input on that as well.

Thanks!
Vinayak