energy.md

Energy trading

There is a data provider that sends us data using AMPQ and we send back few events to initiate trade. The data coming from the data provider has couple types that we have documentation for and example XMLs as well.

There are 62 even types that have to be persisted to a database.

The events are XML messages. The messages are parsed by SweetXml.

Example parser:

  def xml_to_map(xml_doc, :system_info_responses, correlation_id) do
    xml_doc
    |> xpath(
      ~x"//SystemInfoResp"e,
      system_info_response_backend_version: ~x"@backendVersion"s,
      system_info_response_backend_timezone: ~x"@backendTimeZone"s,
      system_info_response_backend_market_timezone: ~x"@backendMarketTimeZone"s
    )
    |> Map.put(:system_info_response_correlation_id, correlation_id)
  end

We need to implement ~40 more event types.

After the XML message is converted to a Map it is sent to a DB GenServer process.

CREATE TABLE system_info_responses (
    system_info_response_correlation_id TEXT NOT NULL,
    system_info_response_backend_version TEXT NOT NULL,
    system_info_response_backend_timezone TEXT NOT NULL,
    system_info_response_backend_market_timezone TEXT NOT NULL,
    system_info_response_inserted_at TEXT NOT NULL,
    system_info_response_updated_at TEXT NOT NULL,
    UNIQUE(system_info_response_correlation_id)
);

The database is Sqlite3 right now and it will be in the future as well. We have 100 messages / second which Sqlite can handle without a hickup.

Services

The following GenServers are implemented:

• AMPQ consumer connection and session
• DBService: SQLite3 writer using Exqlite
• Stats: collecting statistics how fast are we receiving and inserting data
  • We should have latency
• API: Using Plug and Bandit
• SerDe (from_xml, to_xml):
  • SweetXML
  • Saxy
• UI:
  • Using EEx and Bulma

To be implemented:

• 40 message types
• Telemetry (messages sent / s , received / s , throttling budget, etc.)

Extra info

We need our order book every couple of hours and the trading events saved as a continous stream.

Could you not use broadway with sqs to solve your case?
https://github.com/dashbitco/broadway
Brodway telemetry events: https://hexdocs.pm/broadway/Broadway.html#module-telemetry

Suggested architecture:

• A simple elixir/phoenix project with brodway lib for pubsub
• AWS sqs
• LiveDashboard to see telemetry events (just store telemetry data in the database)

Data flow:

• SQS sends the message
• Brodway receives message
• Brodway transforms xml to json (https://www.postgresql.org/docs/current/datatype-json.html#:~:text=PostgreSQL%20offers%20two%20types%20for,sets%20of%20values%20as%20input.)
• A reporter to handle telemetry events and store in the database https://hexdocs.pm/telemetry_metrics/writing_reporters.html
• A livedashboard route to show the metrics https://hexdocs.pm/phoenix_live_dashboard/metrics.html

Hi @l1x. I saw the job offer in the Elixir Jobs and ended here.

Looking to the description, seems that https://elixir-broadway.org/ is great fit to your project. They have a RabbitMQ example project and also a Dashboard. The functionalities of saving in the database and file can be implemented as GenServers or even consumed as events using Broadway. What do you think of this approach?

Regards,
Roger Leite

Hello @l1x. I'm from Elixir Jobs. I had been experimenting with arbitrage crypto trading bots using GenStages and lmdb as the database, quite a similar case but I got data through WebSockets. It was running stable for months without restart. The repository is here https://github.com/angordeyev/crb.

Hi all! Thanks for the applications. I am going through the CVs now.

@l1x my CV

@l1x is this position still available?

Thanks everybody, the position has been filled.