CANdas Demo

CANdas_demo.ipynb

It takes 30 min to convert one blf to csv. Can you help me how to optimize it. And also can you confirm that it takes same for you or not?

Hi :),

yes, converting from blf takes very long and this is in part why I created CANdas. You can do the conversion once and then save the result as .parquet (or .mat), which takes less than a second to load for the same data. Please have a look at the most current commits in the repository, where I have rewritten a lot of the backend (which is now more convenient to use and faster). However there are still some bugs and that's why I didn't create a release yet.

Blf is a tricky format to parse, because you can't easily parallelize it and values can be very arbitrary (mixed floats, ints, strings, etc.). I've started on a Rust version, which is significantly faster, but then I would need to build a separate dbc parser.

Best regards
Julian

Hi :),

yes, converting from blf takes very long and this is in part why I created CANdas. You can do the conversion once and then save the result as .parquet (or .mat), which takes less than a second to load for the same data. Please have a look at the most current commits in the repository, where I have rewritten a lot of the backend (which is now more convenient to use and faster). However there are still some bugs and that's why I didn't create a release yet.

Blf is a tricky format to parse, because you can't easily parallelize it and values can be very arbitrary (mixed floats, ints, strings, etc.). I've started on a Rust version, which is significantly faster, but then I would need to build a separate dbc parser.

Best regards Julian

Yes, Understood. Thank you so much for the thing. And seriously great work.

Hey thank you for making this, I have one question.
One of my data fields seems to be a tuple instead of number, printing it looks like this:

[[array([[1.70202335e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]
 [array([[1.70202335e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]
 [array([[1.70202335e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]
 ...
 [array([[1.70202597e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]
 [array([[1.70202597e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]
 [array([[1.70202597e+09]])
  array([[(array(['Idle'], dtype='<U4'), array([[0]]))]],
        dtype=[('name', 'O'), ('value', 'O')])           ]]

How do i plot the values of this and ditch the string representation of the value?
the value is a stateflow state thats why its named.

Hey @SpieringsAE, can you please create an issue an Github or Gitlab with a code example and example data (blf and dbc, may be shortened version is enough). I can't help you with the given information. Here you can find excellent guide. CAN data is very diverse and my library focuses on cleaning this mess up and making it accessible through a Pandas-like API. I am not sure if it will work with any CAN data, but I will look into it nonetheless :)

@JulianWgs Unfortunately my blf is 273 MB and I have no idea how to make it smaller, same with the generated .mat file.

Its not my log, it was sent to me with the request to analyze it, but I have no tools to do that with, never even heard of the blf file format before.
Still thanks for making this tool, without it I don't think I would have been able to extract any information from it at all.