Skip to content

Instantly share code, notes, and snippets.

@graphitemaster
Last active February 29, 2024 08:49
Show Gist options
  • Star 43 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save graphitemaster/494f21190bb2c63c5516 to your computer and use it in GitHub Desktop.
Save graphitemaster/494f21190bb2c63c5516 to your computer and use it in GitHub Desktop.
Working around offsetof limitations in C++

Working around offsetof limitations in C++:

There is sometimes a situation in which one needs to get the relative offset of a structure field, common examples of this include serialization frameworks which aid to serialize objects, vertex attributes for rendering (D3D, GL.), etc.

The most common technique for getting this information is through the offsetof macro defined in stddef.h. Unfortunately using the macro in C++ comes with a new set of restrictions that prevent some (subjectively valid) uses of it.

Intuition would have you believe that using offsetof on classes/structures with standard-layout would work. However when a base class has data members, the deriving class is not considered standard-layout. Presumably the reason for this is that C++ wants to beable to permit very unusual layouts where a given inherited class might not have the same offset in all instances?

A less relaxed compiler will emit diagnostic for this as a result:

struct vec2 { float x, y; };
struct vec3 : vec2 { float z; };
struct vec4 : vec3 { float w; };

y_offset = offsetof(vec4, y);

Even though vec4 definitely has the following memory layout:

  0   4   8   C
| x | y | z | w |

This becomes a problem and often leads to techniques that lead to illegal code such as the following technique present in many game engines:

vec4 *v = 0;
y_offset = &v->y;

Various people would have you believe it's not undefined since it tends to be the common technique implored to implement the offsetof macro. Modern compilers do provide intrinsics however; (GCC, Clang) now use __builtin_offsetof because they're now beginning to optimize based on the assumption code does not depend on undefined behavior. For more information on why you should no longer write code like that, check out these in depth analysis of the technique:

https://software.intel.com/en-us/blogs/2015/04/20/null-pointer-dereferencing-causes-undefined-behavior

http://www.viva64.com/en/b/0306/

This leaves us in a tough position however. How can we work around this limitation without invoking undefined behavior?

One technique would be to create an object on stack and calculate the offset like so:

vec4 v;
v_offset = size_t(&v.y) - size_t(&v);

However this is not a valid analog for offsetof which is usuable in constant expressions as it yields a constant value. Which means code like the following:

enum { memory_needed = offsetof(vec4, y) };
unsigned char memory_for_x_and_y[memory_needed];

Cannot be realized with this technique.

One option would be to maintain a set of structures with the same layout without any inheritance but that leads to a maintenance hell. So instead I decided to explore C++11s constexpr to see if there was a way to accomplish this.

Unfortunately C++11s constexpr rules are too restrictive to allow us to have a local object in the function, but C++14 did relax those rules allowing us to achieve what we want:

template <typename T1, typename T2>
inline size_t constexpr offset_of(T1 T2::*member) {
    constexpr T2 object {};
    return size_t(&(object.*member)) - size_t(&object);
}

As long as T2 has a default constructor which is constexpr this works. The implicit default constructors for our vec classes are implicitly constexpr so this works:

y_offset = offset_of(&vec4::y);

However C++14 is fairly new and I needed a C++11 compatible option which yields constant values. After a little bit of work, I came up with the following solution:

template <typename T1, typename T2>
struct offset_of_impl {
    static T2 object;
    static constexpr size_t offset(T1 T2::*member) {
        return size_t(&(offset_of_impl<T1, T2>::object.*member)) -
               size_t(&offset_of_impl<T1, T2>::object);
    }
};
template <typename T1, typename T2>
T2 offset_of_impl<T1, T2>::object;

template <typename T1, typename T2>
inline constexpr size_t offset_of(T1 T2::*member) {
    return offset_of_impl<T1, T2>::offset(member);
}

Not as pretty but gets the job done. The compiler diagnostics for misusing this tend to be legible, if you're feeling a bit zealous you could use enable_if to check if the type has a default constructor before stamping out the function. But I found that the diagnostics for that were less legible than not having any check at all.

@schaumb
Copy link

schaumb commented May 12, 2022

Working constexpr code with gcc/clang/vs compilers (c++17):

#include <cstdint>
#include <iostream>
#include <memory>


#pragma pack(push, 1)
template <typename Member, std::size_t O>
struct Pad {
    char pad[O];
    Member m;
};
#pragma pack(pop)

template<typename Member>
struct Pad<Member, 0> {
    Member m;
};

template <typename Base, typename Member, std::size_t O>
struct MakeUnion {
    union U {
        char c;
        Base base;
        Pad<Member, O> pad;
        constexpr U() noexcept : c{} {};
    };
    constexpr static U u {};
};

template <typename Member, typename Base, typename Orig>
struct offset_of_impl {
    template<std::size_t off, auto union_part = &MakeUnion<Base, Member, off>::u>
  static constexpr std::ptrdiff_t offset2(Member Orig::* member) {
      if constexpr (off > sizeof(Base)) {
          throw 1;
      } else {
        const auto diff1 = &((static_cast<const Orig*>(&union_part->base))->*member);
        const auto diff2 = &union_part->pad.m;
        if (diff1 > diff2) {
            constexpr auto MIN = sizeof(Member) < alignof(Orig) ? sizeof(Member) : alignof(Orig);
            return offset2<off + MIN>(member);
        } else {
            return off;
        }
      }
  }
};


template<class Member, class Base>
std::tuple<Member, Base> get_types(Member Base::*);

template <class TheBase = void, class TT>
inline constexpr std::ptrdiff_t offset_of(TT member) {
    using T = decltype(get_types(std::declval<TT>()));
    using Member = std::tuple_element_t<0, T>;
    using Orig = std::tuple_element_t<1, T>;
    using Base = std::conditional_t<std::is_void_v<TheBase>, Orig, TheBase>;
  return offset_of_impl<Member, Base, Orig>::template offset2<0>(member);
}

template <auto member, class TheBase = void>
inline constexpr std::ptrdiff_t offset_of() {
    return offset_of<TheBase>(member);
}


struct s {
  float a;
  char b;
  char bb;
  int c;
  s() = delete; // no constructor
};

#pragma pack(push, 1)
struct s2 {
  float a;
  char b;
  char bb;
  int c;
  double d;
  char e;
};
#pragma pack(pop)

struct a {
  int i;
  int j;
};
struct b {
  int i;
  int k;
};
struct ab : public a, public b {};


struct alignas(16) al {
  float a;
  alignas(8) char b;
  char bb;
  char arr[20];
};

#pragma pack(push, 2)
struct al2 {
  char a;
  int b;
  char c;
};
#pragma pack(pop)


int main() {
    // no constructor, default aligning
  static_assert(offset_of<&s::a>() == 0);
  static_assert(offset_of<&s::b>() == sizeof(float));
  static_assert(offset_of<&s::bb>() == sizeof(float) + sizeof(char));
  static_assert(offset_of<&s::c>() == alignof(s) * 2); // aligned b with bb

  // no alignment
  static_assert(offset_of<&s2::a>() == 0);
  static_assert(offset_of<&s2::b>() == sizeof(float));
  static_assert(offset_of<&s2::bb>() == sizeof(float) + sizeof(char));
  static_assert(offset_of<&s2::c>() == sizeof(float) + sizeof(char)*2);
  static_assert(offset_of<&s2::d>() == sizeof(float) + sizeof(char)*2 + sizeof(int));
  static_assert(offset_of<&s2::e>() == sizeof(float) + sizeof(char)*2 + sizeof(int) + sizeof(double));

  // simply
  static_assert(offset_of<&a::i>() == 0);
  static_assert(offset_of<&a::j>() == sizeof(int));
  static_assert(offset_of<&b::i>() == 0);
  static_assert(offset_of<&b::k>() == sizeof(int));

  // other based
  static_assert(offset_of<&ab::j, ab>() == sizeof(int));
  static_assert(offset_of<&ab::k, ab>() == sizeof(int) * 3);

  // special alignments
  static_assert(offset_of<&al::a>() == 0);
  static_assert(offset_of<&al::b>() == 8);
  static_assert(offset_of<&al::bb>() == 9);
  static_assert(offset_of<&al::arr>() == 16);

  static_assert(offset_of<&al2::a>() == 0);
  static_assert(offset_of<&al2::b>() == 2);
  static_assert(offset_of<&al2::c>() == 6);
}

@schaumb
Copy link

schaumb commented May 13, 2022

Same working code, but with C++11 (gcc requires >=9.0 version)

#include <cstdint>
#include <iostream>
#include <memory>


#pragma pack(push, 1)
template <typename Member, std::size_t O>
struct Pad {
    char pad[O];
    Member m;
};
#pragma pack(pop)

template<typename Member>
struct Pad<Member, 0> {
    Member m;
};

template <typename Base, typename Member, std::size_t O>
struct MakeUnion {
    union U {
        char c;
        Base base;
        Pad<Member, O> pad;
        constexpr U() noexcept : c{} {};
    };
    constexpr static U u {};
};

template <typename Member, typename Base, typename Orig, std::size_t off = 0, class = void>
struct offset_of_impl {
  static constexpr std::ptrdiff_t offset2(Member Orig::* member) {
      return &((static_cast<const Orig*>(&MakeUnion<Base, Member, off>::u.base))->*member) > &MakeUnion<Base, Member, off>::u.pad.m ?
        offset_of_impl<Member, Base, Orig, off + 
        (sizeof(Member) < alignof(Orig) ? sizeof(Member) : alignof(Orig))>::offset2(member) : off;
  }
};

template<typename Member, typename Base, typename Orig, std::size_t off>
struct offset_of_impl<Member, Base, Orig, off, typename std::enable_if<(sizeof(Base) < off)>::type> {
  static constexpr std::ptrdiff_t offset2(Member Orig::* member) {
      return off;
  }

};


template<class Member, class Base>
std::tuple<Member, Base> get_types(Member Base::*);

template <class TheBase = void, class TT>
inline constexpr std::ptrdiff_t offset_of(TT member) {
    using T = decltype(get_types(std::declval<TT>()));
    using Member = typename std::tuple_element<0, T>::type;
    using Orig = typename std::tuple_element<1, T>::type;
    using Base = typename std::conditional<std::is_same<void, TheBase>::value, Orig, TheBase>::type;
  return offset_of_impl<Member, Base, Orig>::offset2(member);
}


struct s {
  float a;
  char b;
  char bb;
  int c;
  s() = delete; // no constructor
};

#pragma pack(push, 1)
struct s2 {
  float a;
  char b;
  char bb;
  int c;
  double d;
  char e;
};
#pragma pack(pop)

struct a {
  int i;
  int j;
};
struct b {
  int i;
  int k;
};
struct ab : public a, public b {};


struct alignas(16) al {
  float a;
  alignas(8) char b;
  char bb;
  char arr[20];
};

#pragma pack(push, 2)
struct al2 {
  char a;
  int b;
  char c;
};
#pragma pack(pop)


int main() {
    // no constructor, default aligning
  static_assert(offset_of(&s::a) == 0, "");
  static_assert(offset_of(&s::b) == sizeof(float), "");
  static_assert(offset_of(&s::bb) == sizeof(float) + sizeof(char), "");
  static_assert(offset_of(&s::c) == alignof(s) * 2, ""); // aligned b with bb

  // no alignment
  static_assert(offset_of(&s2::a) == 0, "");
  static_assert(offset_of(&s2::b) == sizeof(float), "");
  static_assert(offset_of(&s2::bb) == sizeof(float) + sizeof(char), "");
  static_assert(offset_of(&s2::c) == sizeof(float) + sizeof(char)*2, "");
  static_assert(offset_of(&s2::d) == sizeof(float) + sizeof(char)*2 + sizeof(int), "");
  static_assert(offset_of(&s2::e) == sizeof(float) + sizeof(char)*2 + sizeof(int) + sizeof(double), "");

  // simply
  static_assert(offset_of(&a::i) == 0, "");
  static_assert(offset_of(&a::j) == sizeof(int), "");
  static_assert(offset_of(&b::i) == 0, "");
  static_assert(offset_of(&b::k) == sizeof(int), "");

  // other based
  static_assert(offset_of<ab>(&ab::j) == sizeof(int), "");
  static_assert(offset_of<ab>(&ab::k) == sizeof(int) * 3, "");

  // special alignments
  static_assert(offset_of(&al::a) == 0, "");
  static_assert(offset_of(&al::b) == 8, "");
  static_assert(offset_of(&al::bb) == 9, "");
  static_assert(offset_of(&al::arr) == 16, "");

  static_assert(offset_of(&al2::a) == 0, "");
  static_assert(offset_of(&al2::b) == 2, "");
  static_assert(offset_of(&al2::c) == 6, "");
}

@oparkerj
Copy link

Same working code, but with C++11 (gcc requires >=9.0 version)

@schaumb Is there a way to make this work without requiring a newer version of gcc? In older versions, the problem is comparing addresses is not a constant expression. In my use case I also don't care about handling different base classes.

@schaumb
Copy link

schaumb commented Mar 14, 2023

I created another solution working with gcc >= 5.4 version.

The changes:

  • offset_of(mptr) is a macro and not a function.
  • offset_of<base>(mptr) is not working with the macro so I created the offset_of_base(mptr, base) another macro
  • Because offset_of is a macro, only constexpr argument can be used as an argument (it will not work with any non-constexpr variable).

Is this good enough @oparkerj?

@jeaiii
Copy link

jeaiii commented Jun 28, 2023

Nice! The last version works well with large structures, the other ones run out of compiler stack space if you add a large array to some of the cases, but that showed that there was a problem with the step calculation to the next offset to check compared to __builtin_offsetof

Something like this seems more correct alignof(Member) < alignof(Orig) ? alignof(Member) : alignof(Orig))
Or a binary search might overcome both problems

  static_assert(offset_of(&al::arr) == 16, "");
  // arr is at 10 according to offsetof
  static_assert(__builtin_offsetof(al, arr) == 10, "");

https://godbolt.org/z/9dh447qh9

@oparkerj
Copy link

I created another solution working with gcc >= 5.4 version.

The changes:

  • offset_of(mptr) is a macro and not a function.
  • offset_of<base>(mptr) is not working with the macro so I created the offset_of_base(mptr, base) another macro
  • Because offset_of is a macro, only constexpr argument can be used as an argument (it will not work with any non-constexpr variable).

Is this good enough @oparkerj?

Sorry for the late reply. This was quite helpful. I ended up running into some more limitations from my environment, but this helped me learn a lot more about how templates work.

@Somnium7
Copy link

Somnium7 commented Jul 4, 2023

@jeaiii
Copy link

jeaiii commented Jul 19, 2023

@LewisPringle
Copy link

LewisPringle commented Sep 23, 2023

here is a binary search 0 dependency version:

https://gist.github.com/jeaiii/1130f3976fb946cbad2e27c58b8c4d78 https://godbolt.org/z/Yob396xoT

That snippet seems to work fine for gcc and clang, but doesn't work for (the latest) visual studio (on godbolt).

https://godbolt.org/z/Kf4drT61r (KIND OF PARTLY - fixes it for visual studio) - one test case fails...

@jeaiii
Copy link

jeaiii commented Oct 6, 2023

yeah I opened a bug with ms

@jeaiii
Copy link

jeaiii commented Oct 18, 2023

this seems to be the root of the failed test case:

https://godbolt.org/z/YbGj48z1h

struct a {
  int i;
  int j;
};

struct b {
  int i;
  int k;
};

struct ab : public a, public b { };

constexpr ab m{ };

static_assert(&m.j == &(m.*(&ab::j)));

// Fails on MSVC
static_assert(&m.k == &(m.*(&ab::k)));

@jeaiii
Copy link

jeaiii commented Oct 19, 2023

fixing the 0 sized array...just move the termination test to the top so you never create a member_at with N = 0
https://godbolt.org/z/jPn1xPj1n

@jeaiii
Copy link

jeaiii commented Oct 19, 2023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment