How my little C++ meta-serialization library works and how I wrote it

Introduction

A year ago I wrote a small library called MetaStuff. It is used to register meta info about classes, which can be later used for serialization, deserialization, making GUI and so on. The cool thing about it is that you can write a serializer for your own data format, be it JSON, Lua or anything else. The library is very small, it uses C++14, requires no pre-build process, RTTI, macros… It’s just templates!

Let’s look at how registration and serialization work. Suppose you have the following struct:

struct Person {
    int age;
    std::string name;
};

You register it like this:

namespace meta {

template <>
inline auto registerMembers<Person>()
{
    return members(
        member("age", &Person::age),
        member("name", &Person::name)
    );
}

} // end of namespace meta

And now you can serialize / deserialize it to / from JSON! (I’ll be using Niels Lohmann’s JSON for Modern C++ library in this article)

Person person;
person.age = 30;
person.name = "John";

// serialize
json j;
j = person;

std::cout << std::setw(4) << j << std::endl;

// deserialize
Person p2;
p2 = j.get<Person>(j);

std::cout << "Name = " << p2.name << ", age = " << p2.age << "\n";

Output:

{
    "age": 30,
    "name": "John"
}

Simple as that!

One more example. In my game, I have Animation class registered. I also have a bunch of functions for Dear ImGui which can automatically build GUI for me, so I can easily change animation parameters. I just do this:

ImGui::Input(animation);

And I get this (the “Animation properties” table is generated by MetaStuff):

Using MetaStuff

The most important thing about MetaStuff is that it stores information about class members in a tuple without losing their types. To do something for all members of the class, you write this:

meta::doForAllMembers<Person>(
    [&person](const auto& member)
    {
        // do something for member
    });

The type of member inside the lambda is meta::Member<Person, T>, where T is a type of a member you previously registered. This is really nice, because just by calling Member<Person, T>::get(person), you get a reference to an object from person instance, which you can modify and use in other functions.

Let’s look at a simple example.

Person person;
person.age = 30;
person.name = "John";

meta::doForAllMembers<Person>(
    [&person](const auto& member)
{
    using MemberT = meta::get_member_type<decltype(member)>;
    std::cout << "* " << member.getName() <<
        ", value = " << member.get(person) <<
        ", type = " << typeid(MemberT).name() << '\n';
});

Output:

* age, value = 30, type = int
* name, value = John, type =
  class std::basic_string<
    char,struct std::char_traits<char>,class std::allocator<char>
  >

You can iterate over all members and call template functions or functions with overloads. Suppose you have a function:

template <typename T>
void print(T obj) {};

void print(int i)
{
    std::cout << "It's an int! Value = " << i;
}

void print(const std::string& str)
{
    std::cout << "It's a string! Value = " << str;
}

And now we do this:

meta::doForAllMembers<Person>(
    [&person](const auto& member)
    {
        std::cout << "* " << "What is " << member.getName() << " ? ";
        print(member.get(person));
        std::cout << '\n';
    });

Output:

* What is age ? It's an int! Value = 30
* What is name ? It's a string! Value = John

Here you can look at how JSON serialization / deserialization is performed. It’s not that complicated: we just need to write some overloads for non-trivial cases (std::vector should be processed as JSON array and std::unordered_map as JSON object). For the main serialization / deserialization loop, we just need to iterate over all registered members and do something like this:

jsonObject[member.getName()] = member.get(person);

The serializer can handle much more complex cases, than shown previously, such as instance of one class, being in another class: the serialization will work correctly for registered classes! For example, for the following classes:

struct Health {
    int hp = 20;
};

struct Hero {
    Health health;
    int attackPower = 30;
};

we’ll get this JSON:

{
    "attackPower" : 10,
    "health" : {
        "hp" : 20
    }
}

Nice things about MetaStuff

Suppose our Person class now looks like this:

class Person {
public:
    void setAge(int a) { age = a; }
    int getAge() const { return age; }

    // ... same for name
private:
    int age;
    std::string name;
};

We pass getter and setter to MetaStuff during registration:

member("age", &Person::getAge, &Person::setAge)

Now, when we’ll serialize object, Person::getAge will get called when getting value of “age” member.

You can make more complex serialization process easier by using getters and setters in a non-standard way. Suppose that you have Sprite class:

class Sprite {
public:
    // ...
private:
    Texture texture;
};

How do you deserialize it? It’s not possible to store Texture in a JSON, so we’ll store a texture filename instead:

{
    "texture" : "res/images/hero.png"
}

But how do we load texture when reading data from JSON? One solution is to register variable, which will store texture’s filename, and later call some postInit function:

meta::member("texture", &Sprite::textureFilename);

void Sprite::postInit()
{
    texture.loadFromFile(textureFilename);
}

Sprite s;
s = j.get<Sprite>(); // here we'll get textureFilename
s.postInit(); // and here our texture we'll be loaded

This works, but we can do better! Let’s register a loadTexture function as a setter for “texture”.

meta::member("texture", &Sprite::getTextureFilename, &Sprite::loadTexture);

const std::string& Sprite::getTextureFilename() const
{
    return textureFilename;
}

void Sprite::loadTexture(const std::string& filename)
{
    textureFilename = filename;
    texture.loadFromFile(filename);
}

Now, when you’ll deserialize from JSON, loadTexture will be called and the texture will get loaded as you want it to. And when you serialize Sprite class to JSON, texture filename gets saved there.

Now, let’s implement simple meta info holder and see how I came from its design to MetaStuff.

Simple approach (that kinda works)

It all started from a simple, yet somewhat flawed approach. One easy way to build a meta system is to make it with type-erasure and virtual functions. Full working example can be found here.

template <typename Class>
class IMember {
public:
    virtual ~IMember() = default;
    virtual void fromJson(Class& obj, const json& j) const = 0;
    virtual json toJson(const Class& obj) const = 0;
};

IMember is a base class for Member class which will store member info.

template <typename Class, typename T>
class Member : public IMember<Class> {
public:
    Member(T Class::* ptr) : ptr(ptr) {}

    void fromJson(Class& obj, const json& j) const override
    {
        obj.*ptr = j;
    }

    json toJson(const Class& obj) const override
    {
        return json(obj.*ptr);
    }

private:
    T Class::* ptr;
};

Member class stores a pointer to a class member which lets us get it directly from Class instance (see how we need to pass it in fromJson / toJson function).

And now for the class that will store all info about members of a particular class:

template <typename Class>
class ClassMetaInfo {
public:
    static void serialize(Class& obj, const json& j)
    {
        for (const auto& pair : members) {
            const auto& memberName = pair.first;
            const auto& memberPtr = pair.second;

            memberPtr->fromJson(obj, j[memberName]);
        }
    }

    static json deserialize(const Class& obj)
    {
        json j;
        for (const auto& pair : members) {
            const auto& memberName = pair.first;
            const auto& memberPtr = pair.second;

            j[memberName] = memberPtr->toJson(obj);
        }
        return j;
    }

    template <typename T>
    static void registerMember(const char* name, T Class::* ptr)
    {
        members.emplace(name, std::make_unique<Member<Class, T>>(ptr));
    }

    using MemberPtrType = std::unique_ptr<IMember<Class>>;
    using MemberMapType = std::unordered_map<std::string, MemberPtrType>;
private:
    static MemberMapType members;
};

template <typename Class>
typename ClassMetaInfo<Class>::MemberMapType ClassMetaInfo<Class>::members;

ClassMetaInfo class stores all members in members unordered_map. We can’t store objects of different classes inside unordered_map, because Member<Class, int> and Member<Class, std::string> have different types. That’s why they have the common base class and virtual functions which will help us get the original type back (at least for a function call).

Let’s use it:

struct Person {
    std::string name;
    int age;

    static void registerClass()
    {
        ClassMetaInfo<Person>::registerMember("name", &Person::name);
        ClassMetaInfo<Person>::registerMember("age", &Person::age);
    }
};

int main()
{
    Person::registerClass();

    Person p{ "John", 30 };

    json j = ClassMetaInfo<Person>::deserialize(p);
    std::cout << std::setw(4) << j << std::endl;

    Person p2;
    ClassMetaInfo<Person>::serialize(p2, j);
    std::cout << "Name = " << p2.name << ", age = " << p2.age << "\n";
}

Output:

{
    "age": 30,
    "name": "John"
}
Name = John, age = 30

Okay, that worked! This system worked well for me, until I understood that if I wanted to add another format (for example, XML), I’d have to add new toXml / fromXml virtual functions everywhere. Same for ImGui and anything else. The other problem was the need to call registerClass function. If I forgot to do it, the “members” map will be empty for the corresponding class. It’s also very hard to implement serializer which will handle nested classes, as shown with Hero / Health example shown previously.

Implementation of MetaStuff

Let’s see what had to be changed from a previous meta-library to get this functionality.

First of all, instead of storing members in an unordered_map, I store them in a tuple. It looks like this for a Person class:

std::tuple<Member<Person, std::string>, Member<Person, int>> members;

With some template magic and Vittorio Romero’s help (see this awesome video), I implemented a function which calls some lambda for each member of the tuple. And that’s what meta::doForAllMembers does: it iterates over all Member objects inside of members tuple and calls lambda passed as the function parameter for each of them. The lambda which is used is a generic one, so on each call you get a Member<Class, T> object without type erasure!

Let’s look at class registration. To register Person class, you have to write this:

namespace meta {

template <>
inline auto registerMembers<Person>()
{
    return members(
        member("age", &Person::age),
        member("name", &Person::name)
    );
}

} // end of namespace meta

Note the auto as the return type. You don’t want to write std::tuple<Member<Person, std::string>, Member<Person, int>>, the compiler can just deduce it! Imagine the class with 50 members. The return type would be gigantic and the compiler does all the work for you.

Okay, how do we make sure that this registerMembers<T> function gets called? Simple! I just made class called MetaHolder, which uses one nice trick. Let’s look at the implementation:

#pragma once

#include <tuple>

namespace meta
{
namespace detail
{

template <typename T, typename TupleType>
struct MetaHolder {
    static TupleType members;
    static const char* name()
    {
        return registerName<T>();
    }
};

template <typename T, typename TupleType>
TupleType MetaHolder<T, TupleType>::members = registerMembers<T>();

} // end of namespace detail
} // end of namespace meta

How do we force the instantiation of this template to get generated? Simple, we just use some function from namespace meta! For example, if you call meta::doForAllMembers<Person>, it will use MetaHolder<Person, TupleType> class, which will get compiler to generate this class! Its tuple is initialized by calling meta::registerMembers<Person>, when the initialization of members tuple is performed. But wait, how do I get TupleType? I get it from registerMember<T> function! So, TupleType = decltype(registerMembers<T>()).

So, we get a function like this:

template <typename Class>
const auto& getMembers()
{
    return detail::MetaHolder<Class, decltype(registerMembers<Class>())>::members;
}

Doing something for one member

Suppose we want to do some things only for member called “name” using MetaStuff:

meta::doForAllMembers<Person>(
    [&person](const auto& member)
    {
        if (member.getName() == "name") {
            std::cout << "Name starts with " << member.get(person)[0] << '\n';
        }
    });

Sadly, this doesn’t work! Why? Because lambda gets generated for all registered types! Not just for strings, but for Person::age, which is an int. The code member.get(person)[0] is not valid for Member whose type is int, so the compilation fails.

We can use special doForMember function, which will work well:

meta::doForMember<Person, std::string>("name",
    [&person](const auto& member)
    {
        std::cout << "Name starts with " << member.get(person)[0] << '\n';
    });

How does it work? With some SFINAE:

template <bool Test,
    typename F, typename... Args,
    typename = std::enable_if_t<Test>>
void call_if(F&& f, Args&&... args)
{
    f(std::forward<Args>(args)...);
}

template <bool Test,
    typename F, typename... Args,
    typename, typename = void>
void call_if(F&& /* f */, Args&&... /* args */)
{ /* do nothing */ }

And then I do this in doForMember:

detail::call_if<std::is_same<MemberT, T>::value>(std::forward<F>(f), member);

If std::is_same returns true, then call_if with function call gets generated, otherwise call_if function will be empty. Remember that “f” is most likely to be generic lambda, so if it doesn’t get called with some argument, then template instantiation for this type is not generated and everything works as expected.

Conclusion

And that’s mostly how MetaStuff works! The library is a bit more complicated than that, because it lets you register classes without default constructors, register getters and setters and more.

The library is still very simple and needs much more work on it. But it’s certainly one of the hardest things I’ve ever written and I’m very proud of it. I’ve learned a lot about templates, and I hope you did too. Maybe this article even will inspire you to write your own library or start using MetaStuff for you projects.

Thank you for reading!