Skip to content

Instantly share code, notes, and snippets.

@Al12rs

Al12rs/README.md

Forked from Holt59/README.md
Last active May 21, 2020 12:49
Show Gist options
  • Save Al12rs/811072dd4c92c82854989052606635d1 to your computer and use it in GitHub Desktop.
Save Al12rs/811072dd4c92c82854989052606635d1 to your computer and use it in GitHub Desktop.
Download ZIP
MO2 Plugin Python README
Raw
README.md

Mod Organizer 2 - Plugin Python Proxy

This repository contains the implementation of the proxy plugin for python language plugins in Mod Organizer 2.

This README is intended for MO2 developers, not plugin creators. If you are looking for documentation on how to create a python plugin, please refer to the MO2 wiki.

TOC

Introduction

The project is made of two sub-projects: runner and proxy. The proxy only contains the ProxyPython implementation which is the C++ proxy plugin that is actually loaded by MO2. The runner project contains all the bindings between uibase and python.

If you need to modify the python interface, e.g. following uibase changes, you only need to take a look at the runner project.

The runner project

The runner project contains the greater part of the project, with the actual bindings for all the classes in uibase, and C++-python converters.

Forewords

The runner project uses boost::python for the interface, so you might want to get the documentation nearby if you intend to modify it.

In the following, I will start by explaining the organization of the project, detailing what the main 4 components (files) contain, and then explain how to do the most common things (adding a new plugin type, adding a new wrappers, etc.).

Project organization

The parts of the project you are most likely going to modify are the following:

  • proxypluginwrappers: contains wrappers for the plugins.
  • gamefeatureswrappers: contains wrappers for the game features.
  • pythonrunner: contains the actual declaration of the mobase module with all the exposed classes.
  • uibasewrappers: contains wrappers for some uibase classes that are neither plugins nor game features.

Note: A wrapper is a C++ class that implements a C++ interface and delegates virtual function calls to a python object. A wrapper is needed for any class that can be extended through python (plugins, game features, some uibase classes).

The runner projects also contains the following utility files:

  • converters.h contains utility functions to easily create C++/Python conversion functions for Qt or standard classes. Some useful functions:
    • utils::register_qclass_converter to register a Qt class.
    • utils::register_qflags_converter to register a QFlags.
    • utils::register_sequence_container to register a sequence container (QList, std::vector, std::list, ...).
  • tuple_helper.h and variant_helper.h contain converters for tuples and variants (works for both std and boost version). The register functions are in the boost::python namespace.
  • error.h contains custom exceptions that you can throw. In particular, the PythonError, that will fetch the error message from python (does not always work... ).
  • pythonwrapperutilities.h contains utility function that can be used when implementing wrappers.

Those are (should be?) fully documented, so I will not explain them in detail here. You can find examples for those in the 4 main files.

pythonrunner

The pythonrunner.cpp file contains the actual mobase module within BOOST_PYTHON_MODULE(mobase):

BOOST_PYTHON_MODULE(mobase)
{
  // I have no idea why this is required (or even if this is required).
  PyEval_InitThreads();

  // We need to import PyQt5 here, otherwise boost will fail to convert default
  // argument for Qt class.
  bpy::import("PyQt5.QtCore");
  bpy::import("PyQt5.QtWidgets");

  // Registering converters (see below).

  // Exposing classes (See below).

  // Register game features, see the "gamefeatureswrappers" section.
  registerGameFeaturesPythonConverters();
}

Note: bpy is a namespace alias for boost::python.

Registering converters

The beginning of the module declaration contains registration for all converters: Qt classes that are used between Python and C++, QFlags, containers, smart pointers, etc. If a function is exposed through boost::python, all the argument types and return type must have registered converters. If you have a std::function that has arguments, you also needs to register its arguments or return type. For instance, if we were to expose the following function:

std::vector<QString> mapToString(
    td::vector<std::tuple<QDateTime, QUrl>>, 
    std::function<QString(std::set<int>, std::map<int, double>, QString)>
);

You would need to register the following:

utils::register_qstring_converter(); // For QString, this is a special one, see FAQ at the end.

// Register the Qt class:
utils::register_qclass_converter<QDateTime>();
utils::register_qclass_converter<QUrl>();

// Register the tuple:
bpy::register_tuple<std::tuple<QDateTime, QUrl>>();

// Register the containers:
bpy::register_sequence_container<std::vector<QString>>();
bpy::register_sequence_container<std::vector<std::tuple<QDateTime, QUrl>>>();
bpy::register_set_container<std::set<int>>();
bpy::register_associative_container<std::map<int, double>>();

// Register the function:
bpy::register_functor_converter<QString(std::set<int>, std::map<int, double>, QString)>();

You only need to register converters if those are not already registered: do not duplicate converter registration.

A few notes on converters:

  • register_tuple will convert from any Python sequence of the right length and always to Python tuple.
  • register_variant will convert from and to any of the Python types the variant contains (those must have registered converters). This is different from QVariant!.
  • register_sequence_container will convert from any Python sequence, but always to Python list.
  • register_set_container has the same behavior as register_sequence_container. It does not convert to Python set due to the differences between std::set and set (set is more like std::unordered_set in C++ than std::set).
  • register_associative_container will convert from dict-like object and to Python dict.
  • register_functor_converter will convert from any Python callable with the right number of arguments.
    • Only the number of arguments is checked when converting, not their types nor the return type (this is not possible without executing the function), so the conversion could succeed while the actual call might fail.
    • None python object are converted to default-constructed std::function.
    • register_functor_converter does not register a to-Python converter!

Checking for python sequences or dict is made using PySequence_Check and PyDict_Check, so any Python types with the proper requirements should work (e.g. defaultdict and OrderedDict are valid dictionaries).

Some warnings:

  • These converters will not work with pointers (as arguments or return object), e.g. the following cannot be done:
std::vector<int>* myVec(std::set<Qstring> const*);
  • Except for sequence containers, the order of elements is lost when converting.

The register_qclass_converter is different and will register both value and pointer conversions (it does not register value for non-copyable type).

If you need to register an enumeration with an associated Q_FLAGS declaration, you should expose the base enumeration through bpy::enum_ (see below) and then use register_qflags_converter on the actual QFlags, e.g.:

utils::register_qflags_converter<IPluginList::PluginStates>();

bpy::enum_<IPluginList::PluginState>("PluginState")
  .value("MISSING", IPluginList::STATE_MISSING)
  .value("INACTIVE", IPluginList::STATE_INACTIVE)
  .value("ACTIVE", IPluginList::STATE_ACTIVE)
  ;

Exposing classes

Classes, functions and enumerations are exposed using respectively bpy::class_, bpy::def and bpy::enum_. I will not go into details about everything here since the boost documentation for this should be sufficient.

Enumerations

Enumerations are exposed through bpy::enum_. Enumeration values should be ALL_UPPER_CASES to follow python convention and not be exported in the mobase module (see bpy::enum_::export_values).

bpy::enum_<MOBase::IPluginInstaller::EInstallResult>("InstallResult")
  .value("SUCCESS", MOBase::IPluginInstaller::RESULT_SUCCESS)
  .value("FAILED", MOBase::IPluginInstaller::RESULT_FAILED)
  .value("CANCELED", MOBase::IPluginInstaller::RESULT_CANCELED)
  .value("MANUAL_REQUESTED", MOBase::IPluginInstaller::RESULT_MANUALREQUESTED)
  .value("NOT_ATTEMPTED", MOBase::IPluginInstaller::RESULT_NOTATTEMPTED)

  // .export_values() - Don't do that, unless you are in a class scope.
  ;

You can export enumeration values into a class for inner enumerations. This allows typing IFileTree.FILE instead of IFileTree.FileType.FILE. You can find examples of this in the FileTreeEntry and IFileTree classes.

Classes

Classes are exposed through bpy::class_. If your class needs a wrapper (i.e. it needs to be extended in python), you will expose the wrapper rather than the interface.

There are many examples in the actual code so I will not list everything here.

  • If you expose a wrapper, you can mark pure-virtual functions with bpy::pure_virtual. You can provide default implementation for those if you want (see the wrapper section).
    • Currently, the only default implementation are for retro-compatibility with existing plugins. It should be pretty rare to need default implementations.
    • bpy::pure_virtual is not mandatory. Its only effect is changing the error message you get (in Python) if you call a non-implemented pure-virtual method.
  • If you expose a wrapper for a class that can be extended in python, and the original interface has protected member-functions or variables, you need to bring those in the public scope of the wrapper to expose them to python. By convention, protected methods should start with a _, e.g. _parentWidget().
  • Some classes need to expose Qt-specific members in python. This cannot be done using bpy::bases since Qt classes are not exposed through boost::python. You can use the Q_DELEGATE for those (see FAQ).
  • If you expose a (member-)function that has in-out parameters (a non-const reference or pointer), you might need to modify the actual signature (see below).

Be careful when returning objects to pythonr, specifically regarding the return_value_policy. There are many examples in pythonrunner.cpp. In particular QWidget* uses a bpy::return_by_value policy even if we actually reference existing objects. This is due to how we interface with sip.

Here is a mini-example:

bpy::class_<MyPluginWrapper, bpy::bases<IPlugin>, boost::noncopyable>("MyPlugin")

  // Define a method that inheriting classes need to implement:
  .def("methodToImplement", bpy::pure_virtual(&MyPluginWrapper::methodToImplement))

  // A basic method that does not require an implementation and returns the global IOrganizer, so 
  // we use reference_existing_object:
  .def("organizer", &MyPluginWrapper::organizer, 
      bpy::return_value_policy<bpy::reference_existing_object>())

  // Expose a protected member. MyPluginWrapper should have brough `protectedMethod`
  // in the public scope.
  .def("_protectedMethod", &MyPluginWrapper::protectedMethod)
 
  // Returning a QWidget*, we need to use return_by_value.
  .def("_parentWiget", &MyPluginWrapper::parentWidget, 
      bpy::return_value_policy<bpy::return_by_value>())

  // Delegate QObject stuff, e.g. if MyPlugin defines Qt signals.
  Q_DELEGATE(MyPlugin, QObject, "_object")

If a function takes, for example, an int& to be modified, you need to modify the signature. The usual way of exposing such functions to python is to make the function return the int instead of modifying it. If the function already returns something, you can transform it to a tuple, or allow returning either the original return type (if the int& was not modified) or a tuple.

Note: If the reference argument can be modified in python (e.g. the argument does not need to be re-assigned), you can pass it to python using boost::ref.

A full example of this is the IPluginInstallerSimple::install method, you can check its implementation. Below is a mini-example:

// We want to expose: double Foo::bar(QString, int&) const;

// If Foo needs to be extended in Python, we need a wrapper:
class FooWrapper: /* see below */ {
public:
  virtual double bar(QString q, int& i) const override {
    // We will allow Python method to return either a double, if "i" was not modified, or both
    // a double and the new value for i - Do not forget to register converters for both the
    // tuple and the variant:
    using return_type = std::variant<double, std::tuple<double, int>>;
    
    // We call the python method (see the proxypluginwrappers section):
    auto result = basicWrapperFunctionImplementation<FooWrapper, return_type>(this, "bar", q, i);
    
    // We use std::visit and update i (if modified) and return d:
    return std::visit([&](auto const& t) {
      using type = std::decay_t<decltype(t)>;
      
      // The python function returned only d, so i is not modified:
      if constexpr (std::is_same_v<type, double>) {
        return t;
      }
      // The python function returned (d, i):
      else if constexpr (std::is_same_v<type, std::tuple<double, int>>) {
        // Retrieve i:
        i = std::get<1>(t);
        return std::get<0>(t);
      }
    }, result);
  }
};

// When exposing the bar, we need to also return a tuple:
bpy::class_<FooWrapper>("Foo")
  // We use a lambda converted to a function-pointer (+) - If this was not a member function, we would 
  // not have the first argument:
  .def("bar", +[](FooWrapper *foo, QString q, int& i) {
     // Call the original foo:
    double d = foo->bar(q, i);
    
    // Return a tuple containing both the original return value (d) and the argument (i):
    return std::make_tuple(d, i); 
  })
  ;

proxypluginwrappers

These two files contains wrapper for all the plugins defined in uibase. Since a wrapper needs to delegate all methods, even the ones from the parent IPlugin, a utility macro COMMON_I_PLUGIN_WRAPPER_DECLARATIONS is provided.

Here is a typical wrapper declaration for a plugin (in proxypluginwrappers.h):

// The class should inherit the actual plugin type and the corresponding wrapper. If 
// the actual plugin type does not inherit MOBase::IPlugin, the wrapper should also
// inherit MOBase::IPlugin (see IPluginDiagnoseWrapper).
class IPluginShinyWrapper: 
    public MOBase::IPluginShiny, 
		public boost::python::wrapper<MOBase::IPluginShiny> {

  // Add Qt declaration for the plugin:
  Q_OBJECT
  Q_INTERFACES(MOBase::IPlugin MOBase::IPluginShiny)
	
  // Add declaration for common plugin methods:
  COMMON_I_PLUGIN_WRAPPER_DECLARATIONS

public:
  // Add a static className, for logging purpose:
  static constexpr const char* className = "IPluginShinyWrapper";
	
  // Bring get_override:
  using boost::python::wrapper<MOBase::IPluginShiny>::get_override;
	
  // If the parent plugin has constructors, bring them here:
  using IPluginShiny::IPluginShiny;
	
  // If the parent plugin has protected methods, bring them here (required
  // to be able to expose them with bpy::class_):
  using IPluginShiny::superProtectedMethod;
	
  // Add implementation for all pure-virtual methods:
  virtual int isShiny() const override;
  virtual void darken() override;
};

And here is the typical definitions (in proxypluginwrappers.cpp):

/// IPluginShiny Wrapper

// Define the common methods:
COMMON_I_PLUGIN_WRAPPER_DEFINITIONS(IPluginShiny)

// Define the overriden methods. The `pythonwrapperutilities.h` header defines
// multiple utility functions so you should use them instead of raw `get_override`
// for better exception handling.
int IPluginShinyWrapper::isSiny() const {
    return basicWrapperFunctionImplementation<IPluginShinyWrapper, int>(
		    this, "isShiny");
}

void IPluginShinyWrapper::darken() {
    basicWrapperFunctionImplementation<IPluginShinyWrapper, void>(this);
}

There are (currently) 4 functions available in pythonwrapperutilities.h:

  • The first basicWrapperFunctionImplementation overload is the basic one that will call the python method and try to extract an object of the given return type. If the python method is not found, a MissingImplementation exception is thrown, and if an error occurs, PythonError or UnknownException are thrown.
  • The second overload is similar except that it takes a reference to a bpy::object in which the result of get_override will be stored. This is very useful if you want to keep the returned python object alive. See the uibasewrappers section or the FAQ for more details on why you need this.
  • Two basicWrapperFunctionImplementationWithDefault overloads are provided. Those can be used similarly to basicWrapperFunctionImplementation except that instead of throwing a MissingImplementation if the python method is not found, they will call a default method that you can provide.
    • The two overloads (Wrapper * and Wrapper * const) are provided because the default member function can be const-qualified. It does not really matter for the other functions in this header since get_override does not care about const-qualification.

gamefeatureswrappers

These two files contains wrappers for the game features. Note that unlike proxypluginwrappers, these also contain a registerGameFeaturesPythonConverters that register the actual wrappers.

You can modify existing wrapper as you would modify a wrapper for a plugin but if you add a new game feature, you need to add it to the MpGameFeaturesList list at the top of gamefeatureswrapper.h. This is required to get proper extraction from and to the the map of game features of the IPluginGame plugins.

uibasewrappers

This file contains a few wrappers that are neither plugins nor game features. If you use one of the existing wrapper in this file (or if you create a new one and use it), you have to be careful: you need to keep the initial Python object alive to be able to delegate calls to python.

This issue is not present with plugins or game features because plugins are kept alive in PythonRunner::m_PythonObjects and game features are held by the actual game plugin.

If you do the following in C++:

// ISaveGame is wrapped using ISaveGameWrapper in uibasewrappers.h
ISaveGame* getSaveGame() {
  // Simply call the python function:
  return basicWrapperFunctionImplementation<MyWrapper, ISaveGame*>(this, "getSaveGame");
}

And in python you implement getSaveGame:

def getSaveGame(self):
  # We return a new object!
  return MySaveGameImplementation()

The code will most likely not work. Why? Because as soon as you get out of getSaveGame, you lose the initial Python object that holds the ISaveGameWrapper*. You can still use the returned object, but any attempt to access the MySaveGameImplementation python object will fail. In particular, trying to use get_override to call a method overriden in MySaveGameImplementation will result in a missing implementation exception.

The only way to prevent this is to store the actual bpy::object somewhere, e.g. in a member variable of the C++ wrapper. See for instance the implementation of SaveGameInfoWrapper::getSaveGameInfo or SaveGameInfoWrapper::getSaveGameWidget.

The proxy project

Note: This sections is only partially complete since I did not dig into it that much as it's working fine and doesn't require change that often.

The ProxyPython is the main plugin class. The main thing to know about it:

  • It is the ProxyPython that will look through the plugins/ folder and find the python plugins (.py file or folder containing a __init__.py file) and pass them to the python runner for instantiation.
  • It can be disabled / enabled by the user.
  • The user can specify an external python implementation. This has probably not been tested for a very long time and is probably quite useless now that python is shipped with MO2.

FAQ

1. Why is MO2 throwing an exception when I try to create a type inheriting one of MO2 class?

Note: This should probably also be in the plugin creation tutorial.

This often happens if you forget to call super().__init__() with the right arguments. Even if the list of arguments is empty (as in the example), it must be called:

class MySaveGame(mobase.ISaveGame):
    def __init__(self):
		    super().__init__()  # Mandatory!

2. Why are my get_override calls failing with MissingImplementationException when the Python class has the right methods?

In order for get_override to work properly, a reference to the initial python object must exists at any time. This can be on the python side (e.g., by having an attribute), or on the C++ side. For easier plugin creation, I would recommend storing on the C++ side.

Some C++ examples are:

  • The PythonRunner implementation that holds boost::python::object for all the created plugins.
  • The IPluginGame that holds (on the python side), a dict object containing the game features.
  • The SaveGameInfoWrapper (game feature) that holds the widget (m_SaveGameWidget) but also all the saves that were created (m_SaveGames).

3. Why are QString and QVariant converters not registered using register_qclass_converter?

register_qclass_converter is used to register conversion for Qt class that have PyQt equivalent. Since PyQt uses standard Python str instead of QString, we need to use a custom converter.

While PyQt does have QVariant, it is not very convenient since Python developer would have to manually cast to QVariant, so instead we use a custom converter that can create a QVariant from a multitude of python types such as int, str, List[str], etc.

4. What is Q_DELEGATE? And how do I use it?

Q_DELEGATE is a macro that can be used within a bpy::class_ declaration, e.g.:

bpy::class_<IDownloadManager, boost::noncopyable>("IDownloadManager", bpy::no_init)
    .def("startDownloadURLs", &IDownloadManager::startDownloadURLs)
    .def("startDownloadNexusFile", &IDownloadManager::startDownloadNexusFile)
    .def("downloadPath", &IDownloadManager::downloadPath)

    Q_DELEGATE(IDownloadManager, QObject, "_object")
    ;

In this case, we indicate that we want to expose the QObject interface for IDownloadManager. The Q_DELEGATE macro will:

  • Create a __getattr__ method that is used by Python to delegate attribute lookup to QObject when the attribute is not found directly in the IDownloadManager python class.
  • Create a _object method to access the underlying QObject.

It makes it possible to do the following in python:

dm = ...  # Instance of IDownloadManager

# We can connect signals declared in the C++ class:
dm.downloadComplete.connect(lambda i: print("Download {} complete!", i))

wm = ...  # Instance of ISaveGameInfoWidget

# We can call QWidget method on a ISaveGameInfoWidget object:
wm.setLayout(QtWidgets.QHBoxLayout())

Most of the QObject interface has no reason to be exposed, so the only cases where you should need Q_DELEGATE would be when:

  • You need to expose Qt signals to python - this is the case for IDownloadManager and IModRepositoryBridge.
  • You need to expose a class that inherits QWidget. If you do not use Q_DELEGATE in this case, python developers will not be able to call the QWidget method on objects of this class.

5. Why is it not possible to do bpy::bases<QObject>?

I will explain the reason for this here, but you should see the FAQ item above for troubleshooting.

QObject (or any Qt class) is exposed in python using sip, while everything in MO2 is exposed using boost::python. When doing bpy::bases<QObject>, boost::python does not find the PyTypeObject that corresponds to QObject since it is not exposed through a bpy::class_ declaration (registering a converter for it is not enough). It is possible, by playing with internal boost::python stuff, to make boost::python find the PyTypeObject for QObject but... that is not sufficient.

boost::python and sip create classes using their own meta-classes. For boost::python, it is Boost.Python.class. And all classes created by boost::python inherits Boost.Python.instance which is the "top" boost class. Unfortunately, it is not possible to do inheritance between classes that have different meta-classes in python, so it is not possible to inherit both QObject and Boost.Python.instance. The only way would be to provide our own meta-class, but this is not possible with boost::python.

6. Why is the tr method not exposed to avoid having to declare it manually on the python side?

One issue with QObject.tr in PyQt5 is that the context is dynamic, i.e. if class B inherits class A, strings declared in class A will not be translated by an instance of B since the context is the dynamic object (not the static one like in C++), see, https://doc.bccnsoft.com/docs/PyQt5/i18n.html.

It would be quite easy to provide tr since all plugins inherit IPlugin:

.def("tr", +[](bpy::object obj, const char *str) {
    std::string className = bpy::extract<std::string>(
		    obj.attr("__class__").attr("__name__"));
    return QCoreApplication::translate(className.data(), str);
})

...but the issue is the same, since className will be the name of the actual class, not the class containing the strings. It could be possible to go up the class chain to find the first available translation, but I am not sure that it is worth the hassle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment