Building and Testing C++ Python Modules with Bazel
While working on REL, I learned a lot about Bazel and its usage as build system in open source projects. In a series of blog posts, I will share these learnings and describe different approaches. Today’s blog post addresses the integration of C++-based Python modules into Bazel and the modelling of dependencies towards the corresponding Python-based tests.
As part of REL’s Python integration, I created a
rel_py which includes the core REL C++ library and the necessary Python binding (using the ingenious pybind11 framework). If the library is built as dynamic library, the resulting librel_py.so file can directly be imported in every Python script via
import statement. It took me quite some time, though, to model the dependency between a
py_test rule and the mentioned
cc_library. My goal was to add an integration test to Bazel, which uses REL within Python, to read a toy model and test the basic functionality, like accessing all type instances, checking the API etc.
The integration test itself is a simple python script, that imports librel_py.so and interacts with the API. I wrapped it into a
py_test rule. At the moment, it is not possible, though, to model a dependency (
deps) in Bazel from
cc_library, as Bazel only allows dependencies towards rules from the
py_ family. Therefore I tried the
data - attribute, which allows specifying arbitrary dependencies, e.g. to test data. Unfortunately, with this approach, I was not able to specify the correct import paths for the Python runtime. During test execution, Python always complained, that the module that shall be imported cannot be found.
After searching on Stackoverflow and the Bazel bugtracker, I finally figured out the following approach, to get the dependencies right: Apparently it is necessary to define a dummy
py_library first, which is modeled as dependency within the
py_library then uses the
data attribute to point to a
cc_binary rule, which is located in the same folder as the two py-rules, and is actually a copy of the original
cc_library. The disadvantage of this solution is definitely, that the Bazel model is partially duplicated. Nevertheless, the obvious advantage, it now works and I can run an integration test via
bazel test, that builds the Python binding library/binary of REL and runs a Python script, to test the functionality.
My solution in Bazel can be found here: https://github.com/sscit/rel/blob/main/relpy/test/BUILD
Bazel Bugtracker issues related to this topic, that contain additional details: