Draft documentation for Linux system installs

[julian-smith-artifex-com]

I have a draft documentation page for Linux system installs, see: https://ghostscript.com/~julian/docs-PyMuPDF/packaging.html

Please let me know if you have any comments or suggestions.

[dvzrv]

Thanks for providing an overview of the build process!

I believe pip will not be used by any downstream, but instead, more low-level tooling to cater to a PEP517-style workflow.
In your case this is based on build, installer, setuptools and wheel.
For reference, you can have a look at how I am assembling the pieces for the package on Arch Linux: https://gitlab.archlinux.org/archlinux/packaging/packages/python-pymupdf/-/blob/e638e70e1ad7729b74ab0823e81a7611d459bf57/PKGBUILD#L54-82

[julian-smith-artifex-com]

Thanks for this. Yes, i did see that, and i noticed the use of python -m installer to install from a wheel.

I'll try using python -m installer to also do a real system install in our github action.

[mjg]

Hmm. I thought I had shared builds working with mupdf 1.23.7 and PyMuPDF 1.23.8 over the weekend, but mupdf 1.23.8 changed the picture again (as will PyMuPDF 1.23.9). In particular, I'm wondering:

• Why introduce an SONAME without setting it (-Wl,-soname,libmupdf.so.$(FZ_VERSION_MINOR).$(FZ_VERSION_PATCH))? AFAIU, we need that for proper runtime linking, and the softlink comes with it for free (the way we do things on Fedora).
• Which shared library from which target (install, install-shared-c, install-shared-python) is used for what, in particular for which PyMuPDF implementation (legacy/rebased)?
• Why do we need to build without thirdparty stuff if we use shared libs?
• What do we lose when we don't use bundled lcms2 and freeglut?

[julian-smith-artifex-com]

Apologies for breaking things.

• I think i omitted -Wl,-soname,... because it's not required on openbsd and my Linux sysinstall test was working fine; from what you say, this should be easy to fix on Linux so i'll have a look at that tomorrow.

• install-shared-c is required for pymupdf-classic. install-shared-python is required for pymupdf-rebased. Note that the "Packaging for Linux distributions" docs and scripts/sysinstall.py are for the rebased implementation only, and we'll be abandoning the classic implementation in the next few weeks probably.

• I'll get back to you later about thirdparty libraries.

[julian-smith-artifex-com]

Why do we need to build without thirdparty stuff if we use shared libs?

Previously, distro maintainers have said that they want MuPDF and PyMuPDF to use the system's libraries, not MuPDF's internal thirdparty libraries. So for shared installs, MuPDF's Makefile insists that USE_SYSTEM_LIBS is yes, to reduce the amount of variation we have to deal with. We would certainly consider relaxing this constraint if it is a problem.

What do we lose when we don't use bundled lcms2 and freeglut?

I guess that the MuPDF people would say that they are more in control of their thirdparty libraries than if they were having to deal with all the variety of libraries that are installed out in the world. And MuPDF is tested only with MuPDF's thirdparty libraries.

Building PyMuPDF in a non-standard configuration will sometimes cause some changes to behaviour that we might not want to spend large amounts of time fixing. There's one example of this currently in the test suite - the test_color_count test fails due to slightly different behaviour of MuPDF's thirdparty/libjpeg library compared to that in some system installations. Our approach here is to ignore the test on sysinstalls, rather than spend vast effort trying to cope with the minutely varying behaviours of different jpeg libraries.

Having said all of that, i think that lcms2 and freeglut are not used by PyMuPDF.

That came out a lot more verbose than i was hoping. I hope it all makes sense.

[mjg]

Thanks, we'll figure this out.

Please don't read my comments (about weekend etc) as a complaint, it's a data point of when things changed. For example, I set up soname versioning myself and am happy that we have an official so version scheme now. Much better.
I'm also happy to get rid of bundled libraries - so happy in fact that I tried (mupdf) with USE_SYSTEM_LIBS and system libraries only, which failed. And indeed, if I look at the source I see that USE_SYSTEM_LIBS enforces some system libs but not all of them (lcms2, mujs, extract). And it's fine with me once I know :-)

About the change: I added the soname flags and with mupdf 1.23.7, this give me the .so and the .so.SOVERSION (maybe thanks due to the linker or our packaging setup). We (Fedora) ship the .so in the devel package and the .so.SOVERSION in the libs package. During build time of a program, the linker reads the soname flag from the .so (which specifies SOVERSION). Consequently, during dynamic linking at runtime, the system looks for a library with that name and finds the .so.SOVERSION. This is so that developers can link against the .so, and the system can link against .so.SOVERSION at runtime. At least as far as I understand ...

Now, I tried to get mupdf shared libraries to work first, with PyMuPDF legacy, and then switch to PyMuPDF rebased (and 1.23.9), and you helped me understand that different libraries are involved and (?) that they are different from mupdf's own shared library. All of them are meant for external use, not just internally by mupdf resp. the two PyMuPDF implementations, right?

[julian-smith-artifex-com]

Thanks for the reassurances.

I've added -Wl,-soname,... to the link command in the latest MuPDF branch master when building on Linux, so hopefully this will be doing the right thing. It specifies the full leafname, so an example link command is:

cc ... -o build/shared-release/libmupdf.so.24.0 ... -Wl,-soname,libmupdf.so.24.0 ...

I don't know much about the specifics of Linux's sonames, but with this change things still work for me in a Linux system install.

All of them are meant for external use, not just internally by mupdf resp. the two PyMuPDF implementations, right?

Yes, the libraries libmupdf.so and libmupdfcpp.so are for public use; they provide the MuPDF C and C++ APIs. The MuPDF Makefile install rules will also install the MuPDF C and C++ API header files.

Hope that helps.

[mjg]

So, as an update: I seem to have packaging working for Fedora, that is mupdf shared against system libraries (mostly) and PyMuPDF rebased implementation, passing most tests.
This is with PyMuPDF 1.23.21 (plus JM_image_reporter patch) and mupdf master. I had tried with mupdf 1.23.10 initially but needed to backport too many commits from master related to soname etc. Ist there a release from master around the corner, possibly?

As for the tests: I get spurious "Fatal Python error: Aborted" in test_insertimage.py and/or test_inserpdf.py on various architectures, but not always. Could this be some kind of reentry/multithread problem with parallel tests or a timing issue?
E,g. this copr build

Edit: I noticed that PyMuPDF links against both libmupdf and libmupdfcpp. Is this as intended?

[mjg]

So, I think I git the necessary backports from mupdf master to mupdf 1.23.10 to build mupdf as above, as a base for PyMuPDF rebased implementation (which is why I write here).
But: This works everywhere except on aarch64 where I get strange compiler errors about std::cerr << " args=" << args; . I don't get them with mupdf master. Am I missing another backport?

[julian-smith-artifex-com]

[Apologies for taking a while to reply, i'm away quite a lot at the moment.]

I'm glad you're making progress here!

This works everywhere except on aarch64 where I get strange compiler errors about std::cerr << " args=" << args; . I don't get them with mupdf master. Am I missing another backport?

Do you have MuPDF commit d888952ba9c38a0a "scripts/wrap/cpp.py: avoid compiler error in debug diagnostics on Android." ? It looks closely related to the compiler error you're seeing.

Ist there a release from master around the corner, possibly?

There was talk recently about moving MuPDF to 1.24 (which will be basically forked off of master) quite soon, but i don't know any reliable details.

I noticed that PyMuPDF links against both libmupdf and libmupdfcpp. Is this as intended?

Yes, PyMuPDF uses both libmupdf and libmupdfcpp. I think the only place we use the raw C API is in some compiled and optimised routines - the C++ wrappers incur a small overhead because they convert setjmp() exceptions into C++ exceptions for each individual function call, whereas with the C API one can call a sequence of functions inside a single fz_try() block.

[mjg]

Wonderful! I had filtered out "Android", of course, but that was it. Thanks!
I may or may not try to cut down the list of cherry picks. I guess this still counts as "1.23.10 with build system fixes". If PyMuPDF builds against this (like against master) it will make the cut for Fedora 40.