SO answer on storing symlinks: zip - archiving symlinks with python zipfile - Stack Overflow (summary: zipInfo.external_attr |= 0xA0000000).PIP issue with some discussion (closed, and refers to this thread): Symlink (and other) handling of archives.symlink-extraction BPO: Issue 27318: Add support for symlinks to zipfile - Python tracker (summary: if stat.S_ISLNK(member.external_attr > 16)).In this case, the wheel is guaranteed to be intended for local (and therefore platform-specific) use, so as long as project build back-ends don’t try to put symlinks into a wheel when they’re unsupported/disabled on Windows, everything should be fine? Hijacking this thread after the discussion of one implementation for PEP 660 (wheel-based editable installs) where symbolic-links are used ( first mention).
(On linux, the simplest thing is to just dlopen("path/to/libarrow-wheel-14.so") then any future requests for that shared library will be automatically satisfied without going through the normal library search.) Provide a Python API that lets other packages request the library be available at runtime, doing whatever linker finagling is necessary to make that work.Provide a Python API that lets other packages query for build time configuration (linker flags, include dir, etc.), as well as the resulting wheel dependency (maybe if the third-party package is built against pyarrow 14.2.3, then that means its Install-Requires should include pyarrow >= 14.2).Give your library a unique name that designates a specific ABI as shipping inside a Python wheel, like libarrow-wheel-14.so or similar.In that case, the best approach I’ve been able to come up with on Linux is: From your post, I assume you also want the library to also be usable by other packages. Auditwheel gives each vendored library a unique mangled name, which works well for that use case. IMO if you want to ship shared libraries in a wheel, then you should take the search problem seriously, and not rely on the linker’s naming scheme for system libraries. For example, if the user also has a system copy of libarrow.so, and you’re relying on the linker recognizing well-known names like this, then your package might end up using either the system’s copy or the wheel’s copy basically at random, which sounds like a recipe for obscure segfaults. Including libraries like this inside a wheel is a dubious thing to do… I can see how it might solve some problems in the short term but in the long term I think you’ll hit unsolvable problems. Wheels have a different and incompatible way of handling library searching. The whole scheme of symlinks like libarrow.so → libarrow.so.14.0.0 is tightly coupled to how the Linux system linker searches for libraries. Ideally we would like to keep the symlinks inside those wheels. Currently, those symlinks become copies, which doubles the file size (ending up at 50+MB). libarrow.so symlinks to libarrow.so.14.0.0 ). raise BadZipFile("Corrupt extra field %04x (size=%d)" % (tp, ln))Ĭontext: we package rather large.python/cpython/blob/master/Lib/zipfile.py#L466 I notice the zip on the machine I’m using also stores mtime/atime/ctime and uid/gid in the “extra” field but zipfile.py doesn’t currently interpret those. I had happily subclassed Zipfile in wheel to check / generate file hashes… zipfile is a very nice module. The link target just goes where the file contents would otherwise go. AFAICT it’s pretty straightforward: os.stat(filename, follow_symlinks=False).st_mode goes into ZipInfo.external_attr to store the “is a symlink” flag.