Enable shared library build on Travis CI

[tkelman]

Seeing some issues locally with linking the bulldozer shared library, let's see if Travis reproduces it.

[fgvanzee]

@tkelman I assume that answers your question?

That said, there may still be an issue on bulldozer systems, since Travis gives us no guarantee of the architecture on which the tests will be run. (Last I checked, the systems were haswell.)

[tkelman]

My issue may have been caused by building for a series of configurations in a row, something like:

cd build
for i in reference dunnington sandybridge haswell bulldozer piledriver carrizo; do
  mkdir -p ../build$i
  cd ../build$i
  ../configure $i
  make -j4 BLIS_ENABLE_DYNAMIC_BUILD=yes
done

(though also doing a cross compile with some extra flags, so the problem may be coming from there too)

[tkelman]

Okay this might just be a case of my cross-compiler not knowing what to do with bulldozer-specific FMA instructions?

mkdir -p buildbulldozer
cd buildbulldozer
../configure bulldozer
make -j4 CC=x86_64-w64-mingw32-gcc CPICFLAGS="" BLIS_ENABLE_DYNAMIC_BUILD=yes
# ... snip
Archiving lib/bulldozer/libblis.a
Dynamically linking lib/bulldozer/libblis.so
obj/bulldozer/frame/3/gemm/bli_gemm_cntl.o: In function `bli_gemm_cntl_init':
/home/tkelman/github/blis/buildbulldozer/../frame/3/gemm/bli_gemm_cntl.c:121: undefined reference to `bli_dgemm_4x6_FMA4'
obj/bulldozer/frame/3/trsm/ukernels/bli_gemmtrsm_l_ukr_ref.o: In function `bli_dgemmtrsm_l_ukr_ref':
/home/tkelman/github/blis/buildbulldozer/../frame/3/trsm/ukernels/bli_gemmtrsm_l_ukr_ref.c:81: undefined reference to `bli_dgemm_4x6_FMA4'
obj/bulldozer/frame/3/trsm/ukernels/bli_gemmtrsm_u_ukr_ref.o: In function `bli_dgemmtrsm_u_ukr_ref':
/home/tkelman/github/blis/buildbulldozer/../frame/3/trsm/ukernels/bli_gemmtrsm_u_ukr_ref.c:74: undefined reference to `bli_dgemm_4x6_FMA4'
obj/bulldozer/frame/ind/ukernels/gemm/bli_gemm3m1_ukr_ref.o: In function `bli_zgemm3m1_ukr_ref':
/home/tkelman/github/blis/buildbulldozer/../frame/ind/ukernels/gemm/bli_gemm3m1_ukr_ref.c:312: undefined reference to `bli_dgemm_4x6_FMA4'
/home/tkelman/github/blis/buildbulldozer/../frame/ind/ukernels/gemm/bli_gemm3m1_ukr_ref.c:312: undefined reference to `bli_dgemm_4x6_FMA4'
obj/bulldozer/frame/ind/ukernels/gemm/bli_gemm3m1_ukr_ref.o:/home/tkelman/github/blis/buildbulldozer/../frame/ind/ukernels/gemm/bli_gemm3m1_ukr_ref.c:312: more undefined references to `bli_dgemm_4x6_FMA4' follow
collect2: error: ld returned 1 exit status
make: *** [lib/bulldozer/libblis.so] Error 1

[matthew-brett]

I'm getting the same kind of errors, but building with MIngw-w64 compilers under Cywgin, and for sandybridge (but not reference):

Dynamically linking lib/sandybridge/libblis.so
obj/sandybridge/frame/3/gemm/bli_gemm_cntl.o:bli_gemm_cntl.c:(.text+0x23f): undefined reference to `bli_dgemm_asm_8x4'
obj/sandybridge/frame/3/gemm/bli_gemm_cntl.o:bli_gemm_cntl.c:(.text+0x246): undefined reference to `bli_sgemm_asm_8x8'
obj/sandybridge/frame/3/gemm/bli_gemm_cntl.o:bli_gemm_cntl.c:(.rdata$.refptr.bli_sgemm_asm_8x8[.refptr.bli_sgemm_asm_8x8]+0x0): undefined reference to `bli_sgemm_asm_8x8'

This is an old Opteron machine.

[tkelman]

Are your Cygwin packages up to date (x86_64-w64-mingw32-gcc -v)? Were you using the same set of make flags I was? I'm running on a new skylake, but I'd be a bit surprised if the build machine made a difference. Compiler/binutils versions certainly could though.

[matthew-brett]

Yes, I just installed Cywin this evening:

$ x86_64-w64-mingw32-gcc -v
Using built-in specs.
COLLECT_GCC=x86_64-w64-mingw32-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-w64-mingw32/4.9.2/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: /cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-gcc-4.9.2-2.x86_64/src/gcc-4.9.2/configure --srcdir=/cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-gcc-4.9.2-2.x86_64/src/gcc-4.9.2 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/mingw64-x86_64-gcc --htmldir=/usr/share/doc/mingw64-x86_64-gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-w64-mingw32 --without-libiconv-prefix --without-libintl-prefix --with-sysroot=/usr/x86_64-w64-mingw32/sys-root --with-build-sysroot=/usr/x86_64-w64-mingw32/sys-root --disable-multilib --disable-win32-registry --enable-languages=c,ada,c++,fortran,lto,objc,obj-c++ --enable-fully-dynamic-string --enable-graphite --enable-libgomp --enable-libquadmath --enable-libquadmath-support --enable-libssp --enable-version-specific-runtime-libs --with-dwarf2 --with-gnu-ld --with-gnu-as --with-tune=generic --with-cloog-include=/usr/include/cloog-isl --with-system-zlib --libexecdir=/usr/lib
Thread model: win32
gcc version 4.9.2 (GCC)

My make flags are the same as yours:

  make -j4 CC=x86_64-w64-mingw32-gcc AR=x86_64-w64-mingw32-ar \
      RANLIB=x86_64-w64-mingw32-ranlib CPICFLAGS="" BLIS_ENABLE_DYNAMIC_BUILD=yes; \

[njsmith]

Curiously, I can reproduce the same error with a mingw-w64 v5.3.1 gcc, but not with a native linux v5.3.1 gcc.

[tkelman]

Are you configuring from in tree or a separate build dir?

[njsmith]

Using a separate build dir in both cases.

[tkelman]

Actually now on Ubuntu 14.04 I am seeing the bulldozer linker errors from just

./configure bulldozer
make -j4 test

[matthew-brett]

Does it error on a build that isn't parallel?

[tkelman]

It does, that was the next thing I tried

[tkelman]

Found a fix for the bulldozer issue, see PR #40. Meanwhile, adding testing of the shared-library build configuration to Travis is a good thing, right?

[fgvanzee]

@tkelman: Regarding adding the testing of the shared library build to Travis, I think we should only do that if the shared library is built properly (ie: we expect it to work well), and to be honest, I'm not an expert in properly building a shard library. I used to think it was a trivial variation on static archives, but after some conversations with @Maratyszcza, I'm less sure. Perhaps Marat can chime in here.

[tkelman]

These days, -shared is most of it. What is your concern?

okay fine maybe this is a bit on the complicated side after all

[Maratyszcza]

@tkelman Symbol interposition makes everything more complicated.

[njsmith]

The main thing is that you want to make sure that only the symbols that you intend to make public actually are public. (I guess this is what Marat means about interposition?) The generally recommended best practice is to compile with -fvisibility=hidden (which makes all symbols default to private), and then go through and annotate the intended public api with __declspec(dllexport). (That's the traditional windows spelling; there's also a traditional gcc spelling, but I think gcc treats them both identically anyway and I can't be bothered to look it up while on my phone.)

The other thing to worry about is abi stability and versioning.

That said, I'm not sure why it would be bad to start testing that the shared libraries function technically, even if they're not yet ready for public consumption. For example, this would make it easier to tell that future modifications like an -fvisibility=hidden patch didn't break things :-)

[insertinterestingnamehere]

It's not entirely clear to me either what is meant by "symbol interposition". That said, making the default visibility hidden and then explicitly exporting the public API seems like a great approach to me.

[Maratyszcza]

Symbol interposition means that publicly exported symbols can be overriden by symbols in previously loaded dynamic libraries or the executable, i.e. if your internal names are not too unique, horrible things may happen. I agree that -fvisibility=hidden is the right way to deal with it (albeit I'd prefer -fvisibility=internal for performance reasons), but then someone needs to explicitly mark all API functions as public.

[njsmith]

I think one attractive approach would be to start by marking the BLAS/CBLAS symbols as public in shared-library builds, and hiding everything else. That way we could get started on the shared library infrastructure as one piece of work, and then over time expand the exported API incrementally as and when we're able to review the newly public functions and make a commitment that they can be supported over a longer time period in a shared-library stable-ABI style. (Of course the full API would still be available from static BLIS builds, which don't need to worry about ABI stability in the same way.)

[tkelman]

I don't see why you would want to hide BLIS' full API. There isn't that much code written against it relative to classic Fortran BLAS or CBLAS APIs yet, but hiding it makes that code harder to write and use in environments where shared libraries are preferred.

From experience, the Fortran and CBLAS API's definitely need some mechanism of renaming the symbols if you're using ILP64 interfaces with shared libraries if you ever expect to have other code that you did not personally compile loaded into the same address space. It's relatively easy to do this renaming if you start from a static library and a list of symbols, but on OSX it requires a non-standard GPL-licensed tool (http://www.agner.org/optimize/#objconv) that isn't part of binutils, so this should maybe not be BLIS's job to handle immediately.

[insertinterestingnamehere]

For LLP64 platforms, it'd be ideal to use size_t and ptrdiff_t as the default sizes for all the interfaces other than BLAS and CBLAS. At least that way code that uses the new interface could use integer sizes that match the ones used for sizes by the OS. Fortunately, for now, that's not much of a compatibility break since Windows builds of BLIS are so rare. #42 tentatively makes that change.

[njsmith]

I don't see why you would want to hide BLIS' full API.

Because I don't know if Field is ready to commit to maintaining ABI compatibility on the full API, or even necessarily which parts of the API are even intended to be public. And in fact I would like for him to break ABI compatibility :-) Right now all the strided array functions the BLIS API measure strides in units of sizeof(T), while I would prefer that they measure strides in bytes -- it's strictly more general, and since numpy allows for arbitrary byte strides it would reduce the amount of faffing around we have to do later if using the full BLIS API. (It does mean the packing functions would need some provision for unaligned memory access, but they can do a better job of that then anyone else -- the only other alternatives are to make temporary copies of whole arrays before passing them to BLIS, or else to fall back on a naive for-loop implementation.)

From experience, the Fortran and CBLAS API's definitely need some mechanism of renaming the symbols if you're using ILP64 interfaces with shared libraries if you ever expect to have other code that you did not personally compile loaded into the same address space.

Right, the standard BLAS/CBLAS dgemm_ symbols and such should always use 32-bit integers in a "standard" API, with optionally 64-bit versions called dgemm64_ or whatever. In a perfect world BLIS could even export both dgemm_ and dgemm64_ simultaneously, given that it will generally be using 64-bits internally regardless. This probably wouldn't even be too hard to set up, I guess, since there's already a single macro that generates dgemm_/sgemm_/etc. simultaneously; all you'd have to do would be to extend it to insert 64-bit versions too while it's at it.

[tkelman]

If we want to start worrying about API and/or ABI stability then throw on an SONAME for now, but don't hesitate to keep changing things as long as it gets annotated as such. Most of these stability issues apply equally to linking against a static BLIS library, so this is as much a general documentation issue about what's public and/or stable vs what isn't. If it ever becomes a non-theoretical concern that say Julia and NumPy might both want to have incompatible shared-library versions of BLIS loaded simultaneously (and what consequences that would have w.r.t. embedding one language in the other), then we can consider getting really fancy with symbol versioning if we really need to.

[njsmith]

@tkelman: yeah, bumping the SONAME every time anything changes is also an option. But this does still require that someone notice when an ABI-breaking change is made (which in turn probably means that we really do want to export only the actually-intended-for-public-consumption functions, not every single internal function as well). And I'm not so much worried about NumPy and Julia clashing, as e.g. Debian getting cranky and refusing to ship BLIS if the SONAME changes every 2 weeks.

[tkelman]

The likelihood of that right now is why Debian shipping BLIS is still very much theoretical too.

[njsmith]

@tkelman: It's not that theoretical -- one of NumPy's Debian maintainers was playing with this last week. And I think Debian would have no problem at all with shipping a version of libblis.so that just exported the BLAS/CBLAS symbols, possibly alongside a libblis-dev package that contained the .a file with the full API.

[tkelman]

Fair enough. But if I or anyone else wanted to write a Julia package BLIS.jl (I actually started in this direction several years ago, but there's not much there of interest right now) that leverages the full BLIS API, I'll need a shared library. A Debian package with a libblis.so built without the full API (publicly visible) would be useless for such a Julia package, I'd have to check for symbols and skip loading that package and force a compilation from source, making the Julia package take considerably longer to install.

You could argue that maybe the extended BLIS API should perhaps live in a separate shared library than the [C]BLAS compatibility layer, so the Debian maintainer could set up update-alternatives with libblas symlinks, but prevent applications from being able to link against BLIS' extended API via -lblas (which would then break if you were to run update-alternatives and switch to netlib or openblas).

[njsmith]

But if I or anyone else wanted to write a Julia package BLIS.jl [...] that leverages the full BLIS API, I'll need a shared library. A Debian package with a libblis.so built without the full API (publicly visible) would be useless for such a Julia package [...]

Ah, fair enough. But if the BLIS API is not stable, then a Debian package is also useless, isn't it, because you'll want to pin the particular version of BLIS that your wrapper is targeting?

You could argue that maybe the extended BLIS API should perhaps live in a separate shared library than the [C]BLAS compatibility layer, so the Debian maintainer could set up update-alternatives with libblas symlinks, but prevent applications from being able to link against BLIS' extended API via -lblas

Unfortunately, I think this is impossible given ELF's symbol namespace model: if A.so is linked to B.so, and myprog is linked to A.so, then myprog gets equal access to all the symbols in A.so and B.so (the only exception being that symbols in A.so can shadow symbols in B.so, but that's no help here).

Okay, so here's what I'd suggest as an incremental path forward for enabling shared-BLIS:

• Mark BLAS/CBLAS symbols as public
• Use -fvisibility=hidden by default
  • For the BLIS.jl build, you can override this with a custom build configuration using -fvisibility=default, so it'd still be a single switch to enable what you want
• Incrementally mark more symbols as public as they're reviewed and the BLIS project feels able to start providing some sort of ABI backcompat guarantees (even if the guarantee is just "if we change this then we'll notice and bump the soname")

[tkelman]

then a Debian package is also useless, isn't it, because you'll want to pin the particular version of BLIS that your wrapper is targeting?

Only because Debian maintainers have a habit of removing old versions of things much sooner than downstream projects are actually ready to use new API-breaking versions, rather than making old and new versions simultaneously installable. If there are frequent API breakages, debian packaging does more harm than help.

If an API is useful, and designed to be utilized by downstream projects, then exporting it by default has value - otherwise it should just be static anyway. API stability and symbol visibility are mostly orthogonal; visibility is about what is intended to be used right now, stability is about how that contract should evolve over time (and isn't really specific to shared libraries).

[fgvanzee]

Hey everyone, I just wanted to follow up (a few years later, granted), that this issue, which was summarized and restated in #248, has now been largely resolved via #303 and its antecedents. Thanks for your participation, since it helped lead to this improvement.