Bloody upstream

Please note, this post is likely to be interpreted as a rant. From one point of view it is. It’s mostly a general rant geared toward those upstreams that is generally impossible to talk into helping us distribution out.

The first one is the IEEE — you might remember that back in April I was troubled by their refusal to apply a permissive license to their OUI database, and actually denied that they allow redistribution of said database. A few weeks ago I had to bite the bullet and added both the OUI and the IAB databases to the hwids package that we’re using in Gentoo, so that we can use them on different software packages, including bluez and udev.

While I’m trying not to bump the package as often as before, simply because the two new files increase the size of the package four times. But I am updating the repository more often so that I can see if something changes and could be useful to bump it sooner. And what I noticed is that the two files are managed very badly by IEEE.

At some point, while adding one entry to the OUI list, the charset of the file was screwed up, replacing the UTF-8 with mojibake then somebody fixed it, then somebody decided that using UTF-8 was too good for them and decided to go back to pure ASCII, doing some near-equivalent replacement – although whoever changed ß to b probably got to learn some German – then somebody decided to fix it up again … then again somebody broke it while adding an entry, another guy tried to go back to ASCII, and someone else fixed it up again.

How much noise is this in the history of the file? Lots. I really wish they actually wrote a decent app to manage those databases so they don’t break them every other time they have to add something to the list.

The other upstream is Blender. You probably remember I was complaining about their multi-level bundling ad the fact that there are missing license information for at least one of the bundled libraries. Well, we’re now having another problem. I was working on the bump to 2.65, but now either I return to bundle Bullet, or I have to patch it because they added new APIs to the library.

So right now we have in tree a package that:

  • we need to patch to be able to build against a modern version of libav;
  • we need to patch to make sure it doesn’t crash;
  • we need to patch to make it use over half a dozen system libraries that it otherwise bundles;
  • we need to patch to avoid it becoming a security nightmare for users by auto-executing scripts in downloaded files;
  • bundles libraries with unclear licensing terms;
  • has two build systems, with different features available, neither of which is really suitable for a distribution.

Honestly, I reached a point where I’m considering p.masking the package for removal and deal with those consequences rather than dealing with Blender. I know it has quite a few users especially in Gentoo, but if upstream is unwilling to work with us to make it fit properly, I’d like users to speak to them to see that they get their act together at this point. Debian is also suffering from issues related to the libav updates and stuff like that. Without even going into the license issues.

So if you have contacts with Blender developers, please ask them to actually start reducing the amount of bundled libraries, decide on which of the two build systems we should be using, and possibly start to clear up the licensing terms of the package as a whole (including the libraries!). Unfortunately, I’d expect them not to listen — until maybe distributions, as a whole, decide to drop Blender because of the same reasons, to make them question the sanity of their development model.

20 thoughts on “Bloody upstream

  1. I am one of those enthusiastic Blender users you speak of. Actually I’ve found that I cannot rely on the state of Blender in the tree as it is often outdated.I myself have been recommending other Blender users on Gentoo to just download the binary from the website and builds from Graphicall (the site hosting custom community builds of Blender). This recommendation goes against the Gentoo ethos but is based on my own experience as a daily user and that most avid Blender users are artists at heart who doesn’t really care about these technically issues: they just want a tool that works.I have sent out an email to some Blender contacts referring to this blog post and hopefully will get some information about it soon.

    Like

  2. I installed blender once and tried to learn to use it. I got a bit on the way and some months later when I wanted update my system it (blender) failed to compile overnight leaving half of my system updated, the other half not so updated which in turn resulted in hours of frustration and the decision to just give up on using blender alltogether.It’s probably an amazing piece of technology but even if I disregard the issues mention in this post it’s only been a pain for me to use.

    Like

  3. hi diego. i just talked to the blender devs at #blendercoders @ freenode. i do not know whether you already tried that or not but it would be probably helpful if you could tell them the details of the blender packaging problems so that we could keep this great software packaged in gentoo. thank you for all the work you do.

    Like

  4. We really welcome feedback of package maintainers in our developer channels. That’s all open, accessible and very friendly. On the front page of blender.org ‘get involved’ it will send you to the main list:http://lists.blender.org/ma…The issues as mentioned here are mostly new for me. Why not try to at least discuss it with us? There’s a 1000+ users/devs on that list, many of them enthusiast linux users.But to quickly tackle the mentioned issues:- Is Bullet license missing? We will check.- The patches – can you submit these?The suggestion to drop libraries (which?) I cannot understand. We benefit a lot from a wide selection of open source projects in CG and 3D. Libraries we use get selected and evaluated carefully, and will only be included it that provides essential technology we cannot (or should not) develop ourselves.Compared to other 3D programs, Blender’s binary size is microscopic :)Lastly: I would always recommend 3D artists to run the Linux binaries from blender.org anyway. These are stable and maintained, and should run fine for Gentoo users as well.

    Like

  5. It’s not Bullet whose license is missing — check the other linked blog posts, and there is a note with the upstream bug on the library itself. Bullet has instead some patches that add extra APIs.I really don’t care whether you drop the bundled library or not, and I’m not suggesting that you drop the _dependency_ — but distributions don’t like bundled libraries (again, there are all the links from here).Also, if you don’t care about it packaged, I really don’t even want to get near discussing this, and that’s the main reason why I’m not submitting patches or anything. “You would always recommend” to not use a distribution, so why should we waste time trying to fix your program to behave decently enough to be provided by distributions?So basically with the last line you’re explaining exactly why my feeling is to just remove it from Gentoo and not care about it.

    Like

  6. Ton, I’ll try to upstream the patches soon, surely doesn’t make me much willing to spend more time after hearing that our work on making easy and safe using blender in our distribution is considered somehow wrong.

    Like

  7. Look – we all are trying to do good, and we all try to do the best we can for our jobs and to help our own communities best. Let’s try to recognize that this results in a difference in perception, and not consider what I say as an insult or attack on the good work linux distributions do.From our side, we work with users who need Blender, especially for work or jobs they have. We provide them with bi-monthly releases, own daily builds and own build systems – for stability and reporting reasons mostly. (And it’s a cross-platform project).Both our communities might overlap some, but probably differ a lot. I can’t speak on your behalf, nor would I suggest you to do your work better or differently. I just express our willingness to help out and do whatever is possible to make other source distributions possible. But it remains your job, your responsibility, your community.A rational observation could be “Blender is currently too complex to maintain as a source package distribution, it’s too much work to do that, with too little benefit for our users. Our users can get it all via blender.org. Therefore I suggest to drop it”.

    Like

  8. Flameeyes: I humbly suggest getting rid of the flames and trying to discuss constructively.

    Like

  9. Man I follow the blender project in a daily basis and I never see any disrespect from them. This is the first time I come here and I am sure will be the last, so unnecessary rudeness. I think Debian will be my next distro being blender my 3d application of choice. have a nice day everyone.

    Like

  10. wow such harsh words even though Ton suggested he is ready to work out. good thing i moved to arch linux.

    Like

  11. Ton, all the experience we have had with Blender up to now always has been along the lines of “we don’t care about changing this for our builds, so we won’t change it” — one of the patches we’re applying relates to the security of auto-executing scripts, and that one we still have to apply manually for exactly that reason.And I’m pretty sure that Debian has more or less the same problems, so for all those that are saying “I’m moving to Debian”: feel free, you’re not going to find much of a difference, since they are going to use our patch for libav support anyway, and if we drop Blender, they’ll either start to lag behind or just decide it’s too much work for them as well.Basically everything you’ve been saying up to now is that you don’t desire to be a distribution-friendly package. Okay, it is your call, but don’t accuse us of flaming, if we state outright what the problems we have are.Campbell at least clarified some of the bundled libraries problems (libredcode is not really bundled, eltopo – the one that has no license – is no longer used, which means that we really have to get rid of 2.64a at least in our mirrors). But for all the other problems, we have no answer.I’ll be honest and frank: I don’t give a crap about those who complain that I’ve been rude. I’ve been maintaining the ebuild without anybody paying me to, and without any vested interest in it. Half of the bugs submissions we get is “it was released, bump it now!”, and most of the others are “I don’t give a crap about Gentoo policies, use this ebuild” (which is not going to happen).I don’t gain anything to please people that don’t care about my work, just like you don’t expect to gain anything from having your software available in distributions’ repositories. So I’ll be happy to remove the package, and make sure users are on their own to deal with it. It certainly makes our life much easier.

    Like

  12. @Flameeyes,With CMake – glew and openjpeg we have WITH_SYSTEM_GLEW, WITH_SYSTEM_OPENJPEG,I think it would be fine to add others WITH_SYSTEM_BULLET, WITH_SYSTEM_LIBMV for example (though Im not sure libmv is available standalone on many distros).Im not sure if there are any remaining licensing issues, you mentioned Eltopo has no license, but this is now removed from blender’s source.

    Like

  13. Is CMake now the preferred build system? Because that’s one of the problems, it’s hard to tell what should be used. Last time I tried using CMake, some features didn’t build at all, so we had to keep using scons (which is a pain just alone — CMake is far from perfect, but at last it doesn’t require as much smashing to work).libmv might not be available standalone, but that’s not the main issue — we can package it. The problem is when stuff like the patches to bullet happen: we then cannot rely on the packaged library anymore because it’s incompatible, and that’s our big issue right now.

    Like

  14. For Linux, yes, I think CMake is preferred on Linux, its used by Debian, Fedora and Arch.The way we have CMake configured (for the most part) is it detects available system libraries and disables some if they are not available. – rather then failing as scons will.If there is something specific you can point at with CMake, I’ll check on it.You can explicitly enable packages to get around this, suggest checking the arch linux package:https://projects.archlinux….IIRC we are often using an un-patched bullet, however recently Erwin Coumans (the author of bullet), committed some changes to blenders bullet which are not in a releases version of bullet.- Just checked ‘extern/bullet2/patches’ and there are a few of them, we should try get these included in the next bullet release, so agree with you there, this issue just didn’t come up before because nobody asked us about using the systems bullet library.

    Like

  15. Thanks Campbell! — I’ll soon be trying to build with CMake and see if it now works as intended. I’ll also see to prepare some patches for what we need on the CMake side about the libraries that are not used from system now.If the patches come from Bullet’s author and they’ll be folded in on the next release, we’ll gladly apply them over the current version and be done with it — it’s just very difficult to follow the whole thing, when only a bunch of patches are to be found…As I said on the other post, a note like the one left by libmv on their bundled libraries would be terrific for us, as it’ll allow to assess what’s going on with a single look.

    Like

  16. “we need to patch to avoid it becoming a security nightmare for users by auto-executing scripts in downloaded files”There’s also a CMake flag to disable that which Fedora (in theory at least, filed a bug report) uses.I try to make sure that ‘cpack -G RPM’ always produces a valid package so there’s at least one person looking out for y’all poor, unappreciated package maintainers in Blenderland. True, I don’t actually publish my workarounds when issues do arise but I do monitor the mailing list and am more than happy to help where I can.I’m still not really clear on how this kind of stuff is expected to get resolved without trying to communicate *with* upstream, I don’t recall hearing anything about gentoo issues before now (though, honestly, I don’t read every message on the mailinglist). I suppose the plan was to pull the package and have users start a shitstorm on the blender side?

    Like

  17. Hi FlameeyesCommitted WITH_SYSTEM_BULLET,http://projects.blender.org…Erwin’s looking into build issues compiling with a system lib since its not working quite yet.On another topic, really prefer you guys didnt apply your own security patches, CMake’s ‘WITH_PYTHON_SECURITY’ option makes script-auto-execution opt-in rather then opt-out. Last time I checked on the way Gentoo handled this it was pretty much breaking functionality by disabling script execution in a way that wasn’t at all clear to the user how to enable again.Scripts are needed for many rigs for controlling bones in animation so would prefer linux distros dont cripple this feature.

    Like

  18. Campbell, thanks. I’ll make sure to get a snapshot instead of 2.65a then. And I’ll see to coordinate with our security team about the security patch, as I was asked before to keep it around (I would have dropped it already).Dan, no, we’re not expecting for users to “start a shitstorm”, but to start _caring_. If they wanted to get Blender back, they could have started _helping_ instead of just bitching at us because we’re slow to make sure it complies with our policies.Because as much as you wish to think that cpack solves every problem under the sun, DISTRIBUTIONS ARE NOT JUST BINARY PACKAGES. Distributions are integrated systems, including policies and testing. But given the kind of replies I’ve seen on bf-committers, stating that we don’t understand the need for bundled libs, I guess quite a few people have no idea what our work involves at all.

    Like

  19. hi diego,i have spoke with some nice developer of blender team. kaito, serfgo, dingto and other. plz go in irc or send them really request, in objectiv way and ask for what we need.they are about to take realy request

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s