I’ll write another post, expanding from a single tweet of mine:
If you release code and are not interested in helping distributions package it… don’t release code. Honestly.
It might sound harsh and mean, but I mean it, and I think it’s for the best.
You might have a number of different reasons to work on opensource projects, but you can basically find that the people who write open source projects and then don’t care about distributions and other kind of integrators, tend to be those developers that only work for their own ego. I don’t mean they write bad code, sometimes they might actually be the best programmer, but their dislike for distributors is poisonous for the whole opensource environment.
Why do I say this? Well, the problem is that if you release good opensource code, whether it’s packaged by distributions or not, it might very well happen that some other developer will find and use it, that’s what you want, when you release it opensource, isn’t it? Now when that project is no longer a library or backend tool, but rather an user facing tool that the developer wants to see used, there will be requests to package it. But once that happens, you might be stuck with some backend dependency that cannot be clearly packaged. In that situation, the whole frontend can’t be packaged, and that means that the distribution’s users will end up suffering.
This is getting even worse with gems, especially when the most insightful comment in my latest post comes from an user that seems to proclaim himself a troll (or at least the nick suggests strongly so). Indeed, trying to excuse Ruby gems developers for not caring about our needs (which usually boil down to having tests around, a clear repository, the ability to rebuild documentation and tests without bringing in a whole lot of gem-building tools – jewelwer, hoe, and so on – and most importantly code that works and tests green, which is by far not obvious when dealing with gems), is just saying that you care about open source for your own feel good, rather than for the whole environment.
This gets worrisome when it comes to Ruby stuff, especially because of the ties with Rails. I guess the majority of the gems are developed for usage with Rails, which means web-facing code; if they are developed without caring about the opensource environment and their possible users, it would be silly not to expect that at least a part of them are so insecure they are likely to cause vulnerabilities in code that is actually published for the general usage.
So I think I just decided to make two personal rules, that I wish were widespread:
- if you do not plan on helping distributions to get your software packaged, don’t release it opensource;
- if you plan on making your software opensource, do not use software that distributions can’t package.
Packaged distributions are a good thing.That said, I’d like to see more software that doesn’t *need* to be packaged by somebody else in order to fit in well with a development or production environment.When the concept of distributions came about, we didn’t have a lot of tools we have now. With things like distributed version control (git/mercurial) we can refer to a specific version/release of a project by its tag using a url.Why “package” it into something like a tarball, or gem, or deb, or rpm? Why can’t the developer include enough information directly in the source repository for somebody to pull a tagged release directly from there, discover its dependencies, and build/test/install/run it?
>if you do not plan on helping distributions to get your software packaged, don’t release it opensource;>if you plan on making your software opensource, do not use software that distributions can’t package.Why not, who are you to tell other people what to do. If you don’t like their software don’t use it, don’t promote it don’t even criticize it, just make as if didn’t exist at all. Maybe those developers have better things to do than make their projects packagable, but they’ve released the code what more do you want. If you want top of the line support maybe you should pay for it or waste your own time to make it possible. Open source even gives you the possibility to fork a project, if you are so philosophically/technically opposed to their decisions. If people get problems with the software later is their own fault, they should of know better than to use a suboptimal product. You can’t always choose the choices but you can always choose.
To jorge: Diego has done more packaging, forking, maintaining, bugfixing and loads more than most so-called open source developers. The rules presented here are very simple and boil down to simplest form: “don’t be an arse-hole”.Here, let me go one-by-one on your comment:-Not liking software: I don’t like java, so if I ever made a distro, who would use it? Stuff doesn’t get packaged because somebody likes it or not – it is packaged because it is required/requested by USERS not developers- Developers have better things to do: so do I, but if I need something, I would like it better if it’s nicely packaged to fit into my system and work flow. Nobody likes to run stuff from their /home, and most definitely nobody likes to pull shit load of dependencies just for the sake of getting some small app to compile (and run). So if you don’t have the time to maintain what you create, why create it at all? Have you ever bought a car? Try driving it without tanking, because it is bothersome and costs money. Forget tanking – do not change oil – it is even more bothersome and costs a lot more. How long will that car be operational? The same rules can be applied to development – you can say that you developed something but you don’t maintain it, so nobody uses it and thus it ends up as a ego-stroker only in your portfolio.- pay for support? you kidding? packaging is done FOR the developer. If not for the packagers, developer would be bothered by users to make packages, and that wouldn’t be nice if the said developer treated users like those gem developers treat packagers – imagine all that foul mouthed language with repeated question “why you do stuff that nobody can use”… If my software ever gets users and packagers, I swear I would try to help packagers as much as possible(and paying them, donating, flattring are just some examples of support), to spread my software further and further boost my ego.- forking because devs are arse-holes: a lot of that is bad mojo for users. I don’t want to state examples, but let’s just say that having 20 possible apps,having roots in the same baseapp, and all those are distro-bound forks, creates so much confusion that even between users there would be argues about bugs that in some forks are present and in some not.- choice of suboptimals: this is just load of bull crap on your part: Is it my fault that IE is buggy? Or is it the creators’ fault? Now I can choose whatever I want, but what about decade ago? Not so fun. Now imagine the same with gems: we have one version, from some arse-hole of a dev, who doesn’t give a single f*** to bug reports, but keeps adding/removing whatever the hell he wants and we are supposed to be ok with that, because he gives this to us – but what happens when fork gets a lot more users than the original? Problems. Not only for devs, but users suffer too.summing up: don’t be an arse-hole and don’t encourage others to be.Diego: you sir are my hero, how can You put up with this load of crap is simply amazing.
I understand Diego’s frustration, he got some unqualifiable responses about packaging fixes from upstreams.I think he want to say: even when you do not interested in repackaging, or you do not know how can you be interested in it, do not reject the packages contact request, pay him an attention.
errm… package*r*s contact request, of course.
As a developer, I’m interested in talking to packagers to learn more about the process.What information about my software does the packager need to make his job easy? Can I make that information available in a simple, neutral manifest directly in my source repo?Here’s what I’m playing with:A file called “Dependencies.json” containing an array of software projects which the project in question depends on. Each entry contains a url for a project’s homepage, plus other potentially helpful information that may be either used or ignored. Here’s an example entry describing a dependency of my project on PostgreSQL:
"postgresql": { "homepage": "http://www.postgresql.org/", "GitRepository": "git://github.com/postgres/postgres.git", "GitCheckout": "refs/tags/REL9_1_1", "license": "http://www.postgresql.org/about/licence", "build-steps": [ { "command": "./configure --prefix={{target_dir}} --enable-thread-safety --with-bonjour --with-openssl --with-ossp-uuid--with-includes={{target_dir}}/include --with-libraries={{target_dir}}/lib", "must-include": "creating GNUmakefile" }, { "command": "make", "must-include": "All of PostgreSQL successfully made. Ready to install." }, { "command": "make install", "must-include": "PostgreSQL installation complete" }, { "command": "make --directory=contrib/uuid-ossp" }, { "command": "make --directory=contrib/uuid-ossp install" }, { "env": { "PATH": { "match-item": "{{target_dir}}/bin" }}} ], "init-steps": [ { "command": "initdb", "must-include": "Success" } ] },None of this is set in stone of course, and it’d be helpful to collect feedback from people who might use this information.The objective: automate, automate, automate.A software package should be easily useable by a variety of operating system distributions, development environments, deployment and testing tools, etc. If it adequately describes itself using a suitable set of vocabularies, I think that’s a huge step in the right direction.For some of the terms in this JSON, I started with the Description of a Project vocabulary ( http://trac.usefulinc.com/doap ) as inspiration. I think the format could be further improved by looking to recent W3C efforts to represent RDF data using JSON ( http://www.w3.org/wiki/JSON… ).
Simply put… I write a ton of useful code, which I then use for my job. I give it away freely… with the caveat that I may or may give a crap about what you do with it. Also, I may give a crap about helping you with it… It was to scratch my itch. If I have the time to contribute the little handles that make it easier for you to use… great, if not, you have no right to bitch ( unless you’re paying my bills).It’s this kind of sanctimonious crap that makes me wonder what happened to Gentoo.Open source work, like all volunteer work, is a charity… if giving makes you bitch… then either you’re giving too much, or you’re giving for the wrong reasons.
Diego, I suggest to ignore these assholes who say: “I release a code for free and you can’t blame me for anything”Please don’t say it yourself! (I know you never will). You are doing amazing work. I know how frustrating it can be when instead of simply coping an old ebuild we actually need to re-write it, as a simple example.Opensource – is a community of friends who are trying to work together making the software perfect and suitable for everyone. It’s not a garbage bin to where people can through away their code.
Seems like some people miss the point widely. *It’s not a favour to anyone if you make something available you don’t give a crap about fixing.* When that happens you’re “giving” just because you want to feel good about it and not thinking about it the least!If you really cared about community and opensource, you’d know sometimes it’s better _not_ to publish something in the open, as it’ll cause more trouble than it solves. Because if it is used by someone else who don’t know better, it’ll end up used as a basis for castles that could crash down in a breath.I have seen _tons_ of libraries reimplementing @$foo@ before, because other implementations are “complex and slow” … and then implementing their target in a way that causes trouble (because it doesn’t interact correctly with other pieces of software, misinterprets its input, or produce bad output).And if you think that explaining you that is “sanctimonious crap”… I seriously don’t want your code or help.You’re not doing a charity this way, you’re increasing the noise, and lowering the SNR.
The problem with “if you don’t like it, don’t use it” is simple: dependency hell.Even if I don’t want to use Joe’s crappy library, a company I go to work for might be using two or three other packages which depend on Joe’s crappy library.Once something starts getting used, getting rid of it can be like escaping the tentacles and goo of the matrix ( http://billswadleyblog.beyo… ).
The house that you purport to build was not built on the “perfect” stones that you lament the absence of.Some people produce what they have time to do… others, who care more for the ‘castles’ cut the stone, mortar, place, and polish it.You complain that nature has yielded stones that are difficult for you to cut.Sounds like someone who does not know their place in the ecosystem.
Given that I probably have spent for the ecosystem more time than you would ever care to, I can point you that you still haven’t got it.Somebody is giving away stones just because they can. Stones that in too many cases are so crumbly that a rainstorm can flush them away. And they don’t care to at least coat them to avoid that, they don’t care if you point out that they could at least plug the hole the rain is entering from.If you _really_ can’t see how a project that is dropped there and left to rot is dangerous for the whole opensource environment, *you really ought to go away*, as your contributions would be poisonous.Might be harsh, but it’s the reality.Just like a simple codedrop from a big company can’t be said to be “truly opensource”, a codedrop from a developer who has written something and won’t care for it to fit properly in the whole design isn’t “opensource”, it’s “open-ego-source”. You’re just proving my point.
Sometimes I believe that people who do open-ego-soure have never gotten proper education about the software processes. Since first programming classes in my university, everyone is required to: document code, dependencies and usage. Even when requiring only basic “stuff that everybody has”, it is mandatory to document every dependency or your project won’t pass. What I seen in this matter is that most people don’t like to document anything, then after classes like “Team oriented development” those who do not like to document their code get kicks not only from professor but from fellow students also.Now in regards to companies: I see a bit of correlation between documentation and code quality. This means that if code and dependencies are documented, it is easier to keep projects alive and well.Combining this info, from my point of view: undocumented code means to me that it is crappy in quality and author is a little behind in education, thus software is no-good to use. But what to do when it is the only one available to specific task (like mongo driver here)? Diego suggestions are good – cooperate with devs, so their code gets better and more maintainable. If they don’t want to – then why did they release the code: to stroke their ego, otherwise somebody else would write specific piece of code and wouldn’t be an arse about packaging and maintaining code.About noise to sound ratio: think of this situation – some arse releases early implementation of X. Now if you want to use X you must use arse-code, and you depend on it. Your app is great, gets released and all, but when packaging, someone like Diego gets stuck with arse-code. By this time there may or may not be other solutions and changing one dependency may lead to loads of problems inside main app and thus App is great, but because of arse-code dependency, it may not see the light in other distros, or even in no distro at all. This is why people should make an have standards like those posted by Diego.
Every now and then I happen to stumble upon these articles of yours Diego and while they are always preset with a good technical details I can’t help but thinking that something else is wrong here.This time, I feel the user bears more of the blame than either side cares to let on. Most likely they do so unwillingly or unknowingly.I am a user. Every now and then I happen to have a need and the very reason I’m interested in reading these rants of yours is because you, sir, have spent lots of effort making sure others needs can be fullfilled. From my perspective though, the there’s a lot of knowledge about what one need, but not alot of knowledge about how to satisfy it. There is also an incredible lacking in terms of information about suitability of a given solution. Once I’ve found a solution (any solution) I’m usually content. Be that solution horrible for you or not.Now, I do not know what solutions are good and what solutions are bad for you, as I am but a mere user. There is no encouragement for me to look into alternatives that may be less horrible for the packager. There is no warning or informational statement that a user would see and care about that lets the user know of the troubles a packager goes through.The user sees a need for more packagers on the gentoo website pretty much every time they go there, but that appears to users to be more of a “lack of hands” situation than a situation where the hands we have are burdened by the choices users make.Now, I’ve just been typing as I go along and this may or may not hold any value for the discussion, but somehow I’d still like to see the users get more informed. I mean, if I user needs library A which has bad coding practices and high-cost maintanance and there is no library B that does the job (and does it better), then *knowing* it has issues might actually get someone to start up that new project for library B _before_ the world is entirely dependent on library A.
@Jonas: That is a huge part of the problem, judging code quality is one of the more difficult things, so trying to fix it from the user side seems a bit unlikely.Though what often works is trying to do something unusual: Try building it with clang on FreeBSD, …, for example, and see what happens. It often is a clue to the overall quality.If it accepts files as input, try throwing some random mess at it.Or use a valid file and run it through a fuzzer (which randomly corrupts the file) and see what happens.I admit none of this is trivial, but it should be something “ordinary” users are probably able to do with a bit of hand-holding the first time.Of judging availability and quality of documentation can give some clues, too, though that’s not quite the same thing.
Oh, I forgot, while not specific to packaging: Tools like valgrind (looking for memleaks), pmccabe (to check for ridiculously complex functions) or static analysis tools etc. can help, too, but tend to require a bit more knowledge to interpret the results.Of these valgrind IMO works rather good, since getting lots of failures usually means they never used that tool.They might have a good reason to be aware of a failure and keep it in some rare case, but most of the time it’s a sign to avoid the software.A bit more on-topic with packaging, attempting to manually reimplement a “make install” or remove the result of a “make install” might give another clue how much of a pain it is.
I come at this from a python background and I’m curious how python eggs compare to ruby gems. I’m often irritated by python’s pypi which seems to reinvent the package manager. Sure, it’s nice and super convenient when easy_install works getting me the latest version regardless of the ubuntu I happen to be running. However, I can never quite shake the feeling that it’s just a dirty gate-way drug.And then, in the python world, we have this notion of a virtual_env where people are expected to have their own little world totally aside from the system packaging. So, it has this great appeal to make your app totally isolated from the details of the ambient system. But, again, VMs are only getting cheaper these days. Did we really need a programming language layer implementation of jails or VMs?So, Diego, would you mind educating me a bit on why a python egg is better or worse than a gem in the regards of this post?
@jbmohler: Start by trying to compile python using a cross compiler, then worry about other features of this platform 🙂
Diego and some commenters totally miss the point. OSS is not always for users but for programmers. I read a lot of source code illustrating nice ideas or algorithms and never cared about if this specific implementation comes in a package or even compiles. It`s like complaining about the wrong colour of a book which doesn’t fit in your shelf.> everyone is required to: document code, dependencies and usageI prefer a clear and straigthforward API backed up by a usable type system (linear types, dependent types…). But most people never learnt to program or design software and favour drawing funny UML diagrams, writing redundant comments and creating crappy software in ancient languages.
I’m always astonished how people mix up the idea of free software with open source, a term made only for business reasons. For example take comment #9:> If you really cared about community and opensourceReally, “opensource” should never be used to describe the philosophical ideas Stallmann and the FSF has formed over the past 25 years. Opensource has nothing to do with community whatsoever. The OSI came to life to cut all ideology from free software and create a well-fitting development model for businesses. If you want to support free software and the ideas behind it, discard opensource and make the difference clear. IMHO.
Unfortunately for you, you also don’t have something clear: when I say _opensource_ I mean opensource, I don’t mean Free Software. Because it’s not opensource just because the code is there, but also because it can be improved all together:
If you just want to see people as either “big businesses” and “GNU zealots”, feel free, but I’m really neither.
And by that quote, a lot of “open source” software completely misses the point as some of commenters here.One can sum up this as open source is a way of achieving perfection – if that’s not your goal you’re not into open source, but rather showmanship: “Look, here’s MY code, and YOU inferiors can use it, but I won’t do anything about it, because I already got bored with it”.This is even evident by some project admins on source forge – my company needed one simple package couple days ago, but required specific changes to it in order to support newer systems. There was no response, but after we send our patch the admin said that he won’t incorporate it as he’s not interested in maintaining this package any more. So, we wrote our own in few days, that could be spent better if project owner gave it about 10 mins of his time. (To clarify – we wrote package only for stuff we care about, not general purpose like the one mentioned, so our can not replace the mentioned one in all purposes).What’s even worse – in a lot of places the mentioned package was recommended by people just because it is there, not because it is any good (in fact after analysis it became apparent that it’s ridden with bugs and has loads of hard-coded stuff and seemingly wrong assumptions).Now my question: how can anybody convert ego-stroker into open source author?
> One can sum up this as open source is a way of achieving perfectionIn my opinion “perfect” software comes with a proof that it behaves correct, never found one in distfiles. Also there are a lot of bad APIs and bugs in OSS and in fact besides trustworthiness there is no difference in quality to closed source.> but rather showmanshipThe other way around: “Look, I had a specific problem. Here is my solution, feel free to use the idea or the code. Maybe you have to update some library code, but you should get the idea and it should save you some time.” And then comes the ego-packager/user: “It’s OSS! You have to package it for distribution DONTCARE and answer all my questions and apply my patches!!1eleven”.> that could be spent better if project owner gave it about 10 mins of his timeSo there are two possibilities: a) you haven’t offered enough money or b) you’re to stupid to fork a project.
I recently wrote a couple of panel applets for Gnome 2.x – partly because of limited documentation it was a slow process, but I got a couple of reasonably well-written projects in the end.I then set to work packaging the projects, first with autotools and then as .deb files. The autotools packaging (the first time I’d used it) took about 3/4 of the time needed to write the project, and the .deb packaging about the same time.I ended up with something much more convenient to use, which I can recompile easily as Gentoo is updated, but I can see why developers sometimes skimp on packaging.Will
The goal for all concerned is better code with less work. We need to think of this as a manufacturing process. Even more importantly, we need to pay more attention to the “metaprocesses.” By this I mean the processes that are necessary or helpful to creating and improving open source code.1. Some developers don’t give a crap. If their project is important enough, let’s find a way to support them either financially or by volunteer effort to remediate their code.2. There are best practices, but they aren’t well known. It’s important to document these and help others learn them. For most developers, writing code is ad hoc. We need guides, both documentation and mentors. (This applies to all code, not just open source.)3. While some developers don’t care, or are too egotistical to cooperate, we need to encourage egoless programming (see http://en.wikipedia.org/wik…. I believe most developers will cooperate if approached in a positive way. Diego’s approach (“don’t release it”) isn’t optimal.4. If some code uses dependencies that don’t follow the best practices, egoless programming review (if accepted by the developer) can suggest better alternatives.5. We need to recognize that there is a need for volunteers who improve the quality of open source software and explain how they do it. Just as there was a need for distributions, which has been filled by many volunteers, there is now a need for quality improvers. (Part of quality is making the code distribution-friendly.)6. The problem is real. The solution is more cooperation, not ultimatums.Respectfully,Howard
There’s also another side of the problem, the converse one.Certain distributors (e.g. Debian’s) apply modifications to their packages that affect major functionality of the package or fail to update the package even when notified by the developer, while at the same time directing users to report errors to them.Remember the Ion debacle?I understand fixing bugs, but adding features to an obviously active project should be done via upstream and especially not against upstream’s wishes.The proper thing to do instead is a fork instead of the patch. Complete with a new name as well.Now, there’s also another issue. There are projects that are simply badly packaged because of lack of time or testing. Lambasting people for releasing undercooked code is pretty much the worst thing possible in OSS; the maxim should rather be either “patch wanted” or “maintainer wanted”.Not releasing the software means that people who are able to improve the software but don’t have the time/resources to write it from scratch cannot do so.Most of the time, small project suffer just from lack of warm bodies. https://en.wikipedia.org/wi… is a powerful tool and when there’s more than one target, packaging and build system suddenly starts to require thought.
Thank you for your rant! (though I’m only seeing it now)I don’t mind people publishing something and realizing that they don’t want to maintain it. I‘ve given up on a few projects myself. What I consider important is to clearly state that fact: “please don’t use this as a base. If you don’t adopt it, it’s dead.”