|
|
Subscribe / Log in / New account

LCA: How to destroy your community

Did you know...?

LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net.

By Jonathan Corbet
January 18, 2010
Josh Berkus is well known as a PostgreSQL hacker, but, as it happens, he also picked up some valuable experience during his stint at "The Laboratory for the Destruction of Communities," otherwise known as Sun Microsystems. That experience has been distilled into a "patented ten-step method" on how to free a project of unwelcome community involvement. Josh's energetic linux.conf.au presentation on this topic was the first talk in the "business of open source" miniconf; it was well received by an enthusiastic crowd.

[Josh Berkus] If you are a corporate developer, you're likely to realize early on that free software development communities are a pain. They'll mess up your marketing schemes by, for example, taking the software into countries where you have no presence and no plans. They'll interfere with product roadmaps with unexpected innovation, adding features which you had not planned for the next few years - or, worse, features which were planned for a proprietary version. Free software communities are never satisfied; they are always trying to improve things. They tend to redefine partner and customer relationships, confusing your sales people beyond any help. And they bug you all the time: sending email, expecting you to attend conferences, and so on. Fortunately, there are ways to get rid of this community menace. All that's needed are the following ten steps.

#1 is to make the project depend as much as possible on difficult tools. He noted that most companies have no real trouble employing this technique, since it makes good use of the tools they have around anyway. Community-resistant projects should, for example, use weird build systems not found anywhere else. A proprietary version control system is mandatory. Even better are issue trackers with limited numbers of licenses, forcing everybody to use the same account. It's also important to set up an official web site which is down as often as it's up. It's not enough to have no web site at all; in such situations, the community has an irritating habit of creating sites of its own. But a flaky site can forestall the creation of those sites, ensuring that information is hard to find.

2: Encourage the presence of poisonous people and maximize the damage that they can create. There is a special technique to the management of these people which goes something like this:

  1. Take pains to argue with these people at length and to denounce them on the project lists.

  2. Eventually, they should be banned from the community by fiat; it's important to avoid any sort of community process here.

  3. The banned people will take their flames elsewhere. Follow them and continue to argue with them in those external sites.

  4. Eventually the community will complain about this behavior; respond by letting the poisonous people back in. Then go back to step 1 and do it all over again.

Properly managed, one effective poisonous person, according to Josh, can wipe out a community of hundreds.

3: Provide no documentation. There should be no useful information about the code, build methods, the patch submission process, the release process, or anything else. Then, when people ask for help, tell them to RTFM.

4: Project decisions should be made in closed-door meetings. An OK start is to have online meetings with very short notice, though, for best effect, they should be at a time which is inconvenient in the time zones where most community members are to be found. Better is to have meetings via conference call: that excludes about a third of the planet due to sleep requirements, and, for extra value, also excludes a number of people who are at work who might have been able to participate in an online meeting. Best, though, is to hold meetings in person at the corporate headquarters.

5: Employ large amounts of legalese. Working with the project should involve complex contributor agreements, web site content licensing, non-disclosure agreements, trademark licenses, and so on. For full effect, these documents should all be changed without notice every couple of months or so.

6: The community liaison must be chosen carefully. The optimal choice is somebody reclusive - somebody who has no friends and really doesn't like people at all. Failing that, go with the busiest person on the staff - somebody with both development and management responsibilities, and who is already working at least 70 hours per week. It's important, in this case, to not remove any of this person's other responsibilities when adding the liaison duty. It can also be effective to go with somebody who is unfamiliar with the technology; get a Java person to be the liaison for a Perl-based project. Or, if all else fails, just leave the position vacant for months at a time.

7: Governance obfuscation. Community-averse corporations, Josh says, should learn from the United Nations and create lengthy, complicated processes. Keep the decision-making powers unclear; this is an effective way to turn contributors into poisonous people. Needless to say, the rules should be difficult or impossible to change.

[Josh Berkus] 8: Screw around with licensing. Community members tend to care a lot about licenses, so changing the licensing can be a good way to make them go elsewhere. Even better is to talk a lot about license changes without actually changing anything; that will drive away contributors who like the current license without attracting anybody who might like the alleged new license.

9: Do not allow anybody outside the company to have commit access, ever. There should be a rule (undocumented, of course) that only employees can have commit rights. Respond evasively to queries - "legal issues, we're working on it" is a good one. For especially strong effect, pick an employee who writes no code and make them a committer on the project.

10: Silence. Don't answer queries, don't say anything. A company which masters this technique may not need any of the others; it is the most effective community destroyer of them all.

Josh concluded by noting that he saw all of these techniques employed to great effect by Sun. But Sun is far from alone in this regard. Josh has been told by a veteran of the X Consortium that they, too, had made good use of all ten methods at one point or another. Community-destroying skills are widespread in the industry.

But what if you have a different kind of company, one which wants to encourage and build communities? Doing the opposite of all of the above clearly makes a lot of sense. But, Josh said, it all really comes down to trust. A relationship with a development community can be seen like a marriage: one can spend years building it up, but one affair will kill the trust it is based on. Similarly, a company can lose half of its community in a weekend. Those who would avoid that fate must trust their communities and act in a way which will cause that trust to be returned.

Index entries for this article
Conferencelinux.conf.au/2010


(Log in to post comments)

LCA: How to destroy your community

Posted Jan 18, 2010 15:50 UTC (Mon) by ikm (subscriber, #493) [Link]

> Free software communities are never satisfied; they are always trying to improve things

An experience with a project of mine sums it up a bit differently: while 0.1% of the community is trying to improve things a bit indeed, 99.9% of them just whine actively and state that someone else should (which implies me, of course). Their reasoning is that "this would only take a couple of hours", "this would make the project more competitive", "imagine how neat that would be" etc etc, which looks nothing more than an attempt to manipulate me. Sure, I can understand this -- it's just the easiest thing to do. But it just really makes me feel bad each time another newcomer starts this crap all over again, and no one actually wants to do anything harder than that.

The whole experience on that front is that releasing stuff as free software only burdens you. To that end, I think selling it is a better idea: people pay you for the trouble instead of just whining for "moar free stuff".

I guess it depends on a project, of course. And if you do this for fun, that sure doesn't matter -- how could it? I just don't share our fellow editor's sentiments about the unconditional communities' awesomenesses. It really depends.

LCA: How to destroy your community

Posted Jan 18, 2010 20:59 UTC (Mon) by dlang (guest, #313) [Link]

those whiners you complain about are a combination of QA testers (finding bugs) and market research (suggesting new directions)

yes, you may only get 0.1% of people contributing, but if you have a good size community, that 0.1% can be more people than you have as employees. If you keep the community small, the number of contributers will also be small.

LCA: How to destroy your community

Posted Jan 18, 2010 21:09 UTC (Mon) by ikm (subscriber, #493) [Link]

Agreed. However, expanding the community that much would require a lot of publicity. That publicity could either be a result of money investment in an ad campaign, or due to an enormous general usefulness for everyone, so that the natural time to spread would be acceptable. The latter looks possible only for some of the projects, not for any. The former looks much more easy, but requires money.

LCA: How to destroy your community

Posted Jan 18, 2010 21:16 UTC (Mon) by dlang (guest, #313) [Link]

I would argue that publicity itself doesn't help. the program needs to be useful to people, if it's useful word gets around. you do need some publicity, but it's not expensive campaigns.

the real cost of growing a community is dealing with the 'whiners' when the community is small, bringing the new contributers up to speed, working with them to change their patches into something that's acceptable for the project, taking the 'works but is not quite right' patches and assigning someone to clean them up, writing documentation, etc.

when the community is small, these things will distract your paid people, but they are needed to grow the community. it's a long-term investment, and it doesn't always pay off, but when it does it can pay off in a big way.

LCA: How to destroy your community

Posted Jan 18, 2010 21:32 UTC (Mon) by ikm (subscriber, #493) [Link]

I agree again, but would notice that even if program is useful to people indeed (even if highly and extremely useful!), it won't get popular instant by itself. This can take years. With the right publicity, though, it can gain a lot of users in no time. Publicity means a LOT -- look at all of our beloved spammers out there and ask why do they exist ;) Remember the story of Firefox. Look at how fast Google's Chrome is gaining weight (and yet crashes all the time on my machine when I try it!)

Of course, if the program is an outright crap, no amount of publicity would hold the user base :)

LCA: How to destroy your community

Posted Jan 21, 2010 11:44 UTC (Thu) by mgedmin (subscriber, #34497) [Link]

I'm not sure it's a good example. Publicity may be the reason why people
try out Google Chrome (or Chromium), but usability is why they keep using
it. I may be extrapolating from one example here (myself), but despite all
the numerous bugs I find Chromium more usable than Firefox.

LCA: How to destroy your community

Posted Jan 24, 2010 19:36 UTC (Sun) by jospoortvliet (guest, #33164) [Link]

Building a community anew is very hard but there are ways of speeding it up.
If you're part of a larger community, say the KDE community, you can and
should call upon the help of others like the KDE promo team, the forum ppl,
the translators and others.

LCA: How to destroy your community

Posted Jan 19, 2010 0:20 UTC (Tue) by elanthis (guest, #6227) [Link]

This has been my experience to a limited degree as well. I'll admit,
sometimes I've even been one of the freeloading whiners. I'm willing to
bet most of us have, although we may not have thought of it as "whining"
when we did it.

Far too many users expect that because you released a project, you are
honor-bound to improving it any way the users want. It just doesn't work
that way. If I release code, I release it to be nice, not because I want
to be chained to it forever.

As a user, I actually miss being able to buy boxed copies of Red Hat or the
like for this kind of reason. Sometimes I really do have major issues that
need to be solved that I myself can't reasonably solve (e.g. the way my
whole desktop gets corrupted every 60 minutes or so on the Open Source r600
graphics drivers), and I've got absolutely no way to even get a developer
to look at fixing these bugs instead of dumping more time into developing
new code that I can't use because the code it's built on is too unstable.
It's frustrating. Yes, I could contribute (assuming you don't consider
detailed bug reports "contributing"), but as a full-time student with an
almost full-time job and a handful of other projects that I could just as
easily work on in Windows, sinking the time into learning how to debug and
fix a graphics issue in massive stack including X, Mesa, and DRI just isn't
feasible. I'm stuck. I won't tell the developers that they have to fix my
issue -- they don't, they don't owe me a damn thing -- but it's easy to see
why users can be so demanding. I literally can't use Linux for some things
I actually need to do because of a silly bug, was forced to purchase Win7
(the first time I've bought a copy of Windows in almost a decade, excluding
the copy that came installed on my last laptop), all just to do my school
work (graphics programming). I can only imagine how other users that don't
have the know-how or drive to file bug reports are reacting to Linux as a
whole when running into frequent corruption on common graphics hardware.
It _is_ hurting the project.

That said, users need to learn how to work with developers, especially in
Open/Free projects. At the end of day, the user got what he got for free,
and if it just doesn't work and nobody will fix it and the user can't/won't
fix it, the user just needs to go use something else. Most projects really
don't care if the user leaves, and the user needs to understand that.
Nothing is as ineffective as "I'll go use Foo if you don't fix this!"

LCA: How to destroy your community

Posted Jan 19, 2010 8:56 UTC (Tue) by ikm (subscriber, #493) [Link]

> I won't tell the developers that they have to fix my issue -- they don't, they don't owe me a damn thing

Well, my opinion is that you should accurately and neutrally report the issue. Within a project of mine, I like when people do just that. But attempts to demand something or manipulate developers while at it totally ruins it.

Being an active Qt developer, I filed about like 20 bug reports to the Qt tracker. The key to a successful fix was always to provide a meaningful description, a complete test case, and preferably, a patch or the hash of the offending commit. I invested my own time, but it always payed off. The crucial understanding here is that it was *my* problem, not really theirs.

Returning to end user software, lack of any real support sucks. No possibility to pay for it sucks too. Though I'm not sure much people is actually willing to pay. I kinda like the feel of commercial-quality software and approach. Unfortunately, for all but the largest, corporate-backed software, this means closing the sources down.

LCA: How to destroy your community

Posted Jan 22, 2010 3:24 UTC (Fri) by ccurtis (guest, #49713) [Link]

> [...] corrupted every 60 minutes or so on the Open Source r600 graphics drivers [...] because of a silly bug, was forced to purchase Win7 [...] reacting to Linux as a whole when running into frequent corruption on common graphics hardware. It _is_ hurting the project.

I'm confused. I can appreciate some of the sentiment, but why are you insisting on using the open source drivers instead of the closed binary ones like you'd get on Windows, or spending the money on hardware that explicitly does support Linux rather than giving it to Microsoft? (ATI/AMD's support is a "work in progress" AIUI.)

LCA: How to destroy your community

Posted Jan 23, 2010 8:31 UTC (Sat) by sfink (subscriber, #6405) [Link]

I absolutely do consider filing detailed bug reports to be contributing. So much so that I have
it on my resume, under the section labeled "open source involvement". Nobody has ever
commented on it (and yes, I have gotten job offers, so this isn't a completely irrelevant
anecdote. Just mostly.)

But if that's not contributing, what is? It's often more useful than a patch, because most
patches need to be first interpreted and reverse-engineered to figure out what problem they
are trying to solve, and then usually rewritten from scratch to satisfy concerns that the
submitter (justifiably) doesn't care about.

LCA: How to destroy your community

Posted Jan 19, 2010 10:03 UTC (Tue) by nim-nim (subscriber, #34454) [Link]

Developers like to say that complainers that do send fixes are complete overhead. Do remember however that the same level of testing and QA is very expensive to attain in a non-FLOSS context (and QA is necessary for good software).

Though developers like to complain of testers even when they are paid by their company.

LCA: How to destroy your community

Posted Jan 19, 2010 17:49 UTC (Tue) by aliguori (subscriber, #30636) [Link]

One of the costs of being open is that you are subject to trolling. I think it's important to realize that while trolls are very vocal, they do not represent the entire user community.

A user that asks for a specific feature, and is understanding and supportive when it's politely explained that the feature is difficult to implement or lower priority compared to other things is absolutely a positive contributor. A suggestion of a feature is a useful thing even with nothing else and even if you've heard it a dozen times before.

On the other hand, trolls tend to repeatedly complain about the lack of a feature even after being the reasoning is explained. Trolls like to begin sentences with "If you really want to defeat Microsoft...". They do stupid things like reopen bug reports 100 times after it's politely explained that the issue is not in fact a bug.

To survive in an open environment, you have to be able to filter trolls. There's nothing wrong with adding filtering rules to your mail clients to send any mail from an individual that behaves like this to /dev/null. It's the only way to maintain sanity IMHO.

LCA: How to destroy your community

Posted Jan 18, 2010 16:08 UTC (Mon) by lkundrak (subscriber, #43452) [Link]

This definitely does not apply to me. For free-software products developed by Sun I've had issues with (Hudson, OpenGrok, VirtualBox; never needed to be involved with MySQL or Java communities), the Sun engineers were always very kind, responsive and overly helpful. All of those are perfectly maintained and have prospering communities.

An example of outstanding care of a Sun engineer for the community comes to my mind; once a VirtualBox developer analysed and fixed an issue with RPM Fusion-packaged VirtualBox-OSE directly on my machine as the kernel core dump was too big for me to upload. Without me having paid for support.

LCA: How to destroy your community

Posted Jan 18, 2010 16:47 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]

Doesn't all of these projects require you to sign copyright assignment
agreements with Sun? In the presence of those, thriving *developer*
communities are unlikely to happen. How many non-Sun developers does
VirtualBox have for example?

LCA: How to destroy your community

Posted Jan 18, 2010 17:51 UTC (Mon) by cyperpunks (subscriber, #39406) [Link]

Take a walk into the closed land of "Open"Solaris.

LCA: How to destroy your community

Posted Jan 19, 2010 9:33 UTC (Tue) by marcH (subscriber, #57642) [Link]

OpenSolaris is trying to build a community from scratch. They obviously cannot "destroy" it before it even exists.

LCA: How to destroy your community

Posted Jan 20, 2010 16:17 UTC (Wed) by bronson (subscriber, #4806) [Link]

No, but they can prevent it from ever existing. That's even more effective!

LCA: How to destroy your community

Posted Jan 29, 2010 18:51 UTC (Fri) by segedunum (guest, #60948) [Link]

Hmmmm, you mean the thing they've been building for five years?

You've just touched on another tactic hinted at by the article: If you don't have a community, even if you supposedly started one years ago, and people call you out on it then just claim that you're still building it!

LCA: How to destroy your community

Posted Jan 18, 2010 22:39 UTC (Mon) by zooko (guest, #2589) [Link]

When I wanted to contribute patches to make Python's build process on Solaris more like the
build of Python that ships with Solaris, the Solaris engineers were very friendly and helpful. :-)
More so than the Python developers, I'm afraid. :-(

Also I recently had a very pleasurable interaction with the ZFS crypto developers when they asked
for some general advice on crypto design. :-) They didn't end up using most of my proposals,
but I think this was probably for the very good reason that what they have now is "Good Enough"
and they need to hurry up and finalize it and ship it before Oracle takes the reins. In contrast
when I offered suggestions about crypto to the git developers, Linus publicly ridiculed my
suggestions. :-(

Also when I tried to find out why the GPL'ed UltraSPARC T2 processor had its crypto accelerator
omitted from the source code the Sun employees responsible for the T2 were polite and
responsive to my inquiries.

At the moment I can't recall any negative interactions that I've had with any Sun employees on
open source projects, while I find it easy to think of negative interactions I've had with developers
on various other open source projects.

LCA: How to destroy your community

Posted Jan 19, 2010 4:42 UTC (Tue) by lacostej (guest, #2760) [Link]

I agree.

My interaction with Sun has always been good.

You don't make conclusions on one person's experience , but I feel sad to see
Sun pointed out like that when they do provide some great service to the
community.

I would have preferred a talk made on the positive aspects (how to foster
your community), with references to projects properly managed. And I would
have personally given Sun as a reference, in particular with Hudson (a
popular continuous integration / build server) which has a tremendous
community.

LCA: How to destroy your community

Posted Jan 19, 2010 5:05 UTC (Tue) by zooko (guest, #2589) [Link]

Oh, and I forgot to mention in my list of positive interactions with Sun folks that I ran into a guy who hacks the OpenSolaris installer in a coffeeshop the other night and when I told him about the awesome "apt-clone" hack that the Nexenta folks have written he asked me to put him in touch with them in order to share code. Long live the wonderful positive tradition of collegial sharing and peer review!

LCA: How to destroy your community

Posted Jan 20, 2010 6:08 UTC (Wed) by patrick_g (subscriber, #44470) [Link]

Just for the record could you tell us why the UltraSPARC T2 processor had its crypto accelerator omitted from the source code ?

LCA: How to destroy your community

Posted Jan 20, 2010 7:03 UTC (Wed) by zooko (guest, #2589) [Link]

NSA told them to exclude the crypto engine. This bothers me, because by law in the U.S. free/open implementations of crypto are freely exportable and in addition disclosure of crypto *to* other U.S. persons is completely unregulated. So I feel that it was improper for NSA to pressure them to withhold the GPL'ed crypto engine.

LCA: How to destroy your community

Posted Jan 21, 2010 21:50 UTC (Thu) by mrdoghead (guest, #61360) [Link]

This is appalling. And as usual anything the NSA asks the computer corporate structure in the US - and elsewhere, worse - it gets with little argument. And what the intel-LE types somehow don't get all too willingly they find ways to get otherwise. This stuff needs to stop.

LCA: How to destroy your community

Posted Jan 24, 2010 7:21 UTC (Sun) by zooko (guest, #2589) [Link]

I don't know how to fight against this sort of shenanigans. Giant corporations, even though they are powerful in some ways, are ironically very susceptible to semi-legal pressure from shadowy governmnet agencies. If they had resisted NSA's pressure, there could have easily arisen some unfortunately legal snag that could have, for example, prevented them from exporting their wares until it got cleared up, thus costing them millions of dollars per day until the government gave them the go-ahead.

I politely asked the Sun employees that I was talking to if they would give me the crypto engine source under GPL, given that I was a U.S. citizen living in U.S. borders and thus it was clearly legal for them to share it with me. They didn't write back to that request. ;-)

(I had a vague notion that I might try exporting it myself and become and Internet martyr like Phil Zimmermann did in 1991. It's probably for the best that I never had the chance, as my family depends on me to bring home regular paychecks rather than to engage in dangerous civil disobedience.)

LCA: How to destroy your community

Posted Jan 23, 2010 8:37 UTC (Sat) by sfink (subscriber, #6405) [Link]

Sure, personal contact is often great. That's why this FOSS stuff works
in the first place. But this article is more about how a controlling
organization can screw things up, independently of how nice
the actual developers or other surviving community members
might be.

LCA: How to destroy your community

Posted Jan 23, 2010 15:51 UTC (Sat) by zooko (guest, #2589) [Link]

Well, I'm not sure what to say. My communication with Sun about the crypto engine of the T2 was fairly "formal" -- it wasn't with people that I had previously met. Also Sun's "Startup Essentials Program" (a hardware sales division) donated a thumper to my little Free Software startup company (http://allmydata.com). That was nice! Basically, I've had nothing but good experiences dealing with Sun as a seller of hardware, a research institute, a sponsor of open source projects, and a collection of humans. This is in stark contrast to the Linux kernel developers, with whom I've had, I'm sorry to say, mostly unpleasant interactions.

Let's turn the burden of proof around: what is the evidence that some specific corporate policy or habit is bad for open source community involvement? Surely many of the open source projects sponsored by Sun have little or no non-employee participation but this doesn't make those projects any different from *most* open source projects. Almost all open source projects have no contribution from anyone other than their founders or the company that sponsors them. So this is not specific evidence that Sun's policies are particularly destructive.

I would be very interested to learn more specific, reliable patterns about community involvement in open source. I've devoted much of the last fifteen years of my life to such projects, and I can say that the open source projects that I've led have had higher levels of community contribution than average. (Where average is zero.)

That's why I read this article with interest (and also read previous posts on the web about Josh Berkus's speech). Surely the points that Josh Berkus makes seem eminently sensible. I'm sure they are good things to keep in mind. But I'm skeptical that they are actual empirical explanations of the failure of OpenSolaris to attract external contributors, or of the success of Hudson to do so. I suspect there are other factors that are more explanatory.

LCA: How to destroy your community

Posted Jan 24, 2010 3:01 UTC (Sun) by johnflux (guest, #58833) [Link]

> In contrast when I offered suggestions about crypto to the git developers, Linus publicly ridiculed my suggestions. :-(

Ah yes, I remember. Your email:

http://www.gelato.unsw.edu.au/archives/git/0506/5298.html

And his reply:

http://www.gelato.unsw.edu.au/archives/git/0506/5299.html

I understand that you see his reply as a bit stinging, but your whole argument was based on the assumption that you could crack md5 in a way that lets you generate a meaningful exploit and then on top of that manage to inject that into the kernel. I can see why Linus responded with sillyness :-)

LCA: How to destroy your community

Posted Jan 24, 2010 7:09 UTC (Sun) by zooko (guest, #2589) [Link]

> Ah yes, I remember. Your email:
>
> http://www.gelato.unsw.edu.au/archives/git/0506/5298.html
>
> And his reply:
>
> http://www.gelato.unsw.edu.au/archives/git/0506/5299.html

That was one of the conversations that I was thinking of, but the message 5298.html that cite above wasn't written by me. I wrote an earlier message in the thread that IIRC was forwarded to lkml by someone else.

> I understand that you see his reply as a bit stinging, but your whole argument was based on the assumption that you could crack md5 in a way that lets you generate a meaningful exploit and then on top of that manage to inject that into the kernel.

I'm not entirely sure what you mean by "crack md5 in a way that lets you generate a meaningful exploit". In the exchange that you cite above, the other person, <linux@horizon.com>, was right and Linus was wrong in respect to the question of whether git users depend on the collision-resistance property of the hash function or not. The truth is that they do, but in a subtle way that most people (including Linus at least at the time he wrote that) don't understand.

At this time (in 2005), when Linus was deciding to stick with SHA-1 for git, certain certificate authorities were deciding to stick with MD5 for signatures, for the same reason -- it seemed to them that they didn't rely on the collision-resistance property. In 2008 it was demonstrated that they did actually rely on that property:

http://www.win.tue.nl/hashclash/rogue-ca/

A similar attack is probably possible on git. It currently costs substantially more than USD 1 million to build a computer that can generate SHA-1 collisions (how much more is not publicly known, but probably less than USD 10 million). For now, only the rich can play.

So while I'm sure that the cryptographers who generated the rogue Root CA (above) can't inject their own code into your git pulls (because they work at public academic institutions and don't have the budget), I'm not sure that the NSA or the Chinese cyberwarriors can't.

> I can see why Linus responded with sillyness :-)

I understand that it seemed ridiculous to him at the time. However he was qualitatively wrong about the properties that git users rely on, and both he and <linux@horizon.com> were quantitatively confused about the cost to generate SHA-1 collisions. (See the rest of the thread that you cited, in which they talk about SHA-1 collisions costing 2^80 computations, when in fact the known upper bound at that time was 2^69. Today the known upper bound on the cost to generate a SHA-1 collision is 2^63.)

One effect of mocking things that seem ridiculous to you is that it deters certain kinds of people from participating in the conversation. I suppose this could be useful if you are right and they are wrong and progress is achieved by getting them to shut up, but of course you take the risk that you were wrong in the first place and by doing this you stay wrong.

I, for one, was reading that conversation at the time, and decided not to join in and try to explain more, in part because I didn't want to have my feelings hurt by mockery and in part because it didn't seem like I would have a good chance of making my point understood.

So to attempt to swerve back onto the topic of this LWN article, when I offered some suggestions to the engineers who are adding crypto to ZFS, they responded with technical arguments that were expressed in polite language. I was therefore emboldended to think that they might actually be listening, and went on to offer more ideas: http://opensolaris.org/jive/thread.jspa?threadID=117092&... . From my very specific, narrow, limited viewpoint, Solaris open source development has been easier to participate in than Linux development. ;-)

LCA: How to destroy your community

Posted Jan 24, 2010 8:44 UTC (Sun) by johnflux (guest, #58833) [Link]

An interesting response, thanks.

Just sticking to the technical side, Linus has been critical of "masturbating monkeys" crowd (his words), that concentrate "on security to the point where they pretty much admit that nothing else matters to them" (his words again). I am not at all surprised at his response to you, whether you were technically right or not (I can't judge sorry).

On the reaction side - he can be a real ass and you got off lightly compared to his scathing on svn developers (just for an example). You do need a thick skin to work on the kernel, and it has actually been something that people are trying to address.

LCA: How to destroy your community

Posted Jan 24, 2010 15:47 UTC (Sun) by zooko (guest, #2589) [Link]

Yes, I'm quite sympathetic to the sentiment, if not to the terminology, that too many security engineers don't understand the concept of "trade-off", or if they do they seem to think that the security knob should be turned to "11" regardless of the consequences.

I don't fault Linus for being impatient with security worriers in the sense of "not wasting his time listening to them", but I do fault him for being impatient in the sense of insulting them.

One thing that has always bugged me about git's use of SHA-1 is that there was very little engineering cost, and probably not too much cost in CPU cycles, to making it secure -- just use SHA-256 instead of SHA-1. (I believe that the reason git uses SHA-1 is the Monotone did. The earliest prototype of git used MD5 because BitKeeper did.)

The engineering cost for upgrading git from SHA-1 to a new algorithm is much higher. I'm not sure how it can be done well. First of all we probably need to deploy a version of git (let's call it version 2 for this conversation) which allows there to be a slot for a new hash value even though it doesn't read or use that space -- it just uses the SHA-1 hash value which is in the other slot. That way once we *eventually* deploy yet a newer version of git, let's call it version 3, which produces SHA-3 hashes in addition to SHA-1 hashes, people using git version 2 will be able to continue to interoperate. (Although, per this discussion, they may be vulnerable and people who rely on their SHA-1-only patches may be vulnerable.)

As far as I understand people using today's git, git version 1, will not be able to exchange patches in any way with people using some future version (which I called "version 3" above) that uses a new algorithm.

LCA: How to destroy your community

Posted Jan 24, 2010 16:09 UTC (Sun) by johnflux (guest, #58833) [Link]

Looking at: http://www.cryptopp.com/benchmarks.html

It seems, roughly, that SHA256 takes about 40% more time than SHA1. From what I understand, the speed of git is determined most by the speed of the SHA1 implementation (Based on a long thread called 'Linus' sha1 is much faster!'). So roughly, switching would make everything 40% slower.

I think that's a trade-off that they wouldn't be willing to make. However, just playing with the C code of the SHA1 code by the git developers ended up making it nearly twice as fast, so I don't know what the optimal speed difference is against SHA256.

If the numbers stay about the same, I think the git guys wouldn't accept a 40% speed decrease.

(On a side point - subscribing so LWN was worth every penny. How I love to have civil conversations with intelligent people.)

LCA: How to destroy your community

Posted Jan 24, 2010 17:14 UTC (Sun) by zooko (guest, #2589) [Link]

Yeah, I was guessing that git is actually network-bound or disk-bound often enough that the CPU hit doesn't matter, but I'm not sure. (According to http://cryptopp.com/benchmarks-amd64.html , SHA-1 is 192 MB/s, SHA-256 is 139 MB/s. Faster than either your disk or your network?)

If git holds out for SHA-3 then hopefully SHA-3 will turn out to be faster than SHA-256. There's even a chance that it will turn out to be faster than SHA-1 on modern CPUs!

LCA: How to destroy your community

Posted Jan 25, 2010 0:32 UTC (Mon) by johnflux (guest, #58833) [Link]

LCA: How to destroy your community

Posted Jan 25, 2010 3:56 UTC (Mon) by njs (subscriber, #40338) [Link]

SHA-1 can definitely be a bottleneck in some situations. The most extreme -- though I'm not sure whether git does this -- is verifying hashes on an initial clone. (The idea is to prevent one person's disk corruption on an old file or whatever from spreading throughout the network of clones.) Here the disk and network cost is proportional to the size of the delta-compressed repository, but the SHA-1 cost is proportional to the size of the uncompressed repository, which can easily be in the terabyte range.

It can also easily be the bottleneck on, say, committing a large merge (many modified files, all in cache because they were just written).

LCA: How to destroy your community

Posted Dec 29, 2015 18:11 UTC (Tue) by Spitfire19 (guest, #106038) [Link]

I feel that you would also have a big hit when you are performing CI tasks, as for some scenarios you may want your automated tool to delete everything in given directory before pulling and checking out the latest commit.

LCA: How to destroy your community

Posted Jan 27, 2010 11:58 UTC (Wed) by broonie (subscriber, #7078) [Link]

In normal workflows you end up with the files you're accessing in cache so there's no I/O to physical devices and most operations do get CPU bound, especially read only ones. For performance purposes git pretty much assumes that most of the time you're running from hot cache.

OpenOffice in a nut-shell ...

Posted Jan 18, 2010 16:19 UTC (Mon) by mmeeks (subscriber, #56090) [Link]

Wow, this is pure gold. Though, I think rather restrained - this only scratches the surface of a really rich seam of dysfunction, much of it popularized by Sun Microsystems - though: to be fair that is perhaps a function of their early strategic move into Free software - and other big-corporations will no doubt follow at speed.

eg. step 6 - misses the neat consideration of having someone with no technical skills whatsoever as the "community liason". This neatly ensures that step 7. is a given, the documentation they write for step 3. will be hopelessly inadequate, and they will almost certainly build a community of like-minded people, ie. the totally non-technical: a hat-trick of achievement.

OpenOffice in a nut-shell ...

Posted Jan 18, 2010 19:43 UTC (Mon) by AlexHudson (guest, #41828) [Link]

As far as Sun's innovation goes, I feel it is somewhat overshadowed by the rapid progress made by others.

SugarCRM managed to achieve #1, #4, #5, #7 and #9 simultaneously by requiring The Outsiders to submit their patches via a small web form. Somewhat surprisingly, this form once went broken for months without anyone actually noticing - it's unclear to me if any code submitted that way ever made it into the codebase either.

OpenOffice in a nut-shell ...

Posted Jan 18, 2010 22:07 UTC (Mon) by nix (subscriber, #2304) [Link]

That's... truly remarkable. Why has no other project emulated this
stunning decision, I wonder?

(in a past job, I worked on a proprietary system that had a homebrewed
bugtracker for which the bug description had to be typed into a single
Windows input line. Not a text editing canvas: an input line. The
resulting bug descriptions left something to be desired. SugarCRM's
decision strikes me as being of the same degree of jawdropping
craziness...)

OpenOffice in a nut-shell ...

Posted Jan 19, 2010 17:21 UTC (Tue) by AlexHudson (guest, #41828) [Link]

Hehe, look : still it lives!

I don't know if it has changed at all, but the process appears to be improved further from my original understanding. You first create a bug entry covering the issue (sane, aside from login required), then you submit the web form to alert them to the submission (er...?) and they have a lovely set of guidelines to follow (gems like "The proposed change must support all baseline-certified versions of PHP, and must be able to support all databases, etc.").

Beat them hoops. Also, by my reading, they want the description of the improvement in the bug tracker, but the actual code submission via the web - what?

OpenOffice in a nut-shell ...

Posted Jan 19, 2010 21:15 UTC (Tue) by nix (subscriber, #2304) [Link]

Stunning. Nothing screams 'we don't expect contributions, go away' like
constraints like that. The 'no 3rd-party software' rule alone is an
excellent way of forcing wheel-reinvention for no good reason whatever.

OpenOffice in a nut-shell ...

Posted Jan 18, 2010 20:33 UTC (Mon) by hingo (guest, #14792) [Link]

The funny thing is, Josh has held this talk while still at Sun and I think it may have been well received even by Sunnies listening to it.

LCA: How to destroy your community

Posted Jan 18, 2010 16:57 UTC (Mon) by bobsol (subscriber, #54641) [Link]

thanks, Jon. Good coverage of a good talk.

LCA: How to destroy your community

Posted Jan 18, 2010 19:20 UTC (Mon) by dgilmore (subscriber, #40144) [Link]

Sadly this seems to all be very true. This seems to be alot how zimbra works. ive seen the same from the mysql-gui-tool team, even before sun brought them.

LCA: How to destroy your community

Posted Jan 18, 2010 20:08 UTC (Mon) by smoogen (subscriber, #97) [Link]

I think a counterpoint to this should be written. I mean I think every organization has had these things happen to them... and some recover and some do not. Why is it? What things work on rebuilding a community after trust has been broken.

LCA: How to destroy your community

Posted Jan 19, 2010 2:31 UTC (Tue) by Alan_Hicks (guest, #20469) [Link]

An apology goes a long way (assuming you follow through with changes of course).

LCA: How to destroy your community

Posted Jan 19, 2010 11:59 UTC (Tue) by hppnq (guest, #14462) [Link]

Seconded. I think it is actually quite simple: the means, requirements and goals of the bigger organization and those of the smaller organizations of which it is made up are different, and not necessarily aligned.

LCA: How to destroy your community

Posted Jan 19, 2010 4:11 UTC (Tue) by dkite (guest, #4577) [Link]

Hmm. I can think of two other big names that would fit. Funny that Sun, who no longer exists as an entity gets the brunt.

Apple. The Webkit saga was pure ugliness. The thing finally turned around somewhat, but after years, and essentially killing the original project, khtml.

Google. Tesseract and ocropus. Not google directly, but projects by googlites. I suspect that both are a result of not enough resources by the project maintainer, but both had very interesting technical challenges and had some potential of growth. But. 10 rules rule.

Derek

LCA: How to destroy your community

Posted Jan 19, 2010 9:42 UTC (Tue) by marcH (subscriber, #57642) [Link]

> Apple. The Webkit saga was pure ugliness.

Or OpenDarwin (assuming there ever was a community to destroy).

LCA: How to destroy your community

Posted Jan 19, 2010 20:58 UTC (Tue) by lambda (subscriber, #40735) [Link]

The early days of WebKit were pretty bad, but it's now a pretty thriving community that gets most things right.

OpenDarwin, however, was a fiasco, and many of Apple's other Open Source projects aren't nearly as well run as WebKit.

Maemo/Nokia: How to destroy your community

Posted Jan 20, 2010 18:54 UTC (Wed) by jebba (guest, #4439) [Link]

I cut and pasted a few snippets from LWN's post and forwarded it over to maemo-devel. Hopefully they'll resolve the existing problems over there (#1 is particularly painful, imo):

http://lists.maemo.org/pipermail/maemo-developers/2010-Ja...

The responses are interesting. Some Nokians recognize the problems, others circle the wagons...

-Jeff
http://wiki.maemo.org/User:Jebba

LCA: How to destroy your community

Posted Jan 21, 2010 1:39 UTC (Thu) by codewiz (subscriber, #63050) [Link]

At OLPC, we independently discovered and implemented all these techniques
long time ago, confirming our leadership as the most innovative educational
project ever attempted.

As suggested, we even had an affair with the most obnoxious partner one
could think of. Still, our community of enthusiastic volunteers keeps
growing and even dares releasing improved versions of the educational
platform which was ditched one year ago.

This damn community of is just too resilient. What could we possibily do to
get rid of it?

Followup from Josh

Posted Jan 25, 2010 18:54 UTC (Mon) by robilad (guest, #27163) [Link]

http://it.toolbox.com/blogs/database-soup/sun-and-the-ten...

"My presentation of "Ten Ways to Destroy Your Community" on Monday, and even more so LWN's coverage of it, made it seem as if it was entirely about Sun Microsystems and their mistakes. This was not my intention; the mistakes in community relations I was talking about are universal to the sphere of corporate open source. Between projects I myself have worked with, and the people who have shared their stories after seeing "Ten Ways", this includes Red Hat, Microsoft, IBM, Cisco, MySQL AB, SugarCRM, SQL-Ledger, Greenplum (I was responsible for that one), Novell, Compiere, Borland, Google, Apple, Zend, and dozens of other startups and large corporations. If Sun is special in this list, they screwed up more times because they open sourced more internal code than any other large corporation."

I'm curious whether this one will make it to the LWN frontpage as well.

LCA: How to destroy your community

Posted Feb 1, 2010 20:28 UTC (Mon) by donbarry (guest, #10485) [Link]

It would be interesting to tabulate failed projects with the various
failure modes. It doesn't take all! A subset is quite sufficient.

The TWiki project combusted spectacularly with only 4, 6, 7, 8, and 10
at play, forcing *all* community core contributors (16 of 18) to flee and
refound the Foswiki project. Fortunately they picked up another 16 core
contributors in the process so the effect was quite salutory.

LCA: How to destroy your community

Posted Feb 5, 2010 7:35 UTC (Fri) by mdekkers (guest, #85) [Link]

I had the honour of working with Josh on the OOo team for a long time, and kicking off a whole bunch of initiatives, and seeing them crushed one by one by Sun. After Sun started pressuring my employer at the time (a large sun reseller partner) to get me to dance to their tune in the Open Office community, I stopped contributing to Open Office, and shortly after, open source all together.

Josh is right on all counts.

Martijn


Copyright © 2010, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds