Enterprise distributions and free software
| This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible. |
The "enterprise distribution" business is a bit of a funny one; it often seems like an attempt - by vendors and customers both - to fit free software back into the business model long used by proprietary operating systems vendors. It will necessarily create some tensions between the values of the free software community (including free access, rapid development, and collaboration) and those of the business, which include rigid stability and the preservation of competitive advantage. Recent changes made by Red Hat show the effects that this tension can cause, especially when combined with strong, arguably unfair, competition.
The core Red Hat Enterprise Linux (RHEL) distribution is built entirely on free software (the supplementary repository includes a few third-party proprietary bits like Flash). The source for the distribution is freely available and freely distributable by any other party as long as trademark-relevant bits are stripped out. This freedom is exploited by free distributions like CentOS and Scientific Linux; it is also exploited by commercial competitors who are happy to charge money to support the distribution that Red Hat has made.
Naturally, it is the commercial competitors that are creating the most trouble for Red Hat, which invests a considerable amount of money into the development of both the software on which RHEL is based and the creation of RHEL itself. Companies like Oracle also contribute to upstream projects, but on a rather smaller scale, and they contribute little indeed to the development and maintenance of the RHEL distribution. Freed of that expense, these companies are able to offer support for Red Hat's distribution (or derivatives thereof) while charging much less. They are said to be actively courting Red Hat's existing customers, with a certain amount of success. It seems like a clear path to a successful business, even without the darker allusions, heard occasionally, that the real intent is simply to drive Red Hat out of business.
Given this situation, it is not entirely surprising that Red Hat has decided that it needs to respond; a company which makes life too easy for its competitors tends not to stay around for long. The company could try to make its distribution more proprietary to keep others from redistributing it. The use of proprietary kernel modules to this end is not unheard of, but it would be surprising indeed to see Red Hat take that approach. Proprietary user space components, like those used by Google to keep some control over Android, would have fewer licensing difficulties, but it still would be a major change for a (previously) free distribution. The good news is that Red Hat has resisted any such pressures - so far.
What Red Hat is doing is still seen by many as being something other than good news, though. Essentially, Red Hat is asserting stronger proprietary control over the metadata it possesses about the RHEL distribution. That metadata includes its "knowledge base" of problems and solutions, bugs and fixes, etc. What Red Hat wants to do is to exploit the advantage gained from having built and supported RHEL over all these years, and to make it harder for others to exploit that knowledge. The company is trying to compete by providing better support, and one way to have better support is to keep your competitors from making their own support offerings better through the use of your own information and experience.
Kernel source packaging
The move that brought all this to the community's attention was a change in the way that the source RPM file for the RHEL 6 kernel is packaged. Traditional source-packaging practice is to combine an unmodified upstream tarball, a set of distributor patches, and a script (the "spec file") which applies the patches and builds the software. The RHEL kernel package, instead, consists of a single tarball reflecting the kernel in its fully-patched form. There is a long list of patch titles in the spec file, but there is no other information which can be used to recover the specific patches which have been applied to the kernel.
The purpose of this change, evidently, is to make it harder for competitors to figure out why specific patches were made. Many changes applied to the RHEL kernel will have been done in response to specific customer problems; the understanding of those problems and how they were fixed will, at times, be useful in future support situations. By mixing all of its patches into a single, lossy tarball with no changelogs, Red Hat hopes to deprive others of this useful support information.
Will it work? The restriction of information in general should certainly deprive competitors of a useful resource, though the cynical might note that access to that information could be restored via an inexpensive low-end RHEL subscription. Whether broken-out patches for the kernel, in particular, are useful to competitors is not entirely clear. It may well be that Red Hat has irritated - and possibly made life harder for - the development community without bothering anybody else in any appreciable way.
Red Hat claims that the packaging change should not bother the development community at all, noting that the change had been in effect for four months before complaints were heard. One of the foundations of this claim is the fact that the RHEL "2.6.32" is far removed from anything called "2.6.32" anywhere else; the company once described it this way:
This kernel is a sort of Frankenstein's monster of fixes and backports; it is a distant relation to any sort of upstream kernel. Given that, Red Hat has said, most patches to this kernel do not apply in any useful way to the kernels the rest of us are working with. In other words, they claim, there is no value to the community in most of the RHEL kernel patches.
More importantly, though, Red Hat insists that it is a strong believer in working upstream. Any patches developed by Red Hat which have any relevance to upstream kernels are promptly sent upstream for merging. This line of reasoning suggests that there will not be anything of interest to the community in RHEL kernels; everything of value will have already been submitted for merging. The company's history backs up this claim; its history of contributions is matched by few others, and, by all accounts, the practice of contributing upstream is wired deeply into the company's culture.
That said, maintainers of the community stable kernels (and some other developers as well) have been heard to complain that Red Hat's practice is making work harder for them. Beyond that, though, this practice leaves the community with a bit of a bad taste in its mouth. The development of a significant fork of the Linux kernel has become more closed in a possibly ineffective attempt to support corporate interests. The fact that those corporate interests are in the personal interests of everybody whose salary is paid by that company and by everybody who benefits from that company's contributions perhaps softens the blow, but it does not make everything better. Many people are left wondering what the next step will be.
The enterprise distribution business
They may be right to wonder. The "enterprise distribution" business model, as it is currently implemented, is a strange distortion of the development community that supports it. It is said by some to violate the GPL, or, at least, to stretch its spirit to the breaking point; that is a subject which will be examined in another article. Of interest here is the fact that this model also led to the creation of kernels so divergent that its simple fixes no longer apply to the mainline. Your editor can't help thinking of Andrew Morton's classic keynote at the 2004 Ottawa Linux Symposium:
The RHEL kernel, which is certainly differentiated from any other kernel out there, would seem to be well described by Andrew's words. Red Hat's move to lock down information about this kernel is an admitted attempt to increase customer lock-in; they are trying to make it harder to move to another provider of support. Is this good for Linux?
Corporate moves do not happen in a vacuum; Red Hat's competitors will likely come up with responses of their own. Perhaps they will just develop other sources for the information they need. If, instead, they adopt a strategy like Red Hat's, we could end up with a fragmented range of incompatible Linux distributions, each of which tries to lock in customers with its own combination of unique features and proprietary information. Veterans of the Unix wars can be forgiven for shuddering a bit at that thought.
An alternative might be to move away from the idea of "frozen" kernels in enterprise distributions. The truth of the matter is that these kernels are far from frozen; RHEL's "2.6.32" kernel has features from 2.6.38 - and probably beyond. Nobody wants a kernel which was set in stone years before, so these "stable" kernels get no end of features backported into them. The result is a combination which has never been seen before in nature and which may have stability problems of its own. It is also expensive to create and expensive to maintain.
Perhaps enterprise distributions should stick closer to mainline kernels and even move to new releases occasionally? Oracle's "Unbreakable Enterprise Kernel" looks like a move in that direction. Novell did something similar with its SP1 update to SUSE Linux Enterprise 11. This approach would seem to offer some real advantages: users would get new features and fixes from upstream, and distributors would not have to pay the cost of creating and maintaining a highly divergent kernel. There would also be less need to treat related information as proprietary; companies would compete on their ability to support current mainline software, rather than a vendor-specific kernel. That seems like an environment in which Red Hat, which does so much to create mainline kernels, could compete most effectively.
Of course, anybody who listens to business advice from your editor has a strong chance of learning all about the bankruptcy process in short order. Beyond that, this is not a new idea; it was discussed here in 2007. The ultra-stable enterprise distribution model seems to be in no danger of going away, in any case, even though some vendors seem to be experimenting with some changes. There are customers, it seems, so there will be vendors.
In the end, companies need to make money or they disappear. Red Hat has,
so far, found ways to keep its stockholders happy while supporting the
development community which makes it all possible. One can only hope that,
as this market becomes more competitive, the companies involved will
continue to find a way to balance those important interests; the
alternative, certainly, will be destructive to both.
(Log in to post comments)
Enterprise distributions and free software
Posted Mar 8, 2011 0:05 UTC (Tue) by airlied (subscriber, #9104) [Link]
I'm not sure why you think the second set of requirements is necessary for the first statement to be true.
We have the first one now in the enterprise space alone we have, RHEL, SLES/SLED, Ubuntu LTS, Debian stable, and others have come/gone.
In the other spaces we have Ubuntu, Fedora, OpenSUSE etc.
its already fragmented it has been for years, the kernel used is probably the least of anyone's worries.
Enterprise distributions and free software
Posted Mar 8, 2011 0:14 UTC (Tue) by dlang (guest, #313) [Link]
the kernels used by different distros used to be far more fragmented then they currently are, but since there was no lock-in, things were able to be smoothed out.
if there is lock-in, it makes it much harder to fix any of the other problems.
Enterprise distributions and free software
Posted Mar 8, 2011 0:47 UTC (Tue) by blitzkrieg3 (guest, #57873) [Link]
I take issue with this definition of lock-in, which is one I've never heard before. Red Hat doesn't do anything to the OS or to the customer to make it harder for the customer to move, we simply make it harder for competitors to support Red Hat software. Now if you're a business person and you say, "We could move to vendor X for Enterprise Linux support, but their support is not as good as Red Hat", I could see how that might make it "hard to move", but that isn't lock-in.
Enterprise distributions and free software
Posted Mar 8, 2011 1:09 UTC (Tue) by martinfick (subscriber, #4455) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 2:03 UTC (Tue) by airlied (subscriber, #9104) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 0:33 UTC (Tue) by ejr (subscriber, #51652) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 1:33 UTC (Tue) by neilbrown (subscriber, #359) [Link]
The vendor (Redhat / Novell / Canonical / ...) do their own QA, then release. Partners, customers, community all report bugs and these get fixed, so the kernel theoretically gets more and more stable - each customer benefiting from the bugs found by other customers/partners/etc.
The common -stable kernel series is an attempt to provide these benefits across distributions.
There are other (claimed) benefits too such as third-party certification, and incorporation of "valuable" out-of-tree patches.
So I don't think there is "only" one benefit - there are several. I wouldn't claim to suggest what the "main" benefit is though.
Enterprise distributions and free software
Posted Mar 8, 2011 1:46 UTC (Tue) by ejr (subscriber, #51652) [Link]
The third-party certification's immediate impact for me is the availability of binary blobs. I'm not trying to resell, so I can see that difference. My experience is with HPC systems, not customer-service terminals and the like, so perhaps my view is a bit skewed.
So you're right, the one enterprise benefit is a "for me" thing. The stability issue, though, I don't see as a differentiator against large-scale community distributions.
Enterprise distributions and free software
Posted Mar 8, 2011 2:10 UTC (Tue) by airlied (subscriber, #9104) [Link]
I'm not saying its perfect but it less bad than bouncing to an upstream kernel., running Debian stable is fine on 2 yr old hw, even on hw coming out now Debian stable is already behind.
Enterprise distributions and free software
Posted Mar 8, 2011 2:48 UTC (Tue) by ejr (subscriber, #51652) [Link]
And in response to another comment, for *me* the VM performs much better with respect to my multithreaded jobs than older kernels do. But I also turn off swapping for HPC installations; if you swap, you loose the high-performance part. That does eat a little memory for monitors that *could* be swapped out, but it reduces the perturbation when those monitors happen to trigger during a compute job. Turning off swap doesn't turn off mmap-ing of large data sets, so it's not a huge deal for common apps and helps stop student mistakes from crippling the nodes. Setting the ulimit at a tested maximum keeps the OOM killer at bay (or seems to do so).
But I'm also stuck with RHEL given current employer restrictions. It's not terribly OpenMP-friendly, in my experience, leading to many people griping about how "Linux" sucks.
Enterprise distributions and free software
Posted Mar 8, 2011 13:10 UTC (Tue) by SEJeff (guest, #51588) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 13:40 UTC (Tue) by ejr (subscriber, #51652) [Link]
With Debian (and Fedora, and...), you can pull newer versions of tools if you need them. You can at least see how the version chance affects the system even if you decide not to use the Debian version. I don't get why people insist that the named "stable" release is the only thing available. I know apt lets you set defaults to pick and choose easily, and I assume the other distributions' tools do.
I suppose it's another thing I don't get about "enterprise" distributions. You tie your timeline tightly to the producer's decisions. I'd rather be able to tie our cluster upgrade and testing timelines to our calendar.
But, again, I'm talking about systems used for development of some flavor. I suppose shipping a word processing box is different, but you definitely don't want the latest & greatest there, either. Retraining is a major cost.
Enterprise distributions and free software
Posted Mar 8, 2011 19:42 UTC (Tue) by ThinkRob (guest, #64513) [Link]
It's not uncommon to see a production Debian machine with versions of software that are a couple years old. It's also not uncommon to see a production RHEL install with software versions from the better part of a decade ago. RHEL wins. :D
Enterprise distributions and free software
Posted Mar 8, 2011 19:49 UTC (Tue) by skvidal (guest, #3094) [Link]
No one does that kind of maintenance for fun.
Enterprise distributions and free software
Posted Mar 9, 2011 0:45 UTC (Wed) by jengelh (subscriber, #33263) [Link]
Enterprise distributions and free software
Posted Mar 9, 2011 4:31 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
Enterprise distributions and free software
Posted Mar 9, 2011 2:14 UTC (Wed) by ThinkRob (guest, #64513) [Link]
Oh, absolutely. I was merely pointing out that a RHEL release can and often does have a much longer shelf life than a Debian release.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 2:02 UTC (Tue) by kragilkragil (guest, #72832) [Link]
I really hope the business model RH, Novell and Oracle use will die and I hope this is the first indication that it will die eventually. Oracle can just buy one RH license and continue to undercut RH because they don't waste millions on creating totally stupid franken-kernels that only appeal to dumb people that fell for false stability marketing.
A distro should be developed like the kernel itself. There should be one community that builds the distro releases and a lot of companies and individuals should drive it into the directions they want.
Where would Debian be if Novell and RH and Oracle etc had only put their resources into it instead of creating a whole new distro?
Logic dictates that not using free resources and wasting resources will be punished. I don't think we a free market anymore, but I hope that marketforces will bring down non-community distros.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 2:06 UTC (Tue) by airlied (subscriber, #9104) [Link]
good luck.
The sheer number of regressions esp in areas like the VM is quite large, the kernel needs stabilisation periods that upstream doesn't supply, and shoving into Debian stable doesn't do anything other than sticking people with a kernel that isn't going to improve or add features.
The featureset of no major regressions + new HW support is something people think is worth paying for, community distros don't cater to this market at all.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 2:48 UTC (Tue) by dlang (guest, #313) [Link]
I can point at the availability of the systems that I manage and compare it to the availability of the systems running RHEL and make a very good case that my linus kernel on whitebox systems is better than their RHEL on top-tier branded systems
if you blindly install a RHEL upgrade/kernel on your system without testing it, you will have some pretty horrible problems. If you are doing the testing, then it doesn't really matter if you are testing RHEL or linus-kernel based systems (although, if you run into something strange, you can get debugging and fixes for the linus-kernel faster than I've seen happen with RHEL)
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 4:18 UTC (Tue) by raven667 (subscriber, #5198) [Link]
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 13:20 UTC (Tue) by SEJeff (guest, #51588) [Link]
https://bugzilla.redhat.com/show_bug.cgi?id=327591
Redhat took all of the upstream nfs client "fixes" that google primarily wrote and backported them to RHEL4. It did not go so well. The best thing I got out of working on this bug was ninja-level nfs debugging experience. See comment 8 for the secret sauce.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 18:21 UTC (Tue) by ricwheeler (subscriber, #4980) [Link]
Are you confusing google (no interest in NFS) with NetApp which employs Trond, the upstream client maintainer?
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 20:11 UTC (Tue) by vonbrand (guest, #4458) [Link]
If you compare the stability of vanilla Linus kernels on whiteboxen to $ENTERPRISE kernels on top-tier machines, you don't know if the differences are due to the kernel's stability, broken(ish) support for funky hardware on the "top" machines, different workloads/stress, or even simply due to different definitions of "stable" in different environments.
Enterprise distributions suck and free software rules
Posted Mar 12, 2011 21:00 UTC (Sat) by dlang (guest, #313) [Link]
that being said, I used the kernel.org kernel on the top tier boxes for a few years before deciding that they didn't provide any additional reliability.
as for the argument that reliability means different things, in the case I am referring to, reliability is measured and defined by the same people.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 13:02 UTC (Tue) by kragilkragil (guest, #72832) [Link]
It needs to die and I hope it will sooner than later and I hope that the next few quarters RHs numbers won't be shiny and investors will punish them for wasting their money on inefficient development.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 20:26 UTC (Tue) by airlied (subscriber, #9104) [Link]
The QA cycle is longer than the kernel release cycle. QAing a kernel requires a set amount of time, at least for people to be happy that it isn't considerably worse than the previous one. Some of the benchmarks that enterprise customers care about can nearly take longer than the test cycle to get scheduled, they require hw that isn't always available. Never mind tracking down regressions using a bisect on a test that takes a week to run and process the results of.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 20:43 UTC (Tue) by lmb (subscriber, #39048) [Link]
And even if the test cases were only run for, say, for every three upstream releases, I postulate that this would a) greatly reduce the chance that relevant regressions get introduced, b) even rev'ing the "enterprise" kernels every three upstream kernel releases would already be a huge boost over rev'ing them every 3 years.
I'd never have expected that, of all people, the _Linux_ folks would be the ones to claim that Linux/OSS can't work in an mission-critical environment but needs to be curtailed to a legacy "enterprise" model! I'm truly amazed.
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 4:22 UTC (Wed) by airlied (subscriber, #9104) [Link]
Enterprise distros also have a whole bunch of certifications (government, application) etc that it isn't feasible to redo every 3-6 mths it can takes a year or so to get some of them finished.
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 4:27 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
If you can make this model work, you have a brilliant edge over the competition. Feel free to try.
"I'd never have expected that, of all people, the _Linux_ folks would be the ones to claim that Linux/OSS can't work in an mission-critical environment but needs to be curtailed to a legacy "enterprise" model! I'm truly amazed."
Linux can certainly work in a mission critical environment. The debate is not about that at all but whether the current enterprise model is legacy or necessary. I would say that it is possible to tweak the model and vendors occasionally do that but it is not going to go away unless some vendor decides to provide a sustainable alternative.
Enterprise distributions suck and free software rules
Posted Mar 10, 2011 3:56 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
The featureset of no major regressions + new HW support is something people think is worth paying for, community distros don't cater to this market at all.
Actually, Debian does add hardware support, particularly for hardware that may be required during installation. However, we aren't nearly as thorough as Red Hat in doing this, mostly because we don't have the resources for regression-testing driver changes.
I have been meaning to propose some sort of hardware partner program for Debian kernel and X maintainers to work with hardware vendors (or upstream maintainers with good collections of hardware) on testing driver updates. But I think it would have to be somewhat different from commercial partner programs.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 6:06 UTC (Tue) by rilder (guest, #59804) [Link]
A distro should be developed like the kernel itself. There should be one community that builds the distro releases and a lot of companies and individuals should drive it into the directions they want.
"
All the upstream components which make up a distro - Xorg, Glibc et.al. among others are developed in a non-fragmented manner. You don't see two different vendors for Qt, do you ? Building a distro is perhaps the easier part, the actual upstream development which the distros refer to for feature enhancements,bugs etc is where the cake lies. Why else do you think there are virtually hundreds of distros ?
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 12:49 UTC (Tue) by kragilkragil (guest, #72832) [Link]
And it is not like RH is a shining beacon of superb collaboration. All their Gnome3 hackers took a giant dump on freedesktop specs for example.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 20:13 UTC (Tue) by vonbrand (guest, #4458) [Link]
"Most distros are Debian" is just not true in so many different ways...
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 23:42 UTC (Tue) by kragilkragil (guest, #72832) [Link]
That are 129 and it doesn't even include the Ubuntu, Knoppix etc based ones.
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 4:34 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
You have selection bias and discounting all of the Fedora derived and yes, distrowatch doesn't list the majority of them either. If you go and count all of the different distros in Distrowatch, there is a enormous amount of them not related to Debian in any way.
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 9:55 UTC (Wed) by kragilkragil (guest, #72832) [Link]
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 10:08 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 10:35 UTC (Wed) by patrick_g (subscriber, #44470) [Link]
Article from Bruce Byfield here : Linux Leaders: Debian and Ubuntu Derivative DistrosQuote :
"just under 63% of all distributions now being developed come ultimately from Debian. By comparison, 50 (15%) are based on Fedora or Red Hat, 28 (9%) on Slackware, and 12 (4%) on Gentoo."
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 10:44 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
As I already pointed out, Distrowatch list isn't exhaustive. Even my own derivative isn't listed there despite active releases for a long time for unknown reasons. The original source of the distribution isn't that important either. Even if the package format is the same, in practise there are all sort of interesting differences in the installer, packaging guidelines, higher level tools, patches, lifecycle, commercial support, certifications etc.
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 11:33 UTC (Wed) by patrick_g (subscriber, #44470) [Link]
If better data are not available elsewhere I must use this study based on Distrowatch as reference.Do you have more precise figures ?
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 11:44 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 11:54 UTC (Wed) by kragilkragil (guest, #72832) [Link]
Enterprise distributions suck and free software rules
Posted Mar 9, 2011 12:09 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]
Enterprise distributions suck and free software rules
Posted Mar 12, 2011 9:06 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]
GNOME Shell developers have yet implemented a specification that KDE proposed. KDE hasn't implemented some specs that GNOME developers have proposed either. Just because it is hosted by freedesktop.org does not in anyway imply that there is consensus that everybody should use it. This is a common misconception that freedesktop.org is a standards body of some sort. This has never been the case ever since the day Havoc Pennigton from Red Hat launched freedesktop.org back in 2000. Choosing not to implemented a particular spec does not imply lack of collaboration. It can and often is just a technical choice.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 11:54 UTC (Tue) by pkern (subscriber, #32883) [Link]
Compare this to SAP vs. Oracle. The main point of that lawsuit was that SAP wrongfully used their access to Oracle's support site (which they had access to for their own Oracle instances, AFAIR) to offer Oracle support to their clients.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 12:52 UTC (Tue) by kragilkragil (guest, #72832) [Link]
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 18:48 UTC (Tue) by rahvin (guest, #16953) [Link]
No this move is going to cost Oracle some serious development money to re-engineer all of Redhat's work. When they were just copying Redhat it wasn't much of a concern but now that they are going for the value proposition by trying to provide a "better" kernel Redhat needs to react to make them spend what Redhat does to support their Frakenkernel. With everything provided in detail Oracle could have easily reverse engineered the changes and why there were made and cherry picked what they wanted, now they will have to actually spend money to do that or Engineer their own Frankenkernel from the ground up or provide a vanilla kernel. In the end it's possible Oracle simply won't be able to afford it. Either way Redhat has the marketing position to point out that Oracle's kernel is inferior to their own.
I would suggest that Redhat work with the developers of other good community members (Debian, Kernel developers, etc) to reduce the concern but personally I fully support their need to hurt Oracle's business. I believe that Oracle is trying to kill Redhat to make Solaris more popular. Redhat is a key community member and pays for development of a lot of the key pieces of Linux and the community can't afford for the company to go under. There is no doubt Redhat has done questionable things in the past, but they are probably the best commercial community member and I support their need to remove a parasite.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 19:34 UTC (Tue) by foom (subscriber, #14868) [Link]
You think Oracle started developing Unbreakable Linux to destroy Linux in favor of Solaris, 3 years before they acquired Solaris? As evil as Oracle may be, that seems to stretch credulity.
Enterprise distributions suck and free software rules
Posted Mar 8, 2011 20:18 UTC (Tue) by vonbrand (guest, #4458) [Link]
The "Unbreakable Linux" saga was a total flop; that they are now trying to use it's leftovers to leverage Solaris is just (re)use of available assets.
And never forget Hanlon's razor.
Android Kernels are worse
Posted Mar 8, 2011 5:39 UTC (Tue) by Shachar (guest, #67086) [Link]
I have tried to figure out what, exactly, do the new APIs introduced into Android actually do. No documentation, no coherent patches you can inspect. There is a git repository, but there is no order to the changes that helps in any way. Download a kernel source for any HTC device (haven't checked, but I suspect the other Android vendors are the same) and you get the same thing - one tarball of kernel source.
Shachar
Source package with individual patches
Posted Mar 8, 2011 8:20 UTC (Tue) by epa (subscriber, #39769) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 9:37 UTC (Tue) by mjthayer (guest, #39183) [Link]
Of course providing basic support as well (like answering questions about using the software and perhaps helping the customer to decide which bugs are affecting them badly) would probably still be necessary, and could perhaps be done in a manageable way by selling the support on a "number of support employees available full-time to the customer" basis. Again, out of reach for small customers but probably a good proposition for enterprise-size ones.
Enterprise distributions and free software
Posted Mar 8, 2011 13:12 UTC (Tue) by Lennie (subscriber, #49641) [Link]
As there is already a stable kernel I'm not sure if creating their own is the solution.
Enterprise distributions and free software
Posted Mar 8, 2011 14:28 UTC (Tue) by mjthayer (guest, #39183) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 20:17 UTC (Tue) by Fats (guest, #14882) [Link]
You have two packages and you fix an interoperability between them for a customer. Neither of the upstream packages wants to accept the patches as they say it is only the other packages fault or are just not interested in interoperability.
What do you do ? If you don't fork it for all your customers you are basically forking for each customer separately. I don't think this is a better way of working.
Enterprise distributions and free software
Posted Mar 8, 2011 15:13 UTC (Tue) by smoogen (subscriber, #97) [Link]
Customers for the most part want the latest X to work, but don't want anything else to change. And the larger the customer the more they want that lock in for longer times. So there are places still using and paying for 2.4.x kernel (and ancient gcc,perl,python) support but also will pay some consultant to make the latest desktop to work on it.
Enterprise distributions and free software
Posted Mar 8, 2011 15:15 UTC (Tue) by jwarnica (subscriber, #27492) [Link]
The issue is that providing tier-1 support to "everything we can cram on a DVD" is that it is basically impossible. While "Gnome" might be a pretty solid box internally, a given snapshot of "Gnome" might only really work with a specific combination of kernels and userspace utilities. Consider the various layers audio, or network configuration, needs to touch. And consider non-Gnome, Gnome apps, like GnuCash for example. It doesn't work with the latest versions of Gnome.
Sure, you can get it to work, but it doesn't "just work". Getting it to work in many cases might take a lifetime of experiences. You can't possibly build up a knowledge base of quick solutions to common problems if you don't make efforts to control the source of the problems. So every tech support call is about actually debugging a problem, rather then giving someone a quick canned answer. So all of your support calls are an hour long (or longer), and require someone with years of experience to handle.
So instead of having a call center filled with people who have a loaded cost of $15/hour, and answer questions in an average of 5 minutes, you need a call center filled with people costing $75/hour, and who take an hour to answer calls. You do the math on how much a subscription to such a service would cost.
It does happen, but they are called consultants, not distributions.
Enterprise distributions and free software
Posted Mar 8, 2011 20:30 UTC (Tue) by xnox (subscriber, #63320) [Link]
> answer questions in an average of 5 minutes, you need a call center filled with people costing
> $75/hour, and who take an hour to answer calls. You do the math on how much a subscription to
> such a service would cost.
Huh? $75/hour? I'm paid way less than that. Our customers do pay $100 and up per hour. /me should look for a better free software job.
Enterprise distributions and free software
Posted Mar 8, 2011 20:48 UTC (Tue) by jwarnica (subscriber, #27492) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 10:23 UTC (Tue) by lmb (subscriber, #39048) [Link]
In some cases, these customers demand to be lied to - there's no way of meeting the "don't change anything!" at the same time as "keep the new stuff working!" (And "new stuff" isn't just hardware support.)
Clearly, I have a personal opinion on the matter. (Which, strangely enough, doesn't align with my pay cheque. ;-)
It's worth noting that some enterprise products fare quite well by putting the QA on top of upstream; I've got my blinders on here, but it works really well for the cluster stuff so far - we keep releasing, essentially, a QA'ed version of upstream-latest and support that. We work with upstream to ensure that a smooth upgrade is feasible, and that new features do not disrupt existing installations. And upstream for cluster stuff tends to be rather quality-conscious. Yes, even then, there are jumps that need to be carefully planned, but overall, it is sustainable without backports.
Enterprise distributions and free software
Posted Mar 8, 2011 14:22 UTC (Tue) by corbet (editor, #1) [Link]
I don't have opinions, everybody knows that.There are a couple of interesting data points for those who say that enterprises need these frozen-version-number kernels, though.
- The "enterprise real time" distributions have always stuck much closer to mainline, seemingly with no ill effect.
- I've been at talks by representatives of Credit Suisse and NASDAQ OMX, both of which probably qualify as "enterprises." Both run their operations on mainline kernels.
One hears stories of other "enterprises" based on Gentoo. There do seem to be ways to make this work...
Enterprise distributions and free software
Posted Mar 8, 2011 19:03 UTC (Tue) by rahvin (guest, #16953) [Link]
I haven't looked in a number of years but I thought Redhat offered support on their Frakenkernel and Vanilla Kernels for those that want to run them.
Enterprise distributions and free software
Posted Mar 10, 2011 14:10 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
Enterprise distributions and free software
Posted Mar 10, 2011 17:30 UTC (Thu) by lieb (guest, #42749) [Link]
Enterprise distributions and free software
Posted Mar 8, 2011 13:35 UTC (Tue) by AlexHudson (guest, #41828) [Link]
"There would also be less need to treat related information as proprietary; companies would compete on their ability to support current mainline software, rather than a vendor-specific kernel."
Competition doesn't really work like this; you can't organise with other businesses exactly how you're going to compete (well, unless you're a cartel...), you just *compete*. Red Hat could decide that they're going to do this "follow mainline closely", but it really wouldn't take many failed updates to a supposedly "stable" release for customers to leave for the exits (read: Oracle, Novell) extremely quickly.
Enterprise distributions and free software
Posted Mar 8, 2011 13:45 UTC (Tue) by rick.dennis (guest, #61887) [Link]
Personally I don't consider someone's basement or a University an enterprise. It is tens of thousands of 'enterprise class' servers ; support (meaning drivers, testing, the other vendor will take your call) for all their peripherals like 10G, IB, HCA's, enterprise SSD's; thousands of developers/users ; it requires third party software for other critical enterprise functions like clustering and security. (And then running them and supporting that configuration for _years_.)
If a business doesn't have these requirements could they get away with moving "away from the idea of a 'frozen' kernel'"? Perhaps, but then you have another problem: management. Enterprises are typically large companies and those whose core business is not technology aren't particularly concerned with hiring more Linux implementers than is necessary. We're expensive and can be annoying. They'd prefer the indemnification of purchasing some contract with a company that
says if half your Linux team walks out you will be ok. (Regardless of if that perception is real or false.)
2.) Honest question (not rhetorical): (Excluding embedded.) Without the Linux enterprise's, doesn't this just become a hobby OS?
Enterprise distributions and free software
Posted Mar 8, 2011 14:03 UTC (Tue) by lmb (subscriber, #39048) [Link]
The main reason why enterprise distributions exist is that change causes fear in humans, whether justified or not. There are several ways of resolving that; one is to avoid/hide the change (which is what enterprise distributions do - "ohhh, version number is static, must be stable").
The other is to make the risk/cost associated with change drop below the benefits of the newer versions. There is no reason why "upstream tip" should be of lower quality or stability than an enterprise distribution. (That it sometimes, indeed, _is_ of lower quality perhaps tells us something important. But that can be addressed.)
Mind, the first model makes good money; and it satisfies a real and justified need the customers have - managing risk. But it is not the only possible solution. While you won't be able to sell the second to everyone, nor that it would work immediately, or that I have all the answers.
But it's one of these things that I'd love to investigate more. One of these days I'm going to shake a tree until a business angel falls out ... ;-)
Enterprise distributions and free software
Posted Mar 8, 2011 14:26 UTC (Tue) by rick.dennis (guest, #61887) [Link]
I suppose it is fear. Fear that IF we rolled our own and said, yes we will loose support for that 3rd party messaging software or clustering solution, but I'll take on support for that too if things break. Fear that the business is impacted by this, or fear that I could spend my days debugging 3rd party software. (Though really, Sr. management probably has more fear of this and would never allow it to go that far.)
If you mean that the 3rd party software could use a "thawed" kernel in an enterprise distribution -- I don't think we have any evidence (or hope) of tens or hundreds of them all agreeing to support their products on such a kernel. Inevitably it will need to be frozen.
Enterprise distributions and free software
Posted Mar 8, 2011 16:30 UTC (Tue) by lmb (subscriber, #39048) [Link]
If the same - probably less, overall - effort was piled on top of upstream, there's absolutely no reason why that wouldn't work. I clearly didn't say to "roll your own"; there's value in paying someone for this, because it allows them to specialize, and will be there to support users in case something goes wrong. (And also take the responsibility.)
I completely disagree on the "inevitably". Clearly, this is not the case - it doesn't get frozen forever, because every so often, the enterprise vendors do rev up the kernel. And because it only happens so rarely, it is a huge effort, at which time a lot of problems are found. (Because they crept in when nobody was watching.) If appropriate diligence was instead applied to the on-going releases, there's no reason why a smooth, incremental path wouldn't be available.
There's no reason why the latest upstream shouldn't be supportable. In fact, the pool of engineering on it would be even larger than for any current enterprise distribution.
There's one RedHat competitor with less motivation to contribute to a healthy kernel.
Posted Mar 8, 2011 22:38 UTC (Tue) by jpnp (guest, #63341) [Link]
However, there is one competitor of RedHat (incidentally, the one whose distro is most closely related) who also owns a competing Unix implementation. They have less motivation to invest in Linux over their own product (let's call it Solaris). If a weakened RedHat led to less investment in Linux it would only help Solaris' competitive position.
I do have quite a lot of sympathy for RedHat in this position.
Enterprise distributions and free software
Posted Mar 10, 2011 2:18 UTC (Thu) by kurtseifried (guest, #57307) [Link]
========================================
From: ********@oracle.com
Subject: Oracle Linux Suppor
Hi,
Trust all is well.
I am the Country Account manager from Oracle Linux and Virtualisation Business Unit handling Canada. I have an understanding that you have Linux installation in your set up with your current support vendor as Red hat.
As you must be aware of the change in pricing / Support levels Red Hat recently came with the release of RHEL 6. Going forward for any new purchases/renewals Red Hat's basic support would not be available. So your support cost is bound to go high with a change in Support level from Basic to Standard with them. Also the pricing now is according to pair of sockets. So for every additional pair you would be paying even more along with any extra functionality ( clusters, management packs etc ) that you would require.
The reason why I explained the above is to compare Oracle pricing/features vis a vis Red Hat. We help our customers save cost up to 70% of what they currently pay to Red Hat. Our pricing is based on per cpu (2 or less / more than 2). Standard pricing adheres to anything more than 2 cpu's. Also Oracle's Basic Level of support is equivalent to Red Hat's best level (Premium) level of support.
If you think the above could make any business sense to you, Then I would like to welcome an opportunity for a discussion around a Value proposition on Linux Support and help you get the best pricing so that you can save on your Operating System support cost.
Let me know. Looking forward to speaking with you.
========================================
Enterprise distributions and free software
Posted Mar 10, 2011 3:02 UTC (Thu) by foom (subscriber, #14868) [Link]
Enterprise distributions and free software
Posted Mar 10, 2011 11:24 UTC (Thu) by pzb (guest, #656) [Link]
Enterprise distributions and free software
Posted Mar 14, 2011 21:11 UTC (Mon) by leoc (guest, #39773) [Link]
/im.canadian
Enterprise distributions and free software
Posted Mar 14, 2011 20:16 UTC (Mon) by wilck (guest, #29844) [Link]
Most enterprises don't want to buy servers with Linux, they want certain applications that run on certain capable hardware. They want certifications and guarantees. They want 24/7, guaranteed response times, and some party who takes full responsibility to fix problems, preferrably in a guaranteed amount of time. Giving this kind of guarantee requires a massive cut-down in configuration options. Therefore almost every HW and SW vendor supports only a very limited set of OS flavors.
It is pointless to say that vanilla systems can be very stable, have a high uptime, and perform nicely. The question is who is willing to guarantee that for a certain customer application on a certain hardware.
Check out the major HW and SW vendors and see which Linux releases they offer support for. In most cases, you will see RHEL and SLES, and quite often, nothing else. The HW and SW vendors spend engineering + QA efforts to ensure compatibility with distributions. The upgrade-compatibility guarantee of the enterprise distributions helps them save efforts: certify once, and keep certified until end-of-life of the distribution. This upgrade-compatibility implies a commitment of the distribution vendor to ensure API and ABI stability, a commitment that the community deliberately refused to make, as every LWN reader knows. This stability, plus availability of bug fixes and security fixes, is what makes enterprise Linux attractive (new hardware enablement is less important, it can be achieved with virtualization as well).
For a HW or SW vendor, certifying and releasing a product for RHEL5/6 and SLES10/11 is considerable effort, but it's manageable. For many vendors, anything else isn't, unless backed by a very large customer with a very specific demand and highly skilled own staff.
"Never Change a Running System" is a very important paradigm in the commercial Linux world as I know it. Customers can be extremely conservative. When an enterprise Linux distribution finally reaches end of life after 7 or 10 years, customers start requesting support for that distribution for another 5 years or more. The rapidly moving code base of the community distributions is out of the question for these customers, and so are vanilla kernels. A company trying to make money with Linux may consider this irrational, but it would be good advice to be careful telling that to a potential customer.
The enterprise distributions receive QA not only from their vendors, but also from partners and IHVs / ISVs around the globe. That adds up to considerable efforts, and considerable efficiency in finding and squashing bugs. In this way, many bugs have been found in code that had already been reviewed by two upstream communities (upstream project and Fedora/OpenSUSE, and no, it's not always the back-porters and "Frankenkernel" engineers who are messing it up). The superior quality of the upstream community releases is a myth. More often than not, the "thousand eyeballs" turn out to be just two. The upstream review/QA process is good and extremely important, but it's not by definition superior to everything else.
Last but not least, even if that's unpopular on LWN, the KABI commitments of the enterprise distributions make it possible for customers to deploy third-party kernel modules without risk.
I used to think along the lines expressed by many commenters on this page. I even tried to convince my bosses to support vanilla kernels (couldn't figure out .config, though). I had to realize I was wrong. At least the market segment my employer is addressing is served very well with the current "Enterprise Linux" model.
Enterprise distributions and free software
Posted Mar 18, 2011 18:29 UTC (Fri) by geek (guest, #45074) [Link]