RealtimeKit and the audio problem
| Did you know...? LWN.net is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net. |
Skip-free audio and video playback is a fundamental expectation for many - if not most - Linux users. Given the importance of this feature and the increase in hardware performance over the years, one would think that the audio latency problem would have been solved some time ago. The recent posting of (and mixed reception for) the "RealtimeKit" mechanism shows that this issue remains open, though, and that we are still short of a consensus on how it should be solved.
Audio skipping can have a number of causes. If the stream is traveling over the net, those causes may be entirely external to the system involved. The rest of the time, resource contention is usually the problem. And, often, the limiting resource is the CPU. The response time requirements for audio are not especially tight, but, if the processor gets busy with something else for too long, it's still possible that the system will fail to get audio data to the hardware before the stream runs dry. If the sound card runs out of sound to play, it will go silent, thus introducing a skip into the audio stream. Even if the skip improves the material being played (one of the current flood of Michael Jackson retrospectives, say), users still tend to get unhappy.
Efforts to improve audio latency have taken several forms. One is the ongoing effort to identify and fix latency problems throughout the kernel. New scheduling algorithms (the completely fair scheduler, for example) have also helped. But, even after all that work has been done, there seems to be little alternative to running core audio processing code with realtime scheduling priority. And that is where the trouble starts.
More than four years ago, the audio community came forward with its proposed solution to the problem: the realtime security module. This module used the Linux security module (LSM) API to allow the system administrator to grant realtime scheduling privileges to specific users and groups. This solution slightly reduced the security of the system (opening it up to denial of service attacks by the privileged users), but it also solved the latency issues in ways which made audio developers happy.
Kernel developers didn't like this approach, though. The seeming misuse of the LSM API - which is supposed to only limit privileges, never enhance them - was part of the problem. But the realtime module just looked like an ad hoc solution that would not stand the test of time. It was never merged; the kernel developers, instead, opted for an approach based on resource limits. As of 2.6.12, any process is allowed to set a realtime priority up to the value of its RLIMIT_RTPRIO limit. By default, this limit does not allow any realtime scheduling at all, but the system administrator can change the default for specific users or groups by editing the limits.conf file read by the PAM subsystem.
This feature would seem to solve the problem, and, indeed, the media-focused distributions make good use of it. The major distributions tend not to use RLIMIT_RTPRIO, though, because it makes it so easy for a process (malicious or just buggy) to completely freeze the machine. Once a process with realtime priority goes into a tight loop, there is little that the user or administrator can do to stop it short of hitting the reset button. That sort of behavior creates unhappy users and inflammatory bug tracker entries - both things that distributors hate. So those distributors have mostly avoided enabling this feature.
More recent kernels have seen the addition of features which could mitigate this problem somewhat. A new limit (RLIMIT_RTTIME) sets an upper bound on the amount of time a realtime process can monopolize the CPU; after that time, it must make a blocking system call or, eventually, be killed by the kernel. This limit solves the rogue process problem, but it does little against deliberate denial-of-service attacks, which can get around the limit by continually forking new processes. As a result, RLIMIT_RTTIME doesn't make nervous distributors feel much better.
The other feature of note is realtime group scheduling, which allows a group of processes to be given a "realtime bandwidth" value limiting the amount of available CPU time the group can use at realtime priority. That, in turn, limits ability of the group as a whole to completely take over the system. The group scheduling feature looks like it should be a complete solution to the problem; it allows groups of processes to be given access to the realtime scheduler while limiting their ability to affect the overall operation of the system.
When Lennart Poettering set out to solve the audio skipping problem, though, he didn't use any of the above solutions. The security issues associated with resource limits were more than he was willing to deal with, and he describes the group scheduling feature this way:
It is true that a process cannot simultaneously be in a container-related control group and an audio-related control group if both groups want to control scheduling-related parameters.
Lennart's proposal is a new daemon called "RealtimeKit"; it has already found its way into the Fedora Rawhide distribution. The RealtimeKit daemon has a relatively simple job: it grants realtime scheduling priority to processes in response to requests sent via D-Bus. There are, of course, some catches:
- Any process requesting realtime priority must have the
RLIMIT_RTTIME limit set to ensure that it cannot completely
take over the system.
- There are administrator-set limits on the number of processes which can
be running with realtime priority at any given time.
- The requesting process must have the SCHED_RESET_ON_FORK policy flag set in the kernel.
The SCHED_RESET_ON_FORK flag is implemented by a kernel patch written by Lennart and accepted into Ingo Molnar's "tip" tree; this patch has not, as of this writing, been merged for 2.6.31. This flag, when set, prevents any child processes from inheriting any enhanced scheduling priorities from the parent. It thus is effective against fork bombs and other multi-process attacks; it allows RealtimeKit to give realtime priority to a single process in the knowledge that this priority will not be passed on to any others.
As a solution, RealtimeKit looks like it should work, but its reception in the audio development community was chilly at best. As Paul Davis put it:
There are a number of reasons for the objections to RealtimeKit. It adds a dependency on D-Bus regardless of whether audio developers want it. It's far from a POSIX interface. RealtimeKit takes certain decisions (such as whether to use the round-robin or FIFO scheduling classes) out of the developers' hands. It's not sufficient for the needs of pro audio users. And so on. But the real complaints would appear to be these:
- The audio community feels a little burned. They were told four years
ago that they needed to drop their preferred solution (the realtime
security module) in favor of the rlimit-based solution, which is now,
in turn, being pushed aside for RealtimeKit. How long will it be,
they wonder, until yet another solution is put forward as the real
answer?
- Nobody asked the audio developers how they would like a solution to look; instead, RealtimeKit was simply presented to them as the new way to go.
Lennart's response suggests that he's not likely to go asking the Linux audio development ("lad") community for input in the future:
More seriously, he points out that RealtimeKit does not break the existing rlimit-based system. Instead, it just adds an option for distributors who want to make realtime scheduling available to specific processes in a safe way. So nothing which works now will stop working under RealtimeKit. That is little comfort, though, to developers and users who feel that they will be forced to run RealtimeKit to use their audio applications in the future.
The Linux audio community contains no end of highly talented and highly motivated developers. But audio support under Linux still falls short of what it should really be. Audio development has suffered from a lack of consensus on solutions, a lack of communications between different development communities, and a lack of a maintainer with a view of the full problem. So it is not surprising that our audio applications don't always play well together and don't always work as well as we would like.
Lennart has dedicated a good part of his life toward improving this situation. A certain amount of controversy and pain has accompanied this work; one need not look very far to find no end of PulseAudio horror stories. But PulseAudio seems to be getting better, and Linux may well be getting closer to having basic (non-professional) audio "just work" out of the box. The goals of this work are hard to criticize, and criticism of the results might just be on the wane.
Perhaps RealtimeKit is the missing link which will enable distributors to improve audio responsiveness; Fedora looks to be the laboratory in which this particular experiment will be conducted. If RealtimeKit works as advertised, the audio community will eventually move to make use of it - regardless of whether they were asked ahead of time or not. For better or for worse, that's often how our community works: problems are solved not by talk, but by a determined developer who creates code that works.
(See this
README file for more information on RealtimeKit).
(Log in to post comments)
RealtimeKit and the audio problem
Posted Jul 1, 2009 19:44 UTC (Wed) by MisterIO (guest, #36192) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:02 UTC (Wed) by ms (subscriber, #41272) [Link]
Get Jack running with RT priorities to an external USB soundcard (I'm using
a Lexicon Omega). Then do a big apt-get dist-upgrade or something which does
a lot of IO - bonnie++ will probably do it. You'll get sound skips. And
this isn't on poxy systems - I'm on a Quad Core, 2.6GHz, 4GB RAM and
(software) RAID 1. Heavy IO just kills low latency audio.
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:12 UTC (Wed) by MisterIO (guest, #36192) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:24 UTC (Wed) by ms (subscriber, #41272) [Link]
were on, is the solution to only use one or the other?
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:33 UTC (Wed) by dlang (guest, #313) [Link]
as another poster mentioned, the definition of 'acceptable' is very poor.
larger buffers can solve most skipping problems (at the cost of added latency)
for audio-only playback, latency of a half second to a second could be acceptable (as long as you can silence the playback immediatly)
for video/audio playback significant latency is also acceptable, as long as you can know how much it is so that the video and audio can be delayed the same amount.
for recording, latency also isn't important
for live manipulation of sound (record sound, modify it, and play it back) latency is critical.
wipers
Posted Jul 2, 2009 0:00 UTC (Thu) by xoddam (subscriber, #2322) [Link]
wipers
Posted Jul 2, 2009 9:26 UTC (Thu) by nix (subscriber, #2304) [Link]
Or perhaps arts.
:)
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:15 UTC (Wed) by mjthayer (guest, #39183) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:15 UTC (Wed) by macros (subscriber, #6699) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 0:05 UTC (Thu) by xoddam (subscriber, #2322) [Link]
For an example of an operating system that does it 'right', see http://lwn.net/Articles/270081/.
RealtimeKit and the audio problem
Posted Jul 1, 2009 21:34 UTC (Wed) by Martin.Schwenke (subscriber, #7764) [Link]
When doing multitrack recording you can be mixing several existing tracks, applying effects to them, playing the result (also with effects) and recording several new tracks (also with effects!). That can put a very high load on the system. Then, if you get audio skips when recording the 1 time in 20 you're able to play the perfect guitar solo, the recording is screwed.
RealtimeKit and the audio problem
Posted Jul 1, 2009 21:58 UTC (Wed) by dlang (guest, #313) [Link]
it's recording with real-time playback, which is the most demanding thing to do. if you then overload the box, what is the system supposed to stop doing?
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:48 UTC (Wed) by drag (guest, #31333) [Link]
Saying "I don't know what these people are doing when listenning to music" is just about as a ignorant thing as you can possibly say. It's borderline insulting.
Audio processing is a very intensive realtime thing. Very. It's, in fact, absolutely critical for any sort of digital audio workstation environment. It's as critical as having reliable TCP/IP stack for a web server.
> if you then overload the box, what is the system supposed to stop doing?
Disk I/O for starters. I/O to the video card. I/O to the network. In fact it should absolute halt everything non-critical in a effort to keep the important applications working properly and doing what that particular application requires.
What is critical in a file server is decidely uncritical in audio situation.
HOWEVER...
You have to understand that we are not just talking about 'overload'.
Humans can perceive audio latency of about 100ms or so. It requires that a audio professional be able to control audio syncing issues down to about that much, give or take. You have to match beats, match vocals, sync audio and video, etc etc.
--------------------
Take this for example:
I have a MIDI controller. It's a very simple one. Most people would call it a 'keyboard'. But it doesn't play music notes. What it does is that it's a midi controller shaped like a keyboard. I has the traditional piano keys and a few knobs and sliders. It plugs into my Linux box using a USB adapter. It creates a virtual midi device. When I press a key or turn a knob this sends Midi events to the PC.
Midi events are not musical events. They are mearly some sort of 'X happenned' event. Then it's up to various music programs or other midi hardware to interpret that midi event to mean something.
Typically, in Linux, this means using a software synth. A software synth has a number of pre-recorded 'sounds' that play some sort of noise. So my midi keyboard sends that event to the software synth, that synth in turns plays a PCM audio stream that is that particular sound that is then modulated to some sort of note.
Now that noise can be anything. A drum-like noise, a piano, a cat meow, or whatever.
Then I typically route that PCM audio from the software synth through another program that then performs some sort of audio distortion on it. Makes it sound smoother, less digitalized, maybe add some echo or depth or whatever else that I think sounds cool.
This way I can take a noise produced by the software synth and make it sound like, say, a grand piano in a hall or something like that.
Then that modulated PCM audio is then sent to my audio card that then plays the sound out over the speakers.
So the time from when I press the keyboard to the time when I hear the sound will be the total latency:
So in summary that looks like this:
My finger --> Midi keyboard keypress --> USB-Midi event --> Jackd midi routing --> Software synth --> Jackd PCM audio routing --> realtime audio/distortion program --> Jack PCM audio routing ---> sound card --> stereo receiver --> speaker ---> My Ear.
It's suprisingly complicated isn't it? Even with a somewhat fast computer your looking at continious 60-70% CPU load in many cases just to keep the sound card feed and the audio worker happy. Dual/Quad cores can help a lot, obviously.
And that is a SIMPLE setup. Each step along the way will add it's own small amnount of latency. You want the OS to add as little latency as possible for performing tasks that are not immediately critical for the task at hand. Like it doesn't matter if the disk I/O doesn't happen immediately.. the requirements to keep the sound card buffer feed is a much higher priority in this sort of situation.
So as you can imagine a piano player with a half a second of latency from when they press a key to when they hear a sound is completely unacceptable. It's just shit and will screw them up.
I just play around so I am not critical, but it's certainly unbearable to have a big delay. Like I said that most latency below a 100ms will be generally unpercievable. But a small amount preceived audio latency isn't that bad. After all it takes a while for sound to reach your ears. Sound is kinda slow compared to other things.
------------------------------
Now during the normal operation of Linux there are all sorts of things that introduce latency. The kernel blocks when doing lots of different things and all that happy horseshit.
Having high latency is a good thing, often, in a server. When your dealing with network stuff people expect delays and short pauses when doing stuff so that you can take that into advantage to be able to do your processing tasks in a very fast and efficient manner. Having your system continiously interrupting processes every few milliseconds is going to have a very detrimental effect on overal performance.
If you want to look at it like this:
* High latency behavior == efficient processing == good for servers
* Low latency behavior == desirable behavior == good for digital audio workstations.
On early versions of 2.6 they had system latencies... when the entire system would just pause and be crunching on some task or another for 500ms or so. It'll just blank out, go out ot lunch. Make it impossible for any other application to do any work.
And this isn't even under 'high load' or 'moderate load'. It's just normal operating proceedure. (Obviously huge efforts have been made to improve performance)
When your configuring your audio applications to perform in realtime and you get a lag spike of 250ms or 500ms this does not just cause delay it causes huge problems. It causes audio tracks to fall out of sync. Causes video and audio to go out of sync, which causes performance playback problems and bad video performance as the application struggles to keep things in sync. It causes squaks and pops and other audio artifacts in audio playbacks and audio recordings.
----------------------
This is a serious problem in other OSes too.
In Windows XP, even though it is very popular, the Win32 API and audio stack is so miserable and realtime performance is so poor that third parties had to create their own driver model for XP called ASIO.
Even with ASIO, though, you have to be very careful about doing disk I/O in Windows to avoid audio latency and the resulting undesirable audio distortions and artifacts. Typically you have to disable your virus protection and things like that.
RealtimeKit and the audio problem
Posted Jul 1, 2009 23:07 UTC (Wed) by dlang (guest, #313) [Link]
but the fundamental problem is that no system is going to know that the priorities that you want for your audio processing are the righ priorities.
you say kill disk I/O and network I/O
what if you are using one of those for your audio? how is the system supposed to know?
it's a very different thing to say that the system should prioritize your audio over everything else than to say that the system shouldn't have 500ms pauses. the latter is something that everyone can agree to, the former is not.
linux has gotten much better since the early 2.6 days, and I don't think that anyone is saying that there isn't room for further improvement (they may not know where the problems are, but I don't see anyone arguing that there are definantly no problems)
but if you want to dedicate your box to audio processing, you should tell the system that you want to do so. it has no way of (accurately) knowing.
I am not saying that realtimekit is the right answer (I have no idea if it is or not, and am also not a fan of the idea that D-bus is being used for everything nowdays) but the idea that any desktop/server system should out-of-the-box be suitable for heavy audio processing is just wrong.
it should be good for _light_ audio processing (up to the squishy point where the load on the box causes enough contention for resources that you start having an unacceptable number of problems)
RealtimeKit and the audio problem
Posted Jul 2, 2009 6:04 UTC (Thu) by drag (guest, #31333) [Link]
> you say kill disk I/O and network I/O
> what if you are using one of those for your audio? how is the system supposed to know?
Probably by PID number. I was under the impression that the Linux I/O scedualer was capable of prioritizing different things.
The disk access can almost always be delayed because of the nature of cache in ram. A system can't prioritize incomming network packets becuase you need to analyze the packet first to determine weither or not it's priority, but processes can be give priority on network access.
This is all stuff that is already in the kernel.
> it's a very different thing to say that the system should prioritize your audio over everything else than to say that the system shouldn't have 500ms pauses. the latter is something that everyone can agree to, the former is not.
Ya... Its more complicated then you think here, I think.
Very low-latency = low performance. A system that has very good low-latency behavior tends to be inefficient. This means reduced performance for web servers, file servers, database servers, encryption, etc etc.
So it's certainly not something everybody can agree on. So it has to be tunable. Which is the point behind having a configurable system were you can assign "realtime" performance to key processes at the detriment for the rest of the system.
Audio is just one of the many times you need to have good "realtime" performance.
Any time your having a computer interacting with the real world.. You can't pause real-life. You can't put events in a buffer and schedual things so that you can have good performance.. many times the computer needs to react to events outside of it's control in a timely manner.
Robot controls, scientific instrumentation, vehicle navigation systems, vehicle control systems, cell phones, VoIP devices, wallstreet stuff blah blah blah.
It'll be nearly impossible to make Linux ever perform "hard realtime" but certainly things can be improved to the point were the utility of the vanilla Linux kernel can be increased massively.
RealtimeKit and the audio problem
Posted Jul 2, 2009 6:20 UTC (Thu) by elanthis (guest, #6227) [Link]
The application is real-time. It is in full control, not the OS. When it knows it needs to fill an audio buffer, it does that. When it knows it needs to write a sample to disk, it does that. When it knows it has nothing better to do with its time for a little while, it lets the kernel do some stuff.
You are being OS centric. Let's be perfectly clear here: the OS is unimportant. It doesn't matter, at all. Linux, Windows, Solaris, OS X, they're all totally and absolutely irrelevant. The fact is, a computer is only useful for running applications, be they your eCommerce server solution, your audio recording apps, or telecommunications/GSM phone app. The only reason we even run had an OS at all is for convenience: it's easier to be able to run a couple apps together rather than having to reboot to change apps, it's cheaper to use a single computer to run your browser and email client at the same time rather than needing two computers, it's cheaper to let six students share a single mainframe rather than each needing their own, etc. We put a lot of effort into making these OSes powerful and pleasant to use, but at the end of the day, the OS is only there to run our applications.
When the application developers have to start bending over backwards to appease the OS instead of having the OS bend over backwards to run the application the way it needs to be run, something is wrong. It's like selling your car for gas money: you give up something you need in exchange for something you then suddenly no longer need. Without the applications, the OS is entirely freaking useless.
Therefor, if Linux is incapable of knowing what the audio applications want and can't automagically optimize appropriately, the solution is very simple: let the applications tell Linux exactly what they need and how they need it. Which is exactly what real-time systems in the kernel (and other related facilities) are meant to do.
If you dislike the security implications of those facilities, DON'T USE THEM. The audio folks don't give two shits about good server security or what not, because they're not running servers or multi-user desktops or even Web-based network-centric applications. They're running a dedicated set of audio processing apps and little else, and anything else they are running can quite safely be shoved into the background and starved of any CPU or I/O time while the audio app is doing its thing.
RealtimeKit and the audio problem
Posted Jul 2, 2009 7:47 UTC (Thu) by dlang (guest, #313) [Link]
just don't consider the result a general purpose system, it's become a dedicated appliance that could be totally useless for anything else.
in fact, other than possibly the need for drivers, you are better off going with something like DOS for your appliance.
that has a place, but it also will have zero effect on the linux distributions, because they are not making single-purpose appliance builds, they are making an OS that will run many different things, frequently at the same time. and while people don't like their music playback to stutter, they usually aren't willing to sacrafice the lots of performance when playing games, or running autocad, etc just so that the background music never skips
RealtimeKit and the audio problem
Posted Jul 1, 2009 23:47 UTC (Wed) by jwoithe (subscriber, #10521) [Link]
In some cases latency compensation (as implemented in Ardour for example) can help out. In others (such as running incoming audio through realtime effects for live work) there is no getting around the need to lower system latency as much as possible. In this latter case the target latency needs to be somewhat lower than 100 ms - probably closer to 10 ms or there abouts.
For those working with low system latencies (of the order of 10 ms) the chances of audio skips and other related problems are *much* higher than they are for those who are able to work with higher latencies.
RealtimeKit and the audio problem
Posted Jul 2, 2009 23:19 UTC (Thu) by Tuxie (guest, #47191) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:03 UTC (Wed) by MisterIO (guest, #36192) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 6:15 UTC (Thu) by drag (guest, #31333) [Link]
That is a issue of people just not wanting change, it seems. Its not entirely unreasonable.. People have put a lot of work getting Linux audio distros out there right now, and have put work into wikis and other documentation, and lots of other little things and changing the way things work will have a cascading effective of PITA-ness.
But the trade-offs should be worth it.
Combine this realtimekit with things like "policykit" can make things much easier to configure and avoid a lot off the issues you run into when you try to explain to artists about having to editing text files and running various command line stuff. You can make it possible to give users ability to do realtime stuff without also giving them the ability to run sudo or su and things like that.
Also time to time normal desktop stuff needs this. Pulse-Audio is much better then it used to be, but still having realtime performance options for PA does help quite a bit with some people's setup. It should be a easy option to enable for users.
RealtimeKit and the audio problem
Posted Jul 1, 2009 23:21 UTC (Wed) by nix (subscriber, #2304) [Link]
else is going on, as long as the system isn't in single-user mode or
something emergency-maintenancy like that. I'm not going to stop listening
to something just because I've got something that happens to be
heavyweight going on at the same time --- unless, that is, it starts to
skip or the machine starts thrashing or something like that.
(Just now I was doing a foreground compilation, a git gc --aggressive of a
500Mb repository, a fairly large 'git grep's, while a background tinderbox
chewed away on the same machine... oh, and the music was coming over NFS,
just to add insult to injury.
There were no skips, but this *is* a quad-core Nehalem with 24Gb RAM so I
suppose skipping would be rather unlikely.)
RealtimeKit and the audio problem
Posted Jul 2, 2009 10:20 UTC (Thu) by nye (guest, #51576) [Link]
(For my part I only ever get audio skips when my system is swapping like a mad thing. I was under the impression from the rest of this thread that *playback* is a non-issue.)
RealtimeKit and the audio problem
Posted Jul 2, 2009 11:23 UTC (Thu) by nix (subscriber, #2304) [Link]
But I'd be astounded if the machine it was coming from had any problems
with disk I/O (similarly oversupplied with RAM and with half a gig of
cache on its RAID card: music does not need 250Mb/s of disk bandwidth!
Hell, at the quality I listen at, it doesn't need 250Mb/hr...)
And yes, it does seem that recording is the problem area, as so often when
the real world with its harsh latency bounds gets involved.
RealtimeKit and the audio problem
Posted Jul 2, 2009 16:25 UTC (Thu) by NAR (subscriber, #1313) [Link]
Because it's very hard for me to think about a real world example when the music I was listening skipped.Had pulseaudio installed? OK, actually it didn't make audio skip when I last tried, it made the video skip.
RealtimeKit and the audio problem
Posted Jul 1, 2009 19:51 UTC (Wed) by jd (subscriber, #26381) [Link]
Unnecessary communications overhead or context-switching overhead will surely kill any kind of daisy-chaining mechanism for audio processing because of latency, although it shouldn't impact programs that are complete in and of themselves. Now, of course, this begs the question of when these overheads are unnecessary. I'd argue that this is something that can only be answered by experimentation, not by politics.
OS-specific layers are going to complicate things for apps-writers, who are not going to want one audio module for Windows, another for Linux and a third for all the *BSDs. Abstraction layers to hide the specifics just adds to the footprint, to the latency and to the number of places bugs can hide. Certain models are also extremely hard to hide by abstraction. Again, though, what is acceptable can really only be seen by trying the experiment and observing.
As far as I can tell, the underlying problem is that there is no good specification of what "acceptable" even is. There needs to be some solid standard for the "best", "median", "mean" and "worst" cases against which solutions can be measured. The cases don't necessarily refer to the percentage of CPU resources available, they could also refer to the driver architecture used, the hardware bus bandwidth available, the number of audio cards present, the amount of pre-processing required, the complexity of the API, the portability of the API, etc.
If the minimum criteria were laid down in stone, for all the different issues, then there could be no argument over whether a solution was acceptable or not. If it meets the criteria, it's acceptable. If it doesn't, it isn't. It won't completely eliminate the politics (nothing can do that, I fear) but politics will always be loudest where terms are defined the least. Everyone will always try and get their interpretation to be the "one true interpretation" and that's a Bad Idea. It's hard to put a number to things like complexity and portability, but it's surely still easier than the usual wading up to the armpits in virtual blood.
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:05 UTC (Wed) by alankila (guest, #47141) [Link]
The architecture of JACK is based on context-switching and daisy-chaining audio buffers from one process to next. The latency issue is solved by scheduler immediately noticing the next runnable process in the chain.
RealtimeKit and the audio problem
Posted Jul 1, 2009 20:57 UTC (Wed) by dw (guest, #12017) [Link]
Might it be simpler to enhance the driver API to allow flagging of devices with scheduling hints, such that the need to tweak the scheduler at all is contained entirely within the kernel? Maybe a rule like "elevate the opening process to realtime priority if FLAG_LOW_LATENCY is set, and the device is not already open,." That could be set somewhere inside ALSA and propagate automatically to user space processes.
RealtimeKit and the audio problem
Posted Jul 1, 2009 21:42 UTC (Wed) by tetromino (subscriber, #33846) [Link]
So the solution is to either
a. enable rtprio limits for your normal user account (something that I am forced to do, but which carries obvious security risks); or
b. design a secure mechanism for granting realtime priority to specific processes, using policykit or some similar mechanism (which is what Lennart is doing with his RealtimeKit).
Obviously, b. is the correct solution, and I applaud Lennart for trying to make Linux sound Finally Work Right.
RealtimeKit and the audio problem
Posted Jul 1, 2009 21:54 UTC (Wed) by dw (guest, #12017) [Link]
This additionally frees up the kernel folk to replace what could be perceived as a hack, with a better solution later, without braking the user space interface (which simply doesn't exist).
RealtimeKit and the audio problem
Posted Jul 1, 2009 21:56 UTC (Wed) by dw (guest, #12017) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:04 UTC (Wed) by dlang (guest, #313) [Link]
that may be true for you, but is it really true that it's _always_ more important to get a system beep than to write to the database file? I don't think so.
to some extent, the fact that you need priority is a system bug (when the system is not overloaded), and that should be fixed. but that won't be enough for people pushing their systems to the limit with audio processing (but I would argue that if you want to make the audio processing the most important thing on the box, you need to tell the kernel your priorities)
many times when you are running an high-end audio app you are also doing a bunch of number crunching (to compress the data, to display graphs of the data, to uncompress the data) how is the system supposed to know what parts of this you are willing to have slow down?
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:10 UTC (Wed) by dw (guest, #12017) [Link]
RealtimeKit and the audio problem
Posted Jul 1, 2009 22:08 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 0:27 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
Firstly, the rtkit solution requires that any application wishing to use rtkit becomes dependent on dbus. For many authors (particularly of software aimed more at professional use) this is not regarded as a good thing for a number of reasons (technical and otherwise).
Second, as was mentioned in the article, the RLIMIT_RTTIME limit was presented as "the answer" in the 2.6.12 days. Many of the shortcomings of RLIMIT_RTTIME were mentioned at the time RLIMIT_RTTIME came into existance. The stated solution to these was that userspace support of RLIMIT_RTTIME would take care of them. The problem is that the promised userspace support applications/daemons were never written. It is argued that instead of introducing yet another mechanism we should instead be concentrating on making RLIMIT_RTTIME work as it was intended. To take one example, one of the distributor's concerns about RLIMIT_RTTIME is that it grants RT scheduling functionality to all processes started by a given user. Technically though such a broad grant of the RT privilege isn't necessary as demonstrated by other userspace tools such as set_rlimits(*). The reason that it *is* an issue is because the integrated tools to make it happen don't exist.
Third, some developers have claimed (on the LAD mailing list) that in practice rtkit does not in fact protect the system against very much at all. I haven't evaluated the arguments in details but to my knowledge their concerns have not yet been addressed.
Forthly, many (but not all) developers on the LAD list are associated with software which is aimed more at the professional user than the typical "desktop" user. They see rtkit as something originating from the desktop scene which is being forced onto professional users without any thought as to what effect this might have on the pro users.
Finally, rtkit is Linux-specific while many audio developers support their software on a multitude of platforms and operating systems (both free and otherwise). Using rtkit introduces yet another point of difference that they have to manage across the supported systems - an overhead many, it seems, are unwilling to accept.
As a result of these and other issues aired on the LAD mailing list, many audio developers simply don't see rtkit as a viable solution to the audio problem from their point of view. As a result of the clearly polarised reception of rtkit I'm not entirely sure that "the audio community will eventually move to make use of it". My concern is that it will in fact further fragment the community - that we'll see developers of desktop applications embracing it while those of pro applications will not. This will hardly lead to an improved Linux audio experience all-round.
I certainly have no arguments with the motivating reasons behind the development of rtkit - setting up audio under Linux is problematic on a number of fronts and needs to be improved. While I can understand Lennart's frustrations and the reasons behind rtkit's structure I can't help but wonder whether discussing the basic problem with a *small* group of significant stakeholders representing both desktop and pro audio users might have resulted in a system which was acceptable to all.
(*) Disclaimer: I wrote set_rlimits.
DBus
Posted Jul 2, 2009 2:06 UTC (Thu) by dmarti (subscriber, #11625) [Link]
Would it work to have the distribution bundle a simple utility that would do the necessary D-BUS communication, then exec(3) the real application?
DBus
Posted Jul 2, 2009 2:39 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
However, more importantly, even though the "simple utility" could get RT scheduling priorities (and permission to set them) from rtkit, the SCHED_RESET_ON_FORK flag would ensure that all this is lost across the exec() call. In this scenario the "real application" will never inherit the RT privileges.
Regards
jonathan
DBus
Posted Jul 2, 2009 4:34 UTC (Thu) by salimma (subscriber, #34460) [Link]
DBus
Posted Jul 2, 2009 5:08 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
However, there could be another issue. As I understand it, rtkit doesn't grant the requesting process permission to set an RT priority though sched_setscheduler(). Instead it grants a requested RT priority to the process. In other words it functions as a replacement for sched_setscheduler(). As I previously explained, many high-performance audio applications (for example, ardour, jack) are multithreaded and request different scheduling algorithms and priorities for different threads. The simple wrapper application would only be able to grant a single RT priority to what would become the audio application's main thread. That application (or any of its threads) would not be able to alter that priority (or maybe even drop RT scheduling).
Having a single global RT priority for all threads isn't flexible enough to ensure smooth operation of these applications (and in fact, in most cases only a few carefully chosen threads need to run at RT priority anyway).
Regards
jonathan
RealtimeKit and the audio problem
Posted Jul 2, 2009 5:39 UTC (Thu) by PO8 (guest, #41661) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 5:42 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 0:57 UTC (Thu) by PaulWay (guest, #45600) [Link]
One is the SCHED_RESET_ON_FORK flag, which solves the "busy fork bomb" problem. AFAICS this removes the major problem with trusting RLIMIT_RTTIME.
The other is the RealtimeKit daemon, which allows processes to gain realtime priority within the control of the distribution and without the user having to edit nasty config files. It relies on having SCHED_RESET_ON_FORK available, but the reverse is not true.
This makes it a lot easier to accept the two things independently. One is a pure kernel patch, and the other is a userspace control for it.
Have fun,
Paul
RealtimeKit and the audio problem
Posted Jul 2, 2009 2:07 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
> problem. AFAICS this removes the major problem with trusting RLIMIT_RTTIME.
Yes, it addresses the fork bomb problem. However, a thread-based attack (that is, a "busy clone bomb" for want of a better description) is not prevented by the SCHED_RESET_ON_FORK functionality (if I understand things correctly).
Regards
jonathan
RealtimeKit and the audio problem
Posted Jul 2, 2009 4:40 UTC (Thu) by salimma (subscriber, #34460) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 4:54 UTC (Thu) by jwoithe (subscriber, #10521) [Link]
RealtimeKit and the audio problem
Posted Sep 28, 2009 5:31 UTC (Mon) by naptastic (guest, #60139) [Link]
1. Malicious software exploits some other vulnerability first, in which case rtkit or PAM limits are unlikely to do any good anyway.
2. A naive user clicks a link or opens an email with malicious code. Correct me if I'm wrong, but I'm under the impression that neither Firefox nor Thunderbird are usually run with real-time permissions?
3. A user runs malicious code they crafted themselves. It should go without saying that in this instance, real-time priority inheritance is the very least of the administrator's worries.
RealtimeKit and the audio problem
Posted Oct 1, 2009 21:16 UTC (Thu) by nix (subscriber, #2304) [Link]
bug that size in it, though.
(Mind you, it would have been every bit as devastating if it hadn't been
realtime: it *is* still a fork bomb...)
RealtimeKit and the audio problem
Posted Jul 2, 2009 6:18 UTC (Thu) by aleXXX (subscriber, #2742) [Link]
My first impression is that it is not a solution.
* Why are cgroups not a good solution for that ? They can limit the share
of CPU time some threads get. I didn't understand what the problem is
with them.
* Why is a daemon called a "Kit" ? It seems nowadays every new thing is
some "FooKit", where each of those Kits replaces a well-known UNIX
mechanism with something undocumented nobody knows. It also seems that
everything which is called a "Kit" is kind-of implicitely accepted as the
new official standard way to do things.
* Where is the advantage of having a daemon which assigns RT priority to
processes if they ask for it over giving those processes the permission
to assign them RT priority themselves ? Isn't this just a big backdoor ?
Like a "friend" in a C++ class ? "I'm not allowed to do this, but I know
somebody who is, and he does everything for me I ask for" ?
* It is correct that SCHED_FIFO or SCHED_RR would be risky without
limitations. But this RealtimeKit looks like a big hack to get around
unsatisfying behaviour of the scheduler. There is a SCHED_EDF extension
to the normal Linux scheduler in development, where you can assign
periods and budgets to threads (which they cannot overrun). This looks to
me like a much better solution.
Alex
RealtimeKit and the audio problem
Posted Jul 2, 2009 7:40 UTC (Thu) by Frej (guest, #4165) [Link]
* Why are you named "XXX"?. It seems nowadays that every new LWN comment is done by these "XXX" guy's, where each of those XXX guys replaces a well-known-respected kernel guy with someone nobody knows. It also seems that everyone who are called "XXX" is kind-of implicitly accepted as knowledgable people, who should be trusted on how to do things.
PS: I'm guilty of lame comments from time time as well ;). We all have bad days.
RealtimeKit and the audio problem
Posted Jul 2, 2009 8:43 UTC (Thu) by aleXXX (subscriber, #2742) [Link]
<neundorf AT kde.org> (google can tell you the rest)
P.S. this is not me: http://www.myspace.com/alexneundorfband (I didn't
know a second one exists)
RealtimeKit and the audio problem
Posted Jul 2, 2009 9:23 UTC (Thu) by Frej (guest, #4165) [Link]
Cheers!
Names
Posted Jul 2, 2009 9:56 UTC (Thu) by alex (subscriber, #1355) [Link]
RealtimeKit and the audio problem
Posted Jul 2, 2009 9:43 UTC (Thu) by nix (subscriber, #2304) [Link]
name anywhere!
(but, of course, DeviceKit, which might replace it unless it doesn't, is
a 'Kit'.)
As far as I can tell 'Kit' means 'should have a command-line interface
in /usr/libexec and decent manpages but we couldn't be bothered to define
any such thing or write any documentation at all, even one describing how
to set it up: your distro vendors will know through spending hours
grubbing through the code.[1] Here, use our k00l XML-dependent
communications mechanism of the week, which unlike the kernel's own
mechanisms provides no atomicity or delivery guarantees whatsoever and has
an extra layer of security which had horrible bugs for years because
nobody understands it.'
[1] this is unfair to Lennart, perhaps. If PA is any guide RealtimeKit
will have excellent docs. Unfortunately most of the others don't.
ConsoleKit is a particularly appalling example, or was last time I looked
at it.
RealtimeKit and the audio problem
Posted Jul 9, 2009 18:11 UTC (Thu) by jwb (guest, #15467) [Link]
It used to be that Win32 weighed more heavily on the minds of free software. That's why you got HAL, a name directly taken from Windows NT.
RealtimeKit and the audio problem
Posted Oct 13, 2009 2:38 UTC (Tue) by MarkWilliamson (subscriber, #30166) [Link]
Having the kernel handle mechanisms and userspace daemons handle policy is a well established practice and a very sensible one. So having a daemon of some kind that sets up policy or can be consulted to make policy decisions seems reasonable. In this instance, though, it sounds like the userspace process is also handling part of the mechanism too. The kernel's machinery is apparently not suitable for applications to request realtime privileges, so that goes through the daemon too. This looks a bit awkward to me, as it suggests that a daemon has been created to compensate for a lack of kernel flexibility.
RealtimeKit and the audio problem
Posted Jul 2, 2009 9:52 UTC (Thu) by mlankhorst (subscriber, #52260) [Link]
RealtimeKit and the audio problem
Posted Jul 3, 2009 4:00 UTC (Fri) by nedko (guest, #59419) [Link]
Second, JACK is not designed to be run as root user.
RealtimeKit and the audio problem
Posted Jul 3, 2009 16:52 UTC (Fri) by quotemstr (subscriber, #45331) [Link]
Did you read the comment you're replying to, or did you not quite grasp it?The OP said to use the setuid mechanism, which means that the process starts as root, grants itself realtime privileges, then irrecoverably becomes a normal process in every respect except that it can elevate its scheduling priority. Your concerns simply don't apply. In fact, you could write a small wrapper to grant this realtime capability before launching the real program so that the bulk of the code doesn't have to mess with setuid-stuff at all. Err, wait -- it's been written, and it's called set_rlimits. (Granted, set_rlimits should use PAM, but that's a minor detail.)
RealtimeKit and the audio problem
Posted Jul 3, 2009 17:01 UTC (Fri) by nedko (guest, #59419) [Link]
Anyway, I just wanted to be helpful. I don't want to argue over set_rlimits like approach.
RealtimeKit and the audio problem
Posted Jul 3, 2009 12:52 UTC (Fri) by razb (guest, #43424) [Link]
I had the same problem several years ago, but in the servers arena.
in order to solve it I partitioned the solution to two:
1. UNI processors machines
2. SMP processors machines
In UNI processor I created a RT kthread that polls over RTC device.
The observed jitter was 400us.
in SMP processors machines we created a new type of scheduling , called offline. what i did is it to offload a processor and assign it with a task. jitter is 250us under heavy load. solution is open source and can be found at:
http://sos-linux.svn.sourceforge.net/viewvc/sos-linux/off...
Raz
RealtimeKit and the audio problem
Posted Jul 5, 2009 18:55 UTC (Sun) by krasic (guest, #4782) [Link]
I hope some developers here may find it interesting.
We are able to achieve ~1ms range scheduling response even during persistent 100% system load. This represented significant improvements over Linux's CFS and real-time schedulers (see paper and slide links below). That's the good news. The bad news is that we do introduce a new system call, and we require real-time applications to be modified to use the interface. In the paper, we describe how the interface is used in our own video player as well as a modified version of the X.Org X11 server.
How does our approach work (briefly?)
Our approach combines fair scheduling and real-time. We use a new system call "coop_poll()" that is used by application event loops to rendezvous with the kernel, and to share essential timing information (a "release-time") with the kernel scheduler. The kernel scheduler wakes the task up (i.e. return from coop_poll) at the application specified time. When returning, coop_poll provides timing information from the rest of the sytem (kernel and other cooperative tasks). And the whole cycle repeats---the thread later calls coop_poll to yield back to the kernel at appropriate time, in return the kernel scheduler resumes the thread very quickly when its own "release-time" arrives. Aside from short timeslices around release-time, the scheduler timeslices, divides CPU time, among tasks according to a fair queuing approach. Tasks that behave well (use the coop_poll interface as just described) will receive very low latency execution at release-times, and overall fair share of CPU over time otherwise. Our kernel scheduler provides enforcement, so a task that attempts to abuse the interface will lose low-latency execution, although it may regain it if the abuse is not persistent. All tasks receive (weighted) fair CPU allocation. There is no need for root or real-time privilidges.
We presented the work earlier this year at the EuroSys 2009 conference:
(slides)
http://eurosys2009.informatik.uni-erlangen.de/fileadmin/D...
(full paper info)
http://portal.acm.org/citation.cfm?id=1519065.1519077
Having taken a very brief look at Pulse, for example, I think it would be reasonable to modify the Pulse server to take advantage of a coop_poll enabled kernel.
-- Buck
Charles 'Buck' Krasic
Assistant Professor,
Computer Science
University of British Columbia
Vancouver, Canada
RealtimeKit and the audio problem
Posted Jul 6, 2009 9:53 UTC (Mon) by njs (guest, #40338) [Link]
I can't seem to get your paper, but if the graphs for "RT" in your slides were generated using a real high-precision wakeup mechanism then I'm very surprised. Can you confirm what you used? IIRC for 2.6.25 you still have to use a rather complex signal-based approach.
Still, getting RT-like behavior without SCHED_FIFO is quite excellent. Is the idea that coop_poll() is a poll-like syscall with two additional properties: 1) it uses hrtimers for precise wakeups, 2) when a wakeup does occur, the process gets preferential treatment in waking up *immediately*, so long as this doesn't produce unfairness over the long term?
RealtimeKit and the audio problem
Posted Jul 7, 2009 22:21 UTC (Tue) by krasic (guest, #4782) [Link]
Yes, stock linux real-time responds very quickly, but it breaks down if there are multiple CPU intensive tasks at the same time. We give an example of that in our paper (generalized to various number of simultaneous tasks).
A simpler example (not from the paper) is as follows, imagine you run the Pulse Audio server with real-time (SCHED_FIFO, SCHED_RR), and an application such as a video conferencing app, also real-time. Both Pulse and the video app can be CPU intensive. Stock Linux real-time scheduling will not provide fast response to both apps while both are simultaneously active, as the timeslice/quantum is still very large under SCHED_FIFO, and SCHED_RR. Hence one of them can experience large delay.
As for wakeup, we use the hrtimer facility directly in our scheduler.
Your summery of coop_poll() is correct, except the application's responsibility is greater than "so long as this doesn't produce unfairness over the long term". Coop_poll() returns a time value back to the application that indicates when the application should call coop_poll() again. So it isn't just that the kernel provides preferential treatment by waking immediately, the application must also yield immediately when other applications' release-times arrive (well within a slack time).
In steady-state, "cooperative" tasks are never pre-emptively context switched by the kernel, instead they always rendezvous voluntarily with the kernel schedule in coop_poll().
-- Buck
ps Perhaps this link to our paper will work better: http://www.eecg.toronto.edu/~ashvin/publications/timely-s...
RealtimeKit and the audio problem
Posted Jul 8, 2009 0:07 UTC (Wed) by njs (guest, #40338) [Link]
Thanks for the clarifications. I really like coop_poll's semantics (not that my opinion matters much). The classic approach to this problem is "you need low latency so we'll give you high priority whoops but not *that* high priority". Saying instead "you keep the same priority (= available CPU time) but we'll let you request where to spend it and give you feedback on what you got" seems to capture what's going on here much better. (Though it won't help much if it turns out that pulseaudio does, in fact, need more than its fair share of the CPU.)
I'm still a bit confused about how classic RT fails, though. IIUC, under SCHED_FIFO there *is* no scheduling quantum -- processes run until they yield. SCHED_RR does have timeslice-based preemption, but I can't see how any app you might care about on the desktop would ever use it. In practice, an app like Pulseaudio or Ekiga or whatever is going to yield very often, after each slice of work. Under high load it will become runnable again very quickly, but it still gives any other RT apps a chance to take over. So the apps create your effective scheduling quantum, not the kernel.
RealtimeKit and the audio problem
Posted Jul 8, 2009 4:26 UTC (Wed) by krasic (guest, #4782) [Link]
This is a good discussion.
I agree that compute intensive, yet time sensitive, apps could "yield often" as you describe. Indeed, coop_poll relies on that too.
A couple of points about such yielding:
1) if it so, then the difference between SCHED_RR and SCHED_FIFO is moot, as the application would be setting the quantum as you observe;
2) coop_poll allows such applications to yield more intelligently, by providing the application with direct information about when it should yield.
Point #2 can mean far fewer yields/context switches, hence lower overhead and better responsiveness. In the paper we show that a purely periodic approach (i.e. yield often regardless of whether it is absolutely needed or not) has measurably worse performance; an experiment in the paper show this where coop_poll responsiveness is ~1ms vs periodic approach at ~5ms, while at the same time coop_poll has almost 5x fewer context switches. This is part of why I think SCHED_FIFO doesn't cut it. The issue is to divide time between short rapid timeslices for time sensitive events, and longer timeslices for longer, less time sensitive computations. coop_poll can and does do this. With SCHED_FIFO, a similar result could be achieved by dividing the application threads in a way that ensures that any heavy computation runs only in separate, non real-time threads, while time-sensitive actions are within a SCHED_FIFO thread. I think having fewer threads is architecturally preferable, but I guess it is a matter of personal taste.
Also, I'd point out that the share of CPU, e.g. with Pulse, is not that big an issue. Fair share schedulers support weighted fair sharing--e.g. in Linux, the nice value is translated into the fairshare weight (both with CFS and our scheduler). Thus giving a 'nice boost' to Pulse, and other servers such as the X11 server, is safe and arguably sensible thing to do.
-- Buck
RealtimeKit and the audio problem
Posted Jul 20, 2009 16:15 UTC (Mon) by iive (guest, #59638) [Link]
Your approach is heading in the right direction, but it is not there yet. It just implements precise sleep(). It needs one more step.
I'll try to explain it in the given audio context.
Audio cards capture and playback sound via DMA (Direct Memory Access). This means that the CPU is not pulling the card for every sample, it just gives some buffer, says "work here" and the card takes or puts memory values directly through the bus. When the card is done with the buffer it interrupts the cpu (irq) to request more work (actually it does it earlier to prevent underruns).
Now, imagine the following situation. Card plays, irq comes and says "need more", driver provides what it have. When irq handler is done, instead of restoring previously running task, we check if there is task that have open the device and is sleeping at poll/select. If it is, make so that task is immediately scheduled. The task is expected to feeds more data to the driver.
In that case the latency is derivate from the buffered data size. One benefit is that the task switching is almost free as it is done anyway. Second benefit is that only sleeping tasks are waken up (no infinite working). The biggest benefit is that this could work without real-time priority.
One problems is how would concurrent irq bound task operate (aka 2 different devices with 2 different tasks, interrupting each other). Other is that device drivers should communicate with scheduler when to task switch and when not to. The biggest problem is if the hardware device doesn't generate the needed irq, but in that case usually it is the driver that does the pulling (using timer).
You may notice that this whole paradigm is similar to device driver workqueues. In short the real time process is used as userspace driver. The even better thing is that the kernel already provides some api for userspace i/o drivers, including the irq handling, so this may be good starting point to implement that functionality, as they would need it too.
RealtimeKit and the audio problem
Posted Dec 20, 2010 16:41 UTC (Mon) by lievenmoors (guest, #53544) [Link]
this cooperative scheduling scheme...
Now that realtime preemption is becoming
mainstream, it might be time to move on to
something new :-)
Very interesting paper btw...
Thanks for sharing.