Bug 17720

This might also explain https://launchpad.net/bugs/184519, which has 26 duplicates already, with this stack trace:

  http://launchpadlibrarian.net/12890099/Stacktrace.txt

(crash in vt_thread_start). Or was that actually fixed by dfcab49480565a7bcf71752c5b39eb367df81a19 "cleanly shutdown event logging threads"?

Ah, to answer myself, this is probably the case: https://launchpad.net/bugs/196724 is another set of 50 duplicates about the same "crash in thread start" problem, which was fixed with the 0.3 patch I mentioned.

I experience these 63 spawned processes as well, if for nothing else
(all CoW memory etc accounted for) it hogs the scheduler by making 
it slower to sort the CFS RB-tree which is O(log n).

An Ubuntu user proposed the following patch to
autoconfigure the maximum number of consoles from sysfs:

http://launchpadlibrarian.net/15940846/0001-Autoconfigure-maximum-number-of-kernel-consoles.patch

which looks sane to me.

I made a graphical tool to display process migration etc in the system
and the plethora of console-kit-daemons really cloud the view.

(In reply to comment #3)
> I experience these 63 spawned processes as well, if for nothing else
> (all CoW memory etc accounted for) it hogs the scheduler by making 
> it slower to sort the CFS RB-tree which is O(log n).
> 
> An Ubuntu user proposed the following patch to
> autoconfigure the maximum number of consoles from sysfs:
> 
> http://launchpadlibrarian.net/15940846/0001-Autoconfigure-maximum-number-of-kernel-consoles.patch
> 
> which looks sane to me.
> 
> I made a graphical tool to display process migration etc in the system
> and the plethora of console-kit-daemons really cloud the view.
> 

I am that user. The patch broke the logout process of Ubuntu: the
currently logged in user lost the ability to power off the system,
only logout worked.

Sorry for the long delay.  However, it seems like this bug has a few different issues in it.  The issue that is reported in the initial report is actually not a bug but a design decision.  Once we improve or replace the VT subsystem we can do better than this.  Until then we don't have another way.

Aha it's not a bug it's a feature.

Created attachment 24210 [details] [review]
Patch to console kit to use alternative wait mechanism

Here is a patch for console kit that removes the numerous threads by utilizing a new kernel ioctl.  (This requires a kernel patch as well).

Created attachment 24211 [details] [review]
Kernel patch adds a VT_WAITSWITCH ioctl command

This patch adds an ioctl to the kernel which console-kit can use to wait on a terminal switch without creating 60 threads.  It's not incredibly pretty but it seems to work and demonstrates a straightforward solution.

Does anybody know who the official maintainer of this package is?  I am willing to work on getting the patch included in the kernel if this approach is acceptable to the ConsoleKit team, but I don't want to bother with the kernel side of things unless ConsoleKit will incorporate the changes also.  I tried emailing Jon McCann but did not get any response.

(In reply to comment #9)
> Does anybody know who the official maintainer of this package is?  I am willing

William Jon McCann is the official maintainer.

> to work on getting the patch included in the kernel if this approach is
> acceptable to the ConsoleKit team, but I don't want to bother with the kernel
> side of things unless ConsoleKit will incorporate the changes also.  I tried
> emailing Jon McCann but did not get any response.
> 

That's very unfortunate. I can only assume he is very busy. Maybe it helps to ping him again.

> Does anybody know who the official maintainer of this package is?  I am willing
> to work on getting the patch included in the kernel if this approach is
> acceptable to the ConsoleKit team, but I don't want to bother with the kernel
> side of things unless ConsoleKit will incorporate the changes also.  I tried
> emailing Jon McCann but did not get any response.

This kind of hedging typically doesn't work were well in free software projects.
Don't let the lack of preapproval here keep you from getting the kernel side of things fixed.

BTW:

http://lkml.indiana.edu/hypermail/linux/kernel/0906.3/02726.html

Alan wanted cook up his own patch, but uh, he didn't respond in a week or so.

Lennart, was there any followup from Alan on this?

(In reply to comment #13)
> Lennart, was there any followup from Alan on this?
> 

Alan gave up maintainership of the console stuff a few weeks back. gregkh then took this up and one of the first things he did is merge Alan's patch into his tree. Two days ago this was then merged into mainline. So I guess that means the kernel infrstructure should now be able to cut down the threads to two (the ioctl is still blocking)

Lennart, any chance you can make a patch to use this new functionality?

This 3+ years old bug (or feature...) is still giving many people headaches: https://bugs.launchpad.net/ubuntu/+source/consolekit/+bug/148454

Any chance to see those useless processes go away in our lifetime?

If Alan gave up maintainership can someone please work on a fixed version of that ck autoconfigure patch, or something different, an hack, anything?

(In reply to comment #16) 
> Any chance to see those useless processes go away in our lifetime?

They aren't processes, they are threads. And they serve a purpose. 

The current behaviour might be suboptimal, but it is correct, as such it would be nice if this is fixed but we certainly have more burning issues to fix.

Also note that whatever the LP bug says about the 63 threads wasting oh so much resources, that is simply nonsense, the folks complaining have no clue what they are speaking of.

> If Alan gave up maintainership can someone please work on a fixed version of
> that ck autoconfigure patch, or something different, an hack, anything?

The stuff is in the kernel. It's just a matter of someone writing a patch for CK. Would be wonderful if you and your Ubuntu friends might contribute good code once in a while instead of just complaining for 3 years straight... ;-)

Anyway, I might come up with a patch eventually, but it is not a priority item on my todo list, sorry.

(In reply to comment #17)

Maybe we're complaining because we're users, not developers. I didn't *choose* to install consolekit in the first place... but I have to, or stay away from X/Gnome/nm and who knows what else.

Anyway, several console-kit/kernel patches *have* been proposed on LP by those "ubuntu friends" you were talking about; unfortunately, limiting consolekit seems to cause other problems. That's why most people are just waiting it to be fixed in upstream.

(In reply to comment #18)
> Anyway, several console-kit/kernel patches *have* been proposed on LP by those
> "ubuntu friends" you were talking about; unfortunately, limiting consolekit
> seems to cause other problems. That's why most people are just waiting it to be
> fixed in upstream.

if the patches are good, why aren't they submitted upstream?

(In reply to comment #17)
> (In reply to comment #16) 
> > Any chance to see those useless processes go away in our lifetime?
> 
> They aren't processes, they are threads. And they serve a purpose. 
> 
> The current behaviour might be suboptimal, but it is correct, as such it would
> be nice if this is fixed but we certainly have more burning issues to fix.
> 
> Also note that whatever the LP bug says about the 63 threads wasting oh so much
> resources, that is simply nonsense, the folks complaining have no clue what
> they are speaking of.
> 
> > If Alan gave up maintainership can someone please work on a fixed version of
> > that ck autoconfigure patch, or something different, an hack, anything?
> 
> The stuff is in the kernel. It's just a matter of someone writing a patch for
> CK. Would be wonderful if you and your Ubuntu friends might contribute good
> code once in a while instead of just complaining for 3 years straight... ;-)
> 
> Anyway, I might come up with a patch eventually, but it is not a priority item
> on my todo list, sorry.

I have to disagree with the claim that resources are not being eaten up by console-kit-daemon.  According to a top I did on a system that was recently having memory issues, the 64 console kit processes were using over 7.6GB of memory (122+MB each).  as someone else has pointed out, patches have been proposed for this.  Maybe they were not "good code,"  I haven't read it.  It would be interesting to find out why there need to be so many processes at outset rather that spawning a more conservative number of processes (or threads as the case may be; although each one seemed to have a distinct pid) to me even using 122MB total would be silly; what is this program doing that requires so much RAM?

Created attachment 36072 [details] [review]
Patch to make maximal number of watched VTs configurable

I once wrote a small patch for an older ubuntu version that makes it possible to limit the number of watched VTs. The patch is old, for consolekit-0.2.3 and I don't know if it still applies. The main idea was to make the maximal number of threads configurable to the user.

https://bugs.launchpad.net/consolekit/+bug/148454/comments/35

I would like to know if it makes sense to watch Virtual Terminals that have no gettys attached?

A few kernel releases back a new kernel API was added that makes the many threads unnecessary. Instead of coming up with ugly work-arounds for the current "problem", please fix things properly and just prepare a patch that ports CK to the new API entirely.

Created attachment 37970 [details] [review]
New patch that uses new VT_WAITEVENT ioctl

New patch that uses new VT_WAITEVENT ioctl.

I commited a slightly modified version of Kan-Ru's patch now to git. Thanks!

Thanks for fixing this Lennart!

(In reply to comment #25)
> Thanks for fixing this Lennart!

Say thanks to Kan-Ru, he did the patch!

I have now reverted the patch and reopened this bug. VT_WAITEVENT is inherently racy, since events that happen between the time we return from a VT_WAITEVENT and go back into VT_WAITEVENT are completely lost. That has the effect that when coming back from a suspend sometimes VT switches are not noticed and ACLs incorrectly set up.

Kay has now proposed a new kernel interface that helps us to fix this properly. This is currently in discussion on LKML.

The rhbz bug about the race you find here:

https://bugzilla.redhat.com/show_bug.cgi?id=643367

(In reply to comment #27)
> Kay has now proposed a new kernel interface that helps us to fix this properly.
> This is currently in discussion on LKML.

Where could I find the relevant discussions?  A quick search using "VT_WAITEVENT" or "ConsoleKit" didn't find anything.

See the thread starting with this patch of Kay's:

http://markmail.org/message/dhgzoht63uhfieqc

BTW, if somebody cares enough: this bug could be fixed properly now as Kay's patch got merged into .38.

There's now a file "/sys/class/tty/tty0/active" which you can read to get the current VT. And you can poll() for it and will get POLLPRI every time the VT changes.

That said, given that CK is being retired anyway sooner or later (merged into systemd) I do wonder if it is really worth working on this for now.

(In reply to comment #31)
> BTW, if somebody cares enough: this bug could be fixed properly now as Kay's
> patch got merged into .38.
> 
> There's now a file "/sys/class/tty/tty0/active" which you can read to get the
> current VT. And you can poll() for it and will get POLLPRI every time the VT
> changes.
> 
> That said, given that CK is being retired anyway sooner or later (merged into
> systemd) I do wonder if it is really worth working on this for now.


yes it does. there are other distros out there that doesn't use systemd at all and doesn't have the intention to use it.

(In reply to comment #32)
> yes it does. there are other distros out there that doesn't use systemd at all
> and doesn't have the intention to use it.

Are there? If so those folks probably have to become active... We will phase out CK one day, and those folks who find value in sticking with obsolete code will then have come forward and do the work.