This should mostly be a function of the libxtrans source code you build
libX11 with - unfortunately libxtrans is not a proper shared libary, but
a shared set of *.c files that each library includes, so you have to 
install different versions and then rebuild libraries like libX11 against
them.    I would wonder if the abstract socket namespace changes in the
recent release of xtrans 1.1 are involved here.

You are on to something.

I am using xtrans 1.1.

I defined XTRANSDEBUG as 5 and recompiled. xorg-server creates the the normal Unix Domain Socket (unix) but appears to fail when creating the Abstract Socket (local). However, libX11 attempts to connect to the Abstract Socket (local) and fails. When I force the disabling of Abstract Sockets by commenting out its definition in Xtranssock.c and recompile libX11, then libX11 connects to the Unix Domain Socket and works.

Likewise, when I switch to xtrans 1.0.4, everything works.

I have not yet looked into why xorg-server appears to be failing to create the Abstract Socket.

After reading this bug I upgraded xtrans from 1.0.4 (dist’s latest) to
git master and recompiled xcb, libX11 and xserver.

AFAICT, the server is listening on an abstract socket, but nothing is
using it.

(I’m presuming that the sockets in /proc/net/unix starting with @ are
the abstract ones:

,----[ grep -a @ /proc/net/unix ]
| da391b60: 00000002 00000000 00010000 0001 01 10197 @/tmp/.X11-unix/X0
| df236000: 00000002 00000000 00000000 0002 01   305 @/org/kernel/udev/udevd
`----

as those show up as ‘socket’ in lsof(8)’s output, rather than as a
path.)

It is likely relevant that the @/tmp/.X11-unix/X0 line above is followed
by 180 NULs octets in the /proc/net/unix file.  No other line has that.

So I suspect the kernel isn’t properly creating the socket.

I don’t see what is wrong in Xtranssock.c from a quick read thru.

Perhaps the problem you are seeing is also due to the NULs?

Eek.  When reading comment #3 please:

s/kernel/xserver/

:-(

Is this related to not including the NUL in the string length that
was reported in bug 7987?

Yes, I think it is related.

From Xtranssock.c:

    if (abstract) {
        sockname.sun_path[0] = '\0';
        namelen = sizeof(sockname);
    }

    if ((status = TRANS(SocketCreateListener) (ciptr,
        (struct sockaddr *) &sockname, namelen, flags)) < 0)

I'm not sude namelen is correctly computed.  Udevd.c does it like this:

        addrlen = offsetof(struct sockaddr_un, sun_path) + 1 + strlen(&saddr.sun_path[1]);

(where sun_path[0] is already set to '\0'.)

By using strlen they end up with a length which leaves out the trailing '\0's,
whereas Xtranssock’s sizeof() will be 108 + sizeof(sa_family_t) + any padding.

I bet fixing the computation of namelen will fix both bugs.

Created attachment 16178 [details] [review]
xtrans-1.1-abstract_namelen.patch

(In reply to comment #6)

Yes, that length calculation does look wrong and the change you suggest fixed the problem. With the change, abstract sockets are working.

I have attached a patch against xtrans 1.1 that implements your suggested change.

Pushed with commit 960902584a3ef125946beb5ebe331b54d697e9d9.