When max_incomplete_connections is reached, new connections fail and we disconnect the oldest unauthenticated connection. The code contains a "FIXME" explaining the risk with a suggestion of a better solution. We should investigate if we can do better than this.

bus/connection.c:bus_connections_setup_connection():

>   /* And we might also disconnect ourselves here, but again it
>    * only takes effect on return to main loop.
>    */
>   if (connections->n_incomplete >
>       bus_context_get_max_incomplete_connections (connections->context))
>     {
>       _dbus_verbose ("Number of incomplete connections exceeds max, dropping oldest one\n");
> 
>       _dbus_assert (connections->incomplete != NULL);
>       /* Disconnect the oldest unauthenticated connection.  FIXME
>        * would it be more secure to drop a *random* connection?  This
>        * algorithm seems to mean that if someone can create new
>        * connections quickly enough, they can keep anyone else from
>        * completing authentication. But random may or may not really
>        * help with that, a more elaborate solution might be required.
>        */
>       dbus_connection_close (connections->incomplete->data);
>     }

Note: max_incomplete_connections is 64 by default (the system bus uses the default) and 10000 on the session bus.