Bug 101294

When it's frozen, attach gdb to the process, run

 thread apply all bt

and attach the output here.

Created attachment 131700 [details]
Thread dump on Arch Linux mesa 17.1.0 PKGBUILD rebuilt with debug symbols

Created attachment 131721 [details]
Same freeze in QupZilla (main process) with similar backtraces

Looks like a deadlock involving struct pb_cache::mutex and struct amdgpu_winsys::bo_fence_lock. One thread locked the former and wants to lock the latter, another thread the other way around.

I can confirm the issue.

It doesn't always happen, but most of the time - it's fairly easy to reproduce. Usually happens when you have a lot mods (I think).

LIBGL_ALWAYS_SOFTWARE resolves that issue and lets it load, and you can play it just fine.

System the issue was tested on:
- Arch Linux 64bit
- Linux 4.11.6, AMDGPU driver
- Mesa 17.2.0-devel (git-58d337941e) / Mesa 17.1.2
- Radeon R9 285

I think it used to work previously, I'll run some regression tests in the coming days.

Bisected to
commit 2769dadb0fafdbafc98630fdf96924a3bb209ab7
Author: Marek Olšák <marek.olsak@amd.com>
Date:   Thu Apr 13 23:46:59 2017 +0200

    gallium/radeon: always flush asynchronously and wait after begin_new_cs
    
    This hides the overhead of everything in the driver after the CS flush and
    before returning from pipe_context::flush.
    Only microbenchmarks will benefit.
    
    +2% FPS for glxgears.
    
    Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
 

Reverting this commit on the recent git tree fixes the freezes.
Now since the bug doesn't always appear, I'm not 100% sure, but I didn't get a single freeze in 20 runs, while I got 13 out of 20 without reverting it.


Could you maybe look into this? I have no idea how to fix the issue without throwing away all changes from this commit.

The bisected commit only uncovers the existing deadlock scenario.

Summary of the issue.

amdgpu_bo_create
-> pb_cache_reclaim_buffer (lock pb_cache::mutex)
-> pb_cache_is_buffer_compat
-> amdgpu_bo_wait (lock bo_fence_lock) - DEADLOCK

pb_reference
-> pb_destroy
-> amdgpu_bo_destroy_or_cache
-> pb_cache_add_buffer (lock::pb_cache::mutex) - DEADLOCK

amdgpu_cs_flush (lock bo_fence_lock)
-> amdgpu_add_fence_dependency (loop-wait for submission_in_progress) - DEADLOCK


It looks the best way to prevent this deadlock is to unify pb_cache::mutex and bo_fence_lock under one lock, that is, one of them has to go.

Actually, I think it's 3 locks we have to unify:
- bo_fence_lock
- pb_cache::mutex
- pb_slabs::mutex

Hm, I think the proper fix is not to loop on submission_in_progress in amdgpu_cs_flush while bo_fence_lock is held. The looping is there because we need to prepare the request.dependencies list. We can remove that looping rather trivially by passing an amdgpu_fence list (with potential submissions in progress) to the CS thread as dependencies, and the CS thread can initialize request.dependencies accordingly.

Created attachment 132069 [details] [review]
tentative fix

Please test the attached patch. Thanks.

Thanks for explaining, I already feared that the issue was deeper.

I think I found a configuration for Minecraft that triggers the issue in ~100% of all tries.
I tested the patch, and it reliable fixes the freezes.

Can also confirm that this fixed the issue for me it seems.

It also patched nicely on 17.1.0, I should update my PKGBUILD I know...

Fixed by https://cgit.freedesktop.org/mesa/mesa/commit/?id=58af1f6bb074168669aaec2755c7f369a8b58d62

Greetings. I'm also getting a GPU crash / system lockup, when running Minecraft as well as other game engines. It doesn't happen during the splash screen, but probabilistically while the game is running. The problem was introduced at the same date as the making of this report, also around the time my distribution upgraded to Mesa 17.1.

I assume I'm experiencing the same bug described here, since I honestly don't see Mesa 17.1 introducing two completely unrelated crashes that would cause a renderer as simple as Minecraft to crash the OS. However I updated to Mesa 17.1.4, and surprisingly it still occurs.

Therefore it's possible that this was not fixed. Since it's not my bug and I don't wish to inconvenience folks, I won't set the status to reopened... however the developers might wish to take a look at this. Here is my own report:

https://bugs.freedesktop.org/show_bug.cgi?id=101672

(In reply to MirceaKitsune from comment #14)
> I assume I'm experiencing the same bug described here [...]

You're most definitely not. Many different causes can result in similar symptoms. If the symptoms you experience differ from those described in a bug report by even just a detail, it's better to assume you're experiencing a different bug.