| Summary: | [945GM] GPU hang randomly, garbage in batchbuffer. | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | xorg | Reporter: | Fryderyk Dziarmagowski <fdziarmagowski> | ||||||||||||||||||||||||||||||||||||
| Component: | Driver/intel | Assignee: | Chris Wilson <chris> | ||||||||||||||||||||||||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | Xorg Project Team <xorg-team> | ||||||||||||||||||||||||||||||||||||
| Severity: | critical | ||||||||||||||||||||||||||||||||||||||
| Priority: | medium | CC: | axet, cfeck, fdziarmagowski | ||||||||||||||||||||||||||||||||||||
| Version: | 7.4 (2008.09) | Keywords: | NEEDINFO | ||||||||||||||||||||||||||||||||||||
| Hardware: | x86 (IA32) | ||||||||||||||||||||||||||||||||||||||
| OS: | Linux (All) | ||||||||||||||||||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||||||||||||||||||
| i915 platform: | i915 features: | ||||||||||||||||||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||
|
Description
Fryderyk Dziarmagowski
2009-09-15 11:55:46 UTC
Created attachment 29572 [details]
gpu dump (bzip2)
Created attachment 29573 [details]
xorg log
Created attachment 29574 [details]
kernel log
Created attachment 29575 [details]
xorg.conf
You've already using the latest mesa with this patch, right? http://cgit.freedesktop.org/mesa/mesa/commit/?id=acfea5c705f383692e661d37c5cd7da2f3db559b Can you try if this kernel patch provide more info: http://lists.freedesktop.org/archives/intel-gfx/2009-September/004243.html Above Mesa patch is dedicated for i965 and I got no such hardware ;-) (it's i915, but... there are no apps using 3d when it hangs) I've applied mentioned kernel patch. I will post some results after something occurs. By the way, it smells like #23699 Created attachment 29603 [details]
gpu dump 2
I've just triggered a hang running stellarium after back from s2ram (dump attached). Created attachment 29604 [details]
gpu dump before suspend
Created attachment 29605 [details]
gpu dump after suspend
Created attachment 29606 [details]
stellarium showing big triangles after s2ram (compressed with xz)
This screencast shows, that after back from s2ram 3d rendering is broken too.
Unfortunaltely kernel with patch from #5 stays calm as before 00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03) that's a 965, so please test with the mesa patch. I'm really sorry, but I've attached wrong lspci :-[ 00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03) 00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03) 00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03) 00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01) 00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01) 00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2 (rev 01) 00:1c.3 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 4 (rev 01) 00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01) 00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01) 00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01) 00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01) 00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e1) 00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 01) 00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01) 00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01) 02:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 02) 02:01.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller 02:01.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 19) 02:01.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 0a) 02:01.3 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 05) 02:01.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev ff) 0c:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection (rev 02) Just tested the Intel 2009Q3 release, the problem still exists. This time I've made some test using "skyrocket" 3d application (just a screensaver, http://rss-glx.sourceforge.net/). It hangs gpu (i.e after starting it or on resizing the application's window) just after few seconds (with stellarium was a bit harder). upgrading kernel to 2.6.32rc5 with Eric's "for-linus" branch solves the problems described above :-) Unfortunetely I was a bit too fast closing this bug. This time it happened during operations in gnome-terminal, firstly a vertical bar with artifacts appeared on the left side of my laptop (5mm x100mm, I had a impression a text line in midnight commander was not rendered correctly). A few seconds after whole screen stopped to response on mouse movements/clicks. (A good thing about 2.6.32 is, I was able to switch to console with ctrl+alt+Fx. Running to my second machine and remote conection are no longer needed.) This time kernel log show something: Oct 18 18:55:18 aragorn kernel: [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung Oct 18 18:55:18 aragorn kernel: render error detected, EIR: 0x00000000 Oct 18 18:55:18 aragorn kernel: i915: Waking up sleeping processes Oct 18 18:55:18 aragorn ke1.7.0.901rnel: reboot required Oct 18 18:55:18 aragorn kernel: [drm:i915_wait_request] *ERROR* i915_wait_request returns -5 (awaiting 1386051 at 1386039) Oct 18 18:55:18 aragorn kernel: [drm:i915_gem_execbuffer] *ERROR* Execbuf while wedged Last message repeats until a few times per second until reboot. I'm attaching new gpu dump, just after the hang. Once more some details: linux 2.6.32rc5 with Eric's "for Linus" branch Mesa 7_6 git branch intel 2d driver 2.9 git branch xorg xserver 1.7.0.901 Created attachment 30536 [details]
bzipped gpu dump
Small upgrade: Mesa 7.6.1 xorg 1.7.3.901 xorg intel driver 2.9.99.902 kernel 2.6.32.2 Unfortunetely GPU lockup is still here: Dec 23 15:56:13 aragorn kernel: [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung Dec 23 15:56:13 aragorn kernel: render error detected, EIR: 0x00000000 Dec 23 15:56:13 aragorn kernel: i915: Waking up sleeping processes Dec 23 15:56:13 aragorn kernel: [drm:i915_wait_request] *ERROR* i915_wait_request returns -5 (awaiting 330264 at 330250) Dec 23 15:56:13 aragorn kernel: [drm:i915_gem_execbuffer] *ERROR* Execbuf while wedged Attaching new gpu dump. Created attachment 32263 [details]
on more GPU dump (xz compressed)
Upgraded Mesa to 7.7, the problem is still present. I'm able now to trigger same lockup even on "freshly" started machine: it happens during using webbrowser (midori, webkit based). It is still random, but browsing phoronix benchmarks almost always locks up my GPU. I'm getting same drm messages as described above. One more thing from Xorg.0.log (trigger phoronix.com ;-) (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error (EE) intel(0): Failed to submit batch buffer, expect rendering corruption or even a frozen display: Input/output error. Adding Option "DebugWait" "true" to xorg.conf solves the problem. 2D performance suffers a bit, but at least it is stable as in EXA times. ping Created attachment 33504 [details] [review] Record batch buffer at time of error The curse of the empty gpu dump. Can you try the attached patch and upload the i915_error_state following a hang? The fact that DebugWait fixes the issue for you suggests a form of corruption I've been hunting for in the wild. Even more cursed is the fact, that after the patch was applied, I was only once able to lock the gpu. Since debugfs is mounted permanently, no matter how hard I try I'm no longer able to reproduce it (and that was really easy before) The first and only time until now produced something like that: (debugfs mounted after lock up) Time: 1266959695 s 294930 us PCI ID: 0x27a2 EIR: 0x00000010 PGTBL_ER: 0x00000003 INSTPM: 0x00000000 IPEIR: 0x00000000 IPEHR: 0x00000000 INSTDONE: 0x7fffffc0 ACTHD: 0x00000000 seqno: 0x00035937 --- ringbuffer = 0x007bf000 00000000 : 00000000 00000004 : 00000000 00000008 : 00000000 ... (cut: only 00000000 here!) 0001fffc : 00000000 Seeing only zeros in second column I don't really believe it is something valuable. (In reply to comment #27) > Seeing only zeros in second column I don't really believe it is something > valuable. It is. But I guess it is reporting an earlier bug, that should be fixed with commit fd2e8ea597222b8f38ae8948776a61ea7958232e Author: Chris Wilson <chris@chris-wilson.co.uk> Date: Tue Feb 9 14:14:36 2010 +0000 drm/i915: Increase fb alignment to 64k An untiled framebuffer must be aligned to 64k. This is normally handled by intel_pin_and_fence_fb_obj(), but the intelfb_create() likes to be different and do the pinning itself. However, it aligns the buffer object incorrectly for pre-i965 chipsets causing a PGTBL_ERR when it is installed onto the output. (Bug #22936) Can you either apply that patch or use the .33-rc8 + record buffers? sure, I need some more time to do it 8-) ok, upgraded to 2.6.33 with patch from #26 Looks like error state is triggered directly after resuming from s2ram. Just before s2ram: cat /sys/kernel/debu/dri/0/i915_error_state no error state collected and just after: Time: 1267121097 s 317013 us PCI ID: 0x27a2 EIR: 0x00000010 PGTBL_ER: 0x00000003 INSTPM: 0x00000000 IPEIR: 0x00000000 IPEHR: 0x00000000 INSTDONE: 0x7fffffc0 ACTHD: 0x00000000 seqno: 0x00000cc6 --- ringbuffer = 0x00bc8000 00000000 : 00000000 00000004 : 00000000 00000008 : 00000000 ... dmesg shows something what I didn't saw until now: ... i915 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 i915 0000:00:02.0: setting latency timer to 64 render error detected, EIR: 0x00000010 page table error PGTBL_ER: 0x00000003 [drm:i915_handle_error] *ERROR* EIR stuck: 0x00000010, masking render error detected, EIR: 0x00000010 page table error PGTBL_ER: 0x00000003 ... Should I provide something more? Created attachment 33566 [details]
recorded error state
Created attachment 33567 [details]
kernel log
Created attachment 33764 [details] [review] Rebind fbo if unaligned. Hmm, the bo is unaligned following a resume... Please try this patch to rebind the framebuffer with the appropriate alignment, if required. 2.6.33 build fails with the patch from #33: drivers/gpu/drm/i915/intel_display.c: In function 'intel_pin_and_fence_fb_obj': drivers/gpu/drm/i915/intel_display.c:1257: error: implicit declaration of function 'i915_gem_object_fence_offset_ok' make[4]: *** [drivers/gpu/drm/i915/intel_display.o] Error 1 How to fix it? i915_gem_object_fence_offset_ok seems to be defined in drivers/gpu/drm/i915/i915_gem_tiling.c I've upgraded my setup a bit: xserver 1.8 mesa 7.8.0 linux 2.6.33.2 (unfortunately still can't apply rebind_fbo patch) and changed suspending method from plain echo mem > /sys/power/state to high sophisticated pm-utils (it does some magic to suspending, i915_error_state stays clean after resume). With this shiny, new setup it is still very easy to trigger a GPU hang, but for the first time I catched i915_error_state with some content. Created attachment 34640 [details]
i915_error_state running skyrocket
Created attachment 34968 [details]
one more error_state
Catched one more time a error_state. It looks quite different to last one.
ping The rebind bo is now upstream (at last!). Fryderyk, if you can reproduce the skyrocket crash on current trees, please upload a new i915_error_state. I can think of a similar bug that was a result of memory corruption through a batch buffer overrun in mesa that has since been fixed -- so I am optimistic in that this bug is now unreproducible! commit ac0c6b5ad3b3b513e1057806d4b7627fcc0ecc27 Author: Chris Wilson <chris@chris-wilson.co.uk> Date: Thu May 27 13:18:18 2010 +0100 drm/i915: Rebind bo if currently bound with incorrect alignment. Whilst pinning the buffer, check that that its current alignment matches the requested alignment. If it does not, rebind. This should clear up any final render errors whilst resuming, for reference: Bug 27070 - [i915] Page table errors with empty ringbuffer https://bugs.freedesktop.org/show_bug.cgi?id=27070 Bug 15502 - render error detected, EIR: 0x00000010 https://bugzilla.kernel.org/show_bug.cgi?id=15502 Bug 13844 - i915 error: "render error detected" https://bugzilla.kernel.org/show_bug.cgi?id=13844 Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: stable@kernel.org Signed-off-by: Eric Anholt <eric@anholt.net> Glad to hear it! I hope, I will find some time to try something more actual then 2.6.23.x... There is a working theory that 945GM hangs when trying to perform lots of XY_SRC_COPY_BLT. A reproducible test case on a t60 (Core2/i945) is to run x11perf -copypixwin500 which hangs after one pass. The i915_error_state in these cases tend to be fairly random. Can you check to see if your machine is also susceptible to x11perf -copypixwin500? The only issue I have with this x11perf test is a massive slow down when mouse pointer stays idle... after multiple times running it, I was no able to hang the GPU Funny thing: mouse untouched 280 reps @ 0.8198 msec ( 1220.0/sec): Copy 500x500 from pixmap to window and now some with some moves: 8000 reps @ 0.9451 msec ( 1060.0/sec): Copy 500x500 from pixmap to window > --- Comment #43 from Fryderyk Dziarmagowski <freetz@gmx.net> 2010-07-21 12:09:29 PDT ---
> The only issue I have with this x11perf test is a massive slow down when mouse
> pointer stays idle... after multiple times running it, I was no able to hang
> the GPU
So not suffering from the missing bit, but sounds like missing interrupts.
Are you using a compositing WM? Does it make a difference to switch to a
non-compositing WM (or vice versa)? I suspect/hope that the patches Jesse
pushed [2.6.35-rc4] to fix page-flipping on i945 are the answer here.
I've upgraded my kernel to 2.6.34.1 (Xorg to 1.8.2, driver to 2.12.0, Mesa 7.8.2) and I got new surprising results regarding this bug: frozen screen was replaced with a nice X crash! (This is done with GPU killer - skyrocket) [ 81834.638] (EE) intel(0): Detected a hung GPU, disabling acceleration. [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81834.651] (WW) intel(0): i830_uxa_prepare_access: gtt bo map failed: Input/output error [ 81837.926] Backtrace: [ 81837.926] 0: /usr/bin/X (xorg_backtrace+0x3b) [0x809b1c7] [ 81837.926] 1: /usr/bin/X (0x8047000+0x5410e) [0x809b10e] [ 81837.926] 2: (vdso) (__kernel_rt_sigreturn+0x0) [0xffffe40c] [ 81837.926] 3: /usr/lib/xorg/modules/extensions/libdri2.so (0xb77c3000+0x3511) [0xb77c6511] [ 81837.926] 4: /usr/bin/X (0x8047000+0x266ee) [0x806d6ee] [ 81837.926] 5: /usr/bin/X (0x8047000+0x1f7e5) [0x80667e5] [ 81837.926] 6: /lib/libc.so.6 (__libc_start_main+0xe6) [0x47fccb62] [ 81837.926] 7: /usr/bin/X (0x8047000+0x1f401) [0x8066401] [ 81837.926] Segmentation fault at address (nil) [ 81837.926] Fatal server error: [ 81837.926] Caught signal 11 (Segmentation fault). Server aborting What I'm observing now are random screen (text) corruptions in firefox and hard locks when running stellarium (this one kills my laptop, only hard reset helps...) (In reply to comment #44) > > --- Comment #43 from Fryderyk Dziarmagowski <freetz@gmx.net> 2010-07-21 12:09:29 PDT --- > > The only issue I have with this x11perf test is a massive slow down when mouse > > pointer stays idle... after multiple times running it, I was no able to hang > > the GPU > > So not suffering from the missing bit, but sounds like missing interrupts. > Are you using a compositing WM? Does it make a difference to switch to a > non-compositing WM (or vice versa)? I suspect/hope that the patches Jesse > pushed [2.6.35-rc4] to fix page-flipping on i945 are the answer here. No, I don't use compositing at all (due to mplayer tearing) switching on xcompmgr helps: slowdown goes away (still without hang) > --- Comment #45 from Fryderyk Dziarmagowski <freetz@gmx.net> 2010-07-21 12:25:21 PDT --- > I've upgraded my kernel to 2.6.34.1 (Xorg to 1.8.2, driver to 2.12.0, Mesa > 7.8.2) and I got new surprising results regarding this bug: > > frozen screen was replaced with a nice X crash! > > (This is done with GPU killer - skyrocket) > > [ 81834.638] (EE) intel(0): Detected a hung GPU, disabling acceleration. [snip] > Fatal server error: > [ 81837.926] Caught signal 11 (Segmentation fault). Server aborting That crash in particular has been fixed, but we need to find the cause of the GPU hang. > What I'm observing now are random screen (text) corruptions in firefox and hard > locks when running stellarium (this one kills my laptop, only hard reset > helps...) Can you upload some i915_error_state for these hangs? Thanks. (In reply to comment #46) > (In reply to comment #44) > > > --- Comment #43 from Fryderyk Dziarmagowski <freetz@gmx.net> 2010-07-21 12:09:29 PDT --- > > > The only issue I have with this x11perf test is a massive slow down when mouse > > > pointer stays idle... after multiple times running it, I was no able to hang > > > the GPU > > > > So not suffering from the missing bit, but sounds like missing interrupts. > > Are you using a compositing WM? Does it make a difference to switch to a > > non-compositing WM (or vice versa)? I suspect/hope that the patches Jesse > > pushed [2.6.35-rc4] to fix page-flipping on i945 are the answer here. > > No, I don't use compositing at all (due to mplayer tearing) > switching on xcompmgr helps: slowdown goes away (still without hang) ok, forget what I wrote. It looks even worse now: 240 reps @ 0.8121 msec ( 1230.0/sec): Copy 500x500 from pixmap to window (In reply to comment #47) > > --- Comment #45 from Fryderyk Dziarmagowski <freetz@gmx.net> 2010-07-21 12:25:21 PDT --- > > I've upgraded my kernel to 2.6.34.1 (Xorg to 1.8.2, driver to 2.12.0, Mesa > > 7.8.2) and I got new surprising results regarding this bug: > > > > frozen screen was replaced with a nice X crash! > > > > (This is done with GPU killer - skyrocket) > > > > [ 81834.638] (EE) intel(0): Detected a hung GPU, disabling acceleration. > [snip] > > Fatal server error: > > [ 81837.926] Caught signal 11 (Segmentation fault). Server aborting > > That crash in particular has been fixed, but we need to find the cause of > the GPU hang. Could you point me to the fix? > > What I'm observing now are random screen (text) corruptions in firefox and hard > > locks when running stellarium (this one kills my laptop, only hard reset > > helps...) > > Can you upload some i915_error_state for these hangs? Thanks. give me some minutes... Created attachment 37277 [details]
fresh error state
applying Dave's "enable low power render writes on GEN3 hardware" miracle patch does not seems to help here. Hmm, another instance of: 0x0dc07878: 0x7d000003: 3DSTATE_MAP_STATE 0x0dc0787c: 0x00000001: mask 0x0dc07880: 0x00000000: map 0 MS2 0x0dc07884: 0x00000000: map 0 MS3 [width=1, height=1, tiling=none] 0x0dc07888: 0x00000000: map 0 MS4 [pitch=4] 0x0dc0788c: 0x00000000: MI_NOOP 0x0dc07890: 0x00000000: MI_NOOP 0x0dc07894: 0x00000000: MI_NOOP 0x0dc07898: 0x00000000: MI_NOOP 0x0dc0789c: 0x00000000: MI_NOOP 0x0dc078a0: 0x00000000: MI_NOOP 0x0dc078a4: 0x00000000: MI_NOOP 0x0dc078a8: 0x00000000: MI_NOOP 0x0dc078ac: 0x00000000: MI_NOOP 0x0dc078b0: 0x00000000: MI_NOOP 0x0dc078b4: 0x00000000: MI_NOOP 0x0dc078b8: 0x00000000: MI_NOOP 0x0dc078bc: 0x00000000: MI_NOOP 0x0dc078c0: 0x00000000: MI_NOOP 0x0dc078c4: 0x00000000: MI_NOOP 0x0dc078c8: 0x00000000: MI_NOOP 0x0dc078cc: 0x00000000: MI_NOOP 0x0dc078d0: 0x00000000: MI_NOOP 0x0dc078d4: 0x00000000: MI_NOOP 0x0dc078f8: 0x00000000: MI_NOOP 0x0dc078fc: 0x00000000: MI_NOOP 0x0dc07900: 0x00000000: MI_NOOP 0x0dc07904: 0x00000000: MI_NOOP 0x0dc07908: 0x00000000: MI_NOOP 0x0dc0790c: 0x00000000: MI_NOOP 0x0dc07910: 0x00000000: MI_NOOP 0x0dc07914: 0x00000000: MI_NOOP 0x0dc07918: 0x00000000: MI_NOOP 0x0dc0791c: 0x00000000: MI_NOOP 0x0dc07920: 0x00000000: MI_NOOP 0x0dc07924: 0x00000000: MI_NOOP 0x0dc07928: 0x00000000: MI_NOOP 0x0dc0792c: 0x00000000: MI_NOOP 0x0dc07930: 0x00000000: MI_NOOP 0x0dc07934: 0x00000000: MI_NOOP 0x0dc07938: 0x00000000: MI_NOOP 0x0dc0793c: 0x00000000: MI_NOOP 0x0dc07940: 0x06060000: MI UNKNOWN 0x0dc07944: 0x7f800006: 3DPRIMITIVE sequential indirect TRILIST, 6 starting from 0 0x0dc07948: 0x00000000: start This bug is no longer present with latest kernel releases... (tested .33.7 and .35.1). Closing... :) |
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.