Bug 99028

Created attachment 128381 [details]
dmesg-guest.log

Attach the full guest dmesg log.

(In reply to Terrence Xu from comment #1)
> Created attachment 128381 [details]
> dmesg-guest.log
> 
> Attach the full guest dmesg log.

Terence can you also attach gpu crash dump located at /sys/class/drm/card0/error ?

Created attachment 128724 [details]
error.log

error.log from /sys/class/drm/card0/error.

(In reply to yann from comment #2)
> (In reply to Terrence Xu from comment #1)
> > Created attachment 128381 [details]
> > dmesg-guest.log
> > 
> > Attach the full guest dmesg log.
> 
> Terence can you also attach gpu crash dump located at
> /sys/class/drm/card0/error ?

Hi yann, sorry I was uploaded the error log failed and I didn't catch it previously, I just upload the error log successfully.

Yann - please take a look on gpu crash dump.

Terence, please use latest drm-tip ()

it looks like we have a loop of MI_SEMAPHORE_MBOX command (wait upon the semaphore value remained ?) with blitter ring (any relationship with bug 54226?)

You may also try to add drm.debug=0xe in your boot command line to collect more data.

Still a kvm bug.

(In reply to yann from comment #6)
> Terence, please use latest drm-tip ()
> 
> it looks like we have a loop of MI_SEMAPHORE_MBOX command (wait upon the
> semaphore value remained ?) with blitter ring (any relationship with bug
> 54226?)
> 
> You may also try to add drm.debug=0xe in your boot command line to collect
> more data.

Hi Yann, use the newest drm-tip, it still blocked by the Bug99025 :(, I just updated the newest status on it.

This issue actually happened together with bug#99025. It is also a regression. As no GVT code is involved (updated in comment: https://bugs.freedesktop.org/show_bug.cgi?id=99025#c10), suggest i915 team to help to investigate.

Regression = we need the bisect.

I use upstream 4.9 kernel as host and guest kernel, I also see GPU hang in guest.
git bisect tell me the first bad commit is:
commit 49ef5294cda256aa5496ba56bbf859d3c7a17e07
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Aug 18 17:17:00 2016 +0100

    drm/i915: Move fence tracking from object to vma

    In order to handle tiled partial GTT mmappings, we need to associate the
    fence with an individual vma.

    v2: A couple of silly drops replaced spotted by Joonas

    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>

It appears to be using stolen memory in the guest, which is not mediated by gvt-d nor does gvt-d fixup the reported size of stolen memory to be zero.

Indeed guest are using stolen memory. After disabling stolen memory in i915, gpu hang disappear.
While only one guest is using GPU in gvt-d, why guest couldn't use it like host ?

*** Bug 99025 has been marked as a duplicate of this bug. ***

Fixed this issue in Qemu with: 
c2b2e15 vfio/pci-quirks.c: Disable stolen memory for igd VFIO

As the qemu patch in commit#15 breaks windows igd driver load and has been reverted. So this bug will exist again.

Created attachment 130723 [details]
guest igd pci config

From guest igd pci config.txt, we could know stolen memory base is 0xda000000, size is 32M, it's  range is 0xda000000 ~ 0xdbffffff

Created attachment 130726 [details]
host_dmesg_with_stolen_memory_enabled

This message is gotten from host when I boot a guest drm-intel-nightly kernel with stolen memory enabled, I saw gpu hang in guest and reproduced this issue. From this message we could see there are tons of IOMMU read/write fault at 0xdaxxxxx where are stolen memory. So igd are accessing stolen memory.

Created attachment 130727 [details]
host_dmesg_with_stolen_memory_disabled

This message is gotten from host when I boot the same guest drm-intel-nightly kernel with stolen memory disabled.  I didn't see a gpu hang in guest and this bug disappear. From This message we didn't see any IOMMU interrupt fault.

Created attachment 130728 [details]
host_dmesg_boot_guest_4.8_kernel

This host message is gotten when I boot a default ubuntu 16.10 kernel based on v4.8 with stolen memory enabled. From this dmesg, we could see there are several IOMMU interrupt faults at stolen memory. But there are tons of IOMMU interrupt faults at 0xff20c000 where I don't know who owns. while I didn't see gpu hang in this guest during the boot process.  
The access to 0xff20c0000 should be a bug and it has been fixed in drm-intel-nightly.

Anyway once guest enable stolen memory, we could see iommu fault which means igd fail in read/write the target address and couldn't successfully finish the command. So it is better that we could disable stolen memory in kvm guest to prevent this iommu error.

Reference to XiongZhang's patch: https://patchwork.freedesktop.org/series/21818/

Created attachment 130820 [details]
/sys/class/drm/card0/error when gpu hang

the error message:
[    9.853024] [drm] GPU HANG: ecode 9:1:0xeeffefa9, in Xorg [871], reason: No progress on blitter ring, action: reset
[    9.853027] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[    9.853028] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[    9.853030] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[    9.853031] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[    9.853032] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[    9.853223] [drm:i915_reset_and_wakeup [i915]] resetting chip

[   16.896147] drm/i915: Resetting chip after gpu hang
[   16.898713] [drm:i915_gem_reset [i915]] context Xorg[871]/0 marked guilty (score 10) banned? no
[   16.898756] [drm:i915_gem_reset [i915]] resetting blitter ring to restart from tail of request 0x1
[   16.898764] BUG: unable to handle kernel NULL pointer dereference at 0000000000000070
[   16.901051] IP: reset_common_ring+0x8b/0x120 [i915]
[   16.902392] PGD 0

[   16.903508] Oops: 0000 [#1] SMP
[   16.904262] Modules linked in: crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd joydev input_leds serio_raw i2c_piix4 qemu_fw_cfg mac_hid parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid i915(OE) video i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm e1000 floppy psmouse pata_acpi
[   16.907024] CPU: 1 PID: 257 Comm: kworker/1:3 Tainted: G           OE   4.11.0-rc3nightly+ #2
[   16.907710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[   16.908587] Workqueue: events_long i915_hangcheck_elapsed [i915]
[   16.909124] task: ffff89e25e6dcb00 task.stack: ffffa6d2807e4000
[   16.909582] RIP: 0010:reset_common_ring+0x8b/0x120 [i915]
[   16.910081] RSP: 0018:ffffa6d2807e7b60 EFLAGS: 00010206
[   16.910474] RAX: 0000000000003f00 RBX: ffff89e2bbd1f680 RCX: ffff89e2777539f4
[   16.911098] RDX: 0000000000003f40 RSI: ffff89e25d92e000 RDI: ffff89e2777539c0
[   16.911656] RBP: ffffa6d2807e7b78 R08: 0000000000000001 R09: 0000000000000001
[   16.912202] R10: 00000000ffffffff R11: 0000000000000448 R12: ffff89e25df56000
[   16.912749] R13: 0000000000000000 R14: ffff89e2bbd1f680 R15: ffff89e25dba4d38
[   16.913250] FS:  0000000000000000(0000) GS:ffff89e2bfd00000(0000) knlGS:0000000000000000
[   16.913864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   16.914322] CR2: 0000000000000070 CR3: 00000000434a0000 CR4: 00000000003406e0
[   16.914922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   16.915520] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   16.916097] Call Trace:
[   16.916324]  i915_gem_reset+0xbe/0x380 [i915]
[   16.916679]  ? intel_uncore_forcewake_put+0x48/0x60 [i915]
[   16.917111]  ? bit_wait_io+0x60/0x60
[   16.917374]  i915_reset+0xde/0x170 [i915]
[   16.917754]  i915_reset_and_wakeup+0x17c/0x190 [i915]
[   16.918164]  i915_handle_error+0x1db/0x220 [i915]
[   16.918536]  ? scnprintf+0x4d/0x90
[   16.918790]  hangcheck_declare_hang+0xdb/0x100 [i915]
[   16.919211]  i915_hangcheck_elapsed+0x2a9/0x2d0 [i915]
[   16.919613]  process_one_work+0x1fc/0x4b0
[   16.919937]  worker_thread+0x4b/0x500
[   16.920246]  kthread+0x101/0x140
[   16.920478]  ? process_one_work+0x4b0/0x4b0
[   16.920775]  ? kthread_create_on_node+0x60/0x60
[   16.921148]  ret_from_fork+0x2c/0x40
[   16.921450] Code: c8 01 00 00 89 50 14 48 8b 83 80 00 00 00 8b 93 c8 01 00 00 89 50 28 48 8b bb 80 00 00 00 e8 6d 2c 00 00 4d 8b ac 24 60 02 00 00 <49> 8b 45 70 48 39 43 70 74 5d 4d 85 ed 74 20 48 c7 c0 e0 70 61
[   16.922917] RIP: reset_common_ring+0x8b/0x120 [i915] RSP: ffffa6d2807e7b60
[   16.923488] CR2: 0000000000000070
[   16.923762] ---[ end trace 2a911e958217bc24 ]---
error

I did the git bisect again, and the target commit is:
commit c58b735fc762e891481e92af7124b85cb0a51fce
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Thu Aug 18 17:16:57 2016 +0100

    drm/i915: Allocate rings from stolen

    If we have stolen available, make use of it for ringbuffer allocation.
    Previously this was restricted to !llc platforms, as writing to stolen
    requires a GGTT mapping - but now that we have partial mappable support,
    the mappable aperture isn't quite so precious so we can use it more
    freely and ringbuffers are a good user for the otherwise wasted stolen.

    Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
    Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>


After reverting this patch from drm-intel-nightly, I didn't see gpu hang during boot process. So we could confirm this issue is caused by i915 access stolen memory when i915 is as kvm guest where kvm doesn't supply ept and guest domain iommu mapping for stolen memory.

Adding tag into "Whiteboard" field - ReadyForDev
The bug still active
*Status is correct
*Platform is included
*Feature is included
*Priority and Severity correctly set
*Logs included

The i915 driver behavior is correct, if stolen memory is reported, it'll be used. If it's reported as zero, it shall not be used. It was agreed that QEMU/KVM will be changed to correctly report stolen memory as zero and bugs will be fixed outside of i915 to accommodate that. A step two will be to add support to QEMU/KVM for stolen memory passthrough and after that report the stolen memory size as non-zero. Both changes do not require any modifications to i915.