Bug 47280

Summary: vmware driver causes segfault on startup
Product: xorg Reporter: Wil <nullptr>
Component: Driver/VMWareAssignee: Jakob Bornecrantz <jakob>
Status: RESOLVED FIXED QA Contact: Xorg Project Team <xorg-team>
Severity: major    
Priority: medium CC: adam.twardowski, aganders3, andyrtr, jakob, rob.j.david, thellstrom
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Bug Depends on:    
Bug Blocks: 47255, 51139    
Attachments:
Description Flags
Core image of crashed server.
none
X log of startup crash.
none
Core dump of crashed server, vmware driver
none
Core dump of crashed server, cirrus driver
none
Log of crash using cirrus driver
none
Fix
none
Backtrace of vmware crash
none
Fix none

Description Wil 2012-03-13 07:04:25 UTC
Created attachment 58372 [details]
Core image of crashed server.

The vmware driver included in the 1.12.0 X distribution causes a segfault on startup.  I originally posted some details at https://bugs.archlinux.org/task/28882, but the gist of it is that vmware_drv.so loads libvgahw.so as a dependency, and in vgaHWSaveColormap(), function pointers are dereferenced off of the passed-in vgaHWPtr that are NULL.  The particular one that causes the crash is hwp->writeDacMask at hw/xfree86/vgahw/vgaHW.c:1075.

Here's a stacktrace collected from gdb:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) i thr
Id Target Id Frame
* 1 Thread 0x7ffff7fc9880 (LWP 1295) "Xorg" 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff3bfdea1 in vgaHWSaveColormap () from /usr/lib/xorg/modules/libvgahw.so
#2 0x00007ffff3bff7ad in vgaHWSave () from /usr/lib/xorg/modules/libvgahw.so
#3 0x00007ffff4415043 in ?? () from /usr/lib/xorg/modules/drivers/vmware_drv.so
#4 0x00000000004735cc in InitOutput ()
#5 0x0000000000422c0d in ?? ()
#6 0x00007ffff61df38d in __libc_start_main () from /lib/libc.so.6
#7 0x00000000004230cd in _start ()
(gdb) 

I've also attached a core dump from the current ArchLinux x86_64 distribution.
Comment 1 Wil 2012-03-13 07:05:16 UTC
Created attachment 58373 [details]
X log of startup crash.

And, here's an X log.
Comment 2 Rob David 2012-03-13 10:44:37 UTC
Created attachment 58385 [details]
Core dump of crashed server, vmware driver
Comment 3 Rob David 2012-03-13 10:46:05 UTC
Created attachment 58386 [details]
Core dump of crashed server, cirrus driver
Comment 4 Rob David 2012-03-13 10:58:12 UTC
I've also seen this problem running under KVM. I see a pretty much identical log and debug trace to the one Wil reported when I run with the vmware driver. 

#0  0xb778a416 in __kernel_vsyscall ()
#1  0xb742407f in raise () from /lib/libc.so.6
#2  0xb7425a05 in abort () from /lib/libc.so.6
#3  0x081c9459 in OsAbort ()
#4  0x080b9b1c in ddxGiveUp ()
#5  0x080b9bc3 in AbortDDX ()
#6  0x081cee11 in ?? ()
#7  0x081cef35 in FatalError ()
#8  0x081c6f64 in ?? ()
#9  <signal handler called>
#10 0x00000000 in ?? ()
#11 0xb7212ed3 in vgaHWSaveColormap () from /usr/lib/xorg/modules/libvgahw.so
#12 0xb72149e4 in vgaHWSave () from /usr/lib/xorg/modules/libvgahw.so
#13 0xb721c772 in ?? () from /usr/lib/xorg/modules/drivers/vmware_drv.so
#14 0xb7222703 in ?? () from /usr/lib/xorg/modules/drivers/vmware_drv.so
#15 0x080bb6db in InitOutput ()
#16 0x08064400 in ?? ()
#17 0xb740f483 in __libc_start_main () from /lib/libc.so.6
#18 0x080648e9 in _start ()

I also tried changing my virtual machine settings to emulate cirrus graphics, and tried running X with the cirrus driver. I still got a segfault, but this time with rather less informative trace.

#0  0xb7750416 in __kernel_vsyscall ()
#1  0xb73ea07f in raise () from /lib/libc.so.6
#2  0xb73eba05 in abort () from /lib/libc.so.6
#3  0x081c9459 in OsAbort ()
#4  0x080b9b1c in ddxGiveUp ()
#5  0x080b9bc3 in AbortDDX ()
#6  0x081cee11 in ?? ()
#7  0x081cef35 in FatalError ()
#8  0x081c6f64 in ?? ()
#9  <signal handler called>
#10 0x00000000 in ?? ()

I've attached core file from both crashes.
Comment 5 Rob David 2012-03-13 10:59:33 UTC
Created attachment 58387 [details]
Log of crash using cirrus driver
Comment 6 Rob David 2012-03-13 11:03:45 UTC
I've also attached a log of the X server crash with the cirrus driver.

I should also point out this is a 32 bit VM.

Linux (none) 3.2.9-1-ARCH #1 SMP PREEMPT Thu Mar 1 09:10:44 UTC 2012 i686 QEMU Virtual CPU version 0.12.5 GenuineIntel GNU/Linux
Comment 7 Jakob Bornecrantz 2012-03-15 07:48:11 UTC
Which version of the vmware driver was this? Can I get the git id of it?
Hmm it looks like 12.0.0 from the log is this correct?

Cheers, Jakob.
Comment 8 Wil 2012-03-15 07:56:31 UTC
12.0.0, that's correct.  Unfortunately, I don't have the git ID.

I should add that there's now a 12.0.1 package in the arch repos that works when vmwgfx.ko is loaded into the kernel.  But, the X server will still crash if the kernel is not loaded.
Comment 9 Frederik Vos 2012-03-17 08:48:58 UTC
it's working for 32-bits and 64-bits vm's, but:
- vmware hardware version must be version 8, it's not working any longer for version 7
- vmwgfx driver must be loaded
- you'll need at least 32MB VGA memory
- you'll need xf86-video-vmware 12.0.1
Comment 10 Jakob Bornecrantz 2012-03-19 06:57:43 UTC
Created attachment 58683 [details] [review]
Fix

Can you please try this patch? Make sure that it isn't using the new driver.

Cheers, Jakob.
Comment 11 Jakob Bornecrantz 2012-03-21 04:37:06 UTC
Ping?
Comment 12 Rob David 2012-03-21 22:30:33 UTC
I see that arch has now upgraded to version 12.0.2-1 of the vmware driver, which I think includes that patch. Upgraded to that and X is working again in my virtual machine!
Comment 13 Adam Twardowski 2012-05-02 10:33:13 UTC
Created attachment 60912 [details]
Backtrace of vmware crash
Comment 14 Adam Twardowski 2012-05-02 10:34:23 UTC
I'm getting a similar segfault in the vmware driver v12.0.2 from arch linux 64-bit.  If you need more output than what is below let me know and I will have to re-compile X.


#0  crtc_shadow_destroy (crtc=0x20d0cd0, rotate_pixmap=0x0, data=0x26bf750) at vmwgfx_crtc.c:255
#1  0x00000000004acfb4 in xf86RotateFreeShadow ()
#2  0x00007fc95134d46b in vmwgfx_disable_scanout (pScrn=pScrn@entry=0x20cf870) at vmwgfx_crtc.c:120
#3  0x00007fc95134bcb2 in drv_leave_vt (scrnIndex=<optimized out>, flags=<optimized out>) at vmwgfx_driver.c:1119
#4  0x000000000048351c in ?? ()
#5  0x0000000000471116 in AbortDDX ()
#6  0x000000000056a1f2 in ?? ()
#7  0x000000000056a3f5 in FatalError ()
#8  0x00000000005633ce in ?? ()
#9  <signal handler called>
#10 crtc_shadow_destroy (crtc=0x20d0cd0, rotate_pixmap=0x0, data=0x26bf750) at vmwgfx_crtc.c:255
#11 0x00000000004acfb4 in xf86RotateFreeShadow ()
#12 0x00007fc95134d46b in vmwgfx_disable_scanout (pScrn=pScrn@entry=0x20cf870) at vmwgfx_crtc.c:120
#13 0x00007fc95134beb5 in drv_crtc_resize (pScrn=0x20cf870, width=1024, height=768) at vmwgfx_driver.c:204
#14 0x00000000004aad95 in ?? ()
#15 0x00000000004dd05c in ProcRRSetScreenSize ()
#16 0x0000000000434472 in ?? ()
#17 0x0000000000423535 in ?? ()
#18 0x00007fc953118455 in __libc_start_main () from /lib/libc.so.6
#19 0x000000000042380d in _start ()
Comment 15 Alan Coopersmith 2012-06-02 09:45:56 UTC
It looks like the original crash reported here is fixed in the vmware 12.0.2
release, which includes the vgaHWSetStdFuncs patch.

The last crash added here in xf86RotateFreeShadow seems unrelated at first
glance - is it time to move that to a new bug report and mark this one fixed?
Comment 16 Jakob Bornecrantz 2012-06-04 06:04:07 UTC
Created attachment 62503 [details]
Fix

Does this patch help?

Cheers, Jakob.
Comment 17 Jakob Bornecrantz 2013-04-11 18:57:02 UTC
The patch has been applied and released, the second reporter is MIA, closing bug as fixed.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.