2009/11/21

Segmentation fault on Ubuntu 9.10 Server under Windows 7 x64 Virtual PC

I have been using Ubuntu since version 8.10 Intrepid Ibex and I was anxiously waiting the release of Ubuntu 9.10 Karmik Koala some weeks ago. In previous versions of Ubuntu it was a nightmare to have it running under Microsoft virtual environments (i.e. Virtual Server 2007, Virtual PC 2004, Virtual PC 2007 and so on). Problems with screen resolutions, bouncing mouse cursors, and skewing clocks were common and somewhat hard do solve for a novice Linux user I was by those times.

The fact is that I tried Ubuntu 9.10 Server beta, some weeks before the final version was released, under Virtual PC 2007 on Windows Vista Ultimate x64 on my desktop computer. When the bare bone LAMP server was installed, I logged in and installed gnome-desktop, crossing fingers. I was gladly surprised that everything worked fine right after the reboot: no screen flicking, no bouncing mouse, all Ok. Great. It was still the beta but it was a promising start.

When the final version of Ubuntu Server 9.10 was released, I downloaded the ISO and tried to install it on my laptop, a Windows 7 x64 with Virtual PC, the one shipped with Windows 7, not Virtual PC 2007 that you must use in Vista. All my expectations felt helplessly to the mud.

Everything seemed to be fine when the installer told me to reboot the system for first time:

Installation is complete

I rebooted the virtual machine and … oops… segmentation fault. what? I rebooted once again, and the same error: segmentation fault. Sometimes the virtual machine window simply closed, if not, the console showed me the same error: Segmentation fault and rubbish all along the screen.

Segmentation fault 1

Segmentation fault 2 Segmentation fault 3

There was no way I could run Ubuntu 9.10 Server under Virtual PC from Windows 7 x64. I tried various different install configurations (LAMP, DNS, nothing at all), with different RAM sizes, I even tried to change some settings in the guest BIOS, without any luck. In all cases, when the machine booted, I get the segmentation fault error.

After reading some documentation about the general occurrence of a segfault error, and finding that it happens when the code being execute tries to read/write some memory allocation that it should not, or an invalid memory address.

It sounded me like something dealing with Data Execution Prevention or DEP. You can find those settings in your Windows 7 under System Properties –> Advanced Options –> Performance settings –> Data Execution Prevention.

I tried to disable Data Execution Prevention for %windir%\system32\vpc.exe (the executable file for Virtual PC) but since it was a 64bit system I got an error message: You cannot set DEP attributes on 64 bit executables. No luck this way either.

According to Microsoft about Data Execution Prevention:

32-bit versions of Windows Server 2003 with Service Pack 1 utilize the no-execute page-protection (NX) processor feature as defined by AMD or the Execute Disable bit (XD) feature as defined by Intel. In order to use these processor features, the processor must be running in Physical Address Extension (PAE) mode. The 64-bit versions of Windows use the NX or XD processor feature on 64-bit extensions processors and certain values of the access rights page table entry (PTE) field on IPF processors.

XD processor feature? Umhhh, my BIOS (the laptop, physical one) had such a thing… My laptop is a Dell Vostro 1700 and it has a setting called CPU XD Support. Why don’t we try to disable it? I rebooted to check that setting and I saw that it was Enabled (by default). Just for doing one more test, I disabled it and restarted.

CPU XD (Execute Disable) Support

I then started the Ubuntu 9.10 Server virtual machine and… it worked!!! I was even capable of installing gnome-desktop also and everything worked as it worked with Windows Vista in my desktop computer.

But is it safe to disable such a feature for the whole system? Just to be able to try and play with Ubuntu as a VM sometimes? I suppose not. I then rebooted and reset the value to Enabled (by default).

There must be something wrong with either Ubuntu or Virtual PC. Maybe Ubuntu is trying to execute certain memory address that are code for the guest, but data for the host. I don’t know.

At least, I have found a workaround for the problem. Whenever I want to test something in Ubuntu, it costs me a reboot, a change of settings in CPU XD Support value of the BIOS and a restart… ah.. and another reboot to change it back to the safe value.

If someone else finds a better workaround for this problem, I am willing to hear about it!

11 comments:

Steve said...

I had exactly the same issue until I bumped the memory to 1024M instead of the default 512 for the Virtual Machine. Looks good so far. Just logged in with no errors after a clean boot of 9.10.

José Antonio García Barceló said...

You're right. No hassles, no risks, simply brilliant. However, despite this 'workaround' of adding more ram to the VM, there must be something wrong with the memory management in Virtual PC bundled with Windows 7.

Steve said...

I agree with you. There is definitely something wrong in the memory management subsystem. Ubuntu 9.10 *should* run in < 400M. After the bump to 1024, I was able to login, -- but no apps would run and Gnome started to complain (didn't notice this until after my first post). I did an xterm login and found the messages log full of sigseg violations. Simple commands like 'df' and 'find' would only randomly function. After another 512M bump, I have a fully functional Gnome desktop. Ubuntu 9.10 would *never* need this much memory in its own physical box. I hope they are working on this one. This has much potential.

José Antonio García Barceló said...

I'm right after your steps... and agree with you again. Even thought I can now login, I cannot even install telnetd, much less ubuntu-desktop. I get segmentation faults everywhere whenever I do a sudo apt-get install whatever.

Have you tried to turn off your eXecute Disable bit on your BIOS? It might be, after all, that this is the only working option.

shevious said...

Thanks a lot, guys.

I have an intel board PC and I disabled Security->XD Technology in my bios. Then, ubuntu 9.10 works very good with only 256M RAM. (Only with the "vga=791 noreplace-paravirt" trick)

I've also tested with 1G RAM setting with XD enabled. The recovery mode booted well but gnome launched with too many glitches.

I'm using ubuntu 9.04 on VPC since it runs very well in Windows 7. I've tried to install 9.10 many times with no success. I agree that changing bios setting is not safe, but now I at least satisfied with the fact that there is a way! Thanks.

Anonymous said...

Same symptoms...bumping memory to 1024 worked for me as well

Anonymous said...

after reading your comments, im allmost sure it has something to do with swap partition and virtual memory, since increasing the memory of that virtual machine postpones ubuntu´s need to write something to swap/cache/virtual-memory (what ever it is called in linux enviroment) it will postpone the segmentation fault error aswell, im sure that increasing the memory size of that virtual machine isnt a real solution to that problem eventho it looks like it works

Anonymous said...

noreplace-paravirt i8042.noloop clock=pit

Simon L-Deslauriers said...

I have the same thing for a virtual PC downloaded at Nagios for Nagios XI.
The "Segmentation fault" was repeating on and on.

I have raised the memory from 768 to 1024 as suggest in one of the comments above and it corrected my problem.

mpatrix said...

Thanks, with Nagios i change the memory to 1024 and now work fine.

Regards,
mpatrix

Brian Klein said...

1024MB worked for me as well. Thanks!