First sign of 64 bit OSX 10.5 support in Sage 3.0.x

Hello folks,

64 bit OSX 10.5 support has been promised for a while, but I finally managed a couple hours ago to fix a segfault issue in libSingular that made Sage segfault instantly when any multivariate polynomial ring was created. If you check out the 64 bit OSX port page in the wiki you will see that there are currently 35+ ticket listed which need to be fixed. I have patches for all of them, but the quality is uneven and a couple of those patches need to be redone. So it will take a couple more weeks until all those issues are sorted out and polished patches are merged into the official tree.

I have put together a binary [OSX 10.5 x86-64 only]. It is rather hefty at 0.5GB, but I have left a substantial number of hand build spkgs in the dmg more or less by accident. If you plan to build a 64 bit build of Sage on your OSX machine you should wait a couple more days until 3.0.2 is released which will contain the vast majority of fixes. It is even likely that there will be an official 64 bit OSX 10.5 binary, but currently there are 24 doctest failures. Some are trivial to fix, others are caused by the notebook not working due to _ctypes being broken. There are other small issues with pexpect and a couple additional small issues, but overall we are in pretty good shape. On exit OSX also complains about a couple "Non-aligned pointer being freed", but that seems to be mostly cosmetic and will be fixed soon since I got a good idea what is causing this. We will hopefully be at a point where we can call OSX in 64 bit mode fully supported soon, but there will be some more work involved before we are done.

Cheers,

Michael

Stupid Bugs - Part 823

Well, I guess I never mentioned any of the previous bugs, but today I started looking at two rather vexing bugs that hit us on RHEL 5/Itanium. One of them (#2985) was rather odd since I couldn't reproduce it anywhere and the binary did valgrind clean I was running out of ideas. After building Sage 3.0 on Ubuntu LTS 6.06 I hit the same bug there, so it was even more vexing since everything worked on 64 bit Debian testing. But after looking at it over on Ubuntu 6.06 LTS there I still didn't see anything obvious that could be wrong. Switching back to my Itanium test box eventually revealed the problem. Somebody had left a 32 bit x86 binary of optimal in the spkg. Since the makefile for the Reid solver sucked we never ended up building it on a most x86 compatible Linux platforms. So it did start up on Itanium and then more or less die instantly. The bug is now fixed and is merged in 3.0.1.alpha0.

Next up: #2983 - an Itanium specific segfault in Singular. The fun never stops ;)

I have been rather quiet here, but hopefully I will be more active again in the future. Many things are swirling around in my head, but during the 3.0 release cycle I did not have the time to actually write down a long winded rant here. But I think that will change the next week.

Cheers,

Michael

Sage 2.10 vs. MacIntel 64 bit

Well, after talking about for a couple weeks [or is it months by now?] I finally bit the bullet and stated porting the current Sage 2.10.alpha2 to 64 bit MacIntel. I started somewhere around 7 pm locally and now, roughly 12 hours later I struck out. For a first attempt it wasn't too bad. My build notes are a little rough, but surprisingly most issues are fiddling with CFLAGS, CXXFLAGS or CPPFLAGS. That wasn't too much of a surprise since most of the code already runs on MacOSX, but in the end I failed with various spkgs:

• numpy: odd issues with distutils, caused by the fact that I had to be tricky with python and add some flags after the fact for distutils. Consequently scipy and some other spkgs couldn't be build. Josh Kantor is investigating, so hopefully once I wake up it might be solved.
  
• PolyBoRi: SCons issues, mostly ENV related. Here I ran out of time
• twisted: It seems to depend on MacOSX specific python extensions, but I didn't investigate, but should be fixable.
  

But after working around various issues I got all but 5 extension modules to work (four times numpy related and the PolyBoRi extension), but in the end Sage did not start up to do the ceremonial 1+1, but failed in libSingular with:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000018
0x00000001045f0bfb in omInsertBinPage [inlined] () at om_Alloc.c:109
109 after->next->prev = page;
(gdb) bt
Function omInsertBinPage was inlined into function omAllocBinFromFullPage at line 145.
#1 0x00000001045f0bfb in omAllocBinFromFullPage (bin=0x10495aef0) at om_Alloc.c:145
#2 0x00000001045eebde in __omDebugAlloc (size_bin=0x10495aef0, flags=, track=, f=, l=) at om_Alloc.c:145

Since I need to catch some sleep now for a meeting in roughly seven hours I will sign off now.

The plan for the next 24 hours: Merge the easy changes back into 2.10.alpha3 or alpha4, update a bunch of spkgs and get Sage 2.10 toward release candidate status. There is still much to do for 2.10 and we had only planned to officially support MacIntel 64 bit with 2.10.1. I am not sure if we will hit that target, but it isn't looking too bad. Either way, William is excited and I guess there are quite a large number of people out there who would really like to use it. So feel free to help out.

Cheers,

Michael

The Sorry State of Dynamic Libraries on Cygwin

This is a follow up to my previous post on the Cygwin port of Sage 2.9.1.1. What started initially about some linking issue with libpari resolved itself into a very unexpected way: libpari.dll ended up in $PREFIX/bin instead of $PREFIX/lib. After I found the DLL by accident I discovered even more bizar behavior specific to Cygwin:

• Most packages we ship either don't build dynamic libraries or dutifully ignore the request to build them.
• Some put them into $PREFIX/bin instead of $PREFIX/lib. pari isn't the only one example, there are more.
• Some prefix the DLL with cyg instead of lib. I assume that it is to avoid name clashes with "native" Windows Dlls, but if you are stupid enough to put $CYGWIN/lib into your %PATH% my position is that you deserve the pain that is sure to follow.
  
• Most do not create proper def files for imports.
• Naming is woefully inconsistent: .a, .dll, .so, .dll.a and so on.

The problem isn't something that can be fixed overnight. It seems that certain auto-hell macros do various of the things listed above per default and a lot of projects do not test their builds on Cygwin or never hear complaints about those issues. During the 2.10 release cycle we will update a lot of spkgs. While I am at it I will make sure that

• they build properly on Cygwin
• they build shared libraries at all
• they created proper def files
• they put them into the right place.

There are also some other general issues that we need to take care of, so it seems like a good idea to combine fixing the above issues with those. More about that once we get closer to the 2.10 release cycle. This will be fun :)

Cheers,

Michael

Porting Sage 2.9.1.1 to Cygwin

We used to run Sage on Windows on top of Cygwin. When libSingular became a standard component of Sage around the 2.5.1 release the Cygwin port was dropped because Martin Albrecht and I were unable to get libSingular to link against the Sage extension on Cygwin. Both of us spend about a week attempting to fix the issue, but we never found a solution. Since Sage on Cygwin didn't pass a lot of other doctests we finally decided to drop the port and use the VMWare image as the only way to run Sage on Windows.

Recently I came up with some ideas how to solve the libSingular as well as some of the relocation problems that we had on Cygwin. This attempt was also in no small part motivated by a push to get on the way to a native Sage on Windows, discussed in a previous blog post of mine. Over the last 36 hours [of which I slept about 1/4 waiting on some compilation to finish] I finally took the time again to see how much trouble it is to get Sage 2.9.1.1 to compile via Cygwin. It wasn't pretty, but I got nearly all of Sage to compile with fixes/updates to the following spkgs.

• gmp
• cremona
• opencdk
• libgpg-error
• libgcrypt
• GNUTLS
• gdmodule
• linbox
• matplotlib
• mpfi
• quaddouble
• Singular
• libfplll
• polybori
• rpy
• rubiks
• sage-2.9.1.1

Note that I am only discussing the compilation of Sage, I have been unable to actually import any of those Sage modules. I suspect suspect either a broken python install [rpy fails to import as well as matplotlib hangs for hours at 99% cpu use, but numpy imports just fine] and/or some issue with the way libcsage.dll is now build. Consquently I have been unable to doctest the resulting build so far. I will try to build Sage on Cygwin again with my fixes merged probably in the 2.10 time frame as it will take quite a while to clean it all up and feed it back into the build system. Some of the build issues are generic in nature, others can be quickly fixed by adapting the spkg-install script while most actually required fixes to the sources of the spkgs themselves. I plan to clean up the fixes and push them all upstream to their respective maintainers, so that we can avoid patching the code ourselves in the future. It is amazing how just seven months time makes a port bitrot and in that time frame the build system on Cygwin didn't change at all. I was also quite surprised how much open source software out there doesn't compile on Cygwin out of the box. Oh well, the port to MSVC will take more than 36 hours and will be much more painful. Sometimes I wonder if I shouldn't do more useful things in my spare time :)

Cheers,

Michael

A native Sage on Windows?

Since the 2.9 release I spend most of my time fixing various bugs and memory leaks in preparation for the 2.9.1 release, but more about that later. This post is about what initially started out as some observations by Ondrej Certik [sorry, I am lazy here, no fancy accents :)] about the use of Mathematica in a Lie group lecture he was attending. It quickly turned into a somewhat heated discussion about the merits of a native Windows port, certainly to some extend due to a lengthy rant I posted early on in the discussion. Most of the thoughts and arguments I put down in writing in that thread have been on my mind for quite a while, so I finally took the time to put them all into one more or less coherent argument. I will not rehash the complete argument here, but feel free to follow the above link and read the whole discussion at the Google group.

The desire for a native Windows port of Sage has been discussed on and off by various people at Sage Days 5 & 6 face to face and in IRC over the last couple months. While we support Windows via a VMWare image many people (including myself) think that a truly native port could bring Sage and its components to a lot more people. We used to have a Cygwin port, but that was dropped during the 2.5.x development cycle since we didn't manage to make libSingular work despite us trying for several weeks.

During the discussions various point of views came up why we shouldn't do a native Windows port:

• A native Windows port would take away time from people that might be better spend on fixing bugs on OSX, Linux and Solaris.
• It just isn't worth it to port Open Source applications to Windows. If people were really interested in Open Source they would switch to some free operating system.
  
• We should spend time on improving the VMWare image instead.

I pretty much think that all of the above aren't valid or miss the point:

• Bugs that are fixed on on platform are often present on the others, they just haven't caused any trouble yet.
• For the foreseeable future Windows will remain the dominant desktop operating system. While that may be different in certain areas of academia we have to consider targets like high school students and so on. I do not buy the argument that by offering people Open Source applications on Windows those people no longer have an incentive to switch to a free platform. It is quite the opposite: Firefox has brought the idea of Open Source to many more people than the Linux kernel ever will. Take into consideration that I have been running Linux since the time you had to install it from sets of floppy discs that I downloaded at night over 14.4KB modem lines. And I have run it as my primary desktop OS ever since. But for many people switching away from Windows isn't desirable, because Windows is good enough for them. Another quite common scenario is that various lock in factors prevent people from moving away. So in the end we [as in Sage developers] just have to accept reality and do the best we can to bring Open Source math software to the masses, even if they "haven't seen the light yet" and are still using Windows.
  
• Improving the VMWare image is orthogonal to the port issue. The VMWare image has its places and in many cases will be better suited, but it cannot replace a native port.

But there are also certainly problems with the port:

• We will need to rely on non-free compilers, mainly MSVC [the express edition is free as in beer, but certainly not free as in freedom] and Intel Fortran [not free as in beer & freedom].
• It will be a huge amount of work and all that needs to be integrated upstream. So far no entity has been willing to support the developers to do so.
  
• We stand to gain few developers from the effort since most Open Source developers tend to work on anything but Windows and do not have the desire to go back. But the native Windows port will certainly require a lot of attention, maybe even more than the other ports combined.
  

There are also advantages:

• We will potentially gain 64 bit support on Windows. A lot of the components of Sage are only available via Cygwin or MinGW which are currently limited to 32 bits, discounting the fact that Cygwin runs in 64 bit mode on Windows/Itanium.
• The install will be smaller and it offers the possibility to do upgrades via binary packages. While we certainly plan to do something similar via the Debian packaging effort, it seems to be more or less the only way to do efficient upgrades on Windows since few people will have the tool chain to build Sage.
• The potential user base for Sage will increase by a magnitude or more. And we are not elitist snobs that want to exclude people based on their choice of operating system :)
  

So what needs to happen to make this a reality? Mostly [more] developers willing to commit. We are still in the planning stage, but have a wiki page about the port. So if you are interested let us know if you want to help out or just give your perspective on the issue.

Cheers,

Michael