The Sorry State of Dynamic Libraries on Cygwin

This is a follow up to my previous post on the Cygwin port of Sage 2.9.1.1. What started initially about some linking issue with libpari resolved itself into a very unexpected way: libpari.dll ended up in $PREFIX/bin instead of $PREFIX/lib. After I found the DLL by accident I discovered even more bizar behavior specific to Cygwin:

• Most packages we ship either don't build dynamic libraries or dutifully ignore the request to build them.
• Some put them into $PREFIX/bin instead of $PREFIX/lib. pari isn't the only one example, there are more.
• Some prefix the DLL with cyg instead of lib. I assume that it is to avoid name clashes with "native" Windows Dlls, but if you are stupid enough to put $CYGWIN/lib into your %PATH% my position is that you deserve the pain that is sure to follow.
  
• Most do not create proper def files for imports.
• Naming is woefully inconsistent: .a, .dll, .so, .dll.a and so on.

The problem isn't something that can be fixed overnight. It seems that certain auto-hell macros do various of the things listed above per default and a lot of projects do not test their builds on Cygwin or never hear complaints about those issues. During the 2.10 release cycle we will update a lot of spkgs. While I am at it I will make sure that

• they build properly on Cygwin
• they build shared libraries at all
• they created proper def files
• they put them into the right place.

There are also some other general issues that we need to take care of, so it seems like a good idea to combine fixing the above issues with those. More about that once we get closer to the 2.10 release cycle. This will be fun :)

Cheers,

Michael

Porting Sage 2.9.1.1 to Cygwin

We used to run Sage on Windows on top of Cygwin. When libSingular became a standard component of Sage around the 2.5.1 release the Cygwin port was dropped because Martin Albrecht and I were unable to get libSingular to link against the Sage extension on Cygwin. Both of us spend about a week attempting to fix the issue, but we never found a solution. Since Sage on Cygwin didn't pass a lot of other doctests we finally decided to drop the port and use the VMWare image as the only way to run Sage on Windows.

Recently I came up with some ideas how to solve the libSingular as well as some of the relocation problems that we had on Cygwin. This attempt was also in no small part motivated by a push to get on the way to a native Sage on Windows, discussed in a previous blog post of mine. Over the last 36 hours [of which I slept about 1/4 waiting on some compilation to finish] I finally took the time again to see how much trouble it is to get Sage 2.9.1.1 to compile via Cygwin. It wasn't pretty, but I got nearly all of Sage to compile with fixes/updates to the following spkgs.

• gmp
• cremona
• opencdk
• libgpg-error
• libgcrypt
• GNUTLS
• gdmodule
• linbox
• matplotlib
• mpfi
• quaddouble
• Singular
• libfplll
• polybori
• rpy
• rubiks
• sage-2.9.1.1

Note that I am only discussing the compilation of Sage, I have been unable to actually import any of those Sage modules. I suspect suspect either a broken python install [rpy fails to import as well as matplotlib hangs for hours at 99% cpu use, but numpy imports just fine] and/or some issue with the way libcsage.dll is now build. Consquently I have been unable to doctest the resulting build so far. I will try to build Sage on Cygwin again with my fixes merged probably in the 2.10 time frame as it will take quite a while to clean it all up and feed it back into the build system. Some of the build issues are generic in nature, others can be quickly fixed by adapting the spkg-install script while most actually required fixes to the sources of the spkgs themselves. I plan to clean up the fixes and push them all upstream to their respective maintainers, so that we can avoid patching the code ourselves in the future. It is amazing how just seven months time makes a port bitrot and in that time frame the build system on Cygwin didn't change at all. I was also quite surprised how much open source software out there doesn't compile on Cygwin out of the box. Oh well, the port to MSVC will take more than 36 hours and will be much more painful. Sometimes I wonder if I shouldn't do more useful things in my spare time :)

Cheers,

Michael

Sage Bug Day 8 coming up

As discussed in sage-devel we will do another Bug Day before the 2.9.2 release. Due to the AMS meeting date we chose Wednesday, January 2nd, 2008. That way all the fixes should make it into the 2.9.2 release which is currently targeted "just in time" for the meeting. As usual we have a Bug Day 8 wiki page. Please add yourself at the bottom if you plan to participate.

Cheers,

Michael

A native Sage on Windows?

Since the 2.9 release I spend most of my time fixing various bugs and memory leaks in preparation for the 2.9.1 release, but more about that later. This post is about what initially started out as some observations by Ondrej Certik [sorry, I am lazy here, no fancy accents :)] about the use of Mathematica in a Lie group lecture he was attending. It quickly turned into a somewhat heated discussion about the merits of a native Windows port, certainly to some extend due to a lengthy rant I posted early on in the discussion. Most of the thoughts and arguments I put down in writing in that thread have been on my mind for quite a while, so I finally took the time to put them all into one more or less coherent argument. I will not rehash the complete argument here, but feel free to follow the above link and read the whole discussion at the Google group.

The desire for a native Windows port of Sage has been discussed on and off by various people at Sage Days 5 & 6 face to face and in IRC over the last couple months. While we support Windows via a VMWare image many people (including myself) think that a truly native port could bring Sage and its components to a lot more people. We used to have a Cygwin port, but that was dropped during the 2.5.x development cycle since we didn't manage to make libSingular work despite us trying for several weeks.

During the discussions various point of views came up why we shouldn't do a native Windows port:

• A native Windows port would take away time from people that might be better spend on fixing bugs on OSX, Linux and Solaris.
• It just isn't worth it to port Open Source applications to Windows. If people were really interested in Open Source they would switch to some free operating system.
  
• We should spend time on improving the VMWare image instead.

I pretty much think that all of the above aren't valid or miss the point:

• Bugs that are fixed on on platform are often present on the others, they just haven't caused any trouble yet.
• For the foreseeable future Windows will remain the dominant desktop operating system. While that may be different in certain areas of academia we have to consider targets like high school students and so on. I do not buy the argument that by offering people Open Source applications on Windows those people no longer have an incentive to switch to a free platform. It is quite the opposite: Firefox has brought the idea of Open Source to many more people than the Linux kernel ever will. Take into consideration that I have been running Linux since the time you had to install it from sets of floppy discs that I downloaded at night over 14.4KB modem lines. And I have run it as my primary desktop OS ever since. But for many people switching away from Windows isn't desirable, because Windows is good enough for them. Another quite common scenario is that various lock in factors prevent people from moving away. So in the end we [as in Sage developers] just have to accept reality and do the best we can to bring Open Source math software to the masses, even if they "haven't seen the light yet" and are still using Windows.
  
• Improving the VMWare image is orthogonal to the port issue. The VMWare image has its places and in many cases will be better suited, but it cannot replace a native port.

But there are also certainly problems with the port:

• We will need to rely on non-free compilers, mainly MSVC [the express edition is free as in beer, but certainly not free as in freedom] and Intel Fortran [not free as in beer & freedom].
• It will be a huge amount of work and all that needs to be integrated upstream. So far no entity has been willing to support the developers to do so.
  
• We stand to gain few developers from the effort since most Open Source developers tend to work on anything but Windows and do not have the desire to go back. But the native Windows port will certainly require a lot of attention, maybe even more than the other ports combined.
  

There are also advantages:

• We will potentially gain 64 bit support on Windows. A lot of the components of Sage are only available via Cygwin or MinGW which are currently limited to 32 bits, discounting the fact that Cygwin runs in 64 bit mode on Windows/Itanium.
• The install will be smaller and it offers the possibility to do upgrades via binary packages. While we certainly plan to do something similar via the Debian packaging effort, it seems to be more or less the only way to do efficient upgrades on Windows since few people will have the tool chain to build Sage.
• The potential user base for Sage will increase by a magnitude or more. And we are not elitist snobs that want to exclude people based on their choice of operating system :)
  

So what needs to happen to make this a reality? Mostly [more] developers willing to commit. We are still in the planning stage, but have a wiki page about the port. So if you are interested let us know if you want to help out or just give your perspective on the issue.

Cheers,

Michael

Sage 2.9 Released

After two weeks of hard work we finally released Sage 2.9. Since we normally released about once a week it shows that more effort than usual was spend on the process. We closed 110 ticket in the process, as you can see when visiting our trac installaion. We finally merged some long requested packages, namely R and ATLAS. PolyBoRi was also merged as well as updates to FLINT 1.02 and Symmetrica 2.0. You can read the announcement from Google groups with all the gory details or go directly to the download page. While we did spend a large amount of time on getting everything to work perfectly there are still some known problems. Updates can also be a little tricky, but please report all issues as usual to sage-support.

I need to catch up with sleep now. The 2.9.1 release will be chaired by Robert Miller, while I will probably do the 2.9.2 release. Both are planned for a week and two weeks from now respectively.

Cheers,

Michael

Bug Day 7 Wrap Up, Sage 2.9.rc0 Released

Bug Day 7 turned into a 23 hour coding session for some of us. So I am quite tired and don't really up to remember all the details. We merged a lot of tickets, I will write a proper summary for 2.9.rc1 or 2.9.final. I would like to thank everybody who participated in Bug Day 7. The 185 MB tarball is at

http://sage.math.washington.edu/home/mabshoff/sage-2.9.rc0.tar

The release is planned in about 16 hours. So we will mostly concentrate on build and doctest fixes. 2.9.rc0 has a bunch of doctest failures related to calculus and number fields. Those will be fixed before the 2.9 final release.

The following releases are planned for the rest of the year:

• 2.9.1: chaired by Robert Miller, bug fixes, release planned about a week from now
• 2.9.2: chaired by Michael Abshoff, bug fixes, release planned about a week from now

2.9.2 will be the release we will distribute on DVD at the AMS meeting, so we are shooting hard for an excellent release.

Edit: There were some gremlins in this release which we are fixing right now. I also forgot to tag this post with Sage, so it didn't show up on Planet Sage yet.

Cheers,

Michael

Sage 2.9.alpha7 released

Hello,

This is alpha7, as you can guess alpha6 wasn't really ready for prime time. The main problem were little build oddities and some larger integration issues with ATLAS fixed by Josh Kantor and yours truly.
Additionally I merged a bunch of other, non-ATLAS related fixes by William Stein, Robert Bradshaw, Burcin Erocal, Yi Qiang and Mike Hansen. This release will be basis for Bug Day 7 and should have been out hours ago, but I fell asleep waiting for doctests to finish on OSX 10.5. The release compiles on sage.math as well as bsd [OSX 10.5] and all doctests pass. I consider this release quite stable.

The only high priority known build issue is #1497, for which a workaround exists. So if you run FC7, 32 bit on a Dual core CPU please disable power management completely before the build. We are working on a way to detect this issue and stop the build of ATLAS with a menaingful error message.

The 183 MB tarball is at

http://sage.math.washington.edu/home/mabshoff/sage-2.9.alpha7.tar

See the detailed announcement at Google groups.

Michael

Sage Bug Day 7: FRIDAY, December 14, 2007

Since nobody suggested any better date the next bug day will be held on FRIDAY, December 14, 2007. As usual we will start at 9am PST. I am aware that this is only about 12 hours away, but at least William and I will be working on the 2.9 release, which is planned for Saturday. Details as usual at

http://wiki.sagemath.org/bug7

Basis for the Bug Day will be 2.9.alpha7, which will be released in about an hour.

Cheers,

Michael

Valgrind 3.3.0 has been released

Vagrind 3.3.0 has been released. We use it extensively for Sage development and debugging and have been using 3.3.0svn for quite a while because of its increased performance as well for the much better SSE2 and SSE3 support. Among the new experimental tools are DRD and omega. In addition massif has been rewritten and hellgrind has been substantially updated. While I haven't used DRD yet, omega has turned out to be great help in tracking down memory leaks in the sparse linear algebra code of Sage. Memcheck did find those leaks, but didn't point to the location where memory was leaked, but to the location where the leaked memory was allocated. Omega is somewhat slower than memcheck (in the concrete examples I looked at by a factor of 20, on top of the 30 time slowdown of memcheck), but it is well worth the wait. Just make sure to compile your code with "-O0" for omega for optimal results.

An updated optional valgrind.spkg should be available soon.

I am finally blogging.

Hello,

after years of procrastination I am finally starting to blog. Why now? The Sage developers decided to create Planet Sage and it seems like a good idea to use this blog to let people know about the various open source mathematical projects I am involved in. I am also working on ports of Sage to FreeBSD, Solaris and hopefully in the future a native Windows post.

Currently I am release manager for the Sage 2.9 release due out in a couple days. So stay tuned for updates on that.

Cheers,

Michael