dsage.setup() broken in Sage 3.0.2

Hello folks,

unfortunately a severe DSage bug slipped by since it was neither caught by doctesting nor the DSage unit tests. We are tracking the issue at #3311 and already have a patch that fixes the issue here. While poking around two more DSage bugs [#3312 and #3314] come to light, but those two are unreviewed so far. We will accelerate the 3.0.3 release because of the above bugs, but in case you are currently using 3.0.2 and want to use DSage please apply at least the fix from #3311 I linked above. At the ticket are more fixes, but only apply the patch linked since there are dependencies between the other patches.

Cheers,

Michael

Sage 3.0.2 released

Hi,

Sage 3.0.2 has been released. This is another bug fix release with some new features while we are waiting on the coercion rewrite to finish. Interesting new features include (in no particular order):

• Self-orthogonal Binary Codes (Robert Miller)
• Notebook Improvements (William Stein, Tom Boothby)
• Portability of Sage to 64 bit OSX and Cygwin (Michael Abshoff, William Stein)
• Posets and Semi-Lattices (Peter Jipsen and Franco Saliola)
• Frobby for monomial ideals (Bjarke Hammersholt Roune)

As usual check the Sage 3.0.2 release tour in the wiki for details. Some people still have to fill in the details, but I send those people emails to remind them ;)

Sources were posted yesterday and today we posted 15 binaries for various platforms in the usual place. The binaries have been mirrored out to some websites and the other mirros should catch up in the next twentyfour hours. Not all binaries are there yet, i.e. the VMWare binary is still missing, but should show up by Monday at the latest.

Cheers,

Michael

Sage 3.0.2.rc3 released

Hello folks,

here we go with 3.0.2.rc3. It is basically a bunch of bug fixes for rlm's codes code that had a couple small issues left ;) In addition there is one pbuild issue fix that Jaap encountered and that in the past was also hit by David Joyner. You can download the Sage 3.0.2.rc3 sources here.

Unless something goes horribly wrong this will be identical to the final 3.0.2 release. If you added some feature that should be in the Sage release tour please add them to Sage 3.0.2 release tour wiki page. Right now that list consists of

• Franco Saliola and Peter Jipsen's posets and semi-lattive patch
• Robert Miller's self-orthogonal binary codes
• Bjarke Hammersholt Roune's Frobby is now an optional spkg

But feel free to add items. As usual please test and report any issue you hit.

Cheers,

Michael

Sage 3.0.2.rc0 released

Hello folks,

this is 3.0.2.rc0, the likely final release of the 3.0.2 series and hopefully next to identical to the final 3.0.2. What is new?

• Franco Saliola and Peter Jipsen's posets and semi-lattive patch
• Robert Miller's self-orthogonal binary codes
• Bjarke Hammersholt Roune's Frobby is now an optional spkg
• Pbuild should now pass the doctests since #3097 has been fixed

In addition there were the usual bug fixes. Please build and doctest as usual and report any issues you see.

Sources and a sage.math-only binary in the usual places:

• Sage 3.0.2 sources
  
• Sage 3.0.2 sage.math-only binary
  

Cheers,

Michael

Merged in rc0:

#1762: Robert Miller, Michael Abshoff: Create optional graphviz package
#2121: Robert Miller: move libecm wrapper from interfaces to libs
#2519: Franco Saliola, Peter Jipsen: Add support for posets, semi-lattices, etc. to Sage
#3018: Bjarke Hammersholt Roune: Integrate Frobby into Sage
#3097: Gary Furnish, Michael Abshoff: pbuild: make sure the files from setup.py's scripts section are copied
#3104: William Stein: pbori.pyx: Make some doctest long since it uses a lot of RAM
#3112: Robert Miller: Generate self-orthogonal binary codes
#3148: Francis Clarke: improved orthogonal functions
#3218: Michael Abshoff: fix 64 bit OSX build support for mercurial
#3219: William Stein: upgrade to gmp-4.2.2 while we wait for MPIR
#3242: Robert Miller: Fix little bug in G.relabel() for G a graph
#3245: Mike Hansen: provide coefficient and coefficients methods for symbolic expressions
#3257: Gary Furnish: Pbuild ignores gcc specific default settings
#3263: Craig Citro: typo in lseries_ell.py
#3266: William Stein: Sage 3.0.2.alpha1: doctest failure in sage/server/simple/twist.py
#3267: Michael Abshoff: Sage 3.0.2.alpha1: doctest failure in sage/server/support.py
#3269: Jason Bandlow: Improve documentation for combinat/dyck_word.py
#3270: Robert Miller: trivial 100x speedup in coding theory
#3272: Craig Citro: Bug in sparse polynomials over finite fields
#3273: Robert Bradshaw: extend isqrt to work for Python int's in addition to Sage integers and objects with an isqrt method
#3274: Michael Abshoff: OSX: delete libpng*.la since we also nuke libpng*.dylib
#3275: Craig Citro: Make SL2Z distinct

Sage 3.0.2.alpha1 released

Hello folks,

the 3.0.2 release cycle so far has been a little slower than usual and there are a couple contributing factors. While some of us are waiting on the coercion rewrite to progress, others had obligations with school and the end of the semester seems to have kept a lot of people busy. 3.0.2.alpha1 contains *a lot* of porting work for Cygwin and Mac OSX 64 bit. We also merged Dan's new Weyl Character implementation and the usual set of bug fixes.

Since it has been nearly two weeks it is now time to get the release out of the door. So the plan now is to get an rc0 out the door tomorrow and then do a release by Tuesday or Wednesday. We will hopefully merge some more porting fixes, i.e. there are about ten more OSX 64 bit tickets ready to be reviewed. Other than that we have 70+ tickets with patches, so there is plenty to review. Once I catch some sleep I will do another post with details.

Sources and Binaries are in the usual place:

• Sage 3.0.2.alpha1 sources
• Sage 3.0.2.alpha1 sage.math only binary
  

Right now there are two notebook related doctest failures which do not have tickets yet. Please build, test and report issues you see.

Cheers,

Michael

Details for closed tickets in alpha1:

#406: William Stein: notebook -- make tab completion not stick gap. when using the notebook in gap mode
#637: William Stein: notebook improvement -- upload allow txt worksheets.
#1733: William Stein: notebook bug -- %foo (or anything else) in a cell by itself (with nothing else in the cell) does not give an error but it *should*
#1864: William Stein: simple notebook bug -- typing ? in a comment yields introspection but shouldn't
#1892: William Stein: notebook -- uploading a data file should give some help about the DATA variable
#2359: William Stein: notebook -- make it so when you send a kill signal to the notebook server it saves state
#2636: William Stein: notebook -- changing a cell without evaluate should put the red line back to the left
#2860: William Stein: easy-to-fix bug in html.py
#2884: William Stein, Tom Boothby: notebook -- bug; @interact cell eval doesn't clear out the old html output
#2992: William Stein: notebook -- help(foo) in the notebook should not word wrap
#3024: William Stein: notebook -- parses tracebacks in the output of docstrings of help command
#3050: Timothy Clemans: notebook -- add a "remember me" checkbox to the login page
#3051: Dan Bump, Mike Hansen: Implement Weyl Characters
#3053: William Stein: notebook -- new cell_resize doesn't respect %hide at the beginning of a cell
#3069: William Stein: notebook -- typeset checkbox doesn't work after save/reload
#3137: Yi Qiang: view command in misc/latex.py -- fix to not hardcode xdvi command
#3153: Carl Witty: make finite_field_ntl_gf2e use randstate framework
#3155: Timothy Clemans: notebook postdata and behaviour of archive, delete and stop buttons
#3160: Emily Kirkman, Robert Miller: change is_planar for graphs to return bool
#3161: Michael Abshoff: sdist: #3046 seems to have broken sage-banner
#3170: Michael Abshoff: add 64 bit OSX build support to readline
#3171: Michael Abshoff: add 64 bit OSX build support to termcap
#3172: Michael Abshoff: add 64 bit OSX build support to prereq and bzip
#3176: Michael Abshoff: add 64 bit OSX build support to sqlite
#3177: Michael Abshoff: fix 64 bit OSX build support for python
#3178: Michael Abshoff: add 64 bit OSX build support to freetype
#3179: Michael Abshoff: more 64 bit OSX libpng fixes
#3181: Michael Abshoff: add 64 bit OSX build support to iml
#3182: Michael Abshoff: improve 64 bit OSX build support for givaro
#3183: Michael Abshoff: add 64 bit OSX build support to linbox
#3186: Michael Abshoff: fix 64 bit OSX build support for numpy
#3187: Michael Abshoff: fix 64 bit OSX build support for matplotlib
#3188: Michael Abshoff: add 64 bit OSX build support to mpfi
#3189: Michael Abshoff: add 64 bit OSX build support to pycrypto
#3190: Michael Abshoff: add 64 bit OSX build support to zodb
#3191: Michael Abshoff: add 64 bit OSX build support to quaddouble
#3192: Michael Abshoff: fix 64 bit OSX build support for python_gnutls
#3197: Michael Abshoff: fix 64 bit OSX build support for m4ri
#3198: Michael Abshoff: fix 64 bit OSX build support for ecm
#3200: Michael Abshoff: fix 64 bit OSX build support for genus2reduction
#3213: Timothy Clemans: notebook -- Account Settings page for changing password and e-mail address
#3220: William Stein: readline -- fix a couple of issues
#3222: William Stein: sqlite -- add cygwin support to sqlite
#3233: William Stein: cygwin -- make linbox work with cygwin
#3224: Michael Abshoff: add 64 bit OSX build support for lcalc
#3225: Michael Abshoff: add 64 bit OSX build support for cddlib
#3226: Michael Abshoff: add 64 bit OSX build support for gfan
#3234: William Stein: cygwin -- make numpy work with cygwin
#3235: William Stein, Michael Abshoff: cygwin -- mpfi; get it to work with Cygwin by fixing configure.ac
#3236: William Stein, Michael Abshoff: cygwin -- get quaddouble to work with cygwin
#3238: William Stein, Michael Abshoff: libfpll spkg -- update to work with cygwin
#3239: William Stein, Michael Abshoff: cygwin polybori -- add Cygwin build support for polybori
#3230: William Stein: cygwin -- new givaro spkg that works around stupidity in cygwin
#3241: William Stein, Michael Abshoff: cygwin -- new rubiks spkg that builds on cygwin
#3243: William Stein: cygwin -- get log2 to work on cygwin
#3246: William Stein: cygwin -- fix broken gsl.ode setup.py code
#3252: Yi Qiang: add kbase functionality to libsingular

First sign of 64 bit OSX 10.5 support in Sage 3.0.x

Hello folks,

64 bit OSX 10.5 support has been promised for a while, but I finally managed a couple hours ago to fix a segfault issue in libSingular that made Sage segfault instantly when any multivariate polynomial ring was created. If you check out the 64 bit OSX port page in the wiki you will see that there are currently 35+ ticket listed which need to be fixed. I have patches for all of them, but the quality is uneven and a couple of those patches need to be redone. So it will take a couple more weeks until all those issues are sorted out and polished patches are merged into the official tree.

I have put together a binary [OSX 10.5 x86-64 only]. It is rather hefty at 0.5GB, but I have left a substantial number of hand build spkgs in the dmg more or less by accident. If you plan to build a 64 bit build of Sage on your OSX machine you should wait a couple more days until 3.0.2 is released which will contain the vast majority of fixes. It is even likely that there will be an official 64 bit OSX 10.5 binary, but currently there are 24 doctest failures. Some are trivial to fix, others are caused by the notebook not working due to _ctypes being broken. There are other small issues with pexpect and a couple additional small issues, but overall we are in pretty good shape. On exit OSX also complains about a couple "Non-aligned pointer being freed", but that seems to be mostly cosmetic and will be fixed soon since I got a good idea what is causing this. We will hopefully be at a point where we can call OSX in 64 bit mode fully supported soon, but there will be some more work involved before we are done.

Cheers,

Michael

Sage 3.0.2.alpha0 release

Hello folks,

this is Sage 3.0.2.alpha0. What happened? It seems that people were busy and until this morning there wasn't a whole lot to merge. But I had a busy day today and finally these is something to put out. We are still mostly on bug fix only mode, so no big surprises. "sage -sdist" seems to have been broken by David Joyner's #3046, so sage-banner is emtpy [see #3161]. I fixed this in the tarball and it will be fixed in Sage 3.0.2.alpha1.

Binaries and sources in the usual place:

• Sage 3.0.2.alpha0 sources
• Sage 3.0.2.alpha0 sage.math-only binary

I also created md5sums. There are 70+ tickets with patches in trac, among them quite a number of notebook patches. So it would be very nice if people can do some serious review. Note that the notebook patches have some rather large dependency tree available in the wiki here.

Cheers,

Michael

Merged in alpha0:

#336: William Stein, Timothy Clemans: Create an option to clear all cell output
#905: Burcin Erocal, Michael Abshoff: update ipython to 0.8.2
#1230: William Stein: Quit worksheet behaviour
#1557: William Stein: notebook -- usability improvement after uploading file
#2684: Jason Grout: vertices should not default to red
#2768: Jason Grout: add comparison operators to the fast_float mechanism
#2926: Timothy Clemans: notebook -- Minimalistic change password page for notebook user
#2983: Michael Abshoff: Itanium (RHEL 5) -- singular interface problems in matrix_group.py
#3008: William Stein: first cell in notebook is undeletable
#3020: John Cremona, Martin Albrecht: Speed up Finite Fields of characteristic 2 constructors
#3026: Bjarke Hammersholt Roune: multivariate polynomial rings with no variables do not print properly
#3028: Bjarke Hammersholt Roune: Ideals in multivariate polynomial rings with no variables raise exception on comparison
#3065: Didier Deshommes: empty matrices: frobenius() throws RuntimeError
#3105: Francis Clark: new _latex_ and modified __repr__ for elements of relative number fields
#3109: William Stein: elliptic curves -- implement P.divide(n) for P a point on an elliptic curve and n an integer
#3110: Gary Furnish: fix pbuild dependency bug
#3116: Mike Hansen: 1x1 symbolic matrices don't work right
#3121: Jason Grout, William Stein: @interact grid control
#3125: Robert Miller: chromatic_polynomial incorrectly blocks control-c
#3126: Robert Bradshaw: Cython annotation has unicode errors (e.g. from the notebook)
#3129: Bjarke Hammersholt Roune: The singular interface should not claim to support polynomial rings with no variables
#3136: William Stein: the readme for osx should be changed to delete the line about inotebook()
#3138: Bjarke Roune: Singular multivariate polynomial ring has redundant _repr_ method
#3142: Martin Albrecht: MPolynomialIdeal.homogenize bugfix
#3143: Martin Albrecht: remove references to "/home/was"
#3046: David Joyner: version option returning clone branch name
#3150: Carlo Hamalainen: Memory leak in dancing_links.pyx
#3157: Gary Furnish: Executable target for pbuild
#3158: Michael Abshoff: singular-3-0-4-2-20080405.p1 requires flex
#3159: Tim Abbott, Francois Bissey, Michael Abshoff: Patch adding soname to ntl shared library

Stupid Bugs - Part 824

While building Sage 3.0.1 on a pure Solaris install, i.e. no Sun FreeWare build tools in $PATH, I come across the following beauty:

bash-3.00$ which ranlib
no ranlib in /opt/sfw/bin/ /usr/local/bin/ sparc-SunOS/ /usr/bin /home/mabshoff/bin
bash-3.00$ echo $?
0

Sigh. I guess I need to fix that in spkg/base/prereq-0.3-install.

Cheers,

Michael

Sage 3.0.1 released

Hi,

Sage 3.0.1 is out. Compared over 3.0.1.final there was one crucial bug fix to the Singular pexpect interface. Binaries have been build and are currently mirroring out. This is mostly a bug fix release, but there are also some new interesting features. For details check the Sage 3.0.1 release tour, which is as I type this still somewhat work in progress.

The next release is another bug fix release, i.e. Sage 3.0.2, planned about a week from now. It will likely be released with much of the work that will be done during Bug Day 12 this Saturday. Other than that we are waiting for the Coercion team to finish their rewrite which will then be merge in Sage 3.1.

Cheers,

Michael

Sage 3.0.1.rc0 release

Hello,

This is the end of the 3.0.1 release cycle. The build was announced in IRC about eight hours ago, but since I took a long nap in the meantime I never posted to sage-devel ;)

Gary's pbuild has been improved and three bugs have been fixed. Please try it out again for feedback. To use pbuild

export SAGE_PBUILD=yes

before building Sage. The number of threads used during the build of the Sage library is set via SAGE_BUILD_THREADS, i.e.

export SAGE_BUILD_THREADS=8

for eight threads. Switching between pbuild and normal build is not possible in all cases, so in case of problems nuke the build directory on devel/sage.

Things should "just work" in this release [famous last words], but there are some reported oddities with pexpect behaving badly. We have so far been unable to reproduce and of those issue on a local system [Gary claims testdoc.py hangs for him on sage.math - but it doesn't for me], so if you see anything please let us know.

Sources and binaries are in the usual place. This is the end of the release cycle. A handful more patches are planned for 3.0.1.final. Anything non-blocker will be considered for 3.0.2.

Cheers,

Michael

Merged in rc0:

#2755: Andrey Novoseltsev: lattice_polytope.py update
#3060: Marshall Hampton, Michael Abshoff: update optional biopython package to 1.45 release
#3062: Timothy Clemans: implement __oct__ special method for the integers
#3070: Robert Miller: bug in SymmetricGroup(1).cayley_graph()
#3071: Gary Furnish: Using pbuild does not create site-packages sage symlink
#3072: Willem Jan Palenstijn: sage -i numeric-24.2 (and all other experimental packages) fails
#3074: Robert Bradshaw: update Cyton to the 0.9.6.14 release
#3076: Michael Abshoff: spkg-debian in extcode spkg not executable
#3077: Gary Furnish: pbuild does not return properly on failure
#3078: Willem Jan Palenstijn: sage's spkg-install doesn't return failure if build failed
#3082: William Stein: sage-3.0.1.alpha1: a twist.py doctest failure

Sage 3.0.1.alpha1 released

Hello,

this release should have been out two days ago, but somehow general slowness and me spending a lot of time on various porting issues did delay this release more than it should have. Gary's pbuild should now be fully functional, but it wasn't made the default build system yet. If you run

export SAGE_PBUILD=yes

before building Sage pbuild will be used. All the various sage [-b|-ba|-br|...] should work as should -clone & friends. The number of threads used during the build is set via

export SAGE_BUILD_THREADS=8

for eight threads. Switching between pbuild and normal build is not possible in all cases, so in case of problems nuke the build directory on devel/sage. Please try out pbuild and report any trouble. We *really* want to switch to it per default soon.

I am currently seeing an odd doctest failure in

doctest failure in devel/sage/sage/server/simple/twist.py

Other than that things should "just work".

Sources and binaries are in the usual place. We are getting toward the end of the release cycle

Cheers,

Michael

Merged in 3.0.1.alpha1:

#1549: Alex Ghitza: Sage 2.9: fix optional doctests in tut.tex
#2216: Alex Ghitza: Creating an order in a number field --> infinite loop?
#2504: Alex Ghitza: number field .units() method caches proof=False result and returns it for proof=True
#2523: Craig Citro: bug in modular symbols for GammaH subgroup
#2716: Marshall Hampton: convex hulls and polyhedral functions
#2741: William Stein, Timothy Clemans: Implement mesh lines in 3d plots
#2938: Craig Citro: Fix ModularSymbols(GammaH(8,[3])).decomposition() ModularSymbols(GammaH(81, [10])).decomposition();
#3029: Tim Abbott: Move DEB_AUTO_UPDATE_DEBIAN_CONTROL out of Debian packages
#3030: Gary Furnish: Cython working directory command line option patch
#3031: Kiran Kedlaya, Craig Citro: Add zeta_function method for schemes
#3032: Dan Bump: minor docstring cleanup in crystals.py and tensor_product.py
#3034: Tim Abbott: improved cleaning code for Debian packages
#3036: Tim Abbott: SAGE_TESTDIR broken
#3037: Gary Furnish, Robert Bradshaw: update cython to 0.9.6.13-20080426
#3038: Tim Abbott: SAGE setup.py fixes for using Debian packaged polybori, zn_poly
#3039: Tim Abbott: Improve auto-generated version numbers for Debian packages
#3041: Francois Bissey, Michael Abshoff: optimization setting in LinBox.spkg is broken
#3054: Jason Grout: copying a graph doesn't copy _pos or _boundary
#3055: Jason Grout: creating subgraph does not delete _pos entries
#3057: Tom Boothby: MPolynomialRing_generic type-checks to determine commutativity
#3059: William Stein, Timothy Clemans: notebook -- rewrite notebook(...) function to *not* use SSL by default
#3061: Michael Abshoff, Max Murphy: use readlink and realpatch so that symlinking sage works
#3063: Didier Deshommes: empty matrices: norm() returns a ValueError
#3064: Didier Deshommes: empty matrices: density() function throws a ZeroDivisionError
#3066: Didier Deshommes: empty matrices: gram_schmidt() throws a NameError
#3067: Didier Deshommes: matrices: numeric_array() is missing an import

Sage Lisp Wars!

Once upon a time a long, long time ago in a happy world where men were real men and wrote their own device drives in assembler - well, screw that, it is 2008. This is the story about whether Maxima, which is the only Sage component written in lisp, will stay in Sage long term or not.

To make things easier let's get some things sorted out before we get into the story:

Places:

1. "benchmarking CAS" thread in sci.math.symbolic [currently 40 posts]
2. "multivariate factoring - use maxima ?" thread in sage-devel [currently 36 posts]
3. "Project" thread in sage-devel [currently 75 posts]
4. "compiling Maxima by ECL" thread in sage-devel [currently 4 posts]
5. "compiling Maxima by ECL" thread on the maxima-devel mailing list
6. Richard Fateman's email from the above thread about ecl that Juan replied to.
   

This is a tale about my personal comment in [1] stating:

This is motivated by the need for performance and the fact that we want to dump any lisp based code from the core of Sage in the long term.

while taking about the long term plan for symbolics code in Sage. I am doing much of the porting of Sage to pretty much anything that I would consider worth porting to. While some people would question the value of a port to OpenBSD [I don't, but that is a different blog post I had in my head for a while without the time to write it down :)] few people would think that the port of Sage to native Windows in 32 & 64 bit mode would be bad for Sage. But porting Sage to any platform also means porting Maxima and since Maxima is written in common lisp it does require a common lisp implementation. Fortunately Maxima quite happily runs on top of cmulc, sbcl, clisp and gcl. But since Sage is a CAS where every component compiles from source and cmucl as well sbcl require another lisp to bootstrap themselves only clisp and gcl are fit as candidates for Sage. Incidentally sbcl is an amicable fork of cmucl since cmucl can only bootstrap itself off another cmucl instance and in contrast sbcl requires only a common lisp capable lisp instance to do so, but I am getting off path here.

So, there are two candidates to be used as common lisp implementations in Sage. Where is the trouble? Well, the devil is in the details. We need one and only one tarball that builds on a wide variety of systems and way back in the infancy of Sage William decided to go with clisp since it did build for him much more easily on the then supported platforms. gcl is more difficult to build and is lacking support for some platforms we will port to and on others build trouble is pretty much everywhere. See [3] for some examples. But clisp is troubled itself. It has to be compiled with gcc 4.2 and 4.3 with "-O0" due to some compiler bug in gcc. But on Solaris the last clisp release that somebody got to compile and run was 2.38 in 32 bit mode while the current release is 2.44.1. I did do much more than I thought was expected of me to resolve the issue, but in the end there was no solution (see [3] again).

All of the above has lead me to the conclusion that the lisp ecosystem is in "trouble" because its Open Source toolset is in a terrible state. The only ray of hope I have seen in the Open Source lisp ecosystem is ecl, but Maxima does not run on top of it. Way back in 2005 Michael Goffioul did some work to get Maxima to compile on top of ecl, but it seemingly didn't go anywhere. So I saw little chance for Maxima on top of ecl and since I neither have the time nor the expertise to attempt such a port my conclusion was to get rid of Maxime in the core of Sage as quickly as possible. There are various Sage developers who do think alike, so a plan was devised to replace everything use in Maxima with either other components or by code written from scratch. But then in [3] Robert Dodier told us that he had done some work on Maxima on top of ecl in January of 2008 and he was willing to attempt to finish the port since he also saw gcl's use of the Maxima on Windows installer troublesome. Progress in this direction has been made (see [4]) and now I am quite hopeful that we will have Maxima on top of ecl and therefore one of the showstoppers of Sage on Solaris as well as Sage on native Windows has been removed without making Maxima collateral damage in the porting process. Maxima will still have to compete with the other components in Sage and some functionality will be moved into the core of Sage by rewriting it in C/Cython, but I now see its future assured in the long term.

So, what motivated me to write this all down? Most people either don't care or read the long threads on sage-devel and a couple other places or have actually read some or most of the discussions. It was actually the reply to an email of Richard Fateman [6] by Juan Jose Garcia-Ripoll, the ecl maintainer, which deserves to be quote in full:

Hi,

let me first begin by saying that, as politely as I can, Fateman's email are as close to FUD as it can get. He doesn't seem to use ECL at all and just judges from some outdated webpages and his own prejudices about different software libraries.

Regarding the different points which have been raised:

* GNU MP is only used for bignum computations. ECL itself is clever enough to handle fixnums cleverly and even to unbox fixnum computations in compiled code. Incidentally, GCL uses GMP as well, so I do not see the point all.

* The simplistic garbage collector is an option and it is provided for platforms in which Boehm-Weiser does not run. Currently, this means _none_ of the supported platforms.

* Boehm-Weiser is a strong garbage collector and a very powerful one in terms of tunability. You man make it as precise as you want, and the Java people indeed do. ECL uses it and it has seen only performance improvements as we have learnt more and more how to better use it. If the ANSI test suite shows something in that respect is that, under a lot of consing pressure, it does not perform that bad. It does not get so close to SBCL's but I doubt any other free implementation does.

* ECL has a good compromise between all platforms. It provides both C compiled code and a reasonably fast interpreter. Benchmark show that the ECL interpreter is not that far from interpreted CLISP. But on the other hadn CLISP has its own set of optimized bytecodes and when it compiles it optimizes for those bytecodes. AFAI remember, GCL used (and probably still uses) a list based interpreter which runs through forms represented as lists and macroexpanding every form that has to be done so, and every time it uses it. That is terribly inefficient.

* In terms of maintainability it has shown through the years that it is easier for somebody to start coding and hacking ECL and adding new features than with most other platforms. That is how we got ECL ported to the Microsoft compiler and platform and how different pieces of software (sockets, asdf, etc) have been adapted to run here. That by itself is an important value, at least for people who think long term.

* Talking about diverting efforts from the GCL crowd, I am not the best person to speak about it. I am more than pissed of by the GCL community since, shortly after ECL reached most ANSI compliance and portability I was asked to port all that back to the GCL, because they wanted to achieve the same goal. That was back in '01 or '02, do not remember so well. What I remember is that those were not very polite emails and had a kind of "borg" spirit of assimilation, without even caring about the years spent on achieving that. So talk about diverting useful efforts.

So, to the interested parties, if you so much care about Maxima running on just one computer, then stick to sbcl and cmucl which are pretty superior implementations, but please do not scare people from porting useful software to other platforms and environments.

Kind regards

Juanjo

-- Facultad de Fisicas, Universidad Complutense,
Ciudad Universitaria s/n Madrid 28040 (Spain)
http://juanjose.garciaripoll.googlepages.com

So I can only recommend that if you are willing to waste an hour or two to read all the sources and make yourself a picture for yourself. There is much more to the story than I wrote here and I did not go into as much details as I could have since I wanted to finish this blog post in a reasonable amount of time. This is precisely the reason I didn't write it in the first place a couple weeks ago.

So, will there be "Sage Lisp Wars - The Sequel!"? I am sure there will be and you can easily figure out which side of the debate I am on ;) Despite of all the flames I linked to above in the end I feel that the Sage community got something positive in the end by the likely port of Maxima to ecl and at the same time hopefully the Maxima community will see the port to ecl as a benefit to them, regardless how the cooperation between Maxima and Sage will develop.

Cheers,

Michael

Sage 3.0.1.alpha0 released

Hello folks,

this is 3.0.1.alpha0. So far we have only merged bugfixes, nothing invasive has been merged yet and there is nothing on the radar that does look invasive. 24 tickets have been closed up to now and I am not quite sure what the rest of the release cycle will look like because it currently doesn't look like we need a pure bug fix only release quickly.

There are plenty of patches available for review. The coercion rewrite planned for 3.1 seems to be going well.

Sources and binaries are in the usual place:

• 3.0.1.alpha0 source
• 3.0.1.alpha0 sage.math only
  

Cheers,

Michael

Merged in alpha0:

#783: Alex Ghitza: dilog is lame
#1187: Alex Ghitza: bug in G.conjugacy_classes_subgroups()
#1921: Alex Ghitza, Mike Hansen: add random_element to groups
#2302: Michael Abshoff, William Stein, Scot Terry: Add a 64 bit glibc 2.3 based binary of Sage to the default build platforms
#2325: David Roe, Kiran Kedlaya: segfault in p-adic extension() method
#2821: Alex Ghitza: get rid of anything "about this document" sections of any sage docs that say "send email to stein"
#2939: David Joyner: piecewise.py improvements (docstring and laplace fixes)
#2985: Michael Abshoff: ITANIUM (RHEL 5) -- bug in rubik.py's OptimalSolver()
#2993: Michael Abshoff: OSX/gcc 4.2: disable padlock support per default
#2995: Alex Ghitza: some new functionality and doctests for congruence subgroups
#3003: Jason Brandlow: Bugfix for to_tableau() method of CrystalOfTableaux elements
#3005: Craig Citro: modabar -- failure to compute endomorphism ring
#3006: David Joyner: missing elliptic integrals in special.py
#3014: Michael Abshoff: ZZ.random_element -- corrupted docstring
#3017: Michael Abshoff: invalid link after make install
#3022: Tim Abbott: Debian package support for polybori
#3023: Jason Grout: make apply_map deal with empty matrices
#3025: William Stein: Sparse vector spaces don't cast on assignment
#3027: Tim Abbott: Debian lintian fixes

Ubuntu LTS 6.06 x86-64 binaries Available

We finally have Ubuntu 6.06 LTS x86-64 binaries for Sage 3.0 available. it was mentioned in the release announcement, but a last minute bug did delay the release of the binaries since the rubiks.py doctest failed. That has been fixed.

Well, you might ask, Ubuntu LTS 8.04 is out, so what about binaries for that release? And the answer is the same as always: Make a VMWare image with minimal install and development tools, a working ssh access from the outside, a home partition with a devent amount of space [i.e. 20GB] and get it to William or me. From then on we will build you binary releases and the chance that Sage will compile and doctest fine on your preferred distrbution will be greatly improved.

Cheers,

Michael

Stupid Bugs - Part 823

Well, I guess I never mentioned any of the previous bugs, but today I started looking at two rather vexing bugs that hit us on RHEL 5/Itanium. One of them (#2985) was rather odd since I couldn't reproduce it anywhere and the binary did valgrind clean I was running out of ideas. After building Sage 3.0 on Ubuntu LTS 6.06 I hit the same bug there, so it was even more vexing since everything worked on 64 bit Debian testing. But after looking at it over on Ubuntu 6.06 LTS there I still didn't see anything obvious that could be wrong. Switching back to my Itanium test box eventually revealed the problem. Somebody had left a 32 bit x86 binary of optimal in the spkg. Since the makefile for the Reid solver sucked we never ended up building it on a most x86 compatible Linux platforms. So it did start up on Itanium and then more or less die instantly. The bug is now fixed and is merged in 3.0.1.alpha0.

Next up: #2983 - an Itanium specific segfault in Singular. The fun never stops ;)

I have been rather quiet here, but hopefully I will be more active again in the future. Many things are swirling around in my head, but during the 3.0 release cycle I did not have the time to actually write down a long winded rant here. But I think that will change the next week.

Cheers,

Michael

Sage 2.11 is out!

Sage 2.11 has been released. In total 31 people did contribute patches for this release. Due to spring break and the easter holidays things moved a little slower than I had anticipated. But in the end it looks like we put together a pretty decent release. Binaries are available by now.

The high level changes in detail are:

• ATLAS: Michael Abshoff and Burcin Erocal upgraded ATLAS to the 3.8.1 release. In addition tuning info for 32 bit Prescott CPUs as well as Powerbook G4s under Linux was added.
• zn_poly: David Harvey's zn_poly library is now a standard package for Sage. zn_poly is a new C library for polynomial arithmetic in (Z/nZ)[x] where 3≤n≤ULONG_MAX (i.e. any machine-word-sized modulus). The main benefit is speed. The library is used so far only to compute the zeta function for hyperelliptic curves.
• Small roots method for polynomials mod N (N composite): Martin Albrecht implemented Coppersmith's method for finding small roots of univariate polynomials modulo N where N is composite.
• Generic Multivariate Polynomial Arithmetic: Joel Mohler improved the efficiency of the generic multivariate polynomial arithmetic in Sage by roughly a factor of ten.
• k-Schur Functions and Non-symmetric Macdonald Polynomials: Mike Hansen: k-Schur functions s^(k)_\lambda are a relatively new family of symmetric functions which play a role in Z[h1,...,hk] as the Schur functions s_\lambda do in \Lambda. The k-Schur functions, amongst other things, provide a natural basis for the quantum cohomology of the Grassmannian. The k-Schur functions can be used like any other symmetric functions and are created with kSchurFunctions. Non-symmetric Macdonald polynomials in type A can now be accessed in Sage. The polynomials are computed from the main theorem in "A Combinatorial Formula for the Non-symmetric Macdonald Polynomials" by Haglun, Haiman, and Loehr.
• Marshall Hampton did upgrade gfan as well as the optional phcpack spkgs and their interfaces. He also increased doctest coverage to 100% for both interfaces.
• Improved capabilities for solving matrix equations: William Stein implemented code so that one can now solve matrix equations AX=B and XA=B whenever a solution exists. In particular, solving linear equations now works even if A is singular or nonsquare.
• Generators for congruence subgroups: Robert Miller implemented an algorithm for very quickly computing generators for congruence subgroups \Gamma_0(N), \Gamma_1(N), and \Gamma_H(N).
• Various other people fixed a number of bugs and did improve other bits of Sage.
  

Next up is the 3.0 release. We are shooting for a two week release cycle, but we will see how things go :)

Cheers,

Michael

Sage vs. GSoC 2008

Oh well, Sage didn't make it into the GSoC 2008. Looking at the selected projects one can see that mathematical Open Source software is underrepresented. Only the R project (which is technically a statistics software package and not mathematics - some people insist on that distinction) made it in out of many mathematical software projects who applied. In recent years various projects made it into the GSoC via other orgs, i.e. sympy via multiple orgs and at least one Axiom related project via LispNYC.org.

So why didn't Sage make it in? It is hard to tell since there is little transparency during the selection of the mentor organizations. Various people put up theories at Slashdot. Specifically pongo000 commented after hanging out a couple hours in #gsoc when various mentor orgs showed up and inquired why the hadn't been selected. pongo000's observation was that the following issues did play a large role:

• the size of the organization
• whether an org participated in years past
• the quality of the ideas list

So how does Sage fit the bill?

• the size of the organization: not huge, but certainly not at the low end of the spectrum. Interestingly enough OpenOffice.org didn't make it as a mentor org this year. Somebody did show up in #gsoc and inquired why that had happened and was informed that they allegedly didn't apply.
  
• whether an org participated in years past: we applied twice and got twice rejected.
  
• the quality of the ideas list: Some Sage developers did hang out in #gsoc and did talk to lh and ask questions about the application and list of ideas Sage had submitted. One aspect was the quality and accesibility of our idea list. Apparently there are people out there who are not familar with the more abstract aspects of mathematics :) and who do need a little more info about the projects, i.e. it isn't obvious to many people what commutative alegbra consists of. We had initially discussed at Sage Days 8 what kind of application we wanted (new blood vs. tried and true contributors) and come down pretty much in favor of tried and true contributors. While we had at least two projects on the list that didn't require mathematical expertise (the notebook and Cython) the other projects were very specifically tailored for the people who wanted to do them and do contribute to Sage in that area. I won't go into details here, but if we do participate in another GSoC we need to do have more CS that mathematical projects - at least that is worked for the R's project idea page. Another highly recommended feature is a template system that list requirements and difficulty of a given projcet. But in mathematics that is highly subjective, i.e. "Drawing Graphs on Surfaces with Genus greater than 0", is not something the average non-math student will be familiar with or get up to speed in three months. Emily Kirkman on the other hand has been working in Graph theory for a while, so while I am sure it is far from trivial it would have been a nice project. Other projects like "Combinatorial Species/Decomposable Objects" or "Free abelian groups and integer lattices" aren't exactly know to the average College graduate either.
  

So in the end while we aren't happy with the decision of Google it is up to Google in the first place where they spend their money. And it isn't game over for Sage yet and the GSoC 2008. We have various project suggestions that we are trying to get accepted via other mentor orgs like the PSF.

And while I cannot go into details right now I can tell you that this might not be the end of the road, so stay tuned for further announcements.

Cheers,

Michael

Sage 2.10.4 coming up

We are very close to a Sage 2.10.4 release. I didn't announce the 2.10.3 release here, but Marshall Hampton did. My reduced blogging activities were are mix of procrastination, the need for some time away from the computer and Sage Days 8 in Austin, TX. Sage Days 8 was very nice and I had a lot of fun in Austin, but more about that in a later post.

So I am being lazy now and I will more or less copy and paste the announcement from the release notes:

• Memory leaks exposed by modular symbols: Michael Abshoff, Martin Albrecht, Burcin Erocal, Willem Jan Palenstijn, Clement Pernet, William Stein: memory leaks exposed by modular symbols functionality. This ticket is a composite of numerous other memleak fixes merged over a 7 month period. Modular forms are an excellent way to expose memory leaks in pretty much every algebraic component of Sage and all known issues there are now finally fixed.
• SQLAlchemy and DSage: We merged SetupTools and SQLAlchemy into Sage as standard packages. SQLAlchemy is now used as in DSage replacing hand written code with much more efficient classes from SQLAlchemy. SetupTools is required to install SQLAlchemy, but is also useful for a number of optional spkgs like MayaVI and packages from the Enthought Tools. Yi Qiang improved DSage making it more robust and finally adding the documentation to the standard Sage manual.
  
• Graph theory: chromatic polynomial: An algorithm originally written in C by Gordon Royle has been adapted by Robert Miller to replace the old slow method. This algorithm uses a cut and merge algorithm to recursively compute the chromatic polynomial, and is written in Cython.
• Documentation: Many doctest patches written during Doc Day 2 were merged. In addition many people kept up the good work after Doc Day 2 was over and have been submitting patches to increase coverage. We did exceed the target for the release by 0.6% reaching 47.6%.
• Symmetric function updates: Mike Hansen, reviewed by Franco Saliola: Sage 2.10.4 adds support for Macdonald polynomials, LLT polynomials, and Jack polynomials as well as a whole class of user-defined symmetric functions which can characterized by orthogonality and triangularity conditions. Support for working with ribbon tableaux was also added as part of these updates. In addtition, many doctests were added and subtle bugs fixed.
• Notebook Updates Tom Boothby, William Stein and Timothy Clemans: Fixed a bunch of new and old issue that improve the usability of the notebook. Among those are a working trash, fixes to the polling infrastructure, saving the content of unevaluated cells, URL issues to work around problems introduced by restrictive firewalls and small improvements to the interact command.
• Parallel Doctesting: Gary Furnish reviewed by Michael Abshoff: "sage -tp" has been introduced as an experimental multithreaded doctester. The first parameter is the number of threads, and the second parameter is the folder to doctest. Thus "sage -tp 4 devel/sage/sage" tests everything with four threads running. Additional options like "-long" or valgrind options like "-memcheck" do work. The code base is still young and needs more testing. The eventual goal will be to replace the current doctesting infrastructure with this code base.

Sage 2.10.4.final has been released and is currently being build tested on our compiler farm. Once it has passed tests and no blocker issue shows up in testing we will release.

Some times I am right, often I am wrong

Oh well, Ondrej Certik offered a debugging challenge for a problem in Sage he discovered. The problem was that he hit a timeout when starting up a binary copy of Sage on his shiny, new and fast Intel Quad Code box. After some debugging he discovered that Debian's XFS default mount options cause the slowdown, all the details can be found in this blog entry from Ondej's blog. You should go over to his blog and check out the quite interesting details since he took the time to track the issue down.

The challenge boiled down to the whether the timeout bug was in Sage or XFS. The loser would have to publicly claim that he is lame. Since the challenge was issued and resolved while I was asleep I did contribute little to the end result and let's just say that I am not lame. By the process of elimination you can guess the outcame. But I have been wrong about bugs in Sage and elsewhere in computing in general many times before and while I teased Ondrej a little bit about it in IRC it was all good fun. It just shows that Sage is much, much more than a pure technical project and has a social and community side, too.

Cheers,

Michael

Sage 2.10.2 has been released

Sage 2.10.2 has been release roughly three weeks after 2.10.1. The delay is mostly due to Sage Days 7, which took the attention away from most developers and also caused a number of people to come down with nasty colds (me included). Highlights from the release (shamelessly stolen from the announcement):

• John Voight's fast new code for enumeration of totally real fields is now included.
• David Roe's code for unramified and Eisenstein extensions of Qp and Zp is now included.
  
• Clement Pernet, Burcin Erocal and William Stein have implemented an optimized p-adic/modular algorithm for computing Hermite normal forms of matrices over the integers. For random square nonsingular matrices with small entries it is similar to Magma in speed, and vastly faster than the implementations in Gap, NTL, and PARI. For matrices with large entries (e.g., 16 bits or more), it is faster than anything else in the world. For nonsquare matrices it is also reasonably good, though more optimization is needed since Magma is much better in some cases. We also implemented related code for computing determinants over QQ and ZZ, which is again the fastest in the world especially when the matrix entries are large. The main reasons for the speed of our implementation are (1) IML is fast, and (2) we found some tricks that are not in the literature.
• Tim Abbott and Michael Abshoff worked on the Debianization of the build process. Due to a lot of work done by Project Athena at MIT Tim Abbott contributed many build scripts for chroot environments. He also contributed build scripts for nearly all of the SPKGs not yet in Debian. Michael Abshoff did set up a test build server and while it has been shut down for now the Sage project will set up another 64 bit build server in the near future top provide Debian packages for a wide variety of Debian based distributions.

Binaries for 2.10.2 have been uploaded to sagemath.org and are available at various mirrors - even though some might still have to catch up over the next couple hours.

Sage 2.10.3 is coming up and we might do a real quick bug fix only release since Sage Day 8 at Enthought is coming up.

Cheers,

Michael

A Late Sage Days 7 report

Oh well, Sage Days 7 has been over for quite a while and I did procrastinate on this blog post for way too long.

The event was hosted at IPAM at UCLA and it was a lot of fun meeting a whole new set of faces only known from IRC and email in person and also meeting a large set of people I met at previous Sage Days again. I was one of the few Europeans there (and probably the only one who came directly from Europe) and the flights sucked. For some reason I was routed via Las Vegas and the airline managed to lose my luggage on the last leg of my flight itinerary, a 40 minute flight from Las Vegas to LAX. The travel from LAX to Westwood was very pleasant, but finding the UCLA guest house took some more time than I had thought. In the end I finally caught some sleep and due to my charger and backup laptop resting safely in my luggage I didn't respond to email too much for the first 48 hours.

After my luggage showed finally up roughly two days later things got better, but I never caught up with my email until I made it home. I mostly worked on the Debianization of Sage with Tim Abbott, merged a bunch of tickets and spend a lot of time on discussions and bug fixing. Overall it was a very productive Sage Days for me.

I ended up winning the "Guess that spkg" and "Guess that ticket" competitions and I guess everybody did expect me to win since it would have been a major upset if somebody else beat me there. Let's see if we are having another contest at Sage Days 8 in Austin at the end of the month.

Flying back to Europe was alright, even though I didn't sleep the night before and dozed on the plane for a couple of hours. On the way back I visited rpw in Darmstadt, but at that point I was pretty useless from lack of sleep. I also ended up with a rather severe cold which also attacked other participants of SD7, so that I spend the two days after I got home mostly sleeping and away from the computer.

Cheers,

Michael

Bug Day 10

For those who hadn't heard the news: Bug Day 10 will start in a couple hours. Basis will the Sage 2.10.2.alpha0, so if you want to participate you should try to compile it until then.

It has been a while since I blogged, mostly due to Sage Days 7 and a cold that followed, but more about that later. I plan to blog about various Sage related issues in the near future, so things should be more lively again.

Cheers,

Michael

Sage 2.10.1 release plan and an Outlook for 2.10.2

After various release candidates, i.e. Sage 2.10.1.rc0-rc2 we are getting finally close to the final release. The release cycle has been longer than usual since we still have a number of blockers to resolve, i.e. the ATLAS tuning issue as well as fallout from the GNUTLS update which causes segfaults when starting the notebook. It will be the release for Sage Days 7, but we might still push another bug fix only release if it turns out that we have some glaring, must fix bugs in Sage 2.10.1. Sage 2.10.2 (or .3) will probably merged David Roe's unramified and eisenstein extensions of Qp and Zp as well as John Voight's code for enumerating totally real fields with improvements by Craig Citro. We also expect more fixes for the 64 bit OSX port as well as for the Solaris port, but it is uncertain if either one of them will become officially stable in that release.

Cheers,

Michael

Worst SPKG ever!

Well, in case you don't get the Simpsons reference about Comic Book Guy's catch phrase you might want to read those two links. But to get back to the point of this post: I spend about 6 hours or so on ticket #1852. It looked very harmless, i.e. just enable ATLAS during the configure phase and fix some other odd issues, but it turned out to be the worst spkg I ever had to fix. No need to name names here since the subject of this post is mostly tongue in cheek by now. While I did work on the spkg I did feel differently, but those things had a tendency to fade into the background once I have moved on to other things.

Do I blame the person who initially did create the spkg in question? Not really, since we are all busy people and I also had to read the R build manual to fix some of the issues I saw. It was just surprising that this particular spkg was quite bad compared to all the other ones, i.e. we installed about 60 mb unneeded object files somewhere in SAGE_LOCAL and those do add up for the final download size. I am just glad it is done and now I could start bitching about gsl 1.10, see #1623. Since that one is done, too, now it is on to #1631 and then who knows what. So, sooner or later there will be another "Worst SPKG ever!" and this one will be a distant memory ;)

Cheers,

Michael

Sage 2.10.1 Progress

Progress on Sage 2.10.1 has been good so far. We closed 71 tickets as off alpha2 and have about another 30 patches or so under review. The initial load of patches were merged during Bug Day 9, which turned out to be an excellent debug session. The wiki page is somewhat misleading since the IRC log isn't up yet and many people who participated did not add themselves to the wiki page.

But there is some trouble in paradise since the upgrade of GNUTLS and its related components broke the notebook in secure mode. This lead to a revisit in #sage-devel about whether linking a GPLed program like Sage against OpenSSL is legal or not. While opinion diverged in this it remains a fact that we should fix the GNUTLS issue (or is it in Twisted? I am not sure yet) since we can't ship OpenSSL libraries with the binary and the reliance on a properly working system OpenSLL in the past had lead to plenty of trouble in the past.

Besides the above issue what else can we expect of 2.10.1? Better ATLAS build times on Pentium Ms and 32 bit Athlons (not merged yet as of alpha1). a lot of work toward building a 64 bit OSX version of Sage and maybe even a universal binary for OSX (32 bit ppc and x86), so stay tuned.

Cheers,

Michael

Sage 2.10 released!

Sage 2.10 is out. It wasn't planned this way since we had originally shot for some release early next week, ideally Monday. So what happened? 2.10.alpha4 build perfectly on all of our test platforms and the results from the patches created during Doc Day 1 would potentially destabilize the build. All was set in motion to release at this point - after all we had gone nearly twelve days since the last release, which is longer than the usual time between releases - but the release was held up by some tricky, long standing but extremely hard to reproduce bug in the Maxima interface of Sage triggered by some last minute merge of an essential fix. William finally tracked it down and fixed it, which delayed the release about 24 hours. I used that time to catch up on sleep and to prepare myself for Bug Day 9.

The main changes for Sage 2.10 are:

• Python is now built with ucs4
  
• FLINT was updated to the 1.0.5 release
  
• Many bug fixes and also a couple of significant memory leak fixes
  
• Integrate a fix to the MPFR library so we no longer smash the stack with high precisions. This cause a performance regression when multiplying
  
• Fix a long standing, hard to hit bug in the Maxima interface
• When launching external executables LD_LIBRARY_PATH is restored for external binaries, i.e. those which do not reside in SAGE_LOCAL/bin. Before the patch many external executables failed due to library conflicts. This was reported many times on Debian based systems

What does the future hold? Sage 2.10.1 is already getting merges on the way to alpha0. So stay tuned :)

Cheers,

Michael

Sage 2.10.alpha4, Doc Day 1 & Bug Day 9

We are making progress toward Sage 2.10. With 2.10.alpha4 we have closed 72 tickets, but we are working very hard to make this release an excellent one. There are two events coming up this week:

• Doc Day 1 will be held on Thursday the 17th, 2008, i.e. tomorrow, starting at 9am PST. We will write doctests and documentation. Since it is the first of its kind we will play it by ear and see how it goes.
• Bug Day 9 will be held on Saturday the 19th, 2008, starting at 9am PST. It will be business as usual and the plan is to put the finishing touches in the 2.10 release. From experience we will sort out the fallout from all the merges we do at the bug day on Sunday and release either Sunday or Monday night depending on how many issues and last minute problems creep up.

It does look like we will miss most of the goals we set ourselves for 2.10, but we made progress on various fronts and will just shift any issue left open to bug fix releases 2.10.x which are planned weekly until at least 2.10.3

Cheers,

Michael

Sage 2.10 vs. MacIntel 64 bit

Well, after talking about for a couple weeks [or is it months by now?] I finally bit the bullet and stated porting the current Sage 2.10.alpha2 to 64 bit MacIntel. I started somewhere around 7 pm locally and now, roughly 12 hours later I struck out. For a first attempt it wasn't too bad. My build notes are a little rough, but surprisingly most issues are fiddling with CFLAGS, CXXFLAGS or CPPFLAGS. That wasn't too much of a surprise since most of the code already runs on MacOSX, but in the end I failed with various spkgs:

• numpy: odd issues with distutils, caused by the fact that I had to be tricky with python and add some flags after the fact for distutils. Consequently scipy and some other spkgs couldn't be build. Josh Kantor is investigating, so hopefully once I wake up it might be solved.
  
• PolyBoRi: SCons issues, mostly ENV related. Here I ran out of time
• twisted: It seems to depend on MacOSX specific python extensions, but I didn't investigate, but should be fixable.
  

But after working around various issues I got all but 5 extension modules to work (four times numpy related and the PolyBoRi extension), but in the end Sage did not start up to do the ceremonial 1+1, but failed in libSingular with:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000018
0x00000001045f0bfb in omInsertBinPage [inlined] () at om_Alloc.c:109
109 after->next->prev = page;
(gdb) bt
Function omInsertBinPage was inlined into function omAllocBinFromFullPage at line 145.
#1 0x00000001045f0bfb in omAllocBinFromFullPage (bin=0x10495aef0) at om_Alloc.c:145
#2 0x00000001045eebde in __omDebugAlloc (size_bin=0x10495aef0, flags=, track=, f=, l=) at om_Alloc.c:145

Since I need to catch some sleep now for a meeting in roughly seven hours I will sign off now.

The plan for the next 24 hours: Merge the easy changes back into 2.10.alpha3 or alpha4, update a bunch of spkgs and get Sage 2.10 toward release candidate status. There is still much to do for 2.10 and we had only planned to officially support MacIntel 64 bit with 2.10.1. I am not sure if we will hit that target, but it isn't looking too bad. Either way, William is excited and I guess there are quite a large number of people out there who would really like to use it. So feel free to help out.

Cheers,

Michael

The ongoing struggle for Sage 2.10

So far I have released two alphas for Sage 2.10, which can be found in the usual place, i.e. here. Note that there is already an alpha2 directory which contains all so far merged patches and updated spkgs for the next milestone. That way you can easily catch up to my current state. It also enables you to upgrade by downloading the spkgs and dropping them in and downloading the patches and bundles and applying them. Sometimes the order matters, so keeping an eye on the trac timeline helps.

I am not very happy with the progress on 2.10 so far because I ended up auditing the whole test suite with valgrind again and ended up finding three issues that ought to be fixed sooner rather than later. One of them (#1739) has already been fixed, the other two I hadn't even been entered into trac yet. I also build one of the alphas on an Itanium with the tool chain from hell in order to track down a PolyBoRi issue, but I struck out on that one after about a day and a half.

After all this frustration and hair pulling I am finally back to merging and fixing issues I can get done in a reasonable amount of time. But while I do that I will rebuild all of Sage 2.10.alpha2 (once it is out in a little while) with a ucs4 enabled python. The complete rebuild is needed since all python components need to be recompiled and that is only possible if we artificially bump all version numbers of the python related spkgs. We need to do that for the upgrade path anyway, but to experiment a complete rebuild is the easier way. If a ucs4 enabled python works and we have upgraded all the components we plan for Sage 2.10 I will end up scanning all the packages and artificially bump all non-upgraded python dependent spkgs.

You should have gotten the impression by now that 2.10 is more than 3 days away from being released. My current guesstimate is about a week from now, which taken into consideration that many people were at the AMS meeting and the holiday fallout for people to catch up on isn't too bad for a new major release with that many changes compared to 2.9.

Cheers,

Michael

Sage 2.9.3 released, Sage 2.10 release cycle opened

Well, Sage 2.9.3 was actually release two days ago, but since there was never an official release announcement I figured it might be worth mentioning it here. We only merged three tickets, but one of those was a long standing NTL leak which I complained about previously. The issue was fixed by Willem Jan Palenstijn. It turned out to be missing __dealloc__ methods in two of the NTL wrappers classes. I never thought that the memory leak would be caused by missing deallocation in general. I foolishly assumed that since the people who wrote the code used __dealloc__ in all the other classes they would do so in those classes, too. Interestingly enough this leak wasn't caught by doctesting the NTL wrapper, but by padics.py. But I am glad Willem caught it. So why all this focus on memory leaks, even the small ones? There are various reasons:

• Code that leaks makes you look unprofessional. While there may be occasional small leaks (the python interpreter does leak a couple bytes over the run of most Sage session) something as massive as the NTL wrapper leak forces you to quit and restart Sage. And that is totally unacceptable.
• Even small memory leaks add up to large leaks if you hit them often enough. In the past we had some leaks in linear algebra which caused about a total leak of one gigabyte an hour with certain code. We fixed those in the linear algebra code and afterwards the code still leaked about one gigabyte a day. We investigated and as it turned out that one gigabyte a day leak was caused by a single 8 byte leak in a code path he hit very often in that computation. The discovery was live during my Sage Days 5 presentation when I did analyze some of Ifti's example code at the end of the talk. A video of that ought to exist somewhere on Google video.
  
• Many programs run for a long time and usually need all the available physical memory. Once you hit swap because your working set is too large performance goes down the drain. In border line cases not having memory leaks may make a difference. It is also quite bad if you run a lot of independent computations back to back and have to restart Sage periodically because of memory leaks. We also had a bunch of reports about this issue in the past.
• And now the most important reason for some of us: Well, you may say, I have 16, 32 or 64 GB of Ram and why would I care about the little leaks? The answer is clear: performance. If python's garbage collector has to keep track of a billion leaked eight byte segments that cannot be good. In some case the performance difference for long running code was in excess of ten percent by fixing some small eight byte memory leak.
• EDIT: I forgot to mention one important experience. There was a memory leak when computing a cusp and to our amazement the fix (freeing one mpz in a premature return path) broke a doctest. Upon investigation it turned out that the result prior to the memory leak fix was incorrect. I must admit that this was the first and only time I ever had a memory leak affect the correctness of a program. To this day I still cannot understand why it happened.
  

I am also release manager for the 2.10 release of Sage and 2.10.alpha0 was uploaded earlier today. The goals for the 2.10 release are ambitious:

• Update a lot of spkgs to the current release
• Solaris 10 support in 32 bit mode on Opteron/x86
• FreeBSD support out of the box

The currently planned release date is about four days from now, but the way it looks right now that might get pushed back a little. Otherwise we might postpone some of the features to 2.10.1 - life goes on.

Cheers,

Michael

Sage 2.9.2 Released

Sage 2.9.2 has been released. We merged relatively few tickets this round because this release is targeted for the AMS meeting in San Diego that starts this weekend. The big change from 2.9.1.1 is the much improved 3D support via jmol. Robert Bradshaw and William Stein spend a lot of work on this right up to the release. Let's hope everything is working out with that code. Additionally we fixed some long standing build and relocation issues. Please report any problems as usual.

Sage 2.10 will happen in about a week and we plan to update a lot of spkgs as well as add full support for FreeBSD and limited support for Solaris. We will see how that works out, the release date certainly isn't written in stone and depending on how the merge goes we might push it back. If you look at the tracker we have about 272 ticket open against 2.10. I am hoping that the coding spring this weekend will help reduce this number substantially. There are also a bunch of patches ready to go in that we did not merge since we were very risk averse and also had a hard deadline which we cannot move.

Cheers,

Michael

My first year with Sage and an outlook for 2008

To my own surprise I have been involved with Sage for a little over a year now. I thought initially that my first message to sage-devel was in March or April of 2007, but by checking today I saw that it was in December 2006. The first couple months were mostly about fixes to the Cygwin build and general discussion. Only later I started to work on the build system and spkgs portability and eventually I started to manage releases toward the end of the year. I am also maintaining various ports like the Linux/PPC port and am actively working on the Solaris, FreeBSD ports and Cygwin re-port.

So what will the future bring for Sage in 2008? While predictions are notoriously hard and often hilariously wrong I will venture some guesses. I will limit myself to the areas I plan to work on:

• Full Solaris support within three months: We are very close on this and in the next week I will hopefully have time to push all the changes I have accumulated into the official spkgs. There are still some known bugs that seem to be triggered only on Solaris (some libSingular issues), but I am confident we will fix those and by we in this case I mean hopefully Martin Albrecht.
• Full FreeBSD support in one month: This one seems like a no brainer. The port went fairly smooth and aside from the odd issue with Singular and C++ headers it all seems easily mergable. Note that even now you can run the Linux binary on FreeBSD via the Linux emulation.
• Full resurrection of the Cygwin port: I will not give any estimate on this, but I don't think it will take too long to get it to compile. Passing doctests will be another issue.
  
• Massive progress toward a native MSVC port: This one will be the big question mark. There is certainly a lot of potential interest, but so far few people have committed to do any of the work. There is also no funding in sight that would help to get the massive undertaking on the way. But I still maintain that this is certainly doable.
• Sage will be part of distributions: It looks like Pardus might be the first to integrate Sage, but now it looks like Debian might still beat it. There was a lot if activity today in the Google group debian-sage, so I am bullish on that, especially since I will be part of the effort.

So, I will be curious to look back toward the end of 2008 to see what did happen and where I was totally off.

Cheers,

Michael

Hunting Memory Leaks in the NTL Wrapper

Sage's NTL wrapper is more or less famous for leaking memory. During Bug Day 5 at the Clay in Boston I actually did find the last known leak in a live session. The leak was caused by a str() method. As it turns out that mistake had to be corrected 3 or 4 times in the NTL wrapper and a couple times outside of it. Since then the NTL wrapper has been rewritten to be faster and more feature complete, but during the rewrite new memory leaks have been introduced. While I found those before the merge they seemed minor and insignificant. Unfortunately those leaks turned out to cause massive problems in the padics code.

I do occasionally run the whole Sage testsuite under valgrind's memcheck and it takes about a day to finish when skipping memcheck on all external executables. In the 2.9.0->2.9.1 release cycle I fixed 3 memory leaks, one of them in the graph code when converting arbitrary precision numbers to string representation in base 2. Since that function is called quite often when reading or writing adjacency matrices it can be quite a pain. The other two were relatively minor.

But the reason to run memcheck on the test suite was to find and fix all known memory leak issues before the 2.9.2 release. But I have been unable to fix two issues:

• LinBox leaks via Givaro in certain situations, for example when computing characteristic polynomials over certain fields. Clement Pernet and I did investigate the issue during Sage Days 6, but so far we haven't found a solution. Since it is relatively minor, especially compared to the memory leaks we already fixed in LinBox it has been somewhat on the back burner.
  
• The padic doctests leak tremendous amounts of memory. One doctest leaks more than 50 MBytes of memory. That is clearly unacceptable and would make long term computation involving padics nearly impossible. But so far I have been unable to figure out why the code, especially the __pow__ methods are leaking. I attacked the problem with memcheck as well as omega and omega points to some auto-generated Cython code, which is always a bad sign. I guess I need to bother Robert Bradshaw about this :)

So, after having attacked the problem repeatedly with zero progress I have given up on the 2.9.2 release. Maybe with a little distance I will finally figure out what is wrong.

Oh well, some times it just takes time to figure it all out. Then one has to wonder why it took so long to see the obvious.

Cheers,

Michael