Adventures in Open Source Math

dsage.setup() broken in Sage 3.0.2

2008-05-27T19:52:00.000-07:00

Hello folks,

unfortunately a severe DSage bug slipped by since it was neither caught by doctesting nor the DSage unit tests. We are tracking the issue at #3311 and already have a patch that fixes the issue here. While poking around two more DSage bugs [#3312 and #3314] come to light, but those two are unreviewed so far. We will accelerate the 3.0.3 release because of the above bugs, but in case you are currently using 3.0.2 and want to use DSage please apply at least the fix from #3311 I linked above. At the ticket are more fixes, but only apply the patch linked since there are dependencies between the other patches.

Cheers,

Michael

Sage 3.0.2 released

2008-05-25T19:42:00.000-07:00

Hi,

Sage 3.0.2 has been released. This is another bug fix release with some new features while we are waiting on the coercion rewrite to finish. Interesting new features include (in no particular order):

Self-orthogonal Binary Codes (Robert Miller)
Notebook Improvements (William Stein, Tom Boothby)
Portability of Sage to 64 bit OSX and Cygwin (Michael Abshoff, William Stein)
Posets and Semi-Lattices (Peter Jipsen and Franco Saliola)
Frobby for monomial ideals (Bjarke Hammersholt Roune)

As usual check the Sage 3.0.2 release tour in the wiki for details. Some people still have to fill in the details, but I send those people emails to remind them ;)

Sources were posted yesterday and today we posted 15 binaries for various platforms in the usual place. The binaries have been mirrored out to some websites and the other mirros should catch up in the next twentyfour hours. Not all binaries are there yet, i.e. the VMWare binary is still missing, but should show up by Monday at the latest.

Cheers,

Michael

Sage 3.0.2.rc3 released

2008-05-23T19:47:00.000-07:00

Hello folks,

here we go with 3.0.2.rc3. It is basically a bunch of bug fixes for rlm's codes code that had a couple small issues left ;) In addition there is one pbuild issue fix that Jaap encountered and that in the past was also hit by David Joyner. You can download the Sage 3.0.2.rc3 sources here.

Unless something goes horribly wrong this will be identical to the final 3.0.2 release. If you added some feature that should be in the Sage release tour please add them to Sage 3.0.2 release tour wiki page. Right now that list consists of

Franco Saliola and Peter Jipsen's posets and semi-lattive patch
Robert Miller's self-orthogonal binary codes
Bjarke Hammersholt Roune's Frobby is now an optional spkg

But feel free to add items. As usual please test and report any issue you hit.

Cheers,

Michael

Sage 3.0.2.rc0 released

2008-05-23T01:45:00.000-07:00

Hello folks,

this is 3.0.2.rc0, the likely final release of the 3.0.2 series and hopefully next to identical to the final 3.0.2. What is new?

Franco Saliola and Peter Jipsen's posets and semi-lattive patch
Robert Miller's self-orthogonal binary codes
Bjarke Hammersholt Roune's Frobby is now an optional spkg
Pbuild should now pass the doctests since #3097 has been fixed

In addition there were the usual bug fixes. Please build and doctest as usual and report any issues you see.

Sources and a sage.math-only binary in the usual places:

Cheers,

Michael

Merged in rc0:

#1762: Robert Miller, Michael Abshoff: Create optional graphviz package
#2121: Robert Miller: move libecm wrapper from interfaces to libs
#2519: Franco Saliola, Peter Jipsen: Add support for posets, semi-lattices, etc. to Sage
#3018: Bjarke Hammersholt Roune: Integrate Frobby into Sage
#3097: Gary Furnish, Michael Abshoff: pbuild: make sure the files from setup.py's scripts section are copied
#3104: William Stein: pbori.pyx: Make some doctest long since it uses a lot of RAM
#3112: Robert Miller: Generate self-orthogonal binary codes
#3148: Francis Clarke: improved orthogonal functions
#3218: Michael Abshoff: fix 64 bit OSX build support for mercurial
#3219: William Stein: upgrade to gmp-4.2.2 while we wait for MPIR
#3242: Robert Miller: Fix little bug in G.relabel() for G a graph
#3245: Mike Hansen: provide coefficient and coefficients methods for symbolic expressions
#3257: Gary Furnish: Pbuild ignores gcc specific default settings
#3263: Craig Citro: typo in lseries_ell.py
#3266: William Stein: Sage 3.0.2.alpha1: doctest failure in sage/server/simple/twist.py
#3267: Michael Abshoff: Sage 3.0.2.alpha1: doctest failure in sage/server/support.py
#3269: Jason Bandlow: Improve documentation for combinat/dyck_word.py
#3270: Robert Miller: trivial 100x speedup in coding theory
#3272: Craig Citro: Bug in sparse polynomials over finite fields
#3273: Robert Bradshaw: extend isqrt to work for Python int's in addition to Sage integers and objects with an isqrt method
#3274: Michael Abshoff: OSX: delete libpng*.la since we also nuke libpng*.dylib
#3275: Craig Citro: Make SL2Z distinct

Sage 3.0.2.alpha1 released

2008-05-19T18:45:00.000-07:00

Hello folks,

the 3.0.2 release cycle so far has been a little slower than usual and there are a couple contributing factors. While some of us are waiting on the coercion rewrite to progress, others had obligations with school and the end of the semester seems to have kept a lot of people busy. 3.0.2.alpha1 contains *a lot* of porting work for Cygwin and Mac OSX 64 bit. We also merged Dan's new Weyl Character implementation and the usual set of bug fixes.

Since it has been nearly two weeks it is now time to get the release out of the door. So the plan now is to get an rc0 out the door tomorrow and then do a release by Tuesday or Wednesday. We will hopefully merge some more porting fixes, i.e. there are about ten more OSX 64 bit tickets ready to be reviewed. Other than that we have 70+ tickets with patches, so there is plenty to review. Once I catch some sleep I will do another post with details.

Sources and Binaries are in the usual place:

Right now there are two notebook related doctest failures which do not have tickets yet. Please build, test and report issues you see.

Cheers,

Michael

Details for closed tickets in alpha1:

#406: William Stein: notebook -- make tab completion not stick gap. when using the notebook in gap mode
#637: William Stein: notebook improvement -- upload allow txt worksheets.
#1733: William Stein: notebook bug -- %foo (or anything else) in a cell by itself (with nothing else in the cell) does not give an error but it *should*
#1864: William Stein: simple notebook bug -- typing ? in a comment yields introspection but shouldn't
#1892: William Stein: notebook -- uploading a data file should give some help about the DATA variable
#2359: William Stein: notebook -- make it so when you send a kill signal to the notebook server it saves state
#2636: William Stein: notebook -- changing a cell without evaluate should put the red line back to the left
#2860: William Stein: easy-to-fix bug in html.py
#2884: William Stein, Tom Boothby: notebook -- bug; @interact cell eval doesn't clear out the old html output
#2992: William Stein: notebook -- help(foo) in the notebook should not word wrap
#3024: William Stein: notebook -- parses tracebacks in the output of docstrings of help command
#3050: Timothy Clemans: notebook -- add a "remember me" checkbox to the login page
#3051: Dan Bump, Mike Hansen: Implement Weyl Characters
#3053: William Stein: notebook -- new cell_resize doesn't respect %hide at the beginning of a cell
#3069: William Stein: notebook -- typeset checkbox doesn't work after save/reload
#3137: Yi Qiang: view command in misc/latex.py -- fix to not hardcode xdvi command
#3153: Carl Witty: make finite_field_ntl_gf2e use randstate framework
#3155: Timothy Clemans: notebook postdata and behaviour of archive, delete and stop buttons
#3160: Emily Kirkman, Robert Miller: change is_planar for graphs to return bool
#3161: Michael Abshoff: sdist: #3046 seems to have broken sage-banner
#3170: Michael Abshoff: add 64 bit OSX build support to readline
#3171: Michael Abshoff: add 64 bit OSX build support to termcap
#3172: Michael Abshoff: add 64 bit OSX build support to prereq and bzip
#3176: Michael Abshoff: add 64 bit OSX build support to sqlite
#3177: Michael Abshoff: fix 64 bit OSX build support for python
#3178: Michael Abshoff: add 64 bit OSX build support to freetype
#3179: Michael Abshoff: more 64 bit OSX libpng fixes
#3181: Michael Abshoff: add 64 bit OSX build support to iml
#3182: Michael Abshoff: improve 64 bit OSX build support for givaro
#3183: Michael Abshoff: add 64 bit OSX build support to linbox
#3186: Michael Abshoff: fix 64 bit OSX build support for numpy
#3187: Michael Abshoff: fix 64 bit OSX build support for matplotlib
#3188: Michael Abshoff: add 64 bit OSX build support to mpfi
#3189: Michael Abshoff: add 64 bit OSX build support to pycrypto
#3190: Michael Abshoff: add 64 bit OSX build support to zodb
#3191: Michael Abshoff: add 64 bit OSX build support to quaddouble
#3192: Michael Abshoff: fix 64 bit OSX build support for python_gnutls
#3197: Michael Abshoff: fix 64 bit OSX build support for m4ri
#3198: Michael Abshoff: fix 64 bit OSX build support for ecm
#3200: Michael Abshoff: fix 64 bit OSX build support for genus2reduction
#3213: Timothy Clemans: notebook -- Account Settings page for changing password and e-mail address
#3220: William Stein: readline -- fix a couple of issues
#3222: William Stein: sqlite -- add cygwin support to sqlite
#3233: William Stein: cygwin -- make linbox work with cygwin
#3224: Michael Abshoff: add 64 bit OSX build support for lcalc
#3225: Michael Abshoff: add 64 bit OSX build support for cddlib
#3226: Michael Abshoff: add 64 bit OSX build support for gfan
#3234: William Stein: cygwin -- make numpy work with cygwin
#3235: William Stein, Michael Abshoff: cygwin -- mpfi; get it to work with Cygwin by fixing configure.ac
#3236: William Stein, Michael Abshoff: cygwin -- get quaddouble to work with cygwin
#3238: William Stein, Michael Abshoff: libfpll spkg -- update to work with cygwin
#3239: William Stein, Michael Abshoff: cygwin polybori -- add Cygwin build support for polybori
#3230: William Stein: cygwin -- new givaro spkg that works around stupidity in cygwin
#3241: William Stein, Michael Abshoff: cygwin -- new rubiks spkg that builds on cygwin
#3243: William Stein: cygwin -- get log2 to work on cygwin
#3246: William Stein: cygwin -- fix broken gsl.ode setup.py code
#3252: Yi Qiang: add kbase functionality to libsingular

First sign of 64 bit OSX 10.5 support in Sage 3.0.x

2008-05-16T16:17:00.001-07:00

Hello folks,

64 bit OSX 10.5 support has been promised for a while, but I finally managed a couple hours ago to fix a segfault issue in libSingular that made Sage segfault instantly when any multivariate polynomial ring was created. If you check out the 64 bit OSX port page in the wiki you will see that there are currently 35+ ticket listed which need to be fixed. I have patches for all of them, but the quality is uneven and a couple of those patches need to be redone. So it will take a couple more weeks until all those issues are sorted out and polished patches are merged into the official tree.

I have put together a binary [OSX 10.5 x86-64 only]. It is rather hefty at 0.5GB, but I have left a substantial number of hand build spkgs in the dmg more or less by accident. If you plan to build a 64 bit build of Sage on your OSX machine you should wait a couple more days until 3.0.2 is released which will contain the vast majority of fixes. It is even likely that there will be an official 64 bit OSX 10.5 binary, but currently there are 24 doctest failures. Some are trivial to fix, others are caused by the notebook not working due to _ctypes being broken. There are other small issues with pexpect and a couple additional small issues, but overall we are in pretty good shape. On exit OSX also complains about a couple "Non-aligned pointer being freed", but that seems to be mostly cosmetic and will be fixed soon since I got a good idea what is causing this. We will hopefully be at a point where we can call OSX in 64 bit mode fully supported soon, but there will be some more work involved before we are done.

Cheers,

Michael

Sage 3.0.2.alpha0 release

2008-05-11T16:52:00.000-07:00

Hello folks,

this is Sage 3.0.2.alpha0. What happened? It seems that people were busy and until this morning there wasn't a whole lot to merge. But I had a busy day today and finally these is something to put out. We are still mostly on bug fix only mode, so no big surprises. "sage -sdist" seems to have been broken by David Joyner's #3046, so sage-banner is emtpy [see #3161]. I fixed this in the tarball and it will be fixed in Sage 3.0.2.alpha1.

Binaries and sources in the usual place:

I also created md5sums. There are 70+ tickets with patches in trac, among them quite a number of notebook patches. So it would be very nice if people can do some serious review. Note that the notebook patches have some rather large dependency tree available in the wiki here.

Cheers,

Michael

Merged in alpha0:

#336: William Stein, Timothy Clemans: Create an option to clear all cell output
#905: Burcin Erocal, Michael Abshoff: update ipython to 0.8.2
#1230: William Stein: Quit worksheet behaviour
#1557: William Stein: notebook -- usability improvement after uploading file
#2684: Jason Grout: vertices should not default to red
#2768: Jason Grout: add comparison operators to the fast_float mechanism
#2926: Timothy Clemans: notebook -- Minimalistic change password page for notebook user
#2983: Michael Abshoff: Itanium (RHEL 5) -- singular interface problems in matrix_group.py
#3008: William Stein: first cell in notebook is undeletable
#3020: John Cremona, Martin Albrecht: Speed up Finite Fields of characteristic 2 constructors
#3026: Bjarke Hammersholt Roune: multivariate polynomial rings with no variables do not print properly
#3028: Bjarke Hammersholt Roune: Ideals in multivariate polynomial rings with no variables raise exception on comparison
#3065: Didier Deshommes: empty matrices: frobenius() throws RuntimeError
#3105: Francis Clark: new _latex_ and modified __repr__ for elements of relative number fields
#3109: William Stein: elliptic curves -- implement P.divide(n) for P a point on an elliptic curve and n an integer
#3110: Gary Furnish: fix pbuild dependency bug
#3116: Mike Hansen: 1x1 symbolic matrices don't work right
#3121: Jason Grout, William Stein: @interact grid control
#3125: Robert Miller: chromatic_polynomial incorrectly blocks control-c
#3126: Robert Bradshaw: Cython annotation has unicode errors (e.g. from the notebook)
#3129: Bjarke Hammersholt Roune: The singular interface should not claim to support polynomial rings with no variables
#3136: William Stein: the readme for osx should be changed to delete the line about inotebook()
#3138: Bjarke Roune: Singular multivariate polynomial ring has redundant _repr_ method
#3142: Martin Albrecht: MPolynomialIdeal.homogenize bugfix
#3143: Martin Albrecht: remove references to "/home/was"
#3046: David Joyner: version option returning clone branch name
#3150: Carlo Hamalainen: Memory leak in dancing_links.pyx
#3157: Gary Furnish: Executable target for pbuild
#3158: Michael Abshoff: singular-3-0-4-2-20080405.p1 requires flex
#3159: Tim Abbott, Francois Bissey, Michael Abshoff: Patch adding soname to ntl shared library

Stupid Bugs - Part 824

2008-05-09T15:32:00.000-07:00

While building Sage 3.0.1 on a pure Solaris install, i.e. no Sun FreeWare build tools in $PATH, I come across the following beauty:

bash-3.00$ which ranlib
no ranlib in /opt/sfw/bin/ /usr/local/bin/ sparc-SunOS/ /usr/bin /home/mabshoff/bin
bash-3.00$ echo $?
0

Sigh. I guess I need to fix that in spkg/base/prereq-0.3-install.

Cheers,

Michael

Sage 3.0.1 released

2008-05-05T09:17:00.001-07:00

Hi,

Sage 3.0.1 is out. Compared over 3.0.1.final there was one crucial bug fix to the Singular pexpect interface. Binaries have been build and are currently mirroring out. This is mostly a bug fix release, but there are also some new interesting features. For details check the Sage 3.0.1 release tour, which is as I type this still somewhat work in progress.

The next release is another bug fix release, i.e. Sage 3.0.2, planned about a week from now. It will likely be released with much of the work that will be done during Bug Day 12 this Saturday. Other than that we are waiting for the Coercion team to finish their rewrite which will then be merge in Sage 3.1.

Cheers,

Michael

Sage 3.0.1.rc0 release

2008-05-02T23:43:00.000-07:00

Hello,

This is the end of the 3.0.1 release cycle. The build was announced in IRC about eight hours ago, but since I took a long nap in the meantime I never posted to sage-devel ;)

Gary's pbuild has been improved and three bugs have been fixed. Please try it out again for feedback. To use pbuild

export SAGE_PBUILD=yes

before building Sage. The number of threads used during the build of the Sage library is set via SAGE_BUILD_THREADS, i.e.

export SAGE_BUILD_THREADS=8

for eight threads. Switching between pbuild and normal build is not possible in all cases, so in case of problems nuke the build directory on devel/sage.

Things should "just work" in this release [famous last words], but there are some reported oddities with pexpect behaving badly. We have so far been unable to reproduce and of those issue on a local system [Gary claims testdoc.py hangs for him on sage.math - but it doesn't for me], so if you see anything please let us know.

Sources and binaries are in the usual place. This is the end of the release cycle. A handful more patches are planned for 3.0.1.final. Anything non-blocker will be considered for 3.0.2.

Cheers,

Michael

Merged in rc0:

#2755: Andrey Novoseltsev: lattice_polytope.py update
#3060: Marshall Hampton, Michael Abshoff: update optional biopython package to 1.45 release
#3062: Timothy Clemans: implement __oct__ special method for the integers
#3070: Robert Miller: bug in SymmetricGroup(1).cayley_graph()
#3071: Gary Furnish: Using pbuild does not create site-packages sage symlink
#3072: Willem Jan Palenstijn: sage -i numeric-24.2 (and all other experimental packages) fails
#3074: Robert Bradshaw: update Cyton to the 0.9.6.14 release
#3076: Michael Abshoff: spkg-debian in extcode spkg not executable
#3077: Gary Furnish: pbuild does not return properly on failure
#3078: Willem Jan Palenstijn: sage's spkg-install doesn't return failure if build failed
#3082: William Stein: sage-3.0.1.alpha1: a twist.py doctest failure

Sage 3.0.1.alpha1 released

2008-05-01T00:22:00.000-07:00

Hello,

this release should have been out two days ago, but somehow general slowness and me spending a lot of time on various porting issues did delay this release more than it should have. Gary's pbuild should now be fully functional, but it wasn't made the default build system yet. If you run

export SAGE_PBUILD=yes

before building Sage pbuild will be used. All the various sage [-b|-ba|-br|...] should work as should -clone & friends. The number of threads used during the build is set via

export SAGE_BUILD_THREADS=8

for eight threads. Switching between pbuild and normal build is not possible in all cases, so in case of problems nuke the build directory on devel/sage. Please try out pbuild and report any trouble. We *really* want to switch to it per default soon.

I am currently seeing an odd doctest failure in

doctest failure in devel/sage/sage/server/simple/twist.py

Other than that things should "just work".

Sources and binaries are in the usual place. We are getting toward the end of the release cycle

Cheers,

Michael

Merged in 3.0.1.alpha1:

#1549: Alex Ghitza: Sage 2.9: fix optional doctests in tut.tex
#2216: Alex Ghitza: Creating an order in a number field --> infinite loop?
#2504: Alex Ghitza: number field .units() method caches proof=False result and returns it for proof=True
#2523: Craig Citro: bug in modular symbols for GammaH subgroup
#2716: Marshall Hampton: convex hulls and polyhedral functions
#2741: William Stein, Timothy Clemans: Implement mesh lines in 3d plots
#2938: Craig Citro: Fix ModularSymbols(GammaH(8,[3])).decomposition() ModularSymbols(GammaH(81, [10])).decomposition();
#3029: Tim Abbott: Move DEB_AUTO_UPDATE_DEBIAN_CONTROL out of Debian packages
#3030: Gary Furnish: Cython working directory command line option patch
#3031: Kiran Kedlaya, Craig Citro: Add zeta_function method for schemes
#3032: Dan Bump: minor docstring cleanup in crystals.py and tensor_product.py
#3034: Tim Abbott: improved cleaning code for Debian packages
#3036: Tim Abbott: SAGE_TESTDIR broken
#3037: Gary Furnish, Robert Bradshaw: update cython to 0.9.6.13-20080426
#3038: Tim Abbott: SAGE setup.py fixes for using Debian packaged polybori, zn_poly
#3039: Tim Abbott: Improve auto-generated version numbers for Debian packages
#3041: Francois Bissey, Michael Abshoff: optimization setting in LinBox.spkg is broken
#3054: Jason Grout: copying a graph doesn't copy _pos or _boundary
#3055: Jason Grout: creating subgraph does not delete _pos entries
#3057: Tom Boothby: MPolynomialRing_generic type-checks to determine commutativity
#3059: William Stein, Timothy Clemans: notebook -- rewrite notebook(...) function to *not* use SSL by default
#3061: Michael Abshoff, Max Murphy: use readlink and realpatch so that symlinking sage works
#3063: Didier Deshommes: empty matrices: norm() returns a ValueError
#3064: Didier Deshommes: empty matrices: density() function throws a ZeroDivisionError
#3066: Didier Deshommes: empty matrices: gram_schmidt() throws a NameError
#3067: Didier Deshommes: matrices: numeric_array() is missing an import

Sage Lisp Wars!

2008-04-28T04:30:00.000-07:00

Once upon a time a long, long time ago in a happy world where men were real men and wrote their own device drives in assembler - well, screw that, it is 2008. This is the story about whether Maxima, which is the only Sage component written in lisp, will stay in Sage long term or not.

To make things easier let's get some things sorted out before we get into the story:

Places:

"benchmarking CAS" thread in sci.math.symbolic [currently 40 posts]
"multivariate factoring - use maxima ?" thread in sage-devel [currently 36 posts]
"Project" thread in sage-devel [currently 75 posts]
"compiling Maxima by ECL" thread in sage-devel [currently 4 posts]
"compiling Maxima by ECL" thread on the maxima-devel mailing list
Richard Fateman's email from the above thread about ecl that Juan replied to.

This is a tale about my personal comment in [1] stating:

This is motivated by the need for performance and the fact that we want to dump any lisp based code from the core of Sage in the long term.

while taking about the long term plan for symbolics code in Sage. I am doing much of the porting of Sage to pretty much anything that I would consider worth porting to. While some people would question the value of a port to OpenBSD [I don't, but that is a different blog post I had in my head for a while without the time to write it down :)] few people would think that the port of Sage to native Windows in 32 & 64 bit mode would be bad for Sage. But porting Sage to any platform also means porting Maxima and since Maxima is written in common lisp it does require a common lisp implementation. Fortunately Maxima quite happily runs on top of cmulc, sbcl, clisp and gcl. But since Sage is a CAS where every component compiles from source and cmucl as well sbcl require another lisp to bootstrap themselves only clisp and gcl are fit as candidates for Sage. Incidentally sbcl is an amicable fork of cmucl since cmucl can only bootstrap itself off another cmucl instance and in contrast sbcl requires only a common lisp capable lisp instance to do so, but I am getting off path here.

So, there are two candidates to be used as common lisp implementations in Sage. Where is the trouble? Well, the devil is in the details. We need one and only one tarball that builds on a wide variety of systems and way back in the infancy of Sage William decided to go with clisp since it did build for him much more easily on the then supported platforms. gcl is more difficult to build and is lacking support for some platforms we will port to and on others build trouble is pretty much everywhere. See [3] for some examples. But clisp is troubled itself. It has to be compiled with gcc 4.2 and 4.3 with "-O0" due to some compiler bug in gcc. But on Solaris the last clisp release that somebody got to compile and run was 2.38 in 32 bit mode while the current release is 2.44.1. I did do much more than I thought was expected of me to resolve the issue, but in the end there was no solution (see [3] again).

All of the above has lead me to the conclusion that the lisp ecosystem is in "trouble" because its Open Source toolset is in a terrible state. The only ray of hope I have seen in the Open Source lisp ecosystem is ecl, but Maxima does not run on top of it. Way back in 2005 Michael Goffioul did some work to get Maxima to compile on top of ecl, but it seemingly didn't go anywhere. So I saw little chance for Maxima on top of ecl and since I neither have the time nor the expertise to attempt such a port my conclusion was to get rid of Maxime in the core of Sage as quickly as possible. There are various Sage developers who do think alike, so a plan was devised to replace everything use in Maxima with either other components or by code written from scratch. But then in [3] Robert Dodier told us that he had done some work on Maxima on top of ecl in January of 2008 and he was willing to attempt to finish the port since he also saw gcl's use of the Maxima on Windows installer troublesome. Progress in this direction has been made (see [4]) and now I am quite hopeful that we will have Maxima on top of ecl and therefore one of the showstoppers of Sage on Solaris as well as Sage on native Windows has been removed without making Maxima collateral damage in the porting process. Maxima will still have to compete with the other components in Sage and some functionality will be moved into the core of Sage by rewriting it in C/Cython, but I now see its future assured in the long term.

So, what motivated me to write this all down? Most people either don't care or read the long threads on sage-devel and a couple other places or have actually read some or most of the discussions. It was actually the reply to an email of Richard Fateman [6] by Juan Jose Garcia-Ripoll, the ecl maintainer, which deserves to be quote in full:

Hi,

let me first begin by saying that, as politely as I can, Fateman's email are as close to FUD as it can get. He doesn't seem to use ECL at all and just judges from some outdated webpages and his own prejudices about different software libraries.

Regarding the different points which have been raised:

* GNU MP is only used for bignum computations. ECL itself is clever enough to handle fixnums cleverly and even to unbox fixnum computations in compiled code. Incidentally, GCL uses GMP as well, so I do not see the point all.

* The simplistic garbage collector is an option and it is provided for platforms in which Boehm-Weiser does not run. Currently, this means _none_ of the supported platforms.

* Boehm-Weiser is a strong garbage collector and a very powerful one in terms of tunability. You man make it as precise as you want, and the Java people indeed do. ECL uses it and it has seen only performance improvements as we have learnt more and more how to better use it. If the ANSI test suite shows something in that respect is that, under a lot of consing pressure, it does not perform that bad. It does not get so close to SBCL's but I doubt any other free implementation does.

* ECL has a good compromise between all platforms. It provides both C compiled code and a reasonably fast interpreter. Benchmark show that the ECL interpreter is not that far from interpreted CLISP. But on the other hadn CLISP has its own set of optimized bytecodes and when it compiles it optimizes for those bytecodes. AFAI remember, GCL used (and probably still uses) a list based interpreter which runs through forms represented as lists and macroexpanding every form that has to be done so, and every time it uses it. That is terribly inefficient.

* In terms of maintainability it has shown through the years that it is easier for somebody to start coding and hacking ECL and adding new features than with most other platforms. That is how we got ECL ported to the Microsoft compiler and platform and how different pieces of software (sockets, asdf, etc) have been adapted to run here. That by itself is an important value, at least for people who think long term.

* Talking about diverting efforts from the GCL crowd, I am not the best person to speak about it. I am more than pissed of by the GCL community since, shortly after ECL reached most ANSI compliance and portability I was asked to port all that back to the GCL, because they wanted to achieve the same goal. That was back in '01 or '02, do not remember so well. What I remember is that those were not very polite emails and had a kind of "borg" spirit of assimilation, without even caring about the years spent on achieving that. So talk about diverting useful efforts.

So, to the interested parties, if you so much care about Maxima running on just one computer, then stick to sbcl and cmucl which are pretty superior implementations, but please do not scare people from porting useful software to other platforms and environments.

Kind regards

Juanjo

-- Facultad de Fisicas, Universidad Complutense,
Ciudad Universitaria s/n Madrid 28040 (Spain)
http://juanjose.garciaripoll.googlepages.com

So I can only recommend that if you are willing to waste an hour or two to read all the sources and make yourself a picture for yourself. There is much more to the story than I wrote here and I did not go into as much details as I could have since I wanted to finish this blog post in a reasonable amount of time. This is precisely the reason I didn't write it in the first place a couple weeks ago.

So, will there be "Sage Lisp Wars - The Sequel!"? I am sure there will be and you can easily figure out which side of the debate I am on ;) Despite of all the flames I linked to above in the end I feel that the Sage community got something positive in the end by the likely port of Maxima to ecl and at the same time hopefully the Maxima community will see the port to ecl as a benefit to them, regardless how the cooperation between Maxima and Sage will develop.

Cheers,

Michael

Sage 3.0.1.alpha0 released

2008-04-26T01:52:00.000-07:00

Hello folks,

this is 3.0.1.alpha0. So far we have only merged bugfixes, nothing invasive has been merged yet and there is nothing on the radar that does look invasive. 24 tickets have been closed up to now and I am not quite sure what the rest of the release cycle will look like because it currently doesn't look like we need a pure bug fix only release quickly.

There are plenty of patches available for review. The coercion rewrite planned for 3.1 seems to be going well.

Sources and binaries are in the usual place:

Cheers,

Michael

Merged in alpha0:

#783: Alex Ghitza: dilog is lame
#1187: Alex Ghitza: bug in G.conjugacy_classes_subgroups()
#1921: Alex Ghitza, Mike Hansen: add random_element to groups
#2302: Michael Abshoff, William Stein, Scot Terry: Add a 64 bit glibc 2.3 based binary of Sage to the default build platforms
#2325: David Roe, Kiran Kedlaya: segfault in p-adic extension() method
#2821: Alex Ghitza: get rid of anything "about this document" sections of any sage docs that say "send email to stein"
#2939: David Joyner: piecewise.py improvements (docstring and laplace fixes)
#2985: Michael Abshoff: ITANIUM (RHEL 5) -- bug in rubik.py's OptimalSolver()
#2993: Michael Abshoff: OSX/gcc 4.2: disable padlock support per default
#2995: Alex Ghitza: some new functionality and doctests for congruence subgroups
#3003: Jason Brandlow: Bugfix for to_tableau() method of CrystalOfTableaux elements
#3005: Craig Citro: modabar -- failure to compute endomorphism ring
#3006: David Joyner: missing elliptic integrals in special.py
#3014: Michael Abshoff: ZZ.random_element -- corrupted docstring
#3017: Michael Abshoff: invalid link after make install
#3022: Tim Abbott: Debian package support for polybori
#3023: Jason Grout: make apply_map deal with empty matrices
#3025: William Stein: Sparse vector spaces don't cast on assignment
#3027: Tim Abbott: Debian lintian fixes

Ubuntu LTS 6.06 x86-64 binaries Available

2008-04-25T10:49:00.000-07:00

We finally have Ubuntu 6.06 LTS x86-64 binaries for Sage 3.0 available. it was mentioned in the release announcement, but a last minute bug did delay the release of the binaries since the rubiks.py doctest failed. That has been fixed.

Well, you might ask, Ubuntu LTS 8.04 is out, so what about binaries for that release? And the answer is the same as always: Make a VMWare image with minimal install and development tools, a working ssh access from the outside, a home partition with a devent amount of space [i.e. 20GB] and get it to William or me. From then on we will build you binary releases and the chance that Sage will compile and doctest fine on your preferred distrbution will be greatly improved.

Cheers,

Michael

Stupid Bugs - Part 823

2008-04-24T23:17:00.001-07:00

Well, I guess I never mentioned any of the previous bugs, but today I started looking at two rather vexing bugs that hit us on RHEL 5/Itanium. One of them (#2985) was rather odd since I couldn't reproduce it anywhere and the binary did valgrind clean I was running out of ideas. After building Sage 3.0 on Ubuntu LTS 6.06 I hit the same bug there, so it was even more vexing since everything worked on 64 bit Debian testing. But after looking at it over on Ubuntu 6.06 LTS there I still didn't see anything obvious that could be wrong. Switching back to my Itanium test box eventually revealed the problem. Somebody had left a 32 bit x86 binary of optimal in the spkg. Since the makefile for the Reid solver sucked we never ended up building it on a most x86 compatible Linux platforms. So it did start up on Itanium and then more or less die instantly. The bug is now fixed and is merged in 3.0.1.alpha0.

Next up: #2983 - an Itanium specific segfault in Singular. The fun never stops ;)

I have been rather quiet here, but hopefully I will be more active again in the future. Many things are swirling around in my head, but during the 3.0 release cycle I did not have the time to actually write down a long winded rant here. But I think that will change the next week.

Cheers,

Michael

Sage 2.11 is out!

2008-03-31T12:17:00.000-07:00

Sage 2.11 has been released. In total 31 people did contribute patches for this release. Due to spring break and the easter holidays things moved a little slower than I had anticipated. But in the end it looks like we put together a pretty decent release. Binaries are available by now.

The high level changes in detail are:

ATLAS: Michael Abshoff and Burcin Erocal upgraded ATLAS to the 3.8.1 release. In addition tuning info for 32 bit Prescott CPUs as well as Powerbook G4s under Linux was added.
zn_poly: David Harvey's zn_poly library is now a standard package for Sage. zn_poly is a new C library for polynomial arithmetic in (Z/nZ)[x] where 3≤n≤ULONG_MAX (i.e. any machine-word-sized modulus). The main benefit is speed. The library is used so far only to compute the zeta function for hyperelliptic curves.
Small roots method for polynomials mod N (N composite): Martin Albrecht implemented Coppersmith's method for finding small roots of univariate polynomials modulo N where N is composite.
Generic Multivariate Polynomial Arithmetic: Joel Mohler improved the efficiency of the generic multivariate polynomial arithmetic in Sage by roughly a factor of ten.
k-Schur Functions and Non-symmetric Macdonald Polynomials: Mike Hansen: k-Schur functions s^(k)_\lambda are a relatively new family of symmetric functions which play a role in Z[h1,...,hk] as the Schur functions s_\lambda do in \Lambda. The k-Schur functions, amongst other things, provide a natural basis for the quantum cohomology of the Grassmannian. The k-Schur functions can be used like any other symmetric functions and are created with kSchurFunctions. Non-symmetric Macdonald polynomials in type A can now be accessed in Sage. The polynomials are computed from the main theorem in "A Combinatorial Formula for the Non-symmetric Macdonald Polynomials" by Haglun, Haiman, and Loehr.
Marshall Hampton did upgrade gfan as well as the optional phcpack spkgs and their interfaces. He also increased doctest coverage to 100% for both interfaces.
Improved capabilities for solving matrix equations: William Stein implemented code so that one can now solve matrix equations AX=B and XA=B whenever a solution exists. In particular, solving linear equations now works even if A is singular or nonsquare.
Generators for congruence subgroups: Robert Miller implemented an algorithm for very quickly computing generators for congruence subgroups \Gamma_0(N), \Gamma_1(N), and \Gamma_H(N).
Various other people fixed a number of bugs and did improve other bits of Sage.

Next up is the 3.0 release. We are shooting for a two week release cycle, but we will see how things go :)

Cheers,

Michael

Sage vs. GSoC 2008

2008-03-25T17:24:00.000-07:00

Oh well, Sage didn't make it into the GSoC 2008. Looking at the selected projects one can see that mathematical Open Source software is underrepresented. Only the R project (which is technically a statistics software package and not mathematics - some people insist on that distinction) made it in out of many mathematical software projects who applied. In recent years various projects made it into the GSoC via other orgs, i.e. sympy via multiple orgs and at least one Axiom related project via LispNYC.org.

So why didn't Sage make it in? It is hard to tell since there is little transparency during the selection of the mentor organizations. Various people put up theories at Slashdot. Specifically pongo000 commented after hanging out a couple hours in #gsoc when various mentor orgs showed up and inquired why the hadn't been selected. pongo000's observation was that the following issues did play a large role:

the size of the organization
whether an org participated in years past
the quality of the ideas list

So how does Sage fit the bill?

the size of the organization: not huge, but certainly not at the low end of the spectrum. Interestingly enough OpenOffice.org didn't make it as a mentor org this year. Somebody did show up in #gsoc and inquired why that had happened and was informed that they allegedly didn't apply.
whether an org participated in years past: we applied twice and got twice rejected.
the quality of the ideas list: Some Sage developers did hang out in #gsoc and did talk to lh and ask questions about the application and list of ideas Sage had submitted. One aspect was the quality and accesibility of our idea list. Apparently there are people out there who are not familar with the more abstract aspects of mathematics :) and who do need a little more info about the projects, i.e. it isn't obvious to many people what commutative alegbra consists of. We had initially discussed at Sage Days 8 what kind of application we wanted (new blood vs. tried and true contributors) and come down pretty much in favor of tried and true contributors. While we had at least two projects on the list that didn't require mathematical expertise (the notebook and Cython) the other projects were very specifically tailored for the people who wanted to do them and do contribute to Sage in that area. I won't go into details here, but if we do participate in another GSoC we need to do have more CS that mathematical projects - at least that is worked for the R's project idea page. Another highly recommended feature is a template system that list requirements and difficulty of a given projcet. But in mathematics that is highly subjective, i.e. "Drawing Graphs on Surfaces with Genus greater than 0", is not something the average non-math student will be familiar with or get up to speed in three months. Emily Kirkman on the other hand has been working in Graph theory for a while, so while I am sure it is far from trivial it would have been a nice project. Other projects like "Combinatorial Species/Decomposable Objects" or "Free abelian groups and integer lattices" aren't exactly know to the average College graduate either.

So in the end while we aren't happy with the decision of Google it is up to Google in the first place where they spend their money. And it isn't game over for Sage yet and the GSoC 2008. We have various project suggestions that we are trying to get accepted via other mentor orgs like the PSF.

And while I cannot go into details right now I can tell you that this might not be the end of the road, so stay tuned for further announcements.

Cheers,

Michael

Sage 2.10.4 coming up

2008-03-17T05:11:00.000-07:00

We are very close to a Sage 2.10.4 release. I didn't announce the 2.10.3 release here, but Marshall Hampton did. My reduced blogging activities were are mix of procrastination, the need for some time away from the computer and Sage Days 8 in Austin, TX. Sage Days 8 was very nice and I had a lot of fun in Austin, but more about that in a later post.

So I am being lazy now and I will more or less copy and paste the announcement from the release notes:

Memory leaks exposed by modular symbols: Michael Abshoff, Martin Albrecht, Burcin Erocal, Willem Jan Palenstijn, Clement Pernet, William Stein: memory leaks exposed by modular symbols functionality. This ticket is a composite of numerous other memleak fixes merged over a 7 month period. Modular forms are an excellent way to expose memory leaks in pretty much every algebraic component of Sage and all known issues there are now finally fixed.
SQLAlchemy and DSage: We merged SetupTools and SQLAlchemy into Sage as standard packages. SQLAlchemy is now used as in DSage replacing hand written code with much more efficient classes from SQLAlchemy. SetupTools is required to install SQLAlchemy, but is also useful for a number of optional spkgs like MayaVI and packages from the Enthought Tools. Yi Qiang improved DSage making it more robust and finally adding the documentation to the standard Sage manual.
Graph theory: chromatic polynomial: An algorithm originally written in C by Gordon Royle has been adapted by Robert Miller to replace the old slow method. This algorithm uses a cut and merge algorithm to recursively compute the chromatic polynomial, and is written in Cython.
Documentation: Many doctest patches written during Doc Day 2 were merged. In addition many people kept up the good work after Doc Day 2 was over and have been submitting patches to increase coverage. We did exceed the target for the release by 0.6% reaching 47.6%.
Symmetric function updates: Mike Hansen, reviewed by Franco Saliola: Sage 2.10.4 adds support for Macdonald polynomials, LLT polynomials, and Jack polynomials as well as a whole class of user-defined symmetric functions which can characterized by orthogonality and triangularity conditions. Support for working with ribbon tableaux was also added as part of these updates. In addtition, many doctests were added and subtle bugs fixed.
Notebook Updates Tom Boothby, William Stein and Timothy Clemans: Fixed a bunch of new and old issue that improve the usability of the notebook. Among those are a working trash, fixes to the polling infrastructure, saving the content of unevaluated cells, URL issues to work around problems introduced by restrictive firewalls and small improvements to the interact command.
Parallel Doctesting: Gary Furnish reviewed by Michael Abshoff: "sage -tp" has been introduced as an experimental multithreaded doctester. The first parameter is the number of threads, and the second parameter is the folder to doctest. Thus "sage -tp 4 devel/sage/sage" tests everything with four threads running. Additional options like "-long" or valgrind options like "-memcheck" do work. The code base is still young and needs more testing. The eventual goal will be to replace the current doctesting infrastructure with this code base.

Sage 2.10.4.final has been released and is currently being build tested on our compiler farm. Once it has passed tests and no blocker issue shows up in testing we will release.

Some times I am right, often I am wrong

2008-02-26T09:18:00.000-08:00

Oh well, Ondrej Certik offered a debugging challenge for a problem in Sage he discovered. The problem was that he hit a timeout when starting up a binary copy of Sage on his shiny, new and fast Intel Quad Code box. After some debugging he discovered that Debian's XFS default mount options cause the slowdown, all the details can be found in this blog entry from Ondej's blog. You should go over to his blog and check out the quite interesting details since he took the time to track the issue down.

The challenge boiled down to the whether the timeout bug was in Sage or XFS. The loser would have to publicly claim that he is lame. Since the challenge was issued and resolved while I was asleep I did contribute little to the end result and let's just say that I am not lame. By the process of elimination you can guess the outcame. But I have been wrong about bugs in Sage and elsewhere in computing in general many times before and while I teased Ondrej a little bit about it in IRC it was all good fun. It just shows that Sage is much, much more than a pure technical project and has a social and community side, too.

Cheers,

Michael

Sage 2.10.2 has been released

2008-02-23T17:09:00.000-08:00

Sage 2.10.2 has been release roughly three weeks after 2.10.1. The delay is mostly due to Sage Days 7, which took the attention away from most developers and also caused a number of people to come down with nasty colds (me included). Highlights from the release (shamelessly stolen from the announcement):

John Voight's fast new code for enumeration of totally real fields is now included.
David Roe's code for unramified and Eisenstein extensions of Qp and Zp is now included.
Clement Pernet, Burcin Erocal and William Stein have implemented an optimized p-adic/modular algorithm for computing Hermite normal forms of matrices over the integers. For random square nonsingular matrices with small entries it is similar to Magma in speed, and vastly faster than the implementations in Gap, NTL, and PARI. For matrices with large entries (e.g., 16 bits or more), it is faster than anything else in the world. For nonsquare matrices it is also reasonably good, though more optimization is needed since Magma is much better in some cases. We also implemented related code for computing determinants over QQ and ZZ, which is again the fastest in the world especially when the matrix entries are large. The main reasons for the speed of our implementation are (1) IML is fast, and (2) we found some tricks that are not in the literature.
Tim Abbott and Michael Abshoff worked on the Debianization of the build process. Due to a lot of work done by Project Athena at MIT Tim Abbott contributed many build scripts for chroot environments. He also contributed build scripts for nearly all of the SPKGs not yet in Debian. Michael Abshoff did set up a test build server and while it has been shut down for now the Sage project will set up another 64 bit build server in the near future top provide Debian packages for a wide variety of Debian based distributions.

Binaries for 2.10.2 have been uploaded to sagemath.org and are available at various mirrors - even though some might still have to catch up over the next couple hours.

Sage 2.10.3 is coming up and we might do a real quick bug fix only release since Sage Day 8 at Enthought is coming up.

Cheers,

Michael

A Late Sage Days 7 report

2008-02-19T13:07:00.001-08:00

Oh well, Sage Days 7 has been over for quite a while and I did procrastinate on this blog post for way too long.

The event was hosted at IPAM at UCLA and it was a lot of fun meeting a whole new set of faces only known from IRC and email in person and also meeting a large set of people I met at previous Sage Days again. I was one of the few Europeans there (and probably the only one who came directly from Europe) and the flights sucked. For some reason I was routed via Las Vegas and the airline managed to lose my luggage on the last leg of my flight itinerary, a 40 minute flight from Las Vegas to LAX. The travel from LAX to Westwood was very pleasant, but finding the UCLA guest house took some more time than I had thought. In the end I finally caught some sleep and due to my charger and backup laptop resting safely in my luggage I didn't respond to email too much for the first 48 hours.

After my luggage showed finally up roughly two days later things got better, but I never caught up with my email until I made it home. I mostly worked on the Debianization of Sage with Tim Abbott, merged a bunch of tickets and spend a lot of time on discussions and bug fixing. Overall it was a very productive Sage Days for me.

I ended up winning the "Guess that spkg" and "Guess that ticket" competitions and I guess everybody did expect me to win since it would have been a major upset if somebody else beat me there. Let's see if we are having another contest at Sage Days 8 in Austin at the end of the month.

Flying back to Europe was alright, even though I didn't sleep the night before and dozed on the plane for a couple of hours. On the way back I visited rpw in Darmstadt, but at that point I was pretty useless from lack of sleep. I also ended up with a rather severe cold which also attacked other participants of SD7, so that I spend the two days after I got home mostly sleeping and away from the computer.

Cheers,

Michael

Bug Day 10

2008-02-16T07:18:00.000-08:00

For those who hadn't heard the news: Bug Day 10 will start in a couple hours. Basis will the Sage 2.10.2.alpha0, so if you want to participate you should try to compile it until then.

It has been a while since I blogged, mostly due to Sage Days 7 and a cold that followed, but more about that later. I plan to blog about various Sage related issues in the near future, so things should be more lively again.

Cheers,

Michael

Sage 2.10.1 release plan and an Outlook for 2.10.2

2008-01-29T03:12:00.001-08:00

After various release candidates, i.e. Sage 2.10.1.rc0-rc2 we are getting finally close to the final release. The release cycle has been longer than usual since we still have a number of blockers to resolve, i.e. the ATLAS tuning issue as well as fallout from the GNUTLS update which causes segfaults when starting the notebook. It will be the release for Sage Days 7, but we might still push another bug fix only release if it turns out that we have some glaring, must fix bugs in Sage 2.10.1. Sage 2.10.2 (or .3) will probably merged David Roe's unramified and eisenstein extensions of Qp and Zp as well as John Voight's code for enumerating totally real fields with improvements by Craig Citro. We also expect more fixes for the 64 bit OSX port as well as for the Solaris port, but it is uncertain if either one of them will become officially stable in that release.

Cheers,

Michael

Worst SPKG ever!

2008-01-25T16:32:00.000-08:00

Well, in case you don't get the Simpsons reference about Comic Book Guy's catch phrase you might want to read those two links. But to get back to the point of this post: I spend about 6 hours or so on ticket #1852. It looked very harmless, i.e. just enable ATLAS during the configure phase and fix some other odd issues, but it turned out to be the worst spkg I ever had to fix. No need to name names here since the subject of this post is mostly tongue in cheek by now. While I did work on the spkg I did feel differently, but those things had a tendency to fade into the background once I have moved on to other things.

Do I blame the person who initially did create the spkg in question? Not really, since we are all busy people and I also had to read the R build manual to fix some of the issues I saw. It was just surprising that this particular spkg was quite bad compared to all the other ones, i.e. we installed about 60 mb unneeded object files somewhere in SAGE_LOCAL and those do add up for the final download size. I am just glad it is done and now I could start bitching about gsl 1.10, see #1623. Since that one is done, too, now it is on to #1631 and then who knows what. So, sooner or later there will be another "Worst SPKG ever!" and this one will be a distant memory ;)

Cheers,

Michael

Sage 2.10.1 Progress

2008-01-22T19:32:00.000-08:00

Progress on Sage 2.10.1 has been good so far. We closed 71 tickets as off alpha2 and have about another 30 patches or so under review. The initial load of patches were merged during Bug Day 9, which turned out to be an excellent debug session. The wiki page is somewhat misleading since the IRC log isn't up yet and many people who participated did not add themselves to the wiki page.

But there is some trouble in paradise since the upgrade of GNUTLS and its related components broke the notebook in secure mode. This lead to a revisit in #sage-devel about whether linking a GPLed program like Sage against OpenSSL is legal or not. While opinion diverged in this it remains a fact that we should fix the GNUTLS issue (or is it in Twisted? I am not sure yet) since we can't ship OpenSSL libraries with the binary and the reliance on a properly working system OpenSLL in the past had lead to plenty of trouble in the past.

Besides the above issue what else can we expect of 2.10.1? Better ATLAS build times on Pentium Ms and 32 bit Athlons (not merged yet as of alpha1). a lot of work toward building a 64 bit OSX version of Sage and maybe even a universal binary for OSX (32 bit ppc and x86), so stay tuned.

Cheers,

Michael