## Mathematics tools

### maxima optimization for Stack 4.0

maxima optimization for Stack 4.0

Dear All,

I upgrade stack 3.0  to 4.0 this morning, after then, I tried to  optimize maxima, my environment  is

ubuntu 16.04 LTS + SBCL 1.3.1 + MAXIMA 5.39,

while I type  under maxima

I got a message "INFO: Control stack guard page unprotected"

Then the sbcl halted .

It never happened for Stack 3.0

Any idea ?

Thank you very much for your assistance.

C. T.

Average of ratings: Useful (1)
Re: maxima optimization for Stack 4.0

Hi.
Idem for me.
No idea.
Chris ? Tim ?
Thanks.
I killed maxima processus to exit and switched to Linux non optimised.

I ran the heathcheck script.
OK CAS
OK maxima image
OK library version

BUT 2 display bugs :
1. La dérivée de @ x^4/(1+x^4) @ est

2. Deux exemples de tracés. @plot([x^4/(1+x^4),diff(x^4/(1+x^4),x)],[x,-3,3])@ @plot([sin(x),x,x^2,x^3],[x,-3,3],[y,-3,3])@
(no graphics)

Laurent Briend

Average of ratings: -
Re: maxima optimization for Stack 4.0

The @ symbol isn't reconised.

Moodle 3.1.1 (build 20170710)

Stack 4.0.1

Maxima 5.36.1 using Lisp SCBL 1.2.4.debian

Debian server

Laurent Briend

Average of ratings: -
Re: maxima optimization for Stack 4.0

It appears that there has been a change in the delimiters from @ x @ to {@ x @}. This ought to be able to be fixed in old questions with the delimiter script, but that did not work for me, nor did changing delimiters manually in old questions.

I was able to create a new question with the new delimiters and it worked.

Average of ratings: -
Re: maxima optimization for Stack 4.0

I found that editing the question, checking the 'fix dollars' box, and saving updates the question to work with the new delimiter. However, this is a slow process without the optimization turned on.

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hi,

I have a similar problem, when I am trying to optimize Maxima.
When I type in

, Maxima halts. It does not give any errors, just freezes.

Everything is working fine when I use non-optimized version. Also the delimiter update worked for me.

The non-optimized version is too slow for my server, so this is a little bit acute problem for me.

Mikko

Average of ratings: -
Re: maxima optimization for Stack 4.0

I do not think that this problem is caused by the upgrade in STACK, but instead by a security patch to the system (perhaps CVE-2017-1000366). I can not get older versions of STACK to compile an optimized image either anymore with stock maxima.

On CENTOS, I recompiled the maxima package from source, and after that the optimization worked. I have not found a fix yet for ubuntu.

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hi,

I upgraded to Debian Stretch, old maxima erased and new install of maxima.

With this version of Debian :

###

Maxima 5.38.1 http://maxima.sourceforge.net
using Lisp GNU Common Lisp (GCL) GCL 2.6.12

###

Note that it isn't SCBL.

For the optimisation, the documentation for GCL says :

• Get STACK working with Platform type set to 'Linux'. Run the health-check. It is important to do this every time you upgrade your version.
    load("<path>/maximalocal.mac");
:lisp (si::save-system "/path/to/moodledata/stack/maxima-optimised")
quit();

• Go into the STACK settings and set Platform type to 'Linux (Optimised)'.
• Set Maxima command to.

/path/to/moodledata/stack/maxima-optimised -eval '(cl-user::run)'

I tried and :
- OK for
 load("<path>/maximalocal.mac");
- Bug with :
 load("<path>/stackmaxima.mac");
"Maxima encountered a Lisp error:

Condition in MACSYMA-TOP-LEVEL [or a callee]: INTERNAL-SIMPLE-ERROR: Value stack overflow.

Automatically continuing.
To enable the Lisp debugger set *debugger-hook* to nil."

But I have now the linux optimised version of maxima : Healthcheck returns
"unix-optimised
OK    CAS a retourné les données comme prévu. Vous êtes bien connecté à CAS.
OK    Version de librairie STACK-Maxima correcte et attendue (2017082400).
Les résultats de CAS ont été placé dans la base de données."

Any idea to fix the bug and complete the linux optimised version of maxima ?

Laurent Briend from France.

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Daniel

What maxima version did you use? 5.38.1.?

Furthermore, how did you come to the conclusion, that the security patch might be causing this issue?

Cheers, Joël

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Joël,

On my development system, I removed STACK and reinstalled an earlier version of STACK which I had optimized before and still experienced the same issue now. I am assuming that means something else on the system changed. I searched for some the lisp error and found some reports of recent security fixes causing that error.

My development system is Ubuntu 17.04. I followed the instructions from Artem and it resolved the issue for me there. In production, we are running CENTOS 6 will SBLC and Maxima 38.1 which is compiled from the source RPM. After I could not optimize STACK, I rebuilt the maxima from source again. As far as I know that was the only change I made and then the optimization worked.

Daniel

Average of ratings: -
Re: maxima optimization for Stack 4.0

Same issue here and after much fiddling the best I could get is the following:

Go to Site Admin. > Plugins > Question Type > STACK: In the Stack menu keep Platform as 'Linux', keep CAS connection timeout as default (or set higher if you're getting CAS connection errors in the healthcheck script), set Maxima command to '/path/to/maxima', then just run the healthscript.

Assuming all goes well it will spit out OK's at the bottom (if not, first thing to try is clearing cache and checking for typos in maxima path), click 'Create Maxima image' and copy the 'timeout --kill-after=10s 10s /path/to/maxima_opt_auto'

Go back to the STACK menu and now change platform to 'Linux (optimized)', lower the CAS connection timeout (I got it at 5), and set the Maxima command to 'timeout --kill-after=10s 10s /path/to/maxima_opt_auto' (what you got from the healthcheck)

Re-run the healthcheck to make sure it worked okay. I have not used STACK in the past (new to Moodle in general to be honest) so I'm not sure how this compares to optimizing it the way it says in the docs, but it appears that this creates an auto-optimized image that does indeed run STACK much faster.

**
Also (not sure if this is necessary, but just in case it is), assuming you compiled Maxima from source and are getting the following error when you run maxima in terminal:
WARNING: Maxima is unable to set up the help system.
"/usr/share/info/./maxima-index.lisp":
file does not exist.)
Go into that directory (/usr/share/info/) and make sure that the maxima files located there are uncompressed (they were all compressed for me). If you didn't get that error when you compiled - great - but I got every time I compiled Maxima no matter what version I was compiling, and I was fixing this at the same time as I was trying to optimize STACK so I include it here just in case it is necessary to eliminate this maxima error for stack to function/optimize properly.
**

Hope that helps,
A.Z.

Average of ratings: Useful (2)
Re: maxima optimization for Stack 4.0

Hi,

I only want to add that we tried Stack in production and it works well although there were errors during the optimization process (see above).

Laurent Briend

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hmm not even sure if we have the same problem as described above...

We're running STACK 4.0.1 with maxima Version 5.38.0 and SBLC 1.3.11 on CentOS 7.
When trying to optimize STACK, I get the following error:

INFO: Control stack guard page unprotectedControl stack guard page temporarily disabled: proceed with caution
Maxima encountered a Lisp error: Control stack exhausted (no more space for function call frames).This is probably due to heavily nested or infinitely recursive functioncalls, or a tail call that SBCL cannot or has not optimized away.PROCEED WITH CAUTION.Automatically continuing.To enable the Lisp debugger set *debugger-hook* to nil.

Sometimes it runs trough, however the result is always the same, the STACK healthcheck page indicates, that it can't connect to CAS.

Is someone with a similar setup experiencing the same behaviour?

Average of ratings: -
Re: maxima optimization for Stack 4.0

I am pretty sure it is the same issue. The error message varies with OS and lisp implementation, but includes a reference to 'stack guard page'. Are you using maxima-38.0 from EPEL repo or did you try rebuilding it from source? I was receiving a similar error on CENTOS 6, but since I rebuilt the maxima package it optimizes consistently.

Average of ratings: Useful (1)
Re: maxima optimization for Stack 4.0

Maxims 5.38.1 works for me, but I tried 5.38.0 and it does not

Average of ratings: Useful (1)
Re: maxima optimization for Stack 4.0

Hello Daniel

Awesome! Thanks for the information! I'm going to do the same as soon as I have the opportunity, as we don't have a test platform running CentOS, but only a productive one! At the moment we have maxima installed from the EPEL repo.

Cheers

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hi Daniel and Joël, did you ever resolve that issue with the EPEL maxima?

I am just running into the same problems, and would like to solve it, especially since I am writing an instruction manual for the maintainers of our moodle.

Also, I notice I am using a maxima that is newer than Chris allows. Should I expect problems?

Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

OK, I have now tried with a recompiled maxima.

As I tried to say, the problem mentioned above occurs with maxima 5.41.0 and sbcl 1.4.0, which are the current versions from EPEL--but it seems this maxima doesn't play ball with STACK (it shows the graphs but doesn't differentiate), so I downgraded to maxima 5.39.0 (by "adapting" the spec file from the EPEL source). Now works fine with stack, but still doesn't generate an optimised maxima.  I should have trusted Chris's instructions on the maxima version to use!

I further tried two more version of sbcl (since the error seems to occur in its stack handling): a newer 1.4.1 and an older 1.3.18.  No luck, the issue persists. I tried building gcl, but the build fails, so that appears not to be an option as a lisp replacement--and I am loath to do anything that can not be done as a build of an rpm, since I am analysing this for a maintainer.

Since maxima 5.38.1 was mentioned, I also tried that. Same issue.

Clearly I can try all  allowed maxima's with many versions of sbcl, and avoid the infinite recursion, but I am running out of time: there are serious issues  with a RHEL7/Centos7/SL7 installation! Anyone got any other suggestions?

Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Daniel

I've overtake this pending issue at our company. It doesn't seems to work after the updates.

Our actual setup is:

• CentOS 7.0 (3.10.0-693.5.2.el7.x86_64)
• STACK 4.0.1 for Moodle 2.7+ (2017082400)
• Maxima 5.41.0 (5.41.0-1.el7.x86_64)
• SBCL 1.4.0-1 (sbcl-1.4.0-1.el7.x86_64)

I've optimised maxima but the CAS always refused the connection in the Healtcheck. Like e.g "CASText failed validation. CAS failed to return any evaluated expressions. Please check your connection with the CAS".

Output when optimizing the maxima package:

Maxima 5.41.0 http://maxima.sourceforge.net

using Lisp SBCL 1.4.0-0.1.el7.bootstrap

Dedicated to the memory of William Schelter.

The function bug_report() provides bug reporting information.

[ STACK-Maxima started, library version stackmaximaversion ]

(%o1) maximalocal.mac

WARNING: redefining MAXIMA::OPAPPLY in DEFMACRO

WARNING: redefining MAXIMA::OPCONS in DEFMACRO

define: warning: redefining the built-in function intervalp

WARNING: redefining MAXIMA::TEX-MQUOTIENT in DEFUN

WARNING: redefining MAXIMA::TEX-PREFIX in DEFUN

WARNING: redefining MAXIMA::TEX-STRING in DEFUN

WARNING: redefining MAXIMA::TEX-DERIVATIVE in DEFUN

WARNING: redefining MAXIMA::TEX-D in DEFUN

WARNING: redefining MAXIMA::TEX-DABBREV in DEFUN

WARNING: redefining MAXIMA::TEX-INT in DEFUN

WARNING: redefining MAXIMA::TEX-MEXPT in DEFUN

; file: <path>/question/type/stack/stack/maxima/stacktex.lisp

; in: DEFUN MSZ-DISPDPVALUE

;     (MAXIMA::GETCHARN MAXIMA::L 1)

;

; caught STYLE-WARNING:

;   undefined function: GETCHARN

; in: DEFUN MSZ-FLOATGRIND

;     (MAXIMA::GETCHARN MAXIMA::L 1)

;

; caught STYLE-WARNING:

;   undefined function: GETCHARN

; in: DEFUN TEX-MEXPT

;     (MAXIMA::GETCHARN MAXIMA::F 1)

;

; caught STYLE-WARNING:

;   undefined function: GETCHARN

;

; compilation unit finished

;   Undefined function:

;     GETCHARN

;   caught 3 STYLE-WARNING conditions

; file: <path>/question/type/stack/stack/maxima/mathml.lisp

; in: DEFUN MATHML1

;     (MAXIMA::CONCAT "(" (MAXIMA::STRIPDOLLAR MAXIMA::MEXPLABEL) ")")

;

; caught STYLE-WARNING:

;   undefined function: CONCAT

;

; compilation unit finished

;   Undefined function:

;     CONCAT

;   caught 1 STYLE-WARNING condition

define: warning: redefining the built-in function texnumformat

WARNING: redefining MAXIMA::TEXNUMFORMAT in DEFUN

(%o0) "<path>/question/type/stack/stack/maxima/stackmaxima.mac"

(%i1) :lisp (sb-ext:save-lisp-and-die "<path>/maxima-optimised" :toplevel #'run :executable t)

Average of ratings: -
Re: maxima optimization for Stack 4.0

See my discussion below--the version of maxima is definitely too new, but there seem to be serious issues with all versions I have tried.
Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

You folks are way ahead of me!  I will be supporting Maxima 5.4X.Y in due course.....

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Christopher

The problem with the EPEL7 repo is/was, that there was only the version 5.38.0 available. With the STACK update to 4.0 the optimised version was not working too.

There was as Daniel described in this topic by self the workaround with compile the 5.38.1 maxima package. Now the official repo made an big step from 5.38.0 to 5.41.0! That's the acutal situation and the optimization is still not working.

Kind regards

Average of ratings: -
Re: maxima optimization for Stack 4.0

OK,

I have now compiled gcl: The trick was to turn selinux off during compilation (I used gcl-2.6.12-1.fc21.src.rpm, with a tweak to the tex(latex) requirement in the spec file to compile). Compiling maxima takes loooong with gcl, so be patient.

I then hacked the maxima spec  file (3.9.0) to work with gcl.

Unfortunately there is again a stack overflow on load("/var/www/html/moodle/question/type/stack/stack/maxima/stackmaxima.mac");

Maxima encountered a Lisp error:

Condition in MACSYMA-TOP-LEVEL [or a callee]: INTERNAL-SIMPLE-ERROR: Value stack overflow.

Automatically continuing.

Thus this is no solution either--it looks like something is broken somewhere, and most likely in the combination of EPEL patches to maxima and stackmaxima.mac.

Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

I am sorry I do not have a solution at the moment. Our system has been working correctly for the last few weeks until I upgraded the OS and STACK. Now I am unable to create a optimized image again.

Average of ratings: -
Re: maxima optimization for Stack 4.0

Folks,

I've compiled Maxima 5.41.0 with SBCL on Debian 8.

There is an incompatibility with previous versions, and I'll need to post a version of STACK which supports these newer versions of Maxima.  Leave this with me. This is a problem in the display, and is related to the error involving "GETCHARN" above.

I've had no problems in compiling a Maxima image for Maxima 5.41.0 with SBCL on Debian 8, but this problem may be related to the image creation on Centos.  I don't know.

Until I release a supported version, STACK does not work with Maxima 5.4X.Y.

Chris

Average of ratings: Useful (1)
Re: maxima optimization for Stack 4.0

Okay I have my optimization working now. I looked through the maximalocal.mac file and I notice that the additional libraries were listed there as well as in the instructions for optimization with sbcl. It did not make sense to me to load them twice so I removed the lines

load("stackmaxima.mac")$load("stats")$load("distrib")$load("descriptive")$load("simplex")$print(sconcat("[ STACK-Maxima started, library version ", stackmaximaversion, " ]"))$

from maximalocal.mac. Then I completed the optimization as normal.

The set up I am using is CENTOS 6, sbcl 1.3.14, maxima 5.38.1 with the both compiled from source, but I am not sure that it versions matter.

Average of ratings: -
Re: maxima optimization for Stack 4.0

Thanks Daniel,

I think there are two different issues going on here which have become conflated:

1. Support for optimisation on Centos based systems
2. Support for Maxima 5.40.0 and later versions.

I very much want to support both of these, of course.  I'm working on the support for Maxima 5.40.0 onwards today.  There have been a couple of small changes in Maxima which I'm incorporating into STACK.   I'm confident I now have identified all of these but then I need to check back compatibility with a range of Maxima versions, which takes time to run the unit tests on all these.

I'm planning a release of STACK before Christmas to upgrade our servers for January.  Support for Maxima 5.40.0 onwards will be included in this, and available in a separate branch as soon as I've tested it.

Thanks for pointing out the redundant "load" commands.  Since these are (and should) be in maximalocal.mac I've removed these instructions from the docs.  I've done it that way round so that the docs are robust to adding new libraries.

I note, however Daniel you are using 5.38.1, but the problems were apparently with compiling 5.40.0.  Once I've released the version which works here it would be helpful if someone could compile and test this on Centos as I don't have access to a Centos system.

Chris

Average of ratings: -
Re: maxima optimization for Stack 4.0

I agree with you Chris--the issue appears to be working with Centos7 (or in my case SL7, which is essentially the same) and any version of Maxima. The newer maxima issues are separate. It looks like there are no issues on SL6 with maxima 3.81.

It is easy enough for me to test a newer maxima (since everything is now packaged up), but at the moment I have not yet been able to compile an optimised maxima on Centos7

Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

OK,

so not following the old instructions seems to solve the issues: not loading stacklocal before stackmaxima seems to solve all the issues, and I can compile an optimised maxima.

Niels

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Chris,

The optimization issue affects most if not all of the maxima versions. Maxima 5.38.1 did work for me for a while, but stopped being able to produce the image recently. Beside CENTOS the problem has also appeared in the LTS versions of Ubuntu.

There seems to be a problem with loading stackmaxima.mac in maximalocal.mac. The file generated contains

load("stackmaxima.mac")$ However, it does not appear that stackmaxima.mac is in the load path for maxima. If I add a symbolic link to this file in the working directory, the image compilation works, but otherwise gives a stack error. I think this was not noticed earlier because the instructions said to load the library again later with path specified. A security patch changed to behaviour to throw a stack exception when the file is not found. Newer distributions have a different way a resolving the security issue; so they are able to create images okay, but most have newer versions of maxima. Daniel Average of ratings: - Re: maxima optimization for Stack 4.0 Thank you Daniel, This is all very helpful, and it sounds like this should be possible to fix. We need to get the paths correct in order to create the optimised Maxima image. If you look in maximalocal.mac then you will see locally generated paths to tell Maxima where to look for libraries. This includes the path to maximalocal.mac, and to the STACK specific maxima files distributed as part of the question type. It is crucial to have the load("stackmaxima.mac")$
at the end of the maximalocal.mac file.

Without access to the OS it is very hard for me to write and test docs, but I think we should have a dedicated page for this.  Can you help write one?

Chris

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello Chris,

I have done quite a bit more testing, and I believe I made a mistake before. I must have tried the symbolic link at the same time I removed the redundancy. The path specification in maximalocal.mac appears to be correct.

If I repeat the steps as before manually allowing time between steps, it is evident that the exception is thrown when stackmaxima.mac is loaded a second time. I have not been able to reproduce the problem without loading a library twice. I have tried various combinations of versions of maxima and sbcl. So I think that changing the optimization instructions will fix this issue.

The other issue may be more difficult. EPEL seems to be updating maxima whenever Fedora updates the package this means that for CENTOS users, the maxima version may be updated now several times within the lifetime of a stable release. Administrators need to be aware this is happening. It might be better to encourage system admins to compile and install maxima directly from source rather than using the package manager to prevent the version being changed inadvertently. However, I would expect some resistance to that idea.

Daniel

Average of ratings: -
Re: maxima optimization for Stack 4.0

Hello

Did somebody of you get it now working? STACK got some updates... but I can't get it working on our setup.

I've got following error and then it hangs after the load("stackmaxima.mac") command:

[<user>@<host> stack]# maxima
Maxima 5.41.0 http://maxima.sourceforge.net
using Lisp SBCL 1.4.0-0.1.el7.bootstrap
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
WARNING: redefining MAXIMA::OPAPPLY in DEFMACRO
WARNING: redefining MAXIMA::OPCONS in DEFMACRO
define: warning: redefining the built-in function intervalp
WARNING: redefining MAXIMA::TEX-MQUOTIENT in DEFUN
WARNING: redefining MAXIMA::TEX-PREFIX in DEFUN
WARNING: redefining MAXIMA::TEX-STRING in DEFUN
WARNING: redefining MAXIMA::TEX-DERIVATIVE in DEFUN
WARNING: redefining MAXIMA::TEX-D in DEFUN
WARNING: redefining MAXIMA::TEX-DABBREV in DEFUN
WARNING: redefining MAXIMA::TEX-INT in DEFUN
WARNING: redefining MAXIMA::TEX-MEXPT in DEFUN

; file: <path>/question/type/stack/stack/maxima/mathml.lisp
; in: DEFUN MATHML1
; (MAXIMA::CONCAT "(" (MAXIMA::STRIPDOLLAR MAXIMA::MEXPLABEL) ")")
;
; caught STYLE-WARNING:
;         undefined function: CONCAT
;
; compilation unit finished
;         Undefined function:
;             CONCAT
;         caught 1 STYLE-WARNING condition
define: warning: redefining the built-in function texnumformat
WARNING: redefining MAXIMA::TEXNUMFORMAT in DEFUN
[ STACK-Maxima started, library version 2017121800 ]
(%o0) "maximalocal.mac"
INFO: Control stack guard page unprotected

Kind regards

Average of ratings: -
Re: maxima optimization for Stack 4.0

The short version of the discussion above is that maximalocal.mac now loads all the needed libraries so you should not load them explicitly or they may cause an error. The load for maximalocal.mac completes successfully above with a few warnings, but the next load fails. After loading maximalocal.mac you should immediately save the image.

Daniel

    load("maximalocal.mac");
:lisp (sb-ext:save-lisp-and-die "maxima-optimised" :toplevel #'run :executable t);

Average of ratings: -
Re: maxima optimization for Stack 4.0

Thanks so much  @Daniel and all others!

It's now runningin linux optimised mode! I found this discussions helpful:

After creating the maxima-optimised image, it was not running properly in due of SELinux until I have done following:

restorecon -Frv /yourpath-to-moodledata/stack/

The error from SELinux logs /var/log/audit/audit.log was this, when accessing the Healtcheck https://your-moodle/question/type/stack/healthcheck.php:

type=AVC msg=audit(1514452742.849:278923): avc:  denied  { execute } for  pid=27983 comm="maxima-optimise" path="/path-to-moodledata/stack/maxima-optimised" dev="dm-2" ino=100669758 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:object_r:httpd_sys_rw_content_t:s0 tclass=file
type=SYSCALL msg=audit(1514452742.849:278923): arch=c000003e syscall=9 per=40000 success=no exit=-13 a0=20100000 a1=8000 a2=7 a3=12 items=0 ppid=25708 pid=27983 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="maxima-optimise" exe="/path-to-moodledata/stack/maxima-optimised" subj=system_u:system_r:httpd_sys_script_t:s0 key=(null)
type=PROCTITLE msg=audit(1514452742.849:278923): proctitle="/path-to-moodledata/stack/maxima-optimised"

Average of ratings: -