Thread Subject:

Fmincon out of memory

Hi guys, I'm performing a multivariable non linear optimization with fmincon.
Most of the time fmincon runs smoothly but sometimes it remains stucked within some loops I am not able to deal with. In these cases it leads to out of memory errors, which stop the whole optimization. I've tried to change solver (active-set, sqp, interior-point) but the problem present regularly. I changed also tolerances. Since the fmincon is part of a vast monetcarlo simulation running on our hpc (high performance computing), I could amend wrong results a posteriori, without risking to crash the machine. I think the matter resides in the gradient evaluation. Is it possible exit the single fmincon loop which leads to 'out of memory'? Any suggestion is welcome. Cheers, Massimo

fmincon itself would not run easily out of memory (that would require a very large hessian matrix is suppose).

so the problem is very likely to be in the function you try to evaluate (ie. the monte carlo).
would the memory usage depend on the the variables that you optimize for?? or do you possible add things to the memory? (and did you max. out memory?)

The montecarlo simulation is on top of the fmincon. Essentially I am performing a navigation analysis on a trajectory. At some points some manoeuvres are allocated and fmincon performs the optimization of these manoeuvre. The montecarlo consists of repetitive simulation of trajectory propagation and then allocation of manoeuvres. The problem is undependent from the montecarlo and doesn't present always, but only for certain initial condition which depends on the stochastic process which simulates the trajectory propagation.
I even ran the process on high performance computer and it was able to reach 80Gigabytes of memory before crashing, after remaining stucked for some 10 hrs in one fmincon iteration.

I would like to insert some control which restart fmincon or simply exit from it once it has been stucked on one iteration for 1 hour or something. Is there any way to insert
 
"Sargondjani" wrote in message <jsn30h$251$1@newscl01ah.mathworks.com>...
> fmincon itself would not run easily out of memory (that would require a very large hessian matrix is suppose).
>
> so the problem is very likely to be in the function you try to evaluate (ie. the monte carlo).
> would the memory usage depend on the the variables that you optimize for?? or do you possible add things to the memory? (and did you max. out memory?)

"Massimo" wrote in message <jsn9il$nc6$1@newscl01ah.mathworks.com>...
>
> I even ran the process on high performance computer and it was able to reach 80Gigabytes of memory before crashing, after remaining stucked for some 10 hrs in one fmincon iteration.
>
> I would like to insert some control which restart fmincon or simply exit from it once it has been stucked on one iteration for 1 hour or something. Is there any way to insert
============

I think you're thinking about the problem the wrong way. What you should be questioning is whether your own code could be made to consume less memory. 80GB sounds excessive.

"Matt J" wrote in message <jsnn7q$99o$1@newscl01ah.mathworks.com>...
> "Massimo" wrote in message <jsn9il$nc6$1@newscl01ah.mathworks.com>...
> >
> > I even ran the process on high performance computer and it was able to reach 80Gigabytes of memory before crashing, after remaining stucked for some 10 hrs in one fmincon iteration.
> >
> > I would like to insert some control which restart fmincon or simply exit from it once it has been stucked on one iteration for 1 hour or something. Is there any way to insert
> ============
>
> I think you're thinking about the problem the wrong way. What you should be questioning is whether your own code could be made to consume less memory. 80GB sounds excessive.

I have to point out that is fmincon which little by little increases the memory demand when it gets stucked. The code itself run easily with 300Mb in normal situation, but only in particular cases which depends on stochastic variables it becomes foolish. The problem presents when fmincon is not able to make the hessian positive definite, and continues evaluating the derivatives leading to overflowing the memory usage, up the machine crashes.

"Massimo" wrote in message <jsnocs$d2o$1@newscl01ah.mathworks.com>...
>
> I have to point out that is fmincon which little by little increases the memory demand when it gets stucked. The code itself run easily with 300Mb in normal situation, but only in particular cases which depends on stochastic variables it becomes foolish. The problem presents when fmincon is not able to make the hessian positive definite, and continues evaluating the derivatives leading to overflowing the memory usage, up the machine crashes.
================

Still sounds fishy. I don't see why repeating derivative calculations would accumulate memory usage. Doesn't the objective function (and gradient function if you've provided it) free memory every time a function evaluation is complete?

Anyway, if you're right about the problem being related to the Hessian not being positive definite, you should try the trust region algorithm. There's nothing in that algorithm that requires a positive definite Hessian.