A note on solving nonlinear optimization problems in variable precision

Serge Gratton (1, 2) , Ph L Toint (3)

1 IRIT-APO - Algorithmes Parallèles et Optimisation
2 Toulouse INP - Institut National Polytechnique (Toulouse)
3 NaXys - Namur Center for Complex Systems [Namur]

This short note considers an efficient variant of the trust-region algorithm with dynamic accuracy proposed Carter (1993) and by Conn, Gould and Toint (2000) as a tool for very high-performance computing, an area where it is critical to allow multi-precision computations for keeping the energy dissipation under control. Numerical experiments are presented indicating that the use of the considered method can bring substantial savings in objective function's and gradient's evaluation "energy costs" by efficiently exploiting multi-precision computations. 1 Motivation and objectives Two recent evolutions in the field of scientific computing motivate the present note. The first is the growing importance of deep-learning methods for artificial intelligence, and the second is the acknowledgement by computer architects that new high-performance machines must be able to run the basic tools of deep learning very efficiently. Because the ubiquitous mathematical problem in deep learning is nonlinear nonconvex optimization, it is therefore of interest to consider how to solve this problem in ways that are as efficient as possible on new very powerful computers. As it turns out, one of the crucial aspects in designing such machines and the algorithms that they use is mastering energy dissipation. Given that this dissipation is approximately proportional to chip surface and that chip surface itself is approximately proportional to the square of the number of binary digits involved in the calculation [19, 31, 22, 26], being able to solve nonlinear optimization problems with as few digits as possible (while not loosing on final accuracy) is clearly of interest. This short note's sole purpose is to show that this is possible and that algorithms exist which achieve this goal and whose robustness significantly exceed simple minded approaches. The focus is on unconstrained nonconvex optimization, the most frequent case in deep learning applications. Since the cost of solving such problems is typically dominated by that of evaluating the objective function (and derivatives if possible), our aim is therefore to propose

Mots clés: nonlinear optimization; inexact evaluations; multi-precision arithmetic; high-performance computing

Domaines: Analyse numérique [math.NA]

Serge Gratton, Ph L Toint. A note on solving nonlinear optimization problems in variable precision. Computational Optimization and Applications, 2020, pp.1-15. ⟨hal-02943237⟩