Evolution Strategies Converges to Finite Differences
I am in the process of rewriting this paper.
This paper investigates the between the Evolution Strategies method applied by a team lead by Tim Salimans at OpenAI and the class of algorithms which are known as "finite differences gradient approximators". This subject has already been investigated by several groups. In the paper we prove that as the dimension of the object being optimized increases, Evolution Strategies converges to a finite differences method. Establishing this result requires several changes in the standard understanding of a "finite differences" method, which are explored in greater depth in the coming version of the paper. In brief, we contend that the success of Salimans' Evolution Strategies should be interpreted as a coup for traditional finite differences, rather than a success of evolutionary or genetic methods.
Abstract: Since the debut of Evolution Strategies (ES) as a tool for Reinforcement Learning by Salimans et al. 2017, there has been interest in determining the exact relationship between the Evolution Strategies gradient and the gradient of a similar class of algorithms, Finite Differences (FD).(Zhang et al. 2017, Lehman et al. 2018) Several investigations into the subject have been performed, investigating the formal motivational differences(Lehman et al. 2018) between ES and FD, as well as the differences in a standard benchmark problem in Machine Learning, the MNIST classification problem(Zhang et al. 2017). This paper proves that while the gradients are different, they converge as the dimension of the vector under optimization increases.