Make Stochastic Gradient Descent Fast Again (Ep. 113)

<p>There is definitely room for improvement in the family of algorithms of stochastic gradient descent. In this episode I explain a relatively simple method that has shown to improve on the Adam optimizer. But, watch out! This approach does not generalize well.</p> <p>Join our <a href='https://discord.gg/4UNKGf3'>Discord channel</a> and chat with us.</p> <p> </p> References <ul><li><a href='https://koaning.io/posts/more-descent-less-gradient/'>More descent, less gradient</a></li> <li><a href='https://en.wikipedia.org/wiki/Taylor_series'>Taylor Series</a></li> </ul> <p> </p>