Hello
AskProgramming . This question is probably trivial but I wonder because I come from a non-programming background and this semester I'm taking a course on Optimization. It's been math and programming heavy and I struggled a lot in the first assignments where we had to write our own codes to minimize the objective functions even if I understood the math behind the methods (at that time it was single and multivariable nonconstrained optimization so things like Golden-section, Conjugate gradient, Gradient descent, Quasi-Newton methods and so on) due to the fact that for me in the past programming has been mainly a hobby and my ability to translate those equations into code is subpar for not saying straight up bad.
Fast forward a few weeks and for the subsequent assignments the instructor allows us to use existing code and functions to solve the problems and I have been doing much better. For this particular case we're dealing with linear programming, sequential quadratic programming and penalty function methods to solve linear and nonlinear constrained optimization problems. I have used functions from MATLAB and from users who posted them online and I have been able to obtain the results I wanted after some time debugging and changing the code for my particular needs.
Now my question is for someone who is interested in this field (or in other highly mathematical heavy fields such as CFD), how important it is to be able to write code from scratch vs modifying and using existing code, functions and libraries from someone else to do the job?
Here's a dropbox link to a simple data set:
https://dl.dropboxusercontent.com106825941/IPData.tar.gz It's about 84,000 examples polled from an cart-and-pendulum simulation. The columns are:
position theta velocity angular-velocity force-applied-to-cart value
...Where "value" is a simple objective function whose inputs are all taken from the first four columns (x, theta, v, w). All inputs are scaled such that their mean is 0 and they range more or less within [-3, 3]. The output is scaled such that the mean is 0.5, and all values fall within the interval [0.2, 0.8].
A 5-25-1 feedforward network tasked with learning the value function and trained in Matlab can converge on an almost perfect solution with Levenberg-Marquardt or Scaled Conjugate Gradient descent very quickly. However, using very similar network architecture in my own code (one difference being that my output neuron is a sigmoid, while Matlab's output neuron is linear) SGD and RMSprop fail to converge to a good answer. I've tried minibatches with SGD, using the entire dataset per epoch, and lots of different learning rates and learning rate decay values. I've spent a similar amount of time tweaking hyperparameters with RMSProp.
RISO's LBFGS implementation also fails with this dataset, although I haven't put as much time into playing with it.
I see three possibilities:
1.) A bug in my code. This is of course the thing I've been most suspicious of, but I'm begining to doubt this is the cause. My code passes
this test, and successfully extracts gabor shapes from natural images when used for autoencoding. It also passes simpler tests, like learning XOR
* .
2.) Something about this dataset is particularly difficult for stochastic methods. This seems unlikely; if you put together a scatter plot of value-vs-theta-vs-omega, you can see that it's a rather simple structure.
3.) SGD and RMSProp are incredibly sensitive to hyperparameter values, or perhaps weight initialization, and I've just been setting them wrong. Right now I'm initializing weights with a uniform random variable -1
I'm hoping someone can give me some insight into why my SGD and RMSProp are failing here. This should be an easy problem, but I can't find anything to point to that's demonstrably wrong.
*: In regards to XOR, my code also seems to be very senstve to hyperparamters when solving this. A 2-3-1 network needs over 10000 iterations to converge, and won't do so if the batch size is anything other than 1. Starting froma configuration that converges, and reducing the learning rate by a decade and also increasing the number of training iterations by a decade does not result in a network that also converges. This seems wrong.
OutlineOptimization over a SubspaceConjugate Direction MethodsConjugate Gradient AlgorithmNon-Quadratic Conjugate Gradient Algorithm Optimization over a Subspace Consider the problem minf(x) subject to x 2x 0 + S; where f : Rn!R is continuously di erentiable and S is the subspace S := Spanfv 1;:::;v kg. If V 2Rn k has columns v 1;:::;v k, then MATLAB package of iterative regularization methods and large-scale test problems. Numerical Methods and Optimization project: Conjugate Gradient for Singular Systems This is the source code for my blog post on the Conjugate Gradient algorithm. conjugate-gradient Updated Jun 13, 2019; HTML I want to solve a system of linear equations, AX = B, where A is sparse and positive definite. B is a matrix rather than a column vector. So I have to solve multiple system of linear equations (with multiple right hand sides). How can I use conjugate gradient for this in Matlab? I can use the one that works for a column vector B. Conjugate gradient optimizer for the unconstrained optimization of functions of n variables. Add a description, image, and links to the conjugate-gradient-optimization topic page so that developers can more easily learn about it There is nothing terribly special about its implementation of conjugate gradient optimization. It is a translation of Matlab code originally written by Carl Edward Rasmussen , with some minor modifications to allow for different convergence criteria, and to reset to steepest descent under more conditions if desired, e.g. based on orthogonality tests or if the 'beta' update parameter becomes The conjugate gradient method aims to solve a system of linear equations, Ax=b, where A is symmetric, without calculation of the inverse of A. It only requires a very small amount of membory, hence is particularly suitable for large scale systems. It is faster than other approach such as Gaussian elimination if A is well-conditioned. For example, The following matlab project contains the source code and matlab examples used for conjugate gradient. The source code and files included in this project are listed in the project files section, please make sure whether the listed source code meet your needs there. Get MATLAB; File Exchange. Toggle Sub Navigation. Search File Exchange. File Exchange. Support; MathWorks; Conjugate Gradient Method. version 1.0.0 (2.34 KB) by Narayan Das Ahirwar. Unconstrained Optimization Problem Gradient descent is typically run until either the decrease in the objective function is below some threshold or the magnitude of the gradient is below some threshold, which would likely be more than one iteration. The factor of 1/(2*m) is not be technically correct.
