For what values of learning rate 𝑎 does gradient descent converge and its ideal value for quadratic functions.

Mathematics AA HL Internal Assessment For what values of learning rate does gradient descent converge and its ideal value for𝑎 quadratic functions. 20 Pages 1

Introduction Simply put, gradient descent is an algorithm that estimates the coordinates of where a function outputs the lowest value. I initially discovered the gradient descent algorithm while I started working on a machine learning project that involved predicting road traffic and cross walk safety as I found neural networks and ML to be a very interesting topic at the time, especially when artificial intelligence initially became a topic of everyday conversation. As the project was merely for my entertainment and to satisfy my curiosity, I never investigated the parameters that overarch the training of my neural network. As a result the model converged extremely slowly as I chose a small learning rate to guarantee that it will converge. In gradient descent, if the learning rate is too high, it will instead diverge. At the time, I only used a python library that already implemented gradient descent, so it was a black box that meant nothing to me. I’m now curious about the mathematics behind the algorithm as it is the basis of how weights in a neural network are optimized. I’m deciding to study the algorithm mathematically as I want to discover whether there is a perfect learning rate, and if so, can we calculate it? I want to investigate the ideal learning rate parameter for different quadratic functions. Gradient descent is the algorithm that trains all neural networks. It is fundamental to the creation of all neural networks. It is behind things like image recognition machine learning models and language translation models like in google translate. It slowly adjusts the millions of parameters in each of the machine learning models that make them output increasingly accurate predictions after each adjustment for a certain set of input parameters. When we choose a poor learning rate, let;s say too high, each of the adjustments gradient descent makes may cause the algorithm to diverge instead–it overshoots the minima of the functions and begins oscillating and not converging to the optimum. If it is too low, then the neural network will take much longer to train. Doing a mathematical analysis of learning rates in gradient descent through deriving relationships between 2

the variables in the algorithm that I can prove theoretically. Then, I want to verify it computationally after doing the theoretical analysis to ensure that my findings align with my practical results. To conduct my research, I will be basing it around using gradient descent on quadratic functions because they are simple enough to derive exact solutions. I can prove everything rigorously as I can derive closed form solutions while still covering the behavior essential for optimization problems. Gradient descent usually works to optimize multiple parameters. Quadratic functions also are a great middleground as near minima, functions often behave similar to a quadratic function like in the taylor series, the quadratic term becomes dominant when the first derivative nears zero at the minimum. Background The minimum of a function occurs when the derivative of a function equals 0, . In𝑓(𝑥) 𝑓'(𝑥) = 0 multivariate environments, the vector that holds all of the partial derivatives of is , it𝑓(𝑥1, 𝑥2, …, 𝑥3) ∇𝑓(𝑥) contains the direction of the steepest ascent. Its magnitude , contains the “steepness” in that direction||∇𝑓(𝑥)|| (GeeksforGeeks). While doing machine learning, it is impractical to solve for with millions of variables analytically, so we need∇𝑓(𝑥) = 0 an iterative approach instead. This is where gradient descent comes in. Gradient descent is an iterative algorithm that finds the minima of functions. It essentially follows the negative gradient and goes “downhill”, seen in Figure 1. It does this by updating with the rule where is the current position at iteration , is the next position at iteration𝑥𝑘+1 = 𝑥𝑘 − α∇𝑓(𝑥𝑘) 𝑥𝑘 𝑘 𝑥𝑘+1 , is the gradient at the current position, and is the learning rate(Ruder). A gradient is a vector𝑘 + 1 ∇𝑓(𝑥𝑘) α that contains all of the partial derivatives of , (Amari). So, we update our∇𝑓 ∇𝑓 = [ ∂𝑓 ∂𝑥1 , ∂𝑓 ∂𝑥2 , ..., ∂𝑓 ∂𝑥𝑛 ] position by subtracting it by the vector that points to the steepest ascent, allowing us to descend the slope. For a 3

single variable function, the rule becomes . The process begins with . It computes the𝑥𝑘+1 = 𝑥𝑘 − ∇𝑓'(𝑥𝑘) 𝑥0 gradient at that point, uses the rule to update the position, and repeats until we reach a stopping condition. The learning rate, , in the update rule is the coefficient that scales the vector , which means that itα ∇𝑓 controls the magnitude of how much the position is updated in each iteration(Amari). It’s the step size in each iteration. When is too small, it takes more iterations to reach the minimum, while if is too large, theα α algorithm either oscillates, or overshoots the minimum or completely diverges. This comes with the question of whether there is a sweet spot for the learning rate that optimizes the rate of convergence and reliability/stability. I aim to answer this by deriving the optimal learning rate for quadratic functions analytically. Convergence is when the algorithm causes to approach the minimum denoted . In the case of𝑥𝑘 𝑥* investigating parabolas, . Convergence can be defined as for when the iterations𝑥* = 0 𝑘 ∞ lim → 𝑥𝑘 = 𝑥* = 0 converges to a point at . When investigating convergence with an iterative function, we need to have a𝑥* stopping criteria as we cannot run iterations to infinity. Often, the threshold is when , where||∇𝑓(𝑥𝑘)|| < ε is the magnitude of the when is a threshold for how close can be to 0 before we stop. It||∇𝑓(𝑥𝑘)|| ε ||∇𝑓(𝑥𝑘)|| means that when the function has found a value near to 0, within the threshold where there is no significantε descent left, we stop gradient descent as it is considered complete. For our single variable functions, the stopping condition should be , so when the gradient is below the threshold . This works because when the|𝑓'(𝑥𝑘)| < ε ε gradient is a small value, the function is near the minimum(Zhang), so it indicates our proximity to .𝑥* Quadratic functions are written in the form . Importantly, they have a minimum at (0,0)𝑓(𝑥) = 𝑎𝑥2 and are convex functions when which guarantees that we find the global minimum if gradient descent𝑎 > 0 converges. The derivative of the quadratic function in this form is , which makes it possible to𝑓'(𝑥) = 2𝑎𝑥 easily derive a closed form solution. 4

Analysis of 𝑓(𝑥) = 𝑎𝑥2 Analytical Approach to Find Convergence Conditions The investigation will start with a mathematical approach to analytically derive the optimal learning rate for gradient descent. First, the update rule for gradient descent should be applied to the function :𝑓(𝑥) = 𝑎𝑥2 General update rule for gradient descent: 𝑥𝑘+1 = 𝑥𝑘 − α∇𝑓(𝑥𝑘) In this case, the term can be for a single variable function because whichα∇𝑓(𝑥𝑘) α𝑓'(𝑥𝑘) α∇𝑓(𝑥𝑘) = [ ∂𝑓 ∂𝑥1 ] reduces to . Now we need the derivative of :α 𝑑𝑓 𝑑𝑥1 = α𝑓'(𝑥𝑘) 𝑓(𝑥𝑘) = 𝑎𝑥𝑘 2 and we substitute it into the update rule and the update rule for becomes:𝑓'(𝑥𝑘) = 2𝑎𝑥𝑘 𝑓(𝑥) = 𝑎𝑥2 and factor out :𝑥𝑘+1 = 𝑥𝑘 − 2α𝑎𝑥𝑘 𝑥𝑘 which is a geometric sequence.𝑥𝑘+1 = 𝑥𝑘(1 − 2α𝑎) Lets say that and recursively apply gradient descent to find the closed form solution:𝑟 = (1 − 2α𝑎) 𝑥𝑘+1 = 𝑟𝑥𝑘 𝑥1 = 𝑟𝑥0 𝑥2 = 𝑟𝑥1 𝑥3 = 𝑟𝑥2 5

Which reveals a pattern when we substitute into following functions that allows us to generalize:𝑥𝑛 and𝑥2 = 𝑟 · 𝑟𝑥0 𝑥3 = 𝑟 · 𝑟 · 𝑟𝑥0 Which means that for each iteration, we are multiplying by or . So after iterations, we𝑥0 𝑟 (1 − 2α𝑎) 𝑘 are multiplying by times. This allows us to find the closed form solution which is:𝑥0 (1 − 2α𝑎) 𝑘 𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 This equation expresses the position of iteration in terms of directly. We do not need to calculate𝑘 𝑥0 the intermediate values. Proof by Induction of Analytical Approach I will prove by induction that for all .𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑘 ∈ ℤ+ Base case: When .𝑘 = 0 𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑥0 = 𝑥0(1 − 2α𝑎)0 𝑥0 = 𝑥0 · 1 ✓𝑥0 = 𝑥0 6

This is true because of the definition of , so the base case holds true.𝑥0 Now we assume that for any , is true. If the formula is true for then is𝑘 ≥ 0 𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑘 should be true for . This shows the formula holds for :𝑘 + 1 𝑘 + 1 𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑥𝑘+1 = (𝑥0(1 − 2α𝑎)𝑘)(1 − 2α𝑎) we apply the the exponent rule :𝑎𝑚 · 𝑎𝑛 = 𝑎𝑚+𝑛 ✓𝑥𝑘+1 = 𝑥0(1 − 2α𝑎)𝑘+1 Which proves the formula holds for .𝑘 + 1 By principle of mathematical induction, because it holds true for and if it holds for and𝑘 = 0 𝑘 , the expression holds true for all .𝑘 + 1 𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑘 ∈ ℤ+ 7

Determining The Conditions for Convergence Now that has been derived and the form proven, we need to determine when the𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 function converges. Convergence is when . So when we apply this to the expression 𝑘 ∞ lim → 𝑥𝑘 = 0 , then . Now, because is a constant of the initial position, the𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 𝑘 ∞ lim → 𝑥0(1 − 2α𝑎)𝑘 = 0 𝑥0 behavior of the limit is determined by the term .(1 − 2α𝑎)𝑘 A geometric sequence will converge to 0 when when the base, , is . So if𝑢1𝑟𝑛 𝑛 → ∞ 𝑟 − 1 < 𝑟 < 1 we apply this, then: , and simplify− 1 < 1 − 2α𝑎 < 1 , then flip the inequality after multiplying by -1− 2 <− 2α𝑎 < 0 0 < 2α𝑎 < 2 0 < α𝑎 < 1 So when , gradient descent will eventually converge to . It can also be written:0 < α𝑎 < 1 𝑥* 0 < α < 1 𝑎 For any other value outside of this range, gradient descent will either diverge or remain stationary.α This reveals that the learning rate must be a positive value, to iterate towards the minimum, and the0 < α learning rate must be , not too large to avoid divergence from . It also reveals that the upper boundα < 1 𝑎 𝑥* depends on the vertical dilation of the parabola. The steeper the parabola, the smaller the maximum learning rate. These are the necessary conditions for gradient descent to converge on a parabola .𝑓(𝑥) = 𝑎𝑥2 8

Determining the Optimal Learning Rate Now that we have found the bounds for the learning rate that allow it to converge at all, we need to determine what value of in that range is the ideal. Within the bounds for the learning rateα 0 < α < 1 𝑎 different learning rates converge at different speeds. The optimal learning rate should be defined as the value of α in which the number of iterations that are required to reach convergence is minimal. We have the closed form solution and from this, we can see that the common𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 ratio of the geometric sequence formed is , so the term determines how fast shrinks.(1 − 2α𝑎) (1 − 2α𝑎)𝑘 𝑥𝑘 Each iteration of the algorithm multiplies the current position by the term , which means that if the(1 − 2α𝑎) term is near 1, the function will converge slower than in the term is near 0. Since|1 − 2α𝑎| |1 − 2α𝑎| multiplying by a number closer to 0 allows for values to shrink rapidly, it means that we have to minimize in order to minimize the amount of iterations required to reach .|1 − 2α𝑎| 𝑥𝑘 Let’s consider the function . We need to find the range of the function within the𝑓(α) = |1 − 2α𝑎| domain . So then, we should find the sign of the function in the key areas of the function0 < α < 1 𝑎 :𝑔(α) = 1 − 2α𝑎 Left Boundα = 0 𝑓(0) = 1 − 2(0)𝑎 𝑓(0) = 1 𝑓(0) = 1 Mid Pointα = 1 2𝑎 𝑓( 1 2𝑎 ) = 1 − 2( 1 2𝑎 )𝑎 𝑓( 1 2𝑎 ) = 1 − 2𝑎 2𝑎 9

𝑓( 1 2𝑎 ) = 1 − 1 𝑓( 1 2𝑎 ) = 0 Right Boundα = 1 𝑎 𝑓( 1 𝑎 ) = 1 − 2( 1 𝑎 )𝑎 𝑓( 1 2𝑎 ) = 1 − 2𝑎 𝑎 𝑓( 1 2𝑎 ) = 1 − 2 𝑓( 1 2𝑎 ) =− 1 And since the function is linear, we can conclude that for the function when1 − 2α𝑎 0 < α < 1 2𝑎 the function is always positive, while for , it is negative. Therefore, for the function 1 2𝑎 < α < 1 𝑎 , the minimum is at the vertex where . Now, we can try plugging it𝑓(α) = |1 − 2α𝑎| α = 1 2𝑎 𝑓( 1 2𝑎 ) = 0 into the closed form solution that we derived: then we substitute for𝑥𝑘 = 𝑥0(1 − 2α𝑎)𝑘 1 2𝑎 α 𝑥𝑘 = 𝑥0(1 − 2 1 2𝑎 𝑎)𝑘 𝑥𝑘 = 𝑥0(1 − 2𝑎 2𝑎 )𝑘 𝑥𝑘 = 𝑥0(1 − 1)𝑘 and now, when ,𝑥𝑘 = 𝑥0(0)𝑘 𝑘 > 0 𝑥𝑘 = 0 This means that for when gradient descent converges in 1 iteration. Let’s call the optimal learning rateα = 1 2𝑎 α𝑜𝑝𝑡 = 1 2𝑎 10

Convergence Rate for Suboptimal α While the optimal value converges in one iteration, while using gradient descent in practicalα𝑜𝑝𝑡 = 1 2𝑎 applications, we may not use the optimal learning rate, especially as the functions used while performing gradient descent in ML applications are much more complex and model a dataset of which their loss functions may not have an easily derivable optimal learning rate or minimum analytically, which is the point of the iterative method. I will analyze how the number of iterations required for convergence varies with change in .α First, I need to derive an expression for the number of iterations required to converge as a function of .α We have the closed form solution and we need to find the number of iterations required𝑥𝑘 = 𝑥0(1 − 2 1 2𝑎 𝑎)𝑘 for . We can do:|𝑥𝑘| < ε , where|𝑥𝑘| < ε |𝑥𝑘| = |𝑥0||1 − 2 1 2𝑎 𝑎|𝑘 and then solve for .|𝑥𝑘| = |𝑥0||1 − 2 1 2𝑎 𝑎|𝑘 < ε 𝑘 |1 − 2 1 2𝑎 𝑎|𝑘 < ε |𝑥𝑘| |1 − 2 1 2𝑎 𝑎|𝑘 < ε |𝑥𝑘| 𝑘 · 𝑙𝑛|1 − 2 1 2𝑎 𝑎| < 𝑙𝑛 ε |𝑥𝑘| And we have so which means that|1 − 2α𝑎| < 1 𝑙𝑛|1 − 2α𝑎| < 0 and the number of iterations is then:𝑘 > 𝑙𝑛 ε |𝑥𝑘| 𝑙𝑛|1−2α𝑎| or approximately𝑘 = 𝑙𝑛 ε |𝑥0| 𝑙𝑛|1−2α𝑎| because must be an integer𝑘 ≈ 𝑙𝑛 ε |𝑥0| 𝑙𝑛|1−2α𝑎| 𝑘 11

The analysis of this reveals that when: , the convergences happens after the first step whenα = α𝑜𝑝𝑡 = 1 2𝑎 𝑘 = 1 , then k increases as decreases, which means convergence is slower, but there is noα < α𝑜𝑝𝑡 = 1 2𝑎 α overshooting. , then k increases as decreases, which means convergence is slower, and there is overshooting,α > α𝑜𝑝𝑡 = 1 2𝑎 α causing an oscillating motion. The function still converges. , at the boundary, there is no convergence as . Meaning that for each iteration, we areα = 1 𝑎 |1 − 2α𝑎| = 1 multiplying by 1, causing no movement; therefore no convergence.𝑥0 This also confirms that the value is optimal. We can also predict that when looking at aα𝑜𝑝𝑡 = 1 2𝑎 graph of in terms of , it will be a rough U shape, as seen in Figure 2 with a vertex at which I will𝑘 α α = 1 2𝑎 verify computationally. 12

Breakdown of Analytical Findings The formula for the optimal learning rate reveals that we have an inverse relationshipα𝑜𝑝𝑡 = 1 2𝑎 between the coefficient of , , and the optimal learning rate :𝑥2 𝑎 α𝑜𝑝𝑡 𝑎 = 0. 5 → α_𝑜𝑝𝑡 = 1 2·0.5 = 1 𝑎 = 1 → α_𝑜𝑝𝑡 = 1 2·1 = 0. 5 𝑎 = 2 → α_𝑜𝑝𝑡 = 1 2·2 = 0. 25 𝑎 = 4 → α_𝑜𝑝𝑡 = 1 2·4 = 0. 125 And it shows how steeper parabolas, with a larger coefficient , will require smaller learning rates to𝑎 converge optimally. This is because when the coefficient is larger, the gradient produced is steeper, meaning𝑎 that algorithm will take larger steps because it steps in the vector of the negative gradient. Mathematically, determines the step size, so for optimal convergence,α·𝑓'(𝑥) = α·(2𝑎𝑥) = 2α𝑎·𝑥 , which also gives regardless of the current position . The convergence bound is2α𝑎 = 1 α = 1/(2𝑎) 𝑥𝑘 defined by this step size. The convergence bound is and the optimal , is exactly half of the0 < α < 1 𝑎 α α𝑜𝑝𝑡 upper bound . This means the optimal value is exactly at the midpoint of which makesα𝑜𝑝𝑡 = 1 2𝑎 1 − 2α𝑎 sense because the optimal learning rate balances the behavior of both of the upper and lower convergence boundaries, neither undershooting nor overshooting. I will verify this computationally. 13

Computational Analysis of The Relationship Between and𝑎 α Experimental Setup I created an application in python that allows me to set parameters for gradient descent and run many trials of gradient descent at different values for in order to determine the value of computationally. Theα α𝑜𝑝𝑡 parameters that will remain consistent for all of the trials are , but I will be changing𝑥₀ = 10, ε = 0. 001 the values for : . My implementation of gradient descent initiates the position at , then𝑎 {0. 5, 1, 5} 𝑥₀ = 10 computes the gradient at , , then it uses the update rule to update the𝑥𝑘 𝑓(𝑥𝑘) = 2𝑎𝑥𝑘 𝑥𝑘+1 = 𝑥𝑘 − α𝑓'(𝑥𝑘) position, and then checks the criteria to stop . If the stopping criteria is not reached, gradient|𝑓'(𝑥𝑘)| < ε descent continues. The learning rate will sweep from a value of 0.001 to 2 with a step size of 0.001, then I will graph the number of iterations to converge to the learning rate. Results of Experiment Graph of Number of Iterations to Convergence to Learning Rate Logarithmic, 𝑎 = 0. 5 (Photo by author, Figure 3) 14

As we can see in Figure 3, the learning rate, when graphed logarithmically, shows a minimum at α = 1 when . This aligns with our result that the because . The result𝑎 = 0. 5 α𝑜𝑝𝑡 = 1 2𝑎 α𝑜𝑝𝑡 = 1 2·0.5 = α𝑜𝑝𝑡 = 1 of this experiment has confirmed the theoretical result that at , at least for , the algorithmα𝑜𝑝𝑡 = 1 2𝑎 α = 1 converges in exactly 1 iteration, while for other learning rates, there is a significant increase in the learning rate as the learning rate diverges from . It also shows that at the upper bound, , the algorithmα𝑜𝑝𝑡 = 1 2𝑎 α = 1 0.5 = 2 doesn’t converge, but instead diverges at the red points, confirming the theoretical results. It also reveals the U shaped curve that we expected from the expression . We can see that on the right side of the curve,𝑘 ≈ 𝑙𝑛 ε |𝑥0| 𝑙𝑛|1−2α𝑎| the learning rate is smaller, slowing down convergence very dramatically. At , at 10% the optimal, theα = 0. 1 convergence rate is significantly increased and took 88 iterations to converge in Figure 3. This shows that learning rates before optimal show a significantly increased amount of iterations to converge. The same is also true for the opposite. Learning rates that are higher than the optimal value mimic the behavior of the learning rates below the optimal, showing how the absolute difference between the learning rate and the ideal learning rate affects the amount of iterations until convergence. This is, until the learning rate goes outside of the upper bound, which causes gradient descent to diverge or stay stationary. We can also see that as the learning rate approaches each of the bounds, the learning rate begins to increase extremely rapidly, costing up to 9,000 iterations before convergence. 15

This should emphasize the importance of optimizing the learning rate, because a deviation from the by a factor of 2 can cause the gradient descent algorithm to never converge at all, and even a deviationα𝑜𝑝𝑡 smaller can cause learning rates to be orders of magnitude higher. We can check this with the expression we derived: 𝑘 ≈ 𝑙𝑛 ε |𝑥0| 𝑙𝑛|1−2α𝑎| 𝑘 ≈ 𝑙𝑛 0.001 10 𝑙𝑛|1−2(0.1)(0.5)| ≈ 88 𝑘 ≈ −9.210 𝑙𝑛|1−0.1| ≈ 88 𝑘 ≈ −9.210 𝑙𝑛|0.9| ≈ 88 𝑘 ≈ −9.210 −0.10536 ≈ 88 which rounds up, as the iteration must complete.𝑘 ≈ 87. 4145 ≈ 88 16

Now, if we look at the graphs for both , we can see other patterns in the behavior of the𝑎 = 0. 5, 1, 𝑎𝑛𝑑 5 convergence speed: Graph of Number of Iterations to Convergence to Learning Rate Logarithmic, 𝑎 = 1 (Photo by author, Figure 4) Graph of Number of Iterations to Convergence to Learning Rate Logarithmic, 𝑎 = 5 (Photo by author, Figure 5) If we compare figures 3, 4, and 5, we can see a clear pattern in the shape of the graph formed and also the shift in the bounds of the possible learning rates. We predicted that the bounds would be , and0 < α < 1 𝑎 that the upper bound would have an inverse relationship with the coefficient . This is shown to be consistently𝑎 true here, where the bound for , Figure 3, is 2, the bound for , Figure 4, is 1, the bound for𝑎 = 0. 5 𝑎 = 1 17

, Figure 5, is 0.2. We also predicted in theory that the ideal learning rate would be at the midpoint𝑎 = 5 between the two bounds, at . This remains true for all where they are 1 2𝑎 𝑎 = 0. 5, 1 , 5 α = 1, 0. 5, 0. 1 respectively. But what we see here, importantly, is the narrowing of the optimal region and the bound for convergence overall. Since the lower bound is constant, 0, and the upper bound is inversely proportional to , so𝑎 it decreases as increases, the convergence range’s width decreases. The consistent shape of the U in the graph𝑎 can also be observed. Despite the narrowing of the convergence range and the horizontal dilation, the shape of the graph remains consistent. We also consistently observe that at , there is a consistent trend of convergenceα𝑜𝑝𝑡 after only 1 iteration. We also see a consistent shift in the range of the three graphs. For , the range𝑎 = 0. 5, 1 , 5 shrinks as the value of increases.𝑎 There is a strong validation of the theoretical analysis in the experimental data. We can observe all of the patterns discussed in the theoretical analysis. The optimal learning rate shows an inverse relationship with the size of the coefficient and also consistently is at the midpoint of the bounds that allow gradient descent to𝑎 converge . We also see that the range that allows the function to converge decreases inversely to0 < α < 1 𝑎 𝑎 because of the upper bound. This means that as the functions get steeper, choosing the learning rate becomes less lenient. 18

Conclusion The investigation aimed to investigate the relationship between the learning rate and convergence in aα quadratic function with gradient descent. The investigation derived the optimal learning rate parameter and also other relationships between the properties of the quadratic function and the learning rate. We found that the optimal learning rate for a quadratic function was which minimizes the amount of𝑓(𝑥) = 𝑎𝑥2 α𝑜𝑝𝑡 = 1 2𝑎 iterations until convergence to 1. We found the conditions required for convergence in terms of the coefficient of the term which establishes the range for the learning rate that is stable for the specific parabola𝑥2 0 < α < 1 𝑎 with coefficient . Importantly, the optimal learning rate allows the function to converge within𝑎 α𝑜𝑝𝑡 = 1 2𝑎 exactly 1 iteration which is inversely proportional to coefficient . This means in practice, that for steeper𝑎 functions, the learning rate needs to be smaller, or else gradient descent will overshoot the minimum of the function and possibly not converge at all. The theoretical analysis was then verified with experimental trials, showing that in practice, all of the boundaries and the optimal learning rate were consistent with the theoretical analysis. Through the experimental trials, we also determined that the deviation from causes a significantα𝑜𝑝𝑡 increase in iteration count. This emphasizes the importance of choosing an ideal learning rate for a specific machine learning model. 19

Works Cited Amari, Shun-ichi. “Backpropagation and Stochastic Gradient Descent Method.” Neurocomputing, vol. 5, no. 4-5, June 1993, pp. 185–196, www.sciencedirect.com/science/article/abs/pii/092523129390006O, https://doi.org/10.1016/0925-2312(93)90006-o. Accessed 18 Feb. 2026. GeeksforGeeks. “What Is Gradient Descent.” GeeksforGeeks, 23 Jan. 2025, www.geeksforgeeks.org/machine-learning/what-is-gradient-descent/. Accessed 18 Feb. 2026. Rekha. “The Ascent of Gradient Descent.” Medium, 8 June 2020, blog.clairvoyantsoft.com/the-ascent-of-gradient-descent-23356390836f. Accessed 18 Feb. 2026. Ruder, Sebastian. “An Overview of Gradient Descent Optimization Algorithms.” ArXiv.org, 2016, arxiv.org/abs/1609.04747. Accessed 18 Feb. 2026. Zhang, Jiawei. Gradient Descent Based Optimization Algorithms for Deep Learning Models Training. Feb. 2019. 20