I am trying to code this optimizer for a linear regression model. What i want to confirm from is that the update of model parameter are happening even if they cause the increase in cost function, isn't ?
Or we only update the coefficients values if they decreased the value of the cost function?