I am training a model (Recurrent Neural Network) to classify 4 types of sequences. As I run my training I see the training loss going down until the point where I correctly classify over 90% of the samples in my training batches. However a couple of epochs later I notice that the training loss increases and that my accuracy drops. This seems weird to me as I would expect that on the training set the performance should improve with time not deteriorate. I am using cross entropy loss and my learning rate is 0.0002.  Update: It turned out that the learning rate was too high. With low a low enough learning rate I dont observe this behaviour. However I still find this peculiar. Any good explanations are welcome as to why this happens
In  it says     To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good approximation. If we don't, we might end up with $\Delta C&gt;0$, which obviously would not be good! At the same time, we don't want $\eta$ to be too small, since that will make the changes $\Delta v$ tiny, and thus the gradient descent algorithm will work very slowly. In practical implementations, $\eta$ is often varied so that Equation (9) remains a good approximation, but the algorithm isn't too slow. We'll see later how this works.   But just a few paragraphs before we established that $\Delta C\approx−\eta\nabla C⋅\nabla C=−\eta\|\nabla C\|^2$ is obviously always negative (for positive $\eta$). So how can $\Delta C$ be positive if we don't choose a small enough learning rate? What is meant there?
In basic chemistry, we are taught that an atom has roughly the same number of neutrons and number of protons, this doesn't seems to hold for larger atoms, but it is always roughly proportional (i.e. you seldom find an atom with 100 protons but 1 neutron, that just does not happen). Why is that? Although elements like &lt;div&gt;s normally grow to fit their contents, using the float property can cause a startling problem for CSS newbies: If floated elements have non-floated parent elements, the parent will collapse. For example:   &lt;div&gt;   &lt;div style="float: left;"&gt;Div 1&lt;/div&gt;   &lt;div style="float: left;"&gt;Div 2&lt;/div&gt; &lt;/div&gt;    The parent div in this example will not expand to contain its floated children - it will appear to have height: 0. How do you solve this problem? I would like to create an exhaustive list of solutions here. If you're aware of cross-browser compatibility issues, please point them out. Solution 1 Float the parent.   &lt;div style="float: left;"&gt;   &lt;div style="float: left;"&gt;Div 1&lt;/div&gt;   &lt;div style="float: left;"&gt;Div 2&lt;/div&gt; &lt;/div&gt;    Pros: Semantic code. Cons: You may not always want the parent floated. Even if you do, do you float the parents' parent, and so on? Must you float every ancestor element? Solution 2 Give the parent an explicit height.   &lt;div style="height: 300px;"&gt;   &lt;div style="float: left;"&gt;Div 1&lt;/div&gt;   &lt;div style="float: left;"&gt;Div 2&lt;/div&gt; &lt;/div&gt;    Pros: Semantic code. Cons: Not flexible - if the content changes or the browser is resized, the layout will break. Solution 3 Append a &quot;spacer&quot; element inside the parent element, like this:   &lt;div&gt;   &lt;div style="float: left;"&gt;Div 1&lt;/div&gt;   &lt;div style="float: left;"&gt;Div 2&lt;/div&gt;   &lt;div class="spacer" style="clear: both;"&gt;&lt;/div&gt; &lt;/div&gt;    Pros: Straightforward to code. Cons: Not semantic; the spacer div exists only as a layout hack. Solution 4 Set parent to overflow: auto.   &lt;div style="overflow: auto;"&gt;   &lt;div style="float: left;"&gt;Div 1&lt;/div&gt;   &lt;div style="float: left;"&gt;Div 2&lt;/div&gt; &lt;/div&gt;    Pros: Doesn't require extra div. Cons: Seems like a hack - that's not the overflow property's stated purpose. Comments? Other suggestions? How do I add a notification for process completions How can I add a notification of some sort (like playing a .wav file or creating a pop-up on the status bar) when a process finishes. For example, I am compiling a program that takes a couple of hours to finish. I would like to hear/see some sort of message when it completes compiling. Is there a tool for doing this (like tying an alarm program to the pid of a process) or something like that? Are nested functions a bad thing in gcc ? How to determine the value of a domain name? I'd like to register a second-level domain with the .com extension and I noticed that it has already been registered and the owner put it for sale and let the possibility to make an offer in their site or by contacting them.  I saw that domains are generally sold for extremely different prices, so I'm confused.  Are there any general criteria to decide how much I should offer to purchase a domain? For $p$ prime, show that $\Phi_{p^n}(x) = 1 + x^{p^{n-1}} + x^{2p^{n-2}} + \dots +x^{(p-1)^{p^n-1}}$ Well I attempted to try this but I failed to solve it: So $$\Phi_{p^{n}}(x)= \frac{x^{p^{n}} - 1}{\Phi_{1}(x) \Phi_{p^{2}}(x) \dots \Phi_{p^{n-1}}(x)}$$  Now I'm just stuck here. I saw a result somewhere on the net that said $\Phi_{p^n}(x) = \Phi_p(x^{p^{n-1}})$. I would just use that, but I don't see why that result is true or know how to show that's true.  How do I approach this? Find remainder when $f(x^{12})$ is divided by $f(x)$ Book series about preparation for a war at the Singularity I am looking for a book series (almost certainly a trilogy) that starts with some humans in a space station being abducted by a small funny-looking alien and taken to a prison planet. They find out that the way space travel works is by creating copies of the passengers, and the originals remained on the space station. This is actually good for the aliens since they believe every sentient being will resurrect at the Singularity and will be forced to fight a battle, so the more the merrier.  The one other detail I remember is that the world economy has a rampant inflation, so people take any cash they have at the end of the day and buy 20th-century memorabilia that doesn't devalue as much. I am trying to mount my ext4 (sdc6) partition from a live usb because that's how I deleted two unused partitions left from a prior install of ubuntu, using GParted. The swap is mounted. However, I can't edit fstab, or find anywhere to click in nautilus to mount sdc6. In the terminal, I get the error: "can't find /dev/sdc6 in fstab". I'm afraid to exit the usb and update grub if that partition isn't mounted. Is there any easy way to do this, or do I have to install gksu or some other editor? Or do I need to mount it as sda6?  Device     Boot Start     End Sectors  Size Id Type /dev/sda1  *     2048 3905535 3903488  1.9G  b W95 FAT32  Device     Boot     Start        End   Sectors   Size Id Type /dev/sdc1            2048   29362175  29360128    14G 27 Hidden NTFS WinRE /dev/sdc2  *     29362176   29566975    204800   100M  7 HPFS/NTFS/exFAT /dev/sdc3        29566976  487023432 457456457 218.1G  7 HPFS/NTFS/exFAT /dev/sdc4       487024638 1250263039 763238402   364G  5 Extended /dev/sdc5       728393728  740290559  11896832   5.7G 82 Linux swap / Solaris /dev/sdc6       487024640  728391679 241367040 115.1G 83 Linux  Partition table entries are not in disk order. When I compile the Python code below, I get      IndentationError: unindent does not match any outer indentation level     import sys  def Factorial(n): # Return factorial     result = 1     for i in range (1,n):         result = result * i     print "factorial is ",result     return result   Why? What does an unprocessed RAW file look like? How can you tell when a roast is done? How can you tell when a roast is done?  How long should they roast for? Running Synaptic I get the following error message:  E: The package hl1440lpr needs to be reinstalled, but I can't find an archive for it. E: Internal error opening cache (1). Please report.   Upon accepting the message, Synaptic quits. How do I remove "Ubuntu" in the bios boot menu? (UEFI) How to prove sum of powers property of roots of unity? We know that $1+\alpha_1+\alpha_2+...+\alpha_{n-1}=0$ where $\alpha_i$ are the roots of $z^n=1$. How can I prove that: $$1+\sum_{i=1}^{n-1}\alpha_i^m=\begin{cases}0\quad m\in Z,m\not\equiv0\pmod n\\n\quad m\in Z,m\equiv0\pmod n\\\end{cases}$$ For each of the following, give an example of a sequence of functions $f_n$ that converges to f  A. uniformly but not in the mean square sense.  B. in the mean square sense but pointwise nowhere.  I know that for part A the domain of the function cannot be bounded but other then that I'm not sure how to go about the problem Is there a way to copy maintenance plans from one SQL Server 2012 to another? Both servers are not accessible via the same copy of Management Studio at the same time (on different networks). Creation parameters menu greyed out How can I prevent inline math formulas from overflowing into the margin? When I compile my document in LaTeX, some formulas in the document in math mode, between $ $, go into the right margin area beyond all the text.  Is there a way to fix this aesthetic problem? I'm having serious problems installing the Broadcom drivers for Ubuntu. It worked perfectly on my previous version, but now, it is impossible.   What are the steps to install Broadcom wireless drivers for a BCM43xx card?  I'm a user with no advance knowledge in Linux, so I would need clear explanations on how to make, compile, etc.    lspci -vnn | grep Network showed:   Broadcom Corporation BCM4322 802.11a/b/g/n Wireless LAN Controller [14e4:432b]      iwconfig showed:  lo        no wireless extensions. eth0      no wireless extensions.   NOTE: Answer below is updated every time new information is added and confirmed working.