Reference:      To form a richer representation of the data, each hidden layer is   composed of multiple feature maps, $\{h^{(k)}, k=0..K\}$. The weights W   of a hidden layer can be represented in a 4D tensor containing   elements for every combination of destination feature map, source   feature map, source vertical position, and source horizontal position.   What I can't understand: let's say we slide 2 3-D filters across image, producing two feature maps.    How we can be sure, that parameters of our 2 filters won't move close to same value through backpropagation? So that we won't obtain from 3 channels (RGB) two channels, which are highly correlated?  And the more filters I want to introduce, the less obvious for me that some of them (at least) or many of them (in worst case) won't be basically similar vectors in weights space ($w_{1}, w_{2}, w_3 ... w_n .$)
56579
Based from what I have learned, we use multiple filters in a Conv Layer of a CNN to learn different feature detectors. But since these filters are applied similarly (i.e. slided and multiplied to regions of the input), wouldn't they just learn the same parameters during training? Hence the use of multiple filters would be redundant?
79831
Let $T_1,T_2$ be two linear transformations from $\mathbb{R}^n$ to $\mathbb{R}^m$. Let $\{x_1,\cdots , x_n\}$ be a basis of $\mathbb{R}^n$. Suppose that $T_1x_i \neq 0$ for every $i=1,2,\cdots, n$ and that $x_i \bot \ker T_2$ for every $i=1,2,\cdots, n$. Which of the following is/are necessarily true?   $T_1$ is invertible. $T_2$ is invertible.  both $T_1,T_2$ are invertible.  neither $T_1$ nor $T_2$ is invertible.   My Approch:  Given dimension of $\mathbb{R}=n$ and  $x_i\perp Ker(T_2)$  So, $\text{Ker}T_2=0$  Row space and null space are perpendicular to each other.  From Rank Nullity Theorem:  let T : V → W be a linear map. Then the rank of T is the dimension of the image of T and the nullity of T is the dimension of the kernel of T, so we have  $\text{dim}(\text{Im}(T))+ \text{dim}(\text{Ker}(T))=\text{dim}(V)$  Applying above result we get,  $\text{dim}(\text{Im}(T))+ 0=n$  Therefore, $T_2$ is one-one.  I could not solve this problem completely. what to do next?
39365
I have a training dataset of images: X (Visual) and Y (Infrared). Each set has $300$ training examples. I extract feature vectors from both sets of images. Thus my X and Y datasets are respectively $300\times 1920$ and $300\times 1536$ where $300$ is the number of sample observations (in  this case images) and $1920$ &amp; $1536$ are respectively the length of the feature vectors in visual and infrared spectrum.  A testing dataset consists of images of different subjects in both visual spectrum and infrared spectrum. I will have the visual spectrum data as my gallery. I will have the infrared images as the probe data. For each probe image, I need to retrieve the corresponding visual image from the gallery by using some kind of similarity measure.  Basic algorithm idea:   Start Training Phase. Read the images from visual and infrared spectrum dataset. Get the feature vectors by using desired descriptors and populate the matrices X and Y. Use CCA for subspace learning. Get the projection matrices Wx and Wy. Start Testing Phase. Given the gallery images (visual), read these images and use the Wx transformation matrix to convert them into the CCA subspace. For each probe image, convert it into the CCA subspace by using the Wy transformation and compare with each images of the gallery and compute a matching score. The image (or its label) in the gallery having the maximum score is returned.   Could anyone tell me if my approach is correct or not? Pointing me in the right direction would also be helpful. Please see the following paper for reference: .    I work in Matlab and use the following command to perform CCA:  [Wx,Wx,r,U,V] = canoncorr(X,Y);   %// DO CCA   The output I get is this :    Name           Size             Bytes   Class     Attributes    Wx         1920x297            890880  double                 Wx         1536x297            712704  double                 U           300x297             27840  double                 V           300x297             27840  double                 r             1x297               464  double          :   The projection matrices are Wx and Wy since they transform X and Y into the new space. The resulting projections of X and Y into the new space are U and V, respectively. The r vector represents the entries of the correlation matrix between U and V, which is a diagonal matrix.
64830
I wanted to check if I am understanding the concept of the support vectors correctly. Let's say I am not using any kind of kernel, and it is a hard-margin SVM.  In this case, whatever the number of the dimensions is for our features, the minimum possible number of support vectors equals to 2 (1 for +, 1 for -). Am I correct? Do we still need more support vectors even in such an ideal case in $R^{d}$?
44974
It's quite intuitive that most neural network topologies/architectures are not identifiable. But what are some well-known results in the field? Are there simple conditions which allow/prevent identifiability? For example,    all networks with nonlinear activation functions and more than one hidden layer are not identifiable all networks with more than two hidden units are not identifiable   Or things like these. NOTE:I'm not saying that these conditions prevent identifiability (though they seem pretty good candidates to me). They are just examples of what I mean with "simple conditions".  If it helps to narrow down the question, feel free to consider only feed-forward and recurrent architectures. If this is still not enough, I'd be satisfied with an answer which cover at least one architecture among MLP, CNN and RNN. I had a quick look around on the Web but it looks like the only discussion I could find was on Reddit. Come on, people, we can do better than Reddit ;-)
75890
This question can be formulated in two ways. Let there be two $d$-dimensional orthonormal bases $B_{1}$ and $B_{2}$. I refer to the elements of $B_{1}$ by $\lvert\nu_{i}\rangle$ and to the elements of $B_{2}$ by $\lvert\omega_{j}\rangle$. These two bases are mutually unbiased namely: $\lvert\langle\omega_{j}\lvert\nu_{i}\rangle\lvert^{2}=1/d$ for all $\lvert\omega_{j}\rangle$ and $\lvert\nu_{i}\rangle$. We define some vector $\lvert\tau\rangle$ such that $\lvert\tau\rangle$ is orthonormal to one element of $B_{2}$, say $\lvert\omega_{m}\rangle$ and it is mutually unbiased with respect to all the elements in $B_{1}$: $\lvert\langle\tau\lvert\nu_{i}\rangle\lvert^{2}=1/d$ for all $\lvert\nu_{i}\rangle$. Can we conclude that $\lvert\tau\rangle$ is equal to a member of $B_{2}$ say $\lvert\omega_{p}\rangle$ for some $p\neq m$ up to a phase difference?  Another way to make the same conclusion is to ask this: Let there be three orthonormal bases $B_{1}$, $B_{2}$ and $B_{3}$. $B_{1}$ and $B_{2}$ are mutually unbiased, so are $B_{2}$ and $B_{3}$. Can we conclude that $B_{1}$ and $B_{3}$ are either mutually unbiased or have equivalent Hadamard matrices? This question for me is particularly important since in the literature it is always talked about sets of pairwise mutually unbiased bases. But there's no proof given for the fact that three mutually unbiased bases should necessarily be pairwise MUBs.
37915
To use SVM or Neural Network it needs to transform (encode) categorical variables into numeric variables, the normal method in this case is to use 0-1 binary values with the k-th categorical value transformed to be (0,0,...,1,0,...0) (1 is on the k-th position). Is there other methods to do this, especially when there are a large number of categorical values(e.g.10000) such that the 0-1 representation will introduce a large number of additional dimensions(input units) in Neural Network which seems not quite desired or expected?   I am asking about general strategies.
6455
I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.  I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.  I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.  Is there any standard way of handling this sort of imbalance?
26173
Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural networks.
27437
How does the  for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
101080
hello I have a question about weight regularization in Adam apparently the weight_decay in the AdamW function  has the same impact as an L2 regularization my questions are: is that parameter the same as lambda that we have in the regularization term?  how does it exactly work? and what is its impact on the model complexity?
90200
I have been told by a professor that it is not possible to combine different activation functions within a neural network.  And I can't find any examples of anyone doing this.  However, I cannot find any good explanation why.   Conceptually, it seems to make sense:  suppose I have a set of data on some population.  Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data.  We could imagine a logistic model that maps my population data into the categorical variable.  Now suppose I'm training a neural net with one hidden layer.  It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course).  Meanwhile, all the other nodes have a linear activation function.  And then output would be a linear function of all the nodes in the hidden layer.    I have no reason to assume this would improve prediction error.  This is what I want to know:  is it possible to estimate such a model using standard approaches?
115366
For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular inputs, and these three levels split into two parts each, one for training and one for function eval. The last two parts are much closer than to the first. I'm asking about #2, which usually leads to better understanding of #3). (if those are not what 'interpretability' means then what should I be thinking?)  As far as interpretability goes, logistic regression is one of the easiest to interpret. Why did this instance pass the threshold? Because that instance had this particular positive feature and it has a larger coefficient in the model. It's so obvious!  A neural network is  the classic example of a model that is difficult to interpret. What do all those coefficients mean? They all add up in such complicated crazy ways that it is hard to say what any particular coefficient is really doing.  But with all the deep neural nets coming out, it feels like things are becoming clearer. The DL models (for say vision) seem to capture things like edges or orientation in early layers, and in later layers it seems like some nodes are actually semantic (like the proverbial ). For example:    ()  This is a graphic () created by hand for presentation so I am very skeptical. But it is evidence that somebody thinks that is how it works.  Maybe in the past there just weren't enough layers for us to find recognizable features; the models were successful, just not easy to post-hoc analyze particular ones.  But maybe the graphic is just wishful thinking. Maybe NNs are truly inscrutable.  But the many graphics with their nodes labeled with pictures are also really compelling.    Do DL nodes really correspond to features?
103703
As far as I understand, one of the main claimed problems with initializing e.g. a feed-forward neural network (with several $\text{tanh}$ or $\text{ReLU}$ layers) with $W=0$ is that it doesn't break "network symmetry", meaning, backpropagation would propagate the same error through all such units (i.e. "nudging all weights in the same direction"). This is, I presume, undesirable because we would not be learning "different" calculations through different paths of the network.  However, I'm confused why that even matters in this case given the fact that if $W$ ever drops to 0, we will effectively be propagating no gradients at all through the network, since W=0 would multiply all errors from the output and prevent any learning.  Put another way, even if $W=0$ does not break network symmetry (wasting calculations and paths in the network) is it correct to say that if $W=0$ (e.g. by initialization) we are effectively killing gradients in a neural network, and thus no learning can take place?
50890
In a VAE, the encoder learns to output two vectors:  $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$  which are the mean and variances for the latent vector $\mathbf{z}$, the latent vector $\mathbf{z}$ is then calculated by:  $$\mathbf{z} = \mu + \sigma \epsilon$$  where: $\epsilon = N(0, \mathbf{I}_{z \times z})$  The KL divergence loss for a VAE for a single sample is defined as (referenced from  and ):  $$\frac{1}{2} \left[ \left(\sum_{i=1}^{z}\mu_{i}^{2} + \sum_{i=1}^{z}\sigma_{i}^{2} \right) - \sum_{i=1}^{z} \left(log(\sigma_{i}^{2}) + 1 \right) \right]$$  Though, I'm not sure how they got their results, would anyone care to explain or point me to the right resources?
122277
While implementing Batch Normalization for a particular layer 'L' with 'n' hidden neurons/units in a Neural Network, we first normalize the Activation values of that layer using their respective Mean and Standard Deviation, and then apply the Scaling and Offset factor as shown:     X-norm = (X - mu)/sd X' = (Y * X-norm) + B       where    mu = mean of X and it is a (n,1) vector          sd = standard deviation of X and it is also a (n,1) vector         X = Activation values of layer 'L' with dimension (n,m) if mini-batch    size = m   X-norm = normalized X with dimension (n,m)   Y = Gamma / Scaling factor   B = Beta / Offset factor   Now my question is, what are the dimensions of Gamma and Beta ? Are they (n,1) vectors or are they (n,m) matrices ? My intuition says that since they somewhat are analogous to the Mean and Standard Deviation, they should be (n,1) vectors.
39206
I have dataset of 15 features and the goal is to estimate a best fitting curve (regression). Now I want to use a deep learning technique for this purpose. Now I see that there are many architectures that can be used such as CNN, DBM or auto-encoder, etc.. Which one to go for. And I could not find any examples for how to use the same. A sample code would be of great help. Thanks  P.S. Some of the features are categorical. How do we handle that as well?