texts:
I want to extract the total bill from image receipts. I could extract the entire data present in the image but now I am struck with the problem of extracting only the information that I need.

This is the image that I have.

I am pasting the extracted information from the image

m cm lnnk 3mm: :33; no 1 z m

x Visut all! ms“; (or nulnunn mfn an an: nan.

Sub Iota] 19.56
TOTAL 19.56
VISA 1956
Fun 19.56
D!!! You Know 0

For ureat-tastlru dessens under 200
cahries, try our Triple Berry Frozen
Yogurt Sunda: a dish of Frozen Yogurt.
or a Vanma rozen Vugurt Done.

From this data I just want to extract the total bill. To get this I found out that I could use Ad Hoc Normalization (Adhoc retrieval). Can someone provide any insights on Adhoc retrieval. If there are any other option to extract the data from the image please let me do so. I am using tesseract to extract this information. Sometimes it is no giving the proper output. I could use some help in improvising the output given by the tesseract.


texts: Is there any reason not to use classification instead of regression when looking for ranges?

texts:
In the case I want to predict only ranges from a continuous value, is there any reason to use regression instead of classification ? Could it depend on the type of model I am using (neural network, decision tree, bayesian, ...) ?
Example
Let say I have a dataset with images. Each image has one human on it and is labeled with his/her height. Now I am only interested in predicting height ranges, for instance these four classes [ A, B, C, D ] = [ <150, 150-170, 170-190, >190 ] (in cm). Is there any reason why one of the two following approaches should lead to better performances ?

case 1: using regression - First create and fit a model that predicts the exact height from an image, then simply gives its associated height range.
case 2: using classification - First label all the images with the wanted ranges (=classes), then create and fit a classifier to predict this height range.

Note: I am wondering if there is a general answer to this question, not only to this example
EDIT
As @n1tk pointed out, in the post Performance of CNN based deep models with number of classes, the question is answered if we think about increasing the number of classes. In my question, I am wondering about regression vs classification. So try to fit a continuous value vs ranges from this value.


texts: Which dataset I should use when I am retraining my model?

texts:
I trained my deep learning model using x dataset and now I got new dataset and I give it name as y. I want to retrain my model on this new dataset which is y. Do I need to use x+y dataset or just y? Can you tell me what is right approach and what are the effects of that?
If nothing like that and other approaches are available please post it that helps a lot.
Thanks in advance.


texts: How does PCA and clustering treat different type of predictors where order of their values signify the result differently?

texts:
Assume we have 3 variables for 3 countries as shown below:

|  Country |  GDP  |  Child Mortality  |  Health Expenditure  |
|  X       |    5  |                8  |                   4  |
|  Y       |    7  |                3  |                   1  |
|  Z       |   12  |                4  |                   9  |

Now if GDP and health expenditure increase, it is a better country. While if child mortality increases it is a worse country.

Q1. How does PCA and clustering take care of these type of predictors appropriately without having the intelligence of knowing which set of predictors makes it a better country as value increases and which set of predictors make it a worse country as value increases?

Q2. Do we need to translate some predictors like child mortality with some inverse values (eg: divide by 1 or something else on scaled data), so that even for that column an increase in value signifies a better country, so the behavior is similar to columns like GDP / Health Spend etc, so our PCA and clustering algorithms are better able to model and provide good clusters ? Note, that there is no y in the model, we just have a list of countries which we need to cluster.

Anything of this sort is done in modelling and any techniques for these ?

OR what I am seeing here is invalid, and we just dont need to care about any translation (other than simple feature scaling) and directly run PCA/clustering algorithms ? And if we do this how does it take care to create appropriate clusters mathematically ?


texts: Efficient dimensionality reduction for large dataset

texts:
I have a dataset with ~1M rows and ~500K sparse features. I want to reduce the dimensionality to somewhere in the order of 1K-5K dense features.

sklearn.decomposition.PCA doesn't work on sparse data, and I've tried using sklearn.decomposition.TruncatedSVD but get a memory error pretty quickly. What are my options for efficient dimensionality reduction on this scale?


texts: Weigthing discrimnator and generator loss in GAN networks?

texts:
Training of good generator model in vanilla GAN (Generative adversarial networks) https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf is achieved via
minimax game, where Generator is attempting to fool the Discriminator. In the original equation discriminator and generator loss are weighted equally:
$$
\min_G \max_D V(D, G) =
\mathbb{E}_{\mathbf{x} \ \sim p_{data}(\mathbf{x})} [\log D(\mathbf{x})] +
\mathbb{E}_{\mathbf{z} \ \sim p_{z}(\mathbf{z})} [\log (1 - D(G(\mathbf{z})))]
$$
However, one of the many challenges in training GAN's is that discriminator becomes too powerful -  distinguishes almost certainly the fake images from the real one, and this causes the vanishing of gradients for generator. In a simple words it has no clue, how to change behaviour in order to make the generated more realistic. One of the approaches is to add some noise to the real images, but for me it seems, that more efficient strategy would be to put some weights to the generator and discriminator loss in such a way, that discriminator is forced to slow down the learning and enhance the generator training. Modification would look like:
$$
\min_G \max_D V(D, G) =
\alpha (t)
\mathbb{E}_{\mathbf{x} \ \sim p_{data}(\mathbf{x})} [\log D(\mathbf{x})] +
\beta(t)
\mathbb{E}_{\mathbf{z} \ \sim p_{z}(\mathbf{z})} [\log (1 - D(G(\mathbf{z})))]
$$
Where $\alpha(t)$ and $\beta(t)$ are adjusted during the training. Has this approach been pursued in the literature?
I would be grateful for references and results, in case they exist


texts: Comparison between approaches for timeseries anomaly detection

texts:
After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely:

Forecasting with Deep Learning. Eg. RADM or LSTM model
Forecasting without Deep Learning. Eg. Seasonal ARIMA + Kalman Filters
Denoising fixed-length windows with autoencoders (Deep Learning approach). Eg. MAD-GAN, CNN/LSTM autoencoders, variational autoencoders, etc.
Denoising without deep-learning. Eg. Applying filters such as Kalman or Hodrick Prescott, and test if the deviation of  the predicted with the original timeseries is under a threshold.

Maybe there are even more methods that are not classificable into this screenshot.

My question is, depending in the types of datasets that can occur, which approach suits better the need of developing a timeseries anomaly system which:

Detects anomalies in an univariate way, but allows for multivariate posterior integration.
Is suitable for online data-streaming and run-time adaptation/learning
Is capable of learning data distributions holding more than one seasonality.
[Plus] Might allow for semi-supervised improvement at any step.

Note: take into account timeseries that can have different trends; none, one or more seasonalities; cualitative/step/continous timeseries; with more or less missing data, etc.


texts: Downloading a large dataset in google colaboratory

texts: I have a link of a dataset, it is  22 gig but in the collab, there is just 32 gig free disk space. The data is in the zip file and after unzipping collab show crash massage. How can I solve this problem? I can't upload it in drive because it is 15 gig.


texts:
I'm sorry, if I ask my question by the wrong community.
What is a difference between Spark ML and Flink ML and between Spark and Flink in general? The both projects are the projects of Apache, I would like to know why Foundation has two similar projects.

PS I have found some interesting article Fast Big Data: Apache Flink vs Apache Spark for Streaming Data
It has answers on my question.

PPS This question has been already discussed by Stack Overflow community. The topic is What is the difference between Apache Spark and Apache Flink


texts: Unsupervized NER + sequence tagging to extract attributes from product descriptions

texts:
I am trying to extract attribute values from product descriptions in an unsupervised way.

For example, given the product title "Variety pack fillet mignon and porterhouse steak dog food (12 count)", I want to extract values for "12 count" -> size, "fillet mignon" -> flavor, etc.

If I use word2vec for word embeddings, what are the ways to go about word combinations, like "fillet mignon"? It seems to me that I would need to perform some type of BIO / BIOE tagging, since I am working with a dataset that has no tags.

What are the ways to approach this? Is there any technique that can do unsupervised sequence tagging?


texts: Keras/Theano custom loss calculation - working with tensors

texts:
I'm struggling to write some tensor manipulation code for a custom loss function I'm using in Keras.

Basically, I'm trying to modify a binary_crossentropy loss by adding a weight that is calculated from a particular feature.

First thing I do is pass in my extra feature data to the custom loss by appending it to the y_true like this:

y_trainModded = numpy.append(y_train, MyExtraData, axis=1)

Which is then passed to the fit function like:

model.fit(X_train, y_trainModded, epochs=2500, .....)

Then extracted to make it usable like this:

def myCustomLoss(data, y_pred):
    y_true = data[:,:2]
    MyExtraData = data[:,2]
    ...
    ...

So far, that all works fine.  However, I'm struggling with a section where I want to only select the MyExtraData where I predicted '1'.  Intuitively, this would simply be something like:

ExtraDataWherePredicted1 = MyExtraData[y_pred > 0]

However, we're dealing with tensors, not numpy arrays. I tried casting to numpy arrays using eval(), but that didn't work. I also tried various approaches using keras.backend operations such as:

WherePredicted1 = K.greater( y_pred,0)
ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1)

Which I could then use to weight my loss such as:

return K.mean(K.binary_crossentropy(y_pred,y_true), axis=-1)-(K.mean(ExtraDataWherePredicted1))

But anything I try throws out various errors...I just can't figure out how to calculate ExtraDataWherePredicted1. I'm also finding it super hard to debug the loss function because I can't print() anything inside it, so it's very hard to double check to see if the arrays/tensors are what I expect them to be.

Any help would be appreciated!


texts: Applying a keras model working with greyscale images to RGB images

texts:
I followed this basic classification TensorFlow tutorial using the Fashion MNIST dataset.
The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...).
The tutorial uses a simple model:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

This model reaches 91% accuracy after 10 epochs.

I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...).

Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:

  model = keras.Sequential([
     keras.layers.Flatten(input_shape=(32, 32, 3)),
     keras.layers.Dense(128, activation='relu'),
     keras.layers.Dense(10)
  ])

Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?

I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.

Below is my full code:

import tensorflow as tf
import IPython.display as display
from PIL import Image
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import pdb
import pathlib
import os
from tensorflow.keras import layers #Needed to make the model
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

IMG_HEIGHT = 32
IMG_WIDTH = 32

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

train_images = train_images / 255.0
test_images = test_images / 255.0

def make_model():
      model = keras.Sequential([
         keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
         keras.layers.Dense(512, activation='relu'),
         keras.layers.Dense(10)
      ])
      model.compile(optimizer='adam',
                   loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=['accuracy'])
      return model

model=make_model()
history = model.fit(train_images, train_labels, epochs=10)

EDIT: In order to address colt.exe's suggestion below, I converted the CIFAR-10 images from RGB to greyscale, using:

rgb_weights = [0.2989, 0.5870, 0.1140]
train_images = np.dot(train_images, rgb_weights)
test_images = np.dot(test_images, rgb_weights)

However, using the same model with 10 epochs yields an accuracy of about 38%, even worse than before!
I am starting to think that the Fashion MNIST is "easier" to work with since all images have a light background, whereas the CIFAR-10 consist of pictures taken outdoors in a wide variety of contexts.


texts: Advice on dealing with very large datasets - HDF5, Python

texts:
I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets as it offers the versatility to allow many dimensions, has no file size limits and is transferable between OSs.

My question is how to best deal with very large files. I am working with datasets that have 3-dimensions, all of which have large number of components (example size: 62,500 x 500,000 x 3,500, 32-bit int) which is a little problematic. HDF5 is theoretically able to deal with this, however, I've discovered it is really slow to write the data into the file and subsequently slow to retrieve it. Because the number of items is very large, data must be compressed otherwise it becomes insanely large on the HDD, however compression requires chunking and chunking is quite tricky to get right... I receive my data in (1 x 500,000 x 3500) chunks so it makes sense to write it one row at a time so with those dimensions and I ensure that chunks have dimensions (1 x n_values x 3500) to speed-up writing and data retrieval and have relatively small size (< 1Mb) for quick compression/decompression.

Does anyone have any experience in how to efficiently write to HDF5 files when the size of each dimension is v. large? Are there any tricks that could help?

I should also mention that I mostly work in Python, however, would be happy to do it in another language. I've tried increasing the chunk cache via the h5py_cache library.


texts: Evaluating correlations between dichotomous/binary variables? What test?

texts:
I have 4 variables, each are dichotomous, 1 = in this group & 0 = not in this group (for each of the 4 groups).
I have read mixed accounts of whether Pearson's correlation can be use used for binary variables. If not, what is a good alternative?
Would the Cramer V test work with a twoway-chi 2?


texts: What neural network can I use to unmerge numbers?

texts:
I am working on a hobby project where I superimpose two different samples of two different numbers from the MNIST dataset.

However, I'm stuck at what model I could use. I tried messing around with autoencoders for this.

Is there some article or paper which could possibly guide in the right direction as I didn't find anything relevant?

For example, first one is 0 and 5, second is 5 and 6, third is 2 and 0, fourth is 1 and 3 and the last one is 3 and 6.

As you can see from the images that such a task is difficult for humans as well which makes me curious if a computer could notice patterns that we can not in order to perform better.


texts: Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

texts:
I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer which is a fully connected layer. However, I observed that without dropout I get 97.75% accuracy on the test data and with dropout of 0.5 I get 95.36%. I want to ask whether I am doing something wrong or what is the reason for this phenomena?
I change it into eval mode in test but I reach to 96.44% accuracy. Still it is less than without dropout.
Thanks a lot

# RNN Model (Many-to-One)
class RNN(nn.Module):
   def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(RNN, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
                        batch_first=True,bidirectional=True)

    self.fc = nn.Sequential(
        nn.Dropout(0.1),
       nn.Linear(hidden_size*2, num_classes),
        nn.Softmax(dim=1)
    )
def init_hidden(self,x):

     return(Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)).cuda(),
     Variable(torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).cuda()))

def forward(self, x):
    # Set initial states

    # Forward propagate RNN

    hidden = self.init_hidden(x)
    #print(len(hidden))
    out, _ = self.lstm(x, hidden)

    # Decode hidden state of last time step
    out = self.fc(out[:, -1, :])
    return out


texts: SVD++ vs wALS: Which is the more effective for implicit feedback in Recommendation system

texts:
As SVD++ can be used for implicit feedback, I would like to know whether SVD++ can gives better results than the wALS algorithm (paper: Collaborative Filtering for Implicit Feedback Datasets ).

I can't find any benchmark between those 2 algorithms on a implicit feedback dataset (like lastfm dataset).

My use case is "predicting future purchases in e-commerce" (thakns to historical buying behaviours).

Is there any classic new model-based recommendation engine I forget that could work well on implicit feedback datasets ?


texts: Does a CNN think things inside the filter are collocated aka dependent on each other?

texts:
I am running a 1D CNN on tabular data. The rows are data that I have are not sequential, that is to say they are not part of a time series or ordered string, which is why I am not using an LSTM.
So when the filter looks at both row n and row n+1, does it learn from them as if they are spatially collocated or dependent upon each other? Is there a way to prevent this from happening?
Alternatively, is there a better way to look at 2 dimensional data (rows of features) as independent rows?


texts: Graph Embedding Vs Graph Convolution Network

texts:
I'm new in Graph-Embedding and GCN(Graph/Geometric Convolution Network).

I'm confused and not very much sure about "How training works in GCN"?

As per my understanding, GCN training data will be in the form of "Adjacency Matrix" + "Degree Matrix" + Features of nodes(vertex). Using the aggregate of these metrics, we do GCN training. Kindly confirm?

If my understanding is correct, then I believe that even for Graph-Embedding, we use similar type of Input data(as mentioned above) for generating vector representation? Again if  it correct, why GCN is better than Graph Embedding?

Or it's incorrect to co-relate both these terms?


texts: Creating a neural network, composed of n times a different network. Is it possible?

texts:
I'm currently working on a project with a bunch of data of devices that can either belong to people, or not. The ultimate goal is to estimate a number of people detected.
Sadly, it is impossible to get labels for which device should be classified as human and which shouldn't. So a basic binary classification network will not be trainable.

I do however have for each moment in time a number of people, that can be correlated to the number of human-devices that should be detected (it is not the exact sum of devices that should be detected because we won't be able to detect all).

So... I though of this weird construct, where I have a binary classification neural network (green box in the picture) copied n times along itself, and then have its outputs propagate to an outer network which will perform a regression using both the outputs of the classifiers and some other variables (pink) that will determine the scaling (for example, if only 50% of the humans are detectable using the devices, then the scaling will have to be 2. But this is a non-linear function so it'll also be determined by the NN).

The green neural networks should all share the same weights, because they will perform identical work.

Another possibility would be to use the pink scaling variables from the outer (red) NN, to the green-NN's, and then let it result in a non-binary value, but more like a scaled value. So in the same example as above, a 1 would return 2.

Would this be possible in theory? Has it been done before? What would be the best approach for this?

I already had a talk with my adviser about this, but he was very skeptical about it.


texts: Calculate a mean on condtions in a dataframe with dyplr

texts:
My table looks like this:

    Tissue  Dry Amount   Analyte            Area
1   Liver   A   a   3-Phosphoglyceric Acid  66351918.4
2   Liver   B   a   3-Phosphoglyceric Acid  119013081.6
3   Liver   A   b   3-Phosphoglyceric Acid  195732464.0
4   Liver   B   b   3-Phosphoglyceric Acid  247443210.8
5   Liver   A   c   3-Phosphoglyceric Acid  456447252.7
6   Liver   B   c   3-Phosphoglyceric Acid  494555301.1

I would like to get the mean of the two values for the same Tissue Amount and Analyte by ignoring the DRY variable.

I always end up having one value of the two and not the mean.

Summary <- shape.data %>%
+   group_by(Amount,Tissue,Analyte ) %>%
+   summarise(Area_mean = mean(Area))

How can I do this calculation?


texts: Keras custom loss function as True Negatives by (True Negatives plus False Positives)

texts:
I have some model for which I can construct the confusion matrix, although I need a custom loss function which will be as:

true negatives (TN): We predicted no, and it is no.

false positives (FP): We predicted yes, but it is no.

My loss function should be like:

           TN
  1 -  _________
        TN + FP

How do I implement the same in Keras:
There is an explanation for the custom loss function here: StackOverFlow

Although, I am not sure how the differentiation part for the same would be like.

Image source: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/


texts: Classifying videos with varying length using ConvLSTM2D in tensorflow

texts:
I have a collection of videos, where I would like to extract a frame for every second, and then feed them through a ConvLSTM2D for binary classification.

I was under the impression that a LSTM could take varying input sizes, but after many hours of googling it seems like I either need to:

Use padding and masking
Use ragged tensors
Actually use varying input length, but use batch size of 1

I'm not sure how to proceed from here, since I cant find any resources for padding and masking a sequence of images. Ragged tensors are confusing, and I cant find any examples for a sequence of images. When trying to use a batch size of 1, tensorflow still complains that the inputs are not the same size when using model.fit.
The length of the video is actually important, so thats the reason I'm using a variable amount of images, but I could possibly extract a fixed amount of frames.
Any code examples or suggestions appreciated


texts: How to add time as a feature into clustering algorithm?

texts:
I've wrote here about my problem. And now I have a new one.
How to use time as a feature? I have a pandas DataFrame with date-time column('date') and I don't know how to use it as a feature. For now I'm using  TfidfVectorizer.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(min_df=5, max_features=10000, ngram_range=(1, 2))
vz = vectorizer.fit_transform(df['word'])

Where 'word' column is tokens on my text.
I'm using MiniBatchKMeans. Here is code example:

from sklearn.cluster import MiniBatchKMeans

num_clusters = 32
kmeans_model = MiniBatchKMeans(n_clusters=num_clusters, init='k-means++',     n_init=1,  init_size=1000, batch_size=1000, verbose=False, max_iter=1000)
kmeans = kmeans_model.fit(vz)
kmeans_clusters = kmeans.predict(vz)
kmeans_distances = kmeans.transform(vz)

I'm interested at how to add my time feature(my time column) to clusterizator and is there any possibility to make some kind of "weight" factor for time features and for different tokens.


texts: Does ridge regression always reduce coefficients by equal proportions?

texts:
Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization)

"In ridge regression, each least squares coefficient estimate is shrunken by the same proportion"

On a simple dataset, I obtained 2 non-intercept coefficients b1=-0.03036156 and b2=-0.02481822 using OLS. On l2 shrinkage with lambda=1, the new coefficients were b1=-0.01227141 and b2=-0.01887098. Both haven't reduced by equal proportions. What am I missing here?

Note:

the assumption made in an Introduction to Statistical Learning book for the quoted statement is n=p
the scale of both variables in my dataset is same


texts: Are machine learning models good at autoregressive forecasting of time series?

texts:
AR, MA, GARCH and VAR models are standard for autoregressive prediction, which is the forecasting of a variable using its own historical lags for features/regressors.

Support vector regression (SVR), random forests, boosting and ANNs are typical machine learning algorithms.

How good are these at time series forecasting? Are any of them able to forecast farther into the future beyond 1-day forecast horizons by using historical lags for input features? Do any sources give a theoretical basis for machine learners' applicability to autoregression? And to assess their out-of-sample performance, given that the task is autoregressive prediction of time series, is bootstrap aggregation or cross-validation more appropriate?


texts: Classification algorithm that only matches trained examples

texts:
I have 10 categorical features and a multi-class target.
Training data contains rows where the same 10 categorical features may map to a different target class.
What classification algorithm should I choose that fulfils the following criteria:

prediction output should only show results if the prediction input matches a row example that existed in the training data. (I don't want to make assumptions on inputs it has never been trained on, as this model will deal with people's lives in the real world)
prediction output should be targets that match exactly the prediction input, ordered by their frequency in the training data, with the highest as 'the best match' at the top

I realize this looks like it could be 'easily' solved with databases but I would like to use Classification and train a model instead of having a cumbersome db infrastructure with columns, keys, indexes and other boring things.


texts: Text-Classification-Problem: Is Word2Vec/NN the best approach?

texts:
I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context:

Is trained with user generated text paragraphs (like comments/questions/answers)
Each item in the training set will be tagged with . So for e.g. ("category 1", , "text paragraph")
There will be hundreds of categories

What would be the best approach to build such a system? I have been looking at a few different options and the following is a list of possible solutions. Is Word2Vec/NN the best solution at the moment?

Recursive Neural Tensor Network fed with averaged Word2Vec data
RNTN and The Paragraph Vector (https://cs.stanford.edu/~quocle/paragraph_vector.pdf)?
TF-IDF used in a Deep Belief Network
TF-IDF and Logistic Regression
Bag of words and Naive Bayes classification


texts: R - How do I remove a varying number of digits from a date-vector

texts:
I want to remove a varying number of digits from a date vector. My date vectors looks like this:

I want to convert this vector into a date vector, but first I have to get rid of the number in front of it. This is where I struggle since this number can be (here from 88-106) varys in my data.frame from 1-50 000. Does anyone know how to remove these number without destroying the date? I tried the following, but it didn't work out:

I would really appreciate your help! Thanks!


texts: Create clusters based on specific keywords

texts:
I am working on raw text data. I am using clustering to put together common words in the documents. My requirement is to create clusters based on a specific list of words i.e I want to get a group of words that are typically found with the user-given list of words. Visually, the clusters should look like below. Typically, the clustering techniques are focused on creating segregated clusters while I need segregated clusters with some overlap. The image shows the view of the expected results.
I have tried using k-means clustering, the Apriori algorithm, and PrefixSpan in Python. But my desired result is not achieved.
Any suggestion is appreciated.


texts: What methods can be used to detect duplicacy in image dataset?

texts:
I want to remove duplicate images from a dataset of 50Million images. What is the best method to detect all the duplicates?

Do you think one shot learning is good for this?


texts: Restricting the output of a model didn't improve the loss value of the model evaluation

texts:
There is a deep model for prediction.

The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80)

The model Loss value is 70 and I would like to reduce it.

I printed the outputs after evaluating the model by test values and some of the predicted values are more than 80 or less than 0.

I decided to set up the final layer to predict just in 0-80 in the training step, therefore I set a lambda layer after final Dense layer to clip output values.

The codes:

def relu_advanced(x):
return K.relu(x, max_value=80)

def createModel4():
model = models.Sequential()
model.add(Conv2D(256,(3, 3),
                 activation='relu',
                 input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
#model.add(Dense(5*320, activation= 'relu'))
model.add(Dense(5*320))
model.add(Lambda(relu_advanced))

model.summary()
return model

I tested the model with and without the relu_advanced and unfortunately, the Loss value is increased with advanced_relu!

While there is no value much than 80 or less than zero, I don't know what may happen that the Loss is increased?

Thank you


texts: Can I specify the order of yielded instances using keras ImageDataGenerator#flow_from_directory

texts:
I am trying to merge two models,  one is an LSTM that needs an order on a numpy series somehow (I found the appropriate order). This model, along with a convolution model that accepts a flow of images using flow_from_directory.

knowing that order of input of one model is a constraint, I want to order the flow of images ids from directory so that they could learn something together.

Here is my implementation so far without a specific order on

from keras.preprocessing.image import ImageDataGenerator

input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator(rescale = 1./255)

def slice_and_order(gen, cut):
  for i in gen:
    idx = (gen.batch_index - 1) * gen.batch_size
    filenames = gen.filenames #[idx : idx + gen.batch_size]
    filenames = [int(filename.split('/')[1].split('.')[0]) for filename in filenames]
    print(filenames)
    current = int(gen.filenames[idx].split('/')[1].split('.')[0])
    print(current)
    if(current in range(cut[0], cut[1])):
      yield i

def generate_generator_multiple(generator,dir1, cut, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    genX1_ = slice_and_order(genX1, cut)
    with tf.Session() as session:
        res = bounding_boxes(cut)
        genX2 = tf.estimator.inputs.numpy_input_fn(
            res['x'], res['y'], batch_size=2, shuffle=False, num_epochs=1)

    while True:
        val = next(genX1_)
        features, target = genX2()

        yield [val[0], features], target  #Yield both images and their mutual label

_20_ = (14104 // 5)
train_cut = [0, _20_ * 3]
test_cut =  [train_cut[1], train_cut[1] + _20_]
validation_cut = [test_cut[1], 14104]

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1="images/",
                                           cut=train_cut,
                                           batch_size=1,
                                           img_height=96,
                                           img_width=96)

validgenerator=generate_generator_multiple(test_imgen,
                                          dir1="images/",
                                          cut=validation_cut,
                                          batch_size=1,
                                          img_height=96,
                                          img_width=96)

slice_and_order should order genX1 although it doesn't for now.


texts: Is it advisable to rerun LASSO multiple (2) times?

texts:
I have a dataset with large (4500 variables) and 3000 observations (n < p case). I am using LASSO (from the glmnet package in R) to reduce dimensionality of the problem. I have couple of questions.

1) After I ran LASSO the first time, the number of variables with non-zero coefficients goes down to ~80 (which seems reasonable). But the correlation matrix shows that some of these variables are very highly correlated themselves (FORMS distinct groups if you plot it using the corplot package. So I ran LASSO exclusively on these variables whose coefficients were non-zero during the 1st run. This time, the algorithm picked up even less number of variables.I was expecting that they would remain more or less similar. I want to know if running LASSO multiple times is advisable or it introduces bias of any form in anyway that interferes with interpretation/prediction ?

2) My second question is about the magnitude of the coefficients. Some of the coefficients, in absolute value, turns out to be very close to 0 (~1E-14) while some are larger. I mean the spread is very large. Does it indicate that the variables with this small coefficients are making the model worse in any way. Is it advisable to get rid off those variables and rerun LASSO ?

Thoughts/comments are most welcome.

Thanks


texts: Is there a ready-made classifier to determine if webpage is an article or not?

texts:
I am trying to solve a problem, where I have a set of URLs and I need to filter out only those that can be classified as an article (i.e. typically not include privacy policies, terms of usage, webpage for unsubscribing from a feed etc.).
Is there a trained classifier that can be used to filter these webpages? Every webpage classification material I found was about classifying articles into categories, but this doesn't concern me. I need to go a step back and figure out if the webpage is actually an article or not.
If there's not ready made classifier for this, what would be the best approach to create such classifier?
UPDATE: To clarify some questions below. The problem is not to identify an article only by the url, but to find a way of labelling a webpage as article/non-article based upon its contents (html, visual representation?, some other metadata perhaps). From the research I've done since asking the question, I believe this to be a variant of something called blog identification. I understand that the definition of an article is an important piece here and very ambiguous requirement, however that is exactly the hard part I have been struggling with while searching for some algorithm to do just that. If I had a fixed dataset, I should be able to identify common patterns and come up with a definition of an article by myself. But since the set of webpages that need to be filtered is very dynamic and unknown beforehand, I don't want to rely on my own custom rules for defining how an article looks like but would much rather use a method of identification that might have some research or at least a field-test validation behind it. It just seems to me, that this is a problem that someone had to be solving before me and that there might be solutions based upon that already.


texts: Why is the input to an activation function a linear combination of the input features?

texts:
I'm new to deep learning, and there is something about the calculations I do not understand:
For a standard neural network, why is it that only the activation function is not linear, but the input to the activation function is a linear combination of each of the $x_i$'s? For example, with the sigmoid function, it would look like:
$$ \frac{1}{1+ e^{-(w_0x_0 + w_1x_1 + b)}} $$
where $w_i$ are the weights and $x_i$ represents the input to that layer.

For example, why is it that we don't have something like this:
$$ \frac{1}{1+ e^{-(w_0x_0^2 + w_1\sqrt{x_1} + b)}} $$

Is it because it would be redundant if we had enough layers? Or is it because a priori, you wouldn't know what the best function is?


texts: Which Machine learning Algorithm would work best in predicting particle size distribution (PSD)?

texts:
Im a beginner with machine learning and data science. i am working on a project where i have to work with an algorithm/ machine learning model that would predict the particle size distribution ( a distribution function that gives the different size of particles present in a given amount of sample substance e.g sand). The particle size distribution will be given by performing sieve analysis of sand.
below a wikipedia link to give an overview of what im specifically working on.
https://en.wikipedia.org/wiki/Sieve_analysis#:~:text=A%20sieve%20analysis%20
i am really confused about it, just need a push in the right direction.


texts: Use cases for graph algorithms and graph data structures in finance and banking

texts: I work in a bank and most data is in tabular format in relational databases. I have been reading about graph algorithms (page rank), graph libraries (spark graphx) and graph databases (neo4j). I would like to pick a use case from my field (finance). What use case would be suitable for graph algirithms and databases?


texts: How does the naive Bayes classifier handle missing data in training?

texts:
Naive Bayes apparently handles missing data differently, depending on whether they exist in training or testing/classification instances.
When classifying instances, the attribute with the missing value is simply not included in the probability calculation (reference)
In training, the instance [with the missing data] is not included in frequency count for attribute value-class combination. (reference)
Does that mean that particular training record simply isn't included in the training phase? Or does it mean something else?


texts: Why neural networks models do not allow for multiplication of inputs?

texts:
In a neural network, each neuron value is multiplied by the weight of the connection. Then, each neuron's input is the sum of all those values, on which we apply the activation function (sigmoid, relu, tanh, etc.).

Why can't the input be a vector of all those values, so that I can decide to multiply them if I want, or apply another function from the vector space to the reals, and then apply the activation function on that real value? Any implementation I have seen does not allow this, and if I wanted to do it, it seem I would have to implement my whole neural network architecture by myself without the possibility of using Theano, Keras or other libraries. Is there a reason that escapes me of why this would not work? And if there isn't, is there any library out there I could use that already does this without having to create it from scratch?


texts: I need some guidance how to develop myself in deep learning

texts:
Few months ago I've taken an Andrew Ng course at coursera and started my data science/machine learning journey.

During those months, I've take another courses from the internet - about Deep Learning, statistics - read articles from towardsdatascience website, now I'm reading a book "Hands on Machine Learning".

But I feel I'm stuck, I have over a dozen pages of courses at coursera, few books, and I don't know what to do next.
I feel like I'm doing the same things all the time.

I'll be very thankful for some guidance :)


texts: Identify "steady-state" time series window

texts:
I'm new with the time-series analysis.
I have several time-series (noisy of course) part of the same set of measurements (sampled simultaneously).
The time series are the results of a stochastic process (dynamic system) where part of the time-series are inputs to the process and other time-series are the outputs.
I need to find a time window corresponding to a steady operation, i.e. where both inputs and outputs don't change over time (for a deterministic signal would say derivative over time almost zero).
Do you know any way to accomplish this task? The measurements are quite long, I need a rather computationally efficient method ideally without involving any visual inspection.
Thanks!


texts: Multi-class classification as a hypothesis testing problem

texts:
I'm diving into the logistic distribution and its applications in classification problem (see my old question for more details about my idea).

As discussed in my old post, logistic regression, in fact, can be seen as a hypothesis testing problem.

My question is that: is there any hypothesis-testing-based classifier for multi-class classification problems ?


texts: How to create a complex Gaussian random noise with a specific covariance matrix

texts:
I am trying to generate a complex Gaussian white noise, with zero mean and the covariance matrix of them is going to be a specific matrix which is assumed to be given.

Assume i to be a point on the grid of x axis, where there are N points on the axis. The problem is to generate a complex valued random noise at each point (let's call the random value at the point i as $y_i$), which obeys Gaussian distribution and has a covariance matrix of,

$E((y_i)(y_j^*)) = c_{ij}$

where $c_{ij}$ is a given diagonal NxN matrix. * is complex conjugate, E represents the expectation value.

I am trying to generate this on Python.


texts: How to visualize data of a multidimensional dataset (TIMIT)

texts:
I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 PLP values, 13 about first order derivative and 13 about second order derivative).

I would like to improve my dataset. The only thing I've tried thus far is normalizing the dataset using a standard scaler (standardizing features with mean 0 and variance 1).

My questions are:

Since my dataset has high dimensionality, is there a way to visualize? For now, I've just plotted the dataset values using a heat map.
Are there any methods for separating my sample even more, making it easier to differentiate between the classes?

My heat map is below, representing 20 samples. In this heatmap there are 5 different phonemes, related to vowels, in particular, uh, oy, aw, ix, and ey.
As you can see, each phoneme is not really distinguishable from the others. Does anyone know how could I improve it?


texts: What are some (non image) datasets where deep networks are required for good performance?

texts:
It is well known that deep networks tend to outperform shallow networks and other classical machine learning techniques such as boosting on learning tasks involving images. I believe this is because extracting useful high-level features from low-level pixel intensities requires deep models.

I'm curious to know if there are any non-image data sets (e.g., tabular datasets that are typically found in UCI repository) where deep networks are required for good performance and where shallow networks, boosting tend to perform poorly.


texts: Comparing data sets with different measurements

texts:
I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis.

One piece of data (https://www.pandasecurity.com/mediacenter/src/uploads/2014/07/Pandalabs-2015-anual-EN.pdf on page 9) it states that the 'infection rates' of Sweden is 20.88% (bottom 3 ranking), USA at 29.48% (middle ranking) and China (first rank) having 57.24%.

Another (http://www.virusradar.com/en/home/world) , uses a different measurement to define the 'threat rates', which is different to the one above, which has Sweden at 2.13%, US at 2.87%, and China at 15.17%

Another piece of data (https://www.symantec.com/content/dam/symantec/docs/reports/istr-22-2017-en.pdf
 on page 50) states 'identity theft' in Sweden is 6 million(low ranking), US 791 million (high ranking), and China 11 million (middle ranking)

I'm unsure how to compare these, because by just looking with our eyes we can see Sweden is the lowest in all these statistics which I can use to further discuss my argument. However, these are clearly different measurements (one is overall infections, one is just identity theft, etc), and I have much more similar data which provides numbers/percentages of certain types of cyber crime by country.

My goal is to analyse the results so I can compare them to further the goals of the thesis about where cyber crime is most/least prominent and to create visualisations of this.

So would just simply ranking them be fine? (Sweden 3 because it was lowest all times?, USA 2 because it was second twice?, China 1 because it was first twice?) but that sounds very incorrect. Should I convert all my numeric data sets (like the identity theft one) to percentages then compare them all by percentage to rank/discuss accordingly? And compare them then even though they're different measurements of the same thing(country)?

Thanks for any advice.


texts: Train and predict two labels in a single process

texts:
I have a python program that makes predictions using scikit-learn RandomForestClassifier. The label is called "default" and it's the default status of a loan. This works fine.
What I need now is to extend this model, and have another label called "Prepayment Percentage" that needs to be trained using the same data as the "default" label. Ideally, the model will be trained once and the predictions will also run only once for both labels. Is this possible with RandomForestClassifier?


texts: Training the Variational Autoencoder After applying the reparameterization trick

texts:
In a Variational Autoencoders, z cannot be simply sampled from the output of the autoencoder directly since the network would not be differentiable. Instead, we have to sample from a normal distribution who's parameters are the output of the autoencoder, mu and stddev.

I So cannot understand how the network isn't differentiable (lets say we take z = mu + srd). So the reparameterization is not clear to me.

Any help would be much appreciated!!


texts: How to find out if two datasets are close to each other?

texts:
I have the following three datasets.
data_a=[0.21,0.24,0.36,0.56,0.67,0.72,0.74,0.83,0.84,0.87,0.91,0.94,0.97]
data_b=[0.13,0.21,0.27,0.34,0.36,0.45,0.49,0.65,0.66,0.90]
data_c=[0.14,0.18,0.19,0.33,0.45,0.47,0.55,0.75,0.78,0.82]

data_a is real data and the other two are the simulated ones. Here I am trying to check which one (data_b or data_c) is closest or closely resembles to data_a.
Currently I am doing it visually and with ks_2samp test (python).
Visually
I graphed the cdf of real data vs cdf of simulated data and try to see visually that which one is the closest.

Above is the cdf of data_a vs cdf of data_b

Above is the cdf of data_a vs cdf of data_c
So by visually seeing it one might can say that data_c is more closer to data_a then data_b but it is still not accurate.
Kolmogorov-Smirnov (KS) Test
Second method is the KS-test where I tested data_a with data_b as well as data_a with data_c.
>>> stats.ks_2samp(data_a,data_b)
Ks_2sampResult(statistic=0.5923076923076923, pvalue=0.02134674813035231)
>>> stats.ks_2samp(data_a,data_c)
Ks_2sampResult(statistic=0.4692307692307692, pvalue=0.11575018162481227)

From above we can see that statistic is lower in when we tested data_a with data_c so data_c should be closer data_a than data_b. I didn't consider the pvalue as it would not be appropriate to think of it as a hypotheses test and use the p-value obtained because the test is designed with the null hypothesis predetermined.
So my question here is that if I am doing this correctly and also is there any other better way to do it? Thank You


texts: why does making the target variable normally distributed helps?

texts:
while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that?

Should we also transform independent variables to have near normal distributions?


texts: How to apply MultiOutputClassifier to a dataset for Naive-Bayes algorithm

texts:
I have a dataset which is as follows, (it's taken from an article online and I have been trying to Naive Bayesian algorithm on it)
Original Dataset
y attribute
After having done some manipulations (following the article), these are my new datasets for training and testing,
X Train
y Train
Now, it contains a multilabel and I have been asked to look at Multioutput classification for the problem. I have been trying to understand this classification and tried to implement it myself too, but I couldn't get it to done. First of all, I tried following this sample code given on the website,
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np
X, y1 = make_classification(n_samples=10, n_features=100, n_informative=30, n_classes=3, random_state=1)
y2 = shuffle(y1, random_state=1)
y3 = shuffle(y1, random_state=2)
Y = np.vstack((y1, y2, y3)).T
n_samples, n_features = X.shape # 10,100
n_outputs = Y.shape[1] # 3
n_classes = 3
forest = RandomForestClassifier(n_estimators=100, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
multi_target_forest.fit(X, Y).predict(X)

But, since, I am new to all this, I didn't understand anything at all.. I didn't understand why he did the make_classification call, and then shuffled the data and etc.
I tried to implement it on my y_train variable and then placed it in my model.fit for Naive-Baysen algorithm,
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
Yt = np.vstack(y_train).T
n_samples, n_features = X_train.shape # 10,100
n_outputs = Yt.shape[1] # 3
n_classes = 3
forest = RandomForestClassifier(n_estimators=100, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
model.fit(X_train, multi_target_forest)

But it gave the same error which I was receiving previously, which meant that I didn't do the multioutputclassification properly,
ValueError: y should be a 1d array, got an array of shape () instead.

Can anyone help me in telling how to actually implement this classification, so that the Y variable can be used for the Naive Baysen?


texts: When one model is superior in real world use?

texts:
I have an NLP neural network that I have developed with Keras for multi-label classification.

I have fit the model several times and save the best results (via best validation accuracy score) after each set of epochs completes. All of my saved models are in the 96%+ validation accuracy score (according to Keras).

However, when I run these models against real-world data where I also know the result (e.g. effectively a second round of validation) one model in particular outperforms the rest. I can take the champion model (96.29% validation accuracy) and put it up against another model (with something like 96.18% validation accuracy) and the champion model can achieve 90%+ accuracy in the second round of validation while the other model - or any other model - will do nowhere near that. This one model will achieve a minimum 8% accuracy above all other models.

I have double-checked my methodology and I'm nearly positive that all models are being created with the same code and process.

Should I be concerned that this one particular model outperforms the rest? Does it indicate anything in particular in my overall methodology?


texts: Troubleshooting Neural Network Implementation

texts:
I've been going through the Standford/Coursera Machine Learning course; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).
The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.
Here's a brief run through of what I do;
Set all Theta's (weights) to a small random number

for each training example

set activation 0 on layer 0 as 1 (bias)
set layer 1 activations = inputs

forward propagate;
 Z(j+1) = Theta(j) x activation(j)  [matrix operations]
 activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid]

Set Hx = final layer activations

Set bias of each layer (activation 0,0) = 1

[back propagate]
calculate delta;
delta(last layer) = activation(last layer) - Y  [Y is the expected answer from training set]

delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j))

[where ones is a matrix of 1's in every cell; and .* is the element wise multiplication]
[Don't calculate delta(0) since there ins't one for input layer]

DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j))

Next [End for]

Calculate D;

D(j) = 1/#Training * DeltaCap(j) (for j = 0)

D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0)

[calculate cost function]

J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2

Recalculate Theta

Theta = Theta - alpha * D

That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.
EDIT:
Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);

The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.
Edit #2;
I've ran a gradient checking algorithm with the following sort of results;
Numerical calculation: -0.0074962585205895493
Value from propagation: 0.62021047431540277
Numerical calculation: 0.0032635827218463476
Value from propagation: -0.39564819922432665
etc. Clearly there's something wrong here; I'm going to work through it.


texts: Extract data from json format and paste to column using python

texts:
In my column with json data, I have this list I want to extract to column:
"list":[

{
    "id":"list",
    "item":[
        {
            "value":"Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon"
        },
        {
            "value":"Geruchs- und geschmacksneutral"
        },
        {
            "value":"Kältebeständig bis -60°C"
        },
        {
            "value":"Inklusive Rezeptbuch und 50 Eisstielen"
        },
        {
            "value":"spülmaschinengeeignet"
        }
    ],
    "decorators":[
    ]
},

I have extracted other data using this code but the value was string and not a dictionary:
if 'item' in item and item['item']:
                    if isinstance(item['item'], str):
                        cur_model_info[item['id']] = item['item']
                    elif isinstance(item['item'], list):
                        elements = [element['value'] for element in item['item']]
cur_model_info[item['id']] = ','.join(elements)

I tried to use this for the above format of data but I got this error:
TypeError: sequence item 0: expected str instance, dict found
What should I change in order to be able to export in a separate column the data having each element from the list in a new cell of that column?
expected output

Using the above code, the data is exported in this format


texts: Forecast multiple unevenly spaced time series

texts:
I am building a time-series forecasting model to predict some patterns in climatological  data.

The dataset consists of many (2 mln) time series which look for example as:

However the observations all of these time series is unequally distributed (growing trend with years).

Although I am still considering my approach (LSTM, exponential smoothing, etc.), I will have to deal with this unequal distribution of observations. Is there a golden standard for equalizing time series observations?


texts:
I'm studying the Adaboost algorithm. This algorithm updates the weight after training.

This is the table when they explain about weight on Adaboost

I'm confused about what this "weight" means. Is it:

weight for each node?
weight for each model?
table weight column (more confusing)?


texts:
I am trying to start a meta-analysis for which I want to extract some 16S-based information from public databases. Moreover, I want to relate this information with any metadata found in the associated studies (everything from environmental variables to sequencing details).

For this, I realized some databases are available, like NCBI-Nucletotide, NCBI-SRA and EMBL-EBI-ENA, but I am not sure about which one to use or whether I can use them all.

How can I filter only whole 16S sequences? Or even the most used 16S region?

One more thing would be how can I extract all the corresponding metadata?

Has anyone here tried this or anything similar before?


texts: Creating a posterior distribution for classic coin flipping in python using grid search

texts:
I'm reading the book "Bayesian Analysis with Python" and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic coin flipping example. In this example we set a prior on the probability of obtaining a head, then given some data our likelihood is the binomial distribution.

So from my understanding using grid search we will break the interval $[0,1]$ into chunks, in our case 100. We have a prior probability $P(\theta=\theta_{0})$ for each of the points in the discretised grid we multiply this by $P(D|\theta=\theta_{0})$ and we multiply these together to get an un-normalised estimate at each location.

The code is the following

def posterior_grid_approx(grid_points=100, heads=6, tosses=9):
    grid = np.linspace(0, 1, grid_points)
    prior=np.repeat(5,grid_points)
    likelihood=stats.binom.pmf(heads, tosses, grid)
    unstd_posterior = likelihood * prior
    posterior = unstd_posterior / unstd_posterior.sum()
    return grid, posterior

I don't understand why the prior is set to a list of $5$'s isn't this meant to be a probability value? From my understanding they've broken the grid into 100 different points obtained a binomial pmf for each of these locations and then multiplied the number by 5. How is this a valid prior?

I'd really appreciate if someone could clearly explain the reasoning behind this as i like to understand the code as clearly as the concept. Thanks!


texts: Dealing with NA values while using Rotation Forest in R

texts:
I am using Rotation Forest in r.
I have a very big list (which I can't give a sample) that has 50 attributes and 4000 rows.
for every row there is a few missing values in the attributes, it supposed to be like that and I cant fill the missing values.

this is my code:

y <- list[,1]
x <- list[,2:ncol(list)]
rF <- rotationForest(x,y,K = round(ncol(x)/3),L=15)

this is the error:

Error in cov.wt(z) : 'x' must contain finite values only

from what I gathered the problem is with NA in relevance to PCA in the rotation forest.

any suggestion on what I can do?


texts: What is the difference between multiply and dot functions that is used to merge layer in Keras?

texts:
I want to merge two CNN deep learning model using Keras and would like to know what is the difference multiply and dot functions that is used to merge layer?

keras.layers.multiply(inputs)

keras.layers.dot(inputs, axes, normalize=False)


texts: How can the labels of AgglomerativeClustering be re-computed?

texts:
I'm using scikit learn's AgglomerativeClustering on a large data set.
I want to modify the distance_threshold after the model has already been computed. Computing the model is slow (quadratic time) but it should easily be possible to re-compute the labels for a new distance_threshold in linear time because the model stores the children_ and distances_ arrays permanently. But how can the labels be re-computed for a different distance_threshold?
It can be assumed that distance_threshold was originally set to 0, i.e. the entire tree was computed.


texts:
I have some trouble understanding LSTM models in TensorFlow.
For simplicity, let us consider the Example program.

I use the tflearn as a wrapper as it does all the initialization and other higher level stuff automatically.

Till line number 42 net = tflearn.input_data([None, 200]) it's pretty clear what happens.

You load a dataset into variables and make it of standard lengths in this case 200. for both the input variables and also the 2 classes present in this case are converted to one hot vectors.

What I would like to know here is how the LSTM takes the input and across how many samples does it predict the output?

What do these parameters indicate: n_words=20000 & net = tflearn.embedding(net, input_dim=20000, output_dim=128)?

My goal is to replicate the activity recognition dataset in the paper.

For example, I would like to input a 4096 vector as input to the lstm and the idea is to take 16 of such vectors and then produce the classification result.

I think the code would look like this but I don't know how the input to the LSTM should be given

from __future__ import division, print_function, absolute_import

import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdb

train, val = something.load_data()
trainX, trainY = train #each X sample is a (16,4096) nd float64
valX, valY = val #each Y is a one hot vector of 101 classes.

net = tflearn.input_data([None, 16,4096])
net = tflearn.embedding(net, input_dim=4096, output_dim=256)
net = tflearn.lstm(net, 256)
net = tflearn.dropout(net, 0.5)
net = tflearn.lstm(net, 256)
net = tflearn.dropout(net, 0.5)
net = tflearn.fully_connected(net, 101, activation='softmax')
net = tflearn.regression(net, optimizer='adam',
                         loss='categorical_crossentropy')

model = tflearn.DNN(net, clip_gradients=0., tensorboard_verbose=3)
model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,
          batch_size=128,n_epoch=2,snapshot_epoch=True)


texts: Output value of a gradient boosting decision tree node that has just a single example in it

texts:
The general gradient boosting algorithm for tree-based classifiers is as follows:
Input: training set ${\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}{\displaystyle \{(x_{i},y_{i})\}_{i=1}^{n},}$a differentiable loss function ${\displaystyle L(y,F(x)),}{\displaystyle L(y,F(x)),}$ number of iterations M.
Algorithm:

Initialize model with a constant value:
${\displaystyle F_{0}(x)={\underset {\gamma }{\arg \min }}\sum _{i=1}^{n}L(y_{i},\gamma ).}F_0(x) = \underset{\gamma}{\arg\min} \sum_{i=1}^n L(y_i, \gamma)$
For m = 1 to M:

(a) Compute so-called pseudo-residuals:
${\displaystyle r_{im}=-\left[{\frac {\partial L(y_{i},F(x_{i}))}{\partial F(x_{i})}}\right]_{F(x)=F_{m-1}(x)}\quad {\mbox{for }}i=1,\ldots ,n.}r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F(x)=F_{m-1}(x)} \quad \mbox{for } i=1,\ldots,n.$
(b) Fit a base learner (or weak learner, e.g. tree) ${\displaystyle h_{m}(x)}{\displaystyle h_{m}(x)}$ to pseudo-residuals, i.e. train it using the training set ${\displaystyle \{(x_{i},r_{im})\}_{i=1}^{n}}\{(x_i, r_{im})\}_{i=1}^n.$
(c) Compute multipliers and solve for the output values of each leaf. $\gamma$ is the output value of a leaf here: ${\displaystyle F_{m}(x)=F_{m-1}(x)+\sum _{j=1}^{J_{m}}\gamma _{jm}\mathbf {1} _{R_{jm}}(x),\quad \gamma _{jm}={\underset {\gamma }{\operatorname {arg\,min} }}\sum _{x_{i}\in R_{jm}}L(y_{i},F_{m-1}(x_{i})+\gamma ).}$

Output ${\displaystyle F_{M}(x).}F_M(x).$

Now the loss function that they input is logo-loss: $\sum_i −(y_ilog(p_i)+(1−y_i)log(1−p_i))$ where $i$ is counter over all the samples in training set. The output of a tree leaf in the algorithm is assumed to be log-odds of samples belonging to the positive class.
In reality, how the algorithm is executed is - The $\gamma _{jm}$ is computed by expanding the Loss function in Taylor's series of degree 2 and then differentiating that w.r.t $\gamma$
I will write the formulation for the case when there is just one sample ($x1$) in a terminal node.
$$L(y_1,F_{m-1}(x1)+\gamma)=L(y_1,F_{m-1}(x1)) + \frac{\partial L(y_1,F_{m-1}(x1))}{\partial F_{m-1}}\gamma + \frac{1}{2}\frac{\partial^2 L(y_1,F_{m-1}(x1))}{\partial F_{m-1}^2}\gamma^2$$
Now, this formulation is differentiated w.r.t $\gamma$ and set equal to 0 to get the optimal value of gamma
$$\gamma = -\frac{\frac{\partial L(y_1,F_{m-1}(x1))}{\partial F_{m-1}}}{\frac{\partial^2 L(y_1,F_{m-1}(x1))}{\partial F_{m-1}^2}}$$
This computes to $$Residual_1/P_0(1-P_0) \tag 1$$ where $P_0$ is the most recent predicted probability.
The example below is taken from this youtube video - linked to exact time 16:43
For an example case where $F_0(X1) = 0.69$ (these are log-odds)
(converting the log odds to probability) $P_0 = 0.67; Residual_1 = y_1-P_0 = 0.33$
Now they compute $\gamma$ (the output value) for the leaf containing $x1$ by plugging these values into the equation (1) and get the $\gamma$ to be equal to $$\frac{0.33}{(0.67)(0.33)} = 1.49$$
Questions:

How can we approximate the loss function $L$ for each gamma. We need to know radius of convergence around that $\gamma$ I think and then the $\gamma$ thus computed at the end should have lied in that radius of convergence?
The log-loss (binary cross-entropy) is monotonic in predicted probability value. It increases with $P$ if $y = 1$ and decreases if $y = 0$ and thus it is also monotonic in log-odds since log-odds and probability are monotonic w.r.t each other.In that light why is $\gamma$ not equal to $as-high-as-numerical-stability-permists$ when $y=1$ and $0$ when $y=0$?


texts: Data augmentation for tabular data in a multi label classification task

texts: The task at hand is to predict the future lab values for a patient (1 if abnormal and 0 if normal) using the previous numerical data. It is a multi-label, multi-class time series classification task. I want to improve the training using data augmentation. What are the best ways of doing this? maybe SMOTE or GANs?


texts:
I'm trying to get grasp of Bayes regularization algorithm.
List of symbols 1st:
$F$ - objective function
$\gamma$ - regularization parameter
$M$ - number od neural network weights
$N$ - number of data tuples
$e$ - modeling error
$w$ - net weight
$D$ - data set input-output pair
$H$ - Hess matrix of objective function computed based on Jacoby's matrix

I get it that minimalising an objective function:
$$
F = \gamma\sum\limits _{j=1} ^Mw_j^2 + (1-\gamma)\sum\limits_{i=1}^Ne_i^2
$$
is equivalent to maximising likelihood $P(w|D,\gamma)$, which can be computed according to Bayes' theorem:
$$
P(w|D,\gamma) = \frac{P(D|w,\gamma)P(w|\gamma)}{P(D|\gamma)}
$$
Likelihood $P(w|\gamma)$ is assumed to be a gaussian one and can be computed as:
$$
P(w|\gamma)=\left(\frac{\gamma}{2N}\right)^{\frac{M}{2}}\cdot e^{-\frac{\gamma}{2}w^Tw}
$$
Likelihood $P(D|\gamma)$ can be computed too as:
$$
P(D|\gamma)=\left(\frac{\pi}{\gamma}\right)^{-\frac{N}{2}}\left(\frac{\pi}{1-\gamma}\right)^{-\frac{M}{2}}\frac{(2\pi)^\frac{M}{2}\cdot -e^{-F(w)}}{\sqrt{|H|}}
$$
And now my questions are: how is $P(D|w,\gamma)$ computed? Or maybe is there any special assumptions about it? And should I minimalise or maximise $P(w|D,\gamma)$ to minimalise objective function $F$.
I base on those articles:
Article1
Article2

If anyone is able to explain those thing to me, I'd be most grateful.
Thank you,
Max


texts: A Python library to convert text into character-level embedding and LSTM

texts:
I'm trying to recreate an experiment described in a paper where each character in a URL is converted to a 128-dimension embedding. My dataset looks like below.

I'm having trouble figuring out how to do the conversion process in Python. Can anyone suggest a library and function that will allow me to convert the domain column in my dataframe?

The paper describes the process in the diagram shown below.

Bahnsen, Alejandro Correa, et al. "Classifying phishing URLs using recurrent neural networks." 2017 APWG symposium on electronic crime research (eCrime). IEEE, 2017.


texts:
There is this dataframe:
print(df)

df.plot.bar(figsize=(10,5),fontsize=14,)
plt.title('counts',fontsize=15)

plt.show()

barh:
df.plot(kind='barh')

How can I add value labels on this bar and barh chart?


texts:
I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and other recommender systems, which does more or less what I want. I want to know if anyone here has any experience or resources on a more generalized version of this, where I am able to feed in representations on different objects, and learn how similar they are.

For example: Say we have some customers we are sending different advertisements to, and I would like to create a system to map offers to customers. I am thinking in the lines of creating a customer representation, and a representation of the offers, and feeding them in parallel to a neural network that has a label of whether they acted on the advertisement or not. The idea is that I should be able to locate the best offer for any customer given these representations.

I have looked into siamese networks and word2vec, both are close to what I want. The problem differs slightly in that  for the siamese networks, there are identical parallel networks, which I don't want because my inputs are not equivalent. Word2vec type methodology is also close, but I would want a model to process the inputs on "both sides". A combination of the two, is kind of what I am looking for.

If anyone has any resources on a similar problem statement, I would be very interested in it.

Thanks


texts: binary classification for counting - estimating the error on counts due to error on prediction score

texts:
Ok, so I have the following set up:

I have a binary classification problem and I am classifying events into signal and background. Ultimately, I want to count how many background events and how many signal events I have in an unknown data set.

I am using gradient boosted decision trees and I am training my model about 100 times, each time with random selected training and test set (which is called bootstrapping if I am correct).

Now I want to predict the class (signal or background) of instances in a new data set and I dont know the class. I use my ensemble of 100 trained models, so I get a prediction after each of the 100 rounds of training and testing. In the end I get 100 prediction scores (a number between 1 and 0) for each instance in the data set.
Afterwards, I can average over the 100 outcomes and get a mean and standard deviation for every data instance.

After setting a cut value on the prediction score (everything below that value is background and everything above is signal), I can count how many events lie above and how many below and that gives me a number of signal and a number of background events.

Now (finally) my question: can I somehow include the error I gain due to averaging over the 100 outcomes into the error on the counting?
Or is there a (Bayesian) way of not reducing the distributions to the mean and standard deviation but somehow continue with the prediction score distributions to obtain an value for number of background and signal counts?


texts:
One of our famous mathematicians, James Simons, used an extension of the Baum-Welch algorithm to 'crack' the wall street when he started trading on the stock market. Now, as Google, all informations his team used to trade are hidden to the public. I would like to do such thing, but it'll be very hard.

I think it should be great to start with using the Baum-Welch Algorithm. Some guys suggest me to look up A Spectral Algorithm for Learning Hidden Markov Models and Learning Parametric Output HMMs. Another recommend me just starting with deep neural network.

To implement a great model, what are you suggesting to use first? In fact,  I just don't know if I need to learn deep learning first or starting implement my algorithm. What is it advised to learn first?

Thanks in advance!


texts: Keras, re use convoluted layers training for various flattened layers training

texts:
I want to train a range of models, that are very similar; the convoluted layers are the same but the flattened part changes. Example; different density, initially I have model.add(Dense(512)) but I would also like to test 1000 and 4096; I would also like to test adding another layer.
I have no problem creating this models and running them, but it feels like a waste of time to retrain the convoluted layers every time when I'm not modifying anything in them.
Is it possible to save the training done in the convoluted layers and reuse that when training the flattened layers?
This is the base model I have:
model = Sequential()

model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))

model.add(Flatten())
model.add(Dropout(0.2))
# -------------------- Start of Variable Section --------------
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
# -------------------- End of Variable Section ----------------
model.add(Dense(1))
model.add(Activation('sigmoid'))

Other options I want to try (changes to be made inside the variable section) are:
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.2))

or
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))

etc.
To achieve that, where do I save the model? If I remove the flattened section:
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1))
model.add(Activation('sigmoid'))

The model accuracy is unsurprisingly ~none, if I leave the section that is not variable, like this:
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(1))
model.add(Activation('sigmoid'))

I can save the model and load it, but how would I add those flattened layers?


texts: Assigning new word by creating a new column when finding match

texts:
I have a survey df and I want to assign a new value of either existing customer
    or new customer based on their answers. if someone has 3 answers but one of
    them matches "coca cola" I want to give them the value of existing customer
     here is the data frame

ID       Question                                                Answer
101005   what brands did you purchase the past 5 months   Coca-Cola or Pepsi or vitamin water
026458   what brands did you purchase the past 5 months           None
045987   what brands  did you purchase the past 5 months        Coca-Cola

Here is the table I would like to have

ID        Question                                          Answer                      Buyer_Type
101005   what brands did you purchase the past 5 months  Coca-Cola,Pepsi,fanta          Existing Users
026458   what brands did you purchase the past 5 months  None                           New Buyer
045987   what brands did you purchase the past 5 months  Coca-Cola                      Existing Users

Here is what I tried

deux['Buyer_Type'] = deux['answer'].apply(lambda x: 'existing buyer' if x == 'Coca-Cola' else 'new buyer')

For some reasons it is not recognizing 101005 as an existing user.


texts: Can McNemar's test be applied to evaluate multiclass models?

texts:
Full Disclosure:

I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other.

tl;dr

For multiclass classifiers, can you apply McNemar's Test to determine, whether two classifiers are significantly different in how they categorize the same data? Or is McNemar limited to 2-class problems?

Detailed Question

I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Example sources:

example 1

example 2

However, I am not sure if these sources assumed binary classifiers.

Now I would like to know if I can apply McNemar's test my multiclass case. To illustrate let me give you an example. For that let's generate a bit of random data

>>> # number of categories
... k = 4
>>>
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)

Now, given this data, can I apply McNemar?

>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>>
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>>
>>> print(table)
[[559 186]
 [186  69]]
>>> from statsmodels.stats.contingency_tables import mcnemar
>>> print(mcnemar(table))
pvalue      1.0
statistic   186.0

Would the test be applied correctly like this? Or is McNemar limited to 2-class classifiers only?

My classes are imbalanced btw. if that is relevant.


texts: How to extract specific information from raw , unstructured text using NLP and Deep Learning?

texts:
I have data coming from different sources having similar information like the below example where different sources want to specify the age criteria. Is there a NLP or Deep learning based approach which I can use to extract the age rule as shown below from raw unstructured text.

A Criteria
Applicants should be above 21 years of age and up to 65 years or less at the time of maturity.

B Criteria
You are between 25-58 years of age.

C Criteria
Age of the all applicants should be between 21 and 65 .

A Rule
age >=21 AND age <= 65

B Rule
age >=25 AND age <= 58

C Rule
age >=21 AND age <= 65


texts:
I understand the backpropagation algorithm of neural networks, and how the error propagates backwards in layers.  That is, I understand that given a 3-layer feed forward network, the amount to change W1 is dependent on the weights from layers 2 and 3, as well as the derivatives of their activation functions.

Question: When your first layer is an embedding layer (i.e., consider initializing the embedding matrix with glove), how does the network update that matrix using backpropagation? How do you represent that layer as an equation consisting of the input and some weight matrix?


texts: While performing the SMOTE for balancing the class data, what should be the proportion of both class? For instance, if we have 100 instances, what (%) should be the Yes class and what should be the No class?


texts: How to locate the invoice within a camera captured image?

texts: The demand is to locate the invoice within a camera captured image about that invoice. The invoice is always a white paper with printed black or blue characters, tables and red stamps. Sometimes the background behind the invoice is dark(about 60% of all the samples), but others are not. Sometimes there is shadow on it. The question is how to detect the vertices, edges and corners of the invoice in this image? What algorithms should be applied? The android application CamScanner seems to have this function but not is effective every time. So does the CIDetector interface on iOS. What algorithms does CamScanner or iOS use(traditional or deep)?


texts: Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

texts:
I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in a comment on this question that that was wrong. You need to backpropagate the error of every item in the batch individually, then average the gradients you computed through the backpropagation and then update your parameters with the scaled average gradient.

Okay, but why is that not actually the same thing? Isn't the gradient of the average of some points the average of the gradient at these points?

The idea behind SVD is to find the minimum of a cost function $J(\theta)$ of a subset of training items. The cost function is usually defined as the average of some function $J_t(\theta)$ of the errors of the individual predictions and targets for a training item. Let's take MSE as an example. So if we have a batch of $N$ items, we have

$$J(\theta)  : X, Y \mapsto \frac{1}{N} \sum_{i=1}^N (y_i -f(x_i))^2$$

And we want to minimize $J(\theta)$. So we need to find its gradient:

$$\nabla \frac{1}{n}\sum_{i=1}^{n} (y_i - f(x_i))^2$$

But the derivative is linear, so

$$\nabla \frac{1}{n}\sum_{i=1}^{n} (y_i - f(x_i))^2 = \frac{1}{n}\sum_{i=1}^{n} \nabla (y_i - f(x_i))^2$$

What am I doing wrong here?

Another example. Say we do linear regression with an $m x + b$ line fit. Then the partial derivatives taken for $m$ and $b$ are

\begin{align*}
\frac{\partial J(\theta)}{\partial m}   &= \frac{1}{N} \frac{\partial}{\partial m} \sum_{i=1}^N (y_i -f(x_i))^2 & \texttt{factor rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N \frac{\partial}{\partial m} (y_i -f(x_i))^2 & \texttt{sum rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial m} y_i -f(x_i)  & \texttt{chain rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial m} y_i - (mx_i + b)  & \texttt{definition } f\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) (-x_i)  & \texttt{}\\
                                &= -\frac{2}{N} \sum_{i=1}^N  x_i(y_i -f(x_i))  & \texttt{comm., distr.}\\
\end{align*}

\begin{align*}
\frac{\partial J(\theta)}{\partial b}   &= \frac{1}{N} \frac{\partial}{\partial b} \sum_{i=1}^N (y_i -f(x_i))^2 & \texttt{factor rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N \frac{\partial}{\partial b} (y_i -f(x_i))^2 & \texttt{sum rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial b} y_i -f(x_i)  & \texttt{chain rule}\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) \frac{\partial}{\partial b} y_i - (mx_i + b)  & \texttt{definition } f\\
                                &= \frac{1}{N} \sum_{i=1}^N  2(y_i -f(x_i)) (-1)  & \texttt{}\\
                                &= -\frac{2}{N} \sum_{i=1}^N  (y_i -f(x_i))  & \texttt{comm., distr.}\\
\end{align*}

I don't see an error here and the gradient descent also works with these partial derivatives (tested through implementation). So... what am I missing?


texts: I'm not familiar with the ways and tools of data scientists but I have a background in software engineering and mathematics. I'd need to recognize patterns in a two-dimensional data set. I have a lot of measurements of a valve's power consumption over time. The valve is under test. I'd need to tell based on the measurements if the valve is working properly or not. I have sample measurements of properly working and not properly working valves. This is a custom valve so it has no datasheet. I think I could start investigating this on my own but some insight from you would be really great. For example, what are some good tools I can use to analyze the data? Is there any recommended standard way of doing this? Are there any good references I could follow? Thank you.


texts: How to access maximum volume of tweets using Twitter Streaming API, without firehose access?

texts:
Twitter is a popular source of data for many applications, especially involving sentiment analysis and the like.  I have some things I'm interested in doing with Twitter data, but here's the issue:  To get all Tweets, you have to get special permission from Twitter (which, as I understand it, is never granted) or pay big bucks to Gnip or the like.

OTOH, Twitter's API documentation says:

Few applications require this level of access. Creative use of a combination of other resources and various access levels can satisfy nearly every application use case.

Using the filter api with keyword tracking seems like something that would be a big part of this, but you obviously can't enumerate every keyword.  Using a User stream on many User accounts that follow a lot of people might be an option as well, and I'm not sure if it makes sense to think about using the search API in addition.

So here's the question "What combination of other resources and access levels is the best way to get the maximum amount of data from Twitter"?


texts: Document Layout Analysis - state-of-the-art?

texts:
What is the current state-of-the art within document layout analysis?

I.e. detecting columns, separating images from text, distinguishing between page numbering and text and so on.

I am looking for papers and algorithms on the topic.


texts: LDA vs Word2Vec vs Others for predicting recipients of a message

texts:
I'm investigating various NLP algorithms and tools to solve the following problem; NLP newbie here, so pardon my question if it's too basic.

Let's say, I have a messaging app where users can send text messages to one or more people. When the user types a message, I want the app to suggest to the user who the potential recipients of the message are?

If user "A" sends a lot of text messages regarding "cats" to user "B" and some messages to user "C" and sends a lot of messages regarding "politics" to user "D", then next time user types the message about "cats" then the app should suggest "B" and "C" instead of "D".

So I'm doing some research on topic modeling and word embeddings and see that LDA and Word2Vec are the 2 probable algorithms I can use.

Wanted to pick your brain on which one you think is more suitable for this scenario.

One idea I have is, extract topics using LDA from the previous messages and rank the recipients of the messages based on the # of times a topic has been discussed (ie, the message sent) in the past. If I have this mapping of the topic and a sorted list of users who you talk about it (ranked based on frequency), then when the user types a message, I can again run topic extraction on the message, predict what the message is about and then lookup the mapping to see who can be the possible recipients and show to user.

Is this a good approach? Or else, Word2Vec (or doc2vec or lda2vec) is better suited for this problem where we can predict similar messages using vector representation of words aka word embeddings? Do we really need to extract topics from the messages to predict the recipients or is that not necessary here? Any other algorithms or techniques you think will work the best? Should I just use supervised learning instead?

What are your thoughts and suggestions?

Thanks for the help.


texts: Is normalizing the validation set of time series a kind of look ahead bias?

texts:
Here's the data normalization process of a time series in a paper about stock prediction using LSTM:

Split train and test set based on time (e.g. training set: 2001-2010, test set:2011-2012). This looks fine to me.
Normalize the training set by subtracting the mean and dividing them by the standard deviation of the training set.
Train the data, using 20% of samples as the validation set. Keras's model, by default, uses the last 20% for validation.

So, in the training phase, the model knows a little about the validation set through the mean and standard deviation in step 2.

On one hand, the model is training using some of the future information. On the other hand, no information of the test set is leaked to the model. Is this a kind of look ahead bias?

And what is the best practices in this case?


texts: How to realize ARMA-GARCH using a LSTM model?

texts:
I am trying to model the returns of the S&P500 using a multivariate LSTM neural network to compare it to the ARMA-GARCH process. In order to do so,

I need to model the conditional mean of returns.
Then I need to model the conditionnal variance of the residuals (conditionnal mean - returns).

I am trying to create a single neural network that outputs both values; the mean of the returns and the variance of residuals. I can then use both these values to simulate the returns via Monte Carlo and retrieve the confidence intervals.
Can someone help me?


texts: Neural Network Timeseries Modeling with Predictor Variables

texts:
Many have shown the effectiveness of using neural networks for modeling time series data, and described the transformations required and limitations of such an approach. R's forecast package even implements one approach to this in the nnetar function.

Based on my reading, all of these approaches are for modeling a single outcome variable based on its past observations, but I'm having trouble finding a description of a neural-network-based approach that also incorporates independent predictor variables (a sort of ARIMAx analogue for neural networks). I've found references to Nonlinear autoregressive exogenous models (NARX), which seem like they should be what I'm looking for, but all the reading I've been able to find talks more about using this approach for multi-step-ahead prediction of a univariate series.

Can anyone point me in the right direction on this? For bonus points, does anyone know of an implementation of what I'm looking for in R?


texts: Does Gradient Boosting detect non-linear relationships?

texts:
I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn.

My questions are:

1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2, y increases as x approaches negative infinity and positive infinity. What if the graph looks like y=sin(x)?

2) Is the algorithm able to detect interactions/relationships among the features? Specifically, should I add features that are the sums/differences of the raw features to the training set?


texts:
Sklearn has an unsupervised version of knn and also it provides an implementation of k-means.

If I am right, kmeans is done exactly by identifying "neighbors" (at least to a centroid which may be or may not be an actual data) for each cluster. But in a very rough way this looks very similar to what the unsupervised version of knn does.

Then what is the difference between k-means and unsupervised knn?


texts:
in my business we handle all analytics through Excel. This includes mostly scheduling, production planning and accounting operations. We currently are looking into adding a bit of predictive modelling and Excel does suffice to a point, but doesn't have support for complex models.

As I see it, the main benefits of Excel are its ease of use and the ease with which you can find and train people to get accustomed to it. On the other hand more sophisticated environments (e.g. R, python) can handle a wider variety of analytics tasks, but require better trained individuals.

I have also read this question on if Excel is sufficient for Data Science and while it is a bit out of scope for my purpose, the conclusion is that tools like R and Python are much better than Excel.

My question is (in the context of data analytics): "How much far can we get with Excel, without needing to change to a more sophisticated tool?" or "At what point do we need to migrate from Excel to -let's say- R?"

Thank you very much!


texts:
I'm stuying the decision trees in Data Mining.
A weak point of the information gain criterion is that it can lead to an overfitting, a solution can be the use of the gain ratio criterion.

May this last criterion suffer from underfitting ? If yes, when ?


texts: A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?

texts:
For unsupervised text clustering, the key thing is the init embedding for text.
If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model.
BERT can not get good init embedding.
If we do not use deep model, is there better way to get embedding better than glove wordvec?


texts: Is there a proper way of comparing weight coefficients of two logits?

texts:
I'm currently working on web advertising topics and I'd like to know if it does make sense to, let's say, compare the resulting weights of two logits?

Let's take as an example the case where input variables are the same across models. A variable gets a weight of 0.22 as the result of the first logit, and a weight of 0.08 as the result of the second logit (all the other variables remaining the same, and their weights may change obviously)

May I conclude that this variable is of highest importance for explaining hidden patterns within the first dataset, or is such a comparison forbidden? (In fact, when thinking about the problem I couldn't find any obvious mistake...)


texts: Use of negative correlation coefficient in pearson correlation algorithm for recommender systems

texts:
I'm new in recommender systems and I try to find similar users of a base users for user-based collaborative filtering.
When I calculated the similarity score now between two users (based on there ratings with pearson algorithm [or resnick's weighted pearson algorithm]) I get a similarity score from -1 to 1.

Is it a good idea to normalize this values to 0 to 1 (-1 would become 0 and 1 would be 1) to make it comparable to other algorithms?

In fact I try to build recommendations and with a negative similarity score of a user the calculated/predicted rating could be negative as well which make no sense.

Should I normalize/scale "-1 to 1" to "0 to 1" or cut off all users with similarity score below 0?

(maybe the question also could be: "Which users should be taken as mentor to recommend new items on a similarity score from -1 to 1? Or should I take the top n users with highest similarity score?")


texts: Interpreting the new dimensions after PCA

texts:
I have telecom data with large number of dimentions. Now if I apply dimentionality reduction like PCA then from resulting dimention say PC1, PC2 I would loose the meaning or would not understand what they represent.

Are their any techniques other than PCA which can provide any meaning or intution about the new dimentions. Also suggest if there are any research papers in this.


texts: How to create more dimensions for data mining?

texts:
I want to analyze product sales data with WEKA. My data have attributes: sell quantity and sales price. I want to add more attributes, such as mean and standard deviation.

What formula I should use to create more dimensions?


texts: I am wondering why my stacked features do not help me to improve against my loss metric. Here's what I'm doing: I am adding new features which are simple the predictions originated from train, predict of other   models to the original train/test features. Every time I have tried this method, it has failed. I am curious  what the issue could be the with this. Can anyone give me some advice?