texts:
I am creating a mock of sales data. One of the columns is salesperson_id where each id can occur more than once (a salesperson can have multiple sales). I want to generate this column in such a way that if I create a chart of distribution of sale count (not total sales) per salesperson_id, it would be a normal curve.

Let's say I have an array of unique salesperson_id called salespersons, I want to generate n number of records. For example, I have 6 salespersons and I want to generate total of 14 records. This would be the distribution of their counts:

id  count
---------
A   1
B   2
C   4
D   4
E   2
F   1

If arranged in a bar chart, this would look like a normal distribution. I also need some way to control the 'flatness' of the chart.


texts: How can I calculate perplexity for a bigram model?

texts:
I didn't find any function in nltk to calculate the perplexity.

There are some codes I found:

def calculate_bigram_perplexity(model, sentences):
    number_of_bigrams = model.corpus_length #calculate_number_of_bigrams(sentences)
    bigram_sentence_probability_log_sum = 0
    print("num of bigrams", number_of_bigrams)
    for sentence in sentences:
        try:
            bigram_sentence_probability_log_sum -= math.log(model.calculate_bigram_sentence_probability(sentence), 2)
        except:
            bigram_sentence_probability_log_sum -= float('-inf')
    x = math.pow(2, bigram_sentence_probability_log_sum / number_of_bigrams)
    y = math.pow(2, nltk.probability.entropy(model.prob_dist))
    print(f"x = {x} and y = {y}")
    return y

in the code above x is the output of the function, however, I also calculated it from another method:

y = math.pow(2, nltk.probability.entropy(model.prob_dist))

My question is that which of these methods are correct, because they give me different results.
Moreover, my results for bigram and unigram differs:

== TEST PERPLEXITY ==
unigram perplxity:
x = 447.0296119273938 and y = 553.6911988953756
unigram:  553.6911988953756
=============
num of bigrams 23102
x = 1.530813112747101 and y = 7661.285234275603
bigram perplxity:  7661.285234275603

I expected to see lower perplexity for bigram, but it's much higher, what could be the problem of calculation? Please note that I process a text involving multiple sentences... could they be because of sparse data, because I just tested them on one text.


texts: Using a White noise image to minimise the loss in a (convolutional) neural network

texts:
I'm reading someone else's code, which uses vgg19 to transfer the style of an artist to a 'content' image. I'm wondering why they have put

INI_NOISE_RATIO * noise_img + (1. - INI_NOISE_RATIO) * content_img)

instead of just noise_img (INI_NOISE_RATIO=0.7 here).

Why use a mix of the noise image and content image as input for the white noise bit in the content loss? The paper https://arxiv.org/pdf/1508.06576.pdf by Gatys et.al doesn't mention the mixing?


texts: How to get the probability/closeness of a sample belonging to a specific cluster?

texts:
I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect

I got a set of 4 clusters from k-means and now I'm interested in the Cluster No. 1. For this cluster, I take the average of all values for each column and keep it aside.

Now, I have a test sample, for which I run k-means prediction and I get output as 1, meaning it belongs to Cluster No. 1 which is good for me but my use-case here was to calculate that even if that sample didn't belong to Cluster 1, how close was it to falling in that Cluster No. 1

Hence, to resolve this I thought of doing a cosine similarity between my test sample and the one where I take average of all values for each column. Now, in this case, I get a similarity of just 5%

I'm not sure, for my use-case i.e.  (Getting the probability/closeness of a sample belonging to a specific cluster) which is a better interpretation for me?

I know I can use the cluster labels as y variables and make multi-class classification model but I want to keep it as un-supervised as possible. Please guide


texts: Validation Accuracy / Loss plateaus regardless of learning rate (VGG 16)

texts:
I am trying to classify my dataset into two categories using transfer learning with vgg and finetuning the very last layer (fully-connected). When I plot the graph of value vs epochs, I get the following graph:

Each graph shows a unique, different learning rate. It seems like the validation accuracy / loss always plateaus regardless of which learning rate I set; is this the sign of overfitting? What could potentially be the cause of this?


texts: How does backpropagation works through Max Pooling layer when doing a batch?

texts:
Let's assume that we are using a batch size of 100 samples for learning.

So in every batch, the weight of every neuron (and bias, etc) is being updated by adding the minus of the learning rate * the average error value that we found using the 100 samples * the derivative of the error function with the respect to the current neuron weight that is being updated.

Now, when we use a Max Pool layer, how can we compute the derivative over this layer? In every sample that we feed forward, a different pixel (let's say) is chosen as the max, so when we backpropagate over 100 samples in which every time a different path was chosen, how can we do it? A solution I have in mind is to remember every pixel that was chosen as the maximum, and then maybe split the derivative over all the max-pixels. Is this what's being done?


texts: Has anyone heard of a model similar to a random forest which fits a linear regression model in its leaf nodes?

texts:
That is, each leaf node in each decision tree learns a linear model.
Anyone heard of this kind of model? Even better, anyone know of implementations?


texts:
I have been familiarizing myself with Word2Vec and Doc2Vec. After reading multiple papers including the the ones by T Mikolov (the creator of Doc2Vec), I am not clear on how does the neural network for Doc2Vec looks like.

I get Word2Vec: a neural network with 3 layers (1 hidden). The input layer has all words in the vocab. The hidden layer is the same size as that of embedding, while the output layer, again the same size as vocab. This is, of course, the CBOW framework.

How does Doc2Vec change things?


texts: Best practices to image annotation for object detection when objects overlap

texts:
If I have the following example:

How should I annotate the bottom image? I can think of those scenarios:

Create a large box that captures class B and a second box that captures entirely class A. This will lead to overlapping boxes.
Result:

Create two boxes that do not overlap, but also do not covers the objects entirely.
Result:

I think the right choice is scenario 1, but will the algorithm be able to capture this difference? Considering SSD, YOLO.


texts:
Is the size of the output vector of all machine learning algorithms the same? Can't an ML algorithm predict only one value as output?

I have trained an LSTM network with X, Y, heading, speed(from taxi sensor) against feature_x, feature_y. So given a point, I can predict the next one.

Given the LSTM is many-to-one type how does it produce two values as output(x,y)?

Could an MLP also do that?


texts: How to treat a categorical variable which can have multiple values in regression?

texts:
I am working on a regression problem and has a categorical feature 'CATEGORY' which can take as many as 1600 categories.

On top of that, each observation can have any number of categories in that column. For eg.
CATEGORY column of row 1 has 'a' (just one category 'a') but CATEGORY column of row 2 has 'a','b','x' (three categories 'a','b','x') and so on.

What is the most optimal way to deal with such feature for regression problem other than using one-hot encoder or hash vectorizer?


texts: when to use dot product and when to use the common product In neural networks?

texts:
I wanted to know when to use dot product and when to not , I also don't know when we must transpose an array and why should we , could someone help me to understand this ?
If you could give me the derivative of cost function with respect to each Weigh (W1, W2, W3) using notations bellow to compare it with the results I found I'll appreciate that !


texts:
I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set!

So does it make sense to use google translator API to translate the English dataset to German and then train the model on it?

Any idea where I can find German datasets or solve this issue?


texts: I have a problem which is I'm building a chatbot that is used for some booking systems like event booking, I have built it with rasa and botfront for my personal use. So I'm moving to make it as SAAS product where many people can onboard themselves and make there own Dialogflow and deploy it into there website/FB messenger, chatbots that popup in the websites and ask us the question that based on client's dialog-flow. is there any way to do that, or how they did that.


texts: ngram and RNN prediction rate wrt word index

texts:
I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence :

I was expecting to see a plateau sooner on the ngram setup since it needless context. However, one thing I wasn't expecting was that the prediction rate drops.
In my understanding since we already have a context of 3 words, the plateau should converge asymptotically to its highest value.
But both the recurrent network and the Ngram models are experiencing this drop.
I have no idea why it would be.

(Note RNNLM is the name of the framework used to build the recurrent neural net, it uses 500 neurons and 100M direct connections, RNN25 is the same setup but with a training base divided by for)

Here is the sentence size distribution :

Thanks in advance.


texts:
Wanting to understand "the hashing trick" I've written the following test code:

import pandas as pd
from sklearn.feature_extraction import FeatureHasher
test = pd.DataFrame({'type': ['a', 'b', 'c', 'd', 'e','f','g','h']})
h = FeatureHasher(n_features=4, input_type='string')
f = h.transform(test.type)
print(f.toarray())

In the above example, I'm mapping 8 categories into 4 columns, and the output is:

[[ 0.  0.  1.  0.]<-a
 [ 0. -1.  0.  0.]<-b
 [ 0. -1.  0.  0.]<-c
 [ 0.  0.  0.  1.]<-d
 [ 0.  0.  0.  1.]<-e
 [ 0.  0.  0.  1.]<-f
 [ 0.  0. -1.  0.]<-g
 [ 0. -1.  0.  0.]]<-g

In the resulting matrix, I can see repetitions and some categories are represented the same way. Why is that? 8 categories can be mapped into 4 columns if I use a binary representation.

Can someone please explain the output of this technique and maybe elaborate a bit?


texts: improving performance for a limited dataset with noisy images, pattern recognition

texts: I am trying to recognize doodles in noisy images like in this one below. My dataset consists of only 10 000 images and 30 categories I've implemented a CNN but it is giving me a 6% accuracy. I am thinking about removing the noise before feeding the images to my CNN, but I have no idea which methods to use to remove this type of noise and I am not even sure that removing noise improves the NNs performance. Do you have any suggestions about methods to remove the noise or other tips to improve the performance of the model?


texts: PowerBI: Yes/No if date falls within moving month-based period (REVISED)

texts:
Goal: I am trying to filter a line chart that has trends both 3 months back and 6 months out (a total of a 10 month block).

Attempted Method: I tried to create a binary DAX-based yes/no filter as a column in a selected query based on the End date column.

I have the code for my attempt at creating this formula shown below.

My code:

DateBridge =
VAR TodayMonthYr = VALUE(FORMAT(today(),"YYYYMM")
VAR EndMonthYr = VALUE(FORMAT([End],"YYYYMM")

RETURN
IF(
   AND(
       EndMonthYr > TodayMonthYr - 3 ,
       EndMonthYr < TodayMonthYr + 6
   ),
   "Yes" , "No"
)


texts: How do I know what the best number of layers is required to achieve the highest accuracy

texts:
I'm learning from Udacity using this video.
I saw this piece of code:
model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                    #   nn.Linear(64, 32),
                    #   nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1)

        # TODO: Training pass
        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        if e == 1 and running_loss == 0:
            print(loss)

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

Then, I was playing around with layers and neurons.
I have tried:

Adding 1 more layer (64, 32) between (128, 64) and (64, 10), and changing the (64, 10) to (32, 10), increasing the epochs.
Replacing the (128, 64) with (128, 32), increasing the epochs
Adding 1 layer (784, 256), (256, 128), increasing the epochs

Etc.
From what I saw here, they can all achieve good accuracy with enough epochs of training.
Then, my question is how can I find the best number of layers and neurons in each layer (architecture) which can:

Achieve the highest accuracy
Still be the simplest model (least number of parameters)


texts: Chance Curve in Accuracy-vs-Rank Plots in matlab

texts:
I'm working on an image classification problem where each test image (query image) is compared with a set of candidate images (of which several images are considered as "correct answers" or "relevant"). From this comparison, a similarity measure is computed. The candidates are sorted decreasingly based on their similarity to the query image. So, basically, we are generating a ranking such as the ones generated by search engines. To evaluate such systems, it's typical to use mean average precision (MAP) and precision-vs-recall plots. However, I came across something with which I'm not familiar, which is the rank-vs-identification plot. From its name, I assume that this plot shows the identification rate at each rank. My problem is how to compute the chance curve for such a plot. Specifically, I have the following piece of code, which was extracted from the code used to generated Figure 3 of the following paper:

http://crcv.ucf.edu/papers/cvpr2014/Resemblance_CVPR14.pdf

This code is for the chance curve only. I am hoping that someone would help me understand how the chance plot was generated and what the variables x and xMax stand for.

%%A snippet of Matlab code:

fSize = 15;

% Compute Chance
    xMax = 54;
     ranks = [1 2 3 4 5:5:xMax]';
     x = 8;% x value in the equation.
     c = ones(length(ranks),1); % the equation that calculates the chance     value
  for r = 1 : length(ranks)
    if ranks(r) >= xMax; break; end
%     c(r) = 1 - (nchoosek(xMax-1,ranks(r)) / nchoosek(xMax,ranks(r)));
%     c(r) = 1 - (xMax - ranks(r))/xMax;
    c(r) = 1 - newc(xMax-ranks(r),x) / newc(xMax,x);
end

leg1 = {'Chance'};
figure(1);
semilogx(ranks,c,'linewidth',3,'color',[1,0,0]); set(gca,'FontSize',fSize);
hold on;
 psAll = zeros(length(ranks));
 figure(1); legend(leg1,'Location','SouthEast','FontSize',fSize); grid; xlim([1 1000])
 xlabel('Rank','FontSize',fSize,'FontWeight','bold'); ylabel('Identification Rate','FontSize',fSize,'FontWeight','bold');
  xMaxStr = num2str(xMax);
   set(gca,'XTick',[1 10 100 1000]);

% newc function used in the above piece of code

function res = newc(n,x)

res = n - x + 1 : 1 : n;
res = prod(res);


texts: Policy Gradients - gradient Log probabilities favor less likely actions?

texts:
Assume we work with neural networks, with the policy gradients method. The gradient w.r.t to the objective function $J$, is an expectation.

In other words, to get this gradient $\nabla_{\theta} J(\theta)$, we sample N trajectories, then average out their gradient contribution to obtain a more precise value that can "begin flowing into our network" during backprop.
$$\nabla_{\theta} J(\theta) \approx \frac1N \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t}) \right) \left( \sum_{t=1}^T r(s_{i,t}, a_{i,t}) \right)$$

Looking with more detail at some specific trajectory $i$, we sum the gradient at each timestep in this trajectory, then multiply by the the total reward obtained from running that trajectory.

Assuming we use softmax as the network's final layer, the values for each action belongs to the range $[0,1]$

If I remember correctly, the gradient wrt $\theta$  at each timestep of a trajectory is going to be:
$$\nabla_{\theta}\log \pi_{\theta}(a_{t}|s_{t}) = \frac{1}{\pi_{\theta}(a_{t}|s_{t})} \nabla \pi_{\theta}(a_{t}|s_{t})$$

where $\nabla \pi_{\theta}(a_{t}|s_{t})$ is then simply a derivative of softmax wrt its inputs, with the rest of the backprop towards the weight $\theta$

Question:

Let's say at timestep $t_2$ our softmax has output something like this:

[0.5,  0.1,  0.1,  0.1,  0.1,  0.1]

and the total reward for the trajectory was 50

Ignoring the rest of the backprop at the current timestep, $\frac{1}{\pi_{\theta}(a_{t}|s_{t})}$ will give us:

[2,  10,  10,  10,  10,  10]

This means we are already favoring other actions instead of the first action.  This seems counter-intuitive to me.  What if we only got such a large large reward because we took the first action? But the formula encourages us to strengthen the other 4 actions.


texts: Proper loss for multiclass 2D segmentation with coarse masks

texts:
Trying to figure out which is the proper loss for the task that has:
High dimensional grayscale images with relatively small coarse annotated masks
With a high imbalance in classes, as bg is x100 times more frequent than the other two classes.
I have used dice loss and categorical focal loss without class weights(it didn't seem to help)
Also used 100 as a smooth parameter for Dice loss.
What is the loss that could deal well with an imbalanced dataset with coarse masks?


texts: Negative Rewards and Activation Functions

texts:
I have a question regarding appropriate activation functions with environments that have both positive and negative rewards.

In reinforcement learning, our output, I believe, should be the expected reward for all possible actions. Since some options have a negative reward, we would want an output range that includes negative numbers.

This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.

So two questions:

If you do want to have both negative and positive outputs, are you limited to just tanh and linear?

Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?


texts: Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification]

texts:
I'm following this guide to try creating both binary classifier and multi-label classifier using MeanEmbeddingVectorizer and TfidfEmbeddingVectorizer shown in the guide above as inputs.

Both embedding vectorizers are created by first, initiating w2v from documents using gensim library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.

After this step has completed, I tried using these embedding vectorizers as inputs into several model such as OneVsRestClassifier(SVC), RandomForestClassifier and ExtraTreesClassifier. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.

These are the accuracies from each models (binary classifer) :

randomF_countVect: 0.8898
extraT_countVect: 0.8855
extraT_tfidf: 0.8766
randomF_tfidf: 0.8701
svc_tfidf: 0.8646
svc_countVect: 0.8604
ExtraTrees_w2v: 0.7285
ExtraTrees_w2v_tfidf: 0.7241

Multi-label classifier also produced similar result.

I'm not sure that I've done wrong.

Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?

Word2vec initiation

model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}

Models

svc = Pipeline([("count_vectorizer", vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
svc_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
randomF = Pipeline([("count_vectorizer", vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
randomF_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
extraT = Pipeline([("count_vectorizer", vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
extraT_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
etree_w2v = Pipeline([("word2vec vectorizer", MeanEmbeddingVectorizer(w2v)), ("word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
etree_w2v_tfidf = Pipeline([("tfidf word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("tfidf word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])

all_models = [
    ("svc", svc),
    ("svc_tfidf", svc_tfidf),
    ("randomF", randomF),
    ("randomF_tfidf", randomF_tfidf),
    ("extraT", extraT),
    ("extraT_tfidf", extraT_tfidf),
    ("etree_w2v", etree_w2v),
    ("etree_w2v_tfidf", etree_w2v_tfidf)
]
scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean())
             for name, model in all_models],
            key=lambda args: -(args[1]))


texts:
I have the following data set:

I want to use attributes Tags and Authors to classify each record into their respective Rating. In order to do so I want to use a random forest classifier. My concern is how to deal with Tags attribute. Each of the entry has an undetermined number of tags separated by a commas. There are a total of 4412 unique tags and the entry with more tags contains 20 tags. The first entry has tags ["Rhode Island","Economy", "Taxes", "Lincoln Chafee"].
How should I encode this attribute such that I can use Random Forest Classifier from sklearn?


texts: What is the relation between input into LSTM and number of cells?

texts:
I want to train an LSTM network for time-series predictions, and want to get to the bottom of LSTM's.
In my understanding, the number of cells in a single LSTM layer can vary. However, since each cell takes an input at time-step t, wouldn't the number of cells need to be equal to t?

For example (from TensorFlow tutorial):

 t=0  t=1    t=2  t=3     t=4
[The, brown, fox, is,     quick]
[The, red,   fox, jumped, high]

words_in_dataset[0] = [The, The]
words_in_dataset[1] = [brown, red]
words_in_dataset[2] = [fox, fox]
words_in_dataset[3] = [is, jumped]
words_in_dataset[4] = [quick, high]
batch_size = 2, time_steps = 5

Wouldn't the maximum number of cells in each layer be 5? since after the 5th input we no longer have any other information to input. However, I've seen many networks with a higher number of cells than that. Therefore, why is this a possibility?


texts: I get this error Expected Ptr for argument '%s' while resizing image in open cv2

texts:
This is my code:

# loading image from folder
folders = glob.glob('path to folder ')

imagenames_list = []

for folder in folders:
    for f in glob.glob(folder+'/*.jpg'):
        imagenames_list.append(f)

read_images = []

for image in imagenames_list:
    read_images.append(cv2.imread(image, cv2.IMREAD_GRAYSCALE))

re_size = cv2.resize(read_images, (50,50), interpolation=cv2.INTER_LINEAR)

I get this error

Expected Ptr<cv::UMat> for argument '%s'

What does this mean and how can I fix it?


texts:
Multi-State Time Delay Neural Networks (MS-TDNNs) were introduced in

  Haffner, Patrick and Waibel, Alex: Multi-state time delay networks for continuous speech recognition. In Advances in neural information processing systems, 1992.

They are an extension of TDNNs. TDNNs are convolutional neural networks for automatic speech recognition (ASR), where the convolution happens over the time.

The aim of MS-TDNNs seems to be to get rid of the hybrid approach in ASR, where you need dynamic programming / HMMs to chunk the audio stream and then neural networks to recognize the phonemes. Somehow MS-TDNNs seem to do the segmentation as well.

I don't understand how. Could somebody please explain it to me?

(Related side questions: Are MS-TDNNs recurrent networks? Where exactly does the name "multi-state" come from?)


texts: lags number in multivariate time series analysis correlation

texts:
I am trying to calculate the correlation coefficient(Pearson's r) of a financial time series $Y(t)$ and other exogenous variables $X_1(t),..., X_n(t)$. I am trying to understand the impact of my exogenous variables in my target variable.
I am wondering if I sould calculate Pearson's r between $Y(t)$ and $X_1(t),...,X_n(t)$ or with $X_1(t-i),...,X_n(t-i)$ (lagged values of exogenous variables) and if so, how can I determine the timesteps of lagged values ($i$)


texts: How can I dynamically distinguish between categorical data and numerical data?

texts:
I know someone who is working on a project that involves ingesting files of data without regard to the columns or data types. The task is to take a file with any number of columns and various data types and output summary statistics on the numerical data.

However, he is unsure of how to go about dynamically assigning data types for certain number-based data. For example:

    CITY
    Albuquerque
    Boston
    Chicago

This is obviously not numerical data and will be stored as text. However,

    ZIP
    80221
    60653
    25525

are not clearly marked as categorical. His software would assign the ZIP code as numerical and output summary statistics for it, which does not make sense for that sort of data.

A couple ideas we had were:

If a column is all integers, label it as categorical. This clearly wouldn't work, but it was an idea.
If a column has fewer than n unique values and is numeric, label it categorical. This might be closer, but there could still be issues with numerical data falling through.
Maintain a list of common numeric data that should actually be categorical and compare the column headers to this list for matches. For example, anything with "ZIP" in it would be categorical.

My gut tells me that there is no way to accurately assign numeric data as categorical or numerical, but was hoping for a suggestion. Any insight you have is greatly appreciated.


texts: Sensitivity to scaling of features in a multivariate gaussians

texts:
I'm using the HMMLearn python package for hidden markov models.  That implementation is build on multivariate gaussian distributions.

So I have a string of features.  How sensitive are gaussians to vastly different feature scales?  Will it be really skewed if one feature is scaled between 0 and 1, and another is scaled between 0 and 1e8?


texts:
I am using statsmodels in python to perform quantile regression. I am running quantile 99 and quantile 1 to see the extremes quantile of my distribution.

from statsmodels.regression.quantile_regression import QuantReg
from sklearn.datasets import load_iris
import pandas as pd

iris = pd.DataFrame(load_iris()['data'])

y = iris[0]
x = iris.iloc[:, [1, 2, 3]]

qt = QuantReg(y, x)
res99 = qt.fit(q=.99)
res01 = qt.fit(q=.01)
print('Quantile 99 description')
print(res99.predict(x).describe())
print('--------------------')
print('Quantile 01 description')
print(res01.predict(x).describe())
print('--------------------')
print('Target description')
print(y.describe())

The output is

Quantile 99 description
count    150.000000
mean       6.713570
std        1.092909
min        3.972447
25%        5.798665
50%        6.745737
75%        7.447919
max        9.906085
dtype: float64
--------------------
Quantile 01 description
count    150.000000
mean       5.196659
std        0.811265
min        3.110156
25%        4.555466
50%        5.270011
75%        5.777980
max        7.588576
dtype: float64
--------------------
Target description
count    150.000000
mean       5.843333
std        0.828066
min        4.300000
25%        5.100000
50%        5.800000
75%        6.400000
max        7.900000
Name: 0, dtype: float64

That is, the distributions of the percentile 1, the target, and the percentile 99 are not supper different. Is this expected or am I missing something?


texts: When to use t-distribution instead of normal distribution?

texts:
According to the Student's t-distribution wiki article the t-distribution is used instead of the Normal distribution "when estimating the mean of a normally distributed population in situations where the sample size is small and the population standard deviation is unknown".

An unknown population standard deviation implies that it would have to be estimated from the samples itself which is inaccurate with small sample sizes. According to the Z-test wiki article a sample size >= 30 implies the use of a normal distribution, a sample size < 30 implies the use of the t-distribution. (t-test for reference.) Is this assumption common best practice? How does this relate to sample size determination (estimation of a mean)?


texts: Find frequent segments of GPS trajectories

texts:
I have a dataset containing GPS coordinates (latitude and longitude), a timestamp variable and a subject ID to identify different persons. What I want to explore is two things:

Find frequent routes (or segments of route) that are shared among the subjects, for example people going from different places where their home is located and go to work taking the main highway.
Find related subjects. Related to the previous one, which are the subjects that are similar (taking into account the time they use to commute, for example)

I looked at Dynamic Time Warping as shown here but it doesn't take into account that for example, subjects can come from different points (but share a segment).

Tried to implement K-Means, but I'm not sure how my centroids are defined (if I should take into account the time or not) as discussed in here, not sure if the similarity distance is making the correct assumptions in this case.

I looked at some papers on the mater but often they work with pre-defined points A and B and try to cluster, however they hint on some good strategies to transform the data.

I know I'm missing the name of the concept of what I'm looking for, maybe you can guide me on the algorithm/papers to review.

Thanks in advance.


texts:
Applying SVC (Support Vector Classifier) to the 1-d data shown here:

What will be the support vectors for the parameter cost C=0 and C=Infinity?
As far as I read about SVM and hyper parameter C I would say:

For C = 0 there will be no support vectors

For C = Infinity all of the data points will be support vectors

Is it correct?
Note that I'm talking about SVC not SVM


texts:
Background
I have a massive text block of poorly written statements, some facts, some opinions. It's all scraped from various public internet resources and all in english.
I am now using nltk in python to try to work out some basic statements from this massive text block.
My goal is not to capture all statements, false positives are not too bad either. I would simply like to get a few probably true and simple assertions from this text block.
Examples

Bob is married to Lisa since 1983.
Lisa's cat Snowball died two years ago.
Bob and Lisa has a 30 year old son named George.

From these simple sentences, I would like to parse out some simple facts, like:
"Bob is married", "Lisa is married", "George is Lisa's son", "George is Bobs son", "Snowball is a cat"
It does not need to be exactly this.
Actual question
What is this kind of processing called? (Except for that it is a subcategory of Natural Language Processing.)
What are the simpler and the harder approaches to this problem?
How well has this problem been solved by public and proprietary algorithms?


texts: Which model could I use for time-series regression?

texts:
I have a time-series (sensor data) with 2 variables x and y.
                         x     y
2020-05-20 05:15:00    1192  355675
2020-05-20 05:30:00    1171  357600
2020-05-20 05:45:00    1260  392680

Rather than traditional time-series prediction with autoregression, my goal is to estimate y given x and the timestamp.
I've gotten acceptable results using non-temporal models (regression, ensembles, etc), but this way I'm throwing away the time information. The other model I've considered using is Vector Autoregression - given that Granger Causality exists between the two variables - but my understanding is that this requires autoregression input from both variables.
What would be the best choice of model for the relationship between these variables over time? My goal is to be able to use only sensor x, and estimate probably values of y.


texts: Custom loss for low false positive rate (higher precision)

texts:
I am working with a scenario where I need to minimize the false positive rate for the minority class. Additionally my dataset is imbalanced. (10% minority class, 90% majority class). I am using the class_weight in the fit function of keras.

Additionally, I would also like to try a custom loss function to see if this makes a difference. A number of solutions online Keras custom loss function as True Negatives by (True Negatives plus False Positives) discuss a specificity/precision etc loss function. However this cannot be derived, so I don't think this can work.

Any suggestions?

Thanks in advance!


texts: PCA: projection of positive data on negative side of plane

texts: I did PCA on my data and projected the data on first two eigen vectors. After projection I see that the scatter plot of the data starts from [-1,-1]. My data is all positive. Is it correct for the data to be negative in the projected space.


texts: Cross validation for periodic signal in time series

texts:
I have a dataset of 90 periodic signals (current signals). These signals are divided into two big areas: Fault-free area and Fault area.

I sliced the signals using a window of 80ms with an overlap of 40ms in order to increase the dimensionality of the original dataset. Each window is labeled according to the part of the series which it was extracted from. Such approach generated a lot of points which are essentially the same due to the periodicity of the signal. As a consequence, when I try to fit my model and perform cross validation, I receive insane results such as 0.9999% accuracy. This is due to the fact that I  have way too similar data points in my training set and in my validation set.

How can I solve this problem? Do you have any ideas?


texts: statsmodels: significance test (claim test), one sample, mean value - any function available?

texts:
We sample the bacteria contents of drinking water. Fact: when bacteria contents > 88, water is not safe to drink. These are the samples we've obtained:
bac = np.array([248, 37, 146, 19, 66, 236, 164, 30, 13, 144, 242, 20])

Null hypothesis: this water is safe to drink. Test the hypothesis.
With scipy I can do it easily:
from scipy import stats
res = stats.ttest_1samp(bac, 88)
t_statistic = res.statistic
P_value = res.pvalue / 2 # one-sided

P_value is 0.18 so we cannot reject H0.
Is there a way to do the same calculation in statsmodels? Does that library have a function similar to ttest_1samp()?


texts: Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

texts:
I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures precision, all ROUGE implementations I have come across also output precision and the F-score. The original ROUGE paper only briefly mentions precision and the F-score, therefore I am a bit unsure about what meaning they have to ROUGE. Is ROUGE mainly about recall and the precision and F-score are just added as a compliment, or is the ROUGE considered to be the combination of those three scores?
What confuses me even more is that to my understanding ROUGE-1 precision should be equal to BLEU when using the weights (1, 0, 0, 0), but that does not seem to be the case.
The only explanation I could have for this is the brevity penalty. However, I checked that the accumulated lengths of the references are shorter than the length of the hypothesis, which means that the brevity penalty is 1.
Nonetheless, BLEU with w = (1, 0, 0, 0) scores 0.55673 while ROUGE-1 precision scores 0.7249.
What am I getting wrong?
I am using nltk to evaluate BLEU and rouge-metric for ROUGE.


texts:
How much Hardware is required to train it well?(My current PC specs: 8GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU))
I have a dataset of approx 1lakh records.Is it is necessary to train BERT model on such large records or it will give better results on less records as well?
Is BERT capable to classify unidentified text which is not present in train data but in test data?


texts: Optimising character pair substitution costs in Levenshtein distance

texts:
In a typical edit distance algorithm - say Levenshtein - there are hardcoded costs for specific operations, such as insertion, substitution, and deletion. This is obviously a bad assumption (the substitution C->K is small in Capital->Kapital, while the substitution S->T is high in Sailor->Tailor).
After some research, I've found that this problem has been tackled in past, though not on a character cost weighted level.
Can someone think of / propose a way to learn the costs for different substitutions between different character pairs? (I think it best to focus on a single transformation operation to begin with).


texts: Feature Selection Algorithm for Attributes with Logical Relationships (like "AND")

texts:
I'm looking at datasets where the the attributes and the target class have a logical relationsships. All attributes and the target class are binary.

Here's an example: Neither feature#1 nor feature#2 have a significant correlation with the target class. But the conjunction feature#1 AND feature#2 is correlated with the target class.

Are there any Feature Selection Algorithms able to cope with situations like that? I'm thinking that frequent itemsets could be useful.
It would be tremendously helpful, if anyone could point me to a related paper.

Cheers!


texts:
I am having the problem with the following python code which uses the library keras, and I have not been able to find a solution online:

       def model_build():
        model_mag=Sequential()
        model_mag.add(Dense(64, activation='relu', input_dim=5))
        model_mag.add(Dropout(0.5))
        model_mag.add(Dense(64, activation='relu'))
        model_mag.add(Dropout(0.5))
        model_mag.add(Dense(64, activation='relu'))
        model_mag.add(Dense(1))
        model_mag.compile(loss=losses.mean_squared_error,optimizer='sgd')
        return model_mag
       clf = KerasRegressor(build_fn=model_build(),nb_epoch=100)
       # here is the error
       clf.fit(X_train,y_train, nb_epoch=20)

I know that X_train and y_train are well defined above and I can run other classifier methods just fine with them so they are not the problem. It keeps throwing me the following error in the fit line and I can't figure out why:

TypeError: call() takes exactly 2 arguments (1 given)

I am giving two arguments though.


texts:
I have used Xgboost fitted a model with AUC around 0.73 and I printed out my last booster:

booster[599]:
0:[userkn_hometypecnt<22] yes=1,no=2,missing=1
    1:[userkn_60d_opencardniu_days<40] yes=3,no=4,missing=3
        3:[userkn_30d_opencardniu_days<13] yes=7,no=8,missing=7
            7:[userkn_60d_opencardniu_days<24] yes=15,no=16,missing=15
                15:[userkn_timeminperiod_firstday<1029] yes=29,no=30,missing=29
                    29:leaf=0.000352735
                    30:leaf=-0.0100666
                16:[userkn_rate_aopencardniusum_actiondaycnt<0.972506] yes=31,no=32,missing=31
                    31:leaf=0.000398097
                    32:leaf=-0.0129448
            8:[userkn_hometyperate<0.0977183] yes=17,no=18,missing=17
                17:leaf=0.0239075
                18:[userkn_rate_aopencardniusum_actiondaycnt<0.957994] yes=35,no=36,missing=35
                    35:leaf=-0.00201536
                    36:leaf=0.00858442
        4:[userkn_newacitoncntactiondayavg<8.82511] yes=9,no=10,missing=9
            9:[userkn_mingap_importcard_open<297306] yes=19,no=20,missing=19
                19:[userkn_rate_aopencardniusum_actiondaycnt<0.974763] yes=37,no=38,missing=37
                    37:leaf=-0.0138254
                    38:leaf=0.00521038
                20:[userkn_onlinetime_firstday<1961.5] yes=39,no=40,missing=39
                    39:leaf=0.0247849
                    40:leaf=-0.00297016
            10:[userkn_60d_opencardniu_days<59] yes=21,no=22,missing=21
                21:[userkn_rate_repeatcntmaxactionrepeatcnt_actioncnt<0.124787] yes=41,no=42,missing=41
                    41:leaf=0.0101992
                    42:leaf=-0.0222082
                22:leaf=0.0145614
    2:[userkn_hometyperate_firstday<0.25266] yes=5,no=6,missing=5
        5:[userkn_aenterapplyloanpagecntactiondayavg<0.787338] yes=11,no=12,missing=11
            11:[userkn_newacitoncntactiondayavg<8.48678] yes=23,no=24,missing=23
                23:[userkn_worktimeactionrate<0.36514] yes=43,no=44,missing=43
                    43:leaf=-0.0178327
                    44:leaf=0.0168168
                24:leaf=0.0254048
            12:[userkn_newacitontyperate_firstday<0.794737] yes=25,no=26,missing=25
                25:[userkn_newacitoncntactiondayavg<7.14581] yes=47,no=48,missing=47
                    47:leaf=0.0175715
                    48:leaf=-0.00748876
                26:leaf=0.0174804
        6:[userkn_aopencardniurate_firstday<0.0458042] yes=13,no=14,missing=13
            13:[userkn_avgperday_opencardniu_cnt<7.44167] yes=27,no=28,missing=27
                27:leaf=0.00171541
                28:leaf=-0.0229204
            14:leaf=0.00968641

If I am right, the leaf value is the value of logodds and it can be changed into a probability with the sigmoid function. However in the last booster all the leaf values changed to around 0.5 probability.

Which means all the samples will be marked as good/bad cases half and half? So it's no difference with a random guess at a binary classification?

Am I right or any other opinions are quite appreciated!


texts: Find the relation between two logical models, and their inductive bias

texts:
Suppose we want to learn the Boolean function in instance space $X=\{0,1\}^3$. We are given two models to examine: $H_1$ is a set of all logical functions in the conjunctive normal form (CNF), $H_2$ is a set of logical functions which can be displayed in Horn clause (disjunction of literals with at most one positive). Explain what is the relation between $H_1$ and $H_2$, and which sort of inductive bias is implemented in those models.

I apologize in advance if I translated something poorly.
This is my idea. Since CNF holds all logical expressions, that would imply that it can also hold all logical functions, because every expression which is not in CNF can be converted to a CNF. Since CNF can hold all logical function wouldn't it imply that it can hold all sets of Boolean function, and given that $H_2$ is a special case, which is smaller than $H_1$ wouldn't it apply that all solutions can be found in $H_2$.

That's what I got so far, can someone confirm this or deny it, or how much off am I? Any tips would be appreciated!


texts: What is the best way to feature engineer features which have more than one repeated values?

texts:
What is the best way to feature engineer features which have more than one repeated values ?
I want to parse this data and finally keep in a pandas df for further analysis.
Example,
I have data of people's profile which consists of
Name, Age, Gender, Company, Degree

Now it is easy to keep Name , age and gender which has specific single value, but company can have more than one value or multiple value like someone worked with Google or Microsoft or both Google, Microsoft.
Same case with Degree , people can have single as well as multiple values together.
Right now I have kept them as comma separated values like if someone has more than one company then value is  Google, Microsoft.
While I encode them using say sklearn Label Encoder I get different codes like
Google = 1
Microsoft = 2
Google, Microsoft = 3
Which I guess is not very accurate, as when the data increases it will explode with number of combinations also, if I have to find similar features of those who worked at Google I might not get the correct answer as code 2 and code 3 will never match.
Is there a better way to handle such data ?


texts: Tensorflow Fine-tuning a model from my own checkpoint

texts: Assume that I am going to do more training with a similar data set in the future, is there any benefit to me using a fine tune checkpoint from a model that I created from my own training as opposed to the original SSD_Mobilenet_V1 version (for example 5000 images and 50000 steps).  Does it improve any future training or am I just better off using the original one every time I train?  I'm probably searching for the wrong thing, but I cannot find anything that suggests improvement.  In my reading, I thought that the fine tune only helped with the final layer which leads me to believe its a pointless exercise no matter how good the resulting model was.  Can anyone confirm this point and any useful reading for reference?


texts: Why does using a standard scalar on my tf idf matrix make it perform better?

texts:
I have a tf-idf matrix transformed on a list of tweets from a data set i am using. I have a pipeline where i initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as the classifier algorithm.
Pretty much as done here in the examples section. With the pipeline the classifier scores an 87 f1 score. Without the pipe it scores a dismal 53.
Why is this?
I thought tf-idf values were already two-fold normalised so shouldn't the standard scalar have no affect as it just performs normalisation again?


texts: I have 1,000 photoshoot-quality images of pianos on white background with very little noise (people in the background, etc). How would I go about finding the piano image that looks most like the others? Alternatively, how would I find five pianos that look most like the "average" piano. I'm particularly looking for python libraries that may be helpful (I'm currently looking at skimage).


texts: Am I using optim.SGD incorrectly in pytorch?

texts:
I am doing reinforcement learning in checkers.

After each game the network beats itself, I calculate the loss of every individual position in the game, call backward(), and step(). I am beginning to believe that this is not how SGD is supposed to be used, and that I should be feeding it batches, say a whole game at a time.

Strictly  in terms of code:

Do I do this by just wrapping the collective inputs in a one dimensional tensor?
If I give SGD a collection of inputs does it sample from the collection, calculate the average loss, and use that gradient?
Is SGD actually intended to be used in this way with large 'collections'/batches?


texts: Optimization of pandas row iteration and summation

texts:
i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result.

What i am trying to obtain is a summation of IDs in one table (player table) based on each row of a second table (UUID). Functionally each row needs to sum the total of the players table rows that are contained in its Active row and assign the UUID as the index to that row.

My initial thought was to loop row by row and calculate out my results but that has produced quite a slow result, and i suspect is not the optimal way that this could be accomplished. In the version below my estimate total time for the full dataset would be around 66 minutes. Running on a subsample of 10,000 takes around 20 seconds.

Would anyone have a better solution to calculating these results?

Thanks in advance!

UUID Table

This is a subset of the whole table

shape = (2060590, 2)

Player ID Table

This is a subset of the whole table

shape = (39,8)

Final Table

Code

# executes in ~20 seconds
df = None
for ix, i in enumerate(uuid_df[["UUID", "Active"]].sample(10000).itertuples(index=False)):
    # Get UUID for row
    _uuid = i[0]
    # Get list of "Active items" (these are the ones that will be summed)
    _active = i[1]

    # Create new frame by selecting everything from points table where the ID is in the Active List.
    # Sum selected values, convert to a dataframe with UUID as index and tranpose
    _dff = points_table_df.loc[points_table_df.index.isin(_active)].sum().to_frame(_uuid).T

    # Check if first dataframe, if not concat to existing one
    if df is None:
        df = _dff
    else:
        df = pd.concat([df, _dff])


texts:
I'm very new to computer coding and data science in general. I'm trying to open Weka and use it on a practice csv data set called weather.arff.  The manual I have says to type this into command line:

  java weka.classifiers.j48.J48 -t weather.arff

When I open the command line it goes to C:\User\Admin
When I type in the above on command line it gives me message

  Error: Could not find or load main class weka.classifiers.j48.J48

Might someone give me some advice on how to proceed to make the command line work.
Thank you


texts: What is used for Machine Translation besides RNN

texts:
I am doing a university report and it seems that encode-decode RNN are optimal for Machine Translation. I need something else to compare it to but I can't seem to make a proper google search for it.

Can you recommend what else can be used for Machine Translation that I can compare RNN to.


texts: why keras gives me desired results for my Entity Embedding but not pytorch?

texts:
I tried to build Entity Embeddings of categorical data from a dataset.
I took a dataset - "Bike share”.This dataset shows number of bike share/rent/sales in every month.The month column has $12$ categories.

I wanted to make 12 Entity embeddings or 12 vectors, one for each month.I trained the embeddings against the bike share/rent/sales or i should say i used bike share/rent/sales as my labels.To visualize the embeddings I plotted the learned Embeddings/vectors.
On a 3D graph and I expected that the months with similar sales will be close together on the 3D graph but I didn’t get the expected results.
In fact when I made the same model in Keras I got the results I was expecting. For example the batch size of $8$ works very well in keras but same batch size increases loss in pytorch. In pytorch big batch size($40$) reduces loss.

Would you be kind enough to have a look at my code in pytorch. It’s a very short code and will only take $10$ minutes.

Here's the link to my Pytorch code - https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity%20Embedding%20in%20Pytorch%202.ipynb

and keras code -
https://github.com/akshay6893/Entity_Embeddings/blob/master/Entity_Embeddings_keras.ipynb

Here’s the data online - https://www.kaggle.com/marklvl/bike-sharing-dataset.(very small dataset)


texts: Why we use transposed filter as the deconvolution operation instead of the pseudo inverse of filter?

texts:
I am trying to visualization CNN by the method in this paper "Visualizing and Understanding Convolutional Networks"

According to this tutorial A guide to convolution arithmetic for deep
learning (P.19)

We can always rewrite convolution as a matrix multiplication

For example, $Y$ is a feature map &  $X$ is input & $W$ is a filter & $\otimes$ is convolution operation

$$
\begin{bmatrix}
y_{11} & y_{12} \\\\
y_{21} & y_{22}
\end{bmatrix}
=
\begin{bmatrix}
x_{11} & x_{12} & x_{13} & x_{14}\\\\
x_{21} & x_{22} & x_{23} & x_{24}\\\\
x_{31} & x_{32} & x_{33} & x_{34}\\\\
x_{41} & x_{42} & x_{43} & x_{44}
\end{bmatrix}
\otimes
\begin{bmatrix}
w_{11} & w_{12} & w_{13} \\\\
w_{21} & w_{22} & w_{23} \\\\
w_{31} & w_{32} & w_{33}
\end{bmatrix}
$$

In the matrix multiplication form

$$
\begin{bmatrix}
y_{11} \\\\
y_{12} \\\\
y_{21} \\\\
y_{22}
\end{bmatrix}
=
C
\begin{bmatrix}
x_{11}\\\\
x_{12}\\\\
\vdots \\\\
x_{44}
\end{bmatrix}
$$

$$
C=
\left[\begin{smallmatrix}
w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0 & 0 & 0 & 0 & 0\\\\
0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0 & 0 & 0 & 0\\\\
0 & 0 & 0 & 0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0\\\\
0& 0 & 0 & 0 & 0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33}\\\\
\end{smallmatrix}
\right]
$$

So we can visualize filter by multiplying $C^{-1}$.

OK, my question is why we use $C^{T}$ instead of the pseudo inverse $(C^{T}C)^{-1}C^{T}$?


texts: Estimating time to travel between two lat/longs

texts:
I'm trying to create an offline estimator for how long it would take to get from one lat/long to another. Two approaches I have come across are the Haversine distance and the Manhattan distance. What I'm thinking of doing is calculating both of them and then using the average between the two as the distance and then use some average speed to calculate time.

Since this value will be used as an estimator for drivers in a city a straight line distance will grossly underestimate the actual distance drivers will have to travel.

Any thoughts would be greatly appreciated.


texts: How are Q-value and state value different in reinforcement learning

texts:
I have gone through this answer regarding the difference between Q-value and state value. My specific question is:

If Q-value calculates immediate reward after taking a particular action and then calculates the rest by following a policy, how will the expected reward change if value function also follows the same policy? For example, in the image below, how will Q-value and state value change depending on the policy?


texts: How to plot multiple models val and traing acc/loss curve from csv files?

texts:
I trained multiple CNN models, after that, I saved models details (Like , training/Validation Acc/Loss )  by callbacks by using this codes :
tf.keras.callbacks.CSVLogger

Now I have multiple models training-val acc-loss respective values in different csv files.
I want to plots those in one figure from my csv files. How can I do this?


texts: Output trained parameters of Keras model

texts: If I am training on a GRU model, is there a way I can output the learnt parameters so that when I train next time with more data, I can initialize with those learnt parameters as a starting point?


texts: Can we use GAEs with A3C asynchronous reinforcement learning?

texts: The Generalized Advantage Estimator seems very effective with algorithms like PPO in reinforcement learning. But most of the A3C algorithms I have seen use Averaged Advantage estimates. Is there any reason?


texts: Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

texts:
I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge).

I have started running the notebook that includes cells with PySpark/MLlib code for performing cross-validation and prediction using a Pipeline consisting of a VectorAssembler, MinMaxScaler, and GBTRegressor.

When I run this job it appears to be utilizing only CPU (Ganglia UI shows no GPU activity whatsoever, but plenty of CPU being used). Perhaps there is PyCpark code I need to add to my notebook and/or configuration settings for the cluster to allow for running this code with the help of my cluster's GPU?

I am new with Spark/MLlib, it's very possible that I am missing something obvious. Thanks in advance for any suggestions!


texts: What do these terms mean in regards to echo state networks?

texts:
I have been reading about echo state networks and reservoir computing, and have come across these following terms. What are the definitions?

End compatible
Output feedback
State contracting
State forgetting
Input forgetting
Standard compactness conditions
Non-spiky integrator neurons
Teacher output


texts: Feature Importance without Random Forest Feature Importances

texts:
Is their an intuitive way of finding feature importances without just using the random forest feature importances method?
I have a binary logistic regression problem where I have binary features (1 or 0) and a binary target (1 or 0).
I want to see which features are most important towards predicting the target and somehow rank them.
I did an odds ratio for each feature, which gave me some idea of importance.
Are there any other methods?


texts: Apply GroupByKey to identify the Frequently Products Purchase Together

texts:
I've a data set with this fields:

Transaction_ID
Customer_ID
Department
Product_ID

And I'm trying to obtain a tuple with the products associated with each customer transaction. Like:

Transaction_1 -> Product_ID 1, Product_ID 2, Product_ID 3
Transaction_2 -> Product_ID 1, Product_ID 2, Product_ID 4
....

I've this code: But It not return the dataset as I want:

case class transactions (Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)

def csvToMyClass(line: String) = {
    val split = line.split(',')
    transactions(split(0),split(1),split(2),split(3))
}

val  csv = sc.textFile("FILE").map(csvToMyClass)
csv.take(10)
csv.saveAsTextFile("PATH/output.csv")

How can I obtain the list of products associated group by Transaction_ID??


texts: How L2 Regularization penalizes weights in TensorFlow?

texts:
I know that L2 Regularization technique is used to reduce over-fitting and penalizing large weights. In more than one place, I saw that it is used like the code below in TensorFlow library:
reg = tf.nn.l2_loss(w_conv1) + tf.nn.l2_loss(w_conv2) + \
      tf.nn.l2_loss(w_conv3) + tf.nn.l2_loss(w_conv4) + \
      tf.nn.l2_loss(w_conv5) + tf.nn.l2_loss(w_fc1) + \
      tf.nn.l2_loss(w_out)

    loss = tf.reduce_mean(loss + reg * beta)

I am confused about how this code penalizes weights? It adds L2 to weights and then the result to the loss. Could anyone explain this for me?


texts: What approach would you use to determine the effects of an event when you don't know the effects?

texts:
Let's say you have a problem where in time 0 you have an event and then you want to figure out what effects are caused by the event in time 1.  Unfortunately, you do not know what the effects are, but you know the event has occurred.
Could this be a typical ML problem but done backwards where data from time = 1 are your features and data from time = 0 is your response?  As it is a problem that involves determining what the effects are, a way to extract variable importance would be needed, so not sure if I can use NN.
What would be some approaches to solve this problem?
Thanks!


texts: Remove Local Outliers from Dataframe using pandas

texts:
Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe. Any advice would be highly appreciated.

The code to detect the local outliers is below:

def printOutliers(series, window, scale= 1.96, print_outliers=False):

rolling_mean = series.rolling(window=window).mean()

#Print indices of outliers
if print_outliers:
    mae = mean_absolute_error(series[window:], rolling_mean[window:])#mean absolute error is a measure of difference between two continuous variables.
    deviation = 3*np.std(series[window:] - rolling_mean[window:])
    lower_bound = rolling_mean - (mae + scale * deviation)
    upper_bound = rolling_mean + (mae + scale * deviation)
    outliers_lower = series[series<lower_bound]
    outliers_upper = series[series>upper_bound]
    print("values beyond lower bound are: " +  "\n"  + str(outliers_lower))
    print("values beyond lower bound are: " + "\n" + str(outliers_upper))

printOutliers(df['Column1'].dropna(how='any'), 10, print_outliers=True)


texts: I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is it, what is the main difference from the other two (or from the Unsupervised type, specifically), with some good example? Is it a promising alternative worth to explore or just some hyped niche curiosity?


texts: What is difference between feed forward neural network and LSTM?

texts: What is the difference between feed-forward neural network and LSTM? How do they differ in their architecture?


texts:
In ubuntu I can find

~/.keras/keras.json

and change the backend
but in windows, I can not find this file. How can I  change keras backend in windows.


texts: Handling data imbalance and class number for classification

texts:
1 Is there a way to handle data imbalance? ie if data in each class for training is not balanced, say some classes have 50 documents some other have 200 documents. How to handle this?

2 How to handle the classification problem with a large number of classes? I have around 50+ classes (may increase once more data is available) to learn. I am trying out different algorithms and features. Is there any way to handle classification problem with a large number of classes?


texts: Yarn service parameter for pseudodistributed

texts:
I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:

<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>

What is the "mapreduce_shuffle" option?


texts: How to calculate a specific time complexity of inverse calculation of matrix?

texts:
I am a green-hand in calculating the time complexity. Given a calculation as follows:

\begin{equation}
\mathbf{x}=\mathbf{A^T}(\mathbf{AA^T}+\lambda\mathbf{I}_n)^{-1}\mathbf{b}
\end{equation}
where $\mathbf{A}\in R^{n\times p}$, $\mathbf{I}_n\in R^{n\times n}$, $\mathbf{b}\in R^{n}$ and $\lambda\in R$.

Could some one kindly give me the complexity of the above formula in details? Thanks a lot.


texts: Encoding for k-level qualitatative variable

texts: I have a qualitative variable, e.g. userId, which could take around 30,000 different coded values ($k$). I would like to represent this variable as a dummy variable. Coding this into a vector of size $k$ doesn't seem to be a good approach. Is there a more compact method for coding for this variable?


texts: Series Through a GPU's Window To, For Each Item, Output a Prediction and Retrain?

texts:
Perhaps I'm missing something obvious but I've not run across a Keras or PyTorch example of online training and series prediction loop implemented on a GPU with these (seemingly obvious) characteristics:
Allocate an N+1 FIFO in the GPU.
Load an N-input model in the GPU.
Fill the model's FIFO window with the first N+1 items of the L-length series (located in main memory).
LOOP

Retrain the model to predict the newest item with the N oldest items in
the FIFO.
Introduce the next item to the FIFO.
Output a prediction using the oldest N items in the FIFO as input.

END LOOP
This should produce L-N-1 predictions in an output structure in main memory.


texts: Does small batch size improve the model?

texts:
I'm training an LSTM with Keras.

I've noticed that the smaller the batch size, the more the loss decreases during periods: so this makes me think that the network can process fewer items better at a time.

Is it a normal behavior in general?


texts: How to check the randomness or the distribution of an attribute in a dataset?

texts:
I have a dataset. I want to make its randomness nature match with another dataset (2nd one). Basically I want to modify the 2nd dataset values and make them similiar in distribution or nature in accordance with the 1st dataset.
I'm working on Spyder using Python 3.
I want to do it for some experimental purposes


texts:
I've seen a couple questions that have a similar problem, but none of them solved mine. I'm trying to fit a neural network in Keras to a dataset with 22 input features for binary classification. The problem is that I only have 195 training samples. I know it's a small dataset, but I don't know if it's possible to fit a model with reasonable accuracy (I'm aiming for >95% accuracy). The problem I'm having is that my model is only outputting 1 and getting 75% accuracy because my dataset is 75% positive cases. Here's the code I have:

data = pd.read_csv("") #filename omitted, but it loads properly
scaler = MinMaxScaler()
X = scaler.fit_transform(X)
Y = data['status']
X = data.drop(['status', 'name'], axis = 1)
xTrain, xTest, yTrain, yTest = train_test_split(X, Y, train_size = 0.8)
model = Sequential()
model.add(Dense(48, input_shape=(22,), activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'softmax'))
optim = keras.optimizers.adam(lr=0.0001)
model.compile(optimizer = optim, loss = 'binary_crossentropy', metrics = ['accuracy'])
model.fit(xTrain, yTrain, epochs = 20, batch_size = 5, validation_data = (xTest, yTest))

I've tried adding more hidden layers, increasing the number of training epochs, and increased and lowered the optimizer's learning rate, but the accuracy stays the same. Here's the link to the dataset: https://www.dropbox.com/s/c4td650b4z7aizc/fixed.xlsx?dl=0

EDIT: I fixed my problem by applying SMOTE to balance my dataset. The accuracy went up to 96% after I did this and added more hidden layers.


texts: Implementation of NLP to categorize text into two categories

texts:
I can't discuss my actual dataset, so please bear with me.

Let's say I have a dataset that contains a population of 20,000 examinations by a school principal. The principal is to record their examinations of student misconduct incidents. I want to implement NLP that assess the quality very broadly into two categories: "good examination" or "bad examination" of the full population.

An example of "bad examinations" are:"examination results - negative" or "exam results: negative". Or "check student's bags, checked the person. Nothing suspicious found. Or examination results negative". Or "Examination results positive". Or "ABC examined, results negative". ABC could be an abbreviation of the person's name.

A good examination would be where there is a lot of context: "Checked the student's bag and found textbooks, pencils, erasers, binders. No hidden compartments found. Interviewed the student and asked "x", "y", "z" questions. Her story corroborated other reports. Student presented herself in a clam manner. Examination results negative". Other times it could be paragraphs and paragraphs, and at the end "examination negative" or "examination positive"

There are also instances where all what could be listed is "wrong person because of different birth date. Examination results negative" and this is perfectly fine. Would this be a third category?

How would I go about implementing a reliable NLP solution? My first instinct is to take a random sample, classify it manually, and then apply it to the rest of the 20,000 records?


texts: Flattening of a structure field in pyspark is slow?

texts:
I am rather new to (py)spark. I am using a python UDF to compute several outputs. My idea was
to pack all the values in a StructType and then flatten the structure before collecting the result.
The flattening operation is surprisingly slow to me. Here is an example:
import pyspark
import pyspark.sql.functions as F
from pyspark.sql.types import StructType, StructField,FloatType, IntegerType
import numpy as np
import time

num_rows = 10000
num_columns = 100  # number of fields of the UDF output structure

# Define python function
def random_columns():
    """ A function creating random values """
    return np.random.randn(num_columns).tolist()

# Define udf schema
schema_data_list = []
for i in range(num_columns):
    struct_field = StructField("element_" + str(i), FloatType(), False)
    schema_data_list.append(struct_field)
schema = StructType(schema_data_list)

# Define udf function
random_columns_udf = F.udf(random_columns, schema)

if __name__ == '__main__':

    # setup session
    spark = pyspark.sql.SparkSession.builder. \
        appName("application_numpy_standalone"). \
        getOrCreate()

    # create a dataframe
    rdd = spark.sparkContext.parallelize(np.arange(num_rows).tolist())
    df = spark.createDataFrame(rdd, IntegerType())

    # compute something with an UDF
    df_struct = df.withColumn("random_data", random_columns_udf())

    # flatten the results
    df_flat = df_struct.select("value", "random_data.*")

    # time to collect the structured dataframe
    time_start = time.time()
    df_struct_pd = df_struct.toPandas()
    time_complex = time.time() - time_start
    print(time_complex)

    # time to collect the flattened dataframe
    time_start = time.time()
    df_flat_pd = df_flat.toPandas()
    time_flat = time.time() - time_start
    print(time_flat)

On my PC obtaining the toPandas() action on the structured df takes 1.6 seconds, while on the flattened one it takes 21 seconds. I observed a similar behavior on a real cluster of 6 nodes, both with spark 2.4.5 and 3.0. Any idea?
Thanks,
Marco


texts: Knowing when a GAN is overfitting (sequence classification study)

texts:
I have sequences of long, sparse 1_D vectors (3000 digits, made of of 0s and 1s) that I am trying to classify. I have previously implemented a simple CNN to classify them with relative success (with keras).

data:
label, sequence
0,0000000000010000000......
....
1,00000000000000000001........

I am trying to create a GAN that returns a sequence that is emblematic of the sequences that have a '1' label.

I have attempted to re-implement typical analysis of 2D tensors (i.e. images) on my 1-D data.

Thus far I have managed to get functioning code that creates 'fake' vectors that are similar looking to the original ones.... however my problem is that I don't know how to assess what is a sufficient number of epochs... I was advised that assessing the performance on test data might be a good indicator of this. Hence as a separate task to the normal procedure of the GAN training I also assess the generator's performance on test data in each epoch.

See above in the upper-left panel I have plotted the loss profiles of the generator and discriminator, in the upper right I have plotted the accuracy achieved at each epoch when validating on the test data and the bottom three figures are exemplary sequences output at different stages of the training (indicated by arrows).

The code that I use is pretty similar to that at the following link: https://github.com/osh/KerasGAN/blob/master/MNIST_CNN_GAN.ipynb

(If the actual code is needed please comment below and I will include it).

See below another exemplary output (from a different task):

Here the generator seems to be overfitting and generating all 0 sequences- which is undesirable. As an attempt to mitigate this I trained the discriminator to reject all 0 sequences before freezing its' weights and initiating the GAN:

discriminator=load_model('..discriminator..')
X_filler,y_filler=np.zeros((100,3000, 1)),np.zeros(100)
discriminator.fit(X_filler,y_filler,epochs=3,batch_size=100)
discriminator.trainable = False
......initiate GAN training....

Still after doing this the above shown problem persists..

I cannot find a good recommendation as to when to stop training.. The following source recommends:

  Stop criteria: the number of Generator’s failures (failed attempts to
  fool the Discriminator) is [almost] equal to the errors of
  Discriminator to distinguish artificially generated samples from real
  samples.

https://www.quora.com/What-is-the-stop-criteria-of-generative-adversarial-nets-Can-I-treat-it-as-a-multi-objective-problem

But I am not sure how to assess when this is happening based on the test accuracy graph displayed above.

Overall my questions are:

1. When can I know based on test validation, done in each epoch, when to stop training the GAN?

2. How do I know if my generator is overfitting to the and what can I do to mitigate this?


texts: Python inserts newline by writing to csv

texts:
I am trying to scrape http://www.the-numbers.com/movie/budgets/all

but when I write the table into a csv file, there is an additional line with the counter index written in between each movie row... how can I get rid of this? I dont understand how that counter line is being written to the csv...

import csv,os
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen, URLError
from selenium import webdriver

counter = 0
currentDir=os.getcwd()
filename = currentDir + "\\theNumbersScraper.csv"

pagecount = 1
headers=['ID',
 'Release Date',
 'Movie',
 'Production Budget',
 'Domestic Gross',
 'Worldwide Gross']

with open(filename, 'w' ,newline='\n',encoding='utf-8') as csvfile:
        #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers
        #writer.writeheader()
        #csvfile = open(filename, 'w', newline='',encoding='utf-8')
    writer = csv.writer(csvfile,delimiter='|')
    writer.writerow(headers)

#with open(filename, 'a', newline='',encoding='utf-8') as csvfile:
    #writer = csv.DictWriter(csvfile, fieldnames=dictionary)#write headers
    #writer.writeheader()
    #csvfile = open(filename, 'w', newline='',encoding='utf-8')
#    writer = csv.writer(csvfile,delimiter='|')

    while pagecount <5401:

        """
        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1
        #to http://www.the-numbers.com/movie/budgets/all/5401
        #so there are 5400 entries
        """
        request = Request("http://www.the-numbers.com/movie/budgets/all/"+str(pagecount))
        request.add_header('User-agent', 'wswp')
        website = urlopen(request).read().strip()

        soup = BeautifulSoup(website,'lxml')

        """#obsolete
        headertags = soup.find("table").find_next("tr").find_all("th")

        headers= []
        for line in headertags:
            headers.append(line.string)

        headers[0] = 'ID'

        """

        #movie-entries go from http://www.the-numbers.com/movie/budgets/all/1
        #to http://www.the-numbers.com/movie/budgets/all/5401
        #so there are 5400 entries

        all_tr = soup.find_all("tr")
        for movie in range(1, len(all_tr)):
            row=[]
            counter+=1
            row.append(counter)
            td = all_tr[movie].find_all("td")
            for colIndex in range(1, len(td)):
                row.append(td[colIndex].string)
            writer.writerow(row)

        """
        for tr in all_tr:
            row = []
            td = tr.find_all("td")
            for i in range(1, 4):
                row.append(str(counter))
                row.append(td[i].string)
            writer.writerow(row)
            counter+=1
        """
        pagecount +=100

csvfile.close()

The part of interest is actually this block only:

all_tr = soup.find_all("tr")
for movie in range(1, len(all_tr)):
    row=[]
    counter+=1
    row.append(counter)
    td = all_tr[movie].find_all("td")
    for colIndex in range(1, len(td)):
        row.append(td[colIndex].string)
    writer.writerow(row)

The output is:

ID|Release Date|Movie|Production Budget|Domestic Gross|Worldwide Gross
1|12/18/2009|Avatar|$425,000,000|$760,507,625|$2,783,918,982
2
3|5/20/2011|Pirates of the Caribbean: On Stranger      Tides|$410,600,000|$241,063,875|$1,045,663,875
4
5|5/1/2015|Avengers: Age of Ultron|$330,600,000|$459,005,868|$1,408,218,722
6

and so on, with additional counter lines between the output that I don't want, how can I get rid of that? (and where does it come from?)


texts: Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

texts:
A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to.

What I would like to do is take a scikit-learn's SGDClassifier and have it score the same as a Logistic Regression here. However, I must be missing some machine learning enhancements, since my scores are not equivalent.

This is my current code. What am I missing on the SGDClassifier which would have it produce the same results as a Logistic Regression?

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier
import numpy as np
import pandas as pd
from sklearn.cross_validation import KFold
from sklearn.metrics import accuracy_score

# Note that the iris dataset is available in sklearn by default.
# This data is also conveniently preprocessed.
iris = datasets.load_iris()
X = iris["data"]
Y = iris["target"]

numFolds = 10
kf = KFold(len(X), numFolds, shuffle=True)

# These are "Class objects". For each Class, find the AUC through
# 10 fold cross validation.
Models = [LogisticRegression, SGDClassifier]
params = [{}, {"loss": "log", "penalty": "l2"}]
for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:

        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]

        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)
    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)

My output:

Accuracy score of LogisticRegression: 0.946666666667
Accuracy score of SGDClassifier: 0.76


texts: Format for X_train in keras using theano

texts:
I want to try out Keras (Theano backend) for regressions after already using sklearn.

For this I uses this nice tutorial http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/
 and tried to replace the training data there with my own.

import numpy
import pandas
import pickle
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

[X, Y] = pickle.load(open("training_data_1_week_imp_lt_15.pkl", "rb"))

X_train, X_test, y_train, y_test = train_test_split( X, Y, test_size=0.5, random_state=42)
scaler = StandardScaler()
scaler.fit(X_train)  # Don't cheat - fit only on training data
X_train = scaler.transform(X_train)

X_test = scaler.transform(X_test)

print (X_train.shape)

# define base mode
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(8, input_dim=8, init='normal', activation='relu'))
    model.add(Dense(1, init='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# evaluate model with standardized dataset
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=0)

estimator.fit(numpy.array(X_train),y_train)

However, i get the following error:

Exception: Error when checking model target: the list of Numpy arrays that you are
passing to your model is not the size the model expected. Expected to see 1
arrays but instead got the following list of 6252 arrays: ...

The format of X is the usual sklearn format: print (X_train.shape) = (6252, 8)

How do I format my input X correctly.

What I tried was transposing but this did not work.

I also already searched the web but could not find a solution/explanation.

Thanks!

EDIT: here is a small sample file https://ufile.io/8a428

[X, Y] = pickle.load(open("test.pkl", "rb"))


texts:
I am working through a tutorial on neural style transfer with tensorflow and am trying to understand the basic canonical imports. First, we are presented with:

import tensorflow as tf
import tensorflow.contrib.eager as tfe

from tensorflow.python.keras.preprocessing import image as kp_image
from tensorflow.python.keras import models
from tensorflow.python.keras import losses
from tensorflow.python.keras import layers
from tensorflow.python.keras import backend as K

My confusion here is why exactly we need to call python and keras inside of the tensorflow imports. My thoughts are that since python and keras are both dependencies for tensorflow this may be the reason. However isn't python a dependency for all packages?

Why can we not just call:

from tensorflow.keras.preprocessing import images as kp_image

Any help as well as an intuitive explanation would be greatly appreciated.

Thanks.


texts: Perceptron learning rate irrelevant in convergence

texts:
Via this MIT document i studied the single layer Perceptron convergence proof (= maximum number of steps).

In the convergence proof inside this document , the learning rate is implicitly defined as 1 .
After studying it, by myself i tried to re-do the proof inserting this time a generic learning rate $\eta $.

The result is that the result remains the same:

$k\leq \frac{R^{2}\left \| \theta ^{*} \right \|^{2}}{\gamma ^{2}}$

that is the learning rate $\eta$ cancelled out in the proof.

Is it possible , or i make mistakes in my proof ?


texts: better confussion matrix higher LogLoss ? Is that possible>

texts:
I have tried a 2 different versions of a gbm in a multinomial classification problem.  The second model results in better confusion matrix but in worse Log Loss value (at the test sample). How is that possible.

Further are the results of the two models.

I thought that it could be because the Class A is much more oversampled and the small decrease of that class could lead to such a deterioration of hte logloss?

Any ideas?
Thank you


texts: Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How would I calculate the cross entropy loss for this example?


texts: Does high error rate in regression imply the data set is unpredictable?

texts:
I have a data set of video watching records in a 3G network. In this data set, 2 different kind of features are included:

user-side information, e.g., age, gender, data plan and etc;
Video watching records of these users, each of  which associated with a download ratio and some detailed network condition metrics, say, download speed, RTT, and something similar.

Under the scenario of internet streaming, a video is divided into several chunks and downloaded to end device one by one, so we have download ratio = download bytes / file size in bytes

Now, Given this data set, I want to predict the download ratio of each video.

Since it is a regression problem, so I use gradient boosting regression tree as model and run 10-fold cross validation.

However, I have tried different model parameter configurations and even different models (linear regression, decision regress tree), the best root-mean-square error I can get is 0.3790, which is quite high, because if I don't use any complex models and just use the mean value of known labels as prediction values, then I can still have an RMSE of 0.3890. There is not obvious difference.

For this problem, I have some questions:

Does this high error rate imply that the label in data set is unpredictable?
Apart from the feature problem, is there any other possibilities? If yes, how can I validate them?


texts: Estimating test AUC using k-fold CV for imbalanced classification problem

texts:
I have an imbalanced classification problem.

I first partitioned my data into a training set (Dataset A) and a test set (Dataset B).

I then used the R package ROSE to simultaneously undersample the majority class in Dataset A and oversample the minority class. This produced a balanced set (Dataset C) which has as many rows as Dataset A.

I have trained my model on Dataset C and computed training AUC. I now want to estimate (i.e., not using Dataset B) the test AUC using k-fold cross-validation. How should I best do this?


texts:
I've just started reading about AB testing, as it pertains to optimizing website design.  I find it interesting that most of the methods assume that changes to the layout and appearance are independent of each other.  I understand that the most common method of optimization is the 'multi-armed bandit' procedure.  While I grasp the concept of it, it seems to ignore the fact that changes (changes to the website in this case) are not independent to each other.

For example, if company is testing the placement and color of the logo on the website, they find the optimal color first then the optimal placement.  Not that I'm some expert on human psychology, but shouldn't these be related? Can the multi-armed bandit method be efficiently used in this case or more complicated cases?

My first instinct is to say no.  On that note, why haven't people used heuristic algorithms to optimize over complicated AB testing sample spaces?  For an example, I thought someone might have used a genetic algorithm to optimize a website layout, but I can find no examples of something like this out there. This leads me to believe that I'm missing something important in my understanding of AB testing as it applies to website optimization.

Why isn't heuristic optimization used on more complicated websites?


texts:
I am looking for papers which could give a greater understanding on differences between Tree based and linear based models.

Anybody can gove me some pointers?


texts: How can I access to loss value in Keras LSTM implementation?

texts: I use Keras library and it's LSTM model. When I train my network I can see loss value in my program execution console. I like to know how can I access to this value in my code?


texts: Is k-means with Mahalanobis a valid option for clustering?

texts:
I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters.
The steps are:

Create aggregate datasets (initially randomly or other way, doesn't matter)
Estimate mu, sigma for each aggregate/cluster dataset
Recompute clusters by calculating Mahalanobis distance of each point to each cluster and updating the clusters.
go to 2 until clusters do not change.

I have seen this implementation used, also have seen it in dissertations. Yet something don't feel comfortable about it.
There is no way to have a 'absolute' best clustering (i.e. silhouette metric) with Mahalanobis.
You can only estimate the error (e.g. Bayesian Information Criterion) of your model (different normal distributions) on your data. And you can definitely over-fit.
Is this still 'correct' in terms of clustering and methodologically accepted?
Is this still termed 'k-means' clustering?

I guess must be valid since I haven't seen any argumentation against it. Yet I feel obliged to ask, just to make sure, before I use it.

Thanks a bunch.


texts:
For simplicity, I assume that the following timeseries $X_{A}$ and $Y_{A}$ are univariate.
I am familiar with training RNN networks, such as LSTM, on a given timeseries $X_{A} = (X_{A0}, X_{A1}, X_{A2}, ...X_{AN})$ and targets $Y_{A} = (Y_{A0}, Y_{A1}, Y_{A2}, ...Y_{AN})$, such that $Y_{AN} \approx \hat{f}(X_{AN} | X_{A(N-1)}, ... X_{A1})$. Especially, in this direction forecasting problems can be modeled as $Y_{Ai}=X_{A(i+h)}$.
However, I care about a  different problem. If my training set has multiple (univariate) time series $X_{A}$, $X_{B}$, ..., $X_{K}$ along with their target univariate series $Y_{A}$, $Y_{B}$, ..., $Y_{K}$, is it possible to structure this as a single deep learning architecture?
There are two necessities for prediction application:
(1) As new data arrive for a timeseries that actually was in the training set (like $X_A$ in the example), for instance the new values $X_{A(N+1)},X_{A(N+2)}$, to predict  $Y_{A(N+1)},Y_{A(N+2)}$
(2) To also be able to predict $Y_L$ values for a new timeseries $X_{L}$ that was not in the original training set  $\{(X_A,Y_A), .., (X_K,Y_K)\}$
In summary, to be able continue predicting for known timeseries, but also generalize for new timeseries. I am interested in any references towards problem formulation or solution that is similar to what I have described.


texts: How to handle negative integer values in a data set?

texts:
Suppose we have a data set of fields with negative integer values. So can we consider that fields with negative values for further proceedings or do we need to ignore that fields?
If we can consider those negative values then please tell me how will we process it?

The dataset is bank_data


texts: pandas: reshaping the dataset, moving column name as rows

texts:
I currently have the following table:
user_id --------- HR(segment_name) ------ observations.

123                1                     0.9
234 .              0 .                   0.78
567                0                     0.99
789                1                     0.89

Now I would like to convert this table to look like the table below:
user_id ---------- segment --------- HR(segment_values) -------- observations
123                HR                   1                         0.9
234                HR                   0                         0.78
567                HR                   0                         0.99
789                HR                   1                         0.89

I probably need to use the pivote table to convert the first table to the second table. Any ideas how to go about this? Thank you :)


texts: How do I train a model with pre-saved weights?

texts:
I need to train a model but the data after preprocessing is too large to fit in the RAM. Is it possible to preprocess the data in batches and train a model for the small data set and then save the weights and then train it again for the next batches with the saved weights?

the workflow would be this way:

preprocess -> train -> save weights -> preprocess next batch -> train with saved weights ...

If yes, how can I do it?


texts: How to JUST represent words as embeddings by pretrained BERT?

texts:
I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).
I know that almost any task can be solved "inside" BERT, i.e. using fine-tuning in final layer.
Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. "fashion", "nature"). c) Make an unsupervised classification task?


texts:
I'm trying to rank some percentages. I have numerators and denominators for each ratio. To give a concrete example, consider ratio as total graduates / total students in a school.

But the issue is that total students vary over a long range (1000-20000). Smaller schools seem to have higher percentage of students graduating, but I want to standardize it, and not let the size of the school affect the ranking. Is there a way to do it?


texts: Books on time series and sequence classification

texts: Though I have been using traditional machine learning algorithms (Regression and Classification) , I have no experience of using Time series and would like to understand what is time series and different approaches(ex:ARIMA,SARIMA,SARIMAX, LSTM etc) used for time series analysis. I also see that people use LSTM's/RNN for time series/sequence classification. Can you people recommend me a book that discusses the above algorithms and how Time series is different from sequence classification etc?


texts: Relatively high regularization parameters for XGBoost model only way to prevent overfitting

texts:
I am modeling a continuous regression/forecasting problem for very right-skewed data. I've been using ElasticNet and Huber regression with quite a bit of success, and have recently moved into using XGBoost to see if it'll provide any additional value. The dimensions of my training matrix is 60,000 rows by 500 columns.
What I've found is that the much simpler, more interpretable ElasticNet/Huber regression models very often outperform any XGBoost model I've built. The only way I can get XGBoost to compete is by using a ton of different forms of regularization. In particular: the most performant XGBoost models have had reg_alpha/reg_lambda parameters in the [10-150] range; gamma in the [25, 100] range, subsample of 0.5, colsample_by_tree of 0.5, and shallow max_depths, e.g. 3/4/5, with around 150 n_estimators.
From what I've gathered in various tutorials online, gamma values over 10 or 20 seem to be very high, although I completely acknowledge that statement could be very dependent on the characteristics of the dataset being used.
For this super-regularized model, the predictions and feature importances make sense from an intuitive perspective.
I guess I'm just looking for some input – is it insane that I have such high regularization parameters, or am I more justified than once thought in these high values, since the proof seems to be in the pudding with the model's predictive power/generalizability and important features?


texts: I have this data set for crimes of a 12 month time period, over 250k rows. I want to predict future crimes by date and location

texts:
I have this 250k data set with these features

    date_time       FullAddress             call_type priority   lat       long
0   6/14/17 21:54   10 14TH ST\, San Diego\, CA 1151    2.0 32.705449   -117.151870
1   3/29/17 22:24   10 14TH ST\, San Diego\, CA 1016    2.0 32.705449   -117.151870
2   6/3/17 18:04    10 14TH ST\, San Diego\, CA 1016    2.0 32.705449   -117.151870
3   3/17/17 10:57   10 14TH ST\, San Diego\, CA 1151    2.0 32.705449   -117.151870
4   3/3/17 23:45    10 15TH ST\, San Diego\, CA 911P    2.0 32.705722   -117.15035

Date and time , full address , lat and long , and call type , and level of the seriousness of the crime.
I want to predict the time when Future crimes will happen or predict the location it will happen again. How can I make that happen, will I use regression or classification? I already predicted the priority, but how can I predict the time it will happen or the location?

I predicted the priority but doesn't really give me anything. I want to predict time and location or either or.

this is some code i have for my priority prediction

from sklearn.ensemble import RandomForestClassifier
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
my_RandomForest = RandomForestClassifier(n_estimators=100, random_state=0)

my_RandomForest.fit(X_train, y_train)
y_predict_fr = my_RandomForest.predict(X_test)
from sklearn.metrics import accuracy_score
print(y_predict_fr)
accuracy_fr = accuracy_score(y_test, y_predict_fr)
print(accuracy_fr)

[4. 3. 2. ... 3. 1. 2.]
0.95100761598545


texts: Using an autoencoder to mimic independent component analysis?

texts:
I'm trying to use autoencoders in keras to create a linear transformation similar to independent component analysis (ICA) (using this to denoise electroencephalographic data, time series of 64x100000 pts). While the autoencoder does a good job of re-creating the input using a smaller number of neurons in the hidden layers, there's no structure to the weights in the hidden layers, i.e., it doesn't seem to isolate structure in the data, it just mixes everything up in the compressed layers. For example, ICA is able to isolate things like heart-beats, eye-blinks, and brain activity, but the reconstructed time-series in the hidden layers are just linear mixtures of all of these different signals.

Does anyone know how to force the time series constructed in the hidden layers to be temporally independent? I'm thinking I'll have to create some kind of custom regularizer, but I have no idea how to go about this.

Here's my keras code:

import numpy as np
import scipy as scipy
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.layers import BatchNormalization
import matplotlib.pyplot as plt
from keras import regularizers
from keras.constraints import max_norm, non_neg, unit_norm
from keras import backend as K

raw = scipy.io.loadmat('raw.mat')
raw = raw['raw'];

#acts = scipy.io.loadmat('acts.mat')
#acts = acts['acts'];

model = Sequential()
model.add(Dense(32,input_shape=(64,)))
model.add(Activation('linear'))
model.add(Dense(16))
model.add(Activation('linear'))
model.add(Dense(64))
model.add(Activation('linear'))

model.summary()

model.compile(optimizer='Adamax',loss='mean_absolute_error')
xtrain = raw[:,:].transpose()
#xref = acts[:,:].transpose()
model.fit(xtrain, xtrain, verbose=1, batch_size=200, epochs=20)