texts: High level understanding of residual blocks

texts:
I've been wondering what the residual blocks actually do and why they are a necessity for deep convolutional networks. Specifically, why are they so useful in some generator networks? To better extract features? I want an intuition for this concept, but can't find an understandable explanation online.

And what does the summation at the end of the block mean? It shows skipping around the block but in the end it adds it separately anyway? What does x and F(x) really represent in this context? Is F(x) an image feature of a given input and x is the input?

To me it looks like it's for feature detection/recognition because of the way it individually adds that separately calculated sub-results, but I could be wrong. And it doesn't entirely make sense to me this way either, because how does this particular method help to achieve that end? Is it a principle of reaffirmation - the net separately calculates some of the features and through these means knows for sure that a certain texture is what it is instead of something else (less mistakes everywhere)?
But then - why not have this for the discriminator as well? It fundamentally needs this same tech, doesn't it?

Is it just that the generator needs to improve more than the discriminator does? In other words - if the discriminator had these blocks it would start classifying pretty much only the exact replicas of the input image and the generator would never get fool the discriminator? But then - what importance would that be anyway - the generator would get just as good at image generation anyway.


texts:
I read a lot about Orange here in the forum. I know that it is a software developed at the University of Ljubljana. I guess it has some kind of GUI which makes it a "cheap and easy" way to implement ML models.
My question: What is special about Orange? What are the advantages of the software (especially when compared to R or Python)? Is it merely some kind of GUI or does it come with functionality that goes beyond what R or Python can do?
I work with both R and Python and I wonder if it is worth it to have a look at Orange.


texts: Tensorflow RNN not learning when output included in training variables

texts:
I have been attempting to train a RNN on a set of time series data. The goal is to predict one of six categorical outputs. The input is given as 5 time steps of 14 inputs, six of which at one-hot attributes for the output.
There is an output at each time step, but the goal is to use previous recorded time spots and their human-assigned outputs to assign an output to the most recent event.

Confusingly the RNN is unable to learn that one of the inputs is in fact the output of the classification. This is simply a sanity check for me, but it seems that it may indicate a larger underlying problem.

The data is heavily imbalanced, 91%, 4%, 2%, 1%,<1%,<1%, but a cost function is being use to weight mis-classification inversely to it's make-up in the data set. Could the imbalance cause this issue? I'm using 60,000 training examples right now, is this not enough?

I'm working off of this dynamic RNN model: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py.
Note the additional matrix multiplication for a hidden layer. Is this done correctly?

def dynamicRNN(x, seqlen, weights, biases):

# Prepare data shape to match `rnn` function requirements
# Current data input shape: (batch_size, n_steps, n_input)
# Required shape: 'n_steps' tensors list of shape (batch_size, n_input)

# Permuting batch_size and n_steps
x = tf.transpose(x, [1, 0, 2])
# Reshaping to (n_steps*batch_size, n_input)
x = tf.reshape(x, [-1,n_input])
x  = tf.matmul(x, weights['hidden'])+ biases['hidden']
# Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
x = tf.split(0, n_steps, x)

# Define a lstm cell with tensorflow
lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)

# Get lstm cell output, providing 'sequence_length' will perform dynamic
# calculation.
outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32,
                            sequence_length=seqlen)

# When performing dynamic calculation, we must retrieve the last
# dynamically computed output, i.e, if a sequence length is 10, we need
# to retrieve the 10th output.
# However TensorFlow doesn't support advanced indexing yet, so we build
# a custom op that for each sample in batch size, get its length and
# get the corresponding relevant output.

# 'outputs' is a list of output at every timestep, we pack them in a Tensor
# and change back dimension to [batch_size, n_step, n_input]
outputs = tf.pack(outputs)
outputs = tf.transpose(outputs, [1, 0, 2])

# Hack to build the indexing and retrieve the right output.
batch_size = tf.shape(outputs)[0]
# Start indices for each sample
index = tf.range(0, batch_size) * n_steps + (seqlen - 1)
# Indexing
outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)

# Linear activation, using outputs computed above
return tf.matmul(outputs, weights['out']) + biases['out']


texts: Different result between Rapidminer and Python imblearn

texts: I'm currently working on imbalanced classification problem. However i found different result between SMOTE in rapidminer and SMOTE in imblearn (python). rapidminer SMOTE give 15-20% improvement on cross validation, but imblearn only give 5%. is it possible? i'm using random forest as basic learner with default config for both


texts:
I am a newbie to machine learning. I am figuring out a way to predict student outcomes (pass, fail, drop-out)using LSTM? I have attributes to take into account - gender (M/F), previous grades (Pass, Fail, Distinct), age (0-35,35-55, >=55) and interactivity (sum of clicks) of each student with virtual learning environment for each day. I have interaction data for each day. Can I use LSTM/RNN on this kind of data?

The interaction data is of 280 days.

Please help!


texts: Do categorical features always need to be encoded?

texts:
I'm using Spark's Machine Learning Library, and features are categorical. The features are strings, and Spark's MLlib (like many other machine learning libraries) does not accept Strings as inputs.

The normal procedure for overcoming this is to convert Strings to integers, and then encode these integers (using a onehotencoder for example), because converting to integers implies that there is an ordering between features.

My question is - do categorical features always need to be encoded? In what situation could integers be used instead of encoding?

I'm using Logistic Regression and Naive Bayes.
When using integers as features I get an 84% accuracy, when these integers are encoded I get an 82% accuracy.

Is it necessary to encode?


texts: What is the term for data that is too sparse to represent the underlying data model?

texts:
I am giving a presentation on Data Science, and I want to talk about the idea that data that is not "big" enough is a big barrier for Machiene Learning. Looking online, there are concepts like overfitting and underfitting, but I am more looking to talk about data that, even if fitted optimally, would still not actually be a good model for the system.

Is there a good term to use for this?


texts: Need help in selecting a model for a classification problem with binary data

texts:
I have a clinical data set. The goal is to develop a model that predicts race based on binary data, whether or not a gene is present.

I am struggling to find a classification model that works without continuous data. One model that might work is k-modes. Are there any other models that I should consider?


texts: Why embedding or rnn/lstm can not handle variable length sequence?

texts:
Pytorch embedding or lstm (I don't know about other dnn libraries) can not handle variable-length sequence by default. I am seeing various hacks to handle variable length. But my question is, why this is the case? I mean, sequences almost never the same size/length and rnn/lstm should loop through until the end of a sequence. If so, why it will be sensitive to the various length of the sequences of a minibatch?
Pytorch Embedding is a look-up table. To me, I don't see any reason to be sensitive to variable length. Should not be the ideal case is, I can give a minibatch of sentences with a variable number of words? Like the following:

word_embedding = nn.Embedding(17,5)
#each vector is a sentence
word_embeds=word_embedding(torch.tensor([[1,2,3,4,5],[4,5,6,7]]))


texts:
Lets say we have a dataset of 9 dimensional points and I want to apply k-means algorithm.
I was studying an example where they apply PCA before fitting the data into the clustering algorithm. The problem is that I cannot understand why PCA is used in this unsupervised learning setting.
In case we were in a supervised learning setting and we had a labeled dataset I understand that high feauture correlations might lead to bad performance of the resulting predictor. But in this unsupervised learning setting it is not clear to me what's the use of applying PCA. Could you please assist in this ?


texts: Plotting an empty bin in a Seaborn histogram

texts:
I'm going through this YouTube series on simulation by The Coding Train. I'm trying to graph some filtered random numbers, but seaborn is leaving an odd gap in the very middle of the histogram.

My data is filtered by collecting random numbers bigger than the output of some function, like $y = x^2$. I also tested graphing the output from random integers from 0 to 100 and did not get the same gap in the histogram, so I thought the issue might be my data. However, I also made a graph using pygame and there is no gap, I also inspected my data; numbers from 1 to 100 are all represented.

Code:

# Choose a function.
# Make randints a, b.
# If b > function(a), return b.
# Repeat and collect b.
# Graph b as histogram.

import collections
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

sample_size = 10**4
dice_max = 100
def function(x): return (x)**2

def main():
    # Debug: Make random data.
    # values = np.random.randint(0, dice_max, sample_size) # Test different data. Problem goes away.

    # Debug: Inspect values then press enter.
    # for i,(k,v) in enumerate(sorted(collections.Counter(values).most_common())): print(i,k,v); input()

    values = [montecarlo(function) for _ in range(sample_size)]
    pg_histograph(collections.Counter(values))
    sns.set()
    plt.show(sns.distplot(values, bins=dice_max, kde=False, norm_hist=False))

def montecarlo(function):
    """Make randints a, b. If b > function(a), return b."""
    y, b = 1, 0
    while not b > y:
        a, b = (np.random.randint(dice_max) for _ in range(2))
        y = function(a)
    return b

def pg_histograph(bins):
    """Make histograph using pygame."""
    import pygame as pg
    window_width = 800
    window_height = 800
    bar_color = pg.Color('black')
    bg_color = pg.Color('white')
    pg.init()
    window = pg.display.set_mode((window_width, window_height))
    window.fill(bg_color)

    bar_width = int(window_width / len(bins))
    max_count = bins.most_common(1)[0][1]
    for bin_, count in bins.items():
        bar_height = round(count / max_count * window_height) # Fit y values to window.
        x = bin_ * bar_width # Fit x values to bar size.
        y = window_height - bar_height # Bars extend downward from x,y, so they need to be shifted up by bar_height.
        pg.draw.rect(window, bar_color, (x, y, bar_width, bar_height))
    pg.display.update()

main()


texts: Cross between an edit distance algorithm and a phonetic algorithm

texts:
My aim is to find an edit distance algorithm which penalises transformations differently depending on the phonetic context.
Take the Levenshtein algorithm, for example; it penalises the same operation - like a character substitution - equally, regardless of character context.
However, it seems to me that all substitutions/insertions/deletions are not made equal. For example, in some contexts, the transformation C->K should not be as harshly penalised as A->T.
A phonetic algorithm, like Double Metaphone, can help here. It maps both kosmetik and cosmetic to the same phonetic key, but it's aggressive and discards information. There's a clear difference between these two strings even though the phonetic key is the same.
Simply taking the Metaphone key and applying an edit distance algorithm is a poor choice for my goal.
Does a suitable method exist here? Or is the best option to manually fiddle with the internals of an edit distance algorithm?


texts:
how can I calculate the y_score for a roc_auc_score?

I have a classifier, for classes {0,1}, say RandomForestClassifier.

Then, when I apply it to my test data, I will get a list of {0,1}

But roc_auc_score expects y_true and y_score.

As dummy as it might look, after fitting the model, I was making the following:

def find_prob(df):
    def prob(token):
        # the mean is the probability of label {1}
        return df[df.token == token].predicted.mean()
    df['predicted_score'] = df.token.map(prob)
    return df

X_val['predicted'] = model.predict(X_val).astype(int)
X_val = find_prob(tokenize(X_val))

roc_auc_score(y_val, X_val.predicted_score)

where tokenize is only a function which concatenate all the values in a row to a "unique" token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.

Is this the correct approach? Thank you very much.


texts: What is disadvantage of using CART in regression problems?

texts:
Why is CART hardly used for regression?

Is there any significant reason for its unpopularity in regression techniques?


texts: Why does Keras need TensorFlow as backend?

texts: Why does Keras need the TensorFlow engine? I am not getting correct directions on why we need Keras. We can use TensorFlow to build a neural network model, but why do most people use Keras with TensorFlow as backend?


texts: How to use paraphrase_mining using sentence transformers pre-trained model

texts:
I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html
In trial one I run 2 for-loops where in I try to find similarity of given sentence with every other sentence. Here is the code for that -
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')

# Single list of sentences
sentences = ['The cat sits outside',
             'A man is playing guitar',
             'The new movie is awesome',
             'Do you like pizza?']

#Compute embeddings
embeddings = model.encode(sentences, convert_to_tensor=True)

#Compute cosine-similarities for each sentence with each other sentence
cosine_scores = util.pytorch_cos_sim(embeddings, embeddings)

#Find the pairs with the highest cosine similarity scores
pairs = []
for i in range(len(cosine_scores)-1):
    for j in range(i+1, len(cosine_scores)):
        pairs.append({'index': [i, j], 'score': cosine_scores[i][j]})

#Sort scores in decreasing order
pairs = sorted(pairs, key=lambda x: x['score'], reverse=True)

print(len(pairs))
6

for pair in pairs[0:10]:
    i, j = pair['index']
    print("{} \t\t {} \t\t Score: {:.4f}".format(sentences[i], sentences[j], pair['score']))

A man is playing guitar          Do you like pizza?          Score: 0.1080
The new movie is awesome         Do you like pizza?          Score: 0.0829
A man is playing guitar          The new movie is awesome        Score: 0.0652
The cat sits outside         Do you like pizza?          Score: 0.0523
The cat sits outside         The new movie is awesome        Score: -0.0270
The cat sits outside         A man is playing guitar         Score: -0.0530

This works as expected because there can be 6 combinations of the similarity scores between the combinations of 4 sentences. On their documentation page, they mention that this does not scale well because of quadratic complexity and hence they recommend using paraphrase_mining() method.
But when I try to use that method, I do not get 6 combinations but instead only get 5. Why is that the case?
Here is the sample code that I try using the paraphrase_mining() method -
# Single list of sentences
sentences = ['The cat sits outside',
             'A man is playing guitar',
             'The new movie is awesome',
             'Do you like pizza?']

paraphrases = util.paraphrase_mining(model, sentences)
print(len(paraphrases))
5

k = 0
for paraphrase in paraphrases:
    print(k)
    score, i, j = paraphrase
    print("{} \t\t {} \t\t Score: {:.4f}".format(sentences[i], sentences[j], score))
    print()
    k = k + 1

0
A man is playing guitar          Do you like pizza?          Score: 0.1080

1
The new movie is awesome         Do you like pizza?          Score: 0.0829

2
A man is playing guitar          The new movie is awesome        Score: 0.0652

3
The cat sits outside         Do you like pizza?          Score: 0.0523

4
The cat sits outside         The new movie is awesome        Score: -0.0270

Is there a difference between how the paraphrase_mining() works?


texts:
I'm obscuring the data I'm working on, but I think this should get the point across. I'm trying to design a test for the following:

A client sells various popsicles. They think that on days when it is very hot, one particular kind of popsicle sells more than on days when it's not hot. How can I prove if he's correct or not?

My thought is that I could do an A/B test, splitting up between not hot & hot days. For each of those groups, I could find the proportion of sales for that popsicle from all popsicles sold that day. If that proportion rises on hot days, then I can do a test for statistical significance and come to a conclusion.

But I think there have to be other tests/models I can apply based on sales counts alone right? What tests can I run that only involve things like the raw counts of sales on days when it was hot and not? Note that I am planning to use the temperature as a categorical variable but I can also make it continuous if that makes more sense.


texts: Why training error is larger than validation error after PCA?

texts:
We have 4000 features and we are applying Principal Component Analysis to reduce them a small number of features from 20 to 100.

We are performing linear regression.

Both training and validation errors are worse than without any use of PCA.

However we notice that the training error is not lower and actually has become slightly larger than the validation error.

Is this something expected?

Edit:

The variance ratio of the first principal components is high enough in comparison to the others.

What we are noticing is that if you use all of the 4000 principal components then the gap between training error and validation is larger and as you repeat the process and you reduce the principal components to 1000, 200, 100, 10, etc. then the training error comes closer to the validation error (even though both are being increased as expected)


texts: Confused with the derivation of the gradient descent update rule

texts:
I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation:
$$∆C=∇C∙∆w$$
where $∇C$ is the gradient vector/vector derivative of the cost function (MSE) and $∆w$ is the change in weights. It said that the goal is to make the change in cost negative. Good so far. My issue is with the next part. It states that
$$∆v=-η∇C$$
My issue is with this, and why $∆v$ is set to this. Why would we want to change the weights by a small amount of the gradient function?


texts: Unsure of how to implement an equation in PyTorch

texts:
I am trying to implement the SummaRuNNer architecture ( Nallapati et al).

The equation I am stuck at in question is:
$$d = tanh(W_{d}\frac{1}{N_{d}}\sum_{j=1}^{N^{d}}[h^{f}_{j},h^{b}_{j}] + b)$$
Where,
$N_{d} = $ Num of sentences in the document, $h^{f}_{j}$ and $h^{b}_{j}$ are the forward and backward passes of the $j^{th}$ sentence, $[]$ is the concatenation operation, $b$ is a bias term and $W_{d}$ is a learnable parameter.

My question is, what is the best way to implement this in PyTorch.

Also, I am aware that there is already a version of this particular architecture implemented in PyTorch here, however, I am not sure if their implementation of this equation is quite right.


texts:
RStudio automatically recognizes headers in an R script that are set via comments:

I would like to exploit that feature, but I don't quite understand what the rules are for RStudio to recognize them as headers. Can someone explain?


texts: When should the Keras functional API be used?

texts: I have always been using the sequential model for deep learning and Convolution Neural Networks, image classification and prediction. But I am not getting good accuracy everytime. Should I use the functional API? Will it do any good?


texts: How many features does (Random Forest) need for the trees?

texts: How can I determine how many features Random Forest needs to create a tree?


texts: What kind of regression model should I do?

texts:
my research question is the examine the effect of "receiving attention" from other members in an online community on "sustained participation" on the website.

I decided to measure "sustained participation" of each user by  calculating average time difference between the submissions of the user. I calculated it in the following way:

and I measured "attention" by calculating total number of the comments each user received for all the submissions he/she has posted.I also want to consider total number of votes and total number of views as well. I am not sure if it is a good idea to add those as the independent variables into model too or not?

My problem is about Dependent variable:

since some people just participated two times in two successive days and the average between submission days is 1. and some people participated 100 times and their average between submission days of them is also 1. but it is obvious that the second group that have participated 100 times had sustained participation not the first group.

So I need to consider the number of the submissions in the model too! I do not know if there is a way to consider it? how I can handle this problem?

should I group the users and do data analysis separately for them. for example users who have participated less than 10 times in one group! users with 10-20 participation in another group and etc.

I would appreciate if anyone can help me! my paper's due is so close and I need some preliminary results.


texts:
When I first read about neural networks, I learned that Backpropagation is the algorithm used to train the neural network. I am interested if there are other alternatives (or better?) to BP.

What are the other training algorithms used in NN? And is BP is the best one, and that's why almost everyone uses it for training the NN model?


texts: Spark, optimally splitting a single RDD into two

texts:
I have a large dataset that I need to split into groups according to specific parameters.  I want the job to process as efficiently as possible.  I can envision two ways of doing so

Option 1 - Create map from original RDD and filter

def customMapper(record):
    if passesSomeTest(record):
        return (1,record)
    else:
        return (0,record)

mappedRdd = rddIn.map(lambda x: customMapper(x))
rdd0 = mappedRdd.filter(lambda x: x[0]==0).cache()
rdd1 = mappedRdd.filter(lambda x: x[1]==1).cache()

Option 2 - Filter original RDD directly

def customFilter(record):
    return passesSomeTest(record)

rdd0 = rddIn.filter(lambda x: customFilter(x)==False).cache()
rdd1 = rddIn.filter(customFilter).cache()

The fist method has to itterate over all the records of the original data set 3 times, where the second only has to do so twice, under normal circumstances, however, spark does some behind the scenes graph building, so I could imagine that they are effectively done in the same way.  My questions are:
a.) Is one method more efficient than the other, or does the spark graph building make them equivalent
b.) Is it possible to do this split in a single pass


texts: Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv'

texts:
I'm trying to create some charts using Python. I've this dataset in a CSV file:

Banana  Water   Rice
Rice    Water
Bread   Banana  Juice

And I've this code:

 import numpy as np
    from pandas import DataFrame
    import matplotlib
    matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)
    import matplotlib.pyplot as plt

    data = DataFrame.read_csv("test.csv", index_col=1, skiprows=1).T.to_dict()

But when I execute the code I'm getting the following error:

AttributeError: type object 'DataFrame' has no attribute 'read_csv'

How can I solve this problem?

Many thanks!


texts: Implementing Frequently bought together using a DB

texts:
We have a classic structure of an online shop database (products, customers, sales) and we want to implement a Frequently bought together feature. Our software is in ASP.NET and we do not know PHP to reverse engineer how this is being done in Magento.

And all we need is a simple Frequently bought together (not with discounts like Magento offers).

I understand that this is machine learning and one of the more common ways is Jaccard coefficient. Is that the recommended way?

My main question however is what the View I create in the DB should look like? What is the structure of the table that Jaccard coefficient and other relevant tests require so that used best?


texts: Orange file import method for raw binaries

texts:
I tried to add a  binary import method to
    class FileFormat(metaclass=FileFormatMeta)
in io.py, but gave up trying to adapt a copy of the CSV method since so much is specific to CSV issues and the comments are sparse...

Basically, given an Ncols x Mrows raw file of known type and meta data, one should be able to supply the shape, sample rate etc as defaults, and then one "only" needs to satisfy the parameters for Table.from_numpy(). It is the creation of Orange-specific objects that have me stuck, and mainly the domain object:

def data_table_from_bin(cls, data, headers=None):
    ...
    domain = Domain(attrs, clses, metas)

    table = Table.from_numpy(domain, X, Y, M, W)
    return table

Has anyone else tried this, or have suggestions?


texts: Deep Q network disregards input, giving identical output no matter the input state

texts:
I've creating a very simple game,
The board is an array of size 6.
0 is "empty cell" , 5 is "Goal", 8 is "player location"
[8 0 0 5 0 0] for example means the agent needs to move 2 "right" to win.

hitting a "5" ends the game with reward +1
hitting a wall ends the game with reward -1
other moves have reward 0.

the "state" is simply the board, 6-long vector that can be 0\8\5

When i built a regular Q-Table based agent the game converged (to a good result) very fast.

The problem is that the DQN (Deep Q network) agent never converged. i sampled the network after a few thousand iterations and saw that the network for all inputs always has the same result.

The actual result keeps changing, each replay of the memory, but at each point if i ask to predict, no matter the input, the output will be "LEFT: X, RIGHT: Y".
(X and Y are numeric values). the network ignores the input. any suggestions why?

Appendix:
Hyperparams:
batch-size = 64
memory = 100 (tried 1000 as well(
gamma = 0.99  # discount rate
epsilon_min = 0.05
epsilon_decay = 0.95
epsilon_start = 1.0
learning rate = 0.01 / 0.1 / 1.0

the network is ( i tried several):

        # Neural Net for Deep-Q learning Model
    model = K.models.Sequential()
    model.add(K.layers.Dense(8, input_dim=self.state_size, activation='relu'))
    model.add(K.layers.Dense(8, activation='relu'))
    model.add(K.layers.Dense(self.action_size, activation='linear'))

    model.compile(loss='mse',
                  optimizer=K.optimizers.Adam(lr=self.learning_rate))


texts: Finding and ranking best semantic matches between two sets of phrases

texts:
I'm looking for a proper definition for what sort of problem this is, so I can further research it on my own - though I will, for sure, appreciate any specific advice on what are industry standard ways for working on this. While I'm fairly inexperienced with NLP or recommender systems, I've done a fair amount of classical ML before, in case that matters.
The problem - given a "search" query with a list of inputs and a list of expected outputs, retrieve and rank up to N best semantic match for each input. Constraints:

All inputs and outputs are anywhere between a word and a sentence.
Number of outputs >= number of inputs.
All inputs and outputs are unique within themselves, but there can be outputs that are identical to inputs.
Every input is guaranteed to have at least one "good enough" output present.
Every output is guaranteed to be the best match for at most one input.
Labelled data is available, i.e. human-tagged queries. Inputs are sparse, i.e. inputs for queries targeting identical or very similar output sets can be very different.
Data is in English, and I work in Python - if that matters for your suggestions.

An example:
Inputs
(1) Truck
(2) Assortment of lemons, limes, and oranges.
(3) Apples, pears, and oranges.
(4) A tool with broad blade, used for digging.

Outputs
(a) Citrus fruits
(b) Portable telephone that can make and receive calls over a radio frequency.
(c) Fruits traditionally grown in Germany.
(d) Vehicles used for cargo transportation.
(e) Shovel
(f) Motor vehicle used for transportation.

Desired result (numbers are arbitrary, for illustrative purposes)
1 - d (100%), f (75%)
2 - a (95%), c (60%)
3 - c (87%), a (45%)
4 - e (100%)

Similar question suggestions by Stack Exchange are not answering my question, and searching for the answer elsewhere just pits me into endless stream of articles about sentiment analysis of IDMB or Twitter datasets.


texts: How do I develop a system to Recommend a marketing channel using data science?

texts:
I am working on a data science challenge. I would like some help on how to proceed in developing my solution. A snapshot of my dataset is shown below:

This dataset is collected by a drug making company trying to sell its drug to doctors of different specializations.

The drug company has made promotional activity for its brand

The promotional activity has included calls made, emails sent and faxes sent. The dataset shows how many calls have been made ('Calls Made' column), how many calls were successfully completed ('Calls Successfully Completed' column), how many emails were sent ('Emails Sent' column), how many emails were opened ('Emails Opened' column), how many faxes were sent ('Faxes Sent' column). Brand 1 is the brand of the drug company. Brand 2 is the brand of its competitor.

The dataset shows how many sales have been made for various brands. Brand 1 Sales (Company's brand) - number of sales of the drug company's brand Brand 2 Sales (Competitor brand)- number of sales of the competitor's brand Total Branded Market Sales - number of sales of the all brands (from all companies selling branded drugs) in the market Total Market (Branded + Unbranded) sales - number of sales of all branded and unbranded drugs

My task is to develop a ML system using this dataset that will recommend which of the three channels to use (calls/emails/fax) for future promotional activity. How do I develop such a system? My understanding is that this is a classification problem with three classes (calls/emails/fax). For a prospective customer, I have to classify him/her into one of the three classes that would indicate what channel to use.

What kind of feature engineering should I do? What model can I use?


texts: Understanding why ensembling only improves marginally

texts:
I have two models, A and B, trained on Imagenet. Their accuracies on Imagenet validation set are 35.6% and 28.64% respectively, while the accuracy of their ensemble (averaging their scores) is 35.68%. I am interested in finding out why the ensembling isn't effective here.

Specifically, I was going to inspect the confusion matrices for each model, but Imagenet has more than 1000 classes which makes this intractable. Another thing that was suggested to me is Mutual Information, but I can't figure out how to apply it in this context.

So, I have a two part question:

Why doesn't the accuracy of the ensemble degrade (to the average of
two accuracies) or improve?
Is there a way of visualizing/scoring
the output of the two networks to measure correlation?

Edit 1: Both are AlexNet models, but were trained with two different pre-trained weight initializations. The pre-trained weights themselves come from two different self-supervised tasks. Also, when these models were trained (initialized with respective pre-trained weights) on Pascal, there's is a significant boost in the accuracy. Thus, my quest to figure out how do you measure the correlation between models that are ensembling.


texts: Scikit alternative for categorical data modeling?

texts: So, sklearn doesn't support categorical data in its models. Is there a known alternative for categorical data modeling (such as random forests, etc.) for Python?


texts:
First question : can I mix different sorts of inputs types for example, height and age (of course my inputs are normalized)? in general, can we mix different types of inputs in a neural network ?

Second question : can too much different inputs cause overfitting ?

I am using 120 inputs neurons and 20,000 train data and I am overfitting at 53% accuracy (bad)...

Thank you.


texts: Research on Product Search Machine Learning @ Amazon

texts: I am looking for any sort of research paper/ reference where we can read more about machine learning principles using for the amazon's main website. Mainly interested in how they do the product search and show the "similar products others may have bought"


texts: Data normalization before or after train-test split?

texts:
Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

normalized_X_features = pd.DataFrame(StandardScaler().fit_transform(X_features), columns = X_features.columns)
x_train, x_test, y_train, y_test = train_test_split(normalized_X_features, Y_feature, test_size=0.20, random_state=4)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(x_train, y_train)
y_test_pred = LR.predict(x_test)

Normalization after split

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)
normalized_x_train = pd.DataFrame(StandardScaler().fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(StandardScaler().fit_transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)

So far I have seen both approaches.


texts: How to get pixel location in after rotating the image?

texts:
I'm trying to rotate some images with some boundary boxes, but I couldn't get the new bb. So if I have an image of 100x70 and I have a pixel at (19,39) and then I rotate the image with angle = 45,
how can I calculate the new position of this pixel?


texts: While retraining a pretrained model, getting: ValueError: Input 0 is incompatible with layer flatten_1: expected min_ndim=3, found ndim=2


texts: Pandas: access fields within field in a DataFrame

texts:
Suppose I have such a JSON file:

[
  {
    "id": "0",
    "name": "name0",
    "first_sent": "date0",
    "analytics": [
        {
            "a": 1,
            ...
        },
        {
            "a": 2,
            ...
        }
    ]
  }
]

and I want to parse it with Pandas. So I load it with

df = pd.read_json('file.son')

It's all good until I try to access and count the number of dictionaries in the "analytics" field for each item, for which task I haven't found any better way than

for i in range(df.shape[0]):
    num = len(df[i:i+1]['analytics'][i])

But this looks totally non-elegant and it's missing the point of using Pandas in the first place. I need to be able to access the fields within "analytics" for each item.
The question is how to use Pandas to access fields within a field (which maps to a Series object), without reverting to non-Pandas approaches.

A head of the DataFrame looks like this (only fields 'id' and 'analytics' reported):

0    [{u'a': 0.0, u'b...
1    [{u'a': 0.01, u'b...
2    [{u'a': 0.4, u'b...
3    [{u'a': 0.2, u'b...
Name: analytics, dtype: object
0      '0'
1      '1'
2      '2'
3      '3'

The first number is obviously the index, the string is the 'id', and it is clear that 'analytics' appears as a Series.


texts: Should the baseline comparison be done on the training or the test set results?

texts:
I have a classification problem where I want to find out whether feature engineering has improved my final model. Cross-validation is used evaluate the impact of the feature engineering steps, so there is no validation set (only train/test). In short, my situation entails the following:

Collect data
Train baseline model
Feature engineering
Train final model
Compare final model against baseline (question)

Comparing the baseline and final models, I assume, can be done by running both models on the test set, subsequently evaluating the differences in their results (if any). However, I wonder if it is useful to compare the models using the training set as well/instead. It would be great if someone could elaborate on this issue.


texts: How to apply sequentinal MSE model to data that is not binary?

texts:
I have been using this model with binary data to predict likely hood of play from this guide.

import tensorflow as tf
from tensorflow import keras
import numpy  as np
import pandas as pd

model = keras.Sequential()

input_layer = keras.layers.Dense(3, input_shape=[3], activation='tanh')
model.add(input_layer)

output_layer = keras.layers.Dense(1, activation='sigmoid')
model.add(output_layer)

gd = tf.train.GradientDescentOptimizer(0.01)

model.compile(optimizer=gd, loss='mse')

sess = tf.Session()  #NEW LINE

training_x = np.array([[1, 1, 0], [1, 1, 1], [0, 1, 0], [-1, 1, 0], [-1, 0, 0], [-1, 0, 1],[0, 0, 1], [1, 1, 0], [1, 0, 0], [-1, 0, 0], [1, 0, 1], [0, 1, 1], [0, 0, 0], [-1, 1, 1]])
training_y = np.array([[0], [0], [1], [1], [1], [0], [1],[0], [1], [1], [1], [1], [1], [0]])

init_op = tf.initializers.global_variables()

sess.run(init_op)  #NEW LINE

model.fit(training_x, training_y, epochs=1000, steps_per_epoch = 10)

text_x = np.array([[1, 0, 0]])
test_y = model.predict(text_x, verbose=0, steps=1)

print(test_y)

All the current data is binary and model works with binary, is there any model or way to convert non-binary data to binary predict likelihood of product_sold in the below data set?

dataset:

number_infants    cost_of_infants     estimated_cost_infants    product_sold
5                     1000               2000                         0
6                     8919               1222                         1
7                     10000              891                          1

product_sold
1 = yes
0 = no


texts: How to Predict Employee count of businesses using Keras classifiers

texts:
I am trying to predict the amount of employees a business has based on a set of input variables. I am using things like the business's age, transaction details, geographic location, business structure complexity and a few other input features.

I have a proof data-set of about 18,000 examples, which I am using to train Keras classifiers to label unseen data into bands of employee counts (Group 1-5, Group 6-9 etc.).

I am using a classifier for each band which simply run through all the data and then each classifier will classify if that business belongs in that band or not (Group 1-5, Group 6-9 etc.). The results from each classifier are then compiled for each row and the classifier with the highest confidence (probability) is deemed to be the band that the business is in and is then labelled as.

However I am having some trouble with running the whole algorithm. It all works and each classifier can generate a 70%+ classification accuracy however this is only because the data is hugely unbalanced and the classifiers are just randomly guessing and for some reason generating a high accuracy. The classifiers cannot generalize and find patterns in the input data, that maps characteristics of the input data to the desired output label. I believe this is due to my input variables being weakly correlated to my output variable.

I was wondering if anyone could give me some insight into how to improve the true accuracy of my classification, or if perhaps a regression approach may be more effective (Gave worse results when testing)...

Another note is that my input data has quite a lot of categorical features which makes it very sparse (EG: Having 300 different geographic location possibilities)

I have attached an image of an example of one of my classifiers for one employee count band (The 1-5 Group).

Thanks very much


texts: Both train and test error are decreasing in XGBoost iterations

texts:
I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 early stopping rounds. Then when I try to apply the model on a separate set not used for testing and training I can see that a model trained over 100 rounds performs much better than 1000 one. My learning rate is 0.01.

I am wondering whether this is a normal situation (and I should just do early stopping on this independent validation set) or whether I am doing something wrong. Theoretically, the test error should start increasing at some point, but this is just not happening.

I am using XGBoost and I am splitting my dataset in train/test like this:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

Them I train it in the following way:

watchlist =  [(dtest, 'eval'), (dtrain, 'train')]
progress = dict()

# Train and predict with early stopping
xg_reg = xgb.train(
    params=params,
    dtrain=dtrain, num_boost_round=boost_rounds,
    evals=watchlist,    # using validation on a test set for early stopping; ideally should be a separate validation set
    early_stopping_rounds=early_stopping,
    evals_result=progress)

Note that I have used gridsearch to find the optimal hyperparameters (params).

A plot which I get (yellow is my cutoff point identified pretty much via testing on the independent set):


texts: Connect a dense layer to a LSTM architecture

texts:
I am trying to implement an LSTM structure in plain numpy for didactic reason. I clearly understand how to input the data, but not how to output.
Suppose I give as inputs a tensor of dimension (n, b, d) where:
    •   n is the length of the sequence
    •   b is the batch size (timestamps in my case)
    •   d the number of features for each example
Each example (row) in the dataset is labelled 0-1. However, when I fed the data to the LSTM, I obtain as a result the hidden state h_out which has the same dimension of the hidden size of the network. How can I obtain just a number that can be compared to my labels and properly backpropagated?
I read that someone implements another dense layer on top of the LSTM, but it's not clear to me the dimensions that such layer and its weight matrix should have.


texts:
I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three or four paragraphs, the system fails miserably. How can I fix this?

I read the file, split it into lines, sentences and then words. What I get is a list of lists where each internal list contains words of a sentence. Then, I use NLTK to tag each sentence.

def process_file(_file, tagger, stemmer, stopwords, filename, printinfo):
    sentences = []
    _nnp = set()
    words = dict()
    for line in _file:
        for sentence in nltk.tokenize.sent_tokenize(line):
            sentences.append(nltk.tokenize.word_tokenize(sentence))
    sent_count = 0
    for sentence in sentences:
        sent_count+=1
        tags = tagger.tag(sentence)
        for tag in tags:
            if tag[1] == "NNP" or tag[1] == "NNPS":
                _nnp.add(tag[0])
            else:
                if tag[0] not in stopwords:
                    stemmed_word = stemmer.stem(tag[0])
                    if stemmed_word not in words.keys():
                        words[stemmed_word] = 1
                    else:
                        words[stemmed_word] += 1
        print("\r[{0}] Reading file '{1}'[{2:>3.1%}] ".format(printinfo, filename, sent_count/len(sentences)), end='')
return _nnp, words

Main code:

dictionary = dict()
nnp = set()

# Initialize tagger and stemmer
pos_tagger = nltk.tag.PerceptronTagger()
ps = nltk.PorterStemmer()
stopwords = nltk.corpus.stopwords.words('english')
file_count = 0

# For each file, extract words and named entities
for file in files:
    file_count += 1
    _nnp, _dictionary = process_file(open(file, 'r', encoding='utf-8'), pos_tagger, ps, stopwords, file, str(file_count)+"/"+str(len(files)))

    # Extend dictionary
    for word in _dictionary.keys():
        if word in dictionary.keys():
            dictionary[word] += _dictionary[word]
        else:
            dictionary[word] = _dictionary[word]

    # Join sets
    nnp = nnp.union(_nnp)
end = time.time()
print("COMPLETED: Step 2 completed in {0:.3f}s".format(end-start))

Here's a sample of what the text might look like:

In slow motion, afraid of what he was about to witness, Langdon rotated the fax 180 degrees. He looked at the word upside down.

Instantly, the breath went out of him. It was like he had been hit by a truck. Barely able to believe his eyes, he rotated the fax again, reading the brand right-side up and then upside down.

"Illuminati," he whispered.
```


texts: Why we use information gain over accuracy as splitting criterion in decision tree?

texts:
In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on.

I think that using accuracy instead of information gain is simpler approach. Is there any scenario where accuracy doesn't work and information gain does?

Can anyone explain what are the advantages of using Information gain over accuracy as splitting criterion?


texts: What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

texts:
I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding:

hessian regularisation blocks the individual trees ($f_t$) from growing (much like limiting max_depth)
loss regularisation blocks the individual trees ($f_t$) from being kept in the ensemble as it penalises number of trees

However, I have some troubles to find practical implications from that. I mean, is there a situation when hessian regularisation would perform better than gamma regularisation (or worse)? Or is it all dependent on the dataset and the other hyperparameter values? My feeling is that when my model loss is mainly caused by few awfully inaccurate prediction - then hessian regularisation might help more than gamma, and when my model is mistaken by approximetely the same amount for every observation then gamma would be better. But I'm struggle to figure out the proof for that feeling.


texts: context-based model and interaction terms

texts:
I have implemented a prediction model.
Now I am checking if I should split the model to two context-based models (men and women).
I have created a new binary feature that is relevant to only part of the women's group and it can affect the prediction accuracy for this group.
I don’t have a lot of data so I am not eager to split to context-based models.
I know that in this cases it is best to use an interaction terms features (womenxnewfeature) but this feature is only part of the women's’ group and I have heard that adding the feature as it is it’s not good.

Any suggestion on what I can do?


texts: Dealing with missing n-grams in Naive Bayes classifier

texts:
I am doing sentiment analysis on code-mixed text data, i.e English used interchangeably with another language. The dataset I currently have is very small in size, approx 3.5k samples.

I am sure that this dataset will not contain all possible n-grams that are used in the language. Are there any ways I can deal with this problem without increasing the number of samples in the dataset?

For now, I am using a MNB classifier to do the sentiment analysis.


texts:
I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, the with and without batch normalization results have lot of difference. With batch norm results being poorer. As I increase the batch size the performance of with and without batch normalization gets closer. Something funky going on here.

So I would like to ask is the increase in batch size have effect on batch normalization?


texts: I am trying to train a Variational Autoencoder (VAE) to learn a curve, $f_1(x)=x^2+\omega$. It can learn it to a great accuracy, but the problem is that once is trained, for any input to the encoder, the decoder will always output $x^2$. In particular, when training it seems the decoder just learns how to map any input to $x^2$, so the problem seems to be in the decoder, but I am not sure how to solve it. Has anyone experienced this problem? I'd appreciate any suggestion or idea on how to solve this issue.


texts: Feature and the Gaussian Distribution (classification)

texts:
I have a question regarding variable following or not a random distribution.
I selected 4 features negatively correlated to the label (Fraud/No Fraud). The notebook I'm taking the inspiration from plotted the distribution of these feature regarding the label. What came out is that my feature 1 (Fraud only) is following a Normal Distribution.
Here are my questions :

Why is it important to know if my feature is following a Normal Distribution ? -> My guess : some models need it for faster convergence or better results
Is there any interest to visualize my features as Non Fraud vs Fraud and compare the distributions ?
If my features are not following a Normal Distribution but are scaled, should I still force them to a Gaussian like shape ?

Thank you very much !


texts: How the vector of weights assigned to a neural network?

texts:
You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.

We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections?

This is one pattern of assigning weights to neural network. See the another way of assigning the weights.

Please tell which one of the following is the correct way of assigning the weights to the neural network?


texts: I see r""" followed by a comment in quite a few of the source codes for PyTorch and the one I am looking at is here. What is the r for?


texts: How to implement this CNN architecture in Keras

texts:
I am trying to implement in Keras the CNN architecture used by Rajpurkar et al and illustrated below:

I am particularly confused about that max pool that is shown in the right side. Is that a fork from the main sequential line? How do I implement that?

Thanks.


texts: State of the art for Object detection/image recognition

texts:
I was asked to verify the feasibility for solving a particular problem: recognizing for a fashion brand the model of its products.

I have little experience with image recognition in general, I always used Google Vision Api or some pre trained nets from Google/tensorflow like the VGG-16.

I am wondering: Do you think this level of granularity can be achieved?

I guess I should for example train the last layer of a pre-trained network, but I need many images to do that.


texts:
I am new to data science and new to R. I see a lot of resources on data cleaning (i.e., dplyr, tidyr, tidyverse), but not so much on raw value verification. What are the "best practices" I should apply to verify the accuracy of the raw values or detect possible coding errors before cleaning? For example, I have been told you should ensure NAs are true NAs, or ensure there are no spaces in front of numerical or character values after you import data into R.

Thanks in advance.


texts:
Please, I am stuck, I can not understand the number of parameters of a simple RNN, here the example and the model summary.
the example is simple:
x = np.linspace(0,50,501)
y= np.sin(x)
df= pd.DataFrame(data=y, index=x, columns=['Sinus'])

Then I would to build a simple RNNs to predict this sine wave,
test_percent = 0.1
test_point= np.round(len(df)*test_percent)
test_ind = int(len(df)-test_point)
train = df.iloc[:test_ind]
test = df.iloc[test_ind:]

from sklearn.preprocessing import MinMaxScaler
scaled_train = scaler.transform(train)
scaled_test = scaler.transform(test)

from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator
generator = TimeseriesGenerator(scaled_train, scaled_train, length=50, batch_size=1)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, SimpleRNN

n_features=1
model= Sequential()
model.add(SimpleRNN(units=50, input_shape=(50,1)))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam')
model.summary()

I do not understand the number 2600 in the model summary!
because according to this intuition of RNNS

in my case the dimension of U is (50,1), so 50 weights, then for V, also 50 weights, and for W, also 50 weights, for me, only 150 parameters, why this number in the summary: 2600 parameters?


texts: Using Linear Regression to Learn Polynomial Regression

texts:
Let's start by considering one-dimensional data, i.e., $d=1$. In OLS regression, we would learn the function
$$
f(x)=w_{0}+w_{1} x,
$$
where $x$ is the data point and $\mathbf{w}=\left(w_{0}, w_{1}\right)$ is the weight vector. To achieve a polynomial fit of degree $p$, we will modify the previous expression into
$$
f(x)=\sum_{j=0}^{p} w_{j} x^{j}
$$
where $p$ is the degree of the polynomial. We will rewrite this expression using a set of basis functions as
$$
f(x)=\sum_{j=0}^{p} w_{j} \phi_{j}(x)=\mathbf{w}^{\top} \boldsymbol{\phi}
$$
where $\phi_{j}(x)=x^{j}$ and $\phi=\left(\phi_{0}(x), \phi_{1}(x), \ldots, \phi_{p}(x)\right)$. We simply apply this transformation to every data point $x_{i}$ to get a new dataset $\left\{\left(\phi\left(x_{i}\right), y_{i}\right)\right\}$. Then we use linear regression on this dataset, to get the weights $\mathbf{w}$ and the nonlinear predictor $f(x)=\sum_{j=0}^{p} w_{j} \phi_{j}(x),$ which is a polynomial (nonlinear) function in the original observation space. Notes
How does this work? Could anyone give me an example and explain it to me in simple terms? How would I go about implementing this in Numpy?
Thank you in Adance


texts: NLP Library or routine for Quantitative relationships

texts:
I'm searching for generalized methods and libraries to extract quantitative relationships between concepts in natural language.  For instance

Exam score greater than 90%
Exam scores of 90% or greater

Should extract to a structure like

Exam Score,>=,90%

I began with a simple regexp / dictionary method, but it feels very primitive.


texts: What is the meaning of 'concatenate' in this neural network architecture?

texts:
I am trying to understand the lane edge proposal network proposed in LaneNet for lane detection.
My understanding of this is that a number of convolutional and pooling layers are first used to downsample the input images into feature maps, and then three deconvolutional layers (using sub-pixel convolution) are used to upsample back to original image size.
What I don't understand is the dashed 'concatenate' arrows between equally sized conv/deconv layers, they are not mentioned outside this diagram.
Also, the paper also states "Skip connections are used to provide high resolution feature for accurate lane edge localization", but no mention is made of skip connections elsewhere in the paper. My understanding of skip connections is with regard to residual blocks, which doesn't seem to link to this diagram. Unless the concatenate arrows are skip connections? In which case my earlier assumption, that input images simply pass through each conv and deconv block in order, no longer holds.


texts: pandas apply function with multiple condition?

texts:
if i want to apply lambda with multiple condition how should I do it?

df.train.age.apply(lambda x:0 (if x>=0 and x<500))

or is there much better methods?


texts: Best practices for scoring hundreds of models on the same massive dataset?

texts:
I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables.

Currently, my scoring process takes about 5 days, and operates by chunking up the 400m+ records into 100k-person pieces and spinning up n-number of threads, each with a particular subset of the 500+ models, and running this way until all records are scored for all models. Each thread is a Python process which submits R code (i.e. loads an R .rds model and associated dataset transformation logic).

This process takes way too long, is severely error-prone (more of an indicator of the tangled web of code it's become), is expensive (massive cloud instance required), and only allows for models to be built in R (I want to basically be agnostic of the language from which the model is coming, but mainly I want to enable Python and R – that is a non-negotiable requirement).

Does anyone with experience in a similar problem domain have any advice re: how this process could be re-architected to 1) run more efficiently (from a $ PoV) and 2) enable both Python and R models.


texts:
I am trying to create a folder to use in the SAS library. I have tried to do this many different ways. Every time I run the code it always says that the path does not exist.

Below is my error messages from my last attempt:

 72
 73         libname "D:\Users\clkehl01\Documents\myprj1\";
 ERROR: "D:\Users\clkehl01\Documents\myprj1\" is not a valid SAS name.
 ERROR: Error in the LIBNAME statement.
 74
 75         data myprj1.autos;
 76         set sashelp.cars;
 77         run;

 ERROR: Library MYPRJ1 does not exist.
 NOTE: The SAS System stopped processing this step because of errors.


texts: Where to find state-of-the-art time series data deep learning approaches?

texts: Just like this (which stopped updating) that shows the newest deep learning methods utilized in image recognition, is there some similar lists/summaries (in blog posts/paper/competitions etc.) about newest deep learning methods used in time series analysis?


texts: Naive bayes expectation maximization vs logistic regression for binary classification

texts:
Assuming I'm dealing with binary classification.
For what kind of data Naive bayes using expectation maximization would give a better solution and for what kind of data logistic regression would be the better choice?


texts:
I have a dataset which consists of attributes on breakdown of machines.The target variable is machine status which are populated with ones and zeros. The distribution of ones and zeros are given below

 0 - 19628
 1 - 225

0 - signifies the machine is running good and 1 signifies there was a breakdown.

Now, should I go by splitting the dataset using scikit train_test_split method ?. or introduce artificial rows to mitigate the tradeoff between ones and zeros and then split the dataset.

Well, What do I mean by artificial rows ?
Populate  some random data with having target variable as 1's
But that would ultimately mislead the system. I don't see any other options or alternatives.

Is there any way how to make samples balanced?


texts: Reconciling time-based data when data source clock drifts

texts:
How can I reconcile time-based data when the clock on the data source tends to drift and the data may be infrequently retrieved?

I measured the clock to be about 30 minutes behind after 15 hours. UPDATE: the manufacturer confirmed the worst case drift is roughly 10% at the temperature extremes (-40c - 125C)

Due to extremely limited space on the data source device, the data is stored in buckets for each hour and there are 24 buckets, one for each hour in the day. The data is the total count for that hour. UPDATE: we were able to reduce the interval for the buckets to 15 minutes, leaving us with 96 smaller buckets. This is around the point where inaccuracies in the allocation may be acceptable

The device retrieving the data can correct the clock on the data source, but it is not always connected. Data is unlikely to be generated overnight, but may be generated in the morning before the retrieving device is connected. The retrieving device may be connected only infrequently.

One problematic case I have identified is if the retrieving device is connected only in the morning and evening, with more data generated later. In the evening, the data source would be significantly behind, maybe 30m. If the data is retrieved in the first half of an hour, the data source would still be in the previous hour, and data for the rest of the hour would be put in a different bucket after the time is corrected.

Here is a sample of the time drift over about 9 hours: (when the color changes, I stopped rounding down the drift)


texts: how to find the best parameters to solve a differential equation?

texts:
I have a differential equation:
def func(Y, t, r, p, K, alpha):
    return r * (Y ** p) * (1 - (Y / K) ** alpha)

and I want to find the best parameters that fit (r,p,K,alpha). I tried to use curve fit but it was too bad, this is my code
# I chose the value of maxfev randomly
popt, pcov = curve_fit(func, df.index, df.Value,method='lm',maxfev = 8000)

t = np.linspace(0, len(df), len(df))
y0 = popt[0]
params = (popt[1], popt[2], popt[3], popt[4]) # r, p, K, alpha
sol = odeint(func, y0, t, args=params)

and this the plot:

Note: this is how my real data looks like:

what I'm looking for it the best values for (r,p,K and alpha), how to find them?


texts: How to classify all words in a sentence with a context?

texts:
I have the names of the companies (in Russian). The name can contain abbreviations, words with capital letters, words with lowercase letters, and mixed words. The model is trained according to the principle: At the input, the name is given in upper case, at the output - in the "correct" version. For example (adapted):
"SIGMA" LIMITED LIABILITY COMPANY -> "SIGMA" Limited liability company
"SIGMA" LLT -> "SIGMA" LLT
PJSC "GAZPROM"-> PJSC "Gazprom"
STATE BUDGETARY EDUCATIONAL INSTITUTION OF THE CITY OF MOSCOW "LYCEUM NO. 1568" -> State budgetary Educational institution of the city of Moscow "Lyceum No. 1568"

And then I have to predict sentences that I don't know answers for. The size of my dataset is about 5 million rows or 600 MB.
I tried to make a character-based seq2seq model with attention and bidirectional GRU (based on a pytorch guides), but with all the hyperparameters I tried, it seems to be underfitted. It generates the beginning of phrases quite well, but at the end it breaks down.
Now I think that it's better to work with word tokens. But I do not know if there are methods for classifying words from a single text in a context. (Because i need to know, where is the word in sentence, what words are next it predict in't case)
I want to tokenize a sentence, and assign a property to each word, what it is: uppercase, lowercase, or with a capital first letter. Also I need to do something with mixed words like "МосГосПаравоз" (or like "McDonald's" which not only starts with a capital letter, but also contains it inside.).
Perhaps I need a completely different approach. I'll be happy to accept your help


texts: How to get sentiment analysis data widget in python script in orange and how this data used for other purpose and apply on this sentiments data  different methods


texts: Eigen Decomposition of Data Matrix for PCA

texts: In PCA we Eigen decompose the covariance matrix, not data matrix, Is it because most data matrices are non-square. If they were, isn't is correct to eigen decompose data matrix than the covariance matrix?


texts:
One limitation of NDCG and way to overcome the limitation (as mentioned in
https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Limitations) is

"Normalized DCG does not penalize for missing documents in the result. For
example, if a query returns two results with scores 1,1,1 and 1,1,1,1,1
respectively, both would be considered equally good, assuming ideal DCG is
computed to rank 3 for the former and rank 5 for the latter. One way to take
into account this limitation is to enforce fixed set size for the result set
and use minimum scores for the missing documents. In previous example, we
would use the scores 1,1,1,0,0 and 1,1,1,1,1 and quote nDCG as nDCG@5"
In the mentioned scenario, scores are 1,1,1 and 1,1,1,1,1 and both have NDCG =
1
After applying fix scores are 1,1,1,0,0 and 1,1,1,1,1. NDCG is still 1 for
both, so it seems the fix is not working
Could you explain where im wrong and if possible could you elaborate the fix
for NDCG limitation of not penalizing missing documents or point out to some
resource. Thanks


texts: Help understanding input to biaxial network for generating music

texts:
I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then what is passed along the note axis?

The author says:

If we make a stack of identical recurrent neural networks, one for each output note, and give each one a local neighborhood (for example, one octave above and below) around the note as its input, then we have a system that is invariant in both time and notes: the network can work with relative inputs in both directions.

This might mean that the inputs passed to the network along the note axis are fixed representations of notes in the vocabulary. But I am not sure.
I am also having a hard time understanding the input passed to this network as the author explains a few paragraphs below. (Position, Pitchclass, Previous Vicinity, Previous Context, Beat).
Also, at some point, the author talks about RNN along the note axis. But in the architecture, there only seems to be RNN along the time axis. I would really appreciate is anyone could give me some more information to understand how this Biaxial Network is setup. This article by Deep Dark Learning was a little helpful but I am still not fully sure what is going on here.


texts: How to design the generator in generative adversarial network (GAN)?

texts:
I am new to GAN. And recently read a paper about how to implement GAN in recommender system here

I have a question in the paper.

The Eq (20) in the paper should be the  discriminator function.

The generator should be the $G_{\phi}$ which is mentioned in the Eq (21) and Eq (22). However, what is definition of $G_{\phi}$ here?

Thanks,


texts:
OK this is my first time in ML and for starter I am implementing Naive Bayes. I have Cricket(sports) data in which I have to check whether the team will win or lost based on Toss Won|Lost and Bat First|Second. Below is my code:

from sklearn.naive_bayes import GaussianNB
import numpy as np

"""
    Labels : Lost, Draw, Won [-1,0,1]
    Features
    ==========
        Toss(Lost,Won) = [-1,1]
        Bat(First, Second) = [-1,1]
"""
#Based on Existing Data Features are:
features = np.array([[-1, 1],[-1, 1]])
labels = np.array([0,1])
# Create a Gaussian Classifier
model = GaussianNB()

# Train the model using the training sets
model.fit(features, labels)

# Predict Output
predicted = model.predict([[1,0]])
print(predicted)

On running this I get error:

/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:393: RuntimeWarning: divide by zero encountered in log
[0]
  n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: divide by zero encountered in true_divide
  (self.sigma_[i, :]), 1)
/anaconda3/anaconda/lib/python3.5/site-packages/sklearn/naive_bayes.py:395: RuntimeWarning: invalid value encountered in subtract
  (self.sigma_[i, :]), 1)

Update

Code given here


texts:
Hi I am new to Data analytics.I am planning to learn R by doing some real time projects. How should I stream line (set goals) my time in learning R and also I have not learnt statistics till data. I am planning to learn both side by side. I am mid level data warehouse engineer who has experience in DBMS Data-Integration. I am planning to learn R so that I can bring out useful analysis from the Integrated data.

If I be specific, I am beginning in R, so what are the basic statistical concepts I should know and implement it in R. If I want to be an expert or above average person in R how should I plan strategically to become one. Say if I can spend 2 hrs a day for 1 year what level I should reach. FYI am working for a SaaS company. What are the way s in which I can utilize R knowledge in a SaaS environment


texts: Is there a certain threshold over which to accept or reject predictors based on correlation values with the target variable?

texts:
I have been working on the Titanic dataset. After some feature manipulation, I printed out the correlation values between my target variable Survived and all the other predictors. Based on these values, I would like to decide if any variables can be removed from the training set.
Survived  :  1.000000,
Pclass    : -0.338481,
Sex       : -0.543351,
Age       : -0.037059,
Fare      :  0.302002,
Embarked  : -0.167675,
Title     :  0.407753,
IsAlone   :  0.163157

Correlation values of Survived with other predictors.
I am not quite sure of what to do with the Age feature. Maybe try a different kind of feature manipulation on it?


texts: Distance Agnostic classifier using deep learning techniques

texts:
This might be a challenging classification problem.
In this classification, problem distance plays a major role. Here, the Category of the object would change if the distance factor varies. For example, consider objects like Concrete aggregates i.e. 20 mm (millimeter) and 40 mm. 20 mm aggregates might look like 40 mm aggregates when the picture is taken from a close distance and vice versa. In my case, camera calibration is not possible.
So, Looking for an approach using deep learning techniques I should take to develop a distance agnostic classifier whose accuracy will not be compromised for distance variation factor.
Kindly find some sample images on the following links.
20mm concrete aggregate
40mm concrete aggregate
Thanks in advance!


texts: Which one is better performer on wrangling big data, R or Python?

texts: I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and libraries performs the best on wrangling and cleaning Big datasets?


texts: How to improve deep learning model having less data

texts:
I have trained a deep learning model for regression. The accuracy of the model is poor. I am quite new to deep learning. How can I improve it? The target variable Y is obtained by multiplying the features X1 and X2.
DataSet(5800 rows)
X1       |      X2      | Y
1.000000    70.000000   70.000000
0.714286    29.045455   20.746753
0.000000    35.000000   0.000000
0.538462    22.071429   11.884615
0.000000    54.000000   0.000000

Model
#Define a larger model
def larger_model():
  #Create Model
  model = Sequential()
  model.add(Dense(2, input_dim=2, kernel_initializer='normal', activation='relu'))
  model.add(Dense(6, kernel_initializer='normal', activation='relu'))
  model.add(Dense(1, kernel_initializer='normal'))

  #Compile Model
  model.compile(loss='mean_squared_error', optimizer='adam')
  return model

#Evaluate Model

estimator = KerasRegressor(build_fn=larger_model, epochs=10, batch_size=5)
kfold = KFold(n_splits=10)
results = cross_val_score(estimator, X, y, cv=kfold)
print("Results: %.5f (%.5f) MSE" % (results.mean(), results.std()))

Output
Results: -83.81452 (170.38108) MSE


texts:
I would like to use semantic networks to understand changes in texts.
For example when I add/remove some words within a text.
I would like to eat pizza tomorrow evening.
I ate pizza yesterday.

Or just adding a not before a verb, which changes the text meaning:
I would prefer to not eat pizza tomorrow evening.
I didn’t eat pizza yesterday.

I have read about the use of word2vec and neural networks to do this, but I would like if anyone has already tried to do something similar.
Any advice would be greatly appreciated. Thanks


texts: An optimization problem involving perceptron

texts:
What I do not understand is that is there only one input for each weights or there is a vector of inputs from 1 to n?

I also dont get the notation for inputs and outputs? Is that a notation or a power operation?


texts:
I'm currently building my first model with sklearn to predict whether a customer will renew a subscription.  I'm using a random forest because I've heard that they are robust to overfitting.  The confusion matrix generated on the results of my test set (25%) seem to imply that the model is working very well, but the tree I'm generating is massive.  The size of the tree alone, especially considering I'm only using 12 variables, makes it seem like the model is creating too many complex relationships and thus must be overfitting.  If the model is overfitting how can I adjust my code to fix this?  Here is the code I used to tune hyper parameters and fit the model.
pipe = Pipeline([('classifier', RandomForestClassifier())])

param_grid = [
{'classifier' : [RandomForestClassifier()],
'classifier__n_estimators' : list(range(200,500,10)),
'classifier__max_features' : list(range(6,32,5))
}
]

clf = GridSearchCV(pipe, param_grid = param_grid, cv = 5, verbose=True, n_jobs=-1)

predictions = clf.best_estimator_.predict(x_test)
cm = metrics.confusion_matrix(y_test, predictions)
plt.figure(figsize=(9,9))

sns.heatmap(cm, annot=True, fmt=".3f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Values')
plt.xlabel('Predicted Values')
plt.title('Renewal Model Confusion Matrix')

Here is the confusion matrix generated, as well as an image of the tree I'm generating and the code used to generate it.
from sklearn.tree import export_graphviz
import graphviz
import os
os.environ["PATH"] += os.pathsep + 'C:/Users/rich/Anaconda3/envs/32bitpy/Library/bin/graphviz/'

d = export_graphviz(clf.best_estimator_[0][1], out_file='rf.dot',
            rounded = True, proportion = False,
            precision = 2, filled = True)

os.environ["PATH"] += os.pathsep + 'C:/GraphViz/bin'

graph = graphviz.Source(d)
graph


texts:
I am new to python and programming for that matter. I am trying to solve a special type of wave equation using finite element method and carry out simulations in python. I am getting some errors which I do not understand and I need help. Below is the error and subsequently a part of the code.

ValueError                                Traceback (most recent call last)
<ipython-input-138-b33b0055e38e> in <module>
      6     # Finite Element Method
      7     #unew = (dt**2) * Minv @ (f*src[it]  -  K @ u) + 2*u - uold
----> 8     vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v
      9     v = v, vnew
     10     unew = (dt) * vnew

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 1000)

Time extrapolation

for it in range(nt):
    # --------------------------------------
    # Finite Element Method
    #unew = (dt**2) * Minv @ (f*src[it]  -  K @ u) + 2*u - uold
    vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v
    v = v, vnew
    unew = (dt) * vnew
    #uold, u = u, unew
    u = u, unew

# --------------------------------------
# Finite Difference Method
pnew = (dt**2) * Mfd @ ( f/dx*src[it]+ D @ p) + 2*p #- pold
p = p, pnew

# --------------------------------------
# Animation plot. Display both solutions
if not it % iplot:
    for l in line1:
        l.remove()
        del l
    for l in line2:
        l.remove()
        del l
    line1 = plt.plot(x, v, 'k', lw=1.5, label='FEM')
    line2 = plt.plot(x, p, 'r', lw=1.5, label='FDM')
    plt.legend()
    plt.gcf().canvas.draw()


texts: What is the name of this visualization with a circle and internal arcs?

texts:
I came across this visualization where there is a circle with data points on the circle and internal lines (arcs or edges) between the points on the circle. What is the name of this type of visualization?

How would I generate it from a 2-D table like the one below? I want the points A, B, C, D, and E on the circle and the value from point i to j to be an arc between them. Would something like d3.js be able to generate it?


texts: Intuition behind the loss function in Deep Q learning?

texts: I'm currently following a tutorial but I got stuck at the deep Q learning model. According to my understanding of neural networks they predict an approximate function for the inputs given with the help of the loss value, but in the deep Q case, the author of the tutorial said the loss is calculated as Q_target - Q. He told the Q_target is the Q value from the previous iteration. My doubt is if we're using Q value from old iteration to calc lot loss how is the network converging to predict a bellman like equation. Upon asking this doubt he changed his stand, he told that they use the target networks. Each different tutorial I'm trying to refer has a different notion for the Q_target and the loss function. That's why I'm confused. If we're using the "previous iteration's Q value - current Q value" as loss won't our network try to predict the model that predicts values that are closer to the previous iteration's Q values? And moreover for each and every epoch while training for the same state the Q_target will vary each time, so will the network predict a stable model? Can anyone help me by telling me the actual loss function?


texts:
Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ?
Is it allowed ? Or , Should I only fit and transform both on a single set ?


texts: Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

texts:
In this review of categorical encoding, it states early on that

  For regression tasks, Target and LeaveOneOut probably won’t work well

and later repeats that Target/LeaveOneOut (Owen Zhang's approach) encoders are best suited for classification tasks.

I could not seem to find other sources to support this claim, and I was wondering, why wouldn't LeaveOneOut encoding be a valid approach for regression tasks? Is it because it then causes overfitting to certain features? But then, how would this be different from a classification task?


texts:
I am trying to use droplevels.
First I create a subsect of my dataset where the value of attribute Rat must be different from 1 or 5. That value must be a factor. Then, I can see levels are "1", "2", "3", "4" and "5". As I use droplevels() nothing changes. How can I do properly? I wrote this code:

chocolate234 <- subset(chocolate, Rat!="1")
chocolate234 <- subset(chocolate234, Rat!="5")
droplevels(chocolate234$Rat)


texts: Which animal has similar brain complexity to VGGNet

texts:
I know that it sounds weird, but I read somewhere that VGGnet has roughly 14.5k neurons and 138/144M parameters depending on the exact architecture as it's one of the biggest.

Also I read here, that the biggest neural networks trained on super computers have roughly 80M neurons, which is similar to frog's brain.

If we had to compare the complexity of VGGNet to brain of some kind of animal (even simple one) what would it be?


texts: How can I predict the last element of the fixed length=8 sequence after I get each element?

texts:
There are fixed length lists [X1, X2, X3, X4, X5, X6, X7, X8] like this.
I have many lists like them from the past. In the future, I will get new element of current list on weekly bases. one new element for each week.
what machine learning model can I use to predict the last element as soon as I get new element ?


texts: Avoid overfitting in 3 layer neural net for cifar10

texts: I made a neural net with 1 conv and 2 fc layers. I am getting 80-90% accuracy on train set but 60% on train set. I used batchnorm layer too but it didn't remove the overfitting. How can I remove it without increasing the layers?


texts: In Sutton's textbook (Chapter 5) it says "To ensure that well-defined returns are available, here we define Monte Carlo methods only for episodic tasks". Can someone explain what exactly breaks down for non-episodic tasks for Monte Carlo methods in Reinforcement Learning?


texts:
I am trying to build a SVM from scrath and I would like to maximize this Lagrarian expression:

I know what variables means but I would like to know how this maximization is implemeted. Should I start by an alpha close to 0 and increase it until I found support vectors?


texts: How to predict value in every 120 minutes using LSTM in python

texts:
I want to predict value in every 120 minutes continuous using LSTM model. Here I wrote the code for predction. But I'm not getting proper prediction values . Here from start time I need to predict values in every 120 minutes.

model = Sequential()
model.add(LSTM(4, return_sequences=True,input_shape=(x_train_n.shape[2])))
# returns a sequence of vectors of dimension 32
model.add(LSTM(8, return_sequences=True))  # returns a sequence of vectors
of dimension 32
model.add(LSTM(8))  # return a single vector of dimension 32
model.add(Dense(1))
batchsize = 1
model.compile(loss="mean_squared_error",optimizer="adam")
#model.compile(loss='mean_squared_error', optimizer='adadelta',metrics= ['accuracy'])
history = model.fit(x_train_n,y_train_n, batch_size = batchsize,
nb_epoch=30,validation_data=(x_test_n, y_test_n),shuffle =True)\
model.reset_states()
pred1=model.predict(x_test_n)
pred2= model.predict(x_train_n)

#end of the sequences
model.reset_states()

#data=pd.DataFrame(fit1.predict(x_test))

pred1 = scaler_y.inverse_transform(np.array(pred1).reshape ((len(pred1), 1)))
real_test = scaler_y.inverse_transform(np.array(y_test_n).reshape
((len(y_test_n), 1))).astype(int)
real = scaler_y.inverse_transform(np.array(y_train_n).reshape
((len(y_train_n), 1))).astype(int)
pred1 = pred1[:,0]
real_test = real_test[:,0]

#predition in every 120 minutes

sequence_timestep = 120
last_sequence_train = x_train_n[-1]
pred1 = []
def sequence_constructor():
   if len(pred1) >= sequence_timestep:
     new_sequence = pred1[-dimension_seq:]
   else:
      splitter = sequence_timestep - len(pred1)
      part_1 = last_sequence_train[-splitter:]
      new_sequence = np.append(part_1,pred1) #Concatenate 2 list
  new_sequence = np.array(new_sequence)
  return new_sequence
for i in range(1440):
new_sequence = sequence_constructor()
new_prediction = model.predict(new_sequence)
pred1.append(new_prediction)

when I wrote this code error is coming and it's not predicting the values properly.
Here I upload my csv file also, and I wrote what I'm trying to do to predict my value. In my csv file g and p in my two inputs. When the prediction values in every 120 minutes will be the next input of my LSTM .
According to my csv file
10/3/2018 start time = 6:00:00 a.m from that predict the g values in every 120 minutes.
Then again next day start time is again 6:00:00 a.m from that time again new prediction values in every 120 minutes.


texts: correct setting of eval_set in multiclass classification xgboost python , error is “ Check failed: preds.size() == info.labels_.size()”


texts: breaking joined words into meaningful ones during text mining

texts:
I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level.

tidy_complaints <- tidy_complaints %>%
  unnest_tokens(
    output = sentence,
    input = consumer_complaint_narrative,
    token = 'sentences'
  ) %>%
  mutate(sentence_id = 1:n())

However, some of the the complaints contains words that are joined together such as "trailcertified" or "creditorcompany".

Is there any need to break this words up into meaningful words? if yes, how do I accomplish this and still keep the sentence intact?


texts: PCA first dimension do not not capture enough variance

texts: I am doing a PCA as a data exploration step and I realize that the two first principal components capture only 25% of the variance, the ten first principle component capture about 60% of the information, does it worth to interpret those axis knowing that they do not capture enough information


texts:
The Vowpal Wabbit (VW) apparently supports sequence tagging functionality via SEARN. The problem is that I cannot find anywhere detailed parameter list with explanations and with some examples. The best I could find is Zinkov's blog entry with a very short example. The main wiki page barely mentions SEARN.

In the checked out source code I found demo folder with some NER sample data. Unfortunately, the script running all the tests does not show how to run on this data. At least it was informative enough to see what is the expected format: almost the same as standard VW data format, except that entries are separated by blank lines (this is important).

My current understanding is to run the following command:

cat train.txt | vw -c --passes 10 --searn 25 --searn_task sequence \
--searn_passes_per_policy 2 -b 30 -f twpos.vw

where

--searn 25 - the total number of NER labels (?)

--searn_task sequence - sequence tagging task (?)

--searn_passes_per_policy 2 - not clear what it does

Other parameters are standard to VW and need no additional explanation. Perhaps there are more parameters specific to SEARN? What is their importance and impact? How to tune them? Any rules of thumb?

Any pointers to examples will be appreciated.


texts: Doing feature selection based on non-diagonalized covariance matrix vs on eigenvalues?

texts:
Doing feature selection based on non-diagonalized covariance matrix vs on eigenvalues (that equal the diagonal of diagonalized covariance matrix)?

If the values differ in non-diagonalized covariance matrix, then can it be possible that the feature selection done in these two cases offers different outcomes? Is one of them better than another? Do both approaches retain the differences/ratios between variables/values or do they differ even in that sense?


texts:
They asked if the linear assumption is correct, but I just see a graph and based on the data I was taught that a curvature is nonlinear so nonlinear means that a graph is incorrect?

What is the assumption? I know this may seem stupid, but I still haven't grasped the question.


texts: Print the prediction of convolutional neural network

texts:
I have designed a convolutional neural network using tensorflow which looks as follows

#Define a convolutional neural network function
def conv_net(x, weights, biases):
    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
    conv1 = maxpool2d(conv1, k=2)

    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
    conv2 = maxpool2d(conv2, k=2)

    conv3 = conv2d(conv2, weights['wc3'], biases['bc3'])
    conv3 = maxpool2d(conv3, k=2)

    conv4 = conv2d(conv3, weights['wc4'], biases['bc4'])
    conv4 = maxpool2d(conv4, k=2)

    # Fully connected layer
    # Reshape conv2 output to fit fully connected layer input
    fc1 = tf.reshape(conv4, [-1, weights['wd1'].get_shape().as_list()[0]])
    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
    fc1 = tf.nn.relu(fc1)
    # Output, class prediction
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    return out

The code I'm using to train and test the model is as follows :

#Train and Test the Model
with tf.Session() as sess:
    sess.run(init)
    train_loss = []
    test_loss = []
    train_accuracy = []
    test_accuracy = []
    summary_writer = tf.summary.FileWriter('./Output', sess.graph)
    for i in range(training_iters):
        for batch in range(len(X_train)//batch_size):
            batch_x = X_train[batch*batch_size:min((batch+1)*batch_size,len(X_train))]
            batch_y = y_train[batch*batch_size:min((batch+1)*batch_size,len(y_train))]
            # Run optimization op and Calculate batch loss and accuracy
            opt = sess.run(optimizer, feed_dict={x: batch_x,
                                                 y: batch_y})
            loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                              y: batch_y})
        print("Iter " + str(i) + ", Loss= " + \
                      "{:.6f}".format(loss) + ", Training Accuracy= " + \
                      "{:.5f}".format(acc))
        print("Optimization Completed")

        # Calculate accuracy for all test data
        test_acc,valid_loss = sess.run([accuracy,cost], feed_dict={x: X_test,y : y_test})
        train_loss.append(loss)
        test_loss.append(valid_loss)
        train_accuracy.append(acc)
        test_accuracy.append(test_acc)
        print("Testing Accuracy:","{:.5f}".format(test_acc))
        summary_writer.close()

This prints the accuracy but not the predictions. How can I print the predictions?

p.s new to neural network


texts:
Recently I stumbled upon a R package that works similar like caret, but I can't remember what it's name was and I can't find it. It seemed to be less well known, but at least as extensive. Someone able to help?

Edit: I search for this other package to widen my horizon. There is no specific other reason. This question is specific however, since I am not looking for a general recommendation, but I am looking for this specific package I have in mind. I will recognize the package once I read its functions and descriptions. I had searched for a couple of hours myself and did not find it by searching for "package similar to caret". That is why I need help.


texts: Linear Regression with Category variables

texts:
I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance understanding how to go about it.
Specifically, I'm now learning about linear regression with categorical variables, and I understand the gist of it: We just encode the categorical variable to some sort of numerical representation (like one-hot encoding) and put it in the model. Great.
While there are many guides on how to do various encoding methods etc. online, I haven't really found a resource that explains the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?
And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)
I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.
Sorry if this is too many questions in one post, I need some guidance on the concept. All the online resources are telling me how to implement the model, but not when and why to use it.


texts: How standardizing and/or log transformation affect prediction result in machine learning models

texts:
I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data.

I have some general questions about how standardizing and/or log transform predictors affect prediction result in machine learning.

Standardizing (center&scale):  I understand many models require standardizing predictors. For those models that do NOT require standardization, theoretically how does standardization affect the prediction result? I know it changes model interpretation. But my primary goal here is prediction. If it does not matter, then can I always scale my predictor before modeling, at least making training data more consistent when exploring many different models?
log transformation: When predictors are skewed, we often do log transformation to make them normal. How do log-transformation affect the prediction result? For models like random forest, I thought it should not matter. But I did see some difference in prediction performance to my prior experience. Again, prediction is the goal.  in addition, even for simple linear regression, the assumption is residual is normally distributed. It does not require normality of predictors. Do log-transform (normalizing) predictors help make residual normal?
standardization on log-transformed predictors:  For models like elastic net, when predictors are skewed, should I do standardization on log-transformed predictors? As I mentioned earlier, I did see some difference in prediction performance.

I think these are some pretty common data preprocess questions. Some online sources are pretty confusing to me. I look forward to some experts to clarify these importance questions. I believe a clear answer will benefit a lot of people in this community. Thanks a lot.


texts: Metric to evaluate words generated by Neural Network

texts:
I have this task at hand and I would be grateful for some directions. Perfectly not the final solution as I would like to do it myself.

Let's say I need to create new fruit names based on existing ones.

I made it using textgenrnn library.

Now I have a list of potential new names.
I would like to build a metric which would calculate that e.g.:

(Fabic, Alis, Brooty, Morange etc) are potentially good fruit name
(Ssae, Sriew, Adeoie, Seeea) are potential bad fruit names.

Is there any list of metrics for word generating?

As of now, I found following resources:

Evaluating Generative Models for Text Generation https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2737434.pdf
Character-Level Language Modeling with Hierarchical Recurrent Neural
Networks http://150.162.46.34:8080/icassp2017/pdfs/0005720.pdf
Graves A. Generating Sequences With Recurrent Neural Networks
https://arxiv.org/pdf/1308.0850.pdf
Subword language modeling with neural networks.
http://www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf

But since I learnt about them today, I would need a more vanilla introduction to this topic of validation.


texts:
I use deepAR RNN on AWS via python. I run the estimator.fit(inputs=data_channels) and want to get insight on the algorithm performance and the fit that the model obtained, i.e. epoch to epoch convergence, quantile loss, RMSE, etc..

Instead I get a log dump like this:

 [05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.5]): 0.0768988
[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, wQuantileLoss[0.9]): 0.037788
[05/10/2018 09:44:02 INFO 140424081053504] #test_score (algo-1, RMSE): 4433.98308522
#metrics {"Metrics": {"totaltime": {"count": 1, "max": 125569.84615325928, "sum": 125569.84615325928, "min": 125569.84615325928}, "setuptime": {"count": 1, "max": 14.33110237121582, "sum": 14.33110237121582, "min": 14.33110237121582}}, "EndTime": 1525945442.991945, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1525945442.891469}

which is completely unreadable and unprocessable.

Is there a way to get those metrics back to python?

PS. The folder in output_path of estimator contains some files, but they do not seem to have the results I am looking for

I managed to handle this way, yet horribly inefficient:

First I capture the console output with this trick:

f = io.StringIO()
with redirect_stdout(f):
    estimator.fit(inputs=data_channels)
dr_log = f.getvalue()
Then I manually process the long multiline string (blaaah:/)

def dr_metrics(log, r):
rr = list()
final_res = dict()
for line in log:
    if "Epoch" in line:
        l = line.split("]")
        print(l)

    e = l[-3].split("[")[-1]
    b = l[-2].split("[")[-1]
    val = l[-1][:-2].split("=")[1]
    rr.append([e, b, val, "Speed" in line])
    print([e, b, val])
if "Final loss" in line:
    final_res['final_loss'] = line.split(":")[-1].split(" ")[1]
if "#quality_metric" in line:
    if "mean_wQuantileLoss" in line:
        final_res['mean_wQuantileLoss'] = line.split("=")[-1]
    if "RMSE" in line:
        final_res['RMSE'] = line.split("=")[-1]

res_df = pd.DataFrame(rr, columns=["Epoch", "Batch", "AvgLoss", "EpochEnd"])
res_df.head(10)
print(final_res)


texts: Plot 2D function: do I need to pass all the values to the function?

texts:
I'm new with Octave and I need to plot this function:

% Computes the value of the Powell Sum benchmark function.
% SCORES = POWELLSUMFCN(X) computes the value of the Powell Sum function at
% point X. POWELLSUMFCN accepts a matrix of size M-by-N and returns a vetor
% SCORES of size M-by-1 in which each row contains the
% function value for the corresponding row of X.
%
% Author: Mazhar Ansari Ardeh
% Please forward any comments or bug reports to mazhar.ansari.ardeh at
% Google's e-mail service or feel free to kindly modify the repository.
function scores = powellsumfcn(x)
    n = size(x, 2);
    absx = abs(x);

    scores = 0;
    for i = 1:n
        scores = scores + (absx(:, i) .^ (i + 1));
    end
end

I have copied from BenchmarkFcns.

I have no idea about how to plot it.

Do I need to pass all the possible values to the function plot it?


texts: How to present the final model (e.g. random forest)?

texts:
I've run random forest on my dataset (imbalanced, binary target class) and used cross validation to tune the parameter and use recursive feature elimination with cross-validation to get the subset of features.

Then I can present the optimal parameter setting as well as the set of features for model fitting.

But how about the probabilistic threshold?

How can I conclude a threshold as the important part of the final model?

Also using CV?

If this threshold is not stable among folds of CV, what to do next?