texts: Existing library or code for - SVM as classifier in the last layer of neural network/deeplearning

texts:
I have read many articles which replaced cross-entropy or softmax layer with SVM classifier in neural network. Their results suggested SVM-NN (SVM as classifier) outperformed softmax, cross-entropy.
(One such paper - http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf)
I wish to train and test SVM-NN on my datasets. I have two questions

Is there any existing library/code available in R or python
Can some advice on how to implement this. Later I can make it public for others also

Thanks


texts: How much data would be remain unaffected?

texts: our data set has missing values. Further examination tells you that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected (Tell us the %). Why?✱


texts:
I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop).

I'm looking for a way to build the model on any computer and send it up to a server for training and have it return the graphs/accuracies/weights etc. I know there are paid solutions for this but I'm looking for a distributed solution I can run myself.

I have a server set up at home which is about to get a CPU and GPU upgrade. I'd like to be able to set it up so that when I'm working on the LAN, or when I'm working remotely on my laptop, I can send code to the server and have it train the model and return to me the results (or save the results if the sender machine is switched off)

Are there any existing solutions to accomplish something like this? I'm not tied to any specific library, but would prefer to stick with Python if possible


texts: Is there a way to replace existing values with NaN

texts:
I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.

I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is here.

Thanks in advance for the help.


texts: How the combination of cross entropy loss and gradient descent penalizes and rewards

texts:
For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is:

$$L = -\sum_{i=1}^C y_ilog(p_i)$$

where $p_i$ is the classifier prediction score and $y_i$ is the ground-truth labels. To update the parameters, gradient descent is used to make the distribution of $p_i$ as similar as possible to the distribution of $y_i$.
Let's say for an arbitrary point in space ($x_i$) we have the ground truth  label $y_i = [1, 0, 0]$ which means for a $C=3$ class problem, $x_i$ belongs to the class 0. And let's say the softmax classifier has a prediction of $p_i = [0.3, 0.6, 0.1]$ in the first iteration. When optimizing $L$ with gradient descent, the prediction changes like this:

[iteration 0] ....... $p_i = [0.3, 0.6, 0.1]$
[iteration 1] ....... $p_i = [0.4, 0.5, 0.08]$
[iteration 2] ....... $p_i = [0.55, 0.3, 0.02]$
...........................................................
[iteration n] ....... $p_i = [0.98, 0.001, 0.0001]$

So it seems like gradient descent is trying to increase the value for index 0 and decrease value for the other indices.
Based on my understanding, cross-entropy is only evaluated on the correct class, so the other terms will be zeroed out because of the ground truth array $y_i = [1, 0, 0]$:

$L = -\sum_{i=1}^C y_ilog(p_i) = - 1 * log(0.3) = 1.2$

I don't understand the mechanism here. Is it the gradient descent doing these modifications or is it the cross-entropy loss we define or is it the combination of both? How does the prediction score for the correct class increases and the prediction score for the incorrect classes decrease while optimizing?


texts: Real-time or offline Data Augmentation for segmenting microscopy images?

texts:
I'm doing semantic segmentation(for cells) using microscopy images. I'm exploring U-net and FCN DenseNets for the task. In the U-net paper the authors have trained their model only from 30 images but have used data augmentation extensively specially elastic deformations. I too have very little data, 15-20 annotated images. Thus I too plan to use data augmentation. I'm also using elastic deformations.

Other techniques I'm using:

Flipping, Rotation, Translation, Shearing
Random crops, zoom-out, stretching
Image Contrast and Gaussian noise

This will increase my dataset 10-20 folds.

Q.1 My question is shall I use offline augmentation or on-the-fly(Real time) augmentation?

From what I've read so far: use on-the-fly augmentation if dataset if large so to not make the dataset explode in size. I know this is very generalised. I've also seen this on Kaggle: For online data augmentation, the model see one random generated sample only once hence generalises better.

Q.2 Also I'm using so many data augmentation techniques will that be even useful as the data set so small, or shall I restrict myself to only some of these techniques?

Any help is highly appreciated.


texts: CartPole v1 - Simple backprop with 1 hidden layer

texts:
I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to get the average reward to go up beyond about 42 steps per episode. Could anyone help? I've tried looking for a similar solution online but none of them seem to fit the approach that I'm following. Almost all of the solutions involve learning at the end of each episode (by storing SARS' data). Is my approach even correct - as in, is it even possible for the agent to learn the optimal solution if I'm updating the Q-values every time-step, instead of batch updates every episode? Seems like theoretically it should be possible.

Details: After playing around and experimenting with activation functions, stochastic policies and finally settling on a deterministic policy with linear activation function and the parameters mentioned below - i'm able to get my agent to consistently converge (in about 100-300 steps) to an average reward of about 42 steps. But it doesn't go beyond 45. Adjusting the parameters (epsilon, discount_rate, and learning rate) in the program below does not have a huge impact on this.

Here's some snippets from my code below. Basically it consists of 4 features -> 6 hidden layer nodes -> 2 outputs (one for each action).

First, the hyperparameters:

epsilon = 0.5
lr = 0.05
discount_rate=0.9

# number of features in environment observations
num_inputs = 4
hidden_layer_nodes = 6
num_outputs = 2

The q function:

def calculateNNOutput(observation, m1, m2):
    scaled_observation = scaleFeatures(observation)
    hidden_layer = np.dot(scaled_observation, m1)           # 1x4 X 4x6 -> 1x6
    outputs = np.dot(hidden_layer, m2) # 1x6 X 6x2
    return np.asmatrix(outputs) # 1x2

Action selection (policy):

def selectAction(observation):
    #explore
    global epsilon
    if random.uniform(0,1) < epsilon:
        return random.randint(0,1)
    #exploit
    outputs = calculateNNOutputs(observation)
    print(outputs)
    if (outputs[0,0] > outputs[0,1]):
        return 0
    else:
        return 1

Backprop:

def backProp(prev_obs, m1, m2, experimental_values):
    global lr
    scaled_observation = np.asmatrix(scaleFeatures(prev_obs))
    hidden_layer = np.asmatrix(np.dot(scaled_observation, m1))      #
    outputs = np.asmatrix(np.dot(hidden_layer, m2)) # 1x6 X 6x2
    delta_out = np.asmatrix((outputs-experimental_values)) # 1x2
    delta_2=np.transpose(np.dot(m2,np.transpose(delta_out))) # 6x2 X 2x1 = 6x1_T = 1x6
    GRADIENT_2 = (np.transpose(hidden_layer))*delta_out # 6x1 X 1x2 = 6x2 - same as w2
    GRADIENT_1 = np.multiply(np.transpose(scaled_observation), delta_2) # 4 x 6 - same as w1

    m1 = m1 - lr*GRADIENT_1
    m2 = m2 - lr*GRADIENT_2
    return m1, m2

Q-learning:

def updateWeights(prev_obs, action, obs, reward, done):
    global weights_1, weights_2
    calculated_value = calculateNNOutputs(prev_obs)
    if done:
        experimental_value = -1
    else:
        actionValues = calculateNNOutputs(obs) # 1x2
        experimental_value = reward +  discount_rate*(np.amax(actionValues, axis = 1)[0,0])
    if action==0:
        weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[experimental_value, calculated_value[0,1]]]))
    else:
        weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[calculated_value[0,0],experimental_value]]))

Do I need to make any changes in approach/implementation/parameter tuning?


texts: How to manufacture exploding gradients in a neural networks

texts:
it might sound silly but i have this assignment problem where i have to show how exploding gradients occur in neural nets and i need to create a neural netowork which shows this phenomenon. the question is what are the parameters i can tweak to induce this problem deliberately?
currently my neural network has
activation_functions = ['linear','linear', 'tanh', 'ReLU', 'sigmoid']
and neurons = [3, 5, 10, 10, 5, 1] where 3 is input size and 1 is the output size.


texts: Calculate accuracy of crowdsourced responses in realtime

texts:
I have a total dataset of, lets say, 1000 Items. Each item will get a response from the crowd.

After I get all the 1000 responses, I can sample the dataset to calculate accuracy. Sampling would involve checking the responses for items and comparing it with our own knowledge.

Eg: If the task was to tag images as T-Shirts and Jeans. Out of 1000 responses, I can sample a few and open the images and see what they are, and then compare them with the response that I got from the crowd.

But how can I calculate the accuracy of the total dataset in real-time, as the responses are being submitted?


texts: Unsupervised feature reduction for anomaly detection with autoencoders

texts:
I am collecting a big number of generated numeric features for the task of unsupervised anomaly detection.

I can assume that all training data is considered normal.

I expect some of the generated features to be characterised with low standard deviation, for example, some features might be always 0 in the train examples. In contrast, I expect that some of these features will deviate in anomaly instances.

As I have a lot of feature, I strive to perform feature reduction / selection. However, using simple feature selection methods, will completely remove the non deviating features, affecting the upcoming detection for the worse.

I was thinking about using stacked auto-encoders for the sake of feature reduction, so that whenever a feature deviates a lot from the stdv it will affect all resulted features - causing a noticeable anomaly.

Will this technique work? if not, why? and what other technique could work for that.

Also, if it does, and I am planning of using deep auto-encoders for the sake of anomaly detection as well, is the first step of feature reduction redundent?


texts:
Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

Without using any word embeddings for now the best way to go is to use a TF-IDF model.

However, as I am thinking this more carefully I am not entirely surely if this is the best way to go in comparison with a simply TF model.

Specifically, a TF-IDF model will take the inverse document frequency of terms irrespectively to any labels/categories.

So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).

On the other hand, with a simply TF then the word 'positive' would have a very high value and it would be evident that it is directly related to positive reviews.

Why the TF-IDF is necessarily the best way to go in cases like these?


texts: How reliable are model performance reportings?

texts:
I had a conceptual doubt about estimating and reporting a classification model's performance. Say my model works with range of depth values and gives out different readings of test errors. We choose the model with depth having lowest test error rate as M1

Now if we want to report our model's performance on a hidden test set, would it wise to say that M1 would perform equally well on this new test set and with the same test error rate?


texts: Keras high loss and high accuracy in gk bot with reinforcement learning?

texts:
I'm making goal-keeper bot in haxball game. It worked well when i trained less but i worked worse when i trained more.

Last reinforcement state: 5160 episode - 4171281 steps - 0.05 epsilon:

Last fit result= acc: 0.9905, loss: 293408940460887.5000

(previous fit result like: acc: 0.7, loss: 0.012)

Last game result image: https://i.ibb.co/NmLL5b7/hax2.gif

This Keras model:

        NBACTIONS = 3
        WINDOW_WIDTH = 630
        WINDOW_HEIGHT = 400
        IMGHEIGHT = int(WINDOW_HEIGHT/5)
        IMGWIDTH = int(WINDOW_WIDTH/5)
        IMGHISTORY = 4

        model = Sequential()
        model.add(Conv2D(32, kernel_size=4, strides = (2,2), input_shape = (DCQL_Global.IMGHEIGHT,DCQL_Global.IMGWIDTH,DCQL_Global.IMGHISTORY),padding = "same"))
        model.add(Activation("relu"))
        model.add(MaxPool2D(pool_size=(2, 2)))
        model.add(Conv2D(64,kernel_size=4,strides=(2,2),padding="same"))
        model.add(Activation("relu"))
        model.add(MaxPool2D(pool_size=(2, 2)))
        model.add(BatchNormalization())
        model.add(Conv2D(64,kernel_size=3,strides=(1,1),padding="same"))
        model.add(Activation("relu"))
        model.add(MaxPool2D(pool_size=(2, 2)))
        model.add(Flatten())
        model.add(Dense(512))
        model.add(Activation("relu"))
        model.add(Dense(units= DCQL_Global.NBACTIONS, activation="linear"))
        model.compile(loss = "mse", optimizer=Adam(lr=0.0001), metrics=['accuracy'])

Reward rules= goal: -1, save(not missing goal): +1, after kicking the ball if gk follow the ball correctly +0.01 else gk follow the ball incorrectly -0.01 reward.

I'm preprocessing the image like the following link:

https://i.ibb.co/yPdrJKJ/hax.gif

Preprocessing code:

def ProcessGameImage(I):
    GreyImage = skimage.color.rgb2gray(I)
    CroppedImage = GreyImage
    ReducedImage = skimage.transform.resize(CroppedImage,(DCQL_Global.IMGHEIGHT,DCQL_Global.IMGWIDTH))
    ReducedImage = skimage.exposure.rescale_intensity(ReducedImage, out_range = (0,255))
    ReducedImage = ReducedImage / 255.0

    return ReducedImage.astype(np.float)

What's my fault?


texts: Python strategies for handing categorical variables

texts:
I am currently working on a binary classification task where the class is imbalanced.

I have the following categorical attributes with different levels:

time_slot: 8 levels
product_type: 3 levels
state: 40 levels
due_day: 6 levels (Mon - Sat)
lead_time: numerical in days (0-100)

Now, I am planning to use three algorithms to start with:

Logistic Regression, Decision Tree and Random Forest

I am confused as to what sort of encoding strategy is best when it comes to categorical variables?

LabelEncoder, OneHot, BinaryEncoding?

Also, I am thinking of creating bins for lead_time

any pointers/tips will be useful.


texts: Deriving Confidences from Distribution of Class Probabilities for a Prediction

texts:
I run into this problem from time to time and have always felt like there should be an obvious answer.

I have probabilities for potential classes (from some classifier).  I will offer the prediction of the class with the highest probability, however, I would also like to attach a confidence for that prediction.

Example: If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1: 50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1 should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}.

Reporting that I predict class C1 with 60% probability isn't the whole story.  I should be able to derive a confidence from the distribution of probabilities as well.  I am certain there is a known method for solving this but I do not know what it is.

EDIT: Taking this to the extreme for clarification:
If I had a class C1 with 100% probability (and assuming the classifier had an accurate representation of each class) then I would be extremely confident that C1 was the correct classification.  On the other hand if all 5 classes had almost equal probability (Say they are all roughly 20%) than I would be very uncertain claiming that any one was the correct classification. These two extreme cases are more obvious, the challenge is derive a confidence for intermediate examples like the one above.

Any suggestions or references would be of great help.

Thanks in advance.


texts:
I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.

I am writing a custom transformer that will take the dataframe column Company and remove stray commas:

from pyspark.sql.functions import *

class DFCommaDropper(Transformer):

    def__init__(self, *args, **kwargs):
        self.name = CommaDropper

    def transform(self,df):
        df = df.withColumn('Company', regexp_replace('Company',',','')
        return df

The above code is obviously wrong. I'm unsure what/how to initialize this and then how to use the initialized class instance in the transform function.

Thanks in advance for your help.


texts: Derivation of the cross-entropy equation in Michael Nielsen's book

texts:
I am reading the book http://neuralnetworksanddeeplearning.com/chap3.html by Michael Nielsen.
So this is a question mostly for the people familiar with the book and understanding the material.

In the equations (71-75) we are trying to find a cost-function $C$ satisfying:

$$\frac{\partial{C}}{\partial{\omega_j}}=x_j (a-y),$$
$$\frac{\partial{C}}{\partial{b}}=(a-y),$$

where $w_j$ and $b$ are weights and bias of a neuron, $a$ is the output of the sigmoid function

$$a = \sigma(\sum{\omega_{j} x_j + b)}=\sigma(z).$$

We apply the chain rule (equation 73):

$$\frac{\partial{C}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\frac{\partial{a}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\sigma\prime(z).$$

And in the next line the author writes

Using $\sigma\prime(z)=\sigma(z)(1-\sigma(z))=a(1-a)$ the last equation becomes...

Where does this expression come from? $\sigma(z)(1-\sigma(z))$


texts: Graph & Network Mining: clustering/community detection/ classification

texts:
I am working on graphs/networks where nodes and edges have some attributes.

I want to know what algorithm exist for:

1) clustering a graph to k groups: depend only on the structure (edge attribute only)

2) Community Detection: ( same as graph clustering) but the number of communities is unknown.

3) Classification: a supervised method where I have labels and I want to classify the nodes based on their attributes and their connections (edges).

4) Page Rank: detecting the most important nodes in a group (community, cluster) based on their connection

thank you very much.


texts: Prioritized Replay, what does Importance Sampling really do?

texts:
I can't understand the purpose of importance-sampling weights (IS) in Prioritized Replay (page 5).
A transition is more likely to be sampled from experience replay the larger its "cost" is.  My understanding is that 'IS' helps with smoothely abandoning the use of prioritized replay after we've trained for long enough.  But what do we use instead, uniform sampling?
I guess I can't realize how each component in such a coefficient is affecting the outcome. Could someone explain it in words?
$$w_i = \left( \frac{1}{N}\cdot \frac{1}{P(i)} \right) ^\beta$$
It's then used to dampen the gradient, which we try to get from transitions.
Where:

$w_i$ is "IS"
N is the size of Experience Replay buffer
P(i) is the chance to select transition $i$, depending on "how fat its cost is".
$\beta$ starts from 0.4 and is dragged closer and closer to 1 with each new epoch.

Is my understanding of these parameters also correct?
Edit Sometime after the answer was accepted I found an additional source, a video which might be helpful for beginners - MC Simmulations: 3.5 Importance Sampling

Edit As @avejidah said in the comment to his answer "$1/N$ is used to average the samples by the probability that they will be sampled".
To realise why it's important, assume $\beta$ is fixed to 1, we have 4 samples, each has $P(i)$ as follows:
0.1  0.2   0.3     0.4

That is, first entry has 10% of being chosen, second is 20% etc.
Now, inverting them, we get:
 10   5    3.333   2.5

Averaging via $1/N$ (which in our case is $1/4$) we get:
2.5  1.25  0.8325  0.625     ...which would add up to '5.21'

As we can see they are much closer to zero than the simply inverted versions ($10, 5, 3.333, 2.5$). This means the gradient for our network won't be magnified as much, resulting in a lot less variance as we train our network.
So, without this $\frac{1}{N}$were we lucky to select the least likely sample ($0.1$), the gradient would be scaled 10 times.  It would be even worse with smaller values, say $0.00001$ chance, if our experience replay has many thousands entries, which is quite usual.
In other words, $\frac{1}{N}$ is just to make your hyperparameters (such as learning-rate) not require adjustment, when there you change the size of your experience replay buffer.


texts: Finding cause and effect correlation in a dataset

texts:
What methods might one use in order to find and analyze cause-and-effect in a dataset? An example might be something like plotting this:
import matplotlib.pyplot as plt
import numpy as np

n1 = np.array([100, 100, 150, 150, 150, 100, 100, 100, 100])
n2 = np.array([100, 100, 100, 100, 120, 120, 120, 100, 100])

plt.plot(n1, label="feature1")
plt.plot(n2, label="feature2")
plt.legend()
plt.show()

Where we see that some causal happens at feature 1, and a number of iterations later, feature2 experiences some sort of effect. The scales may be different, the iterations may bee slightly different, but looking at it on a graph like this, it appears that they are correlated by some sort of cause-and-effect.
Is there any way that one would be able to programmatically identify something like this? Or maybe give it a score of how correlated through cause-and-effect they might be?


texts: How to compare a sentence with a paragraph and get its probability in terms of correctness?

texts:
This is my first post on stackoverflow network and I am dealing with a machine learning.

Lets say I have a paragraph describing a rabbit and tortoise story. The story concludes that tortoise is a winner.

Now I want to train ML engine based on this paragraph. If I provide a statement as "Rabbit won the race", how can it show me the correctness of my statement in terms of probability of correctness?

Need help.


texts:
I have a problem with R's performance:
here is my Script.

The problem is that I need to use it in a base with  ~6m of clients  (with one linear model per customer) and it's taking to long to process.

Can anyone help me to improve the performance of my script? I think that the problem is in the data.frame rbind function.


texts: Similarity of Dictionaries Using Cosine Similarity

texts:
I'm trying to use Cosine Similarity in Python to compare users, but can't seem to get it quite right.

from math import*

ratings={'Mr. Perfect': {1: 1, 2: 2, 3: 1, 4: 3, 5: 3, 6: 1, 7: 3, 8: 1, 9: 2},
         'The Progressive': {1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3, 8: 3, 9: 3},
         'Mr. Villain': {1: 3, 2: 3, 3: 3, 4: 3, 5: 3, 6: 3, 7: 3, 8: 3, 9: 3}}

def square_rooted(x):

    return round(sqrt(sum([a*a for a in x])),3)

def cosine_similarity(x,y):

    input1 = {}
    input2 = {}
    vector2 = []
    vector1 =[]

    if len(x) > len(y):
        input1 = x
        input2 = y
    else:
        input1 = y
        input2 = x

    vector1 = list(input1.values())

    for k in input1.keys():    # Normalizing input vectors.
        if k in input2:
            vector2.append(float(input1[k]))
        else :
            vector2.append(float(0))

    numerator = sum(a*b for a,b in zip(vector2,vector1))
    denominator = square_rooted(vector1)*square_rooted(vector2)
    return round(numerator/float(denominator),3)

print("Similarity between Perfect and Progressive")
print (cosine_similarity(ratings['Mr. Perfect'],ratings['The Progressive']))

print("Similarity between Villain and Progressive")
print (cosine_similarity(ratings['Mr. Villain'],ratings['The Progressive']))

Basically, found the formula here https://stackoverflow.com/questions/35980910/return-similar-score-based-on-two-dictionaries-similarity-in-python but I keep getting results of 1. Any ideas would be helpful!


texts: How to reshape ndarray to calculate the mean squared error?

texts:
y_train.shape = (7654,1)
y_test.shape = (1914,1)

mae = mean_absolute_error(y_train, y_test) gives:
"ValueError: Found input variables with inconsistent numbers of samples: [7654, 1914]"

So I am trying to reshape y_train to (1,7654) and y_test to (1,1914). Would that work? I tried y_train = y_train.reshape(X.shape[1:]) but that doesn't work.


texts: Why softmax in YouTube’s DNN recommender

texts:
I am confused about the softmax layer of YouTube’s DNN candidate generation. A user may interact with many videos. Softmax is assuming classes are exclusive. For example, logits = [[4.0, 4.0, 1.0]], labels = [[1.0, 1.0, 0.0]], the sigmoid cross entropy loss is 0.45 while softmax cross entropy loss is 1.43.
Is it because in the candidate generation stage, the relative order of items does not matter?


texts: How to compare the topic coherence between models of different number of topics?

texts:
If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence measures because of the summation over all the topics. Is there any flaw in dividing this coherence calculation by the number of topics to get an average for a model of 5 topics and then for a model of 10 topics and comparing the two?

In gensim specifically, I'm using c_v coherence calculation.


texts:
I have a ranked list of rows of 100 lines of data

1-    8  4  0  5  9  3

2-    0  3  3  5  3  2

3-    0  0  2  4  0  2

..

100-  0  2  3  2  2  0

Is it possible to predict

a) when given a new sequence where it would fit?

b) a way to generate a row that would fit into the top 1, top 5, top 10


texts: Unable to use the Python Data Frame method "iloc" on a Data Frame created in pyspark's SQLContext

texts:
After loading a dataset as DataFrame in pyspark's SQLContext, unable to use the Python DataFrame property of 'iloc' on it. Does a DataFrame created in SQLContext of pyspark behave differently and expect different methods than the python DataFrame ?

Error : AttributeError: 'DataFrame' object has no attribute 'iloc'

Full Code: Just loading a data_set and using a custom function to split the datset in to two sets named test_set and train_set

from pyspark.sql import SQLContext
import numpy as np
import pandas as pd

data_set = sqlContext.read.format('csv').options(header='true', inferSchema='true').load('/FileStore/tables/69qouxnt1504605761640/AirQualityUCI.csv')

# Split train and test sets
def split_train_test(data, test_ratio):
    length = data.count()
    shuffled_indices = np.random.permutation(length)
    test_set_size = int(length * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]
    return data.iloc[train_indices], data.iloc[test_indices]

train_set, test_set = split_train_test(data_set, 0.2)


texts: Why a Random Reward in One-step Dynamics MDP?

texts:
I am reading the 2018 book by Sutton & Barto on Reinforcement Learning and I am wondering the benefit of defining the one-step dynamics of an MDP as
$$
  p(s',r|s,a) = Pr(S_{t+1},R_{t+1}|S_t=s, A_t=a)
$$
where $S_t$ is the state and $A_t$ the action at time $t$. $R_t$ is the reward.

This formulation would be useful if we were to allow different rewards when transitioning from $s$ to $s'$ by taking an action $a$, but this does not make sense. I am used to the definition based on $p(s'|s,a)$ and $r(s,a,s')$, which of course can be derived from the one-step dynamics above.

Clearly, I am missing something. Any enlightenment would be really helpful. Thx!


texts: Do word embeddings help with out of vocab tokens?

texts:
I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load existing weights from Glove but am still facing a few problems. The main one being that there are certain "negative" words I know of but that do not appear in the vocab. Because of this, when i try examples with word that do NOT appear in the vocab (like the word "rubbish") the network does not know that this contains a negative sentiment.

Is there a way to use Word2Vec / Glove / etc to pass in the word rubbish, find the similarity to the word garbage, and then pass that known word into the network instead? And if so is that handled by the "Embedding" layer or is it an additional step I need to perform during pre-processing?

Additionally, how can I handle misspelled words? For instance, how can i associate "rubbbbbish" with "rubbish"?

I am new to text classification and would really appreciate a bit of guidance!


texts: Different Time series Modeling techniques?

texts: I started learning Time series modeling and bit confused when I started learning the advanced techniques like Box-Jenkins and Holts Winter techniques. What is the difference between ARIMA, Box-Jenkins and Holt's Winter Models.


texts: How to impute missing value in Test Set using a custom Imputer created on training dataset

texts:
I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique.
Under this technique, I replaced missing values with the mean value of the two categories of the target feature. The code snippet is as below:
dataframe['Feature'] = dataframe['Feature'].fillna(dataframe.groupby('Target Feature')['Feature'].transform('mean'))

Using this strategy I have designed classification models based on Logistic Regression and Support Vector Classifier. Now I have to run my models on a test set but am stuck at the pre-processing stage. The test set also has missing values in the same feature (as in the training set), now how can I update these missing values with the mean values that the models learnt from the training set.


texts: How to extract data from an API every hour in Python?

texts:
I am new to Python and I tried writing a script that extracts air quality json data from an API every hour and logs it into a same excel file. My code doesn't return anything.
Is my code correct ? and how can I make it log into the excel file every hour please ? Thank you very much.

Here is the script:

def write_to_excel():
    request = requests.get("https://api.waqi.info/feed/paris/?
     token=?")
    request_text = request.text
    JSON = json.loads(request_text)

    filterJSON = {
        'time': str(JSON['data']['time']['s']),
        'co': str(JSON['data']['iaqi']['co']['v']),
        'h': str(JSON['data']['iaqi']['h']['v']),
        'no2': str(JSON['data']['iaqi']['no2']['v']),
        'o3': str(JSON['data']['iaqi']['o3']['v']),
        'p': str(JSON['data']['iaqi']['p']['v']),
        'pm10': str(JSON['data']['iaqi']['pm10']['v']),
        'pm25': str(JSON['data']['iaqi']['pm25']['v']),
        'so2': str(JSON['data']['iaqi']['so2']['v']),
        't': str(JSON['data']['iaqi']['t']['v']),
        'w': str(JSON['data']['iaqi']['w']['v']),
    }

    liste.append(filterJSON)
    try:
        os.remove("airquality.xlsx")
    except:
        pass

    pd.DataFrame(liste).to_excel('airquality.xlsx')
    print(liste)

if __name__ == "__main__":
     schedule.every(3).seconds.do(write_to_excel)
     while True:
         schedule.run_pending()

'''


texts:
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:

Scheme 1

Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss)words from each article (for the sake of simplicity let us assume batch size = m)
Set the initial hidden state H0 = $[0,0,..,0]$
Calculate loss and gradient on this batch and update the parameters
Move a word forward and get next S words from each article (hence S2, S3, ... , Ss are the same words as in the previous batch)
Use H1 calculated from previous step and use it as H0 in this iteration
Do this to the end

*In this scheme we would have to use zero padding on the last batch (at the end of articles)

Scheme 2
Same as Scheme 1 but in step 5 we reinitialize H0 to a vector of Zeroes

Scheme 3
Same as scheme 1 but in step 4 we move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration

Scheme 4, 5, 6
Same as Scheme 1, 2, 3 but instead of taking s consecutive words we take first sentence from each article and zero pad them to the length S

What is the right way to go through the data in feeding it to a RNN, Please give reasons as well. I think it should be Scheme 1 but it could be really slow


texts:
I am working on a clustering problem. I am not able to find the right similarity metric for my system.

I have n nodes with ordered vector (eg: [1,0,0,0,1,0,.....] "1" represents represent presence of ith object and "0" represents absence of the object). I want to cluster them.


texts: If we change the $ywx<0$ condition (for performing update) to $ywx<1$ like in SVM (but without adding regularization to maximize the margin), is there any difference from the basic perceptron (the one with the aforementioned $ywx<0$ condition)?


texts:
I have 5-year sales information from a grocery store in Canada. I would like to check the effect of an online campaign that happened in 2017 on food sales. My treatment group is people living in urban areas.
My first question is whether this would be considered a quasi-experiment or not? I ask this because I did not cause the intervention and it is caused by others. Besides, should I use the DiD model or RD? Which one is the best choice? I have already done the analysis by difference-in-difference but I am uncertain if regression discontinuity will work better. Should I do any pre-analysis steps in R for identification strategy?


texts: Convolutional neural network fast fourier transform

texts:
I've read that some convolution implementations use FFT to calculate the output feature/activation maps and I'm wondering how they're related. I'm familiar with applying CNNs, and (mildly) familiar with the use of FFT in signal processing, but I'm not sure how the 2 work together

When I think of convolutions, I imagine taking a kernel, flipping it, multiplying (and adding) the elements of the kernel with the overlapping input, shifting the kernel and repeating the process. How does a FFT fit into this process?


texts: Python: Why does the OutputCodeClassifier provide more estimators than there are classes?

texts:
Why is len(clf) = 6 even though there are just 4 classes?

Data is stored in the following format:

class1 val(1,1), val(1,2)
class2 val(2,1), val(2,2)
class2 val(3,1), val(3,2)
class2 val(4,1), val(4,2)
(...)

clf = OutputCodeClassifier(sklearn.svm.SVC(kernel = 'linear', decision_function_shape = "ovr")).fit(X, Y).estimators_
print(len(clf))

Also, the values in the code matrx are wrong. Since it should be $$a_{i,j} = \begin{cases} 1 &, i = j \\  -1, &,i \ne j \end{cases}$$

Edit:

clf = OutputCodeClassifier(sklearn.svm.SVC(kernel = 'linear', decision_function_shape = "ovr")).fit(X[:, [1, 10]], categories).estimators_

for estimator in clf:
    y = []

    for val in X[:, 1]:
        y = y + [(val*estimator.coef_[0][0] + estimator.intercept_) / estimator.coef_[0][1]]

    plt.plot(X[:, 1], y)

plt.ylim([-10, 10])
plt.show()

this example leads to the error "'ConstantPredictor' object has no attribute 'coef'". How to fix this issue?


texts: How do you apply hypothesis testing to your features?

texts:
How do you apply hypothesis testing to your features in a ML model? Let say for example that I am doing a regression task and I want to cut some features (once I have trained my model) to increase performance. How do I apply hypothesis testing to decide whether that feature is useful or not? I am just a bit confused about what my null hypothesis would be, level of significance and how to run the experimentation to get the p-value of the feature (I have heard that a level of significance of 0.15 is a good threshold, but I am not sure).

For example. I am doing a regression task to predict the cost of my factory, considering the production of three machines (A,B,C). I make a linear regression with the data and I find out that the p-values of machine A is greater than my level of significance, hence, it is not statistically significant and I decide to discard that feature for my model.

The data looks something like this

d = ('Cost': [44439, 43936, 44464, 41533, 46343], 'A': [515, 929, 800, 979, 1165], 'B': [541, 710, 675, 1147, 939], 'C': [928, 711, 824, 758, 635, 901])
df = pd.DataFrame(data=d)

If you want to see the full data, I have taken this example from a video on Youtube. I couldn't really understand how he drew that conclusion about the level of significance and how he used hypothesis testing in this case.

The link is below, it starts from Min 4:00 (it is only 3 mins long after that)

Linear Regression using hypothesis testing


texts: Results are too good.. what is wrong? How to predict correctly?

texts:
I am about to evaluate a neural network and want to check whether the predictions make sense.

The variables:

X_train1.shape
> (1700, 2)

X_val1.shape
> (339, 2)

X_test1.shape
> (701, 2)

hold in [:, 0] the x-values (in this case time formats respectively hour-values) and in [:, 1] the corresponding y-values (in this case current values):

X_train1[0:3]
> array([[-1.72694903,  0.63332341],
       [-1.71673039,  1.00769389],
       [-1.70651176,  1.14177968]])

You can't see time formats in the first column because I had to convert the time formats to numeric values according to

df_train1["timestamp"] = pd.to_timedelta(df_train1['timestamp']).dt.total_seconds()

Beware that I want to make use of an AutoEncoder, where X=Y:

y_train1 = X_train1

and visually:

First, I search the best hyperparameters:

grid_result = grid_obj.fit(X_train1, y_train1)

Then I use the best parameters:

grid_best = grid_result.best_estimator_

Now I want to predict:

prediction = grid_best.predict(X_test1)

in principle this works but I get way too good results though I can see deviations. But I wonder if my concept is meaningful. For example, all the time before, I used predict() to predict new y on X.
When I try to do the same above, means:

grid_best.predict(X_test1[:, 0])

I receive:

  Error when checking input: expected input_152 to have shape (2,) but
  got array with shape (1,)

These are the results:

I train on a period of time of 3 months and predict on 1 month. Why I think that's too good is because I only use five neurons here: (Input 2, latent space 1, output 2). When I use six neurons, it is almost perfect..


texts: Predict time of dispatch for marketing campaign

texts:
What would be appropriate models/algorithms/strategies for predicting best individual send times for marketing campaigns based on past response timestamps?

Data:

Given for example

===========================================================
  customer campaign    campaign_time       response_time
-----------------------------------------------------------
1   100       a     2017-01-01 06:50:01 2017-01-01 08:02:21
2   101       a     2017-01-01 06:50:01 2017-01-01 16:45:31
3   101       a     2017-01-01 06:50:01 2017-01-02 07:20:00
4   100       b     2017-01-07 06:30:21 2017-01-08 08:15:21
5   101       b     2017-01-07 06:30:21 2017-01-07 17:00:12
6   100       c     2017-01-14 06:43:55 2017-01-14 07:59:44
7   101       d     2017-01-21 14:02:01 2017-01-21 16:50:01
-----------------------------------------------------------

two customers 100&101,
four past campaigns a-d.
with each campaign having different times of dispatch,
and multiple,one or no response time(s) (e.g. buying a product) for customers and campaigns

Goal:

Assuming that

campaign_time can vary for 100 and 101 (personalized times
of dispatch), and
past response times are an indicator for when
customers are most receptive for a campaign

I would like to predict the best next campaign_time ( 2017-01-28 ??:??:??) for each customer based on past response_times, so that the number of respondents per campaign is maximized.

Anyone having any experience with something similar or any ideas where to start? I'd be happy to hear some ideas.

To simplify things, I'd consider the first response_time the most valuable one (=> should be predicted) and I'd also abstract from weekdays (=> it's about predicting time 0:00-23:59, marked by the ? above); however it would be nice to have a continous prediction instead of a discretized one (like suggested here).


texts: Validation curve/RandomizedSearchCV difference train and test score

texts:
Ive build a RF model for an imbalanced data set that after feature selection has an F1 score of 54.26%. I am now trying to do hyper parameter tuning using RandomizedSearchCV, after creating validation curve plots for each hyper parameter to identify a more promising grid.

However, when I create the validation curve plots and execute the RandomisedSearchCV both using train data only, there is a huge difference between the train- and validation set (Cross validation "test set") f1 score performance: train f1 score is 99%+ and validation f1 score is close towards test data set f1 score). This is something I would except since I am training on the train data and then predicting on it (basically giving it the answers). Yet when I see other examples online their train- and validation scores lie very close together and they also do the validation curve plots/RandomizedSearchCV on train data only. Anyone has any idea what I could be doing wrong? Many thanks in advance!


texts: LSTM with multiple entries for the same timestamp

texts:
I have a dataset where I have multiple entries for the same timestamp and I want to use LSTM to forecast the next timestamp given the previous 5 timesteps. From https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ i understand how to deal with multivariate time series and in the blog he reshaped the time series such that we forecast the next step based on the previous time steps. But in my case i have multiple entries for the same time step when i do the reshape it creates the wrong relationship since it assumes the time it's sequential but in fact they are the same time step. In this scenario how should I structure my data for training?
The data is a COVID dataset with header below:
Accurate_Episode_Date, Age_Group, Client_Gender, Case_AcquisitionInfo, Reporting_PHU_City, Outbreak_Related, Outcome1

2020-03-30, 70s, MALE, OB, Stratford, Yes, Fatal


texts: Create the most simple/basic deep network where variable initialization does matter a lot

texts:
I'm trying to create the most basic and simple neural network to simulate a situation where the random initialization of the variables matters a lot.
So if I run the network and the initialization over and over again the network will not converge well, unless the random initialization hit the sweet spot.

So basically trying to come up with the most simple example of a non-convex cost function to simulate a situation of non-convergence.


texts: Approaches to A/B testing when you can't randomize on the user level

texts:
I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.

For example, if I own 10 shops and have a certain algorithm that I use to price items, and perhaps I want to test a new algorithm that would result in price changes. There are a few approaches I can see but it isn't clear to me if I'm missing others and would love to hear more.

Standard A/B testing:

If I were to randomly assign shoppers to test or control buckets then this would obviously be a bad experience for them (given that prices are visible).

Switchback experiment

Perhaps instead I decide to run  a switchback experiment, in which the control pricing algorithm runs for 6 hours, then the test pricing algorithm, runs for 6 hours, and so on throughout my test period. This is obviously better than the standard A/B approach but I feel like it still has problems (ie what if the price change results in a change to long term customer behavior but it takes some time for the behavior to change - perhaps the switchback wouldn't capture this).

Causal Impact approach

Another approach might be to use a Causal Impact style analysis, in which I set certain shops as 'control' shops, and other shops as 'test' shops, and then change the algorithm at the test shops and analyze my metric of interest, where we look for a causal effect in the time series at the test shops with respect to the control shops. The drawback here is that you seem highly dependent on how well the controls represent the test shops.

Does anyone have any other suggestions, or even somewhere I can read more about different experimental setups, beyond A/B testing?


texts: Algorithm for user profiling without distinct profiles

texts:
I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide what software and hardware this new user would need.

I am rattling my brain thinking how I could get this to work -
I could use a supervised learning approach and train a model with a dataset of already employed users and the software and hardware they use, however, the variables in this dataset would be extensive as, through one-hot encoding, each user object would have to have variables that represent each possible software or hardware a user could own. Or can my dataset have a collection of user objects but not each object has the same software/hardware variables?

I'm very confused about how to go about this so any insight would help. I'm probably looking at going about this the wrong way.


texts: "Failed to find data adapter that can handle input" when working with mixed data

texts:
I have been trying to make my own generator that would handle an [image, float] value for the x (input), while the y is an int (linear output). My data is compiled into a .csv file. Can someone help me plug this data into a model.fit. I am not very confident in mixed data so there may be more than one problem.
I am currently using:
python 3.8\
tf-gpu 2.4.0rc1\
keras 2.4.3\
pandas 1.1.4

My code:
IMG_SIZE = 400
Version = 1
batch_size = 8

image_list = glob('F:/DATA/Vote/Images/**', recursive=True)
traindf = pd.read_csv('F:/DATA/Vote/Vote_Age.csv', index_col='ID')
del image_list[0] #First path did not lead to an image

def age_link(df, image_name):
    csv_row = df.loc[int(image_name), :]
    return csv_row[0]

def votes_link(df, image_name):
    csv_row = df.loc[int(image_name), :]
    return csv_row[1]

def generator(df, image_list, batch_size, IMG_SIZE):
    while True:
        random.shuffle(image_list)
        for i in range(batch_size):
            image_path = image_list[i]
            image_name = os.path.basename(image_path).replace('.jpg', '')
            image = load_img(image_path, target_size=(IMG_SIZE, IMG_SIZE))
            image = np.array(image) / 255

            age = age_link(df, image_name)
            votes = votes_link(df, image_name)
            age.astype(float)
            votes.astype(int)

            return np.asarray([image, age], dtype=object), votes

inputA = Input(shape=(IMG_SIZE, IMG_SIZE, 3))
inputB = Input(shape=(1,))

y = Conv2D(16, 3, activation = 'relu')(inputA)
y = Conv2D(16, 3, activation = 'relu')(y)
y = Conv2D(32, 3, activation = 'relu')(y)
y = Conv2D(32, 3, activation = 'relu')(y)
y = Flatten()(y)
y = Dense(4, activation="relu")(y)
y = Model(inputs=inputA, outputs=y)

x = Dense(8, activation="relu")(inputB)
x = Dense(4, activation="relu")(x)
x = Model(inputs=inputB, outputs=x)

combined = concatenate([x.output, y.output])

z = Dense(2, activation="relu")(combined)
z = Dense(1, activation="linear")(z)

model = Model(inputs=[inputA, inputB], outputs=z)
model.compile(loss="mean_absolute_percentage_error", optimizer='adam')

model.fit(generator(traindf, image_list, batch_size, IMG_SIZE),
    epochs=200, batch_size=8)

Output Error:
Traceback (most recent call last):
  File "f:\PYTHON\YiffMiner\TrainYIFF_Votes.py", line 91, in <module>
    model.fit(generator(traindf, image_list, batch_size, IMG_SIZE),
  File "C:\Users\Tristan\anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1050, in fit
    data_handler = data_adapter.DataHandler(
  File "C:\Users\Tristan\anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\data_adapter.py", line 1099, in __init__
    adapter_cls = select_data_adapter(x, y)
  File "C:\Users\Tristan\anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\data_adapter.py", line 961, in select_data_adapter
    raise ValueError(
ValueError: Failed to find data adapter that can handle input: (<class 'tuple'> containing values of types {"<class 'numpy.float64'>", "<class 'numpy.ndarray'>"}), <class 'NoneType'>


texts: I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary classification. I could not find it, so I want to know if you guys know any dataset with this characteristics, any suggestions?


texts: Does tensorflow pick samples from a dataset randomly or sequentially when training?

texts:
I have a dataset which consists of more than 10000 images. But similar images are grouped together. I mean first 50 images are very alike then the next 50 images are different(not as much similar as the first ones. I am talking about guns specifically) than the first 50 but they are similar between each other. If I choose a batch size of 50, will it lead to worse results? Or It picks random subsets of the dataset to train at a time?

I am new at deep learning if the answer is obvious sorry.


texts: Clustering Observations by String Sequences (Python/Pandas df)

texts:
I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both columns to index form to simplify the data. As such, the set has 2 columns; 1. A unique ID and 2. A bag of key strings (they are identifying numbers but in string format, separated by spaces). Example format:

ID |  KeyWords
0 |   1 2
1 |   2
2 |   3
3 |   2 3

I initially vectorised KeyWords by TD-IDF and used a simple and dirty KMeans (via sklearn) to cluster them into 13 clusters (selected by the elbow method), which seemed to work and output intuitive and interesting results.

However, the issue I have come across is that there is always one dominant cluster that holds ~80% of the observations. I assume that KMeans was the limiting factor with assumed constant variance, and looked into Latent Dirichlet Allocation (LDA), as well as Gaussian Mixture Modelling (GMM) and BayesianGMM as alternatives. Using the same data format as above, LDA seemed to produce unintuitive results, for which all the topics were very similar and did not make much sense (vastly different to KMeans clusters, expected to see some key similar topics?).

I'm quite new and out of my depth with clustering and have been reading quite a lot about it, but if anyone could provide any insights into immediate logic flaws or a push in the right direction I would very much appreciate it.

Edit:
There's about 3300 unique URLs (or KeyWords in this case). The number of tokens/URLs per document is quite skewed actually. The median is 1, mean is ~1.5 (there is a small number of ID's that have a very large number of tokens, some I can deem as outliers but most I cannot). Another key fact is that in the dominant cluster, almost all of these "big players" who have looked at a large number of KeyWords are all within the dominant cluster. I removed them to see how it played out, and the dominant cluster remains.


texts: "Random Forest" variant of other classifiers

texts:
I am using SVM with linear kernel for one of my multiclass text classification problem and getting accuracy of 0.78. I also tried Random Forest but the best accuracy I could get was 0.72. This set me thinking what if I could simulate Random Forest but with a SVM classifier and not decision tree. I am not sure whether it will result in improved accuracy or not. I wanted to have something in favor before I implemented this and a quick google search didn't help either.

Any thoughts on this?


texts: Best approach to create a text classification model with two inputs?

texts:
I'm looking to train a model with two text inputs (sentences) and a binary classification. Essentially, for 2 given sentences, are they paraphrases or not.
I want to use the Microsoft research Paraphrase Detection dataset as I feel this would be a good place to start.
What would be the best approach to do this?
Thanks in advance, and sorry for my naïvete.


texts: Dealing with a dataset having target values on different scales?

texts:
I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc].

Now , the continuous target variable is sales of a given product in that particular country for a given month in a given year.

It has been given in the problem statement , that the Sales have been taken in the local currency of that country , so now I have values on different scales and I am not sure how to deal with them.

When I grouped the data as per the different countries , (using pandas's groupby function) , I got at least 1000 observations for each country. So maybe I could train a model separately for each country ? All kind of approaches will be appreciated.


texts: Random Forest Classifier cannot recognise parameter grid

texts:
I am trying to run the below code to extract the feature importances of my random forest, but I'm getting the following error TypeError: init() got an unexpected keyword argument 'randomforestclassifier__max_depth'. Can anyone tell me what is wrong?
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.metrics import f1_score

x, y = make_classification(n_samples=10000, weights=[0.99], flip_y=0)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 1)

paramgrid_rf = {'n_estimators': [500],
                'max_depth': [4],
                'random_state': [0],
                'max_features': ['sqrt']
                }

imba_pipeline_rf = make_pipeline(RandomOverSampler(sampling_strategy=0.35, random_state=0),
                             RandomUnderSampler(sampling_strategy=0.9, random_state=0),
                             RandomForestClassifier())

new_params = {'randomforestclassifier__' + key: paramgrid_rf[key] for key in paramgrid_rf}
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
grid_imba_rf = GridSearchCV(imba_pipeline_rf, param_grid=new_params, cv=cv, scoring='f1',
                        return_train_score=True)
scores = cross_val_score(imba_pipeline_rf, x_train, y_train, scoring='f1', cv=cv, n_jobs=-1)
grid_imba_rf.fit(x_train, y_train)
y_pred_rf = grid_imba_rf.predict(x_test)
print('F1 score on validation data: ', f1_score(y_test, y_pred_rf))

rf_final = RandomForestClassifier(**grid_imba_rf.best_params_).fit(x_train_res,y_train_res)
rf_final.feature_importances_


texts: order of features for model tuning vs model fitting

texts:
Assuming that the same columns (i.e., features) are used for hyperparameter tuning and model fitting, and ensemble models are used for modeling (e.g., Random forest or XGboost), then does the order of columns used during the hyperparameter tuning process should be identical to the order of columns used when fitting the model based on the best hyperparameters?

I am using sklearn's make_column_transformer functions in my CV pipeline for the hyperparameter tuning. Unfortunately, this function modifies the order of provided columns when setting the remainder argument to 'passthrough'. Should I ensure that when fitting the model the same order of columns is preserved, or the order does not matter as long as I am using the same features.


texts:
I want to lemmatize set of plural keywords automatically such as 'Web based technologies', 'Information systems' etc. I want to transform them to to 'Web based technology', 'Information system' respectively.

I tried NLTK as follows

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print(lemmatizer.lemmatize("Web based technologies"))

Even though this perform fine for words such as 'cats' to 'cat', for the keywords I have mentioned I get the same plural form. Any idea how to solve this? Or are there any other tools/APIs that I can make use of?

P.S. Given the keywords I only want to get the singular term of it


texts: Improve performances of a convolutional neural network

texts:
I am doing image classificaition, and to do this I have built the following neural network:

def Network(input_shape, num_classes, regl2 = 0.0001, lr=0.0001):

model = Sequential()

# C1 Convolutional Layer
model.add(Conv2D(filters=96, input_shape=input_shape, kernel_size=(3,3),\
                 strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# C2 Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C3 Convolutional Layer
model.add(Conv2D(filters=768, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C4 Convolutional Layer
model.add(Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C5 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C6 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C7 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Flatten
model.add(Flatten())

flatten_shape = (input_shape[0]*input_shape[1]*input_shape[2],)

# D1 Dense Layer
model.add(Dense(4096, input_shape=flatten_shape, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D2 Dense Layer
model.add(Dense(4096, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D3 Dense Layer
model.add(Dense(1000,kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# Compile

adam = optimizers.Adam(lr=lr)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

return model

  # create the model
  model = Network(input_shape,num_classes)
  model.summary()

it works good enough, but I would like to increase its performances.

How could I modify it to do so?

I was thinking about adding layers, which should give better performances, but I haven' t understand well if I should add convolutional layers or dense layers. Moreover I would like to find other ways to increase accuracy than simply adding layers.

Can somebody please help me?

Thanks in advance.

[EDIT] I am considering a training set of 1200 images, which represent 4 wheater conditions : Haze, Rainy, Snowy, Sunny.

With my model, the Test accuracy is 0.797500, and the Test loss is 1.881952.

I would like to increase more my accuracy, but I don' t have other ideas than adding convolutional layers. I could try to change the size of the kernels and other hyperparameters, but I have other ideas.


texts: Is my model to big? I am trying to predict orders for a company, and I don't know if there are typical values for macroparameters

texts:
I am building a model to predict orders, from its time series (univariate), for a company. I am working with 30 layers of 400 LSTM neurons each with the activation function hyperbolic tangent of Yann Lecun, 1 layer of 400 LSTM neurons with the relu activation function, and 3 layers of 500 neurons each, with a Relu Activation Function.

These values have had the best performance yet, but I don't know if there are typical values for training, or if it is an abnormally huge model.


texts: Looking for spatial clusters and anomalies. Is DBSCAN the right tool?

texts:
I have a regular 2D grid of data points (X, Y) with each point having a value.
I'd like to identify clusters and then anomalies that don't belong to those clusters.
I'm trying to understand the best way to do this and it looks like DBSCAN might be an appropriate algorithm to use but I'm a bit confused.
I'm not clear whether I should collapse the 2D (X,Y) data down into 1 dimension and run DBSCAN on the 1D array. That would lose the spatial component of the data but would then identify clusters based on the value of the data point.
Or do I run DBSCAN on a 3D array (the X index, the Y index and it's corresponding data value for that X,Y point)? That sounds intuitively like it might be the better way to approach this but I'm not 100% sure.
If anyone could point me towards an example of using DBSCAN for spatial clustering that'd be really helpful.


texts: Huge cost not converging well with TensorFlow logistic regression

texts:
I try to use Logistic Regression for a dataset which contains 15 numeric features and 4238 rows of examples. The calculated cost started at 415.91, and converged when the cost was reduced to 220.119 only. I think there must be something wrong, but as I am not sure what to do, I would like to share the code with you, it would be helpful for me to know what is not alright in the code and might cause the issue. I would appreciate your advises and experiences a lot!

import tensorflow as tf
import pandas as pd
from sklearn.model_selection import train_test_split

dataset = pd.DataFrame.from_csv('framingham_heart_disease.csv', index_col = None)
print(dataset.shape)
dataX, dataY = dataset.iloc[:,:-1], dataset.iloc[:,-1:]
dataX = dataX.values/50
dataY = dataY.values

trainX, testX, trainY, testY = train_test_split(dataX, dataY, test_size=0.20, random_state=42)

numTrainData = trainX.shape[0]
numFeatures = trainX.shape[1]
numLabels = trainY.shape[1]

X = tf.placeholder(tf.float32, [numTrainData,numFeatures])
yExpected = tf.placeholder(tf.float32, [numTrainData, numLabels])

tf.set_random_seed(1)
weights = tf.Variable(tf.random_normal([numFeatures,numLabels],
                                       mean=0,
                                       stddev=0.01,
                                       name="weights"))
bias = tf.Variable(tf.random_normal([1,numLabels],
                                    mean=0,
                                    stddev=0.01,
                                    name="bias"))

apply_weights_OP = tf.matmul(X, weights, name="apply_weights")
weights_after_nan = tf.where(tf.is_nan(apply_weights_OP), tf.ones_like(apply_weights_OP) * 0, apply_weights_OP);
add_bias_OP = tf.add(weights_after_nan, bias, name="add_bias")
activation_OP = tf.nn.sigmoid(add_bias_OP, name="activation")

learningRate = tf.train.exponential_decay(learning_rate=0.0001,
                                          global_step= 1,
                                          decay_steps=trainX.shape[0],
                                          decay_rate= 0.95,
                                          staircase=True)
cost_OP = tf.nn.l2_loss(activation_OP-yExpected, name="squared_error_cost")
training_OP = tf.train.GradientDescentOptimizer(learningRate).minimize(cost_OP)

sess = tf.Session()
init_OP = tf.global_variables_initializer()
sess.run(init_OP)

numEpochs = 3000
cost = 0.0
diff = 1
epoch_values = []
accuracy_values = []
cost_values = []

for i in range(numEpochs):
    if i > 1 and diff < .0001:
        print("change in cost %g; convergence."%diff)
        break
    else:
        step = sess.run(training_OP, feed_dict={X: trainX, yExpected: trainY})
        # Report occasional stats
        if i % 100 == 0:
            # Add epoch to epoch_values
            epoch_values.append(i)
            # Generate accuracy stats on test data
            newCost = sess.run(cost_OP, feed_dict={X: trainX, yExpected: trainY})
            # Add cost to live graphing variable
            cost_values.append(newCost)
            # Re-assign values for variables
            diff = abs(newCost - cost)
            cost = newCost

            #generate print statements
            print("step %d, cost %g, change in cost %g"%(i, newCost, diff))

I'd expect a better convergence with lower cost, but instead I get this:
step 0, cost 415.91, change in cost 415.91
step 100, cost 229.459, change in cost 186.45
step 200, cost 221.717, change in cost 7.74254
step 300, cost 220.504, change in cost 1.2124
step 400, cost 220.225, change in cost 0.279007
step 500, cost 220.15, change in cost 0.0752258
step 600, cost 220.127, change in cost 0.022522
step 700, cost 220.121, change in cost 0.00689697
step 800, cost 220.119, change in cost 0.00166321
step 900, cost 220.119, change in cost 6.10352e-05
change in cost 6.10352e-05; convergence.

I would appreciate your advises a lot, have a good day people :)


texts: Difference between learning_curve and validation_curve

texts: What is the difference between these two curves: learning_curve and validation_curve ?


texts: Optimisation strategy webstores with shipping costs

texts:
I am scraping the prices of several products on different websites for the past couple of weeks. These prices are stored, plotted and e-mailed to me every day with an update about the price changes. Now I want to take it to the next level, and make a store selection based on several products.

What I want, is pick a number of products, and calculate the most optimal shops where to buy these products, including shipping costs.

For example, I want to buy products A, B and C, at stores X and Y, with the following prices (so product A costs 2 at store X, and 3 at store Y):

   X Y
A  2 3
B  3 4
C  3 2

Not taking into account shipping costs, I would buy A and B at store X, and C at store Y. However, if the shipping costs are too large for store Y, it might be better to buy everything at store X. Making things more complicated, if stores X and Y have shipping costs of 5, but Y is free when buying more then a specific amount, it might be more profitable to buy everything at Y.

Can anyone point me in the right direction how to solve this? The idea for now is to include ~10 products at three stores. I though about linear optimisation, but I do not know how to include the shipping costs.


texts: Alternative means of clustering streams of incoming facial recognition data

texts:
I have a time-series dataset of incoming face data. Each data point is a facial-feature-vector of length 256 that represents the facial features of a person (it is generated by a modified RESNET). Features that are close together are deemed to belong to the same person.

I am (successfully) clustering the incoming face features by DBSCANing. I've recently switched to HDBSCAN also with good results.

My problem is this: DBSCAN and HDBSCAN require I have all the data together at one time. I often have >200,000 features which can be a very large download.

I would much prefer to be able to take every incoming f and assign it to a person without having to collect all the information at one time.

Is there an alternative to this (preferable with a Python implementation)?


texts: Difference in features generated by same filters for color and grayscale images?

texts: Would there be ay difference between the features generated by CNNs if they are fed with same image in color and grayscale format. If I am performing classification with same network for let's say plane and bird classification, would there be any damage in accuracy if I fed grayscale images instead of color images? How will affect the features generated by CNN filters?


texts: how to reduce the load time of tensorflow text classification model.?

texts:
I want to reduce the load time of the model, when testing the model to predict the categories using tensorflow text classification. I am getting the raw data from mongodb collection, which has over 1 million of data. It is taking a long time for prediction.

I have used multilabel binarizer for the multi categories

estimator = tf.contrib.estimator.DNNEstimator(
                        head=multi_label_head,
                        hidden_units=[64, 10],
                        feature_columns=[descriptions_embeddings])

labels = np.array(train_encoded)
features = {'descriptions': np.array(train_descriptions)}
train_input_fn = tf.estimator.inputs.numpy_input_fn(
                                           features,
                                           labels,
                                           shuffle=True,
                                           batch_size=32,
                                           num_epochs=20)

estimator.train(input_fn=train_input_fn)
eval_input_fn = tf.estimator.inputs.numpy_input_fn(
                {'descriptions':np.array(test_descriptions).astype(np.str)},
                  test_encoded.astype(np.int32),
                  shuffle=False)

estimator.evaluate(input_fn=eval_input_fn)
raw_test = ["Pemeriksaan pilihan makanan kita dan makanan yang kita masukkan ke dalam tubuh kita. Berdasarkan memoar Jonathan Safran Foer.", #Documentary
            "When a lawyer shows up at the vampire doorstep, he falls prey to his charms and joins him in his search for fresh blood. Enter Dr. van Helsing, who may be the only one able to vanquish the count."]

predict_input_fn = tf.estimator.inputs.numpy_input_fn(
            {'descriptions': np.array(raw_test).astype(np.str)},
            shuffle=False)

prediction = estimator.predict(predict_input_fn)
for movie_genres in prediction:
top_2 = movie_genres['probabilities'].argsort()[-3:][::-1]
for genre in top_2:
    text_genre = encoder.classes_[genre]
    print(text_genre)

I am expecting to have the minimum load time when testing the model, Does TF serving will be suitable and how i can integrate into it.


texts: How to interpolate and check correlation of two time series with differing cardinality

texts: I want to check how correlated two time series are, but they don't have the same cardinality. They have different number of data points because the timestamps the data are collected are different. Available libraries I have found require that the cardinality be the same. Therefore, I would like to ask if there is a library, an algorithm I can implement myself, or some advice you can give me to approach the problem. Thank you


texts: Extracting sub features from inside a df cell?

texts:
I have a dataframe containing several features of form:

Id, Acol,                   Bcol,   Ccol,           Dcol,
1,  X:0232,Y:10332,Z:23891, E:1222, F:12912,G:1292, V:1281
2,  X:432,W:2932            R:2392, T:292,U:29203   Q:2392
3,  Y:29320,W:2392          R:2932, G:239,T:2392    Q:2391

...about 10,000 Id's

where 1,2,3 are the Id's.
Acol, Bcol, Ccol, and Dcol are the feature columns,
X, Y, Z, W are sub-features of feature "Acol" and so on...

How can I extract the sub-features/features from this sort of dataframe?


texts: Does Sklearn's KNeighborsClassifier Map Input to Output If Dimensions Don't Match?

texts:
I want to classify a hyperspectral image (Indian Pines data set).
The input is of shape (145, 145, 200) = a HSI of 145x145 px with 200 bands.
Each one of the 145x145 pixels should be classified to one of the 15 classes, so 'y' is of shape 145x145 containing the labels for each one of the pixels.
After flattening X (shape is now 145 * 145, 200) and y (shape is now 145 * 145) I used:
X_train_flat, X_test_flat, y_train_flat, y_test_flat = train_test_split(X_flat, y_flat, range(X_flat.shape[0]), test_size = 0.8, random_state = 12)

knn = KNeighborsClassifier(n_neighbors=16)
knn.fit(X_train_flat, y_train_flat)
y_pred_flat = knn.predict(X_test_flat)

This seems to work, but it is not clear to me if KNeighborsClassifier really knows that a 'pixel' is actually an array of values corresponding to the 200 bands and that KNeighborsClassifier should map all of the 200 bands to a single value from y.
For example, the first pixel of X has 200 values (i.e. X[0][0][0], X[0][0][1],.., X[0][0][199]). Can KNeighborsClassifier determine that each value of input X[0][0] belongs to y[0][0]?
I hope I made my question clear enough.


texts: Keras: apply multiple filters to each feature map in CNN

texts:
I am new to Keras, and I want to do the following: take a 2D image, and apply four 2D convolution kernels to it, giving four 2D feature maps. I could accomplish this. But then I want to apply two  distinct 2D convolutions to each of those 4 maps, giving 8 feature maps. Is that possible?
Here's what I have so far:
import keras
from keras.layers import Conv2D

input_img = keras.Input(shape=(N_rows, N_cols, 1))
x = Conv2D(4, (3,3))(input_img)

But then I don't know how to apply 2 kernels to each of the 4 channels, so that I have eight 2D maps.


texts: Is it a good idea to combine fine tuning and feature extraction techniques?

texts:
I have a normal/tumor medical images dataset and, for the same patients, also the relative genomics, and my goal is to predict if a patient has a tumor by combining all the information.
To achieve this, I am using a ResNet50 with imagenet weights to extract features from images, and other methods to extract features from genes. I join the two features and use an SVM to make a prediction.
The accuracy isn't extremely bad, but I wanted to know if it can be increased by performing a fine-tuning over the same images, in place of using a network only trained on the imagenet datasets as it is.
I have researched papers but I found literally nothing, neither in favor nor in contrast with this hypothesis.
Is there some known contraindication?


texts: Keras: visualizing the output of an intermediate layer

texts:
I have read the docs here and I understand the general idea. I am able to visualize the weights of the intermediate layers. However, I'm having trouble visualize the activations. Here's what I have:

I trained my model and saved the weights in a file called weights_file.

Thanks to this jupyter notebook, I got the values of the weights. First I defined my model:

def mlp_model(hid_dim=10):

        model = Sequential()
        model.add(Dense(units=hid_dim, input_dim=X.shape[1], activation='relu'))
        model.add(Dense(Y.shape[1], activation='softmax'))
        model.load_weights(weights_file)
        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

        return model

model_created = mlp_model(hid_dim=15)

To get the weights, I did this:

W = model_created.layers[0].kernel.get_value(borrow=True)
W = np.squeeze(W)
print("W shape : ", W.shape) #(153, 15)

W_out = model_created.layers[1].kernel.get_value(borrow=True)
W_out = np.squeeze(W_out)
print('W_out shape : ', W_out.shape) #(15, 8)

From there I could create Hinton diagrams using this. However, when I try to work with the activations:

get_first_output = theano.function([model_created.layers[0].input], [model_created.layers[1].output])
layer_out = get_first_output([X[0,:]])[0]

I get this error:

TypeError: ('Bad input argument to theano function with name "mlp1_visualize_weights.py:131" at index 0 (0-based).  \nBacktrace when that variable is created:\n\n  File "mlp1_visualize_weights.py", line 213, in <module>\n    mlp_repeat(X, Y, Xtest, Ytest, params_to_use, weights_file)\n  File "mlp1_visualize_weights.py", line 125, in mlp_repeat\n    model_created = mlp_model(hid_dim=hid_val, lr=lrate, reg_val=reg, momentum=moment, nest=nestval, optimizer=optim)\n  File "mlp1_visualize_weights.py", line 105, in mlp_model\n    model.add(Dense(units=hid_dim, input_dim=X.shape[1], kernel_initializer=\'he_uniform\', activation=\'relu\', W_regularizer=l2(reg_val), b_regularizer=l2(reg_val)))\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py", line 426, in add\n    dtype=layer.dtype, name=layer.name + \'_input\')\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 1392, in Input\n    input_tensor=tensor)\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 1303, in __init__\n    name=self.name)\n  File "/mnt/data/caugusta/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 184, in placeholder\n    x = T.TensorType(dtype, broadcast)(name)\n', 'TensorType(float32, matrix) cannot store accurately value [array([ 0.        ,  0.2037037 ,  0.20138889,  0.21100917,  0.62962963,\n        0.6875    ,  0.61206897,  0.44660194,  0.31168831,  0.17391304,\n        0. ...

I would like to look at just one input example, and find the activation and the weights from just that input example. Essentially I'm trying to figure out which features of the data each hidden unit is picking up.

Can anyone explain how to get the activations of intermediate layers in Keras?


texts: unsupported operand type(s) for -: 'list' and 'list' using python

texts:
Here I have a data file and I designed neural network to predict value. I have a three inputs. These three inputs affect to predict value bysubtarcting and adding. If my three inputs are x1,x2,x3 . X1 and X2 add together and that  value  will subtract with the X3 value and give the predict value. This process will happened in x_train. I wrote a code. But it gives me this error. Can nyone help me to solve this problem?
My code is,

data = pd.DataFrame(data,columns=['X1','X2','X3'])
data = data.values
a = []
for i in range(0,len(data)):
    X1 = data[i][0]
    a.append([X1])
b = []
for i in range(0,len(data)):
    X2 = data[i][1]
    b.append([X2])
c = []
for i in range(0,len(data)):
    X3 = data[i][2]
    c.append([X3])
x_train =[]
x =[]
for i in range(0,len(data)):
    x_train.append([a[i] ,b[i], c[i]])
    x.append((a[i] - b[i] + c[i]))
x_train =np.array(x_train)
x = np_array(x)


texts: Asking a specific question from a general one

texts:
Setting

I have data about a board game which has been played multiple times with different players per game (though players reoccur in all kinds of different constellations). During the game one can score points in one of seven categories. The general question is: What is the best strategy? On what category should players concentrate?

Details about the data and performed transformations

The original data has the variables:

game - the number of the game
name - the name of the player
military points
money points
world wonder points
... points
sum of points

which I transferred into

game
name
sum of points (a little repetitive, since its the same for a game and name)
place in game (a little repetitive, since its the same for a game and name)
type of points (one of the seven categories)
number of points
percentage this type of points contributed to the total amount of points of this player for this game

My Question

Asking the right question is been said to be the most important thing in Data Science and in most situations I managed to find a good concrete (programmable) question, but this one seems tough to me.

So: How do I ask a specific question to answer the general question?

Of course I have already asked/answered simple questions regarding the winning players of each game. For example: "What is the type of points that contributes most to the total points for winning players (whose success proved them right)?"

Now I'd like to make analyses over the entire field of players.

An even more advanced question would consider that player reoccur across games.

I have no idea what tag to use here.


texts: Features selection: should I mix features?

texts:
I have sets of features of different nature(for example, 300 features from FFT-transform, 1000 categorical features and so on). However there are only 900 samples and I`m trying to select important features using Lasso.
So the question is: should I perform feature selection differentially on subsets of features or better firstly, concatenate features and only then perform feature selection?


texts:
K-modes algorithm is available here

I want to do clustering of my binary dataset. I need to specify the number of clusters that I need as an output:

KModes (n_clusters, init, n_init, verbose)

My dataset contains 1000 lines and 1000 rows, I want to calculate the distance between my clusters in order to know the exact number of cluster that I need to choose. I don't know how to compare between them. Probably I want to use the hamming distance because it is the most suitable distance to compare between binary data.

Any recommendations that you can give are welcomed.


texts: Deepmind conditional neural process: evaluation

texts:
Going through the Deepmind jupyter notebook conditional neural processes, the plots at the bottom of the notebook show that the ground truth and the predicted distribution only overlap around the "context points". These context points are already in the training set. This comes as a surprise to me because I was expecting that if the model worked, then the ground truth curve would lie inside the predicted distribution at non-context points. So, doesn't this mean that the network failed to model the data? If that's the case, what's the value being shown here?

Edit: looking at Fig 2 in their arxiv publication here and comparing them with the plots in the jupyter notebook, it seems that the publication shows nice plots for which this worked. The plots in the notebook are not as confirming though.

Edit 2: re-running their notebook doesn't yield the same conversion of the loss.

At Iteration: 56200, Thu Jan 24 14:13:40 2019, loss: 1.22

(whereas in the published notebook the loss is already at 0.6 at iteration 20,000)

Edit 3: Edit 2 is not valid. I found that I had made an edit to the code while playing around with it, and it turned out to be critical for the training. (I short-circuited the if self._testing: condition in generate_curves to if True to see its impact and forgot to undo it). The training converges as published in the notebook. I have yet to play with the trained model to see how good the distributions are on non-context points

Edit 4: Now that the network converges, generating new data as validation (with the same method as for testing data), shows that the probability distribution does not capture context points. It also shows a narrow confidence interval in the center, which completely misses the ground truth.


texts:
I'm training a Neural Network and I'm trying to divide my data into training and testing sets. I have a lot of output classes and for some of them I have as little as 2 examples, so I would like to have, in that case, 1 example in training and 1 example in testing. From what I've read, this is using the stratify parameter, but what does stratify mean?

I'm divifing my data into training and testing:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.1, random_state=42, stratify=y)

So, from my understanding, this divides into two sets: training (90% of data) and testing (10% of data) but making sure that there are at least 1 of each class in each set?


texts: Regex remove 1-2 character sequences: hyphens literal, not word boundaries

texts:
I need a regex in R to exclude 1 or 2-character words, but which does not treat hyphens as word boundaries.  Here is an example:

z <- c("regulatory protein SR-B1 na na na na", "Dr Foo is na Editor-in-chief", "na P3 protein is popular na na")

The intended 'gsub' would produce the result

  [1] "regulatory protein SR-B1" " Foo Editor-in-chief"     "P3 protein  popular"

This is not produced from the gsub statements below. Instead:

A - 2-character segments unintentionally dropped from hyphenated words (and hyphens kept).

B - 2-character segments unintentionally dropped within hyphenated words (hyphens also dropped):

C - Produces: Error: '-' is an unrecognized escape in character string starting ""
*\b[[-"

D - Nothing dropped or changed:

gsub(" *\\b[[:alpha:]]{1,2}\\b *", " ", z) # A

gsub(" *\\b[-[:alpha:]]{1,2}\\b *", " ", z)  # B

gsub(" *\\b[[\-][:alpha:]]{1,2}\\b *", " ", z)  # C

gsub(" *\\b[[\\-][:alpha:]]{1,2}\\b *", " ", z) # D

Input would be much appreciated.


texts: Analyzing mobile usage. What kind of approach should I apply?

texts:
I need to analyse a dataset about mobile phone usage (#calls, #sms, #internetConnections) per each cell and hour in the different days.

[date] [CDR/Position] [#calls] [#sms] [#internetConnections]

My purpose is detecting similarities in the data (Monday-Tuesday is similar... or Monday night is different...). After this, I'd like to find the reason they are similar/dissimilar.

What can I apply?


texts: How to use pivot_table on a dataframe? How to handle exceptions.UFuncTypeError?

texts:
I tried to use the pivot_table function on a dataframe. This was working at the beginning now I get all the time errors, and don't really get why.
cols = (['B', 'C', 'D', 'E'])
dataset[cols] = dataset[cols].apply(pd.to_numeric,errors='coerce', axis=1)

score_table = dataset.pivot_table(index='A', values=['B', 'C'], aggfunc = np.mean)
print(score_table)
score_table.plot(kind="bar")
plt.savefig(filename + 'figure_meanscore.pdf')

and thats my error:
numpy.core._exceptions.UFuncTypeError: ufunc 'add' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> dtype('<U32')


texts: How to label "other" while labeling image for object detection/classification?

texts:
I want to train a model to recognize the different categories of food e.g. rice, burger, apple, pizza, orange and other things.

After the first training, I realized that the model is detecting other objects as food. e.g. hand as fish, phone as Chocolate, person as candies.

I get a very low loss because the testing dataset and validation must have at least a pictures of food. But when it comes to a picture of an object other than food, the model fails. How do label the dataset in a way that the model will not do any detection if there is no food on the picture?


texts:
So I have a data set I have successfully used to train a model, and have decent results. I am using a Two Class Boosted Decision tree for a Boolean output. So far so good.

I now want to analyze each column of my data set and remove any column that is not a meaningful influence on the outcome. I see statistics on columns in my data set:

But I don't see whether a column has a strong relationship with the output variable. Any clues?


texts: How to identify recurring patterns in this set of transactional data

texts:
I'm working on a dataset of banking transactions and would like to find recurrent transactions.
I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa.stattools to calculate the autocorrelation function but i'm not getting the expected results:

r = acf(ts, fft=False)

For example this set of transaction (ASSURANCE DESJ) is getting an acf score of 0.3159 when it's obviously a recurring transaction (same amount, same frequency).

Another example of recurring transactions with acf=0.22775:

But this one should not be found as a recurring transactions, and get a score not too far from the previous set (0.26919):

I've been checking a lot of different methods, I acutally came up with a combination of auto-correlation on the regular timeserie, auto-correlation on a timeserie with amount=1, with stationnary checking and other rules to have a not so perfect results. I've also checked at ARIMA and other methodology without luck.

Would you have a better way to detect recurring transactions from timeseries ?

Link to Datasets


texts: Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

texts:
I have several marketing files that I need to download on a weekly basis.
Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, field 7, field 8, field 9. The data contained in the fields are different and specific to each file.
Each file can contain 10 to 1000s records.
What I need to do, is add 2 additional fields to each record in each file. These fields; field A and field B will contain the same info for every record in each specific file. Each file will contain different info to add.
File 1      Need to add Field A and Field B to every record.
File 2      Need to add Field C and Field D to every record.
File 3      Need to add Field E and Field F to every record.
I can edit each file in Excel, add the columns, name the columns, add the 2 fields to the first row and
then copy and paste to all the other rows.
Is there a way to specify the file, field 1 and field 2 in order to add the info more efficiently and quickly?
Thank you!


texts: How to decide for the contamination value (proportion of the outliers) in my dataset?

texts:
I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1.

I am worried if I decide for this parameter with experimenting on my current dataset, it will differ by time as I would be receiving new data points very soon.

How can I know that this parameter better than any other, just by experimenting it with outlier removal?

Thanks.


texts: Optimal points of $f(x,y)=x^2 + y^2 + \beta xy + x + 2y$

texts:
I am self-learning basic optimization theory and algorithms from "An Introduction to Optimization" by Chong and Zak. I would like someone to verify my solution to this problem, on finding the minimizer/maximizer of a function of two variables, or any tips/hint to proceed ahead.

For each value of the scalar $\beta$, find the set of all stationary points of the following two variables $x$ and $y$
$$f(x,y) = x^2 + y^2 + \beta xy + x + 2y$$
Which of those stationary points are local minima? Which are global minima and why? Does this function have a global maximum for some value of $\beta$.

Solution.
We have:
\begin{align*}
f(x,y) &= x^2 + y^2 + \beta xy + x +2y\\
f_x(x,y) &= 2x+\beta y + 1\\
f_y(x,y) &= \beta x + 2y + 2\\
f_{xx}(x,y) &= 2\\
f_{xy}(x,y) &= \beta \\
f_{yy}(x,y) &= 2
\end{align*}
By the first order necessary condition(FONC) for optimality, we know that if $\nabla f(\mathbf{x})=0$, then $\mathbf{x}$ is a critical point.
Thus,
\begin{align*}
f_x(x,y) &= 2x+\beta y + 1 = 0\\
f_y(x,y) &= \beta x + 2y + 2 = 0
\end{align*}
Solving for $x$ and $y$, we find that:
\begin{align*}
x = \frac{\begin{array}{|cc|}
-1 & \beta \\
-2 & 2
\end{array}}{\begin{array}{|cc|}
2 & \beta \\
\beta & 2
\end{array}}=\frac{-2+2\beta}{4-\beta^2}=\frac{2\beta-2}{4 -\beta^2}
\end{align*}
\begin{align*}
y = \frac{\begin{array}{|cc|}
2 & -1 \\
\beta & -2
\end{array}}{\begin{array}{|cc|}
2 & \beta \\
\beta & 2
\end{array}}=\frac{-4+\beta}{4-\beta^2}=\frac{\beta -4}{4 - \beta^2}
\end{align*}
The second order necessary and sufficient conditions for optimality are based on the sign of the quadratic form $Q(\mathbf{h})=\mathbf{h}^T \cdot Hf(\mathbf{a}) \cdot \mathbf{h}$.
The Hessian of $f$ is given by,
$$Hf(\mathbf{x})=\begin{array}{|c c|}
2 & \beta \\
\beta & 2
\end{array}$$
Thus, $d_1 = 2 > 0$ and $d_2 = 4 - \beta^2$. Thus, $f$ has a local minimizer if and only if $4 - \beta^2 > 0$. $g(\beta) = 4 - \beta^2$ is a downward facing parabola. So, the values of this expression positive, if and only if $-2 < \beta < 2$. The function $f$ has no global maximum.
Question. How do I find the actual global minima?


texts: Problem with AI gym from Google Collaboratory

texts:
I am a neophyte trying to learn and nowadays I am studying Raschka's 3rd edition Python Machine Learning.
I am currently using Google Collaboratory and I am having an error when executing the gridworld_env.py file
Here is my code.
I have successfully found the method to execute it. I loaded it to the notebook environment:

and execute it with:
!python gridworld_env.py

but I am getting the following error:
    Traceback (most recent call last):
    File "gridworld_env.py", line 25, in <module>
    from gym.envs.classic_control import rendering
    File "/usr/local/lib/python3.6/dist-packages/gym/envs/classic_control/rendering.py", line 25, in
    <module>
    from pyglet.gl import *
    File "/usr/local/lib/python3.6/dist-packages/pyglet/gl/__init__.py", line 244, in <module>
    import pyglet.window
    File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 1880, in <module>
    gl._create_shadow_window()
   File "/usr/local/lib/python3.6/dist-packages/pyglet/gl/__init__.py", line 220, in
   _create_shadow_window
   _shadow_window = Window(width=1, height=1, visible=False)
   File "/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/__init__.py", line 165, in
   __init__
  super(XlibWindow, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 570, in __init__
  display = pyglet.canvas.get_display()
  File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/__init__.py", line 94, in get_display
  return Display()
  File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/xlib.py", line 123, in __init__
  raise NoSuchDisplayException('Cannot connect to "%s"' % name)
  pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None"

I have searched in the web but until now I have not been able to find how to fix it. Thanks if anybody can send me a tip for solving it.


texts: Deploying models on bigdata platforms like Hadoop and Spark

texts:
Where exactly bigdata platforms fit in to a data science/ machine learning projects ?

Say I have a large dataset for a binary classification problem - cats and dogs.

Now I need to create a model for real time classification

Here is my question.

1 Since dataset is huge I can make use any distributed platform for faster computation and model creation right?

2 Once the model is ready, then there is no need of these distributed platform right ? or are they needed for feature extractions ?