texts: plotting histogram together with variance

texts:
I have a small piece of code that print values gotten from a csv file into a histogram. This is all done using matplotlib library

dev_x= X     #this is where my integers are stored
plt.hist(dev_x, bins=7)

#var=np.var(dev_x)
#print(np.mean(dev_x))
#plt.axhline(var)

plt.xlabel('Values')
plt.ylabel('Quatity')
plt.title('Graph')

When the following lines are commented, I get the result

However, when I uncomment these lines

var=np.var(dev_x)
print(np.mean(dev_x))
plt.axhline(var)

I get only this graph

The code aims to show the histagram of the data and then the mean and the variance horizontal .


texts: First two principal components explain 100% variance of data set with 300 features

texts:
I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans.
My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and lemmatized
I create a data set of about 1.2 million tweet vectors (300 features each) by taking the averaged word vectors multiplied by their tfidf scores, like so:
# trained with same corpus as tfidf
# size=300, epochs=5, and min_count=10
tweet_w2v = Word2Vec.load('./models/tweet2vec_lemmatized_trained.model')

tweet_tfidf = TfidfVectorizer()
with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    tweet_tfidf.fit(infile)

tweet_tfidf_dict = dict(zip(tweet_tfidf.get_feature_names(), list(tweet_tfidf.idf_)))

tfidf_tweet_vectors = []

with open('./corpus/ttokens_doc_lemmatized.txt', 'r') as infile:
    for line in infile:
        word_vecs = []

        words = line.replace('\n', '').split(' ')

        if len(words) == 0:
            continue

        for word in words:
            try:
                word_vec = tweet_w2v.wv[word]
                word_weight = tweet_tfidf_dict[word]
                word_vecs.append(word_vec * word_weight)
            except KeyError:
                continue

        if len(word_vecs) != 0:
            tweet_vec = np.average(np.array(word_vecs), axis=0)
        else:
            continue
        tfidf_tweet_vectors.append(tweet_vec)

I also tried the above code with just average tweet vectors (no tfidf), and my problem still ended up happening.
I am starting to think that maybe my data set just isn't big enough or I am not training my word2vec model properly? I have somewhere around 100 million tweets I can use, but after filtering out retweets and only getting english language, it comes to around 1.3 million.
I'm not sure what's happening and what step I should take next. Any explanation is appreciated.
# Load in the data
df = pd.read_csv('./models/tfidf_weighted_tweet_vectors.csv')
df.drop(df.columns[0], axis=1, inplace=True)

# Standardize the data to have a mean of ~0 and a variance of 1
X_std = StandardScaler().fit_transform(df)

# Create a PCA instance: pca
pca = PCA(n_components=20)
principalComponents = pca.fit_transform(X_std)

# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)


texts: Unable to use readline() for function() input

texts:
I recently started learning R and was trying to make a script that would give specific values of a probability distribution. Ideally, in this script, there would be a list of probability distributions with an associated number. Users would be able to enter a number and it would print out a message while giving them a bunch of variables relevant to the specific probability distribution to enter.

This is my code:

DistriName <- readline(prompt = "Enter Distribution: ")
DistriName <- as.numeric(DistriName)
function(DistriName) {
  if(DistriName = 1) {
    print("You have selected the binomial distribution")
  } else {
    print("I am not familiar with the distribution. Sorry.")
  }
}

For some reason, it keeps giving me this error:

Error: unexpected '}' in "  }"
Error: unexpected '}' in "}"

I have checked multiple times for any missing opening brackets, but could not find anything.


texts: Can anyone explain me the basic difference between bagging and boosting and which technique can be used in which scenario?


texts: Output landscape of ReLU, Swish and Mish

texts:
I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681).

I understand that this figure describes how the loss is being changed, if the change is smooth or not. But the paper don't describe how it was generated, what does the color on the figure mean (no colorbar), or what is the plotted along x and y axis. Can anyone help in this regard?


texts: How do I perform K-Means clustering of the Olivetti Dataset

texts:
This Question pertains to Matrix Factorization and the full question is given below.
Provide for k-means clustering of the Olivetti dataset the following visualizations:

A scatter plot of the r = 2-dimensional representation of the faces in the latent space, together with the faces which represent the two dimensions/features.
A visualization of the faces which define the features of the latent space for r = 5.
The reconstruction of the faces with indices i ∈ {0, 10, 20} when using a rank of r ∈ {5, 25, 50, 100}

To load the Olivetti Dataset run the following code:

from sklearn.datasets import fetch_olivetti_faces
faces = fetch_olivetti_faces()

And print(faces.DESCR) provides a description of the data set.

When I try to K-Means clustering on the model I receive

**TypeError: float() argument must be a string or a number, not 'Bunch'**

Thank you for your time and consideration


texts: Should the minimum value of a cost (loss) function be equal to zero?

texts:
We know optimization techniques search in the space of all
the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or Categorical Cross Entropy, has a global minimum value equal to zero, in the ideal case.

For example, the Gradient Descent, $\theta_j \leftarrow \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$, updates parameters based on the derivation of the calculated cost function value, $J(\theta)$.

I was wondering what will happen if we design a cost function that has a non-zero global minimum in its ideal case. Does it make a difference, e.g. in the convergence rate or other aspects of the optimization process, or not?


texts: How to measure execution time on distributed system

texts:
I'm planning to run experiments with large datasets on distributed system in order to evaluate efficiency gains in comparison with previous proposals.

I have limited number of machines nearly ten machines having 200 GB of free space on hard disk on each. On the contrary, I wished to perform experiments on more than available nodes in order to measure scalability, more precisely. Since I don't have any, I thought about using a commodity cluster. However, I'm not sure about the policies of usage, and I need to reliably measure execution times.

Are there commodity services which will grant me that only my application would be running at a given time?


texts: Find recurrent dates in a small set (and get rid of non recurrent ones)

texts:
I need help in the analyse of a categorization problem.

Given a set of dates (small set: 20 elements maximum), I would like to group dates which are equally distributed (with a tolerance).
It can be, for instance, monthly or weekly separated dates.

Here is an example. Given this repartition :

I would like to categorize into these two groups:

The problem is that I am a developer, not a data scientist. I have an intuition that it should be possible to do a kind of regression.

I have no clue how to analyse this problem. Can you help me with that, please ?

Cheers

PS: I have already seen this thread (Recurring events - finding in a time series) but I have not helped me.


texts: Potential-based reward shaping in DQN reinforcement learning

texts:
I work for quite some time on a RL task which poses a surprising difficulty to the reinforcement learning agent to learn.

My technique is based on the Double DQN, with replay buffer, using recurrent NNs to evaluate Q, with e-greedy action selection (0.1 probability for taking a random action).

The most difficulty goes from the fact that the agent has to make quite a series of right actions in order to stumble upon a positive sparse reward. The number of actions it should make is just about 5 (and the action vocabulary contains just 3 actions), but it should make them in a specific environmental state, which also complicates things.

As a result, replay buffer can have zero positive rewards per 1024 visited states (very sparse).

I tried to shape intermediate rewards based on potential function, in order to inject some positive traits into the buffer.

My question is rather about intuition. Could you say the agent got stuck in a suboptimal policy?

From left to right:

Q values of selected actions
L2 loss
average reward at the end of a trajectory.

Although the Q values and (not shown here) the moving sum of rewards stay positive, the final rewards as trajectory ends cannot break through the zero level (right pic). Anything above the zero level is a good result, and if it is above 1, the result is fantastically perfect.

My quess has been that the agent had found a 'loophole' for keeping the sum of intermediate and final rewards at maximum, while the final rewards are in fact negative.


texts:
Is there a go to source for searching for data sources? Like a way to ask google, specifically to show data files?

Specifically, I'm looking for data on the number of smart phones and tablets sold in countries around the world.

Any ideas where I can look?


texts: Generalization of Correlation Coefficient

texts: The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the general predictive power of x over y when the relationship between x and y is not linear?


texts: Which order is correct Feature Selection then Outlier Detection or vice versa?

texts:
which of these orders is correct?

First (Feature Selection) Second (Outlier Detection)

or

First (Outlier Detection) Second (Feature Selection)


texts: How to calculate events per day in R including dates when no events occurred?

texts:
I would like to create a data frame in which in the first column I will have all the dates from a certain period of time and in the second the number of events that occurred on each date including dates when no events occurred. I would also like to count the events to which specific factors have been assigned

The first data frame in which I have the events with dates for a given date:

Row Sex Age      Date
1    2   36   2004-01-05
2    1   47   2004-01-06
3    1   26   2004-01-10
4    2   23   2004-01-20
5    1   50   2004-01-27
6    2   35   2004-01-28
7    1   35   2004-01-30
8    1   38   2004-02-06
9    2   29   2004-02-11

Where in the column "Sex" 1 means female and 2 male.

Second data frame in which I have dates from the examined period:

Row    Date
 1  2004-01-05
 2  2004-01-06
 3  2004-01-07
 4  2004-01-08
 5  2004-01-09
 6  2004-01-10
 7  2004-01-11
 8  2004-01-12
 9  2004-01-13
10  2004-01-14

I want to get a data frame that looks like this:

Row    Date       Events (All)   Events (Female)   Events (Male)
 1  2004-01-05         1                0                1
 2  2004-01-06         1                1                0
 3  2004-01-07         0                0                0
 4  2004-01-08         0                0                0
 5  2004-01-09         0                0                0
 6  2004-01-10         0                1                0
 7  2004-01-11         0                0                0
 8  2004-01-12         0                0                0
 9  2004-01-13         0                0                0
10  2004-01-14         0                0                0

Could anyone help?


texts: How to predict probabilities in xgboost?

texts:
The below predict function is giving -ve values as well so it cannot be probabilities.

param <- list(max.depth = 5, eta = 0.01,  objective="binary:logistic",subsample=0.9)
bst <- xgboost(param, data = x_mat, label = y_mat,nround = 3000)

pred_s <- predict(bst, x_mat_s2)

I google & tried pred_s <- predict(bst, x_mat_s2,type="response")
but it didn't work.

Question

How to predict probabilities instead?


texts: In a neural network, is it possible to gradient descent with more than one input?

texts:
I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a program to do backpropagation just do understand it more deeply. In linear/logistic regression, it makes sense to do gradient descent on the average of the costs through multiple inputs, and outputs, because the there's just one layer of weights, and the inputs directly affect the outputs.
In case of neural networks, we get back a layer of activations (outputs), and we have the expected outputs with matching shape, so we're getting the costs, by subtracting the expected output, with our actual output, and we're propagating this back with the chain rule. But this way we have to compare our costs with the activations of the neurons, which are unique, and dependent on the inputs. So even if we would take a bunch of inputs, get their costs, and average the costs, how could we decide which neuron activation layer should it be compared to?


texts: Studying social media platforms - key data analysis methods?

texts: I am seeking a basic list of key data analysis methods used for studying social media platforms online. Are there such key methods, or does this process generally vary according to topic? And is there a standard order in which these methods are applied?(The particular context I'm interested in is how the news is impacting on social media)


texts:
I'd like to be able to estimate whether a proposed model is small enough to be trained on a GPU with a given amount of memory

If I have a simple CNN architecture like this:

Input: 50x50x3
C1: 32 3x3 kernels, with padding (I guess in reality theyre actually 3x3x3 given the input depth?)
P1: 2x2 with stride 2
C2: 64 3x3 kernels, with padding
P2: 2x2 with stride 2
FC: 500 neurons
Output: softmax 10 classes
Mini batch size of 64

Assuming 32bit floating point values, how do you calculate the memory cost of each layer of the network during training? and then the total memory required to train such a model?


texts: How to find the driver features towards a particular result in Classification problems

texts: In a classifier model, we can predict the outcome class, but here I need to find out the features that drive towards a particular result in a classification problem, that are a strong indicator of a particular result eg: driver features for loan default or features indicating for a successful sale and so on. Is it the same as finding the important features for the classifier model, if not, how do I proceed towards this. Also which model could be the most useful in such a case. Thanks in advance.


texts: Identifying “clusters” or “groups” in a matrix

texts:
I have a matrix that is populated with discrete elements, and I need to cluster them (using R) into intact groups. So, for example, take this matrix:

[A B B C A]
[A A B A A]
[A B B C C]
[A A A A A]

There would be two separate clusters for A, two separate clusters for C, and one cluster for B.

The output I'm looking for would ideally assign a unique ID to each cluster, something like this:

[1 2 2 3 4]
[1 1 2 4 4]
[1 2 2 5 5]
[1 1 1 1 1]

Right now I wrote a code that does this recursively by just iteratively checking nearest neighbor, but it quickly overflows when the matrix gets large (i.e., 100x100).

Is there a built in function in R that can do this? I looked into raster and image processing, but no luck. I'm convinced it must be out there.


texts: Output of machine learning classification prediction on out-of-sample data has too few observations

texts:
Using the mlr package, I have developed a task using word frequency data to categorize tweets into two categories (TRUE and FALSE). Now I am using this task to classify out-of-sample tweets. Sorry, I cannot share my data, but I will show my code here:
task = makeClassifTask(data = train, target = "category")
mod = train("classif.randomForest", task)

newdata.pred = predict(mod, newdata = outofsample)
newdata.pred

My output is shown here:
Prediction: 17981 observations
predict.type: response
threshold:
time: 0.29
... (#rows: 17981, #cols: 1)

And as a dataframe:
response
<fctr>
1   TRUE
2   FALSE
3   FALSE
4   FALSE
5   FALSE
6   FALSE

I now want to use my categorization to remove any tweets falling in the "FALSE" category. But if I have 17981 tweets, why do I only see 6 observations? I cannot find anything wrong with the "train" dataframe or "outofsample" dataframe (they both have the appropriate number of observations and are listed as dataframes in the global environment, but I did notice that the object "task" creates a list of 6. Is this just a coincidence? How do I retrieve the classification for all of my tweets? And how do I link this information to the out of sample dataset to delete tweets by ID number, if this classification method does not include ID number?
Please don't judge me too harshly, I'm very new to machine learning and R in general. Any advice would be a big help.


texts: Cannot fig out error in my gradient function implementation in python

texts:
Im trying to implement following gradient descent function in Python for logistic regression:

$∇θ(−logL)=−X^T (y−e^{Xθ})$

This is my python implementation:

def gradient(X, y, theta):
    dtheta = -(np.dot(X.T,y - np.exp(X * theta)))
    return dtheta

X is a dataframe of size: (2458, 31),
y is a dataframe of size: (2458, 1)
theta is dataframe of size: (2458,1)

when i pass values to my gradient descent function, it returns a dtheta parameter with size (31,31) due to which i cannot update my theta to pass it to cost function, i cannot fig out where im going wrong.
any help will be appreciated.

Error i keep getting is: ValueError: operands could not be broadcast together with shapes (2458,1) (31,31)

and this is how im implementing the algorithm:

theta = np.random.uniform(low=-0.1,high=0.1, size=(2458,1))
# Iterate and update theta by using the gradient of the negative log-likelihood
max_iter = 100
learning_rate = 1e-3
for i in range(max_iter):
    # Calculate the gradient
    dtheta = gradient(X,y,theta)

    # Update theta

    theta = (theta - learning_rate) * dtheta

    # Calculate the value of the log-likelihood
    cost = negative_loglikelihood(X,y,theta)

    # Print iteration
    print("Iteration %d, cost function %.3f" % (i+1,cost))


texts: Getting Error in RStudio while loading a package- 'tm'

texts:
I've installed the R package tm. Now I am trying to load it, but it's giving me this error and warning everytime, I try.

The package is in this location-

‘C:/Users/Google/Documents/R/win-library/3.3’

Error Message--

Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) :
  there is no package called ‘slam’
In addition: Warning message:
package ‘tm’ was built under R version 3.3.1
Error: package or namespace load failed for ‘tm’

Any thoughts?


texts:
I'm writing a demo code to predict a 2-class classification for a dataset of 10-D inputs. Below, function  _data generates the data:
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

def _data(dimension, num_examples):
    # Create a simulated 10-dimensional training dataset consisting of 1000 labeled
    # examples, of which 800 are labeled correctly and 200 are mislabeled.
    num_mislabeled_examples = 20
    # We will constrain the recall to be at least 90%.
    recall_lower_bound = 0.9

    # Create random "ground truth" parameters for a linear model.
    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    # Generate a random set of features for each example.
    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    # Compute the labels from these features given the ground truth linear model.
    labels = (np.matmul(features, ground_truth_weights) >
              ground_truth_threshold).astype(np.float32)
    # Add noise by randomly flipping num_mislabeled_examples labels.
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

The code below shows my attempt, where predictor is the model and the loss function is chosen to be Hinge loss.
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

dim = 10
N = 100
target, features = _data(dim, N)

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.f_1 = nn.Linear(10, 1)

    def forward(self, features):
        return self.f_1(features)

model = predictor()

optimizer = optim.Adam(model.parameters(), lr=1e-2)
loss = torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

running_loss = 0
for _ in range(1000):
    optimizer.zero_grad()
    output = model(features)
    objective = loss(output, target)
    objective.backward()
    running_loss += objective.item()
    optimizer.step()

    print(running_loss)

My questions:

I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.
I was trying to implement my predictor without using nn.Linear by defining the computations myself as:

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)

    def forward(self, features):
        return torch.matmul(self.weights, features) - self.threshold

but in the optimization process,
model = predictor()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

I get the following error:

ValueError: optimizer got an empty parameter list

I would appreciate direction or comments on how to fix these issues. Thanks.


texts:
Problem
In Keras for Python, I have to use multiple lines of code for a simple XOR neural network:
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD
import numpy as np

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([[0],[1],[1],[0]])

model = Sequential()
model.add(Dense(8, input_dim=2))
model.add(Activation('tanh'))
model.add(Dense(1))
model.add(Activation('sigmoid'))

sgd = SGD(lr=0.1); model.compile(loss='binary_crossentropy', optimizer=sgd)

model.fit(X, y, show_accuracy=True, batch_size=1, nb_epoch=1000)
print(model.predict_proba(X))

Although the network above is quite small, the current implementation may become frustrating with deeper networks.
Question
Is there a built-in shortcut in Keras that quickly adds multiple layers?
For example:
[2,3,4,5] would be a network with 2 input neurons, 3 neurons in the first hidden layer, 4 neurons in the second hidden layer, and 5 output neurons. I use the term built-in because otherwise I can loop through the example array, and add a layer for each element.


texts: How does sklearn's tf-idf vectorizer pick the bigrams and trigrams?

texts:
class sklearn.feature_extraction.text.TfidfVectorizer(
*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None,
lowercase=True, preprocessor=None, tokenizer=None, analyzer='word',
stop_words=None, token_pattern='(?u)\b\w\w+\b', **ngram_range=(1, 1)**,
**max_df=1.0**, **min_df=1**, max_features=None, vocabulary=None, binary=False,
dtype=<class 'numpy.float64'>, norm='l2', use_idf=True, smooth_idf=True,
sublinear_tf=False)

These are the hyperparameters of sklearn's TF-IDF vectorizer.
The ngram_range parameter takes in the possible range of ngrams to extract from the given documents.
I couldn't help but wonder how does the vectorizer come up with the possible ngrams (say bigrams or trigrams) for the documents supplied as word vector features. Does it use only min_df and max_df to come up with them or is there something deeper than what meets the eye going on underneath?


texts:
Hi downloaded the MNIST dataset images and labels and I am trying to train but I am getting low accuracy which is very low . not Even 50 %.

below is my code.

import glob
import imageio
import cv2
import pandas as pd
import numpy as np
images=[]
for image_path in glob.glob(r"C:\\Users\\Downloads\\tmnist_bundle_rgb\\tmnist_bundle_rgb\\imgs\\*.png"):
    im = imageio.imread(image_path)
    a= cv2.imread(image_path)
    gray_img = cv2.cvtColor(a, cv2.COLOR_BGR2GRAY)
    images.append(gray_img)

images = np.asarray(images,dtype=np.float32)

lables = pd.read_csv('C:\\Users\\Downloads\\tmnist_bundle_rgb\\tmnist_bundle_rgb\\index.csv',encoding='utf-8')

lables=lables.iloc[:, 0] # slicing the column

lables = np.asarray(lables) #Array

lables = np.asarray(lables,dtype=np.float32)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(images, lables, test_size=0.1) # split

X_train = X_train.reshape(54252,28,28,1)
X_test = X_test.reshape(6028,28,28,1)

X_train = X_train/ 255
X_test = X_test/ 255

from keras.utils import to_categorical

number_of_classes=10
y_train = to_categorical(y_train,number_of_classes)
y_test = to_categorical(y_test,number_of_classes)

#Model Buidling
from keras.models import Sequential
from keras.layers import Dense, Convolution2D, Flatten
from keras.layers.convolutional import MaxPooling2D
from keras.layers import Dropout

model = Sequential()
model = Sequential()
model.add(Convolution2D(32, (3, 3), activation='relu', input_shape=(28,28,1)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(X_train, y_train,batch_size=32, nb_epoch=5, verbose=1

Output:

  Epoch 1/5 54252/54252 [==============================] - 36s
  663us/step - loss: 2.3024 - acc: 0.1093

  Epoch 2/5 54252/54252 [==============================] - 36s
  664us/step - loss: 2.3017 - acc: 0.1111

  Epoch 3/5 54252/54252 [==============================] - 39s
  721us/step - loss: 2.3016 - acc: 0.1115

  Epoch 4/5 54252/54252 [==============================] - 40s
  733us/step - loss: 2.3018 - acc: 0.1110

  Epoch 5/5 54252/54252 [==============================] - 50s
  912us/step - loss: 2.3015 - acc: 0.1115


texts: Feature Engineering of mixed data type column

texts:
I have a data set in which I have to predict the price of a building. Among many features there is a feature called Availability which has two type values like :

"Ready To Move" and a date (e.g. "20-Jan", "18-May", etc.)

For example:

Availability           Price
------------           -----
Ready To Move           55
18-Apr                  38

How would you engineer this column to find its relationship with price and to draw other conclusions.


texts: Given rankings and the factors that determined the rankings, what is the best method to reverse engineer the rankings?

texts:
I have a list of ranked items and the factors that the group used to create the rankings (like college rankings: 1, 2, 3...), and I'd like to reverse engineer their methodology.

Here's what I tried...

I took the standard score of each factor (factor value - mean /
standard deviation).
I assigned coefficients to each factor and added up all these products to create a score.
I used the scores to rank the items and compared these rankings to the real rankings, trying to minimize the difference.
Rinse and repeat by changing the coefficients.

This is a crude method and I am very new to this.  I was wondering what the best way is to figure out how this group weighted the factors to create the rankings.


texts: Can gradient boosted trees fit any function?

texts:
For neural networks we have the universal approximation theorem which states that neural networks can approximate any continuous function on a compact subset of $R^n$.

Is there a similar result for gradient boosted trees? It seems reasonable since you can keep adding more branches, but I cannot find any formal discussion of the subject.

EDIT: My question seems very similar to
Can regression trees predict continuously?, though maybe not asking exactly the same thing. But see that question for relevant discussion.


texts: Detecting spammers with artificially generated target class

texts:
I have a bunch of data related to my company's incoming and outgoing calls, and I'm trying to detect spam callers in order to blacklist them. The features I've extracted from the data are a bit simple right now:

Caller Number: Who is calling
N_Unique: The amount of different phones called
N_Answered: The amount of times the call was connected
N_Missed: The amount of times the call was ignored/missed
Avg_Duration: The average duration answered calls lasted

Initially my superior sent me the dataset with a field called "IsSpammer", which was binary, and after a very simple Random Forest (1000 trees, default scikit-learn everything else) I got a 100% accuracy even after cross-validation. I took a closer look at the data and realized anyone who called more than 20 different numbers inside the company or had average call duration less than 20 seconds where flagged as spammers (which may be true), but then I asked supervisor how this field was generated, and he told me he set it under that rule the random forest discovered.

I then realized every ML problem I've faced came from a Kaggle contest that gave me a clear field to predict, and a training set that had "correct" information on the fields to predict. I don't know how to proceed with ML if I don't have the information of Spammer/NotSpammer, since apparently if I come up with a clever (or not) rule, then the best I can hope to discover is that rule I'm using in the first place... I'm not sure if getting more features and making and obscure rule using all of them would help. Obviously I lack experience in this, so I figured I would ask.

Is this a problem that could benefit from using machine learning?


texts: shapes recognition for shapes formed by dotted points

texts: In the image, below right corner there is circle shape and triangle shape made with set of co-ordninates.I have searched extensively in the net for algorithm or approach to classify shape correctly .And yet came up with nothing.I need approach or directions to solve this problem.


texts: Measuring stability of data and determining minimum size of dataset

texts:
I come from a different field so please bear with me.
I have a list of users using a certain application, after each usage certain data is collected about how the session went from a network perspective. The goal is to predict how the user's network will behave in the future by looking at stats from previous sessions and using it classify their network into some class from 1-4. This will be done using a very simple heuristic, by just taking the average of a certain network metric over the past N session.
The question(s) is:

What's the best way to measure the stability of this network metric to make sure it's within a given range?
How can one choose N such that the population size is maximized while still being confident to some extent about the classification that is being made.


texts:
Diff = [i - j for i,j in zip(text_features, author_signature)]

Diff is a List ,
text_features    =  [1, 2, 3] ,
author_signature =  [3, 2, 1]


texts: Keras: Dimension Error with Sparse Categorical Crossentropy

texts:
I'm trying to make an NN that, given the time on the clock, would try to predict which class (out of 32 in this example) is making a request to the system. As a first attempt, I've tried to use categorical_crossentropy, but this will obviously not work because the targets are very sparse, so the system will be heavily rewarded by just always predicting the non-requests.

Now I'm trying to use sparse_categorical_crossentropy, but I keep getting a dimension mismatch error (the train and test sets are the same in this case because I just wanted to evaluate performace in the training set at first):

Error when checking target: expected dense_90 to have shape (1,) but got array with shape (32,)

The DataFrame is here (a simple clock and another column for the requests) and the code is:

from keras.models import Sequential
from keras.layers import Dense
from keras.utils import to_categorical
import tensorflow as tf

requests = df['requests'].values
requests_cat = to_categorical(requests, 32)

length = len(df['clock'])
train = np.reshape(df['clock'].values, (length, 1))
train = train.astype(np.int)
target = requests_cat

model = Sequential()
model.add(Dense(25, activation = 'relu', input_shape = (train.shape[1],)))
model.add(Dense(25, activation = 'relu'))
model.add(Dense(32, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy')
model.fit(x = train, y = target, epochs = 100, validation_data = (train, target))

On a sidenote:

This architecture doesn't seem to be the best in this case. As a second prototype I was thinking about doing something with an LSTM, since past requests can affect later ones. Is there a standard architecture for scheduling?
What would be the proper way of splitting sparse sets into training and testing ones?


texts: Can you reuse observations from your train data in your final test data?

texts:
For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.

Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.

I am using XGboost implementation.  Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.

I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :

df2 <- df1[sample(nrow(df1)),]

In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical.

Is it because, a large portion of my final test data already learned from the observations also found in my training data?

If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?


texts: Clustering and graphing similarities of sentence subjects

texts:
I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject.

Ex. "I love reading math books" Subjects for the above sentence = [Math: 10, Reading: 14, Romance: 1, Leisure: 4]

Now I would like to create a graph of nodes, where each node is a sentence and place these nodes at the origin in a 2D plane. Each subject forms the circumference of a circle surrounding the nodes. The "closeness" of the nodes to their respective subjects is represented by their positions in the 2D plane. I figured I could do this by taking each score for a subject of a sentence and apply it as a vector. Then add all the vectors, for all the subjects, together to settle on a final position

The resulting plane could look like this

The idea here is that we can now bind each sentence node to each other with edges to create a proximity graph. Using a Gabriel graph, we can only bind the closest nodes together

The entire goal here is to construct a script from a bunch of sentences where we can flow down the script going from subject to subject without too much discontinuity.

As you can see I already have a method that seems to make sense. I was wondering however if there was already a set of methods for doing this kind of thing in data science. I was looking into spectral clustering, and similarity measure. I even looked into bioinformatics and found Needleman–Wunsch algorithm and Smith–Waterman algorithm. But I'm not knowledgable in data science or bioinformatics. Can I get some directions as to where I should be headed to solve this kind of problem. Is there already an established set of tools and methods for accomplishing it?


texts: How to develop Top-N recommendation for evaluating my system

texts:
I want to evaluate my recommender system with the Top-N recommendation method and I have a problem. In some situations, e.g. N = 5, I don't have 5 items for listing and I cannot do that for evaluating. now, you suppose, I want to increase N from 5 to 10 or 15, I think I will have some problems with some active users.
Would you please give me a suggestion?!!!


texts: Best neural-network loss function for multiple output when order doesn't matter?

texts: I have a problem where my network is to output 3 values, and they are supposed to match three target values, but I don't care about the ordering. At present, I created a loss function which computes all 6 possible ways to match them up and returns the lowest.  Is this the best approach?  I could see this being a confusing surface for the optimizer to navigate.  Another option would be to define a non-arbitrary order (e.g. lowest to highest) and require the network to always output that way, although in my case the values are 2D coordinates, so the ordering would be slightly more abstract, perhaps distance from origin.


texts: Which cross-validation type best suits to binary classification problem

texts:
Data set looks like:

25000 observations
up to 15 predictors of different types: numeric, multi-class categorical, binary
target variable is binary

Which cross validation method is typical for this type of problems?

By default I'm using K-Fold. How many folds is enough in this case? (One of the models I use is random forest, which is time consuming...)


texts: Choosing a suitable learning rate based on validation or testing accuracy?

texts:
I have simulated a neural network with different learning rate, ranging from 0.00001 to 0.1, and recording each test and validation accuracy. The result i obtained is as below. There is 50 epoch for each learning rate, and i note down the validation accuracy at the last epoch, while the training accuracy is computed throughout the process.

Learning rate: 0.00001

Testing accuracy: 0.5850

Validation accuracy at final epoch: 0.5950

Learning rate: 0.0001

Testing accuracy:0.6550

Validation accuracy at final epoch: 0.6400

Learning rate: 0.001

Testing accuracy: 0.6350

Validation accuracy at final epoch: 0.6900

Learning rate: 0.01

Testing accuracy: 0.6650

Validation accuracy at final epoch: 0.6700

Learning rate: 0.1

Testing accuracy: 0.2500

Validation accuracy at final epoch: 0.2100

How does testing and validation accuracy influence which learning rate is better? Would a higher validation accuracy determine the most suitable learning rate for the model?

Hence, is it correct that 0.001 is the most suitable learning parameter since it has the highest validation accuracy at the last epoch?


texts: Is any feed-forward neural network a composable function?

texts:
I am wondering if each feed-forward neural network can be written as a composition of the layer functions.
So if $F(x)$ is the output of a neural network F, do always exists layer functions $f_1$,...,$f_n$ with
$F(x)$ = ($f_1$ o .. o $f_n$)$(x)$? Here "o" denotes the composition symbol.
If the network consists only of layers where each layer is connected to exactly one previous layer, this is obvious. However, this is not always the case, e.g. if you have skip-connections.
So what about the "concat" layer and the "sum" layer?
So, what if $F(x)$ = $G_1(x) + G_2(x)$, or $F(x) = [H_1(x), H_2(x)]$, for some layer functions $G_1$,$G_2$,$H_1$,$H_2$?
Can $F$ still be written as a composition?


texts: Which algorithm should I choose and why?

texts:
My friend was reading a textbook and had this question:
Suppose that you observe $(X_1,Y_1),...,(X_{100}Y_{100})$, which you assume to be i.i.d. copies of a random pair $(X,Y)$ taking values in $\mathbb{R}^2 \times \{1,2\}$. Your plot the data and see the following:
where black circles represent those $X_i$ with $Y_i=1$ and the red triangles represent those $X_i$ with $Y_i=2$. A practitioner tells you that their misclassification costs are equal, $c_1 = c_2 = 1$, and
would like advice on which algorithm to use for prediction. Given the options:

Linear discriminant analysis;
K-Nearest neighbours with $K=5$
K-Nearest neighbours with $K=90$.

What would be the best algorithm for this? I think it should be $5$, as the bigger the $K$, the worse the accuracy gets? What would be your choice and why?


texts:
I am training an SVM on UCI's Bank Marketing Data Set, the bank additional-full.csv. As the data is skewed I am also interested in recall. I am getting accuracy of about 87.95% but my recall is around 51%. I want to know ways to increase recall without decreasing accuracy so much using SVM only.

My code:

from sklearn.svm import SVC

svm_clf = SVC(gamma="auto",class_weight={1: 2.6})
svm_clf.fit(X_transformed, y_train_binary.ravel())

Additional info:

I have not created any new feature (i.e combining features) and considered unknown as label.
I have also removed Duration attribute as suggested by attribute information

I have tried different class_weights, so I can increase recall upto 75.32% but then my accuracy drops to 68%
How can I increase recall in SVM models without decreasing accuracy so much?


texts: Probability or score on how likely an article is motivated by content marketing

texts:
Given a webpage url and the extracted article text from the given page, I want to calculate a probability of the article to be content marketing.

I consider content marketing as differentiated from spam, because it is to admit, content marketing makes some serious efforts to deliver real information to the reader. But the true reason behind the effort of writing an article in the context of content marketing is to sooner or later guide the reader to a newsletter subscription, a product landingpage or to contact the author. In the language of content marketers this kind of qualified reader is considered a "lead".

1. call to actions

I could by using Knn or Bayes simply test the scentences in the article text against a training set of known CTAs.

Okay, this where NLP comes in and this is all nothing else than spam detection.

2. link structure

I possibly can find strong evidence on content marketing motivation, in the outgoing links of a certain page or article. If it links to a landing page styled product page (some heuristics to identify these kind of webpage are needed as well) and a lot of other sites of the website link to a "near by" page.

By "near by" I mean text similarity, optimized for a (and the same) keyword.

question
So what do you think about this? How could scoring look like. How should a probability model look like? Are there aspects, that I miss? Does anyone have some experiences in this field? Is a paper out there? Does anyone know some resources for this?


texts: Why N-pair Loss (NIPS 2016) stops minimizing in Image retrieval task?

texts:
Currently, I am using a Deep Learning model to build a search engine for retrieving images.

With a dataset of pairs of (image, description), I am using a joint model as shown in the figure below to generate embeddings for both image and description.

In the training time, the model must learn to generate similar embeddings for images and descriptions that are in the same pair, and differents embeddings for images and descriptions of different pairs. This is done using the N-Pair Loss (NIPS 2016) described below:

import torch
from torch import nn

class NPairsLoss(nn.Module):
    """
    The N-Pairs Loss.
    It measures the loss given predicted tensors x1, x2 both with shape [batch_size, hidden_size],
    and target tensor y which is the identity matrix with shape  [batch_size, batch_size].
    """

    def __init__(self):
        super(NPairsLoss, self).__init__()
        self.ce = nn.CrossEntropyLoss()

    def similarities(self, x1, x2):
        """
        Calculates the cosine similarity matrix for every pair (i, j),
        where i is an embedding from x1 and j is another embedding from x2.

        :param x1: a tensors with shape [batch_size, hidden_size].
        :param x2: a tensors with shape [batch_size, hidden_size].
        :return: the cosine similarity matrix with shape [batch_size, batch_size].
        """
        x1 = x1 / torch.norm(x1, dim=1, keepdim=True)
        x2 = x2 / torch.norm(x2, p=2, dim=1, keepdim=True)
        return torch.matmul(x1, x2.t())

    def forward(self, predict, target):
        """
        Computes the N-Pairs Loss between the target and predictions.
        :param predict: the prediction of the model,
        Contains the batches x1 (image embeddings) and x2 (description embeddings).
        :param target: the identity matrix with shape  [batch_size, batch_size].
        :return: N-Pairs Loss value.
        """
        x1, x2 = predict
        predict = self.similarities(x1, x2)
        # by construction, the probability distribution must be concentrated
        # on the diagonal of the similarities matrix.
        # so, Cross Entropy can be used to measure the loss.
        return self.ce(predict, target)

During the training process, the loss starts to decrease as expected, but for some reason, it gets stuck at a value (would it be a local minimum?) where all image embeddings are equal to all description embeddings. That is, the similarity matrix has all entries equal to 1.

While what is desirable is that the diagonal value of the similarity matrix was always greater than the other values outside the diagonal.

Could someone give me some clarification of what may be happening?


texts: Pandas - dataframe not storing images properly

texts:
I'm trying to organize image files I have into a Pandas dataframe. I'm using Keras's preprocessing library for working with them. Here's how I load them in:

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
filename_file = pd.DataFrame(columns = ['Filename', 'File'])
for filename in os.listdir(""):
    new_filename = filename
    if(filename[0] == '.'):
        new_filename = filename[2:]
        pic = load_img(new_filename)
        picture = img_to_array(pic)
    filename_file = filename_file.append({'Filename' : new_filename, 'File' : picture}, ignore_index = True)

My goal is to slice off the File column of the filename_file dataframe. I want its shape to be (11000, 350, 350, 3) because I have 11000 images, each one 350 by 350 pixels and 3 for RGB. The issue I'm having is that the dimensions become (11000, 1). How can I get the data in the format I want?


texts: Appropriate similarity measure that highlights symmetrical values of features

texts:
I am trying to find an appropriate distance measure that reflects the differences of the vectors seen in the image below:
The green vector is compared with the blue one and the orange one.

Most of the distance measures (like the Euclidean for example) would yield the same value despite the "phase shift". Therefore the straight line would look "the same" as the blue and the orange line!

The features are ordered but I am also interested in the case they were unordered.

I was also thinking to "split" the vectors in the middle and treat them as bags of two subvectors (like in a multi-instance setting) but then I would not know how to combine the two resulting distances in one.

So I guess my question is two-fold:

Is there a distance measure that highlights these type of value symmetries in a single instance setting?
Is there a way to combine multiple distances (in a multi-instance setting) in a way that symmetry (and therefore order) is highlighted in the final distance result?

Thank you


texts: How is Large BERT is less accurate than basic BERT?

texts:
I'm using BERT for text classification in this NLP competition.
When I use Basic BERT with 12 layers, 3 epochs, and 32 batch sizes, I get a training accuracy of about 0.84 and a val_accuracy of about 0.82. Though when I use the Larger BERT with 24 layers and the same number of epochs and batch sizes, I get a training accuracy of about 0.6 and a val_accuracy of about 0.55.
Why is this happening? Isn't the large BERT supposed to have more layers and parameters so that It can recognize the patterns better? Or maybe it can perform better in the long run with more epochs?
(Both BERTs are uncased, English based and pretrained and I'm using trainable = True in the bertlayer model.)


texts: How do we choose which feature(column) should we normalize or scale

texts:
I am a newcomer to machine learning, I was going through scaling and normalization. When I tried to explore most of the documentation on web, I found people explaining that the dataset needs to be scaled and normalized.
But wanted to understand when to scale or normalize a column? And how to choose which feature/column in a dataset for scaling and normalization.
And how to decide on which normalization algorithm to choose for a column?


texts:
I am working with the following data.

x <- 1:36
y <- c(6.62,6.57,6.57,6.55,6.52,6.56,6.52,6.55,6.54,6.55,6.55,6.62,6.78,6.73,6.6,6.59,6.57,6.57,6.51,6.55,6.39,6.4,6.15,6.22,6.3,6.27,6.27,6.24,6.24,6.23,6.22,6.26,6.23,6.22,6.2,6.14)

The values of y represent monthly totals spanning 3 years beginning with January. Relatively large increases of y tend to occur in January (at values of 1, 13, and 25 for x) and they fall gradually throughout the year. I would like to represent this behavior with an asymmetric sinusoidal wave that best fits the pattern. Per this thread, I experimented with the following R code.

mylm <- lm(y ~ x + sin(2*pi/24*x) + cos(2*pi/24*x))
plot(x, y)
lines(x, mylm$fitted.values, col = "blue")

The approximation is decent, but the symmetry of the fitted wave does not capture the sudden rises in January and the gradual declines across the remaining months.

I don't know the proper vocabulary to fully articulate what kind of wave I am seeking. Perhaps a smooth triangular wave? Thanks.


texts:
I'm trying to train an object detection model to detect some of playing cards for example it will detect 1,2,3, king, queen and jack.
And I'm making a class of not and I have put examples of other cards such as 4,5,6,7,8,9 but for me it doesn't make sense to make this additional class.
So how to solve the problem of detecting other kind of cards as an objects should I make this class or remove it and keep only the required classes?


texts:
I want to train a binary classification algorithm for spam detection using labeled data set of mentions from social media. These mentions have the following features:

  URL, text message, posting date, is deleted, author

where the author has its own fields:

  URL, nick, type (person or community), number of subscribers, registration date, last activity date

I have $\approx 13.5k$ observations with $\approx 450$ spam messages among them.

My suggestions:

transform data features in real numbers
binarize URLs by media sources (e.g. facebook, instagram, etc.)
use SVM with Gaussian kernel for all data

Questions: Am I on the right way? What else can I do? How can I process text data?

Thanks in advance for any suggestions!


texts: Techniques for Ordinal Classification/Regression with Gradient Boosted Trees

texts:
I did some research on how to run ordinal classification decision trees (such as lightgbm, xgboost), and found these articles to be helpful. Both use a k-1 binary classification technique to output ordinal responses.
https://www.hindawi.com/journals/jat/2019/7482138/
https://www.cs.waikato.ac.nz/~eibe/pubs/ordinal_tech_report.pdf
In the latter case, I believe the authors are applying their technique to a single tree. However, my concern with this approach is that algorithms like RF and GBT use multiple learners. The probabilities will most likely not add up to 1, so it doesn't give out true probabilities.
So, my question is:

Subjectively, how big of a concern would this be to you?
Do you have other techniques on how run an ordinal classifier using GBT?

Thank you!


texts: Why are NLP and Machine Learning communities interested in deep learning?

texts: I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.


texts: trying to decrease overfitting with regularisation in CNN

texts:
I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:

training steps: 20,000
learning rate: 0.075
test accuracy :72.9%
train accuracy for final iteration : 98.0%
val accuracy for final iteration : 75.0%

clearly overfitting, so I tried L2 regularization. Here are my parameters and results, for the highest accuracy until now :

training steps: 40,000
learning rate: 0.1
test accuracy :71.1%
alpha for L2 : 0.00075
train accuracy for final iteration : 91.0%
val accuracy for final iteration : 71.0%

Looked promising so I decided to go further with the following hoping to get better accuracy :

training steps: 80,000
learning rate: 0.075
alpha for L2 : 0.005

go the following graph :

Since, the graph was getting saturated so I quit at 50,000 steps.

My question :

Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?

Note : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes


texts: What is important for Pharmaceutical companies to answer with Big Data Analysis?

texts:
I am a data scientist, and I have some biological background (genetics). I have been asked to give a talk for our customers from pharmaceutical industry. I should show them how they benefit from Big data tools such as Spark to get value out of their data. I should do this talk with an example. Does anyone know what is the best example for pharmacutical industry? I mean what data/ challenges would be better to tackle?
Thanks


texts: Is it possible to use a generative model to "share" private data?

texts:
Let's say we have some data set, with lots of instances $X$ and a target $y$. If it is of some importance, you may assume that it is a "real life" data set : medium sized, with important correlations, y is unbalanced...etc.

Let's also say this data set is of relative interest : the field of research is rather active, but there is no (or only a few) available data set. So we are considering publishing our data set.

However, a direct publication wouldn't be possible for privacy concerns. Basic approaches have been considered (pseudonymisation, grouping instances to achieve statistical disclosure control, publishing an old dataset), but mostly set aside.

The research field concentrate on discriminative approaches for learning $y$, based on $X$. However, I recently came to read about generative approaches. So I am curious for (1) if there are generative techniques that could be used to share information about $X$x$y$ without being able to recognise any individual instances.

I think the usefullness of sharing a generative model (/the output of a generative model) to people that are trying to build discriminative models may be limited by the inherent performance of the generative model. Are there more to it than just calibrating a SOTA discriminative model twice (once on real data, once on generated data) and compare performances ? Wouldn't it be better to just share the SOTA algo calibrated on our dataset ?


texts: Replace values that starts with "<" in a data frame in r

texts:
I am new to R and would like some assistance on replacing multiple values in a large dataset with many variables to NA at once in r.
I want all the cells with "<0.01, <0.02, and many others with < sign at the beginning" to be changed to NA.

How do I do it at once? Here is a screenshot of the data.


texts: input transformation for polynomial regression

texts:
First I apologize if the question is not very clear as I'm new to this field. I'm doing a university project to create polynomial regression in python without any kind of libraries. In our class the professor told us you have to transform the values of higher dimension if you want to change a polynomial regression problem into a linear one. for example :

so at first I generate a description file which in each line has the elements to be chosen and last number is parameter to use. and then we have the input variables. All of this is clear to me. but I don't understand how to use the description with my transformed input (out.txt) numbers.

I would be really thankful if somebody could make it more clear to me :)


texts: what does the standard deviation plot around my learning curve indicate?

texts:
I plotted a learning curve below. There is a thick red band around the top portion of my training score. Why is it so high at the beginning?

Below is a snippet of the code used:
train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes, scoring= 'neg_brier_score')
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     # + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training brier score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation brier score")


texts: Problem to classify multilabel dataset while using random forest algorithm

texts:
I am using a large dataset with 4 different multilabel classes. I was trying to apply Random forest algorithm on those data set. After preparing data set I separate X (feature columns) and Y(feature class), split those data into train and testing data, fit training data to model, predict test data and was trying to find the accuracy by using testing data then I faced valueError: 'multiclass-multioutput is not supported'. My data set is:

train.head()

And here is my code after pre-processing:

#define X and y
feature_cols=['MFCCs_ 1','MFCCs_ 2','MFCCs_ 3','MFCCs_ 4','MFCCs_ 5','MFCCs_ 6','MFCCs_ 7','MFCCs_ 8','MFCCs_ 9', 'MFCCs_10', 'MFCCs_11', 'MFCCs_12', 'MFCCs_13','MFCCs_14','MFCCs_15','MFCCs_16','MFCCs_17','MFCCs_18','MFCCs_19','MFCCs_20','MFCCs_21','MFCCs_22']
feature_class=['RecordID','Family','Genus','Species']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X = train[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train[feature_class]

model= RandomForestClassifier(n_estimators=100)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(accuracy_score(y_test, preds,normalize=False))

and then I faced these value error:

How could I solve these problem?


texts: Calculating the F score of Object Detection of Mask RCNN

texts:
I am using Detectron2 Mask RCNN for an object detection problem. The images consist of cells that are very close to each other. I can not use mAP as a performance measure since the annotations are a bit off from the original location and the prediction is actually more accurate and when I use mAP it will give bad results.

Generally, each cell is 30 pixels apart and if the predicted and the actual are less then 30 pixels apart I count that prediction as a TP. I compare each prediction with each actual cell and if an actual cell is left at the end and no predicted is 30 pixels near it I add that to the FP.

This way I have TP, FP and the length of the actual cell is my FN. I calculate the Precision and the Recall and with these, I can calculate the F-Score.

Problem:
When the threshold is small for making predictions  I get an F-score greater than 1.

Here is my code

TP=0
FN=0
FP=0

for d in tqdm(dataset_dicts):

    bboxs = []
    bbox_center_x = []
    bbox_center_y = []

    predicted_centers_x = []
    predicted_centers_y = []

    path = d["file_name"].replace('\\','/')
    im = cv2.imread(path)

    outputs = predictor(im) #going over image by image to find predictions

    i = d['annotations']
    for i in range(len(d['annotations'])):
        bboxs.append(d['annotations'][i]['bbox']) #bboxs have the

    bboxs = np.array(bboxs)
    predicted_bboxs = outputs["instances"].pred_boxes.to("cpu").tensor.numpy()

    for bbox in bboxs:
        bbox_center_x.append((bbox[0]+bbox[0]+bbox[2])/2)
        bbox_center_y.append((bbox[1]+bbox[1]+bbox[3])/2)

    for predicted_bbox in predicted_bboxs:
        predicted_centers_x.append((predicted_bbox[0]+predicted_bbox[2])/2)
        predicted_centers_y.append((predicted_bbox[1]+predicted_bbox[3])/2)

    num_tumor = len(bbox_center_x)
    pred_num_tumor = len(predicted_centers_x)
    if num_tumor > pred_num_tumor:
        FN += num_tumor - pred_num_tumor

    temp_bbox_x = bbox_center_x
    temp_bbox_y = bbox_center_y

    for i in range(len(predicted_centers_x)):
        for j in range(len(temp_bbox_x)):
            if (math.sqrt( (temp_bbox_x[j] - predicted_centers_x[i])**2 +
                (temp_bbox_y[j] - predicted_centers_y[i])**2 ) ) <= 30:
                TP += 1

                break
            if j == ( len(temp_bbox_x)-1 ):
                FP += 1

Precision = (TP/(TP+FP))
Recall = (TP/(TP+FN))

f_score = 2*(Recall * Precision) / (Recall + Precision)
print(f_score)


texts: convert keywords in one column into several dummy columns

texts:
My dataframe has some free text fields named: {'title', 'description', 'location'}

I prepared this text column by: concatenating all into a new column, dropping numbers, dropping words less than 3 char, etc...

As final step in preparing, I removed Stop Words. So, probably at this moment the column only contains relevant keywords.

How can I convert these keywords sentences (several words per row, as space separated words, in just one string column) into dummy columns and get the 0-1 values when, per row, this column contains the keyword?

I was checking the CountVectorizer object, but I couldn't find any use case like this one. So, it is probably not the proper tool to use...


texts: keras evaulate method results vary with equal testset

texts:
I trained segnet on a dataset of remote sensing imagery. When I run model.evaluate a set of metrics is returned. When I compile the model again with the same settings and evaulate the model again, the results differ. For as far as I understand, there is nothing stochastic or random in the compiling of the model, am I right? I thought that the order of the training images in the batches could make a difference, so I tried a batch size of 1, but still I get different results.

Does the initialization of the model result in different results when predicting as well? And if yes, how do I resolve this?

This is the code I'm using for the initialization of the network:

def my_acc(y_true, y_pred):
    class_id_true = K.argmax(y_true, axis=-1)
    class_id_preds = K.argmax(y_pred, axis=-1)
    accuracy_mask = K.cast(K.equal(class_id_preds, 1), 'int32')
    class_acc_tensor = K.cast(K.equal(class_id_true, class_id_preds), 'int32') * accuracy_mask
    class_acc = K.sum(class_acc_tensor) / K.maximum(K.sum(accuracy_mask), 1)
    return class_acc

model = ZF_UNET_224() # This model is from the link in the beginning of the text
model.compile(optimizer=Adam(lr=0.0000001),
              loss = class_weighted_pixelwise_crossentropy,
              metrics=['accuracy',my_acc,'categorical_accuracy'])
model.evaluate(x=imgs_test,y=imgs_mask_test,batch_size = 1,verbose=1)

# result1: [77778.927522078811, 0.94418407911839697, 0.0039057209515957264, 0.94418407911839697]
# result2: [77586.558084239135, 0.077934800141045577, 0.02123359974321911,  0.077934800141045577]


texts: Combining Classifiers with different Precision and Recall values

texts:
Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is high and the recall is low, whereas the precision for B is low and the recall is high. Is there a way to combine these two models where I can get the precision from one and recall from the other, or possibly use the metrics from one to improve those of the other?

For example, let's say these are the metrics:

A

Precision: 0.91

Recall: 0.35

B

Precision: 0.44

Recall: 0.90

As these are binary classifiers, my labels are 1 and 0, and my class of interesting is 1 (so the metrics above are for predicting 1s).

Let's say model A predicts 10 1s, and model B predicts 70 1s.

It is safe for me to say that of the 10 1s that model A has predicted, 9 are true positives.

It is also safe for me to say that of the 70 1s that model B has predicted, 60% are false positives, but the rest are 90% of the true positives in the dataset.

My question is, is there some method for me to combine these results so that I can obtain all 50 true positives from the dataset?


texts:
I have imbalanced data that I pass through a LSTM network.

Here is part of my code :

weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

print (weights)
model = Sequential()
model.add(LSTM(64, input_shape=(look_back, trainX.shape[2]), recurrent_dropout=0.3, return_sequences=True)) #
model.add(Dropout(0.3))
model.add(LSTM(32,  recurrent_dropout=0.3))
model.add(Dense(1, activation='sigmoid'))
print (model.summary())
model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])
model.fit(trainX, trainY, validation_data=(validX, validY), epochs=100, batch_size=16, class_weight=weights, verbose=1 , callbacks=callbacks_list, shuffle=True)

Weights are [0.6147541  2.67857143] seeing good imbalance.

However, when I print confusion matrices (for each set), I only get 0predicted.

How my data are shaped. Actually, each 2-D sequence in input of a LSTM is of size (time_step, features) right ? Here I use time_step of 1 (and features=22 , don't matter). So that means that's the same as if I took each sample of my time serie one-by-one.

When I increase time_stepto something like 5, it works slightly better but I still get a too high number of "false 0" (0 is the biggest represented class) though a few-layers-MLP works very well on the same data with class_weight parameter. Am I doing something wrong with LSTM ?

Here is how the sequences are built: look_back is the size of the sliding window of instants centered around the current time step that I take and the windows overlap :

def create_dataset(feat,targ, look_back=2):
    semi_window=math.floor(look_back/2)
    dataX, dataY = [], []

    for i in range(semi_window,len(targ)-semi_window):
        a = feat[i-semi_window:(i+semi_window+1), :]
        dataX.append(a)
        dataY.append(targ[i])
    return np.array(dataX), np.array(dataY)

I am really waiting for help because it seems to give performances under simple MLP for example, while it's made to learn temporal dependencies on time series...


texts: Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?

texts:
Consider I have one dependent variable to predict 'Attitude' which can take three values 'Positive/Negative/Neutral'.
I have following independent variables or features- Age, Height, Gender, Income etc. I trying to predict Attitude using decision tree classifier.

Attitude ~ Age + Height + Gender + Income (Decision Tree)

I am getting >90% accuracy for the when tree depth is 15. As tree is dividing on continuous variables (i.e. Age, Income and Height) again and again to get leaf with pure classes.

Is this problem of overfitting? Should I convert the continuous variables into categorical variables (like range classes)?


texts:
I am trying to look at the customer retention & churn by using cohorts for an e-commerce usecase. From a business perspective, a client is defined as churned if it hasn't performed any transactions in the following 3 months after the month of its first purchase.

Now, I've done some R&D but all studies that I've found look at the churn on a monthly basis and thus all the coding is done this way.

Now, I am trying to adapt my code in Python so that the aggregation is done on a 3-months level rather than at monthly level.

grouped = txn_new_cif.groupby(['CohortGroup', 'Txn_date'])
cohorts = grouped.agg({'CIF': pd.Series.nunique,
                     'Merchant': pd.Series.nunique,
                     })[![enter image description here][1]][1]

What I would want is that for 2018-05 cohort, the 06-2018 to look at any transactions performed in 2018-06, 2018-07 and 2018-09. If there is any transactions performed by a user across this 3 months interval, then he would be considered active ( For 2018-07 projection, look for transactions in 2018-07, 2018-08 and 2018-09).


texts: How can you determine the growth rate you need to achieve a certain customer base?

texts: Say you have 12 K active customers every month in your platform in August 2016, how can you determine at what rate do you need to grow every month to achieve a certain total (say, 1.5 million) by next year, August 2017. Further, how can you generate different growth scenarios? Say, grow aggressively during the first quarter and then plateau. Are there any time series or optimization tools related to this kind of problem?


texts: How is GPT able to handle large vocabularies?

texts: From what I understand, GPT and GPT-2 are trained to predict the $N^{th}$ word in a sentence given the previous $N-1$ words. When the vocabulary size is very large (100k+ words) how is it able to generate any meaningful prediction? Shouldn't it become extremely difficult to correctly label the next word given that there are 100k possible labels to choose from? Even a large-scale classification problem like ImageNet has only 1k classes to choose from.


texts:
I've recently become interested in possibly of developing some sort of method for ranking athletes of sports such as American football and determining which players are better than others in terms of specific statistics.

My thoughts are that there are two ways to go about doing this. The first would be some sort of mathematical formula which would take in the statistics of a given player and provide some sort of standardized score which could be compared with other players to determine which is better.

My other idea would be to have some machine learning algorithm go through historical data and determine the patterns which indicate how well a certain combination of statistics would perform in the following week of play by using the patterns it recognizes over time.

I'm not sure which approach would be more effective and so I'm hoping that someone has an idea or any advice as to which would be best to look into. Thanks!


texts: What are the true error and the sample error?

texts:
I am a student and I am studying machine learning. I am focusing on the concept of evaluation of an hypotesis.
What I have seen is that there are two types of error: true error and sample error.
The true error of an hypotesis $h$ with respect to a target function $f$ and a distribution $D$ is the probability that an hypotesis $h$ misclassifies an instance $x$ drawn according to $D$, and it is computed as:
$error_D(h)=Pr_{x\in D}[f(x)\neq h(x)]$
while the sample error of an hypotesis $h$ with respect to a target function $f$ and data sample $S$ is the proportion of examples that $h$ misclassifies:
$error_S(h)=\frac{1}{n}\sum _{x\in S}\delta (f(x)\neq h(x))$
where
$\delta (f(x)\neq h(x))=1$ if $f(x)\neq h(x)$ and $0$ otherwise.
I ask this question because I have not clear what these errors are.
Moreover, I have seen that the true error cannot be computed, while we can compute only the sample error. I don't understand why.
Can somebody please help me understand?


texts: How to build a supervised artificial neural network?

texts:
I am trying to build and train a machine learning data science algorithm that correctly predicts what president won in what county. I have the following information for training data.

Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner

I am new to the field and this is my first time building an artificial neural network and I don't know what to do to start (so forgive me if my question is extremely broad). What I have done so far is read the wikipedia page for artificial neural networks. What I want to do next is implement supervised learning using the training data.

I would appreciate any help getting started. Please let me know if you have used any good tutorials or libraries to help build something similar. I was thinking of using the library Lasagne.

A few specific questions I have as I tried to use Lasagne - What is the error I am to calculate when building an algorithm using training data? How many layers do I need and what does each layer signify?


texts: Advantages of pandas dataframe to regular relational database

texts:
In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which are used to store data in many other fields of programming?

While pandas does provide some useful functions for data exploration, you can't use SQL and you lose features like query optimization or access restriction.


texts: Treating continuous data as a classification problem by predicting bins or quintiles

texts:
I currently have a model that has several numeric Y or predicted variables
Sample Data:

Y1
Y2
...
YN

2710
0.32
...
31231

1710
0.52
...
51231

I am currently using regression (multioutput regression with gradient boosting) to predict the Y variables.
Currently I am not happy with the accuracy and was looking at ways to increase accuracy. I was recommended using discretization so I treat Y as bins or quintiles. i.e:

Y1
Y2
...
YN

2000 - 3000
0.3 - 0.4
...
30000 - 40000

1000 - 2000
0.5 - 0.6
...
51000 - 50000

How can I go about doing this? What algorithms or libraries perform this? I am finding online how to discretize features but not so much the data itself. Do I need to manually shape the trained data into bins and then train on it?
I guess what I'm asking is where to go from here coding wise.


texts: Is it possible to build a regression model for predicting movie gross using sections on their wikipedia pages?

texts:
I got this as an assignment from a company recruiter and I've successfully scraped a dataset of about 650 movies with their 'Plot', 'Music' and 'Marketing' sections and gross. I've tried tfidf and count vectorizers and performed LSA/PCA to reduce the dimensions which originally are around 20k terms.

This is really boggling me, due to less instances(650) I guess the no. features should be around 100 or atleast < 600 but that is a drastic reduction of dimensions using PCA or LSA, is it the right thing to do?

Also my regression model is way off target or in some cases overfitting excessively. I've also tried normalizing and de-normalizing the target(gross) for no improvements. I know I should try NN and other models too but the results of linear regression should be atleast comparable too, right?

This makes me ponder if its even possible to model such a problem or am I doing something wrong with latter being more probable as I'm no expert in the field.

PLEASE HELP !


texts: Is there any ongoing research on known patterns ML algorithms can not mine?

texts:
I have been doing ML for sometimes and have explored and implemented Deep Learning as for some projects. ML algorithms assume a certain pattern (Geometric/Bayesian/Distribution) in the data .This can usually be referred as Hypothesis Space in the literature, then we design a solution to best fit that hypothesis to our data.

I wanted to know are there some patterns (Hypothesis spaces), which do not yet have a reliable algorithm to be mined?

Like Decision Trees used the Information Theory to assume that Information Gain from a variable can help us better classify the data. This was a new approach than SVM which assumes a geometrical space to where a plane can classify the data. I hope there must me some new theories which are being worked on.

Let me know if I can improve this question.


texts: Is it possible to train a neural network to solve polynomial equations?

texts:
I randomly generate millions groups of triplet $\lbrace x_0, x_1, x_2 \rbrace$ within range $(0,1)$, then calculate the corresponding coefficients of the polynomial $(x-x_0)(x-x_1)(x-x_2)$, which result in triplet groups normalized in a form of $\lbrace { {x_0+x_1+x_2 \over 3} , {\sqrt{x_0x_1+x_1x_2+x_0x_2 \over 3}} , {\sqrt[3]{x_0x_1x_2}}} \rbrace$;
After that, I feed the coefficient triplets in to a 5-layered neural network $\lbrace 3,4,5,4,3 \rbrace$, in which all the activation function is set to sigmoid and the learning rate is set to 0.1;
However, I only get a very poor cross validation, around 20%.

How can I fix this?

BackGround

My original problem is a dynamic inverse problem.
In that problem, I have hundreds of thousands of observations $O$, from these observations, I need to recover several hundred parameters $P$.
The simulation process from $P$ to $O$ is very easy and cheap to calculate, but the inversion from $O$ to $P$ is highly nonlinear and nearly impossible. My idea is to train a neural network getting $O$ as inputs and $P$ as outputs. To check the feasibility of this idea, I employ a 3-ordered polynomial equation to do the validation.

update half a year later

With more nodes per layer, I have successfully trained a neural network. The topology is set to $\lbrace 3, 64, 64, 64 \rbrace$. And the most important trick is, sorting the generated triplet $\lbrace x_0, x_1, x_2 \rbrace$, ensuring $x_0 <= x_1 <= x_2$ always holds.


texts:
I am working on neural network models for 3D skeletal character animation, where I learn joint positions and orientations. The problem comes with the orientations. There are several ways I can choose to represent a 3D rotation, but all of them have some form of discontinuity that is making my model unable to produce stable correct rotations in all cases. The alternatives I have considered are:

Quaternions. Quaternions (or unit quaternions) are very commonly used for 3D orientation, and are generally fine except a unit quaternion and its opposite represent the same orientation. It is not hard to come up with a loss formula the accounts for this (1 - abs(sum(componentwise_product(q1, q2))) works fine), but the problem is that in some cases the network somehow learns to produce the same quaternions with both signs, and kind of randomly "flips" from one sign to the other at times, with some frames in between, resulting in a sort of "flickering". One possibility would be to take only "half" of the quaternions, e.g. by fixing one component to be positive (then learn by squared differences for example). It doesn't always work, though. For example, suppose I fix the first component to be positive and I have a quaternion like [0.018, 0.743, 0.557, 0.371], and another one like [-0.017,  0.870,  0.348,  0.348]; these are rather similar quaternions, but if I make the first component positive then the second one will be [0.017,  -0.870,  -0.348,  -0.348], which is numerically very different, and the result is that sometimes the network cannot learn the value properly and comes up with some numbers in between (which are incorrect).
Rotation vectors. A rotation vector has the direction of the axis of rotation and the size of the angle of the rotation. They are used for example in the original paper of phase-functioned neural networks, but they still have bad cases. Suppose vector sizes go from 0 to π (greater angles would be taken along the opposite axis). If you have a rotation that is just around the value of π, the rotation vector will be "swinging" from one direction to the opposite one, again making it hard for the network to learn it.
Euler angles. These are just three rotation angles that are applied along specific axes (X, Y or Z) in specific order. They are generally discouraged because they are not very stable and suffer from the infamous gimbal lock, but geometry aside, they still have the same problems. If my angles go from -180º to 180º (or -π to π in radians), values in the "frontier" will always cause instability.
Extensive rotation encodings. One stable way of encoding rotations is giving both sine and cosine values. So I could for example have both the sine and cosine of each Euler angle, or just the nine values of a 3D rotation matrix (which are not sines and cosines but are in a way derived from these, and also a stable representation). However this obviously increases the number of values to learn significantly, and I would be trying to learn independent values that actually have a relationship among them.

I haven't found relevant literature addressing this problem specifically, although maybe I am not searching with the right terms. Has anyone faced this issue before? Or has some idea or alternative that I could consider?


texts: High Correlation between inputs of neural network

texts: I am building a neural network to predict if a video is pornographic or not by analysing the bytes of upload and download at every 0.1 seconds for a total of 25.6 seconds. So, I have 512 input variables, 256 of download and 256 of upload and I realised that for the same second, the upload and download variables are highly correlated. Does this mean that I am just adding complexity by distinguishing between upload and download variables?


texts: Clustering based on distance between points

texts:
I am trying to cluster geographical locations in such a way that all the locations inside each cluster are at max within 25 miles of each other. For this, I am using Agglomerative clustering. I am using a custom distance function to calculate the distances between each location. I do not want to specify the number of clusters. Instead, I want the model to cluster until all the locations within each cluster are within 25 miles of each other. I have tried doing this in both Scipy and Sklearn but haven't made any progress. Below is the approach that I have tried. It only gives me one cluster. Please help. Thanks in advance.

from scipy.cluster.hierarchy import fclusterdata
max_dist = 25
# dist is a custom function that calculates the distance (in miles) between two locations using the geographical coordinates

fclusterdata(locations_in_RI[['Latitude', 'Longitude']].values, t=max_dist, metric=dist, criterion='distance')


texts: Hope you are doing well. I have some problem with mixed. For the classification problem, can we standardize categorical and numerical variables together or just standardise numerical variable or don't need to standardise. For example age[0..120], sex[0..2] , productName[0..5], etc..


texts: If Deep learning or Machine learning is blackbox then why companies are still investing?

texts: I was wondering whether there is mathematical evidence or proof of functions that are happening at the backend of deep learning/machine learning. Particularly in training and testing operations. Secondly, deep learning is invented many years ago but still very minimal hardware systems to do something real. Is this a dark secret of artificial intelligence?


texts: Information Extraction from Free-form text to create Transactions

texts:
I'm working on a use case where the user will be provided a text box to enter the details of the transaction application.
For Example user might enter the below text and I have to parse the data and accordingly create the transaction. (I took basic example as a startup)

  Ex 1 : Transfer $100 from Account 1 into Account 2
  Ex 2: Move Money of total 400 dollars into account owned by Mr.xxxx from Acct#
  Ex 3: Deposit an amount of five thousand dollars into acct#

For above examples I have to parse and convert into a tabular format like
Trx_Type (Deposit or Transfer), Amt, From_Acct#, To_Acct#, Acct_Holder_Name,etc.

I'm trying to look options by doing Named Entity (NER) or Classifier modelling using NLTK in python. Can someone please share ideas from where to start and guide me with a draft way to proceed.
Looking forward for a response. Thanks!


texts: Create continous plot using matplotlib/seaborn with specific hue

texts:
df.head()=

day SBP     exercise
0   146.0   False
1   149.0   False
2   146.0   True
3   148.0   False
4   149.0   False
5   149.0   False

i am looking to plot day vs sbp with each data point coloured as per exercise.
i used following command but couldn't get each point colored as per exercise:
plt.plot('user','SBP', color='green', linestyle='dashed', marker='o',data=df)

can anyone help how i should go about this..?


texts: How many positive responses are good enough for building a marketing response model when the response rate is low(0.5%)

texts:
We are planning a marketing campaign to collect data and the response rates for a random sample.  Total population size is 10 million and historically, response rates are low (0.5 - 0.65 %).
How long do we need to run the test to collect a decent number of positive responses and how many positive responses need to aim for ?
I know there are no general rules for deciding the model data size as it all depends on the attributes and how predictive they are?
I am planning to run the test so that we get 5- 6K positive reponses for our campaign. We may need to send atleast 1 million mailers to get 6K responses. Only reason for proposing 6k number was aiming to collect as much as data and at the same time not spend too much on marketing.
Please let me know your thoughts about test sample size.


texts: When using numerical duplicates for categorical data, new columns should be added or values be converted?


texts:
I am working on a project where I would like to be able to specifically analyze the level of subjectivity in a given text phrase using machine learning. Essentially, I would like to be able to classify a given text based on whether it is written from a subjective or objective point of view. Ideally, I would like to be able to classify on the sentence level (i.e. assign each sentence a classification of being written from a subjective or objective perspective). This will most likely be applied to short form news articles and/or tweets.
My question is: what sort of NLP methodologies are best for this use case?
I have explored a few papers on the topic and it seems that this falls under the umbrella of sentiment analysis. Many papers report that machine learning methods such as SVM and neural network architectures perform well on the problem, given features such as a lexicon containing mappings of words to sentiment and POS tags. I have briefly tested out the TextBlob library's subjectivity score with some mixed results.
As I am new to NLP I am looking for suggestions as to which methodology to apply. Specifically, I would like to ask:

Is the text blob library well suited to this task (are there any other libraries you can recommend)? I have looked into the implementation and it seems to be computing subjectivity based on the presence of modifiers and the number of words belonging to certain POS tags.

What sort of features would you recommend utilizing to train a model for this task? As I mentioned, most papers seem to use POS tag counts and the sum of the polarity scores of the tokens in the text to determine subjectivity, but I wonder if there are other methods that experienced practitioners might think to apply

Are there any architectures that are particularly well suited to this task? The papers I have read report the best results for subjectivity classification using SVM and perceptron architectures but I am interested in exploring other architectures that may be able to analyze the text differently. I have looked briefly into sequence-to-sequence and attention models but based on what I have read they seem to be suited for a different range of tasks than this one - though intuitively I suspect that an attention model may be able to extract some latent representation of the text which could assist in subjectivity classification.

I would really appreciate any help to set me off in the right direction. I hope that my questions are clear. For reference I've included links to some of the papers I have been looking at below.
https://www.sciencedirect.com/science/article/pii/S0950705114002068
https://www.researchgate.net/publication/322005172_Distinguishing_Between_Facts_and_Opinions_for_Sentiment_Analysis_Survey_and_Challenges
https://www.researchgate.net/publication/322512000_Subjectivity_Detection_in_Nuclear_Energy_Tweets
https://www.researchgate.net/publication/343654268_Subjectivity_Detection_for_Sentiment_Analysis_on_Twitter_Data
https://www.researchgate.net/publication/318225737_Using_machine_learning_techniques_for_subjectivity_analysis_based_on_lexical_and_non-lexical_features


texts: Is it possible to create feature groups in Orange?

texts:
Background:
I am trying to use Orange as to classify if a patient has TB based on their coughing sounds.

In the dataset, there are say 100 patients and for each patient we have 10 coughs. For each cough, we have a full feature vector (170 features).

Giving Orange this dataset and training various learning algorithms is fairly straightforward, but the issue that I have is that Orange will consider each feature vector to be independent of another feature vector, which means it will consider every cough of each patient to be independent, and they aren't.

So my question is: Is there a way to tell orange that all 10 coughs of a patient belong to that patient, and when performing leave-one-out or cross validation methods, all the coughs from each patient should be excluded in each fold?


texts: sagemath: compared to r.quantile, what is a faster way to find boundaries for a boxplot?

texts:
I was using the r.quantile method in sagemath to find boundaries for a box plot.
The plot was taking a long time using r.quantile.

r.quantile took more than 20 seconds to find the quartiles for a data set that could be sorted and plotted point by point in less than half a second on the same machine.

What is a faster alternative?


texts: Should I use MongoDB instead of storing data in CSV in python?

texts:
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB instead of storing it in csv.

Will this be beneficial in saving space ?
Overall will this be advantageous to do or will there be any drawbacks apart from me having to change the code ?


texts:
I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here.

The methods provided for checking the quality of the model are very manual and require reading the similar docs, is there a way to calculate some other metrics from the model to try and improve its performance?


texts: Modeling Encoder-Decoder according to instructions from a paper

texts:
I am new to this field and I was reading a paper "Predicting citation counts based on deep neural network learning techniques". There the authors describe the code that they implemented if someone wants to reproduce the results. I tried to do this but I am not sure if I succeeded.
Here is their description:
-RNN module - SimpleRNN
-Output dimension of the encoder - 512
-The output layer - Dense layer
-Activation function - ReLU
-Overfitting prevention technique - Dropout with 0.2 rate
-Epochs - 100
Optimization algorithm - RMSProp
Learning rate - 10^{-5}
Batch size - 256

And here is my implementation. I am not sure if the model I created is sequence to sequence.
epocsh = 100
batch_size = 256
optimizer = keras.optimizers.RMSprop(lr=0.00001)
model =  keras.models.Sequential([
    keras.layers.SimpleRNN(512, input_shape=[X_train.shape[0], X_train.shape[1]],
                           activation='relu', return_sequences=True, dropout=0.2),
    keras.layers.Dense(9)
])

model.compile(loss='mse', optimizer=optimizer, metrics=[keras.metrics.RootMeanSquaredError()])

The summary of this model is:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
simple_rnn (SimpleRNN)       (None, 154521, 512)       266240
_________________________________________________________________
dense (Dense)                (None, 154521, 9)         4617
=================================================================
Total params: 270,857
Trainable params: 270,857
Non-trainable params: 0
_________________________________________________________________

Any help would be appreciated. Thank you!
Update: Here is the dataset.
year  venue  c1  c2  c3  c4  c5  c6  c7  c8  c9  c10  c11  c12  c13  c14
1989    234   0   1   2   3   4   5   5   5   5    8    8   10   11   12
1989    251   0   0   0   0   0   0   0   0   0    0    0    0    0    0
1990    346   0   0   0   0   0   0   0   0   0    0    0    0    0    0

I need to give as an input all the columns until c5, and try to predict the other c's (which are citation count for the upcoming years).


texts: Why does a horizontal cross section of a squared error surface yield ellipsis?

texts: Also, can someone please explain why the descent happens in a direction perpendicular to the contour lines?


texts: Advantages of monotonic activation functions over non-monotonic functions in neural networks?

texts:
What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks?

Do they perform better than non-monotonic ones?
Is this mathematically proven?
Are there any papers/references that are related to this?


texts:
I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point.

I have a few questions:

Is feature selection the first step to take to choose relevant
variables?
Is Logistic Regression only for binary output? For each doctor, can I get probability of drug prescription (e.g. doctor 1 = 0.87, doctor 2 = 0.56)?
How can my model be deployed into production? Is it a huge task?


texts: scikit learn target variable reversed (DecisionTreeClassifier)

texts:
I created a Decision Tree Classifier using sklearn, defined the target variable:
#extract features and target variables
x = df.drop(columns="target_column",)
y = df["target_column"]
#save the feature name and target variables
feature_names = x.columns
labels = y.unique()
#split the dataset
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x,y, test_size = 0.3, random_state = 42)

Additionally I checked the count of each of the two classes (Success, Failure) within y which confirmed to me that each has the correct count.
Then I fitted my DTClassifier:
clf = DecisionTreeClassifier(
    criterion='gini',
    splitter='best',
    max_depth=None,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_features=None,
    random_state=42,
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    min_impurity_split=None,
    #class_weight="balanced",
    presort='deprecated',
    ccp_alpha=0.0,
)
clf.fit(x_train, y_train)

The problem becomes apparent at the visualization step when I plotted the tree, each node shows me class = Failure when Failure is the minority and vice versa. Further down the line plotting the confusion matrix and calculating all the performance metrics it also becomes apparent, that the labels were reversed and I cannot figure out as to why.
Any ideas where I might need to look for the answer? If more code is necessary to give a feedback I can provide.
Much appreciated.


texts:
I am trying to create a script to filter the most "intersting" articles from an rss feed and rank them.

feeds = ['http://feeds.theguardian.com/theguardian/technology/rss',
         'http://rss.cnn.com/rss/money_news_international.rss',
         'https://news.ycombinator.com/rss?format=xml',
         'http://feeds.reuters.com/reuters/companyNews',
         'http://feeds.reuters.com/reuters/businessNews']

I tried using applying the K-means algorithm to RSS feeds to filter the most popular articles from thousands of links in an attempts to reduce my personal RSS reading time.

However, I feel that this might not be state of the art.

Any suggestions on papers, actual implementations or approaches to get a proper list of "must-read" articles.

I appreciate your replies!