texts: How to include categorical fields to enhance a text classification

texts:
I would have a question on how to add more categorical fields in a classification problem.
My dataset had initially 4 fields:
Date             Text                            Short_Mex                        Username        Label
01/01/2020       I am waiting for the TRAIN      A train is coming                Ludo       1
01/01/2020       you need to keep distance       Social Distance is mandatory     wgriws    0
...
02/01/2020       trump declared war against CHINESE technology      China’s technology is out of the games      Fwu32      1

...
I joined this dataset to a new one with labels, having values 1 or 0. This will need for classification.
However I have extracted also other fields from my original dataset such as number of characters, upper case words, top frequent terms, and so on.
Some of these fields may be useful for a classification, since I can assign more ‘weight’ based on a word in upper case rather than lower case.
So I would need to use a new dataset with these fields:
  Date             Text                            Short_Mex                        Username    Upper    Label
    01/01/2020       I am waiting for the TRAIN      A train is coming                Ludo    [TRAIN]       1
    01/01/2020       you need to keep distance       Social Distance is mandatory     wgriws       []      0
    ...
    02/01/2020       trump declared war against CHINESE technology      China’s technology is out of the games      Fwu32    [CHINESE]       1
...

I would like to ask you how to add this information (upper case) as a new info for my classifier.
What I am doing is currently the following:
#Train-test split
x_train,x_test,y_train,y_test = train_test_split(df['Text'], news.target, test_size=0.2, random_state=1)

    #Logistic regression classification
    pipe1 = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('model', LogisticRegression())])

    model_lr = pipe1.fit(x_train, y_train)

lr_pred = model_lr.predict(x_test)

print("Accuracy of Logistic Regression Classifier: {}%".format(round(accuracy_score(y_test, lr_pred)*100,2)))
print("\nConfusion Matrix of Logistic Regression Classifier:\n")
print(confusion_matrix(y_test, lr_pred))
print("\nCLassification Report of Logistic Regression Classifier:\n")
print(classification_report(y_test, lr_pred))


texts: Tweet Classification into topics- What to do with data

texts:
Good evening,
First of all, I want to apologize if the title is misleading.
I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into topics. I am working on topic modelling with LDA getting the right number of topics (I guess) thanks to this R package, which calculates the value of three metrics("CaoJuan2009", "Arun2010", "Deveaud2014"). Since I am very new to this, I just thought about a few questions that might be obvious for some of you, but I can't find online.

I have removed, before cleaning the data (removing mentions, stopwords, weird characters, numbers etc), all duplicate instances (having all three columns in common), in order to avoid them influencing the results of topic modelling. Is this right?

Should I, for the same reason mentioned before, remove also all retweets?

Until now, I thought about classifing using the "per-document-per-topic" probability. If I get rid of so many instances, do I have to classify them based on the "per-word-per-topic" probability?

Do I have to divide the dataset into testing and training? I thought that is a thing only in supervised training, since I cannot really use the testing dataset to measure quality of classification.

Antoher goal would be to classify twitterers based the topic they most are passionate about. Do you have any idea about how to implement this?

Thank you all very much in advance.


texts: What would be the target input for Transformer Decoder during test phase?

texts:
The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer
I am having confusion about what the target sequence will be when the trained model is evaluated?.
Is it that we start with a <SOS> tag for the first timestep and loop through the transformer decoder for each timestep like in RNN's?
It would be helpful if someone can clarify this for me.


texts: About confidence/prediction intervals: parametric methods VS non-parametric (via bootstrap) methods

texts:
About the methodology to find confidence and/or prediction intervals in, let's say, a regression problem, I know 2 main options:

Checking normality in the estimates/predictions distribution, and applying well known Gaussian alike methods to find those intervals if the distribution is gaussian
Applying non-parametric methodologies like bootstraping, so we do not need to assume/check/care whether our distribution is normal

With this in mind, I would basically always go for the second one because:

it is meant to be generic, as it does not assume any kind of distribution
it feels more like experimental as you can freely run as many iterations as you want (well, if it is computionally feasible)

The only drawback I could see is the computational cost, but it could be parallelized...

Can anyone give another point/advice?


texts: Clustering with groups in data related to cluster label

texts:
I want to predict which device got used in which room. Therefore I've got device and sensor data.

My idea was to create a feature vector lie this:

             ----------------------------------------------------------
Data-Vector: | u_1 u_2 u_3 | x_1 ... x_7 | y_1 ... y_12 | z_1 ... z_4 |
             ----------------------------------------------------------
 Categories: | device_data | room 1 data | room 2 data  | room 3 data |
             ----------------------------------------------------------

My device data contains amongst other things:
+ timestamps when the device got turned on/off
+ average power consumption and divergences

My room data contains for example:
+ sensor data of motion detector and timestamps
+ sensor data of lamps (turned on/off) and timestamps
+ weather data

In the feature vector I've got the room data closest to the turn on/off timestamp.

All data points itself are floats.

My idea was to use k-means for clustering.
My problems are:
1. When using k-means, how can I tell which cluster correlates to which room label (room1, room2 or room3)?
2. I think it could be beneficial if I add somehow the information: which sensor is which room.

Can I manipulate the data in k-means algorithm so it will only consider:
the device data and room 1 data for the first cluster and sets everything else to zero
the device data and room 2 data for the second cluster
and so on...

This way I could tell that cluster x correlates to room x.

Or will this somehow break the k-means algorithm?


texts: How to insert sentences into training data which has 2 words, 3 words 4 and 5 etc into training data?

texts:
I have a set of sentences which each contain 2 words, 3 words, 4 words, 5 words etc. When I am trying to give the training data only the first two words in a sentence it is not accepting it. It is showing given 4 expected 2. How do you deal with this? Said another way, how do you to insert each and every sentence into the training data. I am trying to predict the next word.
My code:
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

dtype = torch.FloatTensor

sentences = [ "i like dog", "i love coffee", "i hate milk", "i love football", "hi im great", "i love watching football", "i don't eat chicken"]

word_list = " ".join(sentences).split()
word_list = list(set(word_list))
word_dict = {w: i for i, w in enumerate(word_list)}
number_dict = {i: w for i, w in enumerate(word_list)}
n_class = len(word_dict) # number of Vocabulary

# NNLM Parameter
n_step = 2 # n-1 in paper
n_hidden = 2 # h in paper
m = 2 # m in paper

def make_batch(sentences):
    input_batch = []
    target_batch = []

for sen in sentences:
    word = sen.split()
    input = [word_dict[n] for n in word[:-1]]
    target = word_dict[word[-1]]

    input_batch.append(input)
    target_batch.append(target)

return input_batch, target_batch

# Model
class NNLM(nn.Module):
    def __init__(self):
        super(NNLM, self).__init__()
        self.C = nn.Embedding(n_class, m)
        self.H = nn.Parameter(torch.randn(n_step * m, n_hidden).type(dtype))
        self.W = nn.Parameter(torch.randn(n_step * m, n_class).type(dtype))
        self.d = nn.Parameter(torch.randn(n_hidden).type(dtype))
        self.U = nn.Parameter(torch.randn(n_hidden, n_class).type(dtype))
        self.b = nn.Parameter(torch.randn(n_class).type(dtype))

    def forward(self, X):
        X = self.C(X)
        X = X.view(-1, n_step * m) # [batch_size, n_step * n_class]
        tanh = torch.tanh(self.d + torch.mm(X, self.H)) # [batch_size, n_hidden]
        output = self.b + torch.mm(X, self.W) + torch.mm(tanh, self.U) # [batch_size, n_class]
        return output

model = NNLM()

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

input_batch, target_batch = make_batch(sentences)
input_batch = Variable(torch.LongTensor(input_batch))
target_batch = Variable(torch.LongTensor(target_batch))

# Training
for epoch in range(5000):

optimizer.zero_grad()
output = model(input_batch)

# output : [batch_size, n_class], target_batch : [batch_size] (LongTensor, not one-hot)
loss = criterion(output, target_batch)
if (epoch + 1)%1000 == 0:
    print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.6f}'.format(loss))

loss.backward()
optimizer.step()

# Predict
predict = model(input_batch).data.max(1, keepdim=True)[1]

# Test
print([sen.split()[:2] for sen in sentences], '->', [number_dict[n.item()] for n in predict.squeeze()])

ValueError                                Traceback (most recent call last)
<ipython-input-26-65fcd35f0145> in <module>()
      3
      4 input_batch, target_batch = make_batch(sentences)
----> 5 input_batch = Variable(torch.LongTensor(input_batch))
      6 target_batch = Variable(torch.LongTensor(target_batch))

ValueError: expected sequence of length 2 at dim 1 (got 3)


texts:
I am approaching machine learning for the first time because of my studies. I have been given a bunch of tweets and the goal is to classify them per topic. I really have no clue on how this should be done. Is there a particular way to follow?
Until now, I have only found topics and was thinking about making a DTM-like dataframe for the training data containing not only the number of times not-sparse words occur but also the number of times particular N-grams occur and a ground truth column with the topic.
Is this totally wrong? How else could I train a classifier without having features?


texts: how to update the pre-trained word2vec model with new train data using genism

texts:
Hi I have used the genism to load the Spanish  fasttext word2vec model with following code:

Word2Vec = KeyedVectors.load_word2vec_format('../data/raw/wiki.es.vec',
                                             binary=False)

and now I want to train the word2vec with new training sentences.

I know that in the genism can use the following code to train new sentences if the model is loaded:

 model.build_vocab(sentences)  # prepare the model vocabulary
 model.train(sentences, )

But how to update the model if I used the KeyedVectors.load_word2vec to load the model.

Thanks!


texts: Tensorflow Conv3D with variable input size

texts:
I have a hypotethical question: Is it possible to train Conv3D with variable input size?
Sample dim = Length x Width x Depth ; Depth are fixed per each samples, let's say 500. However Length x Width can vary, e.g.:
Sample 1 = 50 x 4 x 500
Sample 2 = 7 x 7 x 500
Sample 3 = 10 x 13 x 500
.....
Sample n = 5 x 32 x 500

These are for classification problems, the next class could have a different sample size, e.g.:
Sample 4 = 6 x 8 x 500 (from class 2)
Sample 5 = 3 x 32 x 500 (from class 2)
....
Sample m = 10 x 11 x 500 (also from class 2)

Thanks in advance.


texts: Would a neural network trained on extracted features have the same accuracy as a full network with frozen layers?

texts:
Let's say that I train two neural networks on the exact same dataset.

The first network is a VGG19 model with frozen convolutional layers so only the top dense layers are getting optimized.
The second network is composed only of dense layers, with exactly the same architecture and hyper-parameters as the first networks fully connected layers.

Basically, I pass all my data through the convolutional layers of VGG19, saving outputs of the last convolutional layer to disk. Then, I load the data into my second model which is only composed of FC blocks and I train it to be a classifier.
Would combining the two models after training be the same as training a whole VGG19 model (with frozen conv layers) with my own FC layers?
If I understand correctly, the outputs of a trained network are entirely deterministic so in theory it should work. In practice, I have no idea if optimization algorithm takes frozen layers into account when training only the last FC layers.


texts: train_test_split ValueError: Input contains NaN

texts:
I tried to do a stratified sampling by way of train_test_split in order to save myself some trouble later. So I wrote the following lines:

from sklearn.model_selection import train_test_split

X=data_df
y=data_df.pop('class')

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.125, stratify=y)

I got the error :

ValueError: Input contains NaN

Any help is welcome!


texts: With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

texts:
In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.


texts: Do I have to wrap multiclass SVM in OneVsRestClassifier()?

texts:
I am using an SVM for mulitclass classification between 3 labels (1,0,-1). I thought this could simply be done by using SVC(decision_function_shape = 'ovr') in my model. I have only just discovered that there exists an Sklearn package that works as follows:
 sklearn.multiclass.OneVsRestClassifier(estimator, n_jobs=-1)

So my question is, what is the difference between the OneVsRestClassifier() and SVC(decision_function_shape = 'ovr'). And in what scenario could one method or the other yield better accuracy?

The current code for my project is as follows:

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 0)

    pipe = Pipeline([('sc', preprocessing.MinMaxScaler()),
                 ('SVM', svm.SVC(decision_function_shape = 'ovr', kernel = 'poly'))])

    candidate_parameters = [{'SVM__C': [1],
                        'SVM__gamma': [1] }]
clf = GridSearchCV(estimator = pipe, param_grid = candidate_parameters, cv = 5, n_jobs = -1)


texts:
I have a model that i need to train multiple times using epochs, i tried adding this code clf_svm.fit(train_features, train_labels, epochs=10, batch_size=64) and it didn't work. how to add epoches in skleran linear svc? training that model several times and save the model here is the code am working with
# empty list to hold feature vectors and train labels
train_features = []
train_labels   = []
# loop over the training dataset
print ("[STATUS] Started extracting haralick textures..")

for train_name in train_names:
    cur_path = train_path + "/" + train_name
    cur_label = train_name
    i = 1
    for file in glob.glob(cur_path + "/*.jpg"):

        print ("Processing Image - {} in {}".format(i, cur_label))

        # read the training image
        image = cv2.imread(file)
        #resize = ResizeWithAspectRatio(image, width=1250, height=1000) # Resize by width OR
        # convert the image to grayscale
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        # extract haralick texture from the image
        features = extract_features(gray)

        # append the feature vector and label
        train_features.append(features)
        train_labels.append(cur_label)

        # otherwise create the model, train the model and save the model
if os.path.exists("D:/ai training/aphids/joblib_model.sav"):
    print("Loading Trained Model")
    clf_svm = joblib.load("D:/ai training/aphids/joblib_model.sav")
else:
        # have a look at the size of our feature vector and labels
        print ("Training features: {}".format(np.array(train_features).shape))
        print ("Training labels: {}".format(np.array(train_labels).shape))

        # create the classifier
        print ("[STATUS] Creating the classifier..")
        clf_svm = LinearSVC(random_state=9, dual=False, max_iter=1000)

        # fit the training data and labels
        print ("[STATUS] Fitting data/label to model..")
        clf_svm.fit(train_features, train_labels)

        #savemodel
        joblib_file = 'D:/ai training/aphids/joblib_model.sav'
        joblib.dump(clf_svm, joblib_file)


texts: predicting number of customers based on long/lat coordinates

texts: I have a very simple dataset that has as columns the latitude and longitude of bookstores, and in the output column, has the number of customers / year. I've never explicitly deal with longitude/latitude coordinates as predictor variables before, but I was curious if one could use them as predictors to predict the number of customers/year for an arbitrary set of coordinates?


texts: predict_generator gives "Found 0 images belonging to 0 classes."

texts:
I have trained my model on the very big dataset (approx 1200000 train images, and 15000 classes) using Keras' flow_from_directory method. I have successfully trained my model using fit_generator. But, now I am facing the problem for predict_generator.

I got the following error while executing predict_generator:

  "Found 0 images belonging to 0 classes."

However, I have converted my train image dataset into the subdirectory based on class labels. But, for test dataset, I don't have any labels.
So how can I overcome this issue?


texts: my cifar10 kaggle result around 10% is there something wrong with my pytorch code/?

texts:
I'm quite new to pytorch so I want check is there something wrong
I got final submission code score around 10%

here is my code

train_transform = transforms.Compose([
    transforms.Resize(224),
    transforms.RandomHorizontalFlip(p=.40),
    transforms.RandomRotation(30),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

test_transform = transforms.Compose([
    transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

traindata = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/datasets/', train=True,download=False, transform=train_transform)
trainset,valset = random_split(traindata,[42000,8000])
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,shuffle=True)
valloader = torch.utils.data.DataLoader(valset, batch_size=batch_size,shuffle=False)

testset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/datasets/', train=False,download=False, transform=test_transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,shuffle=False)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

torch.manual_seed(777)
if device =='cuda':
    torch.cuda.manual_seed_all(777)
device

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        base = models.resnet18(pretrained=True)
        self.base = nn.Sequential(*list(base.children())[:-1])
        in_features = base.fc.in_features
        self.drop = nn.Dropout()
        self.final = nn.Linear(in_features,10)

    def forward(self,x):
        x = self.base(x)
        x = self.drop(x.view(-1,self.final.in_features))
        return self.final(x)

model = Model().cuda()
[x for x,y in model.named_children()]

criterion = nn.CrossEntropyLoss()
param_groups = [
    {'params':model.base.parameters(),'lr':.0001},
    {'params':model.final.parameters(),'lr':.001}
]
optimizer = Adam(param_groups)
lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)
states = {}

%%time
best_val_acc = -1000
best_val_model = None
print("start")
for epoch in range(10):
    model.train(True)
    running_loss = 0.0
    running_acc = 0

    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        inputs, labels = inputs.cuda(),labels.cuda()

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item() * inputs.size(0)
        out = torch.argmax(outputs.detach(),dim=1)
        assert out.shape==labels.shape
        running_acc += (labels==out).sum().item()
    print(f"Train loss {epoch+1}: {running_loss/len(trainset)},Train Acc:{running_acc*100/len(trainset)}%")

    correct = 0
    model.train(False)
    with torch.no_grad():
        for inputs,labels in valloader:
            out = model(inputs.cuda()).cpu()
            out = torch.argmax(out,dim=1)
            acc = (out==labels).sum().item()
            correct += acc
    print(f"Val accuracy:{correct*100/len(valset)}%")
    if correct>best_val_acc:
        best_val_acc = correct
        best_val_model = deepcopy(model.state_dict())
    lr_scheduler.step()

    torch.save(model.state_dict(), PATH)

print('Finished Training')

from torch.autograd import Variable
results = []
print("start")
i = 0
with torch.no_grad():
    model.eval()
    for num, data in enumerate(test_loader):
        imgs, label = data
        imgs,labels = imgs.to(device), label.to(device)
        output = model(imgs).cpu()
        _, preds_tensor = torch.max(output,dim=1)
        results += preds_tensor.tolist()
print('finish')

ans then my making submission code

  https://www.kaggle.com/c/cifar-10/submit


texts:
I am sure that this has happened to you if you have trained an LSTM model. The LSTM model predicts the same word 2 or 3 times. Now, I am not saying that for every input it has the same output. I am saying that for  every input,  the output that it produces has some repeated words in it. For example:
    Human: What do you like?
    Bot: I quite enjoy enjoy programming!

Here the bot outputted the word enjoy 2 times. Not only in chatbots, I have encountered this issue in every LSTM Model I have trained so far.
What could be the reason of this? And how to stop this?
Thank you in advance!


texts: Does a matrix factorization recommendation engine use user/item related features?

texts:
All the tutorials I can find about matrix factorization recommendation systems start with importing users, items, and user-item-ratings, but then only use the rating matrix to train the recommender (not features of the users or items themselves like "age").

Example

Tutorial: Matrix Factorization for Movie Recommendations in Python

URL: https://beckernick.github.io/matrix-factorization-recommender/

Users (userID, gender, ageGroup, occupationGroup, zipCode)

Items (movieID, title, genres)

Ratings  (userID, movieID, rating, timestamp)

Then we do some cleaning and build a matrix of users-to-items with ratings as the values.

Then we proceed to to only use the matrix built from ratings (not other features of the user like age which I might guess has some predictive value).

# normalize data
import numpy as np
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis=1)
R_demeaned = R - user_ratings_mean.reshape(-1,1)

# import numpy as np
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis=1)
R_demeaned = R - user_ratings_mean.reshape(-1,1)

# SVD
import scipy
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k=2)

# convert to diagonal matrix
sigma2 = np.diag(sigma)

# build predication matrix
all_user_predicted_ratings = np.dot(np.dot(U, sigma2), Vt) + user_ratings_mean.reshape(-1, 1)
preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns, dtype='float')

Theres more in the tutorial but at this point our predication matrix is already set.

I wondered if this is just how matrix factorization recommenders work (aka. not use user/item features) or if online tutorials are just too simple?
If I wanted to incorporate a user feature like age, would it be possible with matrix factorization?


texts:
I was reading the article $\textit{Application of Higher Order Statistics for Atrial Arrhythmia Classification}$, and they mentioned using the third order cumulant $C_3^x = m_3^x(i,j) = E[x(n)x(n+i)x(n+j)]$
and applying independent component analysis on it. The article says,
"In the present study, the performances of two methods are compared: (i) 3rd order HOS cumulants and (ii) HOS bispectrum. The 3rd order cumulant and bispectrum coefficients are subjected to dimensionality reduction using independent component analysis (ICA) and classified using classification and regression tree.."
Can you please explain to me the specifics of how to apply ICA on the 3rd order cumulant to reduce the dimensions. You may assume that $x(n)$
is the heart beat signal and that we only consider the third order cumulant on the domain $(i,j) \in [-2, 2]\times[-2, 2]$


texts: How can I measure if a population has the same distribution as other?

texts:
Is there a population similarity index of some kind which could help me determine if two populations in two different datasets are the same or at least similar?

The datasets have the exact same number of variables.

I would like a measure which assess the similarity variable-by-variable, but an overall measure works too.

The context of my question is that I would like to know if a variable is suitable or not for using in a classification method given the similarity of it between populations.


texts: Can Gaussian dist. use median/MAD instead of mean/SD?

texts: Can Gaussian density distributions be modified using median and median absolute deviation (MAD) as opposed to mean and standard deviation (SD) (since the former are more robust)?


texts: Conflicting directions of weights gradients in gradient descent?

texts:
In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 to eventually update them.

We calculate ∂Err/∂w1, ∂Err/∂w2, ∂Err/∂w3 and update the weights as wi = wi + ∂Err / ∂wi for each of the weights so that we move towards the direction where the loss function reduces in measure.

The problem that I see is that there might be situations all the time when some of the weight delta directions conflict in terms of the loss function. That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err. Isn't this the case? What am I missing?


texts: Metrics show badly performing model for multiclass

texts:
So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy.

I first remove any 0 variance features (~1M -> 2k feats.), then using sklearn.decomposition.TruncatedSVD I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.

Then I run imblearn.under_sampling.RandomUnderSampler (100k samples for majority classes) and imblearn.over_sampling.SMOTE (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.

After this I train several models and then compare different metrics to get a better idea of which is the best choice:

sklearn.linear_model.SGDClassifier(loss='log', alpha=1e-4,tol=1e-4,penalty='elasticnet',max_iter=1000,shuffle=True)
sklearn.linear_model.Perceptron(alpha=1e-3, penalty=None,tol=1e-4,max_iter=1000,shuffle=True)
sklearn.linear_model.PassiveAggressiveClassifier(C=1.0,tol=1e-4,max_iter=1000,shuffle=True)
sklearn.neural_network.MLPClassifier(solver='adam',learning_rate='adaptive',activation='tanh',validation_fraction=0.05,alpha=1e-5,learning_rate_init=1e-4,max_iter=20,hidden_layer_sizes=(100,))
sklearn.naive_bayes.BernoulliNB(alpha=1.0e-5, fit_prior=True)

As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around, AUC and cohen_kappa are the more realiable metrics for multiclass and imbalanced data sets.

I am concerned that the AUC metric show that all models are doing well, but the others, in particular cohen_kappa which is significantly lower (I know though that it is hard to directly compare the metrics).

Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores?


texts: Tools to do Semantic Anotation on 3d CAD Models

texts:
I am trying to do Segmentation on my custom dataset consisting of 3D CAD Models.
However, I am not able to find the proper tools to generate my dataset for Segmentation.
I am able to find the existing dataset with segmentation information, but am not able to find any information on how to create one for our own CAD Models.
I need to train the model on my custom dataset hence creating one.
I will really appreciate any help that is provided.


texts: Q-learning why do we subtract the Q(s, a) term during update?

texts:
I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either.

Everything else makes sence. The q-learning algorithm is an off-policy algorithm, unlike SARSA. The Bellman equation describes q-learning as follows:

$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) - Q(s_t, a_t) \right] $$

"The q value for action $a$ taken in state $s$ at time $t$ becomes equal to: that same q-value plus small amount of: currently received reward (reward could be zero) with some amount $\gamma$ of the best Q-value available in the new state, minus our current value

To me, this $-Q(s_t, a_t)$ term at the very end is redundant. ...If we set gamma to $0.8$, the future rewards will decay anyway.

Yes, if instead we now set $\gamma=0$, then the $-Q(s_t, a_t)$ term will drag our value down - is there a case when it would be useful and what would the result be?

Edit:

wikipeda uses a slighly different form of the bellman equation

$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot argmax(Q(s'_t, a'_t)) \right] $$

It's the same equation as the one above, because we can multiply $Q(s_t, a_t)$ with an $\alpha$ and then factor it out, obtaining the first equation.

This representation makes me kind-off understand that we are linearly interpolating from current to newer Q-value, but I can't tie it to the original representation ...In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?


texts: I am building a anomoly detection model using Pyculiarity (Twitter's anomaly detection). However, the expected value is 'NaN'. What am I missing?

texts:
When I run the following code, I get the expected value as NaN:
import numpy as np
#from statsmodels.tsa.holtwinters import ExponentialSmoothing
from statsmodels.tsa.api import ExponentialSmoothing, SimpleExpSmoothing, Holt
import itertools
from statsmodels.stats.stattools import medcouple```

```detect_ts(df[['Date column','Count']], max_anoms=0.01,  e_value=True, alpha=0.001 ,direction='both') ```

**Output:

{'anoms':             timestamp  anoms  expected_value
 timestamp
 2020-03-23 2020-03-23  30828             NaN
 2020-01-03 2020-01-03  17458             NaN
 2020-01-01 2020-01-01  26396             NaN
 2020-01-02 2020-01-02  18939             NaN, 'plot': None}**

Please help me understand what I am missing


texts: Input variable that influences only trend

texts:
I need to predict cost for the next 4 weeks. Along with categorical variables (available for future as well), I have a numerical variable (a value from the day before prediction - I don't have future values for this variable), let's call it foo, that is telling me if I am spending too much or too little. Basically, it tells me if I need to increase or decrease the cost in the following days. I am not sure how to approach this. How to incorporate this variable in the model.
I thought that I can I maybe declare a categorical variable for future days that would have value 'grow' for the time that it is relevant (for example first 2 weeks), and for the rest value 'na' (last 2 weeks), since I don't know the trend of cost for them? But then how would I create this variable for the past...
Maybe I just need a multiplier, to raise my cost, or lower them depending on foo value from the day before.
Do you have any suggestions?
PS: I am planning to use LSTM, but I am open to other suggestions.


texts: Crop all written letters from image to form a website

texts:
Above image has handwritten hindi / deavanagari letters all on a single

image.
Task is to Prepare a dataset from the given assignment.jpg image using image

processing tools
Idon't have clarity whether i can use photoshop editing tools like adobe photoshop etc.
If image processing tools mean opencv , is it really possible to create like a emnist like dataset from single image.
I am not able to come up with any way to seperate the letters from eachother

Finally i had done it by combining adaptive thresholding,bluring,and morphological, It is helpful to check out find contours for two part letters by @Salman and @Jeru Luke , it really helped me.


texts: How to group by multiple columns in dataframe using R and do aggregate function

texts:
I have a dataframe with columns as defined below. I have provided one set of example, similar to this I have many countries with loan amount and gender variables

      country          loan_amount          gender
1      Austia              175                F
2      Austia              100                F
3      Austia              825                M
4      Austia              175                F
5      Austia             1025                M
6      Austia              225                F

Here I need to group by countries and then for each country, I need to calculate loan percentage by gender in new columns, so that new columns will have male percentage of total loan amount for that country and female percentage of total loan amount for that country. I need to do two group_by function, first to group all countries together and after that group genders to calculate loan percent.

    Total loan amount = 2525
     female_prcent = 175+100+175+225/2525 = 26.73
     male_percent = 825+1025/2525 = 73.26

The output should be as below:

      country            female_percent   male_percent
1      Austia              26.73            73.26

I am trying to do this in R. I tried the below function, but my R session is not producing any result and it is terminating.

  df %>%
  group_by(country, gender) %>%
  summarise_each(funs(sum))

Could someone help me in achieving this output? I think this can be achieved using dplyr function, but I am struck inbetween.


texts: How can I merge 2+ DataFrame objects without duplicating column names?

texts:
This is for work. TLDR: Bottom-line question at the bottom.

I am gathering and parsing test results produced by an old test setup whose output formatting is not likely to change anytime soon. I've made good progress on parsing the output data into lists of strings, booleans, etc., but I'm having trouble pulling the data together into an easily searchable and retrievable whole. The data looks something like this:

- test_case_A
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 100  pass/fail   supporting data
- test_case_B
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 70   pass/fail   supporting data
- test_case_C
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 10   pass/fail   supporting data
- test_case_D
    - pass/fail file
        - big header info with test case metadata
        - test 1    pass/fail   supporting data
        - test 2    pass/fail   supporting data
        - ...
        - test 30   pass/fail   supporting data

I parse them into individual DataFrame objects like so:

df_case_A_pass_fail = pd.DataFrame({
    "case" : "test case A",
    "header" : *big header string*,
    "test ID" : [*list of IDs*],
    "test passed" : [*list of bool*],
    "test data" : [*list of strings*],
    })
*repeat for test case B, C, and D*

Now I try to merge them together.

big_df = fancy_merge_step_probably_involving_reduce_and_a_lambda(...)

Problem 1: The "case" and "header" strings appear to be duplicated all the way down their DataFrame. Like so:

             case        test ID  test passed               header      supporting data
0   "test case A"    "test 1 ID"         True  "big header string"  "all kinds of stuff"
1   "test case A"    "test 2 ID"         False "big header string"  "all kinds of stuff"
...
99  "test case A"  "test 100 ID"         True  "big header string"  "all kinds of stuff"

I checked the DataFrame size according to this get_real_size(...) algorithm and the size explodes exponentially as I merge in more results (~80KB for 1 test case, ~800KB for 2 test cases), so the big header string is definitely getting duplicated. I want to establish one->one relationship between test case and header and one->many between test case and test data, but all I'm seeing is duplication until there is a unique permutation of every line. Am I making the DataFrame wrong for what I need?

Problem 2: (Possibly related to problem 1.) I attempt to merge the DataFrames with an outer join and get the NaN results for places where the column sizes don't match up (expected), but also get duplicated columns "case", "header", "test passed", "test data" (any column that wasn't merged on), appendend with automatic suffixes ("_x", "_y"). I know that pandas does this automatically when there is a column name class, but it is now a problem. Result: searching on column "case" fails because the merged DataFrame has no column "case". All columns formerly named "case" are now "case_x" or "case_y".

I want to query like this:

match1 = (big_df["case"] == "test case D")
match2 = (big_df["test ID"] == "test 3")
single_test_df = big_df.loc[match1 & match2]

match1 = (big_df["case"] == "test case A")
match2 = (big_df["header"] == "header")
header_str = big_df.loc[match1 & match2].values[0]

Question: How do I set up these DataFrame objects and merge them so that I can query the test data as mentioned earlier?


texts:
I currently want to train a CNN but I have two small datasets that are slightly different because of the camera setup that captured the images. I'm interested in ultimately tuning the neural network to only one of the camera setups and using the network for that one setup only from here on out.

Since I'm not going to be able to expand my dataset during this time, I realize I'll need to use both training datasets to get enough of variability. So I'm wondering what approach to this might be better -- should I combine the two datasets and train the CNN, or should I train on one dataset and finetune it to the second dataset? Or is there another method I should be using?


texts: Real time topic identification of news article

texts:
Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)).

Let's say that I'd like to group together all the articles that are speaking about the same thing. However :

I want my algorithm to define the topics by itself (in opposition to "the user tells the existing topics and the algorithm assigns each article to the topics"),
I can't know the precise number topics there are (because, obviously, a new one has to be created each time something new happens),
and, as we are talking about news article, the list of topics should be expanding in real time if something new happens and new articles talk about it.

For simplicity's sake, let's assume all the articles are in the same language.

I've been using TF-IDF for this as of now, because that's what I came up with on the top of my head when I knew nothing about topic modeling, but it's not really good at its job. I'm getting into topic modeling and discovering things such as Latent Dirichlet Allocation, Correlated Topic Model, graph of words etc. I'm currently reading through all I can on the subject, but I'm probably unaware of interesting algorithms that could fit my need.

I'm interested : what approach would you take to tackle my problem ?

Thanks


texts: First order vs Second order online machine learning algorithms

texts:
In this git repo the online learning algorithms are classified as first order and second order. I tried searching what it means, yet I'm unable to understand the different between such first oder and second order online learning algorithms.

What is the difference between first order and second order online learning?


texts: Credit card fraud detection - anomaly detection based on amount of money to be withdrawn?

texts:
I am trying to figure out how the amount of money that a customer would want to withdraw on an ATM tell us if the transaction is fraudulent or not.There are other attributes, of course, but now I would want to hear your views on the amount of money that the customer wants to withdraw.

Data may be of this form:

Let us assume that a customer, for ten consecutive transactions, withdrew the following amounts:

100.33, 384 , 458, 77.90, 456, 213.55, 500 , 500, 300, 304.

Questions:

How can we use this data to tell if the next transaction done on this account is fraudulent of not?
Are there specific algorithms that can be used for this classification?

What I was thinking:

I was thinking to calculate the average amount of money, say for the last ten transactions, and check how far is the next transaction amount from the average. Too much deviation would signal an anomaly. But this does not sound much, does it?


texts:
This is the tflite model being used

This is the code to run tflite model
Object[] inputArray = {imgData};
outputMap = new HashMap<>(1);
outputMap.put(0, outputLocations);
tflite.runForMultipleInputsOutputs(inputArray, outputMap);

Since there is only one output tensor, I have inserted only one element in the outputMap
and this is the shape of the outputLocations
outputLocations = new float[1][2535][6];

Below is the code to extract information
//labels.size() = 1;
NUM_BOXES_PER_BLOCK = 1;
     int outputWidth = 2535;
     float [][] out = ((float [][][]) Objects.requireNonNull(outputMap.get(0)))[0];
     final int labelSizeWithResult = labels.size() + 5; // 1 + 5

      for(int x = 0; x < outputWidth; x++)
           for (int b = 0; b < NUM_BOXES_PER_BLOCK; b++) {
                final float confidence = sigmoid(out[x][b * labelSizeWithResult + 4]);
                 Log.d("Confidence "," "+confidence); //Return NaN  (all values inside the array is Nan
               }

Where am I going wrong?


texts: Please review my sketch of the Machine Learning process

texts:
It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own process by looking at a lot of tutorials that all do it a slightly different way.

I would like to have a standard process to go by, and once I am comfortable with it, I can choose to deviate. I'd like some input from you pillars of the industry. Is this a good routine for a beginner to follow?

Get Data
Clean Data
Split data into Training and Test Data ~(80/20)
Separately, for training and test sets:

Normalize Data (continuous features):

standardize (divide by std. deviation)
center (subtract mean)

Impute missing values
Feature Engineering
Encode Categorical Variables:

Integer Encoding
One Hot Encoding
Target Encoding
Weight of Evidence

Separate labels from Test set if classification problem. Keep aside.
Choose a few models.
for each model, using k-fold cross-validaton:

Train base model on "training set".
Tune and test hyper parameters on "validation set"
Save best scores and parameters

Compare each model's final scores on the never touched test data
Choose the model with highest scores.

Edit: Thank you for the overwhelming number of responses. Lots of times my questions get a single answer or none at all. I appreciate the time taken to help out a beginner.

I have edited the steps above to reflect the wonderful answers below. I hope that this helps another beginner somewhere else.


texts:
I am using Keras to classify images. I am following the Keras blog. The accuracy from  predict_generator is not matching with the accuracy obtained from the confusion matrix, which I am computing using the scikit-learn package. I have included the relevant snippet of the code below

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
import numpy as np
import theano
from sklearn.metrics import classification_report, confusion_matrix
y_actual = np.ones((nb_test_samples),dtype = int)
y_actual[0:2817] = 0

train_datagen = ImageDataGenerator(
             featurewise_std_normalization=False,
             samplewise_std_normalization=False,
     rescale = 1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
                    train_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
                    class_mode = 'binary')

test_generator = test_datagen.flow_from_directory(
            test_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
            class_mode = 'binary',
            shuffle = False         )

model.fit_generator(
                    train_generator,
                    samples_per_epoch = nb_train_samples,
                    nb_epoch = nb_epoch,
                    validation_data = test_generator,
                    nb_val_samples = nb_test_samples)

score =   model.evaluate_generator(
                      test_generator,
                      4938)
print "Test fraction correct (Accuracy) = {:.2f}".format(score[1])
prediction = model.predict_generator(test_generator,nb_test_samples)

for i in xrange(0,len(prediction)):
    if prediction[i]<0.5:
       prediction[i] = 0
    else:
       prediction[i] = 1

#y_predicted = test_generator.classes
print np.sum(prediction)
CM = confusion_matrix(y_actual,prediction)
print CM

If I use y_predicted, I get a perfectly diagonal confusion matrix, when the console output shows an accuracy of 70% which doesn't make any sense at all. What is that I am doing wrong?


texts: UniLM - Unified Language Model for summarization

texts:
The UniLM claims to be the best approach for summarization task. But there doesn't seem to be any tutorial or how-to section in the README.md or any other blog. How exactly can I use this state-of-the-art library for abstractive summary generation?
Github link
Paper
P.S. A newbie in NLP. Sorry if this is a dumb question.


texts: Data visualization with extreme far away points

texts: I want to show points across two groups. However, for both groups, there are some points which are far away from most of the other points within each group, shown below. Any suggestions for this situation? Thank you.


texts: How to find the relationship between one dataset and two others?

texts:
From a STEM problem, I vary a variable x within a range and calculate quantities $U(x)$, $V(x)$ and $W(x)$. I want to figure out an analytical relationship between unknown $U(x)$ and the other two quantities.
However, in my setup, I have 3 vectors: $A(x), B(x)$ and $C(x)$. Each vector results in a value of $V(x)$ and $W(x)$. This means that my $V(x)$ and $W(x)$ are in fact: $V_A(x),V_B(x),V_C(x),W_A(x),W_B(x)$ and $W_C(x)$ (where the subscripts indicate the vector used to calculate the quantity).
However, to calculate $U(x)$, I must choose a pair of distinct vectors. So, my $U(x)$ are in fact $U_{AB}(x), U_{AB}(x)$ and $U_{BC}(x)$ (where the subscripts indicate the chosen pair of vectors).
I have reason to believe that each of my 3 $U(x)$ can be expressed entirely in terms of the 6 $V(x)$ and $W(x)$ terms. However, I am having trouble finding this relationship.
On top of that, $U(x)$ takes on integer values while $V(x)$ and $W(x)$ are smooth functions with only a few discontinuities. So, I tried various combinations of derivatives, sigmum functions, etc to try and recover $U(x)$ without explicitly calculating it. However, I am now out of ideas.
I am not too familiar with data science methods, and so wanted advice on how I could go about finding an analytic relationship between my quantities. I figured I should try to use machine learning (where the software could try various relationships until a match is found), but I haven't done that before. So, I was hoping for advice on a path to take to try and figure my problem out. I have access to supercomputing nodes, if it's anything.


texts:
I'm trying to classify my textile design patterns
(let's just think of it as medieval painting)
what I understand of "multilabel classification" is like this:
it outputs multiple possible result out of all those classes (let's say classes are of some artists, style and technique)
so one example could be

possible classes: leonardo, artist1, artist2, baroque, renaissance, whatever, oil, dessin, watercolor
prediction of img1: leonardo da vinci, artist1, renaissance, oilpainting, wtaercolor

but what I want to do is more like:
 - possible classes:

artist
style
technique

leonardo
baroque
oil

artist1
renaissance
dessin

artist2
whatever
watercolor

prediction: {artist: ['leonardo', 'artist1'], (maybe they drew it together)
style: ['renaissance'],
technique: ['oil', 'watercolor']}

classes are more strictly categorized and but also there could be multiple results from one category of class.
I'm not even sure what should it be called and having hard time to find articels for it.
Can someone please suggest?


texts:
I am currently working with a dataset with complex numbers, and wanted to know how neural network handles situation as such.

What kind of activation should one use, what form of normalisation would sense and so on.


texts: Keras: Merged/Concatenated model perform worst than separate models memes recognition

texts:
I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together.

Right now I have two models, a VGG16 fine tuned CNN for images and a LSTM model for text, each of them with Keras.

Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.

I'm concatenating the output of each model like this:

CNN

input_tensor = layers.Input(shape=(image_size,image_size,3))

vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False)

for layer in vgg_model.layers:
    layer.trainable = False

l2_strength = 1e-5
dropout_prob = 0.5

x = vgg_model.output
x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x)
x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)
x = layers.Dropout(dropout_prob)(x)
x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)

vgg_model = models.Model(vgg_model.input, x)

LSTM

input_text = layers.Input(shape=(1, 512))
x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text)
x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x)

text_model = models.Model(input_text, x)

MERGED

x = layers.concatenate([vgg_model.output, text_model.output])
out = layers.Dense(1, activation='softmax', name='output_layer')(x)

merged_model = models.Model([vgg_model.input, text_model.input], out)

merged_model.compile(loss='binary_crossentropy',
                     optimizer='adam',
                     metrics=['accuracy'])

and I train the model like this:

merged_model.fit(
    [train_image, train_text],
    y = train_y,
    epochs=50,
    batch_size=64,
    validation_split=0.2)

where train_images are the images and train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.

The problem is that both train and validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.

I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.


texts: Multiclass semantic segmentation with some classes possibly not present in some of the images

texts:
Let's assume we have a large annotated dataset with 4 classes. In this dataset, there might be annotated images with less than 4 classes, where the remaining classes might or might not be present. As an example, say we want to detect pedestrians/cars/bicycles/roads in images. In our dataset, there are some annotated images with only 3 classes: pedestrians/cars/bicycles, but this does not mean there are no roads in these images. That is, there might be roads in these images that were ignored by the annotator for some reason, or there might no be roads at all. My question is, how do we take this uncertainty into the loss term?
One option is to work with independent networks for each class. But what if we want to train a single network? how do we add something like a "don't care" term for objects not annotated in an image, but might still be present in the image?


texts: What is the difference between filling missing values with 0 or any othe constant term like -999?

texts:
Most of the text book says to fill missing values use mean/median(numerical) and most frequent(categorical) but I working on a data set which has too many missing values and I can't remove those columns because they are important.

train.isnull().sum()

TransactionID          0
isFraud                0
TransactionDT          0
TransactionAmt         0
ProductCD              0
                   ...
id_36             449555
id_37             449555
id_38             449555
DeviceType        449730
DeviceInfo        471874
Length: 434, dtype: int64

train.shape
(590540, 434)

You can see that DeviceType and DeviceInfo has too many missing values. I am not sure if filling these with most frequent value would be a right choice. So, I am thinking about filling it with some constant term 0 or -999. Is there any difference when fill it with 0 or -999. What is the right way to fill when you have too many missing values?


texts: Multiply Tensorflow sequential layer by fixed integers

texts:
I'm trying to make a simple reinforcement learning model that makes one of three decisions, A, B, or C.  However, C will be effectively a "do nothing" decision.  Because C will never result in an action, it should always have a value of 0.

So all I want to do is multiply the 3 output nodes by 1, 1, 0.  That is, I want to do an element-wise multiplication by a [1,1,0] tensor so that the A and B nodes retain their values from the previous layer and the C node always has a value of 0.

I see that there's a Multiply layers class, which I think might work, but I'm seeing examples with the functional API.  Is there another way to do this with the Sequential API?


texts: How can I get accuracy of a predicted value specifically in Python?

texts:
I am currently working on a disease prediction machine learning model. I used Random Forest Classifier in my model, and now I am trying to get probabilities or accuracies of predicted values, but the code gives me an error. In this program, I want to get the probabilities or accuracies of "each" prediction specifically. For example, I entered the symptoms to predict the disease and the predicted disease is "Allergy". Then, I want my program to show the accuracy of the predicted disease "Allergy" as a percent, but the program gives errors like "classification metrics can't handle a mix of multiclass and unknown targets". I guess I need to use confusion matrix to show the probabilities, but it also gives the same error about the multiclass problem. More clearly, I just want to show the probability of each predicted value as "percent". For instance, the probability, or accuracy of the output "Allergy" disease is 90%, etc. How can I do this operation and how can I solve my problem?
Here the relevant codes:
The numbers represent label encoded symptoms of diseases
p=pickle_model.predict([[22,8,50,9,20,47,50,38,0,0,0]])

According to these symptoms, the output is "Allergy", and its label encoded number is 4.
I tried to filter diseases as 4 to get the predicted value specifically to calculate the confusion matrix and its accuracy.
import numpy as np
from sklearn.metrics import confusion_matrix
y_test_flt = y_test
y_test_flt = y_test_flt.query('Disease==4')
y_pred_flt=[]
#matrix = confusion_matrix(y_test_flt, y_pred)

for y in range(1):
    for i in y_pred:
        if y_pred[i] == 0 in y_pred:
            y_pred_flt.append(y_pred[i])
        else:
            pass
        if i == 55:
               break

y_pred_flt

But in this code, I could not filter y_pred. The output is:
[0,0]


texts:
I have read about SVM and understood that for complex divisions, the SVM theoretically plots the data into a higher dimensional plane such that the in the new dimension the data is linearly separable and to achieve this in a practical way, it uses kernel functions which in place of actually transforming the data into a higher plane gives the boundary distance between the boundary and that data point.
But how does this work in the case of an image classifier?
Let's say we need to classify pictures as either dog or cat. In this scenario, a CNN model would learn features like ear size, face shape, nose shape, and other visual characteristics of the training set to classify between a dog and a cat. But what does SVM learn while training and how does it work in this case?


texts: How to solve this ? Value Error : operands could not be broadcast together with shapes (0,) (100,522)

texts:
I am trying to apply a PSO algorithm to train a neural network applied to "User Identification From Walking Activity Data Set" problem. Can be found here.

So i extrated the dataset, and calculated average value and standard deviation between aceleration x,y and z at every 5 seconds for each person that would be the input of my neural network. After this i try to optimize the weights and bias of the neural network with Pyswarms functions, but i get an error which i dont know where it is coming from.

The Code :

import numpy as np
import statistics as st
import pyswarms as ps
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier

#EXTRACTING THE DATASET AND CREATING INPUTS AND TARGETS/LABELS
def load_input():
  data = np.empty((0,3),float)
  for x in range(1,23):
        fich = np.loadtxt("%d.csv" %x,delimiter=",");
        aux = []
        value = 5.0;#5 seconds for each person
        for y in range(0,len(fich)):
            if fich[y][0] <= value :
                aux.append(fich[y,1])
                aux.append(fich[y,2])
                aux.append(fich[y,3])
            elif fich[y,0] > value or y == len(fich):
                data = np.append(data,np.asfarray([[np.mean(aux),np.std(aux),x-1]]), axis =0);
                y = y-1;
                aux[:] = []
                value = value + 5;
  return data

#OPTIMIZING WEIGHTS AND BIAS
def forward_prop(params):
 n_inputs = 2
 n_hidden = 20
 n_classes = 22

 W1 = params[0:40].reshape((n_inputs,n_hidden))
 b1 = params[40:60].reshape((n_hidden,))
 W2 = params[60:500].reshape((n_hidden,n_classes))
 b2 = params[500:522].reshape((n_classes,))

 z1 = X.dot(W1) + b1 # Pre-activation in Layer 1
 a1 = np.tanh(z1) # Activation in Layer 1
 z2 = a1.dot(W2) + b2 # Pre-activation in Layer 2
 logits = z2 # Logits for Layer 2

 exp_scores = np.exp(logits)
 probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
 # Compute for the negative log likelihood
 N = 923 # Number of samples
 corect_logprobs = -np.log(probs[range(N), Y])
 loss = np.sum(corect_logprobs) / N
 return loss

def f(x):
 """Higher-level method to do forward_prop in the
 whole swarm.
 Inputs
 ------
 x: numpy.ndarray of shape (n_particles, dimensions)
 The swarm that will perform the search
 Returns
 -------
 numpy.ndarray of shape (n_particles, )
 The computed loss for each particle
 """
 n_particles = x.shape[0]
 j = [forward_prop(x[i]) for i in range(n_particles)]
 return np.array(j)

##### MAIN FUNCTION #####
data = load_input();
X = data[:,[0,1]];
Y = data[:,2].astype(int)
# Initialize swarm
options = {'c1': 0.5, 'c2': 0.3, 'w':0.9}
# Call instance of PSO
dimensions = (2 * 20) + (20 * 22) + 20 + 22
optimizer = ps.single.GlobalBestPSO(n_particles=100,
dimensions=dimensions,options=options)
# Perform optimization
cost, pos = optimizer.optimize(f, print_step=100, iters=1000, verbose=3)

Examples of Inputs and target matrixes :


texts: Decision Trees Should We Discard Low Importance Features?

texts:
I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_.

In my model out of around 30 features, 20 of them has importance value of zero. Does that mean that I should discard those low importance value features from my model? As I understood the answer is no, but I don't know the reason behind it. Can anyone explain?


texts: Python model persistence without pickling

texts:
Are there any Python machine learning libraries, which allow model persistence without pickling?

More specifically, I want to avoid problems (and time sinks) that could arise upon sharing models with others. (In my case: academic collaborators in non-computational research fields.)


texts: Partial Derivative - Pattern Recognition Bishop section 2.3

texts:
Please refer to page $93$, Pattern Recognition book by Bishop.
This question is related to partial derivative of a term, the proof refers to appendix C.19 which states:
$$\frac{\partial({a^{T}x})}{\partial{x}} = \frac{\partial({x^{T}a})}{\partial{x}} = a$$
and then moves on to take partial of normal probability distribution w.r.t. $\mu$ to get (eqn 2.120):
$$\frac{\partial(x_{n} - \mu)^{T}\Sigma^{-1}(x_{n} - \mu)}{\partial{\mu}}  = (x_{n} - \mu)$$

  I am unable to see how we get the term. I tried checking elsewhere,
  but the explanation "consider $(x_{n}-\mu)$ to be scalar and
  $(x_{n}-\mu)^{T}$ as vector"  does not resonate, as per my understanding both are
  vectors (as $x_{n}$ is vector) - it's their dot product that results in scalar.

I tried solving it step-by-step:
Expand the term, ignoring $\Sigma^{-1}$ (contains no $\mu$ term) which gives:
$$\frac{\partial{(x_{n}^{T}x_{n} - x_{n}^{T}\mu - \mu^{T}x_{n} + \mu^{T}\mu)}}{\partial{\mu}}$$
As per my understanding $\frac{\partial}{\partial{\mu}}$ of: $x_{n}^{T}x_{n} = 0$; of $-x_{n}^{T}\mu = -\mu^{T}x_{n} = -\mu$  but, not sure about $\mu^{T}\mu$

  Given the context, I request help on partials :

  Is $\frac{\partial}{\partial{\mu}}$[ $x_{n}^{T}x_{n} = 0$,  $(-x_{n}^{T}\mu = -\mu^{T}x_{n}) = -\mu$]
  $\frac{\partial}{\partial{\mu}}(\mu^{T}\mu)$
  I have some understanding of calculus, so guide to minimum that'll help me derive equation 2.120.

Magnus & Neudecker will take time (450+ pages) and so I may have to gallop over  another partial on page 94 $$ \frac{\partial{(x_{n}-\mu)^{T}\Sigma^{-1}(x_{n}-\mu)}}{{\partial{\Sigma}}}$$


texts:
I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find.

I'm playing with the  titanic dataset and I want to work with pipelines (In order to avoid data leakage using cross validation). For that reason I'm using two pipelines (one for numerical, one for categorical) + FeatureUnion().

What's the problem? In the numerical pipeline I fill the NaN values of Age and then I create some buckets for that variable. The result of this pipeline would be a dataframe containing all the numerical features + 1 categorical variable. For encoding categorical variables, I use a pipeline for the categorical variables, and then use FeatureUnion to join both datasets. But the problem is that the new variable I create in the numerical pipeline doesn't go into the categorical pipeline, resulting with a dataframe with one categorical variable that hasn't been encoded. How can I solve this?

CODE:

num_pipeline = Pipeline(steps = [
('selector', DataFrameSelector(numerical_features)),
('imputer', df_imputer(strategy="median")), #Numerical
('new_variables', df_new_variables()) #Numerical
])

cat_pipeline = Pipeline(steps = [
('selector', DataFrameSelector(categorical_features)),
('label_encoder', MultiColumnLabelEncoder()) #Categorical
])

full_pipeline = FeatureUnion(transformer_list=[
("num_pipeline", num_pipeline),
("cat_pipeline", cat_pipeline)
])

Thank you for your time

Best regards

EDIT:

I was thinking about using ColumnTransformer as I think it suits better in my example as I have to apply different transformations for different columns, but the problem is that when working with ColumnTransformer the output would be an array with no columns' names, which I think would be hard to deal if we want to use feature selection. That's why I chose Pipelines rather than ColumnTransformer.

Talking about the option of creating the bucket before going into the pipeline, I can't because it's created based on the variable I'm dealing with missing values.

What would be the best option in this case?


texts:
I am a little confused with the input of LSTM. Basicaly my train input data is of shape (53394, 3). I reshaped my 2D data into 3D data in order to set it according to the input of LSTM. I have two configurations:

trainX = (53394, 3, 1)
model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))

trainX = (53394,1, 3)
model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))

I want to understand how input is feed into the LSTM in both the cases. like one neuron is taking one column value or one complete row with three columns is going as input to one neuron in input layer.


texts: What does RMSE points about performance of a model in machine learning?

texts: I am working on Decision Tree algorithm and at the end I calculate RMSE value based on actual labels and predicted values (for regression). Now what I am having difficulty in is in understanding the significance of the RMSE value that I get. I know that the lower the RMSE better is the performance of the model but what RMSE value is considered reasonable low or high? Suppose my RMSE value comes as 20 so does that mean the performance of my model is 80% (means 80% of predictions are correct)?


texts: Would it work to label 1/5th of my data and once I learn a classifier to slowly provide the unlabeled data?

texts:
I need to classify MOOC video scripts into one of 5 classes which specify the intent of the sentence, e.g. explanation, example, summary etc., sentence-wise.

I have 11 courses extracted and selected equal ratio of files from every course based on the number of files I have for each of them. Say course 1 has 100 files and course 2 has 5 times less, then I selected data for labeling such that the labeled daya for course 1 will be 5 times more than the labeled data for course 2.

Due to lack of time, I wanted to label 1/5th of my data, which is about ~8800 sentences in 111 files. My question is.. can this be enough to predict the rest of the data, or approx ~29600 sentences more.. maybe if i slowly introduce it to the model, not at once.. or i don't know.. any ideas are welcome, since I am not too experiences. Also specific text classification algorithms you think work best, would be very helpful to mention too. I am open to trying several, as long as their implementation is not too much time-consuming. I will of course try the most common ones first such as Naive Bayes and SVM.

Oh also, another noob question - do I need to convert my text into vectors for some reason? Or will the classifier do it's own work based on the textual data?


texts: clustering plus linear model versus non linear (tree) model

texts:
a team has to create models that predict the cost of deploying a machine over time. This is a regression problem.

The team is further divided into two groups, A and B.

Group A puts lots of attention on selecting attributes and finding rules that would segment the training set into clusters that are more or less homogeneous, and then uses a linear model to create predictions within the cluster.

Group B does not do clustering first, but instead includes the same attributes that team A uses for clustering into a non-linear model (let's say an ensemble of random forests or gradient boosting machine).

The results are similar (or slightly better using the non linear model).

How are results measured? mean squared error over a hold-out set.

The explanation appears to be that, by definition, tree models segment the population using attributes, so that the segments are as homogeneous or pure as possible, given the attributes.

So the work team A is doing to cluster the instances, the tree model is is also doing per se - because segmentation is embedded in tree models.

Does this explanation make sense?

Is it correct to infer that the approach of group B is less demanding in terms of time? (i.e. the model finds the attributes to segment the data as opposed to selecting the attributes manually)


texts: Prediction model for marketing to prospective customers (using pandas)

texts:
I'm currently working on a part-time project which involves predicting the likelihood of customers going to buy a product using data analytics. The company I'm interning with has given me a customer CSV file with all current customers and their attributes and needs to make a prediction model to classify whether prospects are feasible to pursue or not.

However since they have given me a list of all their successful customers or leads, in marketing terms, is it possible to train a model like K-means with PCA (and k-fold cross validation?) and get results? I have to train my model to fit a value, say 10, which I will add to the CSV, and further test it.

I am using pandas. Another issue is that there is a lot of demographical data, but I managed to overcome it using get_dummies(). The number of columns escalated from about 10 to 47, though.

I'm just entering into the world of data analysis, hence I'm a bit clueless as to what path to take or whether what I'm doing is right.

The exact analysis is called Predictive Lead Scoring/Analysis, in marketing terminology.

EDIT 1

I followed what @HonzaB did and, hence did get a decision tree. However, since I had 40 columns, it looks like this

I had to take a screenshot of it, as it was over 2 MB.

Obviously it's really big, and I have to prune the tree somehow, but I not sure how to do so on pandas. Also, is there any way that I can just generate the best characteristics as a text file or something that can be understood without the help of a data scientist?

EDIT 2

I've read up on a question that is quite similar to what I need to do. Predictive modeling based on RFM scoring indicators. In it there is a link to a paper([Data Mining using RFM Analysis][3]) that talks about rule-based classification. Ideally this is what I need to do, and what is most suitable to the company's need.

I want to know if it's possible to do this on Python/pandas. Or is it possible to traverse the decision tree and generate the rules?

EDIT 3

I found another website Decision trees in python again, cross-validation that uses cross validation and hyperparameter optimisation to get a better solution. Also they have included Python code to get readable code. It's a feasible solution, however it's quite complicated and I can't understand how it works. Will it work?

PS I solved the "really big decision-tree" problem from Edit 1, by reducing max-depth. I didn't know at all.


texts: How to handle multiple datasets of sequential data

texts:
New to data science and I am working on a personal project in sports analytics.
I have data in the form of one dataset for each year/season.

Each dataset has the team's wins, losses, and various offensive and defensive stats.
These will be pandas.DataFrames indexed by team name.

My goal is to predict wins and losses based on this data for each year, and then create a more generalized model to predict next season using all of the previous years data.

I am having difficulty deciding how best to combine the data. I think that it would be very inefficient to combine all of the datasets into 1 and have variables for each stat in the year prior, 2 years prior, etc.
How is this type of sequential data usually handled?

(For clarification, I plan on using gradient boosting via xgboost package and an LSTM via the Keras package)


texts: Random Forest Class Weighting for Logistic Probabilities

texts:
I have a model at work that I am building and am running into some odd outputs from the random forest as it pertains to the probability of response. In my case, the class distributions are very unbalanced, around 45000 records in my training data, and only 300 are responders. So I took a smaller sample of the non-responders, 1000, so that my training data was 1300 total. This worked well, but it appears like the outputted predicted probability needs to be adjusted in some way given that I altered the distribution of the classes.

When I create a gains table, the predicted response rate of the first decile comes out at 50%, but it should be around 2%. I dont have this problem with running an xgboost model with all training variables.

I know that in traditional logistic regression, you can weight the classes if you take a smaller sample so that it can accordingly adjust the predicted probability.

Is there a way using random forest to solve this problem?


texts:
Is there an existing technique allowing to do PCA maximizing not the variance per se, but the variance away from some direction?

Imagine I have high-dim data with two different labels L1, L2 and I want to do PCA on L1-labeled data in such a way that it contains as little as possible information about the variance of the L2-labeled data.

The simplest way would be to do normal PCA on L2 labelled data, take subspace S generated by first several components and then do another normal PCA on the projection of L1-labelled data on the S* subspace (orthogonal to S).

I am skeptical about this approach though because one can still have some important (fo L2) information in S subspace. E.g. it would be good if one had some control on the angle of the subspace one does second PCA on, rather then requiring it to be strictly orthogonal.


texts: What does depth mean in the SqueezeNet architectural dimensions table?

texts: First time reading the SqueezeNet paper. Based on my understanding, a fire module contains a squeeze layer of 1x1 filters and a expand layer of 1x1 and 3x3 filters. If we take fire2 for instance, the input dimension is 55x55x96 and we take 16 1x1 filters to convolve over it. This returns a 55 x55x16 output. We then take the output and apply two convolutions, one with 64 1x1 filters and the other with 64 3x3 filters. We then concatenate the two results to create a final output of 55x55x128. In this case, what does the depth of 2 mean? Also, how do I calculate the # of parameters for each layer?


texts: Time Series Modelling or Simple regression or something else

texts:
PROJECT: I am working on an e-commerce site where digital products can run out so there is need to reorder them 72h before they run out (reordering them sooner is not a problem but having notification a bit later so if the product would sell better that would be a problem because we cannot reorder products in time).
GOAL: is to know if products run out at least 72h earlier.
DATA COLUMNS: sales datetime, product id, current number of products, price of product, what currency it was purchased in, other data like profit currency was used for the purchase…
SIZE: Before grouping I have a few millions of rows after grouping hundreds of thousands so it is a lot of data point but DASK can handle them.
GRUPPING COLUMNS: I have grouped the data by PURCAHSEDATE & ID so each day has the product that were sold with all its feature. Features have been aggregated mostly buy summing (profit, expenses) and mean (percentage features like margin%)
HOW FAR I HAVE GONE WITH THE PROECJT: I have looked up a couple of Kaggle projects online that were focused on use https://www.kaggle.com/tejasrinivas/simple-xgb-starter
PROBLEMS: A.) Some product has been sold in the past but they are selling out in 1-2 days so it is hard to put trendline on it. B.) Some item just has 1-2 days of data because it just started to sell a few days ago. C.) I also have data of products that have been sold a lot for a mid or long run (hundreds of days thousands of times). So I could do time series modelling on the whole of the sales but for each individual item I don't always have data on it
CURRENT RESULTS: I have used XGBOSOT Regression like It predicts well number of products sales after the days is over with all the features, but that is not the goal  - https://www.kaggle.com/tejasrinivas/simple-xgb-starter
PROJECT RECOMMEND:I am trying to use the following pick ideas from the following competitions: https://www.kaggle.com/c/demand-forecasting-kernels-only/notebooks?competitionId=9999&sortBy=voteCount  ,  https://www.kaggle.com/c/competitive-data-science-predict-future-sales/code
GOAL: simple and easy solution, not LSTM or something complicated but something quick and easy (like xgboost regression so if I have more data I can use rapids.ai to GPU teach it) to implement because as I said it is not a problem if it is missing on the time frame on the positive side and the item gets reordered 96h early and not 72h early. I am guessing that somehow, I should shift the dates but as I said in many case items have not enough dates to shift their sales date.


texts: which script can be used to finetune BERT fro squad question answering in hugging face library

texts: I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used now for fine tuning?


texts: What techniques to use for image matching

texts:
I have a database with around 30,000 pictures. All of them are a different object. They are all from a certain perspective, the pictures itself are the same size but the objects vary in size. I want to build a system that you can query with a new picture, and that will return it's nearest neighbor, given that it is similar enough. The queried images will look relatively similar to the originals, there might be some horizontal and or vertical translations, a bit different lightning and sometimes there will be a sticker on a different place. Some queried objects will not be in the set, and that needs to be returned as well. What are good techniques to try and what would be the downside? Getting multiple pictures of each object is infeasible. Here's some ideas, I'm wondering if there is more to try:

Euclidean distance on the raw data (very sensitive but fast)
Use traditional keypoint matching, linear matching is very slow unfortunately
Use (denoising) autoencoder for lower dimensional feature representation, linear match on this encoded space (smallest Euclidean distance, at least faster linear search)
Learn siamese network for linear matching (don't know how fast this works but seems slow too)
Learn deep binary autoencoder onto 28 bits which allows for very quick narrowing of the search space to do one of the previous methods, by using these bits as memory mapping to a list of candidate solutions

Any other ideas?


texts:
I customized the "https://github.com/matterport/Mask_RCNN.git" repository to train with my own data set, for object detection, ignoring the mask segmentation part. Now I am evaluating my results, I can calculate the MAP, but I cannot calculate the F1 score.

I have this function: compute_ap, from "https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/utils.py" which returns the "mAP, details, memories, overlays" for each image. The point is, I cannot apply the F1 score formula, because the variables "precision" and "recalls" are lists.

How can I calculate the F1 score using Mask RCNN?

I am using the standard formula to calculate the f1-score, as shown in the image, is this the correct formula to calculate f1-score for object detection models?

Here is my code:

def compute_ap(gt_boxes, gt_class_ids, gt_masks,
               pred_boxes, pred_class_ids, pred_scores, pred_masks,
               iou_threshold=0.5):

    # Get matches and overlaps
    gt_match, pred_match, overlaps = compute_matches(
        gt_boxes, gt_class_ids, gt_masks,
        pred_boxes, pred_class_ids, pred_scores, pred_masks,
        iou_threshold)

    # Compute precision and recall at each prediction box step
    precisions = np.cumsum(pred_match > -1) / (np.arange(len(pred_match)) + 1)
    recalls = np.cumsum(pred_match > -1).astype(np.float32) / len(gt_match)

    # Pad with start and end values to simplify the math
    precisions = np.concatenate([[0], precisions, [0]])
    recalls = np.concatenate([[0], recalls, [1]])

    # Ensure precision values decrease but don't increase. This way, the
    # precision value at each recall threshold is the maximum it can be
    # for all following recall thresholds, as specified by the VOC paper.
    for i in range(len(precisions) - 2, -1, -1):
        precisions[i] = np.maximum(precisions[i], precisions[i + 1])

    # Compute mean AP over recall range
    indices = np.where(recalls[:-1] != recalls[1:])[0] + 1
    mAP = np.sum((recalls[indices] - recalls[indices - 1]) *
                 precisions[indices])

    return mAP, precisions, recalls, overlaps


texts: Relative feature importance w.r.t hyperparameters

texts: Could changing the hyperparameters of a model change relative feature importance?


texts: Time Resolution Changes in Time Series Forecasting

texts:
I am doing time series forecasting for estimating monthly sales of certain consumer goods SKU.

I have last 3 years of actual sales data, wherein first 2.5 years of data is monthly sales data, but last 6 months data daily sales data is available.

I can certainly add the daily data to get monthly data, but considering that total data points are then only 36 (for 3 months) and it may be slightly challenging to get various models like ARIMA, Exponential models.

Is there a way to use the daily last 6 months of data to improve the accuracy of the model?


texts:
I'm interested in any research materials on voting patterns.

I have a data set of how PMs (members of parliament) voted in my country during last couple of years. Each PM has 3 buttons: Yes, No, Abstain. There are also two special situations: PM can be absent during the voting and he/she can be present but not vote (it's wrong but they do it sometimes). So any particular voting result is represented as a huge vector whose values are yes/no/abstain/present_not_voted/absent.

I'd like to find some similarities in PMs' voting patterns and cluster them in groups based on these similarities. There are some interesting questions like coalition stability and party loyalty which (in theory) could be answered with the data described above. Unfortunately i deeply lack any references.

While googling i've come across some site which formed an undirected graph based on American senators voting. It looked cool but i didn't find any explanation on how such graph can be formed. Intuitively "similar" is a symmetric binary relation so undirected graph representation sounds quite natural. But how can one define "similarity" for my data? Probably there are some standard ways of doing this. Any help will be highly appreciated.


texts:
I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation.

  It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

  Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

  So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

  It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

As far as I understand, the motivation of this approach is that: target encoding requires the knowledge of output, which is not available on the test set. So if we use the means obtained from the whole train set and apply on test set, that may cause overfitting. So instead, we will use other values derived from its subset.

I found some discussions from this post but have trouble of understanding the following points:

1) It seems to me that the his second-level CV is nothing but taking the average of the whole #2-#20 fold. So basically, this is just one-level cross validation, where instead of using the mean of #1 fold, we use that of #2-#20 fold as the mean value for #1 fold. Am I missing something here?

2) Once we obtain the means of all 20 folds, what will we do next? If we average, this is again nothing but taking average of all train set.


texts: What is the use of fit method in sklearn.preprocessing.Normalizer()?

texts:
According to the documentation of fit(self, X[, y]) method of sklearn.preprocessing.Normalizer(), it does nothing and return the estimator unchanged.
I understand that if I intend to normalize data I can simply pass the data to the Normalizer() function. so what is the use of use of fit method. Moreover, normalization is not a learning algorithm so why is there a fit() method?


texts: Looking for smallest set of rows that form a natural key in a data set

texts: I have several sets of text files on hdfs that are exports from relations.  Unfortunately I do not know the structure of the table is, but I do know that each has a multi-part key that defines a row uniquely.  I know through domain knowledge that the key is multi-part (e.g. reporting-date, and item number) and I can identify some columns that are clearly not in the key (e.g. revenue from sale).  What is an effective way to identify potential sets of columns that are natural keys, because they are not duplicated in the observed data?  I can get several days of logs in a few Gig, so python or sql could work.  This seems like an great application for a dictionary, but I am not sure how to approach this.


texts: Post-classification after inference in deep learning models

texts:
I designed a fire detection using Deep Learning binary classification in Keras (fire vs none). It's a simple model with a few layers. In my training dataset, I included both fire and smoke, and they are both detected (all under "fire"; mostly real fires are detected though. Smoke is less accurate).

Now I need to differentiate these two in my detection results. So basically it would be 3 classes (fire, smoke, none ). One solution that came into my mind was building a binary classification model and pipeline it after the main one. So it gets the main detections as input and prints which of the two classes it belongs to.

Is there any other approaches? What are pros/cons of various approaches?

Here is my simple architecture:

def create_model():
  model = Sequential()
  model.add(Conv2D(32, kernel_size = (3, 3), activation='relu', input_shape=(300, 300, 3)))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  model.add(Flatten())
  model.add(Dense(256, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(2, activation = 'softmax'))

  return model

#....
if retrain_from_prior_model == False:
    model = create_model()
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
else:
    model = load_model("checkpoints/model.h5")


texts: Can i use deep learning in my agriculture PHD?

texts:
My PHD is about yield of soybeans and it is typical agriculture theme, but I am pretty good with programming and Python programming language and I have already some deep learning programs which I made by myself successfully. Problem is that model is not very easy to show on the paper.

Can I show model just like a chart or graph? And maybe somebody already solved issue with transporting deep learning model file to "paper"? Is it normal to use deep learning in my agriculture PHD instead common middle age Linear Regression etc?

Inputs and outputs is ok on paper but deep learning model is a little bit controversial in my scientific circles and I don't know how to present it so I'm asking maybe somebody already has successful experience by presenting deep learning model on the paper in science.


texts: Using memory consuming algorithms for On-Premise applications

texts:
Many cutting edge technologies in Data Science consume a lot of computing power and memory. For example, in NLP, many pre-trained models are big, and so are the matrices operations that they involves.
So, let's say we want to use NLP in on-premise softwares (A software that is installed in the client computer), how could we use powerful NLP technics such as Embedding with Bert, without consuming all the client computer memory and power?
I've made some research and could not find anything really useful.
Thanks a lot
Edit: I know models such as DistilBERT exist, but it won't solve issues like big matrices operations which are memory consuming.


texts: Find specific topics with topic modelling

texts:
I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better?

Topic Modelling with Mallet

I noticed that mallet finds interesting topics, but these topics just match partially with the topic i want to identify. So the idea is to iterate over all interesting k and search for a topic that meets the requirements with the highest percentage.

Find with elbow several (good coherence score) k topics (=list of good number of k)
Have a look at terms with a high frequency and (manually) map these terms to desired topics (= word list for each searched topic)
model topics for all good k and search for a good match with the word list

Example

I would like to classify/identify text that is about the topic "pollution of the environment"

[17, 15, 20, 33, 41, 42, 55, 120] k => with good coherence score
[dirt, pollution, emission, contamination] => high frequency, and these words would match with the topic
found with k=33 a topic which match these keywords.

This could be used to classify a unseen text (and later on for labeling the text for supervised learning)

Topic "pollution of the environment"

[('pollution', 0.09756215849203013),
   ('dirt', 0.09028743250670891),
   ('emission', 0.05491609816030263),
   ('contamination', 0.02589802450774354),
   ('sea', 0.02088654660674448),
   ('ocean', 0.017281515729574187),
   ('climate', 0.014694946490348864),
   ('crisis', 0.013676484852403893),
   ('waste', 0.011380904652591419),
   ('smog', 0.009812797051311068)])


texts: Error with Jupyter Notebook and Google Cloud

texts:
so I integrated my google cloud with the Jupyter Notebook and it seemed to be working as follows:

[I 12:13:51.633 NotebookApp] Serving notebooks from local directory: /home/kerenb1

[I 12:13:51.633 NotebookApp] The Jupyter Notebook is running at:

[I 12:13:51.633 NotebookApp] http://localhost:5000/?
token=63dedf4645de55a797dff864c38721f01740048d93557092

[I 12:13:51.633 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 12:13:51.636 NotebookApp]

To access the notebook, open this file in a browser:
    file:///run/user/1001/jupyter/nbserver-3124-open.html
Or copy and paste one of these URLs:
    http://localhost:5000/?token=63dedf4645de55a797dff864c38721f01740048d93557092

But when I did that, it told me that I had to activate my CloudShell and do Web Preview with the port that was given, but I wanted it on my Jupyter Notebook port so I did that and it said:

Error: Could not connect to Cloud Shell on port 5000.
Ensure your server is listening on port 5000 and try again.

Anybody know how to solve this issue?


texts: Calculating classification metrics when "true" label is also generated by another classification model

texts:
I have a binary classification model $A$ and want to calculate its precision and recall for positive and negative classes. The "ground truth" or "true" labels for this model are obtained from another binary classification model $B$, whose precision and recall for both classes I know.
I'm almost sure that the metrics for $A$ will be probabilistic in nature, but I have no idea on how to calculate them, given the performance metrics of $B$. As far as I understand, if I denote the true label by $y$, the label predicted by $B$ as $y_B$ and the label predicted by $A$ as $y_A$, then I have to calculate $P(y_A=i\ |\ y=j)$ where $i,j\in\{0,1\}$. And of course, I know the values of $P(y_A=i\ |\ y_B=j)$ for all $i,j$. So
$$P(y_A=i\ |\ y=j)=\frac{P(y_A=i\ \cap\ y=j)}{P(y=j)}=\frac{P(y_A=i\ \cap\ y=j)}{P(y=j\ \cap\ y_B=1)+P(y=j\ \cap\ y_B=0)}$$
The denominator terms on the RHS can be estimated from the precision-recall metrics of model $B$, but I have no idea how to deal with numerator. Maybe an entirely different approach is needed. Would appreciate any help in this regard!


texts:
How to vectorize one-line text data? I have used tf-idf including bigrams and trigrams but I am not able to get good results.
I have purchase order descriptions which are one-liners and I need to classify.
It is a multi-class imbalanced data and I have a small dataset to train around 700 PO descriptions. The number of classes is 7 and the class distribution is similar to exponential. One class is dominating.
My take is that TF IDF should not work since the term frequency and the IDF frequency will be very small.
Also, can we make some user-defined functions to create vectors? If yes, what should be it?

Please suggest some alternative approaches as  well.


texts: Sentiment Analysis of Movie Reviews using Python

texts:
I am currently doing sentiment analysis using Python. Here I am taking all the reviews from movie dataset and using Naive Bayes algorithm to predict whether the review is positive or negative. From the input dataset, I am using a logic to remove stopwords and after that training my dataset to predict the result. As of now, my test results are predicted, but few results are predicted wrongly.

This was a great film. Jack Halley is comical as usual and Bela Lugos is  hilarious even in his tiny role. I love this flick.

Example, the above sentence is positive, but it is predicting as negative. Similar instance occurs to many other scenarios. If it has some sarcasm, then there are chances that it may predict the other way. But for few direct reviews, it is predicting the other way.
Could you suggest other ways to better my algorithm, so that it increases the accuracy of prediction. Any other efficient methods apart from removal of stopwords? I have been looking for various suggestions through internet but I am new to this topic and hence couldn't find an effective way to proceed. Any good suggestions would be greatly appreciated.


texts: predicting next observation using HMMLearn.multinomialhmm(discrete hmm)

texts:
I have implemented a HMM using hmmlearn:

states = ['healthy','sick']
observations = ['sleeping','eating','pooping']
model = HMM(n_components=2)
model.n_features = 3
model.startprob_ = [0.7, 0.3]
model.transmat_ = [
    [0.8, 0.2],
    [0.4, 0.6]
]
model.emissionprob_ = [
    [0.2, 0.6, 0.2],
    [0.4, 0.1, 0.5],
]

Also, I have a sequence of observations:

obs = np.array([0,0,1,0,2,0,1,2,0,1,0,2,0,1,1,2,0])
obs = obs.reshape(-1, 1)

Now, I would like to predict the next observation (at t+1), but don't know how to do this.

(I've read the documentation but haven't found anything)