texts: Merging sparse and dense data in machine learning to improve the performance

texts:
I have sparse features which are predictive, also I have some dense features which are also predictive. I need to combine these features together to improve the overall performance of the classifier.

Now, the thing is when I try to combine these together, the dense features tend to dominate more over sparse features, hence giving only 1% improvement in AUC compared to model with only dense features.

Has somebody come across similar problems? Really appreciate the inputs, kind of stuck. I have already tried lot of different classifiers, combination of classifiers, feature transformations and processing with different algorithms.

Thanks in advance for the help.

Edit:

I have already tried the suggestions which are given in the comments. What I have observed is, for almost 45% of the data, sparse features perform really well, I get the AUC of around 0.9 with only sparse features, but for the remaining ones dense features perform well with AUC of around 0.75. I kind of tried separating out these datasets, but I get the AUC of 0.6, so, I can't simply train a model and decide which features to use.

Regarding the code snippet, I have tried out so many things, that I am not sure what exactly to share :(


texts: what are the main differences between parametric and non-parametric machine learning algorithms?

texts: I am interested in parametric and non-parametric machine learning algorithms, their advantages and disadvantages and also their main differences regarding computational complexities. In particular I am interested in the parametric Gaussian Mixture Model (GMM) and the non-parametric kernel density estimation (KDE). I found out that if a "small" number of data points is used then parametric (like GMM/EM) are the better choice but if the amount of data points increases to a much higher number then non-parametric algorithms are better. Could someone please explain both in bit more detail regarding comparison?


texts: Can Expectation Maximization estimate truth and confusion matrix from multiple noisy sources?

texts:
Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$:

$$C_i = \begin{bmatrix} 0.98 & 0.01 & 0.07 \\ 0.01 & 0.97 & 0.00 \\0.01 & 0.02  & 0.93\end{bmatrix} $$

where each column relates to the truth, and each row relates to the observation. Eg. if the true event is $B$ then source $i$ will get it right 97% of the time, and observe $A$ 1% of the time and $C$ 2% of the time. We can assume the diagonal elements are > 95%

Given a sequence of $n$ events, where each event $j$ was observed by source $i$  as $O_{i,j}$, it is trivial to estimate the pmf of the truth $T_j$ by solving $P(T_j | O_{1,j},\dots,O_{m,j})$ using Bayesian formula (given some reasonable priors about the probabilities of the events themselves, say, uniform).

However suppose we didn't have the confusion matrices, nor the ground truth, and instead wanted to estimate them both. One algorithm is:

Start with some reasonable confusion matrix for each source $C_{i,0}$
Fixing the confusion matrices $C_{i,k}$, estimate the most likely truths $T_{j,k}$ using Bayes formula
Fixing truths $T_{j,k}$, estimate new confusion matrices $C_{i,k+1}$ based on how often each source got it "wrong" (allegedly)
Repeat the last two steps incrementing $k$ until convergence

This seems like the EM algorithm but I don't know how to show this formally. (No this is not homework.)

1) Is this EM, or some other standard algorithm in data fusion?

2) Does it have convergence guarantees?

3) Does it have any guarantees about the quality of the solution and how well the final confusion matrices will approximate the true confusion matrices?

4) Are there issues about the number of parameters being estimated vs. the number of samples? Eg. it seems there are $n + 6m$ parameters being estimated - the $n$ truths and the $6m$ elements across all confusion matrices (last cell of each column is determined by the others).

EDIT

These two papers describes exactly the problem and how to use EM to solve it:

Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
http://crowdsourcing-class.org/readings/downloads/ml/EM.pdf

LEARNING FROM NOISY SINGLY-LABELED DATA
https://openreview.net/pdf?id=H1sUHgb0Z

So the answers are:

1) Properly interpreted yes this is EM

2) EM in general converges to local optimum

3) Not really. In this problem though since the sources are 97%+ accurate, I expect the estimates to be pretty good

4) I don't think this is an issue - this is a "non-parametric" EM algorithm as the confusion matrices are not parameterized in anyway way. The sample sizes I deal with are in the 100000's+ so shouldn't be an issue


texts: Python and Titanic competition how to get the median of specific range of values where class is 3

texts:
I am trying to solve Kaggle's titanic competition. In the test set, there is only 1 row having Fare value as null. It's so easy to replace it with median or average of all Fare values.

But I am thinking of getting a graph showing relation between fares and classes (1, 2, 3) and fares with Embark field, to check some patterns and to narrow down the range of fare values before doing the calculations to replace null values.

The Embark field having 3 main values: S, Q and C.

The class field is having: 1, 2 and 3 as classes.

Fares are changing based on the class of course.

What I have done so far is:

fig, axs = plt.subplots(ncols=2, nrows=2, figsize=(20,20))
sns.scatterplot(test['Embarked'], y='Fare' , data=test,  ax=axs[0][0])
sns.scatterplot(test['Age'], y='Fare', data=test,  ax=axs[0][1])
sns.scatterplot(test['Pclass'], y='Fare', data=test,  ax=axs[1][0])
sns.scatterplot(test['Sex'], y='Fare', data=test,  ax=axs[1][1])

And the result was:

There is no patterns at the Embarked, Age and Sex plots.

But at the Pclass plot, we can clearly see that there is stricly defined range for class 3.

I tried to check it at the train set:

So the are nearly the same distribution.

How can fill the null value with median of values range of people having Pclass as 3?

the regular expression would be:

test['Fare'] = test['Fare'].fillna(test['Fare'].median())

But I really need to specify the range of values where Pclass=3


texts: Selection of base model for transfer learning

texts:
Is there a golden rule which gives intuition on which base model needs to be used for a give image classification problem.

Most of the articles gives the below details which says how to train the model based on the dataset.

However I was not able to find good reference for the selection of base model.

Thank you


texts:
I am trying to correctly scale data for LSTM.
I have a set of features like this:
x1(t-1), ... xn,(t-1), y(t-1), where the actual label is y(t).
I would like to know: is it correct to scale the t-1 label y(t-1)?


texts: How to approach graphed data for a binary classification system?

texts:
I'm looking for articles and other reading material on handling and modeling graph-shaped data for classification systems. As I've never working with graphed base machine learning anything from novice to advanced would be good. I'm after solving graph problems, but representing data that natively fits into a graph diagram in a way that would benefit machine learning on it.

I would like to experiment with predicting based on a graph of directed interactions between nodes in a system as well as incidental events occurring on one or more of the nodes (In hopes of deriving information from correlations between # of nodes similar incidents happened at, relations between nodes with similar incidents, etc).

Anything from published articles to experience tips and best practices would be highly appreciated.


texts: Is there a pre-trained network trained on RGB-D (4) channels?

texts:
The most used pre-trained networks for computer vision (e.g. ResNet50) are trained on 3 channels (RGB). At the same time, many cameras used in robotics return RGB-D outputs, that is including depth information.
Fine tuning 3-channels pre-trained network on RGB-D outputs force us to give up one of the channels. Instead, is there a publicly available vision pre-trained network trained on RGB-D (4) channels?


texts: Significance of Number of Calls and Reset Call in Ball Tree

texts:
Why does the Scikit Implementation has functions to reset and get number of calls?
How are these parameter important in Trees?
https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html#sklearn.neighbors.BallTree.reset_n_calls


texts: What commercial software should a data scientist purchase?

texts:
I am setting up a work computer and have free reign here. My typical go-to software packages are all freely available, such as Rstudio and Anaconda.

I have thought about investing in commercial BI software such as Tableau or Spotfire, but nobody else in the group (or myself) is a proficient user. Are there other obvious choices I am missing?

At the outset, much of what this group is working on will be exploratory. They are needing some justification to invest further into "data science".


texts:
The objective function  ()   =   [1∑=1Lossℎ(()⋅())]+2‖‖2

where  Lossℎ()=max{0,1−}  is the hinge loss function,  ((),())  with for  =1,…  are the training examples, with  ()∈{1,−1}  being the label for the vector  ().

how to find the sgd with respect to theta for when  ⋅≤1 is it y*x and is it 0 when ⋅>1


texts:
I m new in text analysis and need your advice to help medical students to write properly and correctly. The students describe sicknesses as they  observe them; however, they must use an "official sickness description". I have the data collected from the students and the correct data. Let me explain:

I have a csv table1 containing 300K rows / one column : each row is the description of health condition written by students ( they can be redundant).

I have an other table2 containing almost 200K rows and 3 columns
Column 1: the official name of the condition ( the correct one the student should use)
Column 2: a code ( number)
Column 3: the translation of "the official name of the condition" in another language.

The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.

I am confused:
Should I go for a recommendation engine or a search/rank algorithms?

What are the steps you can identify to accomplish this task?

PS: the final output might be an API where student start to write the description of the sickness and obtain a list of choices/recommendations where they can pick one.
I hope I was clear ! Thanks


texts: Can I arbitrarily eliminate 20% of my training data if doing so significantly improves model accuracy?

texts:
My dataset contains 2000 records with 125 meaningful fields 5 of which are distributed along highly skewed lognormal behavior.

I've found that if I eliminate all records below some threshold of this lognormal behavior (by combining the fields together then filtering for Nth percentile), my model improves in accuracy from ~78% to ~86%, using a highly tuned random forests classifier. This filter is only done after splitting my data into train, test (which is done after SMOTE).

What makes this particularly odd is that that filter improves results across multiple sampling methods.

Is this filtering acceptable behavior? Why might it be resulting in better predictions?


texts: How to define a distance measure between two IP addresses?

texts: I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the similarities of my data points). What tactic can I use for this?


texts: Machine Learning and Natural Language Processing : Project Initiation

texts:
I am in the research phase of a long project and am willing to get some useful feedback from your side about the most appropriate project path to take.

Current situation:

A large team of so called production editors (PEs) is semi-automatically processing MS word files in order to do the copy-editing and layout.
This is currently semi-automatized with word macros where the PEs will apply different macros on different text paragraphs in order to set the appropriate style, position, font etc.
Still many actions are manual and depending on the human eye checks, resulting at the end many mistakes and an average quality

Plan

Start using machine learning, later maybe neuronal networks, statistics probability and artificial intelligence
The usage of machine learning (to start with) could be in the way that the system will learn the manual tasks done by the user (meaning the tool was not able to do those correctly in the automatized way) and then try to apply those in next similar situation
The term of similar situations is important here as this will imply natural language processing (NLP). The staff is dealing with word documents that could contain anything (within a similar layout)

The dilemma of the day is :) :

Where to start from in order to do some use case analyses? What I currently have:

More than 100k word files that are correctly done (except the user mistakes that may still lay around)
A list of rules that apply to the word documents describing the exact styles that needs to be used (non exhaustive)
Basic Machine Learning knowledge (starting with AI studying)


texts: Set value of a column based on values of other columns (Pandas)

texts:
The last time I used Python is some time ago so some things have faded away.

I have a given dataset, with multiple columns. I want to create a new column and set the values based on multiple values (text or value) of other columns.

So in the example below, c1 consists of [a,a,b,b] and c2 of [a,b,a,b]. Now I want the new column c3 to be [1,2,3,4]

All help is appreciated!

import pandas as pd
data = pd.read_excel('data')

data['c3'] = 0

for i in range(len(data.c3)):
    if data.c1[i] == a and data.c2[i] == a:
        data.c3[i] == 1
    elif data.c1[i] == a and data.c2[i] != a:
        data.c3[i] == 2
    elif data.c1[i] != a and data.c2[i] == a:
        data.c3[i] == 3
    elif data.c1[i] != a and data.c2[i] != a:
        data.c3[i] == 4```


texts: I recently came across graph embedding such as DeepWalk and LINE. However, I still do not have a clear idea as what is meant by graph embeddings and when to use it (applications)? Any suggestions are welcome!


texts: Titanic Kaggle Data: Why am I getting lower accuracy on Kaggle submissions than on held-out data?

texts:
I am going through my first solo machine learning project and would like to gain some insight into what I am doing wrong/what is going on here as I am a bit stuck.

I have been applying machine learning to the Titanic data set with SKlearn and have been holding out 10% of the training data to calculate the accuracy of my fitted models. I also use K-fold cross valdation with 10 folds to evaluate the model performance and choose hyper-parameters. I have so far applied logistic regression and a linear Kernel SVM and in both cases I get 78-80% accuracy on the K-fold validation sets and when applying the fitted classifiers to my held-back previously unseen testing data. However when I predict on Kaggle's test data and submit my predictions it comes back with values around 76% which is significantly less than I'd expect, and well outside the variance in the accuracy values I get with K-fold cross validation.

A link to the Jupyter notebook where I do this is provided below: http://nbviewer.jupyter.org/github/AshleySetter/Kaggle_Competitions/blob/master/Titanic_project/Titanic_machine_learning_clean.ipynb

Could anyone give me some insight into what is going on here and what I am doing wrong?


texts: Intuition behind CNN Training Accuracy being Different than Loaded Model predicted on same exact data

texts:
I am trying to display some metrics in my final evaluation of several CNN models that I have trained using Tensorflow/Keras. I want to list the training accuracy for demonstration sake. However, I am slightly confused about something.
When training the model (note I am using deterministic ops and seeding to reproduce results), my model finishes with around 92% training accuracy. Now, when I go to load this saved model, I extract the SAME exact training data and predict on it. Now I receive nearly a 96% accuracy in my evaluation.
I want to verify that my model loading isn't doing anything unexpected. Why is it that my model training accuracy is different than when I predict on the same training samples? Does it have something to do with the fact that I use Dropout layers during training?
When I load a model and use it to predict, I am only extracting the feature weights that exist in the layer that I "extract", correct? Then I apply those feature weights to the raw image's data features. If I am loading a model's weights, how does my raw image data know to traverse the model's architecture to get the same feature dimensions etc. for prediction? I guess I am just a bit confused on how the CNN model uses .predict() on raw data and could use some clarification.
My model architecture:
def build_cnn_model() -> Sequential:
    """
    This is the CNN model's architecture
    """
    weight_decay = 1e-4
    model = Sequential()
    model.add(Conv2D(32, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding = 'same', input_shape = (32, 32, 3)))
    model.add(BatchNormalization())
    model.add(Conv2D(32, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding = 'same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2)))
    model.add(Dropout(0.2))

    model.add(Conv2D(64, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(64, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2)))
    model.add(Dropout(0.3))

    model.add(Conv2D(128, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(128, (3, 3), activation = 'relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2)))
    model.add(Dropout(0.4))

    model.add(Flatten())
    model.add(Dense(512, activation='relu', kernel_initializer = 'he_normal', kernel_regularizer = l2(weight_decay), name = 'fc'))
    model.add(BatchNormalization())
    # output layer
    model.add(Dense(10, activation = 'softmax'))

    # optimize and compile model
    opt = Adam(learning_rate = 1e-3)
    model.compile(optimizer = opt, loss = 'categorical_crossentropy', metrics = ['accuracy'])

    return model

My pseudo-code loading model code that nets me about 96% training accuracy.
model = load_model(args.model_folder) # load model from filepath
feature_extractor = Model(inputs = model.inputs, outputs = model.get_layer('dense').output)
y_train_score = feature_extractor.predict(xtrain, batch_size = 64)


texts:
I am unable to find details for pretrained word2vec models. If someone is about to use a pre-trained model should it be clear what kind of pre-processing was done before the model was trained? Such as if lower cased was applied or removal of specific stopwords?

Glove: Is not referring to any kind of that information except of recommending stanford tokenizer (splitting words).
Google pre trained model: Doesn't give any info for pre-processing of the pretrained model. Refers in a sentence to use a script for the wikipedia training data, but do not mention how the google news were preprocessed before trained.

And in other cases of pretrained models I cant find this kind of information. Someone could assume that there is a specific way of pre-processing for word2vec but I remember that the instructor of stanford NLP course, Richard Socher, that some people are removing stopwords and some others do not.

I have also noticed that people are using different pre-processing methodology and same pre-trained model for the kaggle competition Bag of Words Meets Bags of Popcorn which doesnt make sense for me.

Any one can give some insights?


texts:
I'm pretty new to anything and everything related to this kind of stuff, I was wondering how would I normalize the coordinates of a entity in my game for a nerual network? Would it just be the same normalizng everything else? I don't assume it'd be the same just because how big these numbers are, you'd have to have a BUNCH of date (I think?)

Edit: also I'd like to know if this would be the proper way of giving your neural network entitie's position in a 3d game?

here's an example of the coordinates, 235.56, -2332.93, -8.57


texts:
I have a binary classification problem:

Approximately 1000 samples in training set
10 attributes, including binary, numeric and categorical

Which algorithm is the best choice for this type of problem?

By default I'm going to start with SVM (preliminary having nominal attributes values converted to binary features), as it is considered the best for relatively clean and  not noisy data.


texts: Credit attribution for prediction in recurrent neural nets

texts:
Consider a recurrent neural net, which has access two inputs sequences

  x1,x2,x3,x4....

and

  s1,s2,s3,s4...

It emits a predictions

  p1,p2,p3,p4....

where

pi = RNN(si,xi,hi)

where hi is the hidden state formed after looking at s,x from t=0....i-1

I am interested in understanding how much of sequence s and sequence x from t=0..i had an effect on the prediction p at t=i.

I want two scalars between [0,1] which sum to 1 to represent the effect of both the sequences. Say the scalars are 0.01 and 0.99 meaning sequence s was mostly ignored.

Thanks!


texts:
I have audio and its spectrogram of the words in English language. (A spectrogram is a frequency domain representation of a signal) Consider the words: chain, change, chair, chapter. As you can notice, the 'ch' sound is common among the words considered.

Is there an algorithm that will allow me to identify which part of the spectrogram (data) is common between the spectrograms of the words? In other words, is there an algorithm that can identify the part of the spectrogram that represents 'ch' sound in each of the words?


texts: Continuous variable to categorical by quartiles?

texts:
Let's stay I have a field with a continuous variable, like a count of people waiting in line. I want to take those values and create a categorical value based on quartiles. Let's say my range of values is 1 to 80 and the quartiles tell me that a "very short" line is less than 5 people, a "short" line in 6 to 30, a "long" line is 31 to 50 and a "very long" line is >=51

I can think of different ways to write this in python/pandas/numpy but something tell me that one of you can come up with a method/snippet that is short and elegant. Note that I want the method to also generate the quartile values, something that I haven't really done in Python before.


texts: Your Prediction Model works. What to do after?

texts:
One of the most crucial skills of a data scientist is not only to be able to build an accurate predictive algorithm but to suggest a set of actions based on that to enhance the goal ratios.

I have build a churn rate predictor for a retail APP which accuracy after testing is about 90%. So now that I know with high accuracy which users will churn out of the APP in the next days, what can I do to try to retain them. Thinking in the overall APP environment the only idea that comes to my mind is:

Push notifications. CRM, use a recommender system to send push notifications based on users profile and micro-moments.


texts: Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

texts:
I'm following a guide here to implement image segmentation in Keras.
One thing I'm confused about are these lines:
# Ground truth labels are 1, 2, 3. Subtract one to make them 0, 1, 2:
y[j] -= 1

The ground truth targets are .png files with either 1,2 or 3 in a particular pixel position to indicate the following:
Pixel Annotations: 1: Foreground 2:Background 3: Not classified

When I remove this -1, my sparse_categorical_crossentropy values come out as nan during training.
Epoch 1/15
 25/196 [==>...........................] - ETA: 27s - loss: nan - accuracy: 0.0348 - sparse_categorical_crossentropy: nan

Why is this the case? If the possible integer values are 1, 2, 3, why would I need to alter them to begin at 0 to be correctly used?
If I include the -1, the training looks correct:
Epoch 1/15
196/196 [==============================] - 331s 2s/step - loss: 2.0959 - accuracy: 0.6280 - sparse_categorical_crossentropy: 2.0959 - val_loss: 1.9682 - val_accuracy: 0.5749 - val_sparse_categorical_crossentropy: 1.9682


texts: Custom loss function with multiple outputs in tensorflow

texts:
my network has two outputs and single input. I am trying to write a custom loss function
$$
Loss = Loss_1(y^{true}_1, y^{pred}_1) + Loss_2(y^{true}_2, y^{pred}_2)
$$
I was able to write a custom loss function for a single output. But for multiple output, I am struck. Below I wrote a mwe I tried
def model(input_shape=4, output_shape=3, lr=0.0001):
    """
    single input and multi-output
    loss = custom_loss(out_1_true, out_1_pred)+mse(out_2_true, out_2_pred))
    """
    input_layer = Input(input_shape)
    layer_1 = Dense(units=32, activation='elu', kernel_initializer='he_uniform')(input_layer)
    #outputs
    y_1 = Dense(units=output_shape, activation='softmax', kernel_initializer='he_uniform')(layer_1)
    y_2 = Dense(units=1, activation='linear', kernel_initializer='he_uniform')(layer_1)

    def custom_loss(y_true, y_pred):

        # both true value of out_1, out_2 are encoded in y_true
        y_true_1     = y_true[:, :1+output_shape]
        y_true_2     = y_true[:, 1+output_shape:]

        #(this part is wrong...I dont know how)
        y_pred_1, y_pred_2 =  y_pred[:, :1+output_shape], y_pred[:, 1+output_shape:]

        #custorm loss for y_pred_1
        entropy_loss = -y_pred_1 * K.log(y_true_1 + 1e-10)

        #mse for y_pred_2
        mse          = -K.mean(K.square(y_pred_2 - y_true_2))

        #net loss
        loss = entropy_loss + C * mse
        return loss

    Network_model = Model(inputs = input_layer, outputs = [y_1, y_2])
    Network_model.compile(loss = custom_loss, optimizer=RMSprop(lr=lr))

    return Network_model

I think, the main issue lies in spiting the y_pred tensor.
PS: For the purpose of mwe, I have use normal cross entropy loss and mse loss functions in the above code. But however, I have a different cost functions.


texts: Where should we release our video dataset for activity recognition?

texts:
We have an activity recognition dataset made of tens of thousands video clips.

Which are the alternatives to release it to the research community?

Is there any public website?
I saw for instance that mendeley has a datasetweb site.


texts: Algorithms to generate a rating system based on history

texts:
I am a beginner in data science. I have a data set of drivers that has the following attributes available-

Time stamp
Speed
Acceleration
GPS co-ordinates

I need to build a driver rating system to rate drivers on a scale of 1-100 on parameter of speeding. The data set has the data from many drivers and also their past data.
Which is the best algorithm that I can use in this situation?


texts:
I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation.

I have a multivariate dataset with n number of input variables each composed of sequences, and a single output sequence which is unrelated to any of the input variables (e.g. using weekly wind speed and humidity to predict temperature).

I've converted the features and label to batches with fixed time steps and n dimensions, however I'm confused which model should be used. Ideally the output should be a single sequence (e.g. annual temperature) however which model would achieve this? Is this something an LSTM could achieve, or is this a problem for a seq2seq model?

Any suggestions/insightful would be appreciated.


texts: Unrelated output by pytesseract image_to_string function

texts:
I'm trying to extract text from an image but pytesseract is giving a totally different output, the image attached below output is "Werle" (complete different word and characters), I tried many different processing methods like image enhancement, rgb2gray, rgb2binary, still didn't work. What confuses me that the text in the image is very clear and straightforward. I also tried to change the notebook from google colab to my local notebook and checked the library version, but same incorrect result.

output >> "Werle"
here's my code:-
ret,frame = cap.read()
crop_img = frame[320:400,430:840]
text = pt.image_to_string(crop_img)

note 1: this problem happens with other images with the same style but different word
note 2: I tried this in my local machine which uses win10 OS and Google Colab


texts: Appraise the statement: “For the model  = 0 + 1 + , 1 reflects the causal effect of  on .” Ask

texts:
not sure if this was the right place to ask my question, but I saw some questions regarding linear regression so I'd thought I would try to get some answers here. I just started learning about linear regression so this is the homework posed to me.

I assume that the statement is true since 1 is the coefficient for . And its the coefficient that would determine if the slope (i.e. the relationship) is positive or negative.

Am I missing out anything or what should I expound on?

Thanks for reading and for the guides and opinions.


texts: Are there any actual, commercial uses of GANs already?

texts:
Doing research on the internet, I found many scientific papers, ideas, and experiments concerning GANs. But I was unable to find a single example of it being already used commercially.

Q1 can you give examples of companies already using GANs in their product?

Q2 if you're unable to give examples, what's the reason for it? Are GANs too young to be already commercialized? Or maybe companies just have no reason to reveal that they're using this framework to train their AI systems?

PS: I'm aware of some "real world" uses:

Some people use it to create fake news, and other nasty things and probably benefit from it.
Neural photo editor available on Github

But it's hard to call then "commercial" uses.


texts: Slightly different results between scipy.stats.spearmanr and manual calculation

texts:
I have the following dataset.
When I calculate the Spearman correlation coefficient with scipy.stats.spearmanr, it returns 0.718182.

import pandas as pd
import numpy as np
from scipy.stats import spearmanr

df = pd.DataFrame(
    [
        [7,3],
        [6,5],
        [5,4],
        [3,2],
        [6,4],
        [8,9],
        [9,7]
    ],
    columns=['Set of A','Set of B'])

correlation, pval = spearmanr(df)
print(f'correlation={correlation:.6f}, p-value={pval:.6f}')

It returns this:

correlation=0.718182, p-value=0.069096

However, when I tried to calculate it manually:

df_rank = pd.DataFrame(
    [
        [5,2],
        [3.5,4],
        [2,4],
        [1,1],
        [3.5,4],
        [6,7],
        [7,6]
    ],
    columns=['Rank of A','Rank of B'])
cov_rank=np.cov(df_rank.iloc[:,0],df_rank.iloc[:,1])[0][1]

cov_rank/(df_rank.std()[0]*df_rank.std()[1])

It returns a different value.

0.7105597124064275

After the two decimal points are different and I do not know why.

The question is if scipy.stats.spearmanr expect the data to be ranked or not.


texts: How to implement your own word list for sentiment analysis?

texts:
I've finished web scraping and cleaning the text i'm interested in and i also have the sentiment word list i want to apply to it. However, i have a few conceptual and implementation problems.
my sentiment word list is a data frame of words assigned to particular features such as negative, positive etc. I currently have a list where each element is a block of text waiting to be analysed. I'm not sure where to go from here. From the resources i have seen so far, they put their text through a sentiment word list from another module that takes care of it for you and simply gives you the results.
I need clarification on the following:

my sentiment word list contains words and they are either True(1)/False(0) on specific features such as positive and negative. So within my text thats being analysed, if it contains the words that are also in my sentiment word list, they will be added to the document matrix, along with their frequency. The overall positivity and negativity will be proportional the the frequency of positive and negative words. Have i understood this correctly?

How do i go about implementing this? i'm not sure on how to use the vocabulary parameter within Count Vectorizer.

from sklearn.feature_extraction.text import CountVectorizer
vc=CountVectorizer(vocabulary=sentiment_words)
vectors=vc.fit_transform(cleaned_text)

The feature names end up being the features i'm using as an independent variable.
(vc.get_feature_names())
['word',
 'negative',
 'positive',
 'uncertainty',]

The feature names should be the actual words contained within my sentiment word list right? i'm assuming i need to change the sentiment word list dataframe but i'm not sure on how to proceed. Also, how would the overall positive and negative metric be assigned here?
I'm quite new to this so any sort of clarification would be great! Thanks!


texts: How can CSO(cuckoo search optimization), PSO(particle swarm opt.) algorithms be utilized for this dataset?


texts:
From a data stream i'm receiving a pair of measurements consisting of a current consumption and a current percentage every second. By accumulating the consumption over time it will represent eventually the maximum capacity when the percentage reaches from 100% to 0%.

I want to predict the maximum capacity in (almost) real time using linear regression with a small sample size window of two percent. However, when i compare the models of these local regressions of every two percent with the model of the whole data regression, i get very different results due to perhaps local fluctuation. (see figure)

Is there a way to bring the local regression models closer to the whole data model? (in a way that i can see the differences due to fluctuation but overall closer predictions to the whole data model)


texts: I hope to explain in a clear way what I would like to do. I have more than 50000 tweets and I would like to add some labels on topics. So I have used LDA for doing this. I have also used k-means to group them and try to predict the cluster (but not the topic). I would like to know if it is possible to associate to clusters the topics found from LDA or if the two approaches are worthless when used simultaneously.


texts: Ordered elements of feature vectors for autoencoders?

texts:
Here is a newbie question; when one trains an autoencoder or a variational autoencoder, does the order of the objects in the training vector $x$ matter?

Suppose I take an MNIST image image $(28\times28)$ and turn it into a feature vector of size $x \in \mathbb{R}^{1\times784}$. Then does it matter if I e.g. flatten the whole image vertically, or horizontally, or some other fancy way? Or if I were to scramble the order of the elements in the feature vector, would that make the VAE or AE mess up?


texts: What are the disadvantages of Azure's ML vs a pure code approach (R/SKlearn)

texts:
Good Day,
Microsoft offers their Azure Machine Learning Platform: https://azure.microsoft.com/en-ca/services/machine-learning/

Azure Machine Learning is designed for applied machine learning. Use best-in-class algorithms and a simple drag-and-drop interface—and go from idea to deployment in a matter of clicks.
...
Use Azure Machine Learning to deploy your model into production as a web service in minutes—a web service that can be called from any device, anywhere, and that can use any data source.

By their demo and their photos online it looks like a simple GUI application you can drag and drop your various pre-processing elements, estimators, and testing schemes and appears to make it easy to get started on Machine learning projects.
What no website selling a product will do is advertise their downsides.  With a GUI like Weka you lack the ability to fine tune and tweek the different parameters.
Azure ML seems to allow hooks into Python or R to give you more control and it sure would be appealing to have to do less work to get the same results.

What are the known disadvantages of using Azure ML for ML projects vs writing the same process in code (e.g. Sci-kit Learn or in R)?


texts: Parallel distribution in networkx In python 3

texts:
I have a graph with 67084 nodes, 91778 edges. I am unable to plot it using nx.draw(G)
Is there a approach to plot it?


texts: Multiple regression (using machine learning - how plot data)

texts:
I wonder how I can use machine learning to plot multiple linear regression in a figure. I have one independent variable (prices of apartments) and five independent (floor, builtyear, roomnumber, square meter, kr/sqm).

The task is first to use machine learning which gives the predicted values and the actual values. Then you have to plot those values in a figure.

I have used this code:

x_train, x_test, y_train, y_test = tts(xx1, y, test_size=3)

Outcome: LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
         normalize=False)

regr.fit(x_train, y_train)

Outcome:nothing

regr.predict(x_test)

Outcome: array([2.37671029, 3.91651234, 2.98472475])

np.mean((regr.predict(x_test) - y_test) ** 2)

Outcome: 2.976924398032532e-26

How can I plot the actual values of the dependent variable and the predicted ones in the same figure?


texts: Negative examples for a Yes/No image classification neural network

texts: I'm trying to retrain a neural network using transfer learning that can classify whether an image has a certain object, say, a car. My positive sample dataset is quite small, only 2500~ images. It works really well with "regular" binary classification (2500 images of cars/2500 images of flowers and it has to differentiate between those two) but the problem is that I'm not sure how to make it classify for all types of images, or how to make it 2500 images of cars/2500 random images and it'd have to classify whether Yes - the image is a car or No - it's not a car. What would those random images be? Thanks!


texts:
Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of results or the computational effort.

In some cases (tree-based models in particular), scaling has no impact on the performance.

There are many discussions out there about when one should scale their features, and why they should do it. Apart from interpretability (which is not a problem as long as the scaling can be reverted), I'm wondering about the opposite: are there cases when scaling is a bad idea, i.e. can have a negative impact on model quality? or less importantly, on computation time?


texts: Convolution with different dilation rates for each dimension

texts:
How can I get a dilated convolution with different dilation ratio on each axis?
Tensorflow/Keras would be best.

For example, the filter in the gif below would have the following properties:

kernel_size = (3,3)
stride = 1
dilation_rate = (4,2)


texts: How to treat data transformation choices as hyperparemeters?

texts:
While reading the book hands-on ML by Aurelien Geron, I came across this line-

Treat your data transformation choices as hyperparameters, especially
when you are not sure about them (e.g., if you’re not sure whether to
replace missing values with zeros or with the median value, or to just
drop the rows).

How exactly do I do that? Is there a way to do it via sklearn or do I have to manually keep several datasets (each with a different transformation) and then fit models onto all of them?


texts: Keras Early Stopping: Monitor 'loss' or 'val_loss'?

texts:
I often use "early stopping" when I train neural nets, e.g. in Keras:

from keras.callbacks import EarlyStopping

# Define early stopping as callback
early_stopping = EarlyStopping(monitor='loss', patience=5, mode='auto', restore_best_weights=True)

# ...THE MODEL HERE...

# Call early stopping in .fit
history = model.fit_generator(..., callbacks=[early_stopping])

Question: I often wonder if it is better to monitor the loss (monitor='loss') or the validation loss (monitor='val_loss'). Are there some takeaways from the literature? What is best practice?

My intuition would be that monitoring the validation loss gives a more direct feedback of the learning process, since in case the validation loss does not increase any further after some epochs, there seems to be little more to learn.

On the other hand - when I use dropout for instance - the loss will often "lag behind" the validation loss (loss $>$ val_loss). In this case, I believe that there still is something to learn for the model, even if this does not immediately translate into improvements in the validation loss.


texts: Why does the learning rate influence whether i get a error from BCE or not?

texts:
When I use a learning rate higher than 0.001, I get this:
Assertion `input_val >= zero && input_val <= one` failed.

This means that the input I gave to BCE is above 1 or below 0 right? Why does changing the learning rate cause this error? Also, I am passing the input through a sigmoid activation function before passing it to the BCE so it should be between 0 and 1.


texts: Binning long-tailed / pareto data before clustering

texts:
I want to cluster a set of long-tailed / pareto-like data into several bins (actually the bin number is not determined yet).

Which algorithm or model would anyone recommend?


texts: What is a good interpretation of this 'learning curve' plot?

texts:
I read about the validation_curve and how interpret it to know if there are over-fitting or underfitting, but how can interpret the plot when the data is the error like this:

The X-axis is "Nº of examples of training"
Redline is train error
Green line is validation error

Thanks


texts: Why initialization of Xgboost DMatrix reducec features number?

texts:
I am trying to understand following case:

when I create new xgbost DMatrix

xgX = xgb.DMatrix(X, label=Y, missing=np.nan)
based on input data X  with 64 features

I got the new DMatrix with 55 features

What the magic is doing here? Any advise would be great!


texts: Direct Discriminative Pattern Mining for Effective Classification - implementation

texts:
I'm looking for any actual working code implementation of the DDPMiner algorithm mentioned in the
Direct Discriminative Pattern Mining for Effective Classification article form 2008
I'm having real trouble trying to implement it myself.


texts: R: comparison (6) is possible only for atomic and list types

texts:
I am getting the error in the subject line for the following piece of code in R.

r<- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)
for (u in 1:nrow(r))
  for (x in 1:ncol(r))
    r_hat<- function(u,x) {

      if (r_hat >r[u,x]) {
        print(r[u,x]-r_hat)
      }
    }

How can I get rid of the error?


texts:
I am trying to understand how the shape of the image changes after deconvolution ?
I am trying to understand the example code of convolutional autoencoder from neon.

layers = [Conv((4, 4, 8), init=init_uni, activation=Rectlin()),
          Pooling(2),
          Conv((4, 4, 32), init=init_uni, activation=Rectlin()),
          Pooling(2),
          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),
          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),
          Deconv(fshape=(4, 4, 1), init=init_uni, strides=2, padding=0)]

The input_shapes and output_shapes of each layer are as follows

Convolution Layer 'ConvolutionLayer': 1 x (28x28) inputs, 8 x (25x25) outputs, padding 0, stride 1
Pooling Layer 'PoolingLayer': 8 x (25x25) inputs, 8 x (12x12) outputs
Convolution Layer 'ConvolutionLayer': 8 x (12x12) inputs, 32 x (9x9) outputs, padding 0, stride 1
Pooling Layer 'PoolingLayer': 32 x (9x9) inputs, 32 x (4x4) outputs
Deconvolution Layer 'DeconvolutionLayer': 32 x (4x4) inputs, 8 x (7x7) outputs
Deconvolution Layer 'DeconvolutionLayer': 8 x (7x7) inputs, 8 x (13x13) outputs
Deconvolution Layer 'DeconvolutionLayer': 8 x (13x13) inputs, 1 x (28x28) outputs

I understand how the shapes change after convolution ('valid'). (Thanks to http://cs231n.github.io/convolutional-networks/)
How does the stride affect the size of matrix when deconvolution (full convolution) is used ?


texts: What are similarity and distance metrics in classification?

texts:
I have an assignment to train a model to classify text data, the brief for the assignment mentions that any for any learning model used I have to provide a reasoning for the similarity or distance metric used. What does this refer to?

My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Minkowski) or L2 (Euclidean). Would this be correct?


texts: Definition of a model in machine learning

texts: This definition does not quite apply since we are not always assuming an underlying distribution. So what is a model really? Can a Gradient Boosted Model (GBM) with specified hyperparameters be considered a model? Is a model a collection of rules?


texts: Is there any implementation of Extended Isolation Forest algorithm in R/Python?

texts: I am using isofor package for regular Isolation Forest but I came by an article about Extended Isolation Forest and need your advise which package has this functions implemented in R/Python.


texts: Embedding layers for categorical features

texts: Is there a threshold where it is computationally more efficient than one hot encoding to create separate keras embedding layers for each categorical feature > than x categories?


texts: How can a GRU perform as well as an LSTM?

texts:
I think I understand both types of units in terms of just the math.

What I don't understand is, how is it possible in practice for a GRU to perform as well as or better than an LSTM (which is what seems to be the norm)? I don't intuitively get how the GRU is able to make up for the missing cell content. The gates seem to be pretty much the same as an LSTM's gates, but with a missing part. Does it just mean that the cell in an LSTM is actually nearly useless?

Edit: Other questions have asked about the differences between GRU and LSTM. None of them (in my opinion) explain well enough why a GRU works as well as an LSTM even without the memory unit, only that the lack of a memory unit is one of the differences that makes a GRU faster.


texts: Fitting lines through large point clouds

texts:
I have a large set of points (order of 10k points) formed by particle tracks (movement in the xy plane in time filmed by a camera, so 3d - 256x256px and ca 3k frames in my example set) and noise. These particles travel on approximately straight lines roughly (but only roughly) in the same direction, and so for the analysis of their trajectories I am trying to fit lines through the points. I tried to use Sequential RANSAC, but can't find a criterion to reliably single out false positives, as well as T- and J-Linkage, which were too slow and also not reliable enough.

Here is an image of a part of the dataset with good and bad fits I got with sequential Ransac:
I'm using the centroids of the particle blobs here, blob sizes vary between 1 and about 20 pixels.

I found that subsamples using for example only every 10th frame worked quite well too, so the data size to be processed can be reduced this way.

I read a blog post about all the things neural networks can accomplish, and would like to ask you if this would be a feasible application for one before I start reading (I come from a non-maths background, so I would have to do quite a bit of reading)?

Or could you suggest a different method?

Thanks!

Addendum: Here is code for a Matlab function to generate a sample point cloud containing 30 parallel noisy lines, which I can't distinguish yet:

function coords = generateSampleData()
coords = [];
for i = 1:30
    randOffset = i*2;
    coords = vertcat(coords, makeLine([100+randOffset 100 100], [200+randOffset 200 200], 150, 0.2));
end

figure
scatter3(coords(:,1),coords(:,2),coords(:,3),'.')

function linepts = makeLine(startpt, endpt, numpts, noiseOffset)
    dirvec = endpt - startpt;
    linepts = bsxfun( @plus, startpt, rand(numpts,1)*dirvec); % random points on line
    linepts = linepts + noiseOffset*randn(numpts,3); % add random offsets to points
end

end


texts: Importing Common Constants to R and Python

texts:
I use Python for data munging, R for data analysis, and I combine both by running the munging/analysis sequentially in a Makefile. My Python and R scripts depend on various constants. I typically store my Python constants in a settings.py file, and I import this file across several Python scripts which depend on those constants. My R constants are typically not used in more than one script, so I store those constants in the script which uses them.

Some constants are shared by both the Python scripts and R scripts. Right now, I am defining them separately in R/Python files. However, it would be better if there was some way of defining constants in common across Python and R files, since this redundancy can lead to mistakes.

What would be the best way for me to define constant in common across files? Obviously, there is the problem that these are two different languages; so, for example, a list in Python is not quite the same as a list in R. However, there might be some solution which is more elegant than simply repeating constants across files.

I believe this would be the best place for this question, but if it belongs in the Python or R StackExchanges, please let me know.


texts:
I'm looking for commercial text summarization tools (APIs, Libraries,...) which are able to perform any of the following tasks:

Extractive Multi-Document Summarization (Generic or query-based)
Extractive Single-Document Summarization (Generic or query-based)
Generative Single-Document Summarization (Generic or query-based)
Generative Multi-Document Summarization (Generic or query-based)


texts: Training on compressed video, testing on uncompressed images, performance degradation?

texts:
In my application, I have to collect training data from a single camera.
At test-time, the camera frames will be fed live to the network, without being saved in a lossy format in between.
Now I wonder whether collecting the training data as compressed video or jpeg-files might degrade the performance of my network.
I've read "Understanding How Image Quality Affects DeepNeural Network" but this paper only covers the opposite situation: Training on uncompressed imagery and testing on compressed images.


texts: Machine learning model to predict the best candidate

texts:
Problem: I would like to build a machine learning model that can predict the best candidate from any given set. What could be a good architecture for such a model?

Given: I have several training examples, each of which consists of:
- a set of candidates.
- a descriptor for the set as a whole.
- a label that tells which one of those candidates is the best in that set.

Details:
- I will have around 10K such sets.
- The number of candidates in every set may be different (may vary roughly from 10 to 100)
- Every set is unordered.
- The descriptor of each set is currently a fixed length one-hot vector. I'm open to add more features to it though.
- Each candidate is represented by a fixed length feature vector. (However in future, the number of features describing each candidate may also differ for every candidate).

What I tried but didn't work:
One approach I tried was a simple MLP that takes one candidate as input and outputs whether or not the candidate is the best. But since this MLP wouldn't know which set the candidate belongs to, it fails in situations where a candidate is the best in one set but the same candidate is not the best in another set.

To get into some more specifics, in my current problem, each candidate is a 2D polygon with a fixed number of line segments. Labelling on the training examples is being done manually to pick the most good looking polygon in a given set of polylines. Each polygon is described by an array of (x,y) coordinates.

One problem I face is that I don't have a natural starting point for a polygon to begin it's array of (x,y) coordinates from. Currently I'm choosing the starting point to be the one with the minimum value of x+y and going counterclockwise from there.

Currently each 2D polygon has the same number of segments. But I would soon need to support polygons with varying number of segments.

In future, I would like to extend this ML model to 3D polyhedrons too, but I don't know how to even build a feature vector to describe for 3D polyhedron yet. I guess that's a problem for another day.


texts: Visualizing many one-to-many relationships

texts:
I am trying to visualize the language tags on github repository data. I have names of 16k github repositories, and all the languages associated with each one of them. Below is a chord diagram I came up with.

However, I find that the chord diagram is not a very good representation of the data because

It does not show one-to-many relationships. For example, most of the repositories with JavaScript will have both html and css. This is represented as JavaScript-HTML and JavaScript-css, but not all together.
The size of the arc does not represent the number of repos in the dataset. For example, the number of repos that use JavaScript is about 6k in my dataset, however, the arc is about 17k. This is because of multiple languages in each repo. For example, if a repo has JavaScript, HTML, CSS and Python, the length of the arc would be 3.

Do you have any suggestions for a better visualization of this data, Thanks. This particular one is a d3 chord diagram, but the visualization can be done using any package (Python or JavaScript or R).

Here is the dataset if you are interested, do this query on Google BigQuery

SELECT
    sample_repos.repo_name,
    sample_repos.watch_count,
    languages.LANGUAGE
FROM
  `bigquery-public-data.github_repos.languages` languages
INNER JOIN
  `bigquery-public-data.github_repos.sample_repos` sample_repos
ON
  languages.repo_name = sample_repos.repo_name
WHERE sample_repos.repo_name IN (
  SELECT repo_name[OFFSET(0)]
  FROM `bigquery-public-data.github_repos.commits`)
ORDER BY sample_repos.watch_count DESC
LIMIT 16000


texts: How can I improve a machine learning model?

texts:
I am a Machine learning newbie and i am trying my hands with a dataset which has 9 features and my aim is to figure out the optimal multi class classification model which fits my dataset.

I applied PCA on my dataset and reduced the dimensionality to 2 and now i can visualize my dataset and this is how it looks like on my training dataset.

I am not sure how to choose a best model when data is distributed in such a way and i am looking for suggestions on what techniques are out there.

Is Getting more data a viable next step?


texts: What are the differences between Knowledge Graph Embeddings (KGE) and Graph Neural Network (GNN)

texts:
From page 3 of this paper Knowledge Graph Embeddings and Explainable AI, they mentioned as below:
Note that knowledge graph embeddings are different from Graph Neural Networks (GNNs). KG embedding models are in general shallow and linear models and should be distinguished from GNNs [78], which are neural networks that take relational structures as inputs
However, it's still vague to me. It seems that we can get embeddings from both of them. What are the difference? How should we choose which approach if we want to get embeddings?


texts: Predicting invoice data of 12 month using only 1 month data

texts:
I have only 1 month of historical data of invoices can I predict next 12 month of data with good accuracy if it is possible then which model should I used for prediction?
Thanks


texts: Sampling from a multivariate von Mises-Fisher distribution in Python

texts:
I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von Mises distribution. Is there any code available? I have not found yet.

-- edit. Apparently, Wood (1994) has designed an algorithm for sampling from the vMF distribution according to this link, but I can't find the paper.


texts: How to predict on part of image after training on other part of image?

texts:
I have images of identity cards (manually taken so not of same size) and I need to extract the text in it.

I used tesseract to predict bounding boxes for each letter and am successful to some extent but some letters are not bounded.

So, I have around 5000 bounding boxes in all images combined.

I want to train it so as to predict bounding boxes for remaining letters.

After predicting the bounding boxes I will try to classify the image into characters.

This is different from conventional machine learning problem where I donot have training and testing data separately.


texts:
I have measurements of 4 devices at two different points of time. A measurement basically consists of an array of ones and zeros corresponding to a bit value at the corresponding location:

whos measurement1_dev1_time1

Name                         Size               Bytes  Class      Attributes

measurement1_dev1_time1      4096x8             32768  logical

I assume that for a specific device the changes between time 1 and 2 of the measurements are unique. However, since I am dealing with 32768 bits at different locations, it is quite hard to visualize if there is some kind of dependency.

As every bit at location xcan be regarded as one dimension of an observation I thought to use PCA to reduce the number of dimensions.

Thus, for every of the 5 devices:

I randomly sample n measurements at point t1and t2 seperatly
I prepare an array as input for pca() with m*n columns (m< 32768; its a subset of all the observed bits, as the original data might be too big for pca) and 4 rows (one row for each device).
On this array A I calculate the pca: ``[coeff score latent] = pca(zscore(A))```
Then I try to visualize it using biplot: biplot(coeff(:,1:2), 'score', score(:,1:2))

However, this gives me really strange results. Maybe PCA is not the right approach for this problem? I also modified the input data to do the PCA not on the logical bit array itself. Instead, I created a vector, which holds the indices where there is a '1' in the original measurement array. Also this produces strange results.

As I am completely new to PCA I want to ask you if you either see a flaw in the process or if PCA is just not the right approach for my goal and I better look for other dimension reduction approaches or clustering algorithms.


texts: How to retrieve tweets from all cities using Twint package in python?

texts:
Using Twint, I'm able to get tweets from each city in the list, but while creating a dataframe I only get tweet from the last city in the list, is it possible to append the tweets based on cities?
india = ["chennai","pune","mumbai"]
for city in india:
    print(city)
    c = twint.Config()
    c.Search = "phone"
    c.Lang= "en"
    c.Pandas = True
    c.Since = "2021-01-01"
    c.Near = city
    c.Limit = 10
    twint.run.Search(c)
data = twint.output.panda.Tweets_df(["date","tweet","near"])
data.head()

        date                              tweet                              near
0   2021-03-23 18:47:09 Cameron FaulknerThe Black Shark 4 is a high-en...   mumbai
1   2021-03-23 18:42:21 Asad just break Zoya's phone so she won't expo...   mumbai
2   2021-03-23 18:40:13 @Babi3Kat Send you phone number an                  mumbai
3   2021-03-23 18:34:12 @sahilbaani @SamsungIndia ????? 5 Saal purane ...   mumbai
4   2021-03-23 18:33:52 @91mobiles They are just started to make more ...   mumbai


texts: How to make a prediction with an existing model in keras?

texts:
I have a pretrained model that i need to load and make a prediction on two images with. The model is a CNN that takes as input 150x150 RGB images and predicts whether it's a cat or a dog. I've sorted out the tensor formatting and model loading, but the result of the prediction looks off and i cant seem to shed light on what the problem is.
My code below:
import keras
import cv2
import os
import numpy
from keras.preprocessing.image import ImageDataGenerator

def loadAndReshape(path):
    targetImage = cv2.imread(path)
    targetImage = cv2.cvtColor(targetImage, cv2.COLOR_BGR2RGB)
    targetImage = cv2.resize(targetImage, dsize=(150, 150)) / 255
    targetImage = targetImage.reshape(150, 150, 3).astype('float32')
    return targetImage

li = []
li.append(loadAndReshape('.//test1.jpg'))
li.append(loadAndReshape('.//test2.jpg'))
targetImages = numpy.array(li)

model = None
targetDatagen = ImageDataGenerator(rescale=1./255)
targetGenerator = targetDatagen.flow(targetImages, batch_size=2)

if os.path.exists('./Model_cats_dogs_small_dataset.h5'):
    model = keras.models.load_model('./Model_cats_dogs_small_dataset.h5')
    result = model.predict_generator(targetGenerator, steps=2)
    resultSecond = model.predict(targetImages)
    print(f'result: {result} and resultSecond: {resultSecond}')

The original model used generators for prediction so i went ahead and used both predict_generator and predict methods to see the differences.
I was expecting that having 2 images and a binary classifier, the prediction result will be an array containing 2 arrays, each with the probabilities for each class. However, for each prediction i just get only one value for each image, like this: result: [[0.17093319] [0.170154  ]] and resultSecond: [[0.47771692] [0.99125683]]
What am i doing wrong and how can i fix this?


texts: Word embedding microservices in the cloud

texts:
I would like to build nlp classifiers for various tasks such as sentiment analysis, topic modeling, name entity recognition etc. . I realized that most of them only involve simple logistic regression or  as small recurrent neural network taking embedded words as an input. At the moment, I store a dictionary of the word embedding for each of my machine learning classifiers but I would like to replace it by a a micro-service which would be used by each of my machine learning model to retrieve the embedding of words. I looked on the internet and could not find any sources which already implement thi (on AWS for example).

So I have two questions:

Does it seems like a good idea to architect a large "farm" of nlp models
Has any of you come across an open source solution which implement such word embedding microservice on the cloud.

Thanks in advance for your help


texts: Recommendations based on other products seen

texts:
I am trying to develop a basic book recommender system to get in touch with the field and start learning methods and how to prepare the data.

The Dataframe I am using is pretty plain, it has the following structure (this is a simplified example):

    number    type   username  product  publishing_dt   price   genres
0       34  access   kerrigan   130365  2019-12-10      16.99   fantasy, kids
1        1   order   kerrigan    76863  2020-01-15      4.66    action, crime
2        1   order  45michael    76863  2020-01-15      4.66    action, crime
3        1   order   kerrigan    76863  2020-01-15      4.66    action, crime
4        1   order  45michael    86833  2020-02-4       15.65   adventure
5        1   order  45michael    86833  2020-02-4       15.65   adventure
6        1   order  45michael   130365  2019-12-10      16.99   fantasy, kids
7        1   order    alicia7   130365  2019-12-10      16.99   fantasy, kids
8        1   order    alicia7   130365  2019-12-10      16.99   fantasy, kids
9        1   order      john5    86833  2020-02-4       15.65   adventure
10       1   order      john5    86833  2020-02-4       15.65   adventure
11      10  access  45michael    63767  2020-01-24      12.99   adventure, fantasy
12       1   order     uololo   830166  2019-11-03      18.45   action, war
13      25  access     7762hc    84325  2019-11-04      25.60   romance
14       4  access   adrian12   997165  2019-12-16      9.99    health, motivational
15       1   order     kylemm   537077  2019-12-25      8.55    history
16      31  access      yvera    76863  2020-01-15      4.66    action, crime
17       1   order   kerrigan  1023897  2020-02-03      7.99    adventure, humor
18       1   order     angel8    86217  2020-02-01      14.99   guide, travel

To make item-based recommendations (which only take into account metadata information) I've used Count Vectorizer and Cosine Similarity.

For purchase-history-based recommendations, for now, I'm also using Cosine Similarity, I am calculating a similarity score between users, the run order would be this one:

Calculate the most similar users to the one we want to get
recommendations for.
From those users, get their purchase history.
Delete from that list those books the user has already bought.
Return the final list with the recommendations.

In this case, I've tried to apply KNN but the truth is I just couldn't and finally discarded it, but I think that with this data Cosine Similarity is a good choice.

The last scenario I would like to contemplate is the classic "According to items you've seen..." (I don't know how to properly translate that into English, but you get the idea). That's why the number field in the Dataframe is there, it shows the number of times certain user accessed the book file but didn't purchase it. So I've been thinking of various options:

From all the books the user has accessed, take, for example, a random one and return new recommendations based on that one. This could end up being a pretty bad choice as people can make clicks on certain items without even being interested in them.
Search for similarities between the books the user has been seeing
and search for new books with those characteristics. This seems like the correct choice, but I don't really know how to take it into action, it looks like a KNN approach.
Simply show the user those items he has been seeing, ordered by
number of accesses (the more he accessed the file, the more he is
interested in it).

Everything in Python, of course, I'm here searching for some help on where to start on this last scenario specifically, and if someone can recommend me a good tutorial or guide on how to apply KNN in similar cases to this, where I don't have any kind of ratings (feature that I've been noticing it has a lot of weight on recommender systems).


texts:
I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).

Any idea?

How can I troubleshoot these kind of CUDA driver issues?

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.6575
pciBusID 0000:01:00.0
Total memory: 11.00GiB
Free memory: 10.71GiB

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device
: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)

2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro
m device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
built graph


texts: Is the early stopping of xgboost using correct

texts:
I'm using xgboost package in R with early stopping at 75 rounds. To monitor the progress the algorithm I print the F1 score from the training and test set after each round. After the algorithm has done 75 rounds, xgboost returns the model with the highest score on the test set, not the training set. My guess is that this has to do with the monitoring functionality and the watchlist parameter of xgboost

My intention of giving the algorithm access to the test set during training (using the watchlist parameter) was to monitor the training progress, and not to  select the best performing classifier with respect to the test set. That would be cheating right? My question is two-fold:

Is this really cheating?

and if it is cheating,

Why is it programmed into the xgboost package?


texts:
I have data on various standardized tests that certain students took and whether or not they ended up passing a specific class in college. Using the pass/fail as the dependent variable and the different scores on the test for the independent variables, I would like to figure out a metric that best predicts the students success in that class; i.e., Student gets these scores on these tests hence the probability that they pass is so and so.

Logistic regression is my first instinct but I was wondering if there are other, possibly better, ways to approach this problem.


texts:
I want to install boruta in my anaconda environment, but if I execute

conda install boruta

It displays,

PackagesNotFoundError: The following packages are not available from current channels:

  - boruta

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

I guess I could install it using the pip that belongs to the anaconda environment, but I am not sure if that is the best way.

Any suggestions?


texts: Does PMML support probability calibration?

texts: As I need to port a decision tree model from Python to Java, I would like to know whether PMML (Predictive Model Markup Language) supports probability calibration.


texts: ALS in Spark: what loss function is it minimizing?

texts:
I’ve playing with the MovieLens ratings dataset under Spark’s ALS and a manual implementation of ALS and comparing results with the same hyperparameters. I’d like to know this exactly in order to make some experiments that could be reproducible. I’ve seen several papers describing the exact loss formula for ALS in different ways. These seem to be quite common:
$$1)\: min \sum (r_{ui} - \mu – b_u – b_i – p_u^Tq_i)^2 + \lambda (\lVert \mathbf{p_u} \rVert^2  + (\lVert \mathbf{q_i} \rVert^2 + b_u^2 + b_i^2)$$
(The ratings are centered by subtracting an overall bias $\mu$, and for each user, a bias $b_u$ which is a part of the model, and part of the regularization, and a similar bias for each movie $b_i$)

$$2)\: min \sum (r_{uj} – u_i^Tm_j)^2 + \lambda (n_{u_{i}} \lVert \mathbf{ u_i } \rVert^2  + n_{m_{j}} \lVert \mathbf{m_j} \rVert^2)$$

(The penalization for each user-feature and movie-feature vector are multiplied by the number of ratings from each user/movie, and the ratings might be centered beforehand, but the biases are not model parameters).

And I’ve also seen other variations where the l2-norms of the vectors are not squared.

I’ve playing with different variations of these formulas and comparing the results on the same data with Spark’s implementation and my manual implementation, and I cannot find any variation which would make the same hyperparameters in both implementations (the number of latent factors and the value of lambda) give similar results with the same data (e.g. under some formulas, with the same number of latent factors and using the same value for lambda that in Spark gives good cross-validated results, in my implementation results in either overfitting badly or making all the predictions close to zero, but a totally different value of lambda gives comparable results).

What formula does Spark use? Also, how are the initial matrices initialized? I’ve seen some suggestions saying to let the first column of the movie-factor matrix be the average rating for the movie and the rest random-normal numbers, while others suggest random uniform numbers with quite big signs. How is the first movie-factor matrix initialized?


texts: Dropping features after final evaluation on test data

texts:
Would you please let me know if I am committing a statistical or machine learning mal-practice in this procedure?
I want to estimate meteorological variable y1 from ${x_1, ..., x_{10}}$ variables. I use data from different weather stations. I keep some weather stations as test sites/data.
I do feature selection and hyper parameter tuning with cross-validation on training data. My model is Random Forest (RF) and two other tree based models.
Before I evaluate my models on test sites I was skeptic about keeping one of the features: Elevation of the weather station, $x_{10}$. This is a static feature that would be present/same in all rows of data related to a station. Knowing a tiny bit about RF made me worried that the model will use this as a kind of "site_id" and possibly overfit to this feature. It wouldn't make me worried if I was using linear/nonlinear regression models.
So I train my models once with and once without $x_{10}$ as a feature.
Then I evaluate my models on the test sites and turns out that the models without $x_{10}$ do significantly better on test sites.
Even before testing this hypothesis about the static feature, I wanted to do similar tests with dropping other features as well, say $x_{9}$.
Now my question is: now that I know that $x_{10}$ hurts my model, I like to retrain my models without $x_{10}$ and test model performance with and without $x_{9}$ in the filtered feature set.
To me it seems like Im using my test data to kind of filter out my feature so it is not right.
But then, I have this information and if $x_{10}$ is hurting my models in the end, why should I go on testing hypothesis and preparing my models with $x_{10}$ being in them?


texts: Difference between RMSProp with momentum and Adam Optimizers

texts:
According to this scintillating blogpost Adam is very similar to RMSProp with momentum.

From tensorflow documentation we see that tf.train.RMSPropOptimizer has following parameters

__init__(
    learning_rate,
    decay=0.9,
    momentum=0.0,
    epsilon=1e-10,
    use_locking=False,
    centered=False,
    name='RMSProp'
)

while tf.train.AdamOptimizer:

__init__(
    learning_rate=0.001,
    beta1=0.9,
    beta2=0.999,
    epsilon=1e-08,
    use_locking=False,
    name='Adam'
)

What is the conceptual difference if we put beta1 = decay and beta2 = momentum?


texts: Q-learning with a state-action-state reward structure and a Q-matrix with states as rows and actions as columns

texts:
I have set up a Q-learning problem in R, and would like some help with the theoretical correctness of my approach in framing the problem.

Problem structure For this problem, the environment consists of 10 possible states. When in each state, the agent has 11 potential actions which it can choose from (these actions are the same regardless of the state which the agent is in). Depending on the particular state which the agent is in and the subsequent action which the agent then takes, there is a unique distribution for transition to a next state i.e. the transition probabilities to any next state are dependant on (only) the previous state as well as the action then taken.

Each episode has 9 iterations i.e. the agent can take 9 actions and make 9 transitions before a new episode begins. In each episode, the agent will begin in state 1.

In each episode, after each of the agent's 9 actions, the agent will get a reward which is dependant on the agent's (immediately) previous state and their (immediately) previous action as well as the state which they landed on i.e. the agent's reward structure is dependant on a state-action-state triplet (of which there will be 9 in an episode).

The transition probability matrix of the agent is static, and so is the reward matrix.

I have set up two learning algorithms. In the first, the q-matrix update happens after each action in each episode. In the second, the q-matrix is updated after each episode. The algorithm uses an epsilon greedy learning formula.

The big problem is that in my Q-learning, my agent is not learning. It gets less and less of a reward over time. I have looked into other potential problems such as simple calculation errors, or bugs in code, but I think that the problem lies with the conceptual structure of my q-learning problem.

Questions

I have set up my Q-matrix as being a 10 row by 11 column matrix i.e. all the 10 states are the rows and the 11 actions are the columns. Would this be the best way to do so? This means that an agent is learning a policy which says that "whenever you are in state x, do action y"
Given this unique structure of my problem, would the standard Q-update still apply? i.e. Q[cs,act]<<-Q[cs,act]+alpha*(Reward+gamma*max(Q[ns,])-Q[cs,act]) Where cs is current state; act is action chosen; Reward is the reward given your current state, your action chosen and the next state which you will transition to; ns is the next state which you will transition to given your last state and last action (note that you transitioned to this state stochastically).

Is there an open AI gym in R? Are there Q-learning packages for problems of this structure?


texts: Is it possible to fix the validation set when tuning hyperparameters using scikit learn?

texts:
I have a question regarding hyperparameter optimization in scikit learn. I am most familiar with tensorflow where you first split your data into three sets: Train, validation and test. Hyperparameters are optimized using the train and validation sets, and then the model is finally evaluated using the test set. All data is normalized using statistics from the test set, and as far as I have understood the rationale behind this is that the model has not "seen" the data from validation and test and therefore can not use their stats in any way.
Anyways, my question is related to the optimizers in scikit. As far as I can see they all use cross-validation. So I have two questions:

Is there a way to set the validation set explicitly in the scikit optimizers?
If not, how do I deal with the normalization? Should I just feed the optimizer with both the train and validation sets, wouldn't that just mix the sets, and somehow mess up the rationale behind initially using the training set for normalization?

Side note: I want to be able to compare models made using both scikit and tensorflow, so I want to follow procedures that are as similar as possible when comparing models.


texts: How can Time Series Analysis be done with Categorical Variables

texts:
Most of the time series analysis tutorials/textbooks I've read about, be they for univariate or multivariate time series data, usually deal with continuous numerical variables.

I currently have a problem at hand that deals with multivariate time series data, but the fields are all categorical variables. Hence, I was wondering if there is any way to use the standard time series analysis techniques (such as ARIMA, ARMA etc.)

Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the location of the problem etc. are stored in the alert. These fields are all categorical variables.


texts: I am trying to understand how to apply the kernelize the logistic regression is there a step by step tutorial with simple math to do so


texts: Error when trying to run RandomForestClassifer with Pipieline and GridSearch

texts:
I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV

I am getting an error when I fit the data. I am not sure how to fix it.

Will appreciate any help on this.

My code is:

 column_trans = make_column_transformer((OneHotEncoder(), ['CategoricalData']),
 remainder='passthrough')
 RF = RandomForestClassifier()
 pipe = make_pipeline(column_trans, RF)

 # Set grid search params
 grid_params = [{'RF__criterion': ['gini', 'entropy'],
 'RF__min_samples_leaf': [5,10,20,30,50,80,100],
 'RF__max_depth': [3,4,6,8,10],
 'RF_min_samples_split': [2,4,6,8,10]}]

 # Construct grid search
 gs = GridSearchCV(estimator = pipe,
 param_grid = grid_params,
 scoring='accuracy',
 cv=5)

 gs.fit(train_features, train_target)  ----This is where I get an error

 ValueError: Invalid parameter RF for estimator Pipeline(steps=[('columntransformer',
             ColumnTransformer(remainder='passthrough',
                               transformers=[('onehotencoder',
                                              OneHotEncoder(),
                                              ['CategoricalData'])])),
            ('randomforestclassifier', RandomForestClassifier())]). Check the list of available parameters with `estimator.get_params().keys()`.


texts: Exporting Correlation Matrix (from function)

texts:
I'm interested in exporting a correlation matrix to csv. I've tried using the to.csv functionality, but my matrix is obtained from a function and I get an "object has no attribute 'to_csv'" error. I appreciate any guidance. Code is below.

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm as cm

def correlation_matrix(df):

    fig = plt.figure()
    ax1 = fig.add_subplot(111)
    cmap = cm.get_cmap('jet', 30)
    cax = ax1.imshow(df.corr(), interpolation="nearest", cmap=cmap)
    ax1.grid(True)
    plt.title('Feature Correlation')
    labels = None
    # Add colorbar, and make sure to specify tick locations to match desired ticklabels.
    cbar = fig.colorbar(cax, ticks=[-1.1, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6,.8,1])
    plt.show()

correlation_matrix(pd.DataFrame(np.c_[Xs,Y]))


texts: Keras Multiple “Softmax” in last layer possible?

texts:
Is it possible to implement mutiple softmaxes in the last layer in Keras? So the sum of Nodes 1-4 = 1; 5-8 = 1; etc.

Should I go for a different network design?


texts: Is there any option to represent 160 discrete variables each one with their own color in scatter plot?

texts:
Good morning,
I want to know if it is possible to represent with different colors in scatter plot more than 10 parameters instead of putting others and assign a color if the amount of parameters overpass 10. I was following the instructions on the web but the discrete variables are not working. I used the color icon and I assigned colors to all my discrete variables, but when I sent this information to scatter plot it didn't work. The program puts random colors ignoring the ones that I selected. I don´t know if there is any trick to make it work.
Is there any method to apply to represent 160 parameters each one with their own color?

I will be really grateful if someone can help me.
Thank you


texts: Learning rate of 0 still changes weights in Keras

texts: I just trained a model (SGD) with keras and was wondering why the change of accuracy and loss from epoch to epoch doesn't really decrease that much when I lower the learning rate. So I tested what happens when I set the learning rate to 0 and to my surprise, accuracy and loss still changed from epoch to epoch and I can't find an explanation for that. Does anyone know why this could be happening?


texts: Identifying trend and seasonality of time series data

texts:
As a part of a statistical analysis engine, I need to figure out a way to identify the presence or absence of trends and seasonality patterns in a given set of time series data. While most answers and tutorials in the Internet outlines methods to predict or forecast time series data using machine learning models, my objective is simply to identify the presence any such pattern.

Example: daily sales data through a year

This data set could show upward trends with monthly seasonality
or no actual trend with yearly seasonality.

If I am not to manually inspect the scatter plot of the distribution, in what ways can I determine the presence of those patterns?

I have looked at the following approaches so far:

Using moving average or exponential smoothing to smooth the time series curve and then check if the resulting line can be approximated to a linear curve, which is supposed to provide an upwards or downward trend if any.
Using auto correlation to check for seasonality, which I am yet to confirm if it's a possible solution or not (any ideas?)


texts: What is the better way to predict classes for the models developed using the functional API in Keras

texts:
We can predict the class for new data instances using the Sequential classification model in Keras using the predict_classes() function. What is the way to predict the class for models that developed using the functional API?

For example, I have a model (functional API based) with sigmoid activation on the last layer to get probabilities in a multi-label classification. When I apply model.predict(), I got a series of probabilities even though the loss is binary_crossentropy.

I understand that I can do this classification manually e.g. following approach.

test_predict_proba = model.predict(x_test, batch_size=batch_size)
class_predict = (test_predicted_proba > 0.5).astype(int)

I am wondering is there any standard procedure to do the same?


texts: Multilabel classifcation in sklearn with soft (fuzzy) labels

texts:
I have a model which is trained in sklearn on a 5-way classification problem, which performs relatively well (there are kNN and SVM versions, and both reproduce a test set with high accuracy).

When the model is applied in "real life", it is highly likely that many samples will contain linear combinations of multiple classes.  So a sample may be 70% class A and 30% class B.

Much of what I have read about multilabel classification in sklearn relates to problems which don't fit this paradigm well, most of them are "tagging" type problems such as movie genre classification.  Is there a way to apply my SVM/kNN models to this type of problem?  I would prefer to only train on single-class examples but can modify the training set to create some multi-class samples too.

It seems I could work this by simply doing an indivdiual binary classifier for each class.  However, this wouldn't give me the relative strength of each label, i.e. the linear coefficient.  Is that possible?


texts: How to understand log-likelihood for generative image model?

texts: I'm reading a paper on generative image modelling. In the paper, the authors compare various approaches by listing their "negative log-likelihoods" (see screenshot). What does this metric translate to in everyday language?


texts:
I am trying to solve a binary classification problem. My labels are abusive (1) and non-abusive (0).
My dataset was imbalanced (more 1 than 0s) and I used oversampling of the minority label (i.e. 1) to balance my dataset. I have also done pre-processing, feature engineering using TF-IDF and then fed the dataset into a pipeline using 3 classification algorithms namely: Logistic Regression, SVM, and Decision Tree.
My evaluation metrics are:
    Logistic Regression:
[[376  33]
 [ 18  69]]
          precision    recall  f1-score   support

       0       0.95      0.92      0.94       409
       1       0.68      0.79      0.73        87

accuracy                               0.90       496
macro avg          0.82      0.86      0.83       496
weighted avg       0.91      0.90      0.90       496

    SVM:
[[383  26]
 [ 23  64]]
          precision    recall  f1-score   support

       0       0.94      0.94      0.94       409
       1       0.71      0.74      0.72        87

accuracy                               0.90       496
macro avg          0.83      0.84      0.83       496
weighted avg       0.90      0.90      0.90       496

    Decision Tree:
[[383  26]
 [ 28  59]]
          precision    recall  f1-score   support

       0       0.93      0.94      0.93       409
       1       0.69      0.68      0.69        87

accuracy                               0.89       496
macro avg          0.81      0.81      0.81       496
weighted avg       0.89      0.89      0.89       496

The issue I'm facing is that certain new abusive text is being predited as non-abusive. I think that I think my false positive (FP) and false-negative rate (FN) are too high and need to be reduced.
Do you have any suggestions on how to reduce FP and FN or any other suggestions to cater for my issue?
Thanks.


texts: What is the purpose of 'oversampling' when the test set is still unbalanced?

texts:
I understand that both training and testing sets should have the same distribution and also understand that we should not touch the test set (in terms of oversampling). But we know that oversampling the training set (specifically in multiclass classification) totally changes the distribution of the training set. For example:

The distribution of my training set before oversampling is: 90%, 5%, 3%, 2% [for classes A, B, C, and D]
The distribution of my training set after oversampling is: 25%, 25%, 25%, 25% [for classes A, B, C, and D]
The distribution of my training set using stratified cross-validation is: 90%, 5%, 3%, 1% [for classes A, B, C, and D] -->as stratifying keeps the distribution of the original data.

Could someone please explain why do we use oversampling when both training and testing sets need to have the same distribution?


texts:
Are there any standard procedure for designing a CNN?

I wrote some Python code for classifying speech signals using the 1D convolutional model in the Keras environment, but I can't meet the accuracy that I expect from it. In the training process the model reaches 98% UAR, but in the test process it overfits and remains at 58%. I used regularizer and dropout and early stopping tricks to overcome overfitting, but unfortunately, I couldn't do it. For more information, I can't augment the database or change the feature vector.


texts:
I am reading the documentation on tf.lattice (https://www.tensorflow.org/lattice/overview)

I am wondering how the training data was created/trained. Is it represented by (x,y) intpositions and some intensity value output?


texts: Predicting correct match of French to English food descriptions

texts:
I have a training and test set of food descriptions pairs (please, see example below)
First name in a pair is a name of food in French
and second word is this food description in English.
Traing set has also a trans field that is True for correct descriptions
and False for wrong descriptions.
The task is to predict trans field in a test set, in other words to predict
wich food description is corect and which is wrong.

dishes = [{"fr":"Agneau de lait", "eng":"Baby milk-fed lamb", "trans": True},
{"fr":"Agrume", "eng":"Blackcurrants", "trans": False},
{"fr":"Algue", "eng":"Buttermilk", "trans": False},
{"fr":"Aligot", "eng":"potatoes mashed with fresh mountain cheese", "trans": False},
{"fr":"Baba au rhum", "eng":"Star anise", "trans": True},
{"fr":"Babeurre", "eng":"seaweed", "trans": False},
{"fr":"Badiane", "eng":"Sponge cake (often soaked in rum)", "trans": False},
{"fr":"Boeuf bourguignon", "eng":"Créole curry", "trans": False},
{"fr":"Carbonade flamande", "eng":"Beef Stew", "trans": True},
{"fr":"Cari", "eng":"Beef stewed in red wine", "trans": False},
{"fr":"Cassis", "eng":"citrus", "trans": False},
{"fr":"Cassoulet", "eng":"Stew from the South-West of France", "trans": True},
{"fr":"Céleri-rave", "eng":"Celery root", "trans": True}]

df = pd.DataFrame(dishes)

    fr                  eng                                          trans
0   Agneau de lait      Baby milk-fed lamb                           True
1   Agrume              Blackcurrants                                False
2   Algue               Buttermilk                                   False
3   Aligot              potatoes mashed with fresh mountain cheese   False
4   Baba au rhum        Star anise                                   True
5   Babeurre            seaweed                                      False
6   Badiane             Sponge cake (often soaked in rum)            False
7   Boeuf bourguignon   Créole curry                                 False
8   Carbonade flamande  Beef Stew                                    True
9   Cari                Beef stewed in red wine                      False
10  Cassis              citrus                                       False
11  Cassoulet           Stew from the South-West of France           True
12  Céleri-rave         Celery root                                  True

I think to solve this as text classification problem, where text is a concatenation of French name and English description embeddings.

Questions:

Which embeddings to use and how concatenate them?
Any other ideas on approach to this problem? BERT?

Update:

How about the following approach:

Translate (with BERT?) French names to English
Use embeddings to create two vectors: v1 - translated English vector and v2 - English description vector (from data set)
Compute v1 - v2
Create new data set with two columns: v1 - v2 and trans
Train classifier on this new data set

Update 2:

It looks like cross-lingual classification may be the right solution for my problem:

https://github.com/facebookresearch/XLM#iv-applications-cross-lingual-text-classification-xnli

It is not clear yet from the description given on the page with the link above, where to fit my own training data set and how to run classifier on my test set. Please help to figure this out. It would be ideal to find end-to-end example / tutorial on cross-lingual classification.


texts: LSTM component dimensions and freedom of design

texts:
Like many, I'm learning about LSTMs using this rather clear blog post and to some extent this construction of an RNN cell in Pytorch. I've been studying this diagram:

and while I understand the motivation behind the design, I'm having some issues working out the dimensions. (Though others have certainly asked LSTM dimension-related questions, I haven't found one that addresses my point of confusion - don't hesitate to direct me elsewhere if you know of a good source.)

Suppose each vector $x_t$ in the input sequence has dimension $m$, the label $y$ for that sequence has dimension $n$, and there are $T$ terms in the sequence. To my understanding, we would put each $x_t$ through the cell, passing along the output from the cell at $x_t$ as part of the input for the cell at $x_{t + 1}$ and also maintaining the cell state. Then we would take $h_T$ as our output and compare it to the label $y$. Then $h_t$ must be of dimension $n$, since it will ultimately be the output, and the cell state has the same dimension as $h_t$, since $h_t$ is just $\tanh$ of the cell state times the result of the output gate.

My question, then, is this: doesn't that mean that the architecture of our LSTM cell is determined the moment we know the dimensions of our inputs and our labels? The forget, input, and output layers in the cell all must take a vector of size $m + n$ ($x_t$ concatenated with $h_{t - 1}$) to the cell state dimension $n$. This seems bizarre to me, as in the exposure I've had to feed-forward networks previously, choosing the number of layers, number of neurons per layer, and so on has been a fairly important step in model design. Here it seems like we don't have those degrees of freedom.

Where have I gone astray?


texts: In regression, is a higher adjusted R-Squared ALWAYS better?

texts:
Bit of a strange question...I'm trying to put together a multiple regression model.  I have daily temperature as one of my control variables.  I really want to control for temperature, but interestingly, removing it seems to very slightly increase my adjusted-R squared (from 0.23 to 0.24 or similar).

I know theoretically, I should just remove it right, as it adds nothing to my model?  But given that the increase is so small and I quite like controlling for it in my model, would it be totally wrong to keep it in?  If I take it out, am I still "controlling" for temperature in my model?


texts: Is it possible to reuse my CNN trained model over a new dataset with different number of classes?

texts:
I had used a trained CNN model (VGG16) over a large dataset with 6 number of classes in FC layer it gave me a good accuracy ( for example over Testing data: loss=0.59, accuracy=0.829).
When I had applied the same model over a dataset with less samples and labels (4 classes) it reduced the accuracy.
Here my question is : is it possible to use model 1 over the 2nd dataset (bec I had saved the attained weights) or it is not possible due to the difference in classes number?
Also if I rerun my model 1 does the accuracy would be improved or no?


texts: Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

texts:
I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around 5000 examples. I have watched G. Hinton's lectures on Deep Learning and he states that each mini batch should ideally be balanced (meaning each batch should contain approximately same number of data points for each class). This can be approximated by shuffling data and then drawing random batch from it.
But, in my mind this will only work if we have some what large and BALANCED dataset. In my case, I think that setting BATCH_SIZE to be >=16 it might have a bad impact on learning, and network will not be able to generalize. Is it better to maybe use SGD and update weights after each sample has been processed (i.e. online training)? P.S. have in in mind that I am using label smoothing (i.e. class weighted loss)