texts: General practices for building an incremental learning model which never forgets?

texts:
I'm new to datascience and appreciate your sage advice! I need to build an incremental learning model, and I know there's a lot that goes into something like that, but I'd like to highlight the most fundamental, abstract requirement I have in my particular case and ask you to focus your attention on that.
However, I build the incremental learning model it must never forget what it has learned. That is when it learns something new it can't forget something it already learned.
I need to know what algorithm or series of algorithms I can use for that.
I'd like to give a toy example to make sure I'm understood. Imagine I decide to use a neural net as the model that learns incrementally. Imagine, as a toy example, that I train it the following 3 inputs → output observations:
0 → 0
2 → 2
4 → 4
We can even visualize this as a 2D graph (y = x):

We train it, so it has 100% accuracy, great. So we deploy it and now it starts seeing more observations and must learn from them incrementally.
In this example, pretend that when it sees a new input it'll ask me if the model output was correct. If it is correct I want it to remember that specific observation, and never change it, it's fixed. This would mean if it is not correct, all the fixed points in the model do not move.
For instance, let's imagine it see a new input: 1, and comes up with a correct model output: 1. It's memory visualized looks like this:
(notice there is a dot at x=1, y=1)

So let's imagine it sees a new input: 3, and comes up with a model output: 3, and asks me if it was correct and I say, "nope, the output should be 0." Now it has to do some training to learn that new answer, but as I said I don't want it to mess up anything it's already learned.
As I understand typical neural net training, they become more accurate on average, meaning when you train it on a new observation it might mess up the weights and biases of already learned observations, but on average, fewer and fewer.
That is not what I want. I want to incorporate the new information into the model without messing up verified observations. Now, some of you might say, "dude, that's just a lookup table." But no, it's not, because I still want the new observation to have an effect on the un-trained, or un-"verified" observations (like 2.5 for example) like this:
(notice how x=0 is still y=0, and x=2 is still y=2, and x=4 is still y=4, and x=1 is still y=1, but everything else moved?)

Alright, so, I hope I've communicated the requirement effectively, my question is: which algorithm do I use to accomplish this model requirement?
The input data and output data of the model will be more complicated than small integers, but I could conform all inputs to be numerical before it gets to the model. I kinda wonder if the boundary to the data mapping (which is the model) can't be directly derived in an N-D space in exactly the same way it is derived on a 2-d grid in the above toy example. Is there a name for doing that?
Perhaps using something as flexible and sophisticated as a neural net to derive that boundary is not actually necessary, idk, I'd like your opinion.
Of course, when you start talking about algorithms, they come with their own requirements for data structures so, I'm not so much asking about algorithms as I am, just asking about how to do this in general. Like best practices, or design patterns, or something.
I'd really appreciate your feedback, expertise, and experience on this problem. Thanks!


texts: Data points are highly overlapped and do not follow smoothness rule assumption

texts:
I am working on a very high dimensional categorical features based data set. There are two output classes and 2-dimensional PCA plot suggests that the data points belonging to both +ve and -ve classes are distributed all over the 2D space (and at many places in the 2D plot; data points from both classes are very close to each other.

I have used many classifiers like DT, LR, SVM (linear), KNN and as expected they failed to generalize well.

Non-linear classifiers RF, SVM (RBF) and CNN gave slightly improved accuracy (5% improvement over baseline) but they also were not able to do anything magical.

Feature selection over large scale categorical variables is a trouble.

I used auto-encoder for reducing it to 2D space but the results were not generalizable.

I have around 300 data points; so yes more data could have been definitely helpful. But currently I do not have any hope to get any additional data sooner.

Main Question: Are there any reliable machine learning or deep learning classification algorithms in literature which work well when smoothness rule is not followed. (Smoothness Assumption: if two samples x and x′ are close in the input space, their labels y and y′ should be the same).

Thanks in advance!!!


texts: randomForest::varImp VS conditional variable importance

texts:
Data:

My training set consists of ~450k obs and 26 variables, out of which 1 is an ordinal factor (order_month, 12 levels) and the rest is numerical. Moreover, some of my predictors are highly correlated (Pearson's > 0.5), which is expected and sometimes intentional.

I trained a good RF model (ROC 0.96, ~90% specificity & ~90% sensitivity when tested on unseen data) and I derived Variable importance for it using varImp().

Problem:

Now, I know that having factorial and / or correlated predictors could result in biased variable selection in RF models, to which conditional inference trees (CIT) should be a remedy.

I tried to compare results of randomForest::varImp() with party::varimp(myforest, conditional=TRUE) however I get an error in the latter:

Error in model.matrix.default(as.formula(f), data = blocks) :
  term 1 would require 9e+12 columns,

suggesting that I have either too many variables or variables with too many levels.

On top of that it takes over 24hrs to run party::cforest() and running predict() on CIT model crashes my computer altogether. Given the above, I have a couple of

Questions:

1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?

2) what are alternative ways of obtaining conditional variable importance if not through party::varimp()?

Big thanks for your suggestions! Kasia


texts:
I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship between these features that would otherwise be lost.

But, I also have some other features which are naturally in [0,1] (since they're derived from ratios) right from the start. Does it make sense to simply skip scaling of these, since dividing them by the same max value as above (e.g. 10,000) kind of ruins them?

Obviously I've tested it out, and found that it works in practice, but are there any sound mathematical arguments against this? I haven't seen or heard any thoughts on this matter.


texts:
I am looking to employ an unsupervised clustering on a dataset where each observation has a mix of textual and non-textual features.

For each observation, I combine the features into a single vector of ~1000 dimensions. To cluster I have two potential ideas:

Using an autoencoder (or an embedding?) to reduce the dimensionality of the data and then cluster using k-means.
Could I use a topic model? If so, isn't this the superior method in most circumstances to the above?

Why are topic models (in my experience) not commonly used for non-textual data? Is this just a relic of their name/original application, or is there something more fundamental?

Thanks!


texts: Model biased towards low frequency data?

texts:
Generally model gets biased towards data_samples/target whose frequency is high in training data set.
Is it possible during training that model gets biased towards low frequency training data set.


texts: Difference between Bernoulli and Multinomial Naive Bayes

texts:
Here is my understanding of the difference between a Bernoulli and a Multinomial Naive Bayes:

Bernoulli explicitly models the presence/absence of a feature, whereas Multinomial doesn't.

Is there something that I am missing?
Can someone explain why this difference matters intuitively? Perhaps using an example where you would obviously use one instead of the other

On the wikipedia page, they say:

  Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.

Why is that?


texts: Replacing mean by median over batch-size to lessen the impact of outliers

texts:
In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less sensitive to the outliers) to replace the average over batch size in the weights update by a median over batch size computation?

For a batch size large enough, this should lessen the impact the contribution from the outliers. It does not seem to be common though, at least to current knowledge. What are the shortcomings of this approach?


texts: Classification problem approach with Python

texts:
I am a Python beginner, just getting into machine learning and need advice on the approach i should use for my problem.

Here is an example of my data-set.

Where the RESULT is a corresponding INDEX in each VALUES array and every row is a separate array, i need to find the probability of each index of an array being the RESULT based on its configuration, where the overall configuration and distribution is most important rather than any individual value.


texts: Update statement for Logistic Regression

texts:
I don't know which the update statement for Stochastic Gradient descent is the right one for the Newton update of the coefficients.

Does it use the sum of the derivatives for ALL variables to update all the  coefficients or is it iterating through each coefficient and using the sum of the derivatives for only that particular variable to update that coefficient?

On page 120 (pg 139 on pdf) of the Elements of Statistical Learning
book https://web.stanford.edu/~hastie/Papers/ESLII.pdf The Newton
update statement (4.23) looks like all coefficients are updated with the sum of the
derivatives of all rows Xi for ALL the variables
In the update statement in this link http://machinelearningmastery.com/logistic-regression-tutorial-for-machine-learning/
the coefficients are updated with the sum of the derivatives for
each that variable and not the sum of the derivatives for all
variables.
The update statement on slide # 23, pg 21 for http://www.hlt.utdallas.edu/~vgogate/ml/2015s/lectures/lr-nb-lec6.pdf
I read the update statement to say the new coefficient for the given
variable is the old weight + the derivative of that given variable.


texts: Are there known techniques to transform features X classified as C to features Y classified as C'

texts:
I don't think the wording of my question is that clear myself, but I don't have any better words suitable for a title (on top of my head at least). I was wondering if given features X that is classified by a model M as class C, is there a way to find the features Y that is relatively "close" to X so that it would be classified as class C' by M.

I was thinking if some sort of clustering can help such as k-means and then getting the centroid of class C' and using that. The final idea is to get the difference between X and Y to be displayed. Does that sound reasonable? I'm not really a data scientist so want to check up on my thoughts.

If someone can suggest a paper or direction that would be much appreciated

EDIT:
For clarification. The purpose of this is that I have a set of people's skills and their jobs and I want to be able to give an advice of what skills a person needs to cultivate for their desired job.

E.g I can program, have a CS degree, experienced with unix etc. and am classified as software developer (skills are codified into numerical values not text anymore) and I want to work as a chemical engineer. I want to know the skills I would need so that I can be classified as fit to be a chemical engineer.

So X is my set of skills, C is software developer, C' is chemical engineer, and Y is the set of skills appropriate for a chemical engineer that I am looking for.


texts: I need to calculate statistical significance between two time series, each with 4500 terms. My null hypothesis is that $H_0: \mu=\mu_0$ (value in time t of first time series is equal to value in time t-1 of second time series). How can i calculate p value? Is Z test approved for p value calculation? How to get p value after z test? I have $\alpha=0.05$.


texts:
I'm reading a sample machine learning auto-generated report at https://app.jadbio.com/share/e18eafb5-ba15-4743-925c-2e3b3fe6bbbb.
In the bottom of the page, I see:

I don't think I understand the plot.

For the Bladder blue group, they are true positives. I would expect the density is high for the blue group near 1.0. But the report shows very low density for the blue group near the 1.0 end.


texts: model predicting probability close to 50 for positive classes in imbalanced training dataset

texts:
I have a binary classification model where I am predicting the positive class which is only 10% of whole training data set. The issue with this imbalanced data set is my model is predicting probability not more than 0.6 for the positive class.
Why is it happening?

I know it is due to imbalanced data set and there are some answers which suggests how to deal with imbalanced data sets.

I have dealt with imbalanced data sets before and have never encountered such a case where probability for a class is not more than 0.6.


texts:
I have many variables that I am using to predict median gross rent. I want to predict forward 3 years this is what my labels look like:

How do I either preprocess the data in order to get a 3-year estimate or what model can I use to predict forward 3 years?


texts: What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying

texts:
I am really in need of some explanation, I am working on an NLP cyber-bullying detection tool which I will deploy to the web using Django framework, however, am stuck on some idea, can someone explain to me...What is the main concept of using lexical, linguistic, semantic or syntactic approach in (NLP) and how is applied in cyber-bullying or what are the step or why does people say I built this cyber-bullying system from a linguistic approach, I know POS-tagging is way of grouping word and look for dependency in other word, my idea of POS-tagging is a synonym of semantic because POS-tagging is a process linking words to it root and representation word in an understandable context correct me if wrong.

I read an article where a paper tackled a project using a predictive analysis approach with feature extraction techniques, Naive Bayes for classification and to train the model, in the discussion they also spoke on how other team used the Semantics approach to classifier cyber-bullying. I know of data cleaning, tokenization stemming and most of feature extraction model, however, am stuck on the problem of approach which is what is the relevant on lexical, semantic or syntactic and how are they been approached.


texts: How tensorflow understand accuracy and loss of training data?

texts:
Code :-
models = keras.Sequential(
    [
        keras.Input(shape=(28*28)),
        layers.Dense(512,activation = 'relu'),
        layers.Dense(256,activation = 'relu'),
        layers.Dense(10),
    ]
)

models.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(lr=0.001),
    metrics = ["accuracy"]
)

models.fit(x_train,y_train,batch_size=32,epochs=5,verbose=2)

Output:-
Epoch 1/5
1875/1875 - 8s - loss: 0.1854 - accuracy: 0.9450
Epoch 2/5
1875/1875 - 11s - loss: 0.0790 - accuracy: 0.9760
Epoch 3/5
1875/1875 - 7s - loss: 0.0561 - accuracy: 0.9822
Epoch 4/5
1875/1875 - 6s - loss: 0.0423 - accuracy: 0.9865
Epoch 5/5
1875/1875 - 6s - loss: 0.0329 - accuracy: 0.9893

<tensorflow.python.keras.callbacks.History at 0x7efda0e109a0>

While learning TensorFlow, My first doubt was what is accuracy and loss? By google, I Learned:-
Loss value implies how poorly or well a model behaves after each iteration of optimization. An accuracy metric is used to measure the algorithm's performance in an interpretable way. The accuracy of a model is usually determined after the model parameters and is calculated in the form of a percentage.
But my doubt is how TF knows about accuracy as we pass only training data, not Test Data. To calculate accuracy I think test data needed. Is TF split train data into another train and test data?


texts: How to identify and extract patterns from emails

texts:
I would like to know if it would be possible to identify some patterns in a text.
For example, looking at emails, there is some common words used at the beginning and at the end.

Dear Mr Pascal,

We regret to inform you that we will not be able to respect the deadline previously agreed for the delivery of your order. Our supplier has warned us today that they are experiencing supply problems, which will result in a delay in our production chain. We count on your understanding and thank you for your patience.

Please accept our apologies.

Best regards,

Matt

If I look at emails, they usually start with Dear/Hi/Hello/Good morning..., then the title/name of a person/company; the body; and the conclusion (I look forward to hearing from you; kind regards, best regards;....).
I would like to ask you, therefore, if there is a way to collect information about these patterns and also if it is possible to classify emails by the patterns.


texts: Tensorflow: can not convert float into a tensor?

texts:
I'm working on a multi-digit image recognition project, but I'm stumbling across a certain error no matter what I do.

Here's a sample of my code:

graph = tf.Graph()
with graph.as_default():
    x = tf.placeholder(tf.float32, shape = (None, 66, 66, 1), name = 'x')
    y = tf.placeholder(tf.int64, shape = (None, 5), name = 'y')
    keep_prob = tf.placeholder(tf.float32, name = 'keep_prob')

...

with tf.Session(graph = graph) as sess:
    sess.run(tf.global_variables_initializer())

    for step in range(num_steps):
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        batch_images = train_images[offset:(offset + batch_size), :, :, :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        if step % 100 == 0:
            acc = accuracy.eval(feed_dict = {x: batch_images, y: batch_labels, keep_prob: 1.0})

When trying to calculate acc, I get the error TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a float into a Tensor.

I don't know why I'm getting this error. My variables are defined in the same graph and I am not overriding my variables.
Any ideas on what is causing this error?


texts:
I have a DL4J project DL4J Framework for a CNN for Classification and Scaling. (Separate projects). My question is that how can I tune my parameters to achieve my goal?
Note that my question is general-purpose and not related to framework or the problem.

How can I approach tuning parameters for CNNs? I know you need to really know the field of machine learning but I would like to know the approach (even if it means throwing me some reading material) in order to execute a Deep Learning model.


texts: A timestep's prediction depends on future data

texts:
Consider an LSTM model with 100 timesteps, each of which with input and target data. Let f(99) be the function mapping the input data of the 99th timestep and hidden state of the 98th timestep to the output data of the 99th timestep.

The mapping f(99) of another LSTM model will be different if the other LSTM model is fitted only to the data in the first 99 timesteps. It follows that future data -- that is, data from the 100th timestep -- was used to generate the 99th timestep's prediction in the first LSTM model.

What are the implications of this for forecasting methodology?


texts: Missing value in continuous variable: Indicator variable vs. Indicator value

texts:
Most data has missing values, and as far as I'm aware, these are the options:

Imputation (mean, hot-deck, etc.)
Indicator variable. A categorical variable that tells what type the primary variable is. For the missing value case, this is binary. Something still has to be imputed, though.
Indicator value. If the model is powerful enough, it can learn to associate a specific imputed value to certain types of predictions.

In my case, a missing value reveals important information, and thus is Missing Not At Random. From what I've read, most imputation methods don't cover this scenario. Thus, I've opted for the indicator value approach.

My question is: Is there any point adding an additional indicator variable, since I'm already using an indicator value? Am I completely misguided and should I be looking into some other approach?

Example:

| Primary variable (-50 to 50)          | Indicator     |
|-------------------------------------  |-----------    |
| 20.5                                  | 0             |
| -14.2                                 | 0             |
|  0.1                                  | 0             |
| 500 (out of the usual distribution)   | 1             |

I can provide more information of my problem, if it's required to answer the question.


texts: What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?

texts: I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the comparison_description.pdf (here). What is the reasoning behind this approach? Is this methodology model agnostic? What are the advantages and disadvantages?


texts: Layer shape computation in convolutional neural net (pyTorch)

texts:
How can you know the expected input size (image input size (tensor size)), for example for this network (cf. pyTorch tutorial example ):

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

Since it is nowhere explicitely stated. Moreover the comment

self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension

is unclear. What does this shape: (16 * 6 * 6, 120) have to do with image size  (e.g. 32x32 as claimed by the authors of the tutorial) ?
I cannot find a way, by looking at the code, to know what input size is expected by the net?


texts: My training accuray is 1.0 but the predictions on the training data are wrong

texts:
My neural network is not working right, and I am trying to find out what is up.

I inserted just three images to a transfer learning (mobilenet) neural network. The three images' classes are:
array([[0., 0., 0., 1.],
       [0., 1., 0., 0.],
       [0., 1., 0., 0.]])

I did 50 epochs on these pictures and by the 20th epoch or so, the training accuracy stayed at 1.0:

Epoch 50/50
3/3 [==============================] - 6s 2s/step - loss: 1.3671 - acc: 1.0000 - val_loss: 1.3770 - val_acc: 0.0000e+00

Then when I went to predict the outcome of the same three images like so:
predictions_test_2 = model_mn.predict(X, batch_size=1, verbose=1)

the predictions were:
array([[0.2473848 , 0.25099277, 0.251868  , 0.24975444],
       [0.24154082, 0.25245225, 0.25358915, 0.25241777],
       [0.24333884, 0.25127387, 0.25357786, 0.25180945]], dtype=float32)

How could that be if the training accuracy is 1.0?!

This is the code:
def mobilenet(img_rows, img_cols, channel=1, num_classes=None):

model = MobileNet( include_top=True,weights='imagenet')

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='softmax')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:8]:

   layer.trainable = False
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))

model.add(Dense(64, activation='relu'))
model.add(Dense(4,activation='softmax'))

# Learning rate is changed to 0.001
sgd = SGD(lr=1e-6,decay=1e-1,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-6, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0000001, amsgrad=True)
model.compile(optimizer=adam, loss='categorical_crossentropy',metrics=['accuracy'])

# checkpoint
filepath="weights-improvement-mn-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
return model

model_mn = mobilenet(img_rows, img_cols, channel, num_classes)

model_mn.fit(X, Y,batch_size=3,epochs=50,shuffle=True,verbose=1,validation_data=(X_vall, Y_vall))


texts: What is the mean of inconsistent in machine learning and why 1NN is well known to be inconsistent

texts:
I really need help to understand the meaning of consistency in machine learning, why it's important, and why 1NN  is considered to be inconsistent.

Thanks


texts:
Considering a TS from 1-5 blocks and using 1-4 block as train data.
Is it invalid to build mean encode on whole train data / or I should mean encode block 1 / block 1-2 / ... / block 1-2-3-4 ?
Edit 1 : By mean encoding, I mean to encode my categorial features by taking target's mean on associate values.


texts:
I am doing some research on machine learning algorithms in the context of a seminar, which focuses on Energy-Based Modeling vs Deep Learning Modeling specifically in working with images. Now I know these are broad topics, but I was wondering what some specific similarities and differences are. I have read countless papers that say "EBMs have properties that make them better than DL in certain tasks" but I haven't really found a comprehensive motivation

What is the problem that EBM is solving compared to DL?
Why are we doing this?

I hope this is the right site for this question (I also checked theoretical CS), This brought me here.
update
Based on my continued reading I believe, that in simple terms my notion of ebm vs DL is wrong. It is a framework that can incorporate deep learning and thus helps improve interpret ability of deep learning models (fundamentally they are black boxes after all!). It is another point of view - a different pair of glasses to see the same thing, if you will. An example would be, that the probabilities generated by probabilistic EBMs are actual probabilities and allow us to make statements about uncertainty (in contrast to the regular soft Mac output). Furthermore their generative nature allows for better calibration in classification and better out of distribution detection.
Maybe this might help or interest someone perusing this modeling framework


texts:
In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object.

However, I am completely lost when it comes to feature extraction techniques in python. My advisor convinced me to use images which haven't been covered in class.

Can anyone direct me towards meaningful documentation or reading or suggest methods to consider?


texts: How to separate words that are together in a large data set

texts:
in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'

i tried this but this is slow

def split(sentence, word_to_index):
# words to index dictionary of valid words
    sentence_words = sentence.split()
    lst12=[]
    for w in sentence_words:
            lst=[]
            ws=w[:]
            # Set the (i,j)th entry of X_indices to the index of the correct word.
            while True:
                for wd in lst:
                    ws = ws.replace(wd,"")
                if ws=="":
                    lst=" ".join(lst)
                    lst12.append(lst)
                    break
                wrdind = ws[:]
                for kl in range(len(ws)):
                    try:
                        word_to_index[wrdind]
                        lst.append(wrdind)
                        break

                    except KeyError:
                            wrdind = wrdind[:len(wrdind)-1]
                            continue
   return " ".join(lst12)
```


texts:
I am building a CNN model for age classification. Assuming age of a person is between 1-100, my last Linear Layer contains 100 output neuron.
Now i want to find an appropriate loss function for this classification problem.
I dont wish to use Regression
My Observations:

I cannot use MSE or BCE loss because they only works element wise so unsuitable as if actual age is 25 then there will be same loss for predicted age 26 and 50. (Prediction will be 100 element vector as last layer has 100 neurons.)
I found about Hinge Loss and Cosine Proximity Loss. But i dont think they can be used in this type of classifiaction either because they are only finding similarity between two vectors without giving any importance or weight to nearby actual-predicted pairs (ex actual age 25 and predicted age 26 should have a very low loss)

Can anyone suggest me a suitable loss function (Preferrably in Pytorch) for this classification problem?
Edit
Lets say I want a Loss Function (L(predicted, actual)) such that (Assuming for 5 class classification)
let actual = [0,0,1,0,0]

L([0,0,1,0,0], actual) < L([0,1,0,0,0], actual) < L([1,0,0,0,0], actual)
L([0,0,1,0,0], actual) < L([0,0,0,1,0], actual) < L([0,0,0,0,1], actual)


texts: After a Feature Scaling do i have the same initial information?

texts:
I'm studying the gradient descent algorithm for single hidden layer neural networks.
Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.

Why mathematically do the normalized features carry the same information of the initial features?


texts:
I have the task of modeling the current to torque mapping for a given motor. I have an experimental set up where I can retrieve current, torque pairs.

Now my initial approach was to model the relationship with a regression curve, but I realized that the motor certainly shows some kind of hysteresis.

How would I be able to compute a current to torque mapping that also includes hysteresis from the data I obtained


texts:
For undergraduate students who understand the definition of
eigenvectors and eigenvalues,
$$A v = \lambda v \;,$$
what is the intuition behind why the eigenvectors of the
covariance (or correlation) matrix correspond to the axes of maximal stretching?
Why specifically does that matrix lead to (e.g.) the largest eigenvector
corresponding to the direction of maximal spread in the data?


texts: MSE relevance as a metric when errors < 1

texts:
I'm trying to build my first models for regression after taking MOOCs on deep learning. I'm currently working on a dataset whose labels are between 0 and 2. Again, this is a regression task, not classification.
The low y values imply that the loss for each sample is quite low, always < 1. My question is then about the relevance of mse as a metric in such a case : since the loss is < 1, squaring it will result in an even smaller value, making the metric value drop very rapidly. In this case, would it be more relevant to use mae ? Or should I multiply the y values so that the order of magnitude of a sample loss would be > 1.
I found this nice article about regression metrics, but didn't find the answer in it. Thanks for your help.


texts: Linear Regression not working due to wrong kind of array

texts:
I try to deal with my homework. The Job is to take this Data and perform a linear regression on it.
The code is published here.
I am quite new to programming in Python and in data science. So I tried transforming as the interpreter suggests, but it didn't work.
My first error was that there was a 2d array expected but 1d given. Then I took the pure array and put it into an empty one suggested by a StackOverflow answer now the error is that a scalar array is given but a 2d array is given.
import pandas as pd
from sklearn.preprocessing import StandardScaler

#Import
data = pd.read_csv('uscrime.txt', sep="\t")
crime = pd.concat([data], axis = 1)
print(crime)

from sklearn.linear_model import LinearRegression
regression = LinearRegression()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(crime.get("M"), crime.get("Crime"), test_size=0.2, random_state=0)

X_train_new = []
X_train_new.append(X_train.values)

y_train_new = []
y_train_new.append(y_train.values)

regression.fit(X_train_new, y_train_new)


texts: Machine learning models for predicting good machine settings given bad machine settings

texts:
I’ve a dataset about a machine that produces a product. Salient features about the dataset:

•   For each data point, some of the features are machine settings (calibration factor, grind time, pressure, water temperature) used to make the product, while few others are stochastic and external in nature, such as water hardness.

•   Output says whether the product produced was of good or bad quality.

Goal:

•   When I’ve a new data point which says that the product was of bad quality, I want to feed in the corresponding features(machine settings + stochastic external features) to an ML model and predict machine settings which will lead to good quality product. Idea is to have an ML based smart controller which would adjust the machine settings to reduce faulty products being produced.

Which ML models would suit this purpose?


texts: How to mathematically define the architecture of neural network model? And the function space associated with it?

texts:
My goal is to properly define a search space for neural architecture search (NAS).

I think a proper definition must handle the following issues.

how to mathematically quantify the topology?
how to define the number in each layer? activation functions?


texts:
I'm trying to create a conditional GAN with the following code:

learning_rate = 0.0002
batch_size = 128
epochs = 10000

image_dim = 784  #28 * 28
Y_dimension = 10
gen_hidd_dim = 256
disc_hidd_dim = 256
z_noise_dim =100 #input noise datapoint

def xavier_init(shape):
  return tf.random_normal(shape = shape, stddev = 1/tf.sqrt(shape[0]/2.0))

weights = {
    'disc_H' : tf.Variable(xavier_init([image_dim + Y_dimension, disc_hidd_dim])),
    'disc_final' : tf.Variable(xavier_init([disc_hidd_dim, 1])),
    'gen_H': tf.Variable([z_noise_dim + Y_dimension, gen_hidd_dim]),
    'gen_final': tf.Variable(xavier_init([gen_hidd_dim, image_dim]))
}

bias = {
    'disc_H': tf.Variable(xavier_init([disc_hidd_dim])),
    'disc_final': tf.Variable(xavier_init([1])),
    'gen_H': tf.Variable(xavier_init([gen_hidd_dim])),
    'gen_final': tf.Variable(xavier_init([image_dim]))
}

Z_input = tf.placeholder(tf.float32, shape= [None, z_noise_dim ], name = 'input_noise')
Y_input = tf.placeholder(tf.float32, shape= [None, Y_dimension], name='Labels')
X_input = tf.placeholder(tf.float32, shape=[None, image_dim], name = 'real_input')

def Discriminator(x,y):
  inputs = tf.concat(axis = 1, values = [x,y])
  hidden_layer = tf.nn.relu(tf.add(tf.matmul(inputs, weights['disc_H']), bias['disc_H']))
  final_layer = tf.add(tf.matmul(hidden_layer, weights['disc_final']), bias['disc_final'])
  disc_output = tf.nn.sigmoid(final_layer)
  return final_layer, disc_output

def Generator(x,y):
  inputs = tf.concat(axis=1, values=[x,y])
  hidden_layer = tf.nn.relu(tf.add(tf.matmul(tf.cast(inputs, tf.float32), tf.cast(weights['gen_H'], tf.float32)), tf.cast(bias['gen_H'],tf.float32)))
  final_layer = tf.add(tf.matmul(hidden_layer, weights['gen_final']), bias['gen_final'])
  gen_output = tf.nn.sigmoid(final_layer)
  return gen_output

output_Gen = Generator(Z_input, Y_input)

Right after executing the Generator i get the following error:
ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_25' (op: 'MatMul') with input shapes: [?,110], [2].

What to do?


texts: How to know if my Decision tree model is good or bad?

texts:
I built a decision tree model and am not sure if it is good or bad. Could you help to evaluate my model?

My code:

from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()
X_new = encoder.fit_transform(X)
#Decision tree model
model = DecisionTreeRegressor(random_state=1)

# Fit model
model.fit(X_new, y)
print(model.predict(X_new))

Result that I got:

[ 2.86       12.83333333  4.5        ...  2.9         2.925
  2.5       ]"


texts: How to setup LSTM problem when using multiple time series inputs?

texts:
Problem:
Suppose I have 50 time series which I want to train on. For each series, given the last say 4 samples, I would like to predict the next value.

Suppose each series is 100 samples. I break each series to 25 sub-samples. So in total, I have 100 * 25 = 2500 sub-samples.

The LSTM input is 3 dimensional input. One for sequence length (4 in this case), one for batch, and one for number of features (1 in this case).

How do I feed the input? This requirement suggests that I have to stack all my sub-samples together, and form the input set as the sub-samples.

If I do that, it would imply that I can feed the input randomly to the network.

If I do that, it would imply that the network is only affected by 4 inputs and any history before that is meaningless (because I may feed the input of 4 samples randomly)

Is this the correct method to setup the problem? Or do I have to batch each sample (4 sub-samples) together?


texts: Taking advatage of the frequency of an outlier

texts:
I'm doing outlier detection (Conditional Outliers) on a multivariate time series. The outliers appear every 2 weeks $\pm$ 4 days.

How can I incorporate this prior in my models, to reduce the number of false positive?  Alternatively, how should I filter the results of my models with this prior knowledge?

All my models use sliding windows and calculate an anomaly score of a series of points.

My rough idea: If an outlier is detected, model a Gaussian distribution where $\mu$ is set 14 days later and $\sigma$ is 4 days wide. Then multiply future predictions with the pmf of the distribution. Assuming the model has some kind of anomaly score, we are reducing the score of points which are unlikely outliers.


texts: Pre-training model with existing seasonal data for new dataset with maximum limit

texts:
I apologise if this is a simple one, I'm not sure if this is not possible or if I'm just not using the right keywords to find the answer;

Say I've got a pizza store, I've been able to successfully run a machine learning model to predict sales in that store and the seasonal changes. Now suppose i want to assume the same seasonal pattern in a brand new store that has only been open for a while. The new store is increasing and i want to use the growth factor of the new store and the seasonal data from the old store.

Another factor is that there is a maximum capacity of production for each store, so i need to be able to set that maximum so that the growth factor doesn't just keep growing beyond what the store can produce.

I'd really appreciate it if someone could help point me in the right direction for a library that would achieve these things and what key words I'm possibly missing. My preferred language is Python.


texts: Dataset for musical Instruments recognition

texts:
I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help
thanks and happy new year


texts: Unable to apply my model on separate test file

texts:
This is the training model and df is the dataframe:
Then, I've applied some operations according to the requirement:

vocab_size = 10000
tokenize = Tokenizer(num_words=vocab_size)

tokenize.fit_on_texts(df.review_description)
print(len(tokenize.word_index) + 1)

X = tokenize.texts_to_matrix(df.review_description, mode='tfidf')

encoder = LabelBinarizer()
encoder.fit(df.variety)
Y = encoder.transform(df.variety)

del df

x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=0)
print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)

del X
del Y

Here is the model operation:

model = Sequential()

model.add(Dense(128, input_dim=x_train.shape[1]))
model.add(Activation('tanh'))
model.add(Dropout(0.2))

model.add(Dense(256))
model.add(Activation('sigmoid'))
model.add(Dropout(0.2))

model.add(Dense(y_train.shape[1]))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'],
              optimizer='adam')

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)

model_output = model.fit(x_train, y_train,
             batch_size=32,
             epochs=100,
             verbose=2,
             callbacks=[es],
             validation_split=0.2)
print('Training Accuracy : ' , np.mean(model_output.history["accuracy"]))
print('Validation Accuracy : ' , np.mean(model_output.history["val_accuracy"]))

Everything is working correctly, just the problem is that I'm doing this on all the train data.
How to apply this on the another "test.csv" file?


texts: Train a competitive layer on nonnormalized vectors using LVQ technique

texts:
How can we train a competitive layer on non-normalized vectors using LVQ technique ?

The net input expression for LVQ networks calculates the distance between the input
and each weight vector directly, instead of using the inner product. The result is that the
LVQ network does not require normalized input vectors. This technique can also be
used to allow a competitive layer to classify non-normalized vectors. Such a network is
shown in figure below.

Using this technique how can we train a two-neuron competitive layer on the (non-normalized)
vectors below, using a learning rate $\alpha=0.5$

$p_1=\begin{bmatrix}
1 \\
1
\end{bmatrix}, p_2=\begin{bmatrix}
-1 \\
2
\end{bmatrix}, p_3=\begin{bmatrix}
-2 \\
-2
\end{bmatrix}$

Present the vectors in the following order : $p_1, p_2, p_3, p_2, p_3, p_1$

Initial weights : $W_1=\begin{bmatrix}
0 \\
1
\end{bmatrix}, W_2=\begin{bmatrix}
1 \\
o
\end{bmatrix}$


texts: Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

texts: I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the structure of it, I still don't understand why $1/10000$ was used as the scaling factor for the $pos$ of a word. Why was this number chosen?


texts: How can I use data on customer interactions to drive communication strategy?

texts:
I have data on my customers (age, location, gender) and number of interactions with customer by channel (#calls, #sms, #letters) and data on whether they have bought a product (yes/no).

I would like to predict what combination of interaction (e.g. 3 calls, 2 letters) is the best approach when contacting different types of customers. What would be an appropriate statistical/ machine learning technique to use to calculate this?


texts: Is there any good practice to cluster 3D data array?

texts:
So I'm not sure what word fits best to describe this data, probably "dimension" would be wrong since it may be used for flat samples with 3 features;

but by 3D data I mean some structure in a form of [samples, timesteps, features]. And there are 2 features in each timestamp.

It looks like [ [ [1,2], [3,4] ], [ [5,6], [7,8] ] ], like an LSTM input.
[1,2] is a timestep and [[1,2],[3,4]] is a sample.

So one way is to just flatten out timesteps and make them into a 1D array. However is there any better way that would somehow utilize the information conducted by "grouping" of features inside a timestamp?

Also how do I properly describe this data structure?


texts: multiple intents for modifying an intent of a sentence?

texts:
Say I have a sentence like 'I refuse to fly' or 'I'd like to fly'. I also have a sentence like 'I don't want to sit'. When training custom intents in one of the available NLU engines (rasa/wit/luis), what's the best way to go for modeling:
Naively I could have: RefuseFlyIntent,WantFlyIntent,and RefuseSit and WantSit

More sophisticated, have set of intents FlyIntent, SitIntent, WantIntent, RefuseIntent, and have my code process the combinations.

same question can apply for other cases, like how to model the difference between You Like To Fly and I Like To Fly

I'm sure there are known methodologies for that, wanted to understand what they are. If you could give me links to literature about it, would be great.

many thanks,
Lior


texts: border_mode for convolutional layers in keras

texts:
Keras has two border_mode for convolution2D, same and valid.
Could anyone explain what "same" does or point out some documentation? I could not find any document on the net (except people asking that it be implemented in theano as well).


texts: Precision decreases with each epoch in CNN

texts:
I've built the following CNN that is used to classify a binary classification set (something a like cats or dogs):

    self.opt = SGD(lr=0.0001)

    self.model.add(Conv2D(filters=16, kernel_size=3, input_shape=(100, 100, 3), padding='same'))
    self.model.add(BatchNormalization())
    self.model.add(Activation(self.activation))
    self.model.add(MaxPooling2D(pool_size=2))

    self.model.add(Conv2D(filters=32, kernel_size=3, activation=self.activation, padding='same'))
    self.model.add(MaxPooling2D(pool_size=2))

    self.model.add(Conv2D(filters=64, kernel_size=3, activation=self.activation, padding='same'))
    self.model.add(MaxPooling2D(pool_size=2))

    self.model.add(Conv2D(filters=128, kernel_size=3, activation=self.activation, padding='same'))
    self.model.add(MaxPooling2D(pool_size=2))

    self.model.add(Dropout(0.5))
    self.model.add(Flatten())
    self.model.add(Dense(150))
    self.model.add(Activation(self.activation))
    self.model.add(Dropout(0.5))
    self.model.add(BatchNormalization())
    self.model.add(Dense(1, activation='sigmoid'))
    self.model.summary()
    self.model.compile(loss='binary_crossentropy',
                  optimizer=self.opt,
                  metrics=[self.precision])

Sadly, my training seems to be doing very badly as the precision is decreasing with each epoch:

What are some common issues when this behaviour is noticed?

Is the most likely cause that my training data is biased for one class? Right now I have 25% class 1 and 75% class 2.

How much does the size of the pictures influence the performance? As of now, all pictures are size 100x100. Does increasing the size lets say to 400x400 make the CNN more capable of detecting the features?

Additional code:

Loading the Images:

def convert_image_to_array(files,relpath):
    images_as_array=[]
    len_files = len(files)
    i = 0
    print("---ConvImg2Arr---")
    print("---STARTING---")
    for file in files:
        # Convert to Numpy Array
        images_as_array.append(img_to_array(load_img(relpath+file, target_size=(soll_img_shape, soll_img_shape)))/255)
        if i == int(len_files*0.2):
            print("20% done")
        if i == int(len_files*0.5):
            print("50% done")
        if i == int(len_files*0.8):
            print("80% done")

        i +=1
    print("---DONE---")
    return images_as_array

from sklearn.model_selection import train_test_split
from keras.preprocessing import image
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

cnn = nn.NeuralNetwork()
cnn.compile()

##### Load Images to train, test and validate

x_train = np.array(convert_image_to_array(X_train,"images/processed/"))
x_test = np.array(convert_image_to_array(X_test,"images/processed/"))

Also the prediction metric:

def precision(self, y_true, y_pred):
    '''Calculates the precision, a metric for multi-label classification of
    how many selected items are relevant.
    '''
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return (precision)


texts: How to make edges appear in a graph when adjacency matrix value is beyond a particular threshold?

texts:
I've created a graph from an adjacency matrix using the igraph package but I want edges to appear in the graph only when the value in the adjacency matrix is beyond a particular threshold. Using the present method I am getting all the edges which have non zero weights.

set.seed(42)
iris_matrix <- as.matrix(iris[,1:4])
image(iris_matrix)
g<-cor(iris_matrix)
library(igraph)
R<-graph.adjacency(g, mode=c("undirected"), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)
plot(R)


texts: Is feature engineering still useful when using XGBoost?

texts:
I was reading the material related to XGBoost. It seems that this method does not require any variable scaling since it is based on trees and this one can capture complex non-linearity pattern, interactions. And it can handle both numerical and categorical variables and it also seems that redundant variables does not affect this method too much.

Usually, in predictive modeling, you may do some selection among all the features you have and you may also create some new features from the set of features you have. So select a subset of features means you think there are some redundancy in your set of features; create some new features from the current feature set means you do some functional transformations on your current features. Then, both of these two points should be covered in XGBoost. Then, does it mean that to use XGBoost, you only need to choose those tunning parameters wisely? What is the value of doing feature engineering using XGBoost?


texts: Combining multiple probabilities from a classifier. Propagating probabilities

texts:
Let's say I have trained a classifier that classifies images of animals into 10 different classes. And let's say that I have 20 different images of a particular animal and because I know the photographer, I know with certainty that all 20 images are of the same animal. So I use my classifier to make a prediction on what animal it is and get 20 predictions one for each image. The model predicts all the images to be a dog with varying probabilities.
image 1: 80% dog
image 2: 90% dog
image 3: 75% dog
and so on.
What is the probability that the animal in question is a dog?
Let's say they predict cat with smaller probabilities, 5%, 2%, 4% ... What is the probability it is a cat?
I've tried a few different approaches, applying Bayes Theorem but I keep getting numbers that add up to be more than one.   Could it really be just the average?


texts: How can I fixed the filter and Kernel Size of a CNN?

texts:
I have created 4 x 4 2d images from a signal. Now, I want to feed this data to a Convolutional neural Network. How I can choose the nubmber of filter, Kernel Size for this small, shape and size of dataset (4 x 4, 320 images). Should I use maxpooling layer ?
Note:
I have already implemented a model for this dataset and accuracy is 78.125%.  The model is,


texts: Time series forecasting. How use future values

texts:
I have a time series dataset containing hourly data from a few year, like below. Let's assume that I want to make prediction for the next 3 hours (2021-01-01 19:00, 2021-01-01 20:00, 2021-01-01 21:00). The output from model (e.g. LSTM) is the following sequence: [Feature_1(t+1), Feature_1(t+2), Feature_1(t+3)]. One of the features of my training data is weather data.

Date
Feature_1
Weather_feature

2019-01-01 15:00
value(t0)
value(t0)

2019-01-01 16:00
value(t0)
value(t0)

....
...
...

2021-01-01 18:00
value(t0)
value(t0)

Can I use data, that I already know in my model? I would like to use the weather forecast for the next hours to improve my predctions.
For example at 2021-01-01 18:00 i know value Feature_1 and  Weather_feature in time t(0) but also i know already Weather_feature in (t+1), (t+2), (t+3). How can i use it in the model?
At the moment, this is what one training sample looks like: [[Feature_1(t-n), Weather_feature(t-n)]...[Feature_1(t-1),Weather_feature(t-1), [Feature_1(t0), Weather_feature(t0)]]


texts: Prediction using words which were not in training in a CNN with pre-trained word embeddings

texts:
In sentence classification using pre-trained embeddings(fasttext) in a CNN,  how does the CNN predict the category of a sentence when the words were not in the training set?
I think the trained model contains weights, these weights are not updated in the prediction stage, are they?. Then, what happens when the words in the sentence (for which the cnn will predict a category) were not seen in the training? I think they do not have a word vector, only the words that were found in the training.


texts: What is the objective that is optimized with Random Search?

texts: I have recently learned about Random Search (or sklearn.model_selection.RandomizedSearchCV in Python) and was thinking about the theory behind the optimization process. In particular my question is, given that one performs Random Search on a certain algorithm (let's say random forest), what are the best hyperparameter based on? More specifically in what sense are they the "best" hyperparameters for the model? Do they maximize accuracy of the model? If not what is the (performance-)criterion that is optimized? Or is it entropy/gini?


texts:
I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results.

If I insepct the results of my Topic Model, the overall Topic assigment for each document is pretty good, but some documents got an assigned topic which does not fit very well.

So overall I got an accuraccy of approx. $85\%$.

Is it possible to reach $100\%$ accuracy in Data Mining?


texts: Delete all rows between two values in a dataframe that repeat multiple times in a column

texts:
I have dataframe and let's say inside of it is a column_A. This column_A has 3 strings as values, call them 'new_records', 'deletions', 'changes'  that repeat across the dataframe multiple times in that order always with multiple rows in between. I want to delete all rows from the beginning of deletions to the end of changes, i.e. I want to leave only new_records in the dataframe. The dataframe looks like this:

column_A         column_B     column_C ....
NEW_RECORDS        val1         val2
string1_new        val3         val4
string2_new        val5         val6
  NaN              val9         val10
  NaN              val11        val12
string3_new
 ...
DELETIONS          val7         val8
string1_del         ...           ...
   NaN              ...           ...
string2_del         ...           ...
  ...
CHANGES             ...           ...
 str1_ch            ...
 str2_ch
  ...
NEW_RECORDS
 str200_new        ...
 str300_new           ...
  NaN
  NaN
  ...
DELETIONS
 NaN
 str100_del
 NaN
 str290_del        ...
  ...
CHANGES
 str1000
 str20000
  NaN
   ...           ...

I want to have at the end only chunks of rows between new_records and deletions values, without rows that belong to the deletions group and changes group. How can I do that?

UPDATE:

There are many rows after the 'new_records' and before the start of 'deletions' group and there are many rows after the start of deletions group and beginning of the 'changes' group. I need to extract only rows that belong to the new_records group. So all rows after the value 'new_records' and before the value of 'deletions' across all dataframe.


texts: Dealing with training set of questionable quality

texts:
Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into the training set is often questionable (I don't even know how to start separating good from bad ones). I have spent much more time trying to pre-process the files in order to increase the validity of the data, than actually building the prototype.

I have only dumped a part of the data that is claimed to be better. There are still data from other sources to complement the missing part in the first set of training data. Given the amount of data, it is practically impossible for me to spend too much time on them.

So the question is, how do people deal with really horribly, inconsistent data in real life (assuming I am working with addresses, where people often write the wrong postcode due to bad city planning or lack of clear instruction, misspell the name of neighbourhood, plus data got "autocorrected" by excel such that "1-2" becomes "1-Feb", etc. etc.)? Specifically, how do I ensure the quality of the train classifier given the training set (and cross validation set) has questionable quality?


texts: How to sum values in a column grouped by another column

texts:
I have this CSV file:
ID,NAME,CITY,COUNTRY,CPERSON,EMPLCNT,CONTRCNT,CONTRCOST
00000001,Breadpot,Sydney,Australia,Sam.Keng@info.com,250,48,1024.00
00000002,Hoviz,Manchester,UK,harry.ham@hoviz.com,150,7,900.00
00000003,Hoviz,London,UK,hamlet.host@hoviz.com,1500,12800,10510.50
00000004,Grenns,London,UK,grenns@grenns.com,200,12800,128.30
00000005,Magnolia,Chicago,USA,man@info.com,1024,25600,512000.00
00000006,Dozen,San Francisco,USA,dozen@dozen.com,1000,5,1000.20
00000007,Sun,San Francisco,USA,sunny@sun.com,2000,2,10000.01

What I want to do is to find the COUNTRY with the largest number of CONTRCNT. Some countries appear more than once in the dataframe so I need to find the country with the largest sum of CONTRCNT.
I thought about summing up the CONTRCNT for all countries and then finding the largest one but I want to do this in a way that is not bruteforce. I actually want to know how I can use Pandas' groupby function to solve this problem.


texts: Adding a column to a dataframe in pandas using another Column

texts:
So I have a column called "plot" in a dataframe and i want to create a new one called "keywords" which only has the important words of plot.
here is the code:

 import pandas as pd
 import numpy as np
 from sklearn.metrics.pairwise import cosine_similarity
 from sklearn.feature_extraction.text import CountVectorizer
 import re
 import nltk
 nltk.download('stopwords')
 from nltk.corpus import stopwords
 df = pd.read_csv('IMDB_Top250Engmovies2_OMDB_Detailed.csv')
 df = df[['Title','Genre','Director','Actors','Plot']]
 df['Keywords'] = ''

 for index,row in df.iterrows():
     plot = row['Plot']
     plot = re.sub('[^a-zA-Z]'," ", plot)
     plot = plot.lower()
     plot = plot.split()
     plot = [i for i in plot if not i in set(stopwords.words('english'))]
     plot = ' '.join(plot)
     row['Key_words'] = str(plot)

And here is the output :(

Link to the csv : https://query.data.world/s/uikepcpffyo2nhig52xxeevdialfl7

Thank you !


texts: Which database to use for storing machine learning data?

texts:
I am currently storing my training data into HDF5 files and I want my team and I to switch for a database for two main reasons:

the data is not used only by me and the different datasets are stored in different folders at different paths etc so I want to create a single database containing all the data
we need to store a lot of metadata like hashes, insertion dates, ground truth (that may differ depending on the application) etc.

I don't know which database to use. My first thought would be something like mongodb which is not relational, but I wonder if there are databases specialized in deep learning training out there. I heard of SciDB but I am not sure what to think about it, and I searched for "databases for machine learning" in google scholar without success. That is why I am asking the question here.
Thank you in advance for your help.


texts: doc2vec - paragraph or article as document

texts:
I'm trying to train a doc2vec model on the German wiki corpus. While looking for the best practice I've found different possibilities on how to create the training data.
Should I split every Wikipedia article by each natural paragraph into several documents or use one article as a document to train my model?
EDIT: Is there an estimate on how many words per document for doc2vec?


texts: Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes

texts:
I am a newbie when it comes to machine learning. I am trying to get hands on experience by analyzing different supervised learning algorithms using scikit-learn library of python. I am using the sentiment140 dataset of 1.6 million tweets for sentiment analysis using various of these algorithms.

I don't know if it is a stupid question, but I was wondering whether if it'd be possible to classify into three classes (positive, negative and neutral) when you've only trained over two classes (positive and negative). The sentiment140 training set consists of only two classes (positive and negative) of 1.6M tweets in total but their test set consists of 500 tweets over three classes (positive, negative and neutral), so it got me thinking.

So is this possible? If yes, how do I proceed to identify the neutral tweets?
Intuitively, I can use to find the conditional probability of each classified tweet in the test set over each class (using predict_proba) and tell if it's neutral if it is below a certain threshold (say less than 0.7) for both the positive and negative classes. Is this the right way to go?


texts: Optimize F-Score only for certain classes, disregard other classes

texts:
I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text.
I use a simple naive Bayes classifier (sklearn) to try to predict a rating given a review which works reasonably well (~65% accuracy - I hope that's decent given that there are 5 classes and I'm only using a naive Bayes?)

classifier = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
classifier.fit(x_train, y_train)
predicted = classifier.predict(x_test)
accuracy = accuracy_score(y_test, predicted) # ~65%

Anyway, what I would like to do is build a classifier that is really good at predicting if a review has the rating 4 or 5 for example. I don't really care how (bad) the classifier is doing for ratings other than 4 or 5.

My ideas were

modify the dataset and change all labels to 1 where the label is not 4 or 5, then fit the classifier again
only use the training examples that have label 4 or 5

However, both approaches do not help at all, the F-Score for classes 4 and 5 do not improve. What can I do to make a classifier better for these 2 classes only?


texts:
I am trying to figure out if it makes sense that the width of the network could be smaller than the input/output size?

So for example, I am giving the Neural Network 2048 numbers, and I am expecting 2048 numbers back. I would also like to use LSTM's, which take a lot of time/space etc to train, and having one or a few 2048 or larger LSTM layers connected to Dense layers would take a lot of space/time to train. But maybe that is the way to go?

Does anyone have any experience with this type of problem? Thank you!


texts: How does the Transformer predict n steps into the future?

texts:
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch implementation requires you to pass the input (src) and the target (tgt) tensors for every step, rather than encoding the input once and keep on iterating for n steps to generate the full output. Am I missing something here?
My first guesses were that the Transformer isn't technically a seq2seq model, that I have not understood how I'm supposed to implement it, or that I've just been implementing seq2seq models incorrectly for the last few years :)


texts:
I came across a Machine Learning software which I won't name, that claims that completely automates the Feature Engineering process from any source without domain knowledge, implying pretty much that turns Data Scientists unnecessary for this step. Now I find that claim a bit suspicious since it goes against the common sense of what I learned Feature Engineering consists of.

But considering that it might be a lack of knowledge from my part to what point we can automate the featuring engineering process? I can think of some general approaches like statistical methods (median,etc..) or several types of coding (binary, polynomial,etc..) for categorical variables, but could anyone give me an overview state of the art approach regarding this subject or point me in the right direction?


texts: Variable Importance changes with oversampling

texts:
I am currently using Xgboost for a binary classification problem with highly imbalanced data in R.

I have used oversampling to train the model. This worked well, now however it comes to measuring variable importance - how does the expected loss change when a certain variable is randomized (this is one of the methods the DALEX package provides).

This has lead to the following: When I compare the variable importance between two models- one trained on a oversampled set and one trained on the original set - I get vastly different results.

It is not surprising that the actual values change, but why do some variables gain and others lose importance as measured above?


texts:
I am reading a paper and encountered this notion:

$$1_{\{Y=1\}}$$

To me it seems to be the expression as below, but I am not entirely sure and I don't think the author explictly explained it:

if Y==1:
  return 1
else:
  return 0

Can someone help me to clarify this notion? Much thanks for your time (:

It appears in:

https://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf


texts: Hidden Markov Models: Linking states to labels after EM training

texts:
The tl;dr version first:

I have the following problem: I implemented Baum Welch for ergodic HMMs. I do it like this:

I pass the model two number C1 and C2, it builds a fully connected state machine with C1 states and C2 emissions. I map all tokens from my training data onto the range [0, C2) and each label the HMM is supposed to assign a token during inference onto [0, C1). Then the HMM goes ahead and does Baum Welch Learning. When it is done it has configured its state machine to maximize the likelihood of the training data (locally).

Now to my problem:

Assume I had two isomorphic initial state machines (isomorphic under consideration of all the probabilities, as structurally all the HMMs are isomorphic anyway, because ergodic). They vary only in their state IDs, so the IDs have been scrambled around from one machine to the other.
Now, after training both HMMs will be isomorphic again when trained with the same data. That means there is absolutely no connection between the IDs I map the labels from my label set to and the IDs of the states of the HMM. So how then can I interpret the HMM after training? How do I know which state corresponds to which POS-tag? Seems impossible, so I guess I am missing some crucial point.

Now a little more detail if the above was unclear:

I take my training data (texts, like newspaper etc.) and count the number of different words (types).

Then I pass count(types and count(labels) the labels being a set of POS-tags. It then randomly constructs a probabilistic fully connected state machine with pow(count(labels), order_of_model) different states. order_of_model being the number of hidden variables (POS-tag ngrams) that get combined into an individual state. It also assigns each of these states an initial and an emission probability for all of the types.

The model assumes a mapping from [0, pow(count(labels), order_of_model)) as state IDs onto external tuples of labels. And a mapping [0,count(types)-1) for the emissions onto words.


texts: The most online tutorials like to use a simple example to introduce to machine learning by classify unknown text in spam or not spam. They say that this is a binary-class problem. But why is this a binary-class problem? I think it is a one-class problem! I do only need positive samples of my inbox to learn what is not spam. If I do take a bunch of not spam textes as positiv samples and a bunch of spam-mails as negativ samples, then of course it's possible to train a binary-classifier and make predictions from unlabeled data, but where is the difference to the onc-class-approach? There I would just define a training-set of all non spam examples and train some one-class classifier. What do you think?


texts: In reinforcement learning, continue update weights after training?

texts: Assuming an rl agent that has to be trained, and then you can mainain same weights over episodes: in a single episode, you have to firtly backup weights, udpate continuosly weights in the episodes, and in the end restore weights? Or Never update weights? I know only deep q learning, I think is this question can be good for other algorithm


texts: Interpret results from a lightFM factorization machines

texts:
I built a recommendation model on a user-item transactional dataset where each transaction is represented by 1:

model = LightFM(learning_rate=0.05, loss='warp')

Here are the results of this recommendation model:

Train precision at k=3:  0.115301
Test precision at k=3:  0.0209936

Train auc score:  0.978294
Test auc score : 0.810757

Train recall at k=3:  0.238312330233
Test recall at k=3:  0.0621618086561

Can anyone help me interpret this result? How is it that I am getting such good AUC score and such bad precision/recall? The precision/recall gets even worse for 'bpr' Bayesian personalized ranking.

Prediction task:

users = [0]
items = np.array([13433, 13434, 13435, 13436, 13437, 13438, 13439, 13440])
model.predict(users, item)

Result from the prediction task:

array([-1.45337546, -1.39952552, -1.44265926, -0.83335167, -0.52803332,
   -1.06252205, -1.45194077, -0.68543684])

How do I interpret the prediction scores?

Thanks


texts: Active learning on image classification model

texts: Say we have a trained image classification model. Theoretically, is it possible to update the model with only a sample without retraining? If not, is there any kinds of active image classification DL model that can allow incremental update?


texts:
I have a multivariable regression problem which basically forms a plane in 3D space. There may be other independent variables at play, but if I can approach the problem with 2 independent variables then I should be able to scale to include the rest.

Broadly, it's a resource problem. I have a required resource (z), which depends on time (x) and number of registered users (y). Registered users is a function of time, but I need the time component because the resource requirement is different at different times of the day. I want to be able to predict the resource requirement at a given time of day if our users continue to registered at a known rate.

Here is a chart of the data:

This is obviously not a linear regression problem, so how do I approach this?


texts: Why is choosing the k in the k-means clustering method based on a feature (take a dead or alive patients scenario as an example, k will be 2) considered clustering rather than classification?


texts: Does XGBoost handle multicollinearity by itself?

texts:
I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, for example: a variable (feature) $A$ also appears as $\frac{B}{A}$ and $\frac{C}{A}$.

My questions are :

How (If?) do Boosted Decision Trees handle multicollinearity?
How would the existence of multicollinearity affect prediction if it is not handled?

From what I understand, the model is learning more than one tree and the final prediction is based on something like a "weighted sum" of the individual predictions. So if this is correct, then Boosted Decision Trees should be able to handle co-dependence between variables.

Also, on a related note - how does the variable importance object in XGBoost work?


texts: How to get value of a matrix from every row with the index generated by argmax in Tensorflow?

texts: I considered gather_nd, but it seemed that gather_nd need a index like [[0,..],[1,..],[2,..]] to get value from every row, and argmax just generated the 1-dimension indexes tensor, it seems that I have to run the argmax, get the narray, use for-loop to add the range index and then feed that into the graph again. I want to know is there any method that can directly get the result?


texts: Is there a real C4.5 implementation in Python ? (handling missing value)

texts:
To my understanding, C4.5 comes with 4 improvements compared to ID3:

Handling missing values in both training data and "test" data,
Handling continuous data
Handling costs on attributes.
The pruning

Source

However, not one of all decision tree python modules that I found, even the so-called C4.5, handles missing values.

Do you know of a library that handles them ? And it would be great if it can work with Pandas DataFrames "out-of-the-box"...


texts: Keras model with second to last sigmoid activated Conv1D layer followed by globalMaxPool outputs values outside [0,1]. Why?

texts:
I am trying to train a binary classifier. It is a residual network with skip layers etc. but ultimately, the bottom two layers are a 1D convolution with sigmoid activation followed by a global max pool afterwards. I believed this should output predictions between 0 and 1.
In fact, I've trained a very similar model previously and it was working as expected but then I decided to change the way I preprocess input data and changed a few minor aspects like kernel size etc. and now when I call model.predict(inputs) I am getting a very broad range of values - at least between [-20,20]. I really don't see why, can anyone look at my model and potentially help me understand where I've gone wrong?
Here is a json string that should allow you to load my model with
model = model_from_json(json_string). Note that the final two layers are called 'conv_final' (conv1D with sigmoid activation) and 'output' (global max pooling)
https://www.dropbox.com/s/3qywmcho5tldurr/final_model.json?dl=0


texts: How does multiple feature files work in Keras

texts:
I was going through a Kaggle kernel, where train data was of different data types, so it consisted of separating out features for different types like:

input_cat = Input((len(a), ))
input_num = Input((len(num_feat), ))
input_words = Input((maxlen, ))

#More layers are added to the inputs
#x_cat, x_num, x_words are updated vars for the 3 inputs

x = concatenate([x_cat, x_num, x_words])
#some final layers added
predictions = Dense(1, activation="sigmoid")(x)

model = Model(inputs=[input_cat, input_num, input_words], outputs=predictions)

Later the model is used to fit training data like:

model.fit([X_train_cat, X_train_num, X_train_words], X_train_targe, #other params)

I get how the separated feature files are trained separately, and then their weights are concatenated, which are then used to pull out predictions.

I just wanted to get a bit intuition behind this or is it as simple as processing different types of feature differently and then combining them for a better model.

Thanks!


texts: Scikit Learn Latent Dirichlet Allocation overload my SWAP and/or RAM

texts:
1) After using LDA of scikit learn I realized it overloads the swap, but never reach more than 20% of the RAM. Do you have any idea why ? I can't really see if this is a problem, but still I would like to understand what happens in memory !

2) The parallelization of this algorithm (using parameter n_jobs=-1) leads to memory errors on my server (192 GB of RAM, 32 CPU cores). I think this is due to the fact that parallelization uses forks to create threads, leading to copying 32 time the current memory (which can be quite big if data are loaded in memory). How could I improve this ?


texts: Spatially constrained geospatial similarity

texts:
What's the current methodology for clustering geospatial data by features?

Example: I have some demographic dataset. Let's say this contains average home price and population density.

So, an example correlation here would be home price vs population density. But, the trick is how the clustering gets pulled. For example, an affluent area with high population density isn't the same as one with low population density. Applying a basic distance metric wouldn't take this into account since low vs highs could offset each other giving similar distances. This leads me to possibly some form of weighted clustering to pull centroids.

Not sure what methodology takes this into account.


texts:
I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.

It looks to me that the crux of the paper is using masked inputs to achieve bidirectionally.

This is an excerpt from the Google AI blog here which states:

  "However, it is not possible to train bidirectional models by simply
  conditioning each word on its previous and next words, since this
  would allow the word that’s being predicted to indirectly “see itself”
  in a multi-layer model.  To solve this problem, we use the
  straightforward technique of masking out some of the words in the
  input and then condition each word bidirectionally to predict the
  masked words."

Can someone please help me understand how does bidirectionally allow the words to see themselves and how masking solves this problem?

Thanks.


texts: Layer notation for convolutional neural networks

texts:
When reading about convolutional neural networks (CNNs), I often come across a special notation used in the community and in scientific papers, describing the architecture of the network in terms of layers. However, I was not able to find a paper or resource describing this notation in detail.
Could someone explain to me the details or point to where it is described or "standardized"?

Examples:

input−100C3−MP2−200C2−MP2−300C2−MP2−400C2−MP2−500C2−output
(source)
input−(300nC2−300nC2−MP2)_5−C2−C1−output
(source)

A good guess seems that xCy are convolution layers (x is number of filters? y is one side of square kernel?). MPz is max-pooling layer (pool size z×z?).

But instead of guessing, I would love to have a reference (which I could possibly also reference in a paper).


texts: How could we use a neuron instead of 100 neurons in 0.01 standard Weight initialization

texts:
Many books explain that we don't need to use 100 neurons and just can use a neuron if all result values(sigmoid(Wx)) are same at each of hidden layers.

but, I don't know that we can just use a neuron.
Thanks !


texts:
Assume that we have a set of elements E and a similarity (not distance) function sim(ei, ej) between two elements ei,ej ∈ E.

How could we (efficiently) cluster the elements of E, using sim?

k-means, for example, requires a given k, Canopy Clustering requires two threshold values. What if we don't want such predefined parameters?

Note, that sim is not neccessarily a metric (i.e. the triangle inequality may, or may not hold). Moreover, it doesn't matter if the clusters are disjoint (partitions of E).


texts: What does Theano dimension ordering mean?

texts: In this code , line 13 is commented as Theano dimension ordering mode. What does this mean?


texts: Understanding LSTM Training and Validation Graph and their metrics (LSTM Keras)

texts:
I have trained a RNN/LSTM model. I would like to interpret my model results, after plotting the graph for Loss and accuracy (b/w training and Validation data set).

My objective is to classify the labels (either 0 or 1) if i provide only a partial input to the model. In such a way I have performed training.

Train_Validate_Test_Split

Train 80% ; Validate 10 % ; Test 10%

X_train_shape : (243, 100, 5)
Y_train_shape : (243,)

X_validate_shape : (31, 100, 5)
Y_validate_shape : (31,)

X_test_shape : (28, 100, 5)
Y_test_shape : (28,)

Model Summary

Model Graph

Model Metrics

Question or Interpretation from the model results

  Q 1 : What can I understand/interpret from loss and Accuracy graph ?
  How can i confirm whether the model trained properly for my data set
  or not ?

  Q 2 : Whether oscillations in both loss and accuracy, have some effect in >model training ? (Or it is a normal behavior) If not, how
  can I regularize my model without oscillations ?

  Q 3 : What can I interpret or understand from my metrics tabular column ? My > Y_test accuracy is more when compared with Train &
  Validation accuracy, What can i interpret from this behavior ?


texts:
In summary, static graphs are easy to optimize but lack the expressivity found in higher-level languages; dynamic graphs provide this missing expressivity but introduce new compilation and execution challenges, especially on heterogeneous hardware and FPGAs.

from

  Roesch, Jared, Steven Lyubomirsky, Logan Weber, Josh Pollock, Marisa Kirisame, Tianqi Chen, and Zachary Tatlock. "Relay: a new IR for machine learning frameworks." In Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 58-68. ACM, 2018.

My question is

Why are static graphs easier to optimize than dynamic graphs?
Why are static graphs less expressive than dynamic graphs?
What compilation and execution challenge dynamic graphs face? How about static graphs? What causes these problems for dynamic graphs?

If such a question is not suitable to be asked here, please redirect me to another stack. Thanks!


texts:
I have my ANN trained on MNIST dataset. Hidden layer has 128 neurons and input layer has 784 neurons. This gave me an accuracy of 94%. However when I added one more layer with 64 neurons in each then the accuracy significantly reduced to 35%. What could be the reason behind this.

Edit : Activation function : sigmoid. 521 epochs.


texts: Weights not converging while cost function has converged in neural networks

texts:
My cost/loss function drops drastically and approaches 0, which looks a sign of convergence. But the weights are still changing in a visible way, a lot faster than the cost function. Should I ensure the weights converge too?

Some details: I just calculated for 1 epoch. My loss function is mean square difference. I use no optimizer. I tried several experiments and different initial weights but all got converged loss function with ever-changing weights.


texts: over-fitting with good enough test accuracy

texts:
Let's make things simple. Imagine an underdetermined linear system with $N$ samples and $p$ features $(N<p)$. Let's say I found one of the possible (among many) solutions of such systems and computed the test accuracy. My train's error is zero in this setting, but not the test's accuracy. Hence, overfitting. Despite that, assume that the test error rate is considered to be good from the perspective of the field experts. Let's also assume that some checks guaranteed that this rather good error rate is not due to test/train separation either.

Is it true to argue that no effective learning has taken place here?
If so, can one conclude from this that a better model with better accuracy in test exists that does not overfit?


texts: Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime as this always leads to high values due to a number of O (other) tags in the dataset.


texts:
I installed Hadoop, Mahout and Spark. I am able to see the Hadoop and Spark MasterWebUI. Moreover, I can also run the following command,

[hadoop@muildevcel01 mahout]$ bin/mahout

However, we I try running the spark-shell I run in the problem stated below,

[hadoop@muildevcel01 mahout]$ bin/mahout spark-shell

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/repl/SparkILoop
        at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.repl.SparkILoop
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more

Question

Any suggestions how I could resolve my problem?


texts: acf function shows error while fiiting time series

texts:
I am fitting a time series model in R. I plotted the time series to check for the pattern of it. It has 30625 observations. The observations are half hourly

plot.ts(ts)

I was checking acf plot to see for auto correlation.

acf(ts)

If ACF plot decreases slowly i.e. it has auto correlation out to higher number of lags, then it probably needs higher order of differencing.

I took the logarithm of the series and then checked its acf plot.

acf(log(ts))

I am getting an error

Error in plot.window(...) : need finite 'ylim' values
In addition: Warning messages:
1: In min(x) : no non-missing arguments to min; returning Inf
2: In max(x) : no non-missing arguments to max; returning -Inf

What does this error means? What should I do remove this error.
I also think the significance bounds given by dotted line are not correctly calculated. What should I do in these two cases?


texts: What model to use for matching two datasets

texts:
I've got 500 images of paper receipts scanned and OCR as one dataset. I also have a dataset of transactions from my credit card statement including amount and date.

What model is best to match the receipts to the transactions?


texts: Can I plug models like Linear Regression into a CNN feature map result?

texts:
I was learning about image recognition on the Orange Software and I saw that I can feed my image database into a CNN(they call image embedding) that has as output a feature map of the image and then I can feed that feature map into models like Logistic/Linear Regression. And that is my objective, compare those models, but I want to do it without Orange.

I am thinking about using VGG16 as my CNN and extract the feature map from it. After that I want to plug those Linear/Logistic Regression to predict my image. Does that work at all? Is it possible?

(Using python and a proper labeled image database. Also I don't want to use the CNN alone, I really would like to use those 'simpler' models consuming the feature map)


texts: Autocorrelation of two functions multiplied and raised to arbitrary powers

texts:
Given a signal $A$ and a signal $B$ with autocorrelation times of $\tau_A$ and $\tau_B$, respectively, where $\tau_A > \tau_B$, is there any general statement that can be made about the autocorrelation time of the product of the two signals $A$ and $B$ together raised to arbitrary exponents, $\tau_{A^\alpha B^\beta}$, with respect to $\tau_A$ and $\tau_B$? For example, the following statement appears intuitively correct when $\alpha = 1$ and $\beta = 1$: $\tau_A > \tau_{A^1B^1} > \tau_B$
But does the situation change if $0 < \alpha < 1$ and $0 < \beta < 1$ or $\alpha > 1$ and $\beta < -1$? Any mathematical proofs or insights on how the autocorrelation time of the product of the two signals, $A^\alpha B^\beta$, varies with respect to $\alpha$ and $\beta$ would be appreciated.


texts: Best model structure for input coordinates producing full images

texts:
I am trying to produce a machine learning model that works as an interpolator of real data. Essentially what would go into the model is an xy coordinate the result would be an image. This is for characterising a camera at the per pixel level.

Ive tried feeding in raw xy coordinates and producing an x*y number of images, using a conv net (unet) but that didn't converge well.

I then tried feeding in full black images with a single xy pixel that was white, but because the output images were mostly black it cheated by making the whole output black. I then tried cropping the output image to be only of the area of interest but again it struggled to produce any real change and by eye just made broadly white images.

Is there a standard way of addressing coordinates in image out? Do I just need to train it for a very long time? Would transfer learning help? Is there a better loss function for images than mse?


texts: Conv bias or not with Instance Normalization?

texts:
It is well known that Conv layers that are followed by BatchNorm ones should not have bias due to BatchNorm having a bias term.

Using InstanceNorm however, the statistics are instance-specific rather than batch-specific yet there are still are two learnable parameters $\gamma$ and $\beta$, where $\beta$ is a learnable bias.

Naturally, Conv layers followed by InstanceNorm layers should also not use bias.

Though some well-renowned implementations on github insist on using Conv bias when followed by InstanceNorm.

What's your take on this?


texts: Interpreting vertical and horizontal parts of ROC curve

texts:
It's not clear to me how I can interpret vertical and horizontal parts of the ROC curve. What important information can I gain from this? This is a text from the book "Human-in-the-Loop Machine Learning" by Robert Monarch:

In this example, we can see that the line of the ROC curve is
near-vertical for the first 20%. This tells us that for the 20% most
confident predictions, we are almost 100% accurate. The ROC curve is
almost horizontal at 1.0 for the final 30%. This tells us that by the
time we get to the 30% least most confident predictions for a label,
there are few very items with that label remaining.


texts: Imbalanced dataset, finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

texts:
I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is. I understand MCC is the same as the Pearson's product moment correlation coefficient (PPMCC) and problem dependent, but my question is: Can we evaluate the statistical significance of it using Pearon's correlation coefficient's relationship to Chi square?
PPMCC= |MCC| = Sqrt[(Chi^2)/n]
In other words, can we conclude from the relation above that By having: imbalanced Binary Classification Problem, N = 1000, and χ² >= 3.85 (p < 0.05, df = 1 received from critical value table)
The following MCC would significant (p<.05) (i.e. we reject the null hypothesis that the correlation is 0):
MCC >= sqrt ( 3.85 / 1000 ) which is MCC >= 0.06
So in other words, we can reject the null hypothesis that the correlation is 0 if MCC is over .06 and accept the alternative hypothesis that we do indeed have a correlation (that is, our classifier is performing better than chance)?
This question is basically the same as the one asked here many years prior that had no sufficient answer: https://stats.stackexchange.com/questions/94563/measuring-statistical-significance-of-binary-classification-using-matthews-corre which had a lot of upvotes but no good answers (the only answer was flawed in my eyes since it starting mentioning ROC curves and didn't seem to answer the question). I think this is an important question without an answer. (the only answer was flawed in my eyes since it starting mentioning ROC curves and didn't seem to answer the question). I think this is an important question without a definitive answer.


texts: Is there a way to put a separate line between clusters for k-Means Clustering?

texts:
k-Means Clustering is a method of vector quantization, originally from signal processing, that is popular for cluster analysis in data mining.

here is a piece of code to perform a 2-d k-Means Clustering

from sklearn.datasets.samples_generator import make_blobs
X, y_true = make_blobs(n_samples=300, centers=3,
                       cluster_std=1.1, random_state=0)
plt.scatter(X[:, 0], X[:, 1], s=50);
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3)
kmeans.fit(X)
y_kmeans = kmeans.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='viridis')

centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200);

outputs a figure (fig_1)

this figure (fig_2) comes from wiki

is there a way to put a separate line shown in fig_2 on fig_1?


texts: Why this sequential model is not starting?

texts:
I am using following code:

input_shape = (75, 75, 3)
x = Input(input_shape)
model = BatchNormalization(axis = 3)(x)

Above code works all right. However, following code does not work:

from keras.models import Sequential
input_shape = (64,64,3)
model = Sequential()
model = model.add(InputLayer(input_shape=input_shape))
model = model.add(BatchNormalization(axis = 3))

But at last line, I get error:

AttributeError: 'NoneType' object has no attribute 'add'

If I change to:

model = model.add(Input(input_shape))

I get following error:

TypeError: The added layer must be an instance of class Layer.
Found: Tensor("input_1:0", shape=(?, 64, 64, 3), dtype=float32)

Where is the problem and how can it be solved?

(PS: If you find this question to be interesting/important, please upvote it.)


texts: Generating a text training dataset from a grammar

texts:
I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar.

More specifically, I would like to build a collection of texts with random 'sentences' and each one is a possible derivation (chosen randomly) from the grammar. I would like also to draw randomly the terminal symbols (the final words). I have looked into (python) nltk and have achieved some document generation, but the derivations chosen are not selected randomly.


texts: Is Elastic Search recommended if attribute getting search is not a huge text document?

texts:
We are currently developing a system with MEAN stack with Mongodb at backend. We have employees name, and Ids in our system and our client wants to get pretty good (Read: Google Like) search in our system to search for employees' records. He needs our system to recommend employees even if he has misspelled the name, etc.

One of the suggestions from our development lead was that we should use elastic search but from what I have seen, elastic search is preferred especially in scenarios where we are searching in a large text documents. I am having a feeling that using elastic search in our case will only increase the overhead as the attributes getting search in our case just names, emails or ids, etc.

Currently, our system only uses some basic regrex but I believe, we can improve our search by putting in some effort and without moving towards elastic search (though the development lead feels that we will only be Reinventing the wheel and we should use technologies that do this stuff for us.)

So, I wanted some guidance (Suggestions) on whether to move towards elastic search or should we stick with our current resources to work on a more optimized search than what we have right now?


texts:
I've trained a recommendation system to recommend steam games based on game tags. An example output is shown below, where GAME is the game recommended based on the similarity score.

Game to recommend for: Total War: WARHAMMER

GAME: Total War: WARHAMMER                     Similarity: 1.0
GAME: Phantom Doctrine                         Similarity: 0.97
GAME: Total War: THREE KINGDOMS                Similarity: 0.96
GAME: Warhammer 40,000: Dawn of War II         Similarity: 0.96
GAME: Total War: WARHAMMER II                  Similarity: 0.95
GAME: Warhammer 40,000: Dawn of War II Chaos Rising Similarity: 0.94

Game to recommend for: Age of Empires II: Definitive Edition

GAME: Age of Empires II: Definitive Edition    Similarity: 1.0
GAME: Rise of Nations: Extended Edition        Similarity: 0.97
GAME: Age of Empires II (2013)                 Similarity: 0.97
GAME: Stronghold Crusader HD                   Similarity: 0.96
GAME: Age of Mythology: Extended Edition       Similarity: 0.95
GAME: Medieval II: Total War Kingdoms          Similarity: 0.95

The model used here is based on embeddings which are determined by a neural network. After training I have two matrices containing the embeddings:

Games Matrix: n games * embedding size
Tag Matrix: n tags * embedding size

The embedding size for both matrices are the same and the similarity score is calculated by the cosine distance of the game in question to all other games.

Would it be possible to find games similar to other games minus a given tag, for example, TOTAL WAR: WARHAMMER has the following tags:

Strategy
Fantasy
RTS
War
Grand Strategy

Say I like this game but I don't like the Fantasy element, could I somehow remove the Fantasy element when making a recommendation? Would a simple operation say Total War: WARHAMMER embedding - Fantasy embedding and then find similar matches work?