texts:
I am new to deep learning and I am trying to build a book recommender system using embedding layers. I use one layer for the book and one for the user.

I am having trouble with fitting the model. More specifically, when I try to feed the first layer with the books' ISBN list I get back the " ValueError : could not convert string to float: '087584877X' ".

Note that a book's ISBN ( as '087584877X' for example )  is either a sequence of numbers or a sequence of letters and numbers. The second category seems to be the problematic one.

I dont get why it should convert the string to float. I though that what the network does is to assign each string to a new integer regardless of the structure of the string.

Here is my code for the books embedding layer :

book_input = Input(shape=(1,))

book_embedding = Embedding(input_dim = n_books+1, output_dim = 4)(book_input)


texts: Ranking tumors from most malignant to most benign

texts:
I have a classification problem classifying tumors as benign or malignant. However, I want to go a step further to provide a ranking of these tumors as most malignant to most benign. Are there any good algorithms out there to help with this ranking? Any suggestions?

The features of the dataset are the radius of the tumor, perimeter of the tumor, concavity, smoothness, etc.


texts: Using spearmint for Bayesian hyperparameter optimization

texts:
I am working on a rnn project using a time series generator after training the model with some parameters from intuition. I would like to perform hyperparameter optimization using spearmint. I installed the package, but when I run:
"mongod --fork --logpath <path/to/logfile\> --dbpath <path/to/dbfolder\>"

it fails. I get the error:
ERROR: child process failed, exited with error number 48 To see additional
information in this output , start without the "--fork" option.

Can someone help me out on how to implement the package?


texts:
PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I simply change the activation function from sigmoid to relu, it drops to less than 50%. Is there a theoretical reason why this happens?

I have found the following example online:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)

Y_train = np_utils.to_categorical(Y_train, classes)
Y_test = np_utils.to_categorical(Y_test, classes)

batch_size = 100
epochs = 15

model = Sequential()
model.add(Dense(100, input_dim=784))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')

model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=1)
print('Test accuracy:', score[1])

This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that?


texts:
I have imbalance dataset for binary classification problem. I want to create a custom cost function that takes into account not only the actual class and probability, but another variable "premium amount" which is a feature in the model.
I have studied few blogs but it doesn't talk about including feature in cost function.
Kindly help!


texts: Jobs for gaining experience in preperation for data science

texts:
Data Scientist positions are not the most common entry level jobs. They require hard work, knowledge and most of all experience. The problem is, I am uncertain what entry level jobs can help you get experience in data science. I apologise for not asking a technical question, but I have been searching an answer and most importantly advice from experts and data science practitioners.

Currently, I am studying to become a Data Scientist and I was wondering what are good entry level jobs that can help me build a good foundation for future success in the field?


texts: How to assess that a cross entropy based model has converged

texts: I have a question regarding cross entropy convergence using Stochastic Gradient Descent. I am a little bit confused about how the convergence should be assessed. Should we treat the model as converged if the loss hit minimum on any single example or may be a certain number of examples? I am asking because what if the model converged on a single example just randomly.


texts: how to implement squared hinge loss in pytorch

texts:
does anyone have any advice on how to implement this loss in order to use it with a convolutional neural network?
Also, how should I encode the labels of my training data? We were using one hot encoding with bce loss before and I was wandering if I should keep it that way also for the hinge loss, since the label itself is not used in the formula of the loss other than for indicating which one is the true class.
The dataset is CIFAR100 by the way.
Thank you in advance!
EDIT: I implemented a version of this loss, the problem is that after the first epoch the loss is always zero and so the training doesn't go further.
Here is the code:
class MultiClassSquaredHingeLoss(nn.Module):
def init(self):
super(MultiClassSquaredHingeLoss, self).init()
def forward(self, output, y): #output: batchsize*n_class
    n_class = y.size(1)
    #margin = 1
    margin = 1
    #isolate the score for the true class
    y_out = torch.sum(torch.mul(output, y)).cuda()
    output_y = torch.mul(torch.ones(n_class).cuda(), y_out).cuda()
    #create an opposite to the one hot encoded tensor
    anti_y = torch.ones(n_class).cuda() - y.cuda()

    loss = output.cuda() - output_y.cuda() + margin
    loss = loss.cuda()
    #remove the element of the loss corresponding to the true class
    loss = torch.mul(loss.cuda(), anti_y.cuda()).cuda()
    #max(0,_)
    loss = torch.max(loss.cuda(), torch.zeros(n_class).cuda())
    #squared hinge loss
    loss = torch.pow(loss, 2).cuda()
    #sum up
    loss = torch.sum(loss).cuda()
    loss = loss / n_class

    return loss


texts: Modified Voting Algorithm to find the best recommendation

texts:
I have to find the best 10 items from the set of items for x number of given features. I don't know what the best recommendation will be. User data is not available to validate it. After reading a number of papers I came to know about voting algorithm.

Trying to implement something like this-

Find x number of features for static variables( that doesn't change over time), dynamic variables (that change very frequently).
Assume some weights for the weights on the basis of requirement for dynamic variables.
Try to come up with 3-4 regression models by combining the static and dynamic variables and find the best 10 items from each
Take the common one out, and then calculate the error.
Back-propagate the error to get the new weights.
Keep on iterating till we get at-least common top 8 elements.

Is the approach right?

Any guidance for what to take as error.

My approach to error calculation was-

Mean of common items feature values (X).
For each model MSE where mean is X and Xi is feature values for uncommon items of that model.

I am new to it. Can anyone please guide me through it.


texts: What is a reasonable way to compare the improvement in accuracy?

texts:
Suppose you have a classification task and accuracy is what you care about. Now an old system $s_1$ has an accuracy of $a(s_1) = 0.9$ and a new system $s_2$ has an accuracy of $a(s_2) = 0.99$. This is an absolute improvement of $a(s_2) - a(s_1) = 0.09$ percentage points and a relative improvement of $\frac{a(s_2)-a(s_1)}{a(s_1)} = \frac{0.09}{0.9} = 0.1$.

However, when you now try to get a system of $a(s_3) = 0.999$ this seems to be much more difficult, although it is only an absolute improvement of $0.009$ and a relative improvement of $0.00\overline{90}$. So neither the absolute nor the relative difference in accuracy seems to capture this well.

Is there a common other way to quantify how much better the system is?


texts: Error: An operation has `None` for gradient with categorical_crossentropy

texts:
I am trying to train my discriminator network using Keras with TensorFlow backend.
The network is meant to classify the input into one of the 9 output labels. I am passing a 2D input (height, width, no channels) and a one-hot vector for the output. I was able to train the network independently using fit(). However, now that I have switched to train_on_batch, it is giving me the error mentioned.

This is my discriminator code:

def build_discriminator(time_steps, feature_size, input_spectrogram=None):
    spectrogram = Input(shape=(time_steps, feature_size))
    # spectrogram = tf.placeholder(tf.float32, shape=(None, time_steps, feature_size))
    layer0 = Reshape((time_steps, feature_size, 1))(spectrogram)
    layer1 = Conv2D(32, kernel_size=(3,3), padding='same')(layer0)
    #model.add(LeakyReLU(alpha=0.01))
    layer2 = MaxPooling2D(pool_size=(4,4))(layer1)

    layer3 = Conv2D(16, kernel_size=(3,3), padding='same')(layer2)
    #model.add(LeakyReLU(alpha=0.05))
    layer4 = MaxPooling2D(pool_size=(4,4))(layer3)

    layer5 = Conv2D(16, kernel_size=(3,3), padding='same')(layer4)
    #model.add(LeakyReLU(alpha=0.05))
    layer6 = MaxPooling2D(pool_size=(4,4))(layer5)

    layer7 = Flatten()(layer6)
    layer8 = Dense(16)(layer7)
    prediction = Dense(9, activation = 'softmax')(layer8)
    # prediction = Dropout(0.1)(layer9)

    model = Model(spectrogram, prediction)

    opt = optimizers.Adam(lr=0.002, beta_1=0.5)
    model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

    return model

This is the code that trains the discriminator:

    x_real = batch_x[:half_batch, :, :]
    labels_real = batch_labels[:half_batch]
    d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))
    # generate 'fake' examples
    x_fake, labels_fake = generate_fake_samples(batch_x[half_batch:, :, :], batch_labels[half_batch:], generator)
    # update discriminator model weights
    d_loss2, _ = discriminator.train_on_batch(x_fake, to_categorical(labels_fake, num_classes=9))
    # update the generator via the discriminator's error
    g_loss, acc = gan.train_on_batch([batch_x, batch_targets], to_categorical(batch_targets, num_classes=9))

It is throwing the error on this line:

d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))

The error traceback:

  File "gan.py", line 126, in train
    d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1513, in train_on_batch
    self._make_train_function()
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 316, in _make_train_function
    loss=self.total_loss)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 75, in symbolic_fn_wrapper
    return func(*args, **kwargs)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/optimizers.py", line 504, in get_updates
    grads = self.get_gradients(loss, params)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/optimizers.py", line 93, in get_gradients
    raise ValueError('An operation has `None` for gradient. '
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

I am using tensorflow-gpu 2.0.0 with keras 2.3.1.
Could someone please help me understand where I am going wrong? Thanks is advance!


texts:
Problem: The following error repetitively comes up when I attempt to use the kable function in r. I have installed library(knitr) so I don't believe this is the issue. Note: The commands work when I type them into my console. The issue is that they are not working when I run them from my script (which prevents knitting the whole document etc).
Error Message:
Error in tempfile("rs_html", tmpdir = ctx$outputFolder, fileext = ".html") : temporary name too long
Examples of commands which do not run:
kable(parameterestimates(fit_uncor))
kable(parallel_table) kable(communalities_ml)  kable(communalities_ml_r)
Any help would be very much appreciated!


texts:
Kernels are used to map datasets into higher dimensions so that they could be linearly separable. However, if we introduce the slack variable in the soft margin SVM, we are allowing some mistakes, and in that case, do we still bother using kernels to try to make the dataset separable in higher dimensions knowing that we are allowing mistakes already? If so, what are the advantages of that compared to not using kernels?

Thanks!


texts: Is converting a numerical target to binary helpful?

texts:
I'm trying to solve an ML problem where the target variable is numeric, let's say the pollution level in a city. But the client is not interested in predicting the actual amount of pollutants, they are just interested in knowing whether the pollution level is high or low based on an agreed upon threshold. (High if the PM2.5 level is above 200, Low otherwise).

Should I treat it as a regression problem and take the numeric PM2.5 levels as target or as a classification problem where I make another feature of high/low pollution level based on the threshold and use that binary variable as a target? What are the advantages and disadvantages of both and What impact it can have on accuracy, if any?


texts: NLP: wit.ai. How to use confidence score?

texts:
I've created an application on Facebook's wit.ai. In my application, I have a single intent with the following phases:

Hello
Hi bud
Hi
Hey there
Hey
Hey bud !! How's it going ?
Hey bud !! Hoss it going ?

Now if I try the easy phase Hello, I get a confidence score 0.535.

But now if I  try the phase shut up, I also get a confidence score 0.535.

Question: How does the Facebook wit.ai work? I don't understand why I would get the same confidence score in both cases.


texts: recommender systems : how to deal with items that change over time?

texts:
Let's say I am building a recommender system where items change through time. We suppose that each transaction is composed of  :

an item $i$ in list of items $(i_1, i_2, i_3, .., i_m)$.
a user $u$ in list of users $(u_1, u_2, u_3, ..., u_n)$.
a date $t$ in list of dates $(t_1, t_2, ... t_k)$.

We suppose that items have underlying features that change over time.For example, If we consider retail products, underlying features could be :

The discount level that is applied on the item when the customer has purchased the transaction (5%, 10%, 20%, 30%, ...).

An other example, if we consider financial stocks, underlying features that change over time could be :

The stock situation at time of the transaction (underpriced or overpriced).
The stock's central bank politics at time of the transaction (low interest rates, medium interest rates, high interest rates).

We suppose that these underlying features have a strong impact on users. It completly drives their decisions to buy or not an object. If we consider two items $i_1$ and $i_2$, at time $t_1$, a given user $u_1$ could prefer $i_1$ over $i_2$ because $i_1$'s underlying features are more interesting than $i_2$'s. If we consider a different time, maybe $u_1$ could be more interested in $i_2$ than $i_1$.

My question is : how to take into account underlying features that change over time in recommender systems such as user-user collaborative filtering, SVD, ALS... ?.


texts: Transformer decoder output - how is it linear?

texts:
I'm not quite sure how's the decoder output is flattened into a single vector.
As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and we input the decoder with a length M sentence, the output of the decoder will give us M x units output. M is not fixed (M should be the length of the decoder's raw input) and will change during the different steps of inference.
How do we go from here to a single vector?

Screen shot from "Attention is all you need"


texts: Python encode piano notes in 88 bits - int too large to convert to C long

texts:
The piano keyboard has 88 keys:

I would like to encode one note or one chord in 88 bit array. I do this, for example:
A4 is 0000000000000000000000000000000000000000000000000000000001000000000000000000000000000000

Next I convert this number to decimal. When I have chord the final number is really big. The problem is when I put this big number into array from NumPy I gets:

note_data = np.array(note_data, dtype=np.int64)
OverflowError: Python int too large to convert to C long

My point is to feed recurrent neural netowrk this kind of data or maybe is better representation on music notes into numbers?

EDIT:

Another example of encode C3 note:

C3 = np.array([0000000000000000000000000001000000000000000000000000000000000000000000000000000000000000], dtype=int)

and error sitll the same:

OverflowError: Python int too large to convert to C long

It's possible to reduce this error?


texts: Large/Medium scale text classification with scikit-learn and beyond

texts:
I'm trying to make text multilabel classification ~40 labels from products description. Labels are unbalanced. There are ~3 labels per sample. And I have ~250k samples.

I digged Kaggle's text classification competition from 2014 and it seems I'm using correct algorithm stack is: scikit-learn/one-vs-rest/tfidf (bigrams)/SGDClassifier (SVM)/f1 score (global - average='micro'). I was able to achieve 0.54 f1 score which for me is bearable as a first step. Now I run out of ideas so I want to ask you for some clues.

I tried so far:

RandomForests - couldn't force them to achieve more than 0.30 f1 score and training time was so long...
KNNs - I'm afraid that they won't be production ready, because of high prediction time
stacking tfidf - PCA/SVD with 100-200 components didn't get significant boost when used with SVM (I must check RFs and KNNs)
WordNet lemmatizer form - it gave some improvements, but it is slow overall
GridSearch over parameters took a long time and didn't give a big improvement

In some moment I discovered that I can add class_weight=balanced, because when using one vs rest... and "rest" can become enormous.

I have problem:

I can use at most of 30k data, because:
LinearSVM is too slow to train with bigrams, but with more data it gives better results
SGD-SVM is perfect in speed, but with more data it has higher much much recall (it simple returns more labels)

Do you know what's wrong with my approach to SGD-SVM?

Do you have some clues what is worth trying? I'm planning to use CNNs and RNNs with theano, but it will take some time. Maybe model ensemble?

What about infrastructure? My laptop is pretty efficient but I would like to set up local grid infrastructure (I was you using LFS job scheduler). Is there anything similar to for scikit-learn? Or maybe cloud: AWS/Azure/Google Cloud, what is the best (I think the cheapest).

How about environment? Jupyter + pycharm works pretty well, but I'm missing some features from RStudio (like showing all variables).


texts:
Have a dataset of a E-learning company  which have information related to the students demographics, package, payment info, services used, class or grade (KG to 12), login details.
 If you want i can share the sample data set.
Could anyone help me out in understanding what business value can be derived from this data. I am really confused. I did some analysis was able to find out the max service consumption. Any help would be appreciated


texts: new and fresh ideas with stack exchange dataset

texts:
I'm a new bee for data analysis . i need to work on a research project in big data analysis. first of all i search for a dataset and i found interest in stack exchange  data dump. however i browse for researches i found a lot .And whatever i thought about a idea based on this dataset , its already done by someone else .
please help me out with a new and useful idea for my research .


texts: How to apply .replace() inside a column in a pandas data frame to clean data

texts:
I have a pandas data frame where I want to replace all the values that are not numeric in one column for ""

The error I'm getting in my code is the following:

ValueError: could not convert string to float: '690,276.00'

Since I'm trying to convert all values into float so I can do internal operations with them.

Part of my cleaning data frame code looks like this:

    # Cleaning:
df_clean = df_read[~(df_read['Ratio of Similarity (Gray)'] <= .2)]
print(df_clean, 'clean 1: Eliminate Ratio of similarity less than 0.2')
df_clean_2 = df_clean.dropna(subset=['buybox_price'])
print(df_clean_2, 'clean 2: Eliminate Nan Buybox Prices')
df_clean_2 = df_clean_2.replace(",", "").replace('', '').astype({'product_ranking':'float64'})
df_clean_3 = df_clean_2[~(df_clean_2['product_ranking'] >= 5000000)]
print(df_clean_3, 'clean 3: Eliminate Product Ranking + than 5.000.000')
df_clean_4 = df_clean_3[~(df_clean_3['buybox_price'] <= 6)]
print(df_clean_4, 'clean 4: Eliminate Buybox Price less than 6$')
# Save Cleaned File
path_file = os.path.join(BASE_DIR, 'csv/amazon_product_comparator.csv')
df_hc = df_clean_4.to_csv(path_file)

The error can be found in the line:

df_clean_3 = df_clean_2[~(df_clean_2['product_ranking'] >= 5000000)]
    print(df_clean_3, 'clean 3: Eliminate Product Ranking + than 5.000.000')


texts: Gathering the number of Google results from a large amount of searches.

texts:
I am trying to build a simple dataset using Google, mainly because it seems like the best option for what I want.

I want to measure fame for a large group of scientists. The quick method is to measure the amount of Google results when searching their name. I do not care about the results, only the number of them. That method has it's flaws I know, so I am not opposed to an alternative.

My scientist data is composed of thousands of entries. Which is causing issues. I tried to programmatically search Google, but less then 1000 searches later they blocked the program. I also looked into their Search API, but that is limited to 100 searches a day unless I pay for more, but since I am a poor college student that isn't an option.

I was hoping someone here may be able to offer suggestions on build a dataset with some way of measuring fame.


texts: Logic behind pre-trained weights and transfer learning

texts:
I am not sure about the logic behind, how pre-trained weights actually make sense and translate into a new problem.
To be more specific; for example in a object detection network, how would a model's weights that was trained, let's say, on the COCO dataset, with 80 categories, would translate into my new problem that only has 2 categories (classes).
How does this make sense? What kind of meaningful features could even be transfered from the previously pre-trained model to my new problem, since the number of categories (classes) have been changed, and also I'm trying to detect completely different type of objects than the previous model? Why do we do transfer learning then?


texts: Reason for square images in deep learning

texts:
Most of the advanced deep learning models like VGG, ResNet, etc. require square images as input, usually with a pixel size of $224x224$.

Is there a reason why the input has to be of equal shape, or can I build a convnet model with say $100x200$ as well (if I want to do facIAL recognition for example and I have portrait images)?

Is there increased benefit with a larger pixel size, say $512x512$?


texts:
I have a dataset that measures the flight details of objects, based on what action was performed. It looks similar to below:
action      velocity        angle       distance
1           25              12          100
1           27              15          110
2           45              24          160
2           48              22          175
3           76              45          267
4           89              67          536
5           null            null        null

When the action does not move the object at all, the velocity, angle, and distance are all null.
I intend to pass this data--after cleaning--to a LSTM RNN. Since LSTM RNNs process numeric data and the null values are relevant to the dataset, I need to treat the nulls for when the object does not move. The velocity and distance can be easily enough handled since when they don't move they are both technically 0, but how do I treat null angles, since 0 and -180 are just a valid angles as 180?
If it is relevant, I intent to convert the angle and velocity into a vector, by converting the angle to radians, then by multiplying the velocity by the sin/cos of the angle in radians. After this conversion, is it then viable to use 0s, the mean, or some other measure to fill the null values?


texts: Why training model give great result but real data gives very bad result: Azure ML Studio

texts:
I am using Two-Class Boosted Decision Tree to train model.

Evaluation result I'd say really good.

But when I am using real dataset - the result is very bad.

What can possibly go wrong that makes such huge difference?

Below is the screenshot of my model:

Two Class Boosted Decision Tree parameters (default):


texts: Preparation for Career in Data Analysis Without College

texts:
I dropped out of college but am interested in a career in data analysis. Now I am self-studying approximately 10 hours per day.  Browsing through job postings on Linkedin has allowed me to compose a rough curriculum.  It would be of great help to me if you would either add a subject I have omitted or eliminate a subject that is not necessary for success in the market place.

Curriculum (in 3-subject groupings):

Group 1

Single-variable calculus
Intro to python
SQL

Group 2

Multi-variable calculus/linear algebra
Discrete math
Data structures and algorithms

Group 3

Calculus-based statistics and probability
Hadoop stack
Differential equations

Group 4

Statistical learning/predictive modelling
Python data analysis techniques/Statistical programming in R
Fundamentals of machine learning

All the while I plan to practice using any data sets I can find online.  Will this be sufficient to land a job in data analysis?  Of course I plan to learn far more than just this, but is this foundation solid enough to land an entry level data engineering/science position?


texts:
I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am interested in knowing if there are any per-existing tools to do it?

I am happy to provide examples if needed :)


texts: Openai Spaces for a modified environment

texts:
I have a 2-dimensional array of normalized data. I am using

space = np.array([0,1,...366],[0,0.000001,.....1])

I need to fit this as an observation space in reinforcement learning. I have extended the open ai gym and created a custom made environment. How to fit in this 2-dimensional array in openAI spaces.
Can I use Box, DiscreteSpace or MultiDiscrete space? Can anyone help me with a sample code to fit this in observation space?


texts: what arguments should I pass to dbscan or optic in order to divide the data in a specific way

texts:
I have thousands of very similar datasets that needs to be divided in diagonal way to two groups.
for example:

and

I tried to play with the argument of dbscan and optic as epsilon and minPoints and even metric and none of them helped me to divide the data properly to 2 groups.

I only succeed to divide the data using dbscan. If I remove the noise between these groups to make them a complete separate 2 groups, I did it using histogram

j = 1
hist, bin_edges = np.histogram(data, bins=500)
max_bin = np.where(np.amax(hist) == hist)[0][0]
max_noise = bin_edges[max_bin+j]
filtered_indicies = data > max_noise
data = data[filtered_indicies]

these lines remove noise from the data, between the groups and also around it when

  j > 1

and that causing me to remove necessary data that I need to reprocess later.

so, I am going back the my main question, how can I know which epsilon, minPoints or other argument of dbscan can help me divide this data properly?
or is there maybe a better way then what I presented here above (histogram) to remove the noise between these groups without removing necessary data?


texts: RL's policy gradient (REINFORCE) pipeline clarification

texts:
I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly non-conventional):

$$\omega = \omega + \alpha \cdot \nabla_\omega log\ \pi(A_t|S_t) * V_t$$

My questions I am unsure about are the following:

Do I calculate the gradient values at each time step $t$ (like in
SGD fashion) or averaging gradient over all timesteps of the episode is a better option?
Do I care about getting the gradient values of the selected action
probability output only (ignoring the outputs for other actions, in
a discrete case)? In other words, do I consider the $V_t$ term for non-selected actions to be 0, which make the gradient values equal 0 as well?
In a discrete case the cross-entropy (the loss) is defined as:
$$H(p,q) = -\sum_x P(x) * log Q(x)$$

(source: wikipedia)

Does that mean that if I substitute the labels (denoted as $P(x)$)
with the $V_t$ terms (non-zero for selected action only) in my
neural network training, I will be getting the correct gradient
values of the log-loss which fully satisfy the REINFORCE definition?


texts: XGBoost custom objective for regression in R

texts:
I implemented a custom objective and metric for a xgboost regression task. In order to see if I'm doing this correctly, I started with a quadratic loss. The implementation seems to work well, but I cannot reproduce the results from a standard "reg:squarederror" objective.
Question:
I wonder if my current approach is correct (especially the implementation of the first and second order gradient)? If so, what could be a possible reason for the difference?
Gradient and Hessian are defined as:
grad <- 2*(preds-labels)
hess <- rep(2, length(labels))

Minimal example (in R):
library(ISLR)
library(xgboost)
library(tidyverse)
library(Metrics)

# Data
df = ISLR::Hitters %>% select(Salary,AtBat,Hits,HmRun,Runs,RBI,Walks,Years,CAtBat,CHits,CHmRun,CRuns,CRBI,CWalks,PutOuts,Assists,Errors)
df = df[complete.cases(df),]
train = df[1:150,]
test = df[151:nrow(df),]

# XGBoost Matrix
dtrain <- xgb.DMatrix(data=as.matrix(train[,-1]),label=as.matrix(train[,1]))
dtest <- xgb.DMatrix(data=as.matrix(test[,-1]),label=as.matrix(test[,1]))
watchlist <- list(eval = dtest)

# Custom objective function (squared error)
myobjective <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  grad <- 2*(preds-labels)
  hess <- rep(2, length(labels))
  return(list(grad = grad, hess = hess))
}

# Custom Metric
evalerror <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  u = (preds-labels)^2
  err <- sqrt((sum(u) / length(u)))
  return(list(metric = "MyError", value = err))
}

# Model Parameter
param1 <- list(booster = 'gbtree'
              , learning_rate = 0.1
              , objective = myobjective
              , eval_metric = evalerror
              , set.seed = 2020)

# Train Model
xgb1 <- xgb.train(params = param1
                 , data = dtrain
                 , nrounds = 500
                 , watchlist
                 , maximize = FALSE
                 , early_stopping_rounds = 5)

# Predict
pred1 = predict(xgb1, dtest)
mae1 = mae(test$Salary, pred1)

## XGB Model with standard loss/metric
# Model Parameter
param2 <- list(booster = 'gbtree'
              , learning_rate = 0.1
              , objective = "reg:squarederror"
              , set.seed = 2020)

# Train Model
xgb2 <- xgb.train(params = param2
                 , data = dtrain
                 , nrounds = 500
                 , watchlist
                 , maximize = FALSE
                 , early_stopping_rounds = 5)

# Predict
pred2 = predict(xgb2, dtest)
mae2 = mae(test$Salary, pred2)

Results:

The custom metric yields a slightly better result MAE=199.6 compared to the standard objective MAE=203.3.

During boosting, the RMSE tends to be lower with the custom objective.

For the custom objective the RMSE is:
[1] eval-MyError:599.490030
[2] eval-MyError:560.677996
[3] eval-MyError:527.867686
[4] eval-MyError:498.216760
[5] eval-MyError:472.167415
...

For the standard objective the RMSE is:
[1] eval-rmse:598.144775
[2] eval-rmse:562.479431
[3] eval-rmse:529.981079
[4] eval-rmse:501.730103
[5] eval-rmse:479.081329


texts: How to find bias for perceptron algorithm?

texts:
My question is very basic. I am starting with ML and am working on the perceptron algorithm. I successfully computed the weights for this input data:

X = [[0.8, 0.1], [0.7, 0.2], [0.9, 0.3], [0.3, 0.8], [0.1, 0.7], [0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

Output_weights = [-0.7, 0.5]

But I didn't take bias into account, i.e. I assumed the discriminator line goes through the origin. Now, let's say I add another point into my training set:

new_X = [4,4]
new_Y = [-1]

How do I proceed if I want to compute the bias as well? In the first iteration for example, I'd set default weights to $[0,0]$, so I find the first point that is incorrectly classified.

Without bias, it is easy. I compute the dot product

0.8*0 + 0.1*0 = 0

should be $-1$, so it is incorrectly classified. I update the weights to:

[-0.8,-0.1]

However, taking bias into account, I get:

0.8*0 + 0.1*0 + bias

Now, how do I update the weights and the bias? What is the procedure?

I have searched several tutorials like this or this but didn't find an answer. A link to some resource would help, too.


texts: Similarity of search results using Jaccard

texts:
I have a set of search results with ranking position, keyword and URL. I want to make a distance matrix so I can cluster the keywords (or the URLs). One approach would be to take the first n URL rankings for each keyword and use Jaccard similarity.

However, I also want higher position ranks to be weighted more highly than lower position ranks - for example two keywords that have the same URL in positions 1 and 2 are more similar than two keywords that have the same URL ranking in positions 39 and 40.

I saw suggested that I could add in higher ranks multiple times, for example:

keyword_1 = [1,1,1,1,1,2,2,2,2,3,3,3,4,4,5]

where the number is the id of the URLs that rank in positions 1-5 of keyword_1.

This means that I can't use for example sklearn Jaccard implementation because sets are assumed.

I had a go at implementing this myself and intuitively the results seem to make sense, but I would like it to run faster, as I could use data for rankings up to 100. There is a lot of looping involved - is there a way of using numpy better to make this code more efficient?

Alternatively, is there a different approach that I haven't found to use already built algorithms?

def jaccard_sim_with_dupes(item1, item2):
        intersection = 0
        for i in item1:
            if i in item2:
                intersection += 1
        for j in item2:
            if j in item1:
                intersection += 1
        union = len(item1) + len(item2)
        return intersection / float(union)

def make_jaccard_distance_matrix(X):
    shape = (len(X), len(X))
    jaccard_distance_matrix = np.zeros(shape)
    for i in range(shape[0]):
        for j in range(shape[1]):
            jaccard_distance_matrix[i][j] = 1 - JaccardDistanceMatrix.jaccard_sim_with_dupes(X[i], X[j])
        if (i % 100 == 0):
            print ("\r{0:.0%}".format(float(i) / shape[0]))
    return jaccard_distance_matrix

Here X is a list of each keyword, where each keyword is represented as a list of URLs, with higher ranked keywords added in multiple times.

X = [[1,1,1,1,1,2,2,2,2,3,3,3,4,4,5], [6,6,6,6,6,1,1,1,1,7,7,7,8,8,9]]

make_jaccard_distance_matrix(X)

returns

array([[ 0. ,  0.7],
   [ 0.7,  0. ]])


texts: Integration of NLP and Angular application

texts:
I'm doing a small POC in which I've trained my Machine Learning model (Naive Bayes) and is saved in ".pkl" (pickle) format. Now my next task is to develop a web application which asks the user to enter the Text for the Text classification analysis. This newly taken (from the user) "TEXT" will be the testing dataset which can be fed to the Naive Bayes model that I built in the earlier stage and make prediction on the "text" taken from the user.

My Question is: Is there any way to convert the text (taken from the user) into numpy array and then transform this numpy array using CounterVectorizer() (which was used for training dataset) and then fed to saved model for predictions ?

I want to make an angular application like this. In this, input is an image and this image can be converted into pixels.

In my case, it is plain text which needs to be transformed into numpy array in Angular application before I feed it to the Trained Naive Bayes Model.

Thanks inadvance!


texts: Can i use other regression types that arent based in decision trees to use it like a weak learners in gradient boosting?

texts: I was thinking if i can use polynomial regression like a weak learners in gradient boosting but i read that decision trees are used for that and i cannot find anything that show me the possibility of other weak learners could be used.


texts: Unable to Predict the custom trained mobilenet in browser using TensorflowJS

texts:
I am new to TensorflowJS and Javascript. I trained a mobilenet with images of diamonds using Tensorflow Object Detection so that it can detect Diamonds. The model is saved in SavedModel format. I converted the SavedModel format into TFJSgraphmodel format using the tensorflowjs_convertor.py and it created a model.json and 29 Shard file.
I am using this [https://github.com/dkreider/tensorflowjs-cat-vs-dog] tutorial which is very similar in functionality of uploading a image and predicting after a button is pressed.
I changed the await tf.loadLayersModel to await tf.loadGraphModel in its index.js file, as converting from SAvedModel to LayerModel is not supported, I have to use loadGraphModel.
I tried the below code to predict on Click
async function initialize() {
    try{
        model = await tf.loadGraphModel('trained-model/model.json');
    }catch(e) {
       console.log("The model is not loaded")
    }

    classifierElement.style.display = 'block';
    loaderElement.style.display = 'none';

    document.getElementById('predict').addEventListener('click', () => predict());
}

async function predict () {

  const imageElement = document.getElementById('img');
  let tensorImg = tf.browser.fromPixels(imageElement).cast('int32').resizeNearestNeighbor([640, 640]);
  const t4d=tensorImg.expandDims(0);
  t4d.print()
  console.log(typeof(t4d))

  data = await t4d.data()

  const prediction = model.executeAsync(t4d).then(prediction=> {
    const data = prediction.dataSync() // you can also use arraySync or their equivalents async methods
    console.log('Predictions: ', data);
  })

But when I run it, I am getting this error:
Uncaught (in promise) TypeError: Cannot read property 'backend' of undefined
    at e.t.moveData (tfjs:17)
    at e.t.get (tfjs:17)
    at Object.dG [as kernelFunc] (tf.min.js:17)
    at n (tfjs:17)
    at tfjs:17
    at e.t.scopedRun (tfjs:17)
    at e.t.runKernelFunc (tfjs:17)
    at e.t.runKernel (tfjs:17)
    at reshape_ (tf.min.js:17)
    at reshape__op (tf.min.js:17)

Kindly help, I am stuck on this for many days.
Thanks


texts: Can I feed only images to simple CNN model without using label data?

texts: I have only image dataset. I want to take only feature map from simple CNN model so can i give only images to model without using label data? How to fit to madel only images?


texts: Estimating X based on Y values in neural network

texts:
Let's say I have a trained neural network model, preferably with couple of dense layers. It has learned to estimate Y values based on X values.
Can I somehow use the same trained model and find approximation of X values that result in a specific Y values?
In other words, I have some Y values, I'd like to find out X values which if I feed to the same model, result those Y values.


texts: data splitting into 3 sets based on years

texts: let's suppose we have a customer data from the year 2015 to 2019, I want to train_test_split() my data such that my data gets divided into three sets, set-1 is from 2015 to 2017 (3 years) on which i will train my model, set-2 i.e. 2018(1 year) on which i will validate my model , set 3 is 2019(1 year) on which I will test my model. I want a code to divide data into 3 sets based on time(years).


texts: predict_proba to print specific class probablity

texts:
I am having 16 labels and predict_proba is giving me probablities of all 16 categories in an array. Is there any way if I pass specific label to predict_proba it can only print me probablity of that category?

preds = model.predict(dataframe)
# getting predicted class , am interested in knowing probablity of this class.
print(preds)
# it is printing array for all 16 labels, I am keen to pass above predicted class and retrieve probability for it.
print(model.predict_proba(dataframe))

Output
[[0.07387347 0.007413   0.00354506 0.02321654 0.09627853 0.00958647
  0.00599333 0.02232621 0.12513558 0.00494633 0.07230524 0.00384056
  0.00378245 0.44431455 0.04089799 0.0625447 ]]

now model had predicted class "XYZ" how to get its index in predict_proba function, if I have that I can simply use (i being index)
print(model.predict_proba(dataframe)[0][i])


texts: How to remove features from a sklearn pipeline after it has already been fitted?

texts:
Background:
I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to fit an estimator that performs automatic feature-selection, for example, lasso regression. Thus, I begin my workflow with n features, but ultimately only m (where m < n) of them will be used to make predictions for the final model. A simple example is provided below:
from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Use Wisconsin breast cancer dataset as an example
X, y = load_breast_cancer(return_X_y = True)

# Produce train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)

# Pipeline preprocesses then applies lasso regression
pipe = Pipeline(steps = [
    ('preprocessing', StandardScaler()),
    ('lasso', LogisticRegression(penalty = 'l1', solver = 'saga', C = 0.4))
]
)

# Fit the pipeline
pipe.fit(X_train, y_train)

# View the coefficients
pipe.named_steps['lasso'].coef_

print('Count of non-zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] != 0))
print('Count of zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] == 0))

Count of non-zero coefficients:  20

Count of zero coefficients:  10

Objective:
I am now looking to apply my model to a new set of observations to produce predictions for them, but it seems computationally wasteful to run all of the n features through the pipeline, as only m of them will ultimately contribute to the final prediction. The preprocessing steps are not theoretically required on the other n-m features once the model has been fit. Of course this example uses a toy dataset, but on larger datasets the excessive processing could be material.
# Predictions require all features to exist in the X dataset
pipe.predict_proba(X_test)

# Passing a subset of the features causes an error - and understandably so since the pipeline
# lacks the context needed to determine which features are included
pipe.predict_proba(X_test[:,np.where(pipe.named_steps['lasso'].coef_[0] != 0)[0]])

ValueError: X has 20 features, but StandardScaler is expecting 30 features as input.

Is there a solution that would allow me to 'pare down' an already-fitted pipeline object so that it can operate on only the m features that are theoretically needed to make my predictions going forward?


texts:
I have a set of customer reports, each is in ms word file. they are all in a similar pattern, for example they start with Name: --, Age: --, Date: --, etc...

is there a way to extract particular strings from each file to form a data set.

In orange, I was able to compile the word documents into corpus which I can display as one column (each report is in one cell). Does orange have a way to extract strings into columns (for example if between "age:" and "gender")?


texts:
When NIFI is used to push data via the Azure Datalakes HDFS interface, one needs to add the respecitve hadoop-azure-datalake-3.3.0.jar file like described in the documentation. However, I'm not completly sure whether everything there applies to this case as there is no local hadoop installation, there just is a NIFI which should use HDFS in Azure.
This documentation also has a troubleshooting entry for my exact error message. However I fail to apply the suggested solution in my NIFI-Case.
How can I add the folder of the above jar file to NIFIs classpath?


texts:
Based on this answer, we know that we can perform build covariance matrix incrementally when there are too many observations, whereas we can perform randomised SVD when there are too many variables.

The answer provide are clear and helpful. However, what if we have a large amount of observations AND variables? e.g. 500,000 samples with 600,000 observations. In this case, the covariance matrix will be huge (e.g. 2,000 GB, assuming 8byte float, and if my calculation is correct) and will be impossible for us to fit it into memory.

In such scenario, is there anything that we can do to calculate the PCA, assuming we only want the top PCs (e.g. 15 PCs)?


texts:
I am trying to use XGBoost in python for logistic regression. I am calling it as follows

import numpy as np
from xgboost import XGBClassifier

x_train = np.array([[1], [2], [3], [4]])
y_train = np.array([0, .25, .75, 1])

params = {
    "objective": "reg:logistic"
}

model = XGBClassifier(**params)
model.fit(x_train, y_train)
print(model.objective)

This outputs an objective of "multi:softprob" instead of "reg:logistic." Therefore, it isn't doing a logistic regression. How can I make sure that XGBoost doesn't switch the objective?


texts: Why does gradient descent gives me much better Relative Squared Error then the Least Squares approach?

texts: Am I doing regression task with 7 dependent variables and 10000 data points. The SGD gives me 22% of mean absolute percentage error on test and train dataset. And Least Squares method using numpy scipy.optimize.least_squares give me only 58% (I have tried different settings.). I thought that Least Squares should give the same or better performance with such size dataset. What can be the reasons?


texts: Back propagation through a simple convolutional neural network

texts:
Hi I am working on a simple convolution neural network (image attached below). The input image is 5x5, the kernel is 2x2 and it undergoes a ReLU activation function. After ReLU it gets max pooled by a 2x2 pool, these then are flattened and headed off into the fully connected layer. Once through the fully connected layer the outputs are converts into Softmax probabilities. I've propagated froward through the network and am now working on back propagation steps. I have taken the derivative of cross entropy and softmax, and calculated the weights in the fully connected layer.
Where I get confused is how to preform back propagation through Max pooling and then ultimately find the derivatives of the weights in the convolution layer.
What I've found online is that you need to find the derivative of the loss with respect to flattened layer, but I am unsure on how you do that.
If I could get some help with an explanation, ideally with equations it would be awesome.
Cross posted in stack overflow (https://stackoverflow.com/questions/63022091/back-propagation-through-a-simple-convolutional-neural-network)


texts: How to train and validate a model continously which affects its own future data?

texts:
We are working with a online marketplace. Our problem is to predict whether certain products are profitable or not for our marketplace in near future(next one month horizon).

For example: Consider 2 products

Toy Giraffe (Predicted not profitable)
Sketch Pen (Predicted profitable)

For the 2nd product which was predicted profitable we take no action and as such we can get the profitability data of next month and check if our prediction was correct.

For 1st product which was predicted non-profitable we take actions which are supposed to help make it profitable. Now 2 scenarios arise: 1. It becomes profitable next month. 2. It doesn't. In both scenarios how do I know whether our initial prediction was correct or not. In scenario 1. It could have gotten profitable since my prediction was wrong or because of the corrective action. In 2. scenario also whether my prediction was wrong or the action taken was bad.

So how do I use the 1st products result in training my model next month?

Simple Tree diagram.

                                        start
                                          |
                                         / \
                                        /   \
                                       /     \
                                      /       \
                                     /         \
                                    /           \
                          Toy Giraffe          Sketch Pen
                            Predicted:         Predicted:
                            Non Profitable.    Profitable.
                                  |                |
                                  |                |
                                  |                |
                            Action Taken       No Action
                                  |
                            Wait 1 Month
                                  |
                                 / \
                                /   \
                               /     \
                              /       \
                             /         \
                            /           \
                           /             \
                          /               \
                     Case 1.             Case 2.
              Is Profitable.           Is not Profitable despite action.
                   |                                  |
                   |                                  |
                  / \                                / \
                 /   \                              /   \
                /     \                            /     \
               /       \                          /       \
              /         \                        /         \
             /           \                      /           \
            /             \                    /             \
      Prediction      Prediction           Prediction      Prediction
     was correct.   was not correct.      was correct.     was not correct.
          |              |                    |                |
  Profitable due      Profitable          Action Failed.    Got non-profitable due to Action.
   to Action.         since our
                   Prediction was wrong.

Under case 1. and case 2. for Toy giraffe nodes how do I identify whether my prediction was wrong/right or the taken action impacted the results. Who among prediction and action was responsible for the end result? The 4 leaf nodes are the possible events that can happen and I need to analyze each case.

References:

How to deal with a machine learning model which affects future ground truth data?
https://stats.stackexchange.com/questions/13533/what-is-reject-inferencing-and-how-can-it-be-used-to-increase-the-accuracy-of

Not sure if second link will be useful technique or not.

Also is Reinforcement learning something to look at for these kind of problems where the reward can be said to be delayed?


texts: How is the min_rank used with the flights database

texts:
I'm exploring the https://r4ds.had.co.nz/transform.html#add-new-variables-with-mutate r for data science handbook and don't really understand the min_rank() operator.  Doing the exercises it asks to Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for min_rank().

here is the code that I performed that doesn't work. Please explain:

min_rank(flights, dep_delay)


texts: How to feed two (or more) time-series data sets to a neural network?

texts:
I want to train the network based on two sets of data. For example, I want the network to predict the humidity based on past humidity trends AND past temperature trends. In this case, how should I organize the input layer? I would think that for just one time-series, I would use a regular sliding window on the series. With two series, do I just present both windows (from the humidity and temperature series) to the input layer that is twice the size of the window? if not, how else can I configure the input layer so that it doesn't confuse between the two sets of training data?

Do I just let the network sort this out by itself or is there a preferred method of presenting two (or more) sets of training data to the network? Thanks.


texts: Where can I find a trained neural network data to play with?

texts:
This is the trained neural network for the XOR operator:

Can I find something like a trained network for recognizing hand writing digits somewhere on the internet? Is there an "official format" for trained neural networks?


texts: Can I get a global attribution when using Shapley values?

texts:
I'm using a shap KernelExplainer to interprete my model and I'd like to display a global attribution for each feature. The idea is to represent, in a numerical way, the plot.
What would be the best way to calculate such attribution?
Is there any built-in function/method/library that I can use?


texts: How can I make a prediction in a regression model if a category has not been observed already?

texts:
I'm researching a regression model to predict a target value that has four features, all of which are categorical.

The categories are not fixed, e.g. one is a customer identifier. How could my model handle making a prediction for a customer identifer it hasn't seen before, based on the remaining features it has been trained with?

I have considered having a model for each of the features that could predict which category label is most similar based on the other three remaining features (or multiple similar category labels could be used and take an average of the target values for those).

My only concern with with this method is that it's not that scalable, I'm going to want to extend the model with more and more categorical features.

Is there some technique that could create an 'unknown' label for each of the features so the model can handle this case or would the prediction likely be completely inaccurate?


texts:
I am using Keras to build my architecture.

The regression problem I am trying to solve has outputs different for different training samples.

Suppose that I have very first two rows of y_train = [[16, 3], [6], ... ] corresponding to my very first two rows of X_train (input data). I would like to assign the output of the dense layer's unit to length of these y_train rows.

For example, for 1st training sample, I would like model.add(dense(2)) as y_train[0] has length 2, for 2nd training sample, I would like model.add(dense(1)) as y_train[0] has length 1, and so on.

I also think that I can call Lambda Layer instead of output dense layer and in that Lambda Layer I can wrap length of every row of y_train and assign it as output units by using output_shape argument of Lambda Layer. But, I don't know how can I access my y_train at training time of my model so that I can use y_train inside lambda layer to make this scenario happen either, nonetheless.

Can anybody help me on this variable length output problem?


texts: How to find a changing relation between continuous variables?

texts:
I have a dataset with 8 continuous variables describing the behavior of a machine:
- input a,b,c for both left and right: a_left, b_left, c_left, a_right, b_right, c_right
- output o, non-linearly related to the inputs (and almost certainly also influenced by other factors)
- timestamp t
I have some environmental data (e.g. temperature) as well but these are fairly constant and I suspect not of significant influence.

Theory:
My theory is that due to performance degradation of the machine due to fouling, the value of o will be lower for the same input values.

Occasionally, the machine is cleaned to restore performance again. I do not know when and how well (i.e. to a "perfect" state vs. to some intermediate state between "perfect" and the previous state). The rate of degradation is also not known and most likely not constant.

Furthermore, a change is one of the input variables has a delayed effect on the output due to inertia.

Actions I have tried:
- plotting the data for some different relations between the input and output
- k-means clustering for some different relations between input and output (without timestamps)

Questions:
- Is it at all possible to verify my theory with this data?
- If so, what techniques are most suited to be applied? I'd prefer anything in Matlab.


texts:
If this is a duplicate, I apologize. I'm not really sure what to even search for to try and find a duplicate/answer!

We are working on a system for providing musical feedback to change the 'mood' of a subject. There are a number of parameters of the music that we can manipulate.

At the same time, we measure the physiology of the subject. I'm using a dynamic Bayesian network to reasonably accurately determine the level of frustration of the subject.

I'm looking for a way to, based on the level of frustration, tell the music generation mechanism "this set of parameters is effective", or not. The music parameters would then adjust automatically and iteratively, as we continue to make judgements of the subject's level of frustration.

I'd appreciate any pointers I can get! Please let me know if I can provide any clarification!


texts: What is the most straightforward way to visualize color-coded clusters along with the cluster centers?

texts:
I have applied the kMeans Clustering algorithm to a dataframe and have gained cluster labels for each row. I had selected only two features.
There are 4 clusters.
I want to visualize the datapoints in 2D plane with color-coded clusters which I want to look like this-

Ignore the labels. I would like to plot cluster centers instead.
I have looked in many blog posts, articles etc. None was helpful.
What is the most straightforward to achieve this?


texts: Multi-label classification model in python?

texts:
Assume you have the following artificial dataset

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))
df['sex']  = [np.random.choice(['male', 'female']) for x in range(len(df))]
df['weight'] = [np.random.choice(['underweight',
        'normal', 'overweight', 'obese']) for x in range(len(df)) ]

This produce the following artificial dataset

df.head()
Out[50]:
          A         B         C         D     sex       weight
0  0.955136  0.802256  0.317182 -0.708615  female       normal
1  0.463615 -0.860053 -0.136408 -0.892888    male        obese
2 -0.855532 -0.181905 -1.175605  1.396793  female   overweight
3 -1.236216 -1.329982  0.531241  2.064822    male  underweight
4 -0.970420 -0.481791 -0.995313  0.672131    male        obese

I am trying to know both the sex and the weigh based on the value of the features A,B,C,D. I learned that this a multi-label classification problem and there is a nice python library that should help (e.g. scikit-multilearn ). However I do not know how this is achieved. Can someone show me how I could train a model and test its accuracy on this artificial dataset? Specifically:
1. Assuming that A,B,C,D are the feature and sex and weight are the label, how to create a test and training sets? The following will not work

X=df[list('ABCD')]
y=[['sex','weight']]
X_train, X_test, y_train, y_test = train_test_split(X, y)

How to you train and test the model?


texts: How to choose solution - Neural Neworks or Scikit-Learn/Numpy/Pandas?

texts:
I am trying to solve a problem - categorising and routing service desk emails to concerned teams for resolution.
Created and tested a model using Scikit-Learn, Numpy and Pandas. - Tokenized the email subject and body, used Bag-of-Words/TfIdf, and applied ML algos like - SVMs, Random Forest, Linear Classification, etc.

Now, as I read more came across NLP and Neural Networks, Keras, PyTorch, Tensorflow, etc.

May I ask how to choose the correct tool or solution for my particular problem? Please advise.


texts:
My documents are only a single sentence long, containing one annotation.
Sentences with the same named entity of course are similar, but not context-wise.
NER training examples (afaik) always has documents sequentially related, aka the next document is context-wise related to the previous document. Consider the example below. The first sentence is about the US, with location annotations. The second sentence is about an organisation but still related to the previous.

The United States of America (LOC), commonly known as the United States (U.S. or US).
The Bank of America (ORG) is a multinational investment bank.

My dataset for example would be:

The United States of America (LOC), commonly known as the United States (U.S. or US) or America (LOC).
The Netherlands (LOC), informally Holland, is a country in Western Europe.
Peter (PER) works at the harbour.

The sentences are not related. When considering a bi-lstm, should I somehow separate the two sentences during training? so that it doesn't think that the annotation of the current sentence is related to the previous sentence too?
Take for example the dataset previewed in this Kaggle notebook. It has a sentence_idx to separate sentences by ID but other than that its just one gigantic long list of words (with features). What happens when a (bi)-lstm finds itself in a completely different context, where the current sentence has absolutely no relation to the previous.
Normally I suppose this isn't a problem because documents are very large, but mine are just a single sentence. Not sharing a context, except that, for example, sentences with a LOC annotation are of course about locations, but not a specific context.
I had a very difficult time describing my problem, questions and edits to make it more clear are very welcome. I believe a similar question is: How to train NER LSTM on single sentence level


texts: Explain FastText model using SHAP values

texts:
I have trained fastText model and some fully connected network build on its embeddings. I figured out how to use Lime on it: complete example can be found in Natural Language Processing Is Fun Part 3: Explaining Model Predictions
The idea is clear - put 1 sentence into Lime, it drop words and generate some new sentences from my and check how score changes.
My next idea - use SHAP values for this.
SHAP values can be used for any deep model, using DeepExplainer. Here is a usage example: Keras LSTM for IMDB Sentiment Classification
But I can't use it for my ensemble, because DeepExplainer needs tensors as input, but I want to fed sentences. I don't want to use BoW or TF-IDF for that - I loose fastText power in that situation.
What I want to achieve - get some shap-plots built on words of my sentence.
Is it possible?


texts: I'm for a function that can plot the following plot using python:


texts: Tuning neural network loss function for space physics

texts:
I am trying to work on neural network classification (with python, Keras) for space physics purposes, where I want to identify specific planetary regions based on multivariate temporal data and multi class classification. Given my objectives, I would like to use a loss function that minimizes such a parameter: absolute_difference_between_predicted_and_real_class * temporal_distance_to_closest_real_class_change

More precisely: I have 3 classes corresponding to specific regions around planets, i.e. close planetary environment (class 1), shock (a physical thin boundary at the limit of inner environment; class 2), solar wind (particle flow due to the Sun; class 3).

With classic algorithms I have 2 issues:

I get predicted class variations that are physically impossible such as class 3 $\rightarrow$ class 1 or vice-versa, and predicted classes too different from real ones (3 instead of 1 or vice-versa, while 2 instead of 1 or 3 is less problematic for me) ; hence I would like to introduce the absolute_difference_between_predicted_and_real_class in the loss function
I have significant uncertainties on the exact time of class changes (3 $\rightarrow$ 2, 2$\rightarrow$1 or vice-versa), so that a wrong predicted class transition may not be a problem if not too far from the "real" (but actually always uncertain) transition; hence I would like to introduce the temporal_distance_to_closest_real_class_change in the loss function

I know I could use post-treatment but I would prefer, if possible, to introduce these constraints directly inside the loss function. In principle I could write a loss function doing that but I don't know how to cope with the constraints of derivability and others needed for the loss function. I also read a bit on ordinal classification, that would correspond to my issue #1, but not sure not adapted to issue #2. For issue #2 I heard about time warping techniques, but not sure to understand to use that in my case.

Overall I'm a bit lost! Any idea or advice?


texts: How to identify multiple lines/clusters in a single dataset

texts:
I'm currently struggling to wrap my head around how multi-linear regression could be done to find separate sets of linear models in a single data set. I can perform regression on single data set for a single regressor and coef with no problem, but what if there are a known-number of lines existing in a single data space?

My first approach was to use hierarchical clustering to identify the points using ML first, but it doesn't seem to capture individual cluster variance in Euclidean space as expected. My second trial was KMeans, which still relies on Euclidean distance so it creates clusters with radii. My last thought process led to Kmedian, but at this point I was wondering what other people might think regarding this problem. If this is the right direction, I know I would have to project points in a better space(i.e. an axis that captures more-or most- variance) before I apply these methods.

I would appreciate any comments or input in any shape or form.

Thank you,

3-line summary: Linear regression on dataset with multiple lines Clustering first, then multiple single linear regression? or have you guys come across a module for this such thing?

I would truly appreciate any insights;


texts:
I have a dataset of about 10000 unlabeled job titles (mostly very short titles) such as head of mobile or lead iOS developer.

I would like to perform classification of those job titles in two different ways :

The 1st classification is sorting job titles according to the type of job it represents (i.e. marketing, IT, healthcare, legal...)

The second classification would be about the level of seniority of a specific job (i.e. executive, manager, associate, trainee...)

I have tried an approach based on Word2Vec (vectors were taken from the Google Word2Vec set)  :

I cleaned and stemmed job titles
I created a vector representing the job title as the average vector of the words that compose it
I created a list of vectors representing the target values (executive could be the average vector of +executive, +chief, -assistant for example)
Find the least cosine distance between the job title vector and the target vectors.

While this approach gets decent result (about 70-80% accuracy) it is not enough for the task I'm planning to do.

So I was wondering if a better approach could be used (except labeling the data by hand and using that to train some algorithm)


texts: What is the use of @ between two matrices

texts:
I was looking into a repo, where I came across this snippet of code:

priors = x[None, :, :, None, :] @ self.route_weights[:, None, :, :, :]

It basically tried to incorporate weights of an input image matrix.
I am really not sure what is the use of @ out here and what are the advantages of using this. Is there any simple alternative too, like won't simple matrix multiplication work?


texts: Classify based on features being in ranges?

texts:
I'm trying to build a classification model where instances are classified based not on some linear combination of their features, but instead based on whether the features are all within some (unknown) ranges. So, for example, if I had two features, and my instances (features, followed by known classification) are:

(0, 5): 0
(2, 7): 1
(3, 2): 0
(4, 4): 1
(3, 9): 0
(6, 6): 0

then perhaps the model would learn that the first feature should be between 1 and 5, and the second feature should be between 3 and 8. (I only want "and"s in the model; I'm not looking for any cases where feature 1 is between a and b OR feature 2 is between c and d.) In essence, the model should find some hyperrectangle in feature space with all edges parallel to feature axes, where 0s are expected to be outside the rectangle and 1s are expected to be inside it. (In the example data, the data points are separable; this is not the case in my real problem. In the spirit of an SVM or logistic regression, the model should do the best it can.)

My first stab at this involved trying to do least-squares regression with a hypothesis function made up of two sigmoids for each feature (one "forward", one "backward"), parameterized to slide back and forth, all of which were multiplied together. Maybe my scipy-fu is weak, but above four features, I started having convergence issues with that approach. Is there some tried-and-true way to solve this kind of classification problem? (Even just a name for this kind of problem would be helpful!)


texts:
I am facing the following problem in machine learning. Given n ordered sets, each containing 6 numbers, generate the next set in the sequence. The numbers in a set are not random.
For instance, it could be that these numbers are generated by an equation, e.g. $(1+x^y)$ so that the form the following sequence:

[{1, 2, 3, 4, 5, 6}, {1, 2, 5, 10, 17, 26}, {1, 2, 9, 28, 65, 126}, ..., {1, 2, 1 + 2**n, 1 + 3**n, 1 + 4**n, 1 + 5**n}]

But this equation or any other rule governing the order is not known. We are given a large n of these sets. What would be the most appropriate way to predict this in Keras?


texts: When I use SHAP for classification problem, it shows an output that is not 0 or 1. How can I overcome this?

texts:
I'm using Pima Indians Diabetes Database(https://www.kaggle.com/uciml/pima-indians-diabetes-database). I made predictions using XGboost and I'm trying to analyze the features using SHAP.

However when I use force_plot with just one training example(a 1x8 vector) it shows that my output is -2.02. This is a classification problem, I shouldn't be seeing such a value. I'm new in SHAP and I don't know what the problem is.

Here is my code:

import numpy as np
import xgboost as xgb
import sklearn as skl
import shap

dataset=np.loadtxt("diabetes.csv", delimiter=",")
X=dataset[:,0:8]
Y=dataset[:,8]
seed=7
test_size=0.33
X_train, X_test, y_train, y_test=skl.model_selection.train_test_split(X, Y, test_size=test_size, random_state=seed)

shap.initjs()
model=xgb.XGBClassifier()
model.fit(X_train, y_train)

predictions=model.predict(X_test)
accuracy=skl.metrics.accuracy_score(y_test, predictions)
print(accuracy*100)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_train)
shap.force_plot(explainer.expected_value, shap_values[0,:].reshape(1, 8), X_train[0,:].reshape(1, 8))

Accuracy of my model is: 77,95.


texts: Negative value in information gain calculation through gini index

texts:
I am trying to  determine the root node for the decision tree on given data

annual income target variable has been renamed as low, mid, and high.

I am using gini index to measure the impurity of my nodes.

The process I am following is simple:

1- calculate the Gini index for the dataset(target is annual income)

gini(annual income)=1-((5/20)^2+(12/20)^2+(3/20)^2) = 0.445

2 - for each variable calculate gini and then remainder and information gain

3 - choose variable with the highest information gain

for remainder i am using this

just instead of entropy, I am using gini

when I am trying to calculate information gain if education becomes root note I am getting a negative information gain (which is obviously not possible)

MY CALCULATION:

as you can see I got a gini index of 0.532 for the node if I do

Information gain (0.445-0.532)=-ve value

can you point towards what am I doing wrong


texts:
in my company we are working on a upset project in which we are trying to solve the following problem:

What we propose to our customer that he/she may be interested in based on the fact that he/she already bought A,B,C.

We have sales records but we also have many products.

We do not have many customer information (at least not in the dataset I'm working with) so as I first step I put all our sales records in a simple format:

CUST_ID A B C D E F G H I

Where each column is 1 if the customer has bought the item or not (One-Hot_Encoding)

mlxtend.frequent_patterns with Apriori and association_rules but I'm not getting good results.

I get so many rules that are quite useless at the moment.

I would love to predict, given a CUST_ID and the product he/she bought, the probability that he/she would buy the missing product.

Can someone point me in some good direction? Feature Extraction? The initial idea was to build a recommender system but I'm a bit lost at the moment...


texts:
I'm new to datascience so please just don't blast me.

In a text book i found:

  Now, the reason we don't initialize the weights to zero is that the
  learning rate (eta) only has an effect on the classification outcome
  if the weights are initialized to non-zero values. If all the weights
  are initialized to zero, the learning rate parameter eta affects only
  the scale of the weight vector, not the direction.

Now, why? Are you able to explain me?


texts: Conditional Entropy and Mutual Information - Clustering evaluation

texts:
First of all, I am doing clustering and I have the true labels for my data.
For evaluation, I am using the weighted average of the entropy values for each predicted cluster. I also came across with Mutual Information as a similar approach while going over the alternatives. On my data, they seem to give similar results.

However there is one issue that puzzles me.

Given the predicted cluster set $U$ and true clusters $V$, mutual information was defined as:
$$ I(U,V) = H(U) - H(U|V) $$
or,
$$ I(U,V) = H(V) - H(V|U)  $$
If my math is correct, the average entropy that I'm using corresponds to conditional entropy term $H(V|U)$ and trying to minimize this aligns with maximizing the mutual information.

What I cannot see is how weigthed average entropy would differ from mutual information and why we would need the entropy terms $H(U)$ or $H(V)$. It feels like minimising one of the conditional entropies should suffice.

To put it another way, as far as I understood from the equations, having high entropy for true or predicted clusters in itself also results in higher mutual information. Does this mean that mutual information favors equally-sized clusters?

Thanks in advance.


texts:
I'm looking for something like K-Means for dividing solid polygons into regions. K-Means clusters discrete points. But I want to cluster (that is, partition) the points of solid polygons.

I don't see any problems with implementing K-Means extension for this case but I want be sure before reinventing the wheel.

So questions:

Are there any algorithms for solid polygon clustering?
Are there any implementations (preferably in javascript)?

I looked at GIS clustering but everything I found is about polygon clustering for zooming. And a polygon is a convex hull of geo markers and deeply inside it is about clustering discrete points.


texts: I am making a classification model. I understand that the regularization minimizes loss functions. Can i use regularization techniques to minimize loss function in classification if doing so is possible. Please help me with some suggestions or useful links.


texts: How to decide power of independent variables in case of non-linear polynomial regression?

texts:
Consider one dependent variable 'Y' and 10 independent variables or features- X1, X2, X3, ... X10.

I want to create a non-linear polynomial regression model such that-

Y ~ a1.X1^b1 + a2.X2^b2 + .... + a10.X10^b10

I was wondering is there any algorithm that will determine best possible values for powers of independent variables that is values of b1, b2, ... b10 from data.


texts: Got some troubles with using OneHotEncoder to multiple categories

texts:
I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book).
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer

num_pipeline = Pipeline([
    ('selector', DataFrameSelector(list(df_num))),
    ('imputer',SimpleImputer(strategy='median', fill_value='num',missing_values=np.nan)),
    ('std_scaler',StandardScaler())
])
cat_pipeline = Pipeline([
    ('selector', DataFrameSelector(list(df_cat))),
    ('imputer',SimpleImputer(strategy='most_frequent', fill_value='categorical',missing_values=np.nan)),
    ('cat_encoder', OneHotEncoder(sparse=False)),
])

from sklearn.pipeline import FeatureUnion

full_pipeline = FeatureUnion(transformer_list=[
        ("num_pipeline", num_pipeline),
        ("cat_pipeline", cat_pipeline),
    ])

df_prepared = full_pipeline.fit_transform(df)
df_prepared.shape
df_total = pd.DataFrame(df_prepared, columns=df.columns)
df_total

Where
df_num = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']
df_cat = ['Sex', 'Embarked']

The problem is that I got the ValueError:
ValueError: Shape of passed values is (668, 10), indices imply (668, 7)

I tried dropping  'Embarked'  from the df_cat
and it created the DF somehow(but it was inaccurate). I assume it didn't work because of 'Embarked'
had 3 categories(10-3 = 7 - that's the number of columns I wanted.)
If I drop 'Sex'
category, I get ValueError: Shape of passed values is (668, 8), indices imply (668, 7) (10-2=8)
How to encode all two categories using Pipeline()
and which wrong steps I've made?


texts: How to get predicted class labels in convolution neural network?

texts:
I have built a convolutional neural network which is needed to classify the test data into either 0 or 1. I am training the CNN with labels either 0 or 1 but while running the below code I am getting the following result.

predictions = classifier.predict(x_test)

print(predictions)

[0.0128037 ]
 [0.01182843]
 [0.01042355]
 [0.00906552]
 [0.00820154]
 [0.00726516]......

logloss_score = log_loss(y_test, predictions)
print(logloss_score)
0.047878393431377855

How do I get the results between 0 and 1? What do I need to modify in the above code?


texts: Confused about the different aspects in Machine Learning

texts:
After reading different articles about ML and algorithms, scientist tends to use different words when describing the different aspects in ML.

So now I'm a bit confused myself and I hope you can correct me if I'm wrong.

1) So to my understanding supervised/unsupervised learning are different categories of machine learning algorithms. Each category contains different algorithms such as Neural Networks and Bayesian?

2) Regression, Classification and Clustering are types of models?

3) A model is the result of a trained algorithm?

I hope that I'm not completely wrong, thanks! :)


texts: Does the performance of GBM methods profit from feature scaling?

texts:
I know that feature scaling is an important pre-processing step for creating artificial neural network models.
But what about Gradient Boosting Machines, such as LightGBM, XGBoost or CatBoost? Does their performance profit from feature scaling? If so, why and how?


texts: If I have a dataset with binary classification and has many attributes with value of  (0 or 1) means the occurrence of attribute is represented by 1 and absence is represented by 0, can I add weight of that attributes as separate column in dataset before applying classifier model?


texts:
I am trying to build an MLP classifier model on a dataset containing 30000 samples and 23 features.
What are the standards I need to consider while selecting the number of hidden layers and number of nodes in each hidden layer?


texts: Evaluating information extraction from structured documents

texts:
I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that:

{
    "company": "STARBUCKS STORE #10208",
    "date": "14/03/2015",
    "address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
    "total": "4.95",
}

Formally, I defined the task as a function f that for a given invoice document D returns a dictionary of entries: {field1:value1,field2:value2,...}.

Initially, I thought about using precision, recall and F1 measures, assuming that each dictionary entry is marked as correct if both a field and its value matches exactly the ground truth. I borrowed this approach from ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction.

However, the problem with metrics based on exact string match is that they penalize even small errors: missing/redundant words, single differences in characters and spacing. Those errors may stem not only from the imperfect information extraction algorithm but also from faulty OCR algorithm. Unfortunately, I am not able to test information extraction in isolation (i.e., given 100% perfectly OCRed input).

Therefore, I am looking for some more relaxed metrics. What are my possibilities? What has been used in this field?

Some ideas that come to my mind:

calculate distance between two texts using Levenstein distance on character/word
consider two texts matching if distance is smaller than threshold X.
or include information about the distance in the final measures, e.g. normalize it and micro-average over each document.


texts: How to set class_weight parameter for cost sensitive learning?

texts:
I'm dealing with a binary classification problem with a balanced data set, however false positives are much more costly than false negatives. Let's just say that an FP is in general 3 times more costly than an FN and the response variable = 1 means a positive identification. How should I set the class_weight parameter in sklearn's RandomForest to reflect this ? From my understanding, I would say:
class_weight = {0:1.0,1:3.0}

I'm not sure if I understand this parameter correctly or should it be the inverse ?
Thanks.


texts: Can hadoop with Spark be configured with 1GB RAM

texts:
I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS.
I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.

As I'm a student, I cannot afford much. Can anyone please suggest me some solution?

Also, it would be great if you could provide any links for setting up multi cluster hadoop with spark on AWS.

Note: I cannot try in GCE as my trial period is exhausted.


texts: How to read contents of a CSV file inside zip file using spark (python)

texts:
I have multiple zip files containing two types of files(A.csv & B.csv)

/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv

I want to read the contents of all the A.csv files inside all the zip files using pyspark.

 textFile = sc.textFile("hdfs://<HDFS loc>/data/*.zip")

Can someone tell me how to get the contents of A.csv files into an RDD?


texts: Using text classification for system calls

texts:
I'm working on a project in which I should classify System calls sequences, my dataset is represented as sequences of integers (from 1 to 340). To do the classification I have inspired from Text classification projects. I'm trying to use on of them but I found a problem in my dataset shape, the code is:

df = pd.read_csv("data.txt")
#df_test = pd.read_csv("validation.txt")

#split arrays into train and test data (cross validation)
train_text, test_text, train_y, test_y = train_test_split(df,df,test_size =
0.2)

#train_text, train_y = (df,df)
#test_text, test_y = (df_test, df_test)
MAX_NB_WORDS = 5700

texts_train = train_text.astype(str)
texts_test = test_text.astype(str)

tokenizer = Tokenizer(nb_words=MAX_NB_WORDS, char_level=False)
tokenizer.fit_on_texts(texts_train)
sequences = tokenizer.texts_to_sequences(texts_train)
sequences_test = tokenizer.texts_to_sequences(texts_test)

word_index = tokenizer.word_index
#print('Found %s unique tokens.' % len(word_index))
type(tokenizer.word_index), len(tokenizer.word_index)
index_to_word = dict((i, w) for w, i in tokenizer.word_index.items())
" ".join([index_to_word[i] for i in sequences[0]])

seq_lens = [len(s) for s in sequences]
#print("average length: %0.1f" % np.mean(seq_lens))
#print("max length: %d" % max(seq_lens))

MAX_SEQUENCE_LENGTH = 100
# pad sequences with 0s
x_train = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)  # former des
sequence de meme taille 150, en ajoutant des 0
x_test = pad_sequences(sequences_test, maxlen=MAX_SEQUENCE_LENGTH)
#print('Shape of data train:', x_train.shape)  #it gives (1,100)
#print('Shape of data test tensor:', x_test.shape)
y_train = train_y
y_test = test_y
#if np.any(y_train):
  #y_train = to_categorical(y_train)
print('Shape of label tensor:', y_train.shape)
EMBEDDING_DIM = 50
N_CLASSES = 2
# input: a sequence of MAX_SEQUENCE_LENGTH integers
sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='float32')

embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM,
                        input_length=MAX_SEQUENCE_LENGTH,
                        trainable=True)
embedded_sequences = embedding_layer(sequence_input)

average = GlobalAveragePooling1D()(embedded_sequences)
predictions = Dense(N_CLASSES, activation='softmax')(average)

   model = Model(sequence_input, predictions)
model.compile(loss='categorical_crossentropy',
          optimizer='adam', metrics=['acc'])
model.fit(x_train, y_train, validation_split=0.1,
      nb_epoch=10, batch_size=100)
output_test = model.predict(x_test)
print("test auc:", roc_auc_score(y_test,output_test[:,1]))

I got the error: ValueError Error when checking target expected dense_1 to have shape(2,), but got array with shape(1,)

Any suggestion, cause I don't know how to proceed .
Thank you


texts: Should a model be re-trained if new observations are available?

texts:
So, I have not been able to find any literature on this subject but it seems like something worth giving a thought:

What are the best practices in model training and optimization if new observations are available?
Is there any way to determine the period/frequency of re-training a model before the predictions begin to degrade?
Is it over-fitting if the parameters are re-optimised for the aggregated data?

Note that the learning may not necessarily be online. One may wish to upgrade an existing model after observing significant variance in more recent predictions.


texts: How do I use keras NN to classify data after training?

texts:
I have defined, trained and saved my tensor keras NN. Now that that is complete how do I use it output classifications to non training data?

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from syslog import syslog_pred

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(128, activation='relu'))
# Add another:
model.add(layers.Dense(128, activation='relu'))
# Add a softmax layer with 8 output units:
model.add(layers.Dense(8, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.load_weights('.my_model')

x = np.array([arr[:-1] for arr in syslog_pred], dtype=np.float32)
dataset = tf.data.Dataset.from_tensor_slices(x)

answer = model.predict(dataset, steps=30)
print(answer)

The code at the end isn't what it should be but I'm a little lost.
Any help would be appreciated!


texts:
I'm studying the training algorithm for the self-organizing maps.
The slides i'm studying says that : after i've found a winner neuron ( in reference to a specific example in the example set), i have to apply this formula to its, and its neighbours.

where $d_{t}$ is the specific example, $m_{ij}$ the neuron ij.
What i've not understood is the meaning of the dist function in the radius . And what this radius means and what is its purpose. Can you help me ?


texts:
We have a dataset where in each attribute class, the values are distributed uniformly something similar to the following image where safety is one attribute of the dataset:

If all other attributes have the same situation then how does SVM classifier perform on this dataset? Which kernel should be used with it?


texts:
What is the difference? Under which criteria should each type of boost be used?

What is the theory behind each of these methods?


texts: Which method should be considered to evaluate the imbalanced multi-class classification?

texts:
I am working on multiclass-imbalanced data. My dependent variable is highly skewed.

           Injury

2(No Injury)         208753
1(Medium Injury)      22318
0(severe Injury)       3394

I have used random forest algorithm with parameter "class_weight='balanced' " to manage the class 2 imbalance.

I get the below results when I use average='micro'.

 [[   34   107   688]
  [  148   778  4592]
  [  905  4635 46730]]
 Accuracy Score: 0.8110616374089428
 precision score: 0.8110616374089428
 Recall score: 0.8110616374089428
 AUC Score: 0.8582962280567071
 F1 score: 0.8110616374089428
 Kappa Score: 0.05522284663052324

For the average = 'macro', the results are below.

[[   31   125   684]
 [  157   838  4559]
 [  890  4694 46639]]
 Accuracy Score: 0.8104816009007626
 precision score: 0.3586119227436326
 Recall score: 0.3602869806251181
 AUC Score: 0.5253225798824679
 F1 score: 0.3592735337079687
 Kappa Score: 0.06376296115668922

So, which results should I consider to evaluate the model? If I have to consider the macro, then my model performance is really bad. Please suggest if there are any methods to improve the precision, recall and AUC score?

If I consider micro results, my precision, recall, f1 score is same. How can I justify this in the project?

I am interested in medium and severe injuries. But individual class precisions 0.0312(severe), 0.1409(medium) for macro are very low. The overall precision score is also very low 0.35. Is it possible to increase these scores? Or Is it fine to consider these low values as my final results for the project? Any suggestions are greatly appreciated. NOTE: I have tried SMOTE oversampling and ensemble cross-validation with different algorithms but I ended up having precision and recall scores less than 50.

Any help would be appreciated.

Thank you.


texts: How to build a classifier that by default will predict that it is for class 1, but if the classifier believes with 80 certainity that it belongs to 0, it will be classed as 0. How to check how certain a classifier is on it's prediction.


texts: All supervised learning techniques have some kind of an inductive bias. What is the inductive bias in Gaussian process models ?


texts: Is there any feature selection method specific for regression analysis?

texts:
Is there any feature selection method that works especially well for regressions?
I used backwards elimination and forward selection before a lot but I've recently read that even though it's historical, it's not correct... but maybe a variance threshold? Correlation threshold? Chi square? I've been searching but everywhere I look I find a new way of doing it... so I don't get a definitive answer
Some context: I'm running a multinomial logit, with 3 classes on the y, and the regressors are a mix of demographics (age, sex, years at job, income, mortgage, debt to income ratio, etc), loan specific variables (loan amount, loan term, APR, rep APR) and some "time" variables (amount of days that client spent in certain status, how many times client entered the status). My objective is to predict.


texts: How to Write Multiple Data Frames in an Excel Sheet

texts:
I have multiple data frames with same column names. I want to write them together to an excel sheet stacked vertically on top of each other. And between each, there will be a text occupying a row. This is what I have in mind.

I tried the pandas.ExcelWriter() method, but each dataframe overwrites the previous frame in the sheet, instead of appending.

Note that, I still need multiple sheets for different dataframe, but also multiple dataframes on each sheet. Is it possible? Or any other python library which can dynamically generate the excel sheet from pandas dataframes?


texts: Can I modify a Logistic Regression classifier to out put more than one class based on the probabilty?

texts:
I'm training a Logistic Regression classifier on text data. I found that many of my data points have more than one target class. Is it possible to modify my model to output more than one class based on the data.
I plan to split multi-class data points in my training set into distinct classes(i.e. if one x has 3 classes, I will split that text into three so that each different text has a unique class associated with it). Then when I predict on the test data if I will output n classes such that

Probability(Class_1)+Probabilty(Class_2)+...+Probabilty(Class_n)>0.95

I will use the prdict_proba method of LogisticRegression for this.
Is it a correct way of doing this?. your help is much appreciated.


texts: variance threshold, returning the names of the selected features

texts:
I am trying the variance threshold method for the first time and I am following the example in sklearn to work on it.
>>> X = [[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]]
>>> selector = VarianceThreshold()
>>> selector.fit_transform(X)
array([[2, 0],
       [1, 4],
       [1, 1]])

However, at the end, it only returns an array of the values of the selected features. How can I get the name of the features that met the variance threshold limit here? I just need the names of the selected features.


texts: How much time do I need for Hyperband tuning?

texts:
I'm using the Hyperband algorithm (Paper, Keras Docs) to tune my model. I've read the paper, but I still don't understand one simple thing.

Is there a simple formula to estimate the time the tuner will take to train? I have the values for max_epochs, factor, the number of configurations (using only hp.Choice right now) and the approx. duration of one epoch (based on observations).

This shouldn't be hard, but the main point is that I get confused with the terms (e.g. paper uses budget, max iterations, brackets; keras docs use max_epochs, hyperband_iterations).


texts: How to evaluate distance in k-means clusters?

texts:
I try to use k-means clusters (using SQLserver + R), and I was wondering how we could estimate a distance correctly.

For instance, if we consider Euclidean distance form the center of the clusters, what happen if we have, for the same dataset, clusters of different sizes ? a "normal" point in a big cluster will have a distance higher than an "outlier" point in a little one.

So:

Is it relevant to center / scale euclidean distance on each cluster ? (and then consider outliers as the ones with the highest scaled distance)
Are there other kind of distance to consider ?


texts: Defining Input Shape for Time Series using LSTM in Keras

texts:
I have been trying to model Time Series forecast using Keras LSTM algorithm. My dataset consists of weekly sales data from Jan-2016 and I also have external features such as Festivals/Events each modeled as a flag (0/1) for each week. So for e.g. let's say 2016W01 has sales of 100 units and has flag value 1 for the NewYearFlag. Likewise, I have created flags for many such events such as Independence Day, Diwali (this is in a different week every year) and so on.

Based on the Time Series decomposition analysis,I also know that 5 week lags are signficant. So it is Week-1, Week-2, Week-3,Week-4 and Week-5 variables.

Essentially,my dataframe row looks like
Week Number (Row Index), SalesQty, SalesQtyW-1,SalesQtyW-2, SalesQtyW-3,SalesQtyW-4,SalesQtyW-5,NewYearFlag,DiwaliFlag, IndepdenceDayFlag, ChristmasFlag.

So I have X = {SalesQtyW-1,SalesQtyW-2, SalesQtyW-3,SalesQtyW-4,SalesQtyW-5,NewYearFlag,DiwaliFlag, IndepdenceDayFlag, ChristmasFlag}
Y = {SalesQty}

So when I pass my dataframe to Keras LSTM layer, I pass (144,9) as there 144 rows (52+52+40) in my training data.

I always get the error - Expected 3 dimensions but got only two (144,9)

I am not sure where I am going wrong. Please help. I checked most of the questions of similar nature but not able to find the right answer. Dont know where am I going wrong!!!


texts:
I'm studying a paper about Named Entity Recognition. The following is a part of the abstract:

To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific
sources of errors: unknown words and label shift or ambiguity.

I don't know what "label shift" definition is. The paper doesn't explain it and I can not find anything that I could understand by Googling it.