texts: Pretrained Models for Keyword-Based Text Generation

texts:
I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2).
An example would be gpt-2-keyword-generation (click here for demo). As the author notes, there is

[...] no explicit mathematical/theoetical basis behind the keywords
aside from the typical debiasing of the text [...]

Hence my question: Are there more sophisticated ways of keyword-based text generation or at least any other alternatives?
Thank you


texts: Transition from clustering to classification?

texts:
To date, I have done several ad-hoc text clustering projects which use combinations of topic modeling, k-means, and other algorithms. Basically, the point of these projects was to produce themes for different events based on associated text. The themes were named manually after the appropriate levels of clustering were determined, and are now stored in a csv in the following format:

event_id    majortheme    minortheme         majortheme_id    minortheme_id
12          Job Failure   TWS Issue          1                major1minor1
14          Job Failure   TWS Issue          1                major1minor1
15          Job Failure   Job Abend          1                major1minor2
16          Access Issue  Unable to Login    2                major2minor1
17          Access Issue  Unable to Connect  2                major2minor2

I want to transition from clustering to classification (from descriptive to prescriptive analytics), i.e. being able to take new events (with new event_ids) and classify them based on previous clusterings. This would be sort of an iterative training dataset, as new classifications would be added to previous clusterings, after verifying that the model hasn't gone completely awry. Using Python, what is the best approach to implement this sort of classification pipeline? Is it as simple as saving my initial clustering results and then just using that data as a training set going forward? Then saving the test predictions to the original training dataset, and so on?


texts: Deciding which samples the model will (probably) classify incorrectly

texts:
Problem:
Given a neural network for image classification, the objective is to develop an algorithm which decides which images are 'problematic' and the model is probably going to classify them incorrectly.
Discussion:
So far, I've thought of two possible approaches:

Feed the given image to the model and then analyse its softmax ouput with various metrics (difference between first and second class confidence, entropy, gini index etc).
Perform some kind of image processing (feature extraction) on the given image, to obtain some features that indicate whether the image is not going to be correctly classified.

Questions:
Can you provide me with more suggestions about the second approach? What type of feature extraction would you think will help distinguish those images?
Any other ideas that are not mentioned here are welcome.


texts: Absolute-value max pooling in 2D convolutional neural networks

texts:
i am fairly new to machine learning, so this may be a silly question. if that is the case, I apologise in advance.
i am training a convolutional neural network on oceanographic images, which include both positive and negative anomalies. The implementation i have tested employs a number of 2D-convolutional layers, followed by max-pooling.
Max-pooling naturally has a bias towards positive values. Does it mean that negative anomalies in my images are going to be given less wight in the training? or is the sign difference accounted for in the convolutional filters?
instinctively, i would like to implement some sort of 'absolute-value max pooling', but since i have found no mention of this in the literature, should i assume this is not needed?


texts: How do you detect seasonality(multiplicative or additive) in a time series data?

texts:
Trying to apply seasonal_decompose on timeseries data. It looks something like this:

            modal_price
Period
2014-11-01  1469
2015-01-01  1258
2015-03-01  1112
2015-04-01  1373
2015-06-01  1370
2015-07-01  1406
2015-08-01  1520
2015-09-01  1860
2015-10-01  1436
2015-11-01  1455

Using freq=11 and seasonalcomponents are all Nan. What should i do in order to detect the seasonality type(multiplicative/additive) of this data?


texts:
So I'm trying to implement Nvidia's end to end driving paper to simply have an agent in Carla follow the lanes. I'm trying to predict the steering angle of the car based on the RGB images from the front camera.

I'm getting and r_score of about 1.20 %

Every image in the dataset has a steering angle.

So here are plots for the distribution of steering angles.

I've also augmented the images but when I run it, the car still fails to follow the lane.

I'm being wrong or should I take another approach? Thanks :)!


texts: Simple neural network implementation in keras

texts:
I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.

def create_baseline():
    model = Sequential()
    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
scores = cross_val_score(pipeline, X, y, cv=kfold)
print("%.2f%%" % (scores[1]*100))

I am using around 5871 malicious samples and 3488 benign samples.

Just to cross check I have implemented without cross validation, a very simple implementation, and I am getting ~99% accuracy in that also.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)

scale = StandardScaler(with_mean=0, with_std=1)
new_X_train = scale.fit_transform(X_train)
new_X_test = scale.transform(X_test)

model = Sequential()

model.add(Dense(23, input_dim=46, kernel_initializer="normal"))
model.add(PReLU(alpha_initializer='zero', weights=None))

model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)

scores = model.evaluate(new_X_test, y_test)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

print(history.history.keys())

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

The plots I am getting are:


texts: Separate code for neural network in different files

texts: I just finished my 300 lines program in which I trained neural network in Keras for pedestrian detection. I would like to separate it in the project but I don't know how to do that. What is the best way to separate the code for Machine learning algorithm in different files so that somebody who see my code can see every file not too big?


texts: Lowering learning rate makes my accuracy on the validation set go down

texts:
I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase the accuracy of the model because all it does is make the step smaller, so if anything it's less prone to over shooting.

So maybe I don't know the learning rate as well as I thought.

Why does this happen?

PS. Is there a good way to figure out the learning rate and other parameters like n_estimators (in this case for XGBoost) and so on other than trial and error?


texts: How to feed tuples of multiple features to the input layer of a Neural Network

texts: I have a dataset for which every object has multiple features, and some of those features exist in tuples, like as in list of $(x, y)$ coordinates for each object. I know I can create an input layer of number of neurons equal to number of features of each object, but how do I pass these 'tuple' sets to the input. Do I break them and append to create a longer list of features, but will that not lead to loss in some correlation between these 2 sets of points from tuples?


texts: Why does TensorFlow convert my decoded image to float32 instead of uint8/16?

texts:
I am using the following code to import a bunch of .png images and decode them using TensorFlow:

from __future__ import absolute_import, division, print_function
import tensorflow as tf
import numpy as np
import os

tf.enable_eager_execution()

NUM_TRAINING_SAMPLES = 333
NUM_CLASSES = 3
BATCH_SIZE = 5
NUM_EPOCHS = 6
INPUT_SIZE = (256, 256, 3)

random_indices = np.random.choice(range(13000), NUM_TRAINING_SAMPLES)
directory = "/home/local/CYCLOMEDIA001/ebos/Downloads/SYNTHIA_RAND_CVPR16"
directory_images = "/home/Downloads/SYNTHIA_RAND_CVPR16/RGB"
directory_labels = "/home/Downloads/SYNTHIA_RAND_CVPR16/GT"
train_images = np.array(os.listdir(directory_images))
train_labels = np.array(os.listdir(directory_images))
train_images = train_images[random_indices]
train_labels = train_labels[random_indices]
train_images = [tf.read_file(os.path.join(directory_images, img)) for img in train_images]
train_labels = [tf.read_file(os.path.join(directory_labels, img)) for img in train_labels]
train_images = [tf.io.decode_image(img, channels=3) for img in train_images]
train_labels = [tf.io.decode_image(img, channels=3) for img in train_labels]
train_images = tf.image.resize_images(train_images, INPUT_SIZE[:2])
train_labels = tf.image.resize_images(train_labels, INPUT_SIZE[:2])

train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
train_dataset = train_dataset.batch(3)
print(train_dataset.output_types)

This returns:

(tf.float32, tf.float32)

However, according to the documentation it should return a tensor of uint8's or uint16's. Why and where does the conversion take place?

I checked all intermediate steps with print statements, which doesn't tell me much as most intermediate lists are of class 'tensorflow.python.framework.ops.EagerTensor'.


texts: Can RNN be replaced with non-recurrent classifier for Sequence Classification problem?

texts:
Setup:
We have sequence of events that are not evenly spaced (not a time series). Length of the sequence is constant.
Goal:
Predict class of the event that is most probable to follow this sequence.
Background:
I know that RNN would probably a good fit for this task, but at the same time I wonder whether parameters sharing in our U,W,V matrices actually hurt accuracy ( even though training process is cheaper). Let's say we are ok to spent more time(and data) for training and don't want to compromise accuracy.
Question:
Is it true that by using regular MLP we can achieve better or at least same performance if we just combine/flatten all features from those sequence events and pass them alltogether as an input? I believe model should still be able to learn interactions between features(that represent different events in a sequence) but not sure how good it will be at it and if not then why?


texts: What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?

texts:
The tag feature-scaling seems to convey that one of the scaling methods is Standard Normal Distribution. Further, I read an Answer on this site saying that Mean Normalization is a form of feature scaling.
What is the difference between two approaches to scaling?
Note: I think that statistics and mathematics of normalization do differ.


texts: how to classify text based on more than one column

texts:
I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled "En" below). But I feel that using the title of each passage (column labeled "Ref" below) would help.
    Ref               En                                                Topics
3   Gittin            modifier meaning board referred unique name mo... dinei-haget
11  Even HaEzer       hand katafres explanation hand slanted obvious... dinei-haget
67  Rest on Holiday   similar two baskets untithed fruit front first... laws-of-holidays
118 Beitzah           mishna states one ate food prepared festival e... laws-of-holidays
131 Sabbath           one may mix water salt oil dip one bread put c... rabbinically-forbidden-activities-on-shabbat

One excellent idea, which @Erwan mentioned in the comments, is to simply include the title together with the text of the passage. But I have two issues with that:

How do I do that?
Don't I want to make the title (or "Ref") count for more weight than the other words in the passage?


texts: What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

texts:
I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation.

I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) grouping them considering the number of events occurred.

My concern is that a lot of the values are 0 simple because some diseases didn't show that particularly symptom, for example (I made up the values for now):

So, I was wondering what was the best way to cluster this dataset. I was looking for and found hierarchical and kmeans, but don't know if i can properly apply to this cenario.
First of all, I switched the absolute values of occurrences to % of total, does that make possible to deal with the 0's ? I thought about it but at the same time 1% is close to 0%, I don't know if the algorithm can also understand as 'flags', since 1% represents that actually that symptom occurr(even at lower rates) and in another disease don't occurr at all.

I heard about PCA to reduce number of variables, and I was also curious if:
1-PCA is applicable to this cenario (dataset with a lot of sparse 0's)
2-PCA could solve my problem(because I think that even when a reduce to 2 3 variables, some rows could still be 0 for that particular column(symptom).

Any help/guidance would be extremely helpful and I thanks all in advance,
Sorry for some english error as well!

Have a great week!


texts:
I am trying to create the following metric for my neural network using keras

$$
s = \left\{
        \begin{array}{ll}
            \sum_{i=1}^{n} e^{\frac{-d_i}{10}}-1 & \quad d < 0 \\
            \sum_{i=1}^{n} e^{\frac{d_i}{13}}-1 & \quad d \geq 0
        \end{array}
    \right.
$$
where $d_i=y_{pred}-y_{true}$

and both $y_{pred}$ and $y_{true}$ are vectors

With the following code:

import keras.backend as K

    def score(y_true, y_pred):
            d=(y_pred - y_true)
            if d<0:
                return K.exp(-d/10)-1
            else:
                return K.exp(d/13)-1

For the use of compiling my model:

model.compile(loss='mse', optimizer='adam', metrics=[score])

I received the following error code and I have not been able to correct the issue. Any help would be appreciated.

  raise TypeError("Using a tf.Tensor as a Python bool is not
  allowed. " "Use if t is not None: instead of if t: to test if a "
  "tensor is defined, and use TensorFlow ops such as "

  TypeError: Using a tf.Tensor as a Python bool is not allowed. Use
  if t is not None: instead of if t: to test if a tensor is defined,
  and use TensorFlow ops such as tf.cond to execute subgraphs
  conditioned on the value of a tensor.


texts: What happens when scikit-learn does a Lasso Model?

texts:
I have started an MLS course. As a beginner and non-mathematician it has been hard.

I am trying to understand the exercise about Lasso Models.
I have done Lasso models on R-cran, but this is my first time with Python.

I have a dataset of cars with seven variables. The exercise consists on doing a Lasso Model to predict the gasoline consumption of the cars, the dependent variable, so $x$ is a table with the rest of variables and $y$ is the consumption.

Then, if I launch Lasso on scikitlearn:

modelLasso = Lasso(alpha=0.1).fit(x, y)

I do not understand the result. Has it generated a prediction on $y$ of every row in the table?
If so, how can I access to the array of predictions of the model and how do I use the model to predict the consumption given new $x$ values?


texts: Is there a link between Training, Test errors based on k fold CV and not doing CV?

texts:
I am using Matlab to train a feedforward NN using Cross validation (CV) approach. My understanding of CV approach is the following. (Please correct me where wrong)

Let X be the entire dataset with Y as the label set. Split X into 90/10 ratio to get: [Xtrain,Xtest] using holdout approach by calling the cvpartition(Y,'Holdout',0.1,'Stratify',true)
Apply CV on Xtrain: For every fold I calculated the accuracy. At the end of the CV loop I have an average accuracy score. Let accCV denote this variable. Inside the CV loop xtrain is further split into [xtrain_cv,xtrain_val].
After CV loop, I reinitialze the weights and re-train a new model using Xtrain. Then I get a training accuracy which I denote by the variable accTrain.
Using the model obtained in Step3 I test for evaluating the model's purpose and consider this to be the generalization performance that is the performance when we have an unseen future data, Xtest. I call this accuracy as accTest.

Question1: Is it possible that accCV will be less than the accuracy over the train set Xtrain when not using the CV approach? That is I call the NN model over Xtrain only once and record the accuracy and denote it by variable accTrain, then is it possible that accCV ~ accTrain?. Or intuitively, accCV should be close to the accuracy when not using CV approach since the dataset is the same which is Xtrain. If this is the case, then why use CV when outside the CV we do not reuse the model that was created inside the CV? What does it tell us?

Question2: If accCV < accTest but the accuracy on the entire dataset Xtrain without using CV is close to that of accTest (accTrain ~ accTest) are we doing something wrong? What is the best case scenario? Is it accCV ~ accTest?


texts: What is the intuition behind using LSTM for classification tasks?

texts:
LSTM is good for sequence prediction, because it can remember the previous context. What is the rationale behind using it in classification tasks ? In particular, they have used it for the following task:

  The Large Movie Review Dataset (often referred to as the IMDB dataset)
  contains 25,000 highly-polar movie reviews (good or bad) for training
  and the same amount again for testing. The problem is to determine
  whether a given movie review has a positive or negative sentiment.

In other words, can the classification problem be reduced to sequence prediction problem in some way?


texts: Global average polling without fc layer, Vanishing gradient or other problem?

texts:
I have built this network with pytorch, it's basically a modification of VGG16, I add some layers and remove some. The output of the network are meant to be softmax scores for F.multilabel-soft-margin-loss in pytorch.

def __init__(self, bn=False):
super(WSDR, self).__init__()
# Network based on VGG16
# ConvReLU means a conv layer following a relu
self.conv1 = nn.Sequential(ConvReLU(3, 64, 3, pd=True, bn=bn),
                           ConvReLU(64, 64, 3, pd=True, bn=bn),
                           nn.MaxPool2d(2))
self.conv2 = nn.Sequential(ConvReLU(64, 128, 3, pd=True, bn=bn),
                           ConvReLU(128, 128, 3, pd=True, bn=bn),
                           nn.MaxPool2d(2))
self.conv3 = nn.Sequential(ConvReLU(128, 256, 3, pd=True, bn=bn),
                           ConvReLU(256, 256, 3, pd=True, bn=bn),
                           ConvReLU(256, 256, 3, pd=True, bn=bn),
                           nn.MaxPool2d(2))
self.conv4 = nn.Sequential(ConvReLU(256, 512, 3, pd=True, bn=bn),
                           ConvReLU(512, 512, 3, pd=True, bn=bn),
                           ConvReLU(512, 512, 3, pd=True, bn=bn),
                           nn.MaxPool2d(2))
self.conv5 = nn.Sequential(ConvReLU(512, 512, 3, pd=True, bn=bn),
                           ConvReLU(512, 512, 3, pd=True, bn=bn),
                           ConvReLU(512, 512, 3, pd=True, bn=bn))

self.gap = nn.Sequential(ConvReLU(512, 1024, 3, pd=True, bn=bn),
                         ConvReLU(1024, 20, 3, pd=True, bn=bn),
                         nn.AvgPool2d(kernel_size=14, stride=14))
# adding this fc layer, things go right
# self.fc = nn.Linear(self.num_classes, self.num_classes)

def forward(self, im_data):
    x = self.conv1(im_data)
    x = self.conv2(x)
    x = self.conv3(x)
    x = self.conv4(x)
    conv5features = self.conv5(x)
    gap = self.gap(conv5features)
    scores = gap.squeeze()
    # scores = self.fc(scores)
    return scores

But this network can not reduce loss correctly. The output of the network become all zeros after only 2 or 3 iterations(one forward and one backward means one iteration), so do the gradients in the network.

I think the problem may be caused by the last global average pooling layer, usually people add another fc layer after that but I didn't. I tried to add a fc, and the problem disappered, the loss start to reduce after adding fc layer.

My question is:

What caused the vanishing gradient and zero output(gap2 feature, scores in the above code all become zeros)?
Why adding a fc layer can solve that problem?
I really don't want the fc layer because I want every channel of the last conv feature to represent one class(for classification), how can I make things right without adding fc layer?

Please leave your advice, any hints would be helpful

Addtional codes:

class ConvReLU(nn.Module):
def __init__(self, in_ch, out_ch, kernel_sz, stride=1, relu=True, pd=True, bn=False):
    super(ConvReLU, self).__init__()
    padding = int((kernel_sz - 1) / 2) if pd else 0  # same spatial size by default
    self.conv = nn.Conv2d(in_ch, out_ch, kernel_sz, stride, padding=padding)
    self.bn = nn.BatchNorm2d(out_ch, eps=0.001, momentum=0, affine=True) if bn else None
    self.relu = nn.ReLU(inplace=True) if relu else None

def forward(self, x):
    x = self.conv(x)
    if self.bn is not None:
        x = self.bn(x)
    if self.relu is not None:
        x = self.relu(x)
    return x

# compute output
output = model(input_var)  # the above model, input_var:image data
loss = F.multilabel_soft_margin_loss(output, target_var)
# compute gradient and do SGD step
optimizer.zero_grad()
loss.backward()
optimizer.step()


texts: For text classification that has innumerable features, how do I choose the number of neurons and layers for MLPClassifier?

texts:
In my use case of text classification (identify the author from a subset of 10 authors), I find that post all processing with trigrams, there are a 100 thousand and odd features with nearly 50k observations/samples to be trained.

It leaves me baffled to decide on the  number of neurons and layers to configure for my MLPClassifier of scikit-learn. If I try a number like 250 neurons with just a single layer, my laptop hangs in about 5 minutes :(

Your advice based on your experience is highly appreciated.


texts:
We are currently developing a customer relationship management software for SME's. What I'd like to structure for our future CRM is developing CRM with a social-based approach (Social CRM). Therefore we will provide our users (SME's) to integrate their CRM into their social network accounts. Also CRM will be enhance intercorporate communication of owner company.
All these processes I've just indicated above will certainly generate lots of unstructured data.
I am wondering how can we integrate big data and data-mining contepts for our project; especially for the datas generated by social network? I am not the expert of these topics but I really want to start from somewhere.
Basic capabilities of CRM (Modules)
-Contacts: People who you have a business relationship.
-Accounts: Clients who you've done a business before.
-Leads: Accounts who are your potential customers.
-Oppurtunites: Any business opportunity for an account or a lead.
-Sales Orders
-Calendar
-Tasks
What kind of unstructured data or the ways (ideas) could be useful for the modules I've just wrote above? If you need more specific information please write in comments.


texts: Is there any book for modern optimization in Python?

texts: I was reading Modern Optimization with R (Use R!) and wondering if a book like this exists in Python too? To be precise something that covers stochastic gradient descent and other advanced optimization techniques. Many thanks!


texts: Deep Learning is an excellent model for classification problem such as image recognition or object detection. Can we use deep learning for regression problems - Time Series prediction ? So if it can, how can we build structure of deep learning. I mean how to building layer to extract features from time series.


texts:
Which is the best python library to extract the location from the text. Spacy and NLTK not recognizing the mountain names, river bodies properly.
Could you please give me your suggestions?
Thank you!


texts: K means library designed for large dataset in python

texts: I find a k means library for large dataset in R (https://www.r-bloggers.com/k-means-clustering-on-big-data/) , may I know is there any similar library in python?


texts: How to transform a folder of images in a csv file

texts:
I have a folder with a lot of images that I want to use to bild a classificator using a SVM model in python with sklearn.
I've always used csv file as train/test set with sklearn, how can I make it? (a csv file with records corrisponding to images and a variable for every pixel)


texts:
There are many ML techniques to estimate latent variables such as the EM algorithm. Is there a technique that allows for thresholds for each of the latent variables?

I have a feature space with 10 variables $(X_1,\dots,X_{10})$ and the outcome $Y$. 7 of the $X$ features are known (I have their observations) and 3 are unknown. Each of the unknown can be within a range from 0 up to a positive constant number.

What ML technique would you recommend for estimating the above latent variables with the setup described above?


texts: What is the difference in computational cost at inference time between object detection and semantic segmentation?

texts:
I am aware that YOLO (v1-5) is a real-time object detection model with moderately good overall prediction performance. I know that UNet and variants are efficient semantic segmentation models that are also fast and have good prediction performance.
I cannot find any resources comparing the inference speed differences between these two approaches. It seems to me that semantic segmentation is clearly a more difficult problem, to classify each pixel in an image, than object detection, drawing bounding boxes around objects in the image.
Does anyone have good resources for this comparison? Or a very good explanation to why one is computationally more demanding that the other?


texts: How to reconsider MSE metric when doing target engineering through a logarithm?

texts:
I am a user of Keras with Python, if I know some mathematics I am not very good about algebraic transformation.
The context is the following.
I am trying to reproduce the performances from a paper. In this paper, each individual has his own model: a neural network with the pattern 8-5-7-1. All these models reached a MSE of 100.
In my work, I do not use the same variables, so I found a different pattern.
I considered 3 individuals (3 models). I have remarked models (of the same pattern) learn better when the skewness of the target is lower. /!\ The target is not transformed at all at this moment.
In concrete terms:

Model 1: skewness 0.41 manage to learn until 100 MSE (reached the goal)
Model 2: skewness 0.66 manage to learn until 200 MSE not less (do not reached the goal)
Model 3: skewness 0.31 manage to learn until 80 MSE (beyond the reached goal)

So this is why I am thinking about making a target engineering by using the logarithm of any base (while it reduces the skewness it is ok). Nevertheless, it transforms the reached metric: 100 MSE as maximum threshold. And I am pretty sure I cannot just compute log(MSE).
I searched the formula, of course: $ MSE = \frac{1}{n}\sum_{i=1}^{n}(y_{i}-\bar{y}_i)^{2} $
But I am not able with my limited skills to reconsider it correctly. I need help.
So MSE is dependent of the n observations of the dataset, more the logarithm transformation on y. How can I implement a function that will allow to transform the new form of MSE obtained through a logarithmic transformation in order to compare it with the previous goal: 100 MSE on a non-transformed target ?


texts: Count the frequncy of words in a cell of a column in a series

texts:
I want to calculate the frequency of the words in obama['text'] (obama is the variable where i have stored this series element ) in a dictionary  and store it in another column . Without using Counter library , how do i do that . The data is in this format :

 URI                                      |              name                |  text
<http://dbpedia.org/resource/Barack_Obama>           Barack Obama               barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureateduring his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief

The output should be in the format in a new column obama['word count']:

{ 2009:4 , the :40 , chicago :10
and so on  }


texts: subrandomsapler dataloader pytorch lengh does not seems right

texts:
split = int(len(train_dataset) * 0.8)
    print(split)
    index_list = list(range(len(train_dataset)))
    train_idx, valid_idx = index_list[:split], index_list[split:]
    print(len(train_idx),len(valid_idx))
48000 12000

I got this 48000 and 12000 index for train_idx and valid_idx

Then I apply this number to the dataloader.

train_loader = DataLoader(train_dataset, batch_size=batch_size, sampler= SubsetRandomSampler(train_idx))
valid_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=SubsetRandomSampler(valid_idx))

  print(len(train_loader.dataset),len(valid_loader.dataset))
  60000 60000

However, the length does not seem to be right.


texts:
See5 / C5.0 is Data Mining Tools available from rulequest

I want to compile C50 for Linux, preferably for CentOS 6.x, but I am unable to compile. I have also tried on Ubuntu, but not success there as well.

I have downloaded C50.tgz from C5.0 Release 2.07 GPL Edition
After extracting when I run ./Makefile it gives below error on Ubuntu18

./Makefile: line 9: CC: command not found
./Makefile: line 10: CFLAGS: command not found
./Makefile: line 11: S: command not found
./Makefile: line 11: LFLAGS: command not found
./Makefile: line 12: SHELL: command not found
./Makefile: line 19: src: command not found
./Makefile: line 48: obj: command not found
./Makefile: line 59: all:: command not found
cat defns.i global.c c50.c construct.c formtree.c info.c discr.c contin.c subset.c prune.c p-thresh.c trees.c siftrules.c ruletree.c rules.c getdata.c implicitatt.c mcost.c confmat.c sort.c update.c attwinnow.c classify.c formrules.c getnames.c modelfiles.c utility.c xval.c\
        | egrep -v 'defns.i|extern.i' >c50gt.c
make: /bin/csh: Command not found
Makefile:75: recipe for target 'c5.0' failed
make: *** [c5.0] Error 127
./Makefile: line 61: CC: command not found
./Makefile: line 61: LFLAGS: command not found
./Makefile: line 61: -o: command not found
./Makefile: line 67: obj: command not found
./Makefile: line 67: c5.0dbg:: command not found
./Makefile: line 68: CC: command not found
./Makefile: line 68: obj: command not found
./Makefile: line 68: -g: command not found
./Makefile: line 74: src: command not found
./Makefile: line 74: c5.0:: command not found
./Makefile: line 75: src: command not found
./Makefile: line 77: CC: command not found
./Makefile: line 77: LFLAGS: command not found
./Makefile: line 77: -O3: command not found
strip: 'c5.0': No such file
./Makefile: line 82: obj: command not found
./Makefile: line 85: .c.o:: command not found
./Makefile: line 86: syntax error near unexpected token `newline'
./Makefile: line 86: `  $(CC) $(CFLAGS) -c $<'

If I run make command, it gives below error.

make c5.0
make: /bin/csh: Command not found Makefile:60: recipe for
target 'all' failed make: *** [all] Error 127

gcc version is:

gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0


texts: anaconda navigator not start direct without source command

texts:
when i run following command
 source ~/anaconda3/bin/activate root
it is work fine. I don't want to run this command every time. give me any solution.


texts: How do I go from number data to string data (NN)?

texts:
I am quite confident with using number data with a neural network, but I want to use string data. My question is where do I begin? Obviously you cant times weights and strings together (x * w) because they are of different data types, so what should I do to the string data to turn it into numbers? If you're interested, here is my dataset. It's by no means the final version but here it is just to give you an example. It's based on spelling:

enormous..........| Difficulty: 2
rhythm..............| Difficulty: 3
hamster.............| Difficulty: 2
walk...................| Difficulty: 1
accommodate...| Difficulty: 3
blue....................| Difficulty: 1
projector.............| Difficulty: 1
regression..........| Difficulty: 2
go.......................| Difficulty: 1
playwright........| Difficulty: 3
weird.................| Difficulty: 3
conscience.......| Difficulty: 2

...so after training, when I will input a word, the network should return  either 1, 2 or 3, depending on how hard the network thinks the word is to spell. To sum up, my question is: I am used to using number data with my projects instead of strings, so what steps should I take when creating a network based around strings for the first time? I use numpy only


texts:
What is the best way to perform custom parameter search CV with the Scikit-learn API? I really like GridSearchCV. However for my case the  param_grid parameter is inflexible because it will search over the entire span of parameter combinations. Ideally, I would like to provide my own parameter space in a dataframe, one column for each parameter.
Thus, is there an appropriate class within Scikit-learn to help me achieve this result?


texts: Large amount of Sigmoid outputs are ones and zeros

texts:
I have Keras neural network for binary classification with final layer having one output with Sigmoid activation. I have noticed that large amount of output numbers are strictly one or zero (rather than between 0 and 1 as expected). What could be the reason for this?

At first I thought maybe network is 100% sure about accuracy of these numbers, but noticed that some of these predictions are actually incorrect.

Edit:

Model:

model = tf.keras.models.Sequential()

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu, input_shape=(X_train.shape[1],)))

model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(1, activation = tf.nn.sigmoid))

model.compile(optimizer = tf.keras.optimizers.Adam(), loss = 'binary_crossentropy', metrics = ['accuracy', roc_auc])

model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test))

y_pred = model.predict(X_test)

yhat = []

for pred in y_pred:
    if pred >= 0.7:
        yhat.append(1)
    else:
        yhat.append(0)

Data consists of 8 columns (features) and 270 000 rows (9th column is 'y' column). Out of these 270,000 rows only 9% contained labels of class 1 (rest were zero), so I downsampled the data (just removed bunch of data with label 0), trained the model and then did the prediction on full data. I modified how ones and zeros were determined by Sigmoid though, I changed threshold from 0.5 to 0.7 (which was the ROC score I got on downsampled data)


texts:
I'm super stuck so any help would be amazing. I'm trying to get the date from Yahoo finance for ASX 200 Stocks. I have a data set with all the tickers under tibble.
The date on Yahoo Data is listed as 'Date' 04 May 2021 - but I keep getting NA.
My Code (and bold is where I think the issue is):
get_stock_prices <- function(code, return_format = "tibble", ...) {

  stock_prices_xts <- getSymbols(Symbols = code, auto.assign = FALSE, ...)

  names(stock_prices_xts) <- c("Open", "High", "Low", "Close", "Volume", "Adjusted")

  if (return_format == "tibble") {
    stock_prices <- stock_prices_xts %>%
      as_tibble() %>%
      rownames_to_column(var = "Date") %>%
      ***mutate("Date"=dmy("Date"))***
  } else {
    stock_prices <- stock_prices_xts
  }
  stock_prices
}

When I run a quick test:
"A2M.AX" %>%
  get_stock_prices(return_format = 'tibble')

I get:
Date        Open  High   Low Close  Volume Adjusted
   <date>     <dbl> <dbl> <dbl> <dbl>   <dbl>    <dbl>
 1 NA         0.555 0.595 0.53  0.565 4816294    0.565
 2 NA         0.575 0.58

I want to get rid of the NA and get the actual date.
Thanks in advance
(and if needed this is where I got the codes from:
asx200 <- read_html("https://en.wikipedia.org/wiki/S%26P/ASX_200") %>%
html_node("table.wikitable") %>%
html_table() %>%
select('Code', 'Company', 'Market Capitalisation') %>%
as_tibble() )


texts: How to know which binning mode to apply to target column

texts:
I am trying to prepare for DP-100 certification and I got some sample questions online. Here is one of the questions:

You are a data scientist using Azure Machine Learning Studio.
You need to normalize values to produce an output column into bins to predict a target column.
Solution: Apply a Quantiles binning mode with a PQuantile normalization.
Does the solution meet the goal?
A.Yes B.No
Correct Answer: B Section: [none] Explanation
Explanation/Reference: Explanation: Use the Entropy MDL binning mode which has a target column.
References: https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/group-data-into-bins

I am not really sure why this solution is incorrect. Can someone please tell in what cases we choose which mode?


texts: Best way to find dissimilarity in a 6x2 DataFrame?

texts:
I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists.

The goal is to find the most dissimilar features between two different playlist.

My question is what is the best way to identify the most dissimilar features between these two playlists?

I started off by getting all the tracks in each playlist and their respective features. I then computed the mean of each of the features.

Here is the DataFrame I ended up with. The data values are the means of all the tracks features to their respective playlist

                   playlist1  playlist2
                   --------------------
danceability      | 0.667509   0.592140
energy            | 0.598873   0.468020
acousticness      | 0.114511   0.398372
valence           | 0.376920   0.287250
instrumentalness  | 0.005238   0.227783
speechiness       | 0.243587   0.088612

I did some digging and found two common procedures:

1. Euclidean Distance

2. Cosine Similarity

I for some reason couldn't wrap my head around which one to use and proceeded to compute the absolute difference between each feature. Simple subtraction because that made sense to me intuitively. The feature with the greatest difference would be the 'most dissimilar'.

With this approach, I ended up using these results and concluded that 'energy' and 'acousticness' are the most dissimilar

                   playlist1    playlist2   absoluteDifference
                   ----------------------------------------------------
energy             |0.871310    0.468020    0.403290
acousticness       |0.041479    0.398372    0.356893
valence            |0.501890    0.287250    0.214640
instrumentalness   |0.049012    0.227783    0.178771
danceability       |0.531071    0.592140    0.061069
speechiness        |0.109587    0.088612    0.020975

Is my intuition correct/incorrect and when would we use the aforementioned techniques? Would any of those techniques be applicable in a situation such as this?

Eventually, I want to take the top two dissimilarities and make them my axis for KNN. My intuition is that I can identify the most dissimilar features of two playlists, I'll have a cleaner and more defined features of the playlist and can more accurately predict which song a playlist ought to belong to.


texts: Multicolinearity & accurate weights of predictors

texts:
Let’s suppose that the stock value of various companies is the target of my models.

I have some “internal” predictors e.g. yearly sales of each company, sum of salaries at each company etc.

I have some “external” predictors e.g. geographical position of each company (latitude & longitude), population in the area in which each company operates etc.

Therefore, each observation at my dataset is about the stock value of a company along with its internal and external predictors.

The purpose of my project is to understand how each of a company's internal predictors affect in a very specific way the stock value of the respective company.

In simpler words, I want to get some accurate weights for the internal predictors which show me how exactly they affect the stock value of the respective company.

However, because there is relatively high multicolinearity between some of the internal predictors as a result I am not really getting very accurate weights for each one of them.

There may also be between the internal and the external predictors but I do not consider this as a problem because I think that you should account for all external predictors when taking the weight of the internal predictors.

However, I am not sure that I have to put all the internal predictors at the same model together because for example I do not want the weight of the yearly sales of a company to be modified because of the presence of other internal predictors such as the sum of salaries at this company.

In this regard, I am starting to think that the best way to go is to have multiple different models where each one of them has to do with one internal predictors separately but at every case all the external predictors.

Does this make sense?

Do you have any better idea?

P.S.
I just found a post which is quite similar to my line of reasoning: https://www.researchgate.net/post/Is_building_separate_models_a_solution_to_multi-collinearity.


texts: Loss function for classifying when more than one output can be 1 at a time

texts:
My desired output is not 1-hot encoding, but
like a 10 D vector: [1, 0, 1, 0, 1, 0, 0, 1, 1, 1]
and the input is like the normal MNIST data set.

I want to use TensorFlow to build a model to learn this,
then which loss function should I choose?


texts:
I'm looking for a stemmer/lemmatizer for Polish language, preferably in Python. What would you recommend?
I have a list of ingredients in a recipe. Plural forms are inflected differently, depending on the counter, e.g.: for tomatoes

5 pomidorów
2 pomidory
1 pomidor

I want my parser to recognize all those ingredients as for one product, hence the need for stemming.


texts: Multilingual data handling for text summarisation

texts:
I have a dataset having text documents as each entry , the documents are of various ethical languages which are not specified. How should be my strategy if I want to summarise the content of the dataset.

For eg. Doc1 : "Ruby is a dog lover.Ruby loves her dogs so much. Ruby has a pet whose name is a dog"

Now summarising it is easy as "Ruby loves dog.Ruby has a dog"

But How should be the approach , when the dataset also has documents in french, chinese, hindi etc...

Some of the problems are: there are cases of code switch, and I don't know what are the languages in the dataset before hand.

What I am guessing at this is to may be change all the dataset to english language and then do the tasks? In case if converting to english is a good choice then which library would be better?


texts: What could be proper terms for a research direction in natural language processing to measure meaningfulness?

texts:
For some time, I did assessments to design metrics on how to recognize well-written and meaningful software requirements. Then I decided to work with Stack Overflow question posts because they are a large corpus and have also votes which indicate human perceived quality1.

Now I am not sure which proper data science/NLP terms would describe the methodology to sort out what is a meaningful text or not.

I've been looking over entropy and semiotics, also research specifically on the Stack Overflow corpus but I fail to find seminal papers dealing actually with defining and measuring meaningfulness of text.

What is meaningfulness?

According to Wiktionary and Wikipedia, meaningfulness is "the state or measure of being meaningful", while meaningful is "having meaning, significant", while meaning is "the information or concepts that a sender intends to convey, or does convey, in communication with a receiver".
I've also asked a question on Linguistics SE to find out whether there are recent results from academic reseach on that and more up-to-date definitions.

Stack Overflow folks would have already implemented some practical entropy detection algorithms to filter out low quality posts (as we know it by the system filter, until you have typed in more content) but for example the following quite meaningless question produces no warning from the - that is, entropy i.e. randomness  appears low enough, but indeed the text is indeed very random and meaningless.

I do understand there is no absolute "meaning" because it depends on the context of the message receiver (semiotics!), but then it should be possible to put an message (SO question) into context of all previously received messages (posted SO questions).

1 The fact that SO content features code snippets additionally to natural language, it's possible a subject to research on its own what does it mean, what contributes etc.; for simplicity, I'll possibly just exclude code for text analysis but for sure it's also worth looking on the correlation between code snippets and votes. As said, it could be a research topic on its own.


texts: Is my implementation of propagation correct ? X is numpy array of shape(9000,784) and Y is one hot encoded array(9000,10) for mnist

texts:
def sigmoid( X ):

     sig= 1/(1+np.exp(-1*X) )
     return sig

def FindCost(output,hypothesis):

      cost = output*-1*np.log(hypothesis) + (1-output)*-1*np.log(1-hypothesis)
      cost = np.sum(cost)/cost.shape[0]
      return cost

def backpropagation(X,Y):

learningRate = 0.2
iterations = 5
#add bias to input data
InputDimention = X.shape[1]
Layer1neurons = 20
Layer2neurons = 15
outputNeurons =  Y.shape[1]

#initialize wts
weight1 = np.random.normal(1,1,(InputDimention,Layer1neurons))   #(no of input dimention, no of neuron in layer 1)
weight2 = np.random.normal(1,1,(Layer1neurons,Layer2neurons))   #(no of neuron in layer 1, no of neuron in layer 2)
weight3 = np.random.normal(1,1,(Layer2neurons,outputNeurons))       #(no of neuron in layer 2,no of neuron in layer 3)
bias0 = np.random.normal(1,1,(1,Layer1neurons)) #weight of bias to first neuron
bias1 = np.random.normal(1,1,(1,Layer2neurons))
bias2 = np.random.normal(1,1,(1,outputNeurons))

for i in range(0,iterations):
    #feed fwd
    net1 = np.matmul(X,weight1) + bias0
    net1 = sigmoid(net1)
    net2 = np.matmul(net1,weight2) + bias1
    net2=  sigmoid(net2)
    net3 = np.matmul(net2,weight3) + bias2
    net3 = sigmoid(net3)

    #feed back
    dJ_dnet3 =  (net3 - Y) / len(Y)
    node_delta3 = dJ_dnet3*net3*(1-net3)
    dJ_dw3 = np.matmul(node_delta3.T, net2)
    dJ_dbias2 = node_delta3.sum(axis=0)

    dJ_dnet2 = np.matmul(node_delta3,weight3.T)
    node_delta2 = dJ_dnet2*net2*(1-net2)
    dJ_dw2 = np.matmul(node_delta2.T,net1)
    dJ_dbias1 = node_delta2.sum(axis=0)

    dJ_dnet1 = np.matmul(node_delta2,weight2.T)
    node_delta1 = dJ_dnet1*net1*(1-net1)
    dJ_dw1 = np.matmul(node_delta1.T,X)
    dJ_dbias0 = node_delta1.sum(axis=0)

    weight1 = weight1 - learningRate*dJ_dw1.T
    weight2 = weight2 - learningRate*dJ_dw2.T
    weight3 = weight3 - learningRate*dJ_dw3.T

    bias0 = bias0 - learningRate*dJ_dbias0
    bias1 = bias1 - learningRate*dJ_dbias1
    bias2 = bias2 - learningRate*dJ_dbias2

cost = FindCost(Y,net3)
print cost
return net3


texts: nlp - opinion mining vs sentiment analysis

texts:
I have been told that nlp possibly holds the key for allowing researchers to infer the affective state of a person when writing. For instance, by using nlp analysis on online note taking you could infer whether a student is stressed. I work in educational cognitive science, so if this were the case it would be a valuable resource for me, however I am struggling to find evidence that this is indeed the case.

I have begun preliminary research and am trying to learn a bit about the technical aspects of nlp - I'm taking an online course by Jurafsky and Manning, another by Michael Collins - and I have been reading about what what can be inferred using nlp, specifically around opinion mining and sentiment analysis.

My question is two part:

Firstly, most resources I have come across say something along the
lines of nlp can be used for opinion mining and sentiment analysis
and we will talk about the implications for opinion mining'. Can
someone point me in the direction of more sentiment analysis oriented
resources?
Secondly, as I understand it, sentiment analysis is deriving the
stated sentiment within text, e.g. I was happy with, I loved, was tasty, enjoyed, hated, frustrated, etc. Can nlp, or something else,
be used to derive unstated affect? And is this the same thing as
sentiment analysis?

[Apologies if this is asked in the wrong exchange. I've been trying to find the best fit, but there were a few candidates and I wasn't sure which would be most appropriate.]


texts: How to improve performance of Tensorflow's CNN example?

texts:
In the example in this link: https://www.tensorflow.org/tutorials/images/cnn, the performance is over 70%, how can I improve it? What parameters do I need to tweak?
Thanks.


texts: Can someone explain what batch size is doing in convolutional NNs?

texts:
I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was simply controlling the number of images sent to the gpu / cpu at any given time, but I can't match that up with this behavior. Can someone explain what batch size controls in the model?  Why the variation in performance? Any good resources?

Sorry for the naive question. This is my first time using these sorts of models, and in the background reading I've done, I've still not fully grasped what this parameter is doing.


texts: Correlation threshold for Neural Network features selection

texts: I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to consider a variable eligible to be an input for my Neural Network?


texts: Is this scheme correct for logistic regression with stochastic gradient descent

texts:
I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging to the correct class oscillates around $0.5$. I have got two classes denoted as $0,1$ and two features. Is this scheme correct?:

Initialize weights $w = [w_0, w_1, w_2]$ to some random number (I choose 1) and loop:

Choose random training sample $x_i = [1, x_1, x_2]$.
Calculate the probability of it belonging to class $1,$
$$P(\hat{y} = 1) = \frac{1}{1+e^{-w \cdot x_i}}$$
Update the weights according to
$$w = w + \alpha x_i(y_i - P(\hat{y} = 1))$$

This scheme makes sense to me since the update to the weights would be big if the probability of belonging to class $1$ was low, if the correct class was $1$ but I can't verify it more than that.


texts: Convergence of vanilla or natural policy gradients (e.g. REINFORCE)

texts:
I am reading in a lot of places that policy gradients, especially vanilla and natural, are at least guaranteed to converge to a local optimum (see, e.g., pg. 2 of Policy Gradient Methods for Robotics by Peters and Schaal), though the convergence rates are not specified.

I, however, cannot at all find a proof for this, and need to know if and why it is true.  I am wondering the following:

What is the proof of this?  Can someone point me to a reference?
What is the best convergence rate out there that's known (if any are proven)?
If I am reading at least the Peters/Schaal paper correctly, this only holds for discounted reward.  Is there any formulation that works for the mean reward of a rollout?  I am in a situation where I am running many simulations of a robot and most directly want to optimize its mean reward.


texts: If I was to train a pixel rnn, by concatenating images with noise vectors, I should be able to get new images from that distribution by starting on the prediction with a new noise vector. Would it be possible to use medication packet leaflets instead of images or as the images, and generate new drugs complete with description this way? If so, how big a dataset would be required and if not, what are the limitations to this approach?


texts: Bias of 1 in fully connected layers introduced dying relu problem

texts:
While implementing AlexNet (model-code), one of the thing I need to do was to initialize the biases of the convolutional layers and fully connected layers.

Normally we initialize biases with 0s, but the paper says:

  We initialized the neuron biases in the second, fourth, and fifth convolutional layers,
  as well as in the fully-connected hidden layers, with the constant 1.

So I went ahead and initialized the biases to 1 as the paper says. But that didn't make the network learn at all. Basically the last fully connected layer was producing a lot of 0s, which is otherwise known as dying-relu-problem. Out of 4096 neurons only 40 or 50 were producing non-zeros.

After lot of debugging, I came to realize that: if I make the fully connected layers' bias to 0 than they are nicely learning. loss decreased nicely.

Now I'm wondering:

How bias plays the role for dying-relu-problem here ?
Can all dying-relu-problem be corrected using bias searching ?


texts:
I am learning reinforcement learning, and as a practice, I am trying to stabilize an inverted pendulum  (gym: Pendulum-v0) in an upright position using policy gradient: REINFORCE.

I have some questions, please help me, I tried a lot but couldn't understand. an answer to any question can help me. Thanks in advance.

1- why the observations in the pendulum code are: cos(theta), sin(theta) and theta_dot? not theta and theta_dot only?

2- the action which I should send to the environment (env.step(action)) is it correct like this (in python):

prob=output.eval(arguments={observations: state})[0][0][0]
# output of the neural network

# Bernoulli dist. either 1 or 0 according to a threshold (rand.)
action = 1 if np.random.uniform() < prob else 0

3- the reward function is defined in the pendulum code as follows, but I couldn't understand why! shouldn't it be something like if the pendulum is upright (with some tolerance) so the reward is high, otherwise zero?

costs = angle_normalise((th)**2 +.1*thdot**2 + .001*(action**2))
# normalize between -pi and pi
reward=-costs

4- The pendulum motion equation is different from what is known (like here), the environment code use the following equation, where is comes from?

 newthdot = thdot + (-3*g/(2*l) * np.sin(th + np.pi) + 3./(m*l**2)*u) * dt


texts: How do I implement a backpropagation neural network with particle swarm optimization for rssi-based distance model?

texts:
I was trying to implement pso-bpnn. Can any one help with this?

I am unable to understand how a bpnn computes distance from rssi values. And how to optimize bpnn weights using pso.


texts: Why and how to match variables in logistic regression?

texts:
I have a dataset of ~4.7K records focused on binary classification with 60 features. class 1 is of 1554 records and class 2 is of 3558 records.

Now I would like to find the risk factors that influences the outcome which is disease present or not. This is a supervised learning problem

I understand that people do matching to ensure that both the classes have similar distribution, so that the comparison results are reliable.

1) I see people usually do matching based on demographics like Age etc. Is it to infer what factors really influence the outcome if we keep Age constant. Am I right to understand this way?

2) If I put all the variables in logistic regression model, doesn't that account for confounding? Why do I have to do matching?

3) Out of 60 features, I would like to do matching based on 4 variables. How do I do this for my full dataset? Is there any python package to do this?

Can someone help me on how to do this?


texts: Is this a good classified model based confusion matrix and classification report?

texts:
I build a classification model. The results below when implemented Decision Tree Classifier.

Prediction accuracy

0.785813630042

confusion matrix

[[ 1302  1581]

[ 2577 13953]]

         precision    recall  f1-score   support

    2.0       0.34      0.45      0.39      2883
    3.0       0.90      0.84      0.87     16530

avg / total       0.81      0.79      0.80     19413

For the Neural Network---

Prediction accuracy

0.863132952145

Confusion Matrix

[[  718  2165]

[  492 16038]]

         precision    recall  f1-score   support

    2.0       0.59      0.25      0.35      2883
    3.0       0.88      0.97      0.92     16530

avg / total       0.84      0.86      0.84     19413

For the Gradient Boosting Classifier---

Prediction accuracy

0.870035543193

Confusion Matrix

[[  971  1912]

[  611 15919]]

         precision    recall  f1-score   support

    2.0       0.61      0.34      0.43      2883
    3.0       0.89      0.96      0.93     16530

avg / total       0.85      0.87      0.85     19413

Gradient boosting classifier gives me good accuracy score but it gives bad recall for class 2 compared with Decision Tree classifier.
Train Data for both classes:

Class 2: 2908

Class 3: 16673

Is recall depend on training data? How do I improve recall for class 2 along with accuracy?


texts: How to combine the features extracted from different CNN architectures?

texts: I want to combine the features extracted from different CNN architectures into my fully connected layer. How to proceed?


texts: How does Elastic's Prelert (formerly Splunk Anomaly Detective App) work?

texts:
Background: In recent months, Elastic has purchased Prelert and will actively incorporate it into the Elastic stack (and also discontinue the Splunk Anomaly Detective App!). I am trying to understand the theoretical methodology and process framework for their predictive analytics approach to time series anomaly detection so I can be an effective and efficient user of the Prelert tool.

Question: What topics to explore, and what is the general process flow/pipeline? How would one start to learn and understand the building blocks of the big concepts involved. I understand from a few posts online (1,2) a bit about how it works, but want to understand the major process items involved. At a high level, I get that there is some training set that fits a distribution via unsupervised ML, and that distribution is used to assess probability of future events - but need to get into the sequential processing blocks involved (for example, data read step, followed by sending it to a distribution-matching step, then some unsupervised ML step where a distribution is selected/adjusted, etc). I understand specifics may be difficult, so guesses and hypothetical ideas from people with experience in this arena would be great.

Note 1: I am not interested in the Prelert's "secret sauce" and the exact proprietary algorithms that they have developed - this would be unethical and I do not want to recreate their product, I just want to understand how to better use it. I am interested in an explanation of the main concepts implemented at a high level but in more detail than the posts (1,2) I found. Example post of the explanation could be in the form:

'distribution is populated via X concept/algorithm and then followed
by selection step by Y machine learning concept/algorithm. Possible
concepts to study are A,B,C. Key search words for literature searches
are Q,W,E. Some references are N,M,L.'

Note 2: I have posted on a related topic before, but perhaps this is an interesting side question for a similar audience.


texts: How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?

texts:
I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine learning algorithm.

Where to look for info?
Googling just gives a lot of examples of shape recognition and 2d to 3d shape reconstruction. I want it other way from 3d to 1d data


texts: What is the meaning of the parameter "metrics" in the method model.compile in Keras?

texts:
I don't have very clear the meaning of the parameter metrics of the compile method of the class model in Keras:

model.compile(..., metrics = ['accuracy'], ...)

the documentation states:

  List of metrics to be evaluated by the model during training and testing

what I don't understand is:

are these the metrics used to evaluate the performance of the network at the end of each training epoch, i.e. at the end of each epoch the code makes the network predict on the training set and calculates the metrics passed

OR

are these the metrics used to train the network, i.e. is the network trained with the goal of obtaining the best possible value for these metrics (I don't know how it could do this, something like: if the recall is low then it corrects the weights more when working with positive samples)?

(or it has another meaning?)


texts: Effective Methods for Background Removal on Images

texts:
I'm interested in learning about how background removal works on images taken of clothing items. Do we need a specific color difference between the background and the clothing item in order to be able to determine object edges? How are edges determined?

Can somebody link me to algorithms/models/methods used effectively?


texts: How to properly rotate image and labels for semantic segmentation data augmentation in Tensorflow?

texts:
What's a proper procedure for doing the image and label rotation for semantic segmentation in dataset augmentation using Tensorflow?

Images

I have seen the function tf.contib.image.rotate(), but this function fills empty space with zeros (from docs):

  Empty space due to the rotation will be filled with zeros.

I would like to fill that empty space with a different value (maybe some constant, like the dataset mean pixel). How this can be done in Tensorflow (I know that there are options in Keras Image Preprocessing, but I need TF)?

Labels

Also, what about the labels? If I just use the same function (tf.contrib.image.rotate()), it will fill the empty space with zeros suggesting that pixel in those places belong to class with id 0 (since I have class labeled with 0). The one solution could be to put ignore label on those pixels (e.g. 255), but, again, the current function doesn't support default fill value ...


texts: "Smearing" probabilities or how to handle imprecise locations for canonically classification-type problems

texts:
I am trying to predict failures at different nodes on a line. Each node has different weather features and hardware/configuration features. For a little under half of the historical failures I have, I know the precise location/exact node where the failure happened, however for the other half I only know a range of nodes the failure could have happened on. My current approach is to "smear" a 100% probability across N nodes that the failure could have happened on. e.g. I know a failure happened between nodes A and D therefore assign A, B, C, and D the value 0.25.

I am currently treating my model as a regression on the probability at each node in keras with a single node output layer but the performance is really poor. For other problems like this, historically I've had plenty of data and more exact locations so I could treat it as a strict binary classification however I've never had to deal with classifications that are "partial" so-to-speak. How would you handle this problem? Is a regression of the probability the best approach? I've had the thought of assigning all possible nodes full 1's instead of fractions and treating it as a standard binary classification but I'm worried that will greatly over represent the unknown location failures.


texts: Linear Discriminant - Least Squares Classification Bishop 4.1.3

texts:
Pls. refer section 4.1.3 in Pattern Recognition - Bishop: "Least squares for Classification":
In a 2 class Linear Discriminat system, we classified vector $\mathbf{x}$ as  $\mathcal{C}_1$ if y($\bf{x}$)>0, and $\mathcal{C}_2$ otherwise.
Generalizing in section 4.1.3, we define $\mathcal{K}$ linear discriminant equations - one for each class:
$y_{k}(\bf{x}) = \bf{w_k^Tx} + \mathit{w_{k0}} \tag {4.13}$
adding a leading 1 to vector $\bf{x}$ yields $\tilde{\mathbf{x}}$.
And the Linear Discriminant function for $\mathcal{K}$ class is given by:
$\bf y(x) = \widetilde{W}^{T}\tilde{x}$. The author progresses and presents sum of squares Error function as:

$E_D(\widetilde{W}) = \frac{1}{2}Tr\{(\tilde{X}\widetilde{W} - T)^T(\tilde{X}\widetilde{W} - T)\} \tag {4.15}$

My doubts are related to above equation 4.15.
Consider a 3-class system with only one observation, $\bf{x}\in \mathcal{C_2}$, my understanding:

Pls. refer $\bf{Y}$ in upper half of diagram. Will only $val(\mathcal{C_2})$ be positive: $\mathbf{x} \in \mathcal{C_2}$,
$y_{2}(\bf{x})$ > $\it{threshold}(\mathcal{C_2})$.  Is the value,
$val(\mathcal{C_k})$, negative for other classes' Discriminant
functions? If not, could you briefly explain the reason?
The error matrix, $\bf{E}$ is 1x3 matrix. $\bf{E}^{T}E$ will be a 3x3 matrix, with diagonal elements representing squared(Error) for a
class. Does $Tr$ in 4.15 stand for $trace$ - sum of diagonal elements?
If so, why do we ignore off diagonal error values/ why don't they
matter?

P.S.: If my understanding is wrong/ grossly wrong, I'll appreciate if you point out the same.


texts: What is the current state-of-the-art within aspect-based sentiment analysis?

texts:
I've been reading Bing Liu's book on Sentiment Analysis. He mentions all of these slightly different approaches seen in research since 2004, but doesn't talk much about efficacy at all.

That leaves me - someone who has not done any sentiment analysis before - wondering what approaches are seeing the best results currently. And it specifically needs to be an approach that can extract the sentiments of individual topics found in the text, not just if a document/sentence is positive or negative.

In case it makes a huge difference: the object of analysis will be reddit comments, not tweets or reviews which seem to be the most common source of data.


texts: matplotlib graph to plot values and variance

texts:
I am really new to the world of matplotlib graphing as well as using those graphs to understand data.

I have written a simple python code where I read a .csv file in and then store the values of one column into a variable. Then plotting them similar to the code bellow:

dev_x= X   #storing the values of the column to dev_x
plt.plot(dev_x)
plt.title('Data')

The graph looks like this, which seems quite messy and hard to understand. So, I am asking for some advice on how to make more cohesive graphs.

This is what my .csv column looks like. It is just many other other rows.

['40' '20' '10' '0' '10' '30' '50' '70' '90' '110' '130' '150' '170' '200'
 '240' '290' '40' '20' '10' '0' '10' '30' '50' '70' '90' '110' '130' '150'
 '170' '200' '240' '290' '40' '20' '10' '0' '10' '30' '50' '70' '90' '110'

At the end of the day I would like a way to display these in a better way so I can also find the variance of this column.


texts: Trying to beat random forest with xgboost

texts:
I have a small time series dataset of about 3000 samples and 5 features. With xgboost, my predictions seem biased (consistently overestimating the target). No matter how many estimators I throw at the problem along with hyperparameter tuning, I can't seem to beat a random forest. How can I go about diagnosing this?

rf = RandomForestRegressor(n_estimators=1000,
                      max_features=1,
                      min_samples_leaf=20,
                      random_state=0, n_jobs=-1)

xgb = xgb.XGBRegressor(n_estimators=100000,
                             max_depth=1,
                             learning_rate=.0001,
                             min_child_weight=1,
                             subsample=.1,
                             colsample_bytree=.1,
                             base_score=0)


texts:
I'm in charge of setting up a patient register (100K+ patients) for a non-profit project with little money. This register should provide the basis for later datascience.
I'm not sure how a good database solution can work the long run.
It must be possible for various clinics to enter the data manually into the system. Since I have experience with Django I have developed a webapp prototype with Django and an SQLite DB (it is not expected that many users access the database at the same time).
But I'm not sure what kind of database would be a good a good choice in the long run. The biggest problem is that a self-made solution may be beautiful for the moment but it binds the project to me and thus puts it in danger in the long run (the typical doctor doesn't know much about programming).
My question is basically about experiences of you experienced datascientists like you for suitable databases and solutions.


texts: Recommendations for statistical models given my dataset

texts:
I'm having a problem when using the k-fold cross-validation with the Random Forest method. One of the outputs is the error "Error in randomForest.default(x, y, mtry = param$mtry, ...) : Need at least two classes to do classification." However, I already have two classes to do the classification, which are "Normal" and "Failure". When posting this question at https://stackoverflow.com/questions/60643415/error-in-randomforest-defaultx-y-mtry-parammtry-need-at-least-two?noredirect=1#comment107290940_60643415 was recommended to me that I ask for "recommendations for statistical models given my data and your prediction/estimation/modeling needs".

Someone could help me?

The R script:

library(caret)
library(randomForest)

data_failures <- read.csv('OUTPUT.csv', header = TRUE, sep = ",", stringsAsFactors = TRUE)

train.control <- trainControl(method = "cv", number = 10)
model <- train(Period_1 ~., data = data_failures, method = "rf",
                                   trControl = train.control)
print(model)

print(class(str(data_failures)))

The ouput:

Random Forest

112 samples
 11 predictor
  2 classes: 'Failure', 'Normal'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 101, 101, 101, 101, 101, 101, ...
Resampling results across tuning parameters:

  mtry  Accuracy  Kappa
   2    1         NaN
   6    1         NaN
  11    1         NaN

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.
'data.frame':   112 obs. of  12 variables:
 $ Period_1 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_2 : Factor w/ 2 levels "Failure","Normal": 2 2 2 1 2 2 2 2 2 1 ...
 $ Period_3 : Factor w/ 2 levels "Failure","Normal": 2 2 1 2 2 2 2 2 2 2 ...
 $ Period_4 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_5 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_6 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_7 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_8 : Factor w/ 2 levels "Failure","Normal": 2 2 2 1 2 2 2 2 2 2 ...
 $ Period_9 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_10: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_11: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_12: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
[1] "NULL"
Warning messages:
1: model fit failed for Fold08: mtry= 2 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

2: model fit failed for Fold08: mtry= 6 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

3: model fit failed for Fold08: mtry=11 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

4: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

A sample of the data:

    Period_1 Period_2 Period_3 Period_4 Period_5 Period_6 Period_7 Period_8
1     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
2     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
3     Normal   Normal  Failure   Normal   Normal   Normal   Normal   Normal
4     Normal  Failure   Normal   Normal   Normal   Normal   Normal  Failure
5     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
6     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
7     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
8     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
9     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
10    Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal
```


texts:
I'm currently building a model to predict early mortgage delinquency (60+ days delinquent within 2 years of origination) for loans originating in 2018Q1. I will eventually train out-of-time (on loans originating in 2015Q4), but for now I'm just doing in-time training (training & testing on 2018Q1) -- and even this I've found challenging. The dataset contains ~400k observations, of which ~99% are non-delinquent and ~1% are delinquent. My idea so far has been to use precision, recall, and $F_1$ as performance metrics.
I am working in Python. Things I've tried:

Models: logistic regression & random forest.
Model selection: GridSearchCV to tune hyperparameters with $F_1$ scoring (results were not significantly different when optimizing for log-loss, ROC-AUC, Cohen's Kappa).
Handing imbalanced data: I tried random undersampling with various ratios and settled on a ratio of ~0.2. I also tried messing with the class weights parameter.

Unfortunately, my validation & testing $F_1$ scores are only around 0.1, (precision & recall are usually both close to 0.1). This seems very poor, since with many problems you can achieve $F_1$ scores of 0.9+. At the same time I've heard there's no such thing as a "good $F_1$" range, i.e. it is task-dependent. Indeed, a dummy classifier which predicts proportional to the class frequencies only achieves precision, recall, and $F_1$ of 0.01.
I've tried to find references on what a "good" score for this type of task is, but I can't seem to find much. Others' often report ROC-AUC or Brier Score, but I think these are hard to interpret in terms of business value added. Some report $F_1$ but see overly optimistic results due to data leakage or reporting testing performance on undersampled data. Finally, I've seen some people weight confusion matrix results by expected business costs as opposed to reporting $F_1$, which seems like it may be a better route.
My questions are: (1) is an $F_1$ score of 0.1 always bad?, (2) does it even make sense to optimize for $F_1$ or should I used another metric?, (3) if $F_1$ is appropriate and a score of 0.1 is bad, how might I improve my performance?


texts: squared error vs absolute error in boosting trees for regression

texts:
From the Element of Statistical Machine learning book, it says:

Using loss criteria such as the absolute error or the Huber loss (10.23) in
place of squared-error loss for regression, and the deviance (10.22) in place
of exponential loss for classification, will serve to robustify boosting trees.
Unfortunately, unlike their nonrobust counterparts, these robust criteria
do not give rise to simple fast boosting algorithms.

My question is why absolute error leads to a slower boosting algorithm? can't we do the same thing using absolute error as when using squared error except for changing the power?
It also mentions squared error simplifies things because the new tree is just fitting the current error. But isn't it the same for absolute error?
Note: this is in a boosting setting and not gradient boosting


texts: Pandas Query Optimization On Multiple Columns

texts:
My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns (say A and B). Now, in the calculation, for each row in the test dataset, I have to get the result of the following query. This time, the program takes forever. I can optimize this on my own writing some extra code (sorting will help in my case and in fact I can skip pandas altogether if I want to), but I wanted to check if there is a better built-in solution should I ever require that (also, I don't want to if I don't have to). The query is like this:

def f(A, B, x, y):
    data = df.loc[df[A].isin([x]) & df.B.isin([y])]
    return len(data)

Here, df is the DataFrame (training dataset). For each row in test dataset, data find all rows where column A has value x and B has value y. As far as I know, isin is slightly faster, so I used it. However, I think this is why it does not run within a second like it did before because each time I run this query, it needs a lot of time. What I have to do next is something like this this.

for i in range(0,l):
    ret = f(A, B, test[A].values[i],test[B].values[i])

Clearly, in every loop, it will calculate the result of the query. Other than this, rest of the functions in my code are mathematical that do not need much calculation.

Let me know if the problem is not clear since I did not use direct codes (replaced it with dummy variables and other lines do not have anything that has high complexity).


texts: How to properly do feature selection when comparing different models?

texts:
Context:
I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random Forest Regressor, Decision Tree Regressor, and Linear Regression. I can tell there is some overfitting going on, as my initial values vs cross validated values are as follows:
RF: 10 Fold R Squared = 0.758, neg RMSE = -540.2 vs unvalidated R Squared of 0.877, RMSE of 505.6
DT: 10 Fold R Squared = 0.711, neg RMSE = -576.4 vs unvalidated R squared of 0.829 and RMSE of 595.8.
LR: 10 Fold R squared = 0.695, neg RMSE = -596.5 vs unvalidated R squared of 0.823 and RMSE of 603.7
I have already tuned the hyperparameters for RF and DT, so I was thinking about doing feature selection as a next step to cut down on some of this overfitting (especially since I know my feature importances/coefs).
On to my questions:
Is feature selection independent of any machine learning algorithms?->I.E., should I be choosing a strict subset of features to apply to all of my machine learning models to keep everything consistent as I'm comparing model performances? Or should I be doing feature selection uniquely to each model(choosing a unique subset of features for each model)? What is generally considered to be too many features? And is using lasso feature selection acceptable in this case (as I have continuous variables)?
And as a follow up question, is it okay to be doing feature selection after tuning my hyperparameters? And would I need to tune them once again after selecting my features?
Finally I noticed that my validated RMSE for DT and LR are lower than their unvalidated RMSE. I think this is inconclusive, but does anyone have any thoughts?


texts: How do I apply Min Max scaling for numerical forecast when both dependent and independent volumes are increasing over time?

texts:
I'm want to build a numerical regression to forecast.

From my initial analysis, it shows linear models (glm) out performs the typical decision tree models (xgboost, ranger...etc). I hypothesized that 2 factors may be at play. Features (independent variables) truly follows a Gaussian Distribution (bell curve) and thus will outperform non-linear models and that decision trees do not extrapolate as well (expecting incremental volumes not seen in historical data).

Due to the complexity of the number of features and it's high levels of interaction effects, I'm keen to try Tensorflow. As i understand it, it can extrapolate well, as it uses many coefficients as per the linear models. I'm experiencing issues with the pre-processing of data. Literature suggest that scaling is important, and my understanding is two types of scaling is generally used. Min Max Normalization and Mean Standard Deviation (Standardization into Z-scores) (see https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/).

As I want to introduce more features into the model (including non-linear ones), I'm going to use the min max normalization. The issue is, due to the continued incremental numerical values (both independent and dependent variables), It doesn't seem right to normalize new datasets with historical min max values.

How should I pre-process (normalize / scale) when i expect future datasets to have increasing numerical values?

Simple example of the scenario described above would be.

Train dataset (2 columns x 10 rows, 1 target, 1 feature)
target values ranges from 1 to 10
feature values ranges from 30 to 50

validate dataset (2 columns x 10 rows, 1 target, 1 feature)
target values ranges from 11 to 20
feature values ranges from 50 to 70

Notice that the validate data has an increasing target, and feature range.

The question now is, if normalization / scaling is required as preprocessing prior fitting a tensorflow model, what impact would an increasing numerical values for both target and feature has on model performance based on validate datasets?

hope the above make sense.


texts: How can I compare the grammatical complexity between two texts using their sentences dependency length?

texts:
This is a continuation to the following thread.
I have two texts, common English texts such as news articles and informative texts versus a technical textbook. I want to compare the grammatical complexity between those texts using their sentences dependency length in order to conclude whether they both have the same level of complexity or not. I was thinking about using the p-value as an evidence against the null hypothesis. Here is how the data would look like:
Text 1
ID  Dependency Length   Sentence Length
0   13                  7
1   5                   3
2   20                  8

Text 2
ID  Dependency Length   Sentence Length
0   8                   5
1   10                  7
2   14                  7

By the way, I am using python.


texts: Predicting when features are time-dependent

texts:
How to predict data with time-dependent features?

For example, I have to predict the result of a Mortal Combat game:

X(i) = [player1_id, player2_id, hero_of_player1, hero_of_player2]

Y(i) = 1 if player1 wins or 0 if player2 wins

I have a dataset, containing games from large period of time. And, of course, performance of each player can variate during that time. Second, there were a lot of game patches, and they tuned some heroes' abilities. As a result overall hero strength can also variate through time as well as certain hero-vs-hero match-ups.

How to track that changes considering also overall and time-dependent impact of player+hero, hero-vs-hero, player-vs-player features?

So far I'm using simple LSTM network on whole game history. So my data has a single sample and total number of steps equal to game history length:

X = data.reshape(1,len(data),len(data[0]) #len(data[0]) corresponds to 4 features for a single record in dataset, but it is onehot-encoded.

m = Sequential()
m.add(LSTM(25, input_shape=(None, len(data[0])), return_sequences=True)
m.add(TimeDistirbuted(Dense(1))

I've got about 60% performance on the real problem. And I think it might be better and I can tune layer sizes, regularize, add stacking LSTMs. But did I choose an adequate NN-structure? What are the most suitable NN-structures for that kind of problems? Sliding-windows and Convolutional LSTMs sounds promising, but I don't have enough intuition about them.


texts: what could it be a good descriptor for simple greyscale symbols?

texts:
I need to find a good method to extract key points from drawing symbols like elements in CAD drawings (eg in 2D objects for the bathroom that can be placed on a floor plan).

I've already tried all type available in openCV/accord (FAST, SURF, Harris, ..) but all of them failed to give me any point.

The images are blank and white (with some sort of grey scale) and I'd like to find the position of a limited set of symbols inside bigger images containing other unknown objects/shapes (that I'm not interested in anyway).

The basic idea is to get the descriptors for my symbols and check if the target image contains similar descriptors.

If I have a good method for descriptors then I can train a classifier to check all the descriptors from the input images and so I can also know the position related to the recognized object.

Framework can be any with preference on OPENCV, Accord.net, tensorflow


texts: There is a project which contains models in DLC format (Snapdragon Neural Processing Engine - SNPE) that I guess are optimized for the Qualcomm Snapdragon 820 chipset (see second link). The project newly also introduced models in Keras format. My question is - will the Keras model run on Google Coral with Edge TPU? If so, how fast? Is it needed to optimize the model for TPU?


texts: Partitioning data into features/labels and train/test after reading from csv file

texts:
I need to read data from csv file and then first partition that data into features and labels and then into training and testing set. However, there are several issues cropping up again and again. Below is the code I tried with error ,

ValueError: could not convert string to float: 'mon'
on line
Y: train_y})

The code for Linear Regression :-

import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
import numpy as np

learning_rate = 0.01
training_epochs = 1000
display_step = 50

data = pd.read_csv('forestfires.csv')
y = data.temp
x = data.drop('temp', axis=1)

train_x, test_x, train_y, test_y = train_test_split(x, y,test_size=0.2)
n_samples = train_x.shape[0]
n_features = train_x.shape[1]

X = tf.placeholder('float', [None, n_features])
Y = tf.placeholder('float', [None, 1])

# Model weights.
W = tf.Variable(np.random.randn(n_features, 1), dtype='float32')
b = tf.Variable(np.random.randn(1), dtype='float32')

# Construct linear model.
prediction = tf.matmul(X, W) + b
loss = tf.reduce_sum(tf.pow(prediction - Y, 2))/(2 * n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

# Start training.
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for epoch in range(training_epochs):
        for (x, y) in zip(train_x, train_y):
            sess.run(optimizer, feed_dict={X: train_x,
                                           Y: train_y})
            # Display logs per epoch step.
            if (epoch + 1) % display_step == 0:
                c = sess.run(loss, feed_dict={X: train_x,
                                              Y: train_y})
                print ('Epoch:', '%04d' % (epoch+1), 'cost=','{:.9f}'.format(c), \
                       'W=', sess.run(W), 'b=', sess.run(b))
    print ('Training Done!')
    training_cost = sess.run(loss, feed_dict={X: train_x,
                                              Y: train_y})
    print ('Training cost=', training_cost, 'W=', sess.run(W), 'b=', sess.run(b), '\n')
    # Graphic display.
    plt.plot(train_x, train_y, 'ro', label='Original data')
    plt.plot(train_x, sess.run(W) * train_x + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

Could anyone help me with reading data properly in a rather general way. Snapshot of the data :-


texts: How do I get latent features for a new row of data when doing non-negative matrix factorization?

texts:
Background: The basic set-up for non-negative matrix factorization (nmf), is that we take a matrix with non-negative elements, X and find two other non-negative matrices H and T such that

  $||X - HT||$ is a minimum

$X$ in this example is a matrix that represents a data set. Rows correspond to samples of data, and columns correspond to features. I understand that nmf is used for feature reduction. My understanding is that you can use the latent features to train a model on.

However, suppose that I use nmf for feature reduction and then get a new row of data and want to find the reduced features for the new data. Do I have to run nmf again with an updated dataset and then retrain my model each time?


texts: XGBoost becomes unstable when predicting more than ~300 classes

texts:
I'm using the Python implementation of XGBoosts (version 0.80) XGBoostClassifier to predict one of a large number of classes.

My feature data consists of a sparse boolean matrix of ~10M rows, ~5k columns, with a density of 0.003. (so about 15 values per row).

My targets consist of 2000+ different classes in a long-tail skewed distribution (~300 classes occur more than 10k times).

For now I'm grouping all of my uncommon classes (<10k) into 'other' so they at least count as negative examples. But ideally, I'd like to be able to use a lot more of the different classes as actual targets.

However, whenever I use, say, 800 different targets (each occurs ~5k+ times), the model becomes unstable: training takes a lot longer, the training loss doesn't seem to converge (I haven't saved the output of this unfortunately) and the predictive quality becomes increasingly poor (predict_proba returning 0.99+ for the wrong classes).

Should I just change some parameter I've overlooked? Am I running out of float precision for my loss? Is it due to sampling in the approx method? Does anyone have any ideas?

My specific classifier:

clf = XGBClassifier(
    n_jobs=8,
    max_depth=7, #arbitrary
    min_child_weight=5, #arbitrary
    subsample=0.5, #arbitrary
    colsample_bytree=0.9, #arbitrary
    tree_method='approx',
    objective='multi:softprob',
    eval_metric='mlogloss',
    silent=False
)


texts: Cluster words into groups of similar meaning (synonyms)

texts:
How can words be clustered into groups of similar meaning (synonyms)?
I started with pre-trained word embeddings (e.g., Google News), which is great, but not perfect - a limitation arises because the word embeddings are based on surrounding words. This introduces challenging results. For example:

polar meanings: word embeddings might find opposites to be similar. Even though these words mean the opposite semantically, they can quite readily be interchanged given the same preceding and following words. For example, "terrible" and "fantastic" are easily interchangeable in the following sentence: "I heard some ____ news today". Another example is "happy" and "unhappy" in the following sentence: "I was very ____ with the service".
euphemisms: word embeddings may find euphemistic words and doublespeak to be similar. Even though these words are perceived very differently, they can still readily be interchanged given the same preceding and following words. For example, "headstrong" and "stubborn" are easily interchangeable in the following sentence: "He is a ____ colleague". Other examples are: "pamper/spoil", "check out/perv", etc.

Is there a way to separate similar words (in a word-embeddings sense) into more semantic clusters?

Possible solution? Make one extra dimension that puts the word on a sentiment spectrum? This one dimension might be enough to sufficiently separate "terrible" from "fantastic", and "happy" from "unhappy", etc.
How might such a feature be engineered?
Note that many words would be "neutral" and so this additional dimension wouldn't really affect them. For example, "Python/R/C++/Java" in the following sentence: "5+ years experience of ____ required".


texts: how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?

texts:
I'm working on the dataset with lots of NA values with sklearn and pandas.DataFrame. I implemented different imputation strategies for different columns of the dataFrame based column names. For example NAs predictor 'var1' I impute with 0's and for 'var2' with mean.
When I try to cross validate my model using train_test_split it returns me a nparray which does not have column names. How can I impute missing values in this nparray?
P.S. I do not impute missing values in the original data set before splitting on purpose so I keep test and validation sets separately.


texts: Book keeping of experiment runs and results

texts:
I am a hands on researcher and I like testing out viable solutions, so I tend to run a lot of experiments. For example, if I am calculating a similarity score between documents, I might want to try out many measures. In fact, for each measure I might need to make several runs to test the effect of some parameters.

So far, I've been tracking the runs inputs and their results by writing out the results into  files with as much info about the inputs. The problem is that retrieving a specific result becomes a challenge sometimes, even if I try to add the input info to th filename. I tried using a spreadsheet with links to results but this isn't making a huge difference.

What tools/process do you use for the book keeping of your experiments?


texts: Categorical and non-categorical data in the same column

texts:
I have a unique dataset that has many columns and most columns contain both categorical and non-categorical data.  For example, let's say that one column is attribute_1 and for observations that have data for attribute_1 the value can be between 100 and 1000.  If an observation does not have data for attribute_1 then they are given a value between -4 and -1, where the value describes why they don't have data for this attribute.

How can I encode the categorical part of the columns while also applying feature scaling to the non-categorical part of the column?  Would it make sense to split the column into two where one is the categorical and another column just for the non-categorical?


texts: Segmentation Network produces noisy output

texts:
I've implemented a SegNet and SegNet ReLU variant in PyTorch. I'm using it as a proof-of-concept for now, but what really bothers me is the noise produced by the network. With ADAM I seem to get slightly less noise, whereas with SGD the noise increases. I can see the loss going down and the cross-evaluation accuracy rising to 98%-99% and yet the noise is still there.

On the left is the actual image, then you can see the mask, and finally the actual output from the network. There's 1024 samples per class, and two classes, which are very consistent as the documents are very structured. I'm using the vanilla SegNet (same kernel, striding and padding) on 224x224.

What could explain this noise, and how could I potentially address the issue?