texts: Why spectral clustering results in disjointed cluster?

texts:
I'm working on a project where I have to dynamically cluster the position of objects with respect to one coordinate. So I'm essentially dealing with subsequent frames and each frame represents a one-dimensional dataset. The intuition behind clustering is to form clusters out of points that are in similar distance to other points within the cluster and can be naturally connected. I use spectral clustering due to its ability to cluster points by their connectedness and not the absolute location and I set rbf kernel due to its non-linear transofrmation of distance. However, in some frames the algorithm results in unnatural assignments. One example is

import numpy as np
from sklearn.cluster import SpectralClustering

X = np.array([[51.08354988], [57.10594997], [70.51259995], [76.74425011],
[61.24844971], [89.00615082], [98.55859985], [61.26575031], [88.35105019],
[87.40859985]])

clustering = SpectralClustering(n_clusters = 4, random_state  = 42,
                                gamma = 5 / (X.max() - X.min()))
clustering.fit(X)

and the result of clustering is presented in a form of swarm plot below, so only x coordinate matters here (each color represents a cluster and labels are array indices):

What I cannot understand is why points marked as red are clustered together as the similarity between points {4, 7} and {5, 8, 9} should be really low. My first thought was that maybe this is caused by unlucky, random initialisation, but I tried with many different random states and the resulting clusters seems to be persistant. So I guess this is connected to the chosen affinity measure (rbf_kernel) and its gamma parameter. As points move with each frame, and distances between them are kind of dependent on their overall range, I tried to set gamma to 5 / (X.max() - X.min()). The intuition behind this was that if the range is bigger, then the distances between points are usually bigger and we should penalise them more to obtain similar values of exp(-gamma * ||x-y||^2) to those obtained within smaller range. But it doesn't seem to work as expected and results in faulty clustering where red cluster is formed out of points divided by green cluster). My expectation would be rather clusters formed as follows: {0, 1, 4, 7}, {2, 3}, {9, 8, 5} and {6} or {0}, {1, 4, 7}, {2, 3}, {5, 6, 8, 9}.

So my question are:

Is affinity choice and its gamma parameter really the problem here?
If so, how can I choose gamma better?
Otherwise, what approach should I consider to deal with faulty assignments with separated points within the same clusters as presented here?
(Side question) Is there any measure/index that would be suitable to automatically compare clusterings with different number of clusters?

@Edit: As it can be observed under this link, those separated clusters occurs for a short period of time, but still, the problem seems to be recurrent.


texts: Managing NaN in target variables (testing)

texts:
Please can someone advise me on how to handle NaN in my target variables set?

I've tried a variety of things but none is working. Here's what I've tried:

Imputing zeros (0) in Y_test
Replacing NaN with whitespace in the Y_test
Emptying the Y_test

The dataset came already separated as training and test set. When I apply my logit classifier or any other classifier for that matter, it keeps returning a TypeError.

  TypeError: '<' not supported between instances of 'int' and 'str'

Please help!


texts: Performing Regression on Text and Image together in the most efficient way

texts:
I have a dataset with texts and images. The texts are present in a CSV file, which I am able to read using Pandas. The CSVs contain the image names, and I have the corresponding pngs which are mentioned in my dataset as well.
The data format is as follows (roughly):

I would like to train a regression model to predict the labels, while making use of Feature 1 and Feature 2, and the images that I have. The image names are the same as what is mentioned in the CSV.
I have tried training a Linear Model just for the CSV using SKLearn, and then training a CNN using Keras for the Images, and then averaging the regression values obtained, however, the results have been very poor (despite appropriate image transformation and pre-processing performed).
Now, I am pondering on how to improve my regression.
Will it be a good data science practice if I convert the images to a column or feature in the dataframe, and then apply a Linear Model? Or, is there a way to verify if the images would help me at all, before working on them. Each image is a very small sized RBG image.
I appreciate on any inputs on what is a right way to approach this.


texts: Mapping xml tags by Rule Learning/Generation Algorithms

texts:
I'm trying to map tags/attributes from xml files from different sources. I've found a paper called Semantic Mapping of XML Tags using Inductive Machine Learning.

Among various things, it also talks about a Rule Generation Algorithm called Data Squeezer.
However, I'm wondering what other (better) algorithms can be used for deep learning xml tags.

Also do we have any such algorithm implemented in TensorFlow or any other open source library ?


texts: How does one derive the modified tanh activation proposed by LeCun?

texts:
In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form:

$$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$

They argue that :

It is easier to approximate with polynomials
It is said that it fit the target that it's second derivative is maximal in 1

I tried to start with a function of the form : $f(x) = a * tanh(b*x)$ and derive the value of $a$ and $b$ to match the aforementionned properties.

Any idea of how those constants are derived ? Under what assumptions ? Does it match its expected properties by construction ?


texts: Simple question about prediction classes of item in question vs not item in question

texts:
Let's say I wanted to use transfer learning to train a model to detect object A vs everything else. In this case, do I provide 2 types of input, images of object A and images of everything else, and then have the final layer of the model output either object A or not-object A?

What about in the case where I want object A vs object B vs everything else. Would it make sense in this case to provide images of A and B and then have only two output classes, but based on the confidence of the output, interpret it as 3 classes? Say that it's object A if the confidence in that is > 50%, object B if the confidence in that is > 50%, and anything else if neither of those two conditions are met?


texts: CNN strategy in recognizing spinned images

texts:
I wrote my CNN code from scratch with some convolution kernels. But my CNN can't recognize flipped/spinned images correctly when there are only a few convolution kernels (3*3). My convolution kernels change very little during training. Why?

When there are over 10 convolution kernels, my CNN starts to recognize fipped images. So more kernels help. However it also starts to make wrong recognition.

How will the resolution of images affect the result compared to the convolution kernel size? The higher the resolution, the higher the dimension of this fitting problem


texts: Octave or python for implementing Machine Learning

texts: When is the need to learn octave or Mat Lab for Machine Learning? Are these required if you are proficient in python?


texts: Giving a score (from 1 - 5) using weights

texts:
I have a dataset with 8 features (columns). Each row is a customer, for which I want to give a score from 1-5 using the 8 features for each customer. The range of values are as follows:

Feature 1: 0-19

Feature 2: 0 - 4651107

Feature 3: 0 - 3525

Feature 4: binary (0 or 1)

Feature 5: binary (0 or 1)

Feature 6: binary (0 or 1)

Feature 7: binary (0 or 1)

Feature 8: 0 - 25857

I tried giving each feature a weight (from 0 to 1). So, the most important feature will have a weight of 0.9, then 0.85, then 0.8, etc... Then the score is given by:

  Score = $\lambda_1x_1$+$\lambda_2x_2$+...+$\lambda_8x_8$ where $\lambda$ are the weights for each feature $x$

I need to somehow scale the final scores from 1-5, which I failed on doing. I'd appreciate an opinion on either of the below:

1) If you think my method is flawed due to the ranges of each features, or something else, please suggest a way from scratch to get the scores :)

2) If you think my method of linear combinations works, then please suggest how to scale the scores from 1-5 :)

3) Anything else! Maybe KNN or something? I don't really know haha

Thank you all very much.


texts: What is the purpose of the discriminator in an adversarial autoencoder?

texts:
This is specific to the generative adversarial network (GAN) proposed in A. Makhzani et al. "Adversarial Autoencoders". In a traditional GAN, the discriminator is trained to distinguish real samples in $p(x)$ from fake generated samples output by the generator. On the other hand, the variational autoencoder uses discriminators over the latent codes $z$ instead of the original samples.

What I cannot understand is how the discriminator is able to distinguish between the latent code's prior distribution $p(z)$ and the posterior distribution $q(z)$. What is the reasoning behind the discriminator model attempting to discriminate something that is not a sample $x$, as in the traditional GAN? And should this discriminator be trained before the rest of the autoencoder?


texts: How to Perform Coordinate Constrained Clustering

texts:
Constrained Clustering typically refers to problems where certain pairs need to be included in or excluded from the same cluster.

However the problem I'm dealing with has a constraint on where the center of my clusters can be located.

Suppose we're working on a 2D plane. We have a rectangular circuit surrounded by a number of red points. We'd like to find a clustering algorithm, to group these red points into N clusters, such that in the final output the centroids for all clusters will fall in the circuit region.

I've considered running a large number of k-means iterations until this constraint is satisfied,  but I'd like to ask if there's a more "reliable" approach.


texts: Eliminate low quality predictions in a classification task

texts:
Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like to classify the text manually.

Let's consider the example provided in http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html so that it's reproducible. In the following example, a classification model is trained and is fitted for test documents. One of the test document is - 'what the heck is this?'. I understand that model is returning the class which has highest probability. However, when the model is unsure, I would like to label the text as 'unable to classify'

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

docs_new = ['God is love', 'OpenGL on the GPU is fast', 'what the heck is this?']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, twenty_train.target_names[category]))

Output

'God is love' => soc.religion.christian
'OpenGL on the GPU is fast' => comp.graphics
'what the heck is this?' => soc.religion.christian

Predicting the probabilities

Here are the probabilities of prediction. Documents 1 and 2 have somewhat clear winner. However, the third document doesn't have one. I have about 100 classes and I would be hesitant to set a manual threshold.

clf.predict_proba(X_new_tfidf)
array([[ 0.16297502,  0.03828016,  0.03737814,  0.76136668],
       [ 0.16387956,  0.36874738,  0.2364763 ,  0.23089675],
       [ 0.28288106,  0.17035852,  0.2484853 ,  0.29827513]])


texts: Multivariate time series forecast with VAR confusion

texts:
I am new to time-series forecasting. I am working now on a task in which I have a data set, containing samples of approx. 15 variables for every hour for several years. Then, I have a test data set (continues at the next time step where training data ended) containing values for all the variables except one. My task is to build a model using training data that can predict that one variable in the test data set.
From reading online, I understood I could use vector autoregression (VAR). I have read many tutorials such as this one. I understand most of it except one thing. When it comes to predicting, they (in the tutorials) predict all the variables. However, I would like to do something different: I would like to predict just the one target variable. And of course take into account values of the other variables in the test data set.
To illustrate this, let's say Var Z is my target variable and this is my training set:
       Var X    Var Y     Var Z
 Day 1     11       20       30
 Day 2     22       40       60
 Day 3     33       60       90

Then this is my test set for which I want to predict Var Z:
       Var X    Var Y     Var Z
 Day 4     44       80       ??
 Day 5     55       84       ??
 Day 6     66       88       ??

But in the tutorials I have seen so far, they always predict all variables!
Question: How to specify I want to forecast only a single variable for certain timestamps and take into account values of other variables at those timestamps? Is VAR not the right tool to use?
I would be most grateful if someone could point me in the right direction. I use Python.


texts: Fetch Intermediate "Keras Model -> ResNet50" output in a Keras model "Source Model"

texts:
I want to create a feature extractor model that has the same input as my Source_Model but the output from the intermediate model- "ResNet50". Fetching intermediate layer output is possible but I want output from the intermediate model - layer "avg_pool".

Model: "Source Model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_5 (InputLayer)         [(None, 32, 32, 3)]       0
_________________________________________________________________
resnet50 (Model)             (None, 2048)              23587712
_________________________________________________________________
dense_4 (Dense)              (None, 64)                131136
_________________________________________________________________
dropout_2 (Dropout)          (None, 64)                0
_________________________________________________________________
dense_5 (Dense)              (None, 10)                650
=================================================================
Total params: 23,719,498
Trainable params: 0
Non-trainable params: 23,719,498
*************************************************************************

def source_ResNet(img_shape, num_classes):

  inputs = Input(shape=img_shape)
  base_model = ResNet50(include_top=False, pooling="avg", weights='imagenet', input_shape=img_shape)
  base_model.trainable = False

  x = base_model(inputs, training=False)
  x = Dense(64, kernel_initializer=initializer, activation='relu')(x)
  x = Dropout(0.3)(x)
  predictions = Dense(num_classes, kernel_initializer=initializer, activation='softmax')(x)

  model = Model(inputs = inputs, outputs = predictions, name="Source_Model")
  model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001), loss=tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True), metrics=['accuracy'])

  return model

def source_feature_extractor(input_model):
  model = tf.keras.Model(inputs=mdl.inputs, outputs=mdl.layers[-3].output)
  model.trainable = False
  return model

# "Source_Model" obj is already created
# I am creating Feature Extractor model -> Fetch output from RESNET50 inside my Source_Model

# Calling Feature Extractor Model
new_source_model = source_feature_extractor(Source_Model)
# It works fine, I am getting output from "dense_4" layer.

# How to get output of "resnet50" model ?
def source_feature_extractor(input_model):
  model = tf.keras.Model(inputs=mdl.inputs, outputs=mdl.layers[-4].output)
  model.trainable = False
  return model
# Above doesn't work, following error comes

"""Graph disconnected: cannot obtain value for tensor Tensor("input_6:0", shape=(None, 32, 32, 3), dtype=float32) at layer "input_6". The following previous layers were accessed without issue: []"""

# This command works
new_source_model .get_layer('resnet50').layers[-1].output

But I need to create a Keras model to pass to my other architecture, How can I do this?

Please help!


texts: Multivariate VAR model: ValueError: x already contains a constant

texts:
I have already read this question and the associated answer.

I have removed any 'all zero' columns, as recommended in the answer. I have 3,169 columns remaining.

datavals_no_con = datavals.loc[:, (datavals != datavals.iloc[0]).any()]

I checked whether any were missed, for some bizarre reason:

varcon = np.asarray([np.var(datavals_no_con[datavals_no_con.columns[i]]) for i in range(len(datavals_no_con.columns))])
print np.where(varcon==0.) #empty array.

Also checked the minimum column variance value, which ended up being 4.306x10^(-7)

This was generated by a column that has no zero entries.

When I run this:

model = VAR(datavals_no_con)

results = model.fit(2)

I still get:

Traceback (most recent call last):
  File "vector_autoregression.py", line 163, in <module>
    results = model.fit(2)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 438, in fit
    return self._estimate_var(lags, trend=trend)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 457, in _estimate_var
    z = util.get_var_endog(y, lags, trend=trend, has_constant='raise')
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/util.py", line 32, in get_var_endog
    has_constant=has_constant)
  File "/user/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/tsatools.py", line 102, in add_trend
    raise ValueError("x already contains a constant")
ValueError: x already contains a constant

How can I resolve this?

EDIT: It occurred to me that the problem would be that x contains a constant, not that x contains all 0s. So the original answer suggested in the previous question was not entirely sufficient.

To test whether any of my columns contained 'all the same value' (e.g. a column of all 0.5), I tried this:

ptplist = []
for i in range(len(datavals_no_con.columns)):
    ptplist.append(np.ptp(datavals_no_con[datavals_no_con.columns[i]], axis=0))

ptparray = np.asarray(ptplist)
print any(ptparray==0.) #FALSE

So none of my columns are constant, unless I'm still missing something.

EDIT 2: I have found the root cause of the problem.

Suppose my input matrix (that is, my set of endogenous variables) is a 5x5 identity matrix, for the sake of argument, and that my lag value is 2 (that is, I'm looking for an AR(2) model: y_{t+1} = A + B_1y_{t} + B_2y_{t-1} + error) :

y = np.eye(5)

1 0 0 0 0 (row 1)
0 1 0 0 0 (row 2)
0 0 1 0 0 (row 3)
0 0 0 1 0 (row 4)
0 0 0 0 1 (row 5)

In the get_var_endog function in /statsmodels/tsa/util.py, under lags=2, the y matrix gets rearranged to this general idea:

[row 2, row 1] (i.e. concatenate these two)
[row 3, row 2]
[row 4, row 3]

And this new matrix could have zero columns, in places where my original data matrix did not. In fact, this is exactly what was happening. Following my example, the np.array Z in get_endog_var looks like this:

0 1 0 0 0 1 0 0 0 0
0 0 1 0 0 0 1 0 0 0
0 0 0 1 0 0 0 1 0 0

So now columns 0, 4, 8, and 9 are completely 0, which throws the ValueError.

Two possible approaches to a solution come to mind:

1) Remove the zero columns from the Z matrix.

2) Edit the original data set such that these zero columns never occur in the first place (much harder, because then the Z matrix here would never have existed, so how can you know which columns to remove...catch 22).

I chose option 1, but now I'm dealing with shape issues down the line. Because, of course, when doing the least squares fit, the shape of the parameters is going to be different from the shape of the original data set (some columns don't exist in the parameters, because I removed them, that do exist in the original data set).

Now, this looks like it should be a relatively frequent problem. A lot of the time, we're working with high-dimensional sparse data, which would generate this issue.
Does anyone have a more robust solution than what I've proposed?


texts: Why is correlation between my independent variables helping my linear regression model?

texts: I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three with a new feature : distance covered which is the sum of the above mentioned three features. When putting it in a linearregression model, when I use the three features and the fourth one as well, I get a better score as compared to using the three featues only or using only the fourth feature. I have read that correlation between features when developing a model should not be there. But when all features (4 of them) having correlation are used to develop a model, the model has a better square (R-square). Why is this happening ?


texts: Is it possible to make use of the CPU RAM, if i'm running of of VRAM, in tensorflow?

texts: I want to train a model running on tensorflow. I have a GPU but it only has 6gb of VRAM. So i was wondering if it is possible to use some of the CPU's RAM to offload the GPU? I know it will be much slower, and i can reduce the batch size and number of layers etc. Can it be done?


texts: what's the motivation behind BERT masking 2 words in a sentence?

texts:
bert and the more recent t5 ablation study, agree that

using a denoising objective always results in better downstream task performance compared to a language model

where denoising == masked-lm == cloze.
I understand why learning to represent a word according to its bidirectional surroundings makes sense. However, I fail to understand why is it beneficial to learn to mask 2 words in the same sentence, e.g. The animal crossed the road => The [mask] crossed the [mask]. Why does it make sense to learn to represent animal without the context of road?
Note: I understand that the masking probability is 15% which corresponds to 1/7 words, which makes it pretty rare for 2 words in the same sentence to be masked, but why would it ever be beneficial, even with low probability?
Note2: please ignore the masking procedure sometimes replacing mask with a random/same word instead of [mask], T5 investigates this choice in considerable length and I suspect that it's just an empirical finding :)


texts: Random forest implementation with probability of choosing column or guarantee of choosing set of columns

texts:
Are there random forest implementations which allows for choosing set of columns which will be always selected for every tree in the forest ? or implementations allowing for specifying probabilities of choosing each column ?

Both cases could be simulated by choosing variables (partially or fully randomly) and building tree containing all variables in new temporal dataset, then repeating this procedure until we obtain assumed number of trees and then merging trees into forest,

but there are drawbacks like
moving large number of data in scripting language until they are sent to low level implementation of forest for building tree or lack of merging procedures, which makes work with set of trees harder.


texts: Meaningful predictive analytics for small (n=114) dataset with just 1 explanatory variable and 1 response variable?

texts: I am given an Excel pivot table that aggregates data from a somewhat sizable data source (a database table with 1.9m records and another of about 490k). The data within the Excel file consists of 3 columns: dates of Mondays which represent their respective weeks, quantity of items, and number of shipments (that it takes for the quantity of items). I am supposed to concoct a model that predicts the number of shipments that would be required for a given quantity of items in the future. What models could I implement for such a small dataset with just 1 explanatory and 1 response variable? I know the run-of-the-mill linear regression with a confidence interval would be a start but the data has a dense cluster and then sparse data with some positive correlation. The color bar represents the date (purple is earlier, yellow is most recent)


texts: Predict which user will buy with an offer - discount

texts:
I have historical data from an e-shop transactions. I want to write a prediction model and check if a specific user will buy with or without a discount, so I can do some targeting offers.

The idea is:

If a user will buy the regular price, will not have an offer.
If a user will not buy the regular price, check if he/she will buy with an offer.

With this way, I will avoid to make an offer to someone who would buy with the regular price.

So, I am still in the brainstorming and trying to find a way for implementing the 1-2. Should I create two separate models to predict the 1) and then the 2) with the second model? Or should I join both in one prediction model?


texts: Duplicated features for gradient descent

texts:
Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens?

I guess that we can not find a unique solution because that's the case for the close form in linear regression, but I do not see how to show that intuitively, or even if it is true or not.


texts: How to conclude, regression results can't be improved anymore?

texts:
I am trying to forecast sector's 1 month forward return using macroeconomic variables (170 variables). I tried a few things:

For variable Selection

Run regression with each one of the independent variables and dependent variable separately. Then, using independent variables whose p-values are less than 0.05 to:
Run stepwise regression for variable selection
Finally, run elastic net with best variables (chosen using the above steps i.e., step 1 and step 2)

My observation:
Results (r2) for test set largely depends on how data is split (I don't know why) and even in the best split, r2 is very less (as per my client) in range of 0.12-0.20. How can I either improve this result or conclude that it can't be improved with linear regression?
I have attached a couple of diagnostic plots for independent variables which have the lowest p-values


texts:
A Tableau Workbook I'm working on has different users logging in to see the data for their company. In the dashboard I have created a dynamic image that shows the customer logo depending on who is logged in. I created this by going to the Map drop down and using Background Images. The field that controls which image is seen, is a customer dimension which lists all of the customers in the database.
The issue I'm having is that some users, internal users, can see all of the customers. When these users log in I need the logo to either disappear or show the main company's logo. The customer column doesn't these users listed because they aren't customers. This means I don't have a column to use as a filter.
I followed every step in this website and hopefully it will provide enough of an example since I can't share the original workbook. Here is the website I used to build this: https://www.thedataschool.co.uk/nora-ann-weststrate/tableau-creating-filter-dependent-images/


texts: Reshaping of data for deep learning using Keras

texts:
I am a beginner to Keras and I have started with the MNIST example to understand how the library actually works. The code snippet of MNIST problem in Keras example folder is given as :

import numpy as np
np.random.seed(1337)  # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils

batch_size = 128
nb_classes = 10
nb_epoch = 12

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
nb_pool = 2
# convolution kernel size
nb_conv = 3

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
..........

I am unable to understand the reshape function here. What is it doing and why we have applied it?


texts: Automomatic typographical error correction

texts:
Does anyone know where I might be able to find a list of the most common typing errors and their corrections? This is separate from more complicated considerations concerning general spelling checking (which can have very many candidates in relation to the correct spelling of the word in question); rather I am looking for a similar list as used by Microsoft Word (for instance correcting

  teh

with

  the

or

  becaise

with

  because

Not only can the manner in which these sort of errors are fixed be hard-coded, their frequent occurrence in text provides significant dividends in textual mining (provided that such a list of errors and corrections can be obtained, of course).


texts: Scatter plot for binary class dataset with two features in python

texts:
I have my dataset that has multiple features and based on that the dependent variable is defined to be 0 or 1.
I want to get a scatter plot such that all my positive examples are marked with 'o' and negative ones with 'x'.
I am using python and here is the code for the beginning.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('/home/Dittu/Desktop/Project/creditcard.csv')

now I know how to make scatter plots for two different classes.

fig = plt.figure()
ax1 = fig.add_subplot(111)

ax1.scatter(x[:4], y[:4], s=10, c='b', marker="s", label='first')
ax1.scatter(x[40:],y[40:], s=10, c='r', marker="o", label='second')
plt.show()

but how to segregate both class of examples and the plot them or plot them with distinct marks without separating?


texts: Distributed Tensorflow VS Tensorflow on Spark

texts:
I am new to Deep Learning and Tensorflow. I learned TensorFlow can be code to run in stand-alone mode and in distributed mode. I also came across Tensorflow on Spark framework that will allow the Tensorflow to run with Spark. Which one is good in runtime performance? I didn't find any resource on this point could someone share their experience?
Thanks


texts: How to do feature engineering for email cleaning / text extraction?

texts:
I have a large batch of email data that I want to analyse. In order to do that, I need to first prepare the data, as the messages are quite often >80% noise. Generally speaking, my dataset's structure is nowhere near that of the ENRON dataset. I need to get rid of signatures, headers and, most importantly, automatically appended legal / security disclaimers.

I have been doing some research and so far I've seen two supervised learning approaches to this problem - one using a multilabel sequential learner on a stream of lines; the other using multiple binary SVMs to find lines that open / close a block of text of a particular type (signature, header etc.).

I am confused with the way feature engineering is done in such problems. The papers I've read suggest a set features that mixes pattern matching and some general text processing (e.g.: line length, starting character). It does not seem obvious how these people have arrived at these particular rules for encoding their data. How do I ensure that the features I identify do a good job of generalising my data and do not introduce heavy bias during classification?

Are there some general principles I should follow when trying to come up with a set of features or is it completely dataset-dependent?


texts:
I see some literature consider L2 loss (least squared error) and mean squared error loss are two different kinds of loss functions.

However, it seems to me these two loss functions essentially compute the same thing (with a 1/n factor difference).

So I am wondering if I have missed anything? Is there any scenario that one should use one of the two loss functions?


texts:
This is a citation from "Hands-on machine learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron:

"Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting, but this also means that predictors end up being less correlated so the ensemble’s variance is reduced."

I can't understand why bagging, as compared to pasting, results in higher bias and lower variance. Can anyone provide an intuitive explanation of this?


texts: How to extract a subset from a data frame?

texts:
Let's say I have a dataframe that contains several columns including a column called A (containing integers).

I would only like to consider the rows of table that correspond to $A \ge 3$.

What would be the syntax for this?


texts:
I am reading notes on using weights for KNN and I came across an example that I don't really understand.

Suppose we have K = 7 and we obtain the following:

Decision set = {A, A, A, A, B, B, B}

If this was the standard KNN algorithm we would pick A, however the notes give an example of using weights:

By class distribution (weight inversely proportional to class frequency)

class A: 95 %, class B 5 %

This results in a class of B.

I can't seem to figure out the math that was left out to obtain B as the answer.


texts: Help training Back-propagation Neural Network With 150k training pairs

texts:
I am currently trying to train my backpropagation to classify 150k training pairs. Each training pair is a vector of 18 Bipolar numbers and it runs through 2 hidden layers with a final output of 1 number(18-18-18-1).

When I feed my neural network with only a couple thousand training pairs it can classify them 100% but when I try to feed more it makes many errors. Does anyone have any recommendations for what can help a back prop net handle more training pairs? Should I do batch updating of weights or update them for every training pair(what I do now)? Also whats the best way to test for convergence with so many training pairs(when summing up error it's ~40k).Any help is much appreciated.

(FYI: each training pair is a loan account I am trying to evaluate with data on balance, age, how long it's been open, average income of zipcode and the target is whether or not it was paid)


texts: Is it feasible to train a single model on 150+ classes?

texts:
I have classes for different regions. (Let say 80 classes for 3 regions each).
Will it be ok if I train my CNN model with 240 classes or should I create 3 models for each region? The classes for each region are similar to the classes of other region with same name but there is some difference in the elements of each class from each region and hence given some other class name.

I currently have one trained model which gives good accuracy for one region with 80 categories. Now that I have to move with remaining classes, it will be time-saving if I know beforehand to train one single model with 240 categories or three different models with 80 categories each.

I have 300 images for each class on average.


texts:
I'm trying to solve a kaggle competition
Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using:
# Process individual chunks to normalize JSON columns and remove unwanted columns
def process_chunk(df):
    df['hits'] = df['hits'].apply(literal_eval)
    df['hits'] = df['hits'].str[0]
    df = df[pd.notnull(df['hits'])]

    df['customDimensions'] = df['customDimensions'].apply(literal_eval)
    df['customDimensions'] = df['customDimensions'].str[0]
    df = df[pd.notnull(df['customDimensions'])]

    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource', 'hits', 'customDimensions']

    for column in JSON_COLUMNS:
        column_as_df = json_normalize(df[column])
        column_as_df.columns = [f"{column}.{subcolumn}" for subcolumn in column_as_df.columns]
        df.drop(column, axis=1, inplace=True)
        df = df.merge(column_as_df, right_index=True, left_index=True)

    # remove some columns to reduce size
    # we only need a few hits columns, I suppose
    want_hits_col = ['hits.type', 'hits.hour', 'hits.hitNumber', 'hits.type']
    all_hits = df[df.columns[pd.Series(df.columns).str.startswith('hits')]].columns
    hits_columns_remove = list(set(all_hits) - set(want_hits_col))

    df = df.drop(hits_columns_remove, axis=1)
    return df

# divide the data into chunks and save processed chunks incrementally
def save_parse(csv_path, chunksize=500):
    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']

    chunks = pd.read_csv(csv_path,
                         converters={column: json.loads for column in JSON_COLUMNS},
                         dtype={'fullVisitorId': 'str'},  # Important!!
                         chunksize=chunksize)
    chunk_counter = 1
    for chunk in chunks:
        processed = process_chunk(chunk)
        # show me the progress
        print("Processed chunk number - {} with shape {}".format(chunk_counter, processed.shape))
        chunk_counter += 1
        # save to csv
        processed.to_csv('train_chunks_appended.csv', mode='a')
    print("Done parsing")

For the first chunk, the process_chunk method outputs current value (df.shape = (211, 59)). Stragly, for second chunk onwards, chunk is not processing properly.
I think there is a problem at merging the normalized json columns back to dataframe at this line:
df = df.merge(column_as_df, right_index=True, left_index=True)

which is why the shape becomes (0, 27) for rest of the chunks. I am not able to figure out why this is happening. Any pointers would be great.


texts: Which one I should choose for random forest?

texts:
I fit a dataset with a binary target class by the random forest. In python, I can do it either by randomforestclassifier or randomforestregressor.

I can get the classification directly from randomforestclassifier or I could run randomforestregressor first and get back a set of estimated probabilities. Then I can find a cutoff value to derive the predicted classes out of the set of probabilities. Both methods can achieve the same goal (i.e. predict the classes for the test data).

Also I can observe that

randomforestclassifier.predict_proba(X_test)[:,1])

is different from

randomforestregressor.predict(X_test)

So I just wanna confirm that both methods are valid and then which one is better in random forest application?


texts:
I'm trying to impute 1D array with shape (14599,) with simple imputer with most_frequent strategy but it said it expected 2D array, i already tried reshaping it (-1,1) and (1,-1) but its error ValueError: could not broadcast input array from shape (14599,1) into shape (14599) how can i impute this since reshaping wont solve the problem? i dont understand why it throws error

##libraries
import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

##codes
plt.close('all')
avo_sales = pd.read_csv('avocados.csv')
avo_sales.rename(columns = {'4046':'small PLU sold',
                            '4225':'large PLU sold',
                            '4770':'xlarge PLU sold'},
                 inplace= True)

avo_sales.columns = avo_sales.columns.str.replace(' ','')

plt.scatter(avo_sales.Date,avo_sales.TotalBags)

x = np.array(avo_sales.drop(['TotalBags','Unnamed:0','year','region','Date'],1))
y = np.array(avo_sales.TotalBags)

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

impC = SimpleImputer(strategy='most_frequent')
X_train[:,8] = impC.fit_transform(X_train[:,8].reshape(-1,1)) <-- error here

imp = SimpleImputer(strategy='median')
X_train[:,1:8] = imp.fit_transform(X_train[:,1:8])

le = LabelEncoder()
X_train[:,8] = le.fit_transform(X_train[:,8])


texts: bad regression performance on imbalanced dataset

texts:
My current dataset has a shape of 5300 rows by 160 columns with a numeric target variable range=[641, 3001].

That’s no big dataset, but should in general be enough for decent regression quality. The columns are features from different consecutive process steps.

The project goal is to predict the numerical variable, with the satisfactory object to be very precise in the area up too 1200, which are 115 rows (2,1%). For target variables above 1200 the precision can be lower than in the area [640, 1200]. The target-variable is normally distributed with its mean ~1780 (25%: 1620, 75%: 1950) and variance of 267.5.

prediction vs actual:

residual plot:

My problem is (see plots above), that no matter what I try, the range of predictions (y_hat) is very limited and rather random (Training RMSE ~300, Test RMSE ~450), best test-mean-abs-error for y-values <= 1200  ~= 120.

I’ve already tried:

feature cleaning
process step wise addition of features to compare model performance/information gain
feature generation
derive new features (by business logic)
generate features

cross-product of features
differences to previous rows
differences between features
differences per feature to mean
durations based on timestamps

normalizing, scaling
log-transformation of target variable
Over- &/ Under-Sampling
various algorithms (using GridSearchCV for hyper-parameter tuning):
sklearn [SVR, RandomForrestRegressor, LinearRegression, Lasso, ElasticNet]
xgboost
(mxnet.gluon.Dense)

What would be your approach? Do you have any advice what technique I could try or what I've probably missed? Or if it's more likely that the training data simply doesn't fit well on the target variable?


texts: U-Net doesn't work with images different from the dataset

texts:
I have implemented a very similar U-Net code from github, but for a different dataset, this one, to segment roads, it works fine using the test folder images, but when i for example, pick a print from bing maps and try to infer with the trained model, this is returned:

Why this is happening?
Already tried change the thresholding values, normalization and etc.
Tensorboard


texts: How can we use the cosine similarity formula on document feature vector without a direction?

texts:
In mathematics, a vector has both magnitude and direction.

In data science, for identifying document similarity we convert the document into a feature vector. Then apply cosine angle formula between the source and target document's feature vector.

However the cosine formula is applicable only for vectors. And a vector should have both magnitude snd direction. For a document that is represented as a vector, where is the direction?


texts: Sensor fusion using recurrent neural network: obtaining a smoothed output

texts:
I am trying to use a recurrent neural network to perform sensor fusion for an inertial measurement unit. IMUs are commonly used in conjunction with a Kalman filter (KF), which performs both fusion of accelerometer and gyroscope data along with 'smoothing', which results in a filtered, final output of orientation (and position if desired). Out of curiosity, I am trying to replicate this functionality using a neural network containing LSTM units: i.e., get the network to compute the hidden representation that Kalman filters estimate using training data. In order to get the network to replicate the working of a Kalman filter fusion algorithm, my first step was to train it on inputs of raw sensor data and the output coming from a KF.

The network takes 6x10 + 6 = 66 inputs (3 velocities from gyro, 3 accelerations from accelerometer from the present instant along with those from the past 10 instants of time) and aims to provide 4 outputs (quaternion representation of orientation). The network structure is

Input -> LSTM units (128) -> Dense layer (128) -> Dense layer (64) -> Output(4)

My initial training data consisted of noisy sensor data from the gyro and accelerometer in the input and smoothed KF orientation data as output, from which I was expecting the network to attempt to learn the nonlinear mapping between the sensors and the orientation. Yet, although the output I am training on is clean, there is a lot of noise in the network's predictions, even on the training data. For example:

Kalman filter output (used as output in training data):

Corresponding predictions from neural network:

The noise I am seeing in the predictions is not unnatural, because the input sensor data is very noisy as well, and the amount of noise in the output corresponds to that of the inputs: but is it not right to expect the neural network to 'learn' the smoothing as well, along with the fusion? Is there any way to reduce the noise and get the network to perform closer to the way the KF does?


texts: SQL Oracle - Excel's BETAINV function (Inverse cumulative function of a beta distribution) in SQL Oracle

texts: Is there a function equal to Excel's BETAINV(probability,alpha,beta,[A],[B]) in Oracle SQL Developer? Or a similar one which I could use to return the inverse cumulative distribution function of a beta distribution?


texts: Find similar observations in two datasets

texts:
I have two datasets A and B. What I would like to do is for each observation in A, I would like to find 5 observations from B that are closest and match to A.

How should I start?

Thank you for your help!


texts: What models are used to get the Next Best Action to convert a physician from a non writer to a write?

texts:
Assume that a sales rep has actions A, B, C, D.... X that he can perform to influence a physician to write his drug

I want to predict what the next best action a sales rep should take to have the highest impact on a physician

For example, if a sales rep has already performed A, D, F then i want to predict the next best action for the sales rep to perform to convert the physician to a writer

What models/algorithms are suitable for this type of a problem?


texts: Is it necessary to perform rolling-window on LSTMs?

texts:
Let's say I have a set of n time-series with sequence length 8

  [[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]]

And let's define the input that LSTM expects as a tensor of shape (samples,sequence-length,features)

I want to predict the last value of each one of them.

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

  [a,b,c,d,e,f,g] -> [h]

In this case the tensor would be (samples,7,1)

Or I can do the classic rolling window, with a window size of, say, 2.

  [a,b],
  [b,c],
  [c,d],
  [d,e],
  [e,f],
  [f,g] -> [h]

In effect this shortens the length of the sequence.

And the input tensor would be (samples,2,1).

What are the trade-offs between performing rolling-windows or giving the "crude" time-series to the LSTM?


texts: Joining two dataframes on the basis of specific conditions

texts:
I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions:

Condition 1: The element in the 'arrivalTS' column in the first dataframe(flight_weather) and the element in the 'weatherTS' column element in the second dataframe(weatherdataatl) must be equal.

condition 2: The element in the 'DEST' column in the first dataframe(flight_weather) and the element in the 'place' column in the second dataframe(weatherdataatl) must be equal.

I tried the joins function but wasn't able to add both the conditions to it. it will be helpful if you could help me join them with the join/merge function. Thanks in advance.

dataframe]3]3


texts: Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?

texts:
Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2.0. This input pipeline meets my needs when data is small enough to fit a PC's RAM.

Now I have some large datasets, more than 10GB, some are larger. I also plan to deploy the models in a production environment, which means there will be new data coming everyday. For DNN model training there is distributed strategy of tensorflow 2.0. But for data preprocessing obviously I cannot use pandas, scikitlearn on the large datasets with one PC. It seems to me I need to use a for-loop where I repeatedly fetch a small part of the data and use it for training?

I am wondering what do people typically use in either experiment or production environment for big data preprocessing?
Should I use Spark(PySpark) and Tensorflow input pipeline?


texts:
I'm familiar with the Expectation-Maximixation algorithm and, until now, I thought it was the only way to maximize the likelihood of the observed data, assuming a Gaussian mixture model.

In the last days while studying for Dirichlet Process I've discovered the Gibbs Sampling that can fit parameters of a GMM.

Now I'm not familiar at all with the Gibbs sampling, and I would really appreciate if you could give me an overview of it, comparing it to EM. What is the basic idea behind it?


texts: Automatically extract original dataset from a line plot image

texts:
Given a line plot image (bitmap, png, etc) form, I'm looking to extract the set of original datapoints without any user input, as shown in the below image:

The outcome should be something like below:

`-  A,B,C
x1: 1,2,3
x2: 2,4,9
x3: 3,8,27
x4: 4,16,23
x5: 5,13,18
x6: 6,18,21
x7: 7,19,23`

Solutions like webplotdigitizer require user input which isn't what I'm looking for.  scatteract is very close but I don't think it works on using line plots.

Can anybody point me to papers or open source projects on this?

Thanks in advance.


texts:
Is there a way to search Kaggle for a list of current and prior challenges, or an external site that does that?
When I search on Kaggle it will only bring up solution notebooks and datasets, it doesn't seem to have a filter on challenge page.


texts: Sparsify meaningful data following a Gaussian distribution

texts:
Let's say I have 1-D data following a Gaussian distribution. I want to extract from this database the meaningful information, that is the information that lies far away from the mean.

One way to do that is to apply a function (e.g. $x \to |x-\mu|$ where $\mu$ is the mean of the distribution) and sort the values I get.
But I want to keep track of how meaningful the data is as I am collecting it all along this process (as I do the bottom-up approach : I gather the samples one by one and create another (smaller) database). Is there an intuitive measure of this concept ?

I thought about comparing the histograms between the initial database and the one I create as I gather the samples. For example, the Wasserstein metric gives us a way to compare two probability distributions. An intuitive effect of the extraction I am making would be that the distance between the histograms does not decrease by much at first, as I only fill the error that does not contribute much to the distance. Is that a good approach ?

Next level : If my data is made of groups of samples (still, the entire database follows a Gaussian distribution), what would be the intuitive way to select groups that have meaningful information ?

I though about giving each group a weight that is the average of the variance of each sample that is in the group, and then sort the groups with the weights they have. But does this really keep track of the useful information ?

Next Next level : What about a n-D data following a (multivariate) Gaussian distribution ?

This is kinda theoric. If you have any questions feel free. Any article on the matter is also appreciated.


texts: How to compare different similarity measurements in text clustering?

texts:
I have a dataset which contains vectors generated from subtitles (each column represents a genre, each row is a movie name), my purpose is to find the most similar movie titles, I want to use different distance/similarity measurements and compare them, what is the best method to use?

For now, I tried L1 distance, cosine similarity, Euclidean distance, Mahalanobis Distance, I got the results of top n most similar titles, but all the results seem very reasonable, how can I compare them to see which method perform best?

I also tried to do k-means, when I implement K-means clustering, it used Euclidean distance by default, how to use other distance to implement K-means?Also any suggestions about other similarity measurements? Many thanks


texts: Strange plot on a simple dataframe Rstudio

texts:
I don't usually use R and I'm having trouble with a simple plot from a dataset. I just wanted to make a scatterplot and look at the values, but it keeps ploting a strange "box" thing which I don't really know if it's an error or if I'm missing data manipulation to make it work.
install.packages("remotes")
remotes::install_github("brunoruas2/gujarati", force=TRUE)
data(package = .packages(all.available = TRUE))
data(package = 'gujarati', Table3_6)

#Exercício 3.20

data(package = 'gujarati', Table3_6)
df <- Table3_6

#Nomes das colunas
names(df)[2] = 'Produçãox1'
names(df)[3] = 'Produçãox2'
names(df)[4] = 'Remuneraçãoy1'
names(df)[5] = 'Remuneraçãoy2'

summary(df)
View(df)

# Plot 1
plot(Remuneraçãoy1 ~ Produçãox1, data=df)


texts: Does the SVM require lots of features most of the time?

texts:
So I know about the curse of dimensionality (too many features too less data).

Say I have a 3000 sample dataset, would 3 features be too less?


texts: How can I create a pixel labelled image for Semantic Segmentation?

texts:
I am following the Semantic Segmentation Examples tutorial by MathWorks.

I understand that I can load pixel labeled images

pxDir = fullfile(dataDir,'buildingPixelLabels');

Define the class names.

classNames = ["sky" "grass" "building" "sidewalk"];

Define the label ID for each class name.

pixelLabelID = [1 2 3 4];

and create a pixelLabelDatastore.

pxds = pixelLabelDatastore(pxDir,classNames,pixelLabelID);

But, how do I create a custom pixel labelled image where every pixel value represents the categorical label of that pixel?

I would then proceed by writing:

pxDir = fullfile(dataDir,'myCustomPixelLabels');

If I understood correctly, imageDatastore holds the actual image and not the pixel labels for that image.

EDIT:

On my system pxDir points to 'C:\Program Files\MATLAB\R2017a\toolbox\vision\visiondata\buildingPixelLabels'. Since I am on Matlab2017a so this example is not included by default, and I cannot compare or view the file to get a better understanding of what I need to do to reproduce this example.


texts: How are data in tensorflow.examples.tutorials.mnist formatted?

texts:
I am analyzing a sample code that implements a Restricted Boltzmann Machine (RBM) using Tensorflow. The input data seems to be the good old MNIST, except that apparently, it is now available in Tensorflow itself.

So, instead of running this sample code on MNIST, I want to run it on my own data. But the problem is that I do not know how

  tensorflow.examples.tutorials.mnist

has been formatted. Is it like a CSV file? If I want to feed my own data, which comes from Wizemann's horse images, how should I format my data that is compatible with how MNIST has been formatted?

I'm very new to this stuff. So, please enlighten me if I am mistaken or my question is absurd.


texts:
I am trying to understand the code here.
The output [12] shows that the model accuracy is above 90% even for the validation set, but the confusion matrix in [16] ca not even achieve 50%  accuracy, and it is also on the validation set, so I do not understand this low accuracy on the confusion matrix. I think it may be due to data augmentation, but I would be thankful if someone could explain it to me and tell me how I could then have a confusion matrix in adequacy with the learning curves. Thanks in advance.


texts: How to use "related" and "unrelated" as classes rather than multiple classes?

texts:
I have a dataset with about 15 feature columns and about 1000 rows that I'd like to use for supervised training.

Every row can be classified as "related" or "unrelated" to another row. About fifteen rows are "related" to one another out of the 1000.

I want to predict whether a row is related or unrelated based on the 15 feature columns.

But my issue is how to do this with classification. It's not really a multi-class problem. I don't want to predict whether something is a member of a specific class based on a featureset. Instead, I want to predict whether a row is "related" or "unrelated" to another row based on the feature set. Is it possible to do this with random forest or gbms?

It's not obvious to me how I'd do this because I can't just have a column called "related" with a binary variable. It seems like what I might need to do is to label each group with a specific class name, then use that as a response. But again, I don't want to predict class membership, just related/unrelated. So maybe I need to rewrite my random forest to search for class membership, but in the course of binary prediction?

In short:

I either end up with one 1000 row dataset with 15 "related" and 985 "unrelated" or I end up with 1000 rows, with 66 different classes. What I really want to think of it as is 66 different datasets with 1000 rows and 15 "related". But I don't know how to use online learning or update a classifier with Random Forest / gbm. So thinking I may need to do a custom net here. But looking for other ideas..


texts: Transfer Learning and Recommender Systems

texts:
I have a task in which I am pretending to have an "unobserved" system, let's call it the target system, that I am using an LSTM from a similar system that has observations to perform the regression. I call these observed systems the source systems.

However, to find the most similar source systems to use I must use some metadata about the systems that are available even for the unobserved targets. The idea is that we can find similarities in the metadata to indicate which source systems will perform well.

To do this I assume I need to create some kind of similarity or distance function on the metadata. Since I don't have too many source systems (about 70), I'm guessing the algorithm must be simple. I'm thinking linear regression on the differences in metadata between the source and the target. If so, how would you define a similarity function based on this regression? Is regression even the best way to go about this?

I'm not too knowledgeable about recommender systems but would this be a case where we would want to "recommend" a source system?

If there are resources about these kinds of problems available please let me know as well, I can't seem to find much in the few transfer learning survey papers I have read.


texts: How to add noise to supervised (binary-classifier)?

texts:
Note: The question is not about validating/testing a trained model.

Say i have an unlabeled features set, I want to approximate the true labels (for the sake of argument lets assume it's a binary classification prob).

I also have a trained model to predict the labels.
Now I want to use this model (since it is trained on the true labels, it should approx. my distribution)  along with some added noise (maybe gaussian) to generate the labels for this unknown dataset.
This should always result in a fixed effect size.

How can I do that? My features are mix of categorical and continuous values.

Would be awesome to illustrate the additive noise via python numpy package.

Apologies, I am a rookie!


texts: Don't understand classification equation for hard margin SVM

texts:
I am trying to get a grasp of hard margin SVMs. In the lecture I am watching the professor talks about a classification equation which when a positive sample is input, returns a value of $1$ or more; and when a negative sample is input, returns a value of $-1$ or less. The graph below shows the vector $\overrightarrow{w}$ which is perpendicular to the separating hyperplane and an arbitrary point with unknown class $\overrightarrow{u}$.

In the lecture it says that:

$$\overrightarrow{w} \cdot \overrightarrow{u_+} + b \geq 1 \textrm{ , for positive class samples}$$

$$\overrightarrow{w} \cdot \overrightarrow{u_-} + b \leq -1 \textrm{ , for negative class samples} $$

To me it seems that we should take the projection of $\overrightarrow{u}$ on $\frac{\overrightarrow{w}}{|\overrightarrow{w}|}$, since this would give the component of $\overrightarrow{u}$ in the direction of $\overrightarrow{w}$. If this component is greater than the distance to the decision boundary/hyperplane, $b$, then $\overrightarrow{u}$ is a positive sample, if it is less then it is a negative sample. In math terms

$$\frac{\overrightarrow{w}}{|\overrightarrow{w}|} \cdot \overrightarrow{u_+} - b > 0$$

$$\frac{\overrightarrow{w}}{|\overrightarrow{w}|} \cdot \overrightarrow{u_-} - b < 0$$

If $\overrightarrow{u}$ lies on the decision boundary then the above expressions will be equal to 0.

If $\overrightarrow{u}$ lies on a support vector hyperplane, then the above expressions will be equal to $m$ or $-m$ for positive and negative samples respectively, where $m$ is the margin from the decision boundary to any support vector.

I don't understand the lecturer's equations. Why is the value for positive sample classification $\geq 1$? Why is the value for negative sample classification $\leq -1$? Furthermore, what does $b$ represent in the lecturer's equations?

Basically, I have trouble understanding the proposed equations, but they should be doing the same thing as mine.


texts: Which algorithms or methods can be used to detect an outlier from this data set?

texts:
Suppose I have a data set : Amount of money (100, 50, 150, 200, 35, 60 ,50, 20, 500). I have Googled the web looking for techniques that can be used to find a possible outlier in this data set but I ended up confused.

My question is: Which algorithms, techniques or methods can be used to detect possible outlier in this data set?

PS:Consider that the data does not follow a normal distribution. Thanks.


texts: How to improve precision under imbalanced classification

texts:
I am using an imbalanced dataset (rare positive cases) to learn models for prediction and the final good AUC is 0.92 but the F1 score is very low0.2.

Is it possible to add some key features which will change the class probabilistic distribution and thus we can get a threshold to generate a higher F1?

Here is an example:

In my original model, I get a class probabilistic distributions shown below:

I can adjust the threshold to make a better precision but meanwhile cut some recall off. It is due to the large overlapping area between two distributions.

Then I use an extreme dataset, i.e. include the target itself as a feature to learn. As a result, I can see I split the distribution completely disjointed.

Dose it mean if I introduce a strong feature, I can split the distribution to some extent and thus promote the precision and thus f1 score? Or please advise how to improve precision under imbalanced classification issue.

Many thanks


texts: Validation Loss is not decreasing - Regression model

texts:
I'm trying to train a regression model with 6 input features.
No. of tuples - 7287.
Train set - 5465
Test set - 1822

I've tried changing no. of hidden layers and hidden neurons, early stopping, shuffling the data, changing learning and decay rates and my inputs are standardized (Python Standard Scaler).
Validation loss doesn't decrease.

NN_model = Sequential()

NN_model.add(Dense(128, kernel_initializer='normal',input_dim = x_train.shape[1], activation='relu'))

NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))
NN_model.add(Dense(128,kernel_initializer='normal', activation='relu'))

NN_model.add(Dense(1, kernel_initializer='normal',activation='linear'))
#adam=optimizers.Adam(lr=0.01,decay=0.0001)

NN_model.compile(loss='mse', optimizer='adam', metrics ['mean_absolute_error','mse'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
NN_model.summary()
history=NN_model.fit(x_train,y_train,epochs=1000,batch_size=64,validation_data=[x_test,y_test],callbacks=[es])

Loss graph looks like:

Mean square error is very high and r2 score is 0.5276 for the train set and 0.3383 for the test set.

I've tried other machine learning models like Gradient Boosting Regressor, Random forest regressor, decision tree regressor but they all have high mean square error.

I'm new to keras and deep learning. How do I solve the issue?

Is this amount of training data enough for the neural network? I can't get more data.


texts: to include first single word in bigram or not?

texts:
in a text such as

  "The deal with Canada's Barrick Gold finalised in Toronto over the
  weekend"

When I try to break it into bigram model, I get this

"The deal"
"deal with"
"with Canada's"
"Canada's Barrick"
"Barrick Gold"
"Gold finalised"
"finalised in"
"in Toronto"
"Toronto over"
"over the"
"the weekend"

my question

shall I include the first word and the last word as single words

"* The"
"Weekend *"


texts: Recurrent Neural Network +LTSM to find the text with High Probability Value

texts:
Dataset contains the features such as description, goal , category etc to predict the probability in decimal such as 0.1,0.8 etc,

Now in the next step need to find out the text words associated with high probability using RNN +LTSM or any other neural network

Please guide with the key steps or the reference sample

Regards,
Swati


texts: What is the difference between tensorflow saved_model.pb and frozen_inference_graph.pb?

texts:
I've re-trained a model (following this tutorial) from the google's object detection zoo (ssd_inception_v2_coco) on a WIDER Faces Dataset and it seems to work if I use frozen_inference_graph.pb from python, but if i take saved_model.pb and put it to tensorflow serving, it predicts a lot of detections all with confidence less than 0.10.

Another thing that I do not understand is that both files are 52Mb however the original files downloaded from the object detection zoo take 98Mb each.

Here's an example of predictions made with the saved_model and tensorflow serving:

And here's example of predictions by the same model, but using frozen_inference_graph.pb:

So what is the actual difference between the saved_model.pb and the frozen_inference_graph.pb?
Are there any ways to inspect what's wrong with the saved_model.pb i use?
How the much smaller size of the re-trained model vs the original could be explained?


texts: temperature variable in boltzmmann-exploration in reinforcement learning

texts:
I have been using epsilon greedy action selection strategy and recently have come across boltzmann(softmax) action selection strategy. One thing I am not clear about boltzmann exploration is the temperature variable. How should we define this variable. Is this a constant variable or should be decreased over the period of training. and how to decide on the absolute value of this parameter?

Thanks


texts: Building a text classification model from scratch

texts:
I do not know if this is the right place to post my question.
I am a beginner in data science and machine learning techniques. I would need to build a modem that can allow me to classify texts and run some sentiment analysis.
Right now I have only data collected and they miss any information about sentiment analysis.

Data collected (texts) are approximetely 50000 and they where already cleaned of punctuation and stop words.
I heard about the possibility to build some neural networks or use a logistic regression, but I do not know anything about specific models to use for that.

Furthermore, I might consider to build a new model from scratch (I know that can take ages and a lot of efforts) but to do it ai would need to know what I would need (for example, already existing model/dataset to train with dictionaries and sentiment analysis).
The problem is that what inwould like to do is to classify a text as positive or negative (sentiment); also, classify it as fake or not fake.

Do you have any suggestion or advice?
If you need further information, I would be happy to provide them.


texts: Maximum Dimensionality of AWS Machine Learning

texts:
I am toying with AWS Machine Learning and I have a dataset with about 200 records with about 220,000 variables each.
Apparently, AWS Machine Learning has a hard limit of 1000 variables.
How can I still analyse my dataset through AWS ML? Should I apply a MDR procedure through my 200 x 200000 dataset or should I split my variables into chunks of 1000 and combine them somehow after processing each of those chunks through AWS ML?


texts:
As my title already says: can the ELBO be negative?

$ELBO_\lambda = KL[q_\lambda(w)||P(w)] - \mathbb{E}_q[\log P(\mathcal{D|w})]  $

Can I theoretically adjust my prior $P(w)$ such that $KL=0$ and also $\mathbb{E}_q>0$? If yes, why call it a lower bound on evidence? Since $P(\mathcal{D}) \geq 0$ per definition and a negative lower bound has no information.


texts: How would I map categories between two similar but different sets of categories

texts:
Having multiple sets of categories from different listings sites (e.g. Yelp, yellowpages.com,  Google My Business...).  I want to figure out what category X on one site is on another site.

We have hundreds of thousands of businesses and the categories on all the sites they are on, so we could see that "Galbi Foo Restaurant" is in category "Restaurants > Korean" on one site and "Restaurants" on the other.

Some examples category mappings that will have to happen:

Nail Salons = Manicure & Pedicure
Eyelash Service = Visagist
Tanning = Sunbed Salon
Specialty Food = Grocery (Specialty Food child node doesn't exist)
Diagnostic Imaging = Radiologist

Where would I start to solve this?  It seems like a classification (logistic regression) problem.  But this ML stuff hasn't clicked with me yet, so I'm likely to drastically over or under complicate these things :).


texts: correcting conditional and marginal distribution in transfer learning

texts: I understand that in case of transfer learning, we can have the target and the source data having different domain distributions. In such cases, authors in many papers suggest bringing the marginal and conditional distributions of the target and the source closer, i.e, minimize the difference between the marginal and conditional distributions. Can someone please help me understand this by giving an intuitive explanation for this? I am unable to understand what exactly the author means when he says by bringing the distributions closer? Explanations using visual representations would be helpful.


texts: How to Transpose dataframe with custom output?

texts:
I have a data like this:

I want to transpose it like this in python:


texts: Why is doing a ShuffleSplit with GridSearchCV decreasing the performance of my model?

texts:
I'm using the Boston Housing dataset in Keras, and neural networks for this regression problem.
The following is the code I use to prepare the data, build the model, and fit it with GridSearchCV.
from keras import models
from keras import layers
import numpy as np
from sklearn.model_selection import GridSearchCV
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import ShuffleSplit

from keras.datasets import boston_housing
(train_data,train_targets),(test_data,test_targets)=boston_housing.load_data()

mean=np.mean(train_data)
std=np.std(train_data)

train_data_norm=(train_data-mean)/std
test_data_norm=(test_data-mean)/std

def build_model():
    model=models.Sequential()
    model.add(layers.Dense(64,activation="relu",
                          input_shape=(train_data_norm.shape[1],)))
    model.add(layers.Dense(64,activation="relu"))
    model.add(layers.Dense(1))
    model.compile(optimizer='rmsprop',loss="mse",metrics=["mae"])
    return model

model=KerasRegressor(build_fn=build_model,epochs=30,verbose=0)

param_grid = {"epochs":range(1,11)}

grid_model=GridSearchCV(model,param_grid,cv=4,n_jobs=-1,scoring='neg_mean_squared_error')

When I run
grid_model.fit(train_data, train_targets)

mean_squared_error(grid_model.predict(test_data),test_targets)

several times, the performance seemed to vary a lot, with the MSE ranging from 60 to 90.
So, I thought that if I used ShuffleSplit, then maybe I could stabilise the scoring... For that, I've used this code:
ss = ShuffleSplit(n_splits=4, test_size=0.1, random_state=0)

grid_model=GridSearchCV(model,param_grid,cv=ss,n_jobs=-1,scoring='neg_mean_squared_error')

grid_model.fit(train_data, train_targets)

mean_squared_error(grid_model.predict(test_data),test_targets)

However, now the MSE is even bigger! I've ran the last statement several times and I get an MSE ranging from 110 to 190. Not only it didn't stabilise, it also got worse. Why is that? Is my code incorrect somehow?


texts: What is an appropriate way to compare classifiers with different sets of classes?

texts:
I have three classifiers for language identification:

A: en, de, ru, fr, ij, kl
B: en, de, ru, fr, xy
C: en, de, ru, fr, no, pq, rs

and I have a balanced dataset which matches the classes of A.

What is a fair way to compare those classifiers?

My thoughts

Accuracy on classes of A: Not fair, because B can't possible recognize ij, kl and will likely make mistakes due to the fact that it knows language xy. (Similar for C)
Smallest common subset: Possible, but not so interesting as those are the "easy" classes.
Precision: Given a language the classifier knows, how often does it actually recognize it?

Unfair for C, as it has more possibilities to make mistakes. Probably this could be somehow compensated? E.g. if it recognizes a class which is not in A, just the next best is taken (until the correct class is predicted or a wrong class from A is preditected)

There could also be a possibility to use the classifier A in combination with another classifier. The weight / importance (however that is measured) of the contribution of the other classifier is its score.

Are there any publications which do something similar?


texts: Keras difference beetween val_loss and loss during training

texts:
What is the difference between val_loss and loss during training in Keras?

E.g.

Epoch 1/20
1000/1000 [==============================] - 1s - loss: 0.1760, val_loss: 0.2032

On some sites I read that on validation, dropout was not working.


texts: MNIST digits... And what technique to find ZIP code on the envelope?

texts:
What techniques are currently available for e.g. locating the ZIP code on the envelope and cutting it to single digits prior to feeding them to digit recognition algorithm (trained on MNIST dataset)?

Or imagine a task of automatic recognition if a phone call recording contains a sound of a train passing a track switch (just yes or no). If present at all, it can be gentle or loud, depending if a window was open. It can be slow or fast, depending how quick the train was going through the switch.

I found so far Spatial Transformations Networks mentioned in this comment


texts:
Thanks also to SE, I've recently changed job and now I'm working in Data Science, mainly on Analytics for the IoT (Internet of Things). Analytics are applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc..

Until now, I've been using R to build Statistical Learning models on datasets which would fit in the memory of my workstation, so I'm not a novice for what it concerns Statistical modeling and Data Science. However, I'm a novice with Python, and I need to learn it, expecially the part of the Python ecosystem related to Data Science, because that's what my team uses.

I don't need to develop the cloud platform: I just need to develop the "core" Analytics. I got the book by Jake Van der Plas:

https://www.amazon.com/Python-Data-Science-Handbook-Essential/dp/1491912057

but I would like to also follow a MOOC on using Python for Data Science. Can you suggest me one?

DISCLAIMER I already asked this question on CV and it wasn't considered very appropriate there. Since here my other question on MOOCs

MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks

was well-received on this site, I thought of asking again here, after deleting the one on CV (no cross-posting). Hope this is fine.


texts:
I have read about linear regression and interpreting OLS results i.e coefficients, t-value, p-value. But unable to find any material related to covariance matrix in linear regression.

I was reading about assumptions in linear regression, came across the term heteroscedasticity and was researching about its consequences. So while going through one of the answer on Stack Exchange about consequences of Heteroscedasticity I came across the term covariance matrix. Here's a link to that StackExchange

What is covariance matrix and how should one interpret it?


texts: Whole time column in csv file convert into UTC (epoch) using python

texts:
I have a datset with time and column. I want to plot a graph with time and value. I tried many methods but didn't come proper graph. Because I have a time series. Then I thought I will convert time into UTC then try to plot it. But I didn't have any idea how to convert whole column into UTC . Can anyone help me to solve this error?

time.strftime("%Y-%m-%d %H:%M:%S",
          time.gmtime(time.mktime(time.strptime("2008-09-17 14:04:00",
                                                "%Y-%m-%d %H:%M:%S"))))

This code is for only one time . If I want to convert whole column then What should I have to do?

def convertTime(s):
tm = ((datetime.datetime.strptime(s, '%d/%m/%Y %H:%M'))-datetime.datetime(1970,1,1)).total_seconds()*1000
return

I wrote this code. But it gave me an error.
my csv file

my code is :

 condition = ""
date_time  = []
x1 = []
x2 = []
x3 = []
def convertTime(s):
  tm = time.strptime(s, " %d/%m/%Y %H:%M")
  return datetime.datetime(tm.tm_year,tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec)
with open('data43.csv','r') as csv_file:
csv_data = csv.reader(csv_file, delimiter=',')
 row_num = 0
for row in csv_data:
 if(row_num == 0):
  condition = row[0]
elif(row_num > 1): #Data starts here
  if(row[0] != ''):
    date_time.append(convertTime(row[0]))
  if(row[1] != ''):
    x1.append(int(row[1]))
  if(row[2] != ''):
    x2.append(int(row[2]))
  if(row[3] != ''):
    x3.append(int(row[3]))
row_num = row_num + 1
fig1 = plt.figure(1)
ax = fig1.add_subplot(2,1,1)
ax.plot(date_time,x1)
ax.stem(date_time,x2,'C1--','C1o',linefmt=None, markerfmt=None, basefmt=None)
ax.stem(date_time,x3,'C2--','C2o',linefmt=None, markerfmt=None, basefmt=None)
ax.legend()
ax.xaxis_date()
ax.get_xaxis().set_major_formatter(DateFormatter('%d/%m/%Y %H:%M:%S'))
plt.xlabel('t')
plt.ylabel('k')
leg = plt.legend( loc = 'upper right')
plt.draw() # Draw the figure so you can find the positon of the legend.
bb = leg.get_bbox_to_anchor().inverse_transformed(ax.transAxes)
xOffset = 0.3
bb.x0 += xOffset
bb.x1 += xOffset
leg.set_bbox_to_anchor(bb, transform = ax.transAxes)
plt.rcParams["figure.figsize"] = [20,20]
ax.plot(style='.-')
plt.show()


texts: Cross error loss function cause division by zero error

texts:
How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by:
-(ylog(actual_output)+(1-y)*(1-actual_output)


texts: Positive or negative impact of features in prediction with Random Forest

texts: In classification, when we want to get the importance of each variable in the random forest algorithm we usually use Mean Decrease in Gini or Mean Decrease in Accuracy metrics. Now is there a metric which computes the positive or negative effects of each variable not on the predictive accuracy of the model but rather on the dependent variable itself? Something like the beta coefficients in the standard linear regression model but in the context of classification with random forests.


texts: Cross validation for Collaborative filter-based recommendation systems

texts:
I am an absolute beginner and am trying to implement collaborative filter for furniture ecommerce (think wayfair). I need some guidance about cross-validation strategy.
Situation:
I am working on a fictitious dataset relating to customers' furniture purchases. Let's assume that the customers buy furniture and related items at a time. "At a time" because they mostly buy them either when furniture breaks or when they need to be replaced because the needs have changed (kids have grown up etc.). Hence, a typical buying cycle is somewhere between 6 and 9 years. Moreover, by "related items" I mean that they would buy related items such as mattresses, bed sheets along with bed.
When the customer visits the furniture store website for the first time (defined as "new customer"), they indicate their preferences in terms of things they are looking for in the furniture: size, handling etc. This was done to avoid "cold-start problem." in recommender system.
Here's how the dataset looks like:

My goal is to recommend products for each new customer who visits the website, which happens every 6 to 9 years because of buying cycle explained above. Predicting right products for the new customer will help reduce sales cycle and possibly improve customer satisfaction.
The question is how should I split the dataset into test and validation?
I read through https://stackoverflow.com/questions/40494952/splitting-in-recommender-system, How to split train/test in recommender systems, and https://arxiv.org/pdf/2007.13237.pdf.
Reading above sources, I believe that realistically, I have three options and here are my thoughts for each:
1) Separate recent k transactions for each customer for testing:
This might suffer from data leakage problem i.e. for some customers I might have data from 2011 to 2020 in training set, but for others, I might have data from 2016 to 2019. I might already know that during Covid (2020), office desk was popular. So, I would automatically recommend this for users with training data until 2019. This will be considered cheating, I think.
2) Holdout: Shuffle and then separate last k transactions for testing:
Timing is important. So, this might not work because I might use a future transaction to predict something in the past, which wouldn't make any sense.
3) Randomly shortlist k customers for testing, and then train using remaining customers:
I am a big fan of this method because my goal is to predict product recommendations for new customers. I won't have cold-start problem because the customers would provide the preferences, and I will be able to use preferences from training customers to match (using collaborative filter) preferences of new customers. However, I am not too sure whether training customers' preferences will represent new customers.
Am I on the right track? I have spent a lot of time on this problem, and I would appreciate any guidance.


texts: Forecasting with a Machine Learning Algorithm

texts:
Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all welcome, sorry. But theoretically subject is inside a gray area and all comments are valuable i think. It is not a widespread approach to use ML for forecasting, as far as I know.

Our sales team publishes a forecasting report at the beginning of each year. But it is mostly far from reality, not very scientific. It includes subjective assesments also.

We decided to construct a ML model to make a forecasting. Not a classical forecasting because there are many input variables that can effect the output. So, it is an experimental project actually.

So far, i have developed an adequate model, which is already far more better than sales one. But still i think it is not a good one, only adequate.

Data includes Customer-FromCountry-ToCountry based sales. So there are many combinations. By looking at similar applications at MachineLearningMastery.com, i have developed my model like that:

customer-from-to-M9-M8.....M1 - month - quarter - movingavg3 - mov6 - mov9 - dptcountryholidaycount - arvcountryholidaycount

m9 to m1 are previous 9 month sales of predicted month M(output variable) M's vary from 0 to 800 mostly. Holidays are from 0 to 6 for example. I Scaled all of these numerical variables between 0-1

Customer-From-To-Month-Quarter variables are categorical and i used OneHotEncoding for these.

I shuffled all the data, so there is not a time ordered dependency inside my dataset.

I have divided the data into 3 segments.
1: above 100 average for last 3 months;
2:10-100 average
3: below 0 average for 3 months.
I run different models for each 3 segments.

There are many 0 values, so i am using a LGBClassifier first to determine if the sales is 0 or not. Then if it is not 0, i am using a LGBRegressor to determine the sales value for that input row. (SGDRegressor for segment 1, it performs better)

I tried to add MonthSeasonality index and QuarterSeasinality index for each combination into dataset as variables. But it didn't perform well.

What can i try to get better score, im not asking about algorithms etc. My main purpose here is to get insights about construction of dataset, preprocessing. Your precious ideas about preprocessing tricks before applying a ML algorithm.

Thanks


texts:
I'm trying to create 3D bars on this map. Can anyone please advise if this is possible, and how?

http://leafletjs.com/examples/choropleth.html

My data: UFO sightings in the USA (location wise).
Count of these sightings per location will be the height of the 3D bar.
Base map is a choropleth with US population density values.

I don't mind integrating Javascript or d3.js into the code to create the 3D bars.


texts: Is there a way to get gini index values for every node in rpart model?

texts:
df <- tibble(x=factor(c("A", "B")), y=factor(c(1, 0)))
model <- rpart(formula=y~., data=df, method="class", control=rpart.control(minsplit=2))

Here model would have 1 parent and two child nodes. How to get gini index values for these nodes from rpart model object?


texts: Why not using linear regression for finetuning the last layer of a neural network?

texts:
In transfer learning, often only the last layer of the network is retrained using gradient descent.
However, the last layer of a common neural network performs only a linear transformation, so why do we use gradient descent and not linear (or logistic) regression to finetune the last layer?


texts:
I need to extract product names from a text column in a dataset. Currently I'm using regular expressions to extract the product names from the middle of the text, but sometimes the product name is misspelled, incomplete or even amended in another word, which means that I am unable to identify and extract the product name.
We currently have around 1500-2000 products on that list and I have a data set with those products already identified from approximately 30,000 lines. Is there an approach that I can use this historical data to improve the identification of products that have not yet been identified?
Just an example:
The product X produced by the Company Y is used to treat skin diseases

Note: The product names doesn't appear in a fixed position.


texts: What is this type of plot and how to interpret it?

texts:
I was presented to this plot and i've seen it before, however i would like to know what does the countour around the main line mean.

Ps: i couldn't find enough information about it to investigate by myself, so i had to create this question.


texts: ResourceExhaustedError when building Sequential model

texts:
i have a big problem when trying to build my model,
input shape: (1447, 224, 224, 3)
output shape: (1447, 154457)

model = Sequential([
        Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='valid', input_shape=(224,224,3)),
        BatchNormalization(axis=-1),
        Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=256, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        MaxPool2D(pool_size=(2, 2), strides=(2, 1)),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid'),
        BatchNormalization(axis=-1),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid', strides=3),
        BatchNormalization(axis=-1),
        Conv2D(filters=512, kernel_size=(3, 3), activation='relu', padding='valid', strides=3),
        AveragePooling2D(pool_size=(6, 1), strides=1, padding='same'),
        ReLU(),
        BatchNormalization(axis=-1),
        Flatten(),
        Dense(154457, activation='relu'),
        Dense(154457)
    ])

i got below error,
ResourceExhaustedError: OOM when allocating tensor with shape[11264,154457] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]

what is the problem here as i am new to neural network thing.


texts: Can I use Tensorboard to manage-fjobs and visualize learning on non-Tensorflow algoithrms? (e.g. Scikit?)

texts:
I am still searching for a great tool that manages jobs and visualizes learning from my models. Tensorboard is obviously one option given it's massive support.

But is it possible to organize jobs and visualize learning on tensorboard for algorithms not from the TF library?


texts: Do I need to buy a NVIDIA graphic card to run deep learning algorithm?

texts:
I am new in deep learning. I am running a MacBook Pro yosemite (upgraded from Snowleopard). I don't have a CUDA-enabled card GPU, and running the code on the CPU is extremely slow. I heard that I can buy some instances on AWS, but it seems that they don't support macOS.

My question is, to continue with the deep learning, do I need to purchase a graphic card? Or is there other solution? I don't want to spend too much on this...


texts: In an SVM, does a more negative/positive decision score mean that it is further from the seperating hyperplane?

texts:
For example, if I have a sample with a decision score of -6 and another  with a score of -3, which sample is closer to the hyperplane?

Also, does the probability of a sample belonging to a class increase with its respective distance from the hyperplane (larger distance from hyperplane = larger probability)?

I am using sklearn's decision_function and predict_proba functions to get these values if it makes a difference.

Thanks.


texts:
I have a list of conferences on different topics, e.g.

Conference on genomics and neurosciences
Advances in string theory and astrophysics
Genomics and neuroscience: 20 years of research
Swiss Physics society meeting on string theory and astrophysics
...

They fall into different classes, like 1 and 3, 2 and 4 together. What is the right tool to group those titles?


texts: How should clickstream data be prepared before user segmentation can be performed?

texts:
I'm interested in doing segmentation/clustering of users in clickstream data and am looking for some good suggestions about how to go about it.

Lets say my data consists of observations made up of visitors to a website. The data is in clickstream/weblog format and so is made up of users cookie data. Lets say I can identify unique users via their IP address (as a basic example). How should I go about preparing my data so that it can be segmented to find users with similar behaviours? One of my thoughts around this, is that because the data is event-driven, the same user can obviously appear multiple times in the data even though it may all be related to the same session of that user. How are these types of problems tackled so that you can perform segmentation based on user behaviour?

Thanks for your suggestions!


texts: Python: validating the existence of NLTK data with database search

texts:
I need to pull the names of companies out of resumes. Thousands of them. I was thinking of using NLTK to create a list of possible companies, and then cross-referencing the list of strings with something like SEC.gov.

I've already been able to successfully pull the candidate's name, and contact info off of the resumes with some RegEx, but this one has me quite stumped.

What I'm thinking is that I could use NLTK to create a list of strings of proper nouns from the resume's, and then search SEC.gov, or some other database.

This is a link to the SEC page I would be searching: SEC company search page

Read Resume1
Get all potential company names as list of strings potentialCompanies
    IF searching for string1 in SEC gets result, THEN add to candidateCompanies
        ELSE remove from potentialCompanies, go to next string

My Questions

To people that have used NLTK, would there be a better way of getting the potential companies from the text besides using proper nouns?

Would there be a better place to search for companies than the SEC site?

I have never done any web scraping before, and don't really know where to start if it is needed.

(I had posted this on Stack Overflow but they told me that it might be better suited for here...)


texts: Decision trees: leaf-wise (best-first) and level-wise tree traverse

texts:
Issue 1:

I am confused by the description of LightGBM regarding the way the tree is expanded.

They state:

  Most decision tree learning algorithms grow tree by level
  (depth)-wise, like the following image:

Questions 1: Which "most" algorithms are implemented this way?
As far as I know C4.5 and CART use DFS. XGBoost uses BFS. Which other algorithms or packages use BFS for decision trees?

Issue 2:

LightGBM states:

  LightGBM grows tree by leaf-wise (best-first).It will choose the leaf with max delta loss to grow. When growing same
  leaf, leaf-wise algorithm can reduce more loss than level-wise algorithm.

Question 2: Is it correct to say that level-wise growth trees will have equal depth for all leaves?

Questions 3: If Question 2 is not correct, then the trees from level-wise and leaf-wise growth will look the same at the end of the traversal (without pruning etc). Is it a correct statement?

Questions 4: If question 3 is correct, how can "leaf-wise algorithm can reduce more loss than level-wise algorithm"? Does it have to do with the post-pruning algorithm?


texts: Ways to Encode context for text classification?

texts:
I have a binary classification task which has the following specification:

Input: Chunk of text (not more than a few sentences, mostly a sentence).
Additional Input: For each input sample there is additional information available (which is also some text of similar length, max 2-3 sentences).
Problem: Classify text content using additional context
Problem Type: Binary classification

Essentially the task boils down to classifying the content conditioned on the context or p(content|context).

I was thinking of effective ways to encode the text for classification using a deep neural network. I searched for recent works but the existing literature mostly use the technique mentioned below.

I was wondering if there is a better way to encode context? Which has shown to be effective in some domain as compared to just concatenation.

Encode then concatenate: This involves encoding the context using ways similar to the content and then concatenating the feature representation before classification. This is the widely used technique. There are a lot of variants of this technique (different ways to encode text e.g. using tfidf rep, word embedding, LSTMs, CNNs ) which are widely explored.

Are there any better ways to bring context information during classification?

P.S: Recursive RNN is something that I have on my TODO.


texts: Understanding the text from the paper 'Efficient BackProp' by Yann LeCun

texts: Sorry, I just started in Deep Learning, so I am trying my best not to assume anything unless I am absolutely sure. Going through comments here someone recommended this excellent paper on backpropagation Efficient BackProp by Yann LeCun While reading I stuck at '4.5 Choosing Target Values'. I can't copy paste the text as pdf is not allowing it so posting the screenshot here. Most of the paper was clear to me but I couldn't understand exactly what the author was trying to convey for this specific part (see the image attached). If I understand correctly, is author reccomending to normalize target values between -0.90 and 0.90 for regression problem instead of -1,1 or is he saying that regression is wrong for classification because it will predict wrong class?