texts: Output a word instead of a vector after word embedding?

texts:
I'm trying to play around with a toy implementation of translation or text-summarization. I understand now that most people use an embedding layer before whatever model they use, which produces something like 300-dimensional vector. But what does the model output? Like for a encoder-decoder model, it's inputs are sequences of those vectors. Then what does the final layer of decoder come out with? Not like event extraction or something like that, which we classify to a small number of classes.

So my core question is: Is the output also a 300-dimensional vector, which then I have to generate the word based on the most similar word vector? Or is it the index of a word from the original word space? In the second case, the network is classifying to more than 100000 classes?

And also, is there any existing package that support this "reverse embedding to word"?

I have not seen any question alike on the site. Please mark it as duplicate if it is.

Any help would be appreciated.


texts: Determining number of clusters in high dimensions

texts: I am doing KMeans clustering for sentence embeddings and my problem is the number of clusters. In general, feature size is an order of a few hundreds (in this case 768) and my concern is the sparsity of space. I tried to use gap statistic, but it just increases monotonically and has no maximum (I ended up with max 2048 clusters). Also, embeddings lie on a n-dimensional sphere rather than fill the space uniformly. My questions is: does it really make sense to use various "clustering metrics" to determine optimal number of cluster when the feature space is large?


texts: Is it possible to predict unknown classes as a composite of known classes?

texts:
In modern NN's, it's a real pain adding new classes on the fly, and if they're not known at the outset, it typically requires retraining the network (either fully or just a handful of layers).
However, let's say the task was to classify species. The network is trained on 20 animals, and up until this point, has no knowledge of a Horse.
If you were to describe a Horse, you might say it was some combination of Cow, Zebra etc. Similarly, if you showed the trained network 100 images of Horses to infer the class, has anyone demonstrated accurately being able to represent a new class as a composite of existing classes? It could predict a Horse as a composite of a cat and a fish, as long as it's consistent, that information is useful.
I.E if for the vast majority of those images you saw a prediction similar to: 0.7 Zebra, 0.2 Cow, 0.1 Dog. I understand the layers in your standard CNN are unlikely to highly correlate a Zebra and a Horse in the same way Humans do, but that's not really an issue. As long as it's prediction for that new unknown class is consistent across those new images, if you had a handful of unknown classes, you could cluster them separately from each other.
I've looked around for papers to see if this has been assessed but struggled to find anything relevant.


texts: Incentivizing curiosity in a sparse reward environment

texts:
I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The problem I'm applying RL to has a continuous observation space (albeit very low dimensional), and a discrete (binary) action space where the agent can either "act" or "do nothing". Opportunity for reward is quite sparse, and comes along randomly approximately once every thousand time steps, and the spacing of the agent's actions around the opportunity has a profound impact on the probability of success. Every time the agent "acts", it uses part of a finite resource, which is replenished randomly according to a self exciting Hawkes process. Put more visually, this is what 70ish time steps of gameplay might look like. "." is do nothing, "x" is shoot, and "O" is when an opportunity comes along.

....x......x....x.x..x......O..x.....x...x..x.x...........x........x..x...

To reiterate, the probability of "O" succeeding depends entirely on the spacing of the two "x" around it.

I'm working on getting a dueling DQN structure to work right now, but the problem I'm having is that the agent seems to get stuck in a local minima, where he shoots at every time step, and only slows down when his resources start to run low. However, this is not the optimal play strategy, as if he were to shoot more widely spaced apart, the probability of success would actually increase (there is a sweet spot). But he seems to be content with the tiny probability of success given xOx even though x....O.....x would be far more likely to succeed.

I have been using a steadily decaying epsilon to try to promote exploration, but the agent really never "explores" all the possible strategies (such as acting once every 5 time steps, once every 10 time steps, once every 100...).

I suspect that one of the reasons for this is because even when epsilon is very high, and the agent takes a random action, on average, it will be taking an action every 2 time steps, so it has no incentive to see what the reward structure looks like if it shoots more infrequently. The reason I'm using RL to solve this problem at all is because I think the optimal solution is probably quite nuanced, where he shoots at the best rate for success when he is high on resources, then shoots more slowly when his resources start dwindling, until they get replenished, at which point he shoots at the optimal rate again.

I recently ran across this paper on curiosity driven exploration, and it seems vaguely applicable to the problem I'm having. But before going down that complicated route, I wanted to post here to see if I'm missing anything obvious. Here is a list of things i've tried in order to promote wider exploration:

Give a small penalty every time the agent shoots since it is using a resource
Have an exponentially growing penalty as the resource runs out to try to get him to shoot slower. This works, but it doesn't incentivize shooting slower when he has plenty of resource
Give a small reward proportional to the "goodness" of the spacing every time he acts to try to help him find the best spacing. This approach was a total fail because he ends up shooting every time step and just gathering all the tiny little rewards, rather than shoot once every n time steps and get big reward.

I really don't know what else to do, and would greatly appreciate any advice.


texts: Choropleth Map with plotly.express doesn't show map, just legend

texts: I copied the "Indexing by GeoJSON Properties" example as seen on the Choropleth Maps in Python page. The code was executed with no error but there was no map showing, only the legend. In addition, the 'box', 'pan', 'zoom' options show but don't work. I have the latest plotly version, installed via pip. any ideas? Thanks.


texts: How big does a Neural Net need to be to learn anything

texts:
Is there literature on how big a net needs to be to learn an arbitrary classification problem where the input is a n-bit integer and the output is one of k categories? I'm interested in both a theoretical bound and the size needed in practice with current training and initialization techniques.

To be specific, the neural net is allowed to see all possible inputs and their respective outputs, so I will not hide anything from the training set, as the function is not assumed to have any structure whatsoever.


texts: Result Dithering? Why randomly shuffle results?

texts:
Most of us want to build a recommendation engine as accurate as possible, however, an experienced chief data scientist believes a practical machine learning algorithm should randomly shuffle the results, therefore non-optimal results. This is known as "Results Dithering". Slide 15 at:

http://cikm2013.org/slides/ted.pptx

While I understand how to do it (adding a small fixed Gaussian noise), but why would we do it to make the performance worse.


texts: Should I use a different ML model for feature sets with missing values while drawing inferences?

texts: I have trained a model with 5 features. While training, I had no missing values, and the models were perfectly trained. However, when I'm using this model to draw inferences, sometimes I'm missing one feature value. How can I address this - should I use a different model with that only uses 4 features, or should I keep using the model with 5 features and draw the inference using the available 4 features?


texts: How to identify a field as holding personal identifiable information from the name of the field itself using ML model in python?

texts:
Is it possible to automatically detect fields holding personal information (name, phone, address, SSN, passport, gov ID...) from its names, using python in order to upload datasets into the cloud after encrypting or anonymizing the PII fields?
I am open to do my own model by training it on a dataset that holds thousands of fields and each one is classified whether personal or not. But apparently I can't find any related datasets.


texts:
I am trying to understand the meaning of error in sklearn KMeans.

In the context of house pricing prediction, the error linear regression could be considered as the money difference per square foot.

Is there a real life meaning about KMeans error?


texts: I have used the t-SNE algorithm to visualize my high dimensional data. However, I was wondering if this is a practical method for inference?


texts: How to use class_weights with one-hot encoding in keras?

texts:
Is it possible to use class_weights with a one-hot encoding?

I've tried sparse_categorical_crossentropy and, for some reason, it's significantly worse than my classic categorical_crossentropy with one hot encoding model.

This is how I'm calculating the class_weights with sparse:

unique_class_weights = np.unique(labels)
class_weights = class_weight.compute_class_weight('balanced', unique_class_weights, labels)
class_weights_dict = { unique_class_weights[i]: w for i,w in enumerate(class_weights) }

Training like:

full_model.compile(loss='sparse_categorical_crossentropy',
                   optimizer='rmsprop',
                   metrics=['accuracy'])

full_model.fit_generator(line_generator(data_train, labels_train),
                         validation_data=line_generator(data_test, labels_test),
                         validation_steps=1,
                         steps_per_epoch=len(data)/GENERATOR_BATCH_SIZE,
                         class_weight=class_weights_dict,
                         epochs=1)


texts: Core components of data literacy for working professionals

texts:
I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government.

I've struggled to develop a concise definition of data literacy, but have determined it consists of four cornerstones: 1) statistical literacy, 2) programming knowledge (Python, R, etc.), 3) domain/ substantive knowledge, and 4) visualization/ communication.

In the latter half of my presentation, I want to provide a list of 10-12 critical questions data literate professionals ask, including:

Where does the data come from and how was it collected? How reliable or "noisy" is the data (e.g. measurement error, missingness, etc.)?
What is the larger population we are inferring from the sample?
What is the context of the data? What is the appropriate frame of reference or point of comparison?
What are the inherent trade-offs (e.g. bias vs. variance) to the methods used to interpret this data?
What's the level of uncertainty with a given model's predictions (e.g. standard errors, confidence intervals, etc.? How do I know when my inference is wrong?
Does this sample of data help me answer the problem, puzzle, or question I’m actually interested in?
Have your findings been replicated? If not, do you have the data and programming scripts so that another data scientist could attempt to replicate your findings?

Any suggestions, comments, or critiques of this framework or particular questions would be much appreciated! Recommendations on good sources (videos, online, books) on data literacy for professionals would be helpful as well!


texts: I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH into a cloud environment, such as a cluster on AWS? Perhaps there are many tasks where a single-node cluster is adequate, and can be run locally more easily.


texts: Why does degradation occur in deep neural networks?

texts:
It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In the original ResNet paper they hypothesize and verify that this is not due to vanishing gradient.

From what I understand, it is difficult for a model to approximate the identity map between layers and furthermore when this map is optimal the model may tend to approximate the zero function instead. If this is the case, why does this occur? Finally, why does this not occur in shallower networks in which the identity map may also be optimal?

I am new to this topic so I apologize if I may not fully understand this problem. Thank you.


texts: Simple logistic regression wrong predictions

texts:
In the following example, the prediction obtained for $x=[0,0]$ after fitting the model is $1$, but expected is $0$.

X = np.array([
    [0, 0],
    [1, 0],
    [0, 1]
])

Y = np.array([0,1,1])

model = LogisticRegression()
model.fit(X,Y)

x = [0,0]
y = model.predict([x])[0]

It only works if I duplicate the training row of ($x=[0,0], y=[0]$)

But why?


texts: Explain to a layperson why text-to-speech (TTS) has made little progress?

texts:
I am a layperson who has searched for years in vain for a human-sounding TTS program, but it seems that over the years, hardly any progress has been made (a purely unscientific evaluation using just my ear and also the ears of many others). It still sounds like the same robotic word-by-word reading-out that I believe was already accomplished decades ago.

This is puzzling (for a layperson like me who doesn't know anything about data science, linguistics, machine learning, etc.). Especially when contrasted to say speech-to-text or driverless cars, which have noticeably improved over the years.

Could you explain to a layperson (like myself) why, in early 2017, do most text-to-speech (TTS) programs still sound so robotic and non-human-like? And why so little progress has been made (especially in comparison to some other achievements)?

Might it just be that there simply isn't much demand for TTS (as compared to speech-to-text or driverless cars) and so few resources have been devoted to this?

(If this isn't the right StackExchange for the question, I apologize in advance. Please migrate this to a more suitable StackExchange site instead, thanks!)


texts: MPI, MapReduce, or Spark for complex datasets and processing

texts:
I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.

This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.

Can you help me to choose an appropriate one?

Please ask if the description is not clear.


texts: Any advantage of sklearn wrappers for xgboost over python API?

texts:
Are there any advantages of using the XGBoost sklearn wrappers XGBRegressor or XGBClassifier over using the Python API with the train method?
Is one faster than the other or does one use less memory than the other?
I am asking for a generic case where the data comes in some form which I convert to a pandas dataframe.


texts: How to explain a figure to highlight importance of features

texts:
I have got this figure by calculating feature importance in my models (test set). As shown in the figure (it is edited but it should be ok for this example), values are very very low (less than 0.025).  Can you please explain what it means in terms of model building and prediction for text classification?
Which kind of considerations can be appropriate in this case? Can one say that the model overfits or is poor performed (despite of accuracy, precision, f1_score and recall of 90 and above)?
Thank you for all your answers.


texts: How is the weight matrix set and its dimension in CNN

texts:
I am trying to calculate the output size of each layer and the number of parameters for 3 class classification using CNN. I have calculated till the final maxpooling layer and would really appreciate help in understanding how to reach the fully connected layer.

In Matlab I checked that the size for fully connected (FC) is 1176 and the weights are 3*1176 I am struggling to understand what these 2 numbers mean and how they have been calculated. I could guess that 3 comes from the number of classes but how did 1176 come? It does not match my calculation.

Question: How to determine the dimension of the last layer- FC? Since I have 3 classes, do I have 3 layers? Please correct me where wrong. Attached is the screenshot which shows the dimensions for the weight as 3*1176 .


texts: After training and saving a model can we give more information as input?

texts:
Suppose my data is a time series with multiple features such as wind, temperature, holidays, etc.. and I'm predicting a target variable Y.
After I go through the whole process of splitting data into training and test and after having good predictions/forecasts I save my model.
Can I now give extra information as input to my model? If I want to predict tomorrow's Y values and I know the wind and temperature forecasts (from weather services) for tomorrow, can't I use them to get a better prediction for tomorrow?


texts: How to improve accuracy in the following code?

texts:
I have the about 43 different categories of traffic signs images data. If I am using the small data of 3 categories the maximum accuracy I am getting is around 65% and I have tried a lot of different layer values as well as losses and optimizers. When I am using the complete dataset I am only getting accuracy of about 5%. Please give some pointers on how to improve my accuracy. Please tell me should I add more layers and if somebody has any good example somewhere, please provide the link.

import cv2
import numpy as np
import os
import sys
import tensorflow as tf
import glob

from sklearn.model_selection import train_test_split

EPOCHS = 10
IMG_WIDTH = 30
IMG_HEIGHT = 30
NUM_CATEGORIES = 10
TEST_SIZE = 0.4
path = []
data = []
label = []

def main():

    # Check command-line arguments
    if len(sys.argv) not in [2, 3]:
        sys.exit("Usage: python traffic.py data_directory [model.h5]")

    # Get image arrays and labels for all image files
    images, labels = load_data(sys.argv[1])

    # Split data into training and testing sets
    labels = tf.keras.utils.to_categorical(labels)
    x_train, x_test, y_train, y_test = train_test_split(
        np.array(images), np.array(labels), test_size=0.2
    )
    # Get a compiled neural network
    model = get_model()

    # Fit model on training data
    model.fit(x_train, y_train, epochs=EPOCHS)

    # Evaluate neural network performance
    model.evaluate(x_test,  y_test, verbose=2)

    # Save model to file
    if len(sys.argv) == 3:
        filename = sys.argv[2]
        model.save(filename)
        print(f"Model saved to {filename}.")

def load_data(data_dir):
    """
    Load image data from directory `data_dir`.

    Assume `data_dir` has one directory named after each category, numbered
    0 through NUM_CATEGORIES - 1. Inside each category directory will be some
    number of image files.

    Return tuple `(images, labels)`. `images` should be a list of all
    of the images in the data directory, where each image is formatted as a
    numpy ndarray with dimensions IMG_WIDTH x IMG_HEIGHT x 3. `labels` should
    be a list of integer labels, representing the categories for each of the
    corresponding `images`.
    """
    subdirs = [x[0] for x in os.walk('C:\\Users\\LENOVO\\traffic\\gtsrb')]
    for i in range(1,NUM_CATEGORIES+1):
        subdirs[i] = subdirs[i] + "\*.*"
    for i in range(1,NUM_CATEGORIES+1):
        for file in glob.glob(subdirs[i]):
            a = cv2.imread(file)
            a = cv2.resize(a,(IMG_WIDTH,IMG_HEIGHT))
            data.append(a)
            label.append(i-1)

    return(data,label)

def get_model():
    """
    Returns a compiled convolutional neural network model. Assume that the
    `input_shape` of the first layer is `(IMG_WIDTH, IMG_HEIGHT, 3)`.
    The output layer should have `NUM_CATEGORIES` units, one for each category.
    """
    model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(
        32, (3, 3), activation="relu", input_shape=(30,30,3)
    ),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),

    # Add an output layer with output units for all 10 digits
    tf.keras.layers.Dense(NUM_CATEGORIES, activation="softmax")])
    model.compile(
    optimizer="Nadam",
    loss="categorical_crossentropy",
    metrics=["accuracy"])
    return (model)
    raise NotImplementedError

if __name__ == "__main__":
    main()


texts: why is SVM cost function the norm of the hyperplane parameters, and not the mean square error?

texts:
working through Dr. Ng's coursera course
cost function intro:
basically, you want to minimize the difference squared between your guess and the actual value
cost function for SVM: this or this
he says the cost function is now the norm of the vector representing the hyperparameters: sqrt(theta_1^2 + theta_2^2... )
Why is that the case?


texts:
I'm trying to visualize data for exploratory data analysis motivated by visualizing  multiple scatterplots of features simultaneously, similar to this question. But I quickly run into problems when using a large number of features (~50) and rows (~50K). While I like using seaborn pairplots the generation of a large number of plot panes can get computational intractable for a large number of features and observations. Subsetting a very large table to a smaller number of features or observations does not seem complete.

My question is: What's an efficient way to plot many features for EDA in python? If there is not an efficient way then is there a defensible way to reduce the number of features or observations


texts: Train-Test split for a recommender system

texts:
In all implementations of recommender systems I've seen so far, the train-test split is performed in this manner:

+------+------+--------+
| user | item | rating |
+------+------+--------+
| u1   | i1   |    2.3 |
| u2   | i2   |    5.3 |
| u1   | i4   |    1.0 |
| u3   | i5   |    1.6 |
| ...  | ...  |    ... |
+------+------+--------+

This is transformed into a rating matrix of the form:

+------+-------+-------+-------+-------+-------+-----+
| user | item1 | item2 | item3 | item4 | item5 | ... |
+------+-------+-------+-------+-------+-------+-----+
| u1   | 2.3   | 1.7   |   0.5 |   1.0 | NaN   | ... |
| u2   | NaN   | 5.3   |   1.0 |   0.2 | 4.3   | ... |
| u3   | NaN   | NaN   |   2.1 |   1.3 | 1.6   | ... |
| ...  | ...   | ...   |   ... |   ... | ...   | ... |
+------+-------+-------+-------+-------+-------+-----+

where NaN corresponds to the situation where a user has not rated that particular item.

Now, from each row (user) of the matrix, a certain percentage of the numeric (non-NaN) values are removed and set aside into a new matrix, representing the test set. The model is then trained on the initial matrix, with test samples removed, and the goal of the recommender is to fill-in the missing values, with the smallest possible error.

My question is, can the train-test split be somehow done user-wise? For example to keep a set of users separate, train the recommender on the rest of the user set and then try to predict the ratings for the new users? I know this goes a bit against the idea that "if a recommender does not know you, it cannot recommend something you like", but I am wondering if some k-NN can be done.


texts: Package RTextTools loaded but facing Error: could not find function "create_matrix"

texts:
install.packages('RTextTools')
library(RTextTools)

data("NYTimes", package='RTextTools')
head(NYTimes)

timesMat <- create_matrix(NYTimes$Title)

The package is installed and I can view the "NYTimes" data set but the create_matrix call throws an error that it can't find the function.

One more thing: call to library(RTextTools) gives the following error messages:

  Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck =
  vI[[i]]) :    there is no package called ‘slam’ Error: package or
  namespace load failed for ‘RTextTools’

Any help appreciated...


texts:
I am trying to analyze the weights in a trained Keras embedding layer. The goal is to understand how well (or how poorly) we're capturing relationships between some specific tokens in our texts, separate from the rest of our model. In my case I have a 10,000 word dictionary, which I have embedded into 128 dimensions.

dictLen <- 10000
embDim <- 128
embedding <- layer_embedding(input_dim = dictLen + 1, output_dim = embDim)

When I export the weights in that layer after training via get_weights(), I get a matrix of 10,001x128 values. So far so good, but I'm a little confused about something:

I'm adding 1 to my dictionary length to account for token 0, the unknown or null token. My understanding is that the embedding layer and subsequent layers will treat that value specially, but then why does it appear to have a trained weight when I export the entire set? And can I safely assume that the first row in the exported matrix is for value 0, and thus token 1 will be row 2 and so on?


texts: Query relating to Pandas Rows manipulation

texts:
I have a query regarding Pandas data manipulation.
Let's say I have a dataframe, df with following structure.
A B C
1 1 7
5 3 3
3 3 2
7 5 2
5 NaN 2

We have 3 columns in the dataframe A, B & C.
B column consists of mean values wrt A.
For example,
Value of B in 3rd row (which is 3) is mean of first 3 rows of A (9/3)
Similarly, value of B in 4th row = (Sum of values in 2nd,3rd and 4th row of A)/3
Now, let's say I have many NaN values in B and there are no NaN values in A, how do I write a function or code to fill the NaN values as per the logic discussed above?
I tried using loc and iloc but I guess I made some mistake.


texts: Why replacing null values with outliers?

texts:
I have been watching a tutorial on stock price prediction with multivariate linear regression and the tutor replaces missing value data, NaN, with the outlier -99999.
Why and how do replacements like that do not skew the data and provide a biased or incorrectly-trained classifier?


texts: I want perform demand forecast for particular item based on attributes.Did I need to train the model with unsold items ? by maintaining sales Quantity as zero or  go with only items sold in training period.


texts: NLP text autoencoder that generates text in poetic meter

texts:
I would like to create an NLP autoencoder that happens to only generate text that conforms to a poetic meter, for example 'iambic pentameter'. That is, the output should be a series of clauses which are 10 syllables long and are read aloud with the verbal stresses as 'duh-DUH duh-DUH duh-DUH duh-DUH duh-DUH'. Shakespeare and his contemporaries used this format for much of their poetry.

The only method I can think of is to collect lines of text which are in this format, and then train an autoencoder on these. Since the network only emits iambic pentameter while training, it should also emit iambic pentameter while predicting. Given a large enough corpus, this seems like it should work.

One problem is that this training only accepts iambic pentameter. To rewrite other text into IP, the training needs to include variations of text in IP and train (not IP sentence) to emit (IP sentence). Generating these variations is straightforward. This crosses the line from training "sentence embeddings" to training "thought embeddings".

Are there other ways to solve this problem? Is there a way to directly alter the sentence embedding space? For example, variational sampling works by applying a nonlinear transformation on the embedding space.

Note: the CMU Pronouncing Dictionary supplies pronunciation stresses for over 100k words, usable for classifying meter:
http://www.speech.cs.cmu.edu/cgi-bin/cmudict

Note: this is a personal hobby project to teach myself deep learning & NLP- I really do not know whether it is achievable with current NLP technology.


texts:
I am using tslearn KShape to cluster time series data. I am generally happy with the results, as upon inspection, the clusters seem to make sense because of the similarity in shape and magnitude. I can see the general shape that emerges as seen in the figures below.

What I find odd is the cluster centers using k_shape.cluster_centers_. There seems to be a significant shift from some of the clusters. I've included two figures illustrating my point. In Cluster 0, the cluster center is plotted as the red line, and there is a significant shift from where that shape actually emerges in the time series. In Cluster 4, the cluster center is much more centered, for lack of a better term.

What could be contributing to the major offset seen in Cluster 0? What I find odd is that not all clusters have such a dramatic offset, and not all are offset to the beginning or the end of the time series. Any thoughts?


texts: When using GridSearchCV with regression tree how to interpret mean_test_score?

texts:
I am using GridSearchCV to tune hyperparameters of regression decision tree. When I do, I get mean_test_score but I thought it would return mean MSE since it is a regressor. how to interpret mean_test_score? Is there a way to tweak GridSearchCV so it returns mean MSE?
here is my code
tree_reg = GridSearchCV(DecisionTreeRegressor(criterion="mse"), {
    "min_samples_split":[2,3,4],
    "min_samples_leaf":[1,2,3]
}, cv=5, return_train_score=False)

tree_reg.fit(X, y)

pd.DataFrame(tree_reg.cv_results_)
>>> params                   split0_test_score  .... mean_test_score
   {"min_samples_leaf":2,        0.998782             0.9989933
    "min_samples_split":3}

   {"min_samples_leaf":2,        0.998823             0.998930
    "min_samples_split":4}
            ...

what does mean_test_score mean?


texts: Gradient descent with vector-valued loss

texts:
My understanding of gradient descent as an optimizer for a neural network is as follows:

Let $w$ be a vector of weights encoding a configuration of the network, and $l : w \mapsto \textrm{network loss}$ a function which calculates the loss over some batch of data given this configuration. Then, the weight update is $-\alpha \nabla l(w)$, where $\alpha$ is the learning rate, since the vector $\nabla l(w)$ represents the direction of greatest increase in the neighborhood around $w$.

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.


texts: Why are CNN layers used for bounding box regression?

texts:
In object detection algorithms such as R-CNN, the position of bounding boxes is tuned using ridge regression. An example is R-CNN, appendix C. This answer also explains hoiw it is done in Fast R-CNN (but doesn't explain why), which is similar. The inputs to the regression function are the outputs of layers of a CNN, usually the final max-pooling layer.
That paper claims that the relationship between the maxpool layer outputs and the location of the boundiung box can be explicitly assumed. Why can this be explicitly assumed? My belief was that convolutional layers are agnostic to object positions within an image because of the nature of convolution. How does a CNN's layer values aid in placing the bounding box therefore?


texts: ideal algorithms to demonstrate overfitting or underfitting

texts:
When one tries to look up concepts such as overfitting and underfitting, the most common thing that pops up is polynomial regression. Why is polynomial regression often used to demonstrate these concepts? Is it just because it can be easily visualised like the graphs here:
https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html
But then most ml algorithm such as kmeans clustering can also be used. Then why is it usually polynomial regression only? Are there any other similar algorithms that could be used?


texts: Create Choropleth Map for categorical data

texts:
I want to create a map where countries specified in dataframe are filled with colour green or red. The colour is depending on value in column 'PREFERENCE' (equals 1 or 0).
Whwn I execute the following code only map with colour grey are displayed. Why is that so?
Is the Chloropeth Map the right solution for this kind of task?

europe_map = folium.Map(location=[51.0, 10.0], zoom_start=4)
europe_geo = "datasets\eu-countries.geo.json"

folium.Choropleth(
    geo_data=europe_geo,
    data=geolocation_df,
    columns=['COUNTRY', 'PREFERENCE'],
    key_on='feature.properties.name',
    fill_color='YlGn',
    fill_opacity=0.3,
    line_weight=1
).add_to(europe_map)

europe_map


texts: Decision Tree - Preprocessing for very sparse features

texts:
How do we pre-process data for very sparse features for a decision tree?

From this Turi documentation for decision trees

It mentioned this:

  Why chose decision trees? Different kinds of models have different
  advantages. The decision tree model is very good at handling tabular
  data with numerical features, or categorical features with fewer than
  hundreds of categories. Unlike linear models, decision trees are able
  to capture non-linear interaction between the features and the target.

  One important note is that tree based models are not designed to work
  with very sparse features. When dealing with sparse input data (e.g.
  categorical features with large dimension), we can either pre-process
  the sparse features to generate numerical statistics, or switch to a
  linear model, which is better suited for such scenarios.

If we have a categorical column that is very sparse (many 0's), then for that category what type of numerical statistics can we transform for each row?


texts:
I am training my GAN, and by looking at the loss curves of the generator and discriminator, I think I am going on the right way because from this blog, my curves look reasonable:

  A stable GAN will have a discriminator loss around 0.5, typically between 0.5 and maybe as high as 0.7 or 0.8. The generator loss is typically higher and may hover around 1.0, 1.5, 2.0, or even higher.

But how do I know when I should stop training. Unlikely to other types of network when we only have a loss curve, GANs have two. I know using FID or Inception scores may work but I am wondering is there any other heuristic way to determine?


texts: Should weight distribution change more when fine-tuning transformers-based classifier?

texts:
I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation:

class PretrainedTransformer(nn.Module):
    def __init__(
        self, target_classes):
        super().__init__()
        base_model_output_shape=768
        self.base_model = DistilBertModel.from_pretrained("distilbert-base-uncased")
        self.classifier = nn.Sequential(
            nn.Linear(base_model_output_shape, out_features=base_model_output_shape),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(base_model_output_shape, out_features=target_classes),
        )

        for layer in self.classifier:
            if isinstance(layer, nn.Linear):
                layer.weight.data.normal_(mean=0.0, std=0.02)
                if layer.bias is not None:
                    layer.bias.data.zero_()

    def forward(self, input_, y=None):
        X, length, attention_mask = input_
        base_output = self.base_model(X, attention_mask=attention_mask)[0]
        base_model_last_layer = base_output[:, 0]
        cls = self.classifier(base_model_last_layer)
        return cls

During training, I use linear LR warmup schedule with max LR=5-e5 and cross entropy loss.
In general, the model is able to learn on my dataset and reach high precision/recall metrics.

My question is:

Should weights distributions and biases in classification layers change more during training? It seems like the weights almost do not change at all, even when I do not initialize them as in the code (to mean=0.0 and std=0.02). Is this an indication that something is wrong with my model or it's just because the layers I've added are redundant and model does not learn nothing new?

Take look at the image of weight from the tensorboard:


texts: Unused bottleneck neurons in autoencoder

texts:
I'm using an autoencoder to compress categorical data, by a factor of about 20x.
For this part of my data set, I have roughly 3500 variables, so my final bottleneck size is 180. I'm getting pretty good losses & accuracy, but that's not what I'm asking about.
After looking at my compressed data, it turns out that quite a few of the encoded variables are zeros. In fact, out of 180 encoded variables, 111 of them are all zeros.
I reran my code twice to confirm that it wasn't a fluke (same parameters, same data, etc), and got similar results: 110 and 96 variables all zeros.
Does this mean I could reduce the bottleneck size by about 100 and still see basically the same losses and accuracy? Are these neurons just unused by my autoencoder?
Here is the code I use to generate it:
x_arr = np.array(x)
x_tr = np.transpose(x_arr)
test_index = np.random.choice(range(len(x_tr)), math.floor(len(x_tr)/5), replace=False)
x_test = to_categorical(x_tr[test_index])
x_train = to_categorical(np.delete(x_tr, test_index, 0))
print("Total data is of shape "+str(x_tr.shape))
print("Training data is of shape "+str(x_train.shape))
print("Testing data is of shape " + str(x_test.shape))
Total data is of shape (2548, 3593)
Training data is of shape (2039, 3593, 4)
Testing data is of shape (509, 3593, 4)

input_data = Input(shape=x_train[1].shape)
f = Flatten()(input_data)
hidden_1 = Dense(hidden_1_size, activation='relu')(f)
hidden_2 = Dense(hidden_2_size, activation='relu')(hidden_1)

code = Dense(code_size, activation='relu')(hidden_2)

hidden_2_rev = Dense(hidden_2_size, activation='relu')(code)
hidden_1_rev = Dense(hidden_1_size, activation='relu')(hidden_2_rev)
output_data = Dense(input_size*4, activation='softmax')(hidden_1_rev)
rshp_output_data = Reshape((-1,4))(output_data)

autoencoder = Model(input_data, rshp_output_data)
encoder = Model(input_data, code)

autoencoder.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[tf.keras.metrics.Accuracy()]) # when data is OHE

history = autoencoder.fit( x_train, x_train, epochs=1000, shuffle=True, validation_data=(x_test, x_test),
             callbacks=kcb.EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True) )

np.savetxt(outdir + "encoded_"+chromosome+"_"+str(start)+"_"+str(end)+".csv", np.transpose(np.array(encoder(to_categorical(x_tr)))), header=';'.join(sampleIDs), delimiter=';')


texts: What is the difference between proposal-based approach and proposal-free approach?

texts:
From here it says that

  Techniques to solve instance segmentation can be roughly grouped into two categories: proposal-based methods and proposal-free methods. In proposal-based methods, a set of object proposals and their classes are first predicted, then foreground-background segmentation in each bounding box is performed. The proposal-free approaches exclude the step of proposal generation.

What is "proposal" in this context? Also, how to "first predict their classes"? There is not much explanation about this topic on the internet and I would appreciate it if someone could explain the differences.


texts: Does matrix shape for training/testing sets have to be in a particular order?

texts: I've noticed that in the Andrew Ng Deep Learning course that for image analysis he always has X_train matrices in the shape of [height, width, 3, num_inputs], or, if flattened, [height X width X 3, num_inputs]. He also has his y_train as [1, num_inputs]. To me, it is more intuitive to flip these so that X_train is [num_inputs, height X width X 3] and y_train is [num_inputs, 1]. Is there any motivating reason or justification that it has to be the way he does it or is it just preference? Is this a standard or does it vary?


texts: Is there any way to define custom entities in Spacy

texts:
1) I have just started working on NLP the basic Idea is to extract meaningful information from text. For this I am using "Spacy".

As far as I have studied Spacy has following entities.

ORG
PERSON
DATE
MONEY
CARDINAL

etc. But I want to add custom entities like:

Nokia-3310 should be labeled as Mobile
and XBOX should be labeled as Games

2) Can I find some already trained models in Spacy to work on ?


texts:
array(['Ruby on Rails', 'Ruby', 'AWS DynamoDB', 'Python', 'MySQL',
         'Swift', 'Android', 'iOS', 'JavaScript', 'React Native', 'ReactJS',
         'TypeScript', 'Vue.js', 'Webpack', 'Amazon Web Services(AWS)',
         'Kubernetes', 'PHP', 'CI/CD', 'Java', 'C#', 'C', 'Node.js',
         'REST API', 'Go', 'Redux-Saga', 'Redux.js', 'Babel', 'GraphQL',
         'Tensorflow', 'PyTorch', 'Jenkins', 'Spring', 'Django', 'Git',
         'AWS EC2', 'CSS', 'HTML', 'MongoDB', 'Docker', 'Scala', 'SQL',
         'Embedded System', 'NLP', 'Apache', 'Kotlin', 'Angular', 'jQuery',
         'C++', 'RxJS', 'AngularJS', 'Redis', 'Next.js', 'NoSQL',
         'GCP(Google Cloud Platform)', 'Elasticsearch', 'OpenStack',
         'JPA(Java Persistent API)', 'TCP/IP', 'Objective-C', 'Realm',
         'Firebase', 'Ajax', 'Linux', 'PostgreSQL', 'ES6', 'AWS Lambda',
         'HTML5', 'AWS S3', 'GitHub', 'RxSwift', 'Terraform', 'AWS EKS',
         'AWS RDS', 'Microsoft Azure', 'Sass(SCSS)', 'CodeIgniter', 'Flask',
         'Nuxt.js', 'Ansible', 'Spring Boot', 'Linux kernel',
         'Apache Kafka', 'Deep Learning', 'Nginx', 'ActionScript', 'OOP',
         'Shell', 'gulp', 'Celery', 'SQLAlchemy', 'ExpressJS', 'RxJava',
         'Apache Spark', 'WebGL', 'OpenGL', 'Machine Learning',
         'MSSQL(Microsoft SQL Server)', 'Database', 'styled-components',
         'MVC', 'Retrofit', 'Machine Vision', 'Oracle', 'web3.js', 'R',
         'AWS ElasticBeanstalk', 'Elastic Stack', 'Laravel', 'ASP.NET',
         'Aurora DB', 'Redux-Observable', '.NET', 'AWS Backup',
         'AWS CloudWatch', 'Kibana', 'Fluentd', 'Logstash', 'JSP',
         'Bootstrap', 'Datadog', 'Rust', 'Azure', 'Apache Hadoop',
         'AWS X-Ray', 'Memcached', 'Jest', 'Mocha',
         'DRF(Django REST framework)', 'Spring Cloud', 'Data Analysys',
         'Big Data', 'GitLab', 'Gradle', 'SQLite', 'Microsoft IIS', 'Unity',
         'Electron', 'MariaDB', 'mSQL', 'gensim', 'Scikit-Learn',
         'AWS Simple Queue Service(AWS SQS)', 'gRPC',
         'Naver Cloud Platform', 'Ubuntu', 'Microservice Architecture',
         'Apache ActiveMQ', 'Oracle Database', 'Apache Subversion(SVN)',
         'Apache Tomcat', 'Red Hat Ceph Storage', 'Puppeteer', 'OpenLayers',
         'Vuex', 'Less.js', 'JIRA', 'Keras', 'NCP(Naver Cloud Platform)',
         'NestJS', 'PKI(Public key infrastructure)', 'AWS ECS', 'Hibernate',
         'UML', 'BitBucket', 'Arduino', 'Raspberry Pi', 'RabbitMQ',
         'Capistrano', 'Bamboo', 'MVP', 'OkHttp', 'Cocos2d', 'Ethereum',
         'Blockchain', 'DSP(Digital Signal Processing)', 'D3.js', 'Cocoa',
         'Axios', 'Ionic', 'WPF', 'AWS IAM', 'Shell Script',
         'Responsive Web', 'Canvas', 'ThreeJS', 'Apache ZooKeeper',
         'Pandas', 'Spring Batch', 'JUnit', 'Spring Data JPA', 'ASP',
         'Grunt', 'WordPress', 'MyBatis', 'AWS ElastiCache',
         'Apache HTTP Server', 'AWS Security Hub', 'Google API', 'Qt',
         'CAD', 'GatsbyJS', 'PostCSS', 'Socket.IO', 'Backbone.js',
         'Azure Linux Virtual Machines', 'Heroku', 'CUDA', 'IOCP', 'Unix',
         'CocoaPods', 'MVVM(Model-View-ViewModel)',
         'Google Firebase Crashlytics', 'Google Cloud Platform',
         'Windows kernel', 'OpenCV', 'Unreal Engine', 'Google Cloud SDK',
         'RxAndroid', 'Windows Embedded', 'Entity Framework', 'Packer',
         'Nexus', 'Consul', 'Selenium', 'Jekyll', 'XML',
         'Dependency Lookup', 'RxKotlin', 'Expo', 'Sketch', 'InVision',
         'Azure Text Analytics', 'Google Dialogflow',
         'Google Cloud Natural Language'], dtype=object)

let say I have this array list
if I want to group them by similar things
is there any pretrained language model to do this job easier?

for example
pytorch and tensorflow should be in one group because
most of the deep learning people are using pytorch or tensorflow


texts: Why underfitting is called high bias and overfitting is called high variance?

texts: I have been using terms like underfitting/overfitting and bias-variance tradeoff for quite some while in data science discussions and I understand that underfitting is associated with high bias and over fitting is associated with high variance. But what is the reason of such association or in terms of a model what is high bias and high variance, How can one understand it intuitively?


texts: Best approach for a simple self driving car

texts:
I'm planning to build a small car with autonomous driving (maybe modifying my current rc car or using a robot car kit, using arduino and raspberry).

I'll use a CNN, and I'm thinking how to collect data (I want to try a similar approach to the Udacity simulator).
My doubt is if is better to aim at supervised learning or reinforcement learning.

I'm more inclined to the supervised learning, but I don't know the best way to record data. If I'll go with the robot car kit, the front wheel doesn't steer (it use different wheels speed to turn left or right). So I could create 3 function for each direction, like "soft left", "medium left" and "hard left" to control the car when I train it, and then predict from 7 possibility:
- soft left, medium left, hard left, center, soft right, medium right, hard right
So 7 possible output.

Otherwise if I use a car with the front wheel that are able to steer, I have 3 possible class, like -1 for left button, 0 for center (any button pressed) and 1 for right button. So 3 possible output, without any gradient for the steering angle.

With the reinforcement learning honestly I don't know where to start. How can I tell to the car when it does something wrong and give her a reward? With a simulator is simple, but with a real car how can I assing a reward score for each frame that I collect? If you have any paper about this I appreciate it.

What's the best approach in your opinion?


texts: I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction methods such a bag of words. You can then use something like liblinear to obtain a SVM LibLinear model that is able to classify your data. On the other hand, you can build a model by concatenating Bert with a Dense layer. You can then fine tune this model and again, you obtain a classifier. Where would you use either one of them and why?


texts:
The ICD code scheme has gone through many revisions: https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems

Is there any compatibility between the different versions? (e.g. can ICD-9 code be directly used where ICD-10 codes are expected?)


texts: How to reach continue training in xgboost

texts:
I read the paper but found nothing talking about how to implement incremental learning.

Can someone share some basic or deep knowledge? not in coding way.

I know how to write code snippet to train incrementally.

When new data comes in, how to train incrementally if I use XGBRegressor? Reserve the old trees and train new data with new trees?

I found nothing talking about this in detail


texts: What is meant by Distributed for a gradient boosting library?

texts:
I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library.

What is meant by distributed?

Have a nice day


texts: What are the practical differences between pooling types at convolution?

texts:
I could find max-pooling in most -- actually, exactly all -- concrete CNN implementations I've seen. However, I see that there are several other kinds (min, average, global average pooling, etc.) but I cannot see what they differ on and what would be the best criteria to select them at application. Has anyone a clue?
I have learned that pooling is a sort of downsampling where we can preserve the locations of the original image that showed the strongest correlation to the specific features we searching for. So far, so reasonable. But what is the point of finding the weakest correlations with min pooling, for example?


texts: How to take the keywords from the given dataset to train GPT-2 based chatbot?

texts:
I am working with a dataset that contains Questions on various Events conducted by a college and the corresponding answers for the queries. I am using this dataset to train a GPT-2 355M model to create a chatbot where users can get their queries answered. But i am not getting good results and i feel that's because the questions in the dataset are in the " -Query " format.
For example, Ques: "Cicada3302 - Do I need to have any prerequisite knowledge to enter this event"
I am confused as to how can I make the chatbot understand that the first words before the "-" is like a keyword for rest of the question ? I am really new to this, so any help will be appreciated.
I have used the gpt_2_simple library for this.
I am attaching the colab link I have written so far, if it might be of any help.
https://colab.research.google.com/drive/1CrzwC9WQwF4YsqD66TY8F7ovTSHKsnFv?usp=sharing


texts: How to grow a list of related words based on initial keywords?

texts:
I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates similar keywords (in this case, other colors). See more examples in this YouTube video.

I would like to reproduce this in my own program. I'm thinking of using Freebase, and it would work like this intuitively:

Retrieve the list of given words in Freebase;
Find their "common denominator(s)" and construct a distance metric based on this;
Rank other concepts based on their "distance" to the original keywords;
Display the next closest concepts.

As I'm not familiar with this area, my questions are:

Is there a better way to do this?
What tools are available for each step?


texts: In industry, what type of new data science algorithms does one develop?

texts:
I've seen several job descriptions for data science which include developing a novel algorithm to be a part of production environments.
Can you give some input of what could be meant here exactly? Would they mean an algorithm that behaves somewhat of an ETL: getting the data, cleaning it, storing it, and running a known model on it? Or something more complex like building variations of known prediction algorithms? Some examples will really be nice, as I'm studying to go into this field.


texts: The OLAP (On-Line Analytical Processing) cube with 4 dimensions

texts:
A typical OLAP cube looks like this:

As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts?
Should I use star schema instead when having more than 3 dims?


texts: Understanding The Vertical and Horizontal stack in conditional gated Pixelcnn paper

texts:
I found some confusion understanding the importance of vertical and horizontal stacks as a solution to the blind spot problem presented in the original pixel cnn architecture discussed in this paper. The vertical and horizontal stacks ideas were presented in the this paper.

Therefore, after browsing, I found this link to explain the concept.
In the vertical stack section of the web page, I still find that pixel f, still cannot see pixels c, d and e.

Any help is much appreciated!!


texts: How do I assess which sentiment classifier is best for my project?

texts:
I wanted to perform Twitter sentiment analysis on Twitter tweets. When I googled it, I found a stanfordNLP code of sentiment class. I used it for finding Twitter sentiment later I came to know TwitterGate model is made for Twitter tweets only.

Then I replaced the stanfordNLP model with TwitterGate model. I did not find any difference in the sentiment results. Both models give the same result.

However, when I tested the same tweets with Monkeylearn API and Datumbox API sentiment, I got confused with results. In NLP and twitterGate most tweets sentiment is given as negative, where the same tweets sentiment is given as neutral in Monkeylearn & Datumbox APIs code.

How can I know which model is giving the correct sentiment?


texts: Keras: Misunderstanding what Keras does when invoking a Model, providing an Input?

texts:
In a class named Generator, I have defined a model with the below method define_model:

def define_model(self):
    conv2d = Generator.__last_block()
    output = Activation('tanh')(conv2d)
    model = Model(self.input_layer, output)
    return model

In another file, I am writing the entry-point of my Python program, which calls the generator's define_model method:

generator = Generator()
generator_model = generator.define_model()

input_low_resolution = Input(shape=low_resolution_shape)
generated_high_resolution_images = generator_model(input_low_resolution)

As I'm reading a course, I however have a queston about this code: what does Keras do when it executes the line generated_high_resolution_images = generator_model(input_low_resolution)?

As far I can understand, it doesn't define a model (my model is already defined thanks to generator.define_model()!). And since the Keras methods train, fit or other aren't called, I deduce that this line doesn't train the model.

By the way, it's a really weird line for me, because it passes a parameter to a reference (the reference to the object is generator_model and the parameter is input_low_resolution). Normally we pass parameters to the reference's methods (constructor and other methods).


texts: Why is the curve for alternative hypothesis shifted compared to that of null hypothesis?

texts: In hypothesis testing the concept of type 1 and type 2 error is very important and very often we come across the graph shown below. I have trouble understanding why is the curve for alternative hypothesis shifted compared to the null hypothsis?


texts: how do the number of classes in an object detection model affect accuracy?

texts:
If I have, say, a Yolo or RetinaNet Object Detection Model... if I train it with 10 vs 50 classes, (assuming 3000 training data images per class), will the model with 10 classes perform similarly to the model with 50 classes? Is there a 'soft limit' to the number of classes a model can successfully hold in 'weight memory' ?.
I notice for most COCO examples, the class # is set at 80. Is there deteriorating performance when they push that number up into the 200s?
Is there any benchmarks done on this type of question? I would assume its a well discussed problem, i.e. splitting object detection of many classes across multiple trained models or packing them all into a single model?


texts: What models are used by Google Cloud Natural Language?

texts: To restate the question posed in the title, what models are used by Google Cloud Natural Language? I know that it says "AutoML," but is there any indication of what kinds of models are used behind the scenes. Any insights  would be greatly helpful.


texts: I'm conducting a machine learning project and therefore looking for a world's wine's dataset to create a classification model based on user's ratings. Can someone recommend me any datasets? Haven't found such public datasets that contain enough data and features.


texts: How to do vocabulary estimation based on observed writings?

texts:
Below is a scatter plot of the data set I am dealing with. The X axis is the total number of words per essay for a particular individual, and they Y axis is the number of unique words. In principle, the number of unique words should approach the individuals vocabulary.

I am attempting to find that individual's vocabulary from the data below, but I don't know what kind of a fit would work. A logarithm would have no limit, a quadratic fit doesn't make sense (the gradient should remain non-negative over the entire domain).

In short, I am looking for a decent model to fit the data below, and don't know where to start.

Thank you.


texts: Is a correlation matrix meaningful for a binary classification task?

texts:
When examining my dataset with a binary target (y) variable I wonder if a correlation matrix is useful to determine predictive power of each variable.

My predictors (X) contain some numeric and some factor variables.


texts: application of Histogram of oriented gradients in colored image

texts:
I recently learned about face recognition with deep learning here. One of the approach involved is Histogram of oriented gradients which is used for face detection as follows (short summary) :

convert image to gray scale
look at every pixel in image and detect surrounding pixel
draw and arrow in direction where surrounding pixels are getting darker
repeat the whole process

this results in HOG version of image ,something like this :

Question Can we use similar method in colored images like how does rgb varies at pixel level or how does pixels vary in colored images to improve the accuracy of face detection and similarity.

Purpose face comparison is a bit tedious in this approach in terms of accuracy. So, i want to find some sort of hash function/value(just using as layman term) to derive unique value from every image. This might improve face comparison easier.

Additionally , sharing any already implemented approach in this direction will be highly appreciated.


texts: What is the difference between data singular and data plural ?

texts: I don't understand the difference between these terms and google didn't help much. Please answer the question even if you have any slightest idea.


texts: cost function for probability estimation on reduced range

texts:
the data: tabular data (low number of columns).  the target is binary, available on the train data only.
the task: i need to predict probability of a the target to be "safe"
the way the prediction will be used: if the probability of safe=True is less than 0.6, the user will be advised to switch item (sorry i cannot give full details of the items).
the user will be presented with alternative items with predicted probability above 0.8.
initial thoughts: i am quite ignorant to probability estimation errors below 50% or above 90%. however, in the range of 50%-90%, i would like to have good accuracy.
therefore, simple logloss is not a very good cost function.
what cost function would be appropriate here?


texts:
I have seen many examples of this syntax that is being used for the loss function specifically:
loss = nn.BCEWithLogitsLoss()(pred, y)
Can anyone explain me what does the (pred, y) do exactly, that it directly computes the loss, instead of calling loss as a function of these 2 arguments again?


texts: Why logistic regression example code does not port to linear regression example?

texts:
I am looking at this tutorial: https://www.dataquest.io/mission/74/getting-started-with-kaggle

Following is the code for linear regression to predict, based on some variables, the survival of the titanic passengers.

# Import the linear regression class
from sklearn.linear_model import LinearRegression
# Sklearn also has a helper that makes it easy to do cross validation
from sklearn.cross_validation import KFold

# The columns we'll use to predict the target
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

# Initialize our algorithm class
alg = LinearRegression()
# Generate cross validation folds for the titanic dataset.  It return the row indices corresponding to train and test.
# We set random_state to ensure we get the same splits every time we run this.
kf = KFold(titanic.shape[0], n_folds=3, random_state=1)

predictions = []
for train, test in kf:
    # The predictors we're using the train the algorithm.  Note how we only take the rows in the train folds.
    train_predictors = (titanic[predictors].iloc[train])
    # The target we're using to train the algorithm.
    train_target = titanic["Survived"].iloc[train]
    # Training the algorithm using the predictors and target.
    alg.fit(train_predictors, train_target)
    # We can now make predictions on the test fold
    test_predictions = alg.predict(titanic[predictors].iloc[test])
    predictions.append(test_predictions)

import numpy as np

# The predictions are in three separate numpy arrays.  Concatenate them into one.
# We concatenate them on axis 0, as they only have one axis.
predictions = np.concatenate(predictions, axis=0)

# Map predictions to outcomes (only possible outcomes are 1 and 0)
predictions[predictions > .5] = 1
predictions[predictions <=.5] = 0

accuracy = sum(titanic["Survived"] == predictions)/len(predictions)

print (accuracy)

This is clear. For logistic regression is much easier:

from sklearn import cross_validation

# Initialize our algorithm
alg = LogisticRegression()
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

Now, why can't I do just, for LinearRegression:

from sklearn import cross_validation

# Initialize our algorithm
alg = LinearRegression()
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

This gives me a totally wrong result. Why is that? What is wrong in the last snippet of code?


texts: What is the immediate reward in value iteration?

texts:
Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration:

$$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \gamma V_i(s'))$$

what is $R_a(s,s')$ ?

The problem I'm having is that terminal states have, by default, $V(s_T) = R(s_T)$ (some terminal reward). Then when I'm trying to implement value iteration, if I set $R_a(s,s')$ to be $R(s')$ (which is wha I thought), I get that states neighboring a terminal state have a higher value than the terminal state itself, since

$$ P_a(s,s_T) ( R_a(s,s_T) + \gamma V_i(s_T) ) $$

can easily be greater than $V_i(s_T)$, which in practice makes no sense. So the only conclusion I seem to be able to get is that in my case, $R_a(s,s') = R(s)$.. is this correct?


texts: Getting unexpected result while using CountVectorizer()

texts:
I am trying to use CountVectorizer() in a loop, But I am getting an unexpected result. On the other hand, if I use it outside the loop then it works fine. I believe there is some small problem with the logic.

from sklearn.feature_extraction.text import CountVectorizer

keys_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']
keys_2 = ['horror', 'fear', 'shock', 'panic', 'scream']
keys_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny amusing fear passionate')

for i in range(3):
    keys = 'keys_' + str(i+1)
    cv = CountVectorizer(vocabulary = keys)
    data = cv.fit_transform([text]).toarray()
    print(data)

cv1 = CountVectorizer(vocabulary = keys_1)
data = cv1.fit_transform([text]).toarray()
print(data)

cv2 = CountVectorizer(vocabulary = keys_2)
data = cv2.fit_transform([text]).toarray()
print(data)

cv3 = CountVectorizer(vocabulary = keys_3)
data = cv3.fit_transform([text]).toarray()
print(data)

Output

  [[0 0 0 0 0 0]]

  [[0 0 0 0 0 0]]

  [[0 0 0 0 0 0]]

  [[1 1 0 0 0]]

  [[0 1 0 0 0]]

  [[0 0 1 0 0]]


texts:
I have those rules:

and those two possible parse trees:

I am asked for the next question:
What is the more natural parsing, the one that leads to the preferred reading of the sentence?
Can anyone explain to me, what is more natural in English and why ?
according to this
A determiner is a noun-modifier that expresses the reference of a noun or noun phrase in the context.

I don't see any possible more natural distinguish.


texts: Are there libraries or techniques for 'noisifying' text data?

texts:
Data augmentation techniques for image data and audio data (eg speech recognition) have proven successful and are now common.

Are there libraries or techniques for augmenting text data?

For example:

  in: 'How are you?'
  out: ['how are you?', 'HOW ARE YOU?', 'hwo are y ou?', 'How're you?', 'how r u', ...]


texts: Distinguish Multi-Task vs Single-incremental Task in Continual Learning

texts:
I read through the Internet and found this:

  Most of Continuous Learning studies focus on a Multi-Task scenario, where the same model is required to learn incrementally a number of isolated tasks without forgetting the previous ones.

And they explain PermutedMNIST is Multi-Task. I have no idea why?

In other words,

  Single-Incremental-Task considers a single task which is
  incremental in nature. In other words, we still add new classes
  sequentially but the classification problem is unique and when
  computing accuracy we need to distinguish among all the classes
  encountered so far.

However, I still can not clarify two these definitions? Could someone help me out and give me example of Multi-Task scenario and Single-Incremental-Task scenario.


texts:
I worked on a machine learning project where we dealt with relatively small data sets. I noticed that the way that we tried to increase performance was basically to try out a bunch of different models with different hyperparameters, try out a bunch of different sets of features, etc. Basically, it seemed like we approached the problem fairly randomly and that we had no real theoretical basis for anything we tried. This disillusioned me a fair amount and made me think if this is what machine learning engineers do in practice.
Have people found that this is fairly common? How can you work on a machine learning problem in a non-random-try-everything kind of way?


texts: Cluster a categorical variable without breaking the existing categories

texts:
Let’s say I am modelling data on flights and delays. I have 14 airlines (carrier variable) in the data below. I would like to know if I can create fewer categories or clusters of this categorical data, so instead of having 14 airlines maybe end up with 3 or 4 clusters of airlines.

require(data.table)
# Data takes ~30 seconds to load
flights <- as.data.table(read.csv(url("https://raw.githubusercontent.com/wiki/arunsrinivasan/flights/NYCflights14/flights14.csv")))
length(unique(flights$carrier))

km <- kmeans(flights[,.(dep_delay,carrier)], centers = 4, nstart = 1)

Is there any methodology that could be used? I tried k-means but it seems it is not the most appropriate.


texts:
When training my LSTM, is there any incentive to pre-pass its inputs through an auto-encoder, or should I always supply as raw data as possible?

I have is a small amount of training data, and it contains only a few very rare significant events (spikes, spaced out in time)


texts:
I am trying to do hyperparameter tunning for the Randomforest regression model.
I'm using RandomizedSearchCV (scikit-learn) and I defined verbose=10.
For that reason, I'm getting messaged while it's running and I would like to understand them a bit better.
Those are my parameters for RandomizedSearchCV:
from joblib import Parallel, delayed, parallel_backend
# Use the random grid to search for best hyperparameters
# First create the base model to tune
rf = RandomForestRegressor(-1)

with parallel_backend('threading',n_jobs=12):
    rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 12, cv = 3, verbose=10, random_state=42)#, n_jobs = -2)
    # Fit the random search model
    rf_random.fit(x_train, y_train)

After running a while )and is still running) I have those messages:

my questions are:
1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough
2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV?
3. "Done one task" - what is the task? one CV?
4. Why do I get score only to some of the parameters?


texts:
I have a list user data: user name, age, sex, address, location etc., and

a set of product data: Product name, Cost, description etc.

Now I would like to build a recommendation engine that will be able to:

1 Figure out similar products

eg :

name  :   category   :  cost    :   ingredients

x     :     x1   :        15  :       xx1, xx2, xx3

y     :    y1   :        14   :     yy1, yy2, yy3

z     :    x1  :          12   :     xx1, xy1

here x and z are similar.

2 Recommend relevant products from the product list to a user

How can I implement this using mahout?


texts: How to use pre-trained weights to initialize the custom CNN?

texts: From this paperhere, it shows that U_Net initialized by VGG received a better result than the one trained from scratch. Now I want to build a custom u_net which has [32,64,128,256] which is different from vgg.features (I have tried to use U_Net with VGG architecture without pre-trained weights and custom U_Net, they all get the same Iou score .This means that the sample architecture suits for my data. ) How can I get the initialized weight from VGG? As far as I know, knowledge distillation could help me to get a simple net whose weight could be used for initialization for my u-net like ([32,64,128,256]). But knowledge distillation on ImageNet data set would cost lost of time, Is there another way to initialize the ‘smaller’ U-Net([32,64,128,256] )?


texts: Can someone explain to me how to use a predictive model to predict something other than the training set

texts:
So let's say I create a logistic model to predict who will open a loan based on a based email list that includes who opened and who didn't that's 90% accurate. The model says age, income, bank engagement are three key variables that decide who opens a loan. Is there a way to apply this model to a different email list with the same variables to predict who will open a loan? Or what percentage of the people will open the loan? Or will I just need to analyze the data on the list myself to determine this.
Sorry, probably a dumb question, but it's one thing I've struggled to figure out on my data scientist journey.


texts: Variance in statistics vs machine learning

texts:
In basic statistics, variance is a measure the variability of the data about its mean. In machine learning, variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data.

Since machine learning is mostly refurbished statistics, how does these two concepts relate to one another


texts: Using categorical_crossentropy for binary classification

texts: Is it ok to use categorical_crossentropy for binary classification or is it better to use binary_crossentropy


texts: What does the color coding and normalized values in confusion matrix actually specify?

texts:
I am unable to infer anything about the model from the following confusion matrix. What is the color coding actually specifying?

For example, when predicted label is 1 and true label is 1, the value in the matrix at that point is 0.20. Does that mean its accuracy? Does it mean that the model is only able to predict 1 only 20% of times when the label is actually 1?

PS: SO URL for a more clear image


texts: Forecasting an individual based on a representative group

texts:
I’m trying to create a model to forecast demand/expenditure for an individual based on historical data of many individuals over time, but I’m having trouble finding examples of this.

To give a simple example of what I’m trying to do:

Let’s say there’s an ice-cream store that children in the surrounding area visit every week. We’ve got the history of all purchases at the store and details of who made the purchase, so we can track the purchase history of each individual. We also have details about each individual, e.g. age, gender, school, no. of siblings etc.

We can assume that a child will buy from the store once a week until they ‘leave’ for good and stop purchasing.

What I want to do is create purchase forecasts for each child over a few weeks. The bit I’m struggling with is where a new child, say Timmy, has only been purchasing for a few weeks and so doesn't have enough of a history to forecast based solely on his own data (but we do have his details).

My assumption is that demand can be better predicted by taking a child's details into account, so I'd like to find a group in the historical data that best represents Timmy, and leverage their data in some way when creating Timmy's forecast. Are there any standard methods/practices for this type of forecasting?

Any advice or guidance would be much appreciated.


texts: How to use Statistical Learning theory in real analysis

texts:
I begin the post trying to say that i don't know if this post is in compliance to community rules, so pardon me for any abuse.

I studied back at the university statistical learning theory. I studied PAC learning, VC Dimension, Uniform convergence etc etc. Recently i watched this talk with Vapnik https://www.youtube.com/watch?v=STFcvzoxVw4&t=1346s in which he claims that deep learning is essentially a "bla bla interpretation" and also claims that 'every problem can be solved with statistical learning theory'.

I'm very confused about this. I can't see how I can apply statistical learning theory on real problems.
Let's suppose I'm facing a new dataset with a clear task of binary classification, with many features and lots of training data. How am I supposed to check for example, if a hypothesis class H is PAC learnable, or in other words if it has a finite VC dimension? Don't take my example too literally, I just want to know if someone can point me out to an article,blog or some kind of answer that clearly shows how we can use this theorems and results in a real analysis.

Thank u


texts: How to visualise a large correlation matrix?

texts:
I have a dataset with 24 variables, 21 of them numeric. As part of model building I decided to look into the correlation between features and so what I get is a large correlation matrix (21 * 21).
Now visualising such large matrices becomes a very messy task and you end up hurting your eyes. So what I have done is set a threshold and to slice out those rows that have greater than this value (say 0.60). However, I'm getting a matrix that has now several NaNs. When I try to drop these null values, the matrix loses all data and what I'm left is a 0*0 matrix.
corr_matrix = data.corr()

threshold = 0.60

high_corr = corr_matrix.loc[corr_matrix >= 0.60]

high_corr.dropna(inplace=True)

print(high_corr)
Empty DataFrame
Columns = []

Visualising the matrix with nans is a good idea but it also results in empty squares. I'm looking for a way where only those rows that have values >= threshold are retained, with no nans. That would make a much smaller matrix which is much less messier when plotted in matplotlib. However I haven't been able to code it that way; can anyone suggest some strategies to deal with such large matrices?


texts: Training a Siamese Neural Network for object similarity assessment

texts:
I am training a Siamese neural network with pairs of similar and dissimilar objects. The features of the objects are binary data on whether they contain some properties or not (2048 features per object).

I then split my dataset into training, validation and test set (60:20:20). Afterwards, I prepared the dataset myself by pairing up at random the objects accordingly yielding 50% similar and 50% dissimilar pairs and I augment the data in the training set by generating extra pairs by random (resulting in 100,000 different instances, again a balanced dataset (50:50), vs. 1,000 instances for the validation set). I then proceed to train the Siamese network and end up estimating the cosine distance between the two outputs to get a similarity metric which is compared to my label with the binary cross entropy loss function. The learning rate used is low (lr = 0.0001) and I am using the Adam optimiser. I have tried producing really small batches (batch_size = 25), adding dropout and increasing the number of instances to avoid overfitting, but the model does not seem to generalise well regardless (see picture). I was wondering if anyone could give me any hint on what is going on - and also why is it that such bumps can be appreciated during the learning process -.


texts: Will I be able to fetch a Data Analyst job if I learn everything in this image?

texts:
India

Asking such a question because I am interested learn everything myself and I am confused about what to read.
And I am interested by reading books, will that suffice? or should I compulsorily attend a course and get certificate(because some say I might require certificate compulsorily)??
Please can somebody throw some light on this.(Please do if you are an Indian based Data Analyst or Data Scientist)
Thank you.

**I AM A COMMERCE GRADUATE AND INTERESTED IN DATA SCIENCE


texts: How to work with predictor variables which are all categorical but the target variable is an integer?

texts: I have historical data with time (hour, day, etc.) and region code linked to number of events in the integer format. I was starting to build a model in R to predict number of events given time and region. I explored SVM in R but seems like it is a non-starter with categorical variables. Any other approach which might of use here. Thanks for your response.


texts:
In CNN transfer learning, after applying convolution and pooling,is Flatten() layer necessary?
I have seen an example where after removing top layer of a vgg16 ,first applied layer was GlobalAveragePooling2D() and then Dense().
Is this specific to transfer learning?
This is the example without Flatten().
base_model=MobileNet(weights='imagenet',include_top=False) #imports the mobilenet model and discards the last 1000 neuron layer.

x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(1024,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
x=Dense(1024,activation='relu')(x) #dense layer 2
x=Dense(512,activation='relu')(x) #dense layer 3
preds=Dense(3,activation='softmax')(x) #final layer with softmax activation

This example is with Flatten().
vgg = VGG16(input_shape=IMAGE_SIZE + [3], weights='imagenet', include_top=False)

# don't train existing weights
for layer in vgg.layers:
  layer.trainable = False

  # useful for getting number of classes
folders = glob('Datasets/Train/*')

# our layers - you can add more if you want
x = Flatten()(vgg.output)
# x = Dense(1000, activation='relu')(x)
prediction = Dense(len(folders), activation='softmax')(x)

# create a model object
model = Model(inputs=vgg.input, outputs=prediction)

What is the difference if both can be applied?


texts: How to handle negative delays when predicting flight delays?

texts:
I am working with the nycflights dataset. My goal is to predict departure and arrival time delays using random forests. My problem is that some flights have a negative delay, meaning they departed/arrived earlier than the scheduled time. So far, I have thought of 2 solutions to this:

Dropping the rows with a negative delay and considering only the records with an actual delay.
Replacing all the negative delays with the mean delay.

I am, however, worried that doing any of the above might have a negative impact on the prediction accuracy, since I am not taking into account the negative delays in any way.

What is the best way, in this case, to handle the negative values?


texts: Learning Rate based on error of the network

texts:
I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be.

Is there an algorithm to dynamically update learning rate based on total error of the network without relying on any hyper-parameters ?


texts: Genetic Algorithm to find best parameter values of an estimaor

texts:
I am making some stochastic training ensemble classes in Python, and I want to get hyperparameters values. Grid search will take too long for moderate data sets, because in my stochastic training I train different learners on different dropouts, as in cross-validation, then I average or weight-average on best error for dropout. Having this set, I want to get best parameter values at every iteration, so I thought to use a genetic algorithm approach.

I found one example on Github, but failed to install, and I am looking for more references of genetic algorithms for doing this. Can anyone help me?


texts: Which type of clustering algorithm to use to identify the "same" item in different data sets?

texts:
I'm trying to find a solution for a data quality problem - specifically, identifying which items in different data sets are used to represent the same things.

As an example, assume that we're a retailer and we buy out a couple of other retailers.  In the process, we also get their systems and databases.  This might lead to some overlap - different systems can represent the same items, customers, etc. in different ways, but with no single unique identifier.

What would the best approach be to determine which rows represent the same thing across data sets in order to come up with a single view of 'unique' entities?

I've done a machine learning course, and I understand the bare minimum.  I believe that the solution to this problem requires a clustering algorithm, but what type?  I may be dealing with a multitude of features in the data - dimensions, names, dates, contact details - and some of those would probably require a higher 'weight' in matching.

Examples (Items):

A:  ABC Notebook, Large, Released: 2018-02-20, 150mm x 100mm.

B:  Notebook (ABC), L, Date: 18/02/2018, 150mm x 100mm

I'd expect these to be treated as the same item.

Examples (Customer):

A:  Doe, Jane, DOB 1970-06-23, 123 ML Ave, F

B:  John Doe, DOB 1971-04-33, 123 ML Avenue, M

C:  J. Doe, Born '71

I'd expected B and C to be identified as the same person, but not A.

For this scenario, I don't expect to come up with something with a 100% accuracy, but I would like to be able to come up with a (narrow) list of possible matches that someone can check.

Can some please point me in the right direction?  Any case studies that I should look at?


texts: Dissimilarity Matrix of non-metric proximity data

texts:
we currently have a coding exercise, where we are asked to implement Constant Shift Embedding (Paper). This in itself is not a big problem. For the algorithm, all you need is a symmetric non-zero diagonal dissimilarity matrix of some non-metric proximity data. With the algorithm you can then embed the information into a vector space and therefore you can use commonly known denoising and dimensionality reduction methods to improve the results of for example k-means clustering.

Given the E-Mail communications based on this data set, how would go about choosing a reasonable dissimilarity matrix?

The data is simply a list of unique pairs, where at least one e-mail has been sent from node A to node B. This gives rise to a graph of around 1000 nodes and 25000 edges.

Creating an adjacency matrix of this undirected graph might be a first step (which is also already provided in the framework).

I'm thankful for any pointers in the right direction.

EDIT: Over night I had an idea:

Let's say we only have 8 nodes. Now compare the proximity elements of two vertices. So if the prox. vectors would for example look like:

1 0 0 0 1 0 1 1

0 1 0 0 0 1 0 1

Their dissimilarity would be 5, since their vectors differ at 5 points.

Now just normalize w.r.t. the total number of nodes, therefore 5/8.

With this, we also incorporate the information of how many neighbors are shared instead of only looking at direct edges, and might therefore receive better results, when we later try to cluster the nodes.

Let me know what you think.


texts: How Ipsative Standardization actually works for time series?

texts:
In this link: http://optimalprediction.com/files/pdf/V2A17.pdf
Ipsative Standardization is explained intuitively and also the why it is useful.

Is there any python code or math formulas online someone could use to calculate the z1 score?


texts: Keras/TensorFlow multi inputs ouputs network that compile and give a stange error

texts:
I'm working on a networks with 4 images in inputs and 4 output. Here is a skeleton :
import tensorflow as tf
import tensorflow.keras as K
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.callbacks import ModelCheckpoint, LambdaCallback
from tensorflow.keras.models import load_model, Model,Sequential

...
inputs=[ Input((ytrain,xtrain,3)) for i in range(4) ]
#Features network
visible1 = Input(shape=(ytrain,xtrain,3))
#few convolutions
extractor = Model(visible1,conv2)

#upscale network
def up():
    little =Input((ytrain//8,xtrain//8,100))
    .....
    big = Conv2D(1, kernel_size=5, activation='relu',padding='same',dilation_rate=1)(c4)
    m = Model(little,big, name='upscaler')
    return m

#Main network
features =[extractor(inputs[i]) for i in range(4)]
CostLayers = [CustomCostLayer(100,i) for i in range(4)]
Costs= [CostLayers[i](features) for i in range(4)]
mapsLow = [tf.nn.softmax(Costs[i],axis=2)for i in range(4)]
upscale = up()
outputs = [upscale(mapsLow[i]) for i in range(4)]
bigNet = Model(inputs,outputs)
bigNet.summary()

# A data generator
X,Y = genLists(lines)
training_generator = DataGenerator(X,Y ,BATCH_SIZE)
bigNet.fit(training_generator,epochs=1500,callbacks=[checkpoint])

The model compile, Training started then after few batch, there is a memory overflow. I noticed something strange in my history
 6/6234 [..............................] - ETA: 3:12:36 - loss: 73945.1797 - upscaler_loss: 9859.3467 - upscaler_1_loss: 15327.2412 - upscaler_2_loss: 22759.8828 - upscaler_3_loss: 25998.7090<0x08><0x08><0x08><0x08><0x08><0x08><0x08><0x08><0x08><0x08>...

Any idea?
Config : uptodate (2 or 3 weeks) keras compiled from source
With a train on batch :
ValueError: No gradients provided for any variable: ['conv2d/kernel:0', 'conv2d/bias:0', 'conv2d_1/kernel:0', 'conv2d_1/bias:0', 'conv2d_2/kernel:0', 'conv2d_2/bias:0', 'conv2d_3/kernel:0', 'conv2d_3/bias:0', 'conv2d_4/kernel:0', 'conv2d_4/bias:0', 'conv2d_5/kernel:0', 'conv2d_5/bias:0', 'conv2d_6/kernel:0', 'conv2d_6/bias:0', 'conv2d_7/kernel:0', 'conv2d_7/bias:0'].


texts: What is the meaning of ValueError in Keras? - 'Tensor conversion requested dtype complex64 for Tensor with dtype float32'

texts:
I have the following part of code -

...
model.add(Generic_act_func())
print('layer 1 output')
print(model.layers[1].output) #Here layers[1] is Generic_act_func() layer specified above
model.add(Lambda(lambda x: tf.spectral.irfft2d(x)))
...

Which when run produces the following output and error respectively -

Output -

layer 1 output
Tensor("generic_act_func_1/Complex:0", shape=(?, 28, 28, 9), dtype=complex64)

Error -

ValueError: Tensor conversion requested dtype complex64 for Tensor with dtype float32: 'Tensor("lambda_1/Placeholder:0", shape=(?, 28, 28, 9), dtype=float32)'
at line

model.add(Lambda(lambda x: tf.spectral.irfft2d(x)))

tf.spectral.irfft2d() requires a complex64 input which the previous layer i.e., Generic_act_func() layer is already producing (as printed by layer 1 output), hence I am not able to figure out why this error. Thanks in advance for any help for solving the error.

Edit - Could there be a mistake wrt passing previous layer outputs to the Lambda layer? Like I have used a variable 'x' in the Lambda function assuming the previous layer output will be captured in this variable. Is this the correct way of adding the Lambda layer? (I am a beginner in Keras, so will be really grateful for any help)


texts:
I have a text file which consists of ~10k triples in the format,

Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(Noun Phrases) from the query and applying similarity search.

It takes a lot of time ~10 minutes to fetch the results and it isn't a scalable solution since, the size of triple file is directly proportional to search time.

How can I reduce the time?


texts:
I would like to know what are the best practices for anonymizing datasets?
Ideally I should be able to get the original data back after performing analysis on the anonymized dataset. Should I be using some encryption functions? Hashing maybe?


texts:
Can anyone help identify what kind of Visualization this is?
(google image search was not informative, unfortunately...)


texts: How to extract characteristics from text using machine learning?

texts:
I would like to develop some kind of model/algorithm that allows me to extract the characteristics of a given product name. (let's say the brand, model and color).

I am looking for a solution similar to the one offered by MonkeyLearn and its model Laptop Feature Extract.

For example:

Given the item "Apple iPhone 6s, 64GB Silver", It should compute:

{
  brand: "iPhone",
  model: "6s",
  capacity: "64Gb",
  color: "Silver"
}

Any suggestions will be appreciated.
Thank you.


texts: Dimension Mismatch Error during dot product in Python

texts:
I have two matrices user_vecs and item_vecs

I am trying to take the dot product of the two to build a recommendation engine:

The shape of the two vectors are as follows:

user_vecs.shape

(20051, 20)

item_vecs.shape

(20,1808)

When i take the dot product of the transpose as follows:

a = user_vecs.dot(item_vecs.transpose())

I get the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-41-f4cd01978711> in <module>
----> 1 a = user_vecs.dot(item_vecs.transpose())

C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\base.py in dot(self, other)
    362
    363         """
--> 364         return self * other
    365
    366     def power(self, n, dtype=None):

C:\ProgramData\Anaconda3\lib\site-packages\scipy\sparse\base.py in __mul__(self, other)
    479         if issparse(other):
    480             if self.shape[1] != other.shape[0]:
--> 481                 raise ValueError('dimension mismatch')
    482             return self._mul_sparse_matrix(other)
    483

ValueError: dimension mismatch

i understand that the dimensions of the two matrices are not matching, but the transpose should have fixed that. Why am I still getting this error?


texts:
I have a dataset as below:

     Key Attr1 Attr2 Attr3 Attr4 Attr5 Attr6
     kd1 l1    l2     l3    l4   l5     l6
     kd1 l1    l7     l8    l9   l5     l10
     kd1 l11   l12    l13   l14  l5     l10
     kd1 ..................................
      .
      .
      .
     kd2 ..................................
     kd2 ..................................
      .
      .
     kd3 ..................................
      .
      .
      .

For each instance, I have multiple combinations of target outputs(Attr1-Attr6).
Whenever I use multilabel libraries, I get a single combination of outputs.

I want a ranked list(top 3) of target label classifications for each key given as input.

For example:

predict('kd1') should return the following:

res = [ [l1,l7,l8,l9,l10], [l1,l2,l3,l4,l5,l6], [l11,l12,l13,l14,l5,l10] ]

Here res[0] is the best combination, res[1]  is the second best combination and so on.

How do I go about that?


texts: What is the difference? "Adding more LSTM layers" or "Increasing epochs"?

texts: To get more accurate results which one is better? Adding more layers or increasing number of epochs? I like to know the difference between effects of these two approaches?


texts: Transfer learning by using vgg in pytorch

texts:
I am using vgg16 for image classification. I want to test my transfered model with the following code:
classes = ['A', 'B', 'C']
len(classes) #3
len(test_data) #171
batch_size=10

# Testing
test_loss = 0.0
class_correct = list(0. for i in range(len(classes)))
class_total = list(0. for i in range(len(classes)))

vgg16.eval()

for data, target in test_loader:
    output = vgg16(data)
    loss = criterion(output, target)
    test_loss += loss.item()*data.size(0)
    _, pred = torch.max(output, 1)
    correct_tensor = pred.eq(target.data.view_as(pred))
    correct = np.squeeze(correct_tensor.numpy())
    for i in range(batch_size):
        label = target.data[i]
        class_correct[label] += correct[i].item()
        class_total[label] += 1

test_loss = test_loss/len(test_loader.dataset)
print('Test Loss: {:.6f}\n'.format(test_loss))

for i in range(len(classes)):
    if class_total[i] > 0:
        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (
            classes[i], 100 * class_correct[i] / class_total[i],
            np.sum(class_correct[i]), np.sum(class_total[i])))
    else:
        print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))

print('\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (
    100. * np.sum(class_correct) / np.sum(class_total),
    np.sum(class_correct), np.sum(class_total)))

I receive following error:
I receive following error:
     15     for i in range(batch_size):
     16         label = target.data[i]
---> 17         class_correct[label] += correct[i].item()
     18         class_total[label] += 1
     19

IndexError: too many indices for array
I do not know why I am getting this error and how I can solve it. I would be grateful if you could help me.


texts: forecast product demand in one week using machine learning approach

texts:
I'm trying to predict product demand in store. The predictors I have include price,  competitor's price, store ID, date.  My target variable is sales volume (in a particular store). What I need to predict is sales in a store one week in advance.

I'm thinking about how I should approach this problem. The goal is to predict sales one week in advance,  I'm a little confused about how I should use predictors for prediction. How I should do the model validation (for predictions one week in advance). How can I incorporate seasonality in the model if I use random forest...

Really appreciate it if anyone can help clarifying this. Thanks a lot!


texts: Is my data good for (DBSCAN) clustering?

texts:
I have a particular dataset consisting of 50k elements with 40 features each. I want to try to cluster the data as it is, without any dimensionality reduction. The main algorithm I am considering is the DBSCAN since is the more versatile and I can accept some poits to result as noise. However how can I judge if the clustering is "significant" since I can't plot the clusters in comparison to the data?
Tring to select the paremeters for the DBSCAN I've done a k-nn analysis, but the results worried me. For example, the following is the third nearest neighbor plot.

As you can see the distances (y-axis) are pretty much "uniform" along the x-axis (object). Does it mean that the data is somehow uniformly sparse and, in this conditions, clustering is useless?

P.S. I have tried to cluster the data anyway, in particulary using epsilon around 2-2.5 and for different values of min_sample. The silhouette score however results very low, about 0.11, and the noiseless fraction of points is about 80%.


texts: Is it possible to increase the number of images of one class using data augmentation, which is not applied on the other class, in the same dataset?

texts:
I have 2 classes for my image classification problem, say class A and class B, and I am using tensorflow and keras for the same. One of them have around 5K images while the other have just around 2K images. Is it possible to increase the number of images in class B to get them near to 5K using data augmentation? I have looked for more images to include under class B, but all I could find were already among the 2k images.
I only see articles which say that data augmentation is done on the dataset, but I need it to be done on one of the classes alone, without affecting the other class in the same dataset
I am using image_dataset_from_directory for loading the dataset. If there are any alternatives for loading the data which aids in the above mentioned situation, please suggest me.


texts: Approaches for implementing Domain specific Question answering System

texts:
Given several wikipedia articles on different movies.
What are the different approaches to implement a QA system to answer different quires related to movies.

Dataset : Wikipedia articles

Input   : natural language query, eg : who directed terminator ?

Output  : The Terminator is a 1984 American science fiction action film directed by James Cameron

Different approaches available are :

1 IR Based approaches

In IR based approaches the similarities between query and passaeges are considered and best suitable passages are selected as answer (as Google does)

2 Ontology based

In Ontology based prior domain knowledge is required about the domain. And documents need to be mapped accordingly.

3 Machine learning based

In ML based approaches large training set of documents is needed for training.

What are the other cons and pros of each approaches?

Which of these approaches suits the best for this use case ? Also is there any other methods ?