texts: Find multiple categories of promises in texts

texts:
I have conversations a customer with an agent (without punctuation). There are phrases of several categories of promises that an agent gave to a customer (call back, make an appointment, etc.). It has been done manually. Altogether 12 categories. Now I'm thinking of creating an algorithm for this. I am thinking to do this task in two steps.

In the first step, I need to create an algorithm that can find an
end and a beginning of all promises. This algorithm has to insert a
start tag and an end tag.
The second step is to create a classifier that would label a promise
to the necessary categories.

As I understand, the second step is well known and this is called text classification. But for the first step, I could not find any articles and github repositories. But I think it is an important NLP task and there must be information on this. Maybe are there approaches that solve two steps at the same time?

Update

Just sample an agent's transcript (in reality it is more difficult):

hi my name is ben how can i help you yes good what about i can help probably yes sir do you have a problem with internet connection i see let do you need a help at place okay i see so what i can do i can arrange appointment with technical will it be good for you great can i help you with something else you okey okey to have a great day you too

Promise here is

i can arrange appointment with technical


texts: Cross entropy and function approximation

texts:
My overall question is: the universal approximation theorems can provide a good heuristics on defining the loss function for supervised regression problems, i.e., because universal approximation functions (say, $F(W, x)$) in theory can approximate any ground truth function (say, $f(x)$) with arbitrary precision, in practice when we have finitely many ($N$) data, we can aim to minimize the empirical risk
$$
\sum_{i=1}^{N} (F(W,xi)−f(xi))^2
$$
with respect to the parameters W's. The above loss function is not even strictly dependent on the universal approximation theorems, but is used to determine coefficients for any model function from the available data.
When it comes to classification tasks, we usually minimize the cross-entropy loss function. I have seen derivations of the cross-entropy loss function starting from two probability distributions. But eventually we are still trying to approximate some underlying ground truth function. Can you provide any derivation/explanation of the cross-entropy loss from the function approximation point of view?
Thanks.


texts: For my current use case, I have a high number of (noisy) samples that I do binary classification on. I use a neural network to approximate this with a sigmoid layer as output and log loss as loss function. My question is about the loss function, I understand that we use log loss to penalize probabilities that are far from the truth because of the log likelihood. The problem (or maybe it is not a problem) is that I want to aggregate my results to get the expected amount of positive samples in big groups. If we look at it from that perspective, looking at it as a mean as opposed to a probability also makes sense. In that case, having 2x 0.4 and 1x 0.7 is the same mistake as 1x 0. and 2x 0.75. Does it still make sense to use the log loss or might it be better to use L2 or L1 regression instead?


texts: How to use a single GPU (vs. CPUs) in Tensorflow for forward inferencing (validation) vs. only for training

texts:
As mentioned in this question, it could be useful to harness a GPU vs. CPU(s) for validation.  For example, in cross-validation, where the number of validation examples can exceed the number of training examples in each epoch, this results in a need for speed in forward inferencing because the process of training to convergence must be replicated, especially when using Monte Carlo cross-validation.

The multi_gpu_model' function oftensorflow.keras.utils` has given me trouble (at least since TF2.2 came out), so I am not considering that at this time, although I would like to in the future for both training and validation (currently using only one GPU).  Obviously it would be faster to use more than one GPU in parallel, but that is not the essence of my question; I think I will get a significant speed increase even using just one for now.

I could be completely mistaken that I will get a speed boost, because a significant part of the calculation time for each validation example is probably consumed by a custom loss function I am currently investigating.  However, that exact same custom loss function (not available in tensorflow at present) is calculated at every training step within the epoch, and that takes less than half a second on the GPU, yet validation for 25 examples takes well over a minute with 10 CPU workers!


texts: What is the best way to compare various classification models?

texts:
If I have two classifiers for example Neural network and support vector machine , now I want to know what would be the best way to identify which is good classifier , should it be based on classification accuracy or by analyzing confusion matrix or average precision score, f1 score etc.

I have seen in most of the papers related to image classification that they only take classification accuracy as the parameter to compare different models.


texts: Composing phrases into a grammatically correct sentence?

texts:
I'm wondering if there exist any models which could take in an ordered list of phrases without punctuation and generate a grammatically correct sentence from it.

For example, for the input: ["My dog", "hates that", "guy named Michael Vick"]

The output could be either "My dog hates that guy named Michael Vick!", "My dog hates that guy named Michael Vick.", "My dog hates that guy named Michael Vick...", or even "My dog hates that guy named... Michael Vick.". Ideally, the probability of the sentence would be returned alongside the output.

Is there a specific name for this problem, so I can do a literature search myself?

Thanks!


texts: How to cluster government census data in order to group Metropolitan statistical areas

texts:
I have collected a bunch of census data from 2012 - 2018. I wanted to apply some clustering algorithms in order to compare Metropolitan statistical area (MSA's). Ideally once I run the clustering algorithm I would like to see which MSA is comparable to another.
The features that I am choosing to govern the clustering is below:
'Bachelors+',
'Estimate  Total  $10,000 to $14,999',
'Estimate  Total  $100,000 to $124,999',
'Estimate  Total  $125,000 to $149,999',
'Estimate  Total  $15,000 to $19,999',
'Estimate  Total  $150,000 to $199,999',
'Estimate  Total  $20,000 to $24,999',
'Estimate  Total  $200,000 or more',
'Estimate  Total  $25,000 to $29,999',
'Estimate  Total  $30,000 to $34,999',
'Estimate  Total  $75,000 to $99,999',
'Median Age',
'Median Gross rent as % of household inc',
'Number of educational and health service workers',
'Number of finance and real estate workers',
'Number of people in management, business, science, and arts',
'Number of service workers',
'Number of tech workers',
'Pct Asian',
'Pct Black',
'Pct Other Race',
'Pct White',
'Total Population',
'Total Population over 25'

Now a question I have is the data I have is on the tract level for every MSA in the United States from 2012 - 2018. Would I first need to aggregate the data so that I have the above features by their associated MSA then do the clustering algorithm from there?
From there how do I identify the MSAs by cluster?


texts: Multicollinearity(Variance Inflation Factor). Variables to remove before doing a model

texts:
I am doing an exercise of a Machine Learning System module in python that takes a dataset of cars (cylinders, year, consumption....) and asks for a model, being the variable to predict the consumption of gasoline. As it has three categorical variables, I have generated the dummies.

In the exercise I need to eliminate the variables with multicollinearity, so I used the method showed on my course notes:

from sklearn.linear_model import LinearRegression

def calculateVIF(data):
    features = list(data.columns)
    num_features = len(features)

    model = LinearRegression()

    result = pd.DataFrame(index = ['VIF'], columns = features)
    result = result.fillna(0)

    for ite in range(num_features):
        x_features = features[:]
        y_featue = features[ite]
        x_features.remove(y_featue)

        x = data[x_features]
        y = data[y_featue]

        model.fit(data[x_features], data[y_featue])

        result[y_featue] = 1/(1 - model.score(data[x_features], data[y_featue]))

    return result

Then if I launch the method it calculates a coefficient for each variable:

In my course notes it is said:

$VIF>5$ is a high value.
$VIF>10$ is a very high value

What should I do? I need to remove the variables that have a $VIF>10$ before executing the model?

The problem I see, for my categorical variable cylinders, is only cylinders_5 has a VIF under 10 so should I remove the others and leave cyclinders_5?


texts:
I am reading a research paper that tries to fix some issues with GAN,  but for one of the equations, the paper does not fully explain where it comes from and why it works. Although the overall algorithm for EGAN is straight forward, I wish to understand it a little better.

The paper uses an evolutionary approach to fix stuff like instability and mode collapsing. A fitness function for evaluating the different generators is give as:

Where f_d is:

and f_q is:

What I'm focusing on is f_d = -log||delta_d - E_x[log(D(x))] - E_z[log(D(G(z)))]||.

In f_d, what does delta_d represent. It was not defined in the paper so I figured maybe it was very typical notation for something so I looked around but could not find anything. The only things I know is from what it says in the paper, and the paper that the writers referenced to create the equation (proofs >.<)

What I know right now is the entirety of f_d represents what the paper calls the diversity fitness score. I also know that the second part of the equation is the same as the loss function for a discriminator so it could be re written as f_d = -log(delta_d + loss_d). But what is delta_d? and how does this equation properly evaluate the diversity of the generated samples?


texts: Discrete Values Matrix: Select Rows and Scattered Plot problem

texts:
I ran in a somewhat basic problem with Orange 3.3.1... but wonder if this is a bug or something I don't understand:

I have a Matrix, all with discrete values.
I select Columns, that goes ok
Select Rows... and use the « is one of » filter with 3 values out of 15 in a particular column.
But when I Scatter Plot. Values matching the filter seems to have been selected, but the X Axis still show X names, all of them,  that are not in « is one of ». On top of it, values have shifted to to wrong X name...

Anyone has an idea???

Regards


texts: Definition of a "lexicon" in Named Entity Recognition

texts:
I am writing a paper on Named Entity Recognition and I mention that in the literature were proposed a lot of methods which make use of lexicons. However, I am struggling to find a definition of a "lexicon" which I can cite. As far as I understand, a lexicon is a list of lemmatas (word parts), whole words or word combinations where these are classified to an entity. So, for example if in a sentence i have "New York is located in" and in my lexicon I have the pattern " X is a located in" -> CITY my system will look up the pattern and see that New York belongs to the entity CITY.
Could someone explain to me if I am wrong? If so, I would like to get a definition of a "lexicon", "gazzetteer" and "dictionary", because I get the feeling that some of these are sometimes used interchangeably in papers.


texts: Why models performs better If normalize test data and train data separately?

texts:
Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found like this one and this one that suggest that applying the normalization to the test set is not really required and it might depends on many factors such as the model used for training or the nature of the test data.

Now, personally, I am more inclined towards applying the normalization on test data as well. But the problem is this: I am working on a neural network model where:

If I apply normalization using the recommended way I get 79% accuracy, (and to be honest it's not interesting for me)
If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

So, I don't know what my neural network performs better on test unseen data if I use the second method. I really I want continue using the second method for this particular model, but I don't feel good about it and feels like it's wrong or cheating.

Now, I have one last argument. The last link I provided, have one answer that says this:

"..This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine..."

The dataset I use is a refined version of its predecessor (NSL-KDD dataset). The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

Ps: Sorry if this is long, it's a research ethics thing. I will follow the approach you guys recommend.


texts: How to properly apply CrossValidation and/or split the dataset?

texts:
I have a particular problem and do not really now how to properly validate my experiments in this scenario.

There is one big data set with 100.000 samples, 99.000 y=0, 1.000 y=1
Each sample has 1.000 Features
There are 10 different subsets of feature combinations which have to be evaluated to get information about more or less expressive feature groups
Because of the different number and type of features each subset, an adequate models architecture has to be figured out manually
The performance of a few different kinds of models have to be evaluated
The performance of several different sampling techniques have to be evaluated
10-fold Cross Validation has to be applied
Random Search has to be applied for each experimental configuration (Combination of subset + model + sampling technique)

These facts lead to the following experimental routine:

train, val = splitDataset()
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, val)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

So, now I'm not sure about whether the CrossValidation is applied correctly. Because the model which has to be validated has already seen the train data (during manuallySearchGoodArchitecture and HPOptimization) and those are also used for testing purpose during CV.

So, is this routine faulty?
What would be the correct way of performing those experiments while applying proper validation technique?
How must the data set be splitted?


texts:
t=tensor([0.1 0.2 0.3 0.4 0.5 0.6])

now i need to modify this existing tensor t as follows:
t=tensor([0.1 0.2 0.7 0.4 0.8 0.6])

I tried as follows:
t=tensor([0.1 0.2 0.3 0.4 0.5 0.6])
a=tensor([0.1 0.2 0.7 0.4 0.8 0.6])
index=range(len(a))
t.index_copy_(0,index,a)

But still it is not updating
how can i modify the tensor in pytorch?


texts: How do you choose the min and max for the min-max normalization on a histogram classifier?

texts:
Please let me know what to do when there is a value in the testing set is bigger than the max value used to min-max normalize the training set building a histogram classifier.

Do I go back and change the bounds of the min-max normalization for the training set? Wouldn't that violate the notion that your training set should generalize to any testing set on its own and that you should retroactively change the what was done during the classifier building on the training set based on future testing sets that you are not supposed to know?

Do I change the bounds of the min-max normalization to the the min and max of the testing set? But, you are supposed to use the same transformation on the testing set as the training set, right?

Or, do I just let there be a bin on either side of the normalized histogram that such that everything that gets (not actually) normalized above (below) the interval [0,1] goes into the bin for all values (below) the interval?

Or, do I just exclude values that get transformed outside of the histogram's interval?

None of these seem right.  Please let me know if I am missing an option.


texts: Development on Sagemaker takes a long time

texts:
I have created a notebook in sagemaker and a python script called my_script.py. I then train a model using the following command:
model = SKLearn(
                entry_point=my_script.py,
                role=role,
                output_path= output_path,
                instance_type="ml.m5.large",
                sagemaker_session=sess,
                framework_version='0.23-1',
                serializer=sagemaker.serializers.JSONSerializer(),
                deserializer=sagemaker.deserializers.JSONDeserializer()
                )

and afterwards deploy it using:
    model_deployed = model.deploy(initial_instance_count=1,
                                  instance_type='ml.t2.medium',
                                  endpoint_name='simpleModel',
                                  serializer=sagemaker.serializers.JSONSerializer(),
                                  deserializer=sagemaker.deserializers.JSONDeserializer()
                                  )

If there are problems in the script my_script.py, I only discover this at the end after about 10 minutes. Does anyone know a more efficient way to work here to create a python script that is bug free?


texts: Feature agglomeration: Is it testing interactions?

texts: I have been looking at feature agglomeration in Python's scikit-learn. According to the user guide, feature agglomeration "applies Hierarchical clustering to group together features that behave similarly". Does this mean it is testing for interactions between features and groups features that do interact together? What does "behave similarly" mean in this context?


texts: Transposed Convolution without using Python built-in functions

texts:
Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions:

cannot use any Python built-in functions
cannot explicitly transpose the matrix
cannot explicitly perform matrix-vector multiplications

I have already written a forward 2D convolution method, and would like to reuse it to perform the backward process. I am not sure how this can be done with the given restrictions.

Zero-padding the filters did not seem to work...

I don't have a specific background in data science so any help is appreciated, thanks!


texts: Am I using GridSearch correctly or do I need to use all data for cross validation?

texts:
I'm working with a dataset that has 400 observations, 34 features and quite a few outliers, some of them extreme. Given the nature of my data, these need to be in the model.

I started by doing a 75-25 split on my data and leaving those 25% aside.

With the train set, I used GridSearchCV with a RepeatedKFold of 10 folds and 7 repeats and this returned my best_estimator results, which when we go in .cv_results_ we see it's the mean_test_score metric. I then called this my "Cross Validation score". Then, with this model fit, I ran it on the test set as grid.score(X_test, y_test) and called this my Test score.

def rf(df, score):

    X_train, X_test, y_train, y_test = train_test(df)

    params = {'n_estimators': [400, 700, 1000],
              'max_features': ['sqrt', 'auto'],
              'min_samples_split': [2, 3],
              'min_samples_leaf': [1, 2, 3],
              'max_depth': [50, 100, None],
              'bootstrap': [True, False]
}

    scorers = {'RMSE': make_scorer(rmse, greater_is_better=False),
               'MAE': make_scorer(mean_absolute_error, greater_is_better=False),
               'R2': make_scorer(r2_score)}

    cv = RepeatedKFold(n_splits=10, n_repeats=7)

    grid = GridSearchCV(estimator=RandomForestRegressor(random_state=random.seed(42)),
                              param_grid=params,
                              verbose=1,
                              cv=cv,
                              n_jobs =-1,
                              scoring=scorers,
                              refit = score)

    grid = grid.fit(X_train, y_train)

    print('Parameters used:', grid.best_params_)

    if score  == 'RMSE':
        print('RMSE score on train:', round(-1*grid.best_score_,4))
        print('RMSE score on test: ', round(-1*grid.score(X_test, y_test),4))

    elif score == 'R2':
        print('R Squared score on train:', round(grid.best_score_,4))
        print('R Squared score on test: ', round(grid.score(X_test, y_test),4))

    elif score == 'MAE':
        print('MAE score on train:', round(-1*grid.best_score_,4))
        print('MAE score on test: ', round(-1*grid.score(X_test, y_test),4))

When I set my metric to RMSE (the most important one), this is what it outputs:

RMSE score on train: 8.489
RMSE score on test:  5.7952

Have I done this correctly? Can I consider this discrepancy acceptable? With Random Forest for example, if I  deliberately ignore the gridsearch parameters and set my min_leaf_node to something like 10, my RMSE goes all the way up to 12 but it becomes very similar between the CV score and my test data. I'm experiencing similar results with SVR and MLP algorithms.

This is part of my thesis and now I have my supervisor telling me I should be using all my data for cross-validation which I don't think is correct.

My conclusion is that given the outliers in the model, without more observations, a discrepancy in results is to be expected, however I don't know if this conclusion is right or if I'm doing something wrong here.

Running my model in a somewhat similar dataset with fewer outliers gives results closer to one another.

RMSE score on train: 5.9731
RMSE score on test:  6.9164


texts: Bad results for training set and good results for test set

texts:
When training an algorithm, I have ROC = 0.5896, Sensitivity = 0.3333, and Specificity = 0.8375. When considering the test set, Sensitivity = 1 and Specificity = 1. This could happen or is a problem?
Another question:
When comparing several learning algorithms, I choose the best based on the results of the test set?
Any help will be appreciated.


texts: Training an LSTM with different time steps and number of features

texts:
I want to use an LSTM using Keras to make course grade predictions. My dataset includes student transcripts, which consist of courses taken and their respective grades of students. For each course, I plan to train an LSTM model that will take as input the student's previous courses and grades and output a predicted grade (from F to A). Each LSTM will be trained on all the student transcripts. The time steps will be semesters and the features of each time step will be the courses (and their grades). However, since not every student has spent the same number of semesters in school (i.e., time steps vary) and students take different courses in each time step (i.e., features vary), I do not know how to use a traditional LSTM.

I did some research and I understand that two ways of dealing with this problem are training one batch at a time and padding.

If I train with one batch at a time, how exactly does Keras do it? More specifically, how will the weight matrix be shaped and what dimensions will it have. Furthermore, when I want to use the model to make predictions, what should the dimensions of the input matrix on which the prediction is to be made be? Lastly, could someone please help me with the code for the one batch training part? Do I have to do a loop over my dataset matrix and train on each student transcript one at a time?

If I use padding, should I make the number of time steps and number of features the max of my dataset? This seems problematic as there are 1000+ different courses and so I would have to use a feature vector of 1000 and assign 0 to courses not taken, hence it seems that the model will take a lot of space and time to train/predict.

Thanks in advance.


texts: I have a linear regression model with 20+ variables and am looking for the best way to present my results in R Markdown HTML. Are there any R packages that make significant coefficients stand out and show key output results? I tried the Stargazer package, but it did not seem to work with the HTML output.


texts: correlation between categorical(ordinal) and discrete(continuous) value

texts:
I am doing my bi variate analysis  but right now looking to see the correlation between  my atributes

some are categorical 5 levels and others amount of money.

how can I see the correlation between them ?

do I have to create class for my money amount?


texts:
I'm struggling to get an LDA in Orange 3.28 (GUI).
Using the good'ol iris dataset, loaded from the dataset widget:

The data, as you can see, is loaded as it should and has "iris" set as target (is it the same thing as class by the way, I could not find the info ?)

The linear projection widget works for PCA and circular :

... but not for LDA:

instead, I get an error to the effect that "y should be a 1d array, got an array of shape (150,0) instead".
What do I do wrong, and how can I get a LDA ?
Bonus question: help says that "This widget displays linear projections of class-labeled data". However I could not find references to "class labeled data" in the manual. The closest I found is the "create class" widget, that creates apparently a new categorical variable based on text. Is this the same as a "target" variable ? The data table widgets refers both to "target with 3 values" (in this case) and allows to "color by instance classes" - is a class same as a target variable, then? If not how is a class defined?


texts: How to use hashing trick with field-aware factorization machines

texts:
Field-aware factorization machines (FFM) have proved to be useful in click-through rate prediction tasks. One of their strengths comes from the hashing trick (feature hashing).

When one uses hashing trick from sci-kit-learn, one ends up with a sparse matrix.

How can then one work with such a sparse matrix to still implement field-aware factorization machines? SKLearn does not have an implementation of FFM.

EDIT 1: I want to perform feature-hashing/hashing-trick for sure in order to be able to scale FFM to millions of features.

EDIT 2: Pandas is not able to scale to many fields. I also want to convert an arbitrary CSV (containing numerical and categorical features) into LIBFFM (field:index:value) format and perform hashing trick at the same time (preferably without using Pandas). Pandas2FFM does not scale even after performing the Hashing Trick.


texts: Memory error when using more layers in CNN model

texts:
On my dell core i7 - 16GB RAM - 4gb 960m GPU laptop, I am working on a project to the classify lung CT images using 3d CNN. I'm using the CPU version of tensorflow. The images are prepared as numpy array size (25,50,50).

My CNN model had 2 conv layers, two maxpool layer, one FC layer and output layer. With this architecture I could train the model with approximately (5000 to 6000) samples. After adding more layers my model now has 6 conv layers, 3 max-pool layers, FC and output layer. My problem is after changing the architecture with just more than 1000 samples my memory gets filled and I get memory error. I tried to make smaller batches, but every time getting same error. I have two questions:

Why by adding more layers the model needs more memory?
Is there any way to deal with this type of problem?


texts:
I have a set of game play event data. One of our goals is to provide a picture of how much time the person has spent playing. This sounds simple but for the fact that events are often dropped and result in vast amounts of time between events.

We are in field trials right now and don't have a huge pool of data to use as reference.

I don't have much experience in statics, though I'm a competent programmer. My question is: What might I do to account for the missing data? How do I ensure that these missing events don't skew my data?


texts: What is the output polytree after aplying the Ramex algorithm to this graph?

texts:
I've been trying to understand the way this algorithm works, but I can't get a consistent result.

It has two phases: the first one coverts a table of events into a graph, and the second where the graph is tranformed into a polytree.

The question is about the second phase, using a back-and-forward heuristic:

This is the heuristic algorithm:

Algorithm 3. Back-and-Forward Heuristic
Input: Network G;
Output: Poly-tree S;
Initialize S;
For each vertex in G
  For each edge in G
    x = arg_max(weighted forward-vertex not visited in G and connected with S; weighted back-vertex not visited in G and connected with S)
  End-for;
  Update solution S with x;
End-for;

The example shown in the paper is this:

Fig. 2. (a) Original cyclic network, (b) Forward Heuristic provides a tree solution, (c) Back-and-forward Heuristic provides a poly-tree solution

Right now I'm having trouble applying the back-and-forward algorithm to the vertex b. I'll describe my thought process:

Initialize S -> Take 'a' and put it in the tree (S).
Look for the edges that connect 'a' (the tree S) to any unexplored node and select the one with higher weight -> 'c'.
Add 'c' to the tree. a--674-->c
'a' is now explored/visited.

Move to next vertex - 'b'.
Look for the edges that connect the tree ('a' or 'c') to any unexplored node and select the one with higher weight -> 'd'.
Add 'd' to the tree. a--674-->c--684-->d
'b' is now explored/visited.

Move to next vertex - 'c'.
Look for the edges that connect the tree ('a', 'c' or 'd') to any unexplored node and select the one with higher weight -> 'e'.
Add 'e' to the tree. a--674-->c--684-->d--1080-->e
'c' is now explored/visited.

Move to next vertex - 'd'.
Look for the edges that connect the tree ('a', 'c', 'd' or 'e') to any unexplored node and select the one with higher weight -> 'h'.
Add 'h' to the tree. a--674-->c--684-->d--1080-->e--930-->h
'd' is now explored/visited.

And now 'b' is left hanging because 'b' and 'd' are visited and the algorithm says the connection must be to a node not visited in G and the polytree shows b-->d.

Is my interpretation wrong?


texts: Reformulating the maximal margin classifier optimization problem

texts:
Ok, so I've been trying to read up on how SVM's work and started with maximal margin classifiers. At page $132$ in ESL (Elements of Statistical Learning) the authors "reformulates" the optimization problem but I can't seem to understand what they are doing from $(4.47)$ to $(4.48)$. Does anyone know?

Here is an excerpt:

Edit: I guess, what I don't understand is why we can arbitrarly set the magnitude of beta to $\frac1M$. What does a positively scaled multiple mean in this case? Just a multiple larger than $0$?


texts:
Source: https://d396qusza40orc.cloudfront.net/exdata%2Fdata%2FNEI_data.zip

Here is my data prep.

NEI <- readRDS("summarySCC_PM25.rds")
Baltimore <- NEI[NEI$fips=="24510", ]
Baltimore$type <- as.factor(Baltimore$type)
Total_Emmisssions <- aggregate(Baltimore$Emissions,
                               by=list(Baltimore$year, Baltimore$type),
                               FUN=sum)
names(Total_Emmisssions) <- c("Year","Type","Emissions")

Plot code

library(ggplot2)
g <- ggplot(Total_Emmisssions,aes(x=Year, y=Emissions, colour=Type))
g1 <- g+geom_point()
g2 <- g1+facet_grid(. ~ Type)
g3 <- g2+geom_smooth(method="lm", se=FALSE)

However, in this plot, the year is always 2002-2008. How do I change the scale / ticks to show from 1999-2008 at an interval of 3 years. Using scale_x_discrete does not work and it messes the graph. Please help.

str(Total_Emmisssions)
'data.frame':   16 obs. of  3 variables:
 $ Year     : int  1999 2002 2005 2008 1999 2002 2005 2008 1999 2002 ...
 $ Type     : Factor w/ 4 levels "NON-ROAD","NONPOINT",..: 1 1 1 1 2 2 2 2 3 3 ...
 $ Emissions: num  522.9 240.8 248.9 55.8 2107.6 ...


texts: How Can I Compute Information-Gain for Continuous- Valued Attributes

texts:
Decision trees handle only discrete values, but the continuous values we need to transform to discrete.  My question is HOW?

I know the steps which are:

Sort the value A in increasing order.
Find the midpoint between the values of $a_i$ and $a_{i+1}$.
Find entropy for each value.

I have this example can someone explain how we calculated the entropy values?


texts: How to avoid confusion between Percentage and Percentage Points?

texts:
Whenever I have to communicate a change in a numerical metric, I found myself struggling if I should use Percentage or Percentage Points. Especially, when the metric is a percentage itself, it can be really tricky.
For example, if we track the Retention Rate of users in a platform and the percentage increased from 10% to 30%, we can either say that we had an increase of 20p.p. or we had an increase of 200%.
Another example, if we classify the risk of a business decision to 5% of revenue drop and an alternative choice has 7% revenue drop. We can say 2p.p increase or 40% increase.
Things to consider

Not every team member has a math/statistical background to understand the difference of the two options without giving an explanation. The difference is not common in the "average-person".
Not everyone has English as his/her first language and the right choice of words might not have a big effect. For example, using "increase by .." or "increase to.." might not have any difference at all.
The use of p.p. or % can be either verbal in person or through an online, video call, an official report or a short message in a chat app (slack, Skype etc)

Why it matters?

The number per se, might have a different impact to others. For example, an increase of 20p.p could "feel" low, while the increase of 200% "feels" enormous.
Using % only ends up to misunderstandings. For example, I found myself in a situation where a team member understood an increase of 50% to a 10% retention rate was ended up to 60% retention rate.

What to do?
How to choose the way I communicate such an increase in the following cases

Verbal in person or video call
An official report or documentation
A short, informal message like in Slack or Skype


texts:
I'm having my first steps in the NLP and at the moment I'm looking forward to building my own documents search engine. I've already got to know with TFIDF in practical way and I've also read about word2vec algorithm. My goal would be to create such document search engine that would catch the meaning of the words in query. I expect my search engine to be used with typing few words into a textbox, maybe a sentence, no more.
I'm aware that TFIDF would not be the best approach to do this as I would really like to catch meaning of words I type into the textbox. I've seen that many people use word2vec to average document vectors and then compare their similarity with words in query. Is it a good idea to take this approach? I've also seen such algorithm like doc2vec, that can represent whole documents as vectors. How would I compare query with such document vectors? Is it even a good idea to take into consideration doc2vec in search engines?
To be honest I have no idea which approach I should take. It would be great if someone would direct me onto a right path. Any ideas or useful links are really welcomed!


texts: How to sample a statistically uniform dataset

texts:
I have a data set of with a distribution that looks something like this:

I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in the green area):

I know how to do this by taking the data, putting into separate "buckets"
 (distribute the data into X buckets, taking up to Y samples from each bucket), but I was wondering if there is there an easier way.

P.S. the result doesn't have to be 100% accurate - a good approximation is enough.


texts: Is there any reason why providing symbolic features into an MLP wouldn't outperform feeding raw pixels to a CNN in a RL task?

texts:
I am tackling a RL problem (relaxed version of Space Fortress) with DQN. The usual approach would be to feed pixels into a CNN but that is usually very slow. I am considering feeding symbolic features such as coordinates, velocity and angles of the relevant elements into a Multi Layer Perceptron because then the training would be faster and I could try more things quicker. My idea is that the perceptual part of the task is not that interesting to learn here and I would like to focus on the RL part instead (I am experimenting with Hierarchical RL).

Given that with all the symbolic features that I would provide it would possible to reconstruct the full image and hence the state of the environment, would there be any reason that a CNN pixel based approach would be better?

PT: The environment has a fixed number of elements

Thanks


texts: Model selection and assessment using leave-one-out cross validation

texts:
Assume $D$ is the training data set with both the value of the predictors $\mathbf{X}$ and the value of the response variable $Y$. I have a loss function $L$ and two models $f(\mathbf{X};\beta)$ and $g(\mathbf{X};\lambda)$, where $\beta$ and $\lambda$ are model parameters. Our goal is to estimate

\begin{equation}
e(f)=\mathbb{E}[L(Y,f(\mathbf{X};\beta))|D]~\mbox{and } e(g)=\mathbb{E}[L(Y,g(\mathbf{X};\lambda))|D]
\end{equation}
Note that it is the expectation of the generalization error of the model $f$ (and $g$) that is trained on the specific training data set $D$, where the expectation is taken based on the same distribution that generates $D$.

Now, if we do a leave-one-out procedure, specifically: let $N$ be the total number of observations in $D$, let $D_{-j}$ be the data set that removes the $j^{th}$ observation. Then,$L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an "almost" unbiased estimator of $e(f)$ right? Theoretically, to get $e(f)$, once should generate infinite new $(Y_i,\mathbf{X}^{(i)})$ from the distribution of $D$, then train the model on $D$ and use that model to make prediction on the new infinite data set and take the average. Now we fit the model on $D_{-j}$, which is only slightly different from $D$. So $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$ should be an "almost" unbiased estimator of $e(f)$. Then you go through all $N$ data points in $D$ to obtain the value of $L(Y_j,f(\mathbf{X};\beta))|D_{-j}$, where $j=1,2,...,N$ (assume $N$ is large), then you take the average, then the result should be very close to $e(f)$ right? Then we do the same thing to model $g$. Then in this case, we get very good estimates of $e(f)$ and $e(g)$ so we can do model selection based on $e(f)$ and $e(g)$. Specifically, if $e(f)<e(g)$, then I should expect that on a LARGE new independent data set $T$, $f$ should perform better than $g$ correct? Also, the quantity $e(f)$ and $e(g)$ computed from $T$ should be very close to the quantity computed from $D$, assuming both $D$ and $T$ are large. Is that correct?

If all the above are correct, then it seems that I did model selection and model assessment in one step. But should I partition the data set into 3 pieces, i.e., first train 2 models on one piece, then apply the 2 models on another piece to do model selection, then apply the 2 models on the 3rd piece to do model assessment?


texts: Difference between Dueling DQN and Double DQN?

texts: I have read some articles, but still can not figure out the difference between the Dueling DQN and Double DQN? What exactly is the difference between them? Also, Does Dueling DQN need to be built on top of a Double DQN? Thanks!


texts: Resampling a normally distributed dataset for regression problems?

texts: I have a dataset from an operating process having 5 measurements and 1 outcome. All values are normally distributed. When I train a regression model on the dataset it performs good on the majority of the dataset - the default operating condition of the process. It performs much worse though on other than default operating conditions, values distant from the mean. If it were a classification problem I would treat this as class imbalance and perform some resampling technique to get balanced classes. How do I treat this for a regression problem?


texts: One-Hot Encoded Matrix Inupt/Ouput for Autoencoder

texts:
I am trying to write an autoencoder to reduce the dimensionality of my genomic data. Currently, my data is in the form of a $273278 \times 1$ vector. Each element of the vector indicates whether a position has no mutations (0), one mutation (1), or two mutations (2). As such, the input and output of my autoencoder looks like this:
$$\begin{bmatrix}
0 \\
1 \\
0 \\
2 \\
\vdots
\end{bmatrix}$$
This uses label encoding to represent the categorical data. This works, but the autoencoder isn't very accurate since the 0, 1, and 2 data are not related to each other.
I am considering using one-hot encoding to create a $273278 \times 3$ matrix where each column corresponds to 0, 1, or 2. As such, the above vector would turn into this:
$$\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0\\
1 & 0 & 0 \\
0 & 0 & 1 \\
\vdots & \vdots & \vdots
\end{bmatrix}$$
However, I am unsure of how to input this matrix into a (keras) neural network. Is there a function to do this? Would flattening this matrix be mathematically appropriate? Is there another method to do this?


texts: What is the best machine learning approach for data error correction?

texts:
I have the following problem: I have a set of raw data entered manually. These data are evaluated by business rules and eventually generate integrity errors, mostly.

Then these errors are manually handled and the correction is performed manually.

What is the best approach in machine learning to implement an algorithm that look at this historical corrections and suggest the correct approach.

Note: all data are structured and business rules are evaluated mostly at foreign keys restrictions.


texts: How do I fit a curve into non linear data?

texts:
I did an experiment in my Uni and I collected data $(ω,υ(ω))$ modeled by the equation:

$$ v(ω)=\frac{C}{\sqrt{(ω^2-ω_0^2 )^2 +γ^2 ω^2}} $$

where $ω_0$ is known. Do you know how can I fit a curve to my data $(ω,υ(ω))$ ? and how to find the parameter $ γ $ through this process ?


texts: About Gaussian kernel in other distances

texts:
I have a question about Gaussian kernel. I read the following site.
Why do we use a Gaussian kernel as a similarity metric?
My question is whether we can extend this to other distances or dissimilarity measures (e.g., Bray-Curtis).
When we would like to use spectral clustering, we need to use an affinity matrix. Is Gaussian kernel specific to Euclidian distance?
It seems that this package supports Bray-Curtis dissimilarity. However, BC dissimilarity is different from distance.
https://www.rdocumentation.org/packages/clusterSim/versions/0.47-1/topics/speccl


texts: How to mitigate the hierarchical error propagation in tree-structured classification

texts:
Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$

We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure below.

Generally, in this kind of hierarchical classification, if a sample is wrongly classified in an upper layer, it is less likely to be corrected again in lower layers.

My question: Is there any strategy to mitigate such hierarchical error propagation in tree-structured classification?

Much appreciated!


texts: How do I evaluate a K-Means unsupervised anomaly detection approach?

texts: how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit learn K-means were used too.


texts: multi class classification : unbalanced data - good testing results poor prediction results

texts:
I have unbalanced dataset with 11 classes where 1 one class is 30% and rest are between 5-12%. I am not a hardcore programmer so I am using the product from https://www.h2o.ai/.
I used GBM and DRF and used the option to balance the classes and the training results are great (98-99% precision and recall) as per the confusion matrix however when I test it on the validation set the only class where I get decent accuracy is the class which is 30% all the others have classification errors close to 100%.
Not sure what approach I am supposed to take.
The 11 classes are 11 market segments and even an accuracy of ~70% for each of the classes is doable for my purposes.

Edit 1: Additional Info: validation the model predicts almost each sample as the 30% class i.e why close to 30% accuracy...similar to a credit fraud detection gone wrong...

Update 1: So I tried 2 more approaches

1) Tried to make it into a 2 class classification by labelling everything other than the 30 % class as  "OTHER" and the results were still poor

2) I removed the 30% class and kept the other 9 as is and then trained a GBM and the results are scary accurate with 85%-15% split between test and validation. However as soon as I do a cross validation the classification is really poor.

Not sure what's going on...maybe I need to rethink my entire approach and redefine the problem and come up with an entirely different hypothesis to begin with.


texts: GloVe word embeddings containing sentiment?

texts:
I've been researching sentiment analysis with word embeddings. I read papers that state that word embeddings ignore sentiment information of the words in the text. One paper states that among the top 10 words that are semantically similar, around 30 percent of words have opposite polarity e.g. happy - sad.
So, I computed word embeddings on my dataset (Amazon reviews) with the GloVe algorithm in R. Then, I looked at the most similar words with cosine similarity and I found that actually every word is sentimentally similar. (E.g. beautiful - lovely - gorgeous - pretty - nice - love). Therefore, I was wondering how this is possible since I expected the opposite from reading several papers. What could be the reason for my findings?
Two of the many papers I read:

Yu, L. C., Wang, J., Lai, K. R. & Zhang, X. (2017). Refining Word Embeddings Using Intensity Scores for Sentiment Analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(3), 671-681.
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T. & Qin, B. (2014). Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 1: Long Papers, 1555-1565.


texts: Which applications can not be handled by very Deep CNN models?

texts:
I wanted to know what challenges very deep models can face even if the accuracy is good. Would they be not suitable for any application given that my model is very very deep?
I wanted to know if models are very deep, will there be any significant delay in obtaining prediction which will make it not suitable for any task?


texts: Getting negative r2_score with new set of dimensions

texts:
I am trying to predict flight take off delay using my current dataset. At this point of time, I only have four dimensions.

scheduled_time_departure (A), flight_id, day_of_week actual_time_departure(B), take_off_delay(B-A)

when I try to predict actual_time_departure(B) using scikit linear regression model using (scheduled_time_departure (A), flight_id, day_of_week) on x-axis, I am getting good r2_score
However when I am trying to predict take_off_delay (which is actually difference between actual_time_departure and scheduled_time_departure), in that case I am getting negative r2_score.

Note:

to convert string to integer I am using LabelEncoder.
scheduled_time_departure and actual_time_departure are in seconds not a timestamp, that is second of day, 86400 is max value it can have.
I even tried doing normalization of scheduled_time_departure
I have ensured that take_off_delay is always positive.
For the case when I was predicting actual_time_departure, I tried using one hot encoder but that agravated r2_score

P.S: I am new to machine learning and data science, let me know if I am making dumb mistake :)
P.P.S: I understand model can be worst if r2 score is negative, however I want to understand the reason.


texts: How to increase accuracy of model from tensorflow model zoo?

texts:
Situation:

My dataset is 70k images of people wearing clothes. Images are labeled: bbox position and class. There are 10 classes. I did 80:20 split. Categories are balanced with exception of one category, but I can accept poor performance on one category.

Goal is cloth recognition in images. When I feed image of person wearing pants and tshirt, I want to see two bboxes of these clothes.

My problems:

I already trained few models from tf model zoo. I did over 100k steps on ssd mobilenet v1 and faster rcnn resnet 101.

Problem with ssd is that it won't converge. Loss is not getting below stable 2 and accuracy is bad. Problem with faster rcnn is that loss is below 1 but it's varying a lot and sometimes it jumps over 1.

What I've done:

I tried different batch sizes for ssd with no luck. FRCNN is locked with batch size 1. I improved dataset multiple times. I went from 50 unbalanced classes to 10 balanced classes. I didn't tweak hyperparameters from models configs besides batch size.

My access to strong gpu is limited for me so I can't just randomly try different hyperparams combinations with hope that it will work. Could you suggest me few things that I can do in order to improve my models? I would be very thankful.


texts: How to customize what happen in fit method in tensorflow 2.0

texts:
I'm studing how to customize what happen in fit method with tensorflow 2.0 and I'm following this link: https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit, but I noticed some difference in code at previous link; in particular it shown first this way to train model:
with tf.GradientTape() as tape:
           y_pred = self(x, training=True)  # Forward pass
           # Compute the loss value
           # (the loss function is configured in `compile()`)
           loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)
           print(loss)

and after this one:
with tf.GradientTape() as tape:
            y_pred = self(x, training=True)  # Forward pass
            # Compute our own loss
            loss = keras.losses.mean_squared_error(y, y_pred)
            print(loss.shape)

As you can see the difference is in the computation of loss function: in the first block they use a compiled loss function, whereas in the second block loss is computed with mean_squared_error function.
My doubts arise when I check the shape of the loss: in first case it is a scalar like this: (tf.Tensor(0.21193008, shape=(), dtype=float32) while in the second the print statement give me a tensor of shape (32,).
My idea it is that in second case the mean over batch isn't computed and I don't know why.
I solved using the tf.reduce_mean but I'm not sure if it is correct. However the loss should be a scalar and therefore I don't understand how the second mode works without tf.reduce_mean and if the usage of this statement is correct.


texts: What is the simplest ReLU network that can learn an upside-down v pattern?

texts:
Input: $x \in \mathbb{R}^+$

Expected output:

$f(x) = \begin{cases}x &\text{for } x \leq 5\\10 -
 x &\text{for } x > 5\end{cases}$

So basically you have a neural network with one input neuron and one output neuron. Let's say the output has the identity as an activation.

What is the simplest network with ReLU activations that fits this function?


texts: Which feature selection technique to pickup(Boruta vs RFE vs step selection)

texts:
I have data with 103 columns. I would like to understand which algorithm is best for feature selection and what may be the logic to call any feature as best.

I run below feature selection algorithms and below is the output:

    1) Boruta(given 11 variables as important)
    2) RFE(given 7 variables as important)
    3) Backward Step Selection(5 variables)
    4) Both Step Selection(5 variables)

I not able to decide which one to pick up; with domain knowledge it appears I must pick up results from Boruta (as it is giving most number of variables and all seems important).

However I don't find any concrete reason to pickup the best combination.


texts: What should be the architecture design for the dataset to be used for a machine learning API

texts:
I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will use the website that I build and the API calls will be made which will run some machine learning algorithms on the dataset. Now I don't have much knowledge regarding designing such systems. One possible solution that I have in mind is.
Migrate the csv data to a proper database and then create an ETL system to access the data and proceed with the computations. Since I don't have much knowledge in this regard can anyone provide help on how I could create an architecture for the task at hand.


texts: Target Variable Encoding for Time Series Change point detection

texts:
I am working on a time series data for which I intend to impliment machine learning model for detecting change point in time series data.

This data is recorded fom machinary and we have to predict process is completed. Now we know when process is completed(actual time at which process is getting completed) and we have to link it to the data that we have recorded from sensors. Our goal is to build model trained on this currently available historical data which will be able to predict the time at which process is getting completed for the future data.

Now my question is I know that process gets completed at 11:50, so I have created target variable which is encoded as 0 from the beginning and then 1 11:50 onwards. This data I fed to Neural networks to predict the variable. Is this a correct approach? Is there any sophisticate method for encoding the target variable than just labelling it as 1 after certain timestamp as it is time series data and my goal is to predict time at which process will get completed for the next sesnor data?


texts: Having a reward structure which gives high positive rewards compared to the negative rewards

texts:
I am training an RL agent using PPO algorithm for a control problem. The objective of the agent is to maintain temperature in a room. It is an episodic task with episode length of 9 hrs and step size(action being taken) for every 15 mins.During the training of an agent, from a given state, agent takes an action.Then I check the temperature of the room after 15 mins(step size) and if this temperature is within limits, I give the action a very high positive reward, and if the temperature is not in he limits, I give a negative reward.Episode ends after 36 actions(9hrs * 4 actions/hour)(step size is 15 mins)

My formulation of reward structure.

zone_temperature = output[4]  # temperature of the zone 15 mins after the action is taken

thermal_coefficient = -10

if zone_temperature < self.temp_limit_min:
    temp_penalty = self.temp_limit_min - zone_temperature
elif zone_temperature > self.temp_limit_max:
    temp_penalty = zone_temperature - self.temp_limit_max
else :
    temp_penalty = -100

        reward = thermal_coefficient * temp_penalty

value of zone_temperature varies from limits in a range of 0 to 5 deg.  So the reward for when the actions are bad(temperature not in limits) varies from 0 to -50 , but when the actions are good(temperature is in limits) reward is +1000 . I had such a formulation so that the agent can easily understand which is a good action and which one is bad. Is my understanding correct and is it recommended to have such a reward structure for my use case ?

Thanks


texts: Feeding large dataset to neural network using tf

texts: I have a csv file of size ~11GB contains 2 million rows located in my local disk as my training data. What's the best way to feed the data to a neural network model? I use tf.keras to build the neural network and my RAM doesn't allow me to import as a single np array. Thank you.


texts: Stacking/Concatenating/Combining two vector space models

texts:
I have two vector-space models,  with different dimensions.

The number of vectors in one model is the same as the number of vectors in the other. I.E: if I have vector representation for a car in one model,  I have vector representation for a car in the other model, but the number of dimensions can be different.

I want to combine these models (and then cluster using the combined model),  I cannot average (BoW) or add these models together as stated earlier they have different dimensions.

I was going to simply concatenate the vectors. Is this valid or  is there better way to do this?

Also before you concatenate should you normalize?


texts: Prioritized Experience Replay - why to approximate the Density Function?

texts:
I am reading about Prioritized Experience Replay, and can't understand the following:

On page 4, every transition can be selected from the table with its own probability.
Here is the cumulative density function (if I understood correctly):

$$P(i) = \frac{p_i}{\sum_k{p_k}} $$

where:

$$p_i = \frac{1}{index In Table}$$

Aterwards, the paper says:

  For the rank-based variant, we can approximate the cumulative density function with a piecewise
  linear function with k segments of equal probability. The segment boundaries can be precomputed
  (they change only when N or α change). At runtime, we sample a segment, and then sample uniformly
  among the transitions within it.

My question is, why do we have to approximate the density if it can be achieved with the following:

roll a dice between 1 and N (with a dice that is exponentially more likely to roll a '1' rather than a '2', etc)
select an item from index according to the dice.

In c++ we have std::exponential_distribution [source] so there is no need to approximate anything. ...If we maintain our table sorted in a descending order.


texts: How label smoothing and label flipping increases the performance of a machine learning model

texts:
I have seen posts and research papers mentioning these techniques for improving the performance of a machine learning model.

These techniques certainly make some sense in the case when we are not sure about how correct is the labelling our dataset.

However, I am wondering if these two techniques are proven to be beneficial for a ML model in the case when the labelling is correct?


texts: From Patch-based Classifier to Full Image classifier

texts:
I was wondering if it is feasible to train patch-based image classifier, due to small amount of data, and then use it in order to initialize training for full image classification, but this time on different set of classes?

Specifically, I am having trouble understanding how a network trained on 227x227 image patches can be used to initialize training a network for lets say 2000x2000 images. Is there anyone who can help me ether understand this problem or point me to the right direction? Thanks!


texts: Neural Network for predict a set in a certain time stamps

texts:
I have a market transactions dataset including time stamps and goods as follow.

John always buy milk and bread in Super Market. Besides that, he also buys some goods like the following:

On Monday, John bought milk, bread {beer, chocolate}.
On Tuesday, John bought milk, bread {potato}.
On Wednesday, John bought milk, bread {chocolate, avocado, peanuts}.

Can we answer the question: "What will he buy on Thursdays?".

For example: He will buy {beer, avocado} besides milk and bread on Thursdays.

Which one should I use? Supervised or unsupervised.

Which model can I use to predict a set of goods in this case?


texts: How to represent relation between users as a feature?

texts:
I'm developing a model for unsupervised anomaly detection. I have a dataset representing communications between users (each example represents a communication): there are many features (time, duration, ...) and the ids of sender and receiver. My question is: how to represent the link between those two users?
I have several ideas, but each of them seems to have serious drawbacks:

Use id as is. Drawback: even if ids are integers, they have no numerical sense (id 15 is not 3 times id 5) and I think this may mislead the system
Use sort of vectors: for example, with 3 users: user1 = (0 0 1), user2 = (0 1 0), user3 = (1 0 0). Drawback : the number of users may vary over time, thus the number of features would vary as well and I would have to re-train my model.
Graph theory: I've heard of that way of representing data, which could fit perfectly my data model. Drawback: I've absolutely no knowledge in graph analysis
Assign each user a id which is a prime number. That way a communication could be represented in an unique way as the product of the 2 ids. Drawback: as for point 1, ids do not have a "numerical sense"

What do you think may be the better way to represent these relations?


texts: Optimization based on validation and not training

texts:
Hello neural network programmers,

I am currently creating a neural network with keras, as I am not that familiar with tensorflow and it's a bit more difficult.

I want my optimization to optimize the validation loss and not the training loss/accuracy.

Ideally, I would want it to look something like this:

  model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['val_accuray'])

but val_accuracy is not something I can eneter into metrics.

Any ideas on how I can implement that?

Thanks!


texts: Accuracy differs between MATLAB and scikit-learn for a decision tree

texts:
Is there any possibility to vary the accuracy of same data set in matlab and jupyter notebook by using python code ?

For same data set, at first I applied it in matlab and get 96% accuracy for decision tree method, then I apply that same data set in jupyter notebook by using python code where I get 53% accuracy for C4.5 (decision tree) by using k-fold cross validation.

I didn't understand where's the problem for getting different accuracy for same dataset and same method.

My procedure in python code is given below:

import pandas as pd
import numpy as np
from sklearn import tree
from sklearn.model_selection import KFold

train=pd.read_csv('E://New.csv')
train.head()

# define X and y
feature_cols = ['Past','Family_History','Current','current or previous
               workplace','diagnosed with a mental health condition by a
               medical professional?','do you feel that it interferes with
               your work when being treated effectively?','Gender']
X = train[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train['Diagonised condition']

kfold = KFold(n_splits=10,random_state=None)
model = tree.DecisionTreeClassifier(criterion='gini')

results = cross_val_score(model, X, y, cv=kfold,scoring = 'accuracy')
result = results.mean()*100

std = results.std()*100
print (result)


texts: Null values as useful information for feature engineering

texts:
I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database because this depends on the chosen sample. Normalization will be done in Python. For my use case, which is a binary classification problem (fraud detection), the presence of null values also correlates with the target variable. Therefore, I would like to preserve this information for the model.

My proposal would be to create an additional binary column called varName_isnull which encodes the presence of a Null in the varName column. In other aggregated features, this binary column would also be used for example to calculate the number of Null values for a particular grouping (i.e. credit card) per unit of time.

Is my proposal reasonable? Are there any alternative representations and if yes what would be their advantages?


texts: There is some way to see the out-of-bag error in Orange3? The widget Test and Score only reports mean squared error, root mean square, mean absolute error, coefficient of determination and root-mean-square deviation.


texts: Is machine learning successful in solving combinatorial optimisation problems in NP-hard? Discuss problem of scheduling using machine learning

texts:
Chess is Exp-complete. I assume Alpha GO is also Exp complete. Correct me if I am wrong. Now deep reinforcement learning has been successful in tackling this problem. What about other more practical problems like scheduling, bin-packing, and routing?
In particular, what is the state of the art for scheduling applications using ML techniques? How should one start in this direction?


texts: One Hot Label Encoding Scikit_learn convert back to Data Frame

texts:
I have a data frame with 4 features and 1 target. The 4 features are 3 categorical and 1 numerical.

I created X which is a new data frame for the 3 categorical features. I use one hot label encoding but now it is a numpy array. Why?

Should i convert it back to a data frame? why not?

what is the best practice to merge X with my 1 numerical feature now ?


texts: Confused about polynomial regression with multiple variables

texts:
I'm trying to create a multivariable polynomial regression model from scratch but I'm getting kind of confused by how to structure it.
So, I have an array of feature vectors such that each vector can be displayed like so:
[height, weight, age]

I know with multivariable linear regression I would create an algorithm like so:
y=B0+B1*x0+...Bn*xn
Where x0 would be the first element of each in the feature vector.
So for multiple variable polynomial regression would it go something like this:
y = B0+B1*x0+B2*x1**2+...Bn*Xn**d
Where d is the degree of the polynomial. Apologies if this is painstakingly obvious and formatted badly, I'm just a small bit lost.


texts: Hyperparameter tuning and cross validation

texts:
I have some confusion about proper usage of cross-validation to

tune hyperparameters and
evaluate estimator performance and generalizeability.

As I understand it, this would be the process you would follow:

Split your full dataset into a training and test set (Python's train_test_split)
Use cross-validation to build a model and tune hyperparameters on the training set (GridSearchCV)
Evaluate the best estimator and assess generalizeability using cross-validation on the test set (cross_val_score)

I've looked through sklearn's cross-validation documentation, and it recommends still having a test set for final evaluation.

A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

sklearn's grid-search information recommends:

When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process: it is recommended to split the data into a development set (to be fed to the GridSearchCV instance) and an evaluation set to compute performance metrics.
This can be done by using the train_test_split utility function.

My issue is that I often see conflicting work (for example, just cross_val_score on the entire dataset, only GridSearchCV on the entire dataset, or just a train_test_split variant), and I am hoping to understand what are best practices and clarify where I may be wrong.
Edit: This Stack Overflow answer seems to answer my question.


texts: Optimizing ad placement using historical data

texts:
Edit: increased generality.

I have an ad placement optimization problem and I am brainstorming to determine which ML techniques are well suited to it.

Basically, I have some objective that involves optimizing ad placements for a given customer based on historical data on impressions, interactions, revenue and viewership history.  There are also certain constraints on the ad placements due to market, content, and repetition etc...

This history goes back over two years and covers a very large customer base.  What I would like to do is use machine learning to optimize ad slot placement that maximizes impressions based on the aforementioned inputs.

This sort of problem can be solved using linear programming methods, but I could also see RNNs as a viable option.  Any insights here would be greatly appreciated.


texts: Bayesian Network Modeling on time series data with constant discrete features

texts:
I'm trying to model a dynamic bayesian network that can infer relationships between traffic sate of different road links over time.
The main theme of mystudy is to model the relationship of change in traffic state of different links over time.
The data:
-Different road links with traffic state (multivariate)in time-series format.
-Geographical features of each road links (e.g. number of lanes, road width)
Here comes the problem.
The geographical features of links are constant over time. (The number of lanes or road width does not change over time).
I want to input these geographical features of links to each corresponding links.
How would I properly reflect the effects of these geographical features in the model?
Should constant variables be removed in the modeling?


texts: Some languages have word endings with their nouns (like Finnish, e.g. "in Berlin" -> "Berliinissä"). I have tried to annotate the characters in the training data as entities, but then I run the model, it doesn't detect the characters inside the word. When those characters are a separate word, only then they're detected. I am unable to think of an implementation to effectively detect named entities within a word. Any suggestions would be helpful.


texts: How to perform Instance Segmentation using Tensorflow?

texts:
I used Tensorflow Object Detection API for a custom dataset based on the instructions at this help document.As required , collected the dataset,annotated it in PASCAL VOC XML format,split into training and test sets,generated tfrecords. Pets configuration for custom object detection and localization is used.

I just want to proceed further and get segmentation boundary rather than bounding box. The link is clear till this point on directory structure and mechanism to prepare input data.

Instance Segmentation help is unclear to me on two aspects:
  -Directory structure
  -Is it required to supply segment boundaries equal in number to that of images in training and test sets ?

Notice that tfrecords are created using this script. Can this be reused to include segmentation data.


texts: Combining neural networks with different variables

texts:
My organisation provides consultation to other firms, in part by making use of neural networks trained with extensive datasets that we have collected over the years.

Whenever available, we would inquire for any similar data our clients may have so that we can tailor a new model based on a checkpoint of our own. This has worked well as the data we used for training is relatively generalised and often collected by organisations in general.

However, recently one of our clients provided data that is highly predictive of the same response variable in our own models, but differs wildly in variable types and size.

An example in order to simplify:

I trained my own network to predict height based on weight using 100,000 records. Oftentimes my clients provide an additional 10,000 records of their employees height & weight which allows me to tailor the network to their needs. A new client, however, has provided me with 10,000 records of their employees' gender, country of origin, and daily dairy consumption. All of which, when trained in a separate network, prove highly predictive of height as well.

How do I effectively combine these separate models into a single classifier or regressor?


texts:
I'm new to anaconda.
How can I install the dlx package (https://pypi.org/project/dlx/) in anaconda?
I found it here: (https://anaconda.org/auto/dlx).
conda install auto::dlx
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - auto::dlx
  - secretstorage
Use "conda info <package>" to see the dependencies for each package.

I get no result from conda info auto::dlx.
Can anyone help me?
SOLUTION:
Thanks to vivekwisdom for leading me on the right path.
PyCharm created a virtual environment. I had to activate it first (source activate <name_of_venv>) and then install dlx using pip. It didn't work via conda install auto::dlx because it was missing cython, which I also installed via pip.


texts: Transfer Learning and Unknown Object Identification

texts:
Is it a good idea to use pre-trained weights of say VGG19 (or other) model for a classification problem for a completely a new class of object altogether?

Whole idea is, I'm working on a binary image classification problem and I want to use transfer learning. I have used VGG19 model and its pre-trained weights. Instead of using the last layer of VGG19, I have used the last fully connected(FC) layer having size [-1, 4096] and plugged few of my FC layers with it.

My questions are:

VGG19 is not trained for that particular class, is it still ok to use pre-trained weights or should I use this or other architecture only?
If it's ok to use pre-trained weights, should I incrementally trained these VGG19 weights or just trained my newly connected layers?

Thanks.


texts: Advantages and disadvantages of using softmax/sigmoid and categorical_crossentropy/binary crossentropy for a binary classification with a CNN

texts:
I'm doing a deep learning model using tensorflow and keras. I have a question about the output architecture. I want to classify between two classes, images with defects and images without defects, I've built a CNN based on VGG16 but smaller.
The problem is basically a binary classification.It's possible to use one unique neuron on the output using the sigmoid activation and the binary crossentropy function as loss. The other option is to use two neurons, using softmax and categorical crossentropy as loss (of course, using a onehot ecoder to represent the 2 labels). I've been looking for a detailed explanation of the difference between this two options but I have not found response of what I really want to know.
This is a related question I've read: https://stats.stackexchange.com/a/260537/295456 Here is said that binary crossentropy is just a special case of categorical crossentropy.
Is there any advantage in any of this approach? I'd like to know the advantages and disadvantages of both options. The network would converge faster/slower? There would be more parameters to train? One of this options can be considered as bad desing?
This is just a schematic code to represent the two aproches:
## Opt 1.
targets = [[0,1], [1,0],...]
...
model.add(Dense(units = 2, activation = 'softmax'))
model.compile(loss = 'categorical_crossentropy')

## Opt 2.
targets = [1,0,1,...]
...
model.add(Dense(units = 1, activation = 'sigmoid'))
model.compile(loss = 'binary_crossentropy')


texts: How to predict whether the client will renew the subscription or not based on given data structure

texts:
I have a requirement where I want to predict whether the client will renew the subscription or not.
And the data is something like below.

Basically client's subscription end date can be anything. And I want to predict whether client A will renew the subscription in Oct or not.
And Similarly client B will renew it in December or not.
And I want to run the model every day because every client's subscription end date can be anything.
Along with this I have various features of the client i.e. Feature 1, 2, 3 etc which can be used to train the model.
My question is, how do I structure the data so that I can run the model everyday and what would be my output variable. What kind of model I can fit.
I was thinking of survival analysis but not sure if  the data structure I have can be used for survival analysis or not.
Need your suggestion to approach this problem.


texts: Data model and algorithm for recommending "related" interests

texts:
On my app, when a user selects an interest (example: ios), I'd like to show related interests (swift, xcode, apple, etc). I have a list of around 700 interests/tags (about 300 of them can be classified as tech). How do I design the system so that when a user selects one of the 700 items, the system can show other relevant items? Note that there is no "learning" involved in this. I'd just go and classify each one myself, and this list will be mostly static.

This is how I have been thinking about it:

I could assign a list of categories to each interest. These categories go from narrow to broad. The broadest category is just "tech", so if a user selects "ios", I could right away eliminate the 400 non-tech interests. Now I can fine-tune by looking at narrower categories (example: "mobile-apps", so I could recommend things that are categorized as "mobile-apps". I could go fine-tune further by looking at a narrower category like "ios-apps". So when a user selects an interest, I start by searching for items with the narrowest categories, and if I need more results, and I need to search for more, then, I go broader. To get started with this, I could simply have up to three categories (narrow, medium, broad) for each interest. Once I identify some categories, I can assign them as narrow, medium, or broad to each one. Now for any given interest, I can find out what other interests share the same narrow, medium, or broad categories.
The other approach I was thinking was to maintain a list of related interests for each interest. This requires more initial work on my part, so for each of the 700 interests, I'd find the 5-10 other interests that relate to it best, so for example, "ios" could have related interests like [swift, xcode, iphone, ipad], and so when a user enters ios, I just have to look up this list. I could maintain this list in a relevancy order. This approach is tedious as when I want to add a new interest to my list of 700 interests, I'd have to go through each and see if this new interest needs to be added as a related interest.

My question is--am I trying to reinvent the wheel? Are there any standard data models and algorithms to do this type of thing? I have been trying to Google this and the results seem to indicate things like interests graphs, cluster algorithms. I have no background in data science or machine learning. Any input is appreciated!


texts: What is the the cost of combining categorical variables?

texts:
I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was considering making a variable location that is a concat of the two columns e.g. state:california city:LA becomes California_LA & state: california city: Missing becomes california_None.

I am building a NN and I want to know what the cost of doing this is. Will I be required to do more computation because the unique values in the location variable will be much greater than state or city?


texts: How to model user's buying behavior on Amazon?

texts:
For our final course project in Data Science, we proposed the following-

  Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that determines a strategic position for placing ads on Amazon. For example, there are millions of products on Amazon. And the dataset gives you an idea of what products are related, what products were brought together, viewed together etc. ( We can construct a graph with this info of also viewed and also bought ) It also gives you the reviews associated with each product over 14 years. Using all these info, we will rate/rank products on Amazon. Now, you are a vendor on Amazon who wants to improve traffic to their product page. Our algorithm helps you identify strategic positions in the graph where you can place your ad so that you can derive max traffic.

Now, our Professor's question is, how will you validate your algorithm without real users? We said-

  We can model a fixed set of users. Some users follow also_bought and also_viewed links to the third hop more often than first or fifth hop. There users' behavior is normally distributed. Some other users hardly navigate beyond the first hop. This set of users' behavior is exponentially distributed.

Our Professor's said - Whatever distribution the users follow, users are navigating using links for similar products. Your ranking algorithm also considers similarity b/w 2 products to rank products. So using this validation algorithm is kinda cheating. Come with some other user behavior, something more realistic and orthogonal to the algorithm.

Any ideas on how to model the users' behavior? I am happy to provide more details about the algo.


texts: YOLO algorithm - understanding training data

texts:
I am taking "Convolutional Neural Networks" on Coursera and it is taught by Andrew Ng. I am in week 3 and confused about YOLO algorithm. I checked the course forums on coursera but I am still not clear and it seems that many people are confused about the same

what is the training data? all the images and the location of
objects in it? so for an image, if there is one car and one man in
it then the input would be vector representation of the image,
location of the centers (x,y), height and width of 2 objects and
object code of what is contained in each box. So in total vector
representation of the image + 4 coordinates of the 2 centers + 4
height and width of 2 boxes + 2 object codes - one for man and one
for car. Right?
what is the test data? vector representation of test images only? The course talks about splitting image into ss grid.
I initially thought that a small square from ss grid is fed
as an input. But i dont think that is the case. The entire image
must be fed. is that correct?
How a square from s*s grid is used?
does size of training image and test image has to be the same? what
they mean by "The car detection dataset has 720x1280 images, which
we've pre-processed into 608x608 images." in week 3 assignment?

update 1------------------------------------------

if a train image has 2 objects and maximum classes is 3 then  then
would input be vector representation of the image + input vector of
8*2?
the length of the input vector would change based upon objects in an individual image?
i am still not clear how ss grid is used. if there is a big car in
the middle of the image and YOLO looks at a small square from ss
grid from the center of the image then it is impossible to detect
that there is a car. We have to provide bigger square from center to
YOLO so that it understands there is a car. So what is the use of
feeding a small square?
How do we feed a bigger square?
is the process of feeding square from s*s grid occurs after multiple
convolution layers (so that basically a smaller square represent a
much larger area)?
do you have an example of an image and its shrink version? is their
some data science around shrinking?

--------------------update 2

i read the answer and looked at videos again and still not clear

2.the length of the input vector would change based upon objects in an individual image? for example if image1 has 5 objects then length of the input is going to be much longer than an image which has only 1 object. How do we feed such kind of data where input is not of fix width. Do we find an image with with maximum objects and decided the length of the input and for rest of images we just pad 0s (to make input vector of the same length)?

I am still  confused about ss grid. I initially asked
How a square from s*s grid is used?

after reading the below answer, my updated question is
i am still not clear how ss grid is used. if there is a big car in the middle of the image and YOLO looks at a small square from ss grid from the center of the image then it is impossible to detect that there is a car. We have to provide bigger square from center to YOLO so that it understands there is a car. So what is the use of feeding a small square?

the answer (in the comment of the orignal answer) says that
3: A grid cell doesn't contain a whole bounding box, but only the mid point of a bounding box.

my confusion is :
as per earlier discussion, we dont feed a grid cell individually. We feed the entire image once. so what is the point of creating the ss grid. If the image is looked only once then how the algorithm detects say two objects - one big car and one small car in a single go? We create the grid and the grid is used only to find midpoint of the object. But then the entire object is identified. I am still not clear with this part.

i feel that i am not the only one who is having hard time understanding YOLO. I saw multiple threads in the comment section of the course asking similar questions and I would appreciate patience and guidance


texts: Catboost Categorical Features Handling Options (CTR settings)?

texts:
I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical features. And learned that one-hot encoding I have been using in past is really bad idea especially when it comes to lots of categorical features with many levels (read these post, and this).

While I've come across methods like target-based encoding (smoothing) of categorical features often based on mean of target values for each feature perhaps this post/kernel in Kaggle. Still I am struggling to find a more concrete way till I found CatBoost an open-source gradient boosting on decision trees released last year by Yandex group. They seem to offer extra statistical counting options for categorical features likely much more efficient than simple one-hot encoding or smoothing.

The problem is the documentation is not helpful how to set CTR settings. I have tried different ways but it just does not work. The doc says CTR setting as simple_ctr, to be given as (CTR setting section):

['CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',
 'CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',
  ...]

Here is a super simple example, the data looks like this:

import pandas as pd
import catboost
data = [{'profit': '342','country': 'holland','account': 'Jones LLC', 'saving': 150, 'debt': -60, 'age': 28},
         {'profit': '875','country': 'germany','account': 'Alpha Co',  'saving': 200, 'debt': -10, 'age': 42},
         {'profit': '127','country': 'italy','account': 'Blue Inc',  'saving': 50,  'debt': -300,  'age': 38 }]
df = pd.DataFrame(data)

Here is a simple Catboost Regressor:

X_train = df.drop(['profit'],axis=1)
Y_train = df['profit']
categorical_features_indices = [0,2]

train_pool = catboost.Pool(X_train, Y_train, cat_features=categorical_features_indices)

model = catboost.CatBoostRegressor(
    depth=3,
    iterations=5,
    eval_metric='RMSE',
    simple_ctr=None)

model.fit(train_pool);

The simple_ctr, one of CTR settings, is the problem!! It is pity because it looks like it the package offers various methods, so far no way to access them.

UPDATE Aug. 9th, 2018: A few days ago I raised this problem to Catboost developer, see here, and they opened a ticket for it to provide a tutorial.


texts: How to import large datasets into Google Colab?

texts: Does anyone know a (free) method for importing large datasets into Google Colab, of multiple GB? Github is severely limited, uploading a folder to google drive takes a long time.


texts: Converting pandas series object to int in pandas

texts:
I started my machine learning journey by deciding to explore recommender systems so that I can apply it in some of the projects for my company. Now, I am using Pandas for data analysis. The data set is the imdv movies data set. Here, I am trying to convert a pandas series object to int but it converts the series to float64.

Here is the screenshot:

'clean_ids' is the method that I am using to do this and you can see that 'id' changes to float64. Please help me with this.


texts: How cross validation works for regression?

texts:
For regression type problem we know the result is a continuous value, so how is it be cross validated?
In classification type problem we know the class label so easy to compare, but how is it compare in regression type problem?


texts: Why do good word embeddings come out of maximizing cosine similarity?

texts:
I have an understanding of the technical details of word2vec.

What I don't understand is why semantically similar words should have high cosine similarity.

From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings


texts: Is high school GPA included in IPEDs NCES databases?

texts: I’m currently developing a computer tool which will sort through the US college and university NCES data provided in the IPEDs website to match users with potential universities. NCES shows university data on admission test scores, but I’m hoping to use the user’s high school GPA. I can’t find anything on accepted GPA scores in the NCES database contains data. It doesn’t seem to be an uncommon trait to search universities by, and would be quite useful to use when collecting/analyzing data. Is there any high school GPA info in NCES? if not, is there a similar database i can use to get GPAs without risk of copyright infringement?


texts: Replace values in dataframe with another dataframes values based on condition

texts: I have 2 dataframes with the same columns names and rows number but the elements' values are different. I need to change the value of each element in the first dataframe to 1 if its value in the the other dataframe is 1. I am stuck on this task. Any help pls?


texts: Multiclassification with large number of labels

texts:
I am attempting to build a classifier with a large input space of one hot encoded vectors. The output should be a vector of labels, with 10000 possible labels each. For example, the labels could correspond to something like: (Class 10, Class 992, Class 800, Class 23, Class 30) etc. My first attempt was to one hot encode this output as well, resulting in an output space of size (10000, 5). However, I quickly discovered that to have an output space this large, I would easily exceed the memory on my GPU (16gb).
Question: What is the best way to encode and implement such an output space in PyTorch?
My first thought was to drop the one hot encoding and encode the outputs exactly as I did above, dropping the word class obviously so that the output would be a 5 dimensional vector (10, 992, 800, 23, 30). However, I wasn't sure which loss function to use here: obviously MSE or other such regression losses would not be appropriate. After doing some research, I found someone who recommended using TensorFlow's softmax_cross_entropy_with_logits (sorry I can't seem to find where I originally saw this anymore).
Using https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/59197990#59197990, I was able to implement a criterion in PyTorch after removing the masked parts, that would accept tensors of size (batch_size, 5) for both labels and outputs and return a number.
However, I'm not sure I'm using this correctly. Attempting to pass the same arguments into the TensorFlow version that this is supposed to be equivalent to throws dimension errors. Also, testing this on some fabricated instances don't seem to match up with what my intuition says it should:
label = torch.tensor([[8.0], [8.0]]) # Corresponds to a two instances of class 8
out = torch.tensor([[7.0], [8.0]]) # Corresponds one incorrect prediction
criterion(out, label) # Expecting a nonzero loss because the first prediction was incorrect
# However, returns tensor(0.)

It is not behaving as expected even with a single label, let alone five. I am thinking with five, if I can get the loss function working for one, to do a balanced sum of each individual.


texts: are OneHotEncoder and keras To_categorical same?

texts:
The length of human_vocab is 18377.
The length of input X is 1000

I'm trying to run to_categorical

np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), X)))

Is this the same if i apply:

onehot_encoder = OneHotEncoder(sparse=False)
onehot_encoder.fit_transform(X)

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

human_vocab contains the corpus data that X doesnt contain
I'm trying to find the replacement of to_categorical due to memory issues to create hot vector


texts: which classification model allows user to choose importance of data inputs?

texts: I am working on a match analytics project where I have to deal with the situation in which I am having some inputs like skills, experience, certifications etc. and my output is candidate selected Yes or No, my problem is, Is there any algorithm which will allow users to give weightage to one Input than the others? For example skills matters more for someone whereas for other user experience matters . I have tried neural network and Naive bayes algorithm but unable to choose importance of variable . Any help will be highly appreciable.


texts: If I call model.cuda() in pytorch where model is a subclass of nn.Module, and say if I have four GPUs, how it will utilize the four GPUs and how do I know which GPUs that are using?


texts:
I have a column called item_colour which describes the colour of products in my dataset.

There are 85 colours and some of the colours are unique or they represent a small part of the data ( I have 100,000 observations). For example, there is only one "almond" colour or only two "sky blue". I want to recombine rare values and put them together into a group called "other colour".

How can I find the threshold to recombine them? For example, combine the colours together which have values less 50 or so.

P.S.I am working with R


texts: What does 'lm' mean in the lmplot() in seaborn?

texts: I was learning seaborn, but could not find the full name of seaborn.lmplot. What is 'l' and 'm' in the name 'lmplot'.


texts:
kmeans = KMeans(n_clusters=4)

model = kmeans.fit(europe_july)
pred = model.labels_
europe_july['cluster'] = pred

pca = PCA(n_components=2)
pca_model = pca.fit_transform(europe_july)
data_transform = pd.DataFrame(data = pca_model, columns = ['PCA1', 'PCA2'])
data_transform['Cluster'] = pred

plt.figure(figsize=(8,8))
g = sns.scatterplot(data=data_transform, x='PCA1', y='PCA2',\
                    palette=sns.color_palette()[:4], hue='Cluster')
title = plt.title('World countries clusters with PCA')

But when I run this code it does not seem to take into account this model.
europe_july['country'] = countries
europe_july['iso_alpha'] = iso_alpha

fig = px.choropleth(data_frame = europe_july,
                    locations= "iso_alpha",
                    scope= 'world',
                    title='2020-11-07 (World)',
                    color= "cluster",
                    hover_name= "country",
                    color_continuous_scale= 'earth',
                    )

fig.show()

Since this is the output that I get, as you can see there is clearly a cluster with only three countries, when there is no such cluster predicted by the model.
This is the output of the predictions for clusters and it matches the visualizations by PCA:
array([2, 3, 1, 0, 2, 0, 0, 3, 1, 3, 3, 3, 1, 3, 3, 1, 1, 3, 3, 1, 2, 0,
       1, 1, 0, 1, 2, 3, 0, 2, 3, 2, 2, 1, 2, 3, 2, 0, 2, 2, 3, 3, 1, 2,
       2, 1, 2, 3, 1, 3, 3, 3, 2, 3, 2, 0, 1, 1, 1, 1, 2, 2, 3, 2, 0, 0,
       2, 3, 3, 0, 2, 2, 3, 3, 2, 0, 3, 0, 2, 3, 1, 0, 2, 2, 1, 2, 1, 3,
       3, 3, 1, 1, 3, 1, 3, 0, 3, 3, 1, 3, 0, 2, 1, 2, 0, 3, 1, 2, 3, 3,
       2, 2, 2, 0, 3, 3, 3, 2, 2, 3, 1, 2, 3, 2, 3, 1, 1, 0, 1, 3, 0, 2,
       2, 1, 2, 1, 3, 0, 3, 0, 2, 2, 0, 3, 1, 1, 2, 3, 2, 1, 3, 1, 3, 3,
       3, 3, 3, 3, 2, 0, 1, 0, 0, 2, 3, 2, 1, 3, 2, 3, 0, 3, 3, 2, 1, 3,
       2, 3, 3, 2, 1, 2, 3, 3, 2, 3, 1, 2, 1, 2, 2, 1, 1, 3, 0, 2, 3, 3,
       3, 3, 3, 3, 2, 0, 2, 1, 0, 2, 2, 2, 1, 0], dtype=int32)

Could someone please guide on why my visualisation is wrong?


texts:
I am working in the problem where the dependent variables are ordered classes, such as bad, good, very good.

How could I declare this problem in xgboost instead of normal classification or regression?

Thanks


texts: What is an optimal local sparse structure of a convolutional vision network?

texts:
I was reading the InceptionNet Paper, where I found quite a few references to developing a sparse network structure, but I am not clear on what this means.
An instance of where this has been mentioned:

The main idea of the Inception architecture is to consider how an optimal local sparse structure of a convolutional vision network can be approximated and covered by readily available dense components. Note that assuming translation invariance means that our network will be built from convolutional building blocks.


texts: Tutorial for restricted Boltzmann machine using PyTorch or Tensorflow?

texts:
I am trying to find a tutorial on training Restricted Boltzmann machines on some dataset (e.g. MNIST), using either PyTorch or Tensorflow. The few I found are outdated.

Can you recommend any?