texts: Dealing with missing data in several features at once

texts:
Good day,
What are the approaches for handling missing data in several features (categorical and continuous) at once? I look through each feature and plotted several histograms of the distribution of them, and I think that a simple substitution of value by some constant value (mean, mode or something else) is not the best choice. I want to build to several models (classifications for categorical and regressions models for continuous values) in order to impute data, but I don't know the right approach for doing this. If I train my models only using non-null rows, then I won't be able to predict values with null values. On the other hand, If I simply drop columns(features) with null values, then I got bad results because features with null values are valuable for prediction. I have 615739 rows * 49 columns data set (after one-hot encoding procedure). If I drop all rows that contain at least one null value, then I will get 451063 rows. Thanks for your advice in advance.


texts: How to categorize unlabelled promotional email data

texts:
I have unlabelled data of promotional emails. I want to categorize those emails based on the topics like fashion, health & wellness, sports, media, Entertainment, etc. Can anyone let me know any effective method or any pre-trained model which i can directly use to categorize those emails (any similar model like YOLO which we have for object detection).
Any kind of help regards to this would be much appreciable. Thanks!


texts: how to detect bifurcation, trifurcation, and crossover on image with python opencv

texts:
how to detect bifurcation, trifurcation, and crossover from the branch point with OpenCV
when the input like this

Expect output :

I'm trying to use binary_hit_or_miss but I don't know how to display the detected area
import io
import numpy as np
from PIL import Image
from scipy import ndimage
from IPython.display import display # to display images

image_string = open('image/skeleton/21_bloodvessel.png', 'rb').read()
img = Image.open(io.BytesIO(image_string))
arr = np.asarray(img)

structure1 = [
    [0,1,0],
    [0,1,1],
    [0,0,0]
]
np_array=ndimage.binary_hit_or_miss(arr, structure1=structure1).astype(np.int)
np_array=Image.fromarray(np_array)
im = display(np_array)


texts: Using historical label as a feature in my ML model?

texts:
I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of 0.1 and so basically when the rate of change is above a certain threshold it gets a label 1, if below -1, and if not then 0.

Because I am using an exponentially smoothed moving average for the label, am I still allowed to use historical labels as a feature in my dataset? So for example if my predictive variable is the direction for t+5, could I use label at point t? Because there is an overlap in the data used


texts:
I'm trying to create a new metric for my model and this is my function

def my_acc(y_true, y_pred):

    if(y_pred > y_true):
      x = ((y_pred-y_true)/y_true)*100
    else:
      x = ((y_true-y_pred)/y_true)*100

    a = 100 - x
    return(a)

When I'm trying to use the above function in the model the following error occurs

def train_model():

  optimizer = tf.keras.optimizers.RMSprop(0.001)
  model = build_model()
  model.compile(optimizer=optimizer,
                loss= 'mean_squared_error',
                metrics=['mean_absolute_error', 'mean_squared_error','accuracy',my_acc])

  logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
  tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

  model.fit(train_dataset, train_labels,
            validation_split = 0.2,
            epochs=500,
            callbacks=[tensorboard_callback])

train_model()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-37-847c0f5f49e0> in <module>()
     15             callbacks=[tensorboard_callback])
     16
---> 17 train_model()

12 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in __bool__(self)
    688       `TypeError`.
    689     """
--> 690     raise TypeError("Using a `tf.Tensor` as a Python `bool` is not allowed. "
    691                     "Use `if t is not None:` instead of `if t:` to test if a "
    692                     "tensor is defined, and use TensorFlow ops such as "

TypeError: Using a `tf.Tensor` as a Python `bool` is not allowed. Use `if t is not None:` instead of `if t:` to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
````


texts:
I often get the problem when this or that alias name is already used somewhere, and I can't easily find that variable or aggregation to release the name.

Is there some place in Tableau where I can view/edit/reset full list of aliases?


texts: Select best answer from several existing ones for a question

texts:
After analyzing questions on a forum, a human support team has created a set of general answers, that can be used to provide basic answers on the forum.

I am trying to build a system that:

Selects best answer from this set of answers for a given question. How to do this?
Estimates acceptability of such an answer. Which metrics to use?

Using document embeddings, such as doc2vec to find similarity between question and answer does not solve the problem, I think.  Other ideas?

Update 1

In my case I don't have labeled data set with good answers to train my model. My problem is unsupervised learning problem.


texts: Confusion matrix - determine the values of FP FN TP and TN

texts: After running my code ,I get the values of accuracy, precision and recall and I want t determine the values of FP FN TP and TN from these metrics. I tried to calculate it using the formula of each metric but I couldn't. Is there any way to do this?


texts:
I was reading about Linear Learning Machines (LLMs) and learned that it is closely related with SVMs. Would like to know an example of any concrete problems that can be classified by LLM as I couldn't find any promising example or explanation for this.
Appreciate your time.


texts: How to setup my Ubuntu PC for these Deep Learning frameworks?

texts: I need to use both NVIDIA Digits and tensorflow Object Detection API for different deep learning purposes. I am a bit lost with the compatibility issues, especially for what concerns CUDA. So which version should I choose for DIGITS, Tensorflow-gpu, CUDA and Object Detection API in order to make everything work?


texts: What is the difference between these 2 Python Pandas queries

texts:
I am trying to make sense out of some data, here is the Pandas data frame:

In [1]: import pandas as pd
In [2]: df = pd.read_csv('c1.csv')
In [3]: df.head(2)
Out[3]:
   year     city        country  avg_temp
0  1849  Abidjan  Côte D'Ivoire     25.58
1  1850  Abidjan  Côte D'Ivoire     25.52

All I want is to know if the year is greater than 2000 and  avg_temp is less than 20, then what's the mean() and I get 2 different answers:

In [10]: (df.query('year > 2000')['avg_temp'] < 20).mean()
Out[10]: 0.5629877369007804

In [11]: ((df.year > 2000) & (df.avg_temp < 20)).mean()
Out[11]: 0.03540828203222504

In [12]: ((df['year'] > 2000) & (df['avg_temp'] < 20)).mean()
Out[12]: 0.03540828203222504

I can't figure out which one is correct


texts:
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.

Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.

Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?


texts: For diagonal difference this logic is showing a garbage value. Where am I doing wrong?

texts:
int main()
{
    int i,a[3][3];
    int diagonal_diff;

    int lsum=0,rsum=0;
    for(i=0;i<3;i++){
       lsum=lsum+a[i][i];
       rsum=rsum+a[i][2-i];
      }
     diagonal_diff= abs(lsum-rsum);
     printf("%d",diagonal_diff);

}


texts: machine learning for a small longitudinal or panel data set


texts: Multiclass classification task where each class is present only once in the test set

texts:
I have a multiclass classification problem where, in the test set, there is only one entry for each possible class. In my particular problem we want to guess the author of a text, and we have 20 different authors. The test set contains 20 texts, one of each author.

I have many texts of each authors in the training data. I can't change the content of the test set, it's these specific texts that I need to classify.

Does this kind of task has a name, so I can google previous work on it more easily ? I'm not talking about author identification specifically, but having 1 entry for each class.
Do you have any suggestion of algorithm that would work well for this task ? Maybe something that gives ranked results or a confidence value ?

Thank you


texts:
I wanna install smote from imblearn package and I got the Following error:

ImportError                               Traceback (most recent call last)
<ipython-input-10-77606507c62c> in <module>()
     66 len(data[data["num"]==0])
     67 #balancing dataset
---> 68 from imblearn.over_sampling import SMOTE
     69 import matplotlib.pyplot as plt
     70 sm = SMOTE(random_state=42)

~\Anaconda3\lib\site-packages\imblearn\__init__.py in <module>()
     33 """
     34
---> 35 from .base import FunctionSampler
     36 from ._version import __version__
     37

~\Anaconda3\lib\site-packages\imblearn\base.py in <module>()
     17 from sklearn.utils import check_X_y
     18
---> 19 from .utils import check_sampling_strategy, check_target_type
     20 from .utils.deprecation import deprecate_parameter
     21

~\Anaconda3\lib\site-packages\imblearn\utils\__init__.py in <module>()
      5 from ._docstring import Substitution
      6
----> 7 from ._validation import check_neighbors_object
      8 from ._validation import check_target_type
      9 from ._validation import check_ratio

~\Anaconda3\lib\site-packages\imblearn\utils\_validation.py in <module>()
     12
     13 from sklearn.base import clone
---> 14 from sklearn.neighbors.base import KNeighborsMixin
     15 from sklearn.neighbors import NearestNeighbors
     16 from sklearn.externals import six

~\Anaconda3\lib\site-packages\sklearn\neighbors\__init__.py in <module>()
      7 from .kd_tree import KDTree
      8 from .dist_metrics import DistanceMetric
----> 9 from .graph import kneighbors_graph, radius_neighbors_graph
     10 from .unsupervised import NearestNeighbors
     11 from .classification import KNeighborsClassifier, RadiusNeighborsClassifier

~\Anaconda3\lib\site-packages\sklearn\neighbors\graph.py in <module>()
      5 # License: BSD 3 clause (C) INRIA, University of Amsterdam
      6
----> 7 from .base import KNeighborsMixin, RadiusNeighborsMixin
      8 from .unsupervised import NearestNeighbors
      9

~\Anaconda3\lib\site-packages\sklearn\neighbors\base.py in <module>()
     20 from .kd_tree import KDTree
     21 from ..base import BaseEstimator
---> 22 from ..metrics import pairwise_distances_chunked
     23 from ..metrics.pairwise import PAIRWISE_DISTANCE_FUNCTIONS
     24 from ..utils import check_X_y, check_array, gen_even_slices

ImportError: cannot import name 'pairwise_distances_chunked' from 'sklearn.metrics' (C:\Users\ASUS\Anaconda3\lib\site-packages\sklearn\metrics\__init__.py)


texts:
What is the difference between fit() and fit_generator() in Keras?

When should I use fit() vs fit_generator()?


texts:
I'am using Rstudio to create some codes. the cods works pretty well, without any error and gives the output desired. But when I turn on the PC after period and re-execute the same code, it generes errors especially in reading external files like

(Error in names(out) <- named :    'names' attribute [10] must be the
same length as the vector [3]).
Error during wrapup: 'names' attribute [10] must be the same length as
the vector [3]

(these 2 different errors are generated from executing the same part of code 2 times. the code want to tokenize a corpus of 10 files and save it in  tables of 3 columns)
I'am using the same dataset from the same location and I'am sure of calling the correct libraries
library("tm")

#seting the corpus
corp <- Corpus(DirSource(path),
               readerControl=list(reader=readPlain,
                                  language='en_CA',
                                  load=TRUE));

myTokenizer <- function(x, n, n_min) {
  library(tokenizers)
  tok <- unlist(tokenize_ngrams(x, n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3,
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
  corp<-tm_map( myTokenizer(x=corp, n=6, n_min=1) )
writecorpus(corp)


texts: Machine Learning - Euclidian Distance Classifier exercise

texts:
I'm taking part in an elective subject at university which mainly focuses on the foundations of Machine Learning.
Now we got our first exercise - this task should be done practically in any language (I've chosen Python). Our teacher doesn't explain the relations between theory and practice well and so it's hard for all of us to follow along - so I decided to post this question here. I don't want anyone to give me a solution, I just don't understand what he wants and may be a hint how to approach this question.:
Here is the full exercise:

Euclidean distance classifier

Develop an Euclidian distance classifier as below: Generate 1000
random points corresponding to each class out of 3 classes with
feature size 2 for a 3-class classification problem. For the
simplicity consider the classes following N([0 1 2], I), N([0 0 1], I)
and N([1 0 0],I) respectively.

Generate the output an 1000-dimensional vector whose ith component
contains the class where the corresponding vector is assigned,
according to the minimum Euclidean distance classifier.

I understand that I should generate random points with two features which are belonging to one of the three classes - okay. But I don't get the second part of the sentence. The classes are normally distributes with a mean(?)-vector of [0, 1, 2], [0, 0, 1] and [1, 0, 0]?

For what does the second paramter I stand in the normal distribution
Does the vector stand for the position/mean of the multivariate normal distribution?
How would you approach this question? Using a k nearest neighbor algorithm?

Thanks for any helpful answers!
Max


texts: How to reference Zero Rule (ZeroR or 0R) baseline classifier

texts:
Does anybody know what is the reference paper for Zero Rule(ZeroR) classifier? I found a paper about 1R but I cannot find any reference to ZeroR baseline classifier.
Any help would be greatly appreciated.


texts:
I've been working in SAS for a few years but as my time as a student with a no-cost-to-me license comes to an end, I want to learn R.

Is it possible to transpose a data set so that all the observations for a single ID are on the same line?  (I have 2-8 observations per unique individual but they are currently arranged vertically rather than horizontally.)  In SAS, I had been using PROC SQL and PROC TRANSPOSE depending on my analysis aims.

Example:

ID    date        timeframe  fruit_amt   veg_amt

4352  05/23/2013  before     0.25        0.75

5002  05/24/2014  after      0.06        0.25

4352  04/16/2014  after      0           0

4352  05/23/2013  after      0.06        0.25

5002  05/24/2014  before     0.75        0.25

Desired:

ID    B_fr05/23/2013   B_veg05/23/2013  A_fr05/23/2013  A_veg05/23/2013   B_fr05/24/2014   B_veg05/24/2014   (etc)

4352  0.25             0.75             0.06            0.25              .                .

5002  .                .                .               .                 0.75             0.25


texts: Assigning tags to posts using predefined set of tags

texts:
I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. Nearly $50$ posts with tags are available for testing the results.

My approach: Remove stopwords, punctuations. Find the similarity(cosine) between word vector of each word of the post and the vectors of all the tags.

Problem: Context sensitive tags like fired(as in from a job) are shown for irrelevant post e.g. 'car back-fired) and only on average $3$ out $5$ most similar tags are relevant.

Gathered more posts($~200$ with average word length $40$) from other websites. Tried preprocessing of the posts: lemmatization and stemming, created dictionary, made bow corpus then used Topic modeling (Latent Dirichlet Allocation)

Used gensim.models.LdaMulticore tried both BOW and tf-idf models but the topics produced had low confidence(of the order $0.07$) for the words in them. Found the relevant tags (using vector similarity) considering only top $10$ words of each topic. But the performance degraded even more as now at most $2$ tags were relevant.

Conditions:The tags are diverse and finding posts/text related to each tag is difficult. Tags are not be modified.

Does anyone have a better approach? Any help would be appreciated.


texts: Passing variables and values from an R script to a shell script

texts:
I'm working with a shell script(#!/bin/sh) and I wanted to know if there is a way to call variables with their values from an Rscript that I have called in my Shell script.
If that doesn't make sense I want to create, for example a data frame
data=data.frame(a=seq(1,5), b=seq(1,5)) in a script called test.r
and then call that variable, with it's content in my shell script, i.e to print it with an echo:

echo $data


texts: Related words extraction in text processing

texts:
I'm working on some aspcet based sentiment analysis project applied on some tweets or reviews, so far I've applied all the known preprocessing methods on my training dataset which is an XML file that you can find on THIS LINK.

What I wanna do next is extracting related words to some aspects, so I can pass them through my classifier to decide whether it's positive or negative, here's an example to make it clearer:

let's take the first example of my dataset (first tweet);

 < sentence id="2339">

     < text>I charge it at night and skip taking the cord with me because of the good battery life.< /text>

     < aspectTerms>
         < aspectTerm term="cord" polarity="neutral" from="41" to="45"/>
         < aspectTerm term="battery life" polarity="positive" from="74" to="86"/>
     < /aspectTerms> < /sentence>

so as you can see in this example there are two terms that I should classify the sentiment expressed about, my problem isn't the extraction of the terms which goes back to the ''aspect terms extraction'' famous problimatic, but it's how to decide what are the words associated to those terms, in the example I presented; ['charge', 'night', 'good'] are associated to the term 'battery life', while ['skip', 'taking']  are associated to the term 'cord'. In other words, I would love to have extract from one sentence (tweet) the words related to my Aspect terms separated from each other.

can anyone give me a hint or some oriontation about how to figure this out?


texts: Fit model function out defined data range

texts:
I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data (x=times,y=observation) that have gaps in time. Whatever is their trend, let's assume it linear for this discussion. The actual data do not have a linear trend, and the gaps are more than one, but for the sake of simplicity I would like to deal with a general example that I can understand and then extend to a more specific case. During the gap in time, there is a decay that makes data deviate from the purely linear trend, until observations start again and the linear trend is recovered. I want to model the decay as part of the function.

from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

def f(x, A, B, decay):
    return A*x + B + decay

x=[1,2,3, 12,13,14]
y=[2,4,6, 5, 7, 9]
popt, pcov = curve_fit(f, x, y)

figure = plt.figure(figsize=(5.15, 5.15))
figure.clf()
plot = plt.subplot(111)
ax1 = plt.gca()

plot.scatter(x,y)
plt.show()

How do I model the decay variable as part of the function and obtain its best-fit value, let's say in the case where the decay is linear or quadratic? How do I plot the whole function ? I have read something about imputing but I it is the first time I read about it and I am not even sure that is what makes my day.


texts: How does backpropagation work with averaging layers?

texts:
I'm studying Word2Vec algorithm, and so far i understood that, in the case of input context bigger than 1 (so multiple words) we have our hidden layer that performs averaging between the inputs (as explained here: Word2Vec CBOW)

I didn't understand how does backpropagation work in this case.

When i start backpropagating from the softmax output layer and i reach for the hidden layer in my process, am i coping the weight over all the "same position" inputs?

To make my question clearer, if i have 2 context words coded as one hot vectors and i call the first weight of the first word as W11 and the first weight of the second word as W21, when i'm backpropagating, those two weights will be updated with se same value, right?


texts: what is the best approach to detect small objects with similar shape?

texts:
I'm working a model which detect different products in supermarket shelf. In the training data, there are a lot of objects with similar shape placed very close to or stacked to each others.(eg: milks with different brands are stacked, placed on the same shelf, the model should be able to detect milk1, milk2). What is the best approach to this problem. I've tried to train a Faster RCNN, but the RPN isn't  working well. I've also tried feature matching, but it cannot detect partially visible objects. Any help will be appreciated!

The training images look like this

Link to FRCNN result when detect 2 type of milk and 1 type of yogurt

faster r-cnn detection result


texts: How does tf.random.log_uniform_candidate_sampler work while generating negative samples?

texts:
I was trying to build a simple negative sampler for a Word2Vec model using TensorFlow by following the tutorial here.
From what I understand, the tf.random.log_uniform_candidate_sampler() takes as input the correct class to be exluded from sampling, the number of samples to return, and the size of the distribution to sample.
In the tutorial, the true_classes is 1, range_max is 8 and num_ns is 4. Which translates to, "Return 4 samples from the distribution [0, 8), but skip 1, since it's a positive class."
However, in the tutorial, the sampler returns the following tensor: tf.Tensor([2 1 4 3], shape=(4,), dtype=int64).
My question is, why does the sampler return the class 1, even after being told to exclude it, since it doesn't constitute a negative sample?
Furthermore, in the following section of the tutorial:
target_index    : 7
target_word     : sun
context_indices : [1 2 1 4 3]
context_words   : ['the', 'wide', 'the', 'shimmered', 'road']
label           : [1 0 0 0 0]

We can see clearly that the first instance of class 1 (index 0 in context_indices) has been flagged as a positive skip-gram, whereas the second instance of 1 (index 2 in context_indices) has been labelled as a negative skip-gram.
I'd love some clarification or a correction in my understanding of this model or tutorial.
Thanks!


texts: Could I use logloss as classifier performance measures when "Test & Score"?

texts: I try orange today, and want to use logloss as the classifier performance measures when "Test & Score", but I don't know how to do that.


texts: Why does reducing the n_estimators in RandomForestClassifier improve accuracy?

texts:
I am taking a course that introduced me to sklearn.ensemble.RandomForestClassifier. At first it uses n_estimators with the default value of 10 and the resulting accuracy turns out to be around 0.28. If I change n_estimators to 15, the accuracy goes to 0.32
Here's some of the code:
pl = Pipeline([
        ('union', FeatureUnion(
            transformer_list = [
                ('numeric_features', Pipeline([
                    ('selector', get_numeric_data),
                    ('imputer', Imputer())
                ])),
                ('text_features', Pipeline([
                    ('selector', get_text_data),
                    ('vectorizer', CountVectorizer())
                ]))
             ]
        )),
        ('clf', RandomForestClassifier())
    ])

I thought that increasing the number of trees (n_estimators) in the RandomForestClassifier would give a better accuracy, but sometimes if I use a value of 100 I can get between 0.30 and 0.32. Could someone please explain? How do you find which is the smallest value for getting the highest possible accuracy?


texts: Mapping of categorical features into binary indicator features

texts:
I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator features, the following question is proposed:

  Is it a good idea to map a categorical feature with n values to log2(n) binary features?

Why wouldn't that be the case, seeing as how much resources could be spared by working with fewer features? Does this approach depend on the model that is being used?


texts: Are there any good general techniques for binning/histogramming arbitrary data?

texts: Let's say we want to histogram a finite set of measurements of some quantity. It is straight forward to calculate the usual statistical quantities for our sample such as the mean and the variance. Let's assume we can clean up our data by identifying outliers and moving them into underflow and overflow bins and thus, define more or less optimal min and max values for the plotting range. But how would one decide on the number and the size of bins? I would like to know if there are methods to find the optimal binning for the cases with fixed and variable bin sizes.


texts: Neural network with different input shapes

texts:
I'm currently designing the architecture of a neural network for the colorization of grayscale images. Later on it should be able to colorize images with different sizes and different aspect ratios. I read that this would not be possible with a common CNN. I also read that the only options are downscaling the images to one specific size or to use a big fixed size (like 3000x3000px) and to fill the remaining space black. Both of these options don't seem to be that elegant. The first one is the opposite of what I want and the second would make the neural network slower.

Then I read about Fully Convolutional Networks and that this problem would not exist there. This would be great if it really works. I would like to know why this special network can deal with different input shapes. And maybe you could show me some tensorflow code of such a network.

By the way, I thought about an Autoencoder combined with a GAN for the architecture.

I know that this question is very general, but I would really appreciate your help.


texts: How to re-train a model from false positives

texts:
I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time?

I've trained a deep model for my binary classification problem (fire vs non-fire) in Keras. I have 4K fire images and 8K non-fire images (they are video frames). I train with 0.2/0.8 validation/training split. Now I test it on some videos, and I found some false positives. I add those to my negative (non-fire) set, load the best previous model, and retrain for 100 epochs. Among those 100 models, I take the one with lowest val_loss value. But when I test it on the same video, while those false positives are gone, new ones are introduced! This never ends, and Idk if I'm missing something or am doing something wrong.

How should I know which of the resulting models is the best? What is the best practice in training/re-training a good model? How should I evaluate my models?

Here is my simple model architecture if it helps:

def create_model():
  model = Sequential()
  model.add(Conv2D(32, kernel_size = (3, 3), activation='relu', input_shape=(300, 300, 3)))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  model.add(Flatten())
  model.add(Dense(256, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(2, activation = 'softmax'))

  return model

#....
if retrain_from_prior_model == False:
    model = create_model()
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
else:
    model = load_model("checkpoints/model.h5")


texts: Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning

texts:
I'm currently implementing PPO for a game with the following characteristics:

Observation space: 9x9x(>150)
Action space: 144
In a given state, only a handful of actions (~1-10) are legal
The state in time-step t can vary a lot from state t+1
The environment is episodic (~25 steps, depending on level) and ends with a win or a loose
In some levels, a random policy (if only legal actions are made) can result in a win, in some levels strategy is needed.

I want the algorithm to learn:

Avoid illegal actions as much as possible, preferably only legal actions
Learn to play the best combination of legal action to win as fast as possible

I've tried different reward functions and the best one so far is:

Illegal action: -0.10
Legal action: +0.10
Win: +1.0

I tested it on a simple level that a random policy will beat. During the time I trained, my policy learned to make around 80% legal actions, but it never won. It will probably go up even more if I run for longer and maybe start winning.

The problem with the above reward function is that it doesn't encourage to win with as few actions as possible, as each legal action get a positive reward.

I've also tried the following reward function:

Illegal action: -0.10
Legal action: -0.02
Win: +1.0

But it converges to around 20% legal moves.

I have two questions:

Anyone familiar with a similar problem that have an idea of how I might design the reward function?
Does anyone know of any papers where they discuss the problem of learning three different objectives: To avoid illegal actions, to win and to win by doing the minimum number of actions?


texts: Why can't the XOR linear inseparability problem be solved with one perceptron - like this?


texts: Doubt in interpretation of Bernoulli Naive Baeyes Algorithm

texts:
We say Bernoulli naive bayes assumes gaussian distribution of all continuous features. What happens if I have categorical features also in the dataset?

What type of prior transformation in data is suitable before applying Bernoulli naive baeyes to a dataset ?


texts: Data science / machine learning books for mathematicians

texts:
I have found other requests for references here. In particular in:
Where to start, which books
and
Books about the "Science" in Data Science?

I have given a glance to:

Artificial Intelligence: A Modern Approach (Russel & Norvig)
Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)
Learning From Data (Abu-Mostafa et al.)
Introduction to Statistical Learning (James et al.)
Elements of Statistical Learning (Hastie et al.)
Pattern Recognition and Machine Learning (Bishop)

Now it is difficult to evaluate if they would fit my needs because only a few pages are generally available online. However my first impression is that they do not. In the appendices of Artificial Intelligence: A Modern Approach I can read:

  Mathematicians define a vector as a member of a vector space, but we will use a more concrete definition: a vector is an ordered sequence of values.

This is exactly the kind of approach I am not looking for.

I'm looking for a book which assumes the reader has a good understanding in set theory, abstract algebra, measure and probability theory, statistics, topology, graph theory, complexity theory, etc and a preference for formal and axiomatic explanations rather than lenghty and so-called "intuitive" approaches based on basic mathematical objects and examples. Furthermore I don't want something that looks like a recipe book from the very beginning. I want a book that formalizes the abstract and common shape of all data science methods as well as their common aim first. Only after that it can start to explain the different categories by explicitely stating which further hypotheses each category is assuming and which cases/problems/domains they are known to handle efficiently or not.

At last, to be clear, I have no problem with being shown concrete examples and their treatment via a specific programming language for example. I just want this to come second as an illustration for the conceptual explanation, not as a substitute.


texts: Correct approach to usage of class labels in cell imaging data

texts:
As part of a group project at university, we are given a series of videos of cell cultures over a 24 hour period. A number of these cells (the "knockout" cells) have had a particular gene removed, which is often absent or mutated in malignancy. We are using a blob detection algorithm to identify the cell centers and radii and further processing to match cells frame-to-frame to build up individual paths, which we then use to calculate various features. We are aiming to train a binary classifier that can identify the potentially cancerous cells given one of these path-derived feature vectors.

Our training data consists of immunofluorescence-tagged videos, where the knockout cells are tagged red and the normal or "control" cells tagged green. These are our "labels" so to speak, and we use our blob detection algorithm twice, once on the red channel and once on the green channel, to separate the two classes in the data. Our test data will consist of grey-scale videos where this tagging isn't present.

My team mate has claimed that separating the red and green channels in the training data is "unfair" since, as well as discriminating between the two classes, the tagging sometimes makes it easier to distinguish between individual cells. This is because it is possible for cells to have a significant overlap between one another, making the individual centers somewhat ambiguous. Where a knockout cell and a control cell are overlapping, separating the two channels removes this ambiguity. Since the separation of the colour channels is only possible in the training data, it may be argued that our "labels" are in fact not only identifying which class an individual cell belongs to, but are are also implicitly adding extra information into the training data by sometimes removing the overlapping ambiguity.

Here's an illustration of what I'm talking about:

If we were to take the red and green channels separately, the blob detection algorithm can easily recover the two cells. Otherwise, it just sees one. In this way, the colour labels are implicitly providing extra information for the blob detection.

My question is, is it okay to use this extra information provided by the labels in the training data by separating the two channels as part of the training process?

EDIT: Added illustration.


texts: Keras functional API Layer name not captured with TimeDistributed wrapper

texts:
class Net2:
    @staticmethod
    def build_cat_branch(inputs,category_size):
        x = TimeDistributed(Dense(category_size))(inputs)
        x = Activation('softmax', name="cat_output")(x)
        return x

    @staticmethod
    def build_t_branch(inputs):
        x = TimeDistributed(Dense(1, activation='relu', name="t_output"))(inputs)
        return x

    @staticmethod
    def build_full_model(timestep_len,hidden_size,category_size,num_features,dropout,rec_drop):
        inputs = Input(shape=(timestep_len,num_features),name="Input")
        bn = BatchNormalization()(inputs)
        lstm = LSTM(hidden_size, return_sequences=True, dropout=dropout, recurrent_dropout=rec_drop,name="LSTM")(bn)
        bn2 = BatchNormalization()(lstm)
        cat_branch = Net2.build_cat_branch(bn2,category_size)
        t_branch = Net2.build_t_branch(bn2)
        model = Model(inputs=inputs,outputs=[cat_branch,t_branch],name="Net2")
        return model

When I try to compile this model I get:

ValueError: Unknown entry in loss dictionary: "t_output". Only expected the following keys: ['cat_output', 'time_distributed_2']

(in my model summary, my layer that I name "t_output" has the name "time_distributed_2" instead)

So, basically, my question is what's going on with the TD wrapper that causes the name attribute to not be part of the object returned by the build_t_branch function?  Clearly, the "cat_output" name is stored correctly, as the loss dictionary recognizes it, but the output layer I have inside a TimeDistributed wrapper is not saving the user defined layer-name.  I know I can get around this by just having all layers defined in a single function without the "branch" functions, but that is besides the point here.  Is this a bug in Keras?  Any way to get around this without the above mentioned fix?


texts: How to run R programs on multicore using doParallel package?

texts: I am running SVM algorithm in R.It is taking long time to run the algorithm.I have system with 32GB RAM.How can I use that whole RAM memory to speed my process.


texts: Making sense of a accuracy plot for a 5 fold training using random forest

texts:
I'm using sklearn.model_selection.learning_curve for 5 fold training of data. The code is as given below.

train_sizes = [1, 100, 500, 1000, 2000, 3000, 3879]
train_sizes,train_scores, validation_scores = learning_curve(estimator = ensemble.RandomForestClassifier(),
                                                          X = X_res,
                                                          y = y_res,
                                                          train_sizes = train_sizes,
                                                          cv = 5,
                                                          scoring = 'accuracy')
train_scores_mean = train_scores.mean(axis = 1)
validation_scores_mean = validation_scores.mean(axis =1)
plt.style.use('seaborn')
plt.plot(train_sizes, train_scores_mean, label = 'Training acc')
plt.plot(train_sizes, validation_scores_mean, label = 'Validation acc')
plt.ylabel('Accuracy', fontsize = 14)
plt.xlabel('Training set size', fontsize = 14)
plt.title('Learning curves for a linear RF model', fontsize = 18, y = 1.03)
plt.legend()

Also, I'm using the random forest classifier model for a multiclass classification problem. When I plot the accuracy graph, it appears like the below.

How's it possible that the mean training accuracy for all training sizes is 100%?

Any help is appreciated.


texts: Could anyone explain how this Gumbel-Max pytorch code implementation works ?


texts: What is the appropriate statistical significance test for multi-class classification?

texts:
I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have the results of two classifiers on the same train/test-set (paired observations).
Some sources suggest to use McNemar’s test for binary classification task. However, is there any generalization of McNemar’s test for multi-class classification problem? If so, what would be the appropriate procedure to carry out these tests?


texts:
My chatbots need to be trained when we get new data or feedbacks from users. Can someone provides ways how these chatbots can learn on themselves and become intelligent day by day?
Some of techniques may be :

Reinforcement learning - though I could not find a successful implementation and have asked a separate question. Any successful implementation reference would be helpful.
Building a pipeline with a user feedback - this has 2 drawbacks. One user or admin has to provide the correct answer. Second, a single feedback data may not be enough to retrain the bot.

Any expert advice on - self optimizing chatbot architecture OR some code reference OR some paper trying to solve this problem (a paper with code may be)?


texts: suggestion to implement undersample and oversample

texts:
My dataset has the following class distribution

CLASS      FREQUENCY
2          22696
4          2541
1          2093
5          1298
3          1116
0          960
6          14

definitely I would want to generate a new sample, for that I will use python imblearn , I have three options:

oversample the minority classes
undersample the majority classes
choose a median class and apply both undersample the majority classes and oversamples the minority classes to equal the median class.

Later I will use the generated dataset to train three estimators RandomForest classifier, SVC and ensemble of both. I will choose the one with the best f-1 score. what would be the best option and which oversampling/undersampling algo should I use?


texts:
What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about the weather or holidays or several other specific topics.

I was looking towards LDA and TFIDF but from what I understand this approach is unsupervised and works well for clustering/grouping large number of documents based on vocabulary frequency. These techniques have a limitation in terms of controlling what topics the algorithm should focus on. Additionally, in my case, I do not have a lot of data to train the model on. So I was thinking about generating lists of tokens characteristic to some specific topics and then measure cosine similarity with word2vec between the vocab used in the document with the list of target tokens.

My questions are:
1. is it the right way forward or there are better ways of achieving this?
2. How the final score should be calculated - is an average of similarities between tokens okay? I am afraid that for e.g. if I create 100 target tokens per each topic, the similarities will somehow cancel out yielding similar score.
3. What I like about LDA is that it shows the probability levels per multiple topics. Is there an algorithm similar to LDA, where I could seed the topics rather than merely stipulate the number of clusters?

Thanks for reading this.


texts: How should I encode 'dynamic' features (with multiple instances) along with 'static' features (single instances)?

texts:
Suppose I have to predict if a certain product from an assembly line in a factory will be a scrap. This product has let's say 'static' data like a certain shape. A certain vendor, etc. And, it can have 'dynamic' data this meaning it can have for example: one or more sets of measurements (pressures,temperatures ,etc) from production processes.
How to treat this 'dynamic' features ?
Somehow it doesn't seem right to repeat the 'static' data for all 'dynamic' events. And using the mean of 'dynamic' features would dilute the information.
I'm thinking to encode this 'dynamic' data in a similar way phrases with variable number of words are encoded in fix length vectors with lstm networks. What do you think ?


texts: Best way to represent data as features vectors in Python

texts:
I have a list of events topic retrieved from a tweets collection.
A set of features have been extracted and their values normalized between 0 and 1. An example of an event:

"paris_attack_news-20150107_100842-20150107_112852": {
    "ages": 0.5557594006583049,
    "density": 0.0012022814250710345,
    "followers": 0.1144661871115895,
    "friends": 0.13507755010659472,
    "hashtagCount": 0.033270950301517985,
    "lifespan": 0.29613227044582224,
    "mediaCount": 0.1095890410958904,
    "mentionCount": 0.020275919732441472,
    "objectivity": 0.2850584551023736,
    "polarity": 0.2963684492294102,
    "retweetCount": 0.21431767337807606,
    "status_count": 0.09222093073720204,
    "truth": 1.0,
    "tweetCount": 0.01300578034682081,
    "urlCount": 0.29494007989347537,
    "verified": 0.3392857142857143
}

Now I need to represent each event as an array of its features:

paris_attack_news-20150107_100842-20150107_112852 = [0.5557594006583049, 0.5557594006583049, 0.1144661871115895, 0.13507755010659472, ...]

After that, I need to manipulate/aggregate the array values in some ways to get a specific events sorting based on the results.

The data are already in a Python Pandas DataFrame (event name as index, features as columns).

Which is the best way (data structure for further storing or libraries such as NumPy, sklearn or similar) to build the arrays starting from this?

P.S.: Then I'll need to apply some machine learning alghoritms to detect if an event is TRUE or FALSE, using the feature named "truth": 1 or 0 as label classification.


texts:
It is often pointed out that sample is an overloaded term in statistics and the sciences being supported by statistics.  In my field (geological sciences) as in most other sciences, the process of collecting meaningful data is critical and discussions about the traps and pitfalls in that process talk about sampling.  Not far down the road from that, particularly when lab results are back, conversations involving statisticians, data scientists, geomathematicians, GIS analysts and even normal geologists are likely to attempt to include multiple meanings of sample in the same sentence!

Q: Have any data scientists (or statisticians) found practical ways to communicate these different meanings?

One way is to always add soil, rock, statistical and so on before sample.  But I was curious if there are any other approaches to effective communication that are in use.


texts: I'm developing an OCR service that scans receipts and assigns points to user, based on amount of money spent. But the problem is that user can forge fake receipts and redeem them for extra points. User takes picture of receipt in app, which sends it to server, so it's not possible for user to modify picture after taking it. But user can print modified receipt (for example with changed receipt id) and take a picture of it. Is there a data science approach to solve this problem?


texts: Recommendation System to integrate with an android app

texts:
I need to build a recommendation system that takes certain parameters as input, computes a score and order suggestions to users based on this score. Well this is what I need to do loosely speaking. I am new to the scene of data science and haven't come across anything that could help me out.

This guy asked a similar question 5 years ago. I basically need something like this:
https://stackoverflow.com/questions/6828013/recommendation-systems-for-a-mobile-market-and-algorithm-suggestion

Apologies for the open-ended and vague question.
Just need to be pointed out in the right direction.

Edit: I do not intend to place the system on the android device. It will be running back-end


texts: Similarity Measure between two feature vectors

texts:
I have face identification system with following details:

VGG16 model for feature extraction
512 dimensional feature vector (normalized)

I need to calculate similarity measure between two feature vectors. So far I have tried as difference measure:

Pairwise cosine, euclidean distance
Dot product (both vectors are normalize, so their dot product should be in range [-1, 1])

These methods are working fine when I want find closest feature vector from set of Feature Vectors. By sorting difference(distance) measure. But this method is relative measurement.
I want to have percentage of similarity. So I can say given image is Person XXX with degree of certainty of x percentage
Lets say that I have 2 feature vectors ( of person a and b).

I want to calculate accuracy level (in a percentage) that person a is
indeed person b, by probability of some percentage.


texts: How to deal with count data in random forest

texts:
I am working on a classification model where my target class is a biased class with the class shape as

    0     1
 20694   101

Most of my features are the count of number of times a certain event was triggered. While exploring these features I found that my target variable is only associated with certain values of features. For example as below

                 0         1
Feature V1
0                12014    75
1                6490     16
2                1177     6
3                402      2
4                176      2
5                100
6                84
7                61
8                39
9                23
10               26
11               14

As we can see that 1 only occurs when V1 has value of 0 to 4.Thus for any unseen data my model would always predict 0 whenever V1 has value greater than 4.

I thought of using bestNormalize package, however the transformations it is suggesting looses correlation when applied to the data.

Any suggestion would be of great help.

Thanks a lot in advance!!

P.S. Happy to share the data if required.


texts:
In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been encoded as 0/0.25/0.5/0.75/1 etc). However, the docs say "Each sample’s missing values are imputed using the mean value from n_neighbors nearest neighbors found in the training set." Because of this, I am getting in-between values like 0.4 for nominal attributes. Is there any way to override this to change from mean to mode for nominal columns?
Also, I looked at missingpy and fancyimpute but they both seem to be using mean as well~


texts:
Say I have $n$ multi-class classification problems $p_1$, ..., $p_n$. Each of these have their own training data. While they are all distinct problems, there may be similarities in their data (which are in my case images), e.g. the data for class $p_1^{_B}$ of problem $p_1$ may be similar in some way to the data for class $p_5^{_F}$ of problem $p_5$. Classically, each classifier tries to seperate each of its training classes from each of its other training classes, but doesn't look at the other classifiers and their training classes. If we instead train the classifiers jointly somehow, can the relationship between the classes (from different problems) be made more explicit? For example, maybe any image classified as $p_3^{_C}$ would also be classified as $p_7^{_A}$ and vice versa, or any image classified as $p_2^{_B}$ or $p_2^{_C}$ would be classified as $p_5^{_F}$, with some degree of accuracy?
Possibly related things that I have explored but don't seem to quite fit (simplified to refer to just two problems $p_1$ and $p_2$ but in principle generalizable to $p_n$):

Transfer-learning seems to be concerned with how to use the knowledge gained from solving (say) $p_1$ to better solve (say) $p_2$. Optimizing the accuracy of any of the classifiers is not myr goal.
Multi-task learning (MTL) seems to be concerned with how to use the knowledge gained from solving both $p_1$ and $p_2$ together, to improve the performance on both. Again, optimizing the accuracy of any of the classifiers is not my goal. But, the core intuition of MTL is that it may be beneficial to exploit the hidden relationships between the two (or more) tasks, and this is essentially (it seems to me) the same premise as in my task.
Multi-label learning seems like a reduced form of my task, arising if each of the classification problems $p$ are binary instead of multi-class. What I find interesting is that label graphs seem to have the same general goals as in my task.

My question is: assuming that my task has already been described and studied in the vast machine learning literature, what is the name for it? This would help me find relevant papers and potentially software, so I don't have to reinvent the wheel. Finally, I am a machine learning beginner, so this question could well be fundamentally misguided. If so, please correct.


texts: Best way to remove useless features (non-english words) when there are more than 100,000 features?

texts:
I am in a situation where i have more than 100,000 features, and i need to select the top features to give them to my final neural network model.
So far i have been using RandomForestClassifier in sklearn, and first i use fit and then i use feature_importances to select the top n features. ( I also use StandardScaler and transform to normalize data)
now i have two questions:

am i approaching this right? is this a correct way remove useless features?

is there any better way to select for example top 200 features to give to my final model when there are more than 100,000 features? my task is sentence classification and these features are actually BOW features of non english words, so basically i want to learn the top most important "words" in my corpus that can be used to classify sentences.

my final model is a neural network, but i cannot give all the features to it and let it decide because of performance issues, i need to filter some of the features and then give them to my neural network model. also my final model is written in pytorch, so currently i am using sklearn to select top n features and then use pytorch to train the final mode, so if there is an easier approach for this please let me know.


texts: Error Analysis for misclassification of text documents

texts:
I am working on a text classification work. The purpose of this work is to classify whether a particular document belong to class A or Class B.

I used KNN algorithm and i am able to get some decent results. However I want to know two things.

Why a particular document has been classified as Class A or Class B? What keyword or information that made a document to be classified as such?
How to perform mis-classification analysis?

Kindly help.


texts: Bias Formula in Machine Learning expanded using ground truth

texts:
Why is Bias calculated for $f(x)$? Shouldn't it be calculated for $Y$ (which is $f(x)$ + Noise $\epsilon$)?

We are fitting our model to $Y$, So shouldn't we be calculating bias wrt to $Y$?

Also, I tried to calculate bias for different polynomial fitting with degree $d=3,9,20$ using $f(x)$. I got the same bias. I got expected bias values when I calculated using $Y$.

But my calculation doesn't match with the actual formula. Basically my doubt is: Why is Bias formula $E[\hat{f}(x_0) - f(x_0)]$ and not $E[\hat{f}(x_0) - Y]$?

Is it because noise is accounted for separately using variance (Irreducible error)? If suppose I haven't separately accounted for noise, then I can calculate bias using $Y$?

Bias should be calculated wrt to the Ground Truth right? According to my understanding ground truth is $Y$ which is $f(x) + \epsilon$. Please correct me if am wrong.


texts: multi-state lstm for time series forecasting

texts:
I have 100's of distinct users and the dataset consists work/hours spent by each user over the past 1 year. Total records are over 40k. The inputs for the LSTM RNN has the categorical feature vectors corresponding to these users.The rnn should learn behavior of each user and should be able to predict the next hour spent in the per user time-series based on the past information of the same user. How to maintain separate hidden states for each user within an LSTM RNN.
I'm refering to below blog post is similar to my problem:
https://towardsdatascience.com/multi-state-lstms-for-categorical-features-66cc974df1dc
Sample data(again referring to the blog) enter image description here

My Queries:
How multi-state lstm's can be implemented using kera?
Is it a custom lstm rnn with stateful nature?
how do we keep separate state for each user with categorical features?
Thanks in advance!


texts: Metric to use to choose between different models - Hyperparameters tuning

texts:
I'm building a Feedforward Neural Network with Pytorch and doing hyperparameters tuning using Ray Tune. I have train, validation and test set, train and validation used during the training procedure.
I have different versions of the model (different learning rates and numbers of neurons in the hidden layers) and I have to choose the best model. But I'm unsure on which metric should I use to choose the best model. Basically I don't know the XXXX in this line of code:
analysis.get_best_config(metric='XXXX', mode='min')

Should it be test loss or validation loss?


texts: Manipulate output of 1 keras model and use that as input to second model

texts:
I am trying to merge two Keras Sequential model. The aim is to get output from 1 model, run the output through a function and then use that as an input to another model. I don't even know whether such model is trainable or not.
The two models are given below
def feature_extractor():
  model = tf.keras.Sequential()
  model.add(tf.keras.layers.Conv2D(16, (3, 3), (2, 2), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(16, (3, 3), (1, 1), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(32, (3, 3), (2, 2), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(32, (3, 3), (1, 1), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(64, (3, 3), (2, 2), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(64, (3, 3), (1, 1), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(128, (3, 3), (2, 2), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  model.add(tf.keras.layers.Conv2D(128, (3, 3), (1, 1), 'same'))
  model.add(tf.keras.layers.LeakyReLU())
  return model

def layer_flow_estimator():
  model = tf.keras.Sequential()
  model.add(tf.keras.layers.Conv2D(64, (3, 3), (1, 1), 'valid'))
  model.add(tf.keras.layers.LeakyReLU(0.2))
  model.add(tf.keras.layers.Conv2D(64, (3, 3), (1, 1), 'valid'))
  model.add(tf.keras.layers.LeakyReLU(0.2))
  model.add(tf.keras.layers.Conv2D(32, (3, 3), (1, 1), 'valid'))
  model.add(tf.keras.layers.LeakyReLU(0.2))
  model.add(tf.keras.layers.Conv2D(32, (3, 3), (1, 1), 'valid'))
  model.add(tf.keras.layers.LeakyReLU(0.2))
  model.add(tf.keras.layers.Flatten())
  model.add(tf.keras.layers.Dense(final_width*final_height*final_channels))
  model.add(tf.keras.layers.Reshape((final_width, final_height, final_channels)))
  return model

the function is as such
def cal_cost(features):
  cost = tf.zeros([6, 32, 56, 128])
  for i in range(6):
    f_width, f_height, f_channels = tf.squeeze(features[i]).shape
    flat_feature1 = tf.reshape(features[i], [f_width*f_height*f_channels])
    flat_feature2 = tf.reshape(features[i+1], [f_width*f_height*f_channels])
    cost_vol = tfp.stats.correlation(flat_feature1, flat_feature2)   #normalize here ############################
    cost[i] = tf.reshape(cost_vol, [f_width, f_height, f_channels])
    print(i)
  inp = tf.zeros([6, 96, 56, 128])
  for i in range(6):
    inp[i] = tf.concat([features[i], features[i+1], cost[i]], axis=0)
  return inp

The flow should be as such - input -> feature_extractor -> cal_cost -> layer_flow_estimator -> output. Please help


texts: Strange out of memory while loading lots of pictures before input CNN for deep learning

texts:
I’m using the TFlearn, and want to classify pictures to two category. But the strange out of memory while loading lots of pictures before input CNN for deep learning. The RAM is 64 G in my deep learning box. I using 482426 pictures(resize to 224 x 224) for training (241213 are A category, the other 241213 are B category ). The process shows the '%MEM: 98.4' whenever the I load these images, so that I can't complete the process. I would like to know how can I edit my code for this situation?

imgs = []

for filename in glob.glob(Learning_Data_Path+"/Training/A/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img)
    imgs.append(img_arr)

for filename in glob.glob(Learning_Data_Path+"/Training/B/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img)
    imgs.append(img_arr)

imgs = np.array(imgs)

y_data = np.r_[np.c_[np.ones(Training_A_num), np.zeros(Training_A_num)],np.c_[np.zeros(Training_B_num), np.ones(Training_B_num)]


texts:
I have created a Tf.Sequential model to when given an input of a number, get either a 1 if that number is bigger than 5 or 0 if it is not.

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [5], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase, or the outputs are too high...

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input with a bias that would point towards 5 being the "decisive" number. I thought that Tensorflow.js automatically assigns biases to weights, so I simply wrote this code:

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

Usually, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and why is what I did incorrect?


texts: How to convert a pytorch geometric graph to Networkx Multigraph?

texts:
I have a weighted graph stored in Data object, and I want to convert this graph to Networkx MultiGraph. Here is what I have tried:
data = Data(x=x, edge_index=edge_index)
print(edge_index).

.

torch.Size([3, 237])

conversion
first trial
from torch_geometric.utils import to_networkx, from_networkx
G=to_networkx(data, to_undirected=True)

ValueError: too many values to unpack (expected 2)

second trial
G = nx.MultiGraph()
edges = data.edge_index.numpy()
edge_list = []
for i in range(data.num_edges):
    edge_list.append(tuple(edges[:, i]))
G.add_edges_from(edge_list)
data.G = G
adj=nx.adjacency_matrix(G)
print('here the adj is', adj.shape)

Result=(18, 18) expected (23,23)

What else can I do? Thank you very much for your Input


texts: Signal classification with convolution neural network

texts:
I want to classify 8 noisy signals with trainNetwork() function that is available in Matlab.  All Matlab examples were for image classification. The arguments are the data and its label, the CNN layers, in which the first layer is imageInputLayer([32 32 3]).
My question is can this function be used to classify 1-D signals? if yes, how can we create such a layer?


texts: How to compare 2 datasets with machine learning

texts:
I have multiple datasets from different sources (multiple cities) but with the same parameters and one output that I want to classify. Each dataset has different number of datapoints (thousands). I am trying machine learning models to figure out which dataset has features of higher predictability of the output classification.
Should I make all the datasets the same size before training models and compare the performance metrics or just use all points even thought the sizes are different?
eg: Both dataset1 and dataset2 have the same columns A, B, C and Output but with different number of rows, lets say n1 and n2 number of rows respectively.


texts: When should I reverse normalizations to evaluate loss?

texts:
If I am training a neural network and have normalized the data before-hand, should I reverse the normalization to calculate the loss?

This tutorial provides an example of this method.

What if I'm using batch normalization? From this post it looks like there are ways to reverse it but my question is more along the lines of should I bother.

My first thought was that it would depend on the problem but I could not think of a problem where I would not want to minimize the error in the actual units of the problem


texts: Is it possible to create a predictive model for a dataset that consists of only positive occurrences of the dependent variable?

texts:
Lets say I want to predict earthquakes.
My dataset would only contain data about earthquake occurrences and no data about non-earthquake occurrences as that would basically be any other period of time which is not kept in the dataset.
In that case, I assume a decision tree or logistic regression would not work as we don't have a dichotomous dependent variable (as the occurrence only gets into the dataset if an earthquake occurred).
Are there any models which would suit this situation, or is a different approach needed?


texts:
I recently learned about Goodhart's Law. Simply put,

  When a measure becomes a target, it ceases to be a good measure.

However, in Data Science, we really do aim at improving our performance by increasing or decreasing a metric, and improve our models based on that. For instance, in Kaggle competitions. Is Goodhart's Law applicable to Data Science? Why or why not?


texts: Nan loss for one Image dataset but finite fraction loss for another dataset?

texts: I am training neural network using chainer library on MNIST dataset and one other dataset. As MNIST dataset is greyscale, hence I converted the other colored dataset to greyscale using cv2 library. I use the same architecture on MNIST, I get an test and train loss as finite digits between 1.5 - 2. But when  use this architecture to other converted image dataset, I get test and train loss as nan starting from the first epoch. What could be the problem, where should I start looking?


texts:
I'm looking on a given solution of the first assignment of cs231n course.

Down below a snippet from the loss function.
I don't really understand lines 140-143. Can you explain why dscores (the derivative of scores) is calculated like that?


texts: Reducing the effect of down voters with rating system

texts:
I  have a site in which users rate things in a 1-5 star system. Once an item reaches the top of the charts, some users tend to start rating it 1 star even though it got a majority of 4-5 stars to get where it's at. It's not rampant, I would say 10-20% of the new votes are 1's. Clearly they are trying to manipulate the rating system, and I want to prevent that.

The current way I am doing that is by having a "reasonable window" of what I consider to be a legitimate vote.

For items with less than 10 votes;
I currently do nothing and take the mean as it's rating.

Once an item starts getting more than 10 votes I tie them to a window of their mean. This window is defined as

Window = 4.5 - Log(TotalVotes, 10);

So a reasonable vote range is then (Mean - Window) thru (Mean + Window)

Once the reasonable vote range is found, the "Rating" is just the mean of all the reasonable votes (those who fall in the reasonable range).

This means an item with a real mean of 4.2 with 100 votes would have a window of 4.5-Log(100,10) = 2.5, so if that item gets a 1 star vote, it will be ignored in the rating. However, the 1 star will still effect the underlying mean.

This has worked good in general but the issue is when an item's Mean - Window is just at the brink of 1.0, as soon as it dips below 1.0 every 1 star vote now is included in the rating and the rating drops significantly even the difference before and after may have just been one more 1 star rating.

I need a better system/way of accomplishing to filter out these 1 star ratings, and not just them but handle the situation in where someone may get their friends to upvote an item 10 votes and all 5 stars, where its true rating may be more 3 star.

Looking for any recommendations of how to handle user-driven rating systems and normalizing outlier votes.


texts: I have a large sparse data matrix (bag of words, over large number of entries). I can easily treat it as a sparse matrix in sklearn models such as RandomForest. But, if I want to use Catboost, I need to turn it into a dense matrix. I was wondering if there is any efficient method to work with Catboost that doesn't cause this? For example, any internal built-in feature such as TFRecords of Tensorflow, to load bacthes.


texts: I am designing a classifier that takes as input features matrices of different dimensions, for example (Nx5, Nx10, Nx100, Nx1000) using visual bags of words of distinct dictionary sizes and methods (SIFT, SURF, ORB). Right now I am training one classifier for each combination of dictionary size and method to extract particular feature descriptions, but the task of fine-tuning all these classifiers is a bit tedious. Is there any method or process to combine such feature matrices into a unique classifier?


texts: Transformer architecture not working on toy problem

texts:
My transformer is not working on a toy problem.

Toy problem
Input : Sequence of random integer, one-hot-encoded. Example :
[[0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
 [0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
 [0, 0, 0, 0, 0, 0, 0, 0, 1, 0],
 [0, 0, 0, 0, 0, 0, 1, 0, 0, 0],
 [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
]

Output : 1 if the first random integer is < 5, 0 otherwise. For the given example, the output would be 1.

Problem
Running my Transformer on this toy problem works perfectly. However, running with a bigger size (one-hot encoding on 800 dimension) does not work anymore : the network always output similar results, no matter the input.
Note : I did change the condition of the output to be x < 800, in order to have class balance.
Why my transformer architecture is not working with this variation of the toy problem ?

Another problem
I also tried to use 2 inputs (similarly generated, with size = 10) and generate the output based on these 2 inputs (1 if the first random integer of both input is < 5 or both > 5, 0 otherwise)


texts: Shall I use ordinal encoding or One-Hot-Encoding when using DBSCAN for content clustering on websites?

texts:
I want to cluster the preparation steps on cooking recipes websites in one cluster so I can distinguish them from the rest of the website.
To achieve this I extracted for each text node of the website the DOM path (e.g. body->div->div->table->tr ....) and did a One-Hot-Encoding before I executed the DBSCAN clustering algorithm.
My hope was, that the DBSCAN algorithm recognizes also not only 100% identical DOM-paths as 1 common cluster, because sometimes one preparation step is e.g. in a   tag and the others are not. But even though I tried a lot to vary epsilon and MinPoints paramters, it does not recognice then all as one cluster.
My question:
Is One-Hot-encoding maybe the wrong way, because a dom path is not really 100% categorical but maybe a kind of ordinal? Because the more common DOM path elements two DOM-paths have the more likely is, that they are building one common cluster.


texts:
i'm trying to implement my own LSTM network. I implemented back propagation algorithm, but it doesn't pass gradient check. Can't realize where is the mistake. Please help

Here is the code problem piece of code:

def backward_propagation(self, x, y, cache):
    # T - the length of the sequence
    T = len(y)
    # perform forward propagation
    cache = self.forward_propagation(x)

    #...

    # delta for output layer
    dy = cache['y'].copy()
    dy[np.arange(len(y)), y] -= 1. # softmax loss gradient

    for t in reversed(range(T)):
        dV += np.outer(dy[t], h[t].T)
        dh[t] += self.V.T.dot(dy[t])
        dhtmp = self.V.T.dot(dy[t])
        dctmp = dct[0] * (1.0 - ct[0]**2)

        for bptt_step in np.arange(t)[::-1]:
            # add to gradients at each previous step
            do[bptt_step] = dhtmp * ct[bptt_step]
            dct[bptt_step] = dhtmp * o[bptt_step]

            dctmp += dct[bptt_step] * (1.0 - ct[bptt_step]**2)

            di[bptt_step] = dctmp * g[bptt_step]
            df[bptt_step] = dctmp * c[bptt_step-1]
            dg[bptt_step] = dctmp * i[bptt_step]

            # backprop activation functions
            diga[bptt_step] = di[bptt_step] * i[bptt_step] * (1.0 - i[bptt_step])
            dfga[bptt_step] = df[bptt_step] * f[bptt_step] * (1.0 - f[bptt_step])
            doga[bptt_step] = do[bptt_step] * o[bptt_step] * (1.0 - o[bptt_step])
            dgga[bptt_step] = dg[bptt_step] * (1.0 - g[bptt_step] ** 2)

            # backprop matrix multiply
            dWi += np.dot(diga[bptt_step].T, h[bptt_step-1])
            dWf += np.dot(dfga[bptt_step].T, h[bptt_step-1])
            dWo += np.dot(doga[bptt_step].T, h[bptt_step-1])
            dWg += np.dot(dgga[bptt_step].T, h[bptt_step-1])

            dUi[:, x[bptt_step]] += diga[bptt_step]
            dUf[:, x[bptt_step]] += dfga[bptt_step]
            dUo[:, x[bptt_step]] += doga[bptt_step]
            dUg[:, x[bptt_step]] += dgga[bptt_step]

            # update deltas for next step
            dhtmp = np.dot(self.Wi.T, diga[bptt_step])
            dhtmp += np.dot(self.Wf.T, dfga[bptt_step])
            dhtmp += np.dot(self.Wo.T, doga[bptt_step])
            dhtmp += np.dot(self.Wg.T, dgga[bptt_step])
            dctmp += dctmp * f[bptt_step]

    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]

Here 'ga' suffix is for 'gate activation' - designation of gate input before non-linear activation function is applied.


texts: Confusion matrix doesn't display properly

texts:
I am trying to plot a confusion matrix using the Logistic Regression for a multi-class dataset.

But the problem is when I plot the confusion matrix it only plot a confusion matrix for binary classification.

Here is where I am plotting it.

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

dataframe = pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

from sklearn.linear_model import LogisticRegression

LRModel = LogisticRegression(C=100, max_iter=5500)

LRModel.fit(X_train, y_train)

predicted_values_ = LRModel.predict(X_test)
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, predicted_values_)

misclassified = (y_test != predicted_values_).sum()
misclassified

import seaborn as sn

# plt.figure()
sn.heatmap(cm, annot=True)
plt.xlabel("Predicted")
plt.ylabel("Actual")

And I get this matrix as shown below.

Can someone tell me where I am doing wrong?

This is where I am using Logistic Regression for multi-class scenario


texts: How to create column for my csv file in python

texts:
i have csv file having these values (without column):-

I:30n
J:0n
J:0n
U:1000n
C:0n
I:12n
I:10n
I:10n
I:10n
I:10n

I want to add column name for these rows values.

  Suppose all I (i.e: I:30n, I:12n, etc)value record should be in
  one column and likewise all J (i.e J:0n, J:0n) should be in one
  column and vise-versa using python.

Can anyone help?


texts:
I am new to data science. I have a dataset of around 200,000 records, having 5 columns. There is a field called, class. For each class, there are one or many divisions. I have to do this:
1. Filter the dataset, such that only those classes with at least 5 divisions turn up.

For each division, I have to calculate attendance from another column.
There is a minimum attendance value for each class. I have to find the percentage of divisions in each class with the minimum attendance.

I started with importing the data in python using Pandas and started writing loops for processing this. But I am sure this is not the right way to do. Can you please give some idea.Can I do this in Excel pivot table?


texts: How to correctly perform data sampling for train/test split in multi-label dataset?

texts:
Problem statement

I have a text multi-label classification dataset, and I've found a problem with the dataset sampling.

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

The preprocessing step simply consists in transforming the labels into OneHot representation and keep only the N most frequent ones. I expect similar (the same) behaviour, but I'm getting really weird results. Let's take a closer look.

Train+Test all together and then split

Start with:

|ID   |TEXT  |LABELS|
|-----|------|------|
|1.txt|the   |A:B   |
|2.txt|lazy  |B     |
|3.txt|fox   |C     |
|4.txt|jumps |B:C   |
|5.txt|over  |C:D   |
|6.txt|crazy |D     |

After preprocessing and split:
Train

|ID   |TEXT|A|B|C|D|
|-----|----|-|-|-|-|
|1.txt|the |1|1|0|0|
|2.txt|lazy|0|1|0|0|
|3.txt|fox |0|0|1|0|

Test

|ID   |TEXT |A|B|C|D|
|-----|-----|-|-|-|-|
|4.txt|jumps|0|1|1|0|
|5.txt|over |0|0|1|1|
|6.txt|crazy|0|0|0|1|

The results are good. Let's take this as reference. F1-Score = 0.61.

Pre-made Train/Test split

Start with:

Train

|ID   |TEXT|LABELS|
|-----|----|------|
|1.txt|the |A:B   |
|2.txt|lazy|B     |
|3.txt|fox |C     |

Test

|ID   |TEXT |LABELS|
|-----|-----|------|
|4.txt|jumps|B:C   |
|5.txt|over |C:D   |
|6.txt|crazy|D     |

After preprocessing and split:

Train

|ID   |TEXT|A|B|C|
|-----|----|-|-|-|
|1.txt|the |1|1|0|
|2.txt|lazy|0|1|0|
|3.txt|fox |0|0|1|

Test

|ID   |TEXT |B|C|D|
|-----|-----|-|-|-|
|4.txt|jumps|1|1|0|
|5.txt|over |0|1|1|
|6.txt|crazy|0|0|1|

The results are totally degradated. F1-Score = 0.15.

What is going on? What could be causing the divergence in results?

Extra information

The labels predicted at the prediction step are not compatible with the labels on the test set. I've taken that into account and is correctly managed, that's not the problem.
The splits are exactly the same. The documents in train/test are the same in both situations.


texts: How do I decide if I need to go for Normalization and not Standardization or vice-versa?

texts: While designing a ML model, how do I decide if I need to go for Normalization and not Standardization or vice-versa? On what factor is this decision made?


texts: Multi-class classification metrics in R and Python

texts: I would like to ask if there is a metric in R and Python that serves not only for binary classification. I've found Matthews Correlation Coefficient works well in Python, but in R is only a binary version. Of course, I'm talking about "real" not just binary metrics - not something like "It's binary, but you can use one vs one or one vs all."


texts: How do I know how to construct the layers of my CNN

texts:
I've done a CNN project with Keras and OpenCV, and I've got roughly 65% accuracy. And now I have to present this work in my University, but I'm afraid if the teachers ask me for how do I knew how to construct the right layers to my CNN.

In fact in my development I took a look in others projects to see how the other people do, and I know that you have to have a number of neurons equivalent to the number and size of your image. But I know that has a mathematic behind this.

So how could I tell if they ask me it?


texts: Can one perform Feature Selection on a subset of training data?

texts:
I have a training data set with almost one million rows and I am considering eight features initially. My machine learning model will be Random Forest regressor.
In Section 3.4.7 of "Feature Engineering and Selection: A Practical Approach for Predictive Models" there is this:

Finally, with large amounts of data, alternative data usage schemes might be a good idea. Instead of a simple training/test split, multiple splits can be created for specific purposes. For example, a specific split of the data can be used to determine the relevant predictors for the model prior to model building using the training set. This would reduce the need to include the expensive feature selection steps inside of resampling.

Based on this and for finding relevant features, I want to do an exhaustive search over all possible subsets of initial set of features on 10% of my training data (~100k rows). The evaluation of each subset will be done by 5fold cross-validating random forest with the same hyperparameters for all of the subsets.
Why 10 %? ans: to reduce computation time and because I think 100k rows would give me a reliable set of features. I have no reference to support my choices. Please let me know if you have one.
After finding THE best subset, I go on and do hyperparameter tuning using 100% of my training data.
Am I doing something wrong? Could you suggest me some references to look for vis-a-vis the parts that I am doing wrong?
I searched about this "specific split" that Max Kuhn talks about, couldn't find any similar suggestion.
More details:
Why do I want to do feature selection if I only have eight features, especially considering that RF is somewhat insensitive to redundant/irrelevant features (with respect to predictive performance, not feature importance/selection)?
Apart from the general idea of decreasing the computational time by using less features, the features are meteorological variables that may not be available/accessible easily and everywhere. So if I am to build a model that can be used in other case studies and research, it would make it easier to use if it has less variables.
Using RF for feature selection inside another method is kind of more useful I think because it can also consider the interaction effects implicitly.
I want to do all these things in an academic work, that's why I need to justify all the steps and decisions that I make. Here I am mostly concerned about the fact that I am using only 100k of my rows for feature selection. One might say, why 100k? why not 200k? and etc.
Another doubt of mine is: since I do a supervised feature selection, I should do the tuning on the remaining 90% of training data, not on 100%. Do you agree?


texts: Which loss function is the best loss function when using XGB regression with highly skewed dataset?

texts:
Which loss function is the best loss function when using XGB regression with a highly skewed dataset?

The skewness of the data is very high. I used XGBoost with objective function of linear regression (but the data was transformed into the log space). It performed better than using gamma objective function. Any other suggestions?


texts: How to split data into training, validation, test data sets if the data is non-stationary?

texts:
When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-stationary. For example, financial time-series data are non-stationary.
So, for this kind of non-stationary data, how do you split data into training, validation, test data sets?


texts: Is it ok to concat train and test sets to for fit_transform before splitting them?

texts:
I'm currently trying to use fancyimpute KNN to impute some data separate train/test sets. However, I'm running into the problem of not being able to do fit_transform on train data then using only transform on the test set because it appears that:

  transform not implemented! This imputation algorithm likely doesn't
  support inductive mode. Only fit_transform is supported at this time.

Is it alright to concat the two sets for imputing before splitting them after? Or is my approach even correct by calling only fit on the train set before applying that to impute the test set? Should I be calling separate fit_transform on both imputation?

Thanks!


texts: What ML techniques can be used to determine context across sentences?

texts:
What ML techniques can be used to determine relevance or context between sentences?

For example:

"I am going to the event at 9pm. Can you meet me there?"

Here, there is context between the two sentences - the person is indicating they will be at the event at 9pm, and is asking the recipient to meet them there.

Conversely:

"I am going to the event at 9pm. Can you meet me at school tomorrow?"

Based on this text alone, it appears as if the sentences represent two disjoint thoughts.

Additionally, although the examples above are of adjacent sentences, the context may exist across multiple sentences, or sentences that are not adjacent:

"I am going to the event at 9pm. I've heard it is fabulous. Can you meet me there?"


texts: Can neural networks be adapted without recreating them completely?

texts:
If I have, for example, a classification network which can tell if there is a dog or a cat in a picture, is it possible to adapt the network so it can also learn to detect a mouse? Without making a new one from scratch.
In this case it doesn't make sense, but I'm wondering if for example Netflix has to retrain its network completely with every new show they add. And if so, when do they do that? The first few days may be the most crucial ones, but also the ones with the least data to train a network from.

The actual problem I do have is that I'm trying to train a network to predict the location of public transport vehicles. That by itself is hard enough, but what do I do if there is a new transport line, which I don't have an input neuron for. - Here I would need to add an input neuron.

Or another idea I'm thinking about is a neural network which can help me order my documents by detecting the company and date. I would like to classify an unknown document layout, a few times, so the neural network is able to classify it by itself, but without losing everything it learned so far. - Here I would need to add an output neuron.

It seems like there is a way to do this, since there are some machine learning algorithms out there which seem to pull off stuff like that. Or do they somehow work around this? If so, how?


texts: How is the output of a maxpool layer window size=1x2 and stride=2 calculated?

texts:
I'm looking at the architecture proposed in the following paper: Baoguang Shi et al, An End-to-End Trainable Neural Network for Image-based Sequence
Recognition and Its Application to Scene Text Recognition.

In the proposed architecture of the model, a MaxPooling Window:1 × 2, s:2 layer is mentioned. I'm not sure what the size of the output of this layer would be.

If i have an input of size (32 x 8), then the output would be:

(32-1)/2 + 1 = 16.5, <- this part doesn't make sense to me

(8-2)/2 + 1 = 4

*ignoring depth and batch size here


texts: Which metric to use for evaluating a rating system

texts:
I have a system which gives a star rating of the quality of work(on scale of 1-5, 1 being extremely poor and 5 being exceptionally good).
An expert labelled a test set with their ratings of quality of work.
The data is imbalanced in the 5 classes(1,2,3,4,5), very few samples for Extremely poor and Exceptionally good ratings.
Is there a metric which can take into account consideration that if system rated extremely poor quality work as very good or exceptionally good, it should be penalised more, as compared to when it rated an extremely poor rating as poor or average? And again due to imbalanced class distribution, straightforward accuracy is not the best measure.
Any suggestions for metric?


texts:
Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular PCA?

The idea is that (hopefully) we could have new features that would improve PCA in a way that wouldn't require retraining every time we add new features. If this isn't possible as explained here, how to achieve this in another way?

We add new features regularly by adding new datasources, or add some new features through feature engineering. Downflow models rely on PCA of that feature space. By training models on PCA and not on features directly simplifies workflows, and also removes burden of retraining models if it was possible to have a PCA (or another feature extraction process) that maps original features to this "stable" subspace.


texts: Cost sensitive classification with individual cost

texts:
I'm currently sitting on a problem, where i'm uncertain if there is not a much simpler solution.

I'm trying to train a DNN with a dataset for a classification task that should be cost sensitive. Classic literature on this kind of task use a cost weight that is constant for any kind of misclassification. My problem needs to use one of the dimensions of the input as the cost of misclassification for that single classification.

My solution would be to use TensorFlow and add the parameter i later need to a collection and then write a custom loss function where i grab the values from the collection for my cost sensitive loss.

So my question would be, does anybody know of any simpler solution, open source implementation etc.?


texts: For a linear model without intercept, why does the redundent term in one-hot encoding function as intercept?

texts:
In this question Elias Strehle pointed out that if we keep all the levels during one hot encoding on a linear model without an intercept, the redundant feature will function as an intercept. Why is this the case?
Isn't that in a linear model, the intercept term ($x_0$, not $\beta_0$) will always have a value of 1? Suppose the feature to encode is gender, then some of the rows of the redundant term will be 0 while others will be 1s.


texts: Selecting Transforms with sklearn pipelines

texts:
So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do :
# I missed the imports just for brevity
pipeline = make_pipeline(StandardScaler(), PolynomialFeatures(), PCA(), LogisticRegression())

So what I want to do, I want to try both StandardScaler, MinMaxScaler and no scaler when I am using other methods such as XGBOOST. Then I want to try to use polynomial features and then use a PCA, only if I used polynomial features. And finally, I want to try Logistic Regression and XGBOOST with GridSearch (This last part I know how to do).
Anybody did something similar?
Thank you!


texts:
i'd like to share with you my path in a clustering exercise (via K-means using python), in order to understand if i made some errors or if there is something  more that can i do.

General Overview

My dataset is composed by a customer base and their digital/non digital interactions (web visits, opened/clicked mail, number of live visits and so on). my goal is to find a way of cluster them in order to understand if there is a cluster let's say more digital (that interact more with digital world like mail, web ecc..)

1) First step

I Applied Elbow method and i getting the optimal number of cluster (i think between 2 and 3) this is the code and the result:

Inertia, or within-cluster sum-of-squares criterion:

x = df[columns_2]
y = df
wcss = []
for i in range(1, 10):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(x)
    wcss.append(kmeans.inertia_)
plt.plot(range(1, 10), wcss)
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()
plot.get_figure().savefig( "Elbow Method.png")

2) Second step

I scaled my 13 metrics (all columns are numeric but not continuos, also they have a different range) and the i ran the k-means, here's the code:

scaler = StandardScaler().fit(x)
std_x = scaler.transform(x)

After a indication of my professor, i fixed n_cluster = 3 instead of 2 and ran the k-means algorithm

kmeans = KMeans(n_clusters=3 ,  init='k-means++', max_iter=300, n_init=10, random_state=0)
kmeans.fit(std_x)

3) Third step

After loooking at a summary clustered data i saw some resonable results (like in 2 of 3 cluster there's seems to be a good attitude with digital interactions, the first cluster is more close to website engagements while the second is more close to the email engagements. The third cluster shows a low engagement level with all digital channels) i decide to run the silouhette score in order to understand how good is this cluster, here's the code:

silouhette = metrics.silhouette_score(data, labels, metric='euclidean')

As you can see i decided to calculate it in all my dataset (so i didn't specified any sample_size parameter) and the result semms to be quite good 0.64

My questions:

1 Do you see any error in what i did? (i mean method and code errors)

2 What do you think about the final result i'm getting?

3 Is there something more that i can do? (in order to check if i'm doing thing in the right way)

Thank you


texts: how to interpret a hierarchical clustering in the heatmap in the picture bellow?

texts:
I am trying to interpret the heatmap which was created based on a agglomerative hierarchical clustering. I am not sure what exactly the heatmap does, having in mind that I see on left hand side clustering done on symptoms and on the top of the heatmap I get a clustering of the bellow labels.
Can someone please help? Here is the picture
Also, this is the code I have used to generate the heatmap:
library(pheatmap)
pheatmap(t(dt_wd[-1]), cluster_rows = TRUE, cluster_cols = TRUE, clustering_distance_cols = "euclidean",
         clustering_distance_rows = "euclidean", clustering_method = "ward",  cutree_cols = 4,
         main = "Cluster of comorbidities based on Covid-19 Symptoms")


texts: Do Data Miners realise Machine Learning cannot establish Causality?

texts:
Do people in Machine Learning (or, more generally, in Data Mining) realise that no causal link can be inferred from a correlation?

My impression of the ML/AI community is that most people involved have a Computer Science background. I think it uncontroversial to point out that this is not a Science in the academic sense (think Physics, Chemistry, Psychology). Therefore, most of these people have no formal training in the Scientific Method, in Statistics or even in Mathematics.

"Correlation does not equal causation" is lesson nr. 1 in Statistics. In order to establish causal links, experiments are required (under some very special circumstances, other techniques are available). To me, this seems a fundamental and disastrous limitation to Machine Learning. ML will do fine with describing data (and classifying it), but it will never substitute an experiment. The application of ML is, therefore, limited in scope.

My question is about the atmosphere within the ML community.

Do people within the field recognise this limitation? Or do people think the causal-inference problem is not fundamental and will be solved in the future by better CPU's and better algorithms?
Do they restrict the use of ML to descriptive analyses? Or do they, erroneously, assume ML can help us understand and influence the world?
Is there a lively debate on this topic within the community? And are there articles/blogs related to these issues?


texts: How can you build a model based on non-independent imbalance data?

texts:
I am trying to predict customer churn based on the data that I have. I am defining churn as an activity that is not followed by another activity within a week. The customer might come back in two months and become active again and those activities are not considered churn.

Therefore, the same user might be considered churned or not churned depending on future behavior. Also, only 0.5% of these observations result in churn.

Now for this data set, I have both dependant and imbalance data. How can I perform modeling, or even simple t-test (as its assumption is independence)?
Any idea or direction is highly appreciated.