texts: What is the difference between (objective / error / criterion / cost / loss) function in the context of neural networks?

texts:
The title says it all: I have seen three terms for functions so far, that seem to be the same / similar:

error function
criterion function
cost function
objective function
loss function

I was working on classification problems

$$E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$$

where $W$ are the weights, $E$ is the evaluation set, $t_x$ is the desired output (the class) of $x$ and $o(x)$ is the given output. This function
seems to be commonly called "error function".

But while reading about this topic, I've also seen the terms "criterion function" and "objective function". Do they all mean  the same for neural nets?

Geoffrey Hinton called cross-entropy for softmax-neurons and $E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$ a cost function.


texts: Orange wont allow me to set target for corpus

texts:
I would like to do a linear regression on an excel file. I have descriptions in one column and category in the other.
I need to be able to set the target to do a linear regression. However, Orange does not allow me to set target variable the way it would when I use the file module. The problem with opening the data as file is that I will not be able to use bag of words (since thats only for corpus).
When I do use Corpus, it automatically detects all columns as meta.
Am I doing something wrong? Is there a work around?


texts: Using flow_from_dataframe with multiple inputs

texts:
I was wondering if it is possible to use the flow_from_dataframe on Keras with multiple inputs. I have a Dataframe with 3 features – TIF image names, and 10 labels – values, and I want to create a CNN with regression. I’ve read the documentation and I created my test and validation generators, with y_col as my multiple label columns. However, I do not understand how I can use multiple x columns.
I tried to pass a list of columns e.g.:
x_col = [feature1 , feature2 , feature3]
y_col = [label1, label2,…,label10]

but I have an error: All values in column x_col must be strings error.
I’d really appreciate it if someone could help me to understand how to accomplish that. Thanks in advance.


texts: Several independent variables based on the same underlying data

texts:
I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle such cases?

Since, logically, they will be highly correlated, it would make sense to only use one of them, preferably the one which conveys more information. But:

Is this the correct approach, or do we actually lose a valuable
information by not including it?
If including it is the correct approach, is there anything else needed to be done and/or checked to prevent messing up the model
(since they will be highly correlated)?

Example 1:

Let's say we have a feature which can be pair of any number from 1 to 3, e.g. (1,1), (3,2), (2,1), etc.
And we also have another feature which tells us how many ones (i.e., 1) are in the previous feature, so for the previous cases this would correspond to 2, 0, 1, etc.
Although this second feature does not provide us with any new information not already present in the first feature per se (i.e. can be deduced from the first feature), it does have some special meaning, i.e. lets say that the number of ones is expected to influence the results (dependent variable).

Example 2:

One variable is a discrete/integer value, and the other one is 0 if the value of the first feature is below some specific value, and 1 if higher or the same.
Just as in the Example 1, the second feature has some special meaning.


texts: How to do postal addresses fuzzy matching?

texts:
I would like to know how to match postal addresses when their format differ or when one of them is mispelled.

So far I've found different solutions but I think that they are quite old and not very efficient. I'm sure some better methods exist, so if you have references for me to read, I'm sure it is a subject that may interest several persons.

The solution I found (examples are in R) :

Levenshtein distance, which equals the number of characters you have to insert, delete or change to transform one word into another.

agrep("acusait", c("accusait", "abusait"), max = 2, value = TRUE)
## [1] "accusait" "abusait"
The comparison of phonemes

library(RecordLinkage)
soundex(x<-c('accusait','acusait','abusait'))
## [1] "A223" "A223" "A123"
The use of a spelling corrector (eventually a bayesian one like Peter Norvig's), but not very efficient on address I guess.
I thought about using the suggestions of Google suggest, but likewise, it is not very efficient on personal postal addresses.
You can imagine using a machine learning supervised approach but you need to have stored the mispelled requests of users to do so which is not an option for me.


texts: What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning?

texts:
I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples.

Determine it as an abnormal sample if it doesn’t match the pattern of normal samples.

I train AE without labels but we need to use ‘label’ to determine which sample is normal or abnormal.

I am wondering what kind of this training is supervised learning,semi-supervised learning or unsupervised learning?


texts: What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV?

texts:
The constructor of sklearn.linear_model.ElasticNetCV takesn_jobs as an argument.
Quoting the documentation here

n_jobs: int, default=None
Number of CPUs to use during the cross validation. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

However, running the below simple program on my 4 core machine (spec details below) shows performance is best when n_jobs = None, progressively deteriorating as you increase n_jobs all the way to n_jobs = -1 (supposedly requesting all cores)
    import numpy as np
    from sklearn.linear_model import ElasticNetCV

    # ------------- Setup  X, y -------------
    IMPORTANT_K = 10
    TOTAL_K = 30
    SAMPLE_N = 1000
    ERROR_VOL = 1

    np.random.seed(0)
    important_X = np.random.rand(SAMPLE_N, IMPORTANT_K)
    other_X = np.random.rand(SAMPLE_N, TOTAL_K - IMPORTANT_K)
    actual_coefs = np.linspace(0.1, 1, IMPORTANT_K)
    noise = np.random.rand(SAMPLE_N)
    y = important_X @ actual_coefs + noise * ERROR_VOL
    total_X = np.concatenate((important_X, other_X), axis=1)

    # ---------------- Setup ElasticNetCV -----------------
    LASSO_RATIOS = np.linspace(0.01, 1, 10)
    CV = 10

    def enet_fit(X, y, n_jobs: int):
        enet_cv = ElasticNetCV(l1_ratio=LASSO_RATIOS, fit_intercept=True,
                               cv=CV, n_jobs=n_jobs)
        enet_cv.fit(X=X, y=y)
        return enet_cv

    # ------------------- n_jobs test --------------
    N_JOBS = [None, 1, 2, 3, 4, -1]
    import time
    for n_jobs in N_JOBS:
        start = time.perf_counter()
        enet_cv = enet_fit(X=total_X, y=y, n_jobs=n_jobs)
        print('n_jobs = {}, perf_counter = {}'.format(n_jobs, time.perf_counter() - start))

What needs to be done to make this work as expected?
Some people seem to think this is broken in Windows as stated here. Indeed I am running Windows 10 on an Intel i7-7700 4.2Ghz machine, but unfortunately the question the above link points to does not have any commentary or answers, nor unfortunately do I have access to a Unix machine to try if this works on Unix.
Some people here say running from an interactive session is a problem. I observe the above program perform in the same way whether run on Jupyter Lab or as a script from the terminal.
I will also add that I have run this on several version combinations of sklearn and joblib, but have not been able to resolve the problem in that way.
One of those combinations is below, put together by conda env create when only specifying the python version python==3.6.10 thereby allowing conda to install the versins it deems are most appropriate.
I also include the numpy, mkl and scipy dependencies that were installed.
intel-openmp              2020.1                      216
joblib                    0.14.1                     py_0
mkl                       2020.1                      216
mkl-service               2.3.0            py36hb782905_0
mkl_fft                   1.0.15           py36h14836fe_0
mkl_random                1.1.0            py36h675688f_0
numpy                     1.18.1           py36h93ca92e_0
numpy-base                1.18.1           py36hc3f5095_1
python                    3.6.10               h9f7ef89_2
scikit-learn              0.22.1           py36h6288b17_0
scipy                     1.4.1            py36h9439919_0

The questions
Does the simple program above give you better performance with increasing n_jobs when you run it?
On what OS / setup?
Is there something that must be tweaked for this to work properly?
All help very much appreciated


texts:
I have a bunch of reviews:

User_id, review
1, "We (a family of 4 adults) chose this and view and loved this place"
1, "My husband and I, with our 2 teen sons, visit this restaurant at least once..."
2,"My partner and I booked table for a short holiday, their wine menu was awesome"
2,"My wife is a fan of jazz and she's expecting, so visited this place "

What techniques/packages are available to, for instance, estimate that:

User Id 1 => family of 4, 2 sons (13-19)
User Id 2 => family of 2, expecting
:
:

I have been googling around, to little help, and other than creating my own labeled dataset, I was hoping there are some NLP techniques that can help bootstrap my training set, which can then be curated by humans.


texts: How to use a CNN to predict free space in a 2d environment with obstacles?

texts:
When given a image(300 x 300) with obstacles(Basic 2D shapes) I want to predict 10 coordinates values in the image that will not collide with the obstacles using a CNN.

For now I am giving the coordinates of those 10 points(as train labels) like following
[[x11,x12, ..., y11,y12..], [x21,x22,... , y21,y22...]]
And I trained the nw but predicted label values are not close to the original one.

IS There a different approach I can take to find the coordinates of those 10 points with a CNN?


texts: Creating Flags Instead of Designated Values

texts: I'm working with http://archive.ics.uci.edu/ml/datasets/Bank+Marketing# dataset in order to create a model. We're going to use it in a presentation to introduce people our new data science environment. In the dataset, pdays column represents how many days passed over after previous campaign to customer. When there is no previous campaign for a specific customer then pdays is set to 999. Is it true to use it as so? Maybe I can create a new flag represents whether there is a previous campaign for a specific user or maybe I can set it simply 0 if there is no previous campaign. For example, if I'm using logistic regression then linear combination of features are used and for customers whose pdays is 0(no previous campaign) then simply pdays column has no effect. What would the correct treatment?


texts: What does the Orange widget "Image Embedding" do exactly?

texts: I would need to know what this widget does. What architecture does it use? What dataset has it been trained on? And so on.


texts: How do the linear layers in the attention mechanism work?

texts:
I think I now the answer to my question but I dont really get confirmation.
When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that there are three linear layers applied on the key, query and value matrix. And then one layer at the end, which is applied on the output of the matrix multiplication of the score matrix an the value.
The three linear layers at the beginnging: When the key/query/value with shape (seq-len x emb-dim) enter the linear layer the output is still (seq-len x emb-dim). Does that mean, the same linear layer is applied on every "index" of the input matrix. Like this (pseudo-code):
fc = linear(emb-dim, emb-dim) # in-features and out-features have the shape of emb-dim
output_matrix = []

for x in key/query/value:
    # x is one row the input matrix with shape (emb-dim x 1)
    x = fc(x)
    # x after the linear layer has still the shape of (emb-dim x 1)
    output_matrix.append(x)

# output_matrix has now the shape (seq-len x emb-dim); the same as the input-matrix

So is this indeed what happens? I couldn't explain why the output is the same as the input otherwise.
The linear layer before the output: So the output of the matrix multiplication of the score matrix an the value is also (seq-len x emb-dim) and therefore the output of the linear layer is too. So the output of the whole attention block has the same shape as the input.
So Im just asking for comfirmation if the explaination I wrote is correct. And if not: What am I understanding wrong?
Extra question: When I want to further use the output of the attention block further for classification, I would have to take the mean along the seq axis in order to get a vector of fixed shape (emb-dim x 1) so I can feed it into a classification layer. But I guess that valueable information is getting lost in that process.
My question: Could I replace the last linear layer with an RNN to get the desired output shape and without losing information?


texts:
Suppose I have a csv file with 20k rows, which I import into Pandas dataframe. I then run models like Random Forest or Logistic Regression from sklearn package and it runs fine. However, when I import into PySpark dataframe format and run the same models (Random Forest or Logistic Regression) from PySpark packages, I get a memory error and I have to reduce the size of the csv down to say 3-4k rows. Why does this happen? Is this a conceptual problem or am I coding it wrong somewhere?

For Pandas dataframe, my sample code is something like this:

df=pd.read_csv("xx.csv")

features=TfIdf().fit(df['text'])
....
RandomForest.fit(features,labels)

And for PySpark, I'm first reading the file like this:

data = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true')\
.load('xx.csv')
data.show()
from pyspark.ml.feature import RegexTokenizer, StopWordsRemover, CountVectorizer
from pyspark.ml.classification import LogisticRegression
# regular expression tokenizer
regexTokenizer = RegexTokenizer(inputCol="converted_text", outputCol="words", pattern="\\W")
# stop words
add_stopwords = ["http","https","amp","rt","t","c","the"]
stopwordsRemover = StopWordsRemover(inputCol="words", outputCol="filtered").setStopWords(add_stopwords)
# bag of words count
countVectors = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=10000, minDF=5)
from pyspark.ml import Pipeline
from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler
label_stringIdx = StringIndexer(inputCol = "Complaint-Status", outputCol = "label")
pipeline = Pipeline(stages=[regexTokenizer, stopwordsRemover, countVectors, label_stringIdx])
# Fit the pipeline to training documents.
pipelineFit = pipeline.fit(data)
dataset = pipelineFit.transform(data)
dataset.show(5)
(trainingData, testData) = dataset.randomSplit([0.7, 0.3], seed = 100)
print("Training Dataset Count: " + str(trainingData.count()))
print("Test Dataset Count: " + str(testData.count()))
from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier(labelCol="label", \
                            featuresCol="features", \
                            numTrees = 100, \
                            maxDepth = 4, \
                            maxBins = 32)
# Train model with Training Data
rfModel = rf.fit(trainingData)
predictions = rfModel.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)

I was trying for lightgbm, only changing the .fit() part:

from mmlspark import LightGBMClassifier
lgb = LightGBMClassifier(learningRate=0.3,
                           numIterations=100,
                           numLeaves=31)
lgb_model=lgb.fit(trainingData)
predictions = lgb_model.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(predictionCol="prediction")
evaluator.evaluate(predictions)

And the dataset has hardly 5k rows inside the csv files. Why is it happening? How can I solve it?


texts: How to fix class imbalance in training sample?

texts:
I was very recently asked in a job interview about solutions to fix an imbalance of classes in the training dataset. Let's focus on a binary classification case.

I offered two solutions: oversampling the minority class by feeding the classifier balanced batches of data, or partitioning the abundant class such as to train many classifiers with a balanced training set, a unique subset of the abundant and the same set of the minority. The interviewers noded, but I was later cut off and one of the knowledge gaps they mentioned was this answer. I know now that I could have discussed changed the metric..

But the question that  pops in my mind now is: is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion? The rule of thumb of machine learning seems to be that the training set needs to be as similar as possible to the testing for best prediction performance.

Isn't just in the cases with have no idea (no prior) about the distribution of the testing that we need to balance the classes? Maybe I should have raised this point in the interview..


texts: Why training and validation similar loss curves lead to poor performance

texts:
I am training a binary classification neural network model using matlab the graph that I got using 20 neurons in hidden layer is given below.
the confusion matrix and graph between cross entropy vs epochs.

to prevent overfitting in a model the training curve in a loss graph should be similar to the validation curve.

but in the current situation the third graph shows curve where validation curve is similar to training although the overall accuracy is low as compared to the curve where the two curve diverges in the above plot.

WHy this is happening and what I am doing wrong in understanding these curves?


texts:
I have a dataset consisting of addresses (points) that have several attributes; one that distinguishes the "sort" of address and one anntribute that contains a numerical value.

I want to cluster these points based on:
1. their distance to each other
2. the sort of address

However, the summed numerical attribute per cluster cannot exceed a certain
threshold value.

In other words, the systeem needs to form clusters, but needs to stop clustering as soon as the sum of the numerical value attached to each address has been reached.

How do I even go about it? I have R, Python and other geo- applications at my disposal.

It seems that none of the existing clustering algorythms work. For k- means for example I need to know the number of clusters beforehand, which I don't.

It seems rather simple, but I can't find a basic methodology to follow.


texts: Why is there a difference of "ML" vs "MLLIB" in Apache Spark's documentation?

texts:
I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.

  https://spark.apache.org/docs/2.2.0/mllib-feature-extraction.html#word2vec
  https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec

mllib seems to be for using RDD's. And ml seems to be using "DataFrames".

What is the difference? Shouldn't they both be using RDDs if this is spark under the hood?

What is a "DataFrame" here? As the documentation doesn't explain it.


texts: Should dimensionality reduction be done before k-means clustering if there are many features?

texts:
My data contains over 200 features and over 500 observations. I want to place the observations into a number of clusters based on the features that make them different.

There are numerous ideas I have and I'm not sure which one is apt:

1) Conduct principal component analysis (PCA) to reduce the features to two dimensions. I've already done this so that I could visualize them on a 2D plot. It would now be quite easy to use k-means clustering with these two dimensions to create the clusters, but I wonder if this isn't a good idea because of all the components that are being lost. But then again, if they're being lost they're probably not that important? Not sure

2) Conduct principal component analysis (PCA) to determine which features are worth including and then conduct k-means clustering on those features. So I probably wouldn't be reducing the dimensions to two, but they would be reduced and then the k-means clustering would be done. This seems like the best idea intuitively to me, but I'm not sure.

3) Forget the PCA and just conduct k-means clustering on all the features I have at the beginning. This feels like it's probably the worst idea because some of the features could be useless but could still be factored into the distance calculations for the clustering, but I'm just including everything I've thought of.


texts: How to determine frame rates to detect for video classification

texts: I am building a Deep learning model for vibration, I have video inputs and I want to analyze this video for vibrations and categories these vibrations into various classes of vibrations, what considerations should I make while choosing the framerate, or sampling frequency of the frames from the video? Is there some sort of calculation or evaluation metric from the same?


texts: Ways to create simulated numerical data based on small sample (using Python)?

texts:
For an internship, I'm being asked to simulate the electrical consumption for virtual appliances (e.g. fridges, freezers).

The company currently has a bunch of recorded second by second data from several different appliances. The data displays strong patterns. It has only two variables - time and (W) consumed. I will need to simulate a large amount of data to train the algorithm. What are the best ways to do this in Python? I have seen some modules that have functionality in that vein but they don't seem suitable for my data format.


texts: How to choose threshold for gensim Phrases when generating bigrams?

texts:
I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA
from gensim.models.phrases import Phrases, Phraser

# 7k documents, ~500-1k tokens each. Already ran cleanup, stop_words, lemmatization, etc
docs = get_docs()

phrases = Phrases(docs)
bigram = Phraser(phrases)
docs = [bigram[d] for d in docs]

Phrases has min_count=5, threshold=10. I don't quite understand how they interact, they seem related? Anyway, I see threshold having values in different tutorials ranging 1->1000, described as important in determining the number of bigrams generated. I can't find an explanation on how to come by a decent value for one's purposes, simply "fiddle and what works best for you". Is there any intuition / formula for choosing this value, maybe something like "if you want x% more tokens added to your dictionary, use y"; or "if your corpus size is x, try y"? I also see scoring='default' can be set to 'npmi' instead. From the linked paper, they say and t is a chosen threshold, typically around 10e−5. Might that be a decent approach if I just want this to work "good enough" without needing to fiddle much? That is, phrases = Phrases(docs, scoring='npmi', threshold=10e-5).
TL;DR: is there a simple or intuitive way to choose a decent threshold (eg, based on corpus size); alternatively would scoring='npmi',threshold=10e-5 be simpler?


texts: Should you turn off label smoothing when validating?

texts: As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand, I haven't seen any authoritative source claiming that it should be turned off during validation, not even the article that introduced the technique mentions it. And afaict, as the feature is implemented in Keras it is not turned off during validation.


texts: How keras.layers.embedding learn word embeddings?

texts:
I was trying some tensorflow tutorials and see that in all of them they use layers.embedding to learn these word embeddings, but how are these learned? , with a NN? which arquitecture? , or word2vec?
Thanks


texts: How to reduce the GPU consumption size while using Elmo Model?

texts:
I am performing an NLP task using Elmo model. Whenever I load the Elmo model, it occupies the 15 GB of my GPU memory. How can I reduce it ?
Below is my code
import tensorflow.compat.v1 as tf
import tensorflow_hub as hub
tf.disable_eager_execution()
from tensorflow.compat.v1.keras import backend as K
sess = tf.Session()
K.set_session(sess)

elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())

def ElmoEmbedding(x):
    return elmo_model(inputs={
                    "tokens": tf.squeeze(tf.cast(x, tf.string)),
                    "sequence_len": tf.constant(batch_size*[maxlen])
              },
              signature="tokens",
              as_dict=True)["elmo"]

And then I am passing the ElmoEmbedding in the Lambda layer as below
input_text = Input(shape=(maxlen,), dtype=tf.string)

embedding = Lambda(ElmoEmbedding, output_shape=(maxlen, 1024))(input_text)

x = Bidirectional(LSTM(units=512, return_sequences=True,
               recurrent_dropout=0.2, dropout=0.2))(embedding)

.....

What do I need to change in the above code ?


texts:
I have set of newspaper articles and I use TextRank algorithms to identify their keywords to perform a classification.

Apart from the important informative keywords, I am also receiving garbage keywords as follows.

"viewed", "flutter", "function", "k", "neighbour"

Is there a way to post process and remove these keywords?


texts: Influence of a data point on the regression result?

texts:
Let's say I perform multiple regression where y = income, x1 = educaiton, x2 = sex, and x3 = religion from 2003 to 2018, where the data is measured daily.

Is there any way to quantify an impact of a single day data (e.x. 2005-07-01) on the regression result?


texts: Clustering pre-defined groups of data points under dimensionality reduction

texts:
I have a dateset of around a million observations, and each observation (300 features) belongs to one of around 300 groups. The set of observations of one group does not directly correspond to the observations of another group in some meaningful way. There is no domain knowledge to help. I wish to cluster the groups themselves, rather than the individual observations. I make use of dimensionality reduction.
To achieve this I have had two ideas. The first, clustering all observations, then deriving the clustering of the groups via this clustering, by for example taking the mean, or majority vote of the individual observations / group. The second idea is to try to create many statistics to describe a group, and then cluster based on these statistics rather than the individual observations of a group.
I am not quite convinced of either idea and have trouble finding any further information on this problem. Is there a standard method of doing this?


texts: What are 2D dimensionality reduction algorithms good for?

texts:
It seems to me that t-SNE and other dimensionality reduction algorithms which reduce the dimensionality to two dimensions are mainly used to get an impression of the dataset. If done well, they look nice (e.g. like this), but I wonder if this is better than just showing random images / grouping them by class on a grid.

I would like to get an answer to the following aspects:

How is t-SNE* better than just taking a random (probably stratified) sample of the data?
How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?
Does t-SNE* give any guarantees?
Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?
There are many dimensionality reduction algorithms. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?

*: You could probably answer this for other dimensionality reduction algorithms as well, but t-SNE seems to be the most popular one.

Please note: I do see the advantage of a reduced dimensionality for compression / easier optimization / faster inference. However, the reduction to 2 dimensions seems to be only for visualization. Hence my question if one can see more in those embeddings than a visually pleasing image of the dataset.


texts: Can the 'bin size' in a histogram be thought of as a regularity constraint?

texts:
When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function?

Also, is there a better way to articulate this reasoning?


texts: What is the effect of highly correlated data on a Convolutional Neural Network?

texts:
A speech audio sample can be converted to MFCC coefficients for further analysis. I wanted to know the effect of correlated data on a CNN. I know the process of computing the MFCC coefficient, which applies Discrete Cosine Transform (DCT) to decorrelate the filter bank coefficients and yield a compressed representation of the filter banks.

I wanted to argue on the usage of filter banks vs MFCC when we want to further analyze using a CNN.

Filter banks result in highly correlated data.
MFCC result in decorrelated data.

Which of the two should i choose if i plan to train a CNN model for speech classification?


texts: Problem trying to build my own sklean transformer

texts:
I build the following sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
   def __init__( self , col2trans ) :
      self._col2trans = col2trans

   def fit( self, X, y = None ):
      return self

   def transform( self , X , y = None ) :
      for col in self._col2trans :
         p = X[ col ].value_counts() / X.shape[0]
         dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
         X.replace( { col : dic } , inplace = True )
      return X

This transformer is replacing catagorical values by their rates.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 0 ] ,
                 [ 2 , 2 , 2 , 'v' , 1 ] ,
                 [ 4 , 5 , 1 , 'w' , 1 ] ,
                 [ 2 , 1 , 9 , 'w' , 1 ] ,
                 [ 1 , 0 , 4 , 'w' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

Value 'v' ( 'w' ) for column 'four' is replaced by 2/5 ( 3/5 ).

My purpose is to fit the transformer on df and apply it to another dataframe df2 :

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 0 ] ,
                    [ 9 , 2 , 2 , 'v' , 0 ] ,
                    [ 4 , 5 , 1 , 'w' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

I am doing this that way :

# Transformer instance
trsf = Cat2Rat( [ 'four' ] )

# Fitting
trsf.fit( df )

# Then applying
trsf.transform( df2 )

But the rates are those of column 'four' values of df2 not df ( on which the transformer was fitted ).

I must have been missing something on the way to properly build such a transformer.

Could someone give some clue on how to fix the transformer so that it gives proper result?

Thanks.


texts: Learning curve using micro F-score and macro F-score

texts:
I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier.

The first plot is made using micro F-score, and the second using macro F-score. I find it quite difficult to interpret both of them. The explanations I found for learning curves use errors.

I think in the first plot there is variability up to around 300 instances, and then it starts to converge until the point when the lines are parallel. Therefore, even adding more data would not help. But what about bias?

Also can you please explain to me the difference between the plots in terms of micro and macro F-scores?


texts: Pattern recognition of two dimensional points with specific Voronoi feature

texts:
In mathematics, a Voronoi diagram is a partition of a plane into regions close to each of a given set of objects.
Each Voronoi diagram has properties depending on the location of the points on the plane. For example, the area of each cell can be considered as these features. Each cell can be placed in a class if specific classes for the size of the area are defined.
Suppose area classes are defined. Also, the specific percentage of cells area with tolerance assign to a specific class. The Voronoi diagram is repeatedly produced With random points. The cases, which satisfy requirements for any class, are specified.
My question is: Can I determine the pattern of points based on specified cases? Can I predict other cases with the same requirements?


texts: Does produced data fall under the "data collection" category?

texts:
I am not sure whether data governance and policy is covered by this community. I'm giving it a try and let me know if I need to adjust.

While surveying all the data of my current employer, and developing a data governance policy, I am able to categorize two different sets of data on our customers:

data we collect on customers
data we produce about customers

Example in the context of my organisation (the data producer and data collector) being a school:

Collected Data: the student's parent's names and date of birth
Produced Data: the student's grades.

However, from the general definition of data collection, it seems that the data produced also falls under the "data collection" term. Is there a generally accepted term to dissociate these two categories?


texts: group similar subjects and train only using them

texts:
I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.
Now to build a model, I don't like to train the usual way (where we build a generalised model).
Instead I would like to group similar patients together like group1, group2, group3, group4 etc.
And when I new subject to predict, I would like to know which group does he belong to and find out the important predictors for each group?
I know this sounds like K-means clustering but am I right to understand that?
Can anybody provide me the list of steps (pseudocode type) on how to do this? Sorry, am new to ML and exploring various ways to do a classification problem


texts: How to predict when an appointment will be scheduled?

texts:
I have a dataset of tens of thousands of appointments. Appointments have a created date and scheduled date. Something like this:
ID   Created      Scheduled
1    08/01/2020   08/05/2020
2    08/01/2020   08/07/2020
3    08/02/2020   08/04/2020
...

I'm trying to predict the probability of all possible schedule dates based on a created date in the future. So basically, if a customer created an appointment tomorrow (August 20th), what's the probability they would schedule their appointment for August 21th, August 22nd, August 23rd, etc. In theory, customers can create appointments into perpetuity but practically no one makes an appointment more than ~2 months in advance.
Some observations I've made are:

Customers prefer making appointments on weekends
Customers prefer making appointments during the last or first few days of the month
Most appointments are scheduled within 2 weeks of the day the customer creates the appointment

I've been struggling with this problem. I first tried to just look at how many days out a customer schedules their appointment. Something like 10% of the time it's the next day, 15% of the time it's 2 days out, etc. But that didn't account for how customers preferred making appointments on weekends and the start/end of the month. So it was incredibly inaccurate.
I'm frankly stumped about how to approach this problem. I would appreciate it if people have ideas on how I can go about this. Thank you! Please let me know if anything here is unclear.


texts: Importing Excel format data into R/R Studio and using glmnet package?

texts:
I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use.  But, when I want to use the glmnet package to develop a regularization model, I invariably run into the following error (after specifying my regularization model and attempting to run it):

Error in storage.mode(y) <- "double":
  (list) object cannot be coerced to type 'double'

Here is what I have already tried to resolve this:

De-format the numbers in Excel (no scientific notation, no %, etc.)
Did copy-paste-special values several times
At time of importing the spreadsheet, converted every column from "include" to "numeric" type
After importing the data, converting it to a matrix.

But, none of the above have eliminated the error.


texts: CNN: How do I handle Blurred images in the dataset?

texts:
I have 30% blurred images in each classes. I have a total of 10 classes. I'm not allowed to drop these blurred images. How do I train the model to get better accuracy for both blurred and nonblurred training dataset ? Currently, I'm at 11% accuracy.
The images were blurred using a Gaussian blur.
I have used a Wiener filter, but not able to restore the image from blurred images.
Please can anyone suggest a good way to train the model.


texts:
How can I connect to Titan database from Python ?

What I understand is that Titan (Graph database) provides an interface (Blueprint) to Cassandra (Column Store) and
bulb is a python interface to graph DB.

Now how can I start programming in python to connect with titan DB?
Is there any good documentation/tutorial available ?


texts: Using classifier's probabilities as independent variables to predict Y

texts:
Suppose you have a classification task y~X with (n_samples,m_features). A colleague told me that it is correct to run r different classifiers to predict y based on X and then use the probabilities given for each classifier as a new matrix Xnew (n_samples_probabilities,r_columns) to train a new classifier y~Xnew

My questions are:

1) Is this something reasonable to apply?

2) If so, is there any mathematical support on this method?


texts:
I am creating an n-gram model that will predict the next word after an n-gram (probably unigram, bigram and trigram) as coursework.

I have seen lots of explanations about HOW to deal with zero probabilities for when an n-gram within the test data was not found in the training data. I understand how 'add-one' smoothing and some other techniques work.

However, I can find nothing about WHY we need to take actions such as these.

For instance, if the test data has "Peace begins with a Smile" and this was not in the training data, so when I supply the model with "Peace begins with a", it will not come up with "Smile" end word. It may provide others or none. If there are none or they have a low probability, then I would supply the shorter n-gram of "begins with a" and see what words and probabilities that provides. If that fails, then "with a" and so on.

I suspect I'm missing something but can't see what.

Please can you help?


texts: (Scikit-learn) differences between LinearSVC, 'linear' kernel SVC and poly kernel SVC with degree 1

texts:
I would like to know the differences between:

linearSVC()
SVC(kernel='lineaer)
SVC(kernel='poly', degree=1)

Will these all determine the same boundary decisions?


texts: Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

texts:
I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading many blog posts/tutorials/documentation about how to present the data in the correct shape (most examples being of slightly less

My Dataset:

several cities
for which I have info about say temperature, car traffic, humidity
for say the last 2 years (one record for each day)

What I want to do:
I'd like to forecast for each city the temperatures I can expect for the next year using a possibly lagged version of temperature, car traffic and humidity (of course there would be several more features but this is just an example for thought).

What I am confused about:
If I have 2 cities, for which I recorded 3 features for 365 days. How should I shape my input so that the model can output a forecast for 365 days for these two cities (i.e. 2 time series of temperatures for 365 days)?

Intuitively the tensor shape would be (?, 365, 3) for 365 days and 3 features. But I'm not sure what to stick into the first dimension and, most importantly, I would be surprised if it had to be for the number of cities. But at the same time, I have no idea how to specify into the model that it has to understand the dimensions properly.

Any pointers will be helpful. I'm pretty familiar with the rest of the problem (i.e. how you build a network in Keras etc since I have done this for other neural networks but more specifically how best to encode the sequence for the desired input.)

Oh and also, I guess I could train and predict for each city independently, but I'm sure everyone will agree there are probably things to be learned that are not particular to any city but that can only be seen if considering several of them, hence why I think it is important to encode it in the model.


texts: How can I improve the results of my clustering

texts:
I am working on a project with the idea to cluster the sound waves of key strokes on a computer. So far what I have done was recorded about 50 keystrokes per key (only have done 1 - 10 so far), found their peaks and isolated the waveforms, then I created a chroma vector as the Feature Vector and run k-means on the resulting feature matrix.

After some very minor filtering I was able to get an accuracy of about 90% (taking accuracy as the mode cluster label for a group of keys as the true value, how many are the true value / total values). However, when I try to essentially classify new data it is far from accurate.

My initial thoughts as to why this probably wont work:

The difference in any audio feature between keystrokes is minimal
Many many more samples are required
Clustering is not the best approach here (I don't have any formal reasoning here but I would guess a supervised alg would work better).
The pre-processing I'm doing is not right. The only things I have done were normalize the audio files and use a smoothing function on the waveforms (no frequency analysis here!).

Any suggestions are appreciated.


texts:
I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary.

I have a lot of Raman spectra data and and most of them differ between both classes even to the naked eye (characteristic peaks or the lack of).
While I do have many datasets I know were not recorded under exact same conditions, I also have a few sets for which exact same conditions apply for both classes. They still look obviously different from each other. Since my primary goal is to develop a robust algorithm with an unsupervised classificator, I now start to wonder:

Assuming the setup and thus noise and any sort of distortion is the same in both classes, even if dominant, do I need to bother with pre-processing when various classifiation models already return satisfying results with the raw data?

Most (if not all) papers I read about Raman spectra pre-processing (there's a lot on that topic) gave me the impression, the authors are always interested in separating the Raman spectrum of interest from all the unwanted other signals (fluorescent background, white noise, cosmic spikes etc). While this certainly would be interesting, I actually only want to sort them.

I'm asking this question from a rather hypothetical standpoint. I know that in reality you can hardly guarantee exact same conditions (especially with random occurrences like cosmic spikes).


texts: Efficient environment for machine or deep learning in Python

texts:
I am getting very frustrated working with either Google Colab or Azure notebooks (they are very slow and glitchy). Usually, I work with Jupyter notebooks to perform Machine Learning or Deep Learning tasks in Python.

Does anyone have any recommendations for a truly good performing alternative? It does not matter if it is free or not.


texts: Demand Forecasting - Error in predictions

texts:
I've been trying to do daily demand forecasting using the H2O AutoML package for using 4 years of daily sales data as the training set. However, the daily estimates always seems to find a range around which they oscillate. The following are the features I use:

Date
Demand of same weekday in the previous week
Demand of same weekday in the second previous week
Median demand of the same weekday in the previous month
Median demand of the same weekday in the previous quarter
Median demand of previous month
Median demand of previous quarter
95% demand quantile of previous month
95% demand quantile of previous quarter
5% demand quantile of previous month
5% demand quantile of previous quarter

Can anyone please suggest where I'm going wrong?


texts: When I use bootstrapping to calculate confidence intervals for accuracy, should I resample the entire dataset or just the test set?

texts:
In this article, the author resamples the entire dataset and retrains the model each time. However, I don't see why you couldn't just resample the test set only, which is what I'm currently doing.

In my case, I'm doing text classification and have around 80,000 unlabeled tweets, and 2,000 labeled tweets. I split the 2000 into 1700/300, then trained and saved a model on the 1700. In order to find confidence intervals for accuracy, I resample the 300 test set with replacement 300 times and use the saved model to predict on that new set of 300, then repeat that a lot of times to get a distribution for accuracy. Finally, I use this saved model to predict labels for the 80,000 unlabeled tweets.

To me, this seems better than resampling the entire set and then retraining (what the article does) because

1) It's much faster

2) I want to know the saved model's accuracy confidence interval, because that's the model I'll be using to predict labels on the large set. I don't care how well it does if it was trained on a different set of tweets.

Why is this approach not the correct way?


texts:
Does this mean that as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college

Any additional things i can interpret from below ?


texts:
I'm using a dataset of movies and would like to group if a movie is the same across different retailers.

Example:

Movie: Beauty and the Beast

Platforms: Google, Netflix, iTunes, Amazon.

I have access to signals like:
Studio, Movie Name, Runtime, Language, Release Year, etc. However, in this case some movies, which are not the same and signals mentioned before, are not capable of finding the right match. I need to do what a human would do: Check Movie cover. Example:

Beauty and the Beast http://www.imdb.com/title/tt2771200/
Beauty and the Beast http://www.imdb.com/title/tt6305650/?ref_=fn_al_tt_1

I have access to the art image.
I'm using Python to do this comparison.

Is there a library that can help me compare 2 images and determine if they are similar?


texts: What are the best activation functions for Binary text classification in neural networks?

texts:
I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification.

I have heard that Relu is best for Binary classification (not sure if im correct)

I have used keras to train a model, which is 2 layer , Dense 512, dropout 0.3, activation = "Relu" for these layers,

But the predictions are not upto the mark.

I have also changed the Dense units to 1024, keeping others same, but still I got bad predictions. (Validation accuracy 50%)

So, can i use other activations, or change my model layers (add few more layers) ??

What can be the best option ?


texts: Clustering time series data using dynamic time warping

texts:
I would like to cluster/group the curves in the attached picture with Python. The data is already normalized and my approach would be to use dtw (dynamic time warping) to calculate the distance and with that feature use a clustering algorithm (like kmeans or DBSCAN) to classify them.
Do I pick out one trajectory as a starting curve to compare the other curves to, or do I calculate an 'average' curve of all curves and use that as the starting curve to compare to?
Moreover I am not completely sure how to interpret the dtw distance. Please have a look at the code below:
from dtaidistance import dtw
import numpy as np
y=np.random.randint(0,10,10)
y1=y+2
dist=dtw.distance(y,y1)

The shape of the two curves should be exactly the same just shifted on the y-axis, but the dtw still calculates a distance greater than 0. Is this because of the fact the two curves are not normalized and then would be aligned exactly over each other? I always had the impression dtw measures the similarity of the shape of two curves and therefore is susceptible to shifting or stretching. I am just worried that if a curve has a different shape, but is near the compared curve, the dtw distance will be way smaller than the dtw distance of two exact same, but shifted curves.


texts: Splitting train/test sets by an identifier?

texts:
I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, the random seed will reset with each updated dataset and take a different train/test split. Doing this, your ML algos will eventually cover the whole dataset, defeating the purpose of the train/test split because it'll eventually train on too much of the whole dataset over time.

The book I'm reading (Hands-On Machine Learning with Scikit-Learn and Tensorflow) gives this code to split train/test by id:

# Function to check test set's identifier.
def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

# Function to split train/test
def split_train_test_by_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]

And it says when there's no ID column given, to create one either by indexing the rows or creating a unique index from one of the variables.

My questions are:

What is the 3rd line doing:

crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32
What is the anonymous function doing in the 2nd to last line?

lambda id_: test_set_check(id_, test_ratio)
In practice, do you commonly split datasets by id in this manner?

Thanks,

Greg


texts: In image classification/recognition how are images coded?

texts:
I am trying to understand how images are coded for classification/recognition.

Suppose that I have images that contain animals (e.g. dogs, cats, birds) -- does each image have a label for that animal or does the label only apply to the portion of the image with the animal in it? If the entire images are coded does it take a classifier longer to learn?


texts:
I have a very big set of high-dimensional, but sparse binary vectors. Each vector represents a "one-hot-style" n-gram sequence of words where each index of the words that occur in the n-gram is set to 1, all others to 0. E.g.

[0,0,0,0,0,1,0,0,0,0,1,0,0,0,...]

An efficient representation for a 5-gram:

[5,10,235,1253,5521]

What I would like to do is generate such n-gram representations from a large text corpus. Hence, this may easily result in a vocabulary size of hundreds of thousands (depending on potential pre-processing steps including stopword removal, stemming, etc.) and even more distinct n-grams (m).
This could be represented as a matrix of size m x vocab_size.

Subsequently, I would like to reduce the number of rows by clustering similar n-grams together while retaining the contained words. For instance:

[5,7,10,12,15],[5,8,10,15,18] -> [5,7,8,10,12,15,18]

My naive approach would be to apply some traditional clustering algorithm in order to assign each entry in the initial n-gram set to a specific cluster and then perform a union over all the n-grams assigned to a cluster.

1st question: what would be an sensible algorithm for clustering sparse binary representations?
2nd question: how could this be done with a very big data set?


texts: What is the meaning of a quadratic relation when r = 0?

texts:
A website (on page 4) says:

  The correlation coefficient is a measure of linear relationship and thus a value
  of  r = 0 does not imply there is no relationship between the variables. For
  example in the following scatterplot which implies no (linear)
  correlation however there is a perfect quadratic relationship.

What is the meaning of a perfect quadratic relationship?


texts: Extracting date, relation and noun phrase from text

texts:
A sentence (Segmented from a document) as below:

This Amendment dated 26th of April 2018 modifies the Agreement dated 20th April 2017.
or
This Amendment depends on the agreement dated 20th April 2017

I would like to extract the following as a CSV

Document_type | Date_of_document | relation | related_doc | related_doc_date
----------------------------------------------------------------------------
Amendment     | 26-04-2018       | modifies | Agreement   | 20-04-2017
Amendment     | Null             | depends  | agreement   | 20-04-2017

What have I done till now

Found and extracted dates using Stanford NER. Got them in a normalized form. Can extract Noun Phrases using TextBlob. But cannot link which Noun Phrase is linked to which date.

Am I doing it wrong? Any tutorials/examples/advice?


texts: Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

texts:
It is common to define the F-measure as a function of precision and recall, as mentioned in [1]:

  $F_{\beta}=\frac{(1+\beta^2)PR}{\beta^2 P+R}$

However I came across some other cases, another definition is used [2] (without weights):

  $F = H(sensitivity, 1- specificity)$

Where H is harmonic mean.

Reference:

F - measure derivation (harmonic mean of precision and recall)
https://link.springer.com/chapter/10.1007/978-3-540-68947-8_133.
https://stackoverflow.com/a/52892413/2243842


texts:
If I have a dataset with events occuring at certain times of day, Hour, how would I go about using this for, say, a classifier? Example:

Hour     | Event | Item
08:45:22 | Buy   | Apple
09:03:10 | Buy   | Orange
10:00:00 | Sell  | Apple

Would I convert the Hour timestamp into a numeric value, such as Unit Timestamp and then do normalization on it, like I would any other numeric value?


texts: How should I convert Logistic Regression's coefs into action strategy?

texts:
I am trying to analyze soccer's data set:

W_OVER_2_5  PREDICTED MATCH_DATE LEAGUE HOME AWAY MATCH_HOME MATCH_DRAW MATCH_AWAY  MATCH_U2_50 MATCH_O2_50
0            0        1105135200    5   260  289    2.05        3.00        4.50        1.65       2.30
0            1        1105308000    16  715  700    2.50        3.30        3.05        1.80       2.14
1            1        1105308000    11  445  479    1.36        5.25       12.00        2.15       1.78
0            1        1105308000    11  453  474    3.00        3.35        2.62        1.75       2.20....

Now, I selected 'the best estimator' -

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,
          verbose=1, warm_start=False)

with best coefs -

1. -2.40477246e-10
2. -5.57611571e-02
3. -1.32010761e-04
4.  1.51666398e-03
5.  7.54521399e-02
6.  6.38889247e-02
7. -2.25746953e-01
8. -3.79313902e-01
9.  3.70514297e-02

Now, I have a question - how should I understand coefs in terms of real strategy? As example,

If `MATCH_HOME` is min among all [`MATCH_HOME`, `MATCH_DRAW`, `MATCH_WAY`] AND `MATCH_O2_50' = 1
THEN PREDICTED := 1
ELSE PREDICTED := 0

PS. I would very appreciated for any science papers about that thematic :)


texts: Gensim word2vec training error on tweets

texts:
I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:

def tokenize_tweets():
    for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):
        tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])
        try:
            mod_text = tokenize(tweet_text)
            tokens = tknz.tokenize(mod_text)
            if len(tokens) > 0:
                yield tknz.tokenize(mod_text)
            else:
                yield ['NULL']
        except UnicodeEncodeError as e:
            yield ['<NULL>']

Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.

Not sure what is wrong with it.


texts: Best way to format data for supervised machine learning ranking predictions

texts:
I'm fairly new to machine learning, but I'm doing my best to learn as much as possible.

I am curious about how predicting athlete performance (runners in particular) in a race of a specific starting lineup. For instance, if RunnerA, RunnerB, RunnerC, and RunnerD are all racing a 400 meter race, I want to best predict whether RunnerA will beat RunnerB based on past race result information (which I have at my disposal). However, I have many cases where RunnerA has never raced against RunnerB; yet I do have data showing RunnerA has beat RunnerC in the past, and RunnerC has beat RunnerB in the past. This logic extends deeper as well. So, it would seem that RunnerA should beat RunnerB, given this information. My real concern is when it gets more complicated than this as I add more features (multiple runners, different distances, etc), and so I'm turing to ML algorithms to help my predictions.

However, I am having difficulty figuring out how to include this in my row data that I can train (after all, correctly formatting data is 99% of proper machine learning), and I am hoping that someone here might have thought along the same lines in the past and might be able to shed some light.

Example:

I am currently trying to include RunnerX-RunnerY past race data by counting all the races that RunnerX and RunnerY have run together and normalizing them on a scale from -1 to 1; -1 indicating RunnerX lost all past races against RunnerY; and +1 indicating that RunnerX has  won all past races against RunnerY; and +1 indicating. And 0 indicating an equal number of wins and losses (or no past races against each other).

For instance, if RunnerA is racing RunnerB, and RunnerA has beat RunnerB in the past, then I want the algorithm to know that (denoted by a +1 on the RunnerB column of row RunnerA); same for vice versa. Taking it another step further, If RunnerA is racing RunnerC (but the two have never raced each other in the past), and RunnerA has beat RunnerD in a past race, and RunnerD has beat RunnerC in a past race, then I want the algorithm to learn that RunnerA should beat RunnerC. I say beat here, but I mean an "average beat" for any RunnerX-RunnerY combinations when data for more than 1 past race is available.

I have set my data up as:

name     track   surface  distance  age    RunnerA   RunnerB   RunnerC   RunnerD
RunnerA  Home    2        400       11     0         1         0         1
RunnerC  Away    2        400       12     0         0         0         -1
RunnerD  Home    2        400       10     0         0         1         0

which shows that RunnerA has beat RunnerB and RunnerD in the past. RunnerC has lost to RunnerD. And RunnerD has beat RunnerC.

The problem:

The problem is that I don't really think this is a correct display of the information for an ML algorithm.

From what I understand, ML data should be row independent. And this data isn't because row 1 (RunnerA) has beat RunnerD, yet the data indicating RunnerD has beat RunnerC is in row 3.

Does anyone have any ideas how I might be able to incorporate this past win-percentage-for-runner-pair-combination data??? I'm totally stuck here. I've read a lot about some algorithms that estimate the win loss by simply totaling win statistics, but those don't say anything about the actual probability of a particular runner to beat another particular runner.

Any pointers would be super helpful.

Thanks!!!


texts: How to build recommendation system that also takes time as a feature

texts:
Let's say a User was Interested in football, so we were recommending him posts about football after few days or weeks he starts watching Baseball.

How'd we go about building a recommender system that also takes recency into consideration
Or
Let's say that it knows when a user was interested in what


texts: How to determine which type of layer one should use?

texts:
Considering the SRGAN, I found it amazingly difficult to find logic on how this architecture was thought.

The first two layers are input layer and 2d convolution layer, whose choice is pretty simple to understand since we want to detect features to scale them.
But after, there are what is called "residual blocks" containing PReLU layers, element-wise sum layers, batch normalization layers, and convolution layers...

Moreover, another element-wise sum is done after these residual blocks plus some batch normalization and convolution again.

How did the author think to use all these types of layers, in this particular order, and what logic does bind these layers? In order to be able to think myself to new architectures and to understand simpler existing ones.


texts: Branch of data science that covers event based time series?

texts:
Let's say I have discrete events in time, e.g. patients getting sick, and I want to predict whether theses events are indicators of some other underlying event, e.g. a disease outbreak. Usually, one would transform the event based time series into a regular time series by for example aggregating the counts of patients over each week. However, I feel that by this aggregation a lot of information is lost and it is also hard use to multiple features of events, e.g. patients' age, patients' sex, etc. My question is very general: is there a branch of statistics / data science that treats time series as composed of events at arbitrary times rather that aggregated values at evenly distributed intervals?

I tried googling it, but it seemed hard to phrase the question in a way that a search engine understands.


texts: Is finetuning from a pretrained model always better than training from scratch?

texts:
At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn't it be better to always start with a pretrained model, as the lower layers of weights has already probably learnt general patterns of images that are transferable across all data sets?

My worry is what if the dataset I want to use for finetuning is highly specialized, highly unnatural and very different from the dataset the pretrained model is trained on. Would this still mean finetuning from a pretrained model wouldn't be the best idea? (e.g. training on X-Ray images instead of natural images for a cifar-1000 pretrained model.)


texts: Is there a way to use Orange3 on a remote server or in the cloud?

texts: My question there is a way to run the Orange, not on a local computer, but on a remote server, even in the cloud? My notebook has only 8GB memory and limited number of CPU core, which is insufficient for real-world data mining task, however I can use much powerful (remote) server machines.


texts:
I have a corpus of manually annotated (aka "gold standard) documents and a collection of NLP systems annotations on the text from the corpus. I want to do a bootstrap sampling of the system and gold standard to approximate a mean and standard error for various measures so that I can do a series of hypotheses tests using possibly ANOVA.
The issue is how do I do the sampling. I have 40 documents in the corpus with ~44K manual annotation in the gold standard. I was thinking of using each document as a sampling unit, and taking 60% of documents for each sample (or 24 documents per sample). However, the issue is that each manually annotated documents does not have the same number of annotations, so that violates using same sample size for each sample.
Any suggestions on how to achieve this bootstrap?


texts: How to plot the bar charts of precision , recall and f-measure?

texts:
I have used 4 machine learning models on a task and now I am struggling to plot their bar charts just like shown below in the image . I am printing classification report to get precision , recall etc . My code is shown:
def Statistics(data):
  # Classification Report
  print("Classification Report is shown below")
  print(classification_report(data['actual labels'],data['predicted labels']))

  # Confusion matrix
  print("Confusion matrix is shown below")
  cm=confusion_matrix(data['actual labels'],data['predicted labels'])

  plt.figure(figsize=(10,7))
  sn.heatmap(cm, annot=True,cmap='Blues', fmt='g')
  plt.xlabel('Predicted')
  plt.ylabel('Truth')
Statistics(data)

How can I plot this type of chart in python
I have saved my actual and predicted labels into csv . There are 4 models applied on data . So I saved their actual and predicted labels into Csv's  just like shown below :
path='/content/drive/MyDrive/CSVs/OriginalCensusRFmodel.csv'
def read_csv(path):
  # Read csv
  data=pd.read_csv(path)
  data=data.drop('Unnamed: 0',axis=1)
  return data
data=read_csv(path)

    def Statistics(data):
      # Classification Report
      print("Classification Report is shown below")
      print(classification_report(data['actual labels'],data['predicted labels']))

      # Confusion matrix
      print("Confusion matrix is shown below")
      cm=confusion_matrix(data['actual labels'],data['predicted labels'])

      plt.figure(figsize=(10,7))
      sn.heatmap(cm, annot=True,cmap='Blues', fmt='g')
      plt.xlabel('Predicted')
      plt.ylabel('Truth')
    Statistics(data)


texts: Is there a flexible way to get the original data indices from each cell of a confusion matrix?

texts:
Let's say I have model A and model B, that I want to compare the performance of by inspecting a confusion matrix. They both produce a list of predictions, pred_A and pred_B, corresponding to a set list of ground truth values ground_truths. I can use something like sklearn.metrics.confusion_matrix in order to generate a confusion matrix for each of these models:
cm_A = confusion_matrix(ground_truths, pred_A) # e.g. [[5,1], [2,3]]
cm_B = confusion_matrix(ground_truths, pred_B) # e.g. [[6,0], [3,2]]

What I want to figure out is how to get the specific indices from each cell of the confusion matrix, corresponding to the ground truth class. This will let me investigate which specific data points are true-positives, false-positives, false negatives and true negatives, and let me search for patterns based on their input features. In the example above, are two of the false negatives the same false negatives in cm_A and cm_B? Or did all of the true positives switch with all of the false negatives between the two model predictions? If I knew their indices in the ground_truths list I could answer the question easily.
I can think of stupid/inefficient ways to do this, but I would love to find a solution that could scale to an arbitrary number of labels.


texts: Why don't the dimensions in this linear regression equation match up?

texts:
I'm going through an article on linear regression, and they give the following formula for computing estimates:

The convention is that all vectors are column vectors. So if n is the number of samples and m is the number of parameters, hw(x) has dimensions n x 1, wT has dimensions 1 x m, and x has dimensions n x m. The problem is that these dimensions don't work out for matrix multiplication.

Where have I gone wrong here?


texts: Python TypeError: __init__() got an unexpected keyword argument 'decision_function_shape'

texts:
I tried creating a SVM Classifier, as:

# Create a SVM Classifier
model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
            decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
            max_iter=-1, probability=True, random_state=None, shrinking=True,
            tol=0.001, verbose=False
            )

(Using Python 2.7)

But getting this error--

  TypeError: init() got an unexpected keyword argument
  'decision_function_shape'

Any thoughts on that? How to sort it out?

Update >>

My sklearn version is 0.16.1. I tried to install the update but it's kept on saying- No matching distribution found for the upgrade.


texts:
Is somebody know a Python equivalent to R NbClust?

I'm searching for a way to determine the optimal clusters number for many 'similar datasets' and NbClust sounds good but it's for R and i'm a Python user...


texts: How to improve tensorflow 2.0 code for policy gradient?

texts:
I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient tape, however, when training the model, I have to convert the weights Tensor to a numpy array, update the weights then reassign the tf.Variable back from the numpy array. I feel this is not performant and could find a better way. Full code is here https://github.com/entrpn/reinforcement-learning/blob/master/tf2_rl/bandits.py

The main code I'm looking to improve as follows:

def train(agent,action,reward, learning_rate=0.001):
    with tf.GradientTape() as t:
        current_loss = loss(agent(action),reward)
    dW = t.gradient(current_loss,[agent.weights])
    weights_as_np = agent.weights.numpy()
    responsible_weight = agent.weights[action]
    responsible_weight_dw = np.array(dW)[0][action]

    weights_as_np[action] = weights_as_np[action] - learning_rate*responsible_weight_dw

    agent.weights.assign(tf.Variable(weights_as_np))


texts: Tensorflow tokeniser: the maximum number of words to keep

texts:
Trying to tokenize the IMDB movie reviews by applying Tensorflow tokenizer.  I want to have a maximum 10000-word vocabulary. For unseen words, I use a default token.
type(X), X.shape, X[:3]

(pandas.core.series.Series,(25000,),
 0    first think another disney movie might good it...
 1    put aside dr house repeat missed desperate hou...
 2    big fan stephen king s work film made even gre...
 Name: SentimentText, dtype: object)

from tensorflow.keras.preprocessing.text import Tokenizer

tokenizer=Tokenizer(num_words=10000,oov_token='xxxxxxx')
# fit on the input data
tokenizer.fit_on_texts(X)

When I check the number of words in tokenizer dictionary I get:
X_dict=tokenizer.word_index

list(enumerate(X_dict.items()))[:10]
[(0, ('xxxxxxx', 1)),
 (1, ('s', 2)),
 (2, ('movie', 3)),
 (3, ('film', 4)),
 (4, ('not', 5)),
 (5, ('it', 6)),
 (6, ('one', 7)),
 (7, ('like', 8)),
 (8, ('i', 9)),
 (9, ('good', 10))]

print(len(X_dict)
Out: 74120

Why do I get 74120 words instead of 10000 words?


texts: How to paste string and int from map to an array in hive?

texts:
I am trying to paste a string and int from map in Hive to an array.
For now, record looks like this:

{"string1":1,"string2":1,"string3":15}

Is there a way to convert it to an array like this:

["string1:1","string2:1","string3:15"]


texts: not quite sure about the difference between RNN and feed forward neural net

texts:
I'm a bit confused after reading this paper: https://arxiv.org/abs/1705.09851

on page 22, the author writes

response:

\begin{equation}
Y = softmax(Z^{L-1})
\end{equation}

and hidden state

\begin{equation}
Z^\ell = max(W^\ell *Z^{\ell-1} + b^\ell, 0)
\end{equation}

which is a relu

But, to me, this looks like a regular feed forward neural net- you multiply your input by a matrix, add a bias unit, then activate. Alternatively, your hidden layer is equal to the activation of the sum of a bias and the previous hidden layer times a weight matrix.

What am I missing?


texts: Is there a difference between np.matrix(np.array([0,0])) and np.matrix([0,0])?

texts:
I was reading this code, for implemnting linear regression from scratch:

# convert from data frames to numpy matrices
X = np.matrix(X.values)
y = np.matrix(y.values)
theta = np.matrix(np.array([0,0]))

When I came accross this line :  np.matrix(np.array([0,0]))

I was wondering why didn't the person just write np.matrix([0,0]).

I ran both in jupyter notebook and got the same output:

theta = np.matrix([0,0])
theta2 = np.matrix(np.array([0,0]))
print(theta,theta2,type(theta),type(theta2))

Output:[[0 0]] [[0 0]] <class 'numpy.matrix'> <class 'numpy.matrix'>

Is there a difference between the two? Does the extra np.array somehow part add to the functionality of theta? Will the final code function properly if I replace the former with the latter?

Thanks.

Edit: Is this the right place to ask this question? I am new here...


texts: Prediction using OSM and Airbnb Listing dataset

texts:
I am building a model to predict the price of an Airbnb listing.
Using OSM("https://wiki.openstreetmap.org/wiki/Downloading_data") and Airbnb listing("http://insideairbnb.com/get-the-data.html") dataset to get the relevant information.
My plan is to model using the Airbnb location, and nearby attraction(such as restaurants, clubs, parks etc).
Using the above model to predict the data
I am not sure if the plan is good for a beginner's project. And want to know how to actually approach the problem.


texts: How to deal with name strings in large data sets for ML?

texts:
My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later.

Some word embedding techniques were used for longer text sequences preferably, not for single-word strings as in this case. So I think these techniques wouldn't be the way that will work correctly. Additionally Label encoding or Label binarization may not be suitable ways to work with names, beacause of many different values on the on side (Label binarization) and no direct comparison between names on the other side (Label encoding).

Are there other approaches to use or transform especially name information in order to work with ML algorithms?


texts: How to extract features from long chemical names?

texts:
I have an interesting problem that I am uncertain about how to even get started. I am working on a binary classifier that will take a chemical name, encoded as a string, and predict whether it is a 'good' or 'bad' name. I have had quite good success doing this by examining the structure of the chemical directly, but I would like to explore whether I can learn anything about the given name of the chemical itself (as the name can contain some structural information about the molecule that my encoding of the molecular structure is missing). I have been searching around trying to find anything built into sklearn to do text feature extraction. There is quite a bit, but it mostly seems to me like they are used for encoding whole sentences or paragraphs. My input would be very long, single words such as:

1-(aminoiminomethyl)-N'-[2,3,6-tri-O-benzoyl-4-O-(2,3,4,6-tetra-O-benzoyl-α-D-glucopyranosyl)-β-D-glucopyranosyl]-
2,4,5-trideoxy-2-[(16-mercapto-1-oxohexadecyl)amino]-1,3-O-(1-methylethylidene)-6-O-undecyl-
octahydro-7-hydroxy-1-[[2-O-(4-hydroxybenzoyl)-α-D-allopyranosyl]oxy]-7-methyl-

And as such, I'm not certain a bag of words or one-hot encoding of the strings would work. Could anyone perhaps point me in the right direction on methodologies or algorithms that might possibly be able to extract features from these strings such that I could train a binary classifier on them?


texts: Is using cross validation on your entire dataset acceptable when dealing with a small sample size

texts:
Normally my practice includes using k-folds cross validation on a subset of my dataset and keep a final test set.

When dealing with an exceptionally small dataset, is using cross validation on the entire dataset acceptable?


texts: R : Counting the number of observations per category

texts:
I'm currently starting out in R and wondering how to count the number of observations per day, per node, per replicate from the below dataset, and store in a different data set.
The original dataset looks like this:

Would like the resulting dataset to look like this:

Can someone help me find out how I could do this in R?
Thanks


texts: Learning curve goes down after converge?

texts: I trained an agent with policy gradient and the learning curve starts to decrease  after it converges. I am wondering if this is overfitting or if this is due to some other issues?


texts: Does not using more filters in deeper CNN creates more images?

texts:
For example, we have applied 32 filters to a single image. Then it created 32 different images (stack of convolutional values).
And in the second layer, if we apply 64 filters, are all these filters going to be applied on all those 32 images? If so, then it will create 64*32 numbers of output or  I am understanding wrong?
I have become confused because when I studied keras documentation, it says that using 64 filters it will create 64 outputs. If anybody enlights me on how the second or deeper layer works in CNN briefly it will be helpful for me.


texts: Counting Number of Parameters in Neural Networks

texts:
Note: This is an academics based problem.

So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of size 5), what will the total number of parameters in this network? How can we compute this?

Additionally, how do we know what shapes are they weights of? How can we determine which activation functions are suitable for such a neural network?

My idea was that (20*10) + (10*5) + (biases = 10+5) = 265. So 265 should be the number of parameters. For shapes/activation functions, from what I understand, it just depends on the data, no? Couldn't think of any way to directly predict it from this limited information


texts: How do I train a contextual bandit policy?

texts:
Say I'm attempting to improve click-through rates on videos on my website. I've been reading the literature on contextual bandits and came across the Microsoft MWT white paper. I believe this is the right method for this case. However, I'm a bit confused on the details of the policy exploration and policy training (Section 3.2 and 3.3).

My first question: Is there a difference between a policy and exploration policy? I want to say no, but want confirmation.

My second question pertains to learning a policy. As an example, say that I initially used an $\epsilon$-greedy exploration policy and collected user click-log data. The paper states that (offline) I want to find a policy out of all allowed policies that approximately maximizes the estimated expected reward (Inverse Propensity Scoring estimator)  $\mu_{ips}(\pi)$ for policy $\pi$ -- Eqn. 4 in Section 3.3. The policy chooses an action $a\in A$ (e.g. a video to serve to the user) given a context $x$ (e.g. user attributes and perhaps attributes of the article/video currently being watched) and an outcome (click or no-click) is observed. The authors propose to reduce the policy training to a cost-sensitive classification problem (e.g. logistic regression, decision trees, or neural nets) where each policy $\pi$ is viewed as a classifier (i.e. for context $x_{i}$, a given policy $\pi$ chooses action $a_{i}$).

However, it's not clear to me what the classification task is. Does it mean that I can use something like a neural net whose output is the policy that minimizes the cost? Is this already performed in ML packages like VW? If so, how?


texts: Xgboost interpretation: shouldn't cover, frequency, and gain be similar?

texts:
I was surprised to see the results of my feature importance table from my xgboost model. Based on the tutorials that I've seen online, gain/cover/frequency seems to be somewhat similar (as I would expect because if a variable improves accuracy, shouldn't it increase in frequency as well?) but my numbers are drastically different. Am I perhaps doing something wrong or is my intuition wrong? Thank you in advance!

Feature       Gain      Cover   Frequency

1:    Var1 0.21943765 0.02821822 0.009433962
2:    Var2 0.18207910 0.05509272 0.066037736
3:    Var3 0.10746529 0.22117710 0.216981132
4:    Var4 0.10294292 0.05267401 0.018867925
5:    Var5 0.06928732 0.10185434 0.141509434
6:    Var6 0.05745753 0.05482397 0.047169811


texts: Keras - Deal with inputs of 2 different shapes

texts:
I'm trying to make a model that deals with categorical data of the shape (891,4,20) and numerical data of the shape (891,4).
To explain further, the categorical data has 891 sequences, 4 different types of categorical, and each is one-hot encoded to be a length of 20.
The numerical data has 891 sequences and have 4 different types.
Here's my attempt so far:

def get_model():
    # Model
    categorical_input = Input(shape=(891,4,20), dtype='float32', name='categorical_input')
    numberic_input = Input(shape=(891,4), dtype='float32', name='numberic_input')

    normalisation = BatchNormalization()(numberic_input)

    dense1 = Dense(16,activation='relu')(categorical_input)
    dense2 = Dense(16,activation='relu')(normalisation)

    concatenated = concatenate([dense1, dense2])

    result = Dense(1,activation='sigmoid')(concatenated)

    model_fnc = Model([categorical_input, numberic_input], result)
    model_fnc.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['acc'])

    return model_fnc

But it seems like Keras won't allow that because of the input shape of categorical and numeric inputs are different. Here's the error message:

ValueError: A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 891, 4, 16), (None, 891, 16)]

I thought that after going through dense1 and dense2 layers, the output of each of those layers should be of the shape (16,), which should be matching for concatenate. Can someone suggest a solution and explain why I am wrong? Thanks!


texts: How to select a model based on statistical analyses?

texts: I've been statistically validating the performance of different Deep Learning models in classifying parasitized and normal cells. In the process, I could find that there is no statistically significant difference in performance between the pre-trained CNN models. In the absence of such significance, which model am I supposed to choose?


texts: How to learn from multiple data sources with different input variables but the same underlying pattern?

texts:
I will explain with an example:

Let's say you have 2 factories that produce pulp paper. Each have similar processes where the laws of physics give the same outcome. Now let's say this 2 factories have equipment and sensors from different manufacturers, so the output of those sensors is not comparable in any way (different number of variables, different metric system etc.). Although for both factories I can caculate the output easily and determine the learning target in a comparable way (eg. metric tonnes of paper).

Is there a way of using deep learning to learn from both datasets at the same time? I mean increase the predictive power upon a sample from factory 1 due to insights on factory 2?

What about having 3 DNN, 2 for reducing feature representation and standardizing output representation and the third one for learning the general pattern common to both and predicting the final output?


texts: Image segmentation network to extract questions from an image of a test paper?

texts:
This is the sample document ->

I want to extract questions along with the options. There are other question papers as which have questions with diagrams in them. I want to be able to extract them as well.
End goal is to store them in a question bank.
I am planning to run an image segmentation algorithm to extract the questions along with their options.
Will that be feasible?
Suggestions are more than welcome.


texts: Why does my model learn with Ragged Tensors but not Dense Tensors?

texts:
I have a string of letters that follow a "grammar." I also have boolean labels on my training set of whether the string follows "the grammar" or not. Basically, my model is trying to learn determine if a string of letters follows the rules. It's a fairly simple problem (I got it out of a textbook).
I am generating my dataset like this:
def generate_dataset(size):
    good_strings = [string_to_ids(generate_string(embedded_reber_grammar))
                    for _ in range(size // 2)]
    bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))
                   for _ in range(size - size // 2)]
    all_strings = good_strings + bad_strings
    X = tf.ragged.constant(all_strings, ragged_rank=1)

    # X = X.to_tensor(default_value=0)

    y = np.array([[1.] for _ in range(len(good_strings))] +
                 [[0.] for _ in range(len(bad_strings))])
    return X, y

Notice the line X = X.to_tensor(default_value=0). If this line is commented out, my model learns just fine. However, if it is not commented out, it fails to learn and the validation set performs the same as chance (50-50).
Here is my actual model:
np.random.seed(42)
tf.random.set_seed(42)

embedding_size = 5

model = keras.models.Sequential([
    keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),
    keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS) + 1, output_dim=embedding_size),
    keras.layers.GRU(30),
    keras.layers.Dense(1, activation="sigmoid")
])
optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)
model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(X_train, y_train, epochs=5, validation_data=(X_valid, y_valid))

I am using 0 as the default value for the dense tensors. The strings_to_ids doesn't use 0 for any of the values but instead starts at 1. Also, when I switch to using a Dense tensor I change ragged=True to False. I have no idea why using a dense tensor causes the model to fail, as I've used dense tensors before in similar exercises.
For additional details, see the solution from the book (exercise 8) or my own colab notebook. As a disclaimer: I am studying ML for fun, not for work or school.


texts: Reverse classification model: Is it possible?


texts: How to test trained PCA used for compression?

texts:
I am working on an exercise for using PCA for compression of images and I don't quite understand how to use it on the test data:

I have 300 images of hand drawn sixes, represented by 28x28 matrices, in the train data, and I have used PCA to find an appropriate low dimensional representation of these images (26 dimensions yields me the sought after 90% threshold), giving me a 300x26 Matrix, that I can use to project my images into that space

Now I have to test this with my test data of 10 similar images - so I have to project them into the same space.

Because I can't just use the train space (dimensions don't agree), if I understood correctly, I should run another PCA to find the Principle Components for these new images, but project them into the 26 dimensions as identified by my train PCA (PCA on just the test suggests that 6 dimensions would suffice, but I want the more accurate 26 from my wide range of training data)

But here is where I'm struggling: how do I centre the test data? To centralise the data before using PCA or the dual PCA, I deduct the mean from the data. When building my test PCA should I deduct the mean of the train data, or the mean of the test data?


texts: The difference between `Dense` and `TimeDistributedDense` of `Keras`

texts:
I am still confused about the difference between Dense and TimeDistributedDense of Keras even though there are already some similar questions asked here and here. People are discussing a lot but no common-agreed conclusions.

And even though, here, @fchollet stated that:

  TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

I still need detailed illustration about what exactly the difference between them.


texts: Feature Encoding for team based sports data

texts:
I am currently playing around with Keras and try to use it with various datasets. Now I have a small datasets of football game results.

date, home_team, away_team, goals_home_team, goals_away_team

Predicting the goals is probably too hard so I combined them into a single feature outcome (win, draw, loss).

date, home_team, away_team, outcome

Using random forest or decision trees, I could simply leave the teams as they are but for a NN I need some encoding. Here is my problem. Which encoding should be used for categorical features that represent a difference in the dataset. Put another way, how would the network know which team is e.g. the winning team.
If I use one-hot encoding, how would the network learn that the outcome is related to the home_team? So I thought about using a label encoder, but that would put an order to the teams.

Is there a "best practice" on how to encode this sort of data?


texts: Reinforcement Learning in NLP for chatbots

texts:
Is anyone aware of any successful implementation of reinforcement learning for NLP. I am looking to for chatbots which can learn automatically.

Tried searching internet but found very few articles like Reinforcement Learning For Natural Language Processing - Medium or papers like A Survey of Reinforcement Learning Informed by Natural Language.

But none of them provides a robust code which shows that it is working. Kindly suggest.


texts: Construct word2vec (CBOW) training data from beginning of sentence

texts:
When constructing training data for CBOW, Mikolov et al. suggest using the word from the center of a context window. What is the "best" approach to capturing words at the beginning/end of a sentence (I put best in quotes because I'm sure this depends on the task). Implementations I see online do something like the this:
for i in range(2, len(raw_text) - 2):
    context = [raw_text[i - 2], raw_text[i - 1],
               raw_text[i + 1], raw_text[i + 2]]

I see two issues arising from this approach.

Issue 1: The approach gives imbalanced focus to the middle of the sentence. For example, the first word of the sentence can only appear in 1 context window and will never appear as the target word. Compare this to the 4th word in the sentence which will appear in 4 context windows and will also be a target word. This will be an issue as some words appear frequently at the beginning of sentences (i.e. however, thus, etc.). Wouldn't this approach minimize their use?
Issue 2: Sentences with 4 or fewer words are completely ignored, and the importance of short sentences is minimized. For example, a sentence with 5 words can only contribute one training sample while a sentence of length 8 will contribute 4 training samples.

Can anyone offer insight as to how much these issues affect the results or any alternative approaches for constructing the training data? (I considered letting the first word be the target word and using the next N words as the context, but this creates issues of it's own).
Note: I also asked this question on Stack Overflow: https://stackoverflow.com/questions/63747999/construct-word2vec-cbow-training-data-from-beginning-of-sentence


texts: Dynamic rnn for toysequence classification

texts:
i am trying to use the dynamic rnn for toy sequence clasification . the code is copied from github link as:https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py.
The errors i am getting during predc function are as:

ValueError                                Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
    509                 as_ref=input_arg.is_ref,
--> 510                 preferred_dtype=default_dtype)
    511           except TypeError as err:

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
   1103     if ret is None:
-> 1104       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1105

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _TensorTensorConversionFunction(t, dtype, name, as_ref)
    946         "Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
--> 947         (dtype.name, t.dtype.name, str(t)))
    948   return t

ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("Reshape_4:0", shape=(?, 1), dtype=float32)'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
 in ()
----> 1 pred = dynamicRNN(x, seqlen, weights, biases)

 in dynamicRNN(x, seqlen, weights, biases)
      2     x = tf.transpose(x, [1, 0, 2])
      3     x = tf.reshape(x, [-1, 1])
----> 4     x = tf.split(0, seq_max_len, x)
      5     lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden)
      6     outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32, sequence_length=seqlen)

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in split(value, num_or_size_splits, axis, num, name)
   1389   if size_splits._rank() == 0 and size_splits.dtype.is_integer:
   1390     return gen_array_ops.split(
-> 1391         axis=axis, num_split=num_or_size_splits, value=value, name=name)
   1392
   1393   if num is None:

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py in split(axis, value, num_split, name)
   7707     num_split = _execute.make_int(num_split, "num_split")
   7708     _, _, _op = _op_def_lib._apply_op_helper(
-> 7709         "Split", split_dim=axis, value=value, num_split=num_split, name=name)
   7710     _result = _op.outputs[:]
   7711     _inputs_flat = _op.inputs

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
    531             if input_arg.type != types_pb2.DT_INVALID:
    532               raise TypeError("%s expected type of %s." %
--> 533                               (prefix, dtypes.as_dtype(input_arg.type).name))
    534             else:
    535               # Update the maps with the default, if needed.

TypeError: Input 'split_dim' of 'Split' Op has type float32 that does not match expected type of int32.


texts: Heavy regression loss for false non 0 prediction

texts:
My regression should predict values  >=0
But a wrongly predicted value >0(e.g. 0.001 instead of 0) is much worse then a a slight missprediction of 0.001 (e.g. 0.002 instead of 0.003)

I am thinking about a costume loss function that weights the false non zeros to return a big loss.

Is there a more elegant way ?

Edit:

this is my unelegant sollution:

def custom_loss(y,yh):
    if y == 0:
        loss = tf.math.minimum(100,yt*100000)
    else :
        loss = losses.mean_squared_error(yh,y)

    return loss