texts: Accuracy changes very little when modifying parameters in a Keras model

texts:
I've seen other similar questions and followed their solutions, to little improvement. I'm making a model to identify the gender of names. As training data I'm using a list of baby names found here: https://www.ssa.gov/oact/babynames/limits.html. I extracted this data to a new data frame, keeping only one instance of those names occurring more than once, and sorted randomly.

Each name string in a column was converted to a numeric array of lenght max_len and normalized by the function:

def text_to_numeric(column, max_len):
    word_characters = []
    for word in column:
        word_characters.append([c for c in word])

    letters_kept = 25
    tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=letters_kept, oov_token='<UNK>')
    tokenizer.fit_on_texts(word_characters)

    word_sequence = tokenizer.texts_to_sequences(word_characters)
    words_pre = tf.keras.preprocessing.sequence.pad_sequences(word_sequence, maxlen=max_len,padding="pre")
    words_pre = tf.keras.utils.normalize(input_data)

    return list(words_pre)

The expected output is an array of 2 element list where [1,0] means “Male” and [0,1] means “Female”. The model, where data_file contains processed names and labels, looks like this:

input_length, input_data, output_data = data_reader(data_file)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(100, input_dim=input_length, activation='relu'))
model.add(tf.keras.layers.Dense(100, activation='relu'))
model.add(tf.keras.layers.Dense(2, activation='softmax'))

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])

model.fit(input_data, output_data, epochs=30, verbose=1, validation_split=0.1)

No matter what, I always get an accuracy of around 75%. I don't know how to choose the model parameters, but I’ve tried with many combinations and the accuracy changes little. So far I've tried: normalizing input, balancing the input dataset so there are the same number of men and women, changing the optimizer, defining an optimizer and change the learning rate, changing layer number, nodes per layer and activation function, increasing number of epochs.

All of this with no significant change in the model's accuracy. Am I missing something or doing something completely wrong? Is this accuracy as good as it gets?


texts:
I'm looking for scalable tools to build kNN graph over sparse data points.

The dimension and number of data points can be both up to millions.

What I have tried already:

sklearn.neighbors.kneighbors_graph: which does brute-force search for sparse data, giving quadratic time.
flann: only supports dense arrays
pysparnn: the running time is not very satisfatory (maybe because it's written in Python)
knn search in mlpack: which only supports dense data
scipy.spatial.KDTree: which converts the sparse data to dense one
SparseLSH: which is implemented in Python, so I'm not quite sure about the scalability
elasticsearch: it seems to only support indexing documents, instead of sparse features.

the reason I thought of elasticsearch is: knn over sparse data can be framed as retrieving the top-k "documents" in IR.

Thanks for any comments/answers :)


texts: Calculating optimal path(s) to increase probability of a classification

texts:
I have trained several classifiers using Python's scikit-learn which are fairly accurate when applied on a test set at identifying different classes with a standardized set of input features. These different classifiers provide a certain probability for the classification.

The input features are controllable physical parameters that I am measuring (e.g. temperature, volume) which intricately influence an output which can essentially be either 1 or 0 (and others in mutli-class cases). I can already do basic identification, but what I am curious about is: given an initial feature vector starting in class 0, are there known methods to find the optimal ways to change my input features so as to increase my probability of going into class 1? The input feature space has a high number of dimensions, and there may be certain constraints on the inputs (e.g. temperature cannot exceed a certain value if volume is kept at a particular value).


texts:
I have 100 2D lists a0 to a99, it is easy to convert them into arrays (I verified already):

print(np.array(a0).shape)   # (2,150)
print(np.array(a1).shape)   # (5,150)
print(np.array(a2).shape)   # (10, 150)
# the sizes of a0 to a99 vary from (1,150) to (10, 150) only

I want to combine these 100 3D arrays into ONE 3D array, for example combined_array:

print(combined_array.shape)  # (100,10,150)
print(combined_array[0].shape) # (2,150)
print(conbined_array[1].shape) # (5,150)
print(combined_array[2].shape) # (10,150)

I use Python 3.7.


texts: Mapping 4 Dimensional array to predicted output text

texts:
Iv been studying machine learning but im struggling with some concepts and cant seem to find particular answers to the question of how theoretical data is mapped into non classification categories.

To be specific , in the problem I am addressing : I have a 4 dimensional image array with R,G,B and Alpha values. I wish to map those input values to expected output values in JSON to train a model which will automatically parse an image into a JSON command based representation:

{
  "name": "img",
  "attrs": {
    "height": "210",
    "width": "400"
  },
  "childs": [
    {
      "name": "path",
      "attrs": {
        "p": "M150 0 L75 200 L225 200 Z"
      }
    }
  ]
}

The issue is I am struggling to conceptually understand how you can map the byte array input or RGB array input values to the expected JSON output to generate the model as the output is a text based prediction alteration of values rather than a classification of particular pixels with a finite set of results.

I know it is possible as I have seen Convolutional neural networks used to map-predict 2 dimensional image data to 3 dimensional model data in OBJ format which is a similar problem to solve however I am uncertain how the mapping from pixel data to expected text based 3D vectors was performed.

Can anyone recommend any articles or outline their experience in representing non-classification/finite result output or atleast point me in the direction of which category of problems this falls under as every example of CNN's etc seem to fall into classification output problems such as object identification rather than numerical prediction output.


texts:
I am trying to train a LSTM on an NER dataset which contains multiple features. But I'm having trouble understanding how to pad multiple features. The dataset contains the following 3 features per row: word, POS, shape. Where POS is part-of-speech tagging and shape is categorical numerical, lowercase, capitalized etc.

So for the LSTM my timesteps need to be the same. In this case its the length of the sentence. The words I can just pad. Either by just adding _PAD_ or something, or in the case of distributed embeddings use the min/max embedding technique.

But I don't know how to pad the additional (categorical) features. How do I pad the features POS and shape? Do I just add another category and use that for padding? If yes: in the case of embedded paddings, using the min/max embedding, would the additional padding category for the POS feature not train the LSTM to learn the min/max embedding on the padded POS?

An example notebook of the dataset using LSTM can be found here. Which pads the words but removes all the other features. How can I expand on this notebook, by adding POS as a feature and use padding?

Should I use one-hot encoding for the categorical values and just use a vector with zero's? If yes, how does one make a vector of zero's in python as one-hot encoding.


texts: Can I create a layer with multiple rnn cell ? [question about a paper]

texts:
I am trying to implement https://dl.acm.org/doi/pdf/10.1145/3269206.3271794  .
Structure:

As it said:

In particular, we integrate the embedding vectors learned from each
individual recurrent encoder into a new conclusive embedding vector to
jointly consider various time series patterns with different ⟨α, β⟩
configurations

For my understanding, it use multiple individual  rnn cell to process different timeseries, then concat all hidden states together to form a 3D input which can use 2d conv extract features .
But I didn't see there is a way to create multiple rnn cells in same layer , do I misunderstand??
If not , could you please give me a guide or an example ?


texts: How does XGBoost compute the probabilities in predict_proba()?

texts:
I'm using the sklearn wrapper for XGBoost.
I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed.

In random forest for example, I understand it reflects the mean of proportions of the samples belonging to the class among the relevant leaves of all the trees.

However in XGBoost I couldn't understand the computation from the documentation or the code. Shouldn't it give different weights for each tree?


texts: Unable to understand the usage of labels argument in sklearn.metrics.f1_score

texts:
I am trying to model a dataset with RandomForest Classifier.
My dataset has 3 classes viz. A, B, C. 'A' is the negative class and 'B' and 'C' are positive classes.

In GridSearch I wanted to optimize on F1-score since the number of samples in all the classes are not evenly distributed and class 'A' has the highest number of samples.

That is where I wanted to understand the usage of labels argument.
The doc says:

  labels : list, optional The set of labels to include when average !=
  'binary', and their order if average is None. Labels present in the
  data can be excluded, for example to calculate a multiclass average
  ignoring a majority negative class, while labels not present in the
  data will result in 0 components in a macro average.

I could not understand it properly. Does it mean, In my screnario I should have labels as labels = ['B', 'C'], just the positive class?
Kindly Help

custom_scoring = make_scorer(f1_score, labels=[???],average='weighted')
clf = RandomForestClassifier(class_weight='balanced', random_state=args.random_state)
grid_search = GridSearchCV(clf, param_grid=param_grid, n_jobs=20, scoring=custom_scoring)


texts: How to arrange the sets to predict y on x in time series?

texts:
I'm performing my first NN with my own data and while I was already tuning the parameters I stumbled over an aspect which confuses me now such that I'm not sure what is right and what is wrong..

Given this (head of the) data df_train1_raw:

                     timestamp       A_phsA
2018-02-05 14:00:00 1.517839e+09    856.436487
2018-02-05 15:00:00 1.517843e+09    859.653339
2018-02-05 16:00:00 1.517846e+09    836.635463
2018-02-05 17:00:00 1.517850e+09    801.097284
2018-02-05 18:00:00 1.517854e+09    794.855960
                       (...)

The timestamp column is basically the index just converted into numeric so I can use these time information for the nn model.

Goal: Predict A_phsA on timestamp

First, I create the train and test sets:

# Prepare data
X_train1_raw = df_train1_raw.values
y_train1_raw = X_train1_raw

# Split data into appropriate sets
## Standardize and scale data

scaler = StandardScaler()
tscv = TimeSeriesSplit(n_splits = 5)
pyplot.figure(1)
index = 1

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

This looks reasonable. When I plot the loss and val_loss values of the nn later it also looks reasonable.

But what struggles me actually is this line at the beginning:

y_train1_raw = X_train1_raw

I can't tell if it is plain stupid or if I can't get my head around it anymore. The reason is, when I look for example at KFold:

X =  list(range(10))
print (X)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y = [x*x for x in X]
print (y)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

kf = KFold(n_splits=5)
X = np.array(X)
y = np.array(y)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print(“X_test: ", X_test)

they have different X and y. Which makes sense I would say.
But when I adjust my code accordingly

X_train1_raw = df_train1_raw.iloc[:, 1].values
y_train1_raw = df_train1_raw.iloc[:, 2].values

I get an error when performing

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

  ValueError: Expected 2D array, got 1D array instead:
  array=[1.5178392e+09 1.5178428e+09 1.5178464e+09 1.5178500e+09
  1.5178536e+09

And I don't understand why. Which part is not correct?

edit: Is it scaler.fit_transform() ?


texts: NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

texts:
I am new to NLP and I just finished reading the paper "Attention is all you need".
I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these visualizations were produced:

I understand that the output of the self-attention sub-layer (for a single head) is a vector of size d_v that is a weighted sum of all the value vectors. Than how do they use this vector to calculate the strengths of the relations between the positions?
Any help and insight would be appreciated, thanks a lot!


texts: Sklearn LocalOutlierFactor contamination parameter

texts:
Can anyone provide an intuitive explanation of the choice of contamination parameter used in sklearn's LocalOutlierFactor implementation when contamination="auto" ?

The sklearn guide suggests "as described in the paper" but I couldn't find anything obvious. Thanks.


texts: Training pipelines where featurization/NLP is more expensive than backprop

texts:
I'm working on a document classification project and I'm using a neural net in tensorflow, where the features are 300-dimensional word embeddings, either from fastext or word2vec (yes I know that there are newer better embeddings -- this is for backwards compatibility).

Each of the documents is very long -- 3000+ tokens.

Featurization involves tokenizing the document, embedding it, and stacking the embeddings into an array of dimension batchsize, max_doc_length, features.  It takes more than 5x as long to make a batchfile (on a CPU) as it does to do one iteration of backprop (also on a CPU).  My problem would be worse on a GPU, if backprop went faster.

My batches are almost 1GB in size, so I'd use almost a whole TB if I were to write everything to disk before starting backprop.  That's too much, especially given that I want to try different featurization strategies.

So, what are some good solutions to this problem?  Are there software packages that are designed to solve this problem?

Right now I've got two processes going in parallel -- one that makes batches and saves them to the disk (in parallel) and the other which loads them, uses them for backprop, then deletes them.  It feels sort of janky, and I can't be the first person who has faced this issue.

Suggestions appreciated!


texts: NLP Style Transfer: Using my iMessage data to do predictive text generation in the style of my friends and family

texts:
So I wanted to use my iMessage data to train a model to see how the people I frequently text would fill in the blanks in sentences like "I am going to the [MASK]" or "I want to [MASK] you." As an iOS engineer (relatively new to ML), I exported my iMessage data as JSON and built an MLM model that incorporates BERT. Basically, the inputs to my model are 1) Message, 2) SenderId, 3) Bert Attention Mask.
The issue I am having is that, when I train and predict with the model, it outputs the same results for every sender -- as if it's completely ignoring the SenderId. You can see my model summary here.
I only have about 22MB of messages (spread mostly amongst 13 contacts including myself), and I'm not sure that is enough data to extract a style for each person. For the contacts I'm trying to imitate, I only have between 200 and 2000 messages. I am using only 9000 messages total.
Any ideas on what I could do to "enhance" the variance across my contacts? Is there just not enough messages? Or is there a tweak I should make to my model? The loss seems to plateau only after like 10-30 epochs, with each epoch taking slightly less than a minute. Also, I am freezing the BERT layers (if I don't, it starts outputting nonsense).


texts: Misleading Pearson Correlation Coefficient?

texts:
I'm looking at the correlation between day of week and number of page views my website gets.  Because day of week is a discrete, non-numeric variable, I've mapped Sunday to 0, Monday to 1, ..., Saturday to 6.

Here's my concern / question: Isn't it possible to get a low Pearson coefficient when, in fact, there is a relationship between day of week and page views, it's just a bit more complicated.

For example, it might be that Mondays and Fridays are highly correlated with page views and the middle of the week is negatively correlated.  So this might average out to a correlation near 0 when there's a real pattern there.

Is there a standard way of trying to figure out what's going on here?  My mapping of days to numbers seems simplistic...

Thanks!


texts:
I wanted to ask you which coding language I should use. I am pretty new to coding and could really use some help.
As I mentioned above I want to create a survey and afterward store my results in a database.
Details:
The survey should be available for a longer period of time (a few years) and the results should be transferred daily to the database.
The survey starts with a general identification (name, age, gender, etc.).
An ID is then assigned to the participants (in case they want to add more answers to the survey AFTER they have completed it).
Afterward, the identification of the actual survey takes place. In this part, I would like to query recurring events. For example, classes attended at the university with subcategories (date, GPA, etc.) Each candidate has attended 1 to x classes.
Here is my first problem, how can I achieve that every candidate can enter all his classes?
For example, is there the following possibility that I ask the following question after each class: Have you attended any other classes? If "NO", the survey is declared closed, if "YES", he can enter another class.
The next problem, I have is that I want to make sure that different people name the same class the same way. So I want to have the possibility that as soon as you are participant 2, you get the already entered classes of participant 1 as a suggestion.
Then I would like to transfer the information into a database. (if possible with MySQL).
And if possible I also would like to visualize it.
I just wanted to ask you which coding language you would recommend.
So far I have little coding experience with Python and Java.


texts: Ordering of standardization, pca, and/or tfidf for neural network

texts:
I have 60k rows of text data. I have tokenized it into 55k columns. I am using a neural network to classify the data but have some questions about how to order my preprocessing steps. I have too much data for my hardware (doesn't fit in memory/too slow) so I am using PCA to reduce dimensions.

Obviously, I need to scale before PCA. I am currently standardizing the columns, but I am wondering if I can use tfidf instead of standardization. Some rows have 50k+ tokens while others have <1k tokens so I am worried these rows have undue influence on the outcome of scaling which will trickle down the pipeline. Is this a good/bad idea? Would I maybe use tfidf then standardize before PCA?

Generally neural nets prefer standardized data. After PCA the first few columns have much greater magnitude than the rest b/c they capture so much variance. Should I standardize after PCA and before training? The reason for standardizing before training is so no feature has bigger influence on the model just b/c the scale is bigger, but isn't PCA telling me that the first few features are actually more important? FWIW, I've tried both and not scaling seems a little better.

What about performing tfidf after PCA and before training? Again, rows with 50k+ tokens will prefer a network with orders of magnitude larger weights than rows with <1k tokens. Wouldn't it be hard for the network to set weights for both types of rows?

Diagram for clarity: data -> tokenize -> ?standardize/tfidf? -> PCA -> ?standardize/tfidf? -> neural net


texts: How to choose the number of output channels in a convolutional layer?

texts:
I'm following a pytorch tutorial where for a tensor of shape [8,3,32,32], where 8 is the batch size, 3 the number of channels and 32 x 32, the pixel size, they define the first convolutional layer as nn.Conv2d(3, 16, 5 ), where 3 is the input size, 16 the output size and 5 the kernel size and it works fine.

in_size = 3
hid1_size = 16
hid2_size = 32
out_size = len(labels)
k_conv_size = 5

class ConvNet(nn.Module):

    def __init__(self):
        super(ConvNet, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(in_size, hid1_size, k_conv_size ),
            nn.BatchNorm2d(hid1_size),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))

        self.layer2 = nn.Sequential(
            nn.Conv2d(hid1_size, hid2_size, k_conv_size),
            nn.BatchNorm2d(hid2_size),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))

        self.fc = nn.Linear(hid2_size *  k_conv_size * k_conv_size, out_size)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)

        return out

I change the output size from 16 to 32 and that of the next layer from 32 to 64 and it still works.
But when I resize the tensor to have the shape [8, 3, 64, 64], it throws a mismatch error that says size mismatch, m1: [16 x 5408], m2: [800 x 4]
I understand m2 is what it's expecting and m1 is what I'm giving.

But I don't understand where the values of m2 and m1 come from and how to change the hid1_size accordingly.

I understand the relationship between the shape of input data and the neurons in the first layer when building regular linear layers but how to define the relationship between the shape of the input and the number of channels produced by the convolutional layer in cnns?


texts: Smallest Possible Dataset for Text Classification using BERT

texts:
What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis?

~100 examples

~1000 examples

...

~10000000 examples

What are your experiences?


texts: Can anyone recommend some good books or articles on working with time series?

texts:
I've read a couple of guides such as this link. I'm having real trouble getting to grips with how to use time series data effectively.

Anything to help with the intuition ideally with some case studies. I am currently reading Applied predictive modelling so anything with a similar pedagogy would be ideal.


texts: Finding variable values to make equations behave like boolean gate

texts:
I came across following problem:

Consider the following simple perceptron, for which the inputs are the always $1$ feature and two binary features $x_1 ∈ \{0, 1\}$ and $x_2 ∈ \{0, 1\}$. The output $y ∈ \{0, 1\}$.
$$y=\begin{cases}
1,  & \text{if $w_0+w_1.x_1+w_2.x_2>0$ } \\
0, & \text{otherwise}
\end{cases}$$
Choose all the correct option below that the weights $w_0,w_1,w_2$ should satisfy such that $y=(x_1 \text{ AND } x_2 )$.
(A) $w_0>0$      (b) $w_0≤0$   (c) $w_1>0$        (d) $w_2>0$

I tried to solve the problem and came up with following:

(Eqn# stands for Equation# in above table)
Eqn#4 $= w_0≤0$  (Option B)
Eqn#2 $→w_0≤-w_1$    (Eqn#5)
Eqn#3 $→w_0≤-w_2$    (Eqn#6)
Eqn#1 $→ 0>-w_1-w_2-w_0$
$→ 0>-w_1+(non-ve)$    … from Eqn#3
$→ -(non-ve)>-w_1$
$→ -ve>-w_1$
$→ w_1>0$ (Option C)
Similarly from Equation 4 and 2, it can be proven that $w_2>0$ (Option D)

But I feel something is wrong with above. I feel option B contradicts with both option C and option D. For example, option C and Eqn#5 makes $w_0=0$ impossible which is allowed by option B. Where did I made mistake? I feel its incorrect to directly conclude $w_0\leq 0$ from Eqn#4 and "stop". What I mean is that I have to continuously keep checking if all facts that I conclude lead to desired effect "together". For example, once I got that option (c) is correct, I have to combine Option C fact with Eqn#4 fact to conclude that $w_0<0$. So finding solution to such problems is kind of continuous process till we get all equations which satisfy each other and target goal (AND gate here). Q1. Am I correct with this? Or I am unnecessarily making things difficult and there is simpler and more importantly "straight forward" way (not involving continuously checking if all equations satisfy each other) to find such solutions?
Q2. This does also mean that Option B should have $<$ instead of $\leq$. Am I correct with this too?
PS:
After getting confused like me, some of my friends have started saying it should be $\geq$ instead of $\gt$:
$$y=\begin{cases}
1,  & \text{if $w_0+w_1.x_1+w_2.x_2\color{red}{\geq}0$ } \\
0, & \text{otherwise}
\end{cases}$$
But I dont feel so as I can still find correct answer with given problem with conclusion that option B should have $<$ instead of $\leq$. Also I dont feel its correct for a function to interpret $0$ as $True$. Q3. Am I correct with this too? (problem doesnt need correction, but option B)


texts: Sentiment analysis with continuous scale for responses

texts:
I am trying to create a sentiment analysis system using deep learning (with R).
I have a review base of customer reviews which for each review the customer make a mark (from 0 to 10).

How can I do this, using marks in the range 0..10?  Is it possible?  In some tutorials, I see that authors use 0, 1 or -1 (0 for neutral, 1, positive and -1 for negative), and I'm not sure how to adjust to marks in the range 0..10.


texts: How many observations in a neural networks dataset?

texts:
I started to study and programming in neural networks for a little while now, but I never read about the minimum number of observations one must collect in a dataset to get robust results. Of course, more observations better results, but, Does exist an empirical or theoretical relationship between variables and observations number?

I mean, neither in econometrics you can compute the minimum number of observations, but it does exist some rule of thumbs that relies the number of exogenous variables to the target variable.

I wonder if there is something similar to that in neural networks too, but, till now, browsing on the internet, I did not find anything of useful.

Any ideas, advises or hint will be appreciated.


texts: Categorical Clustering of Users Reading Habits

texts:
I have a data set with a set of users and a history of documents they have read, all the documents have metadata attributes (think topic, country, author) associated with them.

I want to cluster the users based on their reading history per one of the metadata attributes associated with the documents they have clicked on. This attribute has 7 possible categorical values and I want to prove a hypothesis that there is a pattern to the users' reading habits and they can be divided into seven clusters. In other words, that users will often read documents based on one of the 7 possible values in the particular metadata category.

Anyone have any advice on how to do this especially in R, like specific packages? I realize that the standard k-means algorithm won't work well in this case since the data is categorical and not numeric.


texts: Does the model(best fitting line/curve) changes when the training data is changed in the cross validation?

texts:
From my understanding - a machine learning algorithm goes through the inputs (independent variables) and predicts the output (dependent variable). I believe, what line/curve would best define the training data will depend on the training data. When it decides the best line or curve, the new inputs are plotted and based on plotting, the target variables are found.
My question is, during the cross-validation, the training data changes, due to the difference in splits, shouldn't that change the best fitting line, and shouldn't that make K different models(=number of splits)? If yes (it makes K different model), then what use would that be? If no, then where am I going wrong?


texts: How to validate the model when using Smote technique?

texts:
I am using Smote to treat Class Imbalance and I apply Smote only on the train data. I fit a model using balanced data("Smoted" data).
Now I need to check recall and precision for train and test data(to see if the model is stable or not). For checking recall and precision in train data, should I use the balanced train data(on which Smote was applied) or the imbalanced train (original train) data ?


texts: High dimensional data stream summarization and processing

texts:
Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection?

In fact, I investigated the different methods for data stream summarization (sampling, histograms, sketches, wavelets, sliding windows) and am confused about the choice. In fact, I noticed that sampling and sliding windows are general purpose and keep the raw data, while the others are task-specific and make transformations to the data.

I am interested in the first case but it may be more consuming.


texts:
I have two questions about dense prediction in YOLOv4 paper

What does it mean by the (hard negative, online hard) example mining method is not applicable to one-stage object detector, because this kind of detector belongs to the dense prediction architecture ?
Why dense prediction does not belong to two-stage detector ?


texts: Are there Machine Learning Models for Networks?

texts:
I am working on a regression problem, where the goal is to estimate historic traffic volumes throughout a transportation network. I have traffic counters at 100 locations, so a model can learn the relation between traffic volumes and a number of explanatory variables (e.g., speeds, road characteristics, weather). Afterwards, I can apply the model to estimate historic traffic volumes in places where I don't have traffic counters.

My neural network works reasonably well, but I am wondering if there are machine learning models that could explicitly account for the topology of my road network and the fact that traffic on neighboring road links is highly correlated. I could add "traffic volume at the closest traffic counter" as an input variable to my ANN, but I am wondering if there is a more intelligent approach.

In this regard, I came across Bayesian networks, which can account for the network topology and correlation. However, they seem applicable to cases when we have sensors at 100 locations and we want to predict the traffic state (at these 100 locations) at a future time point. On the other hand, I have measurements at 100 locations and are looking to estimate traffic at a different location for the same time point.

Any suggestion is much appreciated!


texts: why use one regularisation technique over another?

texts:
why should I prefer L1 over L2, in fully-connected-layer or convolution?
why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and one would also have the flexibility to use different regularisation techniques at each layer?
A lot of the times, trying out different techniques and comparing performance may cost time and money. So, when should I use(or prefer) one regularisation technique over other?


texts:
I am trying to replicate the below sql query in R

select a.*, b.key from Table1 a
LEFT OUTER JOIN Table2 b
on a.key = b.key where b.key is null

I have read through this post however I am still struggling to code my specific case. https://stackoverflow.com/questions/1299871/how-to-join-merge-data-frames-inner-outer-left-right

I have tried the below but the result doesnot allow me to filter for b.key IS NULL

LoansToInsert_stg1 <- merge(x = Prior_stg2, y = BankLoans_stg2,
                            by = "Account_ID", all.x = TRUE)

Any insights?

Example:

Key1 <- c("A1","A2","A3","A4","A5")
Key2 <- c("A1","A2","A3","B4","B5")
BV1 <- c(100, 200, 300, 400, 500)
BV2 <- c(150, 250, 350, 450, 550)
df1 <- as.data.frame(cbind(Key1, BV1))
df2 <- as.data.frame(cbind(Key2, BV2))

Expected Output as a new df:

Key1 BV1 Key2 BV2
A1 100 A1 150
A2 200 A2 250
A3 300 A3 350
A4 400 NA NA
A5 500 NA NA


texts: Orange "Scatter plot" widget to excel, best way

texts:
So I have chosen "Orange" for my research as a main app for clustering ant it's analysis.

Excel 2016 is primary app used for dashboard (via "pivot table").

Problem - how to add graphs from "scatter plot" to excel dashboard and sync it at certain time? Is there a way?

Orange provided reporting does not help, maybe there is another - more interactive way? because all I can get now is images via hyperlinks, which is not interactive.


texts: TensorFlow Sigmoid activation function as output layer - value interpretation

texts:
My TensorFlow model has the following structure. It aims to solve a binary classification problem where the labels are either 0 or 1. The output layer uses a sigmoid activation function with 1 output.

model = keras.Sequential([
    layers.Dense(10, activation='relu', input_shape=[len(train_dataset.keys())]),
    layers.Dense(1, activation='sigmoid')
  ])
  optimizer = 'adam'

  model.compile(loss='binary_crossentropy',
                optimizer=optimizer,
                metrics=[tf.keras.metrics.Precision(), tf.keras.metrics.Recall(), tf.keras.metrics.Accuracy()])

The output given is an array of dtype=float32 numbers that lie between 0 and 1.

array([[9.5879245e-01],
       [3.6847022e-01],
       [3.4174323e-04],
       ...,
       [2.6283860e-03],
       [3.2045375e-04],
       [1.0798702e-03]], dtype=float32)

The Tensorflow tutorials state that
"Using the sigmoid activation function, this value is a float between 0 and 1 and represents a probability, or confidence level".
https://www.tensorflow.org/tutorials/keras/text_classification

My question is:

Do I interpret the float values from output as:

"How likely it belongs to the first class label - in this case class 0 is my first class label?" e.g. model.predict() yields 0.99998 and therefore has a 99% chance of belonging to my first class label (class 0) and 1% belonging to the other class (class 1).

or

"The closer the output from model.predict() is to 0.0 the more likely it is class 0 and the closer the output is to 1.0 the more likely it is class 1"


texts:
Just landed on a Entity Resolution (Matching) project.
I am working in python and I was wondering if there's an off-the-shelf implementation of string harmonization functions, including:

removal of trailing spaces
removal of multiple spaces
removal of special characters (eg. ü)
...

All this is easily done with string and Regex operations but I do not want to reinvent the wheel and tap into something already ready (and perhaps learn from that, shall I need to move on to something custom later).
Are there readily available libs for that? Tried nltk, which suggests the use of regex expression, textblob, and googling around but found nothing really helpful so far - probabily because I did not get to use the right keyword.

As an example, it would be useful to have a function that replaces special characters from foreign languages with morphological equivalents - for example "ß" in German with "ss" ("straße" -> "strasse")

Edit: clarification
In my question, I am not really specifying the problem at hand. This is because what I am trying to get to is really a "grammar" that would let me and the other ppl in my team define workflows of harmonization/canonicalisation operations.

I am at the moment Working on this as a chain of functions f: string -> string. Whether some steps are part of it, it will depend on the problem at hand. Workflow will be likely tested empirically (we'll pick the workflow that leads to the best performing algo)


texts: If we have numeric variable, decision trees will use < and > comparisons as splitting criteria. Lets consider this case : If our target variable is 1 for even numeric value, and 0 for odd numeric value. How to deal with this type of variables? How to even identify these type of variables if we have large number of variables? Is there any specific names for these type of variables?


texts: How is the equation for the relation between prediction error, bias, and variance defined?

texts:
I'm reading this article Understanding the BiasVariance Tradeoff. It mentioned:

  If we denote the variable we are trying to predict as $Y$ and our
  covariates as $X$, we may assume that there is a relationship relating
  one to the other such as $Y=f(X)+\epsilon$ where the error term
  $\epsilon$ is normally distributed with a mean of zero like so
  $\epsilon\sim\mathcal{N}(0,\,\sigma_\epsilon)$.

  We may estimate a model $\hat{f}(X)$ of $f(X)$. The expected squared
  prediction error at a point $x$ is: $$Err(x)=E[(Y-\hat{f}(x))^2]$$
  This error may then be decomposed into bias and variance components:
  $$Err(x)=(E[\hat{f}(x)]-f(x))^2+E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big]+\sigma^2_e$$
  $$Err(x)=Bias^2+Variance+Irreducible\ Error$$

I'm wondering how do the last two equations deduct from the first equation?


texts: Statistical loss function for categorical distributions

texts:
For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $[q_1, q_2, \ldots, q_n]$, I have to define a proper loss function measuring the distance between the ground truth $P$ and prediction $Q$.

Since the outputs are probabilistic, I want to use a loss function that amongst others satisfies that the absolute difference between a ground truth $P$ and prediction $Q$ is penalized more as $P$ increases. As an example, $[p,q] = [0.9, 0.8]$ should receive a higher penalty than $[p,q] = [0.8, 0.7]$.

My initial thoughts were to use the Kullback-Leibler divergence, which for $n$-dimensional categorical distributions $P$ and $Q$ is as follows:
$$KL(P||Q) = \sum_{i=1}^{n}p_i\log\big(\frac{p_i}{q_i}\big)$$
A problem that arises when using this divergence is that it is only defined when $q_i=0$ implies $p_i=0$. This problem can be solved by using a 'fuzz factor' $\epsilon>0$:
$$KL_{\epsilon}(P||Q) = \sum_{i=1}^{n}(p_i+\epsilon)\log\big(\frac{p_i+\epsilon}{q_i+\epsilon}\big) $$

I wonder if there are other good metrics that can be used as loss function for this case where the output is a set of parameters from a categorical distribution.


texts: How to condition your neural network to seek a specific amount of output?

texts:
I transformed a time series with collection values from 0 to 100 into a windows of 60 elements with averages and standard deviation, and counts 0-20 ... 80-100 of the next 30 readings, where each minute represents a collection, in a space timeless.
Example:
1 - Divide the data into blocks of 60 elements, where each new block it moves 10 elements
Example: 0 to 60, 10 to 70, 20 to 80 ... etc
2 - Then divide the blocks into 5 parts (12x5 = 60)
3 - Calculate average and deviation of each part
4 - Take the next 30 elements from each block of 60 elements
Example: 60 to 90, 70 to 100, 80 to 110 ... etc
5 - Calculate how many readings there are from 0 to 100, grouping every 20
Example: 0 to 20, 20 to 40, 40 to 60, 60 to 80 and 80 to 100

(0 to 20) 12,18,11,14 = 4
(20 to 40) 20,25,23 = 3
...

The result would be a dataframe like this:

6 - I collect the largest element of each column and divide all elements of the column by it, generating binary values (0,1):
My X consists of mean1,std1 ... mean5,std5
My Y consists of 0-20 ... 80-100
Learning model:
cvscores = []
for train, test in kfold.split(X,Y):

    model = Sequential()
    model.add(Dense(10,
                kernel_regularizer=l2(0.001),
                kernel_initializer=VarianceScaling(),
                activation='sigmoid',
                name='hidden-input'))
    model.add(Dense(5,
                kernel_regularizer=l2(0.001),
                kernel_initializer=VarianceScaling(),
                activation='sigmoid',
                name='output'))

    model.compile(loss='mse', optimizer='adam', metrics=['mse'])

    model.fit(X.iloc[train].values, Y.iloc[train].values, epochs=100, batch_size=5, verbose = 0,
              validation_data=(X.iloc[test].values, Y.iloc[test].values))

    scores = model.evaluate(X.iloc[test].values, Y.iloc[test].values, verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
    cvscores.append(scores[1] * 100)

print("%.2f%% (+/- %.2f%%)" % (np.mean(cvscores), np.std(cvscores)))

y = model.predict(X)

To finish the process I take the values of y multiplied by the division values of step 6, receiving this output:

This neural network aims to answer the following question: In the next 60 readings, how many peaks of 80-100 will happen?
My question is: how to use this model to have an output of the next 60 readings?
Example: model.predict(X) = 60 readings
Index:  | 0  | 1  | ... | 58  | 59  |
        ------------------------------
Output: | 24 | 26 | ... | 42  | 45  |

The shape it is in today gives me an output with 1003 lines the size of my data set, I would like to limit this network to working in search of only the next 60 readings, any suggestions?


texts: Is there an equivalent of the Bandit package in r (Used in implementation of Multi arm bandit algorithm) in python?

texts: I want to implement MAB(Multi arm bandit algorithm ) in python. There is a package called bandit in R Check here, is there anything similar to this in python


texts: What to optimize for when asked to find the most important features

texts:
I have a binary classification problem, let's say people can buy or not buy a certain product. Now unlike a standard prediction task, I only want to find which features are the most important for the person's decision to buy.

Which metric should I use to optimize the algorithm? Maximize out of sample accuracy like when I would be interested in making the best prediction? Or maximize fit and don't care about overfitting? A mixture of both?

I am using xgboost.


texts: Spacy Returns Nonidentical Results for Doc. Examples?

texts:
I am getting started with Spacy, and I was just experimenting the given examples. It is weird though that when I run them over their containers using their experimental console it gives me different results than when I run them locally.

For example, when I run the following example from their console,

import spacy

nlp = spacy.load('en_core_web_md')  # make sure to use larger model!
tokens = nlp(u'dog cat banana')

for token1 in tokens:
    for token2 in tokens:
        print(token1.text, token2.text, token1.similarity(token2))

It gives me:

dog dog 1.0
dog cat 0.80168545
dog banana 0.24327646
cat dog 0.80168545
cat cat 1.0
cat banana 0.2815437
banana dog 0.24327646
banana cat 0.2815437
banana banana 1.0

But when I run the exact same thing locally, it gives me:

dog dog 1.0
dog cat 0.0
dog banana 0.0
cat dog 0.0
cat cat 1.0
cat banana -0.044681177
banana dog -7.828739e+17
banana cat -8.242222e+17
banana banana 1.0

To me the former makes sense, not my local result. And the problem persists for other examples as well. I am so confused! What is happening locally?

Spacy: '2.0.12'
Python 3.6.5 Anaconda, Inc.
Mac OSx 10.13.5

I am not sure if it is relevant, but I was getting this error running it in Jupyter, so I exported the followings, as suggested in the git, in the terminal before launching the Jupyter:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8


texts: What brings the performance difference in Deep Learning with different data augmentation strategies?

texts:
I am studying the performance of deep learning models toward abnormality detection in chest X-rays.

Due to sparsity of data, I augment the data using different augmentation strategies including:

Traditional augmentation methods (Gaussian smoothing, unsharp masking, and minimum filtering)
Generative Adversarial Networks

Contrary to the existing literature, I find that the models showed promising results with traditional augmentation methods (that i have mentioned herewith) than with GAN-generated synthetic images.

What brings this performance difference?


texts: Dimensionality of the target for DQN agent training

texts:
From what I understand, a DQN agent has as many outputs as there are actions (for each state). If we consider a scalar state with 4 actions, that would mean that the DQN would have a 4 dimensional output.
However, when it comes to the target value for training the agent, it is usually described as a scalar value = reward + discount*best_future_Q.
How could a scalar value be used to train a Neural Network having a vector output?
For example see image in
https://towardsdatascience.com/deep-q-learning-tutorial-mindqn-2a4c855abffc


texts: Does a 3-D input shape (time series with features) to an LSTM Model Evaluate the Label at each Time Step?

texts:
I have a problem similar to the one posed in this video:
https://www.youtube.com/watch?v=flMCYqIn3eg
In that video, she had a set of data on a number of individual debtors and needed to find out if they were going to default.  So each of her samples was a unique person and the data was a time series where each time step was a week and for that week she had data points (features) about the person's spending that week.  The goal was to have a binary label classification of yes/no for default or not.  This all makes sense to me except for one part.  From the screenshot at 4:57, it seems as though her original SVM model had all the time series data concatenated as columns so she had many features to evaluate but no time series.  That also makes sense but then it seems as though she only has one final yes/no for whether or not they default (column B).
At 11:08 she has it in LSTM with the time series so each week is it's own slice with the associated weekly features only for that week instead of being one giant array of all features for all weeks.  This also makes sense, but how does that work with the final labeling?
Does she still have only one final label and the entire time series is evaluated just to deliver that one label?  Or is it evaluated at each step of the way to determine at that time step whether or not you get a label that produces an alert for a possible default?  It seems like it is geared toward the former (one final label based on all input data evaluated in series).  But I'm not sure because at 11:49 her code shows her y data being 2 dimensional instead of a 1-d array/vector.  Does that mean she is really having a label for each time step per person, making the shape of y being NumPersons by NumTimeSteps?
In my scenario, I would really like to evaluate for the case where it examines each step of the way.  So to put it into this example context, I want to predict if the person will default say, a month from now (4 time steps).  So at each time step I would like to provide the features for that time step as well as the future (4 time step ahead) historical result of whether or not the person defaulted and I'd like to have it evaluate as a time series at each step of the way to make full use of the history.  Otherwise, as I understand it, it's only evaluating the time series once rather than a in slices.  For example, say you have 200 time steps (weeks) of data per person.  It should start evaluating at the beginning and then move forward while including the new history and the old.  So on week 50 it should be looking at 50 weeks and seeing if the person defaulted and taking that into consideration.  At 75 weeks, it should be looking at 75 weeks.  Maybe the dropout parameters might make it look at less, but the point is to evaluate what you have available up to that point at each step of the way.
Is this possible?  I'm also not sure if she make it to predict specific individuals or if there is just one model that applies across the board.  I believe it's the latter because her diagram at 11:08 doesn't seem to have any features to specify the individual person.  If so, that's good because that's what I am looking for - one encompassing model to look at the underlying connections across all the people rather than having the user be a feature/parameter.


texts: Neural Network accuracy and loss guarantees?

texts:
This question is part of a sample exam that I'm working on to prepare for the real one. I've been stuck on this one for quite a while and can't really motivate my answers since it keeps referring to guarantees, can someone explain what guarantees we can have in a neural network such as the one described?

Network description:

Assume a deep neural network (DNN) that takes as input images images of handwritten digits (e.g., MNIST). The input network consists of 28x28 = 784 input units, a hidden layer of logistic units, and a softmax group of 10 units as the output layer. The loss function is the cross-entropy. We have many training cases and we always compute our weight update based on the entire training set, using the error backpropagation algorithm. We use a learning rate thats small enough for all practical purposes, but not so small that the network doesnt learn. We stop when the weight update becomes zero. For each of the following questions, answer yes or no, and explain very briefly.

Questions:

Will this DNN configuration enable the the weights to minimize the loss value (there may be multiple global optima)?
Does this DNN configuration guarantee that the loss reduces on every step? We use our network to classify images by simply seeing which of the 10 output units gets the largest probability when the network is presented with the image, and declaring the number of that output unit to be the networks guessed label.
Does this DNN configuration guarantee that the number of mistakes that the network makes on training data (by following this classification strategy) never increases?


texts:
I have a question about inverse reinforcement learning. As far as my research went, I found papers that deal with noisy observations of the state, but that's what the agent sees and the data scientist observes as well (that is, the agent measures the current state, i.e. S, with error and the data scientist has access to that observation as well).
But what happens if the information sets are different? What if the agent observes a noisy measure of S, but the data scientist knows S for sure (as if it were ex-post) and does not have access to the agents' measurements?
Are there any papers out there that deal with this issue?


texts: Recommended model for univariate or multivariate multistep ahead time series forecasting

texts:
I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these transactions per account by summing all transactions per day so that I have a total expense amount per day. In other words, for each customer I have a time series for each account spanning several years. Each time series consists of daily expense amounts (and other features about this day and the account).

I now need to forecast the expense amount for an account for the each of the next 7 days. My initial attempt was to use a sequence of single features of the daily expense amount as input into an LSTM. I would predict value t+1, and then use this as the most recent input to predict t+2. I repeat this 7 times to get a 7 day forecast.

The results seem ok for the most part, but not great. I would like to explore more complex modelling to see if I can improve on this initial benchmark model.

This is the best experiment I have come across:

https://machinelearningmastery.com/how-to-develop-lstm-models-for-multi-step-time-series-forecasting-of-household-power-consumption/

Would an encoder-decoder seq2seq LSTM wuit this problem? What about a CNN-LSTM or ConvLSTM? I also would like to use some of the additional input features of the data to enrich the data used to predict the following 7 day expense values. This would now be a multivariate, multistep forecast.


texts: How to generate synthetic dataset using machine learning model learnt with original dataset?

texts:
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset characteristics?
[original data --> build machine learning model --> use ml model to generate synthetic data....!!!]
Is it possible? Please point me to related resource if possible.


texts: Correlation between categorical variables based on the target distribution

texts:
Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way to combine them (lets say that $X_1$ and $X_3$ are very similar in the frequency of the there $Y$ distribution) where $X$ in ($x_1,...,x_n$) and y in ($y_1,...,y_n$)

What is the best way to find all this $X$ subgroups that have similar $Y$ distribution? the reason I'm doing this is because I know that lots of my $X$s are the same but someone labeled them as different.
I have started with doing spearman corr matrix over the frequency table for each $X$ and its $Y$s distribution , but I'm not sure it is the right way to handle it and it gives me some bad results


texts: What is C in sklearn Logistic Regression?

texts:
In sklearn.linear_model.LogisticRegression, there is a parameter C according to docs

Cfloat, default=1.0
Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.

I can not understand it? What does this mean? Is it λ we multiply when penalizing weights?


texts: Best Data Science & Machine Learning Conferences 2019?

texts: What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are particularly interesting to me include model explainability, model pipelines, and model governance. I am specifically looking for a conference that have hands-on Spark training. Do you have any conference suggestions that align with my interests?


texts: History that lead to the word "predict" being used for the application of a model on data

texts:
Background
The framework scikit-learn uses "predict" for the application of model on (new) input data and I have seen many people use that term. In the scientific papers that I have read (and I read a lot, doing foundational research at university, but mostly in the area of unsupervised learning), the word "predict" seems not really that prevalent.
Looking up the word, it seems to me, that predict has a strong connotation about saying something about the future, but very often the "prediction" in the context of machine learning is not about the future at all. In fact, when people use a model to say something about the future, the word "forecast" is mostly used.
Additionally, the word "predict" seems to also have a connotation of "soothsaying", which is also not what we are doing.

Because of that it seems a bit strange that the word "predict" has been used and thus my question:

Question
What I would like to know is the history of what lead the community / so many people / some frameworks to use the word "predict". Maybe there is an original article that introduced the word, or an important talk or something like that. Maybe there was an influential discussion back in the day on why it is a good word that lead to the adoption.
I am also interested in arguments for the adoption of the word by influential people. So, I am not asking for an opinion of the one to answer the question (which again would be opinion-based), but references to and quotes of (historic) documents.


texts: Adjusted R-squared is too high (=1) in Linear Model

texts:
I built a Linear model which has an adjusted r-squared value of 1. I understand that this is a near perfect number. Upon further investigation, I found that one of the 96 independent variables in the dataset is highly correlated with the dependent variable. This is also a variable which I would like to keep (and not drop). Are there any additional steps that I should undertake to handle this?

Sample df reproducing the situation above:

df1 <- data.frame("y" =   c(0.0166 , -0.2380 , -0.3192 , -0.2774 ,  9.3148 , 0.3142) ,
          "x1" =  c(0.0103 , -0.2347 ,  -0.3182 , -0.2793 ,  9.4638  , 0.3297) ,
          "x2" = c( -0.1838 , -0.2458 , -0.2581 ,-0.2533 , 6.7566 ,-0.0835) ,
          "x3" = c(0.3426 ,-0.0543 ,-0.4512 ,-0.0543, 10.4637 , 0.3426) ,
          "x4" = c(-0.161 , -0.270 ,-0.318, -0.280 , 8.279 , 0.169))
df1
df1_lm <- lm(y~. , data = df1)
summary(df1_lm)


texts:
I have a task to find out bad visuals from given video.
Let's define bad images in the video as per the human brain -:

Images which don't contain subject in detail,
are too noisy or too many ads and subscription
letter or underexposed
or overexposed could be called as bad images.
The possibilities of bad images could be greater than this, but this
is just getting started.

Dataset is [collection of videos]
labels is [good:0,bad:1]


texts: Hindsight Experience Replay (HER) results obtained 50 times faster than original paper?

texts:
I am reproducing the results from Hindsight Experience Replay by Andrychowicz et. al. In the original paper they present the results below, where the agent is trained for 200 epochs.

200 epochs * 800 episodes * 50 time steps = 8,000,000 total time steps.

I try to reproduce the results but instead of using 8 cpu cores, I am using 16 CPU cores.

Fetch Push

I train the FetchPush for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using two random seeds.

After 20 epochs = 50,000 iterations we solve this environment. In the paper above, it took the original authors 100 episodes = 4,000,000 iterations to do so.

How is my algorithm converging 50 times faster?

Pick and Place

I train the FetchPickAndPlace for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using three random seeds:

and logger output for the first two epochs, showing that indeed I have 50 episodes per epoch:

Now, as can be seen from my tensorboard plot, after 40 epochs we get a steady success rate, close to 1. 40 epochs * 50 episodes * 50 time steps = 100,000 iterations. Therefore it took the algorithm approximately 100,000 time steps to learn this environment.

The original paper took approximately 50 * 800 * 50 = 2,000,000 time steps to achieve the same goal.

How is it that in my case the environment was solved nearly 20 times faster? Are there any flaws in my workings above? Surely I am doing something wrong, right?

Results are also faster than another paper which also uses 19 MPI workers:

As stated in this paper: "We train for  50 epochs (one epoch consists of 19  2  50 = 1 900 full episodes), which amounts to a total of 4.75 x10^6 timesteps." It took around 2,000,000 timesteps to reach a median success rate of 0.9.

SUMMARY

Any suggestions on what I may be doing wrong would be appreciated.

Code
Reddit thread

EDIT - Logging

Logging process shows that the rank 0 worker is reporting results. Inside her.py)

   if rank == 0:
        logger.dump_tabular()

The function responsible for writing all diagnostics is dumpkvs() inside logger.py:

def dumpkvs():
    """
    Write all of the diagnostics from the current iteration
    """
    Logger.CURRENT.dumpkvs()

Code can be found here:

https://github.com/openai/baselines/blob/master/baselines/logger.py


texts:
I am working on Object Segmentation using Lidar 3D point clouds, basically using Kitti dataset. However, the dataset is imbalanced for classes cyclists and pedestrians. So, I am looking for ways to

1. Augment the point clouds to increase the dataset size for more instances of pedestrians and cyclists and
2. To create an unbiased model which is more accurate and robust

Can someone guide me as to how can I do it for point clouds, like augmentation techniques (apart from flipping in horizontal direction) or adding some kind of noise(awgn)/distortion to point clouds, while preserving the data consistency?


texts:
I am currently working on a dataset to predict customer attrition based on past data and transactions of the customers.

There are 2,40,000 customers in total out of which around 1,77,000 customers are active(as of today) while the remaining ones are inactive (6300).
This is how sample headers look like :

custID|custAge|custGender|TQuantity|TVolume|TValue|TAmount|HolidayStatus|...

Overall, I have 40 predictors which include customer details, transaction details, item details etc.

The data obviously has more active customers than inactive customers i.e. inactive customers form only 2.6% of the entire customer base. Due to this, there are more transactions conducted by active customers(25million/32million) than by inactive (previously active) (6million/32million) ones.

Despite this, I created a logistic regression model using random data (shuf -n 500000 data.csv). The model achieves 96.69% base accuracy in predicting when fed with random data.

The problem: How to make the model predict with greater accuracy on such a biased dataset? or How do I sample the data more appropriately?

Model prediction: With 99.7% probablity, it predicts that the customer will be active whereas the customer is inactive

PS: Changing threshold won't help much


texts: Image Classification with two very similar and random-ish classes

texts:
I am trying to build a neural network which takes a picture of a FOREX chart (currency exchange) and outputs an "up" or "down" prediction. I'm classifying up or down based on where the stock is after one hour, and currently I have 400 examples in each class. However, my training attempts are stuck at around 49-51% accuracy. I've come up with a few potential reasons:

1) A bad model. I don't think this is the case - I've tried transfer learning on Inception V3 and two of my own models. However, is there a different type of model that could somehow account for the randomness of markets?

2) Not enough data: I think this is the most likely explanation. From what I've read, you need thousands of images, not a few hundred,  for good image classification.

3) The problem is impossible. I chose FOREX over stocks because FOREX isn't as heavily affected by headlines during the span of a few hours. Most FOREX trading in that time frame is done by technical analysis, which is doing things with charts - trend lines, moving averages etc.

I know that two is the most likely explanation, but before I go out and spend days collecting more data, I would like a bit of feedback to ensure that this is actually possible.

Thanks!


texts:
I want to extract the values of the below text

Paﬁent Name : Thomas Joseph MRNO : DQ026151?
Doctor : Haneef M An : 513! Gandar : Male
Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17
Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017

extract only the values of the field names for example,

Thomas Joseph from the field name Pateint name, similarly for others field names and save the output to excel

Python code for the above

My attempt -

text = pt.image_to_string(img1)
print(text)
s = re.findall(r'\s:\s(\w+)', text)
print (s)


texts: Why CNN input images are often square shaped?

texts:
Why it's so that in convolutional neural networks we generally take the image dimensions of input image to be generally a square? We even do padding to make it happen. Why not different dimensions?

What I understood is that computer computes multiplication and division (by 2) much faster than the rest.

Can someone shed some light on this?

Any link or reference will be appreciated. I already have CS231n notes and lecture slides.


texts: What's a good book (or other resource) to learn Imitation Learning?

texts: I must learn and apply imitation learning on a robot for my thesis. I'm looking for decent sources of information on this topic.


texts: how do I approach forecasting problems using deep neural networks?

texts:
I am new to machine learning in general, and I have been requested to predict a price given a date. I have been trying to make a neural network for the task but it does poorly in the testing set, so I was hoping for some advice.

the data consists of 300 entries of a string date (YYYY-MM-DD) and the corresponding price starting from 1922.

here is what the price curve looks like:

I had to find a way to featurize the dates somehow so here is what I came up with:

data = pd.read_csv("DataSet", header=0)
dates = pd.to_datetime(data["date"])

# the ratio of the current day of the year to the maximum number of days a year can have
dayOfYear = np.array(dates.dt.dayofyear).reshape(1, -1) / 366

# the ratio of the current week of the year to the maximum number of weeks a year can have
weeks = np.array(dates.dt.week).reshape(1, -1) / 52

# the ratio of the current month of the year to the maximum number of months a year can have
months = np.array(dates.dt.month).reshape(1, -1) / 12

# I wanted to squish the year to be between [0,1] so i shifted the tanh function to the right,
# and divided it by a 100 just so I wouldn't end up with an array of mostly 1s.
# I have no idea if that causes any issues or not, and Idid this based on nothing.
years = ac.tanh((np.array(dates.dt.year).reshape(1, -1) - 1922) / 100)

# the ratio of the current quarter of the year to the maximum number of quarters a year can have
quarter = np.array(dates.dt.quarter).reshape(1, -1) / 4

# the ratio of the current day of the week to the maximum number of days a week can have
# (starting from 0).
weekDay = np.array(dates.dt.dayofweek).reshape(1, -1) / 6

# did the same thing I did with years. the number of days passed since the base date squished to [0,1],
# and the 10000 is there to delay convergence of the tanh function to 1.
# Again, I have no idea if any of this works since I had no upper bound to the data.
delta = np.array((dates - pd.Timestamp('1922-3-31'))).reshape(1, -1)
daysSince = ac.tanh(delta / (np.timedelta64(1, 'D') * 10000))

that's the best I could think of when I tried extracting features from dates.
after that randomized the data, and took the last 50 examples of the 300 to be the validation set.

As for the network, I tried multiple configurations and after hours of trial and error, a 5 hidden layer network is what worked best with me.
the number of neuron along with their activation functions (including the output layer):

[256    ,128   , 110.   ,42.    , 18.  ,      1]
["swish","tanh", "swish","swish","relu", "relu"]

I had to include activations other than relu because a mostly relu network kept calculating linear functions.

I have tried multiple loss functions, and I ended up using the MSE loss function.

I used a learning rate of 0.02 and a learning rate decay coefficient of 0.003273. the learning create decay function is as follows:

learning_rate / (1 + i * learning_rate_decay)

I also used mini batch gradient descent with a mini batch size of 32 and and Adam as an optimizer, and I train the network for 4600 epochs.

with this configuration I get an accuracy of 83% on the training set and 32% on the validation set (any prediction within 5 units of the expected output value is considered = ) .

here is the training set predictions vs the actual values:

the validation set predictions vs the actual values:

and the cost function(MSE) change per 100 epochs:

obviously there are tons of mistakes in this network and I just want to know what am I doing wrong, and can this architecture work or is it unfit for such problems.


texts: How to handle imbalanced Data set features for classification problems?

texts:
I have a Data set with 300 features that are dummies 0 or 1 and two classes for output 0 or 1 and the distribution of classes is 48% and 52%, but the distribution of the features is skewed as in the picture, I use neural network as my classification algorithm but either I get over fitted model with near 100% accuracy on training set and about 58% accuracy on the test set, or I use Early stopping to prevent over fitting and get around 57% accuracy for training and about 54% for testing.
The optimal error for this problem is around 20% so if I at least reach 70% accuracy on the test I am happy, but I cannot even pass the 60% mark, How to deal with that?
Thanks


texts: Python : Feature Matching + Homography to find Multiple Objects

texts:
I'm trying to use OpenCV via Python to find multiple objects in a train image and match it with the key points detected from a query image. For my case, I'm trying to detect the tennis courts in the image provided below. I looked at the online tutorials and could only figure that it can only detect only one object. I thought of inserting a loop in for it to find multiple objects but I failed to do so. Any idea on how to do it?
I used SIFT as ORB does not work that well for my case.
Here's the code and a sample set of images:
import numpy as np
import cv2
from matplotlib import pyplot as plt

MIN_MATCH_COUNT = 10
img1 = cv2.imread('Image 11.jpg',0)          # queryImage
img2 = cv2.imread('Image 5.jpg',0) # trainImage

# Initiate SIFT detector
sift = cv2.xfeatures2d.SIFT_create()

# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)

# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
    if m.distance < 0.7*n.distance:
        good.append(m)
if len(good)>MIN_MATCH_COUNT:
    src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
    dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
    M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
    matchesMask = mask.ravel().tolist()
    h,w = img1.shape
    pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
    dst = cv2.perspectiveTransform(pts,M)
    img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)
else:
    print( "Not enough matches are found - {}/{}".format(len(good), MIN_MATCH_COUNT) )
    matchesMask = None
draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                   singlePointColor = None,
                   matchesMask = matchesMask, # draw only inliers
                   flags = 2)
img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)
plt.imshow(img3, 'gray'),plt.show()

Train Image :

Query Image :

I have already posted this question in Stack Overflow but was recommended by @sophros to try posting it here.


texts: What are kernel initializers and what is their significance?

texts:
I was looking at code and found this:

model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu'))

I was keen to know about kernel_initializer but wasn't able to understand it's significance?


texts: Forecasting one time series with missing data with help of other time series

texts:
I have time series $R$, which shows, how something changes at the regional level.

I have several time series $U_i$, which show, how something changes at a special unit $I$ level.

There are many units in the region. $R$ has no missing data. Different $U_i$ have their own missing periods.

I want to forecast $U$ after a missing period using information of $R$ and information of $U$  when it was availible.

My thoghts till now:

Suppose $R$ is known on interval $[0, 365]$. Suppose $U_i$ is known on interval $[0,300]$. Let's take $R$ and $U_i$ both on interval $[0,300]$, take difference between them and trying to predict that difference with linear regression. So for interval $[301,365]$, I will have differences and to restore $U_i$, I will just have to take out my differences from $R$.

I don't like my solution, because:

We need a model for each $U_i$.
Because, sometimes data is more sparse and I don't even have a $[0,300]$ known interval, so not able to train regression properly.


texts: Using keras LSTM implementation with sparse data

texts:
I am trying to build an Autoencoder with LSTMs. My input data has been one hot encoded, which results in around 13.000 features.

My problem at this point is, that I get the following error message:

...
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 926, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 229, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 208, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 447, in make_tensor_proto
"Cannot create a tensor proto whose content is larger than 2GB.")

ValueError: Cannot create a tensor proto whose content is larger than 2GB.

How can I reduce the size of the tensor without secraficing the deepth of the model respectively without rewriting the code in tensor myself?

I am using tesnorflow as my backend. My model looks as following:

# current values for timestamps and features: 1 and 12113
def build_model(timestamps, features):
    model = Sequential()
    layers = {'hidden1': 64, 'hidden2': 16, 'output': features}

    model.add(LSTM(
            batch_input_shape=(batch_size, timestamps, features), #batch_size is set to 500
            units=layers['hidden1'],
            return_sequences=True,
            stateful=True))

    model.add(LSTM(
            units=layers['hidden2'],
            return_sequences=True,
            stateful=True))

    model.add(LSTM(
            units=layers['output'],
            return_sequences=False,
            stateful=True))

    model.compile(loss="binary_crossentropy",
                    optimizer="rmsprop", # is recommended vor RNNs)
    return model


texts: what is the first input to the decoder in a transformer model?

texts:
The image is from url: Jay Alammar on transformers

K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each decoder layer in the decoder.
The previous output is the input to the decoder from step 2 but what is the input to the decoder in step 1?  Just the K_encdec and V_encdec or is it necessary to prompt the decoder by inputting the vectorized output (from the encoder) for the first word?


texts: Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

texts:
I am a professional java developer with 9 years of experience. Never worked with Python but, I do not have any restraints learning it.

Still I need clarity on a couple of aspects. I hope if anyone can help, it will clear the cobwebs in my mind. I am looking to develop a machine learning project :

With a web interface to present interactive analytical graphs and views.
Employing a web crawler collecting data, both structured and unstructured.
A backend crunching the data collected using machine learning and artificial intelligence and natural language processing algorithms.

While exploring options available, I found both Python and Java have extensive support and frameworks/libraries/modules available to leverage machine learning implementations.

However, working with Java gives me a sense of strongly typed programming and it can leverage the underlying CPU cores via its robust multi-threading APIs. Python on the other hand is scripted language and on the face value looks to lesser efficient when working on complex logic crunching a huge amount of data pertaining to Natural Language Processing.

I might be utterly wrong as I am not very acquainted with Python ecosystem.

Can anyone help me understand on the choice between Java and Python if I keep in mind execution speed, maintainability and community support targeted toward development of a large, scalable and robust machine learning/artificially intelligent application ?


texts: Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

texts:
I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that library uses the WarpLDA sampling algorithm insted of Gibbs?

Direct quote from Griffiths and Steyvers 'Finding scientific topics' paper:
"... we can approximate p(w|K) by taking the harmonic mean of a set of
values of p(w|z, K) when z is sampled from the posterior p(z|w, K) [(Kass
and Raftery, 1995, Section 4.3)]. Our Gibbs sampling algorithm provides
such samples, and the value of p(w|z, K) can be computed..."

Excuse me for being this basic, but does WarpLDA sampling algorithm provide same type of samples so I can use the harmonic mean as an indicator for the topic number?


texts:
I have 3 independent input variables: time, temperature and pressure, and one output - usefulness.

Example:

Number of the sample; Time; Temperature; Pressure; Usefulness
1;1;30;50;9%
2;3;50;50;80%
3;5;50;40;75%
4;7;50;40;?%

I want to train my network and predict the usefulness for the last sample. Which model of forecasting should I use? ARIMA? For coding, I use python.

Thanks!


texts: Is there a way to calculate the cosine distance between 2 time series?


texts: Training with feature metadata - bayesian network (naive bayes)

texts:
I am building a project for fun - reverse astrology!

I am making a dataset like:

planet_name, life_impact

Saturn, impact1

Saturn, impact2

Sun, impact2

Sun, impact3

Sun, impact4

Mars, impact 5

Mars, impact 6

etc…

I have trained a bayesian network over this^. I want the network to learn metadata based learnings also.

ie.
Right now for positive evidence “impact 1” network will contribute probability 1/2 to Saturn and 1/3 to Sun.

Now, I have metadata(duration) for feature: life_impact
duration for impact2 for Saturn is 1/5
duration for impact2 for Sun is 5/5

This should change the probability contribution to sun/saturn: Now contribution should change as per the value of duration also.
eg.
impact 1 with duration 1 should contribute more to Saturn and less to Sun, similarly impact 1 with duration 4 or 5 should contribute more to Sun.

Note that metadata duration might not be there for all impacts, something else might be present like “relevance”

How should I design my features to accommodate this?


texts: Normalise the scores by small and large numbers division

texts:
I am working on a scoring problem use-case where I will score each task based on few aspects and take an average of scores of tasks in each experiment. For each experiment I will have 1 to N number of tasks where N ranges upto 100k. So, this averaged score help me to rank the experiments to understand their efficiency.
Here, the problem I am facing is taking the average ie., since as said, each experiment have 1 to N number of tasks. Lets say we have two cases where
Case 1:.
Experiment X having 10 tasks. Each task is rated 7 out of 10. So the average is 10*7/10 = 7
Case 2:
Experiment Y having 1000 tasks. Each task is rated 7 out of 10. So the average is 1000*7/1000 = 7
Now, ideally since the scores are equal, we can conclude that X==Y. But the fact is Y is better than X because after 1000 tasks it is still having 7 average rating and the experiment Y is chosen 1000 times which itself says that it is more preferred and trusted.
Now, here I want to rate Experiment Y is higher than X by taking no of tasks into account taking into count that the score should not exceed the range 0-10. And, the range of tasks in the experiments can be 0 to infinity.
I tried multiplying the final average score of the experiment with ((N-1)/N) but the dispersion is very small for large numbers of N.


texts: Fixed-radius range search in non-Euclidean space

texts:
I'm trying to find an indexing data structure most suitable for my metric space:

set of IP network related data (IP addresses, ports, TCP flags, ...),
distance function is continuous, non-Euclidean but satisfies non-negativity, symmetry and triangle inequality,

with respect to the fixed-radius range query performance. I want to use it for a clustering algorithm (DBSCAN or similar) on a data sets with millions of elements. So far I have studied:

ball trees/VPT,
MVPT,
GHT, GNAT,
AESA/LAESA,
cover trees,

and basically all other methods from Searching in metric spaces survey, but all mentioned are suited for varying-radius queries. Many related SO answers advise Locality-sensitive hashing (LSH) as a current trend, but I didn't find any clear information whether it exploits the advantage of fixed-range and whether it is usable for non-Euclidean metrics. Any advices/references?


texts:
When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines?
I'm doing a binary classification task where 40% of the samples are class_0 (untreated class), and the other 60% are class_1 (treated/positive class).
I have two baselines: baseline_0 predicts randomly, and baseline_1 predicts class_1 every time.
Because the metrics are calculated relative to class_1, when baseline_1 predicts class_1 on every sample, it ends up with perfect recall (1.0), which inflates the f1 score. Does this mean this baseline model is not good for comparing to my experimental models and I should use baseline_0 instead, or that baseline_1 is good but that the f1 score is not good for making these comparisons?


texts: What is the difference between "Adding more LSTM layers" or "Adding more units on existence layers"?

texts: What is the difference between adding more LSTM layers and just increasing the units of existing layers? Which one is preferred and in which situation?


texts: Any one read the over 100gb csv file and successfully concatenation?

texts:
I have been searching for the deal with large CSV file read method
Its over 100gb and need to know how deal with the chunk file processing
and make concatenation faster

    %%time
    import time
    filename = "../code/csv/file.csv"
    lines_number = sum(1 for line in open(filename))
    lines_in_chunk = 100# I don't know what size is better
    counter = 0
    completed = 0
    reader = pd.read_csv(filename, chunksize=lines_in_chunk)

CPU times: user 36.3 s, sys: 30.3 s, total: 1min 6s
Wall time: 1min 7s

this won't take long but the problem is concat

%%time
df = pd.concat(reader,ignore_index=True)

this part take too long and take too much memory also
is there way to make this concat process faster and efficiently ?


texts:
I am a medical doctor working on methodological aspects of health oriented ML. Reproducibility, replicability, generalisability are critical in this area.
Among many questions, some are raised by adversarial attacks (AA).
My questions are the following : suppose I want to check an algorithms from an AA point of view :

is there a systematic toolbox to be used, relating format of the data, type of models and AA ?
are AA classified in some sense ?
are there gold standards in AA ?


texts:
I have written some code for an SVM and the final output isn’t brilliant.  The data is linearly separable so I don’t understand why the data isn’t perfectly separated. Here is the code:

from matplotlib import pyplot as plt, numpy as np

x = np.array([[-2, 4, -1], [4, 1, -1], [1, 6, -1], [2, 4, -1], [6, 2, -1]])

y = np.array([-1, -1, 1, 1, 1])

def svm_sgd(X, Y):
    w = np.zeros(len(X[0]))    # Parameters
    eta = 1
    epochs = 100000

    # Update rules

    for epoch in range(1, epochs):
        for i, x in enumerate(X):
            if (Y[i] * np.dot(X[i], w)) < 1:
                w = w + eta * ((X[i] * Y[i]) + (-2 * (1/epoch) * w))
            else:
                w = w + eta * (-2 * (1/epoch) * w)
    return w

w = svm_sgd(x, y)
print(w)

# Print Hyperplane —————————————

for d, sample in enumerate(x):
    if d < 2:
        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)
    else:
        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)

plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')
plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')

x2=[w[0],w[1],-w[1],w[0]]
x3=[w[0],w[1],w[1],-w[0]]

x2x3 = np.array([x2,x3])
X,Y,U,V = zip(*x2x3)
ax = plt.gca()
ax.quiver(X,Y,U,V,scale=1, color='blue')
plt.show()

The bottom portion just uses matplotlib to print the data, so it’s not part of the SVM algorithm, and can therefore be ignored.

  The problem ~ can anyone explain to me, based on my code, why the line doesn’t separate the data more perfectly?


texts: CNN 3D line angles prediction regression - results of training of phi depend on theta

texts:
I am a beginner in "deep learning". What I am trying to do, is to predict two angles of a 3D line projected on a 2D image. The toy model is that I create a line going out from the centre of 48x48 array. The angle phi is the angle of the line in the image plane, angle theta is the angle of the line in the plane perpendicular to the image. Theta is just used to calculate each line point distance from the image centre and thus the line brightness at each point, by scaling line value with 1/R^2.

The problem is, that with theta fixed to 0 I can achieve ~3 deg precision of phi determination. However, if theta is allowed to go between 0 and 90 deg, the phi training results drop significantly. Of course, prediction of phi is worse for theta approaching 90 deg, because for theta=90 deg the phi is undefined. However, the results of phi training depend on the theta range allowed. I will have much better results of phi training if I allow theta in the range of 0-45, than if I allow theta in the range of 0-90. I would be grateful if someone could tell me, why this is the case and how to improve the results.

As for implementation. I am using Chainer based CNN. What I do with my layers does not seem to affect results much. At the moment I have 6 2D convolutional layers then 2 linear layers, the last one giving 3 variables. The variables are sin(phi), cos(phi), theta. Using sin/cos it is easier to cover whole 360 deg and take into account circular nature of angle. I output variable as is (linear). The loss function calculates phi as arctan2(cos(phi),sin(phi)), then uses mean_squared_error to compare its phi,theta with the input.


texts: Given data that is labeled as outliers, how can I classify data as outliers?

texts:
I have a dataset that is a mixture of sparse binary features and quantitative features. I only have definite outliers labeled. How should I approach trying to classify unlabeled data?

I considered using OSVM or other methods of one-class classification.

However, in my data the normal data points are clustered close to the mean. The outliers are generally points that deviate from the mean in any direction. My problem is that the outliers form a sort of high dimensional doughnut around the normal data.

Considering that the deviations occur in all directions, what algorithms would be best suited to the task? Keep in mind that I have significantly less normal labeled data points for training although the normal points will outnumber the outliers in the unlabeled data.

PS I posted this question on Cross Validated as well. Which site should this question be posted on?

EDIT:
Mahalanobis is able to work fairly well. However, I have the labeled outliers. Is there someway I could use them to improve accuracy?


texts: Does the Context Vector consist of hidden state and Cell State or just the hidden state?

texts:
LSTM's carry a hidden state and a cell state with them. Now, in a standard encoder-decoder model, we pass the Context Vector from the encoder to the decoder.
Does, this Context Vector consist of just the hidden state carried by the encoder or both, the hidden state and cell state carried by the encoder?


texts: Can I use entropy as a measure for determining significant variables in a cluster after clustering?

texts:
After clustering my data into k groups, I would like to determine for each of the clusters, which dimensions(variables) significantly describe that particular cluster. For example, lets say cluster A contains a majority of people from one age group purchasing a vehicle from particular segment. Cluster B contains people belonging to various age groups but majorly from one or two regions (out of 10-15 regions).

Hypothesis: Using entropy, determine for each cluster, which variables are homogeneous and use those variables to create a definition for that cluster by computing the mode (or some other descriptive statistic). For example,

Cluster A: Customers of 30-40yrs age group, purchasing vehicles from segment C1, etc

Cluster B: Customers from region R1, earning 30K-40K per annum, etc

Are there any pitfalls of using this technique?


texts: Should I Plot my Learning Curve Depending on Various Parameters to Diagnose Overfitting/Underfitting or is One Parameter Sufficient?

texts:
I know that learning curves are a good tool to diagnose overfitting or underfitting for a model.

Their working principle is simple: The training/validation loss/accuracy is plotted depending on specific parameters, e.g., the learning rate, the model architecture complexity, the training set size, the number of training epochs, etc.

However, should I always plot many different learning curves, i.e., depending on these different parameters? Or is it sufficient, if I only plot one learning curve depending on just one parameter? Thus, would it be reliable enough, if I would just plot the learning curve based on the keras (tensorflow 2 as the backend) model.fit history, i.e., the training/validation loss depending on the training epochs.


texts: Joining tables from different locations in Bigquery

texts:
been trying to join two tables in different data set which are in different locations but in same project. Always getting an error that

  dataset not found in US location.

The datasets locations are US and us-east1

Here is what I am doing

select a.*, b.* from `project.dataset1.table1` a join `project.dataset2.table2`
on a.common_col = b.common_col

Please help me out in this?


texts: Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts?

texts: Is this some kind of estimation of the relative positions of words in the training texts? are they creating some kind of statistical "distribution" of words? is "cat" usually 2 or 3 words away from "milk" in English language? things have to have a meaning, havent they?  Is BERT just adding some aditional dimensions to the vector space to include info on the relative positions of words?


texts: Machine Learning with intended missing values

texts:
I have a dataset relating to humans completing reviews, the target variable is whether the review decision is correct / incorrect and one of my features is a trailing 4 week accuracy score for the reviewer.
These accuracy scores are not always available however. My question is around how to model this data - the fact that there is no available accuracy score might be a signal. From my research into this - everything I see tells me that the missing values must be imputed or removed. I am wondering whether there are techniques to incorporate the fact that the data is missing into the dataset.
Perhaps I could convert the score into a categorical variable {low, medium, high, not available] - would this be common practice? I am open to suggestions and would love to hear what is commonly done in these scenarios


texts: Inferring latent variable distribution from binary data

texts:
I want to create a model to infer the behavior of a latent variable.
As an example, let's assume the latent variable represents the level of extro/introversion of a person.

The level of extro/introversion can be represented with some continuous distribution and can assume values in [0,1].

I have a binary training set of facial images of people who are either:

Extreme introverts (for which y=0)
Extreme extroverts (for which y=1)

Is there a way to model the original distribution (which is continuous, and not binary as the training set)?

Obvious solutions worked poorly:

I have tried training a CNN; a CNN with sigmoid activation for one output neuron + binary crossentropy, modeling this as a classification problem, and it worked very well distinguishing between the two classes, however, when evaluated on people who are somewhere in the middle of the intro/extroversion spectrum (y=~0.5), it performed poorly, assigning levels which are either close to 1 or 0.

I have also tried creating a regression based network, with output neuron with linear activation and MSE as a loss function, getting the same performance.


texts: Backpropagating across convolution layers with fixed kernels in Tensorflow

texts:
Background: I am currently working with volumetric data where I feed one 2D slice from the volume into a denoising CNN with learnable kernel weights and calculate a slice-wise denoising loss, and then pass the remaining 2D slices from the volume into a fixed version of this denoising CNN with fixed weights (essentially I am copying the learnable denoising CNN in this fixed version during training using tf.nn.conv2d where a kernel can be fixed). I then concatenate these denoised slices into a volume, and use this volume to calculate a volume-wise loss. These losses are then summed into a final loss where gradients are then taken.
Problem: I want both gradients from the slice wise and volume-wise losses to propagate back to the learnable kernel weights in the denoising CNN. It appears that only the slice-wise loss propagates back to the learnable kernel weights, and not the volume-wise loss. In other words it appears adding the fixed version of the denoising CNN effectively disconnects the graph.
My Attempted Workaround: Sharing Weights -I pass all the slices through the learnable CNN using a for loop, but still only use one slice to learn the loss. I then concateante the denoised slices into the volume, and calculate a data-consistency loss. This fixes the backpropagation problem, but appears to be really memory intensive. It seems like even with sharing weights in a for loop in tensorflow, for N iterations, I get seperate N convolutions rather than passing N pieces of data into one convolution.
Question 1: With respect to sharing weights workaround,Is there a way to use for loops in tensorflow where instead of unrolling to N convolutions, I can pass N pieces of data into one convolution
Question 2: With respect to the fixed convolution approach, Is there a way to effectively reconnect the graph and backpropagate across fixed convolutions to the learnable layers?