texts: How many features do I select when doing feature selection for regression algorithms? Is R2 and RMSE good measures of success for overfitting?

texts:
Context: I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random Forest Regressor, Decision Tree Regressor, and Linear Regression. I can tell there is some overfitting going on, as my initial values vs cross validated values are as follows:
RF: 10 Fold R Squared = 0.758, neg RMSE = -540.2 vs unvalidated R Squared of 0.877, RMSE of 505.6
DT: 10 Fold R Squared = 0.711, neg RMSE = -576.4 vs unvalidated R squared of 0.829 and RMSE of 595.8.
LR: 10 Fold R squared = 0.695, neg RMSE = -596.5 vs unvalidated R squared of 0.823 and RMSE of 603.7
I have already tuned the hyperparameters for RF and DT, so I was thinking about doing feature selection as a next step to cut down on some of this overfitting (especially since I know my feature importances/coefs).I want to do feature selection now with a filter method (i.e. pearsons) as I want to keep the features going into each model consistent.
Question: How would I decide on a number of features to choose using feature selection? Is it arbitrary? Or do I basically just remove all of them that don't have much correlation with the data? Is there a way to spit out an optimized set of features without doing grid search or random search?
Follow up question: Are the R2 and RMSE cross validated values good measures of success for overfitting comparison?


texts: How to apply model to training data to identify mislabeled observations?

texts:
I have a list of people, attributes about those people (height, weight, blood pressure, etc.), and a binary target variable called has_heart_issues. This data represents the full population of data, and I am trying to determine whether anyone who is listed as "No" for has_heart_issues is similar to the people who are listed as "Yes".

To answer this question, I split the data into training (70%) and testing (30%). I trained a random forest model on the training, and I tested it on the testing. The results are good, but I don't know how to apply to the population since I used most of it for training. Is there any way to apply the model to the full dataset (including the training) since I had labels for the full dataset to start with? Essentially, I am trying to determine whether any of the people were mislabeled.

Is it okay to apply the model to the training data to find the "mislabeled" records?


texts: Methodical approach to improve deep neural network performance?

texts:
Working on a binary classification task to identify duplicate documents with 350,000 labeled samples, 70 features. There is about a 10:1 class imbalance. My best performing model so far has been a random forest, but I would like to see if I can use a deep feed forward neural network for better results.

I have been playing with the number of layers and hidden units and verify performance using 5-fold cross validation. Here is an example of one architecture:

for train_validation, test_validation in sfk.split(X_train,y_train):
    model = Sequential()
    model.add(Dense(150, input_dim=X_train.shape[1], activation='relu'))
    model.add(Dense(125, activation='relu'))
    model.add(Dense(100, activation='relu'))
    model.add(Dense(75, activation='relu'))
    model.add(Dense(50, activation='relu'))
    model.add(Dense(25, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])

    model.fit(X_train.iloc[train_validation,:].values,
              y_train.iloc[train_validation].values.reshape([-1,1]),
              epochs=13,
              batch_size=64,
              verbose=2,
              validation_data=(X_train.iloc[test_validation ,:].values,y_train.iloc[val_test].values.reshape([-1,1])))

I run that, get the mean/std of the accuracy and error. Then try adding a layer, removing a layer, adding some units, removing some. There is movement in accuracy and error (in the +/- 0.00x range). This approach is unscientific, what is a better way?


texts: Is one big network faster than several small ones?

texts:
The basis of my question is that a CNN that does great on MNIST is far smaller than a CNN that does great on ImageNet. Clearly, as the number of potential target classes increases, along with image complexity (background, illumination, etc.), the network needs to become deeper and wider to be able to sufficiently capture all of the variation in the dataset. However, the downside of larger networks is that they become far slower for both inference and backprop.

Assume you wanted to build a network that runs on a security camera in front of your house. You are really interested in telling when it sees a person, or a car in your driveway, or a delivery truck, etc. Let's say you have a total of 20 classes that you care about (maybe you want to know minivan, pickup, and so on).

You gather a dataset that has plenty of nice, clean data. It has footage from lots of times of the day, with lots of intra-class variation and great balance between all of the classes. Finally, assume that you want this network to run at the maximum possible framerate (I know that security cameras don't need to do this, but maybe you're running on a small processor or some other reason that you want to be executing at really high speed).

Is there any advantage, computationally, to splitting your network into smaller networks that specialize? One possibility is having a morning, an afternoon/evening, and a night network and you run the one corresponding to the time of day. Each one can detect all 20 classes (although you could split even farther and make it so that there is a vehicle one, and a person one, and so on). Your other option is sharing base layers (similar to using VGGNet layers for transfer learning). Then, you have the output of those base layers fed into several small networks, each specialized like above. Finally, you could also have just one large network that runs in all conditions.

Question: Is there a way to know which of these would be faster other than building them?

In my head, it feels like sharing base layers and then diverging will run as slow as the "sub-network" with the most additional parameters. Similar logic for the separate networks, except you save a lot of computation by sharing base layers. Overall, though, it seems like one network is probably ideal. Is there any research/experimentation along these lines?

Note: I also asked this question on AI.SE here to no response. If I get one here, I'll link to it over there and vice versa.


texts: Out of sample extension for Isomap in Sklearn

texts:
If I'm fitting the isomap class with a certain dataset, then I transform with a different one, does that mean that Sklearn is doing out-of-sample extension ?

I.e.

isomap=sklearn.manifold.isomap()
isomap.fit(X1)

isomap.transform(X2)


texts: What inference can we draw from the frequency distribution of thresholds?

texts:
I've the probability scores of positive class of two models. The frequency distribution of those probability scores(thresholds) are like this

Model #1

Model #2

It's a binary classification problem. And the thresholds obtained are from the test data. The stats in the confusion matrix are nearly same for two models. Can we say the Model #2 is doing well in separating classes? What other conclusion we can draw from the frequency of the thresholds.


texts: Why we take $\alpha\sum B_j^2$ as penalty in Ridge Regression?

texts:
$$RSS_{RIDGE}=\sum_{i=1}^n(\hat{y_i}-y_i)^2+\alpha\sum_{i=1}^nB_j^2$$
Why we are taking $\alpha\sum B_j^2$ as a penalty here? We are adding this term for minimizing variance in Machine Learning Model. But how this term minimizing variance. If I add suppose $e^x$ or any increasing function then it also minimizing the variance. I want to know how this term minimizing the error


texts: What do I need to do to increase model accuracy in sklearn?

texts:
I am new to ML and I have a custom data set that has court cases. The columns are charges(str), prior cases(int) and bond amount(int). I am using charges and prior cases as features and the bond amount is my label. Charges(str) is transformed to an int using LabelEncoder.fit_transform. I want to be able to predict what the bond amount would be given the charges and the prior cases. I am using MLPClassifier as my model and have used a couple others. I am only getting around 45% percent accuracy. How can I increase it? Do I just need to try different models? Is there a problem with how I am transforming the data? Could it be possible that scikit-learn just doesn't cut it for the amount of data I am dealing with? The data is super clean and I have gotten rid of outlier bond amounts, but accuracy only increases by about 2%. Just to give you an idea of what the data looks like here are the heads for X and y. Trial and error have driven me here.
X.head() gives me:
    charges prior_cases
0   2913.02(A)(1)   0
1   2913.02(A)(1)   0
2   2913.02(A)(1)   5
3   2913.02(A)(1)   3
4   2913.02(A)(1)   22

y.head() gives me:
    bond
0   2500
1   1000
2   1500
3   1000
4   5000

After transforming X,
X[:5] gives me:
[[0 0]
 [0 0]
 [0 5]
 [0 3]
 [0 22]]

Here is all the code:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import accuracy_score

# Bring in CSV
final_df = pd.read_csv('merge.csv')

# Drops bonds rows with bonds over $10,000
outlier = final_df[final_df.bond >= 10000].index
final_df.drop(outlier, inplace=True)

# Gets features from CSV
X = final_df[['prior_cases', 'charges']].values
# Gets labels from CSV
y = final_df[['bond']].values

le = LabelEncoder()
# Transform X data
for i in range(len(X[0])):
    X[:, i] = le.fit_transform(X[:, i])
# Transform y data
for i in range(len(y[0])):
    y[:, i] = le.fit_transform(y[:, i])

# Splits the data, 80% for training
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, train_size=0.8)

# Defines model that the data is trained on
model = MLPClassifier(activation='relu', solver='adam', hidden_layer_sizes=(100,), random_state=1, max_iter=500)

# Trains the model
model.fit(X_train, y_train)

# Gives what the predicted label should be
predictions = model.predict(X_test)

# Accuracy of the model
acc = accuracy_score(predictions, y_test)

print('actual:----', le.inverse_transform(y_test[100]))
print('predictions:', le.inverse_transform(predictions[100].reshape(-1,1)))
print('accuracy: ', acc)
```


texts: mAP using Tensorflow object detection API

texts: After I train my object detector using the Tensorflow object detection API(to detect only cars), I get an mAP value around 0.32 while running the eval.py script. However, in the Tensorflow Detection model Zoo page(which can be accessed at https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md), the mAP's for various models are 100 times the value I have obtained but when I look at my model's predictions on the test set, they are actually pretty good. Should I multiply the value which I have obtained by 100 or is there something wrong with the model that I have trained?


texts: Time series binary classification probability smoothing

texts:
Problem
Suppose we have trained binary classifier and want to predict value of [x1, ..., x5] with associated timestamps [t1, ..., t5]. We get the prediction as following: [0.25, 0.99, 0.1, 0.75, 0.79].
Assume that I have the domain knowledge to say that probability of positive class must not change abruptly. Jumps like from 0.99 at t2 to 0.1 at t3 cannot occur in real application.
Questions

Can I enforce smooth output constraint on (any/some) classifier?
Does applying moving average on the prediction probability to smooth it make sense?


texts: How to Implement Biological Neuron Activations in Artificial Neural Networks

texts:
In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation:

Which can be implemented in code as (in Python):

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

How can we implement a biological activation function, such as the Hodgkin-Huxley model, whose mathematical form is:

Where:

Cm: Capacitance
Vm: Membrane potential

As mentioned on the Wikipedia page,

  The typical Hodgkin–Huxley model treats each component of an excitable
  cell as an electrical element (as shown in the figure). The lipid
  bilayer is represented as a capacitance (Cm). Voltage-gated ion
  channels are represented by electrical conductances (gn, where n is
  the specific ion channel) that depend on both voltage and time. Leak
  channels are represented by linear conductances (gL). The
  electrochemical gradients driving the flow of ions are represented by
  voltage sources (En) whose voltages are determined by the ratio of the
  intra- and extracellular concentrations of the ionic species of
  interest. Finally, ion pumps are represented by current sources
  (Ip).[clarification needed] The membrane potential is denoted by Vm.


texts: I'm searching for an online Jupiter notebook environment that allows collaborating in real-time. I've found that I've choice from two services Deepnote.com and DataLore by JetBrains. I have little experience with Deepnote and it starts to perform very poorly when you have a lot of cells in your notebook. If anyone has experience with two environments, does the DataLore suffer the same issue? What advantages and disadvantages do DeepNote and DataLore have relative over each other? Are there any other real-time collaboration environments?


texts: Missing Categorical Features - no imputation

texts:
I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority feature in given row/column.

Is there a better way to approach missing data? Why can't the classifier just ignore the missing feature, and rely on the known features? Why is imputation necessary?

I'm using scikit learn, and am trying to feed in NaN to a classification model (naive bayes, logistic regression, decision tree, random forest), to see what happens.


texts: Can I create random forest with RandomForestClassifier which will consist the same trees?

texts:
Based on answers to this question, I should be able to build a random forest with all the same trees by using bootstrap = False, max_features = None, random_state = 42 parameters.

I wrote quick code to test it, and it seems that different trees are created.

Is it possible to create a random forest using RandomForestClassifier which will produce the same trees?


texts: How best to convert label classification into regression?

texts:
I have a dataset of genes for which I'm trying to predict genes that cause a disease. Originally I was doing this with a multilabel classification. I had 3 groups: I labeled already known disease-causal genes as positive causal examples, I labeled genes with research in the disease as probable, and I labeled a final group as negative (with evidence they are non-causal).
I only have 50 known positive labeled examples, ~150 genes in the probable group, and ~250 in the negative group. I was struggling with the class imbalance so I have converted the problem into a regression with converting the labels to scores between 0 and 1.
However, my question is, I don't have exact guiding criteria to choose which label gets what score between 0 and 1 - can I test the model on several scoring intervals and take the top performing model? Or is this biased?
For example I've run a model where the 3 labels are converted to scores with equi-distant intervals (e.g. 0.9, 0.6 and 0.3, or  1, 0.5 and 0) they all perform almost the exact same. I tried scoring the genes in ways that I thought reflect the labeling criteria (from a biological perspective of how similar the criteria is for known and probable genes are). For example, scoring 1 for known/positive, 0.6 for likely and 0.1 for negative the models perform a bit better.
In general, the models do better about +0.02-0.05 better in nested cross-validation with performance metrics like r2 or MSE when the known and probable genes have closer scores together and the negative group is lower/further away in score - but I'm wondering if I can even use this, from a data science perspective is the converting some scores to be closer together and others to be further away biasing the model or is the +0.05 performance increase genuine? If I have a domain-based knowledge reason for why a known gene is 1 and a likely gene is 0.6 etc. is it then acceptable?


texts: How to use sklearn's Matrix factorization to predict new users' scores

texts:
I'm trying to use sklearn.decomposition.NMF to a matrix R that contains data on how users rated items to predict user ratings for items that they have not yet seen.
the matrix's rows being users, columns being items, and values being scores, with 0 score meaning that the user did not rate this item yet.
Now with the code below I have only managed to get the two matrices that when multiplied together give the original matrix back.
import numpy

R = numpy.array([
     [5,3,0,1],
     [4,0,0,1],
     [1,1,0,5],
     [1,0,0,4],
     [0,1,5,4],
    ])

from sklearn.decomposition import NMF
model = NMF(n_components=4)

A = model.fit_transform(R)
B = model.components_

n = numpy.dot(A, B)
print(n)

Problem is, that the model does not predict new values in place of 0's, that would be the predicted scores, but instead recreates the matrix as was.
How do I get the model to predict user scores in place of my original matrix's zeros without underfitting?
Also, the model wont accept np.nan as matrix inputs so I can't see a way to communicate that the users have not yet rated the items.


texts:
I'm trying to restore this function:

$$ F(x) = x*sin(\alpha x)+b; \space\space \alpha,b \in (-20,20) $$

My NN model(with Keras) is:

1 layer: GRU, 9 neurons, selu activation
2 layer: GRU, 3 neurons, selu activation
1 layer: GRU, 7 neurons, selu activation
1 layer: Dense, 1 neuron, linear activation
kernel init he_normal for all layers

For the training dataset I've generated function values $f(x), x \in -5..5$ (e.g. linspace(-5,5,500) and random integer values for $\alpha$ and $b$, 250 times.

Then I've selected $f[i-1]$ and $f[i]$ (previous steps) and for this "xs" the output is $f[i+1]$. So the training dataset is like:

X:
   first row: previous previous $f$, $\alpha$, $b$
   second row: previous $f$, $\alpha$, $b$

Y:
   current $f$ value

After training over 200 epochs, mae on validation data was 0.1851.

Now if I try to predict new data it seems normal, but when I try to predict new values using previously predicted by model points it falls down and it doesn't looks like sine function at all.

What am I doing wrong?


texts: I know my annotators are not perfect, sometimes making mistakes. What would be the best way to deal with the annotation errors for my training data?


texts: How to report results of RL with high variance?

texts:
I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 runs and 40000 episodes; however, it seems that 500 runs are not sufficient, and because of timing, it is difficult to increase the number of runs.
This graph shows reward_q which is a vector of 40000 and each element number shows the summation of rewards in each episods for 500 runs, and I draw the graph based on number of episodes vs reward_q/n_runs.
Note: The title of the x-axis is not correct.


texts: What is the kernel matrix used for in the kernel trick?

texts:
I have $n$ linearly inseperable datapoints, $x_1 \dots , x_n$. I use the kernel trick to map and compute the dot product in higher dimensions (without actually mapping / transforming the data).

Assume I used the kernel $k(x,y).$ The entry at position $i,j$ of an associated "kernel matrix" is defined as $k(x_i,x_j).$ This matrix is apparently also known as the "Gram" matrix.

What is this matrix used for? Why do I need it? This is in the context of support vector machines, if it matters.


texts: what does `a factor of K-fold` mean in GPU-based training?

texts:
From the book "Deep Learning and Convolutional Neural Networks for Medical Image Computing"

  As we learned about the current state of research on deep learning, I
  was surprised to find that other investigators had used convolutional
  neural networks, one type of deep learning, in the past [22, 23]. But
  there seemed to be something different about the most recent crop of
  deep learning algorithms. They routinely used GPU processing to
  accelerate training by as much as a factor of 40-fold. They also used
  multiple convolution layers and multiple data reduction layers.

does 40-fold mean "K-Fold Cross-Validation"?


texts: Keras model giving error when fields of unseen test data and train data are not same

texts:
I have created a simple Keras deep learning model in python. Total no of variables in training are 195 while in unseen test data are 181.All input fields are categorical(converted by one hot encoding). Since unseen test data has some different categories thats why after one hot encoding fields are not matching with train.

So during predict step on unseen test data, model gives the following error. Is there any way out?

ValueError: Error when checking input: expected dense_30_input to have shape (195,)
but got array with shape (181,)


texts:
Can anybody explain why/if target variable transformations could help when dealing with tree based models?

I've seen this excellent reply which explains quite well why it shouldn't affect if transforming inputs, but I haven't been able to find anything regarding outputs.

Can using a transformation like taking logs or using quantile transform of the response variable help?

Actually, we are using XGBoost and getting better results when using a normal quantile transform of our output and even better results when taking logs (our response variable is a price, highly skewed to the right) but I don't know if this is something justifiable by theory or just random chance.


texts: Behavioural data required to predict churn

texts:
I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all transactions occur within 60 days of one another so this feels reasonable.

I have very limited behavioural data; however. I have a record of a user's transactions and I have access to Google Analytics (GA). GA does not, however, allow me to track the specific actions of a user - everything is aggregated across the entire user base.

I have recommended that we track and keep a record of users' navigation across the website, buttons clicked, times logged in etc. but this has been rejected as it has been deemed unnecessary because we have Google Analytics. Am I being unreasonable? And are there any workarounds. I don't think I'm going to be able to make much of a churn forecasting model with basic information like a user's address, age, and transactions.


texts: I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot the clusters in an n-dimensional space. Which options do I have?


texts: How to apply multiple filter in Data Frame?

texts:
How to implement multiple filters for checking data cell in a range ?
Suppose, I have a list of numbers like,

range_1 = [ 70 ,15,5,7,3,7,8,3,2, 63  ]

# and

range_1 = [ 50, 56, 80, 61, 83, 87, 13, 58, 43, 24, 84, 54, 64,36, 48  ]

And I want to check any column values exist within these two lists.

Any suggestion would be appreciated


texts: How to make two parallel convolutional neural networks in Keras?

texts:
I created two convolutional neural networks (CNN), and I want to make these networks work in parallel. Each network takes different type of images and they join in the last fully connected layer.

How to do this?


texts: How to normalize data for Neural Network and Decision Forest

texts:
I have a data set with 20000 samples, each has 12 different features. Each sample is either in category 0 or 1. I want to train a neural network and a decision forest to categorize the samples so that I can compare the results and both techniques.

The first thing I stumbled upon is the proper normalization of the data. One feature is in the range $[0,10^6]$, another one in $[30,40]$ and there is one feature that mostly takes the value 8 and sometimes 7. So as I read in different sources, proper normalization of the input data is crucial for neural networks. As I found out, there are many possible ways to normalize the data, for example:

Min-Max Normalization: The input range is linearly transformed to the interval $[0,1]$ (or alternatively $[-1,1]$, does that matter?)
Z-Score Normalization: The data is transformed to have zero mean and unit variance:
$$y_{new}=\frac{y_{old}-\text{mean}}{\sqrt{\text{Var}}}$$

Which normalization should I choose? Is normalization also needed for decision forests? With Z-Score normalization, the different features of my test data do not lie in the same range. Could this be a problem? Should every feature normalized with the same algorithm, so that I decide either to use Min-Max for all features or Z-Score for all features?

Are there combinations where the data is mapped to $[-1,1]$ and also has zero mean (which would imply a non-linear transformation of the data and hence a change in the variance and other features of the input data).

I feel a bit lost because I can't find references which answer these questions.


texts: Is it necessary to transform data to normal distribution when removing outliers for xgboost?

texts:
sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can see that the distributions for each numerical column is very skewed.

there is a need to remove outliers for xgboost (since based on residuals) so i would like to remove them. my intended method is as follows:

1) transform all columns to normal distribtuon using box cox (not log transform as  i have some negative fieds in my columns e.g. -4567.

2)remove outliers using quantile based method..

does the above seem along the right lines?

note i know in practice you should not drop outliers but surely this depends on your model. since xgboost fits a loss function on the residials i think it makes sense to remove these values.


texts:
I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I have searched high and low but could not find anything like this available. I already have the news articles in each language.

My question to the community is how would you achieve this as accurately as possible?

I was first looking at Mechanical Turk, where you can hire people to  label each article manually for you. And this may be the best way forward but expensive.

Next, I thought about all of those existing popular libraries (some of whom have already used Mechanical Turk) that do sentiment analysis (AFINN, Bing Liu, MPQA, VADER, TextBlob, etc.)

Sentiment Idea

My current idea is that I run each news article across a few of these libraries (for example AFINN, then TextBlob, then VADER) and for those articles who show positive, negative, neutral unanimously though all three libs are accepted into the corpus. Does that seem like a fairly strong and reasonable verification process?

Language Idea

The next issue pertains to language itself. The 3 lib pipeline above can be executed in English with no issue. However these libraries do not uniformity support many other languages (Spanish, German, Chinese, Arabic, French, Portuguese, etc.) I was thinking about doing what VADER suggests and taking the news stories in non-english languages and sending them though Google Translation API to get them into English and then send them through the existing 3 lib pipeline above. I do realize there will be a loss in semantics for many articles. However, my hope is that enough articles will translate well enough that some pass through the 3 lib pipeline.

I am aware that translating and sending news articles through this triple blind sentiment pipe may take a 100k corpus and yield 10k results. I am fine with that. The accuracy and then price is my concern. I can easily acquire more data.

What would you do that may be a more accurate way of achieving a sentiment corpus of news articles? Is there an existing best used practice for assembling a corpus like this?


texts: Which are the appropriate prameters for lda modeling?

texts:
I try to implement in R test for appropriate metrics for lda.

Here the way I try to use LDA

require(quanteda)
    require(quanteda.corpora)
    require(lubridate)
    require(topicmodels)
dtext <- data.frame(id = c(1,2,3), text = c("This dataset contains movie reviews along with their associated binary sentiment polarity labels. It is intended to serve as a benchmark for sentiment classification. This document outlines how the dataset was gathered, and how to use the files provided.", "The core dataset contains 50,000 reviews split evenly into 25k train and 25k test sets. The overall distribution of labels is balanced (25k pos and 25k neg). We also include an additional 50,000 unlabeled documents for unsupervised learning.", "There are two top-level directories [train/, test/] corresponding to the training and test sets. Each contains [pos/, neg/] directories for the reviews with binary labels positive and negative. Within these directories, reviews are stored in text files named following the convention [[id]_[rating].txt] where [id] is a unique id and [rating] is the star rating for that review on a 1-10 scale. For example, the file [test/pos/200_8.txt] is the text for a positive-labeled test set example with unique id 200 and star rating 8/10 from IMDb. The [train/unsup/] directory has 0 for all ratings because the ratings are omitted for this portion of the dataset."),stringsAsFactors = F)

    corp_news <- corpus(dtext)

    dfmat_news <- dfm(corp_news, remove_punct = TRUE, remove = stopwords('en')) %>%
        dfm_remove(c('*-time', '*-timeUpdated', 'GMT', 'BST')) %>%
        dfm_trim(min_termfreq = 0.95, termfreq_type = "quantile",
                 max_docfreq = 0.1, docfreq_type = "prop")

    dfmat_news <- dfmat_news[ntoken(dfmat_news) > 0,]
    dtm <- convert(dfmat_news, to = "topicmodels")
    lda <- LDA(dtm, k = 10)

Which are the parameters which I have to test to make the perfomance of lda better?
example there methods for the topic selection number
algorithms like gibbs, recover and recoverl2
what other tunning it needs?


texts: How can I get rid of the white space in matplotlib?

texts: I am doing a course in edx. While plotting in python, there is always a blank space before the beginning of the plot. In StackExchange, I have found a solution, which is mentioning the axis limit. However, in maximum cases, the starting points of the plot is unknown. So, Is there any solution to this without manipulating the axis limits manually? A graph is attached herewith for making things more clear.


texts: Plotting different values in pandas histogram with different colors

texts:
I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used pandas.DataFrame.hist() method which gave me a plot as shown below:

I want to represent the distribution for each value in a column with different color. For example, in column 1, all the values corresponding to '0' should be in red color while the values corresponding to '1' in green color and so on. How can I do this? Please help!


texts:
alueError: Unexpectedly found an instance of type <class 'numpy.ndarray'>. Expected a symbolic tensor instance.
That's the error message from Keras Backend I get when I try to predict. I am using a CTCloss function from this example https://github.com/ysoullard/CTCModel/blob/master/CTCModel.py the test set is a numpy array [features_test, input_lengths_test]. I tried converting it to a tensor but didn't convert


texts: normalizing data and avoiding dividing by zero

texts:
I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly detection algorithm and see what happens.

I would like to normalize the data for the autoencoder so my values are either between 0,1 or -1,-1 because my output activation function will either be a sigmoid or tanh. This way my algorithm can train and the input will be in the same range as the output values of the NN.

However, when I normalized with

x(i)-xmean/(xmax-xmin)

I ended up dividing by 0 in several features of the data which gave NaN. Is is possible to normalize my data so it is between -1,1 or 0,1 while avoiding dividing by 0 for my data?


texts: What happens when you have highly correlated columns in a dataset?

texts:
I am doing a regression model. And I was wondering what would be the consequence if we have two or more Highly correlated columns in the dataset? Is that something that can decrease the accuracy of the model?
Answering this question would help decide how to deal with it. Would PCA be the best option here?


texts: I'm trying to implement Topic Modeling via Python & NLP but can't figure out what algorithm should I use. I have studied Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA) and Non-negative matrix factorization (NMF) but how to decide which algorithm fits best for certain task? If I just try all of them in a row then how to measure the result?


texts: How to get phrase tables from word alignments?

texts:
The output of my word alignment file looks as such:

I wish to say with regard to the initiative of the Portuguese Presidency that we support the spirit and the political intention behind it . In bezug auf die Initiative der portugiesischen Präsidentschaft möchte ich zum Ausdruck bringen , daß wir den Geist und die politische Absicht , die dahinter stehen , unterstützen .   0-0 5-1 5-2 2-3 8-4 7-5 11-6 12-7 1-8 0-9 9-10 3-11 10-12 13-13 13-14 14-15 16-16 17-17 18-18 16-19 20-20 21-21 19-22 19-23 22-24 22-25 23-26 15-27 24-28
It may not be an ideal initiative in terms of its structure but we accept Mr President-in-Office , that it is rooted in idealism and for that reason we are inclined to support it .    Von der Struktur her ist es vielleicht keine ideale Initiative , aber , Herr amtierender Ratspräsident , wir akzeptieren , daß sie auf Idealismus fußt , und sind deshalb geneigt , sie mitzutragen .   0-0 11-2 8-3 0-4 3-5 1-6 2-7 5-8 6-9 12-11 17-12 15-13 16-14 16-15 17-16 13-17 14-18 17-19 18-20 19-21 21-22 23-23 21-24 26-25 24-26 29-27 27-28 30-29 31-30 33-31 32-32 34-33

How can I produce the phrase tables that are used by MOSES from this output?


texts: Keras : Classifying names per origin using RNN and an embedding layer

texts:
I am trying to classify names with character RNN using embedding (similar to the PyTorch example https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html)

The problem is that it is showing me 94% from first epoch, and it's not clear on how to read prediction from it, my code is :

    model = tf.keras.models.Sequential(layers=[
  tf.keras.layers.Embedding(input_dim=33,
                            output_dim=50,
                            mask_zero=True),
  tf.keras.layers.LSTM(units=64, return_sequences=True),
  #tf.keras.layers.LSTM(units=64),
  tf.keras.layers.Dense(64, activation="relu"),
  tf.keras.layers.Dense(18, activation='softmax')
])

So the question is how to predict a name using this model?
I tried :

prediction = model.predict([sequences_test])
prediction.shape

and its returning me :

(5015, 18, 18)

now how to check the first prediction?


texts: Why do we divide the regularization term by the number of examples in regularized logistic regression?

texts:
So this is the formula for the regularized logistic regression cost function:

$x^{(i)}$ - the $i$'th training example

$\theta_j$ - the parameter of the $j$'th feature

$m$ - the number of training examples

$n$ - the number of features

$y_{(i)}$ - the actual outcome of the $i$'th training example, such that $y
\in \{0, 1\}$

$\lambda$ - the regularization term

$h_\theta$ - the hypothesis function that produces a prediction in the interval $(0, 1)$

$h_\theta(x^{(i)})$ - predicted value of the $i$'th training example, such that:

$h_\theta(x^{(i)}) = \sigma(\theta^{T}x^{(i)})$, where $\sigma(z) = \frac{1}{1+e^{-z}}$ (the sigmoid/logistic function)

My question is about that last term:

From my understanding, in order to do regularization, we need to find the sum of all the squares of the parameters $\theta$. That is clear enough. Also we multiply this sum by the regularization term $\lambda$, which we can change if we think the data is overfit/underfit. Good. Then, for convenience we divide this term by $2$ such that when we take the derivative we will get rid of that $2$ that would come from the exponent of $\theta$. All clear so far. BUT, why in the world do we also divide by $m$ (the number of training examples)? We divide the left term by $m$ in order to find the average error, and that makes sense since we have $m$ examples, therefore $m$ errors and after we find the sum of these $m$ errors, we would need to divide by $m$ to get the average error. But in this right term that I am confused about, we find the sum of the squares of the features, and the number of features is $n$. If we want to find the average of all the $\theta_j^2$ wouldn't we need to divide by $n$ instead of $m$, since we have $n$ features. Why does it make sense to divide that sum by $m$ and shouldn't we divide it by $n$?


texts: Model accuracy when training on GPU and then inferencing on CPU

texts:
When we are concerned about speed, GPU is way better than CPU.
But if I train a model on a GPU and then deploy the same trained model (no quantization techniques used) on a CPU, will this affect the accuracy of my model? Can the accuracy of the same model degrade on a CPU?
My intuition says, GPU vs CPU should not make any difference if accuracy is concerned.
But I have one doubt that, the GPUs and CPUs have their own different ways to process the information internally. Both of them have different architecture. When a model is trained on a GPU, does the exact same way of processing happens when trained on a CPU but in much slower way? I am not concerned about the accuracy while training, but if a model was trained on a GPU, will it perform exactly with same accuracy on a CPU?


texts: CNN Architecture for Multiple Instance Learning

texts:
I have a binary classification problem where I have a bag of documents (image files) that I need to classify - the bag that is, not the individual document. However, a bag can have a different number of documents and the combination of documents will determine the classification. E.g.

Bag1: A, B, C     label = Pass
Bag2: D, E                Fail
Bag3: F                   Pass

In Bag1, the presence of A, B, or C alone may not be enough to Pass, but together they do. In Bag3, F itself might be enough to pass.

I have already done OCR to represent all the text at the Bag level. For the images, all I can think is to average the pixel values of each image in the Bag to represent it as a single image, and then train with a CNN.

Is there any better architecture or method for handling something like this?


texts:
There is one behavior of labelbinarizer

import numpy as np
from sklearn import preprocessing
lb = preprocessing.LabelBinarizer()
lb.fit(np.array([[0, 1, 1], [1, 0, 0]]))
lb.classes_

The output is array([0, 1, 2]). Why there is a 2 there?


texts: Selecting estimator method for network performance

texts:
Wireless network parameters latency, PER, bandwidth is impacted by Channel occupancy percentage  , RSSI , client type.

To estimate if per , latency will cross threshold from continuously reported input parameters which estimation model can be used.
Does kalman filter apply here.

I understand kalman filter requires output to be related to input in equation. This is not case here. Here , set of input parameters impact output , there is high degree of correlation but not direct formula to derive output from input.
Kindly guide.


texts: How to apply Gradient Descent to a summ of function

texts:
My target is to find a center of a circle that approximate a set of dots

i want to find minimum of a function:
$$\sum_{i=0}^N (\sqrt{(x_i - a)^2 + (y_i - b)^2} - R)^2$$

this function represent an error of a my approximation of a set a dots on a plane with a circle.

I did a bit of a googling and find that Gradient Descent is a decent method for a numerical searching for a minimum of function.

But i have a trouble with understanding how i should correcnt my A,B,R with partial derrivative, when i have function of summ.

picture related for better explanation of my problem:
http://imgur.com/gallery/ecUdXnB


texts: Do you have to normalize data when building decision trees using R?

texts:
So, our data set this week has 14 attributes and each column has very different values. One column has values below 1 while another column has values that go from three to four whole digits.

We learned normalization last week and it seems like you're supposed to normalize data when they have very different values. For decision trees, is the case the same?

I'm not sure about this but would normalization affect the resulting decision tree from the same data set? It doesn't seem like it should but...


texts: How to determine the decrease of resolution in per cent necessary for a partly blurred frame to appear sharp?

texts:
Given is a frame of a video taken with a lens focussing on the background, so the foreground is slightly blurred.
How to determine the decrease of resolution in per cent necessary for the blurred areas in all or most frames to appear sharp?
My approach:

Determine blurried areas(How?)
Average to determine an average amount of blurriness, for instance 4 pixels should be one.
Divide the resolution by 4.


texts: Correct way to set Spark variables in jupyter notebook

texts:
I need to set a couple of variables in my Jupyter Notebook where I have a pre-existing sparkContext and sqlContext and I am doing it wrong. If I don't include an
    sc.stop()
, I get an error that I am trying to instantiate a second context. If I do include it, I get an error that I am trying to call methods on a stopped context.

Can someone tell me the correct way to set these variables?

Here is my code:

from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
conf = (SparkConf()
       .set("spark.yarn.executor.memoryOverhead", "4096")
       .set("spark.kryoserializer.buffer.max.mb", "1024"))

sc.stop()
sc = SparkContext(conf=conf)
sqlContext = SQLContext.getOrCreate(sc)


texts: Hybrid Convolutional and Conventional Neural Networks

texts:
Convolutional Neural Networks are usually the best choice for classification and semantic segmentation of images. categorical/numerical data (age, height, city, etc.) on the other hand are best processed by conventional machine learning models, such as (deep) Random Forests, Support Vector Machines or Conventional Neural Networks.

Are there hybrid architectures combining both convolutional and conventional Neural Networks for classification of datasets consisting of images and categorical data? I am sure this problem has been solved before and I am specifically looking for research papers, tutorials and implementations in one of the common libraries (PyTorch, Tensorflow, Keras, etc.).

A good example for such a "hybrid" dataset is the ISIC dataset, containing thousands of images of skin growths together with information such as the age and the sex of the patient for each image. Hypothetically, a hybrid model that includes this categorical/numerical information could be more succesful in detecting skin cancer, than a model which just uses the images.


texts: How do I prepare data in which each output row depends on multiple input rows?

texts:
My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y.

In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on limits and balances of their credit cards (table 2). The challenge is that customer 1 has two credit cards, whereas customer 2 has one credit card.

How do I map table 2 to table 1 in this scenario?

(Of course I can create summary statistics, but I do not want to bias the model.)

Table 1: Bankruptcy filing

customerId | Customer filed for bankruptcy (Y)
1            1
2            0

Table 2: Customer credit cards

customerId | creditCardId | Credit limit (X1) | Credit used (X2)
1            1              5                   5
1            2              9                   8
2            1              10                  1


texts: An error coming out when fitting a model with keras

texts:
I am trying to build a model using kerras but for some reason I receive a ValueError: Error when checking target: expected activation_5 to have 4 dimensions, but got array with shape (50000, 10) error. I have recently changed the model parameters, so I think it has something to do with that, but I cannot understand what exactly. Before I changed the model everything worked

Here is the code

from keras.models import Sequential, load_model
from keras.layers import Dense, Dropout, Flatten, BatchNormalization, Activation
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.constraints import maxnorm
from keras.utils import np_utils
from keras.datasets import cifar10
from tensorflow.python.keras.callbacks import TensorBoard
from time import time
seed = 21
tensorboard = TensorBoard(log_dir="logs/{}".format(time()))

#loading and preprocessing training and testing data
(train_data, train_labels), (test_data, test_labels) = cifar10.load_data()
train_data = train_data.astype('float32')
test_data = test_data.astype('float32')
train_data = train_data / 255.0
test_data = test_data / 255.0
train_labels = np_utils.to_categorical(train_labels)
test_labels = np_utils.to_categorical(test_labels)
class_num = test_labels.shape[1]

#building model
model = Sequential()
model.add(Conv2D(32, (5,5), input_shape=(32,32,3), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Conv2D(32, (5,5), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Conv2D(64, (5,5), padding ='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same'))
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

epochs = 50
optimizer = 'adam'
model.summary()
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

numpy.random.seed(seed)
model.fit(train_data, train_labels, validation_data=(test_data, test_labels), epochs=epochs, batch_size=64, callbacks=[tensorboard])
scores = model.evaluate(test_data, test_labels, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
```


texts: How to interpret Correlation along with Coefficients of multiple linear regression ?£

texts:
I have 10000 samples. There are 4 independent variables and 1 dependent variable.
The independent variables are all centered with 0 mean.

I found the correlation coefficients between each of these variables which are as below:

I used linear regression model and below is the summary of that model:

Now, based on the coefficients of the predictor variables in the linear regression model, I have been asked to find the significant predictor (s).
Based on just the correlation values, I was thinking X 4 will be the significant predictor but its regression coefficient says a different story altogether. (x4 has the least coefficient value in lm summary output). Can you help me understand what exactly is the correct way to identify the significant predictor (s)?
Also, Additionally even if I remove the x4 variable from the lm model, the Residual standard error remains the same which kinda re-iterates the fact the it is not a significant predictor? Is
my understanding correct here?
Also, I ran the VarImp function available in R which again returned a smaller value for x4.
> varImp(lm_df, scale = TRUE)
     Overall
x1 33.673993
x2 34.858260
x3 33.820908
x4  1.969445


texts: Suspiciously low False Positive rate with Naive Bayes Classifier?

texts:
I am performing phishing URL classification, and I am comparing several ML classifiers on a balanced 2-class data-set (legitimate URL, phishy URL).

The ensemble and boosting classifiers such as Random Forest, Ada Boost, Extra Trees etc AND K-NN  achieve an accuracy about 90%, and a False Positive Rate about 11-12%. (fig.)

On the other hand, classifiers such as SVM, Logistic Regression, Multinomial NB and Bernoulli NB seem to perform poorly with accuracies fluctuating between 70% - 80% and higher false positives.

Here is the thing.
I also tried Gaussian NB and although it yields by far the worst accuracy 58.84% it has an incredibly low False Positive Rate 2.14% (and thus a high FNR)

I have no idea why this is happening. Any ideas?
Why some classifiers perform so poorly and others not?

I parametrized them all with Grid Search, they are used on the same dataset (about 30k records of each class) and I perform a 3-fold cross validation. It doesn't make any sense to me, especially for SVM. At last I use about 20 features.

P.S: I use python's sk-learn library


texts: Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?


texts: How to show percentage (text) next to the horizontal bars in matplotlib?

texts:
I am using seaborn's countplot to show count distribution of 2 categorical data. Fine it works but I want the percentages to show on top of the bars for each of the plot. Please how do I do it?

fig, ax = plt.subplots(1, 2)
sns.countplot(y = df['current_status'], ax=ax[0]).set_title('Current Occupation')
sns.countplot(df['gender'], ax=ax[1]).set_title('Gender distribution')

I have made edits based on the comments made but I can't get the percentages to the right of horizontal bars. This is what I have done.

total = len(df['current_status'])*1.
ax = sns.countplot(y="current_status", data=df)
plt.title('Distribution of  Configurations')
plt.xlabel('Number of Axles')

for p in ax.patches:
        ax.annotate('{:.1f}%'.format(100*p.get_height()/total), (p.get_y()+0.1, p.get_height()+5))
_ = ax.set_xticklabels(map('{:.1f}%'.format, 100*ax.xaxis.get_majorticklocs()/total))


texts: What are the reasons for drawing initial neural network weights from the Gaussian distribution?

texts: Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?


texts:
I'm attempting to implement the policy gradient taken from the "Hands-On Machine Learning" book by Geron, which can be found here. The notebook uses Tensorflow and I'm attempting to do it with PyTorch.

My models look as follows:

model = nn.Sequential(
    nn.Linear(4, 128),
    nn.ELU(),
    nn.Linear(128, 2),
)

Criterion and optimisers:

criterion = nn.BCEWithLogitsLoss()
optim = torch.optim.Adam(model.parameters(), lr=0.01)

Training:

env = gym.make("CartPole-v0")

n_games_per_update = 10
n_max_steps = 1000
n_iterations = 250
save_iterations = 10
discount_rate = 0.95

for iteration in range(n_iterations): # Run the game 250 times
    all_rewards = []
    all_gradients = []
    n_steps = []
    optim.zero_grad()
    for game in range(n_games_per_update): # Run the game 10 times to accumulate gradients
        current_rewards = []
        current_gradients = []
        obs = env.reset()
        for step in range(n_max_steps): # Run a single game a maximum of 1000 steps

            logit = model(torch.tensor(obs, dtype=torch.float))
            output = F.softmax(logit, dim=0)
            c = Categorical(output)
            action = c.sample()

            y = torch.tensor([1.0 - action, action], dtype=torch.float)
            loss = criterion(logit, y)
            loss.backward()

            obs, reward, done, info = env.step(int(action))
            current_rewards.append(reward)
            current_gradients.append([p.grad for p in model.parameters()])
            if done:
                break
        n_steps.append(step)

        all_rewards.append(current_rewards)
        all_gradients.append(current_gradients)

    # Performs the discount and normalises
    all_rewards = discount_and_normalize_rewards(all_rewards, discount_rate=discount_rate)

    # For each batch of 10 games multiply the discounted rewards against the gradients of the
    # network. Then take the mean for each layer
    new_gradients = []
    for var_index, gradient_placeholder in enumerate(gradient_placeholders):
        means = []
        for game_index, rewards in enumerate(all_rewards):
            for step, reward in enumerate(rewards):
                means.append(reward * all_gradients[game_index][step][var_index])
        new_gradients.append(torch.mean(torch.stack(means), 0, True).squeeze(0))

    # Apply the new gradients to the network
    for p, g in zip(model.parameters(), new_gradients):
        p.grad = g.clone()
    optim.step()

When I run the code for 250 interactions I print the average game length I get:

Iteration: 50, Average Length: 18.2
Iteration: 100, Average Length: 23.4
Iteration: 150, Average Length: 29.9
Iteration: 200, Average Length: 11.2
Iteration: 250, Average Length: 38.6

The network isn't really improving and training for longer doesn't help. My two questions are:
1. Is there anything obviously wrong that I'm doing?
2. I've notices the log of the probability is used in the tensorflow implementation, but I'm not sure how to integrate it here


texts: Cluster fusion techniques for global optimal solution

texts: Suppose you have a huge data set. You divide that data set into N number of blocks. Each block is then clustered into M blocks (using any clustering algorithms like K means). Now you have N local clustering solutions. Are there any methods to combine these local clustering solutions to  get a globaly optimal solution?


texts: What activation function should I use for a specific regression problem?

texts: Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain net size and I would like to avoid creating a very big net, also sigmoid is much more prefered but in my case regression will output values from range (0, 1e7)... maybe also sigmoid net with linear head will work? I am curious about your take on the subject.


texts: Saving a different model for every epoch Keras

texts: As part of a project I am currently running a CNN for classifying images. Right now I am training with 50 epochs. I have noticed that after around 20-30 epochs I reach the highest accuracy and then the loss starts to increase significantly. I thought about using Early Stopping but my model loss has a lot of zig-zagging so I think it would be difficult to find the right patience argument to use. I think it would be easier to just save a model for every epoch so I can manually select the one with the highest accuracy. Is this possible in Keras?


texts:
This question is asked several times here on SE, but I havent been able to find the right answer.
I'm trying to build a network with 1 input and 2 outputs. I don't have a lot of data so I would like to use a generator for augmentation (preferably with imgaug).
My code:

seq = iaa.Sequential([ ....
])

gen = ImageDataGenerator(preprocessing_function=seq.augment_image)
batch_size = 64

def generate_data_generator(generator, X, Y1, Y2):
    genX = gen.flow(X, batch_size=batch_size, seed=42)
    genY1 = gen.flow(Y1, batch_size=batch_size, seed=42)
    while True:
            Xi = genX.next()
            Yi1 = genY1.next()
            Yi2 = function(Y2)
            yield Xi, [Yi1, Yi2]

H = model.fit_generator(generate_data_generator(gen, trainX, trainY1, trainY2),
                epochs=epochs,
                steps_per_epoch=len(trainX) // batch_size,
                validation_data=(testX, [testY1, testY2]))

With this, I get the error:

'ValueError: ('Input data in `NumpyArrayIterator` should have rank 4. You passed an array with shape', (115, 16))'

a normal fit like this works just fine, so I don't there is anything wrong with the normal input

H = model.fit(trainX, {"output1": trainY1, "output2": trainY2},
    validation_data=(testX, {"output1": testY1, "output2": testY2}),
    epochs=epochs)

This is my input, I have no idea how I get a shape of (115,16) here

trainX.shape
trainY1[0].shape
trainY2[0].shape

Out[181]:
(115, 158, 100, 3)
Out[181]:
(16,)
Out[181]:
(9,)

thank you


texts: Efficient way of adding new columns to datamart without reprocessing complete pipeline

texts:
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for dependency management and scheduling. The tables that I am building out will have a couple of years of historical data and will be backfilled using Airflow. I have a few seat of initial columns for consumption. As I get more requirements from partner teams, I have to keep adding more columns to the datamart. however, since the script to generate data has older columns and new ones, I will be recomputing all data for existing columns (few years) every time I re-run a DAG. This doesn't seem efficient. The alternative I see is

have new jobs every time there are new columns to be added. Now, I will have multiple jobs for groups of metrics. Also multiple sets of files which have to manually joined by a key for people who are consuming this data. This doesnt seem scalable.
have all fields ready before starting implementation. waterfall model. again, not happening in this agile industry

What are the industry best practices around this ? I asked internally and I was told I would have to re-run the whole job with both old and new columns.


texts: Output of classifier.predict Tensorflow extract probabiltity

texts:
When I do a prediction with my DNN clasifier I get a dictionary like this.

{'probabilities': array([9.9912649e-01, 8.7345875e-04, 8.5633601e-12], dtype=float32), 'logits': array([ 12.641698,   5.599522, -12.840958], dtype=float32), 'classes': array(['0'], dtype=object), 'class_ids': array([0])}

Can someone explain me the values of probability and logits? Why the three values ?

The docs just states

  Evaluated values of predictions tensors.

And do not refer (the docs) to a struct/explanation of the output

Thanks!


texts: Specifying neural network output layout for object detection

texts:
I have a question about if the modeling of the output detection affects the neural nets capability.
In my case I want to train a CNN for object recognition and classification.
As an output I want to get the object classification and a bounding box where it roughly is in the image.
So my thoughts were to model the output as a sequence of vectors:

c = vector for object classification has the length of o (total count of objects to classyify)
l = vector for the bounding box with x, y, width, height

so the complete sequence would look like:

Output = (c + l) x N
where N is the maximum count of objects i would like to detect

Does the order of the subsequence (c + l) have an affect on the effectiveness of the training? Do I have to specify an order, e.g the sequences get ordered by size?

Secondly I have another question, with the specific output format I got numbers in range from 0-1 in the vector c, but numbers from range from 0 -the  size of the image in the bounding box vector l. Does this also affect the training if the activation is set to "relu"? Would it be better to normalize the bounding box vector to the image width and height or the input width and height? So all number have an equal range?

These questions came into my mind while labeling the dataset? So I don't want to waste my time labeling the dataset the wrong way :/

Thank you very much for your help!


texts: Preserve colour in convolutional autoencoder

texts:
at the moment i work with convolutional autoencoder and now I'am looking for paper or methods that adresses a colour preversation. Most of the AE paper use grayscale images and loss functions such as SSIM that preserve the structure very well are also focused on grayscale images. My networks are good in preserving structure (with SSIM as loss) but have a hard time representing the right colour.

I use a all convolution architecture without any pooling. my downsampling is derived by strides > 1 and the upsampling is obtained by transpose convolution.


texts: What does MiniBatchKMeans fit's reassignment_ratio parameter do exactly?

texts:
I am using scikit-learn MiniBatchKMeans to do text clustering.
In the constructor method, there is a parameter reassignment_ratio, which is described in the documentation (link above) as follows:

reassignment_ratio : float, default=0.01

Control the fraction of the
maximum number of counts for a center to be reassigned. A higher value
means that low count centers are more easily reassigned, which means
that the model will take longer to converge, but should converge in a
better clustering.

I cannot wrap my head around that.
If I raise the reassignment ratio, I raise the "maximum number of counts for a center to be reassigned", so a center will be reassigned only if the number of samples (does "counts" stands for "sample" here?) around it is above this threshold.
Shouldn't it be the other way around? That a center is reassigned if the number of samples around it is below reassignment ratio?


texts:
I am starting an exploratory project to determine the time (of day) of which a photo is taken based on image features. I've done some research and downloaded a few databases hoping that "creation date", "last modified", or the file names will properly reflect the time when the original photo was taken, but I have had no luck.

So far I will be resorting to all of my personal photos, which I do not think will be sufficient enough in quantity or diverse enough in quality.

Are there any natural image datasets out there with timestamps or even timestampped file names?


texts: How do Dynamic Memory Network scale to large inputs?

texts:
How do Dynamic Memory Network, for example from the paper Ask Me Anything:
Dynamic Memory Networks for Natural Language Processing, scale to large inputs? The paper states the following:

  In these cases, the input may be a sentence, a long story, a
  movie review, a news article, or several Wikipedia articles.

Did anyone actually implement this network architecture to read an arbitrarily large corpus of text (say 250 Wikipedia articles)? The only example I could find online use rather small inputs (a few lines of text).

What is the bottleneck of using this type of network with large inputs? One solution I see is to encode paragraphs instead of sentences.


texts:
I'm unclear on the exact process of using the validation data.

Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.

Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.


texts:
I am a self-taught web developer and am interested in teaching myself data science, but I'm unsure of how to begin. In particular, I'm wondering:

What fields are there within data science? (e.g., Artificial Intelligence, machine learning, data analysis, etc.)
Are there online classes people can recommend?
Are there projects available out there that I can practice on (e.g., open datasets).
Are there certifications I can apply for or complete?


texts: Input Normalization for Transfer Learning

texts:
If I am training a deep neural net with input features that are physical in nature (e.g. temperature, precipitation, etc), and I want to be able to perform some kind of transfer learning where I train on multiple instances to see how they perform on a different set of inputs entirely. How do I make sure that the inputs being normalized in each instant don't conflict with one another?

For example, Mean temperature of 0 degrees with a standard deviation of 10 will be the same as mean temperature of 80 degrees with a standard deviation of 10 after normalization to a mean of 0 and a std dev of 1.


texts: How to properly use approximate_predict() with HDBSCAN clusterer for text clustering (NLP)?

texts:
I have approached text clustering using HDBSCAN based on this article which describes how to do this in R. I've re-written this in Python using this library. The approach is to first calculate TF-IDF vectors for the documents, then calculate a distance matrix for all vector pairs and fit the HDBSCAN clusterer based on the distance matrix.

I have fitter the clusterer with a subset of my documents since the algorithm is slow and my whole set is a bit big. I've limited it to 5000 samples. The clusters that HDBSCAN has found are acceptable. I will fine-tune them later.

Now, I would like to create a Python method that would take a new document, not being a part of the original training set, and return the cluster label which the new document seems to belong to.

I have approached this task by trying to use approximate_predict(). This is where I have question.

I suspect the process for calculating the cluster label for the new document look like this:

Add the new document to the set of 5000 samples that I've used for the clusterer training
Calculate the disctance matrix for the 5001 samples (the matrix will be bigger than the matrix used for clusterer fitting)
Take the last row of the resulting matrix (should correspond to my new text) and remove the last element from the resulting vector (it should contain the distance of the new document to itself, which we can ignore). Removing the last element is to make the dimension of the last vector match the dimension of the matrix used to fit the clusterer. Otherwise the clusterer would complain.
Use the approximate_predict() method and pass the vector obtained in step 3 to get the cluster label.

My questions are:

Is my approach correct? (it seems overcomplicated but I don't know what it should look like)
Will it perform well in production when I start passing lots of documents to this method? (the processing needed before I actually call approximate_predict() seems huge)
How can it be done differently?
Would it be better to not use the approximate_predict() method, but instead take the cluster labels that HDBSCAN calculated for my my 5000 samples and use it for supervised learning to train a classifier to then classify new documents?

I am looking forward to your answers and I will appreciate them a lot.
I am completely new to this and I don't have anybody around me to discuss this.


texts: Deriving vectorized form of linear regression

texts:
We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ground truth values. We then derive the cost function as follows:
We then compute the partial derivate of $\varepsilon$ with respect to $w_j$:
I'm confused as to where the $j'$ is coming from, and what it would represent.
We then write it as:

Then, we vectorize it as:

I'm confused as to the derivation of the vectorization of $A$ from $A_{jj'}$ likely because I don't know what $j'$ is. But how would the vectorization go in terms of steps and intuition?
Here is the link to the handout.


texts: Is it possible to have stratified train-test split of a set based on two columns?

texts:
Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using sklearn.model_selection.train_test_split. The only thing I have to do is to set the column I want to use for the stratification (in this case label).
Now, consider a dataframe that contains three columns, text, subreddit, and label. I would like to make a stratified train-test split using the label column, but I also want to make sure that there is no bias in terms of the subreddit column. E.g., it's possible that the test set has way more comments coming from subreddit X while the train set does not.
How can I do this in Python?


texts: OneClass SVM classification for text data

texts:
Problem:

I need to classify whether a document is checked or not checked. I have only text dataset for check document and if a document does not consist the check data it can automatically be classified as not checked.

I found that it can be doe using OneClass SVM. Below are the dataset and program implemented by me.

Code:

from sklearn import svm
import pandas as pd
from nltk.stem import PorterStemmer
from nltk.corpus import stopwords

from sklearn.model_selection import cross_val_score
from sklearn.feature_extraction.text import TfidfVectorizer
ps = PorterStemmer()

stop = stopwords.words('english')
data_test = pd.DataFrame({"Keywords":["Barge ID", "Check"]})
data = pd.DataFrame({"Keywords":["Vessel Name", "Barge ID", "Barge/Vessel", "VESSEL"]})

data['Keywords'] = data['Keywords'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data['Keywords'] = data['Keywords'].str.replace('[^\w\s]',' ').replace('\s+',' ')

data_test['Keywords'] = data_test['Keywords'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data_test['Keywords'] = data_test['Keywords'].str.replace('[^\w\s]',' ').replace('\s+',' ')

vectorizer = TfidfVectorizer( max_features = 200, ngram_range = ( 1,3 ), sublinear_tf = True )
data_features = vectorizer.fit_transform( data['Keywords'] )

clf = svm.OneClassSVM(nu=0.1, kernel="rbf", gamma=0.1)
clf.fit(data_features)

test_data_features = vectorizer.transform(data_test['Keywords'])

print(clf.predict(test_data_features))

I tried it for a small sample code but then to it did not work. it is wrongly indicating as -1 for Barge ID and 1 for Check


texts: Suitable methodology to assign job titles to categories

texts:
I am trying to find an automated way to assign job titles to categories.
My input data consists of job titles that are already assigned to a category and I want to be able to assign new job titles to one of those categories that fits best according to the already categorized ones.
The titles are not standardized. So one title could be "Marketing manager" that is already assigned to category A and a new title that should be assigned could be "Head of marketing". Therefore a simple 1:1 mapping of title to category will not be possible.
I appreciate any hints into the right direction.
EDIT:
The prerequisites have now changed so that I have specific job codes rather than varying job titles. So the dataset looks something like this:
category, jobCode, seniority, location
1, 1, CXO, us
1, 4, intern, us
2, 3, manager, uk
2, 4, intern, us

So the task is now to provide a function that uses the components jobCode, seniority and location to output the most suitable category based on the existing assignments.
I used word2vec to assign job titles to the jobCodes. Is this something I should expand on. E.g. just concatenate jobcode seniority and location and treat them as an input for word2veb?
I suppose there as a much more straight forward and suitable algorithm for this kind of problem.


texts: are explanatory variables in multiple linear regression slopes?

texts:
In simple linear regression the formula is: $y = m \cdot x + b$.
 When there are multiple independent variables: $y = b_0 + b_1 \cdot x_1 + b_2 \cdot x_2 + \ldots $

I know that $x_1,x_2,\ldots $ represents different independent variables but what exactly is $b_1,b_2,\ldots$?


texts: Is there any precedent for DL use of pretrained graph embeddings in a similar manner to word embeddings?


texts:
Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from the same image.

My question is: how do you generate fake time series data?


texts: Zero-inflated independent feature in tree-based models

texts:
What is the best approach to include a zero-inflated continuous independent feature (e.g., 90% of the values are Zero, 10% are >0) in a Tree-based models (DT, random forest, gradient boosting. etc). I am considering the following three options:

Option 1: Keep the zero-inflated continues feature as-is.

Option 2: Replace the continuous feature with a binary interaction feature (i.e., 0 for X=0; 1 for X>0)

Option 3: Include both the continuous and categorical features.

The main justification I have for Option 1 is that the continuous feature can be used in more than one split. I am also aware that option 3 indicates including two highly correlated independent features. Will I be losing information if I use option 2?
Update:
I found the following answer; however, I am not sure if it can be generalized for tree-based models


texts: Clustering of Weekday Weekend Time Series Data

texts:
I have a dataset of the number of steps people take throughout a day over a period of months. I aggregated them so that each person will have an average weekday and weekend time series of steps. An example is below:

ID Weekday 00:00 00:30 01:00 ........ 23:00 23:30
A    1       20    30    23             154  256
A    0       15    67    121            35   101
B    1       78    21    46             78   26
B    0       55    301   22             121  451

I have tried clustering using K-means, agglomerative hierarchical and dynamic time warping by combining both weekdays and weekends together (i.e each person only has 1 case of data), and it appears to show 2 distinct groups of people, one group with generally higher levels than the other group.

How can I do clustering while also taking in account whether it is on a weekday or weekend? So for example, there might be a few clusters: one with high weekday activity and low weekend activity, one with high - high, another with low - low, etc?

I am not sure what the terminology is in this case: multidimensional time series? multivariate clustering?


texts: How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

texts:
I think the title is enough for what is my problem here. I have 100 tickers in a folder. I got all csv files to one list. but I wanna see the tickers name which these are csv file names.

what should I do for getting all tickers like that. I'm doing it manually but I need a simple way to do that.
import pandas as pd
import os

#BİST100-2010.01.01-2020.07.01/ is my folder.
log_total = []
for file in os.listdir('BİST100-2010.01.01-2020.07.01/'):
    log_total.append(pd.read_csv('BİST100-2010.01.01-2020.07.01/'+file+''))

by the way after I do that I got this.


texts: Automatic annotation of medical text data

texts:
I have a dataset of 30000 whole-genome sequence analysis. For each sample I have a text description that does not follow any fixed format.
What I want, is an annotation of each sample indicating the disease and the particular tissue that characterize the sample.

I'm not into text mining, so I don't know which tools I can use. Any suggestions?


texts: Primary indexes and index-sequential files

texts:
I am studying the physical organization of databases and right now I trying to understand the concept of primary index or clustering index. The book states the primary index can be realized by storing the tuples on the index itself ( the index stores the tuples ). According to the book, in this case (the index is not separated by the file containing the tuples) the primary index is so called because the storing method can be done by storing according to the search key but also because the index contains the physical storage necessary to store the tuples. The book states also that this kind of index can be also realized by constraining tuples to stay in a certain position according the search key (so they can not be moved from there).
So the questions are:
1) How many ways are used to realize a primary index not separated from the file containing the tuples?
2) When can we call a primary index also index-sequential file?
Thank you


texts:
I tried installing PyFerret on my Macbook through the terminal.
I got into my user directory with the terminal and gave it the following commands from this website https://anaconda.org/conda-forge/pyferret:

conda install -c conda-forge pyferret
conda install -c conda-forge/label/broken pyferret
conda install -c conda-forge/label/cf201901 pyferret

However, I noted that I only needed to download just one of them. I tried removing the files using

conda remove
conda uninstall

but they did not work. Later I tried following the instructions from this website
https://github.com/NOAA-PMEL/PyFerret/blob/master/README.md
which instructs to type the following command

conda create -n FERRET -c conda-forge pyferret ferret_datasets --yes

then it says you can use PyFerret by calling it in the terminal as

conda activate FERRET

but the response I got was

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

I tried using

conda init -bash

and it sort of worked after that, I could use the command to activate PyFerret but it would only say PyFerrer on the name of my command line and I did not really know what was going on.

Could you please instruct me in how to uninstall everything?
 and if possible I would like to be instructed on how to get PyFerret to work as well.


texts: Storing and loading bottleneck features for transfer learning on large data sets (Keras)

texts:
I would like to apply transfer learning on a pretty large image data set in order to solve a classification problem. Currently I load a pre-trained net without the top layers, add my own top layers, freeze the base model and start training.
Since the data set is too big to fit into memory I load the train and validation sets using prefetched TF Datasets:
train_ds = keras.preprocessing.image_dataset_from_directory (training_data_dir,
                                                             batch_size=batch_size,
                                                             image_size=img_size,
                                                             label_mode='categorical')
val_ds = keras.preprocessing.image_dataset_from_directory (validation_data_dir,
                                                           batch_size=batch_size,
                                                           image_size=img_size,
                                                           label_mode='categorical')
train_ds = train_ds.prefetch(buffer_size=buffer_size)
val_ds = val_ds.prefetch(buffer_size=buffer_size)

Now in order to reduce the training time I found a nice approach on this page: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
Before training the model, they first run the base model for each input image and store the output of the base model, that is the bottleneck features. Training is then done on these bottleneck features. That means that during training the base network does not need to be executed, saving a lot of computation time.
Now for their approach they have a data set that fits in memory and the set of all bottleneck features over all images is represented as a single numpy array which is stored in and loaded from a single file.
Unfortunately this is not possible in my case since the original image set is so large it will not fit into memory and also converted to a bottleneck feature set it is still to large. So I am looking for a solution where the bottleneck features of a single image are stored in a single file and where those bottleneck feature vectors can be loaded in a way that is similar to how image_dataset_from_directory loads images.


texts: Keras y-labels range between 0 and 1 instead of binary?

texts: I have X values and corresponding y-labels, until now I used to round my labels <0.5 to 0 and >0.5 to 1. Is it possible to use values between 0 and 1 for "y train"? Using Keras and Tensorflow. Appreciate any suggestions.


texts: Difference between Memory and Disk space

texts:
I was watching this machine learning course by Andrew NG where he states memory requirement and disk space requirement separately while referring to machine learning algorithms.
What exactly is the difference and how are they occupied?


texts: How to break down large SVM classification model?

texts:
I have a classification problem with large number of classes: feature set is 512 Dimension, number of classes are around 3000. This is a face identification problem. (identify among 3000 celebrities, whose face it is. The feature is extracted using FaceNet.)

The issue for training such a SVM model is too slow:

I used sklearn SVC, it came with below result:

RAM usage: > 100GB // I eventually ended up using virtual memory of 100GB
Training time: > 30 hours
Classification time: > 1 hour per face
Other issue: Single CPU usage, no parallelization

To summarize, it is not practical to use above naive way of training for such a SVM model.

My Question: What is a practical way of optimizing SVM training / usage of this scale input data? (~ 3000 classes, feature_size = 512)


texts: How to cluster label (in a multilabel classification problem) which mostly appear together in a class

texts:
To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe:
text     |   genre
===========================
text 1   | [action,mistery,horror,thriller]
text 2   | [drama,romance]
text 3   | [comedy,drama,romance]
text 4   | [scifi,mystery,horror,thriller]
text 5   | [horror,mystery,thriller]

How can I cluster the tag that often appear together? For example, genre "mystery","horror","thriller" often appear together (3 times), genre "drama","romance" often appears together (2 times).


texts: Why does an L2 penalty prefer smaller and more diffuse weight vectors?

texts:
For instance, why is it that it is more favourable for a weight of [0.25, 0.25, 0.25, 0.25] (for which the L2 penalty is 0.25) instead of simply [1, 0, 0, 0] (for which an L2 penalty is 1)?

In this case, both weights would give the same dot product when using W.T * X


texts: How to visualize the weights of the output(classification) layer?

texts:
I'm doing a Convolutional Neural Network of MNIST data set, and how can I visualize the weights of the output(classification) layer? I only found several websites visualizing the filters in CNN model but none for the output layer.

this is my code so far:

model = Sequential()
model.add(Conv2D(
          input_shape=(28,28,1),
          filters=16,kernel_size= (5,5),
          padding='same',
          activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(
          filters=16,kernel_size= (3,3),
          padding='same',
          activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(units=128,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(units=10,activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
train = model.fit(x=x_train,y=y_train,validation_split=0.2,epochs=10,batch_size=200,verbose=2)


texts:
In an RBM, if we represent the weights learned by the hidden units, they show that the neural net is learning basic shapes. For example, for the mnist dataset, they learn features of the numbers they are trying to classify.

In a regular feed-forward net with one hidden layer, I can train the network to recognize digits, and it works, but when I try to visualize the hidden layer weights, I only see noise, no distinguishable feature. Why is that? Hasn't the network learned to recognize the digits?


texts:
I understand this question can be strange, but how do I pick the final random_seed for my classifier?

Below is an example code. It uses the SGDClassifier from SKlearn on the iris dataset, and GridSearchCV to find the best random_state:

from sklearn.linear_model import SGDClassifier
from sklearn import datasets
from sklearn.model_selection import train_test_split, GridSearchCV

iris = datasets.load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

parameters = {'random_state':[1, 42, 999, 123456]}

sgd = SGDClassifier(max_iter=20, shuffle=True)
clf = GridSearchCV(sgd, parameters, cv=5)

clf.fit(X_train, y_train)

print("Best parameter found:")
print(clf.best_params_)
print("\nScore per grid set:")
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

The results are the following:

Best parameter found:
{'random_state': 999}

Score per grid set:
0.732 (+/-0.165) for {'random_state': 1}
0.777 (+/-0.212) for {'random_state': 42}
0.786 (+/-0.277) for {'random_state': 999}
0.759 (+/-0.210) for {'random_state': 123456}

In this case, the difference from the best to second best is 0.009 from the score. Of course, the train/test split also makes a difference.

This is just an example, where one could argue that it doesn't matter which one I pick. The random_state should not affect the working of the algorithm. However, there is nothing impeding of a scenario where the difference from the best to the second best is 0.1, 0.2, 0.99, a scenario where the random_seed makes a big impact.

In the case where the random_seed makes a big impact, is it fair to hyper-parameter optimize it?
When is the impact too small to care?


texts: Relation between using stratify and class weights for imbalanced classes

texts:
I'm working on a multi-class classification problem where the classes are imbalanced (70:25:5).

Train-Test Split

x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42, stratify=Y)

Random Forest Classifier

cls = {0:.9, 1:.09, 3:.01}
RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42, oob_score=True, class_weight=cls)

I do not see a change in the results after assigning class weights. Is there anything that I need to look into in particular here? Thanks in advance!


texts: How to compare a machine learning model and a rule based model

texts:
The difference between machine leaning models and rule based model is that you feed input and output to machine learning models to get rules where you feed input and rules to rule based models to get output.
Say you have a dataset with 10k records. You might use 8k of them to build a machine learning model and the rest to validate your model. On the other hand, you would probably use all the 10k records to create a rule based model. If this was a classification model, how should I compare the 2 models? Accuracy of the machine learning model from the test dataset (2k) and accuracy of the rule based model from all the records?


texts: There could be a problem with the linear layer after the attention inside a transformer?

texts:
My question regards this image:

It seems that after the multi head attention there is a linear layer as they mention also from here:

the linearity is given by the weights W^{o}. my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? I mean, if each token from the attention layer should not depend from the next tokens then these weights seem to mess up the whole thing or they don't? Indeed these linear weights will learn the dependencies among all the tokens and during inference there could be a problem, maybe(?). Thanks for any clarification.


texts: Is there an R build tool (like Maven or Gradle for Java) to get the dependencies and package an R project?


texts:
i read that Json or XML are unstructured data; Are Json or XML data or are they tools to tag the data?

I understand from this wikipedia page (https://en.wikipedia.org/wiki/Semi-structured_data) that semi structured data are data without a formal database structure, but still have some tags;


texts: difference between empirical risk minimization and structural risk minimization?

texts:
I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two.

I read somewhere that perceptron uses Emperical risk minimization where as SVM uses structural.

I know that SVM takes into consideration model complexity so it is somewhat related to structural risk minimization put this is not clear to me whats the difference


texts: I'm trying to perform classification on a large dataset with mixed numerical and categorical features. The dataset is all semi-structured text, so everything is a String. Does anyone know of a library I could use to automatically identify numeric features (e.g. "width (in.)", which corresponds to the width in inches)? This is important because numeric and categorical features are encoded/handled differently in my analytics pipeline. I think this is basically a named entity recognition problem wherein the entity is a numeric variable or feature that exists in the data set as a String/text.


texts: Do we need to increase training data size when increasing dropouts?

texts:
I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to prevent overfitting. While tuning the dropout rate, I am increasing the value from 0.2 -> 0.3 -> 0.4 -> 0.5.

I want to know if I should increase the training data size to have a more accurate comparison. What I mean is suppose I am having training data of size 1 million for a dropout rate 0.2. Should I increase the training data size to 1.5 million for dropout rate 0.3?

Calculation:

1000000 * 0.8 * 0.8 * 0.8 = 512000 (for dropout ratio 0.2)

1500000 * 0.7 * 0.7 * 0.7 = 514500 (for dropout ratio 0.3)


texts: How can I prevent this model to learn more(less) :)))

texts:
As you can see, GradientBoostingClassifier overfit with more training example.
These are my parameter for the model:

{'learning_rate': 0.1, 'loss': 'deviance', 'max_depth': 6, 'max_features': 0.3, 'min_samples_leaf': 80, 'n_estimators': 300}

What should I to make my model better or stop training at 350?


texts: ImageDataGenerator augmenting only one image in training set

texts:
My datset consists of images in 10 classes. I am using 'ImageDataGenerator' to augment my training data using 'flow_from_directory' with training directory containing subdirectories of classes, each containing images belonging to each class.
train_data_generator=ImageDataGenerator(preprocessing_function=preprocess, rotation_range=20,width_shift_range=0.2,height_shift_range=0.2,horizontal_flip=True)

 . I then use 'save_to_dir' attribute in 'flow_from_directory' to save the augmented images. But, after using 'fit_generator' method, I have found only copies of the first image of the last class in the directory. I have ~1200 different images in all. Please help.

Edit
It seems 'ImageDataGenerator' works alright when I don't define a 'preprocessing_func'. This is strange, because the documentation says that the preprocessing function is applied on each input.


texts: Comparing files (text sources) in Orange

texts:
What is the best method to compare text files in Orange?
How do I find the difference or similar between two texts and relations between main nodes?


texts: How to get sentence from embedding vector with Universal Sentence Encoder?

texts:
I'd like to ask, if there is possibility to get sentence (or word) from embedding vector using Universal Sentence Encoder?

First of all, I've clustered my embedded sentences and I've got a vector which is representing center of the cluster, and now I want to convert this to some sentence which will have the best meaning for this specific cluster.

Best regards.


texts: What is the proper way to use early stopping with cross-validation?

texts:
I am not sure what is the proper way to use early stopping with cross-validation for a gradient boosting algorithm. For a simple train/valid split, we can use the valid dataset as the evaluation dataset for the early stopping and when refitting we use the best number of iterations.

But in case of cross-validation like k-fold, my intuition would be to use each valid set of each fold as evaluation dataset for the early stopping but that means the best number of iterations would be different from a fold to another. So when refitting, what do we use as the final best number of iterations ? the mean ?

Thanks !


texts: Does it violate the assumptions of linear regression to perform it on time series data?

texts:
One of the assumptions of linear regression says that the errors must be independent i.e., the residuals must not depend on each other.

Let's say we are using linear regression to model the temperature on a given day. If it is 13:00 and 20 degrees, the temperature at 13:15 will be similar and thus depends on the time before it - it cannot suddenly fall to -20 in the space of 15 minutes. Likewise, the temperature at 20:00 will be more closely related to the temperature at 19:50 than 13:00.

Does the linear regression assumption 'independence of errors' mean that you cannot perform it on time series data?


texts: How to evaluate the quality of speech-to-text data without access to the true labels?

texts:
I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription system. I want to automatically assess the quality of these transcriptions.
Sadly, the quality seems to be disastrous. In some cases it's little more than gibberish, often due to different dialects the machine is not able to handle. We have no access to the original recordings (data privacy), so there is no way whatsoever to get or create the true labels. The system cannot be replaced as we are committed to it.
Again to the question: is there any way to automatically assess the quality of transcriptions with NLP methods? We want to quantify and compare transcription quality to filter out the best samples for semantic inference of our customers' input in a downstream task. I am thinking about something like a coherence measure in order to find the sentences which make the most sense, grammatically or semantically. Sadly, things as BLEU, WER or Rouge do not work in this case.
I'd be grateful for anything pointing in the right direction. Most importantly again, we have no labels and it needs to be scalable.
Thanks a lot!


texts: Why Spyder Python throwing error in I/O operation despite it?

texts: Why Spyder Python stated that my code has error valued I/O operation & [SpyderKernelApp] ERROR | Error in message handler while when I run in PyCharm or copy it in Jupyter the code seems fine?  What does Syderkernel error means?


texts: Collapse categorical variable to reduce levels using a decision tree

texts:
I am using zip codes as an independent variable as part of a binary classification problem. Naturally, this feature has many different levels (around 2,000), so I was wondering if there is a standardized method for reducing the number of categories by grouping them together using a decision tree.
The reason I specifically want to tackle this problem using a decision tree is didactical. I know of other methods, but have never used a decision tree for this purpose.
To be more specific, I would like to reduce the number of levels from 2,000 down to 20 or so different groups.


texts: Google Colab conda packages permanently installation

texts:
I want to install some conda packages on Google Colab. Installation done successfully, but it wont detect it when we restart the new Colab session.

I followed the link "#https://stackoverflow.com/questions/55253498/how-do-i-install-a-library-permanently-in-colab", but still no luck.

Please let me know how to install python packages using conda + pip permanently on Google Colab.


texts:
I've been running kfold cross validation with 10 folds and comparing it against a test set. Logging the score and the stdev along the way.

Once I wad happy with my model I then run the estimator through the cross validation with the full data (including the test set) but ... The standard deviation of the scores increases and the score decreased????

Is this normal? I assumed MORE data decreases variance ? Or is it just due to the randomness in the splits ?