texts: i need explanation to a way of implementing cross validation in trControl function in caret

texts:
given this example :

# Example of Stacking algorithms
# create submodels
control <- trainControl(method="repeatedcv", number=10, repeats=3, savePredictions=TRUE, classProbs=TRUE)
algorithmList <- c('lda', 'rpart', 'glm', 'knn', 'svmRadial')
set.seed(seed)
models <- caretList(Class~., data=dataset, trControl=control, methodList=algorithmList)
results <- resamples(models)
summary(results)

should we always give the entire data set to the train function and it will automate the splitting of test and train and evaluate on the test set
or should i give it the train set  and evaluate by my self on the test data set ?


texts: Performance Metric for topic extraction when there is no ground truth

texts:
I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to index ontology concepts. However, as it is very difficult to find the ground truth about the topics in the documents, it is becoming very difficult to evaluate the model. As manually tagging the correct labels will take a huge amount of time.

Are there any standard ways by which people evaluate models of topic extraction/ entity extraction, when ground truth about the topics is not clear?


texts: Evaluating the performance of a random forest classifier

texts:
I'm using a random forest classifier (in R) to impute missing data in a dataset. Basically, I have a bunch of objects (companies) and I want to guess an attribute (size) from other attributes (capital, owning_group and state). The dependent attribute is a categorical variable (size) with 3 possible values (small|medium|large). A random forest (R package randomForest) on a set of 3 variables provide this output:

ff = size ~ capital + owning_group + state

Call:
 randomForest(formula = ff, data = df, importance = T, ntree = ntree, na.action = na.omit)
               Type of random forest: classification
                     Number of trees: 2000
No. of variables tried at each split: 1

        OOB estimate of  error rate: 32.41%
Confusion matrix:
       large medium small class.error
large    238     17   237  0.51626016
medium    80     25   322  0.94145199
small     73     30  1320  0.07238229

  Overall Statistics

               Accuracy : 0.7297
                 95% CI : (0.7112, 0.7476)
    No Information Rate : 0.8049
    P-Value [Acc > NIR] : 1

                  Kappa : 0.426
 Mcnemar's Test P-Value : <2e-16

Statistics by Class:

                     Class: large Class: medium Class: small
Sensitivity                0.7087       0.84211       0.7294
Specificity                0.8868       0.83981       0.8950
Pos Pred Value             0.5488       0.14988       0.9663
Neg Pred Value             0.9400       0.99373       0.4450
Prevalence                 0.1627       0.03245       0.8049
Detection Rate             0.1153       0.02733       0.5871
Detection Prevalence       0.2101       0.18232       0.6076
Balanced Accuracy          0.7977       0.84096       0.8122

I interpret this output as saying that the model has a 73% accuracy, and that the classifier makes a lot of mistakes for medium and large, but gets small mostly right. Does the P-value indicate that the model is not significant?

Assuming that this precision is OK for my context, how can I validate this model beyond these simple observations?


texts: Why is my LSTM is working best with batch size of 2 and no hidden layers?

texts:
I am building an LSTM for price prediction using Keras. I am using Bayesian optimization to find the right hyperparameters. With every test I make, Bayesian optimization is always finding that the best batch_size is 2 from a possible range of [2, 4, 8, 32, 64], and always better results with no hidden layers. I have 5 features and ~1280 samples for the test I am trying.

Why is this the case? Is the lack of hidden layers due to the fact that I do not have many inputs and samples?

Also, literature always seems to suggest that a batch size of 2 is very low and a middle ground for the optimal batch_size is 32. How can I interpret this result?


texts: How can I compare a NN model and a linear regression?

texts: I have a small dataset (1500 rows) and to predict the imbalanced target, I am running two linear models (linear regression and lasso) and one nonlinear model (Neural Network) on it. I am using Area Under Precision Recall Curve (AUPRC) to compare the three models. The  baseline in the curve is 10%, AUC for linear regression is 11%, AUC for lasso is 11.2%, and AUC for NN is 11.35%. Can I say that the learning models have improved the random guessing? Is the difference between lasso and NN (11.2% and 11.35%) enough to say the relationships are nonlinear rather than linear? Is there any way (test) to show that the scores are significantly different (such as t-test or p-values)?


texts: How to know the model has started overfitting?

texts:
I hope the following excerpts will provide an insight into what my question is going to be. These are from here.

The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting "better". But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.

We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.

We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.

As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost.
Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this.
We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a  better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?


texts: what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

texts:
I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine learning or deep learning.

I had searched for approaches, but unfortunately I didn't find any approach to take.  I had tried with bag of words model and glove word embedding to predict whether a string is name or city etc..

But, I didn't succeed with bag of words model and with GloVe there are a lot of names which are not covered in the embedding example :- lauren is present in Glove and laurena isn't

I did find this post here, which had a reasonable answer but I couldn't the approached used to solve that problem apart from the fact that NLP and SVM were used to solve it.

Any suggestions are appreciated

Thanks and Regards,
Sai Charan Adurthi.


texts: Multiple classification algorithms are predicting always exactly with the same scores. Is that normal? If not, what should I suspect?

texts:
I have been working on a multilabel classification problem. I am using Python machine learning libraries to implement the classification algorithms. For the cross-validation, I am using repeated K-Fold cross-validation. I have experimented with SVM, Logistic Regression, Random Forest, Decision tree, K-Neighbour, and Naive Bayes and using Binary Relevance, Classifier Chain, and Label Powerset transformation methods for all of them.

I noticed, for Classifier Chain, SVM, Logistic Regression, Random Forest, and K-Neighbors are always achieving same subset accuracies and hamming losses. For Label Powerset, SVM, Logistic Regression, and Random Fores are achieving the same scores. However, for the binary relevance, all scores are different. No matter, what random seed I use or how many repetitions of cross-validations I run, they always end up scoring the same. I was wondering if this is a normal occurrence or they are suffering from class imbalance or overfitting or anything of that sort problems?

P.S. I am a beginner. Sorry for bothering with a naive question.


texts: What is the difference between the value -99 and NaN in a data column?

texts: I am new to data science. I was looking into some datasets and I saw some values like -99, which I discovered later that it means that there is a missing value. Does this mean the same thing as NaN? If it is the same thing, why do we use -99 instead of NaN?


texts: Why do we use a d_dimentionap vector for positional encoding in a transformer?

texts: It is mentioned in the paper attention is all you need that for any PE_t+k sinusoidal positional embedding can be calculated using a linear transformation of PE_t. But the same is true if PE is only the first pair of values (i.e. [sin(pos), cos(pos)] and every other [sin, cos]. Furthermore, such a pair is enough to generate unique values for every pos so why not just concatenate a pair at the end or even add them to just the first 2 dimensions if you were adamant on addition?


texts: R packages: How to access csv files in data subfolder?

texts:
I have successfully written an R package and want to ship it with a specific csv file. I placed the file in the data and data-raw subfolders.

read.csv("data/foobar.csv")

The above command fails. How can I read the csv file?


texts: SVM using scikit learn runs endlessly and never completes execution

texts:
I am trying to run SVR using scikit-learn (python) on a training dataset that has 595605 rows and 5 columns (features) while the test dataset has 397070 rows. The data has been pre-processed and regularized.
I am able to successfully run the test examples, but on executing using my dataset and letting it run for over an hour, I could still not see any output or termination of the program. I tried executing using a different IDE and even from the terminal, but that does not seem to be the issue.
I also tried changing the 'C' parameter value from 1 to 1e3.
I am facing similar issues with all SVM implementations using scikit.
Am I not waiting long enough for it to complete?
How much time should this execution take?
From my experience, it should not require more than a few minutes.
Here is my system configuration:
Ubuntu 14.04, 8GB RAM, lots of free memory, 4th gen i7 processor


texts: Why we use an activation function for introducing nonlinearity instead of a polynomial Perceptron implementation?

texts:
I perceive a single perceptron as a single linear function $y = a_1x_1 + a_2x_2 + ... + a_nx_n + b_0$ with a goal to calculate the best weights combination $ w_1, w_2, ..., w_n $ that minimizes the given loss function.
The problem with this type of network is that it would not be able to perform well on a non linear dataset, thus an activation function would be used in order to tackle this. I am wandering what could happen if instead of a linear Perceptron, we introduce a polynomial Perceptron in the form of $ y = a_1x_1^k + a_2x_2^r + ...+ a_nx_n + b_0$ and how this will compare with the original perceptron.


texts: How to use the same scale with new data? - scikit learn - scikit learn

texts:
How do I use the same scale used in preprocessing with new data.

Actual code:

x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df_scaled = pd.DataFrame(x_scaled)

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
pred = clf.predict(X_test)

If I understand it correctly I should have included a scaler variable with the StandardScaler.

  https://stackoverflow.com/questions/38780302/predicting-new-data-using-sklearn-after-standardizing-the-training-data

Something like:

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
scaler = preprocessing.StandardScaler().fit(X_train)
pred = clf.predict(X_test)

What scaler parameters should I use for future data processing?

Thanks!


texts: What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

texts:
I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & sigmoid.

But, I have encountered an explanation by Prof. Andrew NG lecture that explains that a partial solution to the vanishing gradient problem is a better or more careful choice of the random initialization of weights in your neural network.

i.e the solution is:

To set the variance of Wi to be equal to 1/n, where n is the number of input features that are going into a neuron. Along with the assumption that the input features of activations are roughly mean 0 and standard variance 1. So, what it's doing is that it's trying to set each of the weight matrices w so that it's not too much bigger than 1 and not too much less than 1, therefore, it doesn't explode or vanish too quickly.

So, if you are using a ReLu activation function then setting the variance of Wi to be equal to sqrt(2/n) works better**.
and if you are using a TanH activation function then setting the variance of Wi to be equal to sqrt(2/n) works better.
or in some cases, it's being suggested to use Xavier initialization
Also, if we need we can tune of variance parameter as another hyperparameter by multiplying into the above formula and tune that multiplier as part of your hyperparameter search.

Therefore, choosing a reasonable scaling for how to initialize the weights helps weights not to explode too quickly and not decay to zero too quickly, which in turn could help in training a reasonably deep network without the weights or the gradients exploding or vanishing too much and not simply using ReLu!.
Please correct me if my understanding is wrong or incomplete!


texts: How to convert DNA sequences in FASTA format to OneHot Encoded Pandas Dataframe for Neural Networks?

texts:
DNA sequences in FASTA format look like:
CATGCATTAGTTATTAATAGTAATCAATTACGGGGTCATTAGTTCA...

I am trying to convert them into one-hot encoded data in a Pandas dataframe so that I can use various neural networks to analyze them. This has probably been done many times. Can someone point me to references or Python packages for it?


texts: 1x1 Convolution. How does the math work?

texts:
So I stumbled upon Andrew Ng's course on $1x1$ convolutions.
There, he explains that you can use a $1x1x192$ convolution to shrink it.

But when I do:

input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])

out = torch.mul(input_,filter)

I obviously get $28x28x192$ matrix. So how can I shrink it?

Just add the result of every $1x1x192 * 1x1x192$ kernel result? So I'd get a $28x28x1$ matrix?


texts: How are non-restricted Boltzmann machines trained?

texts:
Restricted Boltzmann machines are stochastic neural networks. The neurons form a complete bipartite graph of visible units and hidden units. The "restricted" is exactly the bipartite property: There may not be a connection between any two visible units and there may not be a connection between two hidden units.

Restricted Boltzmann machines are trained with Contrastive Divergence (CD-k, see A Practical Guide to Training
Restricted Boltzmann Machines).

Now I wonder: How are non-restricted Boltzmann Machines trained?

When I google for "Boltzmann Machine", I only finde RBMs.


texts:
I am trying to retrieve the column names of the data set model$data using the following formula:

sample(colnames(model$data),1)

When I run it I receive the following error message:

Error in sample.int(length(x), size, replace, prob) :
  invalid first argument

Appreciate any help!

str(model) looks like this:

> str(model)
    List of 13
     $ data            :List of 1
      ..$ : num [1:1000, 1:56] 1 1 1 1 0 1 1 0 1 1 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : chr [1:1000] "7530" "5975" "552" "815" ...
      .. .. ..$ : chr [1:56] "Agriculture_and_Hunting" "Baking" "Biochemistry" "Braiding" ...
     $ unit.classif    : num [1:1000] 3 5 5 5 16 3 5 1 3 3 ...
     $ distances       : num [1:1000] 0.000806 0.000239 0.000239 0.000239 0.001953 ...
     $ grid            :List of 6
      ..$ pts              : num [1:25, 1:2] 1.5 2.5 3.5 4.5 5.5 1 2 3 4 5 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : NULL
      .. .. ..$ : chr [1:2] "x" "y"
      ..$ xdim             : num 5
      ..$ ydim             : num 5
      ..$ topo             : chr "hexagonal"
      ..$ neighbourhood.fct: Factor w/ 2 levels "bubble","gaussian": 1
      ..$ toroidal         : logi FALSE
      ..- attr(*, "class")= chr "somgrid"
     $ codes           :List of 1
      ..$ : num [1:25, 1:56] 0.000388 0.99996 1 1 1 ...
      .. ..- attr(*, "dimnames")=List of 2
      .. .. ..$ : chr [1:25] "V1" "V2" "V3" "V4" ...
      .. .. ..$ : chr [1:56] "Agriculture_and_Hunting" "Baking" "Biochemistry" "Braiding" ...
     $ changes         : num [1:100, 1] 0.00261 0.00263 0.00262 0.00254 0.00254 ...
     $ alpha           : num [1:2] 0.05 0.01
     $ radius          : Named num [1:2] 3 0
      ..- attr(*, "names")= chr [1:2] "67%" ""
     $ user.weights    : num 1
     $ distance.weights: num 1
     $ whatmap         : int 1
     $ maxNA.fraction  : int 0
     $ dist.fcts       : chr "sumofsquares"
     - attr(*, "class")= chr "kohonen"


texts: Which metrics for evaluating a recommender system with implicit data?

texts:
I am currently in the process of creating a recommender system. This recommender system works with a neural network and then searches for the closest neighbors and thus gives recommendations for a user.
The data is implicit. I only have in the data which products a user has bought.On the basis of this data, I create the recommendations.

What are the best metrics to evaluate this recommender system with implicit data?

Can I evaluate the model and then the search algorithm? If so, are the metrics different? Which metrics do I have to use for what?


texts: Problem while installing tensorflow on window 10 via Anaconda

texts:
(venv) C:\Users\XYZ>pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.13.1-cp35-cp35m-win_amd64.whl

  ERROR: tensorflow-1.13.1-cp35-cp35m-win_amd64.whl is not a supported wheel on this platform.

I am doing this on anaconda prompt.. First I am getting this Error "RemoveError: 'setuptools' is a dependency of conda and cannot be removed from conda's operating environment." While setting virtual environment and then I am getting the Error above.

This is the error I am getting while trying to install tensorflow on Windows via Anaconda. How do I resolve it?


texts: How can collaborative filtering be extended to include more features?

texts:
Looking at the following:
https://realpython.com/build-recommendation-engine-collaborative-filtering/#using-python-to-build-recommenders
I can see that userID, itemID, rating are the standard features used in a collaborative filtering model. However, my question is how can I incorporate more features into the model (such as the reviewText)?


texts: How can I use the Brier Skill Score in cross-validation for imbalanced data?

texts:
I wish to use search for model hyper-parameters (by a grid search) using the Brier score as the scoring method (see code below):
 kfolds = StratifiedKFold(3)
 clf = GridSearchCV(model, parameters, cv=kfolds.split(x, y),
                           scoring='neg_brier_score', return_train_score=True, n_jobs=n_jobs)

The problem is, I have a class-imbalance problem, and I couldn't find an implementation of the Brier score that works with this.  Is there a version of the Brier score that is appropriate problems? And if not, why?


texts: "Memory Error" - Kmeans in python using pandas DataFrame

texts:
I am trying to predict on my "dataset_to_predict" having size of (297000 x 5120). While Memory usage is under 50%. No Specific Error message.

I'm trying to find # of k using elbow method - Got MemoryError:

from sklearn.cluster import KMeans, MiniBatchKMeans
km = KMeans(n_clusters=5)
k_range = range(1,10)
sse = []
for k in k_range:
    km = KMeans(n_clusters=k).fit(dataset_to_predict) #this line throw error
    #km = MiniBatchKMeans(n_clusters=k, batch_size=100, verbose=1).fit(dataset_to_predict) #Also tried on part of the dataset
    sse.append(km.inertia_)

Also, I tried to use Kmeans.fit_predict() method again get the memoryError:

y_predicted = km.fit_predict(dataset_to_predict) #this line throws error
y_predicted

System Specs I have:

Ubuntu 18.04.2 LTS
Memory: 16 GB
Swap: 2GB
Processor: Intel® Core™ i5-7500 CPU @ 3.40GHz × 4


texts: Extracting information with corresponding fields

texts:
I have large pool of scanned county documents. I need to extract information like document title, borrower name&address, lender name&address etc.

The text is like this
Eg: the deed of trust, between abc llc, a limited company, whose address is XXXXXX, herein called "borrower", and  xyz, whose address is XXXXX,herein called "lender".

I used Named entity recognition method to  extract the names, it works well. but how would i know which name is borrower and which one is lender? can anyone help me


texts:
I have a simple question about classification reports and validation generator in keras.
In my case each number in valid_generator.classes corresponds to an artist. But how do i know which number each artist has?

from sklearn.metrics import classification_report, confusion_matrix

valid_generator.reset()

valid_generator = _val_generator.flow_from_dataframe(dataframe = test_info,
    directory = test_folder, x_col = "filename",
    y_col = "artist", seed = 42,
    batch_size = 1, shuffle = False,
    class_mode="sparse", target_size = IMAGE_SIZE)

Y_pred = model.predict(valid_generator)
y_pred = np.argmax(Y_pred, axis=1)

print(classification_report(valid_generator.classes, y_pred))

Output:
     precision    recall  f1-score   support
0       0.43      0.45      0.44        44
1       0.80      0.40      0.53        10
2       0.50      0.35      0.41        20
3       0.33      0.24      0.28        21
4       0.79      0.50      0.61        22
5       0.56      0.58      0.57        77
6       0.62      0.38      0.47        21

etc..
```


texts: How to label a dataset for Machine Learning?

texts:
I have a collection of educational dataset.
The dataset consists of a username and their review for the course.
I want to analyze the data for sentiment analysis.

How can I label the data to train the model for my supervised machine learning model?

Do I need to label manually (Positive, negative, neutral) to train the model?


texts: What is difference between Nearest Neighbor and KNN?

texts: I was taking the tutorial of making Recommendation system , there I read that Nearest Neighbor is different from KNN classifier . Could anyone explain that what is Nearest Neighbor and how it is different between KNN ?


texts:
I am working on a recommendation engine, and I have chosen to use SciPy's cosine distance as a way of comparing items.

I have two vectors:

a = [2.7654870801855078, 0.35995355443076027, 0.016221679989074141, -0.012664358453398751, 0.0036888812311235068]

and

b = [-6.2588482809118942, -0.88952297609194686, 0.017336984676103874, -0.0054928004763216964, 0.011122959185936367]

Running the following code will produce an output of ~1.999:

from scipy.spatial import distance
print(distance.cosine(a,b))

Is there something wrong with my input values? Anyone know why I am getting a result of >1?


texts: Using Spark for finding similar users to a user?

texts:
I read about
https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html

but couldn't find a spark library for this implementation.

I have columnar string dataset.

I have a dataset with around data of 15-20 million users with their show_watched, times_watched, genre, channel and some more columns, I need to calculate lookalike/s for a user(or 100k users).

How do I find lookalikes for them within less time,

I have tried by indexing data in Solr, and then using Solr MLT for finding similar users, but that takes a lot of time, also it uses TF-IDF for MLT and I need users which have times_show_watched close to that user's times_show_watched.

Can anyone recommend a better approach for this, maybe using any other framework for faster processing?

I also tried to implement clustering using Spark MLLIB and later search in which cluster a user belongs so that search space is less, but I couldn't get this approach finished.

I am open to any approaches which would be efficient.

Thanks!


texts: Difference between isna() and isnull() in pandas

texts:
I have been using pandas for quite some time. But, I don't understand what's the difference between isna() and isnull(). And, more importantly, which one to use when identifying missing values in a dataframe.

What is the basic underlying difference of how a value is detected as either na or null?


texts: Transpose convolution math not working out

texts:
I was reading A Guide to Convolutional Arithmetic to understand Transpose Convolution as it is cited in Keras and Theano documentation. I am having trouble understanding the following two statements :

From section 4.1

  Using this representation, the backward pass is easily obtained by
  transposing C; in other words, the error is backpropagated by
  multiplying the loss with C.T. This operation takes a 4-dimensional
  vector as input and produces a 16-dimensional vector as output, and
  its connectivity pattern is compatible with C by construction.

When I try this out in pytorch, the error is certainly not equal to multiplying with C.T

import torch
import torch.nn.functional as F

x = torch.arange(1, 17, dtype=torch.float).resize_(4, 4)
w = torch.rand(3, 3)

Convolve w and x

# Convert x into an "image" tensor with a single channel and part of a mini-batch of size 1
x1 = x.view(1, 1, 4, 4)
x1.requires_grad = True

# Convert w into a conv filter with a single input channel and a single output channel
w1 = w.view(1, 1, 3, 3)
w1.requires_grad = True

y1 = F.conv2d(x1, w1)

Backpropagate

y1.backward(torch.ones_like(y1))
x1.grad

Now create the C matrix as mentioned in the paper.

C = torch.zeros(4, 16, dtype=torch.float)

C[0, :3] = w[0]
C[0, 4:7] = w[1]
C[0, 8:11] = w[2]

C[1, 1:4] = w[0]
C[1, 5:8] = w[1]
C[1, 9:12] = w[2]

C[2, 4:7] = w[0]
C[2, 8:11] = w[1]
C[2, 12:15] = w[2]

C[3, 5:8] = w[0]
C[3, 9:12] = w[1]
C[3, 13:] = w[2]

Multiplying unrolled y1 by C.T will not equal to x1.grad.

torch.mm(C.transpose(0, 1), y1.view(-1, 1)).view(4, 4)

Where is my mis-understanding?

And the second statement that I have not been able to verify has been cited in a lot intuitive answers on StackExchange/Quora, etc.

Section 4.3

  Another way to obtain the result of a transposed convolution is to
  apply an equivalent – but much less efficient – direct convolution.
  The example described so far could be tackled by convolving a 3 × 3
  kernel over a 2 × 2 input padded with a 2 × 2 border of zeros using
  unit strides (i.e., i' = 2, k' = k, s' = 1 and p' = 2), as shown in
  Figure 4.1. Notably, the kernel’s and stride’s sizes remain the same,
  but the input of the transposed convolution is now zero padded.

When I try this in pytorch, the two results are not equivalent except for having the same shape.

F.conv_transpose2d(y1, w1)
F.conv2d(y1, w1, padding=2)

Again, I am pretty sure I am mis-understanding something, but can't put my finger on what.

Any help in clarifying these concepts greatly appreciated.

P.S: I have already taken a look at this question and it does not answer my question.


texts: class_weight on sklearn's DecisionTreeClassifier

texts:
Can class_weight='balanced' on scikit-learn's DecisionTreeClassifier be interpreted as having identical duplicate data points for the minority classes?

I know that doesn't work that way, class_weight works as a misclassification cost. But I want to understand if it would give the same results as oversampling the minority classes.


texts: How does a 3D, 4D,.. hyperplane look like (visualization)?

texts:
I am looking at support vector machine classification algorithm.

It finds the optimal hyperplane. In linear algebra, hyperplane is a space that is one dimension lower than the ambient plane. For example, in a 2D space, the hyperplane is a 1D line. In a 3D space, the hyperplane is a 2D plane. The following image shows such examples.

I am interested to see the visualization of the hyperplane beyond 2D. For example: 3D, 4D...


texts: Can I apply survival analysis to predict if a user will revisit the website?

texts:
I have one business problem in hand which is to predict if a user will revisit the website or not within 6 months. I need to majorly understand what are the factors which make the user return and also need to give business recommendations on what can be done to make a new user return to the website.
My initial idea was to do logistic regression. Lately, I read about survival analysis. I want to know if I can use survival analysis for this problem.
Also, my dataset has 20k users; each user having multiple transactions; the target variable was not given
I aggregated the dataset to one record per user and did some feature engineering to come up with a target variable.
If I want to use survival analysis in this problem, shall I consider only the last transaction of each user or shall I use the aggregated dataset?


texts:
How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 more as output (x_hat, y_hat). I need a loss which looks something like this:

loss = (x-x_hat)**2+(y-y_hat)**2+abs(sqrt((d(x_hat)/dz)**2+(d(y_hat)/dz)**2)-1)

or a variation of it. The thing is that I need the value of $d(x_{hat})\over dz$ and $d(y_{hat})\over dz$ (so the gradient of the output layer with respect to the inner node). I tried several things using hooks but what I found so far didn’t work. I was thinking to do something like this:

loss_1 =  (x-x_hat)**2+(y-y_hat)**2
loss_1.backward(retain_graph=True)
loss_2 = abs(sqrt((d(x_hat)/dz)**2+(d(y_hat)/dz)**2)-1)
loss_2.backward()

After I call loss_1.backward(retain_graph=True), the gradients that I need should be already calculated, I just don’t know how to access them in order to pass them to loss_2 i.e. what is the value of $d(x_{hat})\over dz$ and $d(y_{hat})\over dz$

Can anyone help me with this? Thank you!


texts: I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.


texts: Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

texts:
First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-gradient-descent-in-linear-regression, https://stats.stackexchange.com/questions/212619/why-is-gradient-descent-required but couldn't find my answer.

Gradient descent is: $w_{i}:=w_{i}-\alpha \frac{\delta }{\delta w_{i}}j(w)$ where w is a vector.

At his book "Pattern Recognition and Machine Learning" Bishop says:

"Because the error function is a quadratic function of
the coefficients w, its derivatives with respect to the coefficients will be linear in the
elements of w, and so the minimization of the error function has a unique solution..."

So if we take derivative of $j(w)$ with respect to $w_{i}$ and equal to zero, in the end it will give us the smallest $w$. Which is actually the first exercise.

In gradient descent we take the derivative too, so the problem can't be because of derivative. Such as an equation that its derivative can't be found. If we can find the answer with one iteration(making equation equal to zero for every feature) why do we iterate over and over again instead, which is the case of gradient descent.


texts: treeinterpreter / feature selection with multidimensional data

texts:
In python i have been using treeinterpreter on 2d data. It has worked well and enabled me to look into the blackbox to understand what is going on. I am interested in the why than the predicted output.

However, a new set of data is given to me daily. It is the same categorised data every day but some of it is dynamic. This however has added a new dimension of time. It is not as important as the features / categories but still has an effect.

What is the best way of dealing with this so i can still look into the tree / svm / etc....i am not too fussed which one.

It may be i need to stack the models in order to get the insight - but i am unsure.

example of the data on one day below - the price change and rating can changes daily.

Wine      Country       Area        Grape        Class    Year    Price   Rating
A         France        Burgundy    Pinot Noir   Grand    2014    +6%      94
B         France        Burgundy    Chardonnay   1er      2014    -1%      91
C         France        Burgundy    Pinot Noir   1er      2014    +4%      96
D         USA           California  Pinot Noir            2013    +1%      92
E         USA           California  Blend                 2014    +0.5%    88
F         USA           California  Chardonnay            2014    -4%      81
G         USA           California  Chardonnay            2013    -5%      91


texts: Standard deviation of estimator to predict age

texts:
Hi all I thought I understood regression but this question has me stumped.
Suppose we have two independent models to predict someone’s age. From experience
we know that the first estimator has a standard deviation of 1 around the true value, the
second has a std deviation of 2. If you wanted to predict someone’s age given these
estimators, how would you do it?


texts:
I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. Now I want that when I try to find the distance/similarity between these movies using the feature vectors, I get different values with respect to which movie is the reference point i.e.,
Dist(Mov1, Mov2) != Dist(Mov2, Mov1)

I know this is slightly vague, but I am trying to understand my thoughts and would like your help. I found that there is a field called non-commutative geometry, but that seems a little too much and the last resort to dive into. Kindly direct my thoughts.


texts:
There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME).

What about for anomaly detection in unsupervised learning? Is there any model for which there are libraries that can give you justifications, such as "row x is an anomaly because feature 1 is higher than 5.3 and feature 5 is equal to 'No'"?


texts:
Is it possible for an algorithm to predict a new class that has never been before in training? For example, in my training data I have:

I need a flight to Boston --> flight query

What is two plus two --> math query

Who is the president of the United States --> Donald Trump

How can I make a model that can predict a new class for:

How are you today --> ??? (Should be something like conversation query)

I am almost looking for a way to combine supervised learning and unsupervised learning. Using supervised learning for a model to predict new classes (unsupervised learning)


texts: Time array ( 10 min ) for start time to next time in time series import from csv file using python

texts:
Here I have time series import from csv file. Start time 7.00 a.m next time is 8.00 a.m next time is 9.00a.m like that it will continue till to midnight. Then again next day it will start with time 7.00 a.m . So here I want to write a time array for starting time to next time with time delta 10 min. Can you help me to solve this problem? Here I wrote one code , but I don't know how to add start time from my dataset import from csv file.

my code is:

def datetime_range(start,end,delta):
    current = start
    while current < end:
        yield current
        current += delta
data = pd.read_csv('data6.csv')
dts = [data.strftime('%Y-%m-%d T%H:%M') for data in
       datetime_range(datetime(2018, 9, 28, 7), datetime(2018, 9, 28, 7+2),
       timedelta(minutes=10))]
print(dts)

Output:

['2018-09-28 T07:00', '2018-09-28 T07:10', '2018-09-28 T07:20', '2018-09-28 T07:30', '2018-09-28 T07:40', '2018-09-28 T07:50', '2018-09-28 T08:00', '2018-09-28 T08:10', '2018-09-28 T08:20', '2018-09-28 T08:30', '2018-09-28 T08:40', '2018-09-28 T08:50']

Here I upload how output is came according to my code. But I have to mentioned the start time and end time and it is not read from the csv file. Can anyone give me any suggestions to solve his problem?


texts: I want to be able to collapse and sum values dependent on the gene name

texts:
I have a table that looks like this:

I want add together all the values for each gene for each column. For example, for LINC01128, it should read:
             ConN1   ConN2   ConN3   StN1   StN2   StN3
LINC01128 :    22      14      37     34     54     67

My table is very long and this would need to be done for all the genes.


texts:
I am trying to find the parties between which the agreement is executed from the statement below.

AGREEMENT, dated as of January 10, 2000, is entered into by and between ABC-EFG GROUP Inc. having an address at 418 Mona Drive, Prominade 34, Florida 34673, United States of America and Rob Cummins, an individual residing in the state of Florida, and having an address at 13 test Dr, Arosa, FL 43566

To get this, I tried the following code:
for chunk in doc.noun_chunks: # after loading "nlp = spacy.load("en_core_web_sm")" and doc = nlp("string")
   print(chunk.text, chunk.root.text, chunk.root.dep_,chunk.root.head.text)

which gives me:
AGREEMENT AGREEMENT nsubjpass entered
January January pobj of
ABC-EFG GROUP Inc. Inc. pobj between
an address address dobj having
418 Mona Drive Drive pobj at
United States States conj Drive
America America pobj of
Rob Cummins Cummins conj America
an individual individual appos States
the state state pobj in
Florida Florida pobj of
an address address dobj having
13 test test pobj at
Dr Dr ROOT Dr
FL FL appos Arosa

I don't understand how I am supposed to find the two parties from this output.
I am not specific to spaCy. A suggestion of any other model or method also will work.
Please note that I have already tried NER but the results are very poor.


texts:
I have a general question on reservoir sampling. When I use this method to sample a very large dataset for training machine learning classification algorithms, I am curious as to how to make my pipeline robust to fluctuations in  class distribution across samples.

For example, suppose I am working on a binary classification problem. I want to sample a relatively small subset of data on which to evaluate my algorithms, so I used reservoir sampling. However, it seems to me (thought I could be wrong) that by chance I may draw a sample wherein the class distribution is significantly different from that seen in the larger population of data. If this is a correct inference, can anybody tell me how to remedy the situation? If reservoir sampling is not the answer, what other procedure are out there that I can leverage in this case (apart from using distributed environment)?

I am learning, so some explanation, hint, and direction would be greatly appreciated.


texts: How do I split contents in a text that would include two or more different themes (context) in NLP?

texts:
For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning"
the output should be:

The airline companies have affected by Corona since march 2020
a crime has been detected in Noia village this morning

the text has no Breaks. I know it is not a one-click solution, but if anyone knows a methodology or techniques to solve such a problem, please provide me with resources.


texts: What is the best machine learning algorithm for large, noisy datasets with interaction between variables?

texts:
My initial thought was a neural network but I don't see how a neural network can properly predict interaction between variables (ie. x1 * x2) since each node is just a sum of previous inputs?
Would a decision tree be better suited at capturing the interaction between variables?
My dataset is large, with 400 features and 5,000,000 instances. All data is in percentile and the label is also a percentile. The dataset is quite noisy as well, (customer data, predicting likelihood of becoming a return customer).


texts: sklearn KNN fit throws out error : value too large for dtype('float64')

texts:
I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8  (it's a sum)
I have no Idea how to fix this last error, I tried all previous solutions which are related to my question but nothing worked.
This is the result of : df.isnull().sum():

I am following this tutorial (because I'm just starting to learn about machine learning supervised algorithm KNN):
Tutorial Link
Please help me out!


texts: Machine Learning Recommender high data intensity

texts: We are building a recommender engine to be integrated in an app that, each time an API is called, will pull thousands of records from an Azure SQL database and create recommendations. Currently with Azure, the time taken to query thousands of records is slow (several seconds). Is there any way to improve this speed? Is another database provider such as GCP faster in this case? Thanks


texts:
Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

That is, to get results which say that, for example, the bigram "hello world" is much more common in corpus A than corpus B (ideally with some kind of measure of how much more common).

TF-IDF examples use a larger number of documents to highlight "important" words in each, but I am not sure if that would work in this case?


texts: Determine relationship between users and age?

texts:
I would like to understand how to find an association between users, spam and email's age.
My dataset looks like as follows:
User      Spam      Age (yr)
porn_23    1         1
Mary_g     0         6
cricket_s54 0        4
rewuoiou   1         0
pure75     1         2
giogio35   0         10
viv3roe    1         1

I am looking at the correlation using Pearson. Is it right? I would like to determine the correlation between age and user: spam email should likely come from users having recent email's addresses (fake account / email).


texts: Ljung-box test on weekly percentage of total quarter bookings

texts:
I have a data on the weekly percentage of the total quarter bookings. The data looks as follows (note: weekly percentages add up to 100 for each quarter) :

(not real data)
I used the Ljung-box test to see if the data resemble random data. The Ljung-box test give me a very high p-value (0.7), indicating that the data resembles complete random data.
However if I use the actual numbers of the weekly bookings (not percentage of total quarterly), I get a fairly low p-value in the Ljung-box test, indicating that the data is not random.
Question: Is this expected when using percentages of the total? If so, why?


texts: Methods to visualize the filters in the later layers of a CNN?

texts:
I've extracted the weights from the filters of a pretrained model (AlexNet). I wish to represent these weights visually, this works fine for the first layer as there is only 3 input channels so I can represent the filters as color images with channels (RGB).
However this breaks down in later layers as they often have more than 3 input channels, I can represent them as many greyscale kernels but was hoping someone has found a better solution.


texts: How to match a user with other users with similar interests based on their attributes?

texts:
Information Available

Consider, there are 'n' users and they have these attributes and values

User A:
Row   | Attribute a | Attribute b | Attribute c
Item 1|    0.593    |    0.7852   |   0.484
Item 2|    0.18     |    0.96     |   0.05
Item 3|    0.423    |    0.886    |   0.156

User B:
Row   | Attribute a | Attribute b | Attribute c
Item 7|    0.228    |    0.148    |   0.658
Item 8|    0.785    |    0.33     |   0.887
Item 9|    0.569    |    0.994    |   0.374

Items in this dataset can be described using the attributes a, b, and, c. So, the items might or might not be the same for different users but the attributes explain the taste of the user.

Currently, I have data for about 1000 users in this format and I can create a classifier for one user that says whether the user will like the given item or not.

Goal

What I want to do is to match users who have similar tastes using the info available above. I don't know much about Recommendation Systems and I'd really appreciate if someone could help me out.


texts: Why do we usually have fully connected layers of same sizes in CNNs?

texts:
Is there any specific reason that we observe in CNNs, the fully connected layers usually have the same sizes? You can verify this for many CNNs.
I'm aware that if, for instance, we have a vector of two components and we transform it into a three-dimensional vector space, the basis of that vector will contain still two components at most. Consequently, if we have, for instance, a layer with 100 neurons and the subsequent layer has 120 neurons, our outputs can still be described with at most 100 neurons, although they are transformed from 100-space to 120-space. The point is that this is linear algebra. In fully connected layers, we have nonlinearity other than linear calculations.


texts: What is the best way to use golden feature in machine learning model?

texts:
Let’s say I have a classification model. And my job is to predict the correct class out of 30 different classes. The current accuracy is 60%.

The thing is: I have to consume another team’s classification result which is 80% accurate. So I’m using their prediction result as a feature. I’ll call it “golden feature”. Let’s say I’m aiming >80% accuracy with the golden feature.

Here is my current approach:

(I’m using Deep Learning.) I have several features and each feature has its own weight. I also create a weight vector for one hot vector (1 by 30) of “golden feature” and train all weights together. However the result doesn’t seem to provide much.

I thought about the reason why and realized that the learned vector (30 by 30) won’t be that meaningful. They would be just positive numbers.
(Please yell at me if my reasoning is wrong!)

Has anyone faced the similar problem? Any suggestion will be highly appreciated!
The method that you suggest doesn’t have to be Deep Learning approach.


texts: Unable to resolve Type error using Tokenizer.tokenize from NLTK

texts:
I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution Note','Description','Shortdescription' are text data in English- here is the code that I have written :

#Removal of Stop words:

    from nltk.tokenize import sent_tokenize, word_tokenize
    from nltk.corpus import stopwords
    from nltk.tokenize import RegexpTokenizer

    tokenizer = RegexpTokenizer(r'\w+')
    stop_words = set(stopwords.words('english'))

    tokenizer = RegexpTokenizer(r'\w+')
    dfclean_imp_netc=pd.DataFrame()
    for column in ['Resolution code','Resolution Note','Description','Shortdescription']:
        dfimpnetc[column] = dfimpnetc[column].apply(tokenizer.tokenize)
    for column in ['Resolution code','Resolution Note','Description','Short description']:
        dfclean_imp_netc[column] = dfimpnetc[column].apply(lambda vec: [word for word in vec if word not in stop_words])
    dfimpnetc['Resolution Note'] = dfclean_imp_netc['Resolution Note']
    dfimpnetc['Description'] = dfclean_imp_netc['Description']
    dfimpnetc['Short description'] = dfclean_imp_netc['Short description']
    dfimpnetc['Resolution code'] = dfclean_imp_netc['Resolution code']

My error output is attached below:


texts: Convolutional neural network overfitting. Dropout not helping

texts:
I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each).

I've managed to achieve around 85% classification accuracy on my test set, however I set a goal of achieving 90% accuracy.

My main problem is overfitting. Somehow it always ends up happening (normally after epoch 8-10). The architecture of my network is loosely inspired by VGG-16, more specifically my images are resized to $128x128x3$, and then I run:

Convolution 1 128x128x32 (kernel size is 3, strides is 1)
Convolution 2 128x128x32 (kernel size is 3, strides is 1)
Max pool    1 64x64x32   (kernel size is 2, strides is 2)
Convolution 3 64x64x64   (kernel size is 3, strides is 1)
Convolution 4 64x64x64   (kernel size is 3, strides is 1)
Max pool    2 32x32x64   (kernel size is 2, strides is 2)
Convolution 5 16x16x128  (kernel size is 3, strides is 1)
Convolution 6 16x16x128  (kernel size is 3, strides is 1)
Max pool    3 8x8x128    (kernel size is 2, strides is 2)
Convolution 7 8x8x256    (kernel size is 3, strides is 1)
Max pool    4 4x4x256    (kernel size is 2, strides is 2)
Convolution 8 4x4x512    (kernel size is 3, strides is 1)
Fully connected layer 1024 (dropout 0.5)
Fully connected layer 1024 (dropout 0.5)

All the layers except the last one have relus as activation functions.

Note that I have tried different combinations of convolutions (I started with simpler convolutions).

Also, I have augmented the dataset by mirroring the images, so that in total I have 50000 images.

Also, I am normalizing the images using min max normalization, where X is the image

$X = X - 0 / 255 - 0$

The code is written in tensorflow and the batch sizes are 128.

The mini-batches of training data end up overfitting and having an accuracy of 100% while the validation data seems to stop learning at around 84-85%.

I have also tried to increase/decrease the dropout rate.

The optimizer being used is AdamOptimizer with a learning rate of 0.0001

At the moment I have been playing with this problem for the last 3 weeks and 85% seems to have set a barrier in front of me.

For the record, I know I could use transfer learning to achieve much higher results, but I am interesting on building this network as a self-learning experience.

Update:

I am running the SAME network with a different batch size, in this case I am using a much smaller batch size (16 instead of 128) so far I am achieving 87.5% accuracy (instead of 85%). That said, the network ends up overfitting anyway. Still I do not understand how a dropout of 50% of the units is not helping... obviously I am doing something wrong here. Any ideas?

Update 2:

Seems like the problem had to do with the batch size, as with a smaller size (16 instead of 128) I am achieving now 92.8% accuracy on my test set, with the smaller batch size the network still overfits (the mini batches end up with an accuracy of 100%) however, the loss (error) keeps decreasing and it is in general more stable. The cons are a MUCH slower running time, but it is totally worth the wait.


texts: What dataset was Stanford NER trained on?

texts:
I would like to re-train the Stanford NER library from scratch as a 1 class model.

Only 3,4 and 7 class models are available out of the box.

Is it possible to obtain the data that the model was originally trained on?


texts: Conv1D specify output chanels in tensorflow 2.1

texts:
Hello I'm trying to implement the "Tacotron Towards end to end" paper and in the
Encoder CBHG - a Conv1D bank of K=16, conv-k-128-ReLU
“conv-k-c-ReLU” - denotes 1-D convolution
with width k and c output channels with ReLU activation.
And I wanted to ask how exactly I can implement it in Tensorflow 2.1? because in the documentation I haven't seen how to specify the output channels

Reference to the paper: https://arxiv.org/pdf/1703.10135.pdf


texts: Recognition human in images through HOG descriptor and SVM classifier performs poorly

texts:
I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV.
I've used the excellent tutorial at pymagesearch, which explains what the algorithm does and furnishes hints on how to set the parameters of the detectMultiScale method.

Specifically, I do

# initialize the HOG descriptor
hog = cv2.HOGDescriptor()

# Set the support vector machine to be pre-trained for people detection
hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())

# Detect people in the image
(rects, weights) = hog.detectMultiScale(image,
                                        winStride=(4, 4),
                                        padding=(8, 8),
                                        scale=1.05)

The parameters are chosen according to a fine tuning on both accuracy and performance, following the explanations in the tutorial itself.

My problem is that this method, which seems like the currently best method to recognise humans in a picture according to literature (the original paper is dated 2005) seems to perform pretty poorly on my images.
I have images containing clothes, both with a model and without it and I'm trying this approach to recognise those with the model. On a subset of 300 images which I manually scanned to tag them for containing the model or not, the method fails 30% of the times.

These are some images as examples.

Here it detected a missing human:

Here it did not get the full human:

Here it did not recognise it at all:

I understand that the detector works for upright humans. Should they also be full-figure?
My images encompass half-figure, figures with no head or no feet.

Before this, I have tried a Haar-feature based cascade classifier to recognise the face in an image and the accuracy on the same set of image has been 90%, so I was trying to improve on this. Also, I'm interested in understanding why things are not working here.


texts: I am extracting tweets on brands for sentiment analysis. I am using twitteR package on R. Is there a specific way by which I can only collect tweets from a location ex NY or Paris or Canada etc. I have tried Geocode but here I have to give lat and long data and the vicinity range.


texts: Is it possible to run clustering methods by only knowing the distance between pair of points?

texts: By knowing each data point's coordinate, it is easy to apply them with clustering methods as k-means etc. By if the case is we only know the distances between each pair of data points without knowing the definite location coordinate of every data point, is it possible to apply any clustering methods in this case?


texts: Does severe multicollinearity affect solving linear regression by gradient descent?

texts:
Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario?
Does it converge at the minima?

(My guess is, Cost function of linear regression is always convex. so nothing should stop steepest descent algorithms from coverging)


texts:
I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0's, since there is no continuous flow of cars coming to charge but rather reoccurring charging events as peaks(for example from 7-9 am seems to be a frequent charging timeframe when people are coming to the office)
I have also aggregated weather and weekday/holiday data to be used as features.

I now wish to predict the energy demand for a timeframe of 6h in the future.
So far I have tried SARIMA with terrible results, since the algorithm seems to be obscured by the sparse data.
I have tried different transformations (Box Cox, Normalization, Standardization), differencing, auto-arima for optimal parameters, so far no luck.
I am willing to try different machine learning as well as statistical algorithms.
Does anyone have some recommendations as of what I can do to generate a moderatly accurate prediction with a sparse dataset?


texts: Imbalanced Data how to use random forest to select important variables?

texts:
I am trying to use random forest to select important variables out of 15K features and fit them into logistic regression. My evaluation is based on F1 score. Dataset 2 classes ratio are around: 99.5:0.5.

Here is where 15K features come from:
Initally I have 2000 features, after dumminize them (only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance ratio is also very small. I tried removing near zero variance features before and it significantly reduces the number of features, however the logistic regression result was also not good.

However, after using grid search, random forest still has very low f1 score for cross validation. (less than 0.01)

Also I tried undersampling for the training data to make it 1:1, random forest still has bad F1 score for cross validation. :(

So I was thinking about not selecting the important variables and just fit all features to logistic regression.

Due to memory issue, I am not able to fit all 15K features into logistic regression directly but if I select important variables from random forest, they do not generalize for unseen test data.

Any ideas on how to solve the problem?
I know one alternative could be using hashing so I could fit all 15K features.


texts: Machine learning for missing data in time series

texts:
We have two time series columns - column A is the reference column ( source of truth) and column B is a ''cousin'' of column A, in the sense that it exhibits ( or should exhibit) the same patterns, evolution, rates of changes etc, as column A.

However, at certain periods column B values for wtv reason start exhibiting abnormal values which we are comfortable to classify as outliers. We are happy to remove these value as outliers.

Now given the unequal timeseries ( post outlier removal of column B) would like to explore machine learning techniques that would impute data in column B, using column A as the reference or source of truth, effectivly replicating the behavior of column A to impute missing data in column B.

I've researched many models to use, regression, LTSM etc. hoping for some expertise that would help shed light on what technique would best work for the problem statement at hand.


texts: Evaluating Random Forest regression model that predicts low values for skewed dependent variable

texts:
Background

I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values between 1 and 1000 (median at about 50, 95% percentile at 150). The number of samples are a few millions.

The kind of feature variables that I have are things like country, device info (e.g. iphone, android, desktop computer), browser version, time of visit, day of week, which page they entered on, etc. From these variables it's impossible to predict, with a high degree of certainty, if the user is going to make a purchase, so the predicted value for each visitor should be a fairly low value (somewhere between 0.2 and 4). However by looking at the data it's clear that the feature variables does affect the average value of the visitors (e.g. users on desktop are 25% more likely to make a purchase than mobile users). The predicted values are used in ad bidding.

I started out with a "model" where I simply predicted each visitor's value to the average visitor value. From there I moved on to an extremely basic, naive non ML model where I simply just modify the average value based on just a few feature variables that I've seen has a large impact (on average). E.g. make the desktop visitors worth 25% more than mobile visitors, and increase the value of visitors on week-ends by 10%. I empirically saw clear improvements from using this naive model instead of just using the mean value for each visitor. However, this obviously has problems since the different features might be correlated, and I would also like to be able to use as many features as possible that might improve the predictions.

Therefore I figured some kind of ML-model should be able to perform better than my very naive model would. I decided to start with a Random Forest regression model, since it seemed like a good model to start with (supposed to work fairly good for many different problems, has few hyper parameters to tune, works with highly correlated features, and requires little preprocessing).

The question(s)

I've done quite a bit of feature engineering and I'm now able to train a random forest model. However, I'm not sure how I can best evaluate the performance of my trained model (except for testing it out empirically, which takes quite a bit of time so it's not feasible just for tuning hyper parameters).

Since the predicted values are quite low (around 1) while the actual, rare purchases are high (around 50), the R2 score that I get for my test set is extremely low. After some hyper parameter tuning (n_estimators=300, min_samples_leaf=100, max_features=3, min_samples_split=5) the best R2 score I get for the test set is ~0.006. The R2 score for my naive "model" for the same test set is ~0.003.

Does this mean that my RF model performs much better than the naive one?
Does it make any sense to look at R2 score when it's so low?
Is there any other method of evaluating the model that would better suit my case?

I'd think an R2 score of 0.003 would indicate that the model is quite garbage, however I know empirically that my naive model yields significantly better results for our use-case than just predicting the mean.

I've tried to provide as much (possibly) useful info as possible. I'm a data science novice.


texts: Weights in scikit-learn metric functions

texts: Most of the scikit-learn metric functions have an option to take into account sample weights. What is the purpose or use case to use this options? Unbalanced classes?


texts: How to transform and cummulate data in python

texts:
I would like to transform dataframe and cummulate them using pandas.

year country value
1999 JAPAN   10
2000 KOREA   15
2000 USA     20
2001 USA     13
2002 JAPAN   30

* I want to transform dataframe and cummulate value for each country

year country value
1999 JAPAN   10
1999 KOREA   0
1999 USA     0
2000 JAPAN   10
2000 KOREA   15
2000 USA     20
2001 JAPAN   10
2001 KOREA   15
2001 USA     33
2001 JAPAN   40
2001 KOREA   15
2001 USA     33

I need your help.

Thank you.


texts: What's the difference between setting negative label as 0 or -1 in binary classification?

texts:
I would like know what's the difference between setting the negative labels as 0 or -1 in the binary classification. I guess there is no important difference, just some notation changes in the model.
Thank you in advanced!


texts: I saw some impressive result from LSTM models producing Shakespeare like texts. I was wondering if an LSTM package exists for R. I googled for it but only found packages for Python and Julia. (maybe there are some performance issue which explains why these programs are more preferred over R) Do You know about an LSTM (or at least an RNN) package for R? If exists are there any tutorials for using them?


texts:
I'm trying to plot my test set and test set predictions to check the differences and see how my autoencoder reconstructed the data, but since I have a test set 30x10 I have a huge visualization problem:

How can I solve it? This is the code, I'm just showing the first row (X-test[0]), but I have 29 more and I don't how to show all of them properly.
fig = plt.figure()
plt.plot(X_test[0].T, label='y_true')
plt.plot(x_test_pred[0].T, label='yhat_conv')
_ = plt.legend()
plt.show()
plt.close()


texts: How can decision trees be tuned for non-symmetrical loss?

texts:
Suppose we use a decision tree to predict if a bank customer can pay back a credit. So it is a two class classification problem. Now we can make two mistakes:

$\alpha$ error: The customer can back the credit, but we predict he can't.
$\beta$ error: The customer can't pay back the credit, but we predict he can.

Now we know that $\beta$ errors are 123.4 times as expensive as $\alpha$ errors. But we only have a given set of data. In this set we have $n_1=10000$ customers who paid back the credit and $n_2 = 100$ customers who didn't.

How can the training of the decision tree be adjusted to account for the fact that $\beta$ errors are more expensive?

(Note: This is a theoretical question to learn about decision trees. I know about other classifiers like neural networks and I know of ensembles. However, I only want to know about decision trees here.)


texts: Combining spatial input with a label as input for CNN using Keras

texts:
I also asked this question on Stack Overflow. However, it has not yet been answered and I think this is a more suitable platform to place it.

I'm trying to implement a network set-up similar to this Google Deepmind paper. Their network set-up is as follows:

Mθ is a convolutional network, so I was wondering how they concatenate the input frames with the viewpoints? As far as I knew, the CNN takes into account the spatial information right? So does it even make sense to concatenate the frames with the viewpoints as an input for the CNN?

Thanks in advance!

Edit

I was thinking about it and they might have concatenated the viewpoint on the dense layers behind the convolution layers. Might that be the case?


texts:
Let me describe my problem:
I've got around 1500 items, all of them described by 15 numeric (rational numbers, positive and negative) attributes.

As an minimalized example:

Item A:
- value I: 34
- value II: 12335
- value III: -10
- value IV: 0
....

Item B:
- value I: 500
- value II: -2332
- value III: 0
- value IV: 9
...

No i have to combine them and find the best combination.
There is space for about 70 items and each item can be used n times.

The performance of each combination can be determinated by a simple math formular using 3 of the attributes, while the other attributes has to match min or max requirements to determinate if a combination is valid or not.

because some items have good values for a single attribute, but negative for other attributes, other items have to be taken to compensate these negative aspects

Now I'm looking for an approach/algorithm/tooling to find the best combination or at least get near to it.

My approaches were to (in pseudocode)

//approach A
while(true)
  var best = generateRandomCombination()
  var challenger = generateRandomCombination()
  best = betterPerformanceIndicator(best, challenger)

at some point i have to stop it and use the result.
tried it, let it run for an hour and loggt every single improvement. horrific performance, to much randomness means repetition. and of course it does not came near to a result i'd produce on my own.

//approach B
var best = generateRandomCombination();
while(true)
  var challenger = replaceSingleItemWithRandomOne(best)
  while (!challenger.isBetter(best) && untestedItemsLeft()){
    challenger = replaceWithOtherItem(best);
  }
  best = betterPerformanceIndicator(best, challenger)

This approach is some kind of approximation, but i think because of the needed compensations it won't be very efficient to change only one item.

So i see two possible ways to improve:
- perhaps i should give the items some kind of weighting
- replace more than on item

Do you have any suggestions to improve my approaches or replace it with better ones?
Is this some kind of data-mining problem? What should i learn to deal with it?

(if my question is wrong in here, i'm sorry. would be nice if you would redirect me to the correct stack exchange section)

added information due to comments:

in the beginning i have to fix the container properties, to determinate if there is space for 70, 71 or an other amout of items. And to to determinate the multiplicator of each attribute. As an example: the containers multiplicator of "value I" would be 1,1 - adding Item A and Item B wour result in (34 + 500) * 1,1 = 587,4.
So you see, adding the items is simple math (addition and multiplication)
Concerning speed, my first approaches were written in java using Lists, but i have not tracked the speed of these single operations.
the order of the item does not effect the outcome, so it's about building an unordered set, just like Neil asked.
The formula to compute the total value is:

square root of ( (sum of all valueI) multiplied by (sum of all valueII and valueIII) )
min and max requiremants are just natural numbers, that the sum of all valueN should (not) reach at least


texts: Why my regression model always be dominanted by one feature?

texts:
I am working on a financial predict problem. which means it is a time series prediction problem.
I have three features, which have high correlation(each two's corr is about 0.6)
And I do the linear regression fit.
I assume that the coefficient should be similiar among these three features, but i get a coefficient vector
like this:
[0.01, 0.15, 0.01]
which means the second features have the biggest coff(features are normalized), and it can dominant the prediction result.
I dont know why. I think adding weak features can boost my prediction model, but i think the second feature is dominant in my model, and other features are worthless.
Why one of features can be dominant in the model, did I miss something?


texts: Fully-Configured Deep Learning Virtual Machines in Python (VirtualBox or VMware)

texts:
Fully-Configured environment setup for Deep Learning Virtual Machines in Python (VirtualBox or VMware)

Often when we start working on any new technology, the most common challenge that we face is "How can I not waste time setting up my environment but directly working on the technology?"

If you aren’t a Linux user, it can be really hard to figure out how to get a system fully configured with all the required machine learning libraries and tools like TensorFlow, Theano, Keras, OpenCV, and dlib. A lot of people get stuck while installing everything and give up before ever getting to play around with any code.

Earlier I used to work on Fully-Configured Deep Learning VM in Python VMware shared by Adam G with the below configuration.

  Link for VMware Image

  Link for VitualBox Image

    The virtual machine image has Ubuntu Linux Desktop 16.04 LTS 64-bit pre-installed with the following machine learning tools:

Python 3.5
OpenCV 3.2 with Python 3 bindings
dlib 19.4 with Python 3 bindings
TensorFlow 1.0 for Python 3
Keras 2.0 for Python 3
Theano
face_recognition for Python 3 (for playing around with face recognition)
PyCharm Community Edition already set up and ready to go for all these libraries
Convenient code examples ready to run, right on the desktop!
Even the webcam is preconfigured to work inside the Linux VM for OpenCV /
face_recognition examples
(as long as you set up your webcam to be accessible in the VMware settings).

Currently looking for the latest version of Ubuntu with updated Deep Learning packages.

I've been searching around the Internet for a while but I have not been able to find Fully pre-configured latest VM with the updated versions.

I would really appreciate if anyone here can share his/her method if he/she had used a Fully-Configured Deep Learning VM in Python (VirtualBox or VMware)


texts: Using a past time series to predict how a present time series will pan out?

texts:
Let's say that I have past data indicating how some time series panned out. Now I also have the beginnings of a new time series that I expect to pan out in a similar trend to the old one.

What are some general data science / machine learning / statistics techniques I can use to predict the rest of the new time series? What about using some sort of online model to iteratively predict the old model's trend, and then somehow applying that to the new model?


texts: Putting a model [sklearn] from Git into production

texts:
Currently the DS team provides Python (sklearn) models to put into production. This is considered a devops task. The workaround in lieu of expensive (platform /infrastructure) we went for is using FastAPI to put the models into production.
But now as we move to more CI/CD flow I am trying to figure out if we have open source options where the DS puts their pickle file and function (py) to git and this would trigger automatic build and deploy. [Currently if there is a change to existing model we can do it by using Git/DockerHub/updated docker image but this is because we have already build an image for our model based on FastAPI] .
For a new model with just pickle and the function is it possible to put into production (expose as a REST API)  without a layer like Flask/FastAPI. I feel with annotations it should be possible to expose it as REST APIs for inference.
Apologize if the question is incoherent as I am not familiar with DS and I am just considering it from a general programming perspective. Thank you.


texts: How to apply variational autoencoder for oversampling with cross-validation?

texts:
Currently, I have an imbalanced data set with proportions 84% and 16%. I wanna use VAE as oversampling method and I want to determine the best proportions of data that results in better metrics. Also, I want to use cross-validation for this purpose. The following code is the algorithm that I am using. The problem is that in some cases num_samples might be negative because after diving the data, the ratio changes. For example, in the current model, the proportions are 84 and 16 percent. However, after diving data, this ratio will change like 80-20 %. How should I apply this algorithm such that the results will be fair and comparable?
kfold = StratifiedKFold(n_splits=10, shuffle=False)

# Number of samples to generate
maj = len(y_val_train[y_val_train == 0])
mino = len(y_val_train[y_val_train == 1])

thresholds = np.arange(0.16,0.505,0.02)
measures = np.zeros((np.size(thresholds),3))

no_replication = 10

for train, test in kfold.split(X_train.values, y_train.values):

    X_val_train,y_val_train = X_train.values[train], y_train.values[train]
    X_val_test, y_val_test = X_train.values[test], y_train.values[test]

    counter = 0

    for frac in thresholds:

        num_samples = int(round(1/(1/frac - 1) * maj - mino))

        # Variational Oversampling
        vos = VOS(hidden_dim= 1000,
                                latent_dim=2,
                                minority_class_id=1,
                                verbose=1,
                                epochs=15,
                                num_samples_to_generate = num_samples,
                                random_state = 0,
                                optimizer="Adam")

        #Fit the VAE oversampling model and get new data set
        X_res_val,y_res_val = vos.fit_oversample(X_val_train,y_val_train)

        #--- Fit and predict with Random Forest
        logReg_vae = RandomForestClassifier(n_estimators=50, random_state=0)

        #fit
        lr_vae_fit = logReg_vae.fit(X_res_val, y_res_val)

        #predict
        pred_lr_vae_val = logReg_vae.predict(X_val_test)

        #F1-Score
        mom_vae_val = f1_score(y_val_test, pred_lr_vae_val)
        measures[counter,0] += f1_score(y_val_test, pred_lr_vae_val)
        measures[counter,1] += recall_score(y_val_test, pred_lr_vae_val)
        measures[counter,2] += accuracy_score(y_val_test, pred_lr_vae_val)

        counter+=1
 ```


texts: Explaining GRU equations from two sources

texts:
I really confused about how GRU computation really works. I am not really good at math btw.

I read the original source of GRU equations from Cho, et al. (2014), and also a Colah's blog post on the same topic.

And somehow, GRU equations at Colah's blog post Understanding LSTM Networks are different than equations in the Cho's original paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.

These are the equations from Colah's blog:

$$z_t = \sigma(W_z \cdot [h_{t-1},x_t])$$

$$r_t = \sigma(W_r \cdot [h_{t-1},x_t])$$

$$h_t = (1-z_t) \ast h_{t-1} + z_t \ast \tilde{h}_t$$

$$\tilde h_t = tanh(W \cdot [r_t \ast h_{t-1}, x_t]_j)$$

These are the equations from the original paper:

$$z_j = \sigma([\mathbf W_z \mathbf x]_j + [\mathbf U_z \mathbf h_{(t-1)}]_j)$$

$$r_j = \sigma([\mathbf W_r \mathbf x]_j + [\mathbf U_r \mathbf h_{(t-1)}]_j)$$

$$h_j^{(t)} = z_jh_j^{(t-1)} + (1-z_j) \tilde h_j^{(t)}$$

$$\tilde h_j^{(t)} = \phi([\mathbf W \mathbf x]_j + [\mathbf U(\mathbf r \odot h_{(t-1)})]_j)$$

Can someone explain if Colah's equations are the same as the equations in the paper version?


texts: I am trying to run xgboost in scikit learn.  And I am only using Pandas to load the data into a dataframe. How am I supposed to use pandas df with xgboost?  I am confused by the DMatrix routine required to run the xgboost algorithm.


texts: XGBoost, binary classification: uneven number of observations per user

texts:
I'm working on a binary classification problem with XGBoost and I have a dataset, which has uneven number of observations per user. For some users there are over 100 observations, whereas for some users there are only a few. The "USER_ID" feature is not used as an input for XGBoost.

More specifically, I'm trying to model user physical activity (data collected from wearable trackers) in respect to sleep quality, and some of the variables are demographical features such as as age and sex, alongside steps, heart rate etc. Considering the differing amount of data collected from users, some user behaviours (such night-shift work) are represented more in the data than others due to the number of observations.

How should I take this into account when working with XGBoost?

USER_ID  AGE  SEX  X1  X2    ...  y
1        20   M    65  3000  ...  1
1        ...  ...  ... ...   ...  0
1        ...  ...  ... ...   ...  1
2        30   F    80  2500  ...  0
2        ...  ...  ... ...   ...  1
3        40   M    77  8000  ...  0

The classes are otherwise balanced and I'm able to get good performance for the classifier.


texts: Translating a business problem into a machine learning solution: job-adds website

texts:
I have the following problem:

A company, let's call it X, has a job-adds website. It works as a marketplace where job-seekers and companies that have job vacancies can meet.

Their business model is such that job-seekers pay a subscription to get access to the website after a trial period. Companies, on the other hand, don't pay anything to post job-adds.

In order to attract companies to post jobs on the website they call them and ask if they're interested. They have a database with N companies that did business with them in the past. Out of those "N" they can only call 20% due to budget and personnel constraints.

They select those 20% out of simple heuristic: companies that posted job adds in the last six months.

What they want is to call companies that have high probability of posting a job add that will generate a match. They consider that a match has happened based on a proxy: the company contacted n different candidates for a particular job-add.

I want to find a data-driven approach to select the best companies for them to call. In other words, provide them with a list of companies that have a high probability of generating adds that match with their current database of users.

The problem I'm having is how to transform this business problem into a machine learning solution.

What I have thought so far is to build a classification model. Each row is a job-add. The features are variables related to the company (size, location), to the user (age, CV, last time logged in), and the job-add itself (salary,education required, position, industry). The target variable is a binary of "the job had a mach or not."

The problem with my approach is that the model is not actually learning anything since my target variable is already known for each job-post (I know which job-adds had a match and which didn't).

I'm looking for some ideas on how I can structure this problem in a data-driven framework. Should I continue with my classification model? Maybe some type of collaborative filtering? I'm sure machine learning can provide better results than their simple heuristic. Thanks!

The data I have:

User data, including CV;
Who applied to what job and when;
Data from the companies that posted job-adds;
Contacts made by companies to job-seekers (This just tells me if the company clicked a button to look at the user's contact info);


texts: How can I find contextually related words and classify into custom tags/labels?

texts:
PROBLEM:

Suppose if I have a small dataset containing some words and their tags/labels.
The main task is to provide tags to other words(which are not in the dataset) based on their contextual relationship with the words already in the dataset.

Let's say, for example, my custom dataset includes

              Soap --> label__(cleaning_agent)
              pencil--> label__(stationary_item)
              mobile--> label__(electronics)
              washingmachine--> label(electronics)
              and so on.

I want my program to be able to correctly predict the label of an unknown word, e.g.

         washing powder to its correct category
         label__(cleaning_agent)
         radio to label__(electronics) etc.

ACTIONS:

Now the main problem is to find the relation between two words based on context, but I am not able to decide what can be the parameters for finding that.

I have tried a naive approach using datamuse API and fastText library.

    Naive approach is as follows->
        step 1-> find all the related words of the given word(let's say W) e.g. pencil using datamuse API.
        step 2-> combine them into a string(let's say S) with spaces in between them
        step 3-> use the label name, W, S as the training dataset for fastText.

NOTE: fastText requires label name, word, sentence(can be from news articles, blogs, Wikipedia etc ) as a context for that word.

RESULTS:
fastText is not providing any reliable results.
I am thinking of building a neural network kind of thing for this purpose.
But I am not able to decide what can be the input parameters for our data.

The main problem is about custom word tagging. Our program should be able to tag unknown(not in training dataset) words to their most probable classes based on some score.

As I am new to NLP, I want to know what can be the next move forward.


texts:
I have multiple .csv's files, each of them represent a product.
I am using LSTM to classify these products as good or bad.
All .csv's have been clubbed together in form of a 3d matrix of (#files, time_steps, #features).
I am passing the matrix in batch_size=128.

````below is the psuedo code.
model = Sequential()
model.add(LSTM(#cells(64), input_shape=(time_steps, #features)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimiser='adam',
metrics=['accuracy'])
model.fit(train_X, train_y, batch_size=128, epochs=100, validation_split=0.1)

The problem is that I am getting NaN's in loss function and accuracy too is 0.000.

Can anyone suggest me where to look. Thanks in advance.


texts: Difference between multinomia logistic regression and N 1 vs all binary

texts:
Is there a fundamental difference between building a set on N logistic regressions in 1 vs all fashion, as compared to training a single mutlinomial logistic regression? Put another way, are there any optimization techniques that treat the 1 to N classes logistic regression problem in a way that's markedly different from N independent regressions?

Intuitively is seems like the answer ought to be yes, since there should be a lot of information sharing between various problems if two classes are similar. But since I'm not entirely well versed on how  common 1 to N solvers actually work, I can't tell if I'm right or these problems are treated in ways that fundamentally the same.

I think that we can see that there could be a difference between the two models, but I'm not entirely sure. Googling the matter revealed a few arcane discussing about the subject, but I was unable to find an authoritative discussion of the matter.


texts: Tensorflow + Keras: constructing the composition of two networks

texts:
I have loaded two networks using tensorflow and keras:

import tensorflow as tf
from keras.layers import Input
from keras.models import Model
net1 = tf.keras.models.load_model('net1.h5')
net2 = tf.keras.models.load_model('net2.h5')

I would like to construct the sequential composition of the two nets: the input goes through net1 (inputs of net1 have dimension 10), which outputs x, which is then fed to net2.
I wrote the following:

combined_netInput = Input(shape=(10,))
x = net1(combined_netInput)
combined_netOutput = net2(x)
combined_net = Model(inputs=combined_netInput, outputs=combined_netOutput)
combined_net.compile(loss='binary_crossentropy', optimizer = Adam(lr=0.0002, beta_1=0.5))

The first line is rejected with the following output:

Traceback (most recent call last):
  File "adversarial_examples_pgd.py", line 28, in <module>
    x = net1(combined_netInput)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/sequential.py", line 247, in call
    return super(Sequential, self).call(inputs, training=training, mask=mask)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 751, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 893, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 663, in __call__
    inputs, outputs, args, kwargs)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1708, in _set_connectivity_metadata_
    input_tensors=inputs, output_tensors=outputs, arguments=kwargs)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1795, in _add_inbound_node
    input_tensors)
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 515, in map_structure
    structure[0], [func(*x) for x in entries],
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 515, in <listcomp>
    structure[0], [func(*x) for x in entries],
  File "/home/nf/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1794, in <lambda>
    inbound_layers = nest.map_structure(lambda t: t._keras_history.layer,
  AttributeError: 'tuple' object has no attribute 'layer'

Many thanks in advance for your help!


texts: RMSE in NGBoost example appears to be unreasonably high

texts:
I tried using Standford's ML Group NGBoost.
I tried using this example provided by the authors of the repository.
While training the output is along the lines of :
BestIter=2000 RMSE: Val=15.9316 Test=15.0343 NLL: Test=3.8344

Looking at the dataset used (concrete), I see that a good RMSE for the dataset is given as 4.93 ± 0.14, making the NGBoost result of look really bad.
Am I doing something wrong?
It does not seem very likely that as an example, the authors would give an experiment yielding an error that big compared to the performance reported on the dataset.
Or am I missing something?


texts: Notion of cluster centers and cluster comparison in Density Based Algorithms

texts:
I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main feautures can have quite varying values. Therefore, my focus has been on density based algorithms with quite some success.

However, I am unable to grasp the idea of cluster comparison in such algorithms since the notion of cluster centers cannot be properly defined.

My dataset constists of network flows and I split the dataset in subsets based on an identifier. After applying clustering on each subset I want to be able to compare the clusters that are created on each subset so that I can compare the subsets themselves in some context.

Would appreciate some help from data scientist gurus on how to approach the concept of cluster comparison  or cluster center in such algorithms.

Thanks all!


texts: Are there any multi-type public datasets out there?

texts:
Does anyone know of any good public datasets for supervised learning that would contain data where instances are described with different types of data (ie. timeseries data, categorical data, images, ...)

The types of datasets I have in mind are:

a dataset where each example is described with an image and some numerical features
a dataset where each example can be described with timeseries data and some categorical features
a dataset where each example is described with multiple images (ie. the image taken from two different angles)

An example would be a dataset of daily closing stock price data associated with news sentiment for that particular day.

I'm aware that it's possible to construct a dataset like that on my own with some scraping but I'm wondering whether there are any public datasets out there that serve this purpose.


texts: Say I have have an experiment where I release a single rat into a maze and wait for it to reach the end. Say I also track this rat's position in the maze at various times. Let's do this $n$ times. Now, I have a dataset of $n$ variable-length movement profiles, since some of the animals finish faster, some take longer, etc. Is there a way I can analyze this data to then predict the next move $(x_{t+1}, y_{t+1})$ given an $(x_t, y_t)$? What techniques could I use and how would they work? I was thinking LSTM or RNNs, but unsure what techniques are used for problems like this.


texts: Why might an LSTM be capable of predicting an ARMA signal but not a linear combination of ARMA signals?

texts:
I have an LSTM network and am testing it on some dummy ARMA signals. I'm trying to predict the signal 5 time steps into the future.

The network is capable of outperforming Naive (persistence) when predicting the signal. My next toy example was to make a multi-variate prediction where I take several ARMA signals (ARMA_1 and ARMA_2) and the target variable is a linear combination of these signals (0.5 * ARMA_1 + 0.5 * ARMA_2) 5 time steps into the future. The training dataset is just the original ARMA signals and the target is just the linear combination with the offset.

However, while the training loss improves, the validation loss gets worse pretty much as soon as the predictions deviate away from flat. So the model is overfitting.

The network is already pretty simple (8 hidden nodes in the LSTM and a couple of hidden layers with 16 nodes in them). The networks won't over-fitting in the univariate case so I don't think it should be an issue overfitting in the multivariate case. Any simpler a network and it isn't capable of modelling the univariate case, so I don't want to simplify it further.

There is more than enough samples in the dataset that it should be able to learn the signals. I've done runs with dropout, regularisation and different learning rates to no avail.

I was wondering if anyone had any advice for why the networks is overfitting here but not in the univariate case, even with the same network parameters?


texts:
A tutorial that I am following initialized an ANN with tensorflow using the line

ann = tf.keras.models.Sequential()

Whereas, while going through the docs, I saw

ann = tf.keras.Sequential()

I've tried them both and both seem to work the same. What is the difference?

TIA!


texts: Train a multi-output neural network to learn subset of "valid" response combinations

texts:
I'm working on extending a model of human immediate serial recall task performance, originally described in this paper. This model takes a sequence of items, such as digits or phonemes, stores them as they are presented, and then repeats the sequence back in order. Their model uses one-hot encoding and decoding to describe items to be remembered and recalled (such as digits or phonemes) as atomic units in memory. I'd like to adapt this model to describe items as collections of features instead. For example, rather than encoding the phoneme /b/ as a unique element in a one-hot vector, I'd like to model it as a combination of phoneme features (e.g. voiced = true, manner = stop, place = labial), with each input and output corresponding to one feature.  Similarly, rather than modeling the digit "one" as a unique element in a one-hot vector, I'd like to model it as an input sequence of the phonemes (e.g. [w,^,n]).

This formulation of the problem naturally leads to a set of possible model outputs that is larger than the set of items that occur in the training set. However, the models I've trained so far do not appear to have learned this structure in the data set. When the model produces an incorrect phoneme, it produces phoneme features combinations that do not occur in the training set at about a chance rate.  Similarly, the model does not seem to learn the transition probabilities between phonemes within a word, and instead its incorrect responses seem unrelated across sequential phonemes.  Restated, the model doesn't seem to learn to exploit the set of phonemes or words that are in the training data, and instead seem to "brute-force" the recall task by remembering every input as a sequence of phoneme features.

Therefore, my question is if there are specific model structures or optimizers designed to exploit this cross-output and cross-time information to learn that some output patterns are "valid" (occur within the training set) or "invalid" (never occur within the training set)?  I don't know if this is a problem that has been previously addressed, or whether I'm using the correct terminology, so any guidance would be appreciated.

To date, I've tried a vanilla recurrent layer with a set of dense one-hot output layers to represent the output features.  I've tried adding a softmax-activation dense layer before and after the recurrent layer, but this seemed to impair the model ability to store sequences. For optimizers, I have tried SGD, Adagrad, and Adam optimizers, with Adam generally giving the best results.  Replacing the vanilla recurrent layer with LSTM or GRU units has improved the model ability to recall item sequences, but doesn't address the "invalid" response rate. I've been using Keras to implement the model.


texts: Use all available data to build Logistic Regression model

texts:
Using K-Fold, I chose to use Logistic Regression for a project of mine.

I made it learn on my X_train (80% of data), and tested it on my X_test, with good results.

My question is : now that I need the model to be used, can I rebuilt the model with the same parameters but on all my data ? Will there be overfitting, or maybe as I saw that those parameters doesn't, will it be safe ?


texts: Keras Predictions as a list of file names for each class

texts:
I've created a model using Keras, I have trained it with a training and validation set, and have used a test set filled with random number of images for each class. My test set consists of one folder with 300 images, each of the filenames for each image has the class in the name(e.g dog1.tif, dog2.tif, cat1.tif, horse5.tif). I'd like to double check which images were predicted to each class by looking checking the filenames. How can I get a list of filenames for each of the predicted class?
I first load my saved model
new_model = tf.keras.models.load_model('model')

Define my test set
test_batches = train_datagen.flow_from_directory('test_images',target_size=(224,224),batch_size=10,classes = None,class_mode= None)

Make a prediction
predictions = new_model.predict(test_batches, steps=30, verbose=0)

Then, count find the number of predictions for each class (0 is dog, 1 is cat, 2 is horse)
import collections

collections.Counter(np.argmax(predictions, axis = 1)
Again, Ideally I would like to get a list of files names for each class
or if possible,
Sort the predicted images into a a folder corresponding to the class
Question is cross posted on stack overflow