texts:
I have a data set that's a dictionary of tuples. Each key represents an ID number and each tuple is (yesvotes, totalvotes). Example:

{17: (6, 10), 18: (1, 1), 21: (0, 2), 26: (1, 1), 27: (3, 4), 13: (2, 2)}

I need to find the max key of the set. I want to assign weights so, for instance, key 17 would be ranked higher than key 18 because even though the ratio is much smaller, it has ten times the total votes.

Is there an optimal way to do this? My best guess is simply calculate new ratios by (yesvotes/totalvotes)*(totalvotes+1) but that doesn't seem right... Is there some kind of standardized field of study concerning fair-voting?


texts:
How can one prove that the optimal kPCA solution $a^*=\{a_1...a_K\}$ are the $k$-largest Eigenvectors of the (centered) kernel matrix $K$?

I referred to a lot of resources and couldn't find a proper explanation.


texts: Validation score during training and checkpoint is different in keras

texts:
I have a tabular data with about 1500 columns where every column except the 1st column is sparse. I am trying to train a Feedforward neural network (1 hidden layer with 32 neurons) for a binary classification problem (20% positive classes) using keras. I use SGD with learning rate 0.1 and momentum 0.9. I am using ROC AUC as the metric and I obtain the curves below:

I use a train/valid/test stratified split. When I evaluate the scores with the saved checkpoints I observe the following values:

Why are the values in the keras model history different from the scores from checkpoints?

How do I find where my model starts overfitting from the curves?


texts: how to identify the unknown class in machine learning?

texts:
In the problem of multi-label classification how to identify the unknown class which is not in training labels or classes. In the prediction phase, the classifier puts the data in any of the class even though it's not related to any of the classes or labels?
For binary, the prediction probability of classification is 50/50 so how to interpret these as the confusion or just anomaly but when the data appears in both of the class or the similar to both of the class we can understand this scenario but what about the data which should be in any of the classes.
So, how to handle the same situation for multi-class or a general solution which can provide the proper classification for unknown and non-relevant data.


texts:
I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.

More precisely, I would like this method to be able to leverage noisy data as well, for instance by not fully "trusting" noisy data, or weighting samples, or deciding whether to use a specific sample at all for learning. But primarily, I am looking for inspiration.

Details:

My task is sequence-to-sequence NLP,
I have both clean pairs of sequences of (clean input, clean output) and noisy ones (noisy_input, noisy_output),
I know for certain which samples in my data are noisy, and if possible, I would like the desired method to make use of this information

I am very glad to give more information about my use case if needed.

Edit: Noisy vs. negative examples

  First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples.

My view is that the data I have are noisy examples, but not "negative". Using an example from machine translation from German to English:

clean (equivalent meaning)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are living in a parallel universe.

noisy (meaning overlap)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are weird.

negative (unrelated)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Is Math related to science?


texts: Endogenous and exogenous ordinal variables in R studio Lavaan

texts:
I am new to using R studio, so apologies for the basic question. I have run a number of Confirmatory Factor Analyses using the Lavaan package. Each questionnaire item is on a 4 point Likert scale 0 = never, 1 = sometimes, 2 = often, 3 = always. I have treated items as exogenous ordinal variables. In the Lavaan description it says " If you have an exogenous ordinal variable, you can use a coding scheme reflecting the order (say, 1,2,3,. . . ) and treat it as any other (numeric) covariate."
However, I am now wondering whether it is more accurate to describe questionnaire items as endogenous ordinal variables using the 'ordered' function, or whether my model below is okay. If someone could also explain why this approach is/isn't optimal that would also be helpful.
Here is my model (one general factor and 3 specific sub factors):
library(lavaan)
Model <- " g =~ A_15 + A_19 + A_7 + A_13 + A_14 + A_29
                + A_20 + A_26 + A_30
               f1  =~ A_15 + A_19 + A_7
               f2  =~ A_20 + A_26 + A_30
               f3  =~ A_13 + A_14 + A_29"
fit <- cfa(Model, Data, estimator = "WLSMV", orthogonal = TRUE)
summary(fit, std=TRUE, fit.measures = TRUE)


texts: How to grid search class.weights hyper parameter in Ranger?

texts:
I am currently using ranger for binary classification. My dataset is highly imbalanced (10:1). I went over the documentation, and it appeared to me that class.weights could be a good hyperparameter to tune. But I cannot find a proper documentation/examples on how to carry out grid search for this hyperparameter. Can anyone help me with this?

My code looks like the following right now:

# set up the grid for param searching
grid_search <-  expand.grid(
  .mtry =  seq(5, 30, by = 10),
  .min.node.size = seq(10, 40, by = 15),
  .sample.fraction = seq(0.5, 0.9, by = 0.2),
  .num.trees = seq(400, 1000, by = 200),
  .class.weights = c(0.5, 3), #How to have different weights?
  OOB_error = 0
)

for(i in 1:nrow(grid_search)) {
  rf_model <- ranger(
    formula         = as.factor(trainY$sale) ~ .,
data            = trainX,
num.trees       = grid_search$.num.trees[i],
    mtry            = grid_search$.mtry[i],
min.node.size   = grid_search$.min.node.size[i],
    sample.fraction = grid_search$.sample.fraction[i],
probability     = TRUE,
class.weights   = c(0.5, 3),
seed            = 123
  )
  # add OOB error to grid
  grid_search$OOB_error[i] <- sqrt(rf_model$prediction.error)
  print(grid_search[i,])
}


texts:
I need to explain the word embedding layer of Keras in my paper, mathematically. I know that keras initialize the embedding vectors randomly and then update the parameters using the optimizer specified by the programmer. I want to explain my architecture in an academic paper, therefore I need to explain each layer with a formal formula.
Is there a paper that explains the method in details to reference it? or can you please guide me to how to compose the method formally?

Thanks a lot


texts: coefficients from decision regression tree

texts:
I have a data on which I apply decision regression tree since it gives better numerical values as predicting house prices. I have almost 45 features in which price is the target value. With linear regression, I get bad guesses while decision regression tree(right one) yields far better guesses as shown in the picture.

So, what I don't understand is how it is helpful. I mean in linear regression I can get a formula after training data as y = intercept+0.343543+0.23423+0.3242+..... But I wonder if I get same/similar representation using decision regression tree? If not, how can it be useful?

I cannot use it simply, that is, I cannot write a function taking 45 parameters. In this function, I can predict the price easily.

I'm confused.


texts: Why would a cross-entropy approaching negative infinity?

texts:
I'm learning Deep Learning by Ian Goodfellow. In 6.2.1.1 it says

For real-valued output variables, if the model can control the density of the output distribution (for example, by learning the variance parameter of a Gaussian output distribution) then it becomes possible to assign extremely high density to the correct training set outputs, resulting in
cross-entropy approaching negative infinity.

But the cross-entropy is defined as
$$-\mathbb{E}_{x\sim P_{data}}\log P_{model}(y|x)$$
or in the empirical form
$$-\frac{1}{m}\sum_{i=1}^m\log P_{model}(y|x)$$
As we can see it is bounded below by $-\log 1 = 0$. Then why the book says that it would approach negative infinity?
Any thought would help.


texts: Is data partitioning necessary for an explanatory model and why?

texts:
I've come accross the following paragraph in the To Explain or To Predict? paper by Galit Shmueli.

  In explanatory modeling, data partitioning is less common [than in predictive modeling] because of
  the reduction in statistical power. When used, it is usually done for
  the retrospective purpose of assessing the robustness of ˆf. A rarer
  yet important use of data partitioning in explanatory modeling is for
  strengthening model validity, by demonstrating some predictive power.
  Although one would not expect an explanatory model to be optimal in
  terms of predictive power, it should show some degree of accuracy.

I understand why data partitioning is useful in the case of a predictive model, which is to assess the generalization capacity of a model. However, in the case of an explanatory model, I don't understand why it should show some degree of accuracy in terms of predictive power since it's not the objective of the model.

Here comes my question: is data partitioning necessary for an explanatory model and why?


texts: Machine learning technique to calculate weighted average weights?

texts:
I'm just starting to investigate machine learning concepts, so I'm sorry if this question is very naive, but I'm hoping that it will be an easy one to answer!

I have a document matching algorithm that individually calculates a match for each field (0-1, with 0 = no match, 1 = 100% match), and applies a separate weight to each field match to be used in calculating an overall weighted average relevance "score".

E.g., given a document of 3 fields (d1-3) and an input query against each of the fields (q1-3), field matches are calculated for each pair (m1-3) and then weights (w1-3) are applied using a weighted average for a final relevance score: s = sum(mi x wi)/sum(wi).

For this contrived example, perhaps we can simply say that a document is considered relevant if the score is above 0.5. I.e., there is either a "relevant" (0.5-1.0) or "not relevant" (0-0.5) outcome. But I don't want every field to have equal weight in determining the outcome.

So, my question is simply: What type of machine learning technique is "best" used to calculate the appropriate weights (w1-n), based on past, known results? Is this even an appropriate use of machine learning?

And secondly, if instead of a simple outcome of relevant and non-relevant, I actually want to rank the documents by relevancy, can this also be achieved using a machine learning technique?


texts: Developing an Empirical Model for Non-Linear Data

texts:
I have collected Temp vs. Time data for a controlled environment. I performed three tests, with the thermocouple in the same location for all three tests. I now have three Temp. vs. Time curves for this location. The graphs are shown here:

I am looking to generate an empirical model based on the Temp. vs. Time at this location that can predict the temperature at this location over time. How should I approach this? My initial approach was to identify the local minima and maxima, and average the three points together (for each maximum and minimum). And then correlate that average to a sequence of time, for example, 30C between 750 and 1250s, or something like that. I really don't think this is an efficient approach...

What do you all think would be an efficient approach to this?


texts: LSTM to Predict Pattern 010101… Understanding Hidden State

texts:
I did a quick experiment to see if I could understand what the hidden state in an LSTM does...
I tried to make an LSTM predict a sequence of [1,0,1,0,1...] based off an input sequence of X with X[0] = 1 and the remainder as random noise.
X = [1, randFloat, randFloat, randFloat...]
label = [1, 0, 1, 0...]

In my head, the model would understand:

The inputs X mean nothing, or at least very little (as it's noise) - so it'd discard these values for the most part
Solely the hidden state from the previous sequence/timestep n would be used to predict the next timestep n+1... [1, 0, 1, 0...]
I also set X[0] = 1 so the first initial in an attempt to guide the net to predicting 1 on the first item (which it does)

So, this didn't work. In theory, should it not? Can you someone explain?

## Code
import os
import numpy as np
import torch

from torchvision import transforms
from torch import nn
from sklearn import preprocessing
from util import create_sequences
import torch.optim as optim

Create some fake data
sequence_1 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_1[0] = 1
sequence_2 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_2[0] = 1

labels_1 = np.zeros(50)
labels_1[::2] = 1
labels_1 = torch.tensor(labels_1, dtype=torch.long)
labels_2 = labels_1.clone()

training_data = [sequence_1, sequence_2]
label_data = [labels_1, labels_2]

Create simple LSTM Model
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(LSTM, self).__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, seq):
        lstm_out, _ = self.lstm(seq.view(len(seq), 1, -1))
        out = self.fc(lstm_out.view(len(seq), -1))
        out = F.log_softmax(out, dim=1)
        return out

We try to overfit on the dataset
INPUT_DIM = 1
HIDDEN_DIM = 6
model = LSTM(INPUT_DIM, HIDDEN_DIM, 2)

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for epoch in range(500):
    for i, seq in enumerate(training_data):
        labels = label_data[i]
        model.zero_grad()
        scores = model(seq)
        loss = loss_function(scores, labels)
        loss.backward()
        print(loss)

        optimizer.step()

with torch.no_grad():
    seq_d = training_data[0]
    tag_scores = model(seq_d)
    for score in tag_scores:
        print(np.argmax(score))


texts:
Suppose we have the dataset:

{(3,1),(3-1),(6,1),(6,-1)} {(1,0),(0,1),(0,-1),(-1,0)}
the first set represent the positive label, and de second the negative.

I want manually find the support vectors, solve the equations and determine the coefficients W and bias
w0 of the separating hyperplane.

I know that the SVs are {s1=(1,0),s2=(3,1),s3=(3-1)}

after i wrote a system of equations:

α1k(s1,s1) + α2k(s2,s1) + α3(s3,s1) = -1

α1k(s1,s2) + α2k(s2,s2) + α3(s3,s2) = -1

α1k(s1,s3) + α2k(s2,s3) + α3(s3,s3) = -1

Where k(x,y) is the dot product xy

the resolution of this system is impossible, can someone tell me where I'm wrong?


texts: Does transformer learn from context to context

texts:
We are fine-tuning a transformer-based question answering model to answer the questions about the novels with the chronological plot. We are breaking up the novel into chapters and then using them as contexts with questions and answers, following the SQuAD 2.0 model
Our model is a customized HuggingFace DistilBERT.
My question - is such QA model capable of learning across contexts, so that it associates, for example, relationships between protagonists and places that do not necessarily all appear in the same contexts?


texts: How to structure my data into features and targets for PCA on Big Data?

texts:
I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the targets I got a little bit lost. On the Iris dataset is easy and I understood, but now I am facing a real dataset. My problem is that part of the original data was meant to be into an array and as the database could not hold it then the data was split into columns. However, some columns are referring to the same thing. Please consider the example below.

To be more specific, I designed an example dataframe which is almost the same as mine.

Explained dataset: At a respective date, at a specific place, we had the same four kids being measured. Robin, Ted, Lilly and Jamie are their names. Each number next to their names refers to their height, weight, and other specific measurements. The database could not keep the array for each child with their measurements (height, weight, etc) so the data from the array was split into columns (Robin_1, Robin_2, Robin_3, etc.. for each child).

Specific problem: I do not know how to structure my features/targets. I selected the Date (which is also the index), Country, City and School as being the features. On them I apply the principal components (3 in number -> 3D plot). How should I select the target ? Is there any way on somehow grouping them before clustering? In the final result, I want to cluster each the full Robin, full Ted, full Lilly and so on. I have exactly four kids, but their attributes differ (each one can have different number of attributes (ex. Robin -2, Ted- 10, Lilly-7, etc)) --> four clusters
P.S.: I cannot cluster them by the height, weight, etc. Those are only example attributes.

A visualization image plot of the final wanted result:

Any help would be of use and I would highly appreciate any effort in helping me with this problem. I have been reading a lot but I have not met this problem before. If there is another way to structure/consider them please let me know. Please let me know if I have to be more explicit in a respective part if I did not explained/tell something important.


texts: Ratio between embedded vector dimensions and vocabulary size

texts: Using Embedding layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension vector.  I think that the embedded vector need to have a minimum length to be able to map a given vocabulary.


texts: K-means clustering to separate temperature vertical profiles

texts:
I have temperature measurements from weather stations in a mountainous region and I want to obtain a vertical profile from these data at any given time.
In a simple case one can just plot all values in a temperature vs height chart and apply a linear/polynomial/fancy regression to obtain a curve and therefore an estimate of the lapse rate dT/dz.
In some cases, though, the weather changes abruptly and temperature will differ widely between, say, the eastern and western parts of the region. In such case the cloud of values in the above-mentioned plot will be quite messy and a regression won't be of help - R^2 will be quite low.
I was thinking of using a clustering algorithm in these cases, to split values into two groups (e.g. west and east), each having its own vertical profile, which would then (ideally) be suitable for applying regression. The algorithm could use information about each value's geographical position (lat-lon), which is hidden in the temp vs height plot, to figure out to which group each value belongs.
Can I achieve something like this using k-means? Any suggestion is welcome.


texts: LSTM architecture for multivariate time series

texts:
For a multivariate time series analysis, which of the following LSTM architectures would work better and why?

1) Having two independent LSTM layers (one for the time series variable and one for the features) and later concatenating these independent layers using Keras functional api.

2) Having both the time series variable and the features in one multidimensional array which is used in LSTM.


texts:
I am trying to text mine the reviews of the games and want to find the interesting things from the reviews
example of the data is

'review': ['Simple yet with great replayability. In my opinion does "zombie" hordes and team work better than left 4 dead plus has a global leveling system. Alot of down to earth "zombie" splattering fun for the whole family. Amazed this sort of FPS is so rare.',
'Amazing, Non-stop action of blowing stuff to bits, Decapitation and shooting everything you see. With a combination of action, thriller and emmersive gameplay, as well as enviromental challanges (Jump physics). This game will really put your eyes to the test, can you see the enemys before they see you? Cause their are so many!This is the second level of the killing floor, I quote bill LF4D "Son we just crossed the street" But in reality they only moved up an elevator level on genes. What has yet to come as the game is slowly realsed with thrilling and horryfing creations, But who really cares, Let\'s just blow it up, I invite you to get on the GODAMN KILLING FLOOR, LET'S SHOOT  AND GET PAID!',......
]

I am interested to find the name of games and types of games(strategy, shooting, etc) from the reviews data.

I tried and created the regular expression such that getting the text which has data ( is a game )
so that I will get the reviews

Dying Light is a game.
Magicka is a game

I want a suggestion to improve the results for the above
and my interst is to text mine and find the game type such as a puzzle, shooting,etc


texts: comparing data from first quarter in two years

texts: How can I compare data from two quarters? What software can I use to do it parallelly side by side so I can compare them. What kind of factors can I take into consideration, I'm posting a sample data set pic


texts: Python plot for confusion matrix similar to confusion wheel?

texts:
I have a confusion matrix with 7 classes and would like to represent the matrix in a graph. Something similar like a confusion wheel.

Mainly I need to show, the correct observations in each class and incorrect ones with other classes in a pie chart.

Any python implementations would be great.

http://www.cvast.tuwien.ac.at/ConfusionAnalysis/


texts: Some question about output shape of joint coordinate in cnn-regression

texts:
I have some question about output shape of cnn-regression problem.
Let say I have Image shape with

image shape = (105, 157, 3)

and this image has 14 joint coordinates for each image (14, 3, 10000).

I guess 10000 is the number of pictures

and each joint coordinates looks like

y=array([[145.82985678,  95.5022898 ,   1.        ],
         [111.87785389,  83.38437237,   1.        ],
         [  0.        , -27.29260682,   0.        ],
         [146.22287032,  67.62016267,   1.        ],
         [114.716285  ,  89.0394005 ,   1.        ],
         [147.03073148,  90.65512282,   1.        ],
         [  0.        , -27.29260682,   0.        ],
         [  0.        , -27.29260682,   0.        ],
         [  0.        , -27.29260682,   0.        ],
         [102.99138111,  63.60269094,   1.        ],
         [107.44553454,  91.85599752,   1.        ],
         [112.68571505, 118.12240233,   1.        ],
         [ 95.72063065,  64.38871802,   1.        ],
         [ 76.74681039,  60.37124629,   1.        ]])

If I want to make a neural network then the neural network should have output and output should compare with this y true value

What is the output should shape with?
Which shape do I have to reshape this y true value? and how?

Dataset is from Leeds Sports Dataset (LSP): 11000 training and 1000 testing images from sports activities with challenging in terms of appearance and especially articulations. The majority of people have 150 pixel height. For each person the full body is labeled with total 14 joints.


texts: Can one use non converged results from Logistic Regression?

texts:
I'm running Logistic Regression on a dataset for a classification problem.

I used the model on the dataset when it was normalized and I had no problem with it converging.

Now, I wanted to see the results without feature normalization and I am getting the warning:

ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)

I've tried increasing the number of iterations and I have upped it to 8000 so far and am still getting the error.

I wanted to ask, is this error is critical? Because, I'm still getting my cross validated results even after the error. What exactly is convergence?


texts: How do data types influence hardware (CPU / GPU / TPU) performance?

texts:
I am currently dealing with a relatively big data set, for which I have some memory usage concerns. I am dealing with most of the different data types : floats, integers, Booleans, characters strings for which some were changed to factors.
From a memory usage (RAM) standpoint it is pretty clear what is happening when I switch column type from float64 to float32: the memory usage is divided by two. For string conversion to factor, it is a bit less clear but I get the general idea that regarding to memory theiy are delt with as small integer and remapped if necessary.  It doesn't really matter, as they generally delt with in pre-processing by being encoded as numerical columns some way or another.
Now what I am wondering is how do that data types influence calculations further down the pipeline (pre-processing, calibration). I somehow got the idea that changing data types from float 64 to float 32 would allow for twice faster calculations on CPU, but this seems to be varying wildly. So, I am wondering, for what that matters (pre-processing, calibration), how does data types impact the hardware performance ?
Some clarifications / simplyfiying assumptions :

I have the intuition that some weirds things may happen in the CPU if you mix data types or regarding convergence if you go too low on memory usage (using float16). So let's assume we are just dealing with float64 to float32 conversion.

I am trying to get a langage / library / model agnostic answer. I am mostly using Python + pandas if that matter (to be honest I've never really had to handle data types in R), Python + cudf for GPU. If that matter I am using tensorflow + keras to build a NN. Notably I've called tf.keras.backend.set_floatx('float32'). Similarly I've carrefully made sure that the pre-processing pipeline is handling float32 data types correctly and that code is vectorized.

As I am getting involved in Kaggle competitions with differents accelerator (GPU / TPU), I'd like to get an answer that cover the main kind of hardware (CPU / GPU / TPU). I hope that the answer don't really depends on brands.


texts:
I have an image which is generated from noise using deep learning. When I zoom in on the image it looks like the image provided below. As you can see image is not sharp enough.
I want to sharpen the image such that each pixel has value 255 or 0 (or integers closer to these) instead of the pixel we are seeing on the image provided. Is there any easier way to achieve this or filter I can use in python?


texts:
Is BERT a language model in the sense of a function that gets a sentence and returns a probability?
I know its main usage is sentence embedding, but can it also provide this functionality?


texts: DBSCAN: How does a quantile of kNN relate to the share core points?

texts:
I read this answer by Anony-Mousse to an other question related to density based clustering and how to potentially come up with an eps. It states, that if you want 90% of you points to be core points, you can use the 80% quantile of the 4-NN distances as your eps.

I just don't understand how the 90% core points exactly relate to the 80%-quantile of the 4-NN distances. Of course I get that they are related somehow, but I dont get the exact chain.

I've been thinking (and reading) about this for a while now and just can't come up with the answer, although it's probably pretty obvious?


texts:
I am trying to use NasNetlarge in Keras without the top but I cant get rid of the top:
from keras.applications import NASNetLarge
model = NASNetLarge(input_shape=(224, 224, 3), include_top=False,
                    weights="imagenet")

If I use any shape other than the (331, 331, 3) shape, I get the error that I must use (331, 331, 3) if I include_top=True even though I set include_top=False. If I set the input shape to (331, 331, 3) and include_top to False, it downloads the weights with the top. Am I doing something wrong or is there a bug in the Keras module?


texts:
I would like to build an autoencoder (CNN) to learn a representation of my data.

I never built such a network and I have some experience in supervised learning (classification).

I would like to know if some good practices in training a classifier is also right for an autoencoder:

Does reference architecture exists like ResNet/Inception or something? If not, should I design manually layers?
Does transfer learning/fine tuning works for autoencoder (or is it better to train from scratch)?


texts: Data augmentation when using flow_from_directory in CNN

texts:
I would like to use a small dataset to create CNN model. So, I am using data augmentation to increase the train dataset. Should I use all augmentation techniques (arguments) that listed here?
I have noticed that adding many arguments decrease the accuracy of the model and make the training set harder than the testing set.
What is the best practices to use data augmentation when use flow_from_directory?


texts: Multivariate linear regression in Python

texts:
I'm looking for a Python package that implements multivariate linear regression.

(Terminological note: multivariate regression deals with the case where there are more than one dependent variables while multiple regression deals with the case where there is one dependent variable but more than one independent variables.)


texts: Pytorch DNN feature importance / mean decrese accuracy

texts:
I have been working on a classification DNN for metabolite data. I would like to implement a feature importance calculation to the DNN, and I saw a paper that used mean decrease accuracy for this. When trying to research this, I mainly saw this done on random forest. Is there a fairly easy way to do this with DNN?
I would like to see a simple example of this done with a DNN in Python.


texts: Varying the number of neighbors in nearest-neighbors regression

texts:
For nearest-neighbors regression, it is plausible to increase the number of neighbors used to predict $f(x)$ when there are many data points near $x$.

What is a good algorithm for varying the number of neighbors used?


texts: Differential Evolution optimal tolerance parameter

texts:
I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that it can find the minimum of a function in a reasonable time.

The parameter optimization takes a very long time, and it only has to be done once after which I will just use the best combination and just run those on the other similar datasets, hopefully finding the best value there too.

The global optimizator that I use is called differential evolution and I use the python/numpy/scipy package implementation of it. Here is the wikipedia definition and the relevant papers in the references.

The problem is that it's extremely slow to sample enough combinations of the parameters to find any kind of trend which would suggest me and kind of pattern that I should follow. My problem is especially with the tol parameter which is defined as:

  np.std(pop) <= atol + tol * np.abs(np.mean(population_energies))

Which exits the optimizer if the standard deviation of the population goes below the average of the population energy.

The tricky part is that if the tol=0.01 as per default then the optimizer exits quickly and gives you a low accuracy value, so I run it multiple times, if I set it to a very low number then it doesn't exit at the above criteria but it exits at the maxiter value the maximum number of iterations the function is allowed to do. So either I run it with a big tolerance many times or with a small tolerance few times, the question is which exit clause is better?

What I am measuring is which tol value can give me the best value within a given amount of time, currently 2 hour per set, and I group them as {1e-01,1e-02,...,1e-10} batches, testing which batch will give me the best value in the given time.

I am testing it now as we speak however the low tolerance values are very hard to sample because it takes ~30min for one to be done, so I would need a sample size of 200 from each which would take just too much to compute. My data is 15 dimensional and I only allow 90000 evaluations in total per run (15*popsize*maxiter), which with a low tolerance rate like 1e-10 gets hit very often, which makes drawing samples very slow.

Is there any other way to find the best tol parameter, I would guess a higher one is better because exiting with a condition is more efficient than just brute forcing through an arbitrary set of cycles, but only if the shape of my data is favorable enough, and I have no way of knowing.

Is there any literature on what the rule of thumb best parameters are for the differential evolution global optimizer, that on average is optimal for all functions?

I suspect the optimal range to be around [1e-4,1e-0] but I have no way of proving it, but if it's true then optimizing the other parameters would be a piece of cake as calculations within this tolerance range are very quick.


texts: R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

texts:
See R's {drake}. It allows you to define a reproducible pipeline

plan <- drake_plan(
  raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
  data = raw_data %>%
    mutate(Species = forcats::fct_inorder(Species)),
  hist = create_plot(data),
  fit = lm(Sepal.Width ~ Petal.Width + Species, data),
  report = rmarkdown::render(
    knitr_in("report.Rmd"),
    output_file = file_out("report.html"),
    quiet = TRUE
  )
)

# call the pipeline
make(plan)

The great thing about drake is you that you can reload any of raw_data, data, hist, fit, report at any point. And if you change part of the code and make(plan) and {drake} will figure out which has change and just run that.


texts: Implementing RMSProp, but finding differences between reference versions

texts:
I am researching to implement RMSProp in a neural network project I am writing.

I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a Coursera class presented by Geoffrey Hinton (lecture 6 I think). I don't think the approach has ever been formally published, despite many gradient-descent optimisation libraries having an option called "RMSProp". In addition, my searches are showing up a few variations of the original idea, and it is not clear why they differ, or whether there is a clear reason to use one version over another.

The general idea behind RMSProp is to scale learning rates by a moving average of current gradient magnitude. On each update step, the existing squared gradients are averaged into a running average (which is "decayed" by a factor) and when the network weight params are updated, the updates are divided by the square roots of these averaged squared gradients. This seems to work by stochastically "feeling out" the second order derivatives of the cost function.

Naively, I would implement this as follows:

Params:

$\gamma$ geometric rate for averaging in [0,1]
$\iota$ numerical stability/smoothing term to prevent divide-by-zero, usually small e.g 1e-6
$\epsilon$ learning rate

Terms:

$W$ network weights
$\Delta$ gradients of weights i.e. $\frac{\partial E}{\partial W}$ for a specific mini-batch
$R$ RMSProp matrix of running average squared weights

Initialise:

$R \leftarrow 1$ (i.e. all matrix cells set to 1)

For each mini-batch:

$R \leftarrow (1-\gamma)R + \gamma \Delta^2$ (element-wise square, not matrix multiply)
$W = W - \epsilon \frac{\Delta}{\sqrt{R + \iota}}$ (all element-wise)

I have implemented and used a version similar to this before, but that time around I did something different. Instead of updating $R$ with a single $\Delta^2$ from the mini-batch (i.e. gradients summed across the mini-batch, then squared), I summed up each individual example gradient squared from the mini-batch. Reading up on this again, I'm guessing that's wrong. But it worked reasonably well, better than simple momentum. Probably not a good idea though, because of all those extra element-wise squares and sums needed, it will be less efficient if not required.

So now I am discovering further variations that seem to work. They  call themselves RMSProp, and none seems to come with much rationale beyond "this works". For example, the Python climin library seems to implement what I suggest above, but then suggests a further combination with momentum with the teaser "In some cases, adding a momentum term β is beneficial", with a partial explanation about adaptable step rates - I guess I'd need to get more involved in that library before fully understanding what they are. In another example the downhill library's RMSProp implementation combines two moving averages - one is the same as above, but then another, the average of gradients without squaring is also tracked (it is squared and taken away from the average of squared weights).

I'd really like to understand more about these alternative RMSProp versions. Where have they come from, where is the theory or intuition that suggests the alternative formulations, and why do these libraries use them? Is there any evidence of better performance?


texts: How to print feature names in conjunction with feature Importance using Imbalanced-learn library?

texts:
I used BalancedBaggingClassifier from imblearn library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction with feature names especially when the max_features are less than the total number of features? For example, in the following code total number of features equal to 20 but max_features are 8.

from collections import Counter
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from imblearn.ensemble import BalancedBaggingClassifier
from xgboost.sklearn import XGBClassifier

X, y = make_classification(n_classes=2, class_sep=2,weights=[0.1, 0.9], n_informative=3, n_redundant=1, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)
print('Original dataset shape {}'.format(Counter(y)))

ln = X.shape
names = ["x%s" % i for i in range(1, ln[1] + 1)]

X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
bbc = BalancedBaggingClassifier(base_estimator=XGBClassifier(n_estimators=10,min_child_weight=5,max_depth=3,learning_rate=0.02,colsample_bytree=1,subsample=1,scale_pos_weight=.26), ratio='all',random_state=0, max_features=8)
bbc.fit(X_train,y_train)
for estimator in bbc.estimators_:
    print(sorted(zip(map(lambda x: round(x, 4), estimator.steps[1][1].feature_importances_),names), reverse=True))

I think there is a problem with the above code because always printed features are named x1 to x8 while for example, feature x19 may be among the most important features.

Thanks.


texts: Clustering on imbalanced data that has high correlation

texts:
I am clustering images of two categories, but for the purposes of the experiment, I do not know the labels i.e. this is an unsupervised problem. Via correlation heatmaps and other experiments, I am confident that my images are highly correlated, at least via a Pearson correlation coefficient. However, I face very large imbalanced datasets in my problem, and as a result, my hierarchical agglomerative clustering fails when the imbalance passes a ratio of around $10:1$.

I have no idea of alternative solutions. I have tried reducing dimensionality via PCA, but this does not help.


texts: Confusion regarding classification accuracy calculation and result

texts:
The total number of data points for which the following result is obtained = 1500. Out of which, I have

1473 labelled as 0 and
the remaining 27 as 1 .

As can be seen from the confusion matrix, out of 27 data points belonging to class 1, I got only 1 data point misclassified as 0  . So, I calculated the accuracy for individual classes and got  Accuracy for class labelled as 0 = 98.2% and for the other as 1.7333%. Is this calculation correct? I am not sure...I did get a pretty good classification for the class labelled as 1 so why the accuracy for it is low?
The individual class accuracies should have been 100% for class0 and around 98% for class1

Does one misclassification reduce the accuracy of class 1 by so much amount? This is the how I calculated the individual class accuracies in MAtlab.

cmMatrix  =
1473    0
1       26

acc_class0  = 100*(cmMatrix(1,1))/1500;
acc_class1= 100*(cmMatrix(2,2))/1500;


texts: Available data sets on basic income pilot studies?

texts:
I'm interested in exploring data relating to any UBI/basic income pilot studies that have been conducted but have been unable to find any so far?
thanks


texts: How to estimate student's level according to score and time spend

texts:
I have some data that contains students answer, score, and time spent in many exams.

How to estimate each students level according to this data?

Sample data:

student ID, topic ID, score, time spend

  sid1    ,   tid1  ,  10  , 5s

  sid1    ,   tid2  ,  4   , 15s

  sid1    ,   tid3  ,  3   , 25s

  sid1    ,   tid4  ,  10  , 5s

  sid2    ,   tid1  ,  11  , 15s

  sid2    ,   tid2  ,  6   , 11s

  sid2    ,   tid3  ,  7   , 24s

  sid2    ,   tid4  ,  0   , 53s


texts: Ways to Make Feature Values Relative for Each Participant

texts:
I have a number of participants from a study and am trying to classify them into five "perceived effort" zones. However, I am finding that the accuracy is sitting at around 70% - 75% using various machine learning algorithms, which is OK but I think this could be more accurate by making the features relative for each participant.

Each zone corresponds to the participant's perceived effort:

Zone 1 < 20% effort
Zone 2 < 40% effort
Zone 3 < 60% effort
Zone 4 < 80% effort
Zone 5 maximum effort

I would like to run the same algorithms on data that has feature values relative for each participant, because one participant's zone 3 maybe be the equivalent to another's zone 4. In other words, the zones are not uniform.

One idea I had was to make the feature's relative to to each participants max of that feature. As an example:

Participant 1: Max running speed is 20 mph (running speed = Feature 1)

run    zone        actualSpeed      relativeSpeed
1       5             20            20 / 20 = 1.0
2       4             17            17 / 20 = 0.85
3       3             13            13 / 20 = 0.65
4       2             10            10 / 20 = 0.50
5       1             8             18 / 20 = 0.40

However, a few of my features have values that range from -3.6 to + 2.5, and I am stuck on how to make such features relative to the participant.

Question: How to make sample feature values relative to the groups/participants for supervised classification machine learning?


texts: Clustering analysis for observations with lists as data

texts:
So I have several samples analyzed for their chemical composition. After data analysis, for each sample, I have a list of compounds found and their corresponding relative abundance. Some compounds are unique but most are actually found in most samples.

I want to do clustering analysis based on these list of compounds. How do I go about this? Specifically how to vectorize my dataset since each observation is actually an array with both numerical (abundance) and categorical (compound label) variables.


texts: User-product positive (click data) available. How to generate negative (no-click data)?

texts:
Its very common in recommender that we have user product data which have label as an e.g. "click". In order to learn the model, I need click and no-click data.

Simplest approach to generate is to take user-products pairs which are not found in click data. However, that may be misleading. Example:

user1, product1 (click)
 user2, product2 (click)
 user2, product3 (click)
 user3, product2 (click)

I can take user1 with all products except product1 and label them as "no_click" and so on. But this may not be true. Maybe user1 would have clicked product2 if he was shown product2. But just because he was shown other sets of products - he had no opportunity to decide to click/no-click product2.

So how to address unary data problem?


texts: Tensorflow / Deepmind: how do I take actions from observations for math algorithms related to proofs?

texts:
Crossposted from here
This question is to ask for directions/suggestions/help on the use of deepmind opensource libraries: DeepMind Lab or TensorFlow in Python.
Consider that I'm new to concepts like deep learning and AI.
Questions are:

Are there examples on using Deepmind or Tensorflow for math problems where I need to observe values and take actions?

Using an approach similar to to the one described in this page based on observations, actions, rewards, etc., I'd like to call a learing agent to choose among some values. I was thinking to something like this:

Input: a list of list of tuple (the list will change at each step)
Action: pick up a value from the input (based on experience)
Reward: If the value it returned was good or bad for the rest of the algorithm I'm implementing, I'll reward the deep learning agent.

Additional notes:

I cannot train the algorith in advance

The Input is something like this:
edge: (1, 2), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (0, 1), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (5, 4), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (6, 7), face_down: 3, face_up: 5, face_left: 5, face_right: 5
edge: (3, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (4, 1), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (8, 5), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (3, 8), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (2, 3), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (5, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (0, 5), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (1, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (9, 6), face_down: 3, face_up: 5, face_left: 5, face_right: 5
edge: (0, 3), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (7, 9), face_down: 3, face_up: 5, face_left: 5, face_right: 5

The idea is to use the same aproach deepmind uses to play games, but instead of analyzing pixels and use the pad (up, dowm, left, right, fire, jump), is to let the learning agent to analyze some math values and, as the only action, to select one of them.
Are there other approaches or libraries/framework to address such a problem?


texts: How to define a MLP with multiple outputs

texts:
I need to build a neural network to predict predict outputs based on a set a input features, for a multi-channel system. Once I figure out this problem, I will implement it in Keras. Let me describe it in further detail:

I have a set of input variables ('temperature', 'current', 'voltage' ... )

the outputs are ('gainA', 'gainB', 'gainC', 'gainD')

However, each letter (A,B,C and D) represents a different channel, and the training data contains lots of different configurations of when the channel is ON/OFF. Therefore a typical training dataset would look like:

'ID' |  'Temperature'  | 'current' | 'voltage' | 'gainA' | 'gainB' | 'gainC' | 'gainD'
  1  |       23.1      |     2.1   |    5.1    |    0    |   1.5   |   3.1   |    1.2
  2  |       23.2      |     2.3   |    5.2    |    0    |   1.5   |   2.1   |    1.1
  3  |       23.4      |     2.0   |    5.0    |   1.3   |    0    |   1.7   |    1.0
  4  |       22.8      |     2.3   |    5.4    |   1.5   |   1.3   |   3.2   |     0
  5  |       22.9      |     2.1   |    5.1    |    0    |    0    |    0    |    1.2
  6  |       23.2      |     2.2   |    5.3    |   1.2   |   1.5   |   3.4   |    1.3
...

As you can see from the dataset above, there is '0' for some gains. This is when the channel is OFF. There is only a non-zero value of gain when the channel is ON.

My model needs to be able to predict the gain of a channel if I were to switch it on

For example if we consider row 5 as the test data, I want to predict the numerical value of 'gainC' if I was to activate that channel

'ID' |  'Temperature'  | 'current' | 'voltage' | 'gainA' | 'gainB' | 'gainC' | 'gainD'
  5  |       22.9      |     2.1   |    5.1    |    0    |    0    |[predict this]|1.2

Initially, I considered training 4 models (one for each channel) however that would not work because the combination of gains are correlated and thus a multi-dimensional MLP is required.

If I used the 3 other gains as part of the input, it would just predict the gain to be 0 because that's what the model sees it to be.

Please could someone help me figure out a way of predicting the non-zero value of a gain. Perhaps part of the answer could include the binary representation of the channels. E.g

for ID number 5, the active channels are 0001 and we want to predict 0011

Maybe this could be part of the training data? Or an additional input to the predictor?

I'd appreciate some advice on how to approach this problem. (I'm fairly new to neural networks)


texts: Is there a difference between Vectorization and Featurization?

texts: I have come across articles that mention Featurization and Vectorization interchangeably. Is there a minute difference between each other or are they exactly referring to the same thing?


texts:
I have a huge dataset from a relational database which I need to create a classification model for. Normally for this situation I would use Inductive Logic Programming (ILP), but due to special circumstances I can't do that.

The other way to tackle this would be just to try to aggregate the values when I have a foreign relation. However, I have thousands of important and distinct rows for some nominal attributes (e.g.: A patient with a relation to several distinct drug prescriptions). So, I just can't do that without creating a new attribute for each distinct row of that nominal attribute, and furthermore most of the new columns would have NULL values if I do that.

Is there any non-ILP algorithm that allows me to data mine relational databases without resorting to techniques like pivoting, which would create thousands of new columns?


texts: Do I have to use neural networks in a generative adversarial network?

texts:
I've only seen examples that use (various types of) neural networks for both the discriminative and generative model. Is it not sound to use say, a logistic regression model for the discriminative model instead? Do I have to use a neural net instead of say, a clustering algorithm?

Does the "generate novel samples using one technique, then discriminate between novel and original samples using a different technique, then reinforce anything learned that makes discrimination more difficult" general theory behind generative adversarial networks have to be used with neural networks? Or is that just because most uses of GANs today involve image data, which seems to be particularly conducive to neural network analysis?


texts: Comparing multi-class vs. binary classifiers in predicting a single class

texts:
I've pretty much read the majority of similar questions, but I haven't yet found the answer to my question.
Let's say we have n samples of four different labels/classes namely A, B, C, and D. We train two classifiers:

First classifier: we train a multi-class classifier to classify a sample in data to one of four classes. Let's say the accuracy of the model is %x.
Second classifier: now let's say all we care about is that if a
sample is A or not A. And we train a binary classifier for classifying samples to either A or non-A. Let's say the accuracy of this models
is %y.

My question is, can we compare x and y as a way to measure the performance of classifiers on classifying A? In other words, does a high performance in a multi-class classifier mean that the classifier is capable of recognizing the single classes with high performance as well?
The real-world example of this is that I've read papers that trained multi-class classifiers on a dataset that contains four different types of text. They achieved pretty high performance. But all I care about is for a model to be able to correctly classify one specific type of text. I trained a binary classifier that achieves a lower accuracy. Does this show that my model is working poorly on that type of text and the multi-class classifier is doing better? Or shouldn't I compare these two?


texts:
Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where a is bias term
Do we have any difference between bias and inductive bias.
How Inductive bias is helpful in generalizing
Any Kind of reference is helpful
Thanks in Advance


texts: Structures for incorporating linear functions into a nonlinear optimization problem

texts:
I'm working on a problem which naturally involves both linear and nonlinear operations, and I'd like some help understanding the best way to combine these into a neural network framework. To be more precise: suppose I have a matrix $A$, and some basis of matrices which it can be decomposed into, i.e.,

$$A = \sum_{j}a_{j}A_{j}$$

for some real numbers $a_{j}$ and some pre-specified basis matrices $A_{j}$. Once I have this matrix $A$, I want to apply a function $f$ to it to obtain $f(A) = X$. Note that in this context I mean actually applying $f$ to the full matrix $A$: it is not elementwise applications of $f$, i.e., $f(A)$ is defined by the Taylor series of $f$, and $f$ is a nonlinear function. (In case it matters, the function I have in mind is the matrix exponential, but I don't think this should make much of a difference in the statement of the problem.)

Now, I also have some objective matrix $Y$. My goal is to find $A$ such that $X$ approximates $Y$ as closely as possible. The key here is that $f$ is a nonlinear function, but I want to restrict $A$ to be a linear combination of the $A_{j}$'s. I don't know how to incorporate that constraint naturally in a machine learning context, but it's obviously desirable to do so since I would like to do gradient descent over both the $a_{j}$'s and the parameters which enforce the function $f$ to find the desired  matrix $X$. Any guidance, broadly speaking about what neural network architectures are best to tackle this kind of problem would be very helpful.


texts: How can we conclude that an optimization algorithm is better than another one

texts: When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the papers on a new optimization algorithm, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves. So do we compare best performance or average performance?


texts: Classification of skills based on job ads

texts:
I have around 1,000 job ads in the filed of IT (in excel file). I want to find the skills which are mentioned in each of ads. and then find the similar jobs based on skills.

My method: I created 12 categories Such as programming skills,  testing skills,  communication skills, network skills, ... . Each advertisement may belong to 3-4 categories. In this case, some said multi-variate classification or Multi label classification is useful. But I don't know how to do this kind of classification in RapidMiner.

1- Does anyone know how to do multi-variate classification or Multi label classification in RapidMiner? or is there another way?

2- Do you recommend "classification" in order to analysis required job skills? or another technique?

3- Is there any better way to classify the skills which are stated in job ads?

I'm new in the field of text mining. Please let me know if you have any idea. Thanks


texts: Control which features are used for every task in multioutput classification?

texts:
I would like to perform a multiclass-multioutput classification task, on vectorized textual data. I started by using a random forest classifier in a multioutput startegy:

    forest = RandomForestClassifier(random_state=1)
    multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
    multi_target_forest.fit(X_train, y_train)
    y_pred_test = multi_target_forest.predict(X_test)

When looking on the feature importance for the individual estimators (multi_target_forest.estimators_ ) I've noticed that some features in my dataset are very relevant and useful for some tasks, but are disrupting for another class.
Example:

Task 1: classify documents for Date (q1, q2, q3, q4)
Task 2: classify document for Version (preliminary, final, amendment)

For task 1, features related to dates, such as 'April', are very useful. However, for the second task, the feature 'April' gets a high importance but is a consequence of overfitting to a small dataset. Knowing this I would like to actively remove such features.

Is there a way to control which features are used for every task?

I could just explicitly train separate classifiers for every task, but is that equivalent to multioutput-multiclass? or is there some joined probability calculation going on, that I'll be missing?

Thank you!


texts:
I want to predict conversion rates for an eCommerce store. I have data from Google Analytics with features like averageSessionDuration, bounceRate, numberOfVisitorsBySource etc. and the corresponding conversion rate.
That's the specific background for a generic question: I am not sure how I should select the time units for my data. Should I export the data by day, by week, by month or by year?

When I have one row per day, I have a lot of data (a lot of rows),
but also a lot of noise, because the conversion data for one day is
sparse.
When I have one row per year, I have less noise, but also less data
(less rows).

In general, what's the right approach to select the right time frame per row in such a case?


texts: The graph above shows how accuracy stops increasing after reaching a certain number of features. There are also sudden drops in accuracy at some points. Can this be attrrubuted to overfitting? I am training a decision tree by the way.


texts: Understanding the output of the Random Forest method for classification

texts:
I'm using a Random Forest method to predict the behavior of failures at Period_12. My dataset has information about the eleven periods before, considering 112 subperiods (rows). Each one of these subperiods is classified according to two classes, "Normal" or "Failure". However, the events "Failure" occur much less than the event "Normal". When using the k-fold crossvalidation (named MODEL 1 at the script), I have a Kappa metric equal to zero, while when using the Stratified k-fold crossvalidation (MODEL 2), I have a Kappa metric of 0,325 for mtry=11.

My questions:
a) Why is the Kappa metric so low? It is related with the occurrences of events "Failure" in my dataset? Is there a way to improve this value?
b) When I set folds <- 10 at the MODEL 2, I have an even lower value for Kappa metric.

Someone could help me to understand the output of my model or can suggest a way to improve it?

The script:

library(caret)

data_failures <- read.csv('OUTPUT.csv', header = TRUE, sep = ",", stringsAsFactors = TRUE)

# MODEL 1

train.control <- trainControl(method = "cv", number = 10)

model <- train(Period_12 ~., data = data_failures, method = "rf", trControl = train.control)

print(model)

# MODEL 2
folds <- 4

cvIndex <- createFolds(factor(data_failures$Period_12), folds, returnTrain = TRUE)

tc <- trainControl(index = cvIndex, method = 'cv', number = folds)

model2 <- train(Period_12 ~., data = data_failures, method = "rf", trControl = tc)

print(model2)

The output:

Random Forest

112 samples
 11 predictor
  2 classes: 'Failure', 'Normal'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 100, 102, 102, 100, 100, 101, ...
Resampling results across tuning parameters:

  mtry  Accuracy   Kappa
   2    0.9484848  0
   6    0.9384848  0
  11    0.9384848  0

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.
Random Forest

112 samples
 11 predictor
  2 classes: 'Failure', 'Normal'

No pre-processing
Resampling: Cross-Validated (4 fold)
Summary of sample sizes: 84, 84, 84, 84
Resampling results across tuning parameters:

  mtry  Accuracy   Kappa
   2    0.9464286  0.000
   6    0.9464286  0.000
  11    0.9553571  0.325

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 11.
Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

A sample of the data:

     Period_1 Period_2 Period_3 Period_4 Period_5 Period_6 Period_7 Period_8
1     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
2     Normal  Failure  Failure   Normal   Normal   Normal   Normal   Normal
3     Normal  Failure  Failure   Normal   Normal   Normal   Normal   Normal
4     Normal  Failure  Failure   Normal   Normal   Normal   Normal   Normal
5     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
6     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
7     Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal
8     Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal
9     Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal
10    Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal


texts: What is the value of AIC criterion if RSS is 0?

texts:
The AIC formula is :
$AIC = 2k + n Log(RSS/n)$
So if RSS is equal to 0, it is undefined. How do I deal with this? What value should it take?


texts: How can I train a many-to-one RNN with an array of 2D matrices?

texts: I have eye tracking data for every word of a novel. Features for every word is given separately. I want to take groups of 100 words to make a sample and then use each of these samples as a single training instance (instead of a single word). My samples are 2D matrices of shape (no. of words, features). How can I train my many-to-one RNN with these samples?


texts: Python : Approaches to mine for time ordered sequences

texts:
I have a dataset that looks like this :
timestamp          event

12/1/2020 14:30     A
12/1/2020 14:12     C
12/1/2020 14:10     A
12/1/2020 12:01     B
11/1/2020 21:20     A
11/1/2020 21:00     B
   .....           ...

events are actions by the user on an application. timestamp is when that particular event was raised.
I want to mine for sequences of events in the data to find out what a user's work-flow (market basket analysis with time constraint?) has been inside the application from the day user started using the application.
What would the proper approach to solving this problem be?
Are there any libraries that will allow me to apply sequence mining as a black box??


texts:
I am working on e-commerce, how to rank smartphones in the same category. I want to calculate a weighted average of sentiment score from reviews posted by buyers.

Weights will be based on how many days before review was posted, as people might give more importance to recent reviews than old reviews. Now to find weight for previous day review, i want to understand impact of previous reviews on current review. i.e. impact of t-1, t-2, t-3 on t.

I want to know for how many days should i aggregate review sentiment?


texts:
Having this R statement:

itemsets = apriori(data, parameter=list(support=0.05, confidence=0.5))
plot(itemsets, method="graph",
     control=list(type="items"))

How can I change the colour of my plot? I try to insert "col" but don't allows me to do that because I'm getting the following error:

Error in i.parse.plot.params(graph, list(...)) :
  Unknown plot parameters: col

thanks


texts:
I am trying to implement a Neural Network for binary classification using python and numpy only.

My network structure is as follows:
input features: 2 [1X2] matrix
Hidden layer1: 5 neurons [2X5] matrix
Hidden layer2: 5 neurons [5X5] matrix
Output layer: 1 neuron [5X1]matrix
I have used the sigmoid activation function in all the layers.

Now lets say I use binary cross entropy as my loss function. How do I do the back propagation on these matrices to update weights?

class Layer():
    def __init__(self,number_of_neurons,number_of_inputs):
        self.weights=np.random.rand(number_of_neurons, number_of_inputs)
        self.bias=np.random.rand(number_of_neurons,1)

class NeralNetwork():
        def __init__(self, layer1, layer2,layer3):
            self.layer1 = layer1
            self.layer2 = layer2
            self.layer3 = layer3

    def sigmoid(self,x):
        return 1 / (1 + np.exp(-x))

    def derivative_sigmoid(self,x):
        return x*(1-x)

    def get_cost_value(self,Y_hat, Y):
        m = Y_hat.shape[1]
        cost = -1 / m * (np.dot(Y, np.log(Y_hat).T) + np.dot(1 - Y, np.log(1 - Y_hat).T))
        return np.squeeze(cost)

    def get_cost_derivative(self,Y_hat,Y):
        return  - (np.divide(Y, Y_hat) - np.divide(1 - Y, 1 - Y_hat))

    def train(self,inputs,labels,epocs):
        for epoc in range(1,epocs+1):
            z1=np.dot(self.layer1.weights,inputs)+self.layer1.bias
            a1=self.sigmoid(z1)
            z2=np.dot(self.layer2.weights,a1)+self.layer2.bias
            a2=self.sigmoid(z2)
            #print(a2.shape)
            z3=np.dot(self.layer3.weights,a2)+self.layer3.bias
            a3=self.sigmoid(z3)
            #print(a3.shape)
            if epoc%100 is 0:
                print(a3)

            cost=self.get_cost_value(a3,labels)
            #print(cost)

            layer3_delta=self.derivative_sigmoid(a3)*self.get_cost_derivative(a3,labels)
            print(layer3_delta.shape)
            Dw_layer3=np.dot(layer3_delta,a2.T)
            Db_layer3=layer3_delta
            #print(Dw_layer3.shape)
            layer2_delta=np.dot(self.layer3.weights.T,layer3_delta)*self.derivative_sigmoid(a2)
            #print(layer2_delta.shape)
            Dw_layer2=np.dot(layer2_delta,a1.T)
            Db_layer2=layer2_delta

            layer1_delta=np.dot(self.layer2.weights.T,layer2_delta)*self.derivative_sigmoid(a1)
            Dw_layer1=np.dot(layer1_delta,inputs.T)
            Db_layer1=layer1_delta
            #print(Dw_layer1)

            self.layer1.weights-=((1/epoc)*Dw_layer1)
            self.layer2.weights-=((1/epoc)*Dw_layer2)
            self.layer3.weights-=((1/epoc)*Dw_layer3)

            self.layer1.bias-=((1/epoc)*Db_layer1)
            self.layer2.bias-=((1/epoc)*Db_layer2)
            self.layer3.bias-=((1/epoc)*Db_layer3)

So far I have tried to implement this as shown above. But there seems to be a mistake because, after training, the network doesn't seem to have learned. Please let me know if you have any inputs.


texts:
I would like to use CNN to make classification with 5 classes, but 4 of these classes only have between 16 and 60 images, while the last one has more than 1300. I know 16 or 60 images are not enough, so I want to use transfer learning, fine tuning and data augmentation. However, I have several questions.
As the data augmentation must only be used for the training data, I will have very few images from the 4 classes for the validation set, would it be a problem?
Is it needed to split it in training/validation/test, or is training/validation enough?
Another problem is the unbalanced data: with such a difference between the number of images in each class, would oversampling or undersampling be a good solution?
For transfer learning and fine tuning, should I freeze all the convolutional layers or should I only train a FC layer?


texts: Apply cross validation on regression algorithms

texts:
How can we apply K-fold cross validation on say linear regression? Regression contains weight update and iterations, and so do we integrate K-fold to it?

If done, is it like we take the first K-1 fold to train using linear regression and perform iterations to obtain weight, then take the next fold and proceed the same and so on until all folds are done and pick the best weight or fold from it?


texts: Is my model overfitting when I add new features?

texts:
I'm working on simple 2-class classification problem. Nearly all features we have used (except one) are about the same for both classes: A random forest classifier confirms that one feature has an "importance" of 40% and the rest have about 8% or less.

To try to boost performance, I included 4 new features as an experiment. These features all have VERY different distributions in the two classes (their means are quite different) so they should be useful for discriminating between the classes. However, when I included these, performance dropped significantly. Performance details:

We are trying to achieve 95% precision (that is, 95% of things classified as "class 1" are indeed class 1).
Using the old feature set, we could achieve about 96% recall (96% of things that ARE class 1 are classified as 1).
The new features, however, drive recall down to 95% or below.

I strongly suspect the model is now overfitting. However, I can't find any evidence of this. The accuracy of the model on TRAINING data used to be about 1% higher than on TEST data. With the 4 new features, the accuracy is only 0.8% higher - suggesting the model is actually overfitting LESS than before.

I am currently using the "best performance" numbers of an SVM. I have also tried these things on a random forest model and the results have been the same.

Is it possible I'm measuring overfitting wrong? Or is there another way that adding new features can degrade the performance of a classifier?


texts:
I am trying to create a function that automates the process of taking a CSV file, splits in the data in features and responses, apply different models (regression) to the data and score them according to some metric such as MAE, RSME, etc. Model parameter should be easily interchangeable. Here is a little of what I have so far.

import numpy as np
import pandas as pd

def get_data(file_name):

data = pd.read_csv(file_name)

para = int(raw_input('Enter the no of parameters to be used '))
print(para)

param= []

for k in range(0,para-1):
    param[k]= raw_input('Enter the parameter')

rec = int(raw_input('Enter the no of records in the dataset '))
print(rec)
x_parameter = []
y_parameter = []
x1= []
for i in range(0,para):
    for x1[i] in data[i]:
        x_parameter[i].append(x1[i])

for j in range(0,rec):
    print x_parameter[j]
    print y_parameter[j]

get_data('C:\Users\Douglas\Desktop\trainingset.csv')

from sklearn import model_selection
from sklearn import linear_model
from sklearn import SGDRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressors
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.svm import SVR

dataframe = getdata(file_name)
array = dataframe.values
X = array[]
#confirguartion for cross validation
seed = 7
#prepare models
models = []
models.append(('LR', LinearRegression()))
models.append(('SGDR', SGDRegressor()))
models.append(('KNR', KNeighborsRegressor()))
models.append(('DTR', DecisionTreeRegressor()))
models.append(('GBR', GradientBoostingRegressor()))
models.append(('SVR', SVR()))
#evaluate each model in turn
results = []
names = []
scoring = 'neg_mean_absolute_error'
for name, model in models:
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=sc)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print msg


texts: How can a NN guess the velocity of an object from a video source?

texts:
In the case of a video stream, I'd like to detect the speed (an approximation) of an object, that is moving. What would be the best approach to take?

I am thinking of 3 methodologies to take, though I've not found a lot resources to read about.

Method 1: I could feed Frame images to my CNN and pinpoint the object in the frame. Then calculate the difference of the position boxes.

Frame 1: [  o     ] ---> NN --> 3
Frame 2: [      o ] ---> NN --> 7

7-3 = 4 -> so dX = +4 between 1 frame, so I can estimate the dT (if I know the sampling rate of the frames)

Method 2: Is there any way where the NN can take as a feed-input the context of the previous frame and make the calculation itself?

video lapse 1sec: [  oooo ] ---> NN --> 4m/sec

Method 3: What if I can control the shutter speed of my camera, could I calculate the velocity of an object by the motion blur?

Frame image: [   ---o ] ---> NN --> 4m/$shutterSpeed

Any relevant resource to read would help a lot.


texts: I am new in this field and I am trying to understand the Deep Deterministic Policy Gradient (DDPG) model. I can not understand the use of the target network in this model. Why do we need to define the target network for actor-network and critic-network?


texts:
I have tried searching for this, but it seems like no one addresses a key aspect of this problem (or maybe I'm overthinking this):
So, first let's assume we have a 3x3 image with a single channel, and we apply a 2x2 kernel to it, and let's name the kernel weights $w_1$, $w_2$, $w_3$ and $w_4$. This convolution generates a 2x2 feature map, and let's name the resulting activations $a_1$, $a_2$, $a_3$ and $a_4$.
So, the way you would calculate the activations would be something like:
$\sigma(w_1 * v_1 + w_2 * v_2 + w_3 * v_4 + w_4 * v_5 + b) = a_1$
$\sigma(w_1 * v_2 + w_2 * v_3 + w_3 * v_5 + w_4 * v_6 + b) = a_2$
...
Now, my question is:
To calculate dw1, you'd have different results in each activation (dw1 in a1 = v1, whereas dw1 in a2 = v2). So for these, do you SUM them, or AVERAGE them? I've tried it both ways, and averaging them has gotten me faster training and better results, but it seems like most people tell you to sum?
Here's a code snippet of how I'm implementing this (data is shaped this way because i'm performing convolution through matrix multiplication and index manipulation)
dz is the activation derivative to the current layer, and they all have the following shape:
(number of kernels, feature map size, batch size)
            data = squeeze(sum(data, 2)); // MEAN or SUM across feature maps
            db = squeeze(sum(dz, 2)); // MEAN or SUM across feature maps
            dw = db * data';
            db = sum(db, 2); // Definitely SUM all mini-batch values


texts: Please suggest some good ways to test my RNN

texts:
I have developed an RNN in matlab and now its time to test it. You can set your desired number of layers and nodes and it trains data in random chunks. I also included an annealing function for learning rate that decays gradually from 1 to 1e-5 to avoid overtraining of data.

I would like you to suggest me some simple ways to test my network in a way that i can get some useful feedback about its behavior from the results. I am a little concerned about convergence. I' ve set to do some testing of my own but would like to hear some suggestions from more experienced users.

I try to simulate some 1st, 2nd, 3rd, ... grade function between (0-1) and see how it goes. I will also try to predict the next value of a time series based on previous values, as another testing method. What would be a good result from these tests? An accuracy of ??% and an mse of ?? ?


texts: Why can't I specify the correct NumPy size?

texts:
In the network (model of Keras, Sequential), the input layer must have 4 neurons. The input must be 1 list, the length of which is 4, each element is a number.
print("SHAPE:", np.array([1, 1, 1, 1]).shape)

self.model.fit(np.array([1, 1, 1, 1]),
self.rightAnswer,
epochs = 1,
batch_size = 1)

Here is the conclusion:
SHAPE: (4,)

ValueError: Error when checking input: expected dense_1_input to have shape (4,) but got array with shape (1,)

Why is this happening, and how can I fix it?


texts: How are the same observation sets treated in Random Forests with Bootstrapping?

texts:
Let's assume an extremely small dataset with only 4 observations. And I create a Random Forest model, with a quite large number of trees, say 200. If so, some sample sets that are the same each other can be used in fitting, right? Is it OK?
Even though a dataset is large, the same sample sets can be selected, theoretically. Do the Bootstrapping and Random Forest method not care at all or have a step to avoid such duplicates?


texts: When to use which multiple testing correction?

texts:
There are a large number multiple testing p-value correction methods. e.g.:
   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)
I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.
Any ideas? How do I decide which multiple testing correction I should apply?


texts: I have payments data and I need suggestions from experts what can I do

texts:
I have data for a local payment service that is not compatible with standards for electronic payment services such as Visa cards.

The service depends on sending a message to the buyer from the merchant containing the cost value then the buyer confirms the process

Unfortunately there is not much useful information nor details about the customers such as  age or location, but I have some fields such as payment value, and time of begin transaction and time of end transaction and merchant & customer account number and type of process (Buy, Retrieve), also Point of Sell Number.

I thought about classifying these transactions using k-means algorithm but ..
What can I get insights from these data, Do i can detect the anomalies in them based on time and value only and the rest of the fields mentioned above?


texts: Should I go for a 'balanced' dataset or a 'representative' dataset?

texts:
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should choose a similar data setup for training my models as well. But I came across a research paper or two (in my area of work) which have used a "class balancing" data approach to training the models, implying an equal number of instances of benign and malicious traffic.

In general, if I am building machine learning models, should I go for a dataset which is representative of the real world problem, or is a balanced dataset better suited for building the models (since certain classifiers do not behave well with class imbalance, or due to other reasons not known to me)?

Can someone shed more light on the pros and cons of both the choices and how to decide which one to go choose?


texts:
I have a signal and want to predict y which present Number of requests, using regression models. Currently, I am using OLS regression model to predict y. But the prediction error is very high, as my signal has a lot of variations (ups and downs) as shown below.

I noticed that my model most of the time overestimate y (Number of Requests), especially if the points to be predicted is preceded by large value of y's. As indicated below in the yellow and red circle.

So I am not sure if there's a robust regression models to accommodate this problem of having a lot of variations in my datasets.
Also is there any way to segment out these large values by adapting the window size such that it doesn't include these values?

Could you please advise


texts: Clustering Crime Data which has {latitute, longitude, crime-type} tuples

texts:
I have a data set which has thousands of rows of {latitute, longitude, crime-type} tuples.

Sample Data:

41.757366519   -87.642992854   THEFT
41.910469677   -87.585822373   ROBBERY
41.751270452   -87.690708662   BURGLARY
41.757366519   -87.642992854   THEFT
41.757366519   -87.642992854   THEFT
..             ..              ..
..             ..              ..

I am trying to cluster these based upon the crime types.

For example, if in any  region, THEFT has a high frequency of occurrence, based on the data set, it should show up as a cluster. I have tried clustering using the lat-long data only, and that does not seem to have any meaning for this crime dataset.

I am fairly new to data mining, and gradually figuring my way out.

How can I cluster the data using the latitude and longitude values based such that the clusters are related to each other through the crime-type? Is there any tool available that can use the lat-long data and cluster them on the crime-type basis? Otherwise, I can even write a script once I understand how this can be done.

Also, has anyone had any previous experience in crime-data-mining? In what other ways can I find interesting patterns from a crime data-set?

Thanks a lot!


texts: How to convert sequence of words in to numbers which are input to RNN/LSTM?

texts:
I am watching online videos and tutorials about use of RNN/LSTM for NLP but none of them explain how to convert the sequences of words into digitized input to the neural networks?

I am looking for intuitive understanding but answers with python code are also welcome.

For example, how do I input; "grass is green and sun is hot" to my RNN/LSTM?


texts: Loss function and data distribution for positive / negative examples

texts:
I have an AlexNet architecture that predicts in a supervised-way, whether a given image belongs into classes 1-10. But it should also recognize if it doesn't fit into any of these classes.

My thought was that I just add an 11th class and label all my negative-data with this class and train the softmax-cross-entropy loss with Gradient Descent Optimizer (Adam) as usual.

Now my question is if this is a valid approach on solving my problem?

I have around 7.000 images per class and could obtain up to 70.000 negative examples. I have the strong feeling that training with 70.000 negative-examples my network would always predict the outcome to belong to class 11 (not matching any other). Am I correct with the assumption? If yes: How many negative examples would do the job?


texts: How can I make use of the labels subdivision in a Deep Learning Image classification?

texts:
I want to solve an image classification problem. I want to recognize cats and dogs. However my labels are more detailed than this - they contain information about the dog/cat breed.
 In other words I want to classify images into A and B, but for each photo I know labels such as A.1, A.2, B.1, B.2.

If I simply cast the labels into A and B I lose some of the expertise I have.
I can try to classify into detailed labels first and then cast the results.

However I don't want my neural network to focus on distinguishing between particular sublabels.

Are there any rules of a thumb on it? Any name for such a problem I don't know?


texts: Anomaly detection for time series data with only positive samples?

texts:
I'm having a time series ECG dataset. I want to do anomaly detection (anything different from normal ECG should be abnormal).

The point is I'm having only positive samples with very few negative samples.

How to model this problem ? Is it possible to model this as probability distribution and whenever some negative samples comes just taking the divergence from positive distribution ?


texts:
I am using fastFM to run a factorization model.

Just wanted to know if there is an approach to interpreting the model?
How do I extract the Factorization Model equivalent of Variable Importance?


texts: How are images (roughly) clustered when using KNN

texts: When classifying data by KNN, the classes are calculated by using the distance between datapoints. For example, the case of real estate, where the x-axis is the price and the y-axis the size of real estate. But what are the values of axis when instead of numbers there are pictures? By which category are the distances calculated in this case?


texts: Should I perform customer segmentation before performing churn prediction?

texts:
Imagine a company with multiple lines of revenues coming from diferent products, but all customer can access these different products through the same account and the same online platform.
My goal is to predict the churn for each customer.
Should I perform customer segmentation into clusters and build a churn prediction model for each segment?
The straight foward path would be to get all relevant features for all customers and try to predict the churn for all of them. The different bahaviors and personae will theoretically reflect on the features.
What I do expect to get from the clustering are the personae this company intends to serve.
In this case, a churn will be defined as a custumer who stops consuming any product or close its account.
My objective here is to build a churn prediction model according to each pearsonae behavior pattern and I expect from this approach to have better features for each because now we understand them more and then more acurate models.
This would also lead to more personalized business strategies to avoind the churn.
But I don’t know if this is a good practice and if I should try the straight foward approach or a different one.


texts: Recommendations and Missing Data in Deep Learning

texts:
In this research paper, it is discussed how to combine deep learning with wide (shallow) learning to achieve both generalisation and the ability to learn correlation/association rules.

The input vector of such network is an n-dimensional vector of features $\mathbf{x} = (x_1, x_2, x_3,\dots, x_n)$. While training, the following objective function is maximised:
$$  P(Y=1|\mathbf{x})   $$

Within the context of recommender systems (which is one of the main applications of this paper) how does one cope with the problem of missing data (unseen products)?


texts: What's a classifier capable of predicting a variable number of classes

texts:
I want to solve what I understand as a classification problem regarding tagging. Let's say an Entity can have 0 or more tags and I want to be able to predict which tags (if any) an entity might get

I would like the classifier to output 0 or more tags.

I have never run into this kind of structure with variable length in input data (many tags) and output (0, one or more tags for each entity) before and I'm not even sure what terminology / words to search for.


texts: "R" write/export dataframe preserving the class information

texts:
Short Question

After formatting a dataframe in R with class information like "Date", "Factor", and etc. how to save the dataframe while preserving the class information. Consequently how to load the information back to a new R Project with preserving the above class information.

Long Question
I have created following dataframe with class information,
names <- c("Adrew", "Vikie", "Penny", "Samantha")
date <- as.Date(c("1984-12-28", "1986-10-18", "1986-11-30", "1987-11-30"))
gender <- c("M", "M", "S", "S")

df <- data.frame(names, date, gender)

df$gender <- factor(df$gender,
                   levels = c("M", "S"),
                   labels = c(1,2),
                   order = FALSE)

and the classes are factor, date, factor
method 01
then write the dataframe in csv format and read it
write.csv(df, "df.csv")
dfcsv <- read.csv("df.csv")

and the daraframe classes are changed
method 02
then tried to write the file using table export method
write.table(df, "df.R")
dftable <- read.table("df.R")

again the data frame classes had been changed
Question

is there a proper method to write or save a data frame with preserving the class information like date, factor, etc information intact, and conversely how to load it back.


texts: Unable to perform Keras Reshape to an input to match convolution output

texts:
I am newbie to whole tensorflow and keras. I am trying to implement icGAN in which I feed a conditional vector of length 'ny' after first convolution(I have added network architecture for better insight) . I am having trouble merging the output of first convolution with the condition vector. I tried using reshape but it gave me weird errors. Below is my code snippet. The below code does not give error but I want to know is how should I change the y which will be of shape (batch_dim, 100) to (batchdim, 32,32,100).

    x = Input(shape=img_dim, name="discriminator_input")
    y = Input(shape=cat_dim, name="category_input")
    list_f = [128, 256,512]
    # First conv
    x = Convolution2D(64, 3, 3, subsample=(2, 2),     name="disc_convolution2d_1", border_mode="same")(disc_input)
    x = LeakyReLU(0.2)(x)

    #How to reshape the cat_input.

    # Next convs
    for i, f in enumerate(list_f):
        name = "disc_convolution2d_%s" % (i + 2)
        x = Convolution2D(f, 3, 3, subsample=(2, 2), name=name, border_mode="same")(x)
        x = BatchNormalization(mode=bn_mode, axis=bn_axis)(x)
        x = LeakyReLU(0.2)(x)

Any help would be appreciated.


texts: How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere

texts:
Overall context:

I have a data frame that contains observations for every five minute starting at 5 AM in the morning and ending at 8 PM in the evening for several days. I need to filter all the observations that start from 9 AM in the morning and end at 5 PM in the evening for every day.

The input data frame looks like this:

Date Time
2019-09-20 05:00:00,..,..
2019-09-20 05:05:00,..,..
...
2019-09-20 09:00:00,..,..
...
2019-09-20 17:00:00,..,..
2019-09-20 17:05:00,..,..
...
2019-09-20 20:00:00,..,..
2019-09-21 05:00:00,..,..
2019-09-21 05:05:00,..,..
...
2019-09-21 09:00:00,..,..
...
2019-09-21 17:00:00,..,..
2019-09-21 17:05:00,..,..
...
2019-09-21 20:00:00,..,..

and the output data frame should look like this:

2019-09-20 09:00:00,..,..
...
2019-09-20 17:00:00,..,..
2019-09-21 09:00:00,..,..
...
2019-09-21 17:00:00,..,..

Steps taken so far

In order to extract the rows between 9 am and 5 pm, I determined the number of seconds since midnight for every row by
extracting the hours, minutes and seconds using  vectorized data operations
so input dataframe will have column like:

Date Time, Number of seconds since midnight
2019-09-20 05:00:00,xxxx,..,..
2019-09-20 05:05:00,yyyy,..,..
...
2019-09-21,05:00:00,xxxx,..,..
2019-09-21, 05:05:00,yyyy,..,..

Note that for the same time on every day, the number of seconds will remain the same
Now I was hoping to  extract alll the rows between 9 am and 5 pm by

df[(df['Number of seconds since midnight'] > (nseconds for 9 am from midnight)) &  ((df['Number of seconds since midnight'] < (nseconds for 5 pm from midnight))

but I get the rows from only the last date between 9am and 5 pm.
TO me, it looks it is ignoring all the duplicate rows with the same time.

Can anyone suggest a possible solution that does not iterate over each row and uses the vectorized operations as the database is very large


texts: Training loss = 0, training accuracy =1, validation and test around 85%


texts: Predicting the test data with LinearRegression model gives ValueError: shapes (8523,1606) and (1605,) not aligned: 1606 (dim 1) != 1605 (dim 0)

texts:
Fitting the model, testing and getting the score or r2 does not give the error. But when I try to predict the actual data I get this ValueError:
#test and train shapes
test.shape, train.shape
((8523, 1606), (8523, 1606))

#creating dummies for the training dataset
X = train.drop('Item_Outlet_Sales_log', 1) #drop the log target column
y = train.Item_Outlet_Sales_log

X = pd.get_dummies(X)
train = pd.get_dummies(train)
test = pd.get_dummies(test)

#split the train data into train and test set in order to evaluate the model's accuracy
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

#split the the train data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 42, shuffle = True)
#model 2 fitted with raw data
regression_model_2 = LinearRegression()
#train
regression_model_2.fit(X_train, y_train)

pre = regression_model_2.predict(test)

Error

I tried different approaches but it still complains about the shapes


texts: How to force DecisionTreeRegressor to use polyfit equation instead of mse at leaf level in python SKlearn

texts:
I am using python SKlearn DecisionTreeRegressor for my data. As DecisionTreeRegressor defines constant data at leaf node, my prediction is looking like a step, instead I wanted to force polyfit Equation of my interest. I was looking some thing like to use Decision tree to some depth, then apply normal regression at leaf node to predict continuous data.
For my data set normal regression doesn't work as it looks for global error minimization, so i am looking for this.
I am a beginner, any response would be appreciated. Thanks.


texts: What does 'Linear regularities among words' mean?

texts:
Context: In the paper "Efficient Estimation of Word Representations in Vector Space" by T. Mikolov et al., the authors make use of the phrase: 'Linear regularities among words'.

What does that mean in the context of the paper, or in a general context related to NLP?

Quoting the paragraph from the paper:

  Somewhat surprisingly, it was found that similarity of word
  representations goes beyond simple syntactic regularities. Using a
  word offset technique where simple algebraic operations are performed
  on the word vectors, it was shown for example that vector(”King”) -
  vector(”Man”) + vector(”Woman”) results in a vector that is closest to
  the vector representation of the word Queen [20].

  In this paper, we try to maximize accuracy of these vector operations
  by developing new model architectures that preserve the linear
  regularities among words. We design a new comprehensive test set for
  measuring both syntactic and semantic regularities1 , and show that
  many such regularities can be learned with high accuracy. Moreover, we
  discuss how training time and accuracy depends on the dimensionality
  of the word vectors and on the amount of the training data.


texts: Recommender System: how to treat different events

texts:
I'm trying to build recommender based on user history from e-commerce. There are two(potentially more) types of events: purchase and view.
Is it okay to sum up number of purchases and views for a given item(with purchase and view having different weights)? Or I`ll just mix up user intent this way?


texts: Label description for scikit-learn Diabetes dataset

texts:
I am studying how to build a regression model using the Diabetes dataset from scikit-learn and I cannot find a good interpretation for the target value, meaning that I do not know what value would be good for a patient or not. I read the first article where it was mentioned, but no information regarding the target value except that it is a quantitive measure. Does anybody has any clue?
Thank you!


texts: How do i convert a Named num [1:4] to a data frame in R?

texts:
newbie to r, taking The R Programming Environment from coursera. one of the assignments is to select some columns from a data frame and find the means.

the code below seems to get the correct answer, but the answer should be a data frame.

wc_2 <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans()

How do i convert this to a data frame? i tried: wc_2<-as.data.frame(wc_2) but that gets it column wise. i do not see any way to pass the rows and columns or do a transpose in what i assume is a data frame constructor.

perhaps this can be done as part of the pipe?

thanks

edit: i got it to work:

means <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans()
wc_2<-data.frame(t(matrix(means)))
colnames(wc_2)<-names(means)