texts:
(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able to field any questions on the question that might pop-up.)

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?


texts: Are decision tree algorithms linear or nonlinear

texts: Recently a friend of mine was asked whether decision tree algorithms are linear or nonlinear algorithms in an interview. I tried to look for answers to this question but couldn't find any satisfactory explanation. Can anyone answer and explain the solution to this question? Also, what are some other examples of nonlinear machine learning algorithms?


texts:
I have two images and I've found their keypoints using sift keypoint detector,
Now I have to match their keypoints with HoG features,
I know how to extract HoG description,
but I dont know how to combine it with SIFT and match the keypoints,
any ideas pls?
I am using opencv and python3


texts:
I have a dataset with 100+ feature columns.
My client asked me to choose "the top 10 most important features" from the 100+.

From this post, I learnt that Random Forest can help me ranking the importance of features.

My Question is, is there any other algorithms can do this job? (also if someone can direct me to some tensorflow examples on this.)

P.S. To explain, the dataset is about eCommerce, with 100+ features and one column depicting "how much the user spent on the website". By "important features", I mean the features that can best predict the user-spent column.


texts: How can I monitor the usage of NVLink connections?

texts:
If I'm running a Keras model on some Nvidia GPUs which are connected via NVLink, how can I monitor the usage of the NVLink connections?

I want to be sure the NVLink connections are being used and see at what rate.


texts: How to gauge the Complexity of Pre trained Neural Networks?

texts: What does one mean when they are talking about the simplicity of the networks? Does it mean that the shallower the networks the simpler they are, or does it mean that lesser the number of trainable parameters, simpler the models?


texts: How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

texts:
Let us say we have a dataset with a feature such as Surname.

arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']

And I want to encode this categorical info as a new feature like

arr['Surname_Count'] = [2, 1, 1, 2]

with the caveat that it is done within an Sklearn pipeline. Are there quick ways to do this that do not involve rolling my own partition counting transformer?


texts: Local RTX 2080 is 3x faster than V100 on GCP?

texts:
I have a gaming rig with an i9 CPU, 32GB RAM and RTX 2080, and I have a GCP VM with 4 vCPU, 52 GB RAM and V100.

I try to train the same dataset using the same toolchain on both machines and these are my ETA's:

GCP VM: 16 days
Gaming rig: 5 days

How can a single $600 GPU outperform a 10k GPU?

What's going on here?

And what should I even expect?


texts: Best strategy for extracting specific structured data from unstructured sentences

texts:
Given a list of sentences like this:

4 to 5 hours over a period of 16 weeks
1st session: 2.0-2.5 hours & 2nd session: 1.5-2.0 hours
Approximately 5-6 visits over the course of 5 months. Visit 1, 3, 5: about 1.5 hours. Visit 2, 4: short
15 visits over a period of approximately 74 weeks.
You will come to the organization about 12 times, over a period of a little more than three years. Each visit will take from 3-6 hours.

What tools/strategy should I use if I want to let the model spit out the following data for the above sentences:

Number of sessions
Total duration(h)
Total timespan(w)

Unknown
4-5
16

2
3.5-4.5
Unknown

5-6
4.5
20

15
Unknown
74

12
36-72
156

I'm a ML beginner and wondered if this is achievable with Tensorflow or GPT? For further learning on my own: what is the specific terminology I should google for? Is this NER, text extraction or more like text classification?


texts: How to assign date into pandas Dataframe?

texts:
Im trying to code correlation between exctracted stock prices and data frame with "count" and "date" rows. My dataframe (cvd) looks like this:
``
cases       date
0         0 2020-01-01
1         0 2020-01-02
2         0 2020-01-03
3         0 2020-01-04
4         0 2020-01-05
..      ...        ...

``
and my stock prices (bit) like this
2020-01-01     7200.174316
2020-01-02     6985.470215
2020-01-03     7344.884277
2020-01-04     7410.656738
2020-01-05     7411.317383

2020-12-11    18058.904297
2020-12-12    18803.656250
2020-12-13    19142.382812
2020-12-14    19246.644531
2020-12-15    19417.076172

a simple
cvd.corrwith (bit ,method='pearson')
doesnt work because of error
TypeError: '<' not supported between instances of 'Timestamp' and 'int'
Edit: Also there are some dates missing in the stock prize data, figured that might be a problem?


texts: Measuring distance preservation in dimensionality reduction

texts:
I am looking to compare the distance preserved during dimension reductions for several techniques. I have read some papers on similar topics here and here.

For example, I would like to use the Euclidean Distance to measure the distance preserved during PCA's dimension reduction. However, my point of confusion what are $X$ and $Y$ in
$$d(X, Y) = \sqrt{\sum^n_{i=1}\left(x_i - y_i\right)^2}$$
I understand how to calculate $d\left(X, Y\right)$ given two vectors/matrices, but I don't understand with context to PCA. Let me try to explain.

Let $W_{d \times k}$ be the matrix of $k$ leading eigenvectors, $X_{d\times n}$ be the original data, and $Z_{k\times n}$ be the projection of $X$ onto the reduced subspace.

$Z = W^{T}X$

Back to calculating $d\left(X, Y\right)$. My guess is that the PCA's $X$  correspond to $X$ and $Y$ can correspond to $Z$. But how does this work since $X$ and $Y$ have different dimensions?  I have to be oblivious to something here.

Also, I am not concerned if a Euclidean Distance measure is not a good choice for measuring PCA's distance preservation (unless they are incompatible). This is simply exploration.

Edit:
For example, if I have
$$X_{d\times n} =
    \begin{bmatrix}
    x_{11} & \dots & x_{1n} \\
    x_{21} & \dots & x_{2n} \\
    x_{31} & \dots & x_{3n} \\
    \end{bmatrix}
$$
and say I choose to retain $k = 2$ principal components which are then projected onto
$$Z_{k \times n} =
    \begin{bmatrix}
    z_{11} & \dots & z_{1n} \\
    z_{21} & \dots & z_{2n} \\
    \end{bmatrix}
$$


texts:
I am trying to merge two Keras models into a single model and I am unable to accomplish this.

For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use this as input to the layer $B1$ (of dimension 8 again) in Model $B$ and then combine both Model $A$ and Model $B$ as a single model.

I am using the functional module to create Model $A$ and Model $B$ independently.  How can I accomplish this task?

Note: $A1$ is the input layer to model $A$ and $B1$ is the input layer to model $B$.


texts:
I mean stuff that doesn't use Python, Keras or TensorFlow. I have been looking for an in-depth explanation on how to implement a DCGAN from the ground up so I can have a complete understanding of the concept. Even some literature on GANs from the ground up would be useful.

All of the stuff I have read thus far leaves out a lot of the details, falling back on third party libraries, hiding the implementation.

Is my best bet to read papers? Trudge through references until it makes sense? Is there not a comprehensive text because of the novelty?


texts: I have a dataset with multiple numeric input values and a categorical output. How can I measure model performance with different algorithms. As the results are categorical, we can not obtain r squared or mean square error ect... but I have the accuracy values as performance metrics. What else can be used to measure model performances when the result is categorical?


texts: What is the difference between register_buffer() and parameter.detach() in PyTorch?

texts:
I am writing a PositionalEmbedding() module which is an implementation based on "Attention Is All You Need" using PyTorch. According to the paper, there should be no learnable attribute in the module PositionalEmbedding().
The initialization of the Embedding weights is as follows:
    import math
    import torch

    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            position = torch.arange(0, embedding_number).unsqueeze(1)
            sin_multiplicator = torch.exp(-(math.log(10000) / dimension) * 2 * torch.arange(0, dimension, 2))
            cos_multiplicator = torch.exp(-(math.log(10000) / dimension) * 2 * torch.arange(1, dimension, 2))
            sin_weight = torch.sin(position * sin_multiplicator)
            cos_weight = torch.cos(position * cos_multiplicator)

            weight = torch.zeros(embedding_number, dimension)
            weight[:, 0::2] = sin_weight
            weight[:, 1::2] = cos_weight

Next up, because the weight in the module should not be learnable, detach() the weight came into my mind. I think it may not be the most elegant way, so I made a further investigation. I found that there are two kinds of implementation method like what OpenNMT and FAIRSeq have done respectively.
I follow the method of OpenNMT and my implementation is as follows:
    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            ...
            weight[:, 1::2] = cos_weight
            self.register_buffer('weight', weight)

        def forward(self, position):
            torch.index_select(self.weight, 0, position)

I follow the method of FAIRSeq and my implementation is as follows:
    class TrigonometricPositionalEmbedding(torch.nn.Module):
        def __init__(self, embedding_number, dimension, padding_idx):
            ...
            weight[:, 1::2] = cos_weight
            self.weight = weight

        def forward(self, position):
            torch.index_select(self.weight, 0, position).detach()

So I am curious about the difference between the two method, or what is the difference between registered buffer and detached parameter?
(The question was also posted in PyTorch Forums but no answer yet.)


texts: Validation set after hyperparameter tuning

texts: Let's say I'm comparing few models, and for my dataset I'm using train/validation/test split, and not cross validation. Let's say I'm completely done with parameter tuning for one of them and want to evaluate on the test set. Will I train a new model, on both the training and the validation datasets combined with the best configuration, or will I just run the same model on the test data?


texts:
My model-

#define model
model = Sequential()
model.add(Dense(128, activation='relu', input_dim=n_input_1))
model.add(Dense(64, activation='relu'))
#model.add(Dense(32, activation='relu'))
#model.add(Dense(16, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse',metrics=['mse'])

Now I am trying to tune the hyper parameters by this code-

from sklearn.model_selection import GridSearchCV
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# define the grid search parameters
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1,
cv=3,scoring='neg_mean_absolute_error')
grid_result = grid.fit(scaled_train,y_train_c)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

I am getting this error-

TypeError                                 Traceback (most recent call last)
<ipython-input-10-c6e1e39d878e> in <module>
      9 param_grid = dict(batch_size=batch_size, epochs=epochs)
     10 grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3,scoring='neg_mean_absolute_error')
---> 11 grid_result = grid.fit(scaled_train,y_train_c)
     12 # summarize results
     13 print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\model_selection\_search.py in fit(self, X, y, groups, **fit_params)
    631         n_splits = cv.get_n_splits(X, y, groups)
    632
--> 633         base_estimator = clone(self.estimator)
    634
    635         parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\base.py in clone(estimator, safe)
     58                             "it does not seem to be a scikit-learn estimator "
     59                             "as it does not implement a 'get_params' methods."
---> 60                             % (repr(estimator), type(estimator)))
     61     klass = estimator.__class__
     62     new_object_params = estimator.get_params(deep=False)

TypeError: Cannot clone object '<keras.engine.sequential.Sequential object at 0x0000023DD4D5F488>' (type <class 'keras.engine.sequential.Sequential'>): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

I have no idea whats causing this issue. I am implementing a MLP for time series forecasting.


texts:
I have generated a model for predicting the future input trend with a sample data using linear regression in Knime. I want to validate the model using a different data set.

Suppose the data set I used for creating the model came from the sensor of device A. The prediction is accurate for the dataset from device A. Lets say I saved the model as a PMML file  called A. Can I use the same PMML file A to predict values of Device B (Values of B are not comparable). If not, should I create a different model for all data set that I have?

Question

How do I combine all the models generated so that it predicts any given data set? Is it possible?


texts: Sum up counts in a data.frame grouped by multiple variables

texts:
This is a snippet of the dataset I am currently working on:

> sample
    name sex count
1  Maria   f    97
2 Thomas   m    12
3  Maria   m     5
4  Maria   f    97
5 Thomas   m     8
6  Maria   m     4

I want to sum up the counts grouped by name and sex to finally get this data.frame:

> result
    Maria Thomas
f   194      0
m     9     20

I wrote a simple loop to iterate over the rows and sum up the counts:

result <- matrix(0, nrow=2, ncol=2)
colnames(result) <- unique(sample$name)
rownames(result) <- unique(sample$sex)

for (i in 1:nrow(sample)) {
    sex <- as.character(sample[i,"sex"])
    name <- sample[i,"name"]
    count <- sample[i,"count"]

    result[sex, name] <- result[sex, name] + count
}

Is it suitable to do it this way? Are there any other ways to do it in a more elegant / shorter fashion?

Edit:

I already tried it with aggregate, but the output is in a different format:

> aggregate(sample$count,by=list(sample$name,sample$sex),sum)
  Group.1 Group.2   x
1   Maria       m   9
2  Thomas       m  20
3   Maria       w 194


texts: Neural Turing Machine - how the shifting works?

texts:
I am reading the paper on Neural Turing Machine, page 9, and got stuck on just 1 pesky point.
I can't understand how the shift is done:

Each head emits a shift weighting $\vec{s_t}$ that defines a normalized distribution over the allowed integer shifts (Authors mean entries in s will add up to 1). For example, if shifts between -1 and 1 are allowed, $\vec{s_t}$ has three elements corresponding to the degreee to which shifts of -1, 0 and 1 are performed.
If we index the $N$ memory locations from 0 to $N-1$, the rotation applied to $\vec{w_t^g}$ by $\vec{s_t}$ can be expressed as the following circular convolution:
$$w_{new}(i) \leftarrow \sum_{j=0}^{N-1}w^g_t(j)s_t(i-j)$$
where all index arithmetic is computed modulo $N$.

The authors use notation $w_t(i)$ to signify i'th element in the vector.
I cannot understand how $s_t$ is applied to the $w^g_t$ and what makes it possible to actually perform shift, say by +1

For example, assume we are working on the the i'th element that is at index 3 and $w^g_t$ has 5 entries.
Like in the paper, let's allow $s$ to be able to perform a shift of -1, 0 or 1. Thus, $s$ has 3 dimensions. Now, I want to shift every element in $w$ by +1 forward.
I have a feeling I will get out of bounds error, and modulo won't help too much. We can unfold this computation for the 3rd element:
w[3] = wg[0]*s[(3-0)%5] + wg[1]*s[(3-1)%5] + wg[2]*s[(3-2)%5] + wg[3]*s[(3-3)%5] + wg[4]*s[(3-4)%5]

when simplified this means:
w[3] = wg[0]*s[3] + wg[0]*s[2] + wg[0]*s[1] + wg[0]*s[0] + wg[0]*s[4]

As you can see, I got s[3] at the start and s[4] at the end. ...But we've just agreed s will only have 3 entries {-1, 0, 1} and this means out of bounds error.
And, I can't really understand how what we've done would "shift" the entire $w^g$ into $w_{new}$ once we do it for all 5 entries (we've done the above just for 3rd entry). Can someone give me the intuition for it as well?


texts: How does Mutual Information handle background overlap

texts: I have been reading about mutual information in Image Registration. It's in the literature that MI is better able to handle the cases with the large background where anatomical structures are not aligned than entropy. Can someone provide intuitive information regarding how can MI handle such cases? Thanks in advance


texts: Reclassification machine learning measuring fatigue

texts:
I am very new to data science/ML field, so I apologize in advance if my logic may seem confusing.

I would like to know if it's possible to perform reclassification on a dataset?
So the process I want to do goes like this:

I want to classify physical fatigue using Machine Learning.
I will make the person perform some exercises while collecting sensor data.

I'm going to collect sensor data from smartphone (acceleration, gyroscope, etc) and also EMG, Heart Rate and I'll ask the person to state how tired they are feeling from 1-10 (Rating for Perceived Exertion or RPE).

Then I'll take EMG, Heart Rate data as independent variables and the RPE as dependent variable or labeling method (No-fatigue, mild , moderate  and heavy represented as 1,2,3,4) and perform a logistic regression for classification.

After I get the results I want to use the labels obtained from the logistic regression classification and take the smartphone sensor data as my features and classify my data based on those labels.

I would like to know if this approach would correct or if there is any other way.

I want to do something similar like what the authors did on this paper:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5923826/

http://dcollection.hanyang.ac.kr/public_resource/pdf/000000107948_20200302195433.pdf  Pages [78-81 and 93-99]


texts: PyTorch: Train without dataloader (loop trough dataframe instead)

texts:
I was wondering if it is bad practice to instead of using built in tools such as dataloader just loop trough each row in a pandas df. Lets say I am doing text classification and my training loop looks like:
net.train()

for i in range(0, 5):
    epoch_loss = 0.0
    print('epoch:', i)

    for idx, row in train_df.iterrows(): #is this bad?
        if idx % 1000 == 0:
            print(idx)
        total_loss = 0
        net.zero_grad()
        text_1, text_2 = row['sentence1'], row['sentence2']

        encoded1 = tokenizer.encode_plus(text_1, return_tensors="pt").to(device)
        encoded2 = tokenizer.encode_plus(text_2, return_tensors="pt").to(device)

        cosine_dist = net(
            encoded1['input_ids'], encoded1['token_type_ids'], encoded1['attention_mask'],
            encoded2['input_ids'], encoded2['token_type_ids'], encoded2['attention_mask'],
        )

        gold_label = torch.tensor([row['score'] / 5.0]).to(device)

        raw_loss = loss(cosine_dist, gold_label)
        raw_loss.backward()
        optimizer.step()

        epoch_loss += raw_loss.item()

    print(epoch_loss)

Will this just take longer time, since it does a backprop for each training sample? If using batch training, with e.g batch_size=32, will that only backprop once for 32 samples, meaning it should be faster, but optimizing the average loss in the batch?
Any feedback is appreciated! :)


texts: Math behind L2 Regularization for Logistic Regression

texts: I read that L2 regularization in logistic regression creates a sort of sphere that limits the choice of $w$ weight, but why does this happen?


texts: How to create a dummy model in Tensorflow

texts:
I am a newbie in Machine learning.

I found this example using tflearn somewhere. It is the part of the program where we initialize a dummy model before training it. In the example, there were 4 input layers and 2 output layers. I didn't understand any terms but I ran that program and it worked.

def neural_network_model(input_size):
    network = input_data(shape=[None, input_size, 1], name='input')

    network = fully_connected(network, 128, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 256, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 512, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 256, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 128, activation='relu')
    network = dropout(network, 0.8)

    network = fully_connected(network, 2, activation='softmax')
    network = regression(network, optimizer='adam', learning_rate=LR, loss='categorical_crossentropy', name='targets')
    model = tflearn.DNN(network, tensorboard_dir='log')

    return model

I wanted to reuse this function to create a model that I will train with a data having 24 input layers and 4 output layers but it gave me the following error:

  Cannot feed value of shape (64, 4) for Tensor 'targets/Y:0', which has
  shape '(?, 2)'

Can someone explain me what does all of the term above mean ? And what should I change to adapt with my 24 by 4 layer?


texts: How are dynamic memory networks employed in sequence to sequence modelling

texts:
Dynamic Memory networks are described here
. I understand what is going on for question answering task but when it comes to sequence to sequence modeling, they describe it in 4th paragraph of 2.4 answer module.

  In the sequence modeling task, we wish to label each word
  in the original sequence. To this end, the DMN is run in
  the same way as above over the input words. For word t,
  we replace Eq. 8 with $e^i = h^{i}_{t}$
  . Note that the gates for the
  first pass will be the same for each word, as the question is the same. This allows for speed-up in implementation
  by computing these gates only once. However, gates for
  subsequent passes will be different, as the episodes are different.

I could not understand how the global gates are computed and why would they be equal for each word for first pass. They are going to be computed as a function of question vector (same) and word vector (different for each word). This is how vague i am on what is going on here. Can somebody explain how the piece in the paragraph mentioned come together?


texts: Improving automated ingestion system using Machine Learning and/or NLP

texts:
I'm working on a automated ingestion system which takes a PDF or doc file or a URL. It then parses the file and get me the required text in a json format but there are some error and there are few things that I want to capture differently or add.

As to give you an idea what I really want to do is: Let's say I've a URL of Step by Step Instructions on doing something. I want to scrape/parse this web-page and convert these instructions into smaller and meaningful instruction, so Amazon Alexa or any other voice assistant can speak and give the user instruction through voice.

The output I'm getting by scrapping this webpage is not in proper format. Till now I've been manually improving by adding the desired words and punctuation. But the number of these URL will increase over time so it is not possible to give so much time on improving the instructions by using a human to add the missing words or break a sentence into two meaningful instructions. So how can I automate this part using ML/DL so that this system can be scaled.

So can you guide me on how can I leverage Machine Learning to improve this ingestion system?

Thanks in advance. Let me know if you have any questions regarding this or If I've not made something clear which is required to answer it correctly!!!


texts: How can I create convolutions or linear layers that operate on vectors rather than scalars in pytorch?

texts:
Consider an nn.Linear(2,3) layer transform like the one below.  It uses a 2x3 matrix of scalar weights to create a weighted sum for each scalar element in the output.

Now suppose that instead of scalars I want to operate on 1d vectors (e.g. word embeddings).  So in this case a and b would be vectors and the weights w would each be square matrices.

Can one of the pre-built layer types in nn accomplish this (for linear or convolutions)?  Or is there some way to transform my data to accomplish the equivalent of this with the standard layer types?

thanks!

EDIT: I'm aware that for a given layer like the example I could just do all of the permutations of matrix operations and store the weights for them myself.  What I'm wondering is if there is a better way.

EDIT: Ok, I think what I want does not exist but let me try to clarify on the hopes that maybe there is some similar operation...

Ignoring the bias vector nn.Linear(2,3) is doing the following:

What I'm looking for is the analog of this where a, b, c, ... are vectors and the weighting is a linear transformation of those vectors.  I apologize if my notation is off here but, written as row vectors:

This means that we'd need a 3 dimensional set of weights, not a 2d as in nn.Linear. Again, I guess this is just a different operation entirely but I'm wondering if this kind of thing has a name or if there is some reason it  isn't normally needed (e.g. some transformation of the data that really is equivalent to this).


texts: ROC curves/AUC values as a performance metric

texts:
I want to plot ROC curves using R. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a label vector. The column names of prediction columns are ccs,badaI,badaII and the column name of label vector is value. I am using ROCRlibrary for this as:

library(ROCR)
pred1 <- prediction(df$ccs,df$value)
roc <- performance(pred1,"tpr","fpr");
pred2 <- prediction(df$badaI,df$value)
roc2  <- performance(pred2,"tpr","fpr")
pred3 <- prediction(df$badaII,df$value)
roc3 <- performance(pred3,"tpr","fpr")
auc<- performance(pred,"auc")
auc = round(unlist(auc@y.values),2)
auc2<- performance(pred2,"auc")
auc2 = round(unlist(auc2@y.values),2)
auc3<- performance(pred3,"auc")
auc3 = round(unlist(auc3@y.values),2)
plot(roc,col="black",lty=1, lwd=4, cex.lab=1.5,axt="n")
axis(1,cex.axis=1.0);axis(2,cex.axis=1.0)
plot(roc2, add=TRUE,col="black",lty=3, lwd=4)
plot(roc3, add=TRUE,col="black",lty=2, lwd=4)
abline(0,1,col="gray60")
legend(0.3,0.30,c(paste0("CCS, ","AUC = ",auc),paste0("BADAI, ","AUC = ",auc2),paste0("BADAII, ","AUC = ",auc3)),
      lty=c(1,3,2), col=c('black','black','black'), lwd=4,cex=1.4,bty="n")

While using the above code,I am getting following plot:

I have doubt as:
 While looking at the data, It is obvious that ccs and badaII should have higher AUC values than badaI, but the results are somehow opposite. Can anyone help me in understanding why it is behaving like this?
The dput of the data used, df is:

structure(list(ccs = c(0.16, 0.04, 0.18, 0.09, 0.14, 0.14, 0.04,
0.04, 0.08, 0.76, 0.03, 0.03, 0.68, 0.06, 0.83, 0.15, 0.07, 0.02,
0.93, 0.22, 0.28, 0.11, 0.05, 0.01, 0.17, 0.15, 1, 0.13, 0.23,
0.44, 1), badaI = c(0.61, 0.11, 0.53, 0.79, 0.75, 0.82, 0.57,
0.67, 0.4, 0.95, 0.49, 0.61, 0.97, 0.52, 0.98, 0.7, 0.03, 0.18,
0.85, 0.94, 0.9, 0.77, 0, 0.37, 0.47, 0.88, 0.99, 0.55, 0.86,
0.96, 0.99), badaII = c(0.32, 0, 0.27, 0.12, 0.33, 0.12, 0.56,
0, 0.32, 0.18, 0.18, 0.11, 0.18, 0.54, 0.37, 0.33, 1, 0.39, 0.29,
0.11, 0.32, 0.53, 0.25, 0.21, 0.15, 0.16, 0.85, 0.31, 0.44, 1,
1), value = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1)), .Names = c("ccs",
"badaI", "badaII", "value"), row.names = c(NA, -31L), class = "data.frame")

UPDATE
I use below figure to further explain my intuition. This figure is drawn using the same predictor values.

According to value column, observation with numbers 15, 27, 30, 31 have labels as 1 and the remaining observations have 0 value. While looking at above figure it is clear that CCS or badaIIare best in discriminating the difference between 0 and 1 as compared to badaI, which always provide higher predictor values.In other words, I mean with badaI, it is difficult to predict 1 as its values are higher for both 0 and 1.

I am not able to correlate my intuition with the ROC plot. @TBSRounder, I understood what you have mentioned, but I need to support the above figure with the ROC plot, which I find disappointing. Can anyone help me to correlate the above figure with the ROC plot?


texts: Feature Selection before modeling with Boosting Trees

texts:
I have read in some papers that the subset of features chosen for a boosting tree algorithm will make a big difference on the performanceso I've been trying RFE, Boruta, Clustering variables, correlation, WOE & IV and Chi-square
Let's say I have a classification problem with over 40 variables, best results after a long long time testing :

all variables for Lightgbm (except of one variable with high linearity)
I removed correlated variables for Xgboost (around 8 correlated ones)
I removed variables based on ElasticNet model for Catboost (around 7 ones)

My question is : what's the proper way to choose the candidates variables for modeling a boosting tree (especially for Lightgbm) ?
I'm using R if there is any suggestion for packages ?


texts: Can bidirectional RNN use variable sequence length?

texts:
A bidirectional RNN consists of two RNNs, one for the forward and another for the backward sequential directions, which outcome is concatenated at each time step. Would this configuration restrict the model to always use a fixed sequence length? Or would it still work as the unidirectional RNN, which can be applied to any sequence length?

This question was raised because the bidirectional architecture merges the output of both forward and backward RNNs at each time step. Thus, if the sequence length was 4, the outputs of both forward and backward RNNs would merge this way: 1th forward with 4th backward, 2nd forward with 3rd backward,... 4th forward with 1st backward. However, if a different sequence length was used this merge order would be modified:

Let's say the network was trained with sequence length 4, but at test time a sequence length of 5 was used. Merge would be: 1th forward with 5th backward, 2nd forward with 4th backward... 5th forward with 1th backward. Would this shift in merge order negatively affect the bidirectional RNN performance?


texts:
I have a dataset containing features of different engines showing when they failed. I want to build supervise learning model to predict whether an engine with a certain mileage is going to fail or not. Unfortunately, I have data of engines that they never fail and I need to add them to my dataset (it is true these engines never fail but they will help model to have a better view on lifetime an engine).
Could any one help me how to add these  non-failure data to failure one.

Best,


texts: Is there a machine learning framework that supports partial evaluation ie can return a function?

texts:
Is there a machine learning framework that supports partial evaluation?

For example:

We train on [model, year, km, ..., colour, price].

Today we call
get_prediction(model, year, km, ..., colour)
It returns the value of price, ie a number (or a probability for a label).

Say I would like to call
get_prediction_function(model, year, ..., colour)
It should return a function price(km).  (That function returns a number.)

Essentially we are pulling out a dimension or two.

In theory of course it gives us nothing, it just postpones some calculation.  (In practice this small specialised object is very useful in some production applications.)


texts: Getting kinds on features that has high confidence on classfication

texts: Is there any way get a constraint on features (e.g., ear longer than 100cm and body weight>20kg and ...) or (...)) where it could say 90% guarantee it is a dog or cat? Say if we want to classify dog/cat.


texts: Reason for better performance of variants of SGD when local minimas of Neural Nets are equivalent?

texts:
From some neural net article I read that if you scale up the Neural Net architecture the differnce in different local minimas in the loss surface diminishes. Essentially all local minimas become equivalent.

If that is the case then why do different variants of SGD (like Adagrad, ADAM etc.) works better than plain SGD? I believe reason for using these variants of SGD is to solve "bad" local minima problem, but if all local minimas are more or less the same then what is objective of using these variants?


texts: Neural network recommendations if only few features

texts:
Can there be some general recommendations for architecture of neural network if there are only a few features, say 2-5 features?

What should be the number of hidden fully connected layers here? How many neurons may be there in each layer? Do number neurons in different layers need to stay constant (e.g. 16,16,16) or rise (e.g. 8,16,32) or fall (32,16,8) or rise and fall (e.g. 16,32,16) or fall and rise (e.g. 32,16,32)?


texts:
I have got a X = mx3 array of input data and y = mx1 array of desired outcome data. Now I want to normalize the features accordingly: X[:, 1] holds size of a house, X[:, 2] holds number of rooms, X[:, 0] = 1 is just a constant term. I proceeded as follows:
sigma = np.std(X[:, 0])
mean = np.mean(X[:, 0])
X[:, 0] = (X[:, 0] - mean)/sigma

sigma = np.std(X[:, 1])
mean = np.mean(X[:, 1])
X[:, 1] = (X[:, 1] - mean)/sigma
X = np.c_[np.ones((p,1)), X]

I determined the mean value and the standard deviation of each column separately, does this procedure seem valid?


texts: Is vanishing KL a problem in vision based VAEs as well?

texts:
I came across some work on the problem of a vanishing KL contrbution in Variational Auto Encoders Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing. This work particularly is in the NLP space where they use recurrent neural networks to model sentences which yields to the vanishing KL term problem.
I wonder if there is a similar issue when working with convolutional networks and images.
Cheers


texts: How to fill in missing value of the mean of the other columns?

texts:
I had a movie dataset including 'budget' and 'genres' attributes.

I'd like to fill in the missing value of budget with the mean budget of each genre.

I first create two dataframes with or without budget.

BudgetNull = data[data['budget'].isnull()]
BudgetNotNull = data[data['budget'].notnull()]

Then, calculate the mean budget of each genre based on the BudgetNotNull dataset.

budget_of_genre = BudgetNotNull.groupby('genres')['budget'].mean()

Finally, I'd like to fill in the budget of BudgetNull based on its genre.

For instance, 'Marine Boy' is a action movie, therefore, fill in budget_of_genre['Action'].

How do I do this via for loop? Or there's other ways?


texts: Relation between amount of training samples and model depth?

texts:
When I add more hidden layers to my CNN (e.g. Dense Layers) it seems that the model needs more training samples to produce good results for classes with few training samples. In the single layer case the model provided better results, even for classes with few samples. (multi-class text classification with >10 classes)

Is there any evidence that my assumption is correct?


texts: Why do we need to use sysfunc when we call a SAS function inside a SAS macro

texts:
I saw this piece of code in my project:

%let num = test;
%let x=%sysfunc(trim(&num));

Why could not  I write:

%let x= %trim(&num);

Why did I need to use sysfunc?

Under what circumstances can I call a function inside a macro without using sysfunc?


texts: differences in NN optimization algorithm between Matlab and Scikit-Learn

texts:
I'm trying to build a neural network to estimate the responses of a nonlinear system. My database contains 1000 load cases (inputs) and maximum responses (outputs) from numeric simulations.
I'm using Python Scikit-Learn. I've run a gridsearch on the hyperparameters but the NN does not provide reliable results. I ended up with two hidden layers, 20 and 30 neurons, relu for activation function.
Not anything like what I've seen on articles and tried myself with Matlab: good results with one hidden layer, 20 neurons, logistic for activation function.
I wonder what could be so different between Matlab and Scikit-Learn to yield such a difference in the net architecture and its performance. Can it be the performance of optimization algorithms available in each?


texts:
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.

I work for a business, we sell certain products.
We are running a trial currently on a a subset of these products where we are trying to impact on the margin we make on these products by altering (on purpose) which sales channel we sell them on. However, this may not work in all cases, e.g. we may stop selling the item because it is not sellable on the new sales channels.

I'm trying to do the following:

I have a list of productids where we applied this mechanism
I know how much we sold before and after the trial started, i know the margin made on each sale
I know how much we have received into stock from the item before and after the trial
I know how much we had on stock when we started the trial

I need advice on the following:
* Am i right to assume that the ratio of margin of total sales vs. the units we have receied into stock is the right answer? So if the margin is 10 before the trial and 20 after, but we received 2 items before and 5 after, we are worse off?

How should i involve the stock amount we had on sale when we started the trial in my calculations?
Is there any available reading i should look into to educate myself?
Last question: provided i get answers on the above questions, i'll be able to flag each of my products with a "SuccessfulTrial" flag - true or false. Based on that, i can build some regression to find which product specifics have an impact on the success of this mechanism. Is my thinking right?

Thanks for any answers,


texts: How to use hunspell_stem(or a ordinary function) with tm_map?

texts:
Recently I've been doing some text mining and the stemming processing is taking a lot of time (I'm working with portuguese language).

I found the hunspell package, that provides spell checking, corrections and stemming.

Also, a friend developed another stemming function that I've been studying (RSLP: Removedor de Sufixos da Língua Portuguesa).

But both were made to work on a char vector, and I need to make them work with the tm_map function that read a corpus.

I know that to make them work, I need to split the documents on the corpus into single words, but I don't know how.


texts: Are there any differences in the results of training the deep learning algorithm (CNN) using a device with a GPU and using any device with a cloud service such as AWS and Floydhub?


texts: Guiding principle of Neural Network structure building

texts:
I study Neural Networks and I pretty much understand the logic of the structure with layers, activation functions and connections. But a fundamental question is not clear: how does one put together the actual structure from the several possible combinations? It seems me that most people just combine these quite randomly without any proper reason (or with brute force "grid search" in better cases).

I've read several tutorials but nobody explained the reason of choosing "sigmoind" activation function over "tanh" for instance, not to mention which one to pick in case of different activation functions in different layers. I stress that I understand the working of these functions by themselves, but the logic of the order of them is quite a puzzle.


texts: Layman's explanation of when to use which smoother algorithm/technique: FFT, loess, Savitzky-Golay, etc

texts:
As an analytics practitioner, I frequently come across noisy data (e.g. IoT data). When building a model or machine learning algorithm, it can be advantageous to smooth this data. Over the years, I have come across a variety of smoothing techniques, but have never been able to synthesize succinctly when to use one technique over another.

I am hoping someone can shed some light on any/all of the following techniques. Even better if the response can be kept at a high-level / layman's explanation to facilitate communication to a non-technical audience.

Fast Fourier Transform (FFT)
LOESS / LOWESS
Savitzky-Golay
Gaussian Filter
Moving average
Splines
Kernel Smoothing

In particular, are there conditions/situations in which one should not use one or more of the above methods? Feel free to add other methods/techniques to the arsenal.


texts: Eigenfaces VS Deep neural network for classification

texts:
I just saw a video from Washington University where professor Steve Brunton explain how to use Eigenfaces for classfication e.g image recognition.

Eigenfaces is really simple and does not require any training or parameter tuning etc. He show that it took just few seconds and then it's done. Super fast.

Eigenfaces was solved by using Singular value decomposition.

$$A = USV^T$$
Eigenfaces classification

So what is best if I want to have some kind of simple image classification?
Should I spend time and energy to train a deep neural or should I focus on eigenfaces? Eigenfaces is legacy, but it's still not bad, right?

Is it easy to train a deep neural network with multiple pictures? Or is that a hard task? As I see other people doing deep learning, they just cook up someting and then they have solved world problems. Sounds too good to be true right?

In what situation do you prefer eigenfaces over deep neural network? No?


texts: What does the "dual" parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?

texts:
While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the dual parameter, which is described in the documentation as:

dual : bool, (default=True)

    Select the algorithm to either solve the dual or primal optimization problem.
    Prefer dual=False when n_samples > n_features.

Can someone explain it, preferably in ELI5 style?


texts: What does Logits in machine learning mean?

texts:
"One common mistake that I would make is adding a non-linearity to my logits output."

What does the term "logit" means here or what does it represent ?


texts: In predict next letter model I'm unable to run the code for words which has greater than 4 letters

texts:
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

dtype = torch.FloatTensor

char_arr = [c for c in 'abcdefghijklmnopqrstuvwxyz']
word_dict = {n: i for i, n in enumerate(char_arr)}
number_dict = {i: w for i, w in enumerate(char_arr)}
n_class = len(word_dict) # number of class(=number of vocab)

seq_data = ['make', 'need', 'coal', 'word', 'love', 'hate', 'live', 'home', 'hash', 'star']
# seq_data = ['nithin', 'anishh']

n_step = 3
n_hidden = 128

def make_batch(seq_data):
    input_batch, target_batch = [], []

    for seq in seq_data:
        input = [word_dict[n] for n in seq[:-1]] # 'm', 'a' , 'k' is input
        target = word_dict[seq[-1]] # 'e' is target
        input_batch.append(np.eye(n_class)[input])
        target_batch.append(target)

    return Variable(torch.Tensor(input_batch)), Variable(torch.LongTensor(target_batch))

class TextLSTM(nn.Module):
    def __init__(self):
        super(TextLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=n_class, hidden_size=n_hidden)
        self.W = nn.Parameter(torch.randn([n_hidden, n_class]).type(dtype))
        self.b = nn.Parameter(torch.randn([n_class]).type(dtype))

    def forward(self, X):
        input = X.transpose(0, 1)  # X : [n_step, batch_size, n_class]

        hidden_state = Variable(torch.zeros(1, len(X), n_hidden))   # [num_layers(=1) * num_directions(=1), batch_size, n_hidden]
        cell_state = Variable(torch.zeros(1, len(X), n_hidden))     # [num_layers(=1) * num_directions(=1), batch_size, n_hidden]

        outputs, (_, _) = self.lstm(input, (hidden_state, cell_state))
        outputs = outputs[-1]  # [batch_size, n_hidden]
        model = torch.mm(outputs, self.W) + self.b  # model : [batch_size, n_class]
        return model

input_batch, target_batch = make_batch(seq_data)

model = TextLSTM()

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

output = model(input_batch)

for epoch in range(1000):
    optimizer.zero_grad()

    output = model(input_batch)
    loss = criterion(output, target_batch)
    if (epoch + 1) % 100 == 0:
        print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.6f}'.format(loss))

    loss.backward()
    optimizer.step()

inputs = [sen[:3] for sen in seq_data]

predict = model(input_batch).data.max(1, keepdim=True)[1]
print(inputs, '->', [number_dict[n.item()] for n in predict.squeeze()])

Error:
ValueError                                Traceback (most recent call last)
<ipython-input-8-0f96696141d8> in <module>()
     52         return model
     53
---> 54 input_batch, target_batch = make_batch(seq_data)
     55
     56 model = TextLSTM()

<ipython-input-8-0f96696141d8> in make_batch(seq_data)
     31         target_batch.append(target)
     32
---> 33     return Variable(torch.Tensor(input_batch)), Variable(torch.LongTensor(target_batch))
     34
     35 class TextLSTM(nn.Module):

ValueError: expected sequence of length 3 at dim 1 (got 5)


texts: Training 128x128 autoencoders on 512x512 images, produces strange gridline after recombining

texts:
So I'm training an autoencoder that can recreate 128x128 images, so it can recreate any images by splitting them into 128x128 patches first, running it through the autoencoder, and having them combined with each other to form the original image.

I should be using other dimensions too but right now I'm testing this with 512x512 images. So :

   x = next_train_batch(25)

    for xx in range(0,4):
        x_index = xx*128
        for y in range(0,4):
            y_index = y*128

            this_image = x[:, x_index:x_index+128, y_index:y_index+128, :]
            sess.run(training, feed_dict={x_in: this_image, step_iter_global:step_iter})

This is how I'm doing it now. But this keeps happening:

True, this is just in the beginning of the training process. But I guess I wonder why the autoencoder is having a hard time learning the edges of the patches. These gridlines actually are still present in visualizations of certain layers in the autoencoder (the reason I'm doing this is because I'm trying new layers that could potentially improve autoencoders) even if the output image doesn't have it.


texts: How can I run Labeled LDA over one textual document?

texts: I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help me the steps of this task that I should follow. Otherwise, I will be thankful if you can provide me another python library to do so.


texts: Why use regularization instead of decreasing the model

texts:
Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also benefit the computational time and memory.

My guess would be that different regularization methods make different assumptions of the dataset. If so, what assumptions are made for the common regularizations (L1, L2, dropout, any other)

Thanks in advance!


texts: I am very new to ML and have limited knowledge about it. I am having issue in feature normalization process. I have understood from the post that we need to normalize the training features and scale the test/validation features with the training data. I am facing issue in the implementation as in my case my training samples have fixed dimension but the dimension of validation and test data is variable. So, I can apply zero mean unit variance for training data but I am not sure how can I normalize the validation/test data samples as the sample dimension/length is variable/not fixed.


texts: Can a term weighting function used in text retrieval be compared to one used in text classification?

texts:
I came up with a modified version of TF-IDF function for text retrieval task. I want to do retrieval experiments using Vector Space Model and compare my function to some of those proposed in the literature in terms of known metrics (such as MAP and interpolated precision at recall...etc).
In literature, there are many improved TF-IDF functions that where proposed for text retrieval task and text classification task.
Is it feasible to compare my function to those that were originally used for classification?
I know that the idea of representing documents as vectors of terms weights is the same in both cases. But, can I simply take a function that was mainly developed for improving classification, and apply it in a retrieval process to compare its results to my function's results? Of course, I don't mean those functions that use "class"-related or "category"-related features.


texts:
Can anyone explain why the following code produces input_t with a shape of (32,) instead of (,32), given the fact that inputs has a shape (100, 32)? Shouldn't input_t produce a vector with 32 attributes/columns?

import numpy as np

timesteps = 100
input_features = 32
output_features = 64

inputs = np.random.random((timesteps, input_features))

state_t = np.zeros((output_features,))

W = np.random.random((output_features, input_features))
U = np.random.random((output_features, output_features))
b = np.random.random((output_features,))

successive_outputs = [ ]

for input_t in inputs:
    output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b)
    successive_outputs.append(output_t)
    state_t = output_t


texts: Are Convolutional Neural Networks a one model, multiple output layers.?

texts:
I'm writing a university report on for the Toxici comment classification Kaggle competition, comparing different attempt made with different models and I want to know if Convolutional Neural Networks are One model, multiple output layers model.

ONMO are also known as multi-task learning, this approach would have one input layer, one set of hidden layers, and one output layer for each label.

On the other hand, CNN are several layers of convolutions with nonlinear activation functions like ReLU or tanh applied to the results. It uses convolutions over the input layer to compute the output rather than a fully connected layer, or affine layer.


texts: How to prevent pd.ewma from changing the index?

texts:
I am resampling my time series data using the following code:

bars = transactions.price.resample('4H', base = 3).ohlc()

This allows me to account for DST correctly. The output is:

timestamp
2017-12-31 07:00:00    12668.5
2017-12-31 11:00:00    13183.0
2017-12-31 15:00:00    13124.0
2017-12-31 19:00:00    13178.5
2017-12-31 23:00:00    13130.5

I now want to calculate moving average using pandas .rolling function and trying the following:

ma_rolling = pd.ewma(bars['close'], span=20, freq='4H')

This messes up the time interval, shifting it to even hours:

timestamp
2017-12-31 04:00:00    14006.251358
2017-12-31 08:00:00    13927.838526
2017-12-31 12:00:00    13851.275461
2017-12-31 16:00:00    13787.196295
2017-12-31 20:00:00    13724.649100

Is it possible to specify a base / shift in ewma functions and to preserve the original indices? The .rolling function does not act like this and preserves original indices.


texts: How to add RMSE value on a plot with ggplot

texts:
I added r2 value and the formula of the regression function but I also want RMSE value on my plot, maybe I need to add something but I could not see a proper answer to this question neither here nor google...
ggplot(data = AGB.rf$pred) +
  geom_point(mapping = aes(x = pred, y = obs, color = pred, shape=1))+
  geom_smooth(mapping = aes(x = pred, y = obs), method="lm", se = FALSE)+
  stat_cor(aes(x = pred, y = obs, label =  ..rr.label..),label.y = 3000)+
  #stat_regline_equation(aes(x = pred, y = obs),label.y = 2700)+
  labs(title = "Predicted Values vs. Observed Values with Random Forest Model", x = "Predicted AGB", y="Observed AGB")+
  theme_light() + scale_shape_identity()


texts: What is the relationship between correlation ratio and one-way Anova?

texts:
According to the answer to this post, it is recommended to use one-way anova to compute the dependence between a categorical and a numerical variable.

Besides, the second answer to this post says that:

  The most classic "correlation" measure between a nominal and an
  interval ("numeric") variable is Eta, also called correlation ratio,
  and equal to the root R-square of the one-way ANOVA (with p-value =
  that of the ANOVA).

I would appreciate if you could let me know:

how the correlation ratio is computed using one-way anova?
why this kernel in the following code used np.log(1./anova['p']) when it intends to plot the correlation between a categorical and a numeric feature using one-way anova?

The code:

for cat in cat_features:
    group_prices = []
    for group in train[cat].unique():
        group_prices.append(train[train[cat] == group]['SalePrice'].values)
    f, p = scipy.stats.f_oneway(*group_prices)
    anova['feature'].append(cat)
    anova['f'].append(f)
    anova['p'].append(p)
anova = pd.DataFrame(anova)
anova = anova[['feature','f','p']]
anova.sort_values('p', inplace=True)
# Plot
plt.figure(figsize=(14,6))
sns.barplot(anova.feature, np.log(1./anova['p']))
plt.xticks(rotation=90)
plt.show()

Thanks in advance.


texts:
I heard someone say a neural network needs to sparsely compute the output.

I get what compute means, I get what a sparse matrix is, but what does sparsely compute mean?


texts: When we should use binning to redure noise? or How we find out we have noise?

texts:
I read several times which binning is helpful for reducing the noise of data. But how can we find out our data has noise? What if our data is clean and we reduce the accuracy of data?

Is there any method to measure noise of an attribute?
When should we do binning?


texts: Workng of LME model used for a set of category variable(s) and a continuous variable?

texts:
LME models are being used to analyze the effect of continuos data and category data. Is
this model  appropriate for checking the effect of two independent variables - one with continuous values and another with  nominal category (0, 1) on a dependent  variable? These models, it seems,  use the term random effects for category variables in the linear-mixed effect model. I wish I could get details how the regression coefficients are obtained and combined into the continuous variables effect  to predict the effect on Y dependent variable.


texts: Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

texts:
I'm very new to neural networks and have recently learnt about the loss functions used with neural networks.
This question is in regards to the mean square error metric, defined as (from the textbook I'm using):
$(\frac{1}{n})\sum_{i=1}^{n}(h_{\theta}(x^{i}) - y^{i})^{2}$
Where $h_{\theta}(x^{i})$ gives the predicted value for $x^{i}$ with model’s weights $\theta$ and $y^{i}$ represents the actual prediction for the data point at index $i$.
Looking up about this function online, I've seen different sources say different things. I can't seem to work out what n actually represents.
I understood it as representing the number of neurons in the output layer and hence you'd just be finding the difference between the actual neuron value and the predicted value of the network given the weights.
Some of the sources say it represents the number of training samples. If this is the case however, what does $h_{\theta}(x^{i})$ represent? Is it a sum of the output neuron values itself? Also if n is this, wouldn't that mean you'd have to run the function many times over all the training samples to minimize it? Whereas with the previous understanding of n, you could run it on certain samples and not all of them.


texts: Do all the Policy Gradient Methods work on the same principle/formula?

texts:
Do all the Policy Gradient Methods work based on the same formula?
If Yes, What is that formula?


texts:
I would like to implement machine learning algorithm in C++ without using any C++ machine learning library. So I'm writing this initializer function for generating zero matrices but can't figure out how can I accomplish this. I'm actually trying to write C++ code for simple logistics regression for now.

float * intializer_zero(int dimension){
    // z = wx + b.

float b = 0;
float w[dimension]= { };
return w,b;
}

It's throwing error "cannot convert 'float' to 'float' in return."
How can I write this initializer function in C++?


texts: How does Xgboost learn what are the inputs for missing values?

texts:
So from Algorithm 3 of https://arxiv.org/pdf/1603.02754v3.pdf, it says that an optimum default direction is determined and the missing values will go in that direction. However, or perhaps I have misunderstood/missed the explanation from the article, it doesn't say what exactly is the input.

For example, I have (parent) node A (with 50 inputs) splitting into node B and node C. Now, say of the 50 inputs there are 7 missing values.

The other 43 inputs are split into B and C accordingly. What I seem to be understanding is that, it will allocate the remaining 7 into B and C and determine which one gives a higher gain score; that will be the optimal direction.

However, given the 7 values are missing (Which means I don't know what are these 7 values), how does allocating missing values into any of the child nodes change the gain score, or rather minimize the loss function? This seems to suggest that Xgboost is inputting something for the missing values. I can't seem to find out what is Xgboost inputting for these missing values. I hope this question isn't too vague/general and easy.

Edit: I think "Missing values" may be a vague term. What I meant here is (From wiki) "In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation."

From the author himself (https://github.com/dmlc/xgboost/issues/21), he said
" tqchen commented on Aug 13, 2014
xgboost naturally accepts sparse feature format, you can directly feed data in as sparse matrix, and only contains non-missing value.

i.e. features that are not presented in the sparse feature matrix are treated as 'missing'. XGBoost will handle it internally and you do not need to do anything on it."

And,

" tqchen commented on Aug 13, 2014
Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically "learn" what is the best imputation value for missing values based on reduction on training loss."


texts:
I'd like to detect Dates in Texts. This should include strings like 'last friday', 'next week' and so on.
Unfortunately I couldn't find anything useful in the literature.
Does anyone know a good technique to solve this problem?
I don't have enough data to train a LSTM, so I think a grammar based approach would be great.
All ideas are welcome.
Thank you and best regards.


texts: Analyzing the search space of hyperparameter optimization

texts:
My goal is to train a CNN via transfer learning on a given dataset and to analyze and document the training process. I selected a few CNN architectures and hyperparameters to perform a random search.

For example, some of the hyperparameters are:

depth of the fully connected network (depth)
width of the fully connected network (width)
regularizer function used in the fully connected network (regularizer)
percentage of trainable layers (train_pcnt)
...

So my results look similar to this:

| val_acc | architecture | depth | width | regularizer | train_pcnt | ... |
| 0.863   | inception_v3 | 2     | 256   | l1          | 0.4        |     |
| 0.634   | densenet     | 1     | 32    | none        | 0.8        |     |
| 0.535   | resnet       | 3     | 128   | l1_l2       | 0.5        |     |

As a result of the random search, I found a model with which I'm very satisfied, but now I would like to analyze the search space. Ideally, I would be able to make conclusions like

"The lower the depth of the fully connected network, the better the accuracy.", or
"L1 regularization performs worse than L2 regularization.",

and so on. However, one can expect that most of these hyperparameters strongly interact with each other. Also, it can be assumed that neither the interactions between the hyperparameters nor the interactions between the hyperparameters and the accuracy are solely linear.

That makes it somewhat difficult to simply stuff all the data into a linear regression and to interpret the coefficients.

Is there still a way to analyze such a high dimensional and complex data with a linear regression? Or do there exist better approaches to perform a basic analysis of the search space?


texts: what is the largest network used for image recognition/segmentation?

texts:
What is the largest network (in number of params and layers) considered in the literature for image recognition/segmentation task? I am in particular interested in ResNet architectures.
Any recommendation for literature is appreciated.
For natural language processing, the largest models are of order of billions, such as Megatron-LM, or DeepSeed with Zero.
Is this also the case for image-classification?


texts: What is "experience replay" and what are its benefits?

texts:
I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (particularly, the AlphaGo paper), so I want to understand how it works. Below are some excerpts.

  First, we used a biologically inspired mechanism termed experience replay that randomizes over the data, thereby removing correlations in the observation sequence and smoothing over changes in the data distribution.

The paper then elaborates as follows:

  While other stable methods exist for training neural networks in the reinforcement learning setting, such as neural fitted Q-iteration, these
  methods involve the repeated training of networks de novo hundreds
  of iterations. Consequently, these methods, unlike our algorithm, are
  too inefficient to be used successfully with large neural networks. We
  parameterize an approximate value function $Q(s, a; \theta_i)$ using the deep
  convolutional neural network shown in Fig. 1, in which $\theta_i$ are the parameters (that is, weights) of the Q-network at iteration $i$. To perform
  experience replay, we store the agent's experiences $e_t = (s_t, a_t, r_t, s_{t+1})$ at each time-step $t$ in a data set $D_t = \{e_1, \dots, e_t \}$. During learning, we apply Q-learning updates, on samples (or mini-batches) of experience $(s, a, r, s') \sim U(D)$, drawn uniformly at random from the pool of stored samples. The Q-learning update at iteration $i$ uses the following loss
  function:

  $$
L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]
$$

What is experience replay, and what are its benefits, in laymen's terms?


texts: Demand prediction when having N customers and M products

texts:
I have a csv file (dataset) with the following information:
    date    customer_id    product_id    quantity
11/02/2019        11            2212         2
11/02/2019        12            1116         10
07/04/2020        24            0088         4
22/04/2020        06            2212         7

This dataset represets the sales of a shop during the last 5 years (on daily basis).
The dataset contains N customers (~100) and M products (~2.5k)
I want to build a machine learning model that can predict the sold quantity of each product, for each customer.
For instance, today, I have 10 customers, each bought aroung 20 products with different quantitys.
How can I form this problem? probably it's a regression task, but how can I build a model that can produce the needed outputs (predicting for each customer the product and its quantity)?
I saw some examples online, but all of them has a model for each item, (e.g. Demand Forecast using Machine Learning with Python), and in their cases they have few items.
In my case, since I have M products, should I build M models (prob. not)?


texts:
I'm reading Scikit-learn and I can't understand sample and feature. (n_samples, n_features)
Can anybody describe those by example?


texts: Find all potential similar documents out of a list of documents using clustering

texts:
I'm working with the quora question pairs csv file which I loaded into a pd dataframe and isolated the qid and question so my questions are in this form :

0        What is the step by step guide to invest in sh...
1        What is the step by step guide to invest in sh...
2        What is the story of Kohinoor (Koh-i-Noor) Dia...
3        What would happen if the Indian government sto...
.....
19408    What are the steps to solve this equation: [ma...
19409                           Is IMS noida good for BCA?
19410              How good is IMS Noida for studying BCA?

My dataset is actually bigger (500k questions) but I will use these questions to showcase my problem.

I want to identify pairs of questions that have a high probability of asking the same thing. I thought about the naive way, which is to turn each sentence into a vector using doc2vec and then for each sentence calculate the cosine similarity with every other sentence. Then, keep the one with the highest similarity and in the end print all those that have a high enough cosine similarity. The problem is this would take ages to finish so I need another approach.

Then I found an answer in another question that suggests to use clustering to solve a similar problem. So following is the code I implemented based on that answer.

"Load and transform the dataframe to a new one with only question ids and questions"
train_df = pd.read_csv("test.csv", encoding='utf-8')

questions_df=pd.wide_to_long(train_df,['qid','question'],i=['id'],j='drop')
questions_df=questions_df.drop_duplicates(['qid','question'])[['qid','question']]
questions_df.sort_values("qid", inplace=True)
questions_df=questions_df.reset_index(drop=True)

print(questions_df['question'])

# vectorization of the texts
vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(questions_df['question'].values.astype('U'))
# used words (axis in our multi-dimensional space)
words = vectorizer.get_feature_names()
print("words", words)

n_clusters=30
number_of_seeds_to_try=10
max_iter = 300
number_of_process=2 # seads are distributed
model = KMeans(n_clusters=n_clusters, max_iter=max_iter, n_init=number_of_seeds_to_try, n_jobs=number_of_process).fit(X)

labels = model.labels_
# indices of preferable words in each cluster
ordered_words = model.cluster_centers_.argsort()[:, ::-1]

print("centers:", model.cluster_centers_)
print("labels", labels)
print("intertia:", model.inertia_)

texts_per_cluster = numpy.zeros(n_clusters)
for i_cluster in range(n_clusters):
    for label in labels:
        if label==i_cluster:
            texts_per_cluster[i_cluster] +=1

print("Top words per cluster:")
for i_cluster in range(n_clusters):
    print("Cluster:", i_cluster, "texts:", int(texts_per_cluster[i_cluster])),
    for term in ordered_words[i_cluster, :10]:
        print("\t"+words[term])

print("\n")
print("Prediction")

text_to_predict = "Why did Donald Trump win the elections?"
Y = vectorizer.transform([text_to_predict])
predicted_cluster = model.predict(Y)[0]
texts_per_cluster[predicted_cluster]+=1

print(text_to_predict)
print("Cluster:", predicted_cluster, "texts:", int(texts_per_cluster[predicted_cluster])),
for term in ordered_words[predicted_cluster, :10]:
    print("\t"+words[term])

I thought that this way I could find for each sentence the cluster that it most likely belongs in and then calculate the cosine similarity between all other questions of that cluster. This way instead of doing it on all the dataset I will be doing it on far fewer documents. However using the code for an example sentence "Why did Donald Trump win the elections?" I have the following results.

Prediction
Why did Donald Trump win the elections?
Cluster: 25 texts: 244
    trump
    donald
    clinton
    hillary
    president
    vote
    win
    election
    did
    think

I know that my sentence belongs to cluster 25 and I can see the top words for that cluster. However how could I access the sentences that are in this cluster. Is there any way to do it?


texts: Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

texts:
Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of the original dataset remains in the resulting dataset. A common goal is keeping variance between 90% and 99%.

My question is: is it considered a good practice to try different values of the k parameter (size of the resulting dataset's dimension) and then check the results of the resulting models against some cross-validation dataset in the same way as we do to pick good values of other hyperparameters like regularization lambdas and thresholds?


texts: Number of parameters in Neural Network Language Model

texts:
The Neural Network language Model (NNLM) by Bengio et.al is a structure extensively used in machine translation, text summarization based on deep learning. What's the computational complexity of this model?

Knowing the complexity in terms of number of parameters helps in choosing the size of the training set and in determining what compute infrastructure is required.


texts: I trained a CNN model and a combined CNN-SVM model for classification. I wanted to compare their performance using ROC curve but I was confused which model is better. How to interpret the given ROC curves ?


texts: Prediction that a Employee can take a leave or not based on his previous track record

texts:
which prediction algorithm should I use to find the probability that an employee takes leave and come back on the specified period(i.e.does does not extend his leave).

Thank You.


texts: Ways to convert textual data to numerical data

texts:
I've been looking for ways to wrangle my data which contains both text and numerical attributes.

There are of course several algorithms for numerical data, but I am looking for suggestions regarding how to deal with textual data, for instance: for sorting based on K-means clustering and predicting missing data using kNN. I would really appreciate any thoughts regarding that. I am using scikit-learn.


texts: StandardScaler's mean and standard deviation for real-life data?

texts:
I've heard that we should use train dataset's scale for that of test data so they are in line with each other in terms of scale.
And I know we use transform() function for the test data preprocessing.
But I am wondering how I can apply the same logic to the real-life data set after we successfully save and load a model since I don't think we can use transform() function anymore.
From what I understand, there must be a way to "extract" mean and standard deviation from the train data set and to apply it for real-life model when using the model we saved already, but how can we do that?


texts: PCA method for feature selection - How to solve the raise Exception error ("Data must be 1-dimensional")?

texts:
#Função que permitirá rankear as features mais importantes em um barhplot
def ranks_PCA (x_train, y_train, features_train, RESULT_PATH='Results'):
    print("\nMétodo PCA")

    pca = PCA(n_components=58)
    pca.fit_transform(x_train)

    imp_array = np.array(pca.components_)
    imp_order = imp_array.argsort()
    ranks = imp_order.argsort()

    # Plot PCA
    imp = pd.Series(pca.components_, index=x_train.columns)
    imp = imp.sort_values()

    imp.plot(kind="barh")
    plt.xlabel("Importance")
    plt.ylabel("Features")
    plt.title("Feature importance using PCA")
    # plt.show()
    plt.savefig(RESULT_PATH + '/ranks_DT.png', bbox_inches='tight')

    return ranks

#Função para predição das features dos dados de teste
def predict_PCA(x_test_sel, k_vetor, y_train):
    model = decomposition.PCA()
    model.fit(k_vetor, y_train)
    y_predict = model.predict(x_test_sel)
    return(y_predict)

#Função que calcula o ranking dos dados de treinamento
ranks4 = frk.ranks_PCA(x_train, y_train, features_train, RESULT_PATH)

I have doubts if this implementation is correct to obtain more important features. When trying to run this code, I get the following error:

  Traceback (most recent call last): File "feat_test.py", line 235, in
  'Results/PDBbind2018_F58_Delta_pKd') File "feat_test.py", line 78, in
  run_experiment ranks4 = frk.ranks_PCA(x_train, y_train,
  features_train, RESULT_PATH) File "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\Scripts\feature_ranks.py", line 121, in ranks_PCA imp =
  pd.Series(pca.components_, index=x_train.columns) File
  "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\env\lib\site-packages\pandas\core\series.py", line 305, in
  init data = sanitize_array(data, index, dtype, copy,
  raise_cast_failure=True) File "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\env\lib\site-packages\pandas\core\construction.py", line 482,
  in saniti ze_array raise Exception("Data must be 1-dimensional")

Can anybody help me?


texts: If I understood correctly, "using results of PCA to select features" (as recommended in this answer) implies visually analysing bi-plots of first two principal components - i.e. the angle between a feature and a principal component.  This approach seems impossible with high number of features. Is there a way to interpret the loadings of principal components for automatic feature selection?


texts: Why training a Restricted Boltzmann Machine corresponds to having a good reconstruction of training data?

texts:
Many tutorials suggest that after training a RBM, one can have a good reconstruction of training data just like an autoencoder.
An example tutorial.
But the training process of RBM is essentially to maximize the likelihood of the training data. We usually use some technique like CD-K or PCD, so it seems that we can only say that a trained RBM has high probability to generate data which is like training data (digits if we use MNIST), but not correspond to reconstruction. Or are these two things just equivalent in some way?
Hinton said that it is not a good idea to use reconstruction error for monitoring the progress of training, this is why I have this question.


texts: Reinforcement learning for turn-based AI

texts:
For a side project I'm trying to build a (simplified) AI for Heroes Of Might and Magic, using (as a starting point) deep Q-learning. But I'm having trouble to understand how the "state space" is supposed to be represented.
Here's an idea of what the battle looks like

Roughly speaking:

there are 2 opposing armies
each armies has troops
each troop occupies an hex and contains a number of soldiers
the more soldiers, the more HP and attack a "troop" has
each troop attacks in turns, and during its turn it can either move, or move and attack an adjacent enemy troop
the army that manages to kill all enemy troops win

I've simplified this such that I reduced the board size and am only considering 1 troop per army.
However I am not sure how I should "formalize" the problem into a RL problem, especially concerning the "state" of the game.
For state, I can either:

have a state that exposes the Hit Point, position, attack, etc of the enemy and my own
have a state that is a NMP tensor that contains all the info of the board. Ie the tensor at position (0,0) would contain a 0-vector, but the tensor at position (1,2) where an enemy is would contain a vector of all the enemy statistics

It seems to me that the first solution would be simpler but complex to generalize to a variable number of enemy units
Second solution has all the infos but the state is getting big and very sparse. Also I'm not sure it's the best to input the state in terms of absolute positions, rather than relative position to the unit that has to take an action.
What do you think ? Are there any resources to understand better how such a problem should be transformed into a RL problem?


texts: Why I am having ValueError in this Linear Regression?

texts:
from sklearn.linear_model import LinearRegression

ClosePrices = data['Close'].tolist()
OpenPrices = data['Open'].tolist()

OpenPrices = np.reshape(OpenPrices, (len(OpenPrices), 1))
ClosePrices = np.reshape(ClosePrices, (len(ClosePrices), 1))

regressor = LinearRegression()
regressor.fit(OpenPrices, ClosePrices)

I am having the error

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

What is the solution?


texts: Comparing models when their performance may depend on a continuous variable

texts:
I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form X~A+B+C and M2 may take the form X~D+E.
However, I suspect that the performance of these models may depend on continuous variable F. e.g. for low values of F, M1 performs best; but for high values of F, M2 performs best.
My first thought was to partition the data into two sets corresponding to high and low values of F, then compare the models on each subset of the data. But I have several worries about this approach including that I may lose some information in converting a continuous variable into a discrete one.
Is it possible to (1) test if model performance is dependent on some variable and (2) compare models when observations for different values of F may follow different trends without discretizing F? If either is possible, what is the recommended approach?


texts: Are there any good out-of-the-box language models for python?

texts:
I'm prototyping an application and I need a language model to compute perplexity on some generated sentences.

Is there any trained language model in python I can readily use? Something simple like

model = LanguageModel('en')
p1 = model.perplexity('This is a well constructed sentence')
p2 = model.perplexity('Bunny lamp robert junior pancake')
assert p1 < p2

I've looked at some frameworks but couldn't find what I want. I know I can use something like:

from nltk.model.ngram import NgramModel
lm = NgramModel(3, brown.words(categories='news'))

This uses a good turing probability distribution on Brown Corpus, but I was looking for some well-crafted model on some big dataset, like the 1b words dataset. Something that I can actually trust the results for a general domain (not only news)


texts:
I am taking following Andrew Ng's machine learning course and have just gotten to the polynomial regression. I have a hard time understanding the intuition of how it works and how the model equation is formulated mathematically. I was wondering if anyone can provide me with better clarity on this subject.

In addition to this, in his example and many other examples I have seen online, only use uni-variate$(x_1)$ polynomial regression. Is it possible to use multiple features $(x_1, x_2, x_3, ...)$ when using polynomial regression? If so how would it be done? Is it also possible that one feature has a linear relationship while another does not?


texts:
I am trying to implement k-means clustering algorithm, but I am confused about calculating the distance and update(move) cluster centroids. For example, let's say that I have 2 features. One of them is weight={2,4,6,8,11,14,21} and the other one is height={4,6,7,8,9,12,14}. So, in the coordinate system my points are x1={2,4},x2={4,6},x3={6,7} and so on. Then, I initialize the cluster centroids randomly, doesn't matter how many there are for now, but they have coordinates too. Let's say μ1={4,2}. At this point, I understand how do I calculate distance with Euclidean distance.

My code for calculating distance:

def get_distance(x1,x2,s1,s2):
      return np.sqrt(np.power(s1-x1,2)+np.power(s2-x2,2))

Now I get a distance.

My first question is how cluster assignment step(first step in loop) will know which centroid assign to c(i).I mean,am I supposed to look at each centroid for understand which sample(x(i)) is close to it and then I should assign centroid to c(i), right?

My second question, let's say I got distances and I have c(1,2..,n) array now. Second step in algorithm which is called move(update) centroid step, we are calculate μ. According to formula, this μ is the average of points assigned to clusters so for example μ1=[x(3) + x(4) + x(6)] / 3. However, here our μ was a point in coordinate system, right? I mean, μ1 was {4,2}. How can this be possible? It's a point not a variable. It has coordinates. Well, if it will become a variable let's say μ1=5, how can I subtract ||x(i)-μ|| then? x is a coordinate.

My last question is very simple. For this example, I have two features weight and height. What is the maximum number of features that we can use in k-mean? Is it possible to use k-mean algorithms for many many features? For instance, my first feature is height and the second is weight and the third is width fourth and so on.

I hope, I explained my problem clearly. If not, sorry for the bad English. I think these three questions are independent questions, so you can answer one of them.

Thanks.


texts: K means visualisation after reducing dimensionality with PCA

texts:
In clustering ($K$ means, for example) when I have $N$ features and after creating the model (with this $N$ features) to visualize this model I need to reduce this $N$ dimensions into $2$ or $3$ dimensions, let's  say I will use (PCA) fro example.

My question is how I can analyze the results (grape with principal component)?

this is simple example :

Data without reducing :
AGE,GENDER,SPENT,SALARY,CAR,.....

Data after reducing dimension:

Principal Component $1$, Principal Component $2$, Principal Component 3

what do  PC1,PC2,PC3 means


texts: Downsampling audio files for use in Machine Learning

texts:
I'm trying to use the work (Neural Networks) done in this repo:
https://github.com/jtkim-kaist/VAD

It says this:

  Note: To apply this toolkit to other speech data, the speech data
  should be sampled with 16kHz sampling frequency.

I've got speech data at 48khz. I've read in places that reducing sampling rate is a complicated process, you can't just remove every nth datapoint, you have to filter things...

Is this necessary if I only intend to use the data in the Neural Network toolkit provided by the repo I linked?
If so, is there an industry standard method for changing sample rate?

I realise that it probably depends on what features are being used. However the feature that is used is this:

  MRCG (multi resolution cochleagra) concatenates the cochleagram features at multiple spectrotemporal resolutions

This is a ruddy complicated feature! Lets pretend we're just using a Melspectogram (unless you're willing to answer the question from the perspective of MRCG's).

Neural networks are likely to use features of a Melspectogram that we wouldn't think of. This makes me think it is unwise to train the Neural Net using downsampled speech data unless we intend to predict using 48khz data downsampled to 16khz forever after...

What do you think? Can I use my 48khz data - downsampled with no filtering - with the expectation that the model will work for prediction on real 16khz data?

And then for future readers sake, how about the other way? Say I had an 8khz file, could I increase the sample rate to 16khz without filtering?


texts: Multi-Label Loss function and model training

texts:
I'm working on Multi-Label problem i.e output can predict 1 or more label as an output and hence training data also have multiple labels.

Somehow I'm not able to map such ML model training. Please provide me the answer of below mentioned questions:

How loss is getting calculated for such type of problem?
How the ML model predicts?

I found one old thread Multi Label old query, but again Im not able to understand the solution.


texts: How to use historic data (granularity at day level) for ML modeling?

texts:
There is a scenario where I have to use historic data which is at the day level for the past 5 years.Actually it is water flow data, what quantity of water was flown on that particular day. I have to use this feature along with a few other features like material, coating, etc,. for EDA and prediction. I tried averaging it out but not useful.
Data is like this flow1,flow2, and flow3 (including other features not shown here) for each day on that particular route id. This continues for 5 years for many routes.

 I am not able to figure out how to consolidate this data so that I can feed it to the model.
I am trying to predict the corrosion in the pipeline.
 Any guidance will be helpful.
Thanks


texts: Modify keras_unet.utils.get_augmented to read images from disk

texts:
I want to train a cnn model on a large dataset (10K images and masks).
Currently, I am reading data in batches of BATCH_SIZE = 500 images, augment with
def get_augmented
    return get_augmented(
        x_train, y_train, batch_size=BATCH_SIZE,
        data_gen_args=dict(
            rotation_range=5.,
            width_shift_range=0.05,
            height_shift_range=0.05,
            shear_range=40,
            zoom_range=0.2,
            horizontal_flip=True,
            vertical_flip=False,
            fill_mode='constant'
        ))

The main loop looks like
STEPS_PER_EPOCH = 2
INNER_EPOCHS = 2
EPOCHS = 10

model = init_model(model_filename)  # Define the model
for epoch in range(EPOCHS):  # Number of times I want to iterate over the full dataset
    for batch_id in range(0,20):  # number of chunks of 500 images

        # reads images and masks from disk from batch_id folder
        x_train, y_train, x_val, y_val = read_a_butch_of_masks_and_images(batch_id)

        history = model.fit(
            train_gen,
            steps_per_epoch=STEPS_PER_EPOCH,
            epochs=INNER_EPOCHS,
            validation_data=(x_val, y_val),
            callbacks=[callback_checkpoint]
        )

model.save_weights(new_model_filename)

How to tell get_augmented to read the next batch of images from disk, rather than consume x_train, y_train so that it consumes little memory and goes over all the images??


texts: Interpreting the lm results in R for categorical variables

texts:
I created two categorical variables (Fac1 and Fac2) which can take values 1 or 2.  So, I have 4 combinations.  For the combination (1,1) I created 25 gaussian random numbers with mean 12.5 and variance 3, for (1,2) I created 25 Gaussian Random numbers with (37.5,3), for (2,1) another 25 with (62.5,3) and for (2,2) with (87.5,3).

I created the R program and ran the lm Model.

twodimfac = read.csv("TwoDimFac.csv")
twodimfac$Fac1 <- as.factor(twodimfac$Fac1)
twodimfac$Fac2 <- as.factor(twodimfac$Fac2)

A = twodimfac[,1:2]
x_with_int = lm(twodimfac[,3] ~ . , data = A)
summary(x_with_int)

The result I am getting is

Call:
lm(formula = twodimfac[, 3] ~ ., data = A)

Residuals:
    Min      1Q  Median      3Q     Max
-8.9121 -2.3011  0.2431  1.8639 23.5385

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  12.3355     0.6839   18.04   <2e-16 ***
Fac12        49.5762     0.7897   62.78   <2e-16 ***
Fac22        25.4342     0.7899   32.20   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.948 on 97 degrees of freedom
Multiple R-squared:  0.9812,    Adjusted R-squared:  0.9808
F-statistic:  2530 on 2 and 97 DF,  p-value: < 2.2e-16

I am interpreting Intercept, 12.33554 as a predictor for (1,1).  For other values, the program shows for (1,2) a value which is 12.3355 + 25.4342 = 37.7697, for (2,1) it predicts 61.91175 (which is sum of the intercept coeff and Fac12 coefficient) and for (2,2) it predicts 87.34596 (which is the sum of all coefficients).

How I can interpret the values?  If I have combination of more than 2 factor variables, how one can interpret the coefficients.  Is there any easy formula/function to get it, rather than guessing?


texts: Categorical data for sklearns Isolation Forrest

texts:
I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn.
Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features are categorical (font names, etc.)

I've got a bit too much to use one hot encoding (about 1000+ and that would just be one of many features) and I'm anyway looking for a more robust way of data representation.

Also, I want to experiment with other clustering techniques later on, so I don't want to necessarily do label encoding as it will misrepresent the data in euclidean space.

I have thus a two part question:

How will label encoding (ie. ordinal numbers) affect tree based methods such as the Isolation Forest? Seeing as they aren't distance based, they shouldn't make assumptions about ordinal data, right?
What other feature transformations can I consider for distance based models?


texts: full name of each variable in Plotmo graph

texts:
R function plotmo http://www.milbo.users.sonic.net/gallery/plotmo.html is used for plotting dependence plot from e.g. decision tree model.

In some cases, resulting graph does not print full names of features on each plot. What I am saying is, e.g. the "feature1" is automatically renamed "f1" or something similar.

I want full name. But plotmo just adjust automatically to short name (I believe for memory convenience, but not useful for perception).

Maybe Plotmo has parameter forcing it to produce full name? I just see -main parameter. But what values to set as to produce full original feature names for each generated plot?


texts: Random Forest vs K Nearest Neighbor as non linear classifier

texts:
When classes are non-linearly separable, which of the following methods performs better?
Choose correct one :-

Logistic Regression
Random Forest
K Nearest Neighbor Classification
Linear Regression over most important features

Trail : I think  Random Forest and K Nearest Neighbor Classification are the only non-linear classifiers here. But which one performed better? how to judge here?


texts: Splitting training and test set with financial data

texts:
I am using trees algorithms (decision tree, random forest and XGBoost) to forecast the sign of the returns in the stock market (classification).

I am using this article as a reference:

  http://rpubs.com/raaraa/412512

When splitting the training and the test set, the author uses the following R code:

index <- sample(1:nrow(stock_indicators))
size=0.2*nrow(stock_indicators)
test <- stock_indicators[index, ]
train <- stock_indicators[-index, ]

Is it correct to use a random set of dates to split the test and the training set with time series financial data?

It looks like look-ahead bias to me.

This would be the alternative:

train <-head(stock_indicators,round(0.70 * nrow(stock_indicators)))
test <- tail(stock_indicators,round(0.30 * nrow(stock_indicators)))

But with this split I cannot get significant accuracies even trying with different datasets.

Do you have any suggestions?


texts: Why does adding random pixels stop my model learning in cnn?

texts:
I am using a very simple model to classify a 224x224 RGB image.

For a test, I have labelled my images (2 labels "Green" or "Red", 2,000 images of each) based on colour of a single fixed pixel from upper part of image.

My simple model achieves very high accuracy and very low loss, until I add more "random" pixels to lower part of image.

Why is my model getting confused?

The model easily copes with:

But, when I add more pixels in lower part of image - it achieves only 50% accuracy and high loss:

These project is a bit facile but I am researching a simple image structure I can control.

def VerySimpleNet():
    model = keras.Sequential([
        keras.layers.Conv2D(16, kernel_size=3, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
return model

  EDIT 1

Is it possible to add a bonus to Data Science questions to encourage more answers?

  EDIT 2

I would really be interested if anybody was keen to have a more interactive chat(email,txt?) with me on this question.


texts: Forget Layer in a Recurrent Neural Network (RNN) -

texts:
I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from Colah's blog post "Understanding LSTM Networks":

where:

$x_t$ is input of size $m*1$ vector
$h_{t-1}$ is hidden state of size $n*1$ vector
$[x_t, h_{t-1}]$ is a concatenation (for example, if $x_t=[1, 2, 3], h_{t-1}=[4,
   5, 6]$, then $[x_t, h_{t-1}]=[1, 2, 3, 4, 5, 6]$)
$w_f$ is weights of size $k*(m+n)$ matrix, where $k$ is the number of
cell states (if $m=3$, and $n=3$ in the above example, and if we have
3 cell states, then $w_f=3*3 $ matrix)
$b_f$ is bias of size $k*1$ vector, where $k$ is the number of cell
states (since $k=3$ as the above example, then $b_f$ is a $3*1$
vector).

If we set $w_f$ to be:
        \begin{bmatrix}
    1 & 2 & 3 & 4 & 5 & 6 \\
    5 & 6 & 7 & 8 & 9 & 10 \\
    3 & 4 & 5 & 6 & 7 & 8 \\
    \end{bmatrix}

And $b_f$ to be: $[1, 2, 3]$

Then $W_f . [h_{t-1}, x_t] =$

$$
    \begin{bmatrix}
    1 & 2 & 3 & 4 & 5 & 6 \\
    5 & 6 & 7 & 8 & 9 & 10 \\
    3 & 4 & 5 & 6 & 7 & 8 \\
    \end{bmatrix}
.        \begin{bmatrix}
    1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \\
    \end{bmatrix}
=\begin{bmatrix} 91 & 175 & 133\end{bmatrix}$$

Then we can add the bias, $W_f . [h_{t-1}, x_t] + b_f=$

$$\begin{bmatrix} 91 & 175 & 133\end{bmatrix} + \begin{bmatrix} 1 & 2 & 3\end{bmatrix}=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$$

Then we feed them into a sigmoid function: $\frac{1}{1+e^{-x}}$, where $x=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$, hence we perform this function element wise, and get \begin{bmatrix} 1 & 1 & 1\end{bmatrix}.

Which means for each cell state, $C_{t-1}$, (there are $k=3$ cell states), we allow it to pass to the next layer.

Is the above assumption correct?

This also means that the number of cell state and hidden state is the same?


texts: Looking for an algorithm that compute similarity between a phrase and possible combination of tokens

texts:
I want to find similarity between a phrase and possible combination of tokens that may form the phrase.

For example, phrase = 'sea surface water'

Possible token = ['sea','surface land', 'surface water']

My approach:
First, i generate combinations, c = ['sea','surface land', 'surface water', 'sea surface land','surface land surface water', 'sea surface water']

Then I compute cosine similarity between the phrase 'sea surface water' and each of 'c' elements.

Finally, i use the max cosine similarity to find out the best combination of the tokens that form the phrase, in this case, the combination 'sea surface water' which is made of two tokens 'sea' and 'surface water'

My question, is there any algorithm out there that address the same issue?