texts: How to get accuracy, F1, precision and recall, for a keras model?

texts:
I want to compute the precision, recall and F1-score for my binary KerasClassifier model, but don't find any solution.

Here's my actual code:

# Split dataset in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(normalized_X, Y, test_size=0.3, random_state=seed)

# Build the model
model = Sequential()
model.add(Dense(23, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

tensorboard = TensorBoard(log_dir="logs/{}".format(time.time()))

time_callback = TimeHistory()

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])

And then I am predicting on new test data, and getting the confusion matrix like this:

y_pred = model.predict(X_test)
y_pred =(y_pred>0.5)
list(y_pred)

cm = confusion_matrix(Y_test, y_pred)
print(cm)

But is there any solution to get the accuracy-score, the F1-score, the precision, and the recall? (If not complicated, also the cross-validation-score, but not necessary for this answer)

Thank you for any help!


texts: Speed up convergence of GLM fitting in custom GLM code (using GD)

texts:
So, I've got a little bit of an unusual situation. I do ML on encrypted data, and the ecosystem there is relatively young, so I have to build my own infrastructure, and there are certain restrictions since the data is encrypted. In my problem setup, there is a server and a client, and after every gradient descent step on the client, the client must send the weights to the server for some processing that can only happen there.
Understandably, given network connection latency and all that, I want convergence ASAP.

I wondered if custom weight initialization based on the means and variances of the different features would speed up convergence? Doing a quick google search, it looks like all of the resources talk about NNs, but I wondered if there was any literature around this?

I'm pretty confident that simulated annealing would help but does anyone know of any sort of LR scheduling lit?

I think that Nesterov Accelerated gradients would help, but if anyone has anecdotal experience saying otherwise, I'd love to hear it.


texts: What is this formula, related to simple linear regression, called?

texts:
This is my first post here. I hope I can make myself clear. Right now I'm learning linear regression as part of an introduction class to machine learning.

After going over the steps in the simple regression formulae, I realized that I had been doing something similar in the past to construct lines. In Python:

data = [{'x': 1, 'y': 2}, {'x': 5, 'y': 7}, {'x': 6, 'y': 8}]
coeff = sum([d['x'] / d['y'] for d in data]) / len(data)

Here, we're calculating the mean ratio between the variables, which we can use as a coefficient for constructing a line. Does this method have a name, and how does it relate to simple linear regression?


texts: Using doccano for Aspect Based Sentiment Analysis annotation

texts:
Currently looking for a good tool to annotate sentences regarding aspects and their respective sentiment polarities.

I'm using SemEval Task 4 as a reference. The following is an example in the training dataset:

<sentence id="2005">
    <text>it is of high quality, has a killer GUI, is extremely stable, is highly expandable, is bundled with lots of very good applications, is easy to use, and is absolutely gorgeous.</text>
    <aspectTerms>
        <aspectTerm term="quality" polarity="positive" from="14" to="21"/>
        <aspectTerm term="GUI" polarity="positive" from="36" to="39"/>
        <aspectTerm term="applications" polarity="positive" from="118" to="130"/>
        <aspectTerm term="use" polarity="positive" from="143" to="146"/>
    </aspectTerms>
</sentence>

Can I easily use doccano for such a task? Or would I be better off using some other tool, such as brat?


texts: How to maximize a log linear regression equation satisfying a constraint?

texts:
I have a log linear equation of the form $y = w_1(\log{X1}) + w_2(\log{X2}) + ... + w_n(\log{Xn})$.
How can I find the value of X's that maximize the value of y subject to a constraint $(X_1+X_2+...+X_n \le t)$ in python or excel?
I tried using Excel solver but ran into issues where one of the X variables became 0 in the process and gave undefined results.


texts: Can you estimate average precision from log loss?

texts:
I am doing my final thesis in the field of Deepfakes and their detection. The final outcome is to have a binary classifier which could predict which video was updated and which was not. In other words, if video is fake, output number close to 1 and if video is real, output number close to 0. One of the reasons I started this project is because Facebook released a massive dataset called DFDC and they announced in media that the best solution got only 82% average precision on validation dataset and 65% average precision on the testing dataset. Source: https://ai.facebook.com/datasets/dfdc/
However, they also released a research paper describing the results: https://arxiv.org/pdf/2006.07397.pdf This is the part where I am confused. Research paper does not mention anything about 82% and 65%. Contradictory, in their research paper's figure you can see that Average Precision of the best score was around 0.98 (on the public testing dataset)

However, they also include the log loss scores and I managed to understand how Facebook calculates the values of 82% and 65% on public dataset and private dataset. What they do is described in the answer here: https://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss
which basically if the log-loss is 0.42, they take average precision as:
$$
AP = \frac{1}{e^{0.42}}
$$
I was wondering if it is acceptable to use the above method because when I evaluate my score with scikit library and use: average_precision_score, I get much higher scores if to compare with the method which Facebook used as in https://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss


texts: Explaining XGBoost functioning to non-technical people

texts:
I have been tasked to explain the principle of the XGBoost algorithm to non-technical people (think 1-2 slides in a powerpoint presentation to upper management).

I am currently working with the original papers : here for the paper specific to XGBoost and here for the orignal boosting idea by Friedmann.

I have come up with a short explanation for the boosting principle : the model is built iteratively, by concentrating on where the previous model made errors. XGBoost take advantage of the tree architecture and tools (splitting, pruning) to do so.

What would you add to this short description ?


texts: Treat similar observations in a classification problem

texts:
I have a dataset of about 200k rows and I'm working on a classification problem.
Grouping the dataset by a key variable, I noticed that some rows, with the same key value, have similar values in other columns (not equal). It may be useful to calculate the distance between observations and then deleting the closest ones prior to modelling?
I tried to split the dataset by key and then I calculated the euclidean distance between observations, but now I have a list of distance matrices and I don't know what to do next.
What do you think? I'm on the right way? Do you know some literature about it?


texts: How can I find to starting point of skewed data in python?

texts:
I have a list like this,

import random
import seaborn as sns

years = []

for i in range(1000):
    if i % 100 == 0:
        val = random.randint(1900, 2000)
    else:
        val = random.randint(2000, 2021)

    years.append(val)

sns.distplot(years);

Here is output graph,

As you can see, there is a density after 2000. There is not much data before this point. My question is how can I find this point in skewed data? Is there a formula that gives this? Any idea? Thanks in advance.


texts: Does variable (feature) selection help machine learning performance?

texts:
I am a newbie to machine learning fields. One question I always have when dealing with data problem is that whether feature selection helps the performance of prediction?

I understand that if you have small number of observations but relatively large number of features, it helps to select important feature before prediction. (Right?)

On the other hand, if the sample size is really big (n>>p), does feature selection before prediction help? Especially in random forest, XGBoost, etc, they can "ignore" the features that contributes very less to the model (based on my experience, even I put small number of noise variables, the performance does not change much). Do we need to do a feature selection before these algorithms?

Thanks! Enlighten me!


texts:
My project have to decide if a image is 'pdr' or 'nonPdr', and I have 391 images (22 of PDR class, and the 369 of nonPdr)..

In my first model i was trying this: https://stackoverflow.com/questions/57663233/my-keras-cnn-return-the-same-output-value-how-can-i-fix-improve-my-code
.. and my return was always the same...

Now I made some changes in my model file:

TRAIN_DIR = 'train_data/'
#TEST_DIR = 'test_data/'

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()

    # Get all subdirectories
    FolderList = os.listdir(Path)

    # Loop over each directory
    for File in FolderList:
        if(os.path.isdir(os.path.join(Path, File))):
            for Image in os.listdir(os.path.join(Path, File)):
                # Convert the path into a file
                ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
                # Add a label for each image and remove the file extension
                classes = ["nonPdr", "pdr"]
                LabelList.append(classes.index(os.path.splitext(File)[0]))
        else:
            ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
            # Add a label for each image and remove the file extension
            classes = ["nonPdr", "pdr"]
            LabelList.append(classes.index(os.path.splitext(File)[0]))

    return ImageCV, LabelList

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(605,700,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(4,4), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(2, activation='sigmoid'))
model.compile(optimizer='RMSprop', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)

data = np.array(data, dtype="float") / 255.0

le = LabelEncoder()
labels = le.fit_transform(labels)
labels = np_utils.to_categorical(labels, 2)

model.fit(data, labels, epochs=8, batch_size=20)

model.save('model.h5')

... but running this code give me a Loss = 8.0 and a acc = 0.50

What can I do? I appreciate any answer..

UPDATE I forgot that I reduce my train imgs to 20/20


texts: How to interpret a specific feature importance?

texts:
Apologies for a very case specific question. I have a dataset of genes, with which I am using machine learning to predict if a gene causes a disease. One of the features I have is a beta value (which is the effect size of the gene's impact on the disease), and I'm not sure how best to interpret and use this feature.

I condense the beta values from the variant level to the gene level, so a gene is left with multiple beta values like this:

Gene         Beta
ACE      -0.7, 0.1 ,0.6
NOS      0.2, 0.4, 0.5
BRCA     -0.1 ,0.1, 0.2

Currently I am trying 2 options of selecting a single beta value per gene, one where I select the absolute value per gene (and ignore whether it was a previous negative value) and another where I select the absolute value and return the previous negative numbers back to being negative. I am trying this as for beta values a postive or negative direction indicates the size of the effect a gene has on the disease, so I would think it's important to retain the negative information (as I understand it).

However, I've been advised to use just the absolute values with not retaining negative status, and I'm not sure if there's a way for me to know if one option is better than the other from the machine learning perspective. I am also having a problem in either case where my model values this feature as much more important than any other feature in my dataset. For example gradient boosting gives this an importance of 0.01, the next most important feature is at 0.001.

So my question is, how best can I interpret a highly important feature like this? If it is much more important is it actually a bias and is it likely due to my own handling/preprocessing of the feature or is it acceptable that is it just very important? Would it be possible for me to set my model to re-weight the importance of this particular feature? I have a biology background so not sure what is the normal or least biased approach.


texts: It is obvious that I have to scale the output data if the range of values is between say [-10;10] and the activation function of the output layer takes values in the interval [-1;1]. But I could also scale the activation function by multiplying it with the factor 10 instead. It seems to me that it is more common to scale the data and not the function. Is there a reason for it?


texts:
I am going to write yolov4 real-time object detection, and I have to do it for car then vehicle plate number, but it does not have to find plate number if there is no car, first car then number on car, is that possible?
Is it okay to use the darknet framework?


texts: Help Using Backprop Formulas from Deep Learning Book

texts:
I am trying to see how the fundamental formulas of backpropagation from this online book work on a small example.

The author lists these 4 fundamental equations (where I believe from the reading that the cost (or loss) function C as well as the derivative of the activation function $\sigma$ are general and work with any cost or activation function):

If I take a small example network:

and assume....

C = $\frac{1}{2}(a^3-y)^2$  (squared error)
The activation (a) for each node is just the logistic function sigmoid

From here we have the following:

$z^2_1=w^1_{11}x_1+w^1_{12}x_2+b_1$
$z^2_2=w^1_{21}x_1+w^1_{22}x_2+b_2$
$z^2_3=w^1_{31}x_1+w^1_{32}x_2+b_3$
$a^2_1 =\sigma(z^2_1)$
$a^2_2 =\sigma(z^2_2)$
$a^2_3 =\sigma(z^2_3)$
$z^3_1=w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3$
$a^3=\sigma(z^3_1)$

If I want to calculate $\frac{\partial C}{\partial w^2_{11}}$ it seems from the book that I should calculate $a^1_1\delta^2_1$ where $\delta^2 = ((w^{2+1})^T\delta^{2+1})\odot\sigma'(z^2) $.

Since $W^3$ does not exist I assume it is ignored and we can calculate $\delta^2_1= \delta^{2+1}_1\odot\sigma'(z^2_1)$.

$\delta^3_1 = \frac{\partial C}{\partial z^3_1} = (a^3-y)\sigma'(z^3_1)$
so $\frac{\partial C}{\partial w^2_{11}}= a^1_1*((a^3-y)\sigma'(z^3_1))*\sigma'(z^2_1) $. $a^1_1$ doesnt exist except if this would mean $x_1?$

However, if I just calculate $\frac{\partial C}{\partial w^2_{11}}$ I get $\frac{\partial }{\partial w^2_{11}}\frac{1}{2}([\sigma(w^2_{11}a^2_1+w^2_{12}a^2_2+w^2_{13}a^2_3)]-y)^2=a^2_1(a^3_1-y)\sigma'(z^3_1)$

Where am I going wrong?


texts: Face recognition - How to make an image classifier with large number of classes?

texts:
I am planning to make an image classifier that identifies the face of every player in the English Premier League. I have a couple of questions (since until now I have only worked with small or academic datasets).
My questions:

How do I download this many different images? Since it's pretty hard to manually download the pictures individually, is there a way to automate it?
I'm following this platform and am required to make a different class for each player. How should I go about making different classes for so many players?

PS: I am a newbie, so I apologize if my questions are vague. Please comment if any additional clarification is needed.


texts:
Overfitting Question
Would you consider that overfitting?


texts: One Hot Encoding vs Word Embedding - When to choose one or another?

texts:
A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values)
The usual data science approach would be to perform a One-Hot Encoding.
However, wouldn't it be a bit extreme to perform some One-Hot Encoding with a dictionary quite large (+/- 300 values)? Is there any best practice on when to choose Embedding vectors or One-Hot Encoding?

Additional, information: how would you handle the previous case if new values can be added to the dictionary. Re-training seems the only solution, however with One-Hot Encoding, the data dimension will simultaniously increase which may lead to additional troubles, embedding vectors, on the opposite side, can keep the same dimension even if new values appears.
How would you handle such a case ? Embedding vectors clearly seem more appropriate to me, however I would like to validate my opinion and check if there is another solution that could be more apporiate.


texts: Highly imbalanced class Machine Learning

texts:
I'm currently messing around on a wage dataset (trying to predict who did and didn't earn over 50k p.a. based on a range of factors). One of the variables - 'work-class' is very imbalanced and I was looking for some advice on how to deal with this.

100% of the people within the never-worked and without-pay categories earn over 50k.

Would imputation methods even be useful given the size of groups?

Thanks in advance,
Kelvin


texts:
GameID        Drive        down    yrdln        PlayType       sp
2009091000      1           1        22           run          0
2009091000      1           2        25           pass         0
2009091000      1           3        32           sack         0
2009091000      2           1        42           pass         0
2009091000      2           2        44           run          0
2009091000      2           3        43           pass         0
2009091000      2           4        33           Field Goal   1
2009091001      1           1        5            pass         0
2009091001      1           2        10           pass         1

I am to trying to determine the probability of scoring either a touchdown and/or a Field Goal based on a given starting yard line. The information given above is very similar to the data frame I am currently using, however, there are a few hundred thousand data entries, with thousands of games and multiple drives for each game.
I have been trying to find the first and last row for each drive for each GameID and then determine if a Field Goal or Touchdown was scored at the end of the drive. Ultimately I would like the data organized in a way where each row has a game ID, drive, 1st play yrdln, last play yrdln, Field Goal (yes or no), and sp (scoring play). Or hopefully something close to this.
I don't really know where to begin with something like this and have been failing for a good part of a day trying, so any and all help is greatly appreciated. If anything needs further clarification, please let me know!


texts:
I am reading about k-means clustering method.

How does the method calculate the mean between two coordination points?
How does the method work with more than two dimensions? How can calculate the distance and the mean if there are more than two dimension?

Are there any packages about k-means clustering in R?


texts: Neural Network: One model per user or (one-hot) variable with one model?

texts:
I have about 120 users with a total of 4500 data points. The minimum user has about 5 data points and the maximum has about 100 data points. I would like to build a model that will make predictions for each user.

What is the optimal approach? Do I create a single model for each user or do I create a single model with a categorical variable to specify the user?

I would imagine the single model approach would leverage the correlation between users, but the model per user approach might suffer from not enough data to generalize well.

I would consider a pooling method described here: possible duplicate, but there is not enough information in the independent variables to distinguish the users from one another, which is why I would have to create a categorical variable to distinguish users. Meaning, there are many users with the same input variables, but systematically different outputs.

The input variables include: arrival time, day of week, and temperature. The output variables include departure time and miles charged.


texts: Is each form of word classification also considered to be '(named) entity recognition'?

texts:
In an article that I am writing, I focus on word classification. A typical task that involves word classification is (named) entity recognition. Entity recognition is a rather broad task and seems to cover other sub-tasks as well.
Therefore, it seems fair to me to use the terms interchangeably.
Is this a fair assumption?


texts: Energy consumption time series forcasting

texts: Is there a good java library for doing time series energy consumption forecasting based on weather data and other variables?


texts: Test data for statistical t-test in Python

texts:
first of all sorry if this is not the proper place to ask but i have been trying to create some dummy variables in order to run a students t-test as well as a welch t-test and then run a monte-carlo simulation.Problem is, I am only given the sample size and standard deviation of the 2 populations. How can I go about creating some sort of representation for this data in order for me to run these tests? I wish to run these tests in either python or R. Thanks in advance.

EDIT : both populations come from a normal distribution


texts:
I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved).
Is there a more intuitive explanation of how the algorithm works, at a high-level?


texts: Creating pronunciation dictionary for ASR

texts:
I am working on ASR(automatic speech recoginition) on Somali data as master thesis and now I am stuck with how to create a phonetics or pronunciation dictionary for it. I searched over net and could not find one.

I'm not sure how to tackle this. Can someone guide me ?


texts: Cross entropy loss increase but precision get better

texts:
i am working on classification model.

my test result shows precision is getting better despite loss is increase.

is it just the nature of my data or is there some kind of theoretical explanation?


texts: AUC is high but not able to represent other class properly

texts:
I wanted to know if it makes sense to make 2 ROC curves for each of the 2 classes?
I am doing a binary classification problem but AUC is good at 82%.
But the F Score of the class labelled 1 is very poor (around 0.4).
So the AUC is very good but the F Score is very poor.
What does the ROC curve actually mean in this case?
I always thought, better the AUC score, better the classifier.
But in my case it is not able to capture the class 1 properly.
What should be done in such cases?
Any better metric to evaluate how good the classifier is?


texts:
I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf

In short, the paper describes a 3 network architecture.

The Global Branch is a ResNet or DenseNet (I used a DenseNet) pretrained on imagenet, then trained on the CheXNet dataset. After the training is done, the activations from the last convolutional layer are used to create heatmaps, which are then used to crop the images in the original dataset and create a new dataset.

The Local Branch is a ResNet or DenseNet (I used a DenseNet) with no pretraining. It is trained on the cropped dataset.

Finally, the fusion branch has as inputs the 2 global pooling layers of the Global and Local Branches.

I have managed to train the Global Branch (AUC: 0.77), generate the crops and train the Local Branch (AUC: 0.67). But when I try to train the Fusion Branch, the val_loss doesn't decrease:

This is my code:

def load_model_from_json(models_folder):
    print(" --- Reading model from ", models_folder)
    with open(models_folder + 'model.json', 'r') as f:
        json = f.read()
    print("Read: ", models_folder + 'model.json')
    model = model_from_json(json)

    model.load_weights(models_folder + "model_weights.h5")
    print("Read: ", models_folder + "model_weights.h5")

    return model

global_branch_model = load_model_from_json(self.global_branch_path)

local_branch_model = load_model_from_json(self.local_branch_path)

for l in global_branch_model.layers:
    l.trainable = False
    l.name = 'global_'+l.name
for l in local_branch_model.layers:
    l.name = 'local_'+l.name
    l.trainable = False

global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1')

local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1')

merged = concatenate([global_pooling.output, local_pooling.output])
dense = Dense(512, activation='relu')(merged)
dropout = Dropout(self.hyperparameters.dropout)(dense)
out = Dense(1, activation='sigmoid')(dropout)

fusion_model = Model(inputs=[global_branch_model.input, local_branch_model.input], outputs=out)
loss_function = unweighted_binary_crossentropy

optimizer = AdamW(lr=5e-5)

fusion_model.compile(optimizer=optimizer, loss=loss_function)

fusion_model.fit_generator(
            generator=FusionDataGenSequence(self.labels, self.partition['train'],
                                      current_state='train',
                                      batch_size=self.batch_size,
                                      hyperparameters=self.hyperparameters,
                                      num_classes=self.number_of_classes),
            epochs=self.hyperparameters.epochs,
            verbose=1,
            callbacks=callbacks,
            workers=self.num_workers,
            # max_queue_size=32,
            # shuffle=False,
            validation_data=FusionDataGenSequence(self.labels,
                                            self.partition['valid'],
                                            current_state='validation',
                                            batch_size=self.batch_size,
                                            hyperparameters=self.hyperparameters,
                                            num_classes=self.number_of_classes)
            # validation_steps=1
        )

Can you please help me figure out what I did wrong? Thank you!


texts: Collinearity and Multicollinearity in the features?

texts: What are some advanced or basic methods most used by data scientists/ML Engineers to detect collinearity (or) multicollinearity between features?


texts:
Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

All the values are strings and I need to encode them into vectors for training ML models using this data.
How do I make sure that the strings are always converted to the "same" vectors everytime?
I notice that when I try a test input with the same value as the training data, it gets vectorized into a different integer.
What is the standard procedure for preserving the mapping between String <--> Hashed Integer representation so that i get the same hash everytime?


texts: How to go about training a NER model to extract book citations in free-form?

texts:
I'm doing a project where I wish to create a graph visualization of free-form citations (not academic style citations) across all my e-books. E.g. David Foster Wallace's essays cite a lot of other books by different authors. For that I should be able to detect and extract book and authors names from my own e-books.
I've selected some examples from my e-books that I wish my NER model would tag as "books" (in bold font):

(...) or even the parodistic version of Pater to be found in W. H. Mallock’s The New Republic (...)

Plato words the same conception beautifully in the Republic: (...)

I also wish to tag authors, but I suppose this could be done out-of-the-box with Spacy or other NLP library, with some pre trained PERSON tag.
So, my question is about the best approach to go about creating this NER model.

I could create lots and lots of training samples from my books and create a new NER model. (very time consuming)

Or if there is a dataset or public model with the BOOK or something like WORK_OF_ART tag I could bootstrap my own dataset.

What do you think about this approaches?


texts:
CONTEXT:

I have some simulated data by which I made and trained a model.  during my training, I enjoyed having a large number of samples, and therefore my model is leveraging it by being decently complex. Yet since this model is trained on simulated data, in action, the model must be trained again once the real data is collected. Obtaining those real data is hard and we want to know how many data samples are needed before the model starts giving accurate prediction. So we started to look at the learning curve.

in summary,

  In real-world problems, it's vital to check the learning curve of a model, but not because we'd like to know if excess data is being used, but to make sure enough data has been fed to the model and the model has actually learned something.

SOME EXPERIENCE:

we should be sure of the convergence of the model in order to make a decision about the adequacy of samples. Yet, it might be the case that the convergence rate of the model using a small number of samples is different from once a huge bulk of samples are used for training.

Experimentally I realized that for reaching an optimal solution, you might need to change the learning rate or some structural features of your estimator/optimizer. For instance, once learning a tree-based model, one might want to think again about the minimum number of samples she/he like to have in each leaf.

QUESTION

Is it acceptable to change the meta-parameters in the estimater just for the sake of convergence in small number of samples?
Is it really needed to have convergence at small sizes of training set? My intuition is that the model starts to behave differently (and depart from its transient behavior) as soon as enough number of samples are given.
How in practice learning curve is being used? I assume all the businesses out there are data-hungry except a hand full of companies which already acquired zillions of samples. So I suppose, no one really thinks of the learning curve from the first point of view I mentioned above.


texts:
I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states.

I have finished applying my clustering techniques on my data set and the output of the clusters were the clusters of the states for each year.

Now I can know from my results; which states are close to each other with the birth names and which were not. By looking

I would like to have interesting results to make my project report interesting, Which states are similar in birth names are not enough to make the read be excited about my project.

My questions:

any ideas of what can be learned from my project?
How can I show the comparison between the clustering techniques? anyway other than seeing how Homogeneous the clusters are?


texts: LSTM with return_sequences - "Training a model on multiple timesteps simultaneously"

texts:
So I'm following Tensorflow's LSTM/time series tutorial and there's something I don't understand. I do understand what happens with return_sequences on/off, however, it is stated, that with return_sequences on you allow

Training a model on multiple timesteps simultaneously

I don't quite understand what this means. If you keep it off, the input will go through all the previous timesteps, and make a prediction. Your loss will be the error of this prediction vs. the actual. You're still training on all the timesteps, right (although not simultaneously I guess?).
Now if it's on, it's making a prediction on each timestep - what's the loss in this case? Is it the average error for each prediction in the sequence? Why is it better to switch return_sequences on, as it's done in the tutorial? Does the model learn more or faster in this case? How does it impact the learning?
It is also stated that

With return_sequences=True the model can be trained on 24h of data at a time.

Isn't that still the case with return_sequences=False? In the example they're still teaching the model to predict 1 timestep based on the previous 24.


texts:
I have an implicit dataset. It contains which user click which item. I'm doing collaborative filtering and finally i get the item similarites. So now i have data like;

Item - SimilarItem - SimilarityValue
A    - C           - 0.12
A    - R           - 0.42
A    - Y           - 0.34
A    - J           - 0.62
B    - A           - 0.16
B    - Y           - 0.83
C    - J           - 0.23

or

Item  --  Similarities
A     --  C,R,Y,J
B     --  A,Y
C     --  J,A,D

As i know all the relations between 40K item, can i turn this into a nice visualization showing some of them get together at somewhere and someothers are elsewhere. I can do this either with python or some javascript library.

What should be the way of doing this? Can you show me some examples?


texts: What is difference between intersection over union (IoU) and intersection over bounding box (IoBB)?

texts: Can someone give a detailed explanation IoU and IoBB along with that the differences between them.


texts: Train at 99% and Validation split accuracy is not more than 70%

texts:
I am training a model that get accuracy on train set up to 99%
but the validation split test not increase more than 70-72%
this is how my model is configurated:
model = Sequential()
#model.add(TimeDistributed(Dense(64, activation='linear')))
#model.add(GRU(512, return_sequences=True, activation='linear',))
model.add(LSTM(128, return_sequences=True, activation='linear'))
model.add(Conv1D(64, 64, strides=1, padding='same'))
model.add(Dense(128, activation="linear"))#, kernel_regularizer=regularizers.l1_l2(l1=1e-4, l2=1e-6)
model.add(Dropout(0.2))

#model.add(Conv1D(16, 16, strides=1, padding='same'))
#model.add(TimeDistributed(Dense(32, activation='linear')))
#model.add(GRU(512, return_sequences=True, activation='linear'))
model.add(LSTM(128, return_sequences=True, activation='linear'))
model.add(Conv1D(64, 64, strides=1, padding='same'))
model.add(Dense(128, activation="linear"))#,kernel_regularizer=regularizers.l1_l2(l1=1e-4, l2=1e-6)
model.add(Dropout(0.2))
#model.add(BatchNormalization())

model.add(Dense(1,activation='linear'))

model.compile(optimizer='adamax',loss="mse",metrics=['accuracy'])
model.fit(X_train,y_train,epochs=8000, batch_size=256, verbose=1, validation_split=0.1, callbacks=callback)

what can be the issue?


texts: Imbalanced text classification by oversampling: correction of class predicted probability by prior probability

texts:
My dataset has 3 class and 900 examples for training. Class distribution is 255, 185, and 460.

I found that if I oversample (random) the training data then I have to correct/calibrate the predicted probability of the test data because after oversampling the training and testing data distribution are not same. This is nicely described here, here and here

I have 4 questions:

Should we do this to calculate:

Training loss

My guess: We should not change the posterior/predicted probability for training loss calculation. Because that will cancel out the effect that has been created by oversampling. Am I right?

validation loss
My guess: Since the validation loss has no effect on training (optimization), we can correct the predicted probabilities but then it would not be comparable with the training loss. Since we compare both losses to check overfitting, we should not do any probability correction here. Unless we compute two kind of training losses. One is without probability correction which would be used to compute the gradient and another corrected training loss which would be used to plot with corrected validation loss. Then these two losses would be comparable. Am I right?

validation accuracy

My guess: For validation accuracy, I think we should do it for the same reason we have to do it on testing data. Right?

Is this a mandatory step? I am asking this because this might hurt the overall accuracy. Because this will penalize the probabilities of the classes which have less example.


texts: If I am given 60 features along with test label and I was to find values of other features what is the best way to do it ?


texts:
import numpy as np
from keras.datasets import cifar10
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
from keras import backend as k
from keras.models import Sequential

(Xtr,Ytr),(Xte,Yte)=cifar10.load_data()

Xtr = Xtr.astype('float32')
Xte = Xte.astype('float32')

Xtr = Xtr.reshape(50000, 3072)
Xte = Xte.reshape(10000, 3072)

Ytr = np_utils.to_categorical(Ytr, 10)
Yte = np_utils.to_categorical(Yte, 10)

model=Sequential()
model.add(Dense(100, input_shape=Xtr.shape[1:]))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer="sgd", metrics=['accuracy'])

model.fit(Xtr, Ytr, batch_size=200, epochs=30, shuffle=True,verbose=1)

scores = model.evaluate(Xte, Yte, verbose=0)
print("Accuracy: %.2f%%" % (scores[1] * 100))

And the result:

I am trying to creat a pretty basic 2 layer NN on cifar-10. I know that the data is not preprocessed. But that can't be the reason for learning nothing. Where am I making the mistake?


texts: Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

texts:
In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..

results2 = classify_models(container2, models)

..when feeding in new data (matrixdata2) against the model it produces an error message:

Error in predict.svm(model, container@classification_matrix, prob = TRUE,  :
  test data does not match model !

In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I did create a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am still unsure how to adapt the R code to rectify the issue.

Please can you help me adapt the the R code below to overcome the error?

#Load Libraries
library(RTextTools) #RTextTools available for 3.4.1
library(e1071)
library(gmodels)

setwd(directory/path)
text= read.csv("matrixdata1.csv", header = FALSE)

# build dtm
matrix= create_matrix(text[,1:2])
mat = as.matrix(matrix)

# build the data to specify response variable, training set, testing set.
container = create_container(mat, as.numeric(as.factor(text[,3])),
                             trainSize=1:672, testSize=673:840,virgin=FALSE)

models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168)

#container1
results1 = classify_models(container, models)

text2 = read.csv("matrixdata2.csv", header = FALSE)
matrix2= create_matrix(text2[,1:2])
mat2 = as.matrix(matrix2)

container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE)

#Results from feeding in new data against the model
#When running this code below, it produces the error message outlined above in the description of the problem.
results2 = classify_models(container2, models)

matrixdata1.csv on GitHub

matrixdata2.csv on GitHub


texts: When is something a Deep Neural Network (DNN) and not NN?

texts:
When would a neural network be defined as a Deep Neural Network (DNN) and not a NN?

A DNN as I understand them are neural networks with many layers, and simple neural networks usually have fewer layer... but what a many and a few in numbers? or is there some other definition?

What are networks trained used Tensorflow, Caffee as such? I haven't (as far I know) seen anybody manually design a network with many many layers.
They seem to promote their tools for creating DNN, but is it actually DNN if you only make a network with two layers?


texts:
I am coming from the world of linear/logistic regression where residuals are assumed to be normal and the output includes some sort of standard error of residuals that one can use to make forecasts. Admittedly, I have very little experience in fitting and interpreting multinomial models..

Anyways, I am thinking about a specific problem:

Let's say we have lots of data on voters that have already cast a vote. With this data we wish to train a model to forecast future elections.

From the world of normal regression, there seems to be standard errors in forecasting each vote that could potentially be summed to create a random variable for the sums of votes for each candidate. These random variables could then be used to try to forecast the probability of each candidate winning.

My question is, what is the standard way of assessing the "standard error" of classifier models using tools like Stochastic Gradient Descent Classifiers for example. Or any of the more modern tools for classification. Some sort of bootstrapping approach?


texts: Extracting Features Using TensorFlow CNN

texts:
I'm trying to extract features of set of images. I'm using CNN from this site.

Can anyone please tell me how to do feature extraction of images using CNN? I looked for various places. But nowhere it's clearly mention the feature extraction part.


texts:
I have a classification problem.
In gradient tree boosting I read that-
1. Initially a weak learner is fitted on the entire training dataset.
2. Output of each training row is obtained. In my case it will be {0,1}.
3. Now, the second classifier will train on the residual of the prediction i.e {initial prediction - final prediction}.
My doubt is it seems very odd to calculate the residual in case of classification problems while it is ok when I look at residual calculation in case of mse.
So, what is actually happening in case of categorical target variable. How is the residual being calculated and which loss function is that fitted into?

Thanks in advance.


texts: Is it possible to achieve activity recognition on a single image?


texts:
this is my first post at ds StackExchange, so please be gentle and let me know if something is not clear :)

I have many products (>1M), and I save all the products purchases in a DB with a time stamp. ("purchases data")
each product has 'content features' (e.g product size, product safety rank etc.)

The "purchases data" looks like this:

  | time stamp | product id | content features 1 | ... | content features N |

Where each row is a purchase of a product with id at time stamp.

My main target is to get tomorrow most wanted products,
I translate the problem into predicting demand for the next day, or classify each product id and day as high demanded or low demanded),

I struggle with two main problems with these settings:

Generating demand data: I want to convert the "purchases data" into demand for a day ("demand data")- meaning that I group the data by product id and day,
Then I count the number of rows and save it as 'freq' (and also remove row duplication).
The problem is that the minimum product frequency per day would be 1 and not 0

  For example: if product #1 was purchase at Sunday 3 times and at Wednesday 2 times, the purchases and demand data would be:

"purchases data" fi(product id) is mapping to content feature i:

   | time stamp      | product id | content f 1 | ... | content f N |
   | Sunday  05:20   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  08:11   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  10:25   | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 08:10 | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 16:20 | 1          | f1(1)       | ... | fn(1)       |

"demand data":

   | day             | product id | content f 1 | ... | content f N | freq |
   | Sunday          | 1          | f1(1)       | ... | fn(1)       | 3    |
   | Wednesday       | 1          | f1(1)       | ... | fn(1)       | 2    |

    But if product #1 was not purchase at Monday there wouldn't be any row
    since there is no purchase
    data for this item at that timestamp.

Since there are over 1M products, I want to avoid creating rows with 0 frequency,

Is there a way to create (or to design) demand data from purchases data for a huge amount of items (products) without using 0 demand rows?

Content-based time series: After creating "demand data", I want to use it as a time series.
My problem is that I would need to split the data into over 1M series, one for each product/item id, and these series would also be very sparse...
I want to find a way to use the "content features" as input with the time series and have the model learn to use some kind of averaging of time series based on related content features.

What is the best way to model content based series time prediction?


texts: Filling missing values for Embedded List in Python3

texts:
I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough.

I have a dataset consist of text data. I store them in a list and every row of a list consists of a string value. But every row length is not equal. I want them to be equal, so I can use them in a self-attention model.

The sample of my dataset

In [8]: myList
Out[8]:
[
['the first line of my dataset'],
['the second line'],
['the 3rd'],
['the 4th'],
['the 5th'],
['the 6th'],
['the 7th'],
]

So as you can see the first one is longer than the rest of them. I want to fill with a certain value like # to equalize the word count.

The sample output I'd like to do

In [8]: myList
Out[8]:
[
['the first line of my dataset'],
['the second line # # #'],
['the 3rd # # # #'],
['the 4th # # # #'],
['the 5th # # # #'],
['the 6th # # # #'],
['the 7th # # # #']
]

If this would be a dataframe I could use fillna() function of Pandas library. I tried to apply this:

train_X = pd.Series(train_X).fillna("#").values

but since it is an embedded list(I guess) it didn't work. Is there a better way to do that?

Any recommendation is appreciated.


texts:
Allow me to present some images so that I may explain my problem. The images on the left contain a smooth curve surrounded by lots of outliers/noise. The image on the right depict the desired curve.

Is there any approach that would help in eliminating the noise and retrieving the original curve. Please note, this can be any smooth curve (the image on the right) and equation is not known. No prior knowledge of the curve equation is available.
Please note that I had originally published this question on Match stackexchange. Posting to data science based on feedback.
Thanks,
Sau


texts: Destination prediction with Naive Bayes and sparse output matrix

texts:
Given a dataset of historical cab rides, I'm trying to predict the final zip code destination of a ride based on the following features:

origin zip code (e.g. 10006 Wall Street, Manhattan)
pickup hour of the day
pickup day of the week
pickup month of the year
car class (e.g. standard, premium, luxury, van...)
current zip code at the time of the observation (e.g. 10013 Tribeca, Manhattan)

As an example, a ride started in Wall Street in January on a Saturday at 9am with a Premium car, currently underway and located in Tribeca is likely to end up dropping its passenger in Chelsea.
The classification experiments I've done so far using naives bayes show some interesting results but I can see a couple of issues I'm not sure how to tackle.
First issue: sparsity of the output matrix
The first one is that the output matrix is very sparse, with hundreds/thousands of possible zip codes. I'm not sure how to model that appropriately in order to optimise results and also training time. Modelling this as a multiclass problem sounds a bit naive. Would a regression with lat/lon potentially work better? Is it realistic to think of a combination of both approaches, and in which case would we need two independent models?
Second issue: embedding geographical knowledge
The other issue is that one zip code can have the single highest probability while a group of 5 others, closely packed together, can have lower individual probabilities but indicate a better chance overall. Example, Morningside (upper Manhattan) can have 15% but six other individual adjacent zip codes in Brooklyn could have, together, an overall probability of 60%, meaning that the most likely scenario is that the ride is going to Brooklyn (although we're not sure where exactly) instead of Morningside. For this one, I'm balanced between hardcoding an aggregation by borough from the individual predictions, or alternatively, to have two models (one for the zip codes and one for the boroughs) and merging their predictions afterwards. There might be other approaches like encoding the mutual distances between boroughs, but I'm not sure how to exploit this information in the model.
Any ideas or guidance for either of the above issues would be very helpful.


texts: Association Rule Mining across two market baskets

texts:
I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket.

Imagine customers come to a Store A and buy a certain number of products. The same customers go to Store B and buy another set of products. I want to associate between the two Stores and not within the Store.

So I want to make "A --> B" statements like

  "Customers that Bought x and y from Store A also purchased z from
  Store B"

I could lump all the purchases as one market basket and run some association mining algo on it but in that case the association algo will not control for where the items were purchased.

Clearly one alternative is to find all rules using one market basket and then exclude the ones where the A isn't pure in the A -> B relationship.

Any other ideas would be great.


texts: Why would it be bad to fit a regression model to a binary classification problem?

texts:
Let's say that we have a binary classification problem.

Why would it be bad to fit a linear regression and then classify given a threshold?

The output would be continuos and it could be out of range, but we could convert it to binary with the threshold.


texts: TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

texts:
The input function code:

    def input_fn(params):
        """The actual input function."""
        batch_size = FLAGS.train_batch_size

        name_to_features = {
            "input_ids":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "input_mask":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "segment_ids":
                tf.FixedLenFeature([max_seq_length], tf.int64),
            "masked_lm_positions":
                tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
            "masked_lm_ids":
                tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
            "masked_lm_weights":
                tf.FixedLenFeature([max_predictions_per_seq], tf.float32),
            "next_sentence_labels":
                tf.FixedLenFeature([1], tf.int64),
        }

        # For training, we want a lot of parallel reading and shuffling.
        # For eval, we want no shuffling and parallel reading doesn't matter.
        if is_training:
            d = tf.data.Dataset.from_tensor_slices(tf.constant(input_files))
            d = d.repeat()
            d = d.shuffle(buffer_size=len(input_files))

            # `cycle_length` is the number of parallel files that get read.
            cycle_length = min(num_cpu_threads, len(input_files))

            # `sloppy` mode means that the interleaving is not exact. This adds
            # even more randomness to the training pipeline.
            d = d.apply(
                tf.contrib.data.parallel_interleave(
                    tf.data.TFRecordDataset,
                    sloppy=is_training,
                    cycle_length=cycle_length))
            d = d.shuffle(buffer_size=100)
        else:
            d = tf.data.TFRecordDataset(input_files)
            # Since we evaluate for a fixed number of steps we don't want to encounter
            # out-of-range exceptions.
            d = d.repeat()

        # We must `drop_remainder` on training because the TPU requires fixed
        # size dimensions. For eval, we assume we are evaluating on the CPU or GPU
        # and we *don't* want to drop the remainder, otherwise we wont cover
        # every sample.
        d = d.apply(
            tf.contrib.data.map_and_batch(
                lambda record: _decode_record(record, name_to_features),
                batch_size=batch_size,
                num_parallel_batches=num_cpu_threads,
                drop_remainder=True))
        d = d.prefetch(10)
        return d

The mirrow strategy code:

    distribution = tf.contrib.distribute.MirroredStrategy(
        devices=["device:GPU:%d" % i for i in range(FLAGS.n_gpus)],
        # num_gpus=4,
        cross_tower_ops=tf.distribute.HierarchicalCopyAllReduce())
    run_config = RunConfig(
        train_distribute=distribution,
        # eval_distribute=dist_strategy,
        log_step_count_steps=log_every_n_steps,
        model_dir=FLAGS.output_dir,
        save_checkpoints_steps=FLAGS.save_checkpoints_steps)

    model_fn = model_fn_builder(
        bert_config=bert_config,
        init_checkpoint=FLAGS.init_checkpoint,
        learning_rate=FLAGS.learning_rate,
        num_train_steps=FLAGS.num_train_steps,
        num_warmup_steps=FLAGS.num_warmup_steps,
        use_tpu=FLAGS.use_tpu,
        use_one_hot_embeddings=FLAGS.use_tpu)

    # If TPU is not available, this will fall back to normal Estimator on CPU
    # or GPU.
    estimator = Estimator(
        model_fn=model_fn,
        params={},
        config=run_config)

The problem is that if I have 4 GPU. Each GPU could run 8 batchsize. I set batch_size = 8 not 32. batch_size = 32 will OOM.

Am I right? Will the data be distributed to 4 GPU with different batches?


texts: Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

texts:
I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of the choice of activation function.

Can anybody direct to a good tutorial that include such examples?


texts: I coded a 38 layer CNN and 8 layer CNN but there's something wrong in my 38 layer CNN, which doesn't learn anything. Not able to fugure out what's wrong. They were trained on CIFAR.


texts: is there metric 'multi_logloss' for xgb crassifier?

texts:
lgb has the log_loss metric

lgb = LGBMClassifier(class_weight='balanced', boosting_type='dart',
                         drop_rate=0.9, min_data_in_leaf=100,
                         max_bin=255,
                         n_estimators=500,
                         bagging_fraction=0.01,
                         min_sum_hessian_in_leaf=1,
                         importance_type='gain',
                         learning_rate=0.1,
                         max_depth=-1,
                         objective='multiclass',
                          metric='multi_logloss',
                         num_leaves=31)

is there any  metric='multi_logloss', for xgboost classifier???

xgb = XGBClassifier()


texts: How can I do a train test split for an unbalanced panel data set in Python?

texts:
I have an unbalanced, panel pandas data frame.
I would like to split this data into a training set and a testing set. Python's train_test_split method will not work because it does a random split, and so, it will likely places observations from t + 1 into the training set, and observations from t into the test set.
Which, of course, makes no sense, because the future cannot predict the past.
TimeSeriesSplit will also not work because this function does not take into consideration the panel dimension of my data set.
Is there an easy way to do a train test split on unbalanced panel data sets? This split should (1) take into consideration the panel dimension of the data set, and (2) place earlier observations in the training set and later observations in the testing set.


texts: Is there a theorem of prominent Russian mathematicians that played an important role in the development of machine learning?

texts:
I once attended a seminar in which a statement answering the question asked above was answered in the affirmative.
I do not recall further specifics, however.


texts: percentage of confidance on desion trees results

texts:
I am looking towards a solution where classification algorithms produce output with some confidence value. but I am confused whether classification algorithms are able to produce results with percentage of confidence?
Thanks


texts: Sparse Representation vs Dense Representation

texts:
The one of the benefits of ReLUs is sparsity. Sparsity arises when a≤0 (a = wX+b). The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.
Is it true that sparse representation is more beneficial that dense representation, especially for Neural Networks?


texts:
Something that has always bothered me is summarizing distributions when feature engineering for a machine learning process. Does anybody have best practices for this?

Example: Imagine a dataset showing orders of products by customers. You want to summarize customer behaviour.

In that example, if we're to only focus on customers' order value, the order value for each customer would have some distribution D(x).

Now, can create variables describing features of each customer's distributions (min, mean, median, max, quartiles, IQR, etc.), but are there best practices around which features tend to provide the most information upon extraction? Moreover, is there some way to contain the information of a distribution in a single variable?


texts:
Hi I have tensorflow installed of version 1.14.0 on my ubuntu machine. I am trying to run a code with

import keras

and I get an error

AttributeError: module 'keras.backend' has no attribute 'backend'

i have tried installing specific version of keras(2.0.8) because i have seen an answer which says keras 2.0.8 is required for tensorflow 1.14.0 . But still I get the error.

Can someone help me with this. Really stuck at a vey basic point


texts:
I am trying to use an LSTM to predict daily usage for users. I have data for (say) 90 days of usage for a large number of users. Based on business knowledge (and initial analysis) we know users fall roughly into different categories. E.g. daily users would have a non-zero usage almost every day, weekly users would have one or two days of non-zero usage every 7 days and monthly users would have a couple of days with non-zero usage per 30 days.

Sample data where each column is one day starting from October 1st and each row is data for one user. (The usage 'cycle' of each user might start on any day).

User 1: 10,  8, 10,  9, 0, 0, 11, ...
User 2:  0,  0,  0, 20, 0, 0,  0,  0,  0, 18,  0,  0,  0, ...
User 3: 40,  0,  0,  0, .....

where User 1 might be a "daily" user, User 2 is a "weekly" user and User 3 is a monthly user.

My first question is that can a single LSTM/deep learning model capture these different types of patterns?
The goal is to predict the daily usage (next couple of days based on past 90 days) for individual users.

Currently I am using a really simple LSTM (in Keras):

model = Sequential()
model.add(LSTM(1, input_shape=(90, 1), unroll=True))
model.compile(optimizer='rmsprop', loss='mse')

To help the model 'capture' the fact that different users might have different levels of usage I added the average usage (of non-zero values) for each user as the first data point for each row. The remaining 90 data points for each user remain as shown in the table above.

My second question is if I really need to add the average of the values to 'help' the model?

The problem is that even after 100 epochs the error remains unchanged.
And finally what can I change to make this to work?

Thanks in advance.


texts: Are there any very good APIs for matching similar images?

texts:
I am working with Python. I have 3000 thousands images of front-faced watches like the following ones: Watch1, Watch2, Watch3.

I want to find an API which receives this collection of photos or even others taken under less ideal conditions (different background colour, darker lightning etc) and which finds/matches the most similar watches with one each other.

By similarity I mean that I expect that a round in shape, brown watch with thin lace will be only matched with watches of round shape, of dark colour and with thin lace (from the same collection of photos).

I am aware of APIs of this kind from Google, Amazon, Microsoft, TinEye, Clarifai, Indico etc but I am not sure that they will perform so well in a so specialised application. For example, these APIs are useful for matching car images with car images and not with food images but matching among  the same kind of objects (e.g. watches) based on a very high level of detail (shape, colour, thickness etc) is significantly more demanding.

For instance, this is an application on a specific kind of objects like clothes with Indico: https://indico.io/blog/clothing-similarity-how-a-program-is-more-fashionable-than-me/. However, if you notice it, the results are not that good and essentially they could be retrieved to a great extent even by simply applying PCA and KNN to these images.

Therefore, my question is: Is there any API which can match similar images based on a high level of detail?


texts: Model the predictive relationship between images

texts:
Hello fellow machine learners,

We have numerous pairs of 64 x 64 (or other dimensionality) images (maps). In each pair, the first image demonstrates a physical parameter, e.g. wind speed, at each pixel; the second shows another physical or financial parameter, e.g. temperature or insurance loss, at each pixel. We want to model the predictive relationship from the first image to the second. Our previous models typically consisted of characterizing the field using domain-specific knowledge for dimension reduction. Now my colleague wants to explore the possibility of using statistical learning methods alone.

I did a shallow research and found this could fit in a multi-output regression problem. Online forums suggested papers like Melki and Cano's "Multi-Target Support Vector Regression Via Correlation Regressor Chains", and many Python libraries. However I am a little concerned about the extreme dimensionality, i.e. 4096, and the potential failure of exploiting the spatial structure of our problem if we go down that road.

I know CNNs are suitable for image recognition tasks as such. Has anyone encountered a similar problem and attacked it with neural nets? Any suggestion would be much appreciated!

Thank you!


texts: Difference between parameters and weights in ANN

texts:
I think in writing we often refer to the part of the neural networks that tell us about the data are the parameters when we really mean the weights, is this correct? Would the sentence...

  Better understanding parameters of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.

...be better phrased as...

  Better understanding the weights of the neural network after training on bird migration data can allow us to comprehend the behavior of these animals.


texts: How long does it typically take to train a MNIST data on a Mac Pro?

texts:
My code is below:

# define a simple CNN model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Conv2D(30, (5, 5), input_shape=(1, 28, 28), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Conv2D(15, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(50, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

# build the model
model = baseline_model()
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=500, batch_size=200)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)

My Mac's capacity:

Grphics Radeon Pro 555 2048 MB
Intel HD Graphics 630 1536 MB

Memeory 16 GB 2133 MHz LPDDR3

Processor 2.8 GHz Intel Core i7

It is taking several hours to train this the MNIST image classification dataset on my computer. Is this normal?


texts: error when input function called with shape (2,2)

texts:
I am new to Tensorflow and machine learning.

I am trying to use high level API from Tensorflow.

Please tell me what i am doing wrong.

import tensorflow as tf
import numpy as np

features = np.array([[-1,-2],[1,2]],dtype='int32')
label = np.array([0,1],dtype='int32')

feature_columns = [tf.feature_column.numeric_column('features',shape=[2,2])]

model = tf.estimator.LinearClassifier(feature_columns=feature_columns,n_classes=2)

model.train(input_fn= tf.estimator.inputs.numpy_input_fn(x=features,y=label,shuffle=True))

I am getting an error

ValueError: features should be a dictionary of `Tensor`s. Given type: <class 'tensorflow.python.framework.ops.Tensor'>


texts: How do I change the marker edge color of a seaborn relplot?

texts:
I am trying to output a complex facet grid plot in the format of the following image:

But the problem is that I don't want the edge color of the markers to be white, I want it to be the face color.

In matplotlib, I can change the color of marker edges by calling the following:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.DataFrame({"values_x": np.random.randn(100), "values_y": np.random.randn(100)})

plt.scatter(x=df["values_x"], y=df["values_y"], edgecolors="red")
plt.show()

This produces a plot that looks like:

Now, if I call the following:

sns.relplot(data=df, x="values_x", y="values_y", edgecolors="red")
plt.show()

I still get white marker edges like in the following:

So, how do I correctly change the color of the marker edges within a call to sns.relplot()?

*For reference, I create the top facet grid plot with the following:

sns.relplot(data=df,
            x="C",
            y=metric,
            row="penalty",
            col="solver",
            hue="fold")


texts: Applying Sci-kit Learn's kNN algorithm to Fresh Data

texts:
While I was studying Scikit-learn's kNN algorithm, I realized that if I use sklearn.model_selection.train_test_split, the provided data gets automatically split into the train data and the test data set, according to the proportions provided as parameters.
Then based on the train data, the algorithm looks at the k-nearest neighbor points closest to the test data points to determine whether the test data points belong to a certain criteria or not.
I was wondering whether there was a way to predict the criteria NOT for the test data sets, which were already a part of the provided data set, but brand new data that were not provided during the whole process.
Is there a way to do that using sci-kit learn?


texts: Which of the scikit learn classification algorithms accept Sparse matrices?

texts: If I have to use scikit-learn(sklearn) library for classification and the feature matrix is a sparse matrix then which of the classification algorithms of this library can be used by me?


texts: A classification machine learning flow chart implimenting dimentionality reduction, upsampling, and cross validation

texts:
I would like to make a flow chart for an ML classifier and make sure that my thinking is correct.

Here is a little about my sample:
I have 3 classes and about 160 features. I suspect that some of those features are redundant.
My classes are imbalanced.
Please let me know if I should provide more info about the sample.

This is my first try on tackling a machine learning problem. Based on checking the functionality of each of the options I use below, I chose it in that specific order, but i might be wrong.

Could someone just let me know if this flow chart is logical, and if there was a problem in any of the step, could you possible tell me why it is not supposed to be there?

Flow chart

Get the data
Make it in a readable format and import it
Split the data into train/test
Standardize with StandardScalar
Perform PCA on train data
Perform K-fold cross-validation on train data
Apply SMOTE
Predict (Using SVC)
Make a confusion matrix
Perform GridSearchCV
Perform PermutationImportance


texts: Why is count encoding effective in improving accuracy?

texts:
Can someone please explain why/how Count encoding of categorical features improve accuracy in classification  when compared to simply label encoding them ?
I found one explanation in kaggle " Rare values tend to have similar counts (with values like 1 or 2), so you can classify rare values together at prediction time. Common values with large counts are unlikely to have the same exact count as other values. So, the common/important values get their own grouping. " which doesn't seem convincing or I don't understand the reasoning .
Can someone please explain why it performs better than label encoding ? Label encoding can also find frequent patterns to correlate with the target variable , right ?


texts: Custom loss function which is included gradient in Keras

texts:
I want to make a custom loss function.
Concretely, I use a 2D Convolutional neural network in Keras.
So far, I've made various custom loss function by adding to losses.py.
However, in this case, I encountered the trouble which is explained later.

I have attempted to make a regressor for image tasks.
The data shape of both input & output are (1,128,128,2), where 1 is mini-batch size, 128 is pixel number and 2 is the number of the channels.

Anyway, I want to add new loss function to this task.
I want to compute the difference of pixel gradients between answer & predicted values and add to loss function.
I tried to make such like below.

def continuity(y_true, y_pred):
    import tensorflow as tf
    import numpy as np
    dx = dy = 1/128
    gridSetting = (128,128)

    u_true = y_true[0,:,:,0]
    v_true = y_true[0,:,:,1]
    u_pred = y_pred[0,:,:,0]
    v_pred = y_pred[0,:,:,1]

    cont_true = np.zeros((127,127))

    for j in range(127):
        for i in range(127):
            cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy

    cont_pred = np.zeros((127,127))
    for j in range(127):
        for i in range(127):
            cont_pred[i,j] = (u_pred[i+1,j]-u_pred[i,j])/dx + (v_pred[i,j+1]-v_pred[i,j])/dy

    cont_true = K.variable(value=cont_true, dtype='float64')
    cont_pred = K.variable(value=cont_pred, dtype='float64')
    cont = K.mean(K.abs(cont_true - cont_pred), axis=-1)
    mse = K.mean(K.square(y_pred - y_true), axis=-1)
    cont_mse = cont+mse
    return cont_mse

Error is written as below.

Traceback (most recent call last):
File "DSC_multi-scale_2D.py", line 107, in <module>
autoencoder.compile(optimizer='adam', loss='continuity')
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training.py", line 332, in compile
sample_weight, mask)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training_utils.py", line 403, in weighted
score_array = fn(y_true, y_pred)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/losses.py", line 86, in continuity
cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy
ValueError: setting an array element with a sequence.

Probably, in losses.py, the data type of u_true, v_true... doesn't support numpy.

How can I debug/fix this problem?


texts: ImageDataGenerator for multi task output in Keras using flow_from_directory

texts:
I am creating a multitask CNN model and I have two different classification properties (one with 10 classes, 2nd with 5 classes) and my directory structure looks like this:

    -Train
       - image1.jpg
          ...
       - imageN.jpg

    -Test
       - image1.jpg
             ...
       - imageN.jpg

    -Vald
       - image1.jpg
          ...
       - imageN.jpg

And labels are in a csv file as propA, propB. So, a single image will have two classes, one from property A and one from property B.
The model uses VGG16 :
baseModel = VGG16(weights="imagenet", include_top=False,input_tensor=Input(shape=(img_size, img_size, 3)))
flatLayer = baseModel.output
sharedLayer = Flatten(name="flatten")(flatLayer)
sharedLayer = Dense(1024,name="Shared")(sharedLayer)
sharedLayer = Dropout(0.5)(sharedLayer)
task1 = Dense(512, activation="relu")(sharedLayer)
task1 = Dense(10, activation="softmax",name='PFR')(task1)
task2 = Dense(512, activation="relu")(sharedLayer)
task2 = Dense(5, activation="softmax",name='FT')(task2)
model3 = Model(inputs=baseModel.input, outputs=[task1,task2])

The number of images is large, therefore I cannot load them in memory and need to use flow_from_directory like functionality. But, in my train directory, there are no class sub directories because it is not possible to generate class directories as there are total 15 classes and I am not sure on which property to generate class subdirs for. (and flow_from_directory doesnt work if there are no class subdirs)
The labels are available in  array, propALab and propBLab.
So far, I havenot found anything helpful, can someone help?


texts: What does this formula in Glorot & Bengio mean?

texts:
In this paper, on page 5, we find the formula
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$
I am really struggling to understand what is meant by this formula. I think at least some of the following are true:

We're dealing with a linear neural network, i.e. no activation functions.
$z^i$ is the output of one layer of the network, so for the very first hidden layer (closest to the input) of this network we would have:
$$Var(y)=nVar(x)Var(W)$$
where $y$ is the output vector of the first hidden layer, $x$ is the input vector and $W$ is the matrix of weights connecting that first layer to the input.

However, it's entirely unclear to me what these variances are. I thought maybe $Var(y)$ just meant the empirical variance of the vector $y$, i.e. the sum of the squares of the differences of the elements of $y$ from the mean of $y$, and likewise for $Var(x)$ and $Var(W)$, where the latter is just the variance of all of the entries of $W$. But under this interpretation the formula turns out to be false numerically, so I'm at a bit of a loss to understand what this equation is supposed to mean.


texts: Interpret User Interfaces with Machine Learning

texts:
I am currently working on a prototype of an application that should be able to interact with user interfaces.

Now every user interface has some common elements, like buttons, scrollbars, input fields etc.

I would like to use Machine Learning to "interpret" such user interfaces in a way, in which I later can input a user interface as an image for example, and let the prototype then "try out" the interface, meaning, clicking on buttons, using scrollbars inputting some text into input fields etc.

I know that this would have to be done using Image Recognition, since there are many different UIs.

I am specifically interested in Websites, Adobe Reader with an opened PDF (that in turn can be a form etc.), and Word with an opened Document (again this can contain forms etc.).

Now my main question is if there is already some research going on in this field that I can use, or even an existing tool for parts of the process.

Any help is appreciated :)


texts: Splitting into multiple heads -- multihead self attention

texts:
So, I have a doubt in Attention is all you need:
The implementation of transformers on tensorflow's official documentation says:

Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads.

However, The paper mentions:

Instead of performing a single attention function with dmodel-dimensional keys, values and queries,
we found it beneficial to linearly project the queries, keys and values h times with different, learned
linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of
queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional
output values.

There is no mention of splitting Q,K and V to obtain heads. Instead, the paper says that they are passed through 'h' different dense layers to convert d-model dimensional vectors to 'h' different dk,dk and dv dimensional vectors respectively. So basically the pseudocode, from what I understood, should look something like this:
Q,K & V are d-model dimensional vectors.

for i in range(h):
   Qi = Dense(dk)(Q)
   Ki = Dense(dk)(K)
   Vi = Dense(dv)(V)
   Ai = Attention(Qi, Ki, Vi)

A0, A1, A2, ..., Ah are then concatenated.

Is this right? or am I missing something here?


texts:
# The code is as below.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

bankdata = pd.read_csv("bill_authentication.csv")
X = bankdata.iloc[:, 0:4].values
y = bankdata.iloc[:, 4].values

from sklearn.model_selection import KFold
kf = KFold(n_splits=10, shuffle=True,random_state=50)
for train_index, test_index in kf.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
Predicted=pd.DataFrame({'Actual':y_test, 'Predicted':y_pred})

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
Confusion_matrix = confusion_matrix(y_test,y_pred)
ClassificationReport = classification_report(y_test,y_pred)
AccuracyScore = accuracy_score(y_test, y_pred)

NewData = pd.read_csv("PredectionData.csv")
NewPredictededtion=classifier.predict(NewData)
NewPredictededtion=pd.DataFrame({'Variance':NewData.Variance, 'Skewness':NewData.Skewness, 'Curtosis':NewData.Curtosis, 'Entropy':NewData.Entropy, 'Predicted':NewPredictededtion})

I have problem

  when I run the code I have the prediction result for only one fold

how I edit code to show the predicted result of all folds


texts:
I trained a model in Keras with input dimension 15 and output dimension 1. Then I tried to predict the output for a single input np.array, which I chose to be a toy example np.arange(15). However, the input is not accepted. Can someone tell me where the problem is?
Here is the code for a simplified problem:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense
X = np.arange(15)
Y = 0
model = Sequential()
model.add(Dense(32, input_dim=15, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X, Y, epochs=10, verbose=1, batch_size=batch_size)
model.predict(X)

The following error occurs:
ValueError: Error when checking input: expected dense_4_input to have shape (15,) but got array with shape (1,).
But then again, the input clearly has the correct shape. What is going on here?
Thanks for your help!


texts: How does C have effects on bias and variance of a Support Vector Machine?

texts:
The minimization problem for SVM can be written as-
$$\overset{\text{min}}{\theta} C\sum_{i = 1}^{m}{[y^icost_1(\theta^Tx^i) + (1-y^i)cost_0(\theta^Tx^i)]} + \frac12\sum_{j = 1}^n{\theta_j}^2$$
Now, how can the choice of $C$ lead to underfitting or overfitting?
As I understand, parameters are chosen to make $C\sum_{i = 1}^{m}{[y^icost_1(\theta^Tx^i) + (1-y^i)cost_0(\theta^Tx^i)]}$ part $0$. And we concern ourselves with the second part.
And Andrew Ng says that a large $C$ leads to lower bias and higher variance.
How does this happen? What is the intuition behind this?


texts: Hyper parameters (window size and vector dimensions) tuning in word2vec using Grey Wolf Optimization

texts: Using Grey wolf Optimization, I want to calculate optimal values of two hyper parameters: context window size and embedding size (vector dimensions) for word2vec skipgram model used for word embedding. I want to use this model to find top 25 similar words of a token. But I don't know how to apply GWO to this problem.


texts: reduction of model accuracy while using PCA for a regression problem

texts:
I am trying to build a prection problem to predict the fare of flights. My data set has several catergorical variables like class,hour,day of week, day of month, month of year etc. I am using multiple algorithms like xgboost, ANN to fit the model

Intially I have one hot encoded these variables, which led to total of 90 variables, when I tried to fit a model for this data, training r2_score was high around .90 and test score was relatively very low(0.6).

I have used sine and cosine transformation for temporal variables, this led to a total of only 27 variables. With this training accuracy has dropped to .83 but test score is increased to .70

I was thinking that my variables are sparse and tried doing a PCA, but this drastically reduced the performance both on train set and test set.

So I have few questions regarding the same.

Why is PCA not helping and inturn reducing the performance of my model so badly
Any suggestions on how to improve my model performance?

code

from xgboost import XGBRegressor
import pandas as pd
import matplotlib.pyplot as plt

dataset = pd.read_excel('Airline Dataset1.xlsx',sheet_name='Airline Dataset1')

dataset = dataset.drop(columns = ['SL. No.'])
dataset['time'] = dataset['time'] - 24

import numpy as np
dataset['time'] = np.where(dataset['time']==24,0,dataset['time'])

cat_cols = ['demand', 'from_ind', 'to_ind']

cyc_cols = ['time','weekday','month','monthday']

def cyclic_encode(data,col,col_max):
    data[col + '_sin'] = np.sin(2*np.pi*data[col]/col_max)
    data[col + '_cos'] = np.cos(2*np.pi*data[col]/col_max)
    return data

cyclic_encode(dataset,'time',23)
cyclic_encode(dataset,'weekday',6)
cyclic_encode(dataset,'month',11)
cyclic_encode(dataset,'monthday',31)

dataset = dataset.drop(columns=cyc_cols)

ohe_dataset = pd.get_dummies(dataset,columns = cat_cols , drop_first=True)
X = ohe_dataset.iloc[:,:-1]
y = ohe_dataset.iloc[:,27:28]

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train_us, X_test_us, y_train_us, y_test_us = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
sc_Y = StandardScaler()
X_train = sc_X.fit_transform(X_train_us)
X_test = sc_X.transform(X_test_us)

y_train = sc_Y.fit_transform(y_train_us)
y_test = sc_Y.transform(y_test_us)

#Applying PCA
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)

X_train = pca.fit_transform(X_train,y_train)
X_test = pca.transform(X_test)
explained_variance = pca.explained_variance_ratio_

regressor = XGBRegressor()
model = regressor.fit(X_train,y_train)

# Predicting the Test & Train set with regressor built
y_pred = regressor.predict(X_test)
y_pred = sc_Y.inverse_transform(y_pred)
y_pred_train = regressor.predict(X_train)
y_pred_train = sc_Y.inverse_transform(y_pred_train)
y_train = sc_Y.inverse_transform(y_train)
y_test = sc_Y.inverse_transform(y_test)

#calculate r2_score
from sklearn.metrics import r2_score
score_train = r2_score(y_train,y_pred_train)
score_test = r2_score(y_test,y_pred)

Thanks


texts: How to backpropogate Convolution layer padding inputs with respect to output derivative

texts:
I created a convolution network with 5 Conv blocks, let discuss the issue based on Conv block 4 & 5
Conv Block 4
Input Image size         : 28 * 28,
Padding size 1           : 30 * 30 (image size after padding),
Filter size              : 3 * 3,
Convolution output image : 28 * 28,
Max pooling output       : 14 * 14
Conv Block 5
Input Image size         : 14 * 14,
Padding size 1           : 16 * 16 (image size after padding),
Filter size              : 3 * 3,
Convolution output image : 14 * 14 ,
Max pooling output       : 7 * 7
Full connected layer

Above output image  (7 * 7) has been flattened and passed to Full connected layer and the flattened input has been backpropogated(dL/dO) with respect to loss(gradient checker results are fine).

Now I am trying to backpropogate through the convolution layer as below.

Back Porp - Conv Block 5
Reverse Max pool:  7 * 7 (dL/dO) to 14 * 14 output (dL/dO)
Conv 5 Filter derivative 3 * 3 (dL/dF): (14 * 14 output (dL/dO)) * (16 * 16 Conv 5 input image with padding)
I have issue with the below
Conv 5 input derivative:
input size 16 * 16 with padding (dL/dX) =  (14 * 14 output (dL/dO)) * (3 * 3 filter) after 180 degree
The question is if I have input derivative of (16 * 16) in conv 5 how can I reverse max pool to Conv block 4 because Conv 4 max output was only (14 * 14).
I think conv 5 input derivative should be 14 * 14 so we can reverse max pool to 28 * 28 and backpropogate the Conv 4 block.
Please help me to resolve this, I am sure I am missing something.
Thanks


texts: When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

texts: I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?


texts: How to use Graph Neural Network to predict relationships between nodes with pytorch_geometric?

texts:
Let's say I have a partly connected graph that represents members of many unrelated communities. I would like to predict the possible friendships between members of the same community: on an sliding scale between 0 to 10 how likey would they like each other? I have some characteristics of them whether they are christian, or like sports, and also some geographical features, the distance between them.

The connections could be whether or not they are friends on a social media platform. In the networks, they are not necessarily connected with edges.

I am using pytorch_geometric to build a graph for each community and add edges for connections on the social media platform. One edge for each direction, so the graph is bi-directional. Then I create Data() instances.

Data(x=x, edge_index=edge_index)

Where x is an array with node features and edge_index

x = array([[ 0,  4,  6,  0,  0,  1],
   [ 1,  4,  6,  0,  0,  1],
   [ 2,  4,  6,  0,  0,  1],
   [ 3,  4,  6,  0,  1,  0],
   [ 4,  4,  6,  0,  1,  0],
   ...])

edge_index = [[0, 1],
 [0, 9],
 [0, 10],
 [0, 11],
 [1, 2],
 [1, 7],
 [1, 12],
 [2, 3],
 [2, 6],
 [2, 13],
 [3, 4],
 ...]

Not sure what is the best route from here to train on and predict relationships. What is generally used in this case? There are a few options mentioned in the documentation: EdgeConv, DynamicEdgeConv, GCNCon. I am not sure what to try first. Is there anything available that is made for this kind of problems or do I have to setup my own MessagePassing class?

Data() accepts an argument y to train on nodes. Can I actually use pytorch_geometric for this kind of problem or do I have to go back to pytorch?


texts: finding the mean for each of the channels (RGB) across an array of images

texts:
How can I find the mean for each of the channels (RGB) across an array of images?
For example train_dataset[0]['image'].shape is (600, 800, 3) and len(train_dataset) is 720 meaning it includes 720 images of dimension 600x800 and 3 channels. train_dataset[0]['image'] is an ndarray.
I am looking to end up with 3 numbers each representing the mean for each of the channels across all these 720 images.
I have this very dumb solution but I wonder if there's a better solution?

I also did it this other way and got almost the same answer:


texts: Forecasting with two separate periods of time

texts:
I have a sales forecasting problem where I only have three months of year 2018 ( Jan, fev, march) and three months of year 2019 (oct, nov,dec) and I have to forecast the sales for all the months of 2020.
I am new with the forecasting problems and what I can conclude from these data that it is time series data but it's so limited and the gap between the two years is about 18 months so handling missing data won't be a solution for this. I have been so stack can anyone help me please.
Can you suggest me what to do ?


texts: Is there an encoder which can automatically detect the intrinsic order of an ordinal variable and assign values accordingly?

texts: Given data with an ordinal variable, says "house quality" with values ex (excellent), gd (good), fa (fair) and bd (bad), we obviously cannot just throw data into sklearn's LabelEncoder as the resulting labels can be in wrong order, e.g. {bd: 3, gd: 2, fa: 1, ex:0}. Instead, we need to manually specify an order, right? However, if we do not have domain knowledge, how can we specify the order? Also a manual way is usually prone to error. Thus, I am curious if there is any encoder which can auto detect the correct order in an ordinal variable?


texts: Why should I care about seasonal data when I forecast?

texts: I have a timeseries with hourly gas consumption. I want to use ARMA/ARIMA to forecast the consumption on the next hour, basing on the previous. Why should I analyze/find the seasonality (with Seasonal and Trend decomposition using Loess (STL)?)?


texts:
import h5py #added
hf = h5py.File('../images.h5', 'w') #added
hf.close() #added

h5_file = tables.open_file("images.h5", mode="w")

I also tried:

h5py.File.close(hf)

the error that pops up in both cases is:

ValueError: The file 'restricted_images.h5' is already opened.  Please close it before reopening in write mode.

I've also tried:

if isinstance(obj, h5py.File):   # Just HDF5 files
    obj.close()

while

In[]: hf
Out[]: <Closed HDF5 file>

, the file is not closed yet.


texts:
Does anyone could explain in details how heap structure works in Cluster Algorithm?

I am planning to code CURE in Matlab (Clustering using representatives) but, at first, the paper mentions that all points of datasets are considerered as clusters and heap helps to merge new clusters and I do not know how exactly this data structure works in this algo.

Thanks for your support.


texts:
I read the following claim in this paper https://arxiv.org/pdf/1803.06305.pdf:
In one gate/cell, $W_{∗x} x_t + W_{∗r} y_{t−1}$ can be combined/fused
in one matrix-vector multiplication by concatenating the matrix
and vector as $W_{∗(x r)}[x_t, y_{t−1}]$ (page 3).
How is this the case? I would have thought since $W_{∗x}$ and $W_{∗r}$ are different matrices, the computations would have to be done separately? And does this combination (if it's really possible) save computational time?


texts:
First of all, let me define some terms:

Survival Bias
The data we collected is biased (because we only managed to recognise users who survived through the process)
Landing Page
The first website page which the visitors come from advertising campaigns
(usually comes with Call to Action, so that the visitors will signup)
Signup Rate
number of signup of landing page visitors / total number of landing page visitors

I am currently working on analysis of effect of X(landing page load time) to Y(signup rate).
At the first glance, we can plot the histogram of X as landing page loading speed(we can make them into bins) vs Y the signup rate.
However, most probably the result will be the longer the X, the higher the Y, which doesn't make sense, because longer page load time, will definitely decrease the rate of signup.
After some thought, I think this is because the number of visitors is already biased.
In data collection process, we only know the visitors who waited the page to be loaded completely. In other words, if someone leaves in the middle of pageload event, we have no way to recognise them. (And now, it make sense that the visitors which gone through long loading time is the ones with higher interest, because they are willing to wait). And because of this, we can see that the longer page load time users have higher signup rate.
Now,
i) Longer page load time > higher quality user in average(the one who survived) > higher signup rate
ii) Shorter page load time > lower quality user in average(basically all visitors, because page load time may only takes within 1 second) > lower signup rate
Is there a way that I can still come out the effect X on Y, despite the data is biased now.


texts:
What are the best free Business Intelligence (Reporting) tools, that allows live updates on the reports and also sharing them to other users?
Actually now I’m using the community edition of Pentaho but it’s limited in the number of output rows in the process of reporting. Please I need your suggestions, thank you.


texts: Modeling social media post scheduling optimization

texts:
Problem:

I want to maximize performance for social media posts by optimizing the time when they are published.

Current model:

X: publishing_datetime, post_attribute_1, ..., post_attribute_n
y: performance

Desired model:

X: post_attribute_1, ..., post_attribute_n
y: publishing_datetime

The desired model should predict the optimal publishing_datetime for maximizing performance. Once the data can be modeled like this,  the problem is solved with a regression neural network.

What I've tried:

Filtering the posts with above-average performance and using their attributes and publishing_datetime to form my desired model.

This is not ideal as a lot of data is unused and posts with particularly great performance influence just as much as posts with barely above-average performance.

Any suggestion on how to achieve this model transition?

All ideas and alternative approaches are very welcome. Thanks in advance!


texts: Prediction: plugin a corelation table (neuron) into a Time-Series Neuron in Keras/ TF

texts:
i am adding more details

I have a time series of Babies (1,2,3) showing how many problem they have each week (Born week 1 to week 80) and in which organ (14 organ).

There is a separate numeric time-free correlation tables for babies/organs. showing Xtype blood cells per organ per baby -> Causes problems

My aim is to predict how many problem will NewBaby (Baby4) have, in which organ, per week

Predict this: Week1: 1 problem | Stomach Week1: 1 problem | Liver Week2: 1 problem | Stomach ..

TIME_SERIES_INPUT ------> LSTM (RNN?) -------\
-------------------------------------------------------------------> MERGE --->
AUXILIARY_INPUTS --> [do something] ----------/*

How can I model this problem with Keras/TF? To be specific firstly should I modify the dataset to include correlation tables. Or can I plug a model into LSTM? I have seen some examples on this board but they are mostly CNN that does not apply to my data set Ps I can insert whole dataset if needed

Thanks

DataSet Looks like

Week Baby   Age Organ   Date    IssueFound Problem Type
2   Baby2   A0  Organ2  06/24/16    1   F
2   Baby3   A0  Organ11 05/01/17    1   F
2   Baby3   A0  Organ11 05/02/17    1   F
2   Baby3   A0  Organ13 05/03/17    1   F
2   Baby3   A0  Organ13 05/05/17    1   F
3   Baby2   A0  Organ12 06/29/16    1   F
3   Baby2   A0  Organ2  07/01/16    1   F
4   Baby3   A0  Organ13 05/18/17    1   F
4   Baby3   A0  Organ2  05/19/17    1   F

time-free Co relation Table looks like

New X-Type Blood Cell per Organ per baby

Organ   Baby1   Baby2   Baby3   NewBaby
Bladder Organ1  288 247 97  253
Brain   Organ2  2548    1972    992 251
Ears    Organ3  32  254 162 155
Eyes    Organ4  111 0   459 -89
Heart   Organ5  42  1214    0   142
Kidney  Organ6  4   188 186 -87
Kidneys Organ7  806 1764    796 153
Liver   Organ8  658 657 -20 0
Lung    Organ9  0   484 0   0
Nose    Organ10 3328    9940    4435    3223
Pancreas    Organ11 11  389 207 194
Skin    Organ12 664 766 164 -145
Stomach Organ13 246 -71 0   -70
Tongue  Organ14 1403    0   33  0


texts: How to identify the feature that make the model misclassifed in text classification

texts:
Hi I am working on social media financial THAI text classification, the problem with this one is the confused classes, the misclassified prediction has a pattern that consistent as a pair.
and I want to know how can I identify the word/feature that responsible for the misclassified and lead the model to confuse with the prediction?