texts: Calculate mean by decile in Svydesign object

texts:
So, I´m working with ENIGH - Database, which stands for ¨National Survey of Household  Income and Expenses¨ in Spanish, this is an exercise conducted by the Mexican government and like most surveys of its kind, it works with Weights.

What I´m trying to do is to calculate the mean, maximum and minimum household income by Decile.  In other words What´s the income of each 10%, grouping household base on their income.
To be honest, I haven’t gone that far but this is what I got until now:

I need my svydesign object
Convert that into a table using svytable
Arrange using desc() on my income variable

ENIGH_design <-svydesign(id=~upm, strata=~est_dis, weights=~factor_hog, data = ENIGH)
ENIGH_table <- svytable(ing_cor, ENIGH_design)

Here is where it gets tricky, supposing I have 100 rows, I can’t take the first 10 of them because in reality, when taking weights in mind, the might be 9% or 20% (I´m just throwing numbers) of the actual population.

I could use cut() on my income variable but I would be forgetting about weights and results will only be representative of the sample, not total population.

I think that the best approach would be to use a combination of:

mutate() to create a new variable base
if() in conjugation with mutate to define on which decile each row falls to
group_by() and mean() to calculate what I´m aiming for

This way I will have an extra variable which I could use to calculate whatever I want with whatever other variable I wish to. But again, I haven´t define my groups so it´s pretty much useless.

Thank you for reading. Thank you for your help.

Database available: https://www.inegi.org.mx/programas/enigh/nc/2016/default.html#Datos_abiertos

Here is a glimpse of how my DB looks:

folioviv    foliohog    ubica_geo   est_dis  upm  factor    ing_cor
100587003      1        10010000       2     610    180     22,723
100587004      1        10010000       2     610    180     17,920
100587005      1        10010000       2     610    180     27,506
100587006      1        10010000       2     610    180     56,236
100605201      1        10010000       2     620    178     41,587
100605202      1        10010000       2     620    178     135,437
100605203      1        10010000       2     620    178     62,386
100605205      1        10010000       2     620    178     103,502
100605206      1        10010000       2     620    178     27,323
100606301      1        10010000       3     630    223     68,042
100606302      1        10010000       3     630    223     98,537
100606305      1        10010000       3     630    223     53,237
100606306      1        10010000       3     630    223     132,861
100609801      1        10010000       3     640    232     190,033
100609802      1        10010000       3     640    232     28,654
100609805      1        10010000       3     640    232     74,408
100631401      1        10010000       1     650    171     80,761
100711503      1        10010000       1     770    184     38,640
100711504      1        10010000       1     770    184     81,672

There are many more columns but they aren´t necessary for this exercise.


texts: Find material noun in a sentence computationally

texts:
I want to find all material nouns in a sentence. Using pos_tag set of nltk only give NN, NNS, NNP etc. Is there any existing work to find out material nouns or how can we solve this problem?

Edit: An example will be

Input: Don't forget to bring water bottle, a blanket and shoes **during your visit.**

Output: [water bottle, blanket, shoes]


texts: Annotation tool for classification experiments

texts: What tools are available that provide an interface to present text classification results? I need to keep and compare the results of different classification algorithms for unsupervised data. The tool should allow users to give input (like the good match or not) for the presented examples of the test data.


texts: Keras del stuck with constant loss and accuracy

texts:
I am trying to train a keras CNN against the Street View House Numbers Dataset. You can find the project here. The problem is that during training neither loss nor accuracy change over time. I have tried with 1 Channel (Gray Scale) images, with RGB (3 channels) images, with wider (50,50) and smaller (28,28) images, with more or less filters in the convolutional layers, with wider and smaller patches in the pooling layers, with and without dropout, with bigger and smaller batches, with smaller and bigger learning step for the optimizers, with different optimizers, ...

Still the training gets stuck to constant loss and accuracy


texts:
I'm trying to download the COCO dataset images using the following COCO API command:

from pycocotools.coco import COCO
import requests
catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

... but getting the following error message. Any idea why is it so?

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-9cc4e2f62a0e> in <module>
----> 1 catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

TypeError: getCatIds() missing 1 required positional argument: 'self'

Previously I configured pycocotools using the following instruction:
https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/62381


texts:
I have a set of phrases that lead to a binary outcome (accept/reject) and I was wondering what techniques are most helpful for extracting key phrases that are most likely to determine the outcome, given that I have a training set of data that has the English-language phrase and the observed outcome.
To illustrate the idea let me give a simple example:
Accepted

Sounds great
That would be great
That's fine

key words: great, fine
Rejected

I'm not sure
I don't think so
No way

key words: not, don't, no


texts:
For a project in my data mining class, I am to perform fuzzy-c means clustering on a data set, where each data point has 3 axes.I'm not exactly sure how I would do so, especially given the clustering algorithm I'm using. Here's an example of the data set I'm using;

-
x
y
z

apple
2
5
5

banana
3
2
5

carrot
1
4
4

durian
6
7
1

eggplant
0
3
6

Any help or resources would be greatly appreciated!


texts:
First of all, I'm new in this field and it's my first this kind of work. I'm trying to train EfficientNet (CNN), the code below is working fine, but I can't succeed to add also validation set to the code below. My validation data is val_X and val_y.  I really will appreciate your help.
   def train(net):
        BATCH_SIZE = 64
        EPOCHS = 10
        for epoch in range(EPOCHS):
            for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs = net(batch_X)
                matches  = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)
                loss.backward()
                optimizer.step()
            print(f"Epoch: {epoch}. Acc: {round(float(acc),2)}  Loss: {round(float(loss),4)}")

    train(net)


texts: Upload 'tsv' file to Google Colab / Jupyter Notebook

texts:
TSV(Tab separated Value) extension file can't be uploaded to google colab using pandas

Used this to upload my file

import io
df2 = pd.read_csv(io.BytesIO(uploaded['Filename.csv']))

import io
stk = pd.read_csv(io.BytesIO(uploaded['train.tsv']))

What i want is a tsv file should be uploaded and read into the dataframe stk


texts: How to correctly set a target for a time series based model?

texts:
I need help determining the best way to go about creating the target variable for a machine learning model that is trading a financial instrument (stocks, foreign currencies, crpyto, etc).

Below is some sample data to help with answering the question.

Time: When the data was recorded
Price: Is the current price of the instrument and is the price I would be buying at.
price_good: Price level that I need to sell at to make a profit
price_bad: Price level that I need to sell at to minimize losses
good_id: Returns the ID of the first row where price >= price_good
bad_id: Returns the ID of the first row where price <= price_bad
target: If good_id < bad_id, target = 1. Else target = 0

My ideal scenario is when price_good occurs before price_bad. The issue I'm running into is how to correctly set the target field. I have 2 ways that I can solve this -

Option #1 - This option uses all of the data and has no restrictions on creating the target field but has a slight mismatch on how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           4         nan      1
3          01-03-19   109      120          99           4         nan      1
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan

Option #2 - This option doesn't use all of the data and doesn't allow a target to be set until the initial row reaches it's target which is how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           nan       nan      nan
3          01-03-19   109      120          99           nan       nan      nan
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan

"In production" means how it would work after the model is built. For example,
If my model predicted that id:1 would hit target_good, I wouldn't be able to make any other trades until that trade reaches price_good or price_bad which aligns with option #2. However, I'm losing the information about all of the other trades that I could of made from option #1 if I had more capital to invest.

If I go with option #1, the model will sometimes "overfit" because it has data that is closer together in time and uses mostly the same information.

If I go with option #2, it is a more accurate representation of how it would work in production but I lose a ton of data points. For example, what if I started at id:2 instead of id:1. I would end up with different data points that the model is built on. I could test starting at multiple places to find the most optimal starting place but that would take a ton of iterations and resources to test.

Any tips on how to model this correctly? Thanks!


texts: Neural Network unable to track training data

texts:
I am new to ML and this is my first Tensorflow project. I am doing regression with Neural Networks on a dataset with 17 features and 1 outcome. But for some reason my network is unable to follow the training data. I am getting massive errors in results, as can be seen from the plots below. I have also tried experimenting with different parameters (learning rate, nodes per layer, number of layers etc) but nothing seems to work.
I have pasted the Tensorflow code here. I have also provided my cost and training plots as well as a link to my datasets. I'd be grateful if someone could please help me figure out what I am doing wrong.
Thank you!

Links to the datasets-
Features arranged in columns-
https://drive.google.com/file/d/1U182Lhf67WygeSbv6BNEx5LHyL7Ba13O/view?usp=sharing
Output column -
https://drive.google.com/file/d/10XWo1d5mhIsxccQBgAyGDWDAVgu2BjAA/view?usp=sharing

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
validation_filename_X = "validation_set_X.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
validation_filename_Y = "validation_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
validation_features = np.loadtxt(validation_filename_X, delimiter=',')
validation_observations = np.loadtxt(validation_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 5
n_nodes_hl2 = 5
n_nodes_hl3 = 3
no_features = 17
learning_rate = 0.001
epochs = 2000

cost_history = []

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:
    sess.run(init)

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
#        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history.append(sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_training_features})

    plt.plot(range(len(cost_history)), cost_history)


texts: Machine Learning using NVIDIA DIGITS - Can't Classify Left or Right Direction of a Ball Throw

texts:
I'm using NVIDIA DIGITS 3.0 to do training for detecting direction of a ball throw.

My Dataset contains 400+ binary images each for left and right throw with the following specs in DIGITS:

Image Type: Grayscale, JPG
Image Size: 256 x 256
Resize Transformation: Fill

My Classification Model specs:

Solver Type: NAG
Networks: GoogLeNet
The rest are default values

Problem:

I separated out 8 images (4 for left, 4 for right) from my 400+ training dataset to do testing on my model. The results returned from my classification model always classifies all 8 are left throws at 90%+ accuracy even though 4 of which are clearly right throws.

2 of the test images are shown below. My subject always stands at the same position to throw the ball, which is more or less constant. I've tried to change various parameters in the classification model like no. of epochs, subtract image mean, etc. to no avail.

In a previous dataset, my model classifies all throws as right throw, similarly only one direction is detected. I'm unsure whether the problem lies in my dataset images or the training model. I would greatly appreciate any advice.

Left Throw:

Right Throw:


texts: Is Pearson coefficient a good indicator of dependency between variables?

texts:
Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a good answer since as demonstrated in the figure below, dependency between two variables may not be linear and the Pearson coefficient can be close to $0$ (parabola, circle). I have seen people on Kaggle always starting with correlation matrix and discarding data that are not correlated.

My question is: is Pearson coefficient always a good measure of correlation between variables and we should always rely on it? Can you give practical example when some variable in your problem was important, but Pearson indicated there is hardly any correlation?


texts: Target and output shape/type for binary classification using PyTorch

texts:
so I have some annotated images that I want to use to train a binary image classifier but I have been having issues creating the dataset and actually getting a test model to train. Each image is either of a certain class or not so I want to set up a binary classification dataset/model using PyTorch. I had some questions:

should labels be float or long?

what shape should my labels be?

I am using a resnet18 class from torchvision model, should my final softmax layer have one or two outputs?

what shapes should my target be, during training, if my batch size is 200?

what shape should my outputs be?

Thanks in advance
Quote Delete


texts:
My dataset has 3 fields ColumnX, ColumnA, Column B as below.

Column X, Column_A ,      Column_B

AA, 1     ,        KEYWORD1 KEYWORD2 KEYWORD3

AA, 2    ,         KEYWORD4 KEYWORD5 KEYWORD6

XX, 3 , KEYWORD3 KEYWORD7

XX, 4, KEYWORD6 KEYWORD8

YY, 5, KEYWORD9

I removed the stop words in Column_B and kept the keywords.

So here Can I train a model which can classify the data with Column_X and Column_B and given an input from user which matches any keyword in Column B , returns Column_A

Sample output:

user gives an input : KEYWORD9.

my output should show ColumnX: YY and ColumnA: 5

I know it can be done with basic python but I want to use ML here


texts: Boostrap parameter in random forest regressor?

texts:
There's one parameter in RandomForestRegressor which is bootstrap. By default
bootstrap=True

  bootstrap : boolean, optional (default=True)

  Whether bootstrap samples are used when building trees.

So from the docs if I set bootstrap=False then I guess bootstrap samples are not used but I'm really confused on what is bootstrap samples mean here?

There were explanations but it's really confusing. Can someone please explain it in a simpler term? And also does bootstrap=True help in improving the model accuracy?

Thank you.


texts: Why don't we set initial hidden states in RNN to random small numbers like we do to the weights?

texts: I'm following a couple of tutorials on RNNs and the instructor said that we should always set the initial hidden state in our RNNs to a tensor of all zeros and I couldn't really understand why. Even when I looked online, it said that fixing the hidden state to zero is a common choice and that this has been shown to force the RNN to focus on the input sequence regardless of the initial choice of the hidden state. What do they mean by that exactly?


texts:
I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it.

Initially, I used TextBlob library for sentiment analysis to generate a polarity score. But the polarity score being generated for news headlines are not accurate. It is classifying negative news as positive news.

For example: Goldman Sachs CEO apologizes to people of Malaysia. This news is being classified as positive.

Post that, I've tried to build a custom model using spaCy library. I trained the custom model on 500 manually labelled news headlines. After training the model, I ran it on new set of headlines. The accuracy has improved but there is still scope for improvement. The accuracy is particularly poor with regard to financial news and editorial headlines.

My expectations is to create a model that can accurately predict the sentiment of news headline. So that I can plot sentiment trend about the subject over a time period.


texts: Data prediction using scikit-learn and a list

texts:
I have a group of lists detailing temperatures over differing amounts of time. My goal is to use machine learning to identify periods in which a machine is turned on and off, where turning on the machine drastically increases temperature, and turning off the machine returns it to idle temperature.

The rub is that there is a level of temperature inertia- take the following example.

Among my control data is the following list (truncated for simplicity's sake):

At minute intervals, the temperature starts at 73 degrees. It is turned off after 12 minutes, when the temperature is 83 degrees. However, the peak is reached 3 minutes after the stopping point, at 86 degrees.

Given a control group of a list with labelled starts, ends, and peaks, how would I go about using supervised learning to create an algorithm that could predict stops using a list with only starts and peaks?


texts:
I am working on a binary classification problem.

I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below

logreg=LogisticRegression(random_state=41)
logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')
logreg_cv.fit(X_train_std,y_train)
y_pred = logreg_cv.predict(X_test_std)

But I came across the term bootstrap validation on how it can be used to avoid test data poor performance due to split imbalances (in test data).

Can someone guide me on what is bootstrap validation and how to implement this?


texts: Training value neural network AlphaGo style

texts:
I have been trying to replicate the results obtained by AlphaGo following their supervise learning protocol. The papers specify that they use a network that has two heads: a value head that predicts the winner of the game and a policy head that predicts the next move based on the current state of the board.

For the policy head, no problem, the CNN learns to predict the next move. However, the value head does not learn anything and always falls back to always predicting 0 instead of -1/1 for white/black victory. It feels like the model is not complex enough to understand how to predict the outcome of the game so it learns to predict 0 instead to minimize the loss.

Things that I tried and didn't work:

Divide the network into two: one value net and one policy net. But even in this case the value net does not learn.
Label the winners as 0/1 instead of -1/1 and therefore use a sigmoid instead of a tanh activation function, but that didn't work either.
Different number of residual units (3/5/7/11).
Only using 2 channels as input, one for black stones and one for white stones.
Copy the weights of the first layer of the policy net and freeze them to force the value net to use the already learned features.

Here is the model:

def residual_block(s):

    shortcut = s
    s = Conv2D(256, (3, 3), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(s)
    s = BatchNormalization()(s)
    s = ReLU()(s)
    s = Conv2D(256, (3, 3), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(s)
    s = BatchNormalization()(s)
    s = add([shortcut, s])
    s = ReLU()(s)

    return s

def model():

    inputs = Input(shape=(2, 19, 19, ))

    # Convolutional block
    x = Conv2D(256, (3, 3), strides=(1, 1), padding='same')(inputs)
    x = BatchNormalization()(x)
    x = ReLU()(x)

    # Residual block
    for i in range(11):
        x = residual_block(x)

    # Value head
    x = Conv2D(1, (1, 1), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(x)
    x = BatchNormalization()(x)
    x = ReLU()(x)
    x = Flatten()(x)
    x = Dense(256, kernel_regularizer=l2(0.0001))(x)
    x = ReLU()(x)
    x = Dense(1, activation = 'tanh')(x)

    return Model(inputs=inputs, outputs=x)

And here is the training :

batch_size = 32
gen_train = generator(batch_size, sgf_paths)
gen_test = generator(batch_size, sgf_paths)

optimizer = SGD(lr=0.01, momentum=0.9, decay=0., nesterov=True, clipnorm=1.)
checkp = ModelCheckpoint(filepath="weights_ValueNet.h5", verbose=1, save_best_only=True)

model = model()
model.compile(loss="mean_squared_error", optimizer=optimizer)
hist = model.fit_generator(
        gen_train,
        steps_per_epoch = 2048,
        epochs = 100,
        shuffle = True,
        verbose = 1,
        validation_data = gen_test,
        validation_steps = 300,
        callbacks = [checkp])


texts: Impulse Response Function - Negative Shocks on R

texts:
I have two questions on how to produce impulse responses using R

(1) Impulse responses to a negative shock in the independent variable (money supply)

(2) Impulse responses at 2 standard deviations

The code I used to generate the impulse responses to a positive shock at 1 standard deviation is the following:

m1 <- read.csv("m1.csv", header=T)

m1

varm1 <- VAR(m1, p=8, type="cons")

irfm1 <- irf(varm1, impulse="m1", response= c("gdp"), boot = FALSE)

plot(irfm1)

irfm1


texts:
I need to find a decent online annotation tool to transcribe audio.

There are some requirements for a potential tool:

I should be able to deliver audio files to a few labelers. I should
be able to track which files went to which labeler.
It should be safe in terms of data storage.

Any suggestions?


texts: how to handle values that only appear once in a column?

texts:
Counting the values of a column using pandas I got the following result:

Human                 195
Mutant                 62
God / Eternal          14
Cyborg                 11
Human / Radiation      11
Android                 9
Symbiote                8
Kryptonian              7
Alien                   7
Demon                   6
Atlantean               5
Alpha                   5
Asgardian               5
Cosmic Entity           4
Inhuman                 4
Human / Altered         3
New God                 3
Animal                  3
Saiyan                  2
Eternal                 2
Frost Giant             2
Human-Kree              2
Demi-God                2
Human / Cosmic          2
Vampire                 2
Metahuman               2
Amazon                  2
Icthyo Sapien           1
Czarnian                1
Rodian                  1
Martian                 1
Clone                   1
Zombie                  1
Maiar                   1
Yoda's species          1
Human-Vulcan            1
Zen-Whoberian           1
Mutant / Clone          1
Korugaran               1
Dathomirian Zabrak      1
Parademon               1
Kaiju                   1
Flora Colossus          1
Human-Spartoi           1
Yautja                  1
Ungaran                 1
Human-Vuldarian         1
Neyaphem                1
Xenomorph XX121         1
Bizarro                 1
Human / Clone           1
Gungan                  1
Bolovaxian              1
Talokite                1
Luphomoid               1
Tamaranean              1
Kakarantharaian         1
Spartoi                 1
Strontian               1
Gorilla                 1
Name: Race, dtype: int64

I am new to data science, but I think that all those value appearing only once in the dataset is not going to help the classifier, so is there a good way to handle those values? I was thinking about grouping all those values that appear less than 5 times, or maybe I should remove the lines. By the way, I do not know if it is important to know, but I want to apply the gaussian naive bayes, knn and logistic regression to this dataset. This column is a feature to predict a binary value.


texts: How to stop a neural network from regressing to the mean

texts:
I'm very familiar with neural networks for classification, but I'm trying a regression task for the first time. I'm finding that the network tends to go towards guessing a mean for the whole dataset rather than making case specific predictions.
What are the potential reasons for this behaviour and how can I stop it?
Hint
In my specific case this behaviour is present in both the training and validation set. The training set is very small, and it's questionable as to whether the inputs have a strong correlation with the target.
Also might be worth noting that I'm training on images. So I'm using a CNN.
Update
I did some inspection of my model and found that it drives the output of my CNN backbone to 0 and simply uses the single bias on my fully connected layer to produce the output. I also found that if I freeze the bias during training, I can increase the learning rate indefinitely without the loss blowing up...


texts: Sequence prediction with unlimited predictions

texts:
I have a special kind of prediction problem.

I have observed $M$ sequences $X_m = [x_1, x_2, ..., x_N]$ where the distance $d$ between $x_n$ and $x_{n+1}$ is drawn from the same normal distribution, eg $d \sim N(\mu, \sigma^2)$. I can learn the parameters $\mu, \sigma$.

Now I need to predict/generate a whole sequence at once where a prediction $\hat{x}_n$ is considered correct if it falls within some absolute tolerance of the true data point $x_n$. There is one caveat: I can make as many predictions as I want without penalty but there must always be a minimum distance $\epsilon$ between predictions $\hat{x}_n$ and $\hat{x}_{n+1}$, where we can safely assume $\epsilon << d$. Intuitively, this makes me want to predict a pattern rather than trying to predict each point individually.

To re-iterate: a prediction that is outside the tolerance of a true data point is not penalized. We only need to maximize the number of correct predictions (that fall within the tolerance of a true data point).

Example 1

Prediction:       [10, 20, 30]
True observation: [11, 21, 31]
------------------------------
3 correct predictions if tolerance >= 1, else 0 correct predictions

Example 2

Epsilon = 4 (eg we can predict with a minimum distance 4)

Prediction: [6, 10, 15, 22, 30, 35]
True observation:      [11, 21, 31]
-----------------------------------
tolerance = 1 => 2 correct predictions (10, 30)
tolerance = 2 => 3 correct predictions (10, 22, 30)

What would be a good way to approach this problem? Are there problems that are similar?

Edited for clarity.


texts: Seq2Seq Model training: Encoder vs. Decoder

texts:
Can someone point me to an article which explains how the model training is done in Seq2Seq? I know "Teacher Forcing" is used but what I found so far hasn't been detailed enough. What I am most confused about is where the training happens? The back-propagation goes back to the encoder?

Any insight here would be really appreciated. Thanks!


texts: How do I calculate the accuracy for graph mining in terms of (top 1%)?

texts:
I have 3600 samples in my dataset. I split the dataset into the train (2700) and test (900).
My problem is related to new link prediction. I am using the Common Neighbor (CN) method. Using CN, we can predict new links based on the score. The score comes from the CN formula. CN gives a score for every possible edge/path/link. A higher score means a higher possibility to be a new link.
Now, I applied my train dataset to the CN methods. From CN, I have received 4200 new links with a score. From this 4200, I took only the top 1% (based on the score given by CN). So, the top 1% is 42.
Now, I tried to match the 42 with my test dataset. Means, actually, how many edges/links/path I can recover which were missing in the training dataset.
Among these 42 (CN output), 38 samples are perfectly matched with the test dataset (means same edge/link/path of the train and test). But, I did not give or these 38 edges/link/paths were missing in the training dataset.
True Positive: 38
False Positive: 42-38 = 04
False Negative: 900-38 = 862
True Negative: 900 - (38 + 04 + 862) = -04

From this, I am trying to calculate Precision, Recall, Accuracy. But, what is happening, I am getting very little Accuracy and Recall.
Can you give me an idea to improve the Accuracy and Recall?


texts: How does attention for feature fusion works

texts:
I am struggling to understand how would a self-attention layer be used for features of different modalities fusion.
What I understand until now is that :
Every unique modality is fed into a self-attention layer, this layer produces attention scores for every feature of that modality. So these scores give us information about which features are most important in that modality.
And then I have read that using these scores we can find out which of the modalities is most important, and when fusing the final feature vectors, use that as weights for the modality, and the attention scores as feature weights.
Am I understanding this correctly? How is the modality weight score computed and how are feature weights assigned?  By multiplying the features to their scores and concatenating ?
Here are several papers that have used attention for feature fusion, but I do not understand how it actually works and the math behind it.
Self-attention multimodal fusion,
Self attention for feature fusion in NLP,
Self attention for feature fusion in sensor data


texts: How to find similar meaning log messages in the source code?

texts:
Currently we have 10 thoundands log messages in the source codes which is maintained by different groups.
Some messages may be similar, but the sentence is not the same. My purpose is to create a centralized file which contains the unified message for reference.
For example:

printf("Failed to open the file %s", file)
printf("%s: failed to open file %s", TIMESTAMP, file)
printf("Failed to create the file %s")

I can extract all the message body which is in bold.They are not strict English sentence. but manually categorizing them takes too much time.
Is there a way to automatically categorizing the logs?
I just want to reduce the manual work.


texts: Anconda R version - How to upgrade to 4.0 and later

texts:
I use R through the anaconda navigator, which manages all my package installations. I need to use qgraph for a project, which is dependent on mnormt library, which in turn needs RStudio verion >4.0
I think the solution to my problem would be to upgrade Anaconda R to the latest R version, but the r-base search in conda has no R version greater than 3.6. Any help in upgrading Anaconda R to newer versions would be very useful.
The problem is in installing qgraph, which stops with errors in installing dependencies (pysch)
During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_TIME failed, using "C"
3: Setting LC_MESSAGES failed, using "C"
4: Setting LC_MONETARY failed, using "C"
Error: .onLoad failed in loadNamespace() for 'mnormt', details:
  call: library.dynam("mnormt", pkg, library)
  error: shared object 'mnormt.dylib' not found
Execution halted
ERROR: lazy loading failed for package 'psych'


texts: I have multiple labels per image. is it better to train taking each each label separately or should i mark all the labels present as 1 in the same image? which method is better? i will be using CNN architecture


texts: LeCun paper on deeplearning (Nature, 2015)

texts:
As I was reading Y. LeCun's paper on Deep Learning (Nature, vol. 521, 2015), I came across a figure (the 1st one in the paper) which was associated to the backward pass during backpropagation through a convolutional network. I understood everything apart from the very last sentence of the legend that accompanied the figure: "Once the $\frac{∂E}{∂zk}$
is known, the error-derivative for the weight $w_{jk}$ on the connection from unit j in the layer below is just $y_j\cdot\frac{∂E}{∂zk}$".

It must be purely mathematical but I can't figure out why the error-derivative mentioned above equal $y_j$ times $\frac{∂E}{∂zk}$. Can someone explain to me this result?

Thx.

PS : link to the figure mentioned above is https://www.researchgate.net/figure/Multilayer-neural-networks-and-backpropagation-a-A-multi-layer-neural-network-shown-by_fig4_277411157


texts: Impose similar metric on segments to model

texts:
I am training a binary classifier in a dataset using AUC as a score. The dataset has two main groups (we will refer to them as good and bad population). A property that this dataset has is having a higher proportion of target = 1 in the bad population.

For this reason, a relatively dummy classifier would give higher scores to the bad population and lower scores to the good population. In fact, the AUC of the classifier could be pretty high globally, and, when looking at the AUC inside both populations separately, the AUC might be really low in both of them.

I want to avoid this behavior. In fact, I am willing to sacrifice some AUC in the global population such that the AUC in each group is not very low. An idea that I had was using the harmonic mean of the AUC of both groups as a metric instead of the general AUC. However, this might not really help a classifier in a natural way.

Are there any papers/techniques/software that can help me in solving this problem in a more natural way?


texts: Predicting results of tennis matches based on historical data

texts:
I am writing my Bachelor thesis in Python about predicting results of tennis matches based on historical data. I have started from Logistic Regression but my model isn't efficient. If you could look at it and tell me, what should I change. I am just getting started and I am not really sure if I understand these methods. I am using data from this repository https://github.com/JeffSackmann/tennis_atp, but i changed it a bit for my needs https://drive.google.com/file/d/1w8weQxhsMBvYwrDPVTuiM3diFxtsXaky/view?usp=sharing.
Performance of this model isn't really good and I would like to improve it.
I will deal with missing data, by changing it to mean, median or mode and check which one is best.
import numpy as np
import pandas as pd
import random
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression

def df_mean(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].mean(), inplace=True)
   return frame

def df_median(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].median(), inplace=True)
   return frame

def df_mode(frame):
   index = np.where(frame.columns.values == 'minutes')[0][0]
   columns = frame.columns[index::]
   for i in columns:
      frame[i].replace(np.NaN, frame[i].mode(), inplace=True)
   return frame

For the first time i chose mean
df = df_mean(pd.read_excel('matches 00-19.xlsx')).dropna() df.index = range(61475)

frame = df[['winner_name', 'loser_name', 'winner_name']]
frame.columns = ["Player1", "Player2", "Winner"]
X = frame[["Player1", "Player2"]]
y = frame[['Winner']]

In the data I had, in the dataframe the winner was always on the left and the loser on the right so I had to mix it.
for i in range(X.shape[0]):
   rand = random.getrandbits(1)
   if rand == True:
      X.iloc[i][0], X.iloc[i][1] = X.iloc[i][1], X.iloc[i][0]

y1 = pd.DataFrame(data=np.zeros(shape= (len(y),1)), columns= ['Winner'], dtype= np.int64)
for i in y.index:
   if y['Winner'][i] == X['Player2'][i]:
      y1['Winner'][i] = 1

y = y1

I had to change names of the players to numbers so I made a set of all the players and than made a dictionary of it. I know I could use 'winner_id' but those numbers are too big.
s1 = set(frame['Player1'])
s2 = set(frame['Player2'])
set_merged = sorted(s1.union(s2))

d1 = {}
for i in range(len(set_merged)):
   d1[set_merged[i]] = i

keys = list(d1.keys()) X2 = pd.DataFrame(data=np.zeros(shape= (len(X),2)),
columns= ['Player1','Player2'], dtype= np.int64)
for i in X['Player1'].index:
   if X['Player1'][i] in keys:
      X2['Player1'][i] = d1[X['Player1'][i]]

for i in X['Player2'].index:
   if X['Player2'][i] in keys:
      X2['Player2'][i] = d1[X['Player2'][i]]
X = X2

X_train = X[:49180]
y_train = np.ravel(y[:49180])

X_test = X[49180:]
y_test = np.ravel(y[49180:])

model = LogisticRegression()
model.fit(X_train, y_train)
model.score(X_test,y_test)


texts: How to interpret gradient descent in boosting ensembles?

texts:
I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) sequentially -- each subsequent one is learning from the errors of the previous ones (by upweighting the misclassified examples, if I see correctly) and combining the results.
(Subquestion: does this combination mean that we use all the subsequently trained constituent estimators, maybe with different weights, or we just take the final one, which is assumed to be the most accurate?).
However, I cannot figure out how gradient descent and learning rate comes into the picture here. Trees themselves are not gradient based learners, and combining the output (either way) doesn't require any optimization. So what is its role?


texts: Which type of regression has the best predictive power for extrapolating for smaller values?

texts:
I have a data set which deals with response variable in the order of 10-20. The scatter plot for such a regression appears linear, but the problem being when I predict for test cases using values very small compared to the trained sample the predicted response variables appear in negative values. Please note that the values of the data set predicted should not be negative.

Here is the 3d scatter plot of my data

Is there a type of regression which has better predictive power which can perform such operations without such an error?


texts: Euclidean distance for more than two datapoints

texts: I have n 3-dimensional vectors. Is there a way to find distance between all using euclidean distance?


texts: it is possible to use features maps of CNN to localised important areas in image?

texts:
I'm new in deep learning and CNN, I understand how convolutional and pooling layers work, I understand how and why feature maps are created.  How I can localize from the feature maps important area in the original image? is that possible?
I.e. I use my own  data  ( medical image )
Feeding those images to my neural network I can receive maps features and calculate from them the probability of each class.
But how from that feature maps I can get know where exactly those areas are?
how can I backtrack the weights?
any idea?


texts: What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?

texts:
Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer.
I have a pretty basic idea how the complexity of algorithms are calculated, however, with the presence of multiple factors that affect the performance of a GRU network including the number of layers, the amount of training data (which needs to be large), number of units in each layer, epochs and maybe regularization techniques, training with back-propagation through time, I am messed up. I have found intriguing answers for neural networks complexity out here-  https://ai.stackexchange.com/questions/5728/what-is-the-time-complexity-for-training-a-neural-network-using-back-propagation, and bi-directional recurrent neural networks here- what is the complexity of a bidirectional recurrent neural network? but that was not enough to clear my doubt.
I am aware that back-propagation through time is used for training the recurrent neural network. But I am not able to understand how this happens for the bi-directional versions of the recurrent neural networks?
So, I was hoping if anyone help me with how to:

Derive the time-complexity of GRU networks for training via back-propagation through time?
Understand with an example the training of bi-directional recurrent neural networks using back-propagation through time? (I tried following the original paper https://ieeexplore.ieee.org/document/650093, but it was kind of confusing for me when they perform the backward pass for training)
Suggest me some research papers, where authors have taken up the time-complexity of recurrent neural networks in their applications and explanation (which I have tried searching, but with not much success)


texts: Changing default values of ANNIE resources in GATE from Java code

texts:
In GATE, default values for ANNIE are set during initialization, but sometimes based on requirements they have to be changed.
My Requirement : I want to extract English sentences without considering the "nextline character" but considering "full stop" which gives correct sentences. For that, I need to change the default value of transducerURL in SentenceSplitter in ANNIE.
This can be done in two ways:

Using ANNIE_with_defaults.gapp - changing initparams value in Sentencesplitter and accessing from java:
 Gate.setGateHome(new File(Configuration.GATE_HOME));
 Gate.init();
 // Load ANNIE with defaults from the plug-ins folder
 File pluginsHome = Gate.getPluginsHome();
 File anniePlugin = new File(pluginsHome, ANNIEConstants.PLUGIN_DIR);
 File annieGapp = new File(anniePlugin, ANNIEConstants.DEFAULT_FILE);
 annieController = (CorpusController) PersistenceManager.loadObjectFromFile(annieGapp);

I am not being able to find where to change its value in gapp file.

After initialising GATE, accessing ResourceData using CreoleRegister and changing the default value of parameter transducerURL as shown below:
 String resourceClassName = "gate.creole.splitter.SentenceSplitter";
 ResourceData resData = Gate.getCreoleRegister().get(resourceClassName);
 //System.out.println(resData.getParameterList());
 ParameterList params = resData.getParameterList();
 List<List<Parameter>> param =  params.getInitimeParameters();
 System.out.println(param);
 //System.out.println(param.get(0));
 for(List<Parameter> plist : param)
 {
     for(Parameter pm : plist)
     {
         System.out.println("Name : "+pm.getName()+", "+pm.getDefaultValue());
     }
 }

But Parameter does not have setDefaultvalue method.
How can I fix this?


texts: What is the effect of KL divergence between two Gaussian distributions as a loss function in neural networks?

texts:
In many deep neural networks, especially those based on VAE architecture, a KL divergence term is added to the loss function. The divergence is computed between the estimated Gaussian distribution and prior. Since Gaussian distribution is completely specified by mean and co-variance, only those two parameters are estimated by the neural network. For Gaussian distributions, KL divergence has a closed form solution. By minimizing KL divergence, we bring the estimated distribution closer to the prior.

My question is, since Gaussian distribution is completely specified by mean and co-variance, why don't we just take MSE between estimated parameters and prior parameters? Minimizing MSE between mean and co-variance also brings the two distributions closer. Does taking KL divergence have any significance?


texts: Why do the training curve fall sharp suddenly?

texts:
I am training a CNN classifier on a binary balanced dataset. The dataset has 4500 numbers of tweet data along with the class of the tweet. During training, I am applying, GLOVE embedding of 300 dimensions, 'adam' solver to run the model for 33 times of epochs. Besides, the sequence length I have considered is 31.

I have applied 200 filters which include a number of convolution2dlayers,batch normalization layers, relu layers, dropout layers and max-pooling layers. The drop out I have considered is 0.2 and the max pool layer is of size [1 sequence length].

The training curve is approaching smoothly until the end period where it has fallen sharply. Here, I have attached the training plot I receive:

Would you please explain to me why does this sudden fall occur? And how could I get rid of this?

Thanks,


texts: Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?

texts:
I have scoured the internet and books, but everything seems to use num_steps and batch_size or similar terms interchangeably and I can not get a grasp on their specific use, or where batches fit in when talking about RNNs ( I understand their use with Gradient Descent, and maybe I am misunderstanding RNNs and I need to be set straight at the fundamental level- if that is the case, please do so.

So where does batching apply in Recurrent Neural Networks? At the level of instances within time steps or at the level of timesteps themselves.

By my current understanding the data fed into the model is structured like this:

data = [
    "time_step_1": [
        "instance_1": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_2": [
        "instance_2_but_first_in_time_step_2": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_3": [
        "instance_3_but_first_in_time_step_3": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_4": [
        "instance_4_but_first_in_time_step_4": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],
    "time_step_5": [
        "instance_5_but_first_in_time_step_5": [
            "feature_1": "some_value1",
            "feature_2": "some_value2",
            "feature_3": "some_value3",
            "feature_4": "some_value4"
        ],

    ]
]

Except: That many people (and examples) have more instances per time step.

A typical training loop looks like:

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        run(model, data[time_step_count])

And that is if you had (for example) 10,000 instances at each time step, then batching would allow the weights to be updated more often than having to wait for the entire 10,000 instances to be processed. Then the loop would look like:

batch_size = len(data) // do_num_batches

for epoch in range(do_num_epochs):
    for time_step_count in range(len(data)):
        for batch_count in range(do_num_batches):
            batch_data = data[time_step_count][batch_count * batch_size : (batch_count + 1) * batch_size]
            run(model, batch_data)

But there is no point in processing the data in batches if there is only a single instance of data per time step is there?

I made a chart to try to explain ( to myself as well ) what the flow and data structure would look like:

I am just trying to understand the role of the batch in all of this, how it fits in?

(Edit/Addition)

After some more research on the subject, I am fairly certain that batches are a way to feed in sequences in chunks, and control the frequency of weight updates. By utilizing batches, the weights are updated at batch level, thus letting you group time-series together in groups. I illustrated what I believe to be right below, in the below example there are:

4 instances,
each with 5 features
across 30 time steps

In the first example (mini-batch), there are 3 batches, of batch_size= 10 in that example, the weights would be updated 3 times, once after the conclusion of each batch.

In the second example, is online learning with an effective batch_size=1 and in that example, the weights would be updated 30 times, once after each time_series

In the final example (Batch learning), the entire data set is processed at once, the weights are updated only once (1) at the conclusion of the 'batch'.

In each of these example, the entire data set would be run through many multiple of times, each one of these cycles is an epoch.


texts:
I want to train an LSTM with attention for translation between French and a "rare" language. I say rare because it is an african language with less digital content, and especially databases with seq to seq like format. I have found somewhere a dataset, but in terms of quality, both french and native language sentences where awfully wrong. When I used this dataset, of course my translations where damn funny ...
So I decided to do some web scraping to build myself my parallel corpus and it might be useful for research in the future.
It worked well and I managed to collect some good articles from a website containing some articles (monthly, since 2016 in both languages). Now the tricky part is putting everything into sentence to sentence format. I did a trial with a text and its translation just by tokenizing into sentence and I noticed that for example I had 23 sentences for French and 24 for native language.
Further checking showed that some small differences where notices in both languages, like a sentence where a comma was replaced in the other language by a dot.
So my question is :
Is it mandatory to put my articles into sentence-French to sentence-Native language format ? Or can I let it as text / paragraphs ?


texts:
I am working on LSTMs and LSTM AutoEncoders, trying different types of architectures for multivariate time series data, using Keras.
Since it is not really practical to use relu in LSTM because of exploding gradients, I added a Dense layer following LSTM, so it is like:
model = Sequential()
model.add(LSTM(number_of_features, batch_input_shape = (batch_size, time_steps, number_of_features), return_sequences = True))
model.add(Dense(number_of_features))

What I want to know is:

Is this fully connected Dense layer connected to only the last step in LSTM? Or does it add a fully connected Dense layer for all time steps?

When I checked the number of parameters to be sure about this. Dense layer has number_of_features $\times$ (number_of_features + 1) parameters, which implies this Dense layer is applied to all time steps in LSTM network. This makes sense since I set return_sequences = True, but even when I set it to False, this does not change, which made me doubt my understanding.
So,

How does Dense work with LSTM with Return_Sequences?
What is its different from TimeDistributed layer?
Why changing return_sequences to False did not result in a reduction in number of parameters of Dense layer, from number_of_features $\times$ (number_of_features + 1), to (number_of_features + 1)?


texts: Python Time series: extracting features on a rolling window basis

texts:
I have a long univariate time series, and before performing some machine learning models with it, I want to extract as many features as I can from the time series on a rolling-window basis.

As a quick example, for a window of size 10, I would like to calculate statistics like mean and std deviation for the first t=0:9 points in my dataset, and have those two results occupy one row in a some new feature table, and the next row in the table will have mean and std deviation calculated on points t=1:10, and so on and so forth, until the end of the data.

Is there an efficient way to do this in Python?


texts: How to handle non-numeric user id's in a recommendation model

texts: Hi i'm working on a practice project on amazon movie reviews and i did everything that was asked except building a model that will recommend movies to users which have not been watched nor been rated. but when i'm trying to train and split the model I have a problem on assigning x/features and y/target as user_id's are not numeric e.g 'A3R5OBKS7OM2IR'. (The data contains 207 columns with user_id column which contains 4.9k users and 206 movies(movie 1 to movie 206 with ratings corresponding with the respected user_id)


texts: Could principle component decomposed coordinates value be correlated to each other?

texts:
I am wondering if we have a A= n*p matrix of samples and we run a PC decomposition on it.

Say the eigenvector matrix is E, so the samples in the eigenvector space should be

  B= A*(E)^-1

So, I am wondering if there is any rule that the columns of B to be correlated? That is, the first PC loadings and the second PC loadings to be correlated?
 Thanks!


texts:
I have 1299 images in 4 classes (374/269/284/372). I want to use the VGG19 model, add a dense layer at the top and fine-tune it with my images. As I only have 1299 images, I also want to use data augmentation.
Here is the code (it is not exactly the code I used everytime, as I tried several things. When I did not use preprocess_input, I did X = X/255.) :
drive.mount('/content/gdrive')

DATADIR = '/content/gdrive/My Drive/'
CATEGORIES = ['A','B','C','D']
training_data = []

def create_training_data():
    for category in CATEGORIES:
        path = os.path.join(DATADIR,category)
        class_num = CATEGORIES.index(category)
        for img in tqdm(os.listdir(path)):
            img_array = cv2.imread(os.path.join(path,img))
            img_array_RGB = cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB)
            new_array = cv2.resize(img_array_RGB, (224,224))
            training_data.append([new_array,class_num])

create_training_data()

X = []
Y = []

for features, label, in training_data:
    X.append(features)
    Y.append(label)

X = np.array(X)
Y = np.array(Y)

IMG_SIZE=224
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 3)

Y_class = Y
Y = to_categorical(Y, num_classes=4)

X_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.3, random_state=0, stratify=Y)

Y_train_int, Y_val_int = [np.where(r==1)[0][0] for r in Y_train], [np.where(r==1)[0][0] for r in Y_val]

from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
        horizontal_flip=True,
        vertical_flip=True,
        brightness_range=[0.2,0.6],
        fill_mode='wrap')

datagen.fit(X_train)

vgg_model = VGG19(weights='imagenet', include_top=False)

X = preprocess_input(X)

x = vgg_model.output
x = GlobalAveragePooling2D()(x)

x = Dense(512, activation='relu')(x)
x = Dropout(0.3)(x)

predictions = Dense(4, activation='softmax')(x)

model = Model(inputs=vgg_model.input, outputs=predictions)

for layer in vgg_model.layers:
    layer.trainable = False

layer_num = len(model.layers)
for layer in model.layers[:21]:
    layer.trainable = False

for layer in model.layers[21:]:
    layer.trainable = True

#class_weights = class_weight.compute_class_weight('balanced',
#                                                 np.unique(Y_train_int),
#                                                 Y_train_int)

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(datagen.flow(X_train,Y_train, batch_size=64), epochs=30, batch_size=128, shuffle=True, validation_data = (X_val,Y_val))

However this does not produce relevant results.
I tried several modifications (changing the learning rate, batch size, loss as sparse_categorical_crossentropy by changing from one hot encoding to integer, adding weights to classes, and I also tried to build a CNN from scratch with keras, but nothing seems to give better results (predicting the class with the most images)
Here is an example of plots : https://i.stack.imgur.com/KODAr.png
I don't really know what else I could try, but I really think a good model could classify these images better than this.
Edit :
I changed some parameters :

Data augmentation (I deleted the brightness)
batch_size = 32 for the data augmentation and 64 for the model
added weights to the classes
added x = Dense(256, activation='relu')(x) after x = Dense(512, activation='relu')(x)

And now the learning curve and confusion matrix look like this (only 10 epochs) : https://i.stack.imgur.com/IsJTh.png
The learning curve is really fluctuating and the model doesn't seem to learn, but the accuracy is higher than before.
I don't know what else to do now.


texts: Do I need to square a column if I want a neural network to try using that?

texts:
In the classical linear regression implementation, if I suspect the square of the values of the column is correlated to the target, then I actually need to create a new column with the squares for the algorithm to make use of that.
Is this also necessary when using neural networks? I know it's a broad question - are there cases where this is necessary and cases where it isn't?


texts: How Adaboost calculates error for each weak learner in training?

texts:
I am studying the Adaboost classification algorithm because i would like to implement it from scratch. I understand how it works, but i am not able to understand where some steps are placed.
I will describe the Adaboost training steps in my understanding (sorry for any incorrect formalism):

Initialize a weak learner $k$
Define a weight for each sample in the dataset equally $w =\frac{1}{N}$
Fit $k$ to the dataset
Calculate error $e = \sum_{i=0}^{N}e_iw_i$
Calculate importance $\alpha$ of $k$, i.e. $\alpha = \frac{1}{2}log(\frac{1-e}{e})$
Recalculate weights for the correct classified samples: $w_{t+1} = w  e^{\alpha}$
Recalculate weights for the incorrect classified samples: $w_{t+1} = w  e^{-\alpha}$
Normalize new sample weights: $w_{normalized} = \frac{w}{\sum_{i=0}^N w_i}$
For all sequent learners, select samples based in weighted random choice until get a dataset with same size as the original and do the same proccess.

My question is: how error is obtained? Regarding the implementation, should i firstly fit the dataset and then get the error from predicting the same dataset? This doesn't seems correct.
I've tried to read different sources about this and even a great explanation from the Statquest channel wasn't able to make this clear.
Thanks!


texts:
I would like to implement a machine learning algorithm for data like this.

id                Date     Amount isFraud
1  100 2018-03-20 10:20:10  110.0      No
2  101 2018-03-25 11:20:20    2.0     Yes
3  101 2018-03-25 11:20:22    2.5     Yes
4  101 2018-03-25 11:20:25    1.7     Yes
5  101 2018-03-25 11:20:27    2.1     Yes
6  101 2018-03-25 11:20:30    2.9     Yes
7  102 2018-03-29 11:00:11  290.0      No
8  110 2018-04-15 08:10:05    1.5     Yes
9  110 2018-04-15 08:10:09    2.3     Yes
10 110 2018-04-15 08:10:13    2.7     Yes
11 110 2018-04-15 08:10:17    1.9     Yes
12 110 2018-04-15 08:10:21    2.9     Yes
13 110 2018-04-15 08:10:27    2.2     Yes
14 110 2018-04-15 08:10:45    2.1     Yes
15 113 2018-04-15 10:10:05  190.0      No

Algorithm needs to work in the following way. Transactions with lower amount are classified as scam. Frequent transactions from same id with same amount or lower amount are also scam. The above dataset was created for this purpose. For these kinds of problems, should we write our own algorithms by specifying the conditions? Or can we use any existing algorithms?


texts: Finding the relation between two dimensions in a multi-dimensional problem

texts:
I have a collection of data points. Each point has 6 dimensions (x1, x2,...x6). I want to find a relation between two dimension (e.g. x1 vs x2). What I have been doing so far is look for points where the other dimensions (x3 to x6) are relatively constant, by defining a band. This way I would get several groups of data points where only the two dimensions of interest would change.

I was wondering if there is a better way of analyzing the relationship between these two dimensions. I looked at PCA, but I have a feeling that it does not help me much. If I reduce the problem to two dimensions the axes are basically meaningless.

Can you guys give me some directions to look at?


texts: Question about bias in Convolutional Networks

texts:
I am trying to figure out how many weights and biases are needed for CNN.

Say I have a (3, 32, 32)-image and want to apply a (32, 5, 5)-filter.
For each feature map I have 5x5 weights, so I should have 3 x (5x5) x 32 parameters. Now I need to add the bias. I believe I only have (3 x (5x5) + 1) x 32 parameters, so is the bias the same across all colors (RGB)?

Is this correct? Do I keep the same bias for each image across its depth (in this case 3) while I use different weights? Why is that?


texts: Establishing data science programs as an independent discipline

texts:
What are the benefits of establishing data science programs as an independent discipline rather than creating data scientists in different fields? (for students, universities and the society as a whole)

I am authoring a book on data science and I have mentioned some university programs as models for establishing data science programs in bachelor's, master's and doctorate level. I am thinking of some reasons to compare this model with enriching well-established programs and adding data science skills.


texts:
I have panel data for 3 countries, ranging over 3 years. The dataset is called CarProduction

Country    Year       cars       Fuel_price     PPP      Manufact      PublicTransport

USA        2015       500           5         10000          9                2
USA        2016       700           5.2       10500          8.75           2.2
USA        2017       780           5.4       11000          8.6            1.9
China      2015       150           9          4000          11               3
China      2016       200           8.6        4500          11.5             4
China      2017       340           9.4        6000          15.6             5
Italy      2015       200           9          4000          11               5
Italy      2016       300           8.6        4500          11.5           6.2
Italy      2017       440           9.4        6000          15.6           6.7

I know that the glmnet package can be used in R to run a LASSO regression.

Considering that I have panel data, how do I run LASSO regression on panel data?

I will be implementing OLS for comparison purposes. This is the regression I would ordinarily run, accounting for fixed effects:

ans <- plm(cars ~ Fuel_price+PPP+Manufact+PublicTransport, data=CarProduction, model="within")


texts: Machine learning algorithms for correct words formation from jumbled words

texts:
https://www.google.com/search?q=jumbled+words&oq=jumbled&aqs=chrome.1.69i57j0l4.3399j0j9&client=ms-android-lava&sourceid=chrome-mobile&ie=UTF-8
Can Machine learning algorithms solve the input dataset of jumbled words and form the correct words from them?


texts: Behavioral Differences between Standard Autoencoder and Variational Autoencoder

texts:
If I have a convolutional network which compresses from 256x256x3 images to, say, 100 hidden continuous states, then back to 256x256x3 and then train it with stochastic gradient descent (where the output is a denoised input), is it reasonable to expect it to learn the same or similar features as a similarly architected network, but one that uses variational methods (reconsruction + latent loss) techniques?  Assuming they both have the same number of conv/max/norm steps between the input and the FC layers?

What about if I have a really small dataset, say 10 examples, and want to overfit?  Will both of them arrive at similar representations at the hidden layer?


texts: Machine Learning - Range of Hypothesis space and choiceof Hypothesis function type

texts:
I am new to machine learning and seek your help in clarifying my elementary doubts. I did a fair amount of googling, but find most literature jumping directly into math.

What I know is that given a labelled training data, a ML algorithm chooses from a hypothesis space H a hypothesis function h.

As an example, assume that a feature vector in training data contains 3 features (x1 through x3)

Now the data from the training set is taken and plugged into a formula (function type). If x is the feature vector and w represents the coefficients of the formula, then output y = A pre-determined f(w,x).

My questions are:

1. Who decides the range of each of the coefficients?
2. Who decides the formula? Is the formula fixed for a ML algorithm?
3. What exactly is the hypothesis space? Is it range of w, or is it different formulas or both?

I acknowledge I have asked more than one question and against the rules, but it was convenient to logically group them in a single post.


texts:
What are the best options for a big dataset classification?
I am thinking of 2 solutions-

1.Autoencoders

2.PCA(Principal Component Analysis)

I think the first approach is better.Does it works for my problem?Or maybe someother solution is better?

Thank you


texts:
I am learning this post,
"A few useful things to know about machine learning"

The author says

  A classifier must be represented in some formal language that the
  computer can handle.Conversely, choosing a representation for a learner is tantamount to
  choosing the set of classifiers that it
  can possibly learn. This set is called
  the hypothesis space of the learner.

and put KNN and SVM in the instance-based category.

I assume the instance here mean each individual example, for instance, KNN needs to compute the distance to each example.

Assume there are 300 examples to train a KNN classifier, how to understand choosing this specific representation is tantamount to choose hypothesis space?

to clarify the question.

does KNN have a hypothesis space? if yes, Is there a way to clearly define the hypothesis space with the instances?


texts:
I am currently trying to recreate the result of this paper, in which they do feature extraction from a "spectogram" of log-melfilter energies..

Since the paper doesn't state what kind of feature I am seeking, i am currently trying to extract features, and match them to MFCC features. The paper states a technique called LWS (Limited weight sharing) in which the spectogram frequency axis will be divided into section, and each section don't share their weight with others.

So i've divided the my input image into 13 section to receive 1 output features  from a (6,3,3) input image. 6 for the number of rows, 3 as each column represent [static delta delta_delta] data of the given log melfilter energi, and the last 3 is the color channels.

If i'd used 13 filterbanks, and made the plot, will the result of this be that each (1,3,3) matrix would result in one feature, but that seemed a bit too good to be true, so i decided to use 78 filterbanks and divide it into 13 section which should result in one feature can be extracted from a matrix of size (6,3,3)

I am training the network with this model structure:

def create_model(init_mode='normal',activation_mode='softsign',optimizer_mode="Adamax", activation_mode_conv = 'softsign'):
    model = Sequential()

    model.add(ZeroPadding2D((6,4),input_shape=(6,3,3)))
    model.add(Convolution2D(32,3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(Convolution2D(32, 3,3, activation=activation_mode_conv))
    print model.output_shape
    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))
    print model.output_shape
    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(Convolution2D(64, 3,3 , activation=activation_mode_conv))
    print model.output_shape
    model.add(MaxPooling2D(pool_size=(2,2),strides=(2,1)))
    model.add(Flatten())
    print model.output_shape
    model.add(Dense(output_dim=32, input_dim=64, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=13, input_dim=50, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=1, input_dim=13, init=init_mode,activation=activation_mode))
    model.add(Dense(output_dim=1,  init=init_mode, activation=activation_mode))
    #print model.summary()
    model.compile(loss='mean_squared_error',optimizer=optimizer_mode)

    return model

This model keeps for some reason providing me very bad results..
I seem to keep getting an loss of 216, which is nearly 3 times the data range...

I did a grid seach to find out which parameter (activation function, init_mode, epochs and batch_size would be best, which are those chosen in the function above (eventhough there wasn't much change in the outcome..)

What can i do to get better results?
Is the CNN network poorly designed?


texts:
I'm trying to understand how does the KernelDensity class in scikit-learn work. Consider the following two cases which build a kernel from two different arrays (a). I'm wondering why the result of scoring on the array b in both cases is the same? Shouldn't a kernel from 10 same points be different from that from 2 points? The one from the 10 points should indicate more density. So, why the final scoring result is the same in both cases?
case A:
a = np.array([[1],[1],[1],[1],[1],[1],[1],[1],[1],[1]])

kde = KernelDensity(bandwidth=0.1)

kde.fit(a)

b = np.array([[1]])

log_dens = kde.score_samples(b)

print('Probability is: {}'.format(np.exp(log_dens)))

Probability is: [3.9894228]
case B:
a = np.array([[1],[1]])

kde = KernelDensity(bandwidth=0.1)

kde.fit(a)

b = np.array([[1]])

log_dens = kde.score_samples(b)

print('Probability is: {}'.format(np.exp(log_dens)))

Probability is: [3.9894228]


texts: What is the right loss function for semantic segmentation or do I have to use all of them?

texts:
I'm doing my PhD research about image semantic segmentation and now I'm trying to understand what kind of loss function do I have to use with a CNN like U-Net.
I have found the paper "A survey of loss functions for semantic segmentation" very interesting. It talks about fourteen loss functions organized in (check paper's table 1 for more details):

Distribution-based Loss.
Region-based Loss.
Boundary-based Loss.
Compounded Loss.

I'm confuse. Without any previous idea about what a loss function does, I thought that doing a subtraction between the ground truth and the prediction is enough to get a value to know how good is the prediction. But reading the paper, I guess that I need a loss function that tells how good are each of the pixels in the prediction image. Knowing how good are each of them, I can improve the CNN doing backpropagation on each of its outputs (I think the convolutional network outputs are all of the pixels in the prediction image).
If I have four types of loss functions, checking how good the predicted image is in distribution, or region, or boundary, or in compounded... what is the right loss function for me? Because I think I need to check that the prediction is good on the four types. This is why I thought that doing a subtraction between the ground truth and the prediction it will check the performance on these four types of loss.
By the way, I'm researching about semantic segmentation on magnetic resonance brain images to find tumours, WMH, etc.


texts: Finding orthogonal input patterns associated with logistic function output

texts:
I've been given this problem but cannot seem to get an analytical solution. I've tried satisfying the logistic function with several vectors but have difficulty finding ones which are also orthogonal. The problem begins with me being given a neural unit weight vector,
$$w =\begin{bmatrix}
           1 \\
           \frac{1}{4} \\
           \frac{1}{9} \\
         \end{bmatrix}$$
The neuron output u is related to its input pattern v∈R3 by
$$u = f(w^Tv + 1)$$
where
$$f(x) = (1 + e^{-x})^{-1}$$
To find a pair of orthogonal vectors, v1 and v2, for which the unit output $u_{i} = 0.5, i=1,2.$

So far I know that $w^TV = 0$ for the function output to be 0.5.


texts: How to plot time series data from SQLite database

texts:
I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data from .sqlite file.

import sqlite3
from sqlite3 import Error

def create_connection(db_file):
    """ create a database connection to the SQLite database
        specified by the db_file
    :param db_file: database file
    :return: Connection object or None
    """
    conn = None
    try:
        conn = sqlite3.connect(db_file)
    except Error as e:
        print(e)

    return conn

def main():
    database = r"E:\Data\SINCAL_Models\AusNetData\20180128_PQV\pq.sqlite"

    # create a database connection
    conn = create_connection(database)
    with conn:
        print("Database connected:")

if __name__ == '__main__':
    main()

The database file is shown in attached picture. I am trying to plot the yellow highlighted variables to get the plot like here


texts:
I'm trying to predict age from a given picture. I built the model below but the problem is that I'm getting very large loss value with low accuracy while fitting the model.

I think the problem is choosing the wrong loss function (here mean_squared_error). What can be the problem here?

import tensorflow as tf
from tensorflow import keras

X = X.reshape(-1, image_size[0], image_size[1], 1)
model = keras.models.Sequential()

model.add(keras.layers.Conv2D(32, (5, 5), activation='relu', input_shape=X.shape[1:]))
model.add(keras.layers.MaxPooling2D((2, 2)))

model.add(keras.layers.Conv2D(32, (3, 3), activation='relu'))
model.add(keras.layers.MaxPooling2D(2, 2))

model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))

model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(60, activation='relu'))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(1, activation='softmax'))
model.compile(optimizer='adam', loss=keras.losses.mean_squared_error, metrics=['accuracy'])

model.fit(X, Y, epochs=170, shuffle=True, validation_split=0.1)

As another question, are my layers correct to predict a number for a given picture?


texts: how to deal with two high correlations feature which both has a low correlation with target

texts:
I am doing a prediction of house trade money.
Here is the correlation matrix of features whose correlations are larger than 0.3 as follows:

train_corr_full.unstack().sort_values(kind='quicksort', ascending=False).reset_index()
corr = corr[(corr['level_0'] != corr['level_1'])]
corr

    level_0             level_1              0
13  pv                  uv                   0.911879
14  uv                  pv                   0.911879
15  area                bathroom             0.721935
16  bathroom            area                 0.721935
17  area                tradeMoney           0.687447
18  tradeMoney          area                 0.687447
19  bathroom            tradeMoney           0.580745
20  tradeMoney          bathroom             0.580745
21  bathroom            room                 0.421762
22  room                bathroom             0.421762
23  remainNewNum        totalNewTradeMoney   0.417114
24  totalNewTradeMoney  remainNewNum         0.417114
25  buildYear           totalFloor           0.393571
26  totalFloor          buildYear            0.393571
27  remainNewNum        tradeMeanPrice       0.314611
28  tradeMeanPrice      remainNewNum         0.314611

corr[corr['level_0'] == 'tradeMoney']

    level_0     level_1             0
18  tradeMoney  area                0.687447
20  tradeMoney  bathroom            0.580745
34  tradeMoney  tradeMeanPrice      0.282720
45  tradeMoney  totalFloor          0.249755
47  tradeMoney  tradeNewMeanPrice   0.236713
55  tradeMoney  room                0.215041
79  tradeMoney  buildYear           0.123065
81  tradeMoney  totalTradeMoney     0.122407
95  tradeMoney  remainNewNum        0.100921
116 tradeMoney  pv                  0.072919
134 tradeMoney  uv                  0.040452
137 tradeMoney  totalNewTradeMoney  0.038420

  Features descriptions:
  area: the area of the house.
  room: the number
  of the room in this house.
  bathroom: the number of the bathroom in
  this house.
  totalFloor: total number of floors of this building.
  tradeMeanPrice: the mean price of second-hand housing transaction
  price this month.
  tradeNewMeanPrice: the mean price of new housing
  transaction price this month.
  buildYear: the age of this house.
  totalTradeMoney: the total price of second-hand housing transaction
  price this month.
  totalNewTradeMoney: the total price of the new
  housing transaction price this month.
  remainNewNum: the number of
  houses that haven't been sold this month.
  pv: the number of times that
  the website was viewed by tenants.
  uv: the number of tenants that
  viewed the website.

It shows that there is a high correlation with pv and uv but both of which has a low correlation with the target.
What I did before is that I would drop the features whose correlation is lower than about 0.15 and do more EDAs and feature engineerings in those high correlations feature.
And even if consider context I also think the pv and uv is not good enough for the prediction.

Could anyone give some hints on it?
Thanks in advance.


texts: Best practice to test (unit and integration) a neural network implementation

texts:
I'm implementing my own neural network (with the term implementing i mean writing the code that run the neural network itself, not training).

I implemented it for didactical purpose, but i does not know how to proceede for a validation of what i have done.

So i would ask if there is some best practice for neural network test (eg : is there a dataset of neural nets with weight, input and expected output)


texts: Why is Random Forest feature importance biased towards high cadinality features?

texts: I understand how a random forest algorithm works but could someone tell me the rationale behind Random Forest feature selection being biased towards high cardinality features?


texts: Can LSTM be used for non time series data?

texts:
I have a dataset -

This is a TOR network traffic dataset with labels added as TOR/ Non TOR.
I want to run an LSTM on it and classify it as Tor/Non Tor. Is that possible since this is not a time series data?


texts: Text classification using deep learning with removing name, place address etc

texts:
I want to prepare a deep learning model for text classification(Document classification), But in my training content many place have name, address, brand name, etc.. which will do confuse to model as deep learning is work with sequence of words, I was thinking if we will train model by removing these details, but in predict content when we get name, address, brand name then it also confuse...
Can someone give the idea how i handle this situation in deep learning?
And which deep learning algorithm and library should use for best accuracy and speed for text classification.


texts:
I have been reading through this book and am trying to do the exercises.

The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic argument to prove "the weight decay will tail off when the weights are down to a size around $1/\sqrt{n},$ where $n$ is the total number of weights in the network."

For more context on the same problem (Parts 1 and 2):

Heuristic argument for Weight decay and regularization

Speed decay proof for L2 regularization and non-normalizied weight initiation

The relevant equation appears to be this:

$$C= -\frac{1}{n_t} \sum_{xj} [y_j \ln{a^L_j}+(1−y_j)\ln{(1−a^L_j)}]+\frac{\lambda}{2n_t}\sum_{w}w^2.$$

(The book seems to use the $n$ notation to represent two different things. I've changed it so that $n_t$ refers to the size of the training set, and $n_w$ refers to the number of weights in the network)

I think it has something to do with the last term, $\frac{\lambda}{2n}\sum_{w}w^2$, because if we substitute $w=\frac{1}{\sqrt{n_w}}$ then the whole term simplifies to $\frac{\lambda}{2n}$. This would mean that the partial derivative with respect to the weight becomes $\frac{\partial C}{\partial w}$ instead of $\frac{\partial C}{\partial w}+\frac{\lambda}{n}w$. Is this enough of an explanation to conclude that weight decay falls off once the weights are around $\frac{1}{\sqrt{n}}?$


texts:
I have time series of parameters A, B, C and D. All of them are under influence of the same major conditions, but each one has minor differences. They are placed in different locations, A, B, C are in local1 and D is in local2.

I would like to know which one (A, B, C) has the major similarity to D. How should I approach this issue?


texts: How to use a set of pre-defined classifiers in Adaboost?

texts:
Suppose there are some classifiers as follows:
dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

In AdaBoost, I can define a base_estimator and also the number of estimators. However, I want to use these 7 classifiers. In other words, n_estimators=7 and these estimators are above ones. How can I define this model?


texts: Bubble Chart Data Visualization in imPower/Microstrategy ordering incorrectly

texts:
Apologies if this isn't the right community for this, I could not find a "data visualization" site so I felt that this was the closest fit.

I'm currently attempting to build a bubble chart visualization to show how a group of individuals has progressed on an online training course. For context, here is the editor menu I have set up:

Here is the visualization produced:

The visualization is correctly sizing bubbles by % completed (and the data is accurate) and is correctly splitting and coloring by E-mail. The problem is that it is not correctly ordering horizontally by % completed. Even though the X axis is % Completed, bubbles are being ordered alphabetically by E-mail.

Neither I nor my colleagues have any idea why this is happening. It's been an issue for the past 2 days, and I'm out of ideas.


texts:
The code here:

https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py

Specifically line 53:

xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)

Why is the cross entropy multiplied by original_dim? Also, does this function calculate cross entropy only across the batch dimension (I noticed there is no axis input)? It's hard to tell from the documentation...


texts: Generating Z-scores after bootstrap sampling

texts:
I have a very basic question. I have performed bootstrap sampling to a list of values to a sample of an unknown distribution. I have computed the mean of the empirical bootstrap sample, as well as the distribution of the standard deviation for the empirical bootstrap sample.
Now I need to calculate Z-scores (and ultimately probabilities) for my original list of values. I know the formula for calculating Z-scores, but it uses the mean and standard deviation. Since I have a distribution of standard deviations instead of a single value, can anybody give me some direction of what I need to do to obtain the Z-scores?


texts: What is an alternative name for "Unstructured Data"?

texts:
I'm writing my thesis at the moment, and for some time - due to a lack of a proper alternative - I've stuck with "unstructured data" for referring to natural, free flowing text, e.g. Wikipedia articles.

This nomenclature has bothered me from the very beginning, since it opens a debate that I don't want to get into. Namely, that "unstructured" implies that natural language lacks structure, which it does not - the most obvious being syntax. It also gives a negative impression, since it is the opposite of "structured", which is accepted as being positive. This is not the focus of my thesis, though the "unstructured" part itself plays an important role.

I completely agree with the writer of this article, but he proposes no alternative except for "rich data", which doesn't cover my point. The point I'm trying to make that the text lacks a traditional database-like (e.g. tabular) structure of the data, with every piece of data having a clear data type and semantics that is easy to interpret using computer programs. Of course I'd like to condense this definition into a term, but so far I've been unsuccessful coming up with, or discovering an acceptable taxonomy in literature.


texts: Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

texts: If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?


texts: Time series forecasting: prediction and forecast far from the reality

texts:
Apologies for the awkward title, but I hope to be able to regain your confidence.

Let's start with the final output I got, so at least you can understand why I'm not happy/concerned about the outcome.

The graph shows the past 2 years website traffic. I was interested to understand whether my learnings could have been applicable to my main profession. I truly believe this is the case, but the results I got aren't very encouraging.

The plot has been obtained running the data past an ARIMA model after different normalisations occurred (e.g. differencing, log, etc.)
The best ARIMA model was found to be 11, 1, 11. This is supported from the below:

You may be tempted to say P = 15, but in fact, after several computations, the best RSME was given by the above. See below (though I'm not sure that RSME can be really applied here)

I understand that almost 69% of confidence is far from being perfect, but I would have in some case expected to return something better than the above output.

Do you have any ideas of what I could have done wrong, what I may improve and how?
Bear in mind I'm learning, so please excuse my limited understanding on complex terminology. The plainer you go, the best chance for me to understand.

Thanks for your support.

EDIT:

Thanks to all comments below, I made some progress. I was also able to get through the SARIMA and counter verify the data with the new data acquired since I started this learning project.

See below the new plot.

Although a perfectly matching fit with the "training" data, when it comes to the forecast, the data hardly correspond.
Given my in-expertise, is the above plot something that you would deem normal? Or Do I need to re-work a lot more to get anything better?


texts:
I am studying the CNN architecture of the AlexNet, and I have seen that it has convolutional layers without pooling in between:

but I don' understand why this is done. Wouldn't be better to have something like CONV - POOLING - CONV - POOLING , and so on, instead of CONV - POOLING - CONV - CONV -CONV - POOLING?

Why is this done? Thanks in advance.


texts: How exactly do I extract the important features from strings for machine learning?

texts:
Forgive me for my ignorance. Linked below is an image of my dataset with 1000 tuples.

https://i.stack.imgur.com/WHIlx.png

I have the following questions
(1) How exactly do I go about extracting information from the Ad topic line?
(2) How should I approach the categorical variables Country and City? ( I've heard about encoding but won't the large number of countries/cities be a problem?)

Thank you so much for your time. All help is greatly appreciated.


texts: How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background?

texts:
I am currently a postdoc and my PhD was in applied mathematics in the area of numerical analysis and electromagnetic/acoustic wave propagation. There was no statistical element to my PhD, it was completely deterministic. I took several probability/statistics and one machine learning module 5-6 years ago during my BSc, and a stochastic ODE module during my MSc but that's about it..its been all applied mathematics since then.

I am considering leaving academia and entering industry and it seems like there are far more jobs in the area of data science/machine learning than there are for my skillset.

If I left academia and began 'studying up', how long do you think it could take me to gain the skills required for a data science/machine learning position in industry?
It seems like there is a very wide variety of science/machine learning techniques and obviously there isn't time to learn all or even most of them. So what approaches are absolutely essential for data science/machine learning in industry these days and what is the most efficient route to gaining these skills?


texts: What affects blurriness of images outputted by vanilla VAE?

texts: I know that MSE between reconstruction (m(z)) and original image (X) affects blurriness, because it averages across pixel values. I also know that variance of the Gaussian distribution p(x|z) (p(x|z) = N(m(z),sigma^2*I) also affects it, because 1/sigma^2 multiplies MSE in the cost function, and thus the smaller the variance the better is the reconstruction. My intuition is that MSE and variance should somehow be connected (because the MSE is dervied from the log likelihood of the p(x|z)), and not affect the blurriness of the images separately. Am I wrong?


texts: Autoencoder for Dimensionality Reduction - varying result - parameter tuning

texts:
I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question. The problem and steps taken to solve problem are as follows:

There exists a data set with 5200 rows and 113 features from industrial sensors [Numeric Type].
The task is to use Autoencoder for the unsupervised dimensionality reduction purpose.
The goal is to gain a result with 3 features so as to plot the data for visiualization and further machine learning models input.
The overall construction of Autoencoder is as:

Preprocessing step using minmax scaling:

train_scaled = minmax_scale(clean_Data_numeric, axis = 0)
df = pd.DataFrame(data=train_scaled)
df.head()

Splitting the data set:

X_train, X_test = train_test_split(train_scaled, train_size = 0.8, random_state = seed(2017))

Constructing the Autoencoder:

ncol = train_scaled.shape[1]
encoding_dim = 3
input_dim = Input(shape = (ncol, ))

# Encoder Layers
encoded1 = Dense(40, activation = 'relu')(input_dim)
encoded2 = Dense(20, activation = 'relu')(encoded1)
encoded13 = Dense(encoding_dim, activation = 'relu')(encoded2)

# Decoder Layers
decoded1 = Dense(20, activation = 'relu')(encoded13)
decoded2 = Dense(40, activation = 'relu')(decoded1)
decoded13 = Dense(ncol, activation = 'sigmoid')(decoded2)

# Combine Encoder and Deocder layers
autoencoder = Model(inputs = input_dim, outputs = decoded13)

# Compile the Model
autoencoder.compile(optimizer = 'adadelta', loss = 'mean_squared_error')

Autoencoder summary:

Model: "model_17"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_14 (InputLayer)        (None, 113)               0
_________________________________________________________________
dense_112 (Dense)            (None, 40)                4560
_________________________________________________________________
dense_113 (Dense)            (None, 20)                820
_________________________________________________________________
dense_114 (Dense)            (None, 3)                 63
_________________________________________________________________
dense_115 (Dense)            (None, 20)                80
_________________________________________________________________
dense_116 (Dense)            (None, 40)                840
_________________________________________________________________
dense_117 (Dense)            (None, 113)               4633
=================================================================
Total params: 10,996
Trainable params: 10,996
Non-trainable params: 0

Fitting:

autoencoder.fit(X_train, X_train, nb_epoch = 5, batch_size = 32, shuffle = False, validation_data = (X_test, X_test))

Evaluation:

encoder = Model(inputs = input_dim, outputs = encoded13)
encoded_input = Input(shape = (encoding_dim, ))
encoded_train = pd.DataFrame(encoder.predict(train_scaled))
encoded_train = encoded_train.add_prefix('feature_')
print(encoded_train.shape)
encoded_train.head()

-------------------------------------
(5287, 3)

    feature_0   feature_1   feature_2
0   0.021782    11.836038   1.777279
1   0.018666    11.803829   1.770958
2   0.000000    11.711332   1.765472
3   0.025804    11.797852   1.765566
4   0.001563    11.896879   1.787951

Now, The problem is each time I run the autoencoder with different setting like

nb_epoch = 5, batch_size = 32, shuffle = False
nb_epoch = 10, batch_size = 64, shuffle = True

I will get totally different result for 3 features, and in case of plotting the features, The total shape of 3D points are different each time of running. Thus,

How can I be certain that I am using right parameter and setting for my autoencoder?
Which parameter should be used for this setting which be ended up in accurate 3D representation of points by 3 features?

Thank You


texts: Majority voting in scikit-learn Random forest

texts: My main concern is that i need to understand that how does the random forest do majority voting in scikit learn source code. I did not find that specific code in source code of RandomForest. if anybody knows, then kindly explain that. Thanks


texts: Uploading images folder from my system into Google Colab

texts: I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder into my notebook and use it?


texts: Can BERT embeddings be used to reproduce the original content of the text?

texts:
From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different depending on the context)

Is there a way to "reverse" these contextualized embeddings to produce an output related to the original content of the text? For instance, how would I do machine translation, or style transfer?