texts: Azure ML / AutoML: problem with univariate time series forecasting

texts:
I'm having troubles generating univariate time series forecasts with Azure Automated Machine Learning (I know...).
What I'm doing
So I have about 5 years worth of monthly observations in a dataframe that looks like this:

date
target_value

2015-02-01
123

2015-03-01
456

2015-04-01
789

...
...

I want to forecast target_value based on past values of target_value, i.e. univariate forecasting like ARIMA for instance.
So I am setting up the AutoML forecast like this:
# that's the dataframe as shown above
train_data = Dataset.Tabular.from_delimited_files(path=datastore.path(my_remote_filename))

# ...other code...

forecasting_parameters = ForecastingParameters(
    time_column_name='date',
    forecast_horizon=2,
    target_lags='auto',
    freq='MS'
)

automl_config = AutoMLConfig(task='forecasting',
                             debug_log='automl_forecasting_function.log',
                             primary_metric='normalized_root_mean_squared_error',
                             enable_dnn=True,
                             experiment_timeout_hours=8.0,
                             enable_early_stopping=True,
                             training_data=train_data,
                             compute_target='my-cluster',
                             n_cross_validations=3,
                             verbosity=logging.INFO,
                             max_concurrent_iterations=4,
                             max_cores_per_iteration=-1,
                             label_column_name='target_value',
                             forecasting_parameters=forecasting_parameters)

What the problem is
But AutoML does not seem to generate the forecast for target_value based on past values of target_value. It seems to use the date column as the independent variable!
The feature importance chart also shows date as the input feature:

As a side note: running multivariate forecasts works fine.
When I use a dataset like this, feature_1 and feature_2 are used (i.e. as the X) to forecast target_value (i.e. the y)

date
feature_1
feature_2
target_value

2015-02-01
10
7
123

2015-03-01
30
2
456

2015-04-01
20
5
789

...
...
...
...

My questions therefore
How do I need to set up a univariate AutoML forecast to forecast target_value based on past observations target_value?
I assumed generating lagged values for target_value etc. is exactly what AutoML is supposed to do.
Thanks!


texts: Graph isn't an attribute in TensorFlow? Very basic question

texts:
Hi am am just starting to learn how to use TensorFlow and this is one of my first projects. I am simply trying to create a graph.

This is my code:

    import tensorflow as tf
    graph1 = tf.Graph()

I cannot seem to create a Graph. The compiler threw an AttributeError (not exactly sure what that is)...

    AttributeError                            Traceback (most recent call last)

The compiler said the issue was with line two in creating the graph.

    ----> 2 graph1 = tf.Graph()

    AttributeError: module 'tensorflow' has no attribute 'Graph'

If it is of any use I am using a Jupyter Notebook in JetBrain's PyCharm. I believe I imported everything correctly but I am not sure if using PyCharm will mess anything up. Any help you be very appreciated!! :)


texts: Resampling pandas Dataframe keeping other columns

texts:
I'm facing a problem with a pandas dataframe. Actually my Dataframe contains 3 columns: DATE_TIME, SITE_NB, VALUE.
For some SITE_NB there are missing rows. For example:

DATE_TIME;SITE_NB; VALUE

2011-01-03 01:00; 1; 10.7

2011-01-03 04:00; 1; 3.2

2011-01-03 05:00; 1; -2.1

So here, rows for 2011-01-03 00:00, 2011-01-03 02:00 and 2011-01-03 03:00 are missing. What I want is add these rows with the same SITE_NB (=1) and with VALUE (=NaN)

I want to do the same for all different SITE_NB in my dataframe. So for each SITE_NB, add missing rows based on DATE_TIME with a frequency of 1 Hour, and putting NaN in VALUE for freshly added rows.

I tried resampling but did not get the right output...

Can somebody help me to solve this issue?

Thanks!


texts: Dimension of weight matrix in neural network

texts:
Why would the dimension of $w^{[2]}$ be $(n^{[2]}, n^{[1]})$ ?

This is a simple linear equation, $z^{[n]}= W^{[n]}a^{[n-1]} + b^{[n]}$

There seems to be an error in the screenshot.
the weight, $W$ should be transposed, please correct me if I am wrong.

$W^{[2]}$ are the weights assigned to the neurons in the layer 2

$n^{[1]}$ is the number of neurons in layer 1

Screenshot from Andrew Ng deeplearning coursera course video:


texts: Two parallel models for semantic segmentation in Keras

texts:
I want to build two parallel models for image semantic segmentation in Keras.

    input1 = Input(shape=(480,480,3))
    input2 = Input(shape=(480,480,1))

    c1_1 = Conv2D(filters=64, kernel_size=(3,3), activation='relu',  padding='same')(input1)
    c1_1 = MaxPool2D(strides=(2,2))(c1_1)

    c2_1 = Conv2D(filters=16, kernel_size=(3,3), activation='relu',  padding='same')(input2)
    c2_1 = MaxPool2D(strides=(2,2))(c2_1)

   (...)

    # Merge to models:
    c = concatenate([c1_n, c2_n], axis=3)
    c = UpSampling2D(size=(2,2))(c)
    c = Conv2D(filters=512, kernel_size=(3,3), activation='relu', padding='same')(c)

   (...)

   output_layer = Conv2D(6, kernel_size=(1,1), activation='softmax')(c)

   model = Model([input1, input2], output_layer)

   model.compile(optimizer=Adam(2e-4), loss='categorical_crossentropy', metrics=['categorical_accuracy'])

   def my_generator(x_train, y_train, batch_size):
     data_generator = ImageDataGenerator(
        (...).flow(x_train, x_train, batch_size, seed=42)
     mask_generator = ImageDataGenerator(
        (...).flow(y_train, y_train, batch_size, seed=42)

     while True:
        x_batch, _ = data_generator.next()
        y_batch, _ = mask_generator.next()
        yield [x_batch[:,:,:,:3], x_batch[:,:,:,3]], y_batch

     #X = [72, 480, 480, 4]
     #Y = [72, 480, 480, 5]
     model.fit_generator(my_generator(X, Y, 7),
                       steps_per_epoch = 60,
                       validation_data = (X_test, Y_test),
                       epochs=150, verbose=2)

But i have an error:

ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[ 0.33891213,  0.37238494,  0.33054393,  0.        ],
     [ 0.34728033,  0.38493724,  0.33472803,  0.        ],
     [ 0.35146444,  0.39330544,  0.35983264,  0.        ],
     ....

Can you help me with my error or tell me whether this assumption is correct?


texts: How can one encode data of different dimensions into a tensor in Pytorch?

texts: I'm relatively new to Deep Learning. In my Deep Learning class our input was always images or vectors of numbers which fit a nice rectangular format. My question is: if you have data that doesn't fit into nice rectangular shapes, how do you combine them to go through a single feed-forward network? In my case, I am trying to encode all the information about a game for a reinforcement learning task. This will include a grid/matrix of data (similar to a chess board), as well as metadata like time remaining and other information that's not part of the board. The tensor that goes into my network needs to maintain the spatial relationship with the board, but not imply a spatial relationship with the metadata. Tips?


texts: What do each of the three SVM classes in R represent?

texts:
Inspired by this post, I took a look at this doc

SVN in R output this:

Statistics by Class:

                     Class: 1 Class: 2 Class: 3
Sensitivity            1.0000   0.9412   1.0000
Specificity            1.0000   1.0000   0.9762
Pos Pred Value         1.0000   1.0000   0.9796
Neg Pred Value         1.0000   0.9865   1.0000
Prevalence             0.2778   0.1889   0.5333
Detection Rate         0.2778   0.1778   0.5333
Detection Prevalence   0.2778   0.1778   0.5444
Balanced Accuracy      1.0000   0.9706   0.9881

What are each of the three classes?


texts: What is the concept of error bars in barplots in seaborn library?

texts: I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in documentation. What do these error bars represent and how are they calculated ?


texts:
I have transcripts of phone calls with customers and agents. I'm trying to find promises which were made by an agent to a customer.

I already did punctuation restoration. But there are a lot of sentences that don't have any sense. I would like to remove them from the transcript. Most of them are just a set of not connected words.
I wonder what approach is the best for this task?

My ideas are:

• Use tf idf and word2vec to create vectors from all sentences. After that we can do some kind of anomaly detection e.g. look for and delete vectors that are highly deviated from most other vectors.

• Spam filters. Maybe is it possible to apply spam filters for this task?

• Crate some pattern of part of speech tags that proper sentence must include. For example, any good sentence must include noun + verb. Or we can use for example dependency tokens from spacy.

update

Example of a sentence that I want to keep:

  There's no charge once sent that you'll get a ups tracking number.

Example of a junk sentence:

  Kinder pr just have to type it in again, clock drives bethel.

Another junk sentence:

  Just so you have it on and said this is regarding that.


texts: principles of time series analysis by neural network models

texts:
I can understand for speech signals, words are correlated and therefore one should have a reason to believe that recurring NNs or LSTMs could predict by running some complex algorithm with weights and an activation function.

But for random digital signals like stock prices, why can the prices be predicted in any way? Tomorrow's price has nothing to do with those in the past.


texts: My target variable is whether an application is accepted or not.  It is a highly imbalanced target with 98.5% of applications accepted.  I am unclear about the concept of downsampling.  If I were to downsample the applications, do I have to maintain the current ratio of accepted to rejected applications while lowering the total number of applications in the training data or can I change the ratio of accepted to rejected apps to say 50% accepted, 50% rejected?  What are the benefits of doing either of these approaches?  Help is welcome.  Thanks!


texts: Any heuristic for minimal DCGAN latent space dimension?

texts:
I am highly interested in approaching minimal latent space dimension (as many other may be) for DCGANs or autoencoders.

In this example of DCGAN on the MNIST dataset, the person uses a 100-dimensionnal latent space. Since the dataset is made of 10 distinct classes it is reasonable to assume a lower bound of 10 for the minimal DCGAN latent space.

But what about examples with no distinct classes such as the latest face generation by Nvidia?

If no heuristic exist, is there some way that could minimize the amount of time spent tuning the parameter?


texts: How to set seed for random drawings with Numpy?

texts:
I'm trying to set the seed for a few lines of code in a jupyter notebook.
However, when I tried running numpy.random.seed(0) in the initial cell, in the later cells the random generator was not 'seeded'. I had to write a set seed in all the cells where I was running the random draws... Why is that? Is there a way to avoid having to write the same thing in every cell where I have a random draw?
Also, I checked the numpy documentation  and it seems that numpy.random.seed() is not the best practice. I don't understand the example given there... How would I go about, according to the best practices shown in the example, in order to do something similar to numpy.random.seed(constant)?


texts: Error and odd behaviour using Mahalanobis distance in multi-dimensional risk calculations

texts:
I am trying to use an approach detailed in a 2015 paper by NASA (https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20150018915.pdf) on using the Mahalanobis distance to aggregate a number of Likert based risks into a single risk metric, called the MRISK.  The Likert scale is 1 to 5 for each risk.
The NASA paper explains how to combine 3 risks.  According to the paper, the approach is to define:
$$
 MRISK = 4\left(\frac{d_{min}}{d_{min}+d_{max}}\right) + 1
$$
where
$$
 d_{min} = \sqrt{\left(\pmb{x} - \pmb{x}_{min}\right)\pmb{S}^{-1}\left(\pmb{x} - \pmb{x}_{min}\right)^T}
$$
and
$$
 d_{max} = \sqrt{\left(\pmb{x} - \pmb{x}_{max}\right)\pmb{S}^{-1}\left(\pmb{x} - \pmb{x}_{max}\right)^T}
$$
where $\pmb{S}$ is the variance-covariance matrix, which is derived from the risk vectors.
As we are using a Likert scale of 1 to 5, by definitions given in the paper we should have:
$$
\pmb{x}_{min} = (1, 1, 1) \quad\mbox{and}\quad \pmb{x}_{max} = (5, 5, 5)
$$
I believe I've noticed an error in the NASA paper, and in addition to the error I've noticed some odd behaviour about the results from the MRISK calculation which are not intuitive/obvious.
Dealing firstly with the error, taking the variance-covariance matrix given in the NASA paper:
$$
 S = \begin{bmatrix}
       0.733181 & 0.21235 & 0.54867\\
       0.21235 & 1.67585 & 0.62334\\
       0.54867 & 0.62334 & 1.75407 \\
     \end{bmatrix}
$$
the paper calculates the MRISK for the consequence vector (2,1,5) as:
$$
 d_{min} = \left[(2,1,5) - (1,1,1)\right] \pmb{S}^{-1} \left[(2,1,5) - (1,1,1)\right]^T = 10.60895
$$
and
$$
d_{max} = \left[(2,1,5) - (5,5,5)\right] \pmb{S}^{-1} \left[(2,1,5) - (5,5,5)\right]^T = 26.5657
$$
and consequently
$$
 MRISK = 4\left(\frac{10.60895}{10.60895+26.56579}\right) + 1 = 2.14152
$$
However, it is clear they've not followed their own definition of $d_{min}$ and $d_{max}$ as they've not taken the square-root of these values.  I believe the correct values are:
$$
 d_{min} = \sqrt{10.60895} = 3.257
$$
and
$$
 d_{max} = \sqrt{26.56579} = 5.154
$$
and so
$$
 MRISK = 4\left(\frac{3.257}{3.257+5.154}\right) + 1 = 2.54893
$$
The error in the paper can be highlighted by calculating the MRISK for the consequence vectors (2, 2, 2), (3, 3, 3) and (4, 4, 4).  I would expect the MRISK to be 2 for (2, 2, 2) and this is indeed what you get when you take the square-roots when calculating the Mahalanobis distance (for ref, $d_{min} = 1.29629$ and $d_{max}=3.88887$).   However, if you don't take the square-roots and use the approach in the paper, then the MRISK for (2, 2, 2) is apparently only 1.4 as $d_{min}^2 = 1.68037$ and $d_{max}^2=15.12334$.
In addition to the error, I've noticed some odd behaviour about the MRISK for certain vectors.  For example, for consequence vector (2, 1, 1) the MRISK (when taking square-roots when calculating $d_{min}$ and $d_{max}$) is 1.93924, since $d_{min} = 1.33633$ and $d_{max} = 4.354789$.  However, for consequence vector (2, 1, 2), the MRISK is 1.89119, $d_{min} = 1.22911$ and $d_{max} = 4.287625$
At the moment those results don't make sense to me, for if the risk increases for one of the factors (keeping the other 2 the same), then I would have thought the MRISK would also increase.  I can understand why the MRISK for say (1, 2, 1) would be different to (2, 1, 1), as we've increased one risk and decreased another, but I don't understand why the MRISK for (2, 1, 2) is lower than for (2, 1, 1) when "intuition" would say that the MRISK for (2, 1, 2) should be higher than for (2, 1, 1)
So, my questions are:

Has anybody else identified this errors in the NASA paper, and is my analysis correct, ie has NASA indeed made a mistake here?
Can somebody provide an intuitive explanation for why the MRISK for (2, 1, 2) is lower than for (2, 1, 1) when I would expect the MRISK for (2, 1, 2) to be higher than for (2, 1, 1).


texts:
I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the exact same values as per the training data. However, my test tuples can have values that exceeds the maximum value of the data in the training set. For example, in the training set I have following values for a given feature.

Maximum : 20457
Minimum: 3

In the testing tuple, I sometimes get values like 35002. This is present in most of the features.

The problem would have been solved if I knew the maximum and minimum values for all feature, but it no possible. The maximum value can go up to any value.
How do I do data normalization in a scenario like this? Can someone please help me with this?


texts:
How do I correct my data or format it so that it is presentable, and fix my graphs?

Dataset is 345551 rows × 7 columns.
I am using numpy, pandas, seaborn and matplot lib.
It seems that my pricing data is being displayed in scientific notation.

When I fit a linear regression model I get the following coefficients

property_type= -3.096186e+05
new_build= -1.909146e+04

When I use a train/test split and check my predictions they don't make sense.

index=246862
actual=440000
predicted=4.252606e+05


texts:
What I have so far

I have a set of images that I am trying to classify. I can also extract different feature descriptors from the images using algorithms such as hu moments, color histogram, and SIFT. I can the build a vocabularies from each of these algorithms.

What I do not understand is how can I combine these vocabularies together. From my understanding these feature descriptors can be of different size depending on what the features are. What would be the proper way to build this vocabulary from these different algorithms.

What is confusing me is how would we be combining feature descriptors from different algorithms which have different sizes? SIFT will give a m x 128 array where m is the number of feature descriptors of set size 128. A color histogram may give a count of each color in the range of 0-255. How would I combine these two in a meaningful way?

Here is the paper I was reading which gave me the idea of combining the vocabularies.

A Visual Vocabulary for Flower Classification

Here is what I was reading before:

Implementing Bag of Visual words for Object Recognition

If anyone can point me in the right direction I would greatly appreciate it.


texts: Training an ensemble of small neural networks efficiently in TensorFlow 2

texts:
I have a bunch of small neural networks (say, 5 to 50 feed-forward neural networks with only two hidden layers with 10-100 neurons each), which differ only in the weight initialization. I want to train them all on the same, smallish dataset (say, 10K rows), with a batch size of 1. The aim of this is to combine them into an ensemble by averaging the results.

Now, of course I can build the whole ensemble as one neural network in TensorFlow/Keras, like this:

def bagging_ensemble(inputs: int, width: int, weak_learners: int):
    r'''Return a generic dense network model

    inputs: number of columns (features) in the input data set
    width: number of neurons in the hidden layer of each weak learner
    weak_learners: number of weak learners in the ensemble
    '''
    assert width >= 1, 'width is required to be at least 1'
    assert weak_learners >= 1, 'weak_learners is required to be at least 1'

    activation = tf.keras.activations.tanh
    kernel_initializer = tf.initializers.GlorotUniform()

    input_layer = tf.keras.Input(shape=(inputs,))
    layers = input_layer
    hidden = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)\
                (input_layer)
    hidden = []
    # add hidden layer as a list of weak learners
    for i in range(weak_learners):
        weak_learner = tf.keras.layers.Dense(units=width, activation=activation, kernel_initializer=kernel_initializer)\
                        (input_layer)
        weak_learner = tf.keras.layers.Dense(units=1, activation=tf.keras.activations.sigmoid)(weak_learner)
        hidden.append(weak_learner)

    output_layer = tf.keras.layers.Average()(hidden)  # add an averaging layer at the end

    return tf.keras.Model(input_layer, output_layer)

example_model = bagging_ensemble(inputs=30, width=10, weak_learners=5)
tf.keras.utils.plot_model(example_model)

The resulting model's plot looks like this:

However, training the model is slow, and because of the batch size of 1, a GPU doesn't really speed up the process. How can I make better use of the GPU when training such a model in TensorFlow 2, without using a larger batch size?

[The motivation for using this kind of ensemble is the following: Especially with small datasets, each of the small networks will yield different results because of different random initializations. By bagging as demonstrated here, the quality of the resulting model is greatly enhanced. If you're interested in the thorough neural network modelling approach this technique comes from, look for the works of H. G. Zimmermann.]


texts: How to model & predict user activity/presence time in a website

texts:
I need to make a prediction model based on some historical data from a website's user login system. Suppose my dataset has some features like user login time and logout time for each day for a specific user. Login and logout times can be multiple in a day for a specific user. Suppose, If the user login 5 times in the website in a day, there will be five entry points as rows to the dataset for that user, logout also works like this. Now from the login and logout time, I need to find out the active time that user was logged in to the website as well as predicting the inactive time in which user is not available/present in the website. How can I do this? Which algorithm should I use and which prediction model (Linear regression/Logistic Regression/Time series) need to choose in this case? It will be very much helpful if you can suggest me for this specially to implement in R. Thanks.

Edit:

Actually I need to find out/predict a time in which the user is active in a website during the day. I have a dataset with 3 columns listed as "user_id", "login_time" and "logout_time". Now I am trying to make another column "active_time" in which I'm trying to compute the user's active time in the website by subtracting the login time from logout time and it can be multiple as user can access website multiple times in a day. Now I need to predict the time in which the user is active in the website where active time is the target variable and login, logout time as predictor. I also trying to make a linear regression model for this prediction. But I don't know whether my process is correct or not for this problem. Can anyone please let me know which type of model I need to build for this prediction? Is it will be Linea regression, logistic regression or time series ?


texts: Different t-test results from R and LibreCalc

texts:
while I was trying to apply one-tailed t-test to two vector of data on R, I just wanted to apply same thing on librecalc also and then I noticed that p-values are different. Here is a summary what I have done so far:

in R I applied: t.test(data[,1],data[,2],paired = TRUE,alternative = 'greater') and then result is:

Paired t-test

data:  ttest[, 2] and ttest[, 3]
t = -134.33, df = 49, p-value = 1
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
-8.318141       Inf
sample estimates:
mean of the differences
            -8.2156

It returns 1 as p-value. However, same thing on Librecalc, it gives following:

Paired t-test
Alpha   0,05
Hypothesized Mean Difference    0
    Variable 1  Variable 2
Mean    33,003568   41,219168
Variance    0,155220041404082   0,038756230791837
Observations    50  50
Pearson Correlation 0,04472285907661
Observed Mean Difference    -8,2156
Variance of the Differences 0,187038752653061
df  49
t Stat  -134,325507363656
P (T<=t) one-tail   7,16710643183049E-65 //(Here is the p-value)
t Critical one-tail 1,67655089261685
P (T<=t) two-tail   1,4334212863661E-64
t Critical two-tail 2,00957523712924

Except the p-values, all the other results are same as you see. Can someone tell me what is going on here, or what should I do for a one-tailed t-test to compare two algorithms. (I have 50 different runs' results of both algorithm). Here is the data:

31,3888 41,028
33,3064 41,2048
32,6824 41,5216
33,4552 40,9848
32,6832 41,3152
33,2704 41,212
32,8344 41,2104
32,6376 41,2904
33,3552 41,22
32,6    41,2152
32,6752 41,4784
32,9464 41,1352
33,2064 41,4216
33,1752 41,0568
33,0952 41,2808
32,5272 41,7328
33,1624 40,8696
33,0192 41,0544
33,1744 41,1048
33,4464 41,0488
33,7448 41,3832
33,0568 41,4656
32,836  41,1192
33,1    41,0832
33,2144 41,4944
33,0544 41,2736
33,3232 41,4232
32,8616 41,2304
33,228  41,2184
33,2472 41,02
32,8952 40,9808
33,5056 41,3808
33,5248 41,348
33,4416 41,1072
33,0416 41,016
33,0504 41,168
32,4064 41,184
32,6256 41,572
32,488  40,7848
33,4304 41,2536
33,2128 41,2248
32,9576 41,5392
33,0136 41,4608
32,9296 41,0872
32,632  41,0128
33,2616 40,968
32,7664 41,0192
32,8952 41,3344
33,3512 41,2984
32,4408 41,1216


texts: Weights for keywords in a set of documents using Term Frequency and Inverse Document Frequency

texts:
I have a set of 270 documents. I have a list of keywords that are spread across these 270 documents. I would like to rank these keywords based on the TR-IDF method. From what I read, this method helps to determine the importance of a word in a particular document as the TR of a word is of one particular document.

Could someone tell me how can I extend this approach to get the importance of a word across all 270 documents?

Eg: The word "abc" and "xyz" and spread across all the documents. So
 I can have a document matrix(word and the tr-idf) for "abc" and "xyz"
 for all the documents. Now how do I decide which word is important
 overall, not just in one document?


texts: reinforcement learning: Decompose a policy gradient

texts:
I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f

Couldn't figure out how the first equation becomes the second equation?

In the second equation, why the first Expectation value have only s_0, a_0, s_1, a_1 ... s_t, a_t but no reward involved? Also, the second Expectation Value has only r_t+1, s_t+1, ... r_T, s_T, but no action involved? Could anyone please share the thoughts/intuition behind this? Thank you!


texts: How do I model time series data that contains a unique ID and a constant time step?

texts:
My data currently looks like this (after processing, 15 mins timesteps):
date                ID     count
2017-06-01 00:00:00 1      4208
2017-06-01 00:00:00 2      3993
2017-06-01 00:15:00 1      3740
2017-06-01 00:15:00 2      3482
...

I want to predict the count for a given ID in a given timestep (eg: count for ID 1 at 20:00:00-20:15:00 interval) using the past data. I believe that it can be modelled using LSTM since we are using a time sequence, but I'm unsure on how to format the inputs/outputs for it. I would appreciate if someone could share any related work or an overview of how to model such a problem.


texts: How can I use time stamps for classification?

texts:
My data consists of entries when an event is True, namely, when a train crossing is down. So I will have entries within a day like so (just examples):

start,end,interval-length

10:43:02,10:46:02,180,
10:49:06,10:51:06,120

The state is of course True for these entries, and the times not logged throughout a day are when the state is False, aka traversable. (however I have a strong feeling I should explicitly pad these times in, so that I have full 24-hour cycles logged)

The data presented is a time series, based on which I wish to make predictions for coming days. I want to be able to ask "What are the chances that the crossing is down at time $t$, of day $d$?" (the second argument is important because I will have 7 different models, one for each day of the week due to the differing train traffic schedules in a week), taking into consideration the previously observed times.

This of course needs to take into account the fact that when the events happen, they have a certain length (as shown by my data). This is not inherently a classification problem because I am also interested in a confidence of the prediction, which is not as simple as taking a built model's performance parameters. Assuming though that this could be a promising way to tackle the problem, how can I build a model so that the times later on in a day don't wrongfully bias it?

Consider the simple example of some dataset, where days are represented as indexing numbers 1-7. An unsupervised classification algorithm would be biased to guess more towards saturday and sunday, because they are higher numbers (and in this case the solution is to have 7 different columns, each with boolean values instead - but I can't do anything similar to this in my case). I hope this example clears up my question a bit.


texts: Best measure to indicate quality of LDA model

texts:
On my corpora, I am running LDA with different settings (I experiment with different number of topics, different different ngrams and TFIDF or regular BOW).
Now, I want to rank these setups to select one best topic model to continue working with.
In order to rank them, I have calculated both the coherence value as well as the perplexity for all the different settings, as is done here. In the link, the number of topics is selected using the coherence value. However, a rationale for why coherence is chosen is missing.
Therefore, I have a few questions: are there more common measures for this task? What are likely reasons for picking one of these metrics over the other?


texts: Predict multi-steps for Multi-Time series output with Keras

texts:
I have a dataset like the following one:

Each column is a different numerical feature. Each row represents a timestamp. I want to create an LSTM model that can make prediction of the future time-steps for all the features. For example, I want to use the first 2000 examples to train my model and use the next 1000 to test it. The problem is that I do not know how to proceed.

Since we do not have y's value in this dataset, I was thinking of creating them by shifting the time t+1 to t has a new column y. I explain myself: for example I will have a new column 14, with the value 996.52 for timestamp 0, which is the value at time-step 1 for the feature 0. And so on for all the time-steps and all the features.

The problem is after that I do not know how to feed my LSTM using Keras to make several steps predictions with such a dataset.


texts: Backpropagation with log likelihood cost function and softmax activation

texts:
In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$
Suppose we have 10 output neurons, when back propagating the error, only the gradient w.r.t. $y^{th}$ output neuron is non-zero and all others are zero. Is that right?

If so, how is the below equation (81) true?
$$\frac{\partial C}{\partial b_j^L} = a_j^L - y_j $$
I'm getting the expression as
$$\frac{\partial C}{\partial b_j^L} = y_j (a_j^L - 1) $$


texts: Merge two or more ML models with different size of output togetger

texts:
I have several models that classify the input (word embedding) into several classes. My problem is that I need to train these models separately and need to merge the output of these models together to get a label.
For simplicity assume only two models:
Model 1: predicts A, B or C
Model 2: predicts D or E

Then, I need to classify the input X so that I get the joint probability over A, B, C, D and E.
I have tried this code in Keras/Python, in which I define neural networks and remove the softmax activation function after training:
inputs = tf.keras.Input(shape=dim_in)
x = layers.Dense(dim_out)(inputs)
outputs = layers.Activation(activation="softmax")(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs, name='keras_model')
model_without_softmax = tf.keras.Model(inputs=model.inputs[0], outputs=model.layers[-2].output, name='keras_model', trainable=False)

model.compile(optimizer=tf.keras.optimizers.Nadam(),
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['sparse_categorical_accuracy'])

Then I get final prediction with softmax over joint arrays of predictions:
prediction = softmax(np.append(prediction_1, prediction_2))

It wasn't working because the output values from the neural networks were on a different scale when classifying into 2 or 3 classes. Thus, I tried dividing into a number of classes but without success:
prediction = softmax(np.append(prediction_1/len(idx2label_d1), prediction_2/len(idx2label_d2)))

Can you point me to right direction for the general formula of merging the neural network's outputs together?


texts: Training data requirements for NLP models

texts: Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the labels are, the event frequencies, the chosen algorithm, etc. Are there any general frameworks I can use to estimate how much data I would need? I'm going through an exercise of having data owners labeling their text data and i'd like to give them a recommendation for how much data they should label other than more is better.


texts: Implementation of reliable rule learning

texts: I want to perform "reliable rule learning", i.e. mining a set of rules with a very low number of false negatives. I recently read the paper "Reliable agnostic learning" by Kalai et al. (https://doi.org/10.1016/j.jcss.2011.12.026) and they basically describe what I want: Rules are determined to reliably classify data points, and the reliability is partly reached by allowing "I don't know" as an additional answer. Sadly, their paper is purely theoretical and I could not find a corresponding implementation. Is there an implementation of reliable rule learning or a similar algorithm? At best, open source and Java-based?


texts:
I've been reading about one-hot encoding for categorical values. Could similar mechanism make sense for continuous values? I'm looking at a problem, where I try to predict the volume of product X, with a variables being the price of product Y.

The reason is to increase the strength of the signal from the price of Y, which otherwise could be understated because overall prices fluctuate in relatively low range. The idea would be to e.g. remove attribute of "Price Y" and replace it with "Promotion Y" based on a set threshold.

E.g.

Price X | Price Y | Vol X | Vol Y  | Promotion X  | Promotion Y
100     | 90      | 40    | 50     | 0            |  0
75      | 75      | 60    | 60     | 1            |  1
100     | 75      | 30    | 70     | 0            |  0

I could just test it out on a dataset, but this could be very specific to this very set and I just want some more theoretical suggestion if and where such approaches would make sense?


texts: Preparing for interview - Logistic regression question

texts:
So I'm doing some exercises to prepare for a interview test. However there's one of the assignments I don't understand. Maybe some of you can explain what they want me to do? It would help me to understand the underlying questionframes I may come across.
This is the assignment:
Logit Regression
Have the function LogitRegression(arr) read the input array of 4 numbers x, y, a, b, separated by space, and return an output of two numbers for updated a and b (assume the learning rate is 1). Save up to 3 digits after the decimal points for a and b. The output should be a string in the format: a, b
  def LogitRegression(arr):

  # code goes here
  return arr

  # keep this function call here
  print(LogitRegression(input()))

Logistic regression is a simple approach to do classification, and the same formula is also commonly used as the output layer in neural networks.
We assume both the input and output variables are scalars, and the logistic regression can be written as:
y = 1.0 / (1.0 + exp(-ax - b))
After observing a data example (x, y), the parameter a and b can be updated using gradient descent with a learning rate.
Examples:
Input: [1, 1, 1, 1]
Output: 0.881, 0.881
Input: [2.2, 0.0, 5.1, 5.7]
Output: 7.3, 6.7
What I don't understand is, do they only give me 4 scalars and want me to train on x, y and then predict a and b. Or is a and be supposed to be weights and bias and I need to return the trained ones? The output numbers in the second example doesn't match probabilities so it can't be that. I might be overthinking this and it's just a simple thing I need to do?
This is the code I've tried:
arr = np.array([2.2, 0.0, 5.1, 5.7])
arr2 = np.array([1.0, 1.0, 1.0,1.0])

def Logit(arr):
    learnrate = 1
    X = arr[0]
    y = arr[1]
    weights = arr[2]
    bias = arr[3]

    y_hat = 1/(1+np.exp(np.dot(X, -weights) - bias))
    new_weights = weights + learnrate * (y - y_hat) * X
    new_bias = bias + learnrate*(y - y_hat)
    print(new_weights, new_bias)

Logit(arr)

Output:
2.900000098664297 4.700000044847409


texts: Does statsmodels compute R2 and other metrics on a validation-/test- set?

texts:
Does statsmodels compute R2 and other metrics on a validation set?
I am using the OLS from the statsmodels.api
when printing summary, an r2 and r2_asjusted are presented.
I did not trust those 0.88 and computed an own adjusted R2 with scikit-learn r2_score and the adjusted r2 function from this answer resulting in 0.88 as well. So the question arose.


texts: What is the state of the art/research metric to compare ellipses but jaccard coefficient?

texts:
Im looking for the, if there is one, metric to compare ellipses with each other. Last time a had a similar dataset (malaria cells, now its pupiles) i used jaccard coefficient but that was more because of i didnt had the time to do further research on this topic.

I jused the jaccard coefficient like that:
- transform the multi-d data in 1D to make the comparation even possible.

Even tho it worked quite well i didnt like it that much and im quite sure there a better metrics out there for that matter.

I hope you all stay fit in this very challenging time for some


texts: PMML XML Model == same as config file or implementation?

texts:
there is the PMML standard defined as XML Schema for Data Mining Model portability between different DM software. My question is, does this xml file serve as a sort of "config file" with the specifics of the underlying domain-model relation (with the specific model implementation logic for the data mining problem concerned), for the other software that you want to import your model?
Or is the PMML model the implementation itself, I mean is there any logic implemented to execute the model with the PMML xml file itself (e.g if you add some test data and let it run through your xml) or must there exist an implementation of the learning algorithm in the software and the PMML is just the configuration of the specific model?


texts: Conditional Multivariate Gaussian Distribution - Section 2.3, equation 2.74

texts:
Please refer section 2.3 (page 86-87) in Pattern Recognition and Machine Learning - Bishop

$$\mathit{N}(\mathbf{x}|\mathbf{\mu}, \Sigma)$$
where

$
\mathbf{x} =
\begin{bmatrix}
x_{a} \\
x_{b}
\end{bmatrix}
$,
$
\mathbf{\mu} =
\begin{bmatrix}
\mu_{a} \\
\mu_{b}
\end{bmatrix}
$,
$
\mathbf{\Sigma} =
\begin{bmatrix}
\Sigma_{aa} && \Sigma_{ab}\\
\Sigma_{ba} && \Sigma_{bb}
\end{bmatrix}
$

The equation below expresses the Quadratic term of exponent in the Bivariate Gaussian probability distribution (ref. book eq. 2.70; ignoring $-\frac{1}{2}$):

$$(x - \mu)^{T}\Sigma^{-1}(x - \mu) = (x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a)
+(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b)
+(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a)
+(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b)$$

  Author states "..conditional distribution $p(x_{a} | x_{b})$ can be
  evaluated from the the joint distribution $p(x) = p(x_{a}, x_{b})$ by
  fixing $x_b$ to the observed value and normalizing the resulting
  expression to obtain the valid probability distribution over $x_{a}$
  .." and then filters out only the linear terms with $x_{a}$ and $x_{a}^{T}$
  component. The result is presented in equation 2.74 as:

$$x_{a}^{T}\{\Lambda_{aa}\mu_{a} - \Lambda_{ab}(x_b - \mu_b)\}$$

I follow the steps, and see that above is due to $x_{a}^{T}$, $x_{a}$ in the first 2 terms of equation 2.70 - where "term(s)" refer to the components in 2.70 added using $+$ operator.

  However, I seem to be getting two additions elements in $x_a$, in
  excess of the those present in 2.74 - due to the linear $x_a$ factor
  in $3^{rd}$ term (of equation 2.70). Excess term(s):  $$ (x_{b} -
 \mu_{b})^{T}\Lambda_{ba}x_a $$

Where am I going wrong? Please guide.

  Note: I am taking the $x_a$ too to be linear component, whereas 2.74 is (seemingly)
  considering only the $x_{a}^{T}$ as linear term. Seems like I am losing track
  somewhere in the middle (perhaps in equation 2.71 - related extract,
  below):

$$ -\frac{1}{2}(x - \mu)^{T}\Sigma^{-1}(x - \mu) = -\frac{1}{2}{x^{T}\Sigma^{-1}x} + {x^{T}\Sigma^{-1}\mu} + constant \ldots eq. 2.71$$

...where 'const' denotes terms which are independent of ${x}$, and we have made use of the symmetry of $\Sigma$.


texts:
I used SGD as optimizer and its accuracy result is about 97%

and I have changed optimizer to Adam

surprisingly, my accuracy became 49%

I only changed optimizer and didn`t change anything else

but the result became very different

is it normal?

Does optimizer highly affect on accuracy?


texts: Should one normalize the frequency values when feeding it as an input to machine learning model?

texts:
Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to my model(K-means) is by taking difference of the time steps of the current web-Page ID load to next web_page ID load.

Now, there are some features in the data like "scroll" (which represents a human scrolling on that webpage) which is occurring multiple times for the same web page ID. Since I am only using delta the way I want to encode this "scroll" as a feature is how many times it happened between the delta(time difference). This gives the frequency.

Now the question is should I do some processing on this raw frequency I calculated, or can I directly feed it to my model. In case more processing is needed, what do you suggest?


texts:
I am learning R and have a tutorial that wants me to remove the percent sign.  The code they want me to use is:

 chocolateData$Cocoa_Percent <- sapply(chocolateData$Cocoa_Percent, function(x) gsub("%", "", x))

My question pertains to what I may be typing wrong, or if there is a different way to do this with R?


texts: Application of Machine learning in modelling experimental system as a PhD topic

texts:
I am working as a control Engineer in a high energy physics lab. I am looking for a PhD thesis topic.
Our experimental system is highly non linear and can't be modeled using standard white box approach. I am thinking of developing Neural Net models of the sensors attached to the experimental system. Also in this way I can model the system with primary inputs and outputs.

So the basic idea is to have a data driven system identification and data driven sensor model.

I wanted your feedback if this can be potential thesis problem.
Thanks.


texts: Find outliers in Hive - SemanticException

texts:
I'm trying to find some outliers on my database using HIVE and I'm using Standard Deviation technique. My query is:

SELECT ID
FROM data
WHERE ID < (AVG(ID) + STDDEV(ID))
  AND ID > (AVG(ID) - STDDEV(ID));

When I run this code I'm getting the following error:

 Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 3:12 Not yet supported place for UDAF 'AVG'

How to solve this problem? Many thanks!


texts: A practical reason to use Cross-entropy as a error-function in Neural networks?

texts:
Cross-entropy tends to allow errors to change weights even when nodes
  saturate (which means that their derivatives are asymptotically close
  to 0.)
  Link

Why is the above statement true? Figures and examples if possible.


texts: How to dowload Wikileaks Cable Leaks documents as text corpus?

texts: I'd like to perform NLP analysis on Wikileaks US Diplomatic Cable Leaks documents (https://wikileaks.org/plusd/), preferably as Python's NLTK3 corpus od Mongo DB documents. I couldn't find any option for download these in any raw text format, so I'm afraid I'm forced to apply some kind of scraping I guess, but I'd be thankful if anyone would give a clue for some simpler solution, if exists any.


texts: Inference of root mean square value in terms of house prediction

texts:
The objective of the task is to predict the housing prices. A model is created based on California housing dataset to predict housing prices and is subjected to evaluation using the below code.

    from sklearn.metrics import mean_squared_error

    housing_predictions = lin_reg.predict(housing_prepared)
    error = mean_squared_error(housing_labels,housing_predictions)
    rmse = np.sqrt(error) --> prediction error

    // value I got for rmse was - 68628.1981

The average housing prices range from 120,000 - 265,000. But the rmse value is
68628.1981.

  Questions

By looking at rmse value, What should I infer?. I know the model is underfitting. I wanted to know the intuition behind the rmse value.


texts: Relation mining of multivariant categorical timeseries without excluding the temporal nature

texts:
To all:

I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following:

I have nominal multivariant timeseries that look like the following:

          Time Var1 Var2 Var3 Var4 Var5 ... VarN
             0     A     A   B    C    A   ... H
             1     A     A   B    D    D   ... H
             2     B     A   C    D    D   ... H
             ..

And so on from times 0 to 1,000,000. What I would like to do is search the time series for rules of the type:

Given Var3 is in state B in the previous step and Var5 is in state D in the previous step, than Var1 will be in state B. What I want to do is have the rules that include the time interval explicitly. A simpler case of interest would simply be to reduce the time series to

               Time    Var1 Var2 Var3 Var4 Var5 ... VarN
                0        0    0    0     0   0   ... 0
                1        0    0    0     1   1   ... 0
                2        1    0    1     0   0   ... 0

Where the the variable is 1 if its state is different from the previous step and zero otherwise. Then I just want to have rules that say something like:

If Var4 and Var5 changed in the previous step than Var1 will change in the current step. Which would be easy for a lag of one, as I could just make the data into something like:

   Var1 Var2 Var3 Var4 Var5 ... VarN Var1_t-1 Var2_t-1 Var3_t-1 ...

and then do sequence mining, but if I want to have rules that aren't just a single lag but could be lags from 1 to 500 than my data set begins to be a little difficult to work with.

Any help would be greatly appreciated.

Edit to respond to comment:
Each column could be in one of 7 different states. As far as a target, it is non-specific, any rules between the columns would be of interest. However, predicting columns 30-40 and 62-75 would be particularly interesting.


texts: Is it bad to have a lot of one class of Data [K-NN classifier]?

texts:
I am trying to train a sklearn K-NN classifier on a labeled text dataset (in Irish). There are 5 classes, 0-4, but there is a lot of variation between how many there are in each class.
What I have done is I've gotten a corpus of Irish text, iterated through every word and stripped a few letters from it based on a linguistic form it took (or not). The problem is, class 4 (which means no action was performed) accounts for 16.5M out of 20.1M entries and it goes all the way down to class 3 with only 36,000 entries.
Gathering more data probably won't help as this basically represents the proportion of times these forms of words appear in real life.
Is this bad for classification and will it bias the classifier in any way? If it does, is that bias actually of help?
Any help is appreciated.
Justin


texts: LSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?

texts:
From this answer I know that

the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately.
The error gradient that you want to calculate for gradient descent is $\nabla_{\theta} C(X, \theta)$, which you can therefore write as:
$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$
The derivative of the sum of any two functions is the sum of the derivatives, i.e.
$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$
In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:
$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

Question:
When working with LSTM specifically, am I indeed allowed to apply this $\frac{1}{|X|}$ at the end of the backprop once I've summed-up gradients of weights across all timesteps?
Or should I always apply $\frac{1}{|X|}$ straight away, to any gradient flowing into the top layers of my network?

This bothers me because LSTM is not merely multiplying things like a basic RNN would. Instead, LSTM has summation, which then gets multiplied with other things as we descend to earlier timesteps.
For example, this happens when we have to add some gradient flowing from all the 4 gates the LSTM into grad_wrt_resultAtPreviosTimesep (It will be required to compute grad during previous timestep once we get there)
Using an overly-simplified algebra example:
I feel that multiplying immediately by $\frac{1}{X}$ would represent the left side of this expression:  $(\frac{1}{X}a + 40)*100 \neq \frac{1}{X}
(a+40)*100$. On the other hand, multiplying by $\frac{1}{X}$ in the end of backprop would represent the right side.
Edit:
Just as a reminder, here is what LSTM looks like at each timestep, taken from this blog:


texts: What does it mean for a method to be invariant to diagonal rescaling of the gradients?

texts:
In the paper which describes Adam: a method for stochastic optimization, the author states:

The method is straightforward to implement, is computationally
efficient, has little memory requirements, is invariant to diagonal
rescaling of the gradients, and is well suited for problems that are
large in terms of data and/or parameters.

What does "invariant to diagonal rescaling of the gradients" imply here?


texts: Replace Values with corresponding numbers in the same dataframe

texts: Please help me to replace NR values with corresponding values as in the dataframe.


texts:
Im trying to learn and get the hang of tflearn.  One thing that I have tried to do is save my model, and then reload it.  I get an error that says:

W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for quicktest.model

Below I have my model as well as that checkpoint file that gets created with the saving of a model.  I have googled this error and tried many different file changes but nothing has fixed it.

    import tflearn
    data, labels = load_csv('my_csv', target_column=0, categorical_labels=True, n_classes=2)
    net = tflearn.input_data(shape=[None, 32])
    net = tflearn.fully_connected(net, 32)
    net = tflearn.fully_connected(net, 2, activation='softmax')
    net = tflearn.regression(net)
    model = tflearn.DNN(net)
    model.fit(data, labels, n_epoch=1, batch_size=10, show_metric=True)
    model.save('quicktest.model')
    model.load('quicktest.model')

checkpoint

model_checkpoint_path: "quicktest.model"
all_model_checkpoint_paths: "quicktest.model"


texts:
I built a neural network for my NLP preblem using GloVe and an embedding layer.
Each word is converted to a vector of 100 dimensions and input_length is 300. Word2Vec has 68,546 words.
How does the math work?
My understanding is that the input is (None, 300), which is label encoded texts.
This is converted to vectors and the process is not subject to backpropagation.
So, in a way the input has the shape of (None, 300, 100)
Question 1:
Assume these vectors are not trainable, is it same as grayscale images at this point since its 2 dimensional?
My embedding layer has such shape
Layer (type)                 Output Shape              Param
=================================================================
embedding (Embedding)        (None, 300, 100)          6854600
#param is 6854600 and thats understandable because its 68546 words * 100 dimensions
#param is the same wether the next layer is Dense or Conv1D.
Question 2:
How does the math work here? Are those parameters weights or biases?


texts: PCA and the relationship between number of samples and number of dimensions

texts:
I'm performing PCA on a high dimensional dataset: 800,000 samples by 300 features for which my aim is to identify clusters using Kmeans.
After keeping only the numerical features that weren't noisy or sparse and scaling everything, performing PCA... I still found about 70% of my variance was explained by 48 principle components.
I believe, were I to select only the first 48 principle components that attempting to use K means clustering  won't work effectively. I don't think I have enough samples and that the euclidean space between points in this many dimensions is too sparse.
What I'm trying to understand and build intuition on however is this relationship between dimensions and samples. Given a high enough number of samples, I should be fine to cluster in a corresponding high number of dimensions, right? So is there a method I can use to determine this? Or is there a rule of thumb?
I roughly understand that the relationship between dimensions and features is an exponential one and Wikipedia's article on the 'curse of dimensionality' discussed how a equally distributed latice of 100 points in 2 dimensional space would need to be: 10^(2*5) in 10 dimensional space. But with regards to clustering where I might expect groupings in N dimensional space, i'm not sure evenly spaced sample distributions is a accurate comparison.
Any thoughts?
https://en.wikipedia.org/wiki/Curse_of_dimensionality


texts: fast ML algorithms for binary classification with (large+sparse) binary input data

texts:
I'm sorry that this is so very broad, but as a non-ML scientist it feels to be almost impossible to keep up with recent developments (esp. in deep learning etc.). Hence, I'm asking for guidance on how to handle this specific use case:

The goal is to predict a binary output from ~50,000 binary input variables (the input data being rather sparse with about 1,000 1s on average). The training dataset includes several thousand (fairly balanced) labeled samples. I already have a non-ML solution to this giving good results but it is computationally expensive. Thus, my questions:

Which ML algorithms work well (i.e. train reasonably fast on a small HPC-cluster) on binary data of that scale.
Do they allow to extract information about the inputs (i.e. the magnitude of loadings of the individual binary variables).
How large are the performance advantages of having binary data? As opposed to using the 50k binary input variables I could run a PCA and use the first couple hundred PCs (it takes about 500 to recover 90% of the variance) for training/prediction. What would the advantages/caveats be?

The order of the input variables is not really "random", but their importance might be. Hence I think CNNs would not be the best idea, but are non-convoluting NNs even feasible at this scale? Additionally, it is usually only a few input variables that mostly decide the output, if that makes any difference in model selection.

I have worked with ML in the past, but this is several years ago and my theoretical knowledge is more than rusty. Also, the variety of NN architectures / frameworks etc. has exploded since then, hence I wanted to ask for some input before blindly trying out everything.


texts: Why does feature scaling improve the convergence speed for gradient descent?

texts:
From this article, it says:

  We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.

For linear regresion, from the equation $a_{n+1}=a_n-\alpha\nabla F(a_n)$ it wasn't obvious to me why larger range variables converge slower or how/why an oscillation occurs.


texts: Listwise learning to rank with negative sample relevance

texts:
Typical listwise learning to rank (L2R) algorithm tries to learn the rank of docs $\{x_i\}_{i=1}^m$ corresponding to a query $q$. If we use correlation efficient to label the relevance between docs and query, then the label $y_i\in[0, 1]$. The larger the $y_i$, the more relevant of the doc $x_i$ to $q$. Most L2R algorithms, such as approximated rank/NDCG and ListMLE, focus more on the ranking accuray of positve correlated doc (i.e. $y_i$ close 1) by giving larger weight in the loss function. This is common in information retrieval problem.

However, if the relevance between docs and query can also be negative, i.e. $y_i\in[-1,1]$, and the ranking accuaracy of both large positive and large negativa docs are important, how can L2R algorithms handle this problem ?


texts: Regularization - Combine drop out with early stopping

texts:
I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection.

When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a patience of 5 epochs when my validation loss is increasing.

Does it make sense to use early stopping in combination with dropout?


texts: correlated variables & model performance: optimal trade-off

texts:
on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift.
assuming little to medium interpretability is needed, we have a scenario when even medium-ish gains in predictive performance are very very welcome but, in order to do so, we have to step up the number of moderate-to-strongly correlated variables (0.6-0.8 range, linear correlation). this provides benefits (using R2 here) in the order of 5 to 10 percentage points. these gains absolutely do make a difference for the problem at hand.
but I am concerned that I am implicitly tethering to the same phenomenon (therefore making the model less robust) especially when distribution shifts are on the horizon for the variables upon which the model relies the most.
so, do you have any approaches or even rule of thumbs that can be generally applicable in these scenarios?


texts:
I am currently preparing for an exam on neural networks. In several protocols from former exams I read that the activation functions of neurons (in multilayer perceptrons) have to be monotonic.

I understand that activation functions should be differentiable, have a derivative which is not 0 on most points, and be non-linear. I do not understand why being monotonic is important/helpful.

I know the following activation functions and that they are monotonic:

ReLU
Sigmoid
Tanh
Softmax: I'm not sure if the definition of monotonicity is applicable for functions $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ with $n, m > 1$
Softplus
(Identity)

However, I still can't see any reason why for example $\varphi(x) = x^2$.

Why do activation functions have to be monotonic?

(Related side question: is there any reason why the logarithm/exponential function is not used as an activation function?)


texts: Understanding `get_combination_wise_output_matrix` when investigating a multi-label classification problem

texts:
I am currently working on a multi-label classification problem. I am using the scikit-multilearn library (further reading here)

I understand that train / test split is important for these types of problems - ensuring label combinations are represented well in both (scikit-multilearn implements their own split method for this purpose).

My current goal is to understand how this works so that I can evaluate it properly - I am reading from here primarily.

I am a little confused by the output of the get_combination_wise_output_matrix method. I understand at a high level it is giving me a breakdown of counts of label combinations for whatever order I specify.

Here is an example output I have observed:

The first question I have is does (5, 5): 1, mean anything other than the label at index 5 appears once, and this number is repeated for... reasons?

Additionally I haven't yet found any literature evaluating how / when to use different orders (my current approach to this is to qualitatively evaluate a few different options that seem reasonable given the specific problem I am trying (desperately) to solve

Thanks in advance for any help!


texts: Attitude to text mining and preparing tokens, irrelevant words, low accuracy

texts:
For purpose of quite big project I am doing a text mining on some documents. My steps are quite common:

All to lower case
Tokenization
Stop list and stop words
Lemmatizaton
Stemming
Some other steps like removing symbols.

Then I prepare bag of words, make DTF and classify to 3 classes with SVM and Naive Bayes.

But the accuracy I get is not too high (50-60%). I think that may be because in array of words after all the steps are still many words very irrelevant like first and last name from document. What is the attitude in such case? What can be done during preprocessing to make classifiers work better with higher accuracy?

I was thinking about preparing some dictionary consists of all relevant words to my area but it could be too hard and for sure some important words will be missed.

Any advice what could be done here?


texts: Generate 3D models from screenshots or screen recordings

texts:
I was wondering if there existed any software or libraries that could turn a series of screenshots or screen recordings (such as livestreams or videos) of simple geometries into models that I could import into something like Blender or a CAD Software.
There are some 3D models which I have found online that can be explored and disassembled, but that cannot downloaded and I was wondering if there existed a way to construct these 3D models based off a series of screenshots or a recording of the on-screen model being manipulated in space. Another possible use case would be to take snippets of a livestream or a video of someone doing 3D modelling to model along with them.
I have found this question asking for a similar solution, but my problem is that I'm not working with real-life photographs and that I don't have any reference points. Moreover, these 3D demo websites don't take into account natural lighting and other environmental noise.


texts: A question about dual representations, kernels and notations used in Bishop's book

texts:
I'm having hard times about kernel functions and dual representations on 'Pattern recognition from and machine learning' by Bishop. Here it is the page I'm trying to understand:
Of course setting the gradient of $J(\textbf{w})$ to $\textbf{0}$ and solving for $\textbf{w}$ leads to the (6.3). Now he writes about the design matrix $\Phi$.
For me (without initially considering nonlinear transformations of dataset), the design Matrix $\textbf{X}$ is a $N \times p$ matrix with ($n$- number of samples, $d$- number of components/features)
$$\textbf{X} = \begin{bmatrix}x_1^{(1)}  & \dots  & x_1^{(d)}\\ \vdots & & \vdots \\ x_n^{(1)} \quad & \dots & \quad x_n^{(d)} \end{bmatrix}$$
Considering a nonlinear transformation $\phi: \, \mathbb{R}^D \rightarrow \mathbb{R}^K$ such that
$$
\mathbb{R}^D \ni \textbf{x}=(x^{(1)}, \dots , x^{(d)}) \mapsto \begin{bmatrix}\phi_1(\textbf{x}), \dots , \phi_K(\textbf{x}) \end{bmatrix}$$
I'm assuming the application $\phi$ takes a ROW-vector and brings a ROW vector because I originally assumed that the $D$- dimensional features of matrix $\textbf{X}$ were rows. Is this correct?If yes, why? This reasonment would bring me considering the design matrix $\Phi$ as the $N \times K$ matrix
$$\Phi = \begin{bmatrix}\phi_1(x_1) & \dots & \phi_K(x_1)\\
\vdots & \dots & \vdots \\ \phi_1(x_N) & \dots & \phi_K(x_N) \end{bmatrix}$$
So why does he write that $n$-th row corresponds is given by $\bf \phi(\textbf{x}_n)^T$?
Also how the hell does he obtain the 6.5 by substitution? I really can't figure it out. Thank you for your help.


texts: How to make ML understand that (-) sign means "between numbers"?

texts: When I try to work on this data error is thrown that non-numeric data can't be plotted. How can I make it identified by python that 25-29 means between 25 and 29 age?


texts: Is there any augmentation tool for images and bounding boxes?

texts: I don't have a lot of training data and I'm looking for some tools in python or executable program like labelimg that do some heavy augmentation on images, even better if they also change bounding boxes coordinate accordingly. Any help will be appreciated!


texts:
So I have one csv with 'films' in a database that have a unique ID

movie_title      | ID       | popularity
spider-man        unique ID   4.2
captain marvel    unique ID   9.9

Then about every week, I would like to update that popularity with a new one from another list of movies.

New movie list:

movie_title      | ID       | popularity
spider-man        unique ID   5.5
captain marvel    unique ID   7.4

How can I merge these two csv and output a new csv with only the value popularity updating? I have other values that also change over the week, but I just did popularity for simplicity. The unique ID must be in the new csv as well not only the changes; because, when I resubmit back to the database that is the only way that it will update with the ID.

The second list does not contain the ID's.


texts: Generating prediction when adding new rows into a relational dataset

texts:
There is a PostgreSQL table that I regularly update with new book reviews. Meanwhile, I also have trained a Tensorflow model that takes book review text and return a predicted book review category (an integer).
What is the best practice to automate category prediction whenever a new row is inserted into a relational table?
What works for me right now is a Postgres trigger function with plpythonu that uses Python's subprocess: after row insertion into the target table, the trigger runs a .py file to update the same table with psycopg2. But this processes takes a lot of redundant steps, time and computation so I was hoping to improve it.


texts: what does "blocks" mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

texts:
According to Hortonworks

  HDFS has demonstrated production scalability of up to 200 PB of
  storage and a single cluster of 4500 servers, supporting close to a
  billion files and blocks.

what does "blocks" mean in this post?


texts: Are support vector machines and logistic regression equivalent if data is linearly separable?

texts: I understand that SVMs separate data drawing an hyperplane with the biggest margin, but doesn't logistic regression do the same thing if data is linearly separable?


texts: looking for databases of audio with labelled 'true' and 'deceptive' sections

texts: For lie-detection research I'm looking for databases of speech audio with labelled 'true' and 'deceptive' sections - does anyone have any leads?


texts: Is there a good deployment module using Golang's amazing single binary compilation model?

texts:
From my limited understanding, it's quite annoying trying to deploy models like PyTorch. So one would need something like ONNX.
But another approach like Golang with a single binary is VERY attractive. I have used Golang before and it's reasonably fast and compiles to one binary easily. For Rust, compiling to a single binary is a nightmare.
So I wonder if anyone has seen work that translates XGBoost trees to Golang and uses arrow as the data exchange format? That way, it makes deployment super easy, wouldn't it?


texts: Best format for recording time stamp and GPS

texts: In most data acquisition settings it is useful to tag your data with time and location. If I write the data to csv file, what are the best formats that I can use for this two variables if I want to create a heatmap on Google Maps?


texts: How does a zero centered activation functions like tanh helps in gradient decent?

texts:
I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation function can overcome this problem?

Since, even in the case of tanH function, if all X are positive then too the sign remains same.

Forgive my english, not a native speaker.


texts: In a dataset, how to detect simple relationship between two columns (datatype not restricted to numerical values)?

texts:
Suppose, I have a dataset with the following features:
(FirstName, LastName, FullName, SchoolJoiningDate, SchoolGraduationDate)
My aim is to validate the data before feeding it into the ML pipeline. Some type of relationships which can exist on the columns of this dataset are:

FullName = FirstName + LastName
SchoolGraduationDate >= SchoolJoiningDate + 10 years

Similarly, there can other columns with different data types. I understand, the correlation between two numerical columns can give a sense of how the two numerical columns are related but the limitation of using correlation is as follows:

Cannot help with the actual relationship between two columns (like addition, subtraction)
Cannot be used for other non-numerical datatypes (like text, dates)

Is there any way I can detect such relationships between two columns?


texts: how to configure and access data on spark

texts: friend this week i am very busy on installation and configuration of spark envirnoment  i did the installation and I begin basic operations like creating folder and adding text file to the folder,count the number of word found on the text file. however after i add csv file when i try access the data from the csv file i got the following error.


texts: How to approach training a machine to read a form

texts:
After many rewrites I'm still not entirely thrilled with how I've presented this.... please delete if inappropriate.

Background

As part of my job, we record observations on underwater paper and then have to enter the data manually at the end of the day, a time consuming and error prone process

We have set forms that we write data into as seen in the attached image

Manually entering the data for the top two rows isn't a problem (reef, site, date, observer, slope and benthic cover)

However, manually entering the data for each of the categories e.g Acropora, isopora etc is problematic.

The sheet shows the genus of coral on the left and what transect it was counted on.

What I would like to do

I have been thinking about automating the entry process and so far have been looking at using pytorch to train an algorithm to count the tally marks and write it to a csv

I have quite a large training set that I could use

I don't have much of a background in python but am quite comfortable in R as I use it most days for work and was thinking I could use the reticulate package.

There are different forms for other tasks that we also manually enter but I plan to carry the method over to the other sheets

Actual Question

I haven't found an off the shelf software package to do it so I assume I'll have to write something

So far, I'm thinking I'm going to approach this by first identifying each cell ie T1, Acropora then the machine counts the tally marks in that cell.

Assuming that is a practical way to approach it, what would you call that process so that I can research it and find a way to script it?

I'll take any suggestions on how to approach this :)


texts: Custom loss function with additional parameter in Keras

texts:
I'm looking for a way to create a loss function that looks like this:
The function should then maximize for the reward. Is this possible to achieve in Keras?
Any suggestions how this can be achieved are highly appreciated.

def special_loss_function(y_true, y_pred, reward_if_correct, punishment_if_false):
    loss = if binary classification is correct apply reward for that training item in accordance with the weight
           if binary classification is wrong, apply punishment for that training item in accordance with the weight
                    )
    return K.mean(loss, axis=-1)

The approach I've been looking at for my example is to pass in the weights along with y_true and then cut the tensor into two, separating out the weights and the y_true as seen below. Would an approach like this be possible at all or would this interfere with the normalization process etc?

def decompose_y_true(y_true_and_weights):
    y_true = y_true_and_weights[:,1]
    weights= y_true_and_weights[:,1:]
    return y_true, weights

def custom_loss(y_true_and_weights, y_pred):
    y_true, y_weights = decompose_y_true(y_true_and_weights)
     loss = # some loss operation
    return K.mean(loss, axis=-1)

Much more elegant would be if I could pass in my weights over the sample_weights parameter in the fit function, but it seems there are some limits what shape those weights can have, and also there's no way to retrieve them within the loss function as far as I can tell. Or is there any way to pass this into the loss function somehow so I can operate on them from there?


texts: I was wondering, if there is any way to install pytorch in windows like the way we can install tensorflow. My machine is not supporting docker.


texts: Tracking time-series latency using conjugate priors

texts:
I need to do a project using Bayesian statistics for a class and I am trying to apply it to my work.
I help manage a time series database with 40,000+ different time series that we collect. The time series have various frequencies and some of them have frequencies that change over time (was sampled hourly now every 15 minutes for example). Other time series might be seasonal gauge data, only on a few months out of the year. Lastly, some time series that have hourly frequency but only come in once a day or every 6 hours.
We have a few hundred time series that are considered critical that we are warned if the latency (current time - last timestamp) is greater than a specified time. This leaves the vast majority of time series that we are not tracking and I want some way of automatically doing that.
I want to track the latency using conjugate priors. I think I could model it as a gamma likelihood and gamma prior then estimate the posterior every hour. I would then use this to see if the latency falls out of a certain credible interval.
I see a few pitfalls and that this is more or less a time series as well so will it suffer from autocorrelation? Can I do something like take the first difference and proceed?
I have been trying to figure out a good way to track this data but it has been difficult because it is time-consuming to compute. Bayes offers an on-line learning that would be fast and efficient, I would also probably be able to do it all in pure SQL. I am not certain that my approach is valid, however.
Thanks for any input.


texts: Higher variance in PCA can mean, that data structure is less informative?

texts:
I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with respect, how much one explains variance in one data structure to the other.

Therefore I decided to use PCA, nevertheless, I don't know if it safe to say such statement: If the variance explained by the same number components N (e.g. 2) is higher in a data structure than in the other, the data structure loses some part of the information. For example as on the image below, if Data_structure_A gives more variance, then it loses more information than Data_structure_B?


texts: How to calculate Precision and Recall using confusion matrix in Matlab?

texts:
I am working on 3 class problem.How to calculate precision,recall ,f-score,MCC of each class while using  MATLAB. Here is my confusion matrix:

2775  0    0
1    591   0
4     0   845

I am calculating Accuracy from Confusion matrix in this way:

Accuracyy = 100*sum(diag(confusionMat))./sum(confusionMat(:));

I want to measure below performance measures for each class.I know the formulas but how to execute this in MATLAB. Please help!!! I would really appreciate! Thank you!

Precision=TP / (TP + FP)
Recall= TP / (TP + FN)
F-score = 2*TP /(2*TP+ FP + FN)
Matthews Correlation Coefficient (MCC)


texts: How to split train/test datasets having equal classes proportion

texts:
I would like to know how I can split in an equal number the following
Target
0    1586
1     318

in order to have the same proportion of 0 and 1 classes in a dataset to train, if my dataset is called df and includes 10 columns, both numerical and categorical.
I would consider the following
y=df['Target']
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.1, stratify=y)

so to do a stratification, but I do not know if it is right and I would appreciate if you could confirm it or provide an alternative to do that.
Sample of data
Fin                 Eco   Target
High percentage     12      1
Low percentage      5       0
Medium percentage   48      0
NA                  3       1
TBC                 NA      1
Low percentage      25      0
Medium percentage   12      0

How can I check if it is actually splitting in equal classes proportion my dataset?
I think the best way to train a model should be having an equal proportion of 0 and 1 values. Right now I have 5 times data with Target=0.


texts: Why MLP only learns bias for unbalanced binary classification?

texts:
I just started training a MLP model on a data set which has the following statistics. Notice that both train and validation sets are unbalanced (88.4% negatives).

Note: The unbalance isn't caused by sampling and is because that's the natural ratio of the classes. E.g. when I look back in a 3 month period, I see I have eaten 88 apples and 12 oranges; and looking forward, I might still favor apples more than oranges by the same ratio. I may be wrong here but I think I shouldn't give class weights nor resample to balance the two classes, because that's their natural ratio.

Train data:
    Shape (features): (891473, 122)
    Shape (labels): (891473,)
    # of classes: [(0, 788118), (1, 103355)]
    % of classes: [(0, 0.88406266931247501), (1, 0.11593733068752503)]
Validation data:
    Shape (features): (251141, 122)
    Shape (labels): (251141,)
    # of classes: [(0, 222009), (1, 29132)]
    % of classes: [(0, 0.8840014175303913), (1, 0.1159985824696087)]

I'm using three 8-sized dense layers each following a ReLU layer, finally a sigmoid since it's binary classification.

However, the training process converged after only 2 epochs and basically only learned the bias term and and ended up predicting everything to be negative. The confusion matrix stays unchanged like this since the 2nd epoch till the end (20 epochs):

Train confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884
Validation confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884

I have tried to increase the size of the dense layer (e.g. from 32, 128, 512, 1024). They all yielded the same result.

I'd expect that with 3 x 1024 neurons the model would be complex enough to learn something beyond a single bias term. So I'm quite confused as what I am doing wrong to make it not learning anything other than the bias.

Question:

What could be the possible reasons for the model to learn not much?
What should I try and/or what kind of info should I be looking for at this point to move this forward?

Thank you for your time in advance~!

The following is a snippet of my code to show what I was trying to do:

learning_rate = 1e-4
batch_size = 10000
epochs = 5

DENSE_SIZE = 1024

train = (training_features.values, training_labels.values)
validation = (holdout_features.values, holdout_labels.values)
test = (eval_features.values, eval_labels.values)
eval_sets = [
    ('Train', train),
    ('Validation', validation),
    ('Test', test),
]

model = Sequential()
model.add(Dense(DENSE_SIZE, activation='relu', input_shape=train[0].shape[1:]))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# initiate optimizer
opt = RMSprop(lr=learning_rate)

# Let's train the model
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

model.fit(
    train[0],
    train[1],
    batch_size=batch_size,
    epochs=epochs,
    validation_data=validation,
    shuffle=True,
    verbose=2,
    callbacks=callbacks,
)

EDIT:

The three answers as of writing all suggest that the inefficiency in learning is associated w/ the imbalanced classes and I should try to balance the classes by either using class weights or resampling.

While I'm happy to see an apparent agreement of what the problem is what , could anyone provide more insights about why imbalanced classes cause difficulty even for a complex model? In theory a complex enough model can at least memorize all the rarely positive samples and achieve a higher-than-bias accuracy on the training set.


texts: Transfer Learning with CNN layer trainable True - Accuracy not improving

texts:
I am working on a image classification problem with 4 classes. And I am using Transfer Learning (Resnet50) to train the model. Below are the observation.

Pre-trained weights are from imagenet dataset.  And images used in this problem are different from imagenet dataset.

1. Transfer Learning (Resnet50) with CNN layer trainable false
After training the model, model accuracy was around 95% (which seems to be fair).

2. Transfer Learning (Resnet50) with CNN layer trainable true
Since the images are different from imagenet dataset I planned to train the CNN layers as well. However I am not seeing any improvement in the accuracy of the model. After training for 50 epochs it reached an accuracy of 60% which is quite poor considering I got a better accuracy in case 1.

Could you please help me to increase the accuracy of the model in case 2?

Kindly let me know if you need more details.

Updates:
Class distribution as given below
Class A : 15000
Class B : 11000
Class C : 15000
Class D :  7000

Code snippet :

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = True

    x = base_model.output
    x = Flatten()(x)

    for fc in fc_layers:
        print(fc)
        x = Dense(fc, activation='relu')(x)
        x = Dropout(dropout)(x)

    preditions = Dense(num_classes, activation='softmax')(x)
    finetune_model = Model(inputs = base_model.input, outputs = preditions)

    return finetune_model

base_model = ResNet50(weights = 'imagenet',
                       include_top = False,
                       input_shape = (HEIGHT, WIDTH, 3))

finetune_model = build_finetune_model(base_model,
                                      dropout = dropout,
                                      fc_layers = FC_LAYERS,
                                      num_classes = len(class_list))

Thank you,
DeepGuy


texts: What makes binary cross entropy a better choice for binary classification than other loss functions?

texts:
I'm reading this
post where I came across this quote "Cross-entropy is the default loss function to use for binary classification problems."

But what about it makes it the default and presumably best loss function for binary classification?


texts:
I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic.
Problem is I dont have any idea where to start not even sure what kind of data i should collect to make a cost function.
Please share how to proceed in this case. I understand the point after defining cost function gradient descent can be used to find , but i have no idea how to define such a cost function.


texts: How to pass input to deep learning models for Multiple choice question answering problem?

texts:
I'm currently working on a multiple-choice question answering system. The training set consists of a question, answer and 4 options and I need to predict the correct answer among 4 options. Sometimes there is one paragraph too,  For example :
1.Which among the following is measured using a Vernier Caliper?

[A] Dimensions
[B] Time
[C] Sound
[D] Temperature

Answer : A [Dimensions]

Chapter text: [Book chapter related to Dimension, time, sound and temperature ]

How to feed this input to any of deep learning models?
I thought two approaches :

Using  tokens

and correct and as one hot encoding => [1, 0, 0, 0 ]

Using concatenation

Generating fix sized word embedding for each text :
 - Chapter text = [1,1024]
 - Text         = [1,1024]
 - option_a     = [1,1024]
 - option_b     = [1,1024]
 - option_c     = [1,1024]
 - option_d     = [1,1024]

final_input = concat( [ Chapter text, Text, option_a, option_b, option_c, option_d] ) ==> [1,6144]
and correct and as one hot encoding => [1, 0, 0, 0 ]

Is it good representation for understanding and reasoning over text for mcqa task?


texts: How to deal with missing values in the survey data to perform Paired sample t test

texts:
I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it  providing efficient services to the patients?
We have two columns 'Expected service quality' and 'Perceived service quality'. We will perform this equation= Service quality=Expected-Perceived to see if the service quality is positive or negative. If the difference is negative, it states we need improvement in that area.
Now the problem is the two columns have many missing values and I have to perform paired t test on the data. In the first column there are 15 missing values and in 2nd I have 56 missing values. How do I deal with these missing values?


texts: Can dropout and batch normalization be applied to convolution layers

texts:
Can dropout be applied to convolution layers or just dense layers. If so, should it be used after pooling or before pooling and after applying activation?
Also I want to know whether batch normalization can be used in convolution layers or not.

I've seen here but I couldn't find valuable answers because of lacking reference.


texts: Convolution and Pooling as Infinitely strong priors

texts:
I am currently reading about Convolutional Neural Networks in Deep Learning Book. I am stuck on section 9.4 titled "Convolution and Pooling as Infinitely String Priors".

Could someone please intuitively explain to me what prior probability distribution is and what is its association to Convolution and Pooling operations in context of CNNs. Thank you!


texts: What algorithm can help me discover synonyms?

texts:
In the  context of text mining, I'd like to discover potential synonyms in my dataset. The current dataset is stackexchange's stackoverflow data on archive.org. The result doesn't have to be perfect, I can post-process it by hand. But I need help to have a clue of which term seems to be "similar" to another term.

Here examples of synonyms I am looking for:

postgresql, postgres, pgsql, psql
mobile, phone, android, iphone

Also in best case, it should be possible to also guess multi word synonyms like:

rdbms, relational database management system
obama, barack obama

The algorithm doesn't need to compute whether those are many-way synonyms (like rdbms and relational database management system) and one-way synonyms (like iphone is mobile but not all mobile are iphone).

I read that word2vec can be helpful but I am not sure how to use it.


texts:
Hi I would appreciate it if someone can point me in the right direction.  I'm looking for an algorithm or mathematical theory which I would use to compute the similarity between two ordered lists, where each list element can have n sub-elements.  I will explain with an example:

Suppose I go to a baseball game and I record the sequence of strikes and balls for each of the first 30 players at bat.  My list looks like this, where P is a player, S is a strike and B is a ball. Order matters.

L1: {P1=(S,S,S)}, {P2=(B,B,S)}, {P3=(B,B,S,S)}, ...

My friend goes to a baseball game and does the same thing.  Later, we meet up and compare our lists.  We find that our lists are almost identical except that I recorded a strike for player 16 where my friend recorded a ball.  What are the chances we were at the same game and one of us made a mistake at player 16?

Thanks in advance...


texts: Ideas to check if a given sentence A is the response to a query B

texts: I've a sentence 'A' which is a possible response to one of the x number of questions. I need to compare the response with each sentence to see which question's response sentence 'A' is or not a response at all. Any ideas on how to go about this. Pure nlp based solution will be helpful as there are no existing labelled data for the same.


texts: Can I train two stacked models end-to-end on different resolutions?

texts:
Is it possible to stack two networks on top of each other that operate on different resolutions of input data?

So here's my usecase: like Google, I want to recognize text in images. Unlike Google, I have access to very limited computational resources. My solution for this is that instead of using a sequence-to-sequence network on the whole scene image, I first run object detection using YOLO and then pass the crop to a sequence-to-sequence model, Attention-OCR.

To further reduce processing time, I run the object detection on low resolution, and crop the detection result in the higher resolution input image, so I still have access to the high resolution input when using sequence recognition on my crop. To detect that there is text, the low resolution is adequate, but to read the text, the network needs the higher resolution input.

This all works fine, but I suspect I would get better performance if I could train the whole system end-to-end, so the crop gets directly optimized to result in the best reading of the text. I could stack the sequence-to-sequence model on top of the object detection model to do this, but then the text reading operates on the same low resolution input that is used for the text detection.

Does anyone have an idea how to solve this, or can anyone point me to research that is related to this?


texts: Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

texts:
I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources.

http://www2007.org/papers/paper570.pdf
https://chrisjmccormick.wordpress.com/2015/06/12/minhash-tutorial-with-python-code/

The data that I am working with consists of interactions between users and items. There are 2.2M distinct users and 440M distinct items. Across all of the data, there are only 905M records, so it is very sparse.

In my approach, I am calculating H minimum hash values for each user by reordering the items (of which there are 440M). Users have a wide range of item interactions. The user with the most interactions as 2.5M interactions, the lowest is 1 interaction, the average is 403, and the median is only 26.

In google's doc about Google News, they recommend concatenating 2-4 keys (LSH) and doing this 10-20 times. I imagine this works well when a user has interacted with a smaller amount of items like news articles, but it is woefully low for what I am doing. When I test this number of keys for users that have 1,000+ interactions, many do not have any concatenated min has matches with another user. This is a problem because I can manually calculate cosine or jaccard similarity for some of these users and see an acceptable amount of similarity for my needs. I have found better results by not concatenating hash keys and using as many as 200.

For most of my hash key groups, there are around 2M distinct hash keys for the 2.24M users. As such, there is a fairly low amount of collision.

Do you guys have any tips for increasing the amount of clustering? I am considering using 1,000 hash keys and pairing users if they match on more than one. Thanks in advance.


texts: Good database/plug-in to scrape for academic paper info?

texts:
I am trying to scrape the web to find information about a set of academic papers.
Unfortunately, for many of the papers, I only have the author's name, a year, and part of a title.
(For example, [BANDURA A, 1997, SELF EFFICACY EXERCI])
I have tried using a web driver to look up the papers on Scopus but many of the papers are not there.
Does anyone know of a good academic paper database and/or plug-in that I could search through using the information I have?
Is there a better way to solve this problem?


texts: Can I create a word cloud of crowdfunding donors using word cloud?

texts:
I have a table like this:

FirstName    SecondName     Amount
Lorenzo      Perone         100
Mario        Rossi          25
...          ...            ...

I'd like to create a "word cloud" using "Amount" as weight, is it possible using the "word cloud" tool?
Thanks.


texts: Keras: DepthwiseConv3DTranspose or doing transposed Conv. with a Conv. layer

texts:
I am building an autoencoder for 3D images and would like to use Depthwise convolutions.
For the encoder, I found an
 implementation of a depthwise 3D convolutional layer (DepthwiseConv3D).

For the decoder, I need a DepthwiseConv3DTranspose layer.

I did not find an implementation of this anywhere.
I read that "a deconvolution layer performs also convolution".

This made me think: is it possible to choose parameters so that DepthwiseConv3D performs a transposed convolution?


texts:
I want to use R for implementing a fuzzy inference system. There are 4 input variables and one output. Each rule is dependent on all input variables and based on there membership the output class is decided.

Can anyone suggest a good library and source examples to begin with? The library that have good visualisation support also.
Should I go with 'frbs' R package?


texts: I am very new to Java and primarily a Python and R user. I want to use OpenNLP for certain NLP tasks that I have been entrusted with. Is there any good tutorial or book on OpenNLP that anyone can recommend?


texts:
I tried to install Basemap and it gives me this:

preparing transaction: done

verifying transaction: done

executing transaction: failed

ERROR conda.core.link:_execute(507): An error occurred while uninstalling packag

e 'defaults::conda-4.5.12-py37_0'.

PermissionError (13, Access is denied)

Attempting to roll back

Rolling back transaction: done

PermissionError (13, Access is denied)

Question: What should I do next? I will appreciate your response as I have been on this for some time now.

Thanks.

NOTE: I have also tried to install cartopy but I ran into the same problem.

NOTE: I use Windows 7 32 bits. I use python 3.7.1


texts: Resource and useful tips on Transfer Learning in NLP

texts:
I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?

I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-

Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?

Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.


texts:
I have just finished writing a neural network implementation. I don't do the training myself so the only thing it should do is forward propagation, an incremental update of the input neurons and reading a csv file containing the weights/biases. I don't really know how to test these things (especially the first and last) though.
How do you test that forward propagation is working as it should?