texts: The memorisation capacity of an LSTM (real numbers)

texts:
My question is the following:
It is known that a LSTM can remember sequences of one-hot encodings which represent integers (i.e. output $x_1, ... x_n$ after receiving $x_1, ... x_n$ as inputs, $x_k \in \{0,1\}^m$, where $m$ is the number of distinct integers).
Is it theoretically possible for the LSTM to learn to remember sequences of real numbers instead (that can be expressed in a finite number of bits), i.e. if $x_t \in \mathbb{R}$ instead.
The task I'm concerned with is much simpler - I just want to output the first input $x_1$ after reading the entire sequence $x_1, ... x_n$. I have done some small experiments with $x_t \in \mathbb{R}$, using square loss. There seems to be some level of success, however the results aren't very interpretable (when I look at the weights). Can anyone shed some light on this, specifically:

Does such a configuration of weights exist? (the questions following this quesetion suggests that it does exist)
If so, what are they and if not, why not?

The LSTM model is specified by:
The input, forget and outputs gates:
$$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$$
$$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$$
$$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$$
And the internal state $c_t$ and hidden state $h_t$:
$$c_t = f_t * c_{t-1} +  i_t * \text{tanh}(W_c[h_{t-1}, x_t] + b_c) $$
$$h_t = o_t * \text{tanh}(c_t)$$
As requested, this is the assignment question:

Memory Task Description
Consider the following task: Given an input sequence of $n$ numbers, we would like a system which, after reading this sequence will return the first number in the sequence. That is given an input sequence: $(x_1, x_2, \cdots x_n)$, $x_i \in \mathbb{R}$ the system has to return, at time $t=n$ after 'reading' the last input $x_n$, the first input $x_1$.

Given the task above, consider the above recurrent models (RNNs/LSTMs/GRUs). Which of these arhitectures can (theoretically) perfom the task above? In answering this questions, please consider a simple one-layer model of RNNs/GRU/LSTM with a one-dimensional input $x_t$, a $32$-dim hidden and output layer, followed by a transformation to a one-dimensional final output which should predict $x_0$.
Whenever the answer is positive, give the gates' activations and weigths that will produce the desired behaviour. Whenever the answer is no, prove that there exists no such parameters that an arbitrary input sequence can be transformed to produce the first symbol read.


texts: TF tells me to use Keras optimizer, tells me the opposite when I change it?

texts:
I have this piece of code:

model = Sequential()
model.add(Dense(32, input_dim=9, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='relu'))

learning_rate = 0.001
(1) optimizer = tf.train.AdamOptimizer(learning_rate)
(2) optimizer = tf.keras.optimizers.Adam(lr=learning_rate, decay=1e-6)

model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'])

For the optimizer, at first I had line (1). It worked, but I got the following warning when making ModelCheckpoints:

  WARNING:tensorflow:TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers).

Apparently, you have to use the Keras optimizer if you want to use this. Ok, no problem, so I commented (1) out, and inserted (2).

Now it doesn't even want to compile anymore, and gives me this error instead:

  ValueError: optimizer must be an instance of tf.train.Optimizer, not a class 'tensorflow.python.keras.optimizers.Adam'

This sounds a bit contradictory to me? What am I doing wrong?


texts: What recommendation engine for a situation where users can only see a fraction of all items?

texts:
I want to add a recommendation feature to a document management system. It is a server on which most company documents are stored. Employees browse the web interface and click to download (or read online) the documents they want.
Each employee only has access to a subset of all documents:

My goal: Recommend to an employee the documents recently opened by their teammates, or the spreadsheet that serves as an annex to the document they just opened, or anything they might want to read.

There are many recommendation engines for publicly available data (all Netflix users can see all movies), but the situation here is special: Each employee only has the permission to a fraction of all documents, whereas in Netflix any user has access to all movies.

Example: Employee1 can read DocumentA but not DocumentB. Employee2 can read both and Employee3 cannot read any.

Of course, I must not recommend to an employee documents to which she/he does not have access. Furthermore, I guess I should consider the popularity of a document only in the context of the employees who have access to it. To make things even more complicated, employees sometimes move from a project to another, which impacts the documents to which they have access.

Is there a name for this kind of problem?
Can it be reduced without precision/efficiency loss to a more common kind of problem?
If not, what approach would work well for this kind of problem?

Note: A Netflix-like recommendation engine is not good enough. A document with 50 views should be prominent if only 10 employees (including me) have access to it, but not prominent if 100000 employees have access to it.

In case it is needed, here are a few data specifics: The average company has 1000 employees, about 10000 documents, an employee clicks about 5 documents per day. Each project has an average of 10 employees having access to it, and has about 100 documents. Each employee works on an average of 5 projects in parallel.


texts: How to define the shape of hidden and meory state in Numpy, keras?

texts:
I was very confused about the concept " number of units ". Now I clearly understand it. It means the number of LSTM cell in each time-step.  I thought it means the number of neurons in each gate.
I went through Numpy LSTM codes to understand LSTM model. And I have two question:

Each gate includes neural network layer (yellow Color). It means that we have to define the size of network. As you can see below, the shape of hidden (a) and memory (c) is (n_a, m). In this code, m means the number of training examples. But what is the n_a? I assume that it is the number of neurons in the each gate (neural network). If it is correct, why we don't define this prarameter in keras?
As you know that, each gate includes neural network. Lets assume that number of units is 1. It means that we have only one LSTM cell at each time step. In this case, Can I say that each gates in this LSTM cell include neural network which has only one neuron? In this one neuron, input will be multiplied with weight and added with bias.

Thank you


texts: Does the time to train a model using keras increase linear with epoches?

texts:
Training a model in keras can go very fast. I trained the following model. To train the model with 1 epoch finishes in a few second. However, using 20 epochs it lasts half an hour or so? An epoch means how often each sample in the training set used to train the model. Therefore, I would naively expect a linear dependency. Which factors do lead to a non-linear behavior?

def keras_model(X_train, y_train):
    from keras.models import Sequential
    from keras.layers import Dense, Dropout, Activation
    from keras.optimizers import SGD

    model = Sequential()
    # Dense(64) is a fully-connected layer with 64 hidden units.
    # in the first layer, you must specify the expected input data shape:
    # here, 20-dimensional vectors.
    model.add(Dense(63, input_dim=63, init='uniform'))
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))
    model.add(Dense(63, init='uniform'))
    model.add(Activation('tanh'))
    model.add(Dropout(0.5))
    model.add(Dense(22, init='uniform'))
    model.add(Activation('softmax'))

    sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)

    model.compile(loss='categorical_crossentropy',
                  optimizer=sgd,
                  metrics=['accuracy'])

    model.fit(X_train, convert_labels_to_matrix(y_train),
              nb_epoch=20, batch_size=15)

    return model


texts:
I m studying visual analytics and i have a theoretical question about this topic.

My professor introduced this schema in him slide

For connect data to visualisation.
Some topic is very easy to understand and regards the rappresentation of data, plot and other tool, other topic instead regards the "data" in particular the data reduction using algorithm like PCA.

I m not very clear why using PCA over the data for reduction some data that we want rappresent.
Ad example, if i have dataset with all movie(it s just an example) with some attribute ad example

id,title,year,review, userReview ecc...

Where is the correlation of this data?Why i need apply a PCA for reduce big database and lose some of this components that I can visualize in some plot or schema that for me is important
For example could be interesting see all best movie that have high average review, using PCA we just cluster user in different movie (i saw example on Iris database here: https://medium.com/@kyasar.mail/pca-principal-component-analysis-729068e28ec8 )

Anyone can clarify me?


texts:
I often see that numeric values in machine learning is scaled to 0-1 range. Why is it better?

I have some temperature values in my training set.
What if I will have some values to predict that will be outside training set values?

I mean that eg in training set I will have range of temperatures like 5-20 and MinMaxScaler will fit to these values, and then I will have 25 to predict.


texts: How to do Predictive Modeling for Different Companies

texts:
There is Company X

Company X wants to predict if its opportunities will Win or Lose

I trained my model based on the data of company X
and did the prediction of new opportunities

Now Company Y and Company Z wants to buy this product

Should I predict their opportunities based on my trained model of (Company X) ?

OR

Do a new model that is trained based on (Company: X+Y+Z)

OR

Do 3 separate models for each company

What is the best approach if I want other companies to use my model? In this case its 3 only, what if I have 20 companies

Thanks


texts: Can we have a dataset with slight difference in target values for same value of feature variable?

texts:
I am trying to generate a dataset which involves 1 feature variable(X) and 1 target variable(y).

The feature variable represents values on the X-axis on the graph and target variable represents values on Y-axis.

Datatype of X: integer

Datatype of y: floating point

I have N such graphs for same values of X, but a slight variation in y values.

One of the graph is as follows:

I want to fit the data into a regression.

Now, my question is how to generate the dataset for this use case. Should I include values from all graphs into a single dataset? But, in this case, for every unique value of X, I will have N rows with same value of X and a different value of y?

I am doubtful about this approach.

Any help is greatly appreciated!


texts: RL Advantage function why A = Q-V instead of A=V-Q?

texts:
In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s)

Preliminary, from this post:

  First recall that a policy $\pi$ is a mapping from each state, $s$, action $a$, to the probability $\pi(a \mid s)$ of taking action $a$ when in state $s$.

  The state value function, $V^\pi(s)$, is the expected return when starting in state $s$ and following $\pi$ thereafter.

  Similarly, the state-action value function, $Q^\pi(s, a)$, is the expected return of when starting in state $s$, taking action $a$, and following policy $\pi$ thereafter.

In my understanding, $V(s)$ is always larger than $Q(s, a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$. So, why is the advantage function defined as $A = V - Q$ rather than $A = Q - V$ (at minute 1:12:29 in the video)?

Actually, V might not be larger than Q, because $s$ might actually contain a negative reward. In such a case how can we be certain what to subtract from what, such that our Advantage is always positive?

$Q(s, a)$ returns a value of entire total reward that's expected ultimately, after we pick an action $a$. $V(s)$ is the same, just with an extra reward from current state $s$ as well.

I don't see why a value of $Q - V$ would be useful. On the other hand, $V - Q$ would be useful because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$.


texts: class weights formula for imbalanced dataset

texts:
I am trying to make some semantic segmentation. I have 7 imbalanced classes in my case.
I found several methods for handling Class Imbalance in a dataset is to perform Undersampling for the Majority Classes or Oversampling for the minority classes. but the most used one is introducing weights in the Loss Function. And I found several formula to calculate weights such us:
wj=n_samples / (n_classes * n_samplesj)
or wj=1/n_samplesj
which is the best one?


texts: Perform PCA on columns of different length

texts:
I have about 20-30 columns, all with different lengths. The first column has 25000 rows, the second column has 19000 rows, and it's different for all the columns.
All are the survey data with 0 (No),1(Yes), and 666 (for missing data) values. I want to perform PCA on these data. As each column is of unequal length, I am having a hard time doing PCA due to 'NaN' values for most of the columns that are shorter in length.
I don't want most of the information to be lost, so I am not willing to delete the rows of longer length columns and make every column equal in length.
Please advise me the possible solutions.


texts: Is is possible to train on Hypergraphs in Keras?

texts:
I know that vanilla Keras doesn't support operations with graphs.
For example, the Spektral library, based on Keras API, provides some layers to work with simple graphs. However, it doesn't support graphs with multi-node connections (i.e. hypergraphs).
Is there any way to train a deep learning model on the hypergraph input data?


texts:
I have a weekly dataset and I have to normalize this data.
Data is something like this :

1. week   50
2. week   51
3. week   50
4. week   54
5. week   150
6. week   155
7. week   ...

The important thing is, the difference between week 3 and week 4 (50-54) is not same with week 5 and week 6. And also there is a huge different between week 4 and week 5.

My question is how can i handle all of this things ?

Is the standard normalization functions(for example scikit normalization) can do it for me and should I normalize this data 0-1 or -1 to 1 ?

Sklearn normalization page

NOTE I am working with python and generally scikit-learn library.

Any help is appreciated.


texts:
I have a dataset of photos labeled as triplets based on the style of the interior of flats.
E.g. photo A is more similar in style to photo B than to photo C
I have about 20000 such entries of about 4000 photos
I would like to use Keras to embed this information in some multi-dimensional space and then display it in 3D using Tensorflow Projector.
I imagine I need a single Dense layer with number of outputs equal to the number of dimensions of the embedding.
I know there is a loss function for this implemented in tensorflow_addons called tfa.losses.TripletSemiHardLoss().
I read about this loss function and I cannot understand what the shape of the inputs of my dense layer should be.
At first, I thought I will need to have three one-hot-encoded vectors of 4000 elements,  i.e. if A is more similar to B than to C to have someting like:
[1, 0, 0 ... 0],  [0, 1, 0 ... 0], [0, 0, 1 ... 0]
Clearly that's not the case, but I can't wrap my mind what it should be.
I need some help to get me started.


texts: What is the difference between Latent and Explicit Semantic Analysis

texts:
I'm not quite sure what "latent" refers to in this context.

In Computing Semantic Relatedness using
Wikipedia-based Explicit Semantic Analysis they say

''Our semantic analysis is explicit in the sense that we manipulate
 manifest concepts grounded in human cognition, rather than
 'latent concepts' used by Latent Semantic Analysis''.

What does that mean in simple terms?

I had a look at the related Wikipedia articles, ( Latent (wikipedia.org/wiki/Latent_semantic_analysis), Explicit wikipedia.org/wiki/Explicit_semantic_analysis) ), and I wasn't able to make heads or tails of it.

Perhaps someone with a better appreciate of the nuance involved here might be able to provide me with a clear indication of the similarities and differences, the pros and cons between these two methods for accessing document/text fragment relatedness to a particular concept.


texts: Using tensorflow-gpu 2.0.0rc0. I want to choose whether it uses the GPU or the CPU.


texts: How to design a LSTM network with different number of input/output units?

texts:
I want to design a LSTM network with 25 input units(T_x=25), and 5 out units(T_y=5), but I don't know how to convert 25 y^s(y_hat), to 5 y^s?

For example the following picture shows a LSTM network with equal input and outputs(equal to Tx). But what if I have 25 inputs and need 5 outputs not 25!

Should I add the output of the first unit(y<1>) to the input of the second unit(x<2>)? Or maybe adding it to the activation or memory of the second unit(a or c)? Or I must forget the 20 first outputs(y<1> to y<20>) and only use the last 5 outputs(y<21> to y<25>)?


texts: Trying to train Stylegan, stopping with errors after first tick

texts:
I'm just getting started with GANs and have prepared a dataset for Stylegan of around 5500 256x256 images to train it on. I've pored through the scant resources outlining the training process and have all of the software set up, using pretty much default settings for the training.py and training_loop.py files aside from specifying GPU number and dataset paths.

When I run the train.py script it runs for a single tick every time before giving me this error about input depths vs filter depths. These don't seem to be settings in Stylegan so I don't really know how to troubleshoot the issue. I've tried changing many different params in the training files and it always fails after one tick. Do these errors look familiar to anyone? My dataset seems like it shouldn't cause any issues.

Traceback (most recent call last):
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/futureprojects/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.(0) Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3
 [[{{node InceptionV3/_Run/InceptionV3/InceptionV3/import/conv/Conv2D}}]]
 [[InceptionV3/_Run/InceptionV3/Reshape/_4879]]
(1) Invalid argument: input depth must be evenly divisible by filter depth: 1 vs 3
 [[{{node InceptionV3/_Run/InceptionV3/InceptionV3/import/conv/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

I haven't been able to make much sense of it from what I've read of Tensorflow's docs yet.


texts:
I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.

There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.

The data that I am processing look like this below all recorded in 15 minute intervals.

            Date_Time    kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00

The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)

EDIT rolling average plot

Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...

In python if I do a df.describe:

Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...

Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?

Any tips to try or avenues to march down is greatly appreciated :)

EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data


texts: How to do an automated SWOT analysis with Text Mining

texts: has somebody an idea/approach how to do text mining for a SWOT analysis? (e.g. sentiment analysis) I need to assign categories (Strengths, Weaknesses, Opportunities, Risks) to words in a document and then rank them.


texts: out of memory error when consrtucting 2d list from 2 numpy arrays

texts:
I am working on lung CT images from luna16 dataset, the dataset have a 3d lung image and a label from CSV file, I have a code for constructing 2d list from 3d array 25x25x25 (the 3d image) and a label [0,1] or [1,0] from CSV file, after creating the 2d list I want to save it in numpy file, below is my code for creating the 2d list and saving it in numpy file:

def getIDlist(csv_Dist,Data_Dist):
    # receive marked coords and ID in annotations.csv, and return the distination with coords.
    print('loading')
    data = np.loadtxt(csv_Dist, delimiter = ',', dtype = 'str')
    # delete the header file via 1:0, and receive the ID, x, y, z, r via 0:5 to a list.
    ID_coords = data[1:,0:5][0:10000] # get list of 'seriesuid' 'coordX' 'coordY' 'coordZ' 'class' (without header).
    # define the output file.
    ID_dist = []

    print('strat finding')
    process_bar = ShowProcess(len(ID_coords))

    for ID,x,y,z,label in ID_coords:
        ID = ID +'.mhd'
        found = 0
        for parent, dirnames, filenames in os.walk(Data_Dist):
            for filename in filenames:# loop inside all files
                if ID == filename: # ID + .mhd in csv equal to filename in files
                    process_bar.show_process()
                    ID = parent + '\\' + ID# ID gets full path of the founded file
                    ID_dist.append([ID,x,y,z,label])# ID_dist gets info of founded files
                    found = 1
                    #print("found: ", found)
                    break
            if found == 1:
                break
        if found == 1:
            continue

    process_bar.close()
    return ID_dist

def get3Dmatrix(ID_dist):

    print('preparing the 3d matrix')
    matrixlist = []
    for Dist, xcoords, ycoords, zcoords, label in tqdm(ID_dist):
        # read the image
        imagearray,origin,spacing = load_itk_image(Dist)
        # resample in to 1mm*1mm*1mm
        imagearray = resample(imagearray,spacing,(1,1,1))

        # transfer world coordinates to voxel-coordinates, divide new spacing 1mm
        z = int(round((float(zcoords)-float(origin[0]))/1))
        y = int(round((float(ycoords)-float(origin[1]))/1))
        x = int(round((float(xcoords)-float(origin[2]))/1))

        # get the 3D array with shape 25*25*25
        imagearray = imagearray[z-13:z+12,y-13:y+12,x-13:x+12]

        #converting the label number into a one-hot-encoding
        if int(label) == 1:
            label=np.array([0,1])
        elif int(label) == 0:
            label=np.array([1,0])

        # put it into output file
        matrixlist.append([imagearray,label])# 2d list consist of 3d array + label of all cases.
    return matrixlist

 def main():
    start_time = time.time()
    # get ID_list from the csv and data dist.
    ID_list = getIDlist(candidates_V2_Dist, Data_Dist)# nested list - get file name with dist + x,y,z,class
    # Data_set[i][0] is the 3D array, Data_set[i][1] is the label
    Data_set = get3Dmatrix(ID_list) # 2d list consist of 3d array + label of all cases.
    print("Begin saving in numpy file")
    np.save(output_path+'np_ds(10000)-25-25-25(zyx)_one_hot.npy', Data_set)
    print("%s time takes in seconds" % (time.time() - start_time))

if __name__ == "__main__":
    main()

my problem is:

1- with approximatly 550 samples, the RAM gets fulled and I get memory error, I am working on dell inspiron core i7 with 16 gb ram laptop.

2- it takes 34 seconds for creating each sample, and I see this is huge amount of time for only one sample.

I did a lot of search in google and asked a question in some other forums but didn't get any answer, can anyone help me, please? realy I am confused with that error.
image below is the error message:


texts: Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

texts:
I was wondering if my methodology makes sense. I am using GridSearchCV with cross-validation to train and tune model hyperparameters for a bunch of different model types (e.g. Regression Trees, Ridge, Elastic net, etc.). Before fitting the models I leave out 10% of the sample for model validation using train_test_split. (see Screenshot). I select the models with the best parameters to make predictions on the unseen validation set.

Am I missing something, as I haven't seen someone doing this when evaluating model accuracy while tuning for model parameters?


texts: Should I update my regularisation L1 and L2 regularisation parameters in online setting?

texts:
I have been working on online learning for a few weeks now, especially with Vowpal Wabbit and logistic regression. My understanding of the online learning algorithms and the problem is alright but I can't get my head straight about the regularisation issue.

In a standard machine learning problem, one uses a validation dataset in order to tune regularisations parameters for L1 and L2. But how do you choose those regularisation parameters in an online setting? Do you just fix them from start, or should I update them while training occurs?


texts: Should we also include negative instance in cross-validation process of one-class classifiers?

texts:
For a one-class classifier to do text classification, only positive instances are used for training.
However, in the cross-validation process to select the best hyperparameters, should we also include negative instances to evaluate the classifier? or use only positive instances? What's the tradition?


texts: generate graph from .eps file (preferably using R)

texts:
Using the R package 'Deducer', I saved a graphic (chart) as an .eps file.

I can open the .eps file, it's just a bunch of text.

How do I re-generate the graphic (chart) from the .eps file (preferably using Deducer or R)?


texts: How can be proved that the softmax output forms a probability distribution and the sigmoid output does not?

texts:
I was reading Nielsen's book and in this part of chapter 3 about the softmax function, he says, just before the following Excercise, that the output of a neural network with a output softmax layers forms a probability distribution and the sigmoid output does not always forms it.
Now I've been wondering about the output of a neural network, if I have a sigmoid output layer, say for one observation the output is 0.7 for class 0, should the probability for class 1 be 0.3? Or, in this binary classification example, using a softmax output, the first output neuron would be 0.7 for class 0 and 0.3 for class 1 in that particular observation?


texts: weight sharing among neurons at same depth

texts:
I'm trying to understand some visual illustrations about the wight sharing in the Convolutional Neural Network as following:

In this picture we see that for different outputs different inputs share the same weights, as the weights are associated with the color (same color - same weight).
So in the Stanford example we can see it eather:

As for example, for the o(0,0,0)=5 and for o(0,1,0)=7 both we used the w0[0],w01,w2 weights,
and it fits the picture above.
Now there is this picture:

And I'm not fully understand the 3 estimation. It says that the same color means the same weight,
but as I can tell in the picture we see a filter, each neuron, let's mark it as $f$ refers to the  multiplication of $f(x)=wx(+b?)$, and as much as I can tell from the Stanford exmaple, the weight can be all different and not the same color, and the weight sharing isn't representable on the filter, but only in any view that includes the input, as in the first image, so isn't it a bit misleading ?
Or maybe I just didn't understand it and anyone could explain it more briefly ?


texts: How can I determine best performing groups in a series

texts:
Not sure if this is right place for this question, but taking an example of the following trend, how can I determine that groups A and B are the areas where the result has been best?

I have already tried by doing an average and finding all those above the average. But finding out a group and driving a conclusion is tricky part. How can I do this?


texts: Matrix based Visualization Meaning - Assocation Rules

texts:
I didn't found any good resource that explains me very well what type of information I can extract from a Matrix based Visualization like this:

I do't uderstand what RHS and LHS represents

Anyone can explain me the meaning of this chart?

Many thanks!


texts: Combining Linear Regression and Time Series

texts:
Does anyone know of a predictive model that can combine the linear regression model and time series model?

I have some data about some products. The data has two parts, some attributes about the products themselves, and some time series data of month-by-month sales amounts for each product. We want to predict future month-by-month sales amount for each product. with the products attributes data, we can build a regression model, or an svm model, to predict sales amounts. With the time series sales amount, we can build a time series model to predict sales amount, but is there any predictive model that can combine the regression, svm, and time series models to do the prediction?


texts: What is the generalization of binary/boolean matrix factorization to fuzzy logics called?

texts:
Given a matrix of boolean values $\mathbf{X} \in \mathbb{B}^{M \times N} = \{\top, \bot\}^{M \times N}$, the binary/boolean matrix factorization (BMF) problem is to find $\mathbf{U} \in \mathbb{B}^{M \times K}$ and $\mathbf{V} \in \mathbb{B}^{K \times N}$ for some fixed $K$ that minimize $\sum_{i, j} d(x_{ij}, \hat{x}_{ij})$, where $\hat{x}_{ij} = \bigvee_k u_{ik} \land v_{kj}$ and $d$ is some boolean metric.

BMF can be generalized to t-norm fuzzy logics (with involutive negation) by replacing $\mathbb{B}$ with the closed unit interval $[0, 1]$, $\land$ with a t-norm $*$, and $\lor$ with the t-conorm that is dual to $*$ assuming the canonical negator $1 - x$. The objective can similarly be restated in terms of minimizing a unit metric, rather than a boolean metric.

I have found it difficult to locate literature on this generalization of BMF because I do not know how it is commonly referred to. Any pointers to theoretical or experimental results on the topic are appreciated.


texts: Remove all columns where the entire column is null

texts:
I have a very dirty csv where there are several columns with only null values.

I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal to the number of rows.

clean_df = bucketed_df.select([c for c in bucketed_df.columns if count(when(isnull(c), c)) not bucketed_df.count()])

However, I get this error:

SyntaxError: invalid syntax
  File "<command-2213215314329625>", line 1
    clean_df = bucketed_df.select([c for c in bucketed_df.columns if count(when(isnull(c), c)) not bucketed_df.count()])
                                                                                ^
SyntaxError: invalid syntax

If anyone could help me get rid of these dirty columns, that would be great.


texts:
I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to develop a model to detect these anomalies.
My problem is that the anomaly data looks like normal data, which means they are not outliers or has a certain pattern. If I plot the normal data over anomaly data they are in the same distribution.
I tried several anomaly detection approaches:

Multivariate Gaussian Distribution Approach to identify anomalies

I tried to create new features that anomaly data will be outliers and then I can use Multivariate Gaussian Distribution Approach, but could not able to find any combination to isolate the anomalies.

I guess there is no point of using a classification algorithm since dataset is highly imbalanced.

I tried OneClassSVM, DecisionTree, RandomForest but AUC is 0.5 (as good as random).

How to implement a model for this kind of scenario?
Other methods which I can think about:

Develop a NN with AutoEncoders
Try Generate Synthetic Samples and resample the dataset


texts: Calculating an estimate of KL Divergence using the samples drawn from distributions

texts:
Given two sets of samples drawn from two different distributions, is it computationally possible to get an estimate of KL-Divergence between the two distribution using these samples?
Here I am assuming the dimensionality of the two distributions is high (say d). To compute the estimate, we first need to discretize the entire space and then estimate probabilities based on the frequencies. Let us say, we discretize each dimension into p bins. Then the total number of grids in the space will be $p^d$. So we need to compute the probabilities of the two distributions for $p^d$ grids, which is exponential in time. Hence I assume we cannot compute an estimate of KL Divergence using the samples for any practical problem.
I wanted to check if this explanation is correct or if I am missing anything. Could someone assert if this rationale is correct?


texts: Python Seaborn: how are error bars computed in barplots?

texts:
I'm using seaborn library to generate bar plots in python. I'm wondering what statistics are used to compute the error bars, but can't find any reference to this in the seaborn's barplot documentation.

I know the bar values are computed based on mean in my case (the default option), and I assume the error bars are computed based on a Normal distribution 95% confidence interval, but I'd like to be sure.


texts:
I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing,
I would like to predict errors or failures but till now they have not been recorded yet. So I thought to using these time series to understand if there are abnormal behaviours (which could lead to a faillure), maybe clustering time series. Beacuse I suppose that the machine, in normal condition, has the same parameters for each work.
Is there a procedure/documentation/ best practises in order to do anomaly detecion with time series?
Thanks a lot!


texts: How is the backbone of two neural networks trained?

texts:
Suppose, I have a backbone network(convolutional neural network). After this network ends, the output is fed into two neural networks. Both building on the outputs of the feature extractor(CNN). Now if I want to train this complete network from scratch on two different tasks, the weights of the layers after the backbone network can be updated easily, but how should I update the weights of the backbone network. I mean I can compute gradients with respect to two losses, shall I take the mean for the gradients in the backbone or it has to be some weighted sum? if it is the weighted sum then how would the parameters of the weighted sum be updated?
Thanks


texts: Identifying hierarchical structures from user submitted tags

texts:
I am working with a dataset of shops, where users have submitted some number (up to 5) of "tags" for each shop, which is supposed to describe an aspect of the shop.
From the tags, I want to create a hierarchical structure categorizing each attribute, something of a "phylogenic tree" structure as described in this article (How Quizlet uses hierarchical classification to label content with academic subjects).
e.g. From this data:

shop_id
tags

1
[Fast Food, Fried Chicken, Western]

2
[Fast Food, Chinese]

3
[Western, Pasta, Sit-down]

etc.
etc.

I would want to create a hierarchical structure of

category 1
category 2
category 3

Fast Food
Western
Fried Chicken

Fast Food
Chinese

Sit-down
Western
Pasta

Would love some suggestion on how to approach this problem!


texts:
I have an idea for a side-project: I'd like to be able to start with an image of a Scottish kilt, and automatically determine what tartan is used. For example, this is my (MacGill) tartan.

Ideally, I'd like to be able to start with any picture of people wearing kilts, find all the kilts, and determine the tartan for each. However, I imagine that it would make things considerably easier if the image used is a square of tartan like the MacGill example above, nicely lined up so that the pattern forms horizontal and vertical lines. That way I wouldn't have to worry about tilt, skew, lighting, motion-blur, fabric grain, and all the other real-world factors that would complicate things.
I'm a seasoned software developer, but have no data-science expertise. At all. I see this as an opportunity to learn. However, the field is huge, and I would assume that only 1% of the techniques, tools, languages, etc. that I see mentioned on this site are relevant to this domain. I'd like some advice on how to narrow my search.
And, I suppose, first and foremost I'd appreciate advice on whether this is even achievable.
Any pointers gratefully received.


texts: Why do trained RL agents still display stochastic "exploratory" behavior on testing data?

texts:
I am training a PPO2 RL model using stable baselines. One thing I found is that a trained agent will still display some stochastic behavior on test data, as shown by the predict method of PPO2; the deterministic flag should be set to True to see deterministic (un-stochastic) behavior.

I understand that when training an RL agent, there is some amount of random exploration so  that an agent can learn the optimal value-policy network. However, I thought that once an agent was trained, and used to act (predict) on new test data, the model parameters would be used and no exploration would take place. Therefore, when testing, I thought the agents actions and subsequent rewards would be deterministic (once the test data is static).

Why is the trained agent exploring on test data / Why is the trained agent still stochastic when predicting test data? To what degree is the trained agent stochastic (will it follow its model predictions 90% of the time and guess the other 10%)?


texts: Which one to choose to identify patterns of user activity: Sequence analysis or process mining?

texts:
I have the following user activity data that where for each user the activity type they were engaged are recorded along with the phase:
User |  Phase |  ActivityType  |  Date
321     1        A                12/20/2020 15:00
321     1        B                12/20/2020 16:00
321     2        A                12/21/2020 12:00
321     1        C                12/21/2020 13:00
321     3        B                12/22/2020 11:00
322     1        A                12/20/2020 15:00
322     1        A                12/20/2020 16:00
322     2        B                12/21/2020 12:00
322     1        C                12/21/2020 13:00
322     3        D                12/22/2020 11:00

For each user, I also have the satisfaction score about the application.
User | Satisfaction
321    90
321    60

What I want to see is if there are any emerging groups of users with a certain pattern of activities. Then, I want to compare the satisfaction scores across these groups to check if the specific pattern of activities yield higher satisfaction or not.
To perform this analysis, I identified two approaches: process mining (with PM4PY, python library), and sequence analysis (with TraMineR, an R library).
However I am not sure which one would fit better to my needs. I am a total beginner in both areas. Any insights to help me make a good decision here?


texts: How to combine two time dependent datasets?

texts:
I'm very new to data science so please be gentle.
I have a dataset that contains record of occurrence of fire for the past 35 years (+-700.000 rows). Each date and time can have more than one occurrence as two fires can happen at the same time in different locations. It's features are approximately as follows:
Date and Time | Borough | County | Area Burned | Type | Cause | etc.

I'm planning to combine this with a new dataset that I'm in the process of constructing. This new dataset would have the following features:
Date and Time | Latitude | Longitude | Av. Temperature | Av. Wind Speed | Av, Humidity | etc.

My idea is to try to predict the likelihood of a fire occurring and it's potential severity from a given set of atmospheric parameters. But I'm a bit unsure what would be the best way to combine them.
If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way? The problem I see is that the locations where there was no fire would not be represented, so I'd have to add a bunch of blank rows for all locations where there was no fire to balance the set.
Any ideas on the best strategy for this?


texts: What can I infer from a linear correlation of regression coefficients?

texts:
I am working on a dataset for classification where each observation is a series of values of a certain measurement $Y$ for a fixed range of values of measurement $X$ (i.e. a discrete mapping from $A \subseteq X$ to $Y$). (I'm not allowed to reveal what $X$ and $Y$ are due to confidentiality issues.) I am doing some single-variable regression on this relationship to obtain a smooth $X$ $\to$ $Y$ function for each observation. The basis functions I'm using for this are
$$\left\{x\mapsto 1,\ x \mapsto \frac{1}{\sqrt{x}},\ x\mapsto e^{-x \cdot 10^{-3}}\right\}$$
This is done by applying each of these functions to the range of $X$ values; then this transformed feature set is passed to a standard linear regressor. (The $10^{-3}$ factor in the exponential is to avoid numerical issues; if I just use $e^{-x}$ the coefficient for it becomes 0 due to the $e^{-x}$ features vanishing to 0 for large values of $x$.) This essentially amounts to finding the best-fitting linear combination of the basis functions.
It seems like this choice of basis functions yields curves (dashed) that fit pretty well to the original data (solid):

(Again, I'm not labelling the axes due to confidentiality issues.)
When I went to inspect how well the regression coefficients separated instances of each class, I found this:

The coefficient of 1 (that is, the independent term) and that of $\frac{1}{\sqrt{x}}$ seem to be linearly correlated. This lead me to think that perhaps one of the two basis functions was redundant. However, when I removed either, the regression performed worse. For example, removing the independent term:

So it is not necessarily true that one of these two basis functions are redundant. My question is: if the values of a subset of the regression coefficients seem to be linearly correlated, what can I infer from it, if anything?


texts:
Imagine situation, input is some text file and information are spread according to rows. I want to use rows as input of model.

Model is planned as LSTM with softmax as output layer. My problem is the output; I want to classify text file to some categories. So I imagine this like, I open text file, do necessary preprocessing and row by row feed into the model and after I have applied all the rows to the file, I tell softmax which category is right for this document. But I don't know how to make softmax layer "wait" until last row of file is processed by LSTM layers.

For example, I have file with 1000 rows and every row has 5 features. So my dataset is [1000x5] size but I don't have 1000 y I have 1 y which told me category of this document. Documents can be different size so I can't use whole document as input. I also don't see as option make y size 1000 where everything is category index.

When I think about this seems to me, when person implement LSTM from scratch, is possible to make LSTM run(save every state) and after several iterations perform softmax and than apply BTT or more suitable algorithm and perform weights update.

Every LSTM keras tutorial which I saw uses y which is the same shape as number of samples. Maybe I didn't see right ones.


texts: Highly accurate convnets appear to have random-looking visualized weights?

texts:
I'm building a TensorFlow convoluted neural network that isn't getting the accuracy that I hoped for. So I figured I would visualize the learned weights to see where the network might be stumbling. As a benchmark I started visualizing the weights for a totally different project, Google's MNIST convnet example, which has very high accuracy (99.2%).

I assumed that a very accurate model would have intuitive weights, but in reality I got weights that looked completely random. Other people seem to get similarly random results. See martin-gorner's comment with visuals on Mar 18. His are results are similar to mine.

More Googling shows that other people see essentially random behavior for non-MNIST datasets. Is this common? If so, it would seem that visualizing weights is a fruitless exercise unlikely to bring any helpful insights to the modeler. Is this true? If not, why do so many people use this visual (clear edges in the first layer, and composite images in subsequent layers) when describing how convnets work? Stanford's course material also seems to at least show edges (you have to scroll down about half of the page) when visualizing the weights. I'm not sure what to think.


texts: How to save multi-output predicted masks into two different folders after using model.predict_generator

texts:
I have a multi output segmentation task, the training process went well, but when Im trying to get the prediction I found difficulties to separate the two output into two different folders, in my dataset I have one image and two relating masks, I used this test generator:

def testGenerator(test_path= "data/membrane/test/image",num_image = 1584,target_size = (224,224),flag_multi_class = False,as_gray = False):
  for i in range(num_image):
    img = io.imread(os.path.join(test_path,"%d.jpg"%i),as_gray = as_gray)
    img = img / 255.
    img = trans.resize(img,target_size)
    img = np.reshape(img,img.shape) if (not flag_multi_class) else img
    img = np.reshape(img,(1,)+img.shape)
    yield img

and this is the code for model.test_generator:

testGene = testGenerator("data/membrane/test/image")
model = new_model()
model.load_weights("new_model.hdf5")
(conv11, convc11) = model.predict_generator(testGene,10,verbose=1)

saveResult(conv11,"data/membrane/test/results/road")
saveResult(convc11,"data/membrane/test/results/cl")

for visualization I used this :

def labelVisualize(num_class,color_dict,img):
    img = img[:,:,0] if len(img.shape) == 3 else img
    img_out = np.zeros(img.shape + (3,))
    for i in range(num_class):
    img_out[img == i,:] = color_dict[i]
    return img_out / 255

def saveResult(save_path,npyfile,flag_multi_class = False,num_class = 2):
    for i,item in enumerate(npyfile):
    img = labelVisualize(num_class,COLOR_DICT,item) if flag_multi_class else item[:,:,0]
    io.imsave(os.path.join(save_path,"%d_predict.tif"%(i)),skimage.img_as_ubyte(img))


texts: How does reweighting samples actually affect each new weak learner in adaboost?

texts:
In Adaboost, when you reweight the samples, how does the training process for the next weak learner in the boosting algorithm take in to account the weights? Is it reflected in the loss function of the next weak learner? The ESL book doesn't really talk about this, and I was a bit confused on how the weights manifest itself in the actual learning process for the next model to be added.
In addition, if say we are using trees as the weak learners, how is each subsequent tree determined? Is there a preset random selection of variables to be considered for partitioning like in random forest?


texts: How does action get selected in a Policy Gradient Method?

texts:
As I understood, in Reinforcement-Learning a big difference between a Value-based method and a Policy-gradient method is how the next action is selected.

In Q-learning (Value-based method), each possible action gets a score. We then select next action that has a highest score, with a slight possibility to select any other action at random.

How does a policy gradient method decide upon an action? We are no longer allowed to select the highest-scored action, as this will leave us with a non-differentiable policy, that can't be improved.  The actions still have scores - do we select them at random ...using the "cummulative distribution function"? Also, am I using the right term?

In other words, do actions that have a higher score occupy more space and are more likely to be selected?

Edit: there is a really awesome lecture about policy grads. Unfortunatelly the video is deliberately unlisted, so that normal people can't really get to it - but I am for free education, so here it is:  CS294-112 9/6/17


texts: Neural Nets: Backpropogation from Conjugate Gradient Descent

texts:
In neural nets, the back-propagation algorithm is 'based on' gradient descent, optimizing over a cost function $C(w,b)$ of the weights and biases of the network.

Conjugate gradient descent has way faster convergence than standard gradient descent (and stochastic gradient descent). Is there a similar back-propagation algorithm based off of conjugate gradient descent?


texts: Updating the weights of the filters in a CNN

texts:
I am currently trying to understand the architecture of a CNN. I understand the convolution, the ReLU layer, pooling layer, and fully connected layer. However, I am still confused about the weights.

In a normal neural network, each neuron has its own weight. In the fully connected layer, each neuron would also have its own weight. But what I dont know is if each filter has its own weight. Do I just have to update the weights in the fully connected layer during back propagation? Or do the filters all have a separate weight that I need to update?


texts:
I barely know about Data Analysis tools and techniques, so bare with me if I'm asking something too trivial.

I'm looking for a Sentiment Analysis tool to process comments in Spanish. I do know some options for Sentiment analysis but those all work for English.

Is there a model/tool that already works with Spanish?

I'm language agnostic so it does not matter if it's a Java, Python or even Go code.


texts: Program to fine-tune pre-trained word embeddings on my data set

texts: I am looking for a program that would allow me to fine-tune pre-trained word embeddings on my data set. Ideally, open source and working on Linux or Windows.


texts:
Im a 26 year old guy with an MBA and I work as an ERP system administrator.
I have been interested in the field of data science for a while now. I've always liked statistics and various analytical tasks.

I would really like to try and work within this field, but it feels a bit overwhelming. People mention that you have to learn Python, R, SQL, Machine Learning, advanced algebra, data modeling, big data (hadoop etc), predictive analytics, various business intelligence tools, VBA, Matlab etc etc.

As of today, I have some SQL knowledge and have a general understanding of BI, big data, good excel skills etc. I am willing to learn some of the aforementioned areas, but I don't have the money or time to go back to full time university studies again.

So here is my question: is there any recognized "light version" of data scientist on the market? What are they usually called? What skills do they need to master? What should I learn in order to work with big data sets and analytics without having to study full time for another 5 years?

I live in Scandinavia so the job market is probably different here, but I thought it would be interesting to hear some answers.


texts: LSTM Neural Network gets stuck in a specific state when trying to predict new states over many time periods

texts:
I have built an LSTM neural network for category, or latent state, prediction.
The data is more or less of the form:

x1 = continuos number from current record
x2 = continuous number from current record
x3 = continuous number from current record
x4 = state value for the current record

Only one one-dimensional variable to be predicted.

y1 = state value for the next record

Fitted with an LSTM model of the form below:
model_lstm = tf.keras.Sequential()
model_lstm.add(tf.keras.layers.LSTM(10, return_sequences = True, input_shape = (xtrain.shape[1],xtrain.shape[2])))

model_lstm.add(tf.keras.layers.Dense(units=5, activation = "softmax", input_shape = (xtrain.shape[1],xtrain.shape[2])))

loss_fn =  tf.keras.losses.SparseCategoricalCrossentropy()
opt = tf.keras.optimizers.Adam(learning_rate=.001)
model_lstm.compile(loss=loss_fn, optimizer=opt, metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])

Fitting an LSTM model of say this form makes it fit almost perfectly on training and validation data when presented with an individual sequence:

[x1_t1, x2_t1, x3_t1, x4_t1] = [y1_t1]
[x1_t2, x2_t2, x3_t2, x4_t2] = [y1_t2]
.
.
.
[x1_tn, x2_tn, x3_tn, x4_tn] = [y1_tn]

however, it fails when simulating, a sort of walk where the predictions are used directly into the next inputs...

[x1_t1, x2_t1, x3_t1, x4_t1] = [y1_t1]
[x1_t2, x2_t2, x3_t2, y1_t1] = [y1_t2]
.
.
.
[x1_tn, x2_tn, x3_tn, y1_t(n-1)] = [y1_tn]

In fact it simply gets stuck in a state and doesn't move. Is there any solution to this? I am impressed with the accuracy but obviously it's not very useful unless a person only wanted one prediction at a time that he/she can immediately correct per time step output.


texts: The data in our relational DBMS is getting big, is it the time to move to NoSQL?

texts: We created a social network application for eLearning purposes. It's an experimental project that we are researching on in our lab. It has been used in some case studies for a while and the data in our relational DBMS (SQL Server 2008) is getting big. It's a few gigabytes now and the tables are highly connected to each other. The performance is still fine, but when should we consider other options? Is it the matter of performance?


texts:
I got interested in autocompletion using deep learning and tutorials that I found where conditioned always on specific number of characters (given 40 characters predict the next character or the whole complete word). But in real world the autocomplete is done after entering a first letter already.

I was checking out how autocompletion works in Whatsapp I had following observations:
 #1. Autocomplete starts working already after a first letter.
 #2. Before I would enter a space, Whatsapp suggests the continuation of a word, example: entering 'L' would result in suggesting 'L', 'Like', 'Last'.
 #3. If I enter a space after a word, autocomplete would suggest me whole next word, example: entering 'Last ' would result in suggesting 'time', 'night', 'week'.
 #4. If I enter 'I am ru' it will suggest 'running', 'run'. (Pay attention: grammatically, run is wrong)

One could use RNNs/LSTMs with one-hot encoding or word-embedding to do such things.

However, I wondered what the train dataset should look like:

is it char-based model with different n-grams words, example:
1.1 char-based 1-gram ('l', 'i', 'k', 'e') word would cover the cases #1 and #2 because I would be able to sample the next character.
1.2. char-based with 2-gram word ('l', 'i', 'k', 'e', ' ', 's','u','n') would cover #3 as I would be able to predict the whole complete next word (character-based) and it also preserves the dependence on the previous word.

By using 1.1 and 1.2 I could cover cases #1, #2 and #3 and model 2-words dependence.

is it word-based model with different n-grams, example:
2.1 word-based with 1 grams, eg. "like sun" -> sequence: "like ", predict: "sun". This would allow me to model #3
2.2  word-based with 2 grams, eg. "like sun and" -> sequence: "like sun ", predict: "and". This would allow me to model a bit longer dependence.

By using 1.1 and 2.1 I would cover cases #1, #2, #3.

Questions:
I wonder what would be better to do: use 1.1 and 1.2 or 1.1 and 2.1.

Also, let me know if my way of thinking is correct or if I am completely wrong somewhere or if I miss something or if you think there are some easier way of doing it. Would be glad about any suggestions.


texts: Test predictive power of classes in multiclass classification

texts: How can we confidently identify if the data has lower predictive power. I saw that the classifiers are giving poor accuracy (< 50%) on test data and approx 97% on train data. Correlation of a class with itself is less than other classes. KS 2 tailed test shows similar kinds of values for classes with KS statistic O(0.3 or 0.4). What more I can do to verify if the data has good predictive power or not


texts: How can I use octave function minimizers in c++?

texts: I have downloaded and built octave library and it works fine . But I cannot call function minimizers like fminunc() , fmingc() etc to minimize my functions for performing logistic regression or using it in neural networks . Can these functions be accessed from c++ ? If yes, then how ?


texts: Matrix multiplication issue (shapes not alligned)

texts:
I am building an RNN using numpy only and have started on the forward propagation section. However i am having some issues aligning my matrices. The issue is on this line:

h = np.dot(u, x) + np.dot(aprev, w) + bh

More specifically, the problem is with this part:

np.dot(u, x)

I tried playing around with it by transposing different parts but I still receive the error:

ValueError: shapes (8,8) and (4,8) not aligned: 8 (dim 1) != 4 (dim 0)

How can i overcome this? The entirety of my code is pasted below, and note that at the beginning i transposed x, to make data entry and inserting data into its corresponding timestep easy. '.T' can be easy to miss sometimes.

import numpy as np

# Data Processing

x = np.array([
    # t/no. of inputs
    [1, 2, 3, 4, 5, 6],
    [7, 8, 9, 10, 11, 12],     # Samples
    [13, 14, 15, 16, 17, 18],
    [19, 20, 21, 22, 23, 24]]).T

# Model Parameters

numInputs = x.shape[1]
timeSteps = x.shape[0]
numNeurons = 8

u = np.random.random((numNeurons, numInputs))
v = np.random.random((numInputs, numNeurons))
w = np.random.random((numNeurons, numNeurons))
bh = np.zeros((numNeurons, 1))
bo = np.zeros((numInputs, 1))
aprev = 0

# Training

def fprop(x, aprev, u, v, w, bh, bo):
    h = np.dot(u, x) + np.dot(aprev, w) + bh
    a = np.tanh(h)
    o = np.dot(a, v) + bo
    yhat = (np.exp(o))/(np.sum(np.exp(o)))
    aprev = a

for t in x:
    fprop(t, aprev, u, v, w, bh, bo)


texts:
can anyone help me in finding website or blogs for best data science use case in Supplier payment

Thank you in advance


texts: Best way to implement 10-fold cross validation for CNN model

texts:
I was performing a binary classification problem with 15000 RGB images using a scratch build CNN model. While it comes to evaluate the model, I can do it in two ways:

Splitting data Train and Test and use 10 fold cross-validation for the training data. Later with the best model, I would use the unseen Test data. In this way I got appx. 91.5% avg. accuracy for both test and validation.
Just use 10 fold cross-validation and got 92.5% avg accuracy(slightly better result than the previous one.)

Which option would be the best for reporting the performance of my model in the research article?
TIA


texts:
I just installed tensorflow environment with conda.
but problem is conda does not have some packagefor example 'fastai'
pip3 only has this package
as you know pip install and conda install has different package directory
so if conda does not have the package what you have to do?
 I dont want to move pip package to conda manually....


texts: Is a BiLSTM layer required if we use BERT?

texts:
I am new to Deep learning based NLP and I have a doubt -
I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a language model that scans the contexts in both the directions and embeds words according to the context. Now my question is - if context is captured during word embedding with BERT, why do we need another layer of BiLSTM?


texts: How do I get similarity with autoencoders

texts:
I have build an autoencoder to extract from a very high dimensional (200 dimensions) space a smaller but significant representation (16 dimensions).

Now that I have these "encoded" vectors, I would like to compute some kind of similarity score, or clustering.

I am not sure which notion of distance to apply at this point. Any ideas how I can get similarity/clusters considering that I have used autoencoders?


texts: Association rules: Find the recipe for a list of ingredients

texts:
Assume I have big database of recipes. For each recipe I have a list of ingredients.
Now I want to find all association rules in the form of (ingredient₁, ingriedient₂, …) → recipe.

Is the Apriori algorithm suitable for my problem? As far as I am able to understand the Apriori algorithm is intended to find rules like X → Y where X and Y are subsets of the same superset. But in my case, X and Y are subsets of completely disjoint supersets.
Is it possible to fit the Apriori algorithm for my problem? Are there other algorithms that suit my problem better? Or am I completely on the wrong track? I would be very grateful for any tips or hints.


texts: Comparing English word pronunciation complexity

texts:
I'm trying to figure out a way to compute a score for the pronunciation of a given english word, so I can use that score to compare the pronunciation complexity between english words.

Eg: Given words apple and banana, determine which one is harder to pronounce based on that score.

Please share your thoughts.
Thanks.


texts: Train Accuracy vs Test Accuracy vs Confusion matrix

texts:
After I developed my predictive model using Random Forest I get the following metrics:

        Train Accuracy ::  0.9764634601043997
        Test Accuracy  ::  0.7933284397683713
         Confusion matrix  [[28292  1474]
                            [ 6128   889]]

This is the results from this code:

  training_features, test_features, training_target, test_target, = train_test_split(df.drop(['bad_loans'], axis=1),
                                                  df['target'],
                                                  test_size = .3,
                                                  random_state=12)
clf = RandomForestClassifier()
trained_model = clf.fit(training_features, training_target)
trained_model.fit(training_features, training_target)
predictions = trained_model.predict(test_features)

Train Accuracy: accuracy_score(training_target, trained_model.predict(training_features))
Test Accuracy: accuracy_score(test_target, predictions)
Confusion Matrix: confusion_matrix(test_target, predictions)

However I'm getting a little confuse to interpret and explain this values.

What exactly this 3 measures tell me about my model?

Thanks!


texts: How to favour a particular class during classification using XGBoost?

texts:
I am using a simple XGBoost model to classify 2 classes (0 and 1) in a binary context. In case of the original data, the 0 is the majority class and 1 the minority class. The thing which is happening is that in case of classification, most 0s are being classified correctly, with many going into 1s, but most 1s are being misclassified into 0s.

I am fairly new to this, and having looked at various documentations and questions on SE, am really confused as to how I can specify my XGBoost model to favour class 1 (to be precise, if most 0s are misclassified into 1s, that is not a problem, but I want that most 1s are correctly classified as 1s (to increase the true positives, if there are false positives that is something which isn't much of a problem). The segment of code I am presently using to train and test the XGBoost are as follows (afterwards I use the confusion matrix in which the true positives (1s) are highly misclassified into 0s).

from xgboost import XGBClassifier

# fit model on training data
model = XGBClassifier()
model.fit(X_train, labels) # where labels are either 1s or 0s

# make predictions for test data
y_pred = model.predict(X_test)
y_pred = y_pred > 0.70 # account for > 0.70 probability
y_pred = y_pred.astype(int)

print(y_pred)

I just want to know if there is a simple way to specify to the XGBoost model any parameter in my code, so that the true positive rate can be increased? I can compromise of false positives being high, but I want the number of 1s to be correctly classified as 1s, instead of most of them going into 0s. Any help in this regard is appreciated.

UPDATE:

I have now tried to use scale_pos_weight in the XGBoost, with its value set to 0.70 (a random figure), but it is still landing most samples to 0, instead of 1.


texts: Finding linear transformation under which distance matrices are similar

texts:
I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ matrix) is approximately "the same" (i.e. has a low variance) among all sets $S_i$.

To illustrate, it might be the case that for each vector in each $S_i$, the first $\frac{k}2$ numbers are random noise, and the latter $\frac{k}2$ are the same for each $i$. In this case, $W$ would recover the last $\frac{k}2$ elements from each vector. But in practice the structure of the vectors may be more complex than in this toy example.

Is there a method to learn $W$ - either using direct linear algebra methods, or through learning (e.g. by optimizing a specific loss on the distance matrices), without finding the naive solution of $W$ being the zero mapping, or other mapping that maps all vectors to the same vector?

EDIT:

One possible approach I was considering is to apply a variant of Canonical correlation analysis (CCA) on the distance matrices. CCA projects two "views" of the same data to a shared subspace that maximizes the total correlation between the views; is there a variant of CCA in which the views are not pairs of vectors, but rather sets of vectors that each has some internal structure (as expressed by the distance matrices)? while in CCA we maximize the correlation between the pairs, here I'd like to maximize the correlation between the distance matrices of the different sets.

Thanks


texts: Can Orange read an excel file on Sharepoint or Onedrive?

texts: I would like to share my Orange file through a team SharePoint site, including its linked (excel) datafile. Is this possible?


texts:
I'm trying to cap outliers in a column of my pandas DataFrame.
Here's the boxplot for a column of my original data.

So, using code from this stackoverflow answer, I tried capping outliers.

Here's how capped column looks like, with new outliers after upper bound.

My problem now is, I can't keep capping outliers, yes! What do I do now?


texts: Best common metric for comparing classic time series forecasting methods (ARIMA/Prophet) with ML approach?

texts:
I am new to time series forecasting and looking to compare the performance of ARIMA/Prophet with an XGBoost model in predicting future stock market values based on historical stock market data and social media sentiment scores.
I am more familiar with ML, so would usually use an evaluation metric like R^2 to assess model performance for this sort of problem.
Are there any common evaluation metrics which forecasting methods like ARIMA/Prophet use to assess their accuracy, so I can do a like-for-like comparison with the XGBoost's prediction accuracy?


texts: Binary classification model with time series as variables

texts:
This is probably a simple question. Assume I'm interested in modelling a binary variable, with various covariates, including ones that are time series observations. In the usual modelling approach, one can try searching for various features from the timeseries data, such as standard deviations, averages, max and etc, to make a flat model matrix.

My question: what are the tools/approaches that allow for a (relatively) simple inclusion of time series data to a classification problem?

I don't think panel regression would work since the time series data is very different among the rows, sometimes is very sparse and asynchronous.  Melting the data, due to the structure, obviously wouldn't work too. Descriptive statistics is the easy way, but there should be something else?

I'm not experienced in working with neural networks, but maybe there's a NN approach that could find meaningful structures in the time series data?

I'm also thinking about clustering different time series based on their similarities and check for significance, but again, is there something robust to different length/sparseness of the time series?


texts: B.Tech Project for final year of College

texts:
I am an Engineering student in NSIT, Delhi, India. In our last year of B.Tech degree we have a B.Tech project, aka, BTP. I am in ECE(Electronics and Communication Engineering) dept.

Usually, my ECE batch mates would perform the project majorly regarding electronics, but I want to do in data Science or data manipulation under other departments(COE or IT). I approached a COE mentor under whose supervision I would like to do the project. But I have been struggling in choosing the right topic for project.

There are a lot of suggestion online, and many are preferably done by students who just want to complete a project instead of it been already done and executed by others. I want some decent approach suggestions for choosing the right project topics as I am quite confused right now.

  What student level data Science or, specifically, Data manipulation
  projects can I do in my final year of College?


texts: Deep Continious Clustering algorithm - just one output cluster

texts:
I use the DCC algorithm to cluster some data.
The whole algorithm is available here, but shortly it is:

construct mkNN graph of the data points (the connected components of it are the clusters).
pretrain an Autoencoder for data dimension-reduction.
train the Autoencoder witn it's objective wih another objective of the clustering which tries to reduce the distance between the dimension-reducted data points.
the final connected components of the becoming close points(if close enough (by a threshhold) they remain connected, if they moved far from each other - the edge is gone and some others appear and e.t.c) are the clusters.

full algorithm with link to the article

Could you possibly guess why am I getting the most of the data in the first 0-index cluster (consistently, with tuning all of the hyperparameters) ?
The data has good potential for a very good clustering as I can see from the tSNE visualization of the output. But the clustering isn't good. I thought maybe to enlarge the k parameter of the mKNN graph construction, but it didn't work for that, just reduces the whole amount of clusters, but the data samples continue to clash into one cluster.
Any suggestion and theoretical discussion is welcomed.


texts: How is the evaluation setup for YouTube faces of FaceNet?

texts:
The YouTube Faces database (YTF) consists of 3,425 videos of 1,595 different people. Given two videos, the task for YTF is to decide if they contain the same person or not. Having $n$ comparisons, the classifier might get $c \leq n$  right. Then the accuracy would be $\frac{c}{n}$.

FaceNet is a CNN which maps an image of a face on a unit sphere of $\mathbb{R}^{128}$. It was evaluated on YTF. How did they decide which person is in the video?

(I can imagine several procedures how this could be done, but I couldn't find it in the paper. One example, how it could be done, is by evaluating all images $x_i^{(k)}$ with $i = 1, \dots, \text{length of video }k$ and averaging the results - but I would like to know what they did / how this is usually done.)


texts: Can pytorch / keras support dataloader object of Image and Text?

texts:
Is there a way of creating a dataloader object, or the equivalent in Keras, where every observation is an image AND some text? I could create two models to do classification but I want to see if I can build it all in one to pull information from both to get the prediction.

Are there any examples of this someone can point me to? Or is it even possible?

The closest I've found is in fast.ai there's this article that combines tabular and text.


texts:
I read the article on Vanishing Gradient problem, which states that the problem can be rectified by using ReLu based activation function.

Now I am not able to understand that if using ReLu based activation function solves the problem, then why there are so many research papers suggesting the use of Newton's method based optimization algorithms for deep learning instead of Gradient Descent?
While reading research papers, I was having the strong impression that vanishing gradient problem was the core reason for such suggestions but now I am confused whether Newton's method is really needed if Gradient Descent can be modified to rectify all the problems faced during machine learning.


texts: Effect of a single variable in a linear model

texts:
In a multiple linear regression (MLR) model, people normally look at the relationship between each predictor and the response variable to see if it is linear. However, in my example below, I show that it can produce a totally different result from what you'll get out of a partial dependence plot. Am I right to assume that modelling the relationship between one variable and the response variable in an MLR should be avoided and we shall look at the partial independence plot to decide about the relationship?

The correlation coefficient also shows a negative direction, while the partial dependence plot shows a positive relationship between X1 and y. Which one shall be used to decide about the relationship between each predictor and the response?

import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.inspection import plot_partial_dependence
import matplotlib.pyplot as plt
X1 = np.array([1,2,3,7,5,4,5,9,8,7])
X2 = np.array([10,4,30,5,2,-1,0,-10,-20,-30])

y = 1*X1 + 3*X2

# Linear model using only X1
lin_model_1 = LinearRegression()
X1_df = pd.DataFrame(X1, columns=["X1"])
y_df = pd.DataFrame(y, columns=["y"])
print(y_df['y'].corr(X1_df['X1']))
lin_model_1.fit(X1_df, y)
print(lin_model_1.intercept_, lin_model_1.coef_)
y_pred_1 = lin_model_1.predict(X1_df)
fig, ax = plt.subplots(figsize=(12, 6))
plt.scatter(X1, y)
plt.plot(X1, y_pred_1, color='red')
plt.show()

# Multiple linear regression using both X1 and X2
X_df = pd.concat([pd.Series(X1), pd.Series(X2)], axis='columns', keys=["X1", "X2"])
lin_model = LinearRegression()
lin_model.fit(X_df, y)
print(lin_model.intercept_ , lin_model.coef_)

fig, ax = plt.subplots(figsize=(12, 6))
ax.set_title("Linear Regression Partial Dependence")
lin_disp = plot_partial_dependence(lin_model, X_df, ['X1'], ax=ax)
plt.show()

Update:
there is no strong collinearity between X1 and X2: 0.66 but the X2 and y are strongly correlated (almost 1). I haven't tried to generate another example, but I guess it's always some sort of multicollinearity between the variables that might result in such situations. Is it safe to say so?


texts:
In Deep Mind's Rainbow paper, how come A3C algorithm be so slow? twice slower than DDQN... Was it trained on a single actor? :D

It's on page 1 of the paper

Wasn't A3C supposed to be something a lot more powerful?

For example as splendid as this, taken from here:


texts: loss/val_loss decrease but acc/val_acc are consistent

texts:
I don't know why I am getting such good results.

Epoch 3/10 2937/2937 [==============================] - 12s 4ms/step -
loss: 0.2836 - acc: 0.4679 - val_loss: 0.1937 - val_acc: 0.1980

Epoch 4/10 2937/2937 [==============================] - 12s 4ms/step -
loss: 0.1355 - acc: 0.4679 - val_loss: 0.0866 - val_acc: 0.1980

>Epoch
5/10 2937/2937 [==============================] - 13s 4ms/step - loss:
0.0580 - acc: 0.4679 - val_loss: 0.0342 - val_acc: 0.1980

Epoch 6/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0223 - acc: 0.4679 - val_loss: 0.0120 - val_acc: 0.1980

Epoch 7/10 2937/2937 [==============================] - 14s 5ms/step -
loss: 0.0082 - acc: 0.4679 - val_loss: 0.0040 - val_acc: 0.1980

My training and label sets are float number arrays in range [-0.05, 0.05] and I am using Keras.sequential.model.lstm. Why might this be happening? Previously, I had the opposite problem here: loss/val_loss are decreasing but accuracies are the same in LSTM!, but I couldn't understand the problem.

EDIT: I changed my code from:

model.compile(optimizer = 'adam', loss = 'mean_square_error', metrics=['accuracy'])

to:

model.compile(optimizer = 'adam', loss = 'mean_absolute_error', metrics=['accuracy'])

But the result is same.

I then changed the above line of the code to:

model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['mean_squared_error'])

But it didn't work and the result is as follow:

Train on 2937 samples, validate on 735 samples Epoch 1/10 2937/2937
[==============================] - 90s 31ms/step - loss: 1.6645 -
mean_squared_error: 0.0019 - val_loss: 0.7620 -
val_mean_squared_error: 0.0010

Epoch 2/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.5503 - mean_squared_error: 0.0019 - val_loss: 0.3890 -
val_mean_squared_error: 0.0010

Epoch 3/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.2837 - mean_squared_error: 0.0019 - val_loss: 0.1938 -
val_mean_squared_error: 0.0010

Epoch 4/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.1355 - mean_squared_error: 0.0019 - val_loss: 0.0866 -
val_mean_squared_error: 0.0010

Epoch 5/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0580 - mean_squared_error: 0.0019 - val_loss: 0.0342 -
val_mean_squared_error: 0.0010

Epoch 6/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0223 - mean_squared_error: 0.0019 - val_loss: 0.0120 -
val_mean_squared_error: 0.0010

Epoch 7/10 2937/2937 [==============================] - 13s 5ms/step -
loss: 0.0082 - mean_squared_error: 0.0019 - val_loss: 0.0040 -
val_mean_squared_error: 0.0010

Epoch 8/10 2937/2937 [==============================] - 14s 5ms/step -
loss: 0.0035 - mean_squared_error: 0.0019 - val_loss: 0.0017 -
val_mean_squared_error: 0.0010

Epoch 9/10 2937/2937 [==============================] - 13s 5ms/step -
loss: 0.0022 - mean_squared_error: 0.0019 - val_loss: 0.0011 -
val_mean_squared_error: 0.0010

Epoch 10/10 2937/2937 [==============================] - 13s 5ms/step
- loss: 0.0019 - mean_squared_error: 0.0019 - val_loss: 0.0010 - val_mean_squared_error: 0.0010


texts:
I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:

x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)

lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]

sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

At human check I see that as a result, all predicted labels are a constant sequence.

What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).


texts:
Following is what I learned about the process followed during building and pruning a decision tree, mathematically (from Introduction to Machine Learning by Gareth James et al.):

Use recursive binary splitting to grow a large tree on the training
data, stopping only when each terminal node has fewer than some
minimum number of observations.
Apply cost complexity pruning to the large tree in order to obtain a
sequence of best subtrees, as a function of α.
Use K-fold cross-validation to choose α. That is, divide the training
observations into K folds. For each k = 1, . . .,K:
(a) Repeat Steps 1 and 2 on all but the kth fold of the training data.
(b) Evaluate the mean squared prediction error on the data in the
left-out kth fold, as a function of α.
Average the results for each value of α, and pick α to minimize the
average error.
Return the subtree from Step 2 that corresponds to the chosen value
of α.

My question: Going through the above algorithm means we automatically choose the most optimum tree based on minimum average error. Why then, do we have to select maximum depth as the pruning parameter. Shouldn't optimum depth be decided by the algorithm itself and not us?

If anything, shouldn't we be choosing 'K' here, as in how many-fold validation would we like to do?


texts: Can I completely cancel the effects of using a smaller batch size by reducing the learning rate?

texts:
I'm having the problem that the data from a regular sized batch (e.g., 32, 64) doesn't fit in my GPU. Among other solutions, I'm considering reducing the batch size, as is normally suggested. Of course, this will make the neural network more unstable, so other things need to be considered.

For instance, I'm thinking about reducing the learning rate in order to offset the instability brought by the smaller batch size. Would the higher instability be completely canceled by using a smaller learning rate? Or, does this bring other problems of its own, and does not solve the problem completely?


texts: I am building a machine learning based model (random forest in scikit-learn) to predict maize yields in the U.S. based on data on historical maize yields and temperature and precipitation information. Since there is a trend in maize yields, I detrend the yield data first, then fit the model and then predict for current year. However, I assume that the predictions from the model are detrended as well. How do I go about making predictions that are not detrended? Any pointers and literature reference would be helpful


texts:
I am seeking advice on a solution I am building as a part of my data science course.
I have to build a web application that solves a problem created by COVID-19.
My idea - For the first stage of my project development, I decided to build a solution that tracks hospital resource utilization by state(USA) as an effect of rising COVID cases and gives a warning beforehand in case COVID cases are rising and not enough resources are available.
Datasets - The data i am considering for this project are:

USA Hospital Beds - https://aws.amazon.com/marketplace/pp/prodview-yivxd2owkloha?qid=1599355365023&sr=0-13&ref_=srh_res_product_title
A COVID time-series data(There are many).
Dataset giving population vulnerability and other similar datasets.

I am not very experienced in working with sequential data, to be honest this is my first one. I am confused about which kind of architecture will be appropriate to approach a problem like this.
First i considered time series analysis. but, according to my current understanding time series just looks at trends and seasonality in the data. My problem is much of a cause and effect problem so now I am leaning towards RNN/LSTM/Attention.
Can you validate my choices or provide a suggestion on better alternatives?


texts:
I am using Boruta feature selection (with Random forest) to decide the important features in the below data set.

   Gender Married Dependents     Education Self_Employed  ApplicantIncome  \
0   Male      No          0      Graduate            No             5849
1   Male     Yes          1      Graduate            No             4583
2   Male     Yes          0      Graduate           Yes             3000
3   Male     Yes          0  Not Graduate            No             2583
4   Male      No          0      Graduate            No             6000

 CoapplicantIncome  LoanAmount  Loan_Amount_Term  Credit_History  \
0                0.0         NaN             360.0             1.0
1             1508.0       128.0             360.0             1.0
2                0.0        66.0             360.0             1.0
3             2358.0       120.0             360.0             1.0
4                0.0       141.0             360.0             1.0

Property_Area Loan_Status
0         Urban           Y
1         Rural           N
2         Urban           Y
3         Urban           Y
4         Urban           Y

Please help me in clarifying the below doubts
1) whether I need to convert the all categorical variable into numeric variable (using one hot encoding) before applying Boruta?
2) Whether the NA values would be taken care by Boruta or do we need to remove NA values before feeding into Boruta ?

In case of Regression problem, whether the Boruta approach (Random forest classifier -> Boruta) remains the same as in classification problem ?

Thank you.


texts:
Are there any heuristics for deciding on where to start with the number of layers for a neural network? I built one with with 7 layers that has 40 input features and this took around 8 hours to train completely, but now I want to add a bunch of polynomial features which will increase the input size of my network to a few thousand features. Since there are so many features will my network benefit from having more layers? If so, what is a good starting point for the number of features?

Here is some information about my problem: suppose I have a set of usage data from a website. So if a user logs into the website one day their usage statistics during that day is calculated. I am using a few statistical tools to summarize the data and I want to make a prediction about whether or not the user will use the website in the next 60 days. My plan is to try and engineer all the possible monomial quadratic features from this summary and feed this information into a DNN. Without the feature engineering I have roughly 60 different features, but with it I will have at least an addition 3600 features.


texts: How to measure F1 score and NMI for clustering task?

texts:
I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section.

Although I have looked at the code, I am not sure about the actual algorithm.


texts: Deep learning technique to recognize a person's silhouette

texts: I'm reading Nielsen's book on neural networks. In the first chapter you construct a neural net to recognize handwritten digits by training the network on 60,000 samples. I wonder how would you build a neural net to recognize a person by his/her silhouette? Is there some special technique to make it possible?


texts: I work on a dataset with numeric values.The class labels has also numeric values.I made the 6 numeric class labels into one witch contain values like e.g happy_pleased. I want to insert the new .arff file into weka but i have a problem with the class @attribute as i declare it as nominal. Witch is the right type to declare the class label? I tried nominal but nothing happened


texts: Daily-level variables and weekly-level variables at the same model

texts:
I am running a model which has NPS (Net Promoter Score) as its target and various predictors among which some CSAT (Customer Satisfaction) data.

The rest of the predictors are good to use at daily level but NPS & CSAT are not exactly because on daily basis too few NPS/CSAT surveys are filled in (in my data) so on daily level the ratings are not exactly so representative. (option 1 - see below)

The obvious option is to use all the variables at weekly level to avoid these biases but then I am losing some valuable information from the rest of the variables. (option 3 - see below)

However, I was thinking that I can use the NPS & CSAT as constants (weekly averages) at daily level and use the rest of the data properly at daily level. (option 2 - see below)

Therefore there are the following options:

Use all variables at daily level.
Use NPS & CSAT as constants (weekly averages) at daily level and use the rest of the data properly at daily level.
Use all variables at weekly level.

Is option (2) actually better than option (1) or at both cases you introduce some significant amount of noise to the model?


texts: How to identify text similarity based on training data?

texts:
I have a set of documents  (1 to 11) for which the labeling is done.
Lets Assume:
Doc No: 1,3,5,7 - Belongs to Type A
Doc No: 2,4,9 - Belongs to Type B
Doc No: 8,10 - Belongs to Type C
Doc No, 6,11 - Belongs to No one

Now, let us say I have new incoming docs - 11,12,13 .. and so on, and I would like to know which Type (A, B, C or none ) they belong to based on the text similarity of existing docs in that Type. Can someone please suggest how I can achieve this?
Should I create my own corpus of data and consider it a supervised problem?


texts: LSTM doesnt find finer dependencies than the Random Forest model

texts:
I am currently implementing a forecasting model. The whole builds on a regression. The first model is an LSTM network based on Keras and the second a Random-Forest model based on sklearn.

I use the same input values and target variables for both models. The forecasts of the LSTM network are closer to the actual observed values (better rmse and r2) (probably due to the memory / lookback), but the Random Forest model provides better altitudes and depths. It seems that the Random-Forest model has found better dependencies than the LSTM model.

LSTM:

Random-Forest:

What is the best way for the LSTM model to understand finer dependencies? Should I use an ensemble or stack method from LSTM + RF? Does it make sense?


texts: Conjugated gradient method. What is an A-matrix in case of neural networks

texts:
I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $u$ and $v$ are conjugated with respect to $A$ if $u^TAv=0$. I also read that $A$ is symmetric, positive definite matrix.

I am trying to find out how is that related to training of neural network by minimalising mean-square error function using CG method. What will be $A$ matrix in that case? How $A$ matrix is connected to weights of neural network. And is it still symmetric and positive definite? I read What is conjugate gradient descent? this thread and resources linked there, but I still can't figure it out.

I'm sure I'm missing something simple, but could you give me a bit of explanation?

Thank you,

Max


texts: Reason for having both low loss and same predicted class?

texts:
I'm training a cNN for binary classification. I used a batch size of 128, and the loss is decreasing and accuracy is increasing over epoches. The accuracy reached over 0.99 eventually, and the loss reached below 0.3. But after a few more epoches, the model converges to loss 0.6 and accuracy 0.5. An inspection of the model shows that it always predicts 0.5. I'm using binary cross entropy as the loss function. For each epoch, all data points are shuffled. I'm using SGD, learning rate is 0.01.

Am I hitting a local minimum with low accuracy rate but fairly good loss value? What is the suggested approach to deal with this? Also, why is it possible to have a low loss function with one single predicted class?


texts: Balancing on the particular imbalanced classes of image dataset

texts:
I have a dataset that has 12 classes in the base directory. However, these 12 classes consist of several amounts of Images. The number of images of 12 classes is inconsistent therefore its impacts the total accuracy. Thus, should I apply the data augmentation to the particular classes that have a low amount of data?
Image data of each class:
#Dummy Classes
[AAAA: 713
ABCD: 274
ACBD: 335
ADBC: 576
BBBB: 538
BACD: 607
BCAD: 253
BDAD: 257
CCCC: 463
CABD: 309
CBAD: 452
CDAB: 762]

Therefore, if should I have to apply data augmentation to increase the amount of data in that particular classes that have a lower amount of image, as a result, I have been applied data augmentation but it does not increase the image data. Besides that, I want to generate the augmented data with the raw data that means the input and out directory will be the same. Therefore, it is possible to assist me to solve this problem?
Notebook: Google Colab
OS: Windows 10
Code of Augmentation for particular (Individual Classes):
from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=45,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range = 0.2,
    zoom_range = 0.2,
    horizontal_flip=True,
    fill_mode = 'reflect', cval = 125)

i = 0

for batch in datagen.flow_from_directory(directory = ('/content/dataset/ABCD'),
                                         batch_size = 317,
                                         target_size = (256, 256),
                                         color_mode = ('rgb'),
                                         save_to_dir = ('/content/dataset/ABCD'),
                                         save_prefix = ('aug'),
                                         save_format = ('png')):
  i += 1
  if i > 100:
    break

Output: Found 0 images belonging to 0 classes.


texts:
I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings.
What algorithms are suggested to do this? I don't know number of clusters in advance (and as more data is coming clusters can change as well), what features are normally used to represent each sentence?
I'm trying now the simplest features with just list of words and distance between sentences defined as:
$|A \cup B$ \ $A \cap B|$/$|A \cup B|$
(A and B are corresponding sets of words in sentence A and B)
Does it make sense at all?
I'm trying to apply Mean-Shift algorithm from scikit library to this distance, as it does not require number of clusters in advance.
If anyone will advise better methods/approaches for the problem - it will be very much appreciated as I'm still new to the topic.


texts: How many neurons a deep learning model (specifically a CNN) does have?

texts:
I found total number of neurons of ResNet-50 model is 26,560 and 94,059 in two different papers. Their titles are below:
1: DeepXplore: Automated Whitebox Testing of Deep Learning Systems
2: Testing Deep Neural Network based Image Classifiers
I found another article (link) that clearly describes how many neurons a CNN does have. If I follow this technique then the neuron count does not match with the count mentioned in those papers.
Could anybody please help me how I can calculate number of neurons in a DL model? Also which layers have neurons and which don't have.


texts: How do I encode the categorical columns if there are more than 15 unique values?

texts:
I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have more than 15 unique values such as country.
Do I still use one-hot encoding? Or is there an alternative? Is using regression on this dataset a good idea?


texts:
Are there any ways to check if more data can help the quality of kmeans clustering?

The closest research I could find was for EM algorithms.

Any thing similar for evaluating kmeans ?


texts: Discrepancy in probability calculations in paper 'Multi-digit Number Recognition...'

texts:
In the paper, 'Goodfellow, I., et al. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. ICLR, 2014', on page 10 there is a table which calculate $\log(P(S|H))$ given $\log(P(s_1,s_2,...,s_l))$ and $\log(P(l))$.

From the equation given in page 4 it should be

$$\log(P(S|H)) = \log(P(s_1,s_2,...,s_l)) + \log(P(l))$$

but if you see from the first two lines in the lower table it is:

$$\log(P(S|H)) = 10\cdot\log(P(s_1,s_2,...,s_l)) + \log(P(l))$$

and on third line it is correctly specified as:

$$\log(P(S|H)) = \log(P(s_1,s_2,...,s_l)) + \log(P(l))$$

Did I miss anything??

Here is the table of probability (idealized) predicted by the model: