texts: How to interpret the confusion matrix and compare the result of features extraction with LBP and Haralick

texts:
I'm begginer in deep learning so I tried to execute a code of liveness face detection from github in this link :https://github.com/imironica/liveness , so when I tried to run features extraction with LBP and HAralik, I got a multiple results in confusion matrixes like this:
1- test features with LBP:
Train Nearest neighbors (3)
Accuracy: 0.7621621621621621
Confusion matrix:
 [[1625   45]
 [ 703  772]]

Train SGD
Accuracy: 0.8158982511923688
Confusion matrix:
 [[1588   82]
 [ 497  978]]

Train Naive Bayes
Accuracy: 0.8756756756756757
Confusion matrix:
 [[1544  126]
 [ 265 1210]]

Train Decision Tree Classifier
Accuracy: 0.6791732909379968
Confusion matrix:
 [[1569  101]
 [ 908  567]]

Train Adaboost Classifier
Accuracy: 0.6515103338632751
Confusion matrix:
 [[1630   40]
 [1056  419]]

Train Gradient Boosting Classifier
Accuracy: 0.6419713831478537
Confusion matrix:
 [[1622   48]
 [1078  397]]

Train Random Forest Classifier
Accuracy: 0.7106518282988871
Confusion matrix:
 [[1641   29]
 [ 881  594]]

Train Extremelly Trees Classifier
Accuracy: 0.724006359300477
Confusion matrix:
 [[1638   32]
 [ 836  639]]

Train Linear SVM with C=1
Accuracy: 0.8076311605723371
Confusion matrix:
 [[1593   77]
 [ 528  947]]

Train SVM with C=10
Accuracy: 0.8082670906200318
Confusion matrix:
 [[1591   79]
 [ 524  951]]

2-and when I tried test features with Haralik I got:
Train Nearest neighbors (3)

Nearest neighbors (3): 0.753577106518283

[[1580   90]

 [ 685  790]]

Train SGD

SGD: 0.7424483306836248

[[1598   72]

 [ 738  737]]

Train Naive Bayes

Naive Bayes: 0.7093799682034976

[[1521  149]

 [ 765  710]]

Train Decision Tree Classifier

Decision Tree Classifier : 0.6511923688394277

[[1527  143]

 [ 954  521]]

Train Adaboost Classifier

Adaboost Classifier : 0.763751987281399

[[1537  133]

 [ 610  865]]

Train Gradient Boosting Classifier

Gradient Boosting Classifier: 0.7939586645468999

[[1544  126]

 [ 522  953]]

Train Random Forest Classifier

Random Forest Classifier: 0.7007949125596185

[[1564  106]

 [ 835  640]]

Train Extremelly Trees Classifier

Extremelly Trees Classifier: 0.7364069952305247

[[1543  127]

 [ 702  773]]

Train Linear SVM with C=0.01

Linear SVM with C=0.01 : 0.5303656597774244

Train Linear SVM with C=0.1

Linear SVM with C=0.1 : 0.729093799682035

Train Linear SVM with C=1

Linear SVM with C=1 : 0.7335453100158983

Train Linear SVM with C=10

Linear SVM with C=10 : 0.7869634340222575

Train Linear SVM with C=100

Linear SVM with C=100 : 0.7939586645468999

Train Linear SVM with C=500

Linear SVM with C=500 : 0.7936406995230525

Train Linear SVM with C=1000

Linear SVM with C=1000 : 0.7968203497615263

Train Linear SVM with C=2000

Linear SVM with C=2000 : 0.794912559618442

Train SVM with C=0.01

SVM with C=0.01 : 0.5310015898251192

Train SVM with C=0.1

SVM with C=0.1 : 0.5306836248012718

Train SVM with C=1

SVM with C=1 : 0.7297297297297297

Train SVM with C=10

SVM with C=10 : 0.7456279809220986

Train SVM with C=100

SVM with C=100 : 0.7802861685214626

Train SVM with C=500

SVM with C=500 : 0.7910969793322734

Train SVM with C=1000

SVM with C=1000 : 0.7926868044515103

Train SVM with C=2000

SVM with C=2000 : 0.7939586645468999

Train SVM with C=4000

SVM with C=4000 : 0.7926868044515103

Train SVM with C=10000

SVM with C=10000 : 0.7930047694753577

Train SVM with C=2000000

SVM with C=2000000 : 0.7364069952305247

so now I'm trying to understand what is the difference between LBP and Haralik in features extraction and which is the best to choose?


texts:
So I was training a fairly shallow convnet, because my deepnet based on vgg19 wasn't working. 2 conv layers and 2 flat layers, the second flat layer was the output.

It converged quickly to all zeros in the second conv layer. First conv layer wasn't all zeros. First flat layer appears to have learned the ditribution of classes.

So it appears that my network's strategy os to ignore the inputs and just predict the class distribution. I tried class weights, same result just a more uniform distribution gets predicted.

I changed the learning rate, optimizer, and even tried gradient clipping. I augmented the data, and introduced regularization on the data, and network via dropout layers. No luck. Same result.

Why would my network exhibit this behavior and what can be done about it?

EDIT:

Here is a code snippet that defines the network that I am using:

def discriminator_model():

model = Sequential()
model.add(Conv2D(32,
                 (10, 10),
                 strides=(2,2),
                 input_shape=( 256, 256, 3),
                 kernel_initializer='random_uniform',
                 bias_initializer='zeros'))
#model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(228, activation='sigmoid'))
return model

After training I was trying to see what the layers had actually learned with the following code:

from keras import backend as K

inp = d.input                                           # input placeholder
outputs = d.layers[3].output
functors = K.function([inp], [outputs])
print(functors([x])[0].shape)
arr = functors([x])[0].reshape(64,29,29,1)
for i in range(32):
     #p = (255./(np.max(arr[i])-np.min(arr[i])))*(arr[i]-np.min(arr[i]))
     p=255.0*arr[i]
     plt.imshow(p.reshape(29,29))
     plt.show()

Here is an example of what I am getting as output from the

And here is example output from the first layer for comparison, which I generated with similar code:

Also tried changing loss functions, removing pooling layers, etc. I know it is weird. I've never come across anything quite like this before.


texts: How to evenly distribute data to multiple GPUs using Keras

texts:
I am using Keras=2.3.1 with Tensorflow-gpu=2.0.0 backend. While I trained model on two RTX 2080 ti 11G GPUs, it allocates all data to '/gpu:0',and nothing changed with '/gpu:1'. Surely, the second GPU not used at all.
However, every GPU could work if I selected only one GPU.
Moreover, the two gpus can be run parallelly in Pytorch.
Follow some instances, I try to run multi-gpus with these codes:

Below is NVIDIA-SMI output when I run a multi-gpus model.

and cuda = 10.1, cudnn = 7.6.5.


texts: Predict Customer Next Purchase with Sequence

texts:
Suppose I buy products: [1,2,3,4]

Another customer X bought: [2,3]

Most probably customer X next purchase will be: 4

Sequence is very important in my problem

I tried association analysis using R, but it don't take under consideration the sequence

Please advise what algorithms I need to solve this?

Do I need to First do Clustering to find similar customers?


texts:
I have a dataset as following:

This is test case 1. My goal is to fill the missing years data. As the age sex and smoking is not changing so I have to predict the condition and percent data for year 0 to all the way 54. I found high correlation between condition and percent variable. This seems easy. But I am a bit confused now. Should I have to use multivariable regression? what would be the most best method to approach?


texts: Label distribution over training, validation and test

texts: I am wondering over whether the number of classes distributed over my training, validation, and test label affects the model.


texts: Alternative methods for novelty detection and correlations

texts:
Hey mates I have the following project:

Imagine having two datasets A and B. Each dataset consits of 101 time series with the same lenght and identical time stamps. The two datasets where taken from the same experiment, therefore the data structure is identical. From the 101 time series there is one particulary signal that is of interest in both datasets. Lets call that signal X(t)_101. Now we have the following case that the signal X(t)_101 from dataset A (good dataset) is significantly different than from dataset B (bad dataset). The goal now is to analyse the remaining 100 signals and find the signals that have the biggest effect on making our signal of interest X(t)_101 go bad. So in the end I would get some sort of ranking, on which signal is responsible for the change of our signal X(t)_101.

So far I am looking into novelty detection and correlation, are there any other keywords I should consider for this topic?

Thanks


texts: Information Extraction from image / text - approach?

texts:
I need assistance with a ML project I am currently trying to create.

I receive a lot of invoices from a lot of different suppliers - all in their own unique layout. I need to extract 3 key elements from the invoices. These 3 elements are all located in a table/line items for all the invoices.

The 3 elements are:

1: Tariff number (digit)
2: Quantity (always a digit)
3: Total line amount (monetary value)

Please refer to below screenshot, where I have marked these field on a sample invoice.

I started this project with a template approach, based on regular expressions. This, however, was not scaleable at all and I ended up with tons of different rules.

I am hoping that machine learning can help me here - or maybe a hybrid solution?

The common denominator

In all of my invoices, despite of the different layouts, each line item will always consist of one tariff number. This tariff number is always 8 digits, and is always formatted in one the ways like below:

xxxxxxxx
xxxx.xxxx
xx.xx.xx.xx

(Where "x" is a digit from 0 - 9).

Further, as you can see on the invoice there is both a Unit Price and a Total Amount per line. The amount I will need is always the highest for each line.

The output

For each invoice like the one above, I need the output for each line. This could for example be something like this:

{
    "line":"0",
    "tariff":"85444290",
    "quantity":"3",
    "amount":"258.93"
},
{
    "line":"1",
    "tariff":"85444290",
    "quantity":"4",
    "amount":"548.32"
},
{
    "line":"2",
    "tariff":"76109090",
    "quantity":"5",
    "amount":"412.30"
}

Where to go from here?

I am not sure of what I am looking to do falls under machine learning and if so, under which category. Is it computer vision? NLP? Named Entity Recognition?

My initial thought was to:

Convert the invoice to text. (The invoices are all in textable PDFs, so I can use something like pdftotext to get the exact textual values)
Create custom named entities for quantity, tariff and amount
Export the found entities.

However, I feel like I might be missing something.

Can anyone assist me in the right direction?


texts: How to load numerous files from google drive into colab

texts:
I am trying to load in 30k images (600mb) from Google drive into Google Colaboratory to further process them with Keras/PyTorch.

Therefore I have first mounted my Google drive using:

from google.colab import drive
drive.mount('/content/gdrive')

Next I have unzipped the image file using:

!unzip -uq "/content/gdrive/My Drive/path.zip" -d "/content/gdrive/My Drive/path/"

Counting how many files are located in the directory using:

len(os.listdir(path-to-train-images))

I only find 13k images (whereas I should find 30k). According to the output of unzip, the files appear to be unzipped correctly.

Also, I found there are some issues with loading in many files from a google directory: https://github.com/googlecolab/colabtools/issues/510.

Does anyone know where I am going wrong? Or whether there is a workaround?


texts: Surface Pro 6 vs Macbook Pro for Professional Data Science Practice

texts:
[I strongly agree this is totally very opinionated question, thus narrators feel free to vote to close it if you feel it is right, but I find endless pros and cons on the Internet, I've decided to ask the community here.]

Surface Pro 6 or Macbook Pro for Data Scientist Job?

About 8 years ago I was a Windows user. The most annoying part was that it was a quite unstable. It is noted that I was not a developer or a data scientist back then. Anyways, I moved to Mac OSx both at work during my PhD studies since it was a better choice for doing Computational Physics. I totally felt great with such a smooth, stable OS that allows access to Linux kernel as well, and eventually I bought a Macbook Pro for my personal use.

At present I am about to start my new role in industry as Data Scientist. The whole team is based on Microsoft, even very few developers in the team using the latest Surface Pro 6. They give me a choice of either Surface Pro 6 or Macbook Pro, although I felt it was implied they would rather me going with Surface Pro 6 (still I can choose!). The configues will be exactly the same as I wish, so this wouldn't matter.

Obviously I am leaning towards Macbook Pro as I have been with for some time now and am quite efficient and productive. Nevertheless, majority of other employees are Windows-base, and I feel if I go with Mac I would be an outlier.

At the same I am not feeling completely stranger around Windows and I am not totally blind! In fact recently I hear in the news a lot about Microsoft drastic changes naming: incorporating Linux kernel starting from end of this month (would be so sure if it would work as the Mac one though) in Windows 10 Pro, Visual Studio Code is a product of Microsoft that has been gaining lots of interest and I am using and loving it and perhaps would run even more smoother in Windows machine than Mac, and not that it is very relevant but Widows acquired GitHub. it is worth  to mention that I will have GPU if I go with Surface Pro 6, and has a nice touchscreen feature too!

I guess I still looking for an actual use case, where someone did in fact had such a transition recently, and would appreciate sharing your thoughts and feedbacks.


texts:
In Orange3, I have a table with row duplicates. How can I simply remove the duplicated rows to keep only one copy of each?
I feel like Orange3 is great but cannot do simple tasks. I hope I just did not find the right widget.


texts: Widening prediction intervals when data is missing

texts:
Question: How can I get a prediction interval that widens when data is missing?
Problem:
I'm trying to predict a likely range of customer spending.  Prediction interval seems like an excellent answer; my feature distributions are unknown, and are probably unknowable.  I have a reasonably rich dataset, but the any given customer is likely to be missing some random features.
Intuitively, I really want my intervals to widen when data is missing -- the less information informs my prediction, the less precise my estimate can be, right?
Attempts:
For this style of problem, I typically use ensemble tree models like LGB or more recently NGBoost -- trees seem to do a good job of inferring the existence and behaviour of demographics of which I have no prior knowledge.  These models deal with missing data by imputing -- either (a) before prediction, or (b) in-model, but allocating values to the side that better reduces loss.  Both of these techniques have the opposite of the desired effect!
I've also had a look at Bayesian models, thinking they should be able to do this just by eliminating the terms for the missing features, but... none of them seem to do so.  I can't find anything that works appropriately, and rolling my own from first principles seems like an incredibly deep rabbit hole.
TLDR: Can you suggest a quantile regression model that widens to account for missing data?


texts:
I am going to work on a project which requires implementation of A2C model using Tensorflow 2.0. I am new in the Machine Learning field and also in Python. These are topics which I have covered theoretically:

Different methods of Machine learning (supervised, unsupervised, reinforcement)
Linear and Logistic Regression
Required knowledge on Statistic and Probability
Neural network
Policy gradient
Gradient Descent
Basic of Tensorflow 2.0 (basic operations, preprocessing of Data)

Now I am a bit confused about what structure should I follow to become well familiar with Reinforcement learning, A2C model with Tensorflow and complete the project. I need some structured guidelines to study (even if possible from scratch).


texts: How to ensemble classifier incorporating all features in python?

texts:
I am doing a text classification task(5000 essays evenly distributed by 10 labels). I explored LinearSVC and got an accuracy of 80%. Now I guess whether accuracy could be raised by using ensemble classifier with SVM as base estimator?

However, I do not know how to employ an ensemble classifier incorporating all the features? Please note that I do not want to combine the different features directly in a single vector.

Therefore, My first question: in order to improve the current accuracy, is it possible to use ensemble classifier with svm as base estimator?
My second question How to employ an ensemble classifier incorporating all features?


texts: How to conclude the generality of any classification methods?

texts:
Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$.
The task $A$ is measured by a consistent measure.
For instance, the task A can be a binary classification. In this case, F-score, ROC curve can be used.
I did a survey on some research are and found that

$M_1$ is evaluated with dataset $D_1$ (open) using pre-processing $P_1$ only (seems the seminal work).
$M_2$ is evaluated with dataset $D_1$ (open), $D_2$ (private) and compared with $M_1$, claiming $M_2$ has more accurate result, but using different data pre-processing $P_2$.
$M_3$ proposes new way with dataset $D_3$ (private) and did not provide any comparison against $M_2$ and $M_1$

I'm trying to work on this area, but there are lots of inconsistency.
None of the methods are validated with validation data. They just used train and test data.
I think some parameters are tuned for test dataset although authors do not claim so.
Since this field is not a data science-oriented and the amount of dataset is few, this may happen.
Which method can we consider as a state-of-the-art?
How can we conclude the generality of each of the method?


texts: emphasise some observation weights more than the others

texts:
I want to emphasise (increase the weight) of only a subset of data. Lets say I have old and fresh data, I would like to say that old data has to have more weight and therefore has more influence in the decision than the new data.

In scikit-learn I found only class-weight parameter, but it does not change the weight of the samples, only of all samples within the class.

Is there a way to incorporate this emphasis into the gradient boosted trees in spark or xgboost in python?


texts: Understanding Training and Test Loss Plots

texts: I have attached a figure that contains 6 subplots below. Each shows training and test loss over multiple epochs. Just by looking at each graph, how can I see which one is the best? Which ones are overfitting or underfitting. Which ones are getting worse with each epoch?


texts: Reconstructing original data points from t-SNE output

texts: I have been trying to understand t-SNE for a while now and I have this very basic question on the comparison of PCA and t-SNE, on which I would really appreciate some help. In case of PCA suppose the Eigen vectors (say first $m$ PCs are retained) are $U_{n\times m}$ and your data matrix is $X_{n\times p}$ with covariance matrix $C_{n\times n}=XX^T$, then the reduced data set is obtained by performing a projection $X_r=U^TX$. From this $X_r$ one can get back $X$ by simply multiplying $X_r$ with $U$. Thus $U$ acts as the basis functions on which the data is projected. I was wondering what would be the basis function for t-SNE type dimensionality reduction algorithm. I know it's non-linear so such a linear projection would not be available, but is there a way to get back the original data set from the reduced data set as obtained as the output of t-SNE algorithm? Apologies if this is too intuitive or fundamental and is simply because of my lack of understanding t-SNE. Any links to material related to this would be appreciated


texts:
i have a question about the tutorial of tensorflow to train the mnist database
how do i create my own batch without using next_batch() , the idea is to train with a batch of 50 ,then 100 and so but it has to be in order

here is the code

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.initialize_all_variables())
for i in range(20000):
    batch = mnist.train.next_batch(50)#revisar aqui
    if i%100 == 0:
        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
        print("step %d, training accuracy %g"%(i, train_accuracy))
    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})


texts: Why does logistic function use e rather than 2?

texts:
sigmoid function could be used as activation function in machine learning.

$${\displaystyle S(x)={\frac {1}{1+e^{-x}}}={\frac {e^{x}}{e^{x}+1}}.}$$

If substitute e with 2,

def sigmoid2(z):
    return 1/(1+2**(-z))
x = np.arange(-9,9,dtype=float)
y = sigmoid2(x)
plt.scatter(x,y)

the plot looks similar.

Why does the logistic function use $e$ rather than 2?


texts: User defined aggregations on data of around 200GB where row order matters

texts:
I am working with "medium large" data of around 200GB. The data are long form log files, where there are several thousand logs for each "entity". The entities are actually flights and each log entry occurs at a different time stamp. Temporal order matters (I need to sort on timestamp).
to prepare the data, I need to perform several operations on each flight -- order them by time, select the last 300 observations, rescale by mean and standard deviation. Nothing fancy.
Using a small sample of the data, I wrote a simple R program that split the dataframe by entity ID and applied a short function to each piece.
Now I need to scale up the code. I considered using Spark's groupBy and user defined functions, but I know that Spark DataFrames are inefficient when you need to index the rows and relate one row to its neighbour under the indexing. And in any case, 200GB of data do not require a massively distributed solution.
So I would like to know what are the best tools to scale up a moderate sized data analysis, when I need to split a large file into smaller dataframes and apply a function to each, where the order of the rows needs to be controlled.


texts: Why does averaging a sentence's worth of word vectors work?

texts:
I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/.  The goal is to predict the label using a Random Forest classifier.

Each text sentence as been vectorized using the GoogleNews word vectors. The embedding source can be found here: https://github.com/mmihaltz/word2vec-GoogleNews-vectors

In the example I am following along with there is one step that irks me - there is a step that converts my array of vectorized tokens to a single vector by taking the mean over the tokens e.g.

def transform(self, data):
        v = self.word_vectors.get_vector('king')
        self.D = v.shape[0]

        X = np.zeros((len(data), self.D))
        n = 0
        emptycount = 0
        for sentence in data:
            tokens = sentence.split()
            vecs = []
            m = 0
            for word in tokens:
                try:
                    vec = self.word_vectors.get_vector(word)
                    vecs.append(vec)
                    m += 1
                except KeyError:
                    pass
            if len(vecs) > 0:
                vecs = np.array(vecs)
                X[n] = vecs.mean(axis=0)    # take the mean of the vectors?  what does it mean?
            else:
                emptycount += 1
            n += 1
        print("Number of samples with no words found: %s / %s" % (emptycount, len(data)))
        return X

I am leaving out some boilerplate but later on I run the model and the results are surprisingly good:

model = RandomForestClassifier(n_estimators = 200)
model.fit(XTrain, YTrain)
print("train score:", model.score(XTrain, YTrain))
print("test score:", model.score(XTest, YTest))

> train score: 0.9992707383773929
> test score: 0.9378711740520785

I understand that the random forest model expects to have one row per example so it is unable to consume a sequence of embeddings like a RNN might.  So you are required to convert to single row (1-D array).

My question is: WHY does it work?  It seems at odds to me that the averaged word vectors would be able to capture anything about the context or meaning of a sentence by merely averaging over the encodings.

Best case scenario I would expect this technique breaks down for larger blocks of text because you would tend to squash all your examples into the same neighborhood of your input space.

It would be great to get some clarification on this.


texts: What kind of "vector" is a feature vector in machine learning?

texts:
I'm having trouble understanding the use of Vector in machine learning to represent a group of features.

If one looks up the definition of a Vector, then, according to wikipedia, a Vector is an entity with a magnitude and direction.

This can be understood when applying Vectors to for example physics to represent force, velocity, acceleration, etc...: the components of the Vector represent the components of the physical property along the axes in space. For example, the components of a velocity vector represent the velocity along the x, y and z axes

However, when applying Vectors to machine learning to represent features, then those features can be totally unrelated entities. They can have totally different units: one feature can be the length in meters of a person and another can be the age in years of the person.

But then what is the meaning of the Magnitude of such a Vector, which would then be formed by a summation of meters and years? And the Direction?

I do know about normalization of features to make them have similar ranges, but my question is more fundamental.


texts:
I'm really struggling to understand how the parameters of Keras early stopping callbacks play out, especially in the presence of a baseline.  What I want is simply for the training to stop within 2 epochs of the validation accuracy hitting 95%.  So I try:
(trainX, testX, trainY, testY) = train_test_split(Tensor, Labels1Hot, test_size=0.2)
when2stop = EarlyStopping(mode='max',monitor='val_accuracy',verbose=1,patience=2,baseline=0.95)
history = model.fit(trainX, trainY, epochs = 100, batch_size = 500, validation_data = (testX, testY), callbacks=when2stop)

This just stops after just 2 epochs, even if the val accuracy has in fact improved. But the val accuracy is way off 0.95. Is this a bug or am I mis-understanding the baseline and patience settings?
Using: Tensorflow 2.4


texts: How much data should we use during training, and how much in testing? Can anyone explain why does it always seem to be 70:30 or 80:20 ratios?


texts: Measuring the success of text summarization

texts:
I am trying to make a text summarization program that will take a text article and reduce it to a para or 2.

Since I am a newbie with no idea of NLP, it is hard to approach and break down the problem. So I was wondering if there was a measure that is used to check for effectiveness and correctness of text summarization. I tried googling this, but nothing that suits my purpose.

Does something like this even exist? or am I going in the wrong direction?


texts:
I have a large raw dataset on crime and I want to cluster the data using k-means, However, I get an Error when I enter this code:

Rawdata.3means <- kmeans(Rawdata, centers = 3).

Error:

Error in kmeans(Rawdata, centers = 3) :
  more cluster centers than distinct data points.
In addition: Warning message:
In storage.mode(x) <- "double" : NAs introduced by coercion

How can I resolve this error?


texts:
Currently we are regularly analyzing sets of paragraphs every month. I would like to automate this and split each paragraphs into chunks of data.

To do this I would like to employ a neural network. However, I am not really very familiar with creating neural networks.

Any ideas or starting point on how to do this using Neuroph or maybe in other framework/approaches?

Edit for more info as suggested.

I have very little experience on neural networks though I have some introduction with it in college. However I am very much familar with java

The data is around 3 megabytes only and consists of rules and relationships for a single domain. This means that the data is complex but relatively limited though still free-form English language.


texts: word2vec: usefulness of context vectors in classification

texts:
I've been working on a NN-based classification system that accepts document vectors as input. I can't really talk about what I'm specifically training the neural net on, so i'm hoping for a more general answer.
Up to now, the word vectors I've been using (specifically, the gloVe function from the text2vec package for R) have been target vectors. Up to now I wasn't aware that the word2vec training produced context vectors, and quite frankly I'm not sure what exactly they represent. (It's not part of the main question, but if anybody could point me to resources on what context vectors are for and what they do, that would be greatly appreciated)
My question is, how useful are these context word vectors in any kind of classification scheme? Am I missing out on useful information to feed into the neuralnet?
How would, qualitatively speaking, these four schemes fare?

Target word vectors only.
Context word vectors only.
Averaged target and context vectors.
Concatenated vectors (i.e. a 100-vector word2vec model ends up with a length of 200)


texts: Neural networks: generating g prime(z) in back propagation

texts:
I am doing Andrew Ng's excellent new Deeplearning course. I have an issue with implementing back propagation in a one hidden layer network that looks like this:

I am trying to derive the derivative of Z1 using the formula:

My understanding is that g prime Z1 in this formula is the derivative of g(z) with respect to Z which I believe is

    sigmoid(Z) * (1-sigmoid(Z))

where the function sigmoid generates the sigmoid function with respect to Z.
My final equation looks like this:

  dZ1 = np.dot(W2.T, dZ2) * (sigmoid(Z1)*(1-sigmoid(Z1)))

Where Z1 is the output of the first (and only) hidden layer.

Where am I going wrong?


texts: different scored probability distributions for a classification model using python vs Azure ML Studio

texts:
I have a classification model which I initially built in Azure ML studio and then created a similar model in python (its similar because the algorithms in python VS Azure are a bit different so they are not exactly the same). What I am observing is that the average scored probabilities for both classes is very different between the two models. The azure results are more definitive which means the ones which were categorized as class 1 have probabilities a lot closer to 1 and the reverse for class zero. Python one is interesting because the probabilities are more normally spread between 0 and 1. both results are pretty similar in terms of my desired metric so I can use both of them but I wonder why there is this huge difference between these two models. I appreciate any hint.
Thanks


texts: Text classification problem using Python or R

texts:
I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems.

I have a dataset with two columns, "Titles" and "Description". Titles column has names of clinical lab tests and description column has description about results of corresponding laboratory test( can be seen below). There many titles specific to a particular lab test.

Title                       Description
Complete blood test         RBC: Normocytic and Normochromic
COMPLETE Blood test\        Platelets: Adequate on the smear
Blood glucose               COLOUR - COLOURLESS
Complete blood picture      WBC: Total and Differential counts are within normal limits

I have only shared a small part of the data frame.

Problem 1: I have manually grouped the similar looking titles. For example, I have grouped the titles in the above data frame as "Blood Test". Is there a way to use NLP technique to group similar looking titles ( As shown below).

Problem 2: Based on the description, i have manually labeled a particular outcome for a lab test as normal or abnormal. Again i am looking for a way to do this without having to manually label the outcome (As shown below).

 Title                       Description                           Outcome
Blood Test                   RBC: Normocytic and Normochromic       Normal
Blood Test                   Platelets: Adequate on the smear       Normal
Blood Test                   COLOUR - COLOURLESS                    Normal
Blood Test                   WBC: Total and Differential counts     Normal

Any suggestions or resources to help me get started would be appreciated.


texts: When do we scale features and should it be done to label encoded features?

texts:
So, I am trying to make a linear regression model for predicting car prices for which I have the following data set:  Data Set

Since the unique values are a lot for the categorical features, I label encoded the categorical features and converted the data set like this: Modified Data Set

Thereafter, I scaled the continuous features ( Engine, Power) like so:

# Scaling continuous features (No scaling for categorical variables)

X_train[['Engine', 'Power']] = scale.fit_transform(X_train[['Engine', 'Power']])

est = sm.OLS(y_train,X_train).fit()
print(est.summary())

X_test[['Engine', 'Power']] = scale.transform(X_test[['Engine', 'Power']])
y_predicted = est.predict(X_test)
y_predicted.sort_values()

# R2 value for y_test and y_predictions

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_predicted)
print(r2)

and got an R2 value of 84.8% compared to 33.5% if I also scaled label encoded features. I want to know why this happened? Also, my predicted values using this model on testing data is resulting in negative prices as well and most of these prices are not predicted well with the R2 score of predicted and true data being 68%.


texts: What is a good way to start Data Analysis of unknown dataset (JSON data)

texts:
I am working with an organization to analyse their data residing in Mongodb and to look for any trends/patterns in the data. I am quite new to the professional field of Data Analysis but have a good background of Statistics and Data Mining (University coursework). I will be doing a proof of concept on the data to understand if the data the organization is gathering is good for Analytics and if no what enhancements should they include in their datasets to make it better for Analytics. I do have some predefined questions that I am planning to answer but looking strategically what would be a good way to approach this kind of problem.

I have previously worked with some of the Kaggle datasets and projects in University where datasets were fixed and questions to be answered were given.


texts: Can somebody explain me in a simple language how multiple instance ranking algorithm works? What is ranking function? how mathematically it is expressed?


texts:
I have input RGB images as follows:

I have a dataset of manually annotated images highlighting the outline(edges) from the input images I am attaching an example.

My aim is to train a ML algorithm which learns how these outlines are mapped. And when given a new image it should produce similar output. Is there any way that this can be done?

Ive tried canny-edge detector,sobel and other type of open cv transforms. But they do not distinct object from their background properly. So I am looking into ML.


texts: Neural network model for sparse multi-class classifier on Tensorflow

texts:
The problem I'm trying to solve is the following:

the data is Movielens with N_users=6041 and N_movies=3953, ~1 million ratings.

For each user, a vector of size N_movies is defined, and the values of the vector are 1 if the user rated the movie before time T, 0 if not. For instance if the user rated movie 3 and 5 the input vector is [0,0,1,0,1].

The goal is to predict the movie the user will rate in the future (between time T and T+delta T). The labels are vector of size N_movies, and if the user rate the movie 4 the labels vector is [0,0,0,1,0].

I'm currently trying to get some initial results based on Fully connected layers, but it seems that it cannot optimize the loss at all. The representation might be too sparse, but it seems that the neural network should be able to learn at least some features.

Is it possible to make this model work, is there any problem with the loss function, or the optimizer?

from __future__ import print_function
import tensorflow as tf
from tensorflow.contrib import rnn
import csv
import bisect
import glob
import re
import numpy as np
import random
import data
import config

cfg = config.Config()
graph_data = data.Graph_data(cfg)

X = tf.placeholder("float", [None, cfg.N_movies])
Y = tf.placeholder("float", [None, cfg.N_movies])

def Dense(x):
    hidden_layer_1 = tf.layers.dense(inputs=x, units=500, activation=tf.nn.relu)
    hidden_layer_2 = tf.layers.dense(inputs=hidden_layer_1, units=50, activation=tf.nn.relu)
    output_layer = tf.layers.dense(inputs = hidden_layer_2, units= cfg.N_movies, activation=tf.nn.softmax)
    return output_layer

logits = Dense(X)
cross_entropy = tf.reduce_sum(- Y * tf.log(logits), 1)
loss_op = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=cfg.learning_rate)
train_op = optimizer.minimize(loss_op)

init = tf.global_variables_initializer()

with tf.Session() as sess:
  sess.run(init)

  for step in range(1, cfg.training_steps+1):
    batch_x, batch_y = graph_data.train_next_batch(cfg.batch_size)
    sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})
    if step % cfg.display_step == 0 or step == 1:
      loss = sess.run(loss_op, feed_dict={X: batch_x,
                                                                  Y: batch_y})
      print("loss = ",loss)


texts: two autoencdoers learned by two similar vectors (each one with its own). Similarity of hidden layer vectors will be the same?

texts: If I will train one autoencoder with one vector only and second autoencoder with second vector only, does it mean if vectors were similar, that the hidden layer vectors of both autoencoders will be similar as well. Autoencoders structure is identical. The quantity of hidden layer neurons is smaller that input. So i want to reduce dimentionality. I understand that it will be overfitting. But for me it looks like you not learn features but just mimick your input function with the smaller dimensional vector while loosing information. So you receive kinda low dimentional approximation of learned vector. So if vector V1 was similar to V2 and I will use V1 to train Autoencoder A1 and V2 to train Autoencoder A2, does it mean that hidden layer vector of A1 and hidden Layer vector of A2 won't be that similar like V1 and V2 but still aproximately quite similar?


texts: Appropiate ways of treating Inf or NA ratio variables for xgboost or random forest

texts:
I have several variables ordered by date, and I would like to take the ratios of growth from one month to the next one. I understand that it is normal to loose the first value.
But my question is, how should i replace the infinite values in ratios where the lag value was zero (by replacement of NA or because it was actually zero)
For example:
example <- data.frame(
  date= c("01-01-2020",
            "01-02-2020","01-03-2020","01-01-2020",
            "01-02-2020","01-03-2020"),
  id = c("a", "a", "a", "b", "b", "b"),
  var1 = c(3L, 4L, 5L, NA, 4L, 5L),
  var2 = c(0L, 3L, 5L, 1L, 3L, 3L)
)
example

example_g <- example %>%
  group_by(id) %>%
  mutate(var1_lag = (var1 - dplyr::lag(var1, 1))/dplyr::lag(var1, 1),
         var2_lag = (var2 - dplyr::lag(var2, 1))/dplyr::lag(var2, 1)) %>%
  ungroup()
example_g

What are good approaches for random forest, neural nets or xgboost?
I would think that for tree based methods simply replacing Inf/-Inf by something like 99999/-99999 would suffice, but i'm not sure if i'm missing something, and if that would work for neural nets.
Has anyone faced a problem like this? what did you do?


texts: How can I create a "trained" dataset for categorizing news articles?

texts:
I am trying to automatically categorize news articles according to their primary topics, i.e. Politics, Entertainment, Sports, Business, Technology, Health, etc.

There are some labeled datasets out there, but ideally I'd like to create my own (for potential commercial usage later on). I'm using python, but an answer clear enough with relation to any language would be sufficient.

So, how would be the best way to go about this task?

My current thoughts are:

Determine the most popular keywords for each category, then associate
each keyword/keyword set with each respective category, and then use
an algorithm to apply a "category" label to a large set of scraped
articled based on the predefined keyword sets.
Another option is to scrape article from specific sections of news
sites where the categories are already specified, and apply them to
each individual constituent article.

After I have the trained dataset, I plan to implement Naive Bayes classification method to automatically categorize future articles.

...

As you can see I have some ideas, but because the web is a large and magical place, I assume someone with previous experience doing something like this may be able to reduce my effort expenditure by guiding me toward the most feasible solution..


texts: General unsupervised learning strategy when using convolutional autoencoder (CAE)

texts:
I am working on implementing an autoencoder for unsupervised learning, and I have some questions about the overall process. From what I was reading here, @rjpg suggests the following general approach:

Slice the NN in half on the last encoder (before the decode starts)
Freeze the the weights of the encoders
Add or concat a new NN in front of the last encoding layer
(post) Train this concat NN with a softmax output layer to classify the digit

I haven't been able to find an example of this using Keras.  I have seen examples in R or H20, but the code is so encapsulated, it is difficult for me to implement a similar solution in Keras.  I have three questions related to how to implement this:

Does this strategy seem reasonable?
How to freeze the weights of the encoders in Keras?
What is meant by adding or concatting a new NN in front of the last encoding layer, or why is this needed?

Thanks!


texts: What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted

texts:
I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is doing a good job in itself not fitting this data. However I want to quantify this 'goodness' with a statistical measuring technique.

I know that R squared is the most popular for regression, however I think that it is not the right approach for me. After trying r squared I am getting very low values because my algorithm is not fitting outliers which r squared does not take into account. rsquared does not know that I have outliers which the algorithm is on purpose not fitting with the rest of the data.

Is there another approach to this? maybe a different algorithm, or some changes I can make to r squared?


texts: Reducing the dimensionality of word embeddings

texts: I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some dimensionality reduction method to scale the word embeddings with 300 dimensions down to 50 dimensions?


texts: Tensorflow - simple multi-layer perceptron not stabilizing around mean of normally distributed y-values

texts:
I'm building an FX trading model where I'm trying to predict the +/- movement of a currency pair 5 minutes into the future.  I've had some promising results adapting the model as a classifier (i.e., buy if the currency pair is expected to increase by more than some threshold, sell if the pair is expected to decrease by some threshold, or do nothing otherwise), but when trying to create even the simplest multi-layer perceptron regression model, the mean of predicted values bounces around the mean of actual y values (as expected, this is very close to zero).

The two images below depict the train predictions (green) versus the actual y_train values (blue) after two different epochs after a number of epochs had already been completed.  Rather than move towards the true y mean in some stable way, the prediction mean flies right past it and will seemingly do this over and over with more epochs.

So far, I've tried:

Using a smaller learning rate
Adding more epochs.  The training set has almost 300k items in it, so the classification model is able to learn in only about 5 epochs, but increasing this to 50 or 100 for regression still doesn't seem to help.
Changing the loss function from mean squared error to mean absolute error
Changing the optimizer from Adam to other options like RMSProp (eventually will predict the same value for all items), Adagrad, Adadelta, Nadam, etc.
Using elu or leaky relu instead of relu.  Also, if I don't add 1 to the y values before training, the model ends up predicting the same value for every item after only a few epochs, which might be an extension of the same problem.
A bias constraint on the output node to keep it close to zero (the true y mean).  Ultimately, it doesn't matter and the model still pushes away from zero.
Batch normalization

This is happening even with a very simple model such as this:

self.model = models.Sequential()
self.model.add(layers.Dense(self.num_layers, activation="relu" input_shape=(self.x_train.shape[1],)))
self.model.add(layers.Dense(1))
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])

Any suggestions as to how I can stabilize the model?


texts: Feature extraction using autoencoder and assigning sub-features to the classes

texts:
I have a dataset with N records and D numerical attributes belonign to C different classes. I use a Stacked Autoencoder for feature extraction for a classification task:
it takes an input vector $x \in \mathbb{R}^d$, and maps it to a hidden representation $y \in \mathbb{R}^{d^{'}}$ ($ d^{'} < d $).

My question is: if we consider outputs of the middlemost layer ( i.e. $y \in \mathbb{R}^{d^{'}}$) as our feature set, for each class of data ($ c_i \in C $),  how can we realize which subset of this feature set has the greatest impact on identifying the class $m_i$.

As an example: for MNIST we have N=60000, D=784, and C=10. An autoencoder with this architecture:

d=784
inp = Input(shape=(d,))
x = Dense(d, activation='relu')(inp)
x = Dense(d//2, activation='relu')(x)
x = Dense(d//8, activation='relu')(x)
y = Dense(d//128, activation='relu')(x)
x = Dense(d//8, activation='relu')(y)
x = Dense(d//2, activation='relu')(decoded)
x = Dense(d, activation='sigmoid')(x)
model = Model(input_img, z)

produces a $y\in \mathbb{R}^{6}$. For example, here we see the outputs of layer y for digit 5 and 9:

  class  y_1    y_2     y_3     y_4     y_5    y_6
  9      1.09   9.59    1.58    8.47    1.14   7.25
  9      2.13   1.34    4.00    8.59    1.53   1.36
  5      7.19   7.52    4.58    5.04    1.09   5.35
  5      9.80   1.55    1.46    5.06    6.49   3.51

If we connect layer y to a softmax dense layer:

out = Dense(num_classes, activation='softmax')(y)
encoded = Model(input_img, out)

the new model encoded give us a good accuracy about 98%. So, if we consider new representation, y, as an efficiently extracted feature vector, is it reasonable to assign a subset of this feature vector to each digit?
Or, what is the right way to relate the vector y to the C different classes?


texts: Latent variable graph in Variational Autoencoder

texts:
I followed this Keras documentation guide about Auto Encoders.
At the end of the documentation there is the graph of the latent variable z:

But I can not understand and how to interpret the plot, and how should the plot change as the hyperparameters of the model change?


texts:
I'm trying to implement the validation curve based on this SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs X and y are shaped (266531, 23) and (266531,).

The curve looks like this:

And my code is:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil

np.random.seed(0)

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.show()

UPDATE

A comment suggested I made X and y identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.


texts:
If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate?

Surely, momentum would catch up its magnutude pretty quickly to any needed value anyway, why to bother scaling it with learning rate?

$$v_{dw} = \beta v_{dw} +(1-\beta)dW$$
$$W = W-\alpha v_{dw}$$

Where $\alpha$ is the learning rate (0.01 etc) and $\beta$ is the momentum coefficient (0.9 etc)

Edit

Thanks for the answer! To put it more plain: momentum controls "how well we retain" the movement, and learning rate is "how fast do we reGain" the movement


texts: Can i use chi square test to remove a particular variable from the model?

texts:
I have 5 variables - $v_{1}$, $v_{2}$, $v_{3}$, $v_{4}$, and $v_{5}$. All are categorical variables.

I conducted a $\chi^{2}$ (chi-square) test to understand the relationship between these variables. I could see that $v_{3}$ and $v_{4}$ have p-values of 0.  I understand that these two variables are dependent on each other. Should I remove one of the variable from further analysis?

What if the same $v_{3}$ and $v_{4}$ are independent of other variables, like $v_{1}$, $v_{2}$, and $v_{5}$?


texts: Decomposable output regression neural network

texts:
For a sub problem I need a network that produces a single continuous number $y$, however during training I have more information available about the output, the output is a weighted sum between a 4 other numbers $y_{1..4}$, the weights are known and constant. All of $y_{1..4}$ are dependent on the input $x$ and I feel like incorporating this knowledge into the network architecture might be beneficial for training. My current idea is to have a number of shared layers, then have 4 individual intermediate layers that all regress to the specific $y_{1..4}$ and then have $y=\sum_{i=1}^4a_iy_i$, with $a_i$ being the weight for $y_i$.

My question is about a good loss function, I'm interested in regressing the mean so using $L_2$ loss makes sense but only doing it at the end doesn't incorporate the prior knowledge. You could add all the $L_2$ losses for the individual $y_i$, they should probably be weighted by how much weight they have in the total sum. However then we don't directly optimize $y$, sometimes the network could be more biased than normal by making a mistake in the same direction for all $y_{1..4}$. My idea is now to do it on the individual $y_i$ but also on $y$, weighing the sum of the $L_2$ of the individual $y_i$ equally to $L_2$ of $y$. The weight between these two parts of the loss function could be different and I guess is a hyperparameter of the model. My question is if there are any papers that have worked with a similar idea to see which of the options work better or if there is another approach to do this. Splitting the value and advantage streames in deep reinforcement learning is what inspired me to look at this.

EDIT: The goal is indeed to get the lowest $L_2$ loss on the final $y$, but I feel like the network should be able to learn better because I have more information with regards to how $y$ is built up. My question is how to best incorporate this additional information, I know I can test some different scenarios but was wondering if there is any literature about similar approaches before implementing things that end up inferior.


texts: Gan in Keras - You must feed a value for placeholder tensor

texts:
i have a problem with the training of my GAN.

I get the following error message:

Traceback (most recent call last):
File "/home/user/PycharmProjects/pythonTest/main.py", line 172, in
<module>
train(1)
File "/home/user/PycharmProjects/pythonTest/main.py", line 145, in train
g_loss = gan.train_on_batch([[noise], word_as_sequence, [len(word)]],
[valid_label])
File "/home/user/.local/lib/python3.6/site-
packages/keras/engine/training.py", line 1217, in train_on_batch
outputs = self.train_function(ins)
File "/home/user/.local/lib/python3.6/site-
packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "/home/user/.local/lib/python3.6/site-
packages/keras/backend/tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "/home/user/.local/lib/python3.6/site-
packages/tensorflow/python/client/session.py", line 1458, in __call__
run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed
a value for placeholder tensor 'ocr_input_img' with dtype float and shape
[?,128,512,1]
 [[{{node ocr_input_img}}]]

my Code looks like the following:

...
img = self.generator([noise, word_embedded])

self.discriminator.trainable = False
self.text_recognition.trainable = False

valid = self.discriminator(img)

ctc = self.text_recognition([img, labels, label_length])

combined = Model([noise, word_sequence, labels, label_length], [ctc])

I cant understand what the problem with the text recognition is. The Discriminator have the same input shape  for the image as the text recognition. When i try to run it without the text_recognition and only with the Discriminator it works.

The Part to start Gan training:

g_loss = gan.train_on_batch([[noise], word_as_sequence, [len(word)]], [valid_label])

The Output of the Generator has the shape=(?, 128, 512, 1) dtype=float32


texts: Tradeoffs between Storm and Hadoop (MapReduce)

texts:
Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing via MapReduce in a Hadoop Cluster) is a batch processing system, and Storm is a real-time processing system.

I have worked a bit with Hadoop Eco System, but I haven't worked with Storm. After looking through a lot of presentations and articles, I still haven't been able to find a satisfactory and comprehensive answer.

Note: The term tradeoff here is not meant to compare to similar things. It is meant to represent the consequences of getting results real-time that are absent from a batch processing system.


texts: why do transformers mask at every layer instead of just at the input layer?

texts: working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one mask at the first layers of encoder and decoder?


texts: I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?


texts: Join the List and the Data frame for matching list elements in Python

texts:
I have a list =[a,b]  and

Dataframe as below

If the text in the 'Text column' matches  with List, then need to add additional column matchedItem

So output should be

Please suggest on this..How to get this output after the join (with exact match and partial match)..

also alternatively output in another dataframe(partial match)

Code

To resolve

I am trying something like this one

df["kpiName"] = getKPIName(df1.text.str)
kp1=['a','b']

def getKPIName (str):
    return [kp1.index(x) for x in kp1 if str in x]

But this results in
error
TypeError: 'in ' requires string as left operand, not StringMethods


texts: Trying to input a list of strings into my LSTM model

texts:
I'm training a model for dialogue act classification. I'm trying to write it so that I can enter a singular list of strings and receive a prediction for each of the strings. I've come to understand that LSTM models require 3D inputs, but my data that I'm using as input is just a list of strings. The structure of my input is simply a numpy list of strings. I haven't been able to shape my data, due to the fact that it's not always divisible, reshaping has been difficult. How can I input a list of strings to model.predict() to generate predictions?
I've attached the model's creation below.

# Parameters
vocabulary_size = metadata['vocabulary_size']
num_labels = 41
max_utterance_len = 106
batch_size = 100
hidden_layer = 128
learning_rate = 0.001
num_epoch = 10
model_name = model_name + " -" + \
             " Epochs=" + str(num_epoch) + \
             " Hidden Layers=" + str(hidden_layer)

model = Sequential()
model.add(LSTM(hidden_layer, input_shape=(max_utterance_len, num_labels), return_sequences=True, kernel_initializer='random_uniform', recurrent_initializer='glorot_uniform'))
model.add(TimeDistributed(Dense(hidden_layer, input_shape=(max_utterance_len, hidden_layer))))
model.add(GlobalMaxPooling1D())
model.add(Dense(num_labels, activation='softmax'))

optimizer = RMSprop(lr=learning_rate, decay=0.001)
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
print(model.summary())

test_predictions = batch_prediction(model, test_data, test_x, test_y, metadata, batch_size, verbose=False)
val_predictions = batch_prediction(model, val_data, val_x, val_y, metadata, batch_size, verbose=False)


texts: LabelEncoder with a Multi-Layer Perceptron?

texts:
So we're working on a machine learning project at work and it's the first time I'm working with an actual team on this. I got pretty good results with a model that uses the following SKLearn pipeline:

Data -> LabelEncoder -> MinMaxScaler (between 0-1) -> PCA (I go from 130 columns to 50 prime components that cover the variance) -> MLPRegressor

One of my colleagues mentioned that I shouldn't normally use LabelEncoder to encode training data, as it's meant for encoding the target variable. I did some research and now and I understand why

  LabelEncoder only is not a good choice, since it brings in a natural
  ordering for different classes.

Then, however, my colleague mentioned that in this case it shouldn't make much of a difference as I'm using a neural net (~MLPRegressor). My question is - (if he's right - is he?) why? He basically commended me saying this usually would be a bad idea but in this case it should work.

I will try to move to one-hot encoding (currently I'm only stuck with it because I run out of memory while doing PCA on that many columns, but that's another question and I'll do some research on that separately), but for now I'd like to know if using this kind of encoding can result in inaccurately good results (I'm having an r^2 score of around 0.9 and my boss literally won't believe I achieved a result that good haha).


texts: Would keeping all punctuation make any sense in word2vec?

texts:
I am trying to learn how word2vec works to get to more complicated stuff like LSTMs.
Because I will use the same training data (so with the same vocabulary) and I want to predict punctuation too, I decided to keep it.

Punctuation is obviously limited to .!,?, anything else is discarded (The sentence itself to avoid loss of meaning). Also, every symbol is converted into a string-like representation to avoid any kind of problems with encoding.

Now, does it make sense to keep punctuation? Should I discard it completely (if I do discard it, how can I generate punctuation for seq2seq model later?)?


texts: Orange: Group samples by a "splitting" feature for cross-validation?

texts:
I need to split my datasets based on my own feature column in order to hold together certain data rows (e.g. from one patient, or compound) for cross-validation (CV).

In Orange 3.11, Test&Score contains "Cross validation by feature", but it is disabled. The widget documentation contains a screenshot still without it.

Is this a feature that is currently under development? Or do I have to define the relevant column differently to enable the option? (It's "meta" right now.) Or is there a different way to achieve this particular CV, i.e. use a pre-defined column for a "Leave-group-out" CV?


texts: Why doesn't the binary classification log loss formula make it explicit that natural log is being used?

texts:
I'm completing a DataCamp course where we are introduced to the log loss formula for binary classification:

Two scenarios are given to show how the formula is used. One with p=0.1 and one with p=0.5.  The answers the instructor displayed were 2.3 and .69, respectively.  However, using a calculator, the answers for log(0.1) and log(0.5) are -1 and -0.30, respectively. I later tried using natural log instead and got the same answers as the instructor, except negative.  Specifically, the calculator returned -2.3 for ln(0.1) and -0.69 for ln(0.5).

Is it common in math for log to implied to be "ln" or "log e" without stating it explicitly in the formula?  Also, is there something about the log loss binary classification formula that suggest that the absolute value of the result should be taken?


texts: is it possible get a overfit underfit comparation between models, with this chart? (homework)

texts:
I am trying to interpret this chart.
I am not sure how to interpret this, because, I think that the fact of the for examples LGBM Validation error, is wide and similar to train boxplot, there arent problem of overfitting, but when I see another type of charts of the execution of LGBM, I can see that really the LGBM is overfitted, so really I don't know how to interpret this of the correct way.

But I don't know how could interpret beyond this:
LightGBM is maybe the best option because it is faster and finally you can get enough accuracy with that, and in comparison with the other two, bagging have less overfit because of the differences between the error is less.
Any idea?
Thanks


texts:
When I evaluate the model I seem to be getting a decent RMSE score but when I try to actually see the predictions when I call the model all my values are the same.

 xdata = xgboost.DMatrix(X_train, y_train, feature_names=all_vars)
 xdata_val = xgboost.DMatrix(X_valid, y_valid, feature_names=all_vars)
 xgb_parms['seed'] = random.randint(0,1e9)
 model = xgboost.train(xgb_parms, xdata)
 model.eval(xdata_val)
 ypred = model.predict(xdata_val)

I believe the error is on my last step, what am I doing wrong?


texts: Books about the "Science" in Data Science?

texts:
What are the books about the science and mathematics behind data science? It feels like so many "data science" books are programming tutorials and don't touch things like data generating processes and statistical inference. I can already code, what I am weak on is the math/stats/theory behind what I am doing.

If I am ready to burn $1000 on books (so around 10 books... sigh), what could I buy?

Examples: Agresti's Categorical Data Analysis, Linear Mixed Models for Longitudinal Data, etc... etc...


texts: Reinforcement Learning in continuing tasks (non-episodic)

texts:
I'm looking for an algorithm for Deep Reinforcement Learning in non-episodic or continuing tasks. To be explicit, I'm looking for an algorithm that allows the agent to learn online without separate the experience into episodes with finite rewards.
Until now, I've found little information, and the closest article that I have read is this website, where the use of mean reward is explained based on Sutton's book. The explanation makes sense to me, but I'm not completely sure, and I'm relatively new on the field. I've read about R-Learning in the Sutton book; however, I've found little information about this and how to apply it. Also, they give little information about it, and it is not included in the second edition of the book.
Originally I wanted to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing tasks; however, I'm not sure if I should modify my model (from an episodic setup).
If you could recommend me any methodology to work on a continuing problem, I would be very thankful. I've heard about applications of RL in stock predictions.
I really appreciate any help you can provide.
Tomás


texts: How to resolve IndexError while doing Monte Carlo for 1000 runs?

texts:
Below code runs without any problem, however when I run the same code using Monte Carlo Analysis for 1000 runs, it gives IndexError. Can someone explain why this happens. Thanks
X = df1.drop("Gender", axis = 1)
y = df1.Gender
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2)
nb = CategoricalNB()
nb.fit(X_train,y_train)
nb_pred = nb.predict(X_test)
nb_accuracy = accuracy_score(y_test,nb_pred)
nb_accuracy

output: 0.6279486413854882

X = df1.drop("Gender", axis = 1)
y = df1.Gender
for i in range(1000):
    X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2, random_state = i)
    #CategoricalNB Naive Bayes Model
    nb = CategoricalNB()
    nb.fit(X_train,y_train)
    nb_pred = nb.predict(X_test)
    nb_accuracy = accuracy_score(y_test,nb_pred)
print("CategoricalNB Naive Bayes Accuracy:{}".format(nb_accuracy))

IndexError: index 74 is out of bounds for axis 1 with size 74


texts:
I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example:

EPISODES = 91
GAMMA = 0.2
EPSILON_DECAY = 0.999
MIN_EPSILON = 0.01
MAX_EPSILON = 1

My questions are:

Is it correct if epsilon doesn't reach MIN_EPSILON?
Is there something wrong with the reward - the reward is not higher and higher but it is behaving otherwise - it decreases in time?


texts: Why does my minimal CNN example show strongly fluctuating validation loss?

texts:
I'm fairly new at working with neural networks and think I am making some basic mistake. I am trying to assign simulated images to 5 classes to test, what (if any) networks are helpful for a large data problem we have in our group. I am training the ResNet50 CNN included in Keras on 40000 (256 by 256 pixel greyscale) images. While training loss is improving quickly within the first epoch, validation loss fluctuates wildly in (to me) fairly random manner.

I am trying to use as many high level functions as possible and have ended up with the following code:

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.resnet50 import ResNet50
from keras.models import Model
from keras.layers import GlobalAveragePooling2D, Dense

inputShape = (256, 256,1)
targetSize = (256, 256  )
batchSize = 128

# First, load resnet
base_model = ResNet50(include_top =False,
                weights =None,
                input_shape=inputShape)
x = base_model.output

# modified based on the article https://github.com/priya-dwivedi/
# Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

x = GlobalAveragePooling2D()(x)
predictions = Dense(5, activation= 'softmax')(x)
model = Model(inputs = base_model.input, outputs = predictions)

# Compile model, might want to change loss and metrics
model.compile(loss='categorical_crossentropy',
        optimizer='adam',
        metrics = ["acc"]
        )

# Define the generators to read the data. Training generator performs data argumentation as well
train_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,
        )

train_generator = train_datagen.flow_from_directory(
        'trainingData/FolderBasedTrain256',
        target_size=targetSize,
        color_mode = 'grayscale',
        batch_size=batchSize,
        save_to_dir = 'trainingData/argumentedImages',
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        'trainingData/FolderBasedValidation256',
        target_size=targetSize,
        batch_size=batchSize,
        color_mode = 'grayscale',
        class_mode='categorical')

# Fit the Modell, saving the history for later plotting
history = model.fit_generator(
        train_generator,
        epochs=20,
        validation_data=validation_generator,
        #steps_per_epoch = 62,
        steps_per_epoch = 31,
        #callbacks=[checkpointer],
        validation_steps=9)

I can always create more images or train longer, but to me, this looks as if something fundamental went wrong somewhere. I would be very gratefull for any and all ideas. Thank you!

EDIT: I was told to stress that validation and training set are both created by exaclty the same simulation routine, so they should be relativly easy to classify

EDIT2: Found the error! My batch size did not fit with the amount of data and the steps by epoch, resulting in the CNN not seeing all the training data. Now everything converges nicely and I can evaluate the choice of modell. Thanks to all contributers


texts: Is random forest better suited than neural network in my scenario?

texts:
I am learning Neural network and facing this scenario.

Say in my input, X has 20-30 features, and Y is a classification (e.g. 1,2,3,4, 5).

What I need is to find the features that contribute most to the output Y (i.e. the most important features).

Base on my limited understanding about neural network and machine learning, random forest is the model that can give out this information. For neural network, I am not really sure it can achieve what I want.

Is it possible that someone give more ideas?


texts: Problem importing CNTK in Azure jupyter notebook

texts:
Vineeth Sai indicated in this that with the following code:

pip install cntk

the problem is solved. However, I am getting the error shown in attached image:


texts:
how is countvectorizer used in real production environment?

do you keep training the model with new features/vocabulary everyday and save the vocab into a flat file and reload them up on the next day?

do you use a pipeline to streamline the process?

what is the best practice?

we are going to implement a combination of countvectorizer,tfidf and some machine learning algo in production system soon and any tips or practical experiences will be appreciated.

thanks!


texts: What is effect when I set up some self defined predisctor variables?

texts:
I run a random forest model on a dataset to assess the credit risk. I found the periodic income and periodic liability were important features in my estimator. Then I try to set a ratio metric which was periodic income/periodic liability which measured the relevant scale on these two variables. Then I found this ratio was also important but the model performance like f1 score, confusion matrix were not improved.

In my case here, does it mean the ratio was redundant and took no extra effect in enhancing the prediction? In math, periodic income , periodic liability and ratio represented different quantitative meaning.

Can I say in machine learning scope, if Variable A and B were all important from the fitting result, then the ratio of them or even any other algebraic operation on them can be expected as important as well but no extra contribution on the model fitting?

Could someone give me advice and thanks heaps.


texts: Convolutional Neural Network overfitting

texts:
I built a CNN to learn to classify EEG data (only about 4000 training examples, 2 classes, 50-50 class balance). Each training example is 64x512, with 5 channels each

Ive tried to keep the network as simple/small as possible for testing:

ConvLayer (4 filters)
MaxPool
Dropout 50%
Fully connected (50 neurons)
Dropout 50%
Softmax

Im also using weight decay (L2 reg, lambda = 0.001)

The problem is no matter how I play with the filter parameters (size, stride, number) my network keeps overfitting. It fits the training data 100%, but no matter what I do I can't get the test accuracy over 65%.

Why is such a small network overfitting? I thought it was a sample size issue, but I've read a number of research papers on EEG and BCI and they occasionally have even smaller sample sizes than I do

What else can be done to regularize a CNN?


texts:
I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars)

In[10]

lines = spark.read.text("Dataset.csv").rdd
print(lines.take(10))

Out[10]

[Row(value='0,0,0,290'), Row(value='1,1,1,112'), Row(value='2,2,2,87.8'), Row(value='3,3,3,69.7'), Row(value='4,4,4,65.7'), Row(value='5,5,5,62'), Row(value='6,6,6,61.6'), Row(value='7,7,7,60.7'), Row(value='8,8,8,57.7'), Row(value='9,9,9,56.2')]

In[10]

# Need to convert p[1] from str to int
parts = lines.map(lambda row: row.value.split(","))

print(parts.take(2))

Out[11]

    [['0', '0', '0', '290'], ['1', '1', '1', '112']]

In[12]

# RDD mapped as int and float from Dataset

ratingsRDD = parts.map(lambda p: Row(userId=int(p[1]),repoId=int(p[2]),repoCount=float(p[3])))
ratings = spark.createDataFrame(ratingsRDD)
print(ratings.head(10))

Out[12]

    [Row(repoCount=290.0, repoId=0, userId=0), Row(repoCount=112.0, repoId=1, userId=1), Row(repoCount=87.8, repoId=2, userId=2), Row(repoCount=69.7, repoId=3, userId=3), Row(repoCount=65.7, repoId=4, userId=4), Row(repoCount=62.0, repoId=5, userId=5), Row(repoCount=61.6, repoId=6, userId=6), Row(repoCount=60.7, repoId=7, userId=7), Row(repoCount=57.7, repoId=8, userId=8), Row(repoCount=56.2, repoId=9, userId=9)]

In[13]

(training, test) = ratings.randomSplit([0.8, 0.2])

In[14]:

# Build the recommendation model using ALS on the training data
# Cold start strategy is set to '"drop" to make sure there is no NaN evaluation metrics which would result in error.
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="repoId", ratingCol="repoCount"
        ,coldStartStrategy="drop") #Cold-start is set to DROP
model = als.fit(training)

In[15]

#Evaluate the model by computing the RMSE on the test data
predictions = model.transform(test)
type(predictions)

predictions.show(3)

Out[15]

    +---------+------+------+----------+
    |repoCount|repoId|userId|prediction|
    +---------+------+------+----------+
    +---------+------+------+----------+

My model is giving NULL values. Is there a problem in my dataset, or wrong assumptions on Training.

Note that my ratingCol in ALS is Number of Stars which is an explicit rating and not implicit rating.


texts: Training a classifier when some of the features are unknown

texts:
I am training a classifier in Matlab with a dataset that I created.
Unfortunately some of the features in the dataset were not recorded.

I currently have the unknown features set as -99999.

So, for example my dataset looks something like this:

class1: 10 1 12 -99999 6 8
class1: 11 2 13 7 6 10
...
class2: 5 -99999 4 3 2 -99999
class2: -99999 16 4 3 1 8
...
class3: 18 2 11 22 7 5
class3: 19 1 9 25 7 5
...

and so on, where the -99999 are the places where the features werent able to be measured. In this case, each class has 6 features.

I don't want to bias my classifier with the unknown features so I thought it would be a good idea to set the unknowns to -99999 so it would be way out of the range of normal features.

I tested the classifier with the -99999's and it was 78% accurate.
Then I changed the -99999 to 0's and tested the classifier again, this time it was 91% accurate.

So my question is, what is a general rule for training a classifier when some of the features were not recorded? Was I right to assume setting the unknowns to a very high negative value? But why was it more accurate when I set the unknowns to 0s?

Thanks for reading!


texts: Linear approximation of the given equation with python3

texts:
I was given a set of raw datum and have to model it by means of some machine learning techniques. After some research, I decided to do with the method of linear approximation.

Description of the equation.

z - depth (meters)
T(z) - temperature at the depth z
T(zᵢ) - temperature at the depth zᵢ
T₀ - temperature at the surface (It is constant and known)
K - coefficient of geothermal gradient (How the temperature changes with respect to the depth)
Mᵢ - flow rate of the liquid at the depth zᵢ

As it is shown from the equation we can find the temperature of the liquid in the any depth of the well bore.

I have list of depths, temperature and the flow rate of the liquid. I have to model an equation according to these datum by means of python3 and find K and c. Currently I use matplotlib library for such type of calculations.


texts: Clustering of longitudinal user generated data; determine at what point in time does the user "become" the final clustering outcome

texts:
I analyse a lot of telephone call log data sets (akin to user generated data) and I use k-means clustering a lot to look at the types of callers that exist in the data set. The callers are clustered on three main usage characteristics:- the number of calls each caller makes, the average duration of calls and the standard deviation of call duration. You can use this to infer a lot about their interaction with the telephone service.
Anyway, say if I have call log data set that spans over several years and I cluster that data in it's entirety, which would capture the behaviour of all callers in the data set. I usually find five robust types of callers within the data (using k-means) and I label the five caller types with appropriate names which describe their behaviour.
I would like to find out, at what point in the timeline of the caller's "tenure" or existence within the data, does their final caller type become consolidated or set in stone. For instance, If I have a caller type who rings hundreds of times over several years, I may label that caller a "regular" caller; is it possible to determine at what point within the data (or over the caller's tenure/period of time using the telephone service) this caller's behaviour becomes consistent of that to a "Regular" caller? This could be measured in the number of calls, number of days spent with the service etc.
I realise that it might sound like a complicated question but if there is any types of analytic or machine learning techniques which would help me measure this, then I would really appreciate any suggestions :) many thanks!


texts:
I was going through some material and the term "localized filters" was used, can someone please explain what it means and what are the advantages of having a localized filter.

EDIT-1
My question is in the context of the word used in NIPS 2017 tutorial on Geometric Deep Learning on Graphs and Manifolds at 7:23 it is said that "In many successful instances of CNN models, these filters are localized in space. And the intuition why they have to be localized can be understood from the geometric point of view, that if I apply my filters to the inputs that are slightly deformed then many of the filter responses are going to be stable." I am not able to understand the last line.
For this purpose, I have read stuff related to localization of rings and related things in Algebra but can't relate from there as well.


texts: Controlling the sampling from Variational AutoEncoder (VAE)

texts:
Suppose a Variational Autoencoder (VAE) is trained with mnist data. To sample, one draws from normal distribution. My question is: suppose I am interested in generating only 1s and no other digits. How can do I do that? do I sample until I generate a 1 and then I keep sampling from the neighborhood of that point? or is there a more controlled way to tell the VAE what digit to generate? Thanks
Edit*
The ideal encoder would take any input and generate a sample coming from $\mathcal{N}(0,1)$. If the input is 1 or 2 or 3, the samplings will all come from $\mathcal{N}(0,1)$, so, how does the decoder know if the input was 1, 2, or 3?


texts: In PCA, every principal component a eigen vector?

texts:
In pca, we convert predictors into principal components for dimensionality reduction. My assumption is every principal component is a eigen vector with eigen value as sum of squared distance of orthogonal projection of data points or vectors on it..
Is my assumption correct. If not pls clarify


texts: How to automatically segment multidimensional data?

texts:
How to partition the time-series multidimensional data in the figure below into segments using an unsupervised algorithm, so that the information within the same segment remains consistent, while the information in adjacent segments differs? Note that the algorithm should be adaptive because we do not know how many segments each time-series data should be divided into.
The data can be found here.


texts: Multiple Object recognition in image using deep Learning

texts: I am working on recognizing object classes in images using neuronal nets so I could make classifiers for cats, dogs... using Imagenet and some Conv nets famous architectures but my problems is if I had an image that contain a human and a cat and a car how can I detect and recognize 3 of them?


texts: Transformer-based architectures for regression tasks

texts: As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving regression tasks? Could one build a regressive auto-encoder for example? How would normalization fit into this (as LayerNorm destroys some of the information from the input)?


texts: Is there a way to set a different activation function for each hidden unit in one layer in keras?

texts: I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?


texts: How many features do you generally use for your ML Model?

texts:
I am working on a certain kaggle competition and users there say that they are using >5000 features and training a XGBoost or Random Forest on it.

The mentioned post is here: https://www.kaggle.com/c/walmart-recruiting-trip-type-classification/forums/t/17258/feature-counts

I tried doing so myself after exploding the feature space and creating interaction features, but it takes forever to run on my 16GB macbook.
For example: (96000,1000) data took around 5 hrs to finish to train an RF with n_estimators=1000,max_depth=40, cv=3.

Tuning such a model would take weeks and it only uses 1000 predictors....

Now here are some of my doubts:
1. Is my python scikit environment properly loaded. I use anaconda distribution. Are these running times normal?


texts: Multivariate Linear Regression with exponential trend-line

texts:
The dataset that I'm working with has 2 independent variables (qty, volume) and 1 dependent variable (cost).  When I plot individual X with Y, it turns out qty vs cost gives an exponential decay trend line while volume vs cost gives a linear relationship.
I'm trying to come up with a linear model (since it's beginner-friendly) to predict the cost when a new input, volume set is given.  I tried to excel and while the statistic output looks reasonable, the predicted values are off quite a lot.  I read up on some articles and there is something regarding log transformation so that both qty and volume have a linear relationship with cost.  I'm not sure if log transformation applicable in this situation.  What would be the best way to approach this problem?
I plan to use R instead of excel next.
Here are some outputs of what I've done

Note: Thickness, width, and height essentially become volume so it doesn't affect the data much


texts: How gradients are flown back to Network in siamese architecture? How weights of all CNN models are same even when using different models

texts:
TL;DR : Intuition behind the gradient flow in Siamese Network? How can 3 models share same weights? And if 1 model is used, hpw Gradients are updated from 3 different paths?
I am trying to build to a Siamese Network and as far as I can know, if I have to build a Triplet Loss based Siamese, I have to use 3 different networks. So for simplicity, let us say that my architecture is something like: Please correct the architecture if wrong
    I1 = Input(shape=image_shape)
    I2 = Input(shape=image_shape)
    I3 = Input(shape=image_shape)

    res_m_1 = ResNet50(include_top=False, weights='imagenet', input_tensor=I1, pooling='avg')
    res_m_2 = ResNet50(include_top=False, weights='imagenet', input_tensor=I2, pooling='avg')
    res_m_3 = ResNet50(include_top=False, weights='imagenet', input_tensor=I3, pooling='avg')

    x1 = res_m_1.output
    x2 = res_m_2.output
    x3 = res_m_3.output

    # x = Flatten()(x) or use this one if not using any pooling layer

    ##### ------- ---------------------------- --------- ########

    'NEED HELP AFTER THIS ONE; HOW TO BUILD ARCHITECTURE'

    ########### ------------------------------------ ###########

    siamese_model = Model(inputs=[I1,I2], outputs=final_output)

    siamese_model.compile(loss=some_triplet_loss,optimizer=Adam(),metrics['acc'])

    siamese_model.fit_generator(train_gen,steps_per_epoch=1000,epochs=10,validation_data=validation_data)

My Understanding and Question:: If there are 3 networks in the architecture, how come they can produce the output with the same weights? How come these networks are sharing weights?
Also, Let us suppose it is just one network (unable to assume, how, please help), then at the first epoch, it'll give outputs with default weights (if used ImageNet). But when the gradients flow back to the network, how are these updated? Because There are 3 different paths going from the same model and how the gradients will flow back to theese paths? Parallelly is not possible (I can't think how) and if sequentially, how that one either because outputs were provided sequentially but gradients can't flow back that way?


texts: What model should I use for multiple time series input

texts:
I want to predict bacteria plate count in the water from time series(around 10000 values in a row) of water temperature on a one minute granularity, and other daily climate data including min and max temperature, rainfall, solar exposure, day_of_the_week etc for a sequence of 20 days each before the sample was collected for test. There are 700 different locations in the building.

My initial approach was to join all time series of the same location into one row/one series. I used RNN LSTM but the testing accuracy is not so good(65%-75%). I guess the reason could be that joining different time series(eg. water temperature and rainfall) may have led to gradient explosion due to the contrasting nature of data, thus I tried gradient clipping and there was no improvement in testing accuracy. I understand that in general RNN is good for time series data and CNN is good for image preprocessing. But my case is slightly different as I'm having multiple time series joined together to form one time series. I'm wondering if CNN or GRU would be a better model to use.

The data looks like this:

     x0  x1    x2   x3   x4   x5   ... x10000 Date       max_t1...max_t20 min_t1...min_t20 rf1... rf20 sol1...sol20 d_wk1... d_wk20
1    40 31.05  25.5 25.5 25.5 25   ...  33    2019-01-01 26.2  ...        20.2  ...         0 ...      32.4...       4 ...
2    35  35.75 36.5 36.5 36.5 36.5 ...  29    2019-01-03 24.8. ...        18.4  ...         0 ...      28.8          6 ...
⋮     ⋮   ⋮      ⋮    ⋮    ⋮     ⋮          ⋮

where

max_t1, ..., max_t20 represent max temperature from day1 to day20(Date day);

min_t1, ..., min_t20 represent min temperature from day1 to day20(Date day);

rf1, ..., rf20 represent rainfall from day1 to day20(Date day);

sol1, ..., sol20 represent solar exposure from day1 to day20(Date day).

d_wk1, ..., d_wk20 represent which day it was of the week from day1 to day20(Date day)

These are all the features beside water temperature data(so there are around 100 new columns in total).

Update Of Question:

I have checked many CNN case studies online but very few are on non-image data. I fit the data using CNN on Keras but the accuracy level is very low(<60%). What could go wrong? Is there anything I can do? A friend suggested XGBoost- is there a way to apply the algorithm to my data? Any idea is appreciated.

So far I've used something like this:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,1:10170], merge[['Result_cat','Result_cat1']].values, test_size=0.2)

import numpy as np
import keras
import tensorflow
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.optimizers import SGD
from tensorflow.python.keras.optimizer_v2.adam import Adam

model = Sequential()

model.add(Dense(1000, input_shape=(10167,)))
model.add(Activation("relu"))
model.add(Dense(512, activation='softmax'))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))

model.fit(X_train, y_train, batch_size=10000, epochs=1000)
score = model.evaluate(X_test, y_test, batch_size=10000)


texts: Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, random search etc. that could allow faster discovering of best parameters)? I heard of hyperopt but seems there are some problems, and not sure it can train traditional machine learning algorithms?


texts: Threshold to consider new feature as a new finding to a model?

texts:
I am working on binary classification problem with 5K records and 60 features.

Through feature selection, I narrowed it down to 14 features.

In existing literature, I see that there are well-known 5 features.

I started my project with an aim to find new feature that can help improve the predictive power of the model

However, I see that with well known features (reported in literature), it produces an AUC of 84-85 and having all my 14 features decreases it to 82-83.

So I tried manual add and drop and found out that if I add only one feature (let's say magic feature), it increases the AUC to 85-86.

I see that there is a difference of 1 point in AUC.

1) Is it even useful to be happy that this adds some info to the model?

2) Or me looking at AUC is not the right way to measure model performance?

3) Does it mean the other new features (9 out of 14) that I selected based on different feature selection/ genetic algorithm aren't that useful? Because my genetic algorithm returned 14 features, so I was assuming that was the best subset but still through my previous experiments I know that model had better performance when it had 5 features. Any suggestions here? What can I do?

4) I am currently using train and test split as my training and testing data. I applied 10 fold cv to my data. Should I be doing anything different here?

5) If I add around 16-17 features, I see the AUC is increased to 87 but this can't be over fitting right? Because if it's overfitting, shouldn't I be seeing the AUC as97-100or just100? I know we haveoccam razor's principleto keep the model parsimonius but in this case, just having 16-17 features in model is not too complex or heavy. Am I right? Because it's increasing theAUC`. Any suggestions on this?


texts: Very simple real-valued time-series dataset for RNN prototyping

texts:
Is there a simple real-valued time-series dataset on which a vanilla RNN model can be trained. With "very simple" I mean only two to four real-valued inputs per time step and a single real-valued output per time-step.

Background: I am doing research in the intersection of machine learning and formal methods. To test a new technique for formally verifying RNNs, we need to start with a quite simple setup.

Thanks!


texts: How pairplot is constructed? Based on what rule? Why people use it?

texts: I am curious how pairplot in seaborn library is constructed. I have seen similar plot, such as Pearson Correlation plot. What is the difference between these two plots? What is exact the use case of them?


texts: How to build a predictive model on Time series data

texts:
I have a time series data containing hotel occupancy rates from January 2014 to December 2017. I have run a forecasting model (Holtswinter in R). Now, I want to predict the significant external factors driving this occupancy rate. I have considered weather data (Temperature, Visibility etc.,) and Google Trends data (based on certain key words for trend tracking) as Independent variables and Occupancy rate as Dependent variable. This is clearly a non-linear regression model. What is the best algorithm to predict the significant variables impacting occupancy rate? Please refer to the sample data in the image.

Any insights on how to proceed with the model and/or any code snippets in R is/are surely appreciated. Thank you!


texts:
I am about to start a job in which I will be working with large datasets and will be expected to find trends, etc...  I have found lots of resources on where to learn ML and other hard skills and feel that I am (semi) competent on this end.

I am interested in knowing if there are specific soft skills that are helpful as a data scientist.  What are things you wish you knew starting out?

While Kaggle is very useful when learning, it also presents clear objectives.  How do you handle being given a dataset, but no clear objective?

Let me know if this is too broad, I can think of more specific questions.


texts: The affect of bootstrap on Isolation Forest

texts:
I've been using isolation forest for anomaly detection, and reviewing its parameters at scikit-learn (link). Looking at "bootstrap", I'm not quite clear what using bootstrap would cause. For supervised learning, this should reduce overfitting, but I'm not clear what the effect on anomaly detection should be.
I think it would require the trees to achieve more "consensus" about what the anomaly is, therefore, reducing the effect of any single feature. I.e, an anomalous observation would probably need to be anomalous consistently and over a number of features (?).
Is this a correct interpretation of this parameter?


texts: spark ml StringIndexer vs OneHotEncoder, when to use which?

texts:
Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder.
The OneHotEncoder docs say

For string type input data, it is common to encode categorical features using StringIndexer first.

In what situations would I want to take the extra step of transforming StringIndex'ed output to one-hot encoded features?


texts: How to define observation space for a custom environment in OpenAI gym?

texts:
I am a newbie in OpenAI gym. I need help to define the observation space for a sensor network environment. Suppose, there are N sensor nodes in a network. Each node has three features. So, I have total N*3 values that represent the state of the network. How can I define the observation space for this sensor network? I looked into the definition of OpenAI gym observation space (i.e. using box, tuple, dictionary, etc) but I am not sure which one will be applicable for this environment and how?
Thanks in advance.


texts: Why do you need to use group lasso with categorical variables?

texts:
From what I've read you should you use group lasso to either discard the dummy encoded variables (of the category) or use all of them. If you use normal lasso then some of the variables in the group can be discarded (set to zero) and some might not, but why is this an issue?

Let's say we have a categorical variable with 3 levels (A, B, C) and we dummy encode it to get columns A, B (C when A=B=0). Now if we, with normal lasso, only keep A, shouldn't the interpretation then be that when A=1 we get A and when it is 0 we get either B or C, where it doesn't matter that much which one (B or c) it is. What am I missing?


texts: visualize sampling distribution of a covariance matrix

texts: I am looking to show how covariance matrices vary over samples. Any suggestions or ideas for how it could be done graphically? I want to do this in a non-parametric sense without assuming particular distributions for variables.


texts:
I've got a collection of yearly data (one value per year per category), and I'd like to find series that are most similar to one another.

Example data is here.

I don't know much about data science, but it seems like cosine similarity might be the way to go? If so, how do I account for the nil values in the datasets?